California’s vulnerable water system faces many threats

On a February morning in 2021, a water treatment plant operator in Oldsmar, Fla., noticed something unusual: An unidentified user had remotely accessed the plant’s computer system and was moving the mouse around the screen.

The operator watched as the intruder clicked into various software programs before landing on a function that controls the amount of sodium hydroxide, or lye, in the plant’s water system. The hacker then increased the amount of lye — a potentially dangerous substance used to control acidity — from 100 parts per million to 11,100 parts per million.

The plant operator reversed the change almost immediately, and officials said there was never any threat to public safety. But the incident has highlighted the threats facing major drinking water systems across the country.

“Water systems, like other public utility systems, are part of the nation’s critical infrastructure and can be vulnerable targets when someone desires to adversely affect public safety,” Sheriff Bob Gualtieri of Pinellas County, Fla., said at the time.

In California, where epic Sierra Nevada snowpack and “the Big Melt” have substantially increased the stakes for reservoir managers, officials say they’re taking steps to protect the state’s water systems from hackers, terrorist attacks and natural disasters, such as the flooding that temporarily severed the Los Angeles Aqueduct — the city’s water lifeline to the Owens Valley.

But experts say the challenges are numerous. Many of the systems in California and nationwide are still operating with outdated software, poor passwords, aging infrastructure and other weaknesses that could leave them at risk.

“We’ve seen a steady rise in both the prevalence and the impact of cyberintrusions, as well as an extraordinary increase in ransomware attacks, which have become more destructive and more expensive,” said Joe Oregon, chief of cybersecurity for Region 9 of the federal Cybersecurity & Infrastructure Security Agency.

Andrew Reddie, an assistant professor of practice in cybersecurity at UC Berkeley’s School of Information, said much of the problem is “driven by the fact that the infrastructure is really, really old, and ultimately predates the era that we find ourselves in now, where we actually bake cybersecurity into these … systems by design.”

“You can point to any number of critical infrastructure, including things like dams and water treatment plants, that are not terribly well-protected in terms of passwords,” he said.

A lot of older infrastructure is not “air gapped” from the internet, he said, referring to a separation between operational technology and internet technology. That could enable a bad actor to do things like change chemical levels or open sluices to manipulate flows in water channels or dams.

Compounding the problem is a lack of central regulation or uniform protocols. Multiple agencies — including the Environmental Protection Agency, the National Institute of Standards and Technology, the American Water Works Assn. and the Department of Homeland Security and CISA — provide some degree of risk management oversight, or offer frameworks and recommendations. But many of the day-to-day decisions are left up to individual operators.

“A lot of the responsibility does certainly fall on the stakeholders’ shoulders to manage their own information systems effectively to prevent any type of cyber compromise or cyber incidents,” said Oregon, of CISA.

The agency estimates that about 63% of the nation’s 91,000 dams are privately owned. Federal, state, and local governments and utilities own 35%, and the remaining 2% have “undetermined ownership.”

Despite the risks, experts said it’s important for water systems to be networked in order to expedite maintenance and monitoring. In California, reservoirs are often intentionally spread far apart to maximize rainwater capture and other benefits, so sending physical crews to respond to every potential problem would be time-consuming and expensive, said Ethan Schmertzler, chief executive of Dispel, a cyberdefense firm.

“It all depends upon how water systems are connected, and most water systems in the United States are not — it’s not one national water system,” he said. “The good news is each community is divided into their own command and control systems. The downside is, they’re all divided into their own command and control systems.”

Though most standards are not mandatory, cybersecurity recommendations — and spending — have vastly improved in recent years, he said. Recent legislation through the National Defense Authorization Act will soon compel utilities to report cybersecurity threats to CISA, which will help the federal agency better spot trends, share information and render a response.

John Rizzardo, security coordinator with the State Water Project at the California Department of Water Resources, said the agency operates with an ethos of “layers upon layers of security,” for both physical and cyber threats. Because the agency is also an energy provider in the state, “we probably employ more security features than a lot of just the water industry,” he said.

An aerial view of water flowing out of the compromised Oroville Dam spillway on Feb. 21, 2017.

(Marcus Yam / Los Angeles Times)

That doesn’t mean it is immune, however. CISA pointed to the Oroville dam crisis of 2017 as an example of the nation’s need for “comprehensive oversight and guidance over dam resilience.” During that incident, hillside erosion on the dam’s emergency spillway threatened a major flood event and prompted the evacuation of about 200,000 people, though disaster was ultimately averted.

Rizzardo said the agency has since shored up the spillway and made significant security upgrades, and is working to implement the same standards across all State Water Project facilities. The Department of Homeland Security runs national security drills for the dam sector every two years, he said, which the agency also participates in.

But even with the best protocols in place, “there’s still going to be a risk of a cyber or physical attack,” Rizzardo said. “It could happen — we’re doing our best to prevent it — but if it does happen, we do practice our emergency action plans regularly so that we’re prepared if there is some kind of attack that we can try to mitigate, to reduce the consequences.”

Indeed, the Oldsmar incident was not a one-off. Only weeks later, a ransomware attack on the Colonial Pipeline — a vital U.S. oil conduit between the Gulf of Mexico and the East Coast — spurred fuel shortages, flight cancellations and state of emergency declaration from President Biden.

Earlier this year, Biden unveiled a national strategy for cybersecurity that calls for a “more intentional, more coordinated and more well-resourced approach to cyberdefense.”

Similar attacks have threatened other water systems, including an Iranian attack on a New York dam in 2016, in which hackers tried but failed to take control of a sluice gate.

In January 2021, an unnamed water treatment plant in the San Francisco Bay Area also suffered a cyberattack, NBC News first reported. Hackers accessed the plant’s system through a remote access TeamViewer account and deleted programs used to treat drinking water. The programs were reinstalled the next day and no failures were reported. (The Northern California Regional Intelligence Center, which compiled a report on the incident, said it could not provide more details as an investigation is ongoing.)

One of the largest water providers in the country is the Metropolitan Water District of Southern California, a massive regional wholesaler that supplies 26 agencies serving 19 million people, including the Los Angeles Department of Water and Power.

General manager Adel Hagekhalil said in an email that America’s Water Infrastructure Act of 2018 served as a “catalyst for utilities to evaluate their resilience to risk and create emergency plans for responding to all hazards.”

“We are constantly taking steps to ensure the security of our water supplies against physical and cybersecurity threats,” Hagekhalil said. He noted that community water systems serving more than 3,300 people are required to actively update their risk and resilience assessment and emergency response plans every five years.

Additionally, the MWD employs cybersecurity experts and constantly monitors network and computer activity to “detect unusual events quickly so they can be addressed,” he said. Computer and network access is tightly controlled and employees are also required to take annual cybersecurity training. The agency also conducts periodic emergency management exercises at different facilities to simulate responses to physical threats such as earthquakes, floods, fires and terrorist attacks, which include first responders and law enforcement agencies, he said.

But the U.S. is home to more than 55,000 public water systems and 16,000 wastewater systems, said Jennifer Lyn Walker, director of infrastructure cyberdefense at the Water Information Sharing and Analysis Center. One of her primary concerns was that there is often a “lack of awareness” about the potential for cyberthreats and other such vulnerabilities.

“Physical threats are so much more top of mind, or more easily identified or more easily understood than the cyberthreat,” she said. “The concern is a lack of preparedness.”

However, most large systems in California “are doing what needs to be done” when it comes to cybersecurity, she said. Small and medium-size systems, which often have fewer resources than major providers, may need assistance however, and could benefit from the guidance of larger operators.

“A smaller system that just barely services 5,000 people — that’s still 5,000 people’s lives that could be at risk if something should happen, and that’s from physical or cyber [threats],” she said.

Reddie, of Berkeley, said more auditing would provide a better understanding of which systems are networked, as well as which systems follow best practices. He also recommended educating workforces about proper cyberhygiene.

Even with such steps in place, however, vulnerabilities remain. Ongoing investigations into the Oldsmar incident indicate that it may not have been the work of an outside hacker at all, but might have been caused by an internal employee. Should that prove to be the case, it would highlight that insider threats can also be cause for concern, Reddie said.

“These individual firms need to be thinking about what’s their model for the type of threat actor that they’re likely to see,” he said. “Like, is this going to be a state actor? Is it going to be a disgruntled employee? Is it going to be, you know, a script kiddie in a basement?”