In this four part series, we’ll explore critical infrastructure IT and OT in the modern era, focusing on Electric Utilities in the United States.
Part 1: Understanding IT and OT
Part 3: Bridging the Gap
Part 4: Looking Ahead
Now, let’s take a look at the threat landscape utilities are faced with, that led to this project being undertaken.
What are the common threats to IT and OT systems?
As mentioned previously, the CISA guidance on the Volt Typhoon campaign should be eye-opening for anyone watching the state of cybersecurity in critical infrastructure. Chinese-origin threat actors have successfully compromised utility IT systems and moved laterally into their OT environments, establishing footholds. Significantly more information has been made available to utilities who are members of E-ISAC and released as TLP:RED; if your utility is part of their covered audience and you’re not a member, go apply now at that link and dig deep into what’s available.
These threat actors are using living-off-the-land toolsets, that is, using built-in tooling like powershell’s commandlets or netsh commands typically meant for administration tasks, to enumerate your environment and establish persistence without detection. Being that these tools are legitimately used by support staff, many companies ignore alerting on their use or, at worst, whitelist their use entirely—leading to no detection of threats as they spider through infrastructure.
We got to see living off the land tactics in action: one of our utility’s technology integration vendors was compromised in a ransomware event that could have led to further infection of their entire supply chain, including several hundred utilities. This vendor maintains active 24-hour VPN connections to manage on-prem and cloud resources at their customer environments, where they utilize administrative access to Windows and Linux systems that are often domain-joined to the utility. By whitelisting powershell.exe within their EDR system, this vendor’s alerting effectiveness was compromised—their EDR simply ignored those living-off-the-land exploits as they happened.
Take a look at how a threat actor could compromise your systems if they get a foothold in one. That vendor’s system was in the same network as other unrelated systems and servers (no segmentation), had default firewall rules the vendor deployed (no zero trust or reduction of attack surface), and most importantly the IT group used their admin credentials to log into that system—anyone familiar with Windows knows the NTLM hashes of those domain admin credentials are stored in the operating system, and as we’ll dive into later, it’s trivial to exploit that and gain access elsewhere.
Thankfully the vendor had other tooling in place that caught the exploit as it spread internally, and damage was kept to a minimum, but that risk is still real and present across the entire utility industry as you’re reading this today.
Why does this matter to the utilities?
When a utility deploys a SCADA (Supervisory Control And Data Acquisition) or ICS (Industrial Control System) environment, they typically are motivated by the same business drivers across the industry:
- Safety
- Reliability
- Availability
Safety in the context of a SCADA or ICS environment constitutes the safety of personnel—being able to remotely actuate high-voltage systems without exposing people to the dangers of an arc flash, for example—as well as the safety of our very expensive, capitalized equipment that is needed for grid operations. Being able to monitor the health of multimillion-dollar transformers against internal arcing, or monitor coolant pressure in a $200m GE turbine so it doesn’t blow up, are rather major parts of any modern SCADA environment. These SCADA systems and their remote field device controls can be used to isolate damaged grid components and reduce mean-time-to-restoration of the electric grid as well, improving utility operations and granting better key performance indicators, for those who care about metrics. Protecting human life, protecting utility grid assets, or protecting customer’s or member’s power needs are chief among every utility’s concerns and a very large motivator for implementing this computerized automation and logic into traditionally manual systems and processes.
Reliability and Availability are also concerns—having a SCADA system that can actuate breakers remotely and remove the need for arc flash suits is useless, if these systems have constant failures in communication. Good system design will include planning for partial system degradation, such as a relay failure or switch gear maintenance that removes those devices from service, and will have other ways to isolate system problems reliably without impacting the utility significantly. Modern SCADA systems should be designed with levels of redundancy that allow single-component failures while the system as a whole still stays reliably operational—an example of this redundancy in your everyday life is having both regular brakes on your car, and an emergency brake that can be used to stop you when the main brakes fail.
Cybersecurity becomes a concern when it can impact Safety, Reliability, and Availability. If a cyber threat actor can compromise a system and actuate your field RTUs (Remote Terminal Units), they can put lineworkers in the substation at risk as well as damage the very expensive equipment sitting in the field. A cyber incident could also impact reliability and availability by remotely damaging or disabling control systems or modifying your Data Acquisition ‘points’ to make everything look okay—this in fact was how the now-famous Stuxnet was able to succeed, by making the computer systems “read” good temperature and speed values while manipulating those datapoints in the field, damaging the reactors significantly by the time any personnel noticed something was wrong. Cyber threats in the form of Ransomware or other malware can just straight-out damage and destroy your entire operational SCADA system as well—removing all eyes into any field devices or automation systems until you clean up or replace the infected SCADA components, a very costly and time-consuming process.
Now that we’ve established the business drivers of larger SCADA system design and the security risks inherent in today’s SCADA systems, in the next post we’ll take a look at our example use case: our utility’s need for a new SCADA system network design.