Notes on Volt Typhoon

So this tweet asks for specifics on Volt Typhoon. I don’t have any, I’ve just read the documents by Microsoft and CISA discussing specifics. What I may have to offer is context. So I’ll try that in this blogpost.

Critical infrastructure

According to this story, the Chinese hacking group is targeting critical infrastructure. According to FBI directory Christopher Wray:

“There has been far too little public focus on the fact that People’s Republic of China hackers are targeting our critical infrastructure -- our water treatment plants, our electrical grid, our oil and natural gas pipelines, our transportation systems. And the risk that poses to every American requires our attention – now.”

I first want to stress that such warnings have little credibility. They’ve been crying wolf for almost 30 years that foreign attackers are going after our critical infrastructure. Yet, bad things continue not to happen.

Secondly, I want to describe how such warnings are created. It’s because people assume that if they get hit with a cyberattack that they were targeted.

When I scan the Internet, people falsely believe I’m targeting them. A water treatment plant might detect my scans and notify CISA that they are being targeted by a hacker. But of course, I don’t know who they are — I was scanning everybody.

More illustrative is an incident I had with an intelligence agency. They have public network assets and clandestine network assets. When they detected a scan hitting both their public and clandestine assets together, they not only thought they were targeted, but wanted to know how I knew about their clandestine assets. I didn’t know, of course, I scanned everyone. They didn’t see how I was hitting all the networks they weren’t monitoring.

That they’ve been hit but not targeted, it’s extraordinarily difficult for people to understand. It’s especially hard for the US government to understand.

The cyber operations of China and Russia are largely opportunistic, they’ll hack into anything they can, and they can be indiscriminate because they don’t really care if they get caught. They choose the means first, then spew them against any possible target.

US cyber operations are the opposite. They are directed. We choose the targeted first, then the means to hack them. We avoid detection like the plague — the NSA will choose not to hack a target if that means even a small risk of exposure. The NSA often makes the fallacy of assuming foreign adversaries follow the same rules we follow.

The US government, like CISA, spends a lot of time monitoring critical infrastructure and military targets. Thus, when there is an indiscriminate hacking campaign, they’ll often see it as targeting the things they monitor, without really questioning whether its targeting the things they aren’t monitoring.

They aren’t dumb. Those in the trenches understand this effect, and can describe it with Bayesian statistics. No such incident happens without some techie pointing what I just described. But when these things rise above the techies into leadership, such nuance often gets lost. Leaders already have their conclusions, like China targets critical infrastructure, so any data is molded to fit those preconceived beliefs.

You may have read stories about hackers targeting critical infrastructure, but they are generally warped to fit the narrative. For example, in 2021, it was reported hackers broken into a water treatment plant and tried to poison the water. The story was obvious bunk (as I commented at the time), but it took two years for the FBI to confirm it was bunk — that the cause was almost certainly operator error.

FBI director Wray has all sorts of information that’s not public. The CIA may have a mole inside the Chinese hacker group revealing their plans, for example. After almost 30 years of crying wolf, this time it may be true. But for all of Wray’s rhetoric sounds ominous, he’s given us no new words, no new reason to believe him.

Lastly, consider the context of such testimony. It’s not a warning to the public, so that the public can learn to defend themselves. It’s to congress, to push some political agenda. It’s suspect on first principles

Botnet sources

Regardless of who they are targeting, Volt Typhoon is effective. We are finding them deep in sensitive networks.

The warnings you should heed are from Microsoft and CISA, who in their reports detail the tactics used by the group.

The technique starts with a botnet. It’s standard practice now to run attacks through a botnet, firstly to hide the source of an attack, and secondly to make the source diffuse enough that it doesn’t appear an attack, just normal traffic.

Microsoft described this in a separate (likely Russian caused) incident: they failed to detect a password spray attack because it came from thousands of residential machines rather than a single IP address, distinctive networks (data centers), or Tor exit nodes. Since residential networks (ISPs like Comcast) are where most all the normal traffic comes from, the source IP address is becoming nearly useless.

So that’s the first lesson defenders need to learn: stop relying upon the source IP address.

Public facing products (with a public IPv4 address)

Next Volt Typhoon appeared to hacked their victims through Fortinet FortiGuard. There have been several high-profile vulnerabilities in these devices over the past few years. The important thing to know about the Fortinet vulnerabilities is that all devices were likely hacked within moments of a workable exploit becoming public.

In the past, if you left a public facing device unpatched, eventually it’d be found and hacked. These days, it’ll get hacked within minutes — at least, for all IPv4 devices. A tool like masscan can be used to scan the entire IPv4 Internet in under 5 minutes, automating the process.

In the past, you’d have months before you needed to worry about patching exposed devices. Nowadays, as soon as the vendor (of a product found on the public Internet) releases a patch, hacking groups start reverse engineering it. Once they find how to exploit it, they’ll exploit the entire Internet quickly.

Today, due to a vulnerability in Ivanti (another public facing product), CISA recommended an emergency disconnect of all those devices. If they are exposed to the Internet, they are already hacked. CISA detected that the bug was already being exploited — in today’s world, if it was exploited somewhere once then it’s almost certainly been hacked everywhere. There’s little difference from a hacker’s point of view whether they hack one IPv4 address or all of them.

The upshot is that you need an inventory of all publicly facing products, like Fortinet, Ivanti, Citrix, etc. When a critical bug is found that either can easily be exploited, or worse, has exploits detected in the wild, then it’s already been exploited as far as you are concerned. The obvious action is to immediately disconnect it.

I’ve gotten pushbacks from such extreme statements. People point out that organizations are still struggling to patch in a timely manner, so there’s no way they can follow advice like “immediately disconnect”.

That’s true in a fashion. I’m the first to point out “cybersecurity advice” by experts is out-of-touch with the reality of what IT departments are capable of. But that’s because experts are also out-of-touch with the risk. The risks to private devices is dramatically less than public devices. When the statement is “hackers control your security product”, running to the basement and pulling the electricity breaker shutting down the entire company is not too extreme an action.

I’ve done that (metaphorically) a couple times, notified the CEO that there’s an extreme situation and I’m going to bypass all processes and piss off a lot of people in order to address it. It’s very satisfying doing such a thing. Of course, I was in positions where the CEOs would meekly give me permission, which most IT/cybersec people don’t have.

The point of that paragraph is that part of your process needs to be that somebody is assigned the ability to pull the plug purely on their own authority. In a lot of companies, it’s going to be the CEO by default, to which the process should be that IT/cybersec workers need to have an emergency red phone line to the CEO. I’m a great fan of “process”, but part of process is how to cope with the unexpected emergency that doesn’t fit within your processes.

Living off the land

The NSA has been warning us for a while that adversaries are shifting to “living off the land”. That means instead of uploading hacking tools/malware to a hacked computer, they use the existing tools already installed.

This makes them hard to detect.

It’s part of the general trend as mentioned above with botnets — hackers avoid leaving behind IoCs. IoC stands for “indicator of compromise”. When a nation-state is detected, analysts release things like the IP addresses they are using, the hashes of the tools/malware, and so on. These IoCs can then be imported into tools that will look for these signs on the local IT infrastructure.

I’m highly critical of these efforts because they are of low quality. A few years ago, there was a news story about a Vermont nuclear power plant getting hacked. Russian hackers had been using Yahoo email as part of their hacks, so the government added the IP addresses of Yahoo email servers to their IoCs. An IT worker at the powerplant blindly added these to their detections, so that the next morning, when an employee logged into Yahoo email as normal, alarm bells went off.

But apparently, such IoCs are making a difference, as shown by the efforts nation-state hackers are spending at avoiding them.

The Microsoft and CISA documents do a good job describing how your processes need to change to detect these new threats. It’s less important about how to detect Volt Typhoon specifically and more a description of the changing methodology.

Organizations on the leading edge of cybersecurity are already doing such things. The Microsoft/CISA items are only minor tweaks to their infrastructure. A lot of orgs have threat hunting teams that’ll jump right on this. That’s the lesson you should be taking away from Volt Typhoon, that the detection isn’t revolutionary.

Except most of organizations are behind the curve, such that they have no ability to implement what those Microsoft and CISA documents describe.

The original tweet asking me to comment on this mentioned WMI. The command-line tool wmic has long been used by hackers to launch code in ways that older tools wouldn’t detect. Here, among it’s uses is to launch netsh interface portproxy, a common technique for setting up proxies, which allow hackers to mask the source of their attacks. It’s part of the living-off-the-land, using built-in Windows tools to accomplish these goals rather than uploading custom code to do it.

Conclusion

This blogpost is in response to this tweet asking for specifics about Volt Typhoon. I don’t really know the specifics, and my skills are getting rusty. Using ntdsutil to generate install media, then scrape passwords from that? I’d never heard of this!

But as an old timer, maybe I can offer perspective. A threat hunting team on the leading edge of defending against such things would have much better discussion on this.