Notes on Microsoft's Midnight Blizzard attack
I’m not liking this post from Alex Stamos critical of Microsoft “addiction to security revenue”. He pretends that instead of fixing security problems for free, Microsoft is charging for the fixes. It’s a cheap accusation that he doesn’t substantiate, knowing that his readers hate Microsoft anyway, and that he needs no substantiation.
But I’ve read the stuff Microsoft has made public, and I don’t see the same thing.
Note: I have an immense amount of respect for Alex Stamos. You should serious consider anything he writes.
There was no product flaw
Stamos translates what Microsoft says to mean the following:
Stamos: Since the techniques outlined in the blog only work on Microsoft-hosted cloud identity and email services, this means that other companies were compromised using the same flaws in Entra (better known as Azure Active Directory) and Microsoft 365.
But Microsoft was clear in its blogpost, that no flaws were exploited.
Microsoft: The attack was not the result of a vulnerability in Microsoft products or services.
The cybersecurity community has an unhealthy obsession with vulnerabilities. Vulns are scary, because they mean that even if you do everything right as a defender, a vulnerability in software can still get you hacked.
The reality is that vulns are rarely the cause of hacking incidents. The reason people get hacked is because they did something wrong.
That’s the case here. Microsoft makes it clear what’s to blame, and it’s not the products or services.
Moreover, it’s not clear that it’s Microsoft-specific. Sure, the tools the hackers use are specific to Microsoft’s infrastructure, but the general hacker technique of password spraying isn’t.
Legacy tenant
Stamos thinks Microsoft is spinning the issue by calling it a “legacy” system that was attacked.
Stamos: Calling this a “legacy” tenant is a dodge
As an attacker, I disagree. It’s common for hackers to target legacy systems. Legacy is an important descriptive word.
I see this when masscanning the entire Internet: big corporations attach unimportant machines to the Internet all the time that nonetheless have full enterprise access. For example, when pentesting an insurance company once, we entered through a system intended for retiree’s. Since it wasn’t mission-critical, it wasn’t maintained. Yet, it was still part of the Active Directory domains, with real accounts, and thus integrated into the whole trust maze that is Microsoft.
In this section, Stamos is right about the AD trust maze, and how the only good tools for discovering problems are the ones hackers develop for themselves. Every big corporations needs to restructure their system from the ground up. The “typical” pentest should consist of giving the pentester a standard employee desktop — breaking in from the outside is a secondary concern.
But the flaw is less Microsoft’s than their customers. Microsoft’s tools are actually pretty good if you structure your AD systems as Microsoft suggests. The problem is that nobody seems to be able to do this — even, as it appears here, Microsoft itself. Part of the problem is legacy. For example, nowadays Microsoft uses 2FA for all their employees, but in older systems, they didn’t.
I’m more on the red team here, attacking Microsoft with no experience on the blue team side. I don’t know how much blame Microsoft’s deserve for this maze. But I know it’s complex, not something you can wave aside by claiming Microsoft should’ve made it “secure-by-default” — a facile phrase that claims it’s a simple moral weakness, to which everyone who doesn’t understand complexity will agree to.
Anyway, back to “legacy” systems. On average, most all systems in production are “legacy”. Last year when I was writing about Twitter firing Peiter Zatko, one of the concerns was legacy CentOS systems. Well, at the time, the suggested CentOS distribution Amazon AWS suggested was from 2016 — six year out-of-date.
The problem is worse for non-production systems, such as the retiree’s website mentioned above.
When I masscan the Internet, it’s pretty easy measuring how out-of-date a system is. All the major protocols tell\ you, like SSH, SMB, MS-RPC, and SSL. Scan the Internet for SSL ports and you’ll be shocked at the percentage of systems have expired certificates., for example.
Here’s how to you trap hackers. Put up a system with an expired certificate for “ad.microsoft.com”, and you’ll see a flood of attacked against it, a lot more than a system “ad.microsoft.com” with a valid certificate.
One specific subcategory of “legacy” is “test systems”. It’s common for online services to have “production” systems that aren’t actually quite in production. They roll out changes to those test systems first that are indistinguishable from the customer facing systems, except that customers aren’t interacting with them, and won’t notice if something fails. After the test is done, people lose track of them, because they aren’t essential anything.
What I read from Microsoft’s tea leaves is these were installed several years ago to test Microsoft’s own efforts to move its internal infrastructure to its Azure cloud — dogfooding, as they call it. That means they have legacy test systems integrated with Microsoft’s actual user accounts.
Thus, far from appearing a doge, Microsoft’s description of “a legacy non-production test tenant account” sounds perfectly reasonable. I see such things everywhere. It’s important to distinguish the fact that the latest systems wouldn’t behave this way.
Microsoft upsell
Microsoft has a long blogpost educating it’s customers on how to mitigate the threat (well, not long itself, but a lot of links to other long pages). Only rarely are any of its recommendations to pay for additional services from Microsoft.
Stamos identifies three products Microsoft recommends that cost extra money. These are cherry picked — he’s ignoring the vast wealth of information Microsoft has published that requires no additional money.
In fact, most of the cybersecurity industry could be characterized as customers ignoring the features that Microsoft supplies for free (or cheaply) and instead buying inferior alternatives from security vendors.
Stamos describes these upsell features as the following:
Stamos: This is morally indefensible, just as it would be for car companies to charge for seat belts or airplane manufacturers to charge for properly tightened bolts.
It’s a grossly unfair characterization. It’s not like “seat belts”, it’s like “anti-lock breaks” when they were new. When they were invented, they were an expensive option on cars. Customers who cared more about safety would pay extra.
Then government mandated them, so they are now standard on all modern cars. This raises the price of cars, forcing people to pay for them who don’t care about the safety. On the other hand, by making them standard, it dramatically reduces their price, making them a lot cheaper for those who do want them.
These optional Microsoft security features are the same way. They cost a lot of money. They require expensive log retention, and even more expensive processing of those logs. Certainly, you could insist that they be standard, making all Microsoft’s services more expensive for everyone. It’s not a moral travesty, it’s a tradeoff.
Microsoft’s “Entra” (Azure’s cloud Active Directory) has multiple tiers, from a free service, to premium ones providing more features and more security. It’s absurd claiming there should be a one-size-fits all with maximum security features for the maximum price.
Microsoft prices these things differently because they cost money to provide them. Log retention is expensive. Analyzing logs is even more expensive. Development of new algorithms to detect new attacks costs a lot more money. Microsoft charges for them because developing and providing these services is expensive.
There is a lot of discussion in the industry about minimal logging to detect attacks. I’d agree that like anti-lock breaks, that any cloud service with logins should log all login attempts. But those additional services Stamos cherry picks describe esoteric log entries that most people haven’t heard of. They are ones that retroactively, we’d say are obvious for detecting Midnight Blizzard’s attack. But they never appeared on anybody’s “necessary logging” list before these attacks.
None of these extra Microsoft features are necessary, in much the same way as anti-lock breaks aren’t necessary.
I understand the need to charge for log storage or human services, but we should no longer accept the idea that Microsoft’s basic enterprise offerings (including those paid for by the US taxpayer) should lack the basic features necessary to protect against likely attacks.
Microsoft’s blogpost is full of free things to do, or at least, which don’t require giving more money to Microsoft. The necessary things include adding MFA, filtering for weak passwords, and removing unnecessary trust relationships — all free.
The three items mentioned by Stamos are not necessary. It’s like the difference between a firewall and an intrusion detection system (IDS). Firewall rules are necessary, IDSs are not. Sometimes an IDS will help identify attacks, sometimes it’s useless, depending upon the attack. The same is true here. All three of those “upsell” items are not critical, and even if you have them, aren’t guaranteed to find attackers. Instead, since most people aren’t being attacked by Russia, they’d be a huge expense or no gain. Moreover, they are not only expensive on Microsoft’s end, they also require dedicated staff trained on using them to audit logs on the customer’s end. Even if they were free, they’d be expensive to use.
Conclusion
Stamos is preaching to a choir who doesn’t care if his descriptions are wrong, or if his claims are substantiated. He makes the following two statements that are wrong:
that there is some flaw in the products that need to be fixed, that wouldn’t exist if they were secure-by-default — there is no flaw
that the items Microsoft is “upselling” are necessary — they aren’t necessary, all the necessary things are free
These are clearly and unambiguously incorrect. Microsoft’s Active Directory is certainly complex, and I suppose that’s a flaw, but that’s more Microsoft’s customers demanding this than Microsoft pushing this. Microsoft does indeed push simplifications on customers, but they refuse to adopt those measures.
None of those “upsell” items are necessary to secure yourself. You shouldn’t even consider them until you’ve first done all the free things Microsoft mentions in their blogpost. If you’d done all the free things Microsoft has suggested for years, you wouldn’t even need those optional features. In any case, it’s right that Microsoft charges for them, because they are expensive to provide, in terms of storage (log retention), computation (algorithms), and human efforts needed to investigate the things it flags.
I don’t know. I’m an attacker rather than a defender. There may be more going on behind the scenes that anonymous leakers might know. It’s just that from the public information, none of Stamos’s accusations are substantiated.