The value of pi π is 3.141592653589793. That’s the value held in an IEEE 754 64-bit floating point, how modern CPUs do number crunching. If you need pi π for a calculation, that’s the value you’ll end up using 99% of the time. This is the value NASA uses for all its calculations, because it’s good enough even to pilot spacecraft to the next star.

Math nerds

My tweet discussing this went a little viral. Techies can’t help but say “well actually…”. In this case, they say the actual value of a float64 holding pi is the following 49 digit number:

3.141592653589793115997963468544185161590576171875

They can cite support for this. The AIs will often agree with them, such as Gemini:

They are all wrong.

The reason is that there is a difference between math and engineering. Mathematically, the value of this float64 equals this. For engineering, though, if we start with 16 significant digits we cannot get 49 digits at the end. Anything after digit 16 has been hallucinated.

Sig Figs

In science and engineering we are taught about significant digits, or significant figures, or abbreviated sig figs. Not all the digits in a number are significant.

It’s 100 miles between Nashville and Huntsville. You might claim, therefore, that it’s 160.9344 kilometers between the two cites. After all, that’s the exact mathematical definition. Back in 1959 there was an international agreement setting the conversion factor from miles to kilometers exactly at 1.609344. Therefore, 100 miles exactly equals 160.9344 kilometers.

But the original number was only a rough estimate, the true value is 103 miles. The original number only had about two significant digits. Therefore, when translating a rough estimate to decimal, we keep the number of significant figures and discard the rest. Hence, it’s 160 km between the cities.

Adding extra digits is just hallucinating them, making them up.

Where float64 expansion goes wrong

I wrote 50 lines of code to explain the issue. This code deconstructs the floating point number, does the same calculation every does to reach the 49 digit number, and shows where the hallucination happens (line #14).

We extract the mantissa as the number 7074237752028440 and extract the exponent as 2251799813685248.

To calculate the result, we simply divide 7074237752028440 ÷ 2251799813685248. If you plug this into a calculator, you’ll get that long 49 digit number.

By default, when we do such math, we assume integers have infinite precision. We can assume that a number like 42 also means 42.00000000…. — the integer followed by an infinite number of zeroes.

But in our Nashville example above, we can’t assume “100 miles” is “100.00000…” followed by infinite zeroes. Indeed, even the last zero isn’t significant, only the first two digits.

The same is true in the floating point calculation. The number 7074237752028440 is not followed by an infinite number of digits, 7074237752028440.000000…. Instead, it has exactly those 16 decimal digits and no more.

In my code, the hallucination of extra digits happens on line #14. This shows one form of long division. This line multiples the remainder by 10, padding the integer with a zero, hallucinating it.

If you don’t set the max_digit parameter, then this loop can hallucinate a lot of extra digits, going past 16 digits to produce 49 digits.

Fewest digits

Modern languages like JavaScript, Python, and Ruby get the right answer.

$ node
> 3.14159265358979311599796346854418516159057617187
3.141592653589793
> 3.14159265358979311111111111111111111111111111111
3.141592653589793
> 3.141592653589793
3.141592653589793
> 1.50000
1.5

In this example, JavaScript uses float64 internally. When it prints the number, it prints the fewest digits. That means at least limiting output to the significant digits, but often even shorter if extra zeroes aren’t needed for padding (even if the zeroes are significant).

It’s actually a difficult algorithm (like Dragon4) because you want fast output so you can spit gigabytes of JSON to a file, with these numbers formatted correctly.

The point is simply that while AIs often get the wrong answers, these programming languages get the right answer.

Conclusion

I’m writing up this blogpost because I wrote up some code.

I’m making an unusual claim, that these AIs and many nerds are hallucinating insignificant digits. This trolls some nerds who deconstruct the floating-point number to show division by two integers, thus “proving” they are not making up digits.

But the error is their assumption the integers had infinite precision. The mantissa doesn’t, it has exactly 52 binary digits or 16 decimal digits. Anything past that is hallucinating precision that doesn’t exist.

Recently, Daniel Lemire had a blogpost pointing out that AI produced surprisingly fast code for parsing IPv4 addresses. As a “parser” guy, I have some comments.

The summary is this:

• his benchmarks were flawed, though the code is indeed fast

• it’s fast because of good compilers and good CPUs, not good AI

• let’s look at some other algorithms

Large test sizes

The reason the AI code appeared so fast is that his benchmark was flawed. He only parsed 15,000 addresses, instead of a larger test like 1,500,000 addresses.

Instead of his original benchmark taking 4.4 nanoseconds per IP address, it now takes over double that, 11.05 nanoseconds. These are per address, calculated by total time divided by the number of addresses.

One possible reason is that the larger test size is evicted from the fast L1 cache into the slower L2 cache. You can see that with the memcpy part of the benchmark, because L2 bandwidth is lower.

But this shouldn’t be a latency problem. CPUs predict access patterns, so should automatically put something into the L1 cache before it’s needed. Also, you can fit 10 addresses into a single 128-byte cacheline, so it would only impact 10% of the addresses.

I’ve concluded it’s probably branch prediction. Modern CPUs have added branch prediction algorithms that depend upon data accesses, which will get disrupted when data access patterns become too large. In other words, branch prediction “overfit” the original test case size.

A failed branch prediction can add around ~3 to ~4 nanoseconds. I tracked some low-level CPU hardware counters that suggest there are 1.7 branch misses, which roughly lines up with the extra 6.5 nanoseconds in time.

The hardware counters are flaky, and speculative execution mitigates a lot of the cost of a miss, so I’m not really confident in this conclusion.

I wrote my own code to test this, which I describe in a section below.

The conclusion is simply that a short test doesn’t reflect real-world behavior, you need a large test case. The size of that test case matters, simply repeating a short test case isn’t sufficient to properly measure this. In real world parsing, input is constantly changing, so wouldn’t match the short test case, but the long.

Smart compilers and CPUs

The reason that AI generated code is fast is because modern CPUs have a lot of tricks, such as the data-dependent branch prediction mentioned above.

Remove those tricks and go back to an old CPU, and the benchmark doesn’t show such an advantage. It’s not bad, it becomes merely average.

There’s a similar story with compilers. Using modern compilers to build code for old CPUs still improves their speed. The recognize common idioms, replacing the code you think you wrote with something more efficient. The website GodBolt.org is does a great job compiling code for you on virtually all compiler versions so that you can see the difference in how modern compilers generate code compared to old ones.

AI produces average code. Modern compilers and CPUs have optimized the s**** out of average code, making it perform fast. It’s to the point where you should ask the AI to produce typical code, and not try to steer it toward weird optimizations, because that’ll likely slow things down instead of speed things up.

My fastip project

I’ve created a project, fastip, to benchmark this against other algorithms. I’ll describe the other algorithms below.

I implement a bunch off different algorithms, and then benchmark mark them twice, once with the short test-case, and again “+” with the long test case. The AI generated algorithm is the first one, though I ported it to C from C++, and it appears a bit slower, at 5.4 nanoseconds per IP address, going to 11.9 nanoseconds for the larger test case..

The hardware counters are flaky, so it’s not correctly reporting branches or misses.

I changed the test to also run on the e-cores. Modern CPUs have both powerful, power hungry cores (p-cores) and efficient, slower cores (e-cores). My tests also run on the other cores, where I get better hardware counters:

As you can see, the original algorithm now takes 27.2-ns vs. 5.4-ns. That’s because the cores are running at a slower clock rate (1.1-GHz instead of 4.1-GHz), but also because they are slower per clock cycle (4.7 IPC instead of 7.1 IPC).

The thing to note is that I’m getting branch/miss hardware counters to the right. I can see that the algorithm has 40 branches, and that in the larger data set, 1.6 branch misses.

This confirms branch misses are a problem, but introduces other problems. Branch misses are roughly the same cost, ~3ns to ~4ns, so doesn’t explain the 25ns slowdown we see here.

Another way I tested this is using an algorithm that gets rid of all branches. I’ve marked this in the screenshot below, in the [brch] column, where using SWAR techniques, the number of branches go from 40 to 2. And those branches come from simply calling the parse function.

By “branches” we mean all “if” conditional statements, “for” or “while” loops, “gotos”, or “function calls”. By “SWAR” we mean using techniques from SIMD/GPUs applied to normal C code, which among other things, removes branches. You can go read the code if you are interested to see how we can get rid of such things.

The [swar] algorithm for parsing addresses experiences no slowdowns when moving to the larger data set, implying that indeed, branches could be the problem. It has no branch mis-predictions because it has no branches.

Similarly, the next algorithm, [from], is branch heavy, with branch misses regardless of input size, again implying this could be related to the problem.

I have a bunch more algorithms. These are vibe coded, and not particularly optimized. Like the [swar] algorithm (also vibe coded), they are meant simply as a different way of looking at how the CPU operates rather than trying to be fast.

Three of these, [dfa], [fsm], and [fsm2] are different ways of using state-machines as parsers. They are unoptimized and therefore slower, but when optimized, they are usually the fast way of parsing. The reason is that they also optimize the surrounding code that calls them, namely, handling fragmented input.

A typical use of parsers first reassembles fragments then does an initial pre-parse, such as finding the end-of-line in text. State-machines avoid this.

Then, within the state-machine framework, you can use non-state-machine techniques to optimize things, getting the benefit of both worlds.

Conclusion

The high instructions-per-clock of 7 IPC from the original blogpost made me suspicious. I found that the size of the input test case matters. I’m still unsure why it matters, though I suspect it’s branch prediction related. I created a project that investigates this.

They looked redacted. They had black bars over the text, preventing humans from seeing the text. But, the text was still there. A redaction needs to remove the text — but the FBI’s attempt at redaction did not.

To understand this, open a word processing program like Microsoft Word or Google Docs. They have an option to change the background of text, the “Highlight Color”. An example is this below, where I show some text highlighted with yellow, and some other text that I’ve highlighted with black.

Now if I choose a black background color, the text becomes unreadable, because it’s foreground color is also black. But this doesn’t mean the text isn’t there. You can select the text and change it back to white (or yellow) in order to make it readable again. Or, you can select the text and paste it into a different document. Here is a link to the Google Doc so that you can see this for yourself.

We are not hacking when we select the text in an editor, copy, thend paste it somewhere else in order to read it. We aren’t particularly smart, it’s just that FBI agents are particularly stupid, believing that if they can’t read the text that it has been “redacted”. It hasn’t, the text is still there.

Changing the highlight color to black is something you can try at home, but what happened with the Epstein PDF files is slightly different. Instead of changing the background, they put black bars in front of the text. But anybody reading the PDF file online can still select the text, copy it, and paste the text somewhere else in order to easily view it.

Thus, when you see this sort of thing in a PDF file, you don’t know if it’s actually been redacted, whether the text is still there, or whether its been removed. You have to select the text and copy it.

This is one of the many non-redacted redactions in the Epstein dump. I include the PDF with this post so that you can scroll down to section 80 and see for yourself that the text hasn’t been redacted. Select the entire paragraph, copy, and paste somewhere else.

2022

823KB ∙ PDF file

Download

Download

Recently on the socials there was a discussion whether VPNs are needed to protect your privacy while on WiFi. As an expert, I have comments. More to the point, I have evidence.

The last part about “evidence” is important because of tweets like the one shown below, where the author demands people believe him because he’s an expert. He was wrong, though. I’m a bigger expert, but more to the point, as you’ll see below, I can show the evidence.

Cybersecurity has changed a lot over the years, and a lot of expert advice may be outdated.

Because SSL/TLS is used everywhere, it’s relatively safe to use public WiFi. It’s not 100%, you may have your email program misconfigured to send plaintext, or somebody may trick you into ignoring warnings caused by a man-in-the-middle attack, but it’s safe enough that experts themselves use public WiFi.

You still still admonitions to avoid public WiFi by those giving cybersecurity advice. It’s one of the ways of measuring the trustworthiness of advice — especially by those claiming the mantle of an official government department qualified to give such advice. Such advice almost always sucks.

The things that still leak over public WiFi are DNS lookups, website names in SSL/TLS headers, and the IP addresses of websites. I’ll talk about all three in this post.

To fix plain-text DNS, you can now send DNS requests to public resolvers over SSL/TLS. Google’s 8.8.8.8 resolver, CloudFlare’s 1.1.1.1 resolver, and many others support it. You can configure your browser to use them, and/or configure Windows 11 to use them more broadly than just the web browser.

The screenshot below is how I enabled it in my browser (Brave); ask the AI how to enable it in your browser.

The above tweet mentions encrypted DNS as well as “ECH” or “Encrypted Client Hello”.

SSL/TLS famously encrypts the connection, but it starts with an unencrypted “handshake” or “hello” step. The server name (“SNI”) is sent during that process.

The latest version, TLS 1.3, supports an optional encryption of this handshake, “ECH”, that will hide details, including the server name.

But it’s not widely supported as the above tweet claims. Anybody can verify this trivially by “sniffing” their own packets and seeing the names of websites appear in those packets.

I’m at a bar using public WiFi and did just that, visiting “PornHub”. As you can see, the name “pornhub.com” is right there in the packet.

Don’t trust me because I’m an expert, trust the evidence. This guy demanded you trust him as an expert, but yet, he obviously got things wrong. He didn’t use a sniffer.

But the thing that really irritates me about the “trust me, I’m an expert” approach is his response to the screenshot debunking him.

The answer is that it doesn’t matter which DNS I’m using. If he were an expert, he’d know that. Furthermore, when confronted with evidence, the correct response is to either admit you were wrong or provide evidence of your own.

Instead, he attacked a possible flaw, to cast doubt on the screenshot. In the world of toxic twitters, this is the sort of things pundits do to satisfy their fanboys. The fanboys will believe my evidence has been debunked, when in fact, nothing of the sort happened.

In any case, I addressed the flaw:

As you can see, I’m getting DNS from 1.1.1.1 over TLSv1.3, but still ECH is not used by PornHub.

I admit, things can get more complicated. When I told the browser to use 1.1.1.1, it didn’t. Instead, it used “chrome.cloudflare-dns.com” server, which maps to 162.159.61.3. It’s fine, it’s the same service. To configure 1.1.1.1, I have to manually enter the URL:

Yes, such details are boring, but I’m trying to make it easy for anybody to replicate this rather than trust my word as an “expert”.

Besides DNS and SNI, the third place that leaks the website is the IP address. In the above screenshots, you’ll notice that PornHub is located at 66.254.114.41. This maps to reflected.net, a CDN or “Content Delivery Network” — not actually PornHub.

CDNs effectively mask IP addresses of the server. Content providers, like PornHub, give the content to CDNs, which in turn, give it to the consumer. That CDN’s address may simultaneously be hosting content from PornHub, Disney, the BBC, and the Department of Defense.

CDNs put their servers in major cities around the world. While you may think you are accessing a website thousands of miles away, most of the content comes from a data center located less than 100 miles away. One of these days I’ll do a robust study measuring the average distance an Internet packet travels; I bet it’s less than 100 miles (200km).

In any case, CDNs really handle “bulk” content, like the home page and videos. If I surfed PornHub enough, I’m sure it would eventually leak an IP address that can be tied to it. Unfortunately, I’m in the state of Georgia right now which blocks PornHub, so all I can see are a home page and some help pages.

Conclusion

David Plummer is right: you are far less exposed on a public WiFi than you were in the past. With near universal SSL/TLS support, encrypted DNS, encrypted Hellos, and content deliver networks, adversaries can’t see much.

But at the same time, SSL/TLS is only “nearly” universal, encrypted Hellos are not widely supported, and not everything goes through a CDN. Adversaries can still see enough.

The “expert” is correct knowing that such things exist, but he’s not an expert enough to simply put Wireshark on his network and see the exposure for himself.

I don’t know how much VPNs help. As far as the local WiFi is concerned, you are anonymous. MAC addresses are randomized these days so they can’t really identify you. VPN providers, however, are not so anonymous. Most have your user account information. Some allow anonymous payments with cryptocurrency, but still, if you connect to their service from home, they’ll have your home IP address. They promise to ignore such things, and some probably do. But ultimately, you are just shifting trust one the local bar to a remote company in Sweden.

My point is that I’m an expert and I don’t know what you should do. I can educate you on the choices, but I can’t make choices for you. Don’t trust anybody that claims to be an expert making choices for you, telling you what to do (or not do). Especially when the evidence shows they are wrong.

ESR, a famous nerd, disagrees with my claim that the “OSI Model” was based upon IBM “mainframes”. It was, but it’s complicated.

ESR’s version of history is right—that the OSI Model was driven by European academics opposed to American DoD researchers. But there’s another view of this history, where these academics were still influenced by Big Government, Big Telcos, and Big Mainframes.

In this post, I describe how they are related. I explain how the idea of a “model” comes from mainframe networks, how the lower layers were based directly on the mainframe network stack, and how the upper layers were inspired by mainframe thinking.

What’s a Mainframe?

The “mainframe” comes from the era where computers were so expensive that an organization could only afford one of them. Such early computers didn’t even have the typical user interface of a keyboard and screen. They were just buttons and panels of “blinking lights” that were programmed with punch cards.

As Moore’s Law turned, there was a fork in technology. One fork led to the democratization of computing, with more and more power on a user’s desktop. But another fork led to smarter devices attached to the mainframe, with the mainframe still in control.

In this fork of technology, users had simple “terminals” that merely displayed content rather than running code. The app’s code still just ran on the central mainframe.

IBM was the biggest supplier of mainframe computers. In the 1970s, IBM accounted for about half of the entire computer industry. American researchers and companies like Intel and Xerox often lived in a separate non-IBM world, which is why we have non-mainframe technology today. But much of the rest of the world still lived in the grasp of IBM.

Where OSI Comes From

The lead author was Hubert Zimmermann of OSI, a French researcher who helped develop the CYCLADES network, from where the Internet gets its “end-to-end” principle.

But the Europeans in turn were building standards from the top down. At the top were the big national telephone companies and big industrial firms, with “stakeholders” and “committees”. As such, the original OSI Model was based upon industry more than academia. (The Internet was built bottom-up — nerds just built things, whatever worked became the standard).

The original 7 layers were contributed by Charles Bachman, an engineer working for Honeywell, one of IBM’s mainframe competitors. Honeywell was building a network stack modeled on IBM’s SNA.

Hence, the original OSI layers matched very closely with IBM’s “SNA” network layers, especially the bottom 3 layers. The easiest way of understanding what the OSI standards actually mean is to read documentation from the 1970s about how SNA actually worked. You really aren’t going to understand what the “Session Layer” really intends unless you learn about SNA’s “Data Flow Control (DFC)” functionality.

The important thing about IBM’s mainframe network stack was that it’s essentially a single product. Each layer specifies a piece of the larger product.

That’s where the OSI Model goes wrong. It sees the entire stack as having a fixed number of layers, where each layer has a different purpose.

That’s not how things work on the Internet. Layering happens, but in an ad hoc fashion. In RFC 791, the [Internet Protocol] runs over some sort of local link or local network, but the local details don’t matter. It might be Ethernet, or it might be pigeons. The local network may itself have sublayers, but that fact is opaque to the Internet as a whole.

The RFC 791 model is really only two sublayers: the Internet Protocol layer and the transport layer. Everything above that is just some sort of payload. What runs on top is as opaque as whatever links things below.

The idea of fixed vs. ad hoc layers led to much debate about SSL/TLS. People wanted to make SSL fit the fixed OSI Model, which really had no place for it. People struggled to accept that this was just an ad hoc layer, that it used transport below, providing encryption to the payload above.

The point is that the idea of fixed layers with assigned functionality comes from the IBM mainframe world. It is largely foreign to the Internet standards.

Layer #2 and SDLC

IBM mainframes inspired the fixed-function model as a whole, but also defined several of the layers specifically.

The OSI “Data Link” layer comes directly from IBM’s “SDLC,” which stands for “Synchronous Data Link Control”. It’s right there in the names, “Data Link”.

In the beginning (as far back as 1800s), all you had was a “link”—a wire connecting two points. (There were also multidrop wires, but forget about that for the moment.)

Such links could have “dumb” devices on either end, devices without a CPU or even transistors. The early teletypes were just that sort of dumb device.

The early RS232 serial link standard had 25 pins. Only 3 wires were strictly necessary: transmit, receive, and a common ground. Other pins were used for control. For example, if the sender was transmitting faster than the receiver could keep up, the receiver would send current down a dedicated pin to tell the sender to pause. This could be handled with mechanical solenoids rather than needing software.

The invention of the 8-bit microprocessor changed things. Suddenly, it became practical to put smart software on either end of a link. Link “protocols” changed from being separate wires to data sent back and forth in packets. If the sender was transmitting too fast, the receiver would send a message in the other direction.

This was the birth of IBM’s “SDLC.” It was a packet protocol for serial links. Among its features was adding checksums and serial numbers to packets so that if they got corrupted, they could be resent. This was a big problem with the low-tech cables and connectors of the time, where electronic noise would corrupt packets.

IBM’s SDLC quickly inspired standardization. One standards effort was a slight variation called “HDLC,” with some small variations. Another variant was called “LAPB,” used in the telco X.25 standards.

The original Ethernet had no such thing. When Ethernet was standardized and pigeonholed into Layer #2, they needed to add something compatible with SDLC. This was called “LLC” and is standardized as IEEE 802.2, and runs as a sublayer on top of the Ethernet “MAC” sublayer, both part of Layer #2.

You don’t see LLC today because it was never actually needed. It exists only to make the non-mainframe Ethernet conform to the mainframe model.

By the time LLC was create, it wasn’t really needed. Cable technology had reached the point where they would reliably transmit packets without corruption, and networks had reached the point where you wanted to retransmit lost packets “end-to-end” anyway, meaning not simply across the local link, but between the remote ends across the Internet.

Network Layer #3 Was Connection-Oriented

Hubert Zimmermann helped design CYCLADES, the early French network that influenced the design of the TCP/IP Internet. But the OSI Network Layer #3 did not work like CYCLADES—it worked like IBM’s SNA and telco X.25. It was “connection-oriented,” not “connectionless.”

Let me explain the difference.

The telephone system works according to connections or circuits. When you make a phone call using a traditional wired phone, you establish a virtual circuit consisting of a 64-kbps stream of bits flowing in each direction.

This is the “T carrier” system from the 1960s when the phone system was made digital using the newly invented transistor. Smaller streams are combined into larger streams, like the 1.544-mbps T1 line and 45-mbps T3 line. The telephone switch would then forward streams, so multiple incoming streams on one line may then be split up to flow out of different lines—each stream flowing toward its own destination.

When you dial the phone, every switch in between the caller and callee is contacted, and a 64-kbps stream is reserved. If there is congestion, the caller will instead hear the error message “no circuits are available” and the call won’t go through. You often hear this on New Year’s calling to wish family and friends well, or after natural disasters trying to call loved ones in the area.

Once a call succeeds, then congestion won’t happen after that point. You’ve got a 64-kbps stream until the call ends.

Such streams of bits are inefficient for computer networks because they are flowing all the time, even when the computers have nothing to transmit. Computer data is bursty, sending a lot for a short period of time, but silent the rest of the time.

Computers want to send data in packets rather than in streams.

To handle this, the major telephone companies (the “telcos”) developed the X.25 packet switching standard. Your computer would then request the packet-equivalent of a virtual circuit. Each packet switch between the source and destination would be contacted, and a “connection” established.

One of the properties of such connections would be to require a minimum transfer rate, such as 1-mbps. As long as you transmitted less than that minimum, your packets were guaranteed to go through the network. You could transmit faster, but those packets would be delivered “best effort.” When there’s congestion, such “best effort” packets could be dropped.

Likewise, when you used less than your guaranteed bandwidth, the switches in between would be using that opportunity to forward best-effort packets for other users.

Thus, the packet-switched X.25 network was much more efficient, and therefore cheaper, for computer users.

The Internet is a packet-switched network as well, but it’s only “best effort”. You don’t establish a connection through the network ahead of time; you simply send the packet. You can’t reserve a minimum amount of bandwidth. If there’s congestion somewhere in the network, that packet will be lost—”dropped” by the router that’s unable to forward it out a congested link. Each packet finds its own way through the network, so when you send two back-to-back packets, they may follow different paths and arrive in the wrong order.

The Internet is therefore a connectionless network.

Both mainframe and telco X.25 networks demanded the “reliability” of a connection-oriented network, so therefore the OSI Network Layer #3 specified it. The only option was a connection-oriented network. This is what Hubert Zimmermann put in the standard even though he himself helped develop CYCLADES, which had a connectionless network.

The standard was quickly amended to allow either a connectionless or connection-oriented network layer, so this is probably of little historic significance. The point is only that the mainframe ideals came first.

The other thing you need to know is that Ethernet was a Layer #3 protocol, a “Network Layer” protocol.

The Network Layer #3 is defined where a relay receives packets from one link, examines the destination address, then forwards that packet out the correct link in that direction.

This describes an Internet router (or X.25 switch). It also describes an Ethernet switch, because an Ethernet switch is a Layer #3 device.

When Andrew Tanenbaum published his first “Computer Networks” textbook based upon the OSI Model in 1980, he clearly puts Ethernet as Layer #3, because it was.

However, if you Google or ask the AI today, everyone will tell you that Ethernet is Layer #2, part of the “Data Link Layer.”

As mentioned above, it didn’t actually conform to the OSI definition. They had to add LLC to make Ethernet look more like SDLC/HDLC/LAPB, so that it could then carry SNA and X.25 traffic.

In other words, once they decided upon a rigid model with fixed functionality at each layer, they had to make Ethernet conform to its assigned layer, and added LLC to it.

The real model of today’s Internet is that one network may be layered on another. That might mean putting Internet packets inside Ethernet packets inside your home and office, which is common on the edges of the network.

On backbones, we see something else, like a system called MPLS to carry Internet traffic. MPLS is a network technology that itself may be layered on top of Ethernet, giving three layers of networks.

Internet traffic can be tunneled through VPNs, which layers the Internet on Internet, adding yet more ad hoc layers.

The point is that OSI envisioned a fixed, connection-oriented Network Layer #3 that would need something like an SDLC-like Data Link Layer #2 beneath it. It was designed this way because that’s how IBM’s mainframe network worked.

But today’s Internet doesn’t work that way. The Internet Protocol can be encapsulated in anything, maybe within local Ethernet networks (with no LLC), maybe carried by pigeons.

Upper Three Layers (#5, #6, #7)

Back in the late 1970s, the lower 3 (or 4) layers existed as practical, real things that engineers could touch. Hubert Zimmermann had them in the CYCLADES. Xerox had them in its highly influential PUP and XNS standards. They were visible in the emerging TCP/IP standards of the future Internet.

The upper three layers were more theoretical. They roughly existed in IBM’s mainframe networks.

You can’t understand the OSI Session Layer #5 without looking at IBM’s mainframe network stack. The most prominent feature was the fact that some mainframe communications can be “half-duplex,” meaning only one side can transmit at a time. Another prominent feature is that transactions can be “batched” so that they all succeed or fail together, instead of some operations succeeding while later ones fail.

The history of the OSI Presentation Layer #6 is even more difficult to understand.

Back in the day, each computer had its own way of representing data. They would have different word sizes, like 12-bit or 36-bit words. They would have different character sets, like the famous difference between IBM’s EBCDIC and US-ASCII. Structured data would often be written to disk by simply dumping the contents of memory, meaning memory layout was the file layout.

The lifecycle of data was to be created, processed, and eventually destroyed all on the same machine. If it was transferred between machines, it was usually between machines of the same type.

In those early days, the following rule was followed: the format of data was the property of where it was located.

As networking started to mean connecting computers of different types together, this became a problem. You couldn’t simply copy data from one machine to another because it would then have the wrong format. You had to convert it.

Who was responsible for the conversion? The sender? The receiver?

The answer they came up with is that this would be handled by the network stack itself. Networking computers of different types always required data conversion, so of course that should be a layer in the stack.

This was an issue for file transfer, but also for terminals. Different terminals had different character sets but also different control codes for drawing things on the screens. On Unix systems, among the features of the Telnet protocol is to communicate terminal type, so libraries like “ncurses” can be used to send the correct codes to do things like draw boxes on the screen.

At the time, sending the right terminal control codes was one of the most important features of the network for everyone involved in defining the OSI layers. They had terminals on their desktops, not personal computers.

These concerns don’t exist anymore.

The most important change in thinking is that data conversion is the wrong solution. The format of data is a property of the data itself, not where it’s located.

A PDF or JPEG is the same format regardless of what device holds it. Trying to convert data will only corrupt it, especially if one side supports features the other side doesn’t, causing them to be removed. And it’s rare for two types of formats to support precisely the same feature set.

Not only do you not want conversion to be located in the network stack, you don’t want it to happen at all.

In the past, when the lifecycle of data was contained in a single computer, the lifecycle of data today is to be transferred among computers of different types. You might take a picture on your Android phone, send it to somebody using an iPhone, through a server running Windows. Conversion never happens in this sequence of events.

At the top of the network stack is the Application Layer #7. Is this simply the payload that’s above the network, outside of it? Or is this the highest part of the network stack, inside the stack?

Nobody really knows these days.

In the mainframe view of the world, you had services like VTAM and FTAM, the parts of the network stack that dealt with “terminals” and “file transfer” respectively. An application would never transfer a file itself; it would instead request file transfer from the FTAM service.

Therefore, in the mainframe view of the world, the entire network stack is integrated, and applications are fundamentally not really network aware. They require services from the operating system to accomplish goals without being precisely a network application.

That’s not how it worked on the Internet. If you have an email application, it’s very much a network application. It’s not a program that requests services from an email subsystem using some sort of “email API.” Instead, it implements the email protocols itself, using the “Sockets API” to send and receive payloads itself. Its implementation of email protocols is above the network stack—payload, not inside the network stack.

Now, web apps are a little bit different. A lot of programmers use web-specific APIs, both on the client side and server. The “web” is therefore considered an “inside” part of the stack.

But it’s still wrong to think of the web as conforming to the OSI Application Layer #7. It’s an ad hoc layer applied on top of the existing Internet, not something designed to fulfill fixed functionality defined in a specific layer.

For example, in HTTP/1.1, the web protocol runs on top of “Transport Layer Security” or TLS. In HTTP/3.0, TLS capabilities are integrated directly into HTTP—I suppose a sort of “web layer security” instead of “transport layer security”. It’s arbitrary and ad hoc.

The Internet exists as a series of layers, but not the mainframe (and OSI) of fixed layers.

Layers #1 and #4

I don’t really discuss these layers in this post.

Layer #1, the physical layer, transcends this discussion. It existed 100 years before all this nonsense and isn’t mainframe related. It’s just a fact of life: at some point, you connect two things with a wire. In much the same way that your model should include a “payload” above the network stack, you should think of a “physical wire” below the stack, outside of the stack.

Layer #4 is probably the layer that’s actually based upon Zimmerman’s work. It’s definition is that it’s end-to-end — that whatever functionality it provides, it does so on the ends of the network instead of in between. To understand it, look at TCP/IP rather than IBM SNA.

Conclusion

The point is that both stories are correct. ESR is correct in saying that OSI was driven by European academics. But from another viewpoint, it’s really based upon IBM’s mainframe networking and is the antithesis of what we see as the Internet today. The fixed-functional layers was wrongthink that came from IBM. Data Link #2 was IBM’s SDLC. The Network Layer #3 was connection-oriented. The upper 3 layers aspired to reach the wrong goals—mainframe goals.

The entire OSI Model is a lie. It was based upon IBM mainframe thinking and is not the actual model used by the Internet. I know that everything you read, everything you Google, every answer you get from AIs will disagree with me. Nonetheless, everyone is wrong.

The thing to know about The New York Times is that it only quotes people who matter, their fellow elites. You can see that in this story about the recent Louvre jewel heist, where they quote George Clooney. As the star of the Ocean’s 11 heist movies, he matters.

By contrast, technical experts are nobodies, including specialists in museum security in general, and the Louvre in particular.

One reason experts don’t matter is that technical details are boring. The thieves couldn’t simply smash through the Louvre’s windows; they had to cut through them with angle grinders. Such details are important, but dull to the average reader.

The other reason techies don’t matter is that readers already have a ready-made explanation: “They didn’t take security seriously”. No technical information is needed to support this claim.

Such narratives let Times readers feel morally superior about the root cause of the failure while remaining ignorant of how anything actually works. The failure, we’re told, stems from sloth, greed, villainy, or some other moral flaw. Readers already believe this cliché and are simply looking for stories that confirm it.

The reality is that security is hard and full of trade-offs.

For example, the Louvre is an ancient castle, a heritage building. There’s a limit to how much you can harden its windows without damaging or defacing the structure. There’s a reason there are no iron bars or armored panes. Modern art museums often have no windows at all precisely because they’re housed in modern buildings.

There’s also the issue of visitor accessibility. Many visitors are disappointed to find the Mona Lisa behind thick, bomb-proof glass rather than displayed like the other paintings.

Security trade-offs aren’t only about money. But money matters, too. Politicians give the Louvre a fixed budget, and its administrators must choose affordable security measures within that constraint. They’re not greedy or negligent; they’re making trade-offs with limited resources.

Many reports claim the windows, alarms, and cameras were outdated — implying laziness. In reality, upgrading heritage buildings is prohibitively expensive. To criticize those choices responsibly, you’d have to explain where the money would come from.

I’m not a museum expert, of course. I’m just echoing comments from museum-security professionals I’ve seen in online discussions. But I am a cybersecurity expert, and the same reasoning applies there. Non-techies often blame moral weakness for security failures when they’re usually the result of technical and financial trade-offs.

Rather than simple negligence, the real cause is often that the thieves outsmarted the defenders. Statistically, you might have ten brilliant defenders studying decades of heists and covering every expected attack — but thousands of creative thieves dreaming up new ones. The defenders are smarter than any single thief, but not smarter than all thieves combined.

Those defenders will do what they always do: analyze this attack, adjust their plans, and move on. Then the next big heist will happen, and we’ll once again hear that it was because “they didn’t take security seriously”.

And, of course, we’ll get more quotes from movie stars.

In my last blog post, I described how the New York Times doesn’t cite technical experts, but rather “People Who Matter”—people with impressive résumés, like having been the “Director for Cyber Incident Response at the U.S. National Security Council at the White House”, the sort of person who speaks at the World Economic Forum in Davos on cybersecurity.

In this blog post, I double down on this claim, pitting myself (a techy) against Anthony J. Ferrante, the Person Who Matters quoted by the New York Times on yesterday’s UN SIM farm story, who has also been at the center of other stories.

This is the same guy who five years ago did forensics on Jeff Bezos’s iPhone to conclude that Saudi Crown Prince MBS had hacked Bezos. This was widely covered in the press and is generally assumed to be true.

But as a techy, I know the forensics analysis to be flawed. The forensics did not find a “smoking gun” to prove this—they only found unexplained anomalies. That’s a common flaw among forensics people, where anomalies become confirmation bias, where anything unexplained is used to support a foregone conclusion.

In particular, Ferrante’s team was unable to decrypt the video file sent by Saudi Crown Prince MBS to Bezos, the one they assumed contained malware.

They couldn’t decrypt the video due to lack of technical expertise. The file was end-to-end encrypted with WhatsApp. End-to-end means that nobody in the middle could decrypt the file, not even Facebook’s WhatsApp service itself. However, if you had one of the phones on either end, then you could decrypt it. Bezos’s iPhone had the decryption key—techies just needed to know where to find it.

The tools Ferrante used (like Cellebrite) are well known and respected in the forensics industry, but the tools didn’t have the ability to automatically decrypt WhatsApp. Forensics people are usually only trained on using tools effectively, not on how such things work underneath. When the tool doesn’t do it, they can’t do it themselves. Hence, they couldn’t decrypt the file.

In a blog post I wrote at the time, I described in great detail how techies can decrypt the file. I sent the same video to myself, performed forensics on my iPhone, then described step-by-step how to find the decryption key and decrypt it. I wrote code to help. Any competent forensics techy can follow the instructions and do the same.

Ferrante’s team could have done this and could have conclusively proven malware if it was there. But they didn’t have the knowledge. Instead, they had to rely upon guesses, using unexplained anomalies for their conclusions.

One of the anomalies was that the encrypted file was slightly larger than the original video, by 14 bytes. Ferrante attributed the difference to malware code. As my techy blog post explains, this is an artifact of encryption, which prepends a 10-byte “authentication code” on the front and appends 4 bytes of “padding” on the end. The reason for the anomaly is conclusively explained—the file was exactly the size it was supposed to be. If there were malware code inside, it was carefully constructed to be the same size.

The biggest unexplained anomaly was network traffic. Soon after the message from Saudi Crown Prince MBS was received, the phone appeared to send out bulk data, as if it were stealing all the photos and messages on the phone.

As another of my blog posts explains, such anomalous traffic is normal for Apple iPhones. They don’t smoothly log how much is being transmitted day by day. Instead, they can keep track of traffic for months and then create a single log entry with a large number. That’s what my Uber app did—slowly uploaded data (likely location information) day by day for months, and then the day I closed the app, reported 56 megabytes of upload.

This doesn’t conclusively explain the anomaly Ferrante’s team saw, but it does demonstrate that their conclusions aren’t really warranted. Abnormal iPhone traffic is, well, normal. This is the typical confirmation bias that plagues forensics.

I’m not claiming Ferrante and his team are incompetent. They delivered the standard results you get from standard forensics, which were inconclusive. Their conclusions were unwarranted, but that’s what the client wanted. The client wanted reasons to blame Saudi Crown Prince MBS.

My skills here are pretty rare. You wouldn’t expect the average forensics team to have known them. But they aren’t that rare. There are plenty of small, boutique firms that could have done this, but they are run by techies. They aren’t run by People Who Matter, like former members of the FBI and National Security Council. A billionaire like Bezos only wants People Who Matter, so wouldn’t get such techies.

The New York Times is a paper by the (self-appointed) elite for the elite. Their readers expect to hear from those with pedigrees, with FBI, Security Council, and WEF credentials. They don’t care how impressive a techy GitHub account is (mine is very impressive, by the way)—techies don’t matter. If techies had something useful to say, they’d tell their manager, who’d tell their manager, who’d tell the CEO or VP, who’d then get quoted in the New York Times.

As a result, you get these stories from the New York Times that aren’t about truth, but narratives. Bezos and his security chief Gavin de Becker wanted a story about Saudi MBS hacking. The Security Service wanted a story about how they foiled a plot against the UN. They matter, so their stories matter—even if I, as a techy, know them to be bogus.

Today, the Secret Service announced they foiled some big national security threat. Major news organizations (e.g. NYTimes) have repeated their claims without questioning them.

The story is bogus.

What they discovered was just normal criminal enterprise, banks of thousands of cell “phones” (sic) used to send spam or forward international calls using local phone numbers. Technically, it may even be legitimate enterprise, being simply a gateway between a legitimate VoIP provider and the mobile phone network.

The backstory is a Secret Service investigation into threats sent to politicians via SMS messages. The miscreant used one of this spam farms to mask their origin. When the Secret Service traced back the messages, using radio “triangulation” (sic) to find the mobile phones, they found these SIM farms instead.

One of the reasons we know this story is bogus is because of the New York Times story which cites anonymous officials, “speaking on the condition of anonymity to discuss an ongoing investigation”. That’s not a thing, that’s not a valid reason to grant anonymity under normal journalistic principles. It’s the “Washington Game” of “official leaks”, disseminating propaganda without being held accountable.

The Secret Service is lying to the press. They know it’s just a normal criminal SIM farm and are hyping it into some sort of national security or espionage threat. We know this because they are using the correct technical terms that demonstrate their understanding of typical SIM farm crimes. The claim that they will likely find other such SIM farms in other cities likewise shows they understand this is a normal criminal activity and not any special national security threat.

Their official statements are obvious distortions, like being within 35 miles of the UN building. Their unofficial statements are designed to exaggerate even more, like “never before seen such an extensive operation”. The Secret Service doesn’t normally investigate such crime, so of course they are unlikely to have seen such an extensive operation.

Another way you know that the NYTimes is lying is because of the independent “experts” they quote to confirm it.

For decades now, when the NYTimes has a cybersecurity story from anonymous government officials, they quote James A. Lewis to confirm it. This guy used to work for CSIS (Center for Strategic and International Studies) but apparently has changed employers recently. Whenever I blog/tweet about bogus NYTimes cybersecurity stories, I point out this relationship with James Lewis. When you see anonymous government officials and James Lewis quoted in a NYTimes story, you are seeing government propaganda.

Another “expert” the NYTimes quotes is Anthony Ferrante [update]. He’s got the resume that the NYTimes loves. I’m famous among hackers for my technical expertise, but I would never be quoted in the NYTimes, because I don’t matter. The NYTimes only quotes people who matter, meaning, people involved at high levels of government, people with their resume posted on WEF.

Both of these “experts” claim things that are objectively silly. Ferrante says “my instinct is this is espionage” and “could be used for eavesdropping”. This is false, this arrangement cannot be used for eavesdropping and there’s nothing particularly related to espionage here. Lewis claims “only a handful of countries could pull off such an operation, including Russia, China and Israel”. That’s false, I can pull this off, personally. It’s just a SIM farm. Sure, there’s some capital involved, on the order of $1 million, but it could be setup and managed by a single person. It likely wasn’t setup all at once with that much money, but has been slowly growing for years as profits are funneled back into setting up more SIM accounts

Who are you going to trust, these Washington insiders, “people who matter”, or an actual hacker like myself?

I say “phones” above in quotes above because the actual hardware isn’t like the phones you have in your pocket. Your Android/iPhone is a computer with a single “baseband” radio that talks to the cell tower, and maybe two SIMs in case you have two different phone accounts. That’s what a SIM is — a chip that locks you to a specific phone account.

A “SIM box” has single computer (often running Linux), maybe 20 baseband radios, and maybe 100 physical SIM cards. It rotates among the SIM accounts when spamming SMS messages.

A SIM card may be the same sort of prepaid $10/month SIM you buy at Walmart that allows 1000 SMS messages per month. There are other types of accounts they might use, so they aren’t necessarily walking out of a Walmart with bags full of prepaid SIMs after clearing off the shelves, but it’s close enough. They are trying to fly under the radar, appearing to the mobile networks as normal users.

The Secret Service hypes this as some sort of national security threat that can crash cell towers. The reality is that this is just a normal criminal threat that sometimes crashes cell towers. SMS is an ancient technology that works slowly even in modern cell networks. Too many SIM boxes spamming SMS in one location can indeed overwhelm a cell tower. You actually don’t need a bunch of SIM boxes to do it — you can sometimes crash a cell tower with a single baseband radio. Ask me how I know.

The point is: while criminals do sometimes crash or overload cell towers, an actual foreign threat can do this much easier than using SIM farms. In any event, there are thousands of cell towers around New York City satisfying 10 million subscribers, so crashing a few won’t make much difference.

The correct quote from any expert is that this looks like a normal criminal SIM farm, that’s used for a wide range of purposes, often SMS spam. They are pretending to be thousands of normal mobile phone users to prevent the mobile phone companies from shutting them down. Some miscreant likely used the service to hide the origin of threats sent as SMS messages to politicians, which is why the Secret Service is involved. Theres no evidence the Secret Service is involved due to some actual national security or espionage threat — that’s just propaganda they are hyping.

35 miles radius centered on UN building:

Some comments on this blogpost;

On Hacker News there’s an article entitled a Brief History of Threads, but it really isn’t. I thought I’d write my own history.

What is a “thread”?

What we are talking about are kernel threads. A single process has one or more threads of execution that all share the same process resources, like file handles and memory space, but they have their own unique stack and program counter (the thing that points to the code being executed). These threads are scheduled separately by the operating-system with preemptive multitasking and running on multiple CPUs.

In the old days, a process had only one thread. If you needed to do something that required multiple threads, you created multiple processes. This was resource intensive.

We are not talking about userspace threads. I need to discuss this first because there is a lot of confusion about this. For example, the original article claims that threads first appeared back on IBM S/360 mainframes in the 1960s. That’s not true. Userspace threads appeared there, not kernel threads.

A userspace thread where you create additional stacks in userspace and use cooperative multitasking to switch between them. You can do this in basic C code by using malloc() to allocate memory for another stack, and setjump() and longjump() to swap stacks. You call some sort of yield function to cede control back to a scheduler, which swaps stacks and continues executing another userspace thread. This yield function is often hidden behind APIs, so you call functions without really paying attention to the fact that it might yield control.

Such userspace threads are damn useful. For example, the Java runtime supports “green threads” that does this for you underneath. On network programming, when waiting to receive incoming data, your thread needs to stop and wait. In the Java runtime, it can transparently switch userspace threads without interacting with the kernel. You get most the benefits of kernel threads, but greater efficiency in userspace.

Likewise, the Go language “goroutines” are essentially userspace threads. People love the Go programming language because of the ease in creating scalable network services.

Userspace threads are cooperative multitasking. If they sit in an infinite loop without either yielding control, or calling a function that transparently yields control, then no other userspace thread will run. This can be bad — writing code for userspace threads means writing code differently.

Kernel threads are very different. They use preemptive multitasking, for one thing, so they can interrupt infinite loops.

In the past, before kernel threads, the kernel would maintain a list of all running processes. A timer chip would provide an occasional interrupt, like every 10 milliseconds. This would save the state of the current process, then transfer control to the kernel. The kernel would look through the list and find some other runnable process, then transfer control to that process. In that way, a single CPU could appear to run multiple processes at the same time.

Kernel threads is the same concept, except now a process can have multiple threads of execution. The scheduler now uses a table of all threads to choose from instead of all processes.

Some early implementations simply used the same process table, but allowed processes that would share address space and resources, making them effectively the same thing as threads. A process with two threads would then appear as two entries in the process table.

The point is that kernel threads are different than userspace threads. Userspace threads have always existed, kernel threads are a relatively recent invention.

We normally discuss threads as being a lighter weight alternative to forking a full process, but we should also see them has a heavier weight alternative to userspace space threads.

There are two histories here.

• One is the history of full operating systems, like Unix, that had memory protection and preemptive multitasking. They had heavy weight processes and progressed to adding threads.

• The other is the history of toy operating systems (from the modern perspective) that only had cooperative multitasking with userspace threads, like Windows 3.0, and Macintosh prior to MacOS X. These eventually grew up to full operating systems

OS/2 (1987)

The first mainstream operating-system with threads was OS/2 released in 1987 for the Intel 80286 processor.

This operating-system is largely forgotten by history, but was a seminal development at the time. Modern Windows is based upon Windows NT, which is in turn based upon OS/2.

Around 1987, the world was dominated by MS-DOS for PC-compatible desktop computers. This was a toy operating-system. There were versions of Unix available for the PC, like Microsoft’s own Xenix, but they were unsatisfactory for a number of reasons, the most important of which is that it couldn’t run MS-DOS programs (backwards compatibility).

IBM paid Microsoft to develop OS/2 — a full operating system with backwards compatibility with MS-DOS.

OS/2 had all the features of a full operating system, meaning memory protection and preemptive multitasking. It ran multiple copies of MS-DOS simultaneously, though only one could be “full screen” at a time.

There are lots to be said about this, but the thing to remember is that the first 1.0 release of OS/2 had threads as we know them today.

I’m not sure why OS/2 had threads. The main justification for other operating systems was multi-CPU support, because a programmer would want to split one process across multiple CPUs, and each CPU would need their own thread.

Mach and NeXT (1989)

The “Mach” kernel was a project from 1985 with a bunch of innovations. Among its features were multi-CPU support, and hence, threads.

It was first released as part of a commercial operating-system in 1989 with Steve Jobs’s “NeXT” computer. If you’ll remember, Steve Jobs had been fired from Apple in the 1980s, so he created a competitor company, NeXT. It was based upon the same CPU as the Macintosh, but with a Unix operating system. That operating system’s kernel was based upon Mach, with a BSD shell around it.

In 1996, Apple purchased NeXT to be the basis for the next generation Macintosh, where the operating-system was renamed MacOS X (today, “macOS”). Though MacOS X wasn’t released until 2001, conceptually, it’s had kernel threads support since 1989.

BeOS (1991)

Some other former Apple engineers split off to create an alternative, called BeOS. It also supported multiple CPUs and hence, multiple threads.

The thing that both NeXT and Be were chasing was the fact that Apple was failing with its “Pink” operating system. Everyone knew that personal computers had to move from toy operating-systems (MacOS, MS-DOS, Windows) to a full operating-system (like Unix). But at the same time, big companies always struggle with a v2.0 rewrite of their successful project. Apple had started their “full operating-system” project in 1988 called “Pink”, and it was already flailing in 1990.

Engineers knew they could split off and form a small team that could beat Apple to the goal, even though Apple already had 2 years of development behind them. They knew that Apple would eventually give up and buy a competitor.

Which is exactly what Apple did, though they chose NeXT over Be, making BeOS a failure. But engineers still consider it an elegant system.

The signature feature of BeOS was getting multi-media done right. Getting low-latency, reliably streaming audio from a computer is a surprisingly difficult job, which is why Linux still struggles at it. Since this was important to consumers, BeOS solved that first, and sorta built the rest of the system behind it. Threads and multi-CPU support were integral to this.

Solaris (1992)

Sun Microsystems was the most aggressively innovative Unix vendor around 1990. Their bombshell release of Solaris 2.0 in 1992 came with multi-CPU and threads, though they didn’t provide much support to developers until Solaris 2.2 in 1993.

It’s important to understand this in context of the Unix wars of 1988. All the vendors supported slightly incompatible versions, making it difficult to write software for one vendor’s Unix that would run on another.

Therefore, multiple standardization efforts were started.

• One effort was POSIX, which simply standardized the APIs. This wasn’t highly regarded at the time, as people believed they needed to share the operating system code to be truly compatible.

• One effort was “System V release 4” or “SVR4”, which combined the code from AT&Ts SRV3, BSD (including SunOS), and Microsoft Xenix into a single operating-system.

• A third effort was “OSF/1”, based upon the Mach microkernel, in the same fashion as NeXT chose.

In terms of number of licenses, Xenix was the most important. It accounted for half of all Unix licenses. That’s because it ran on cheap PCs. But in terms of revenue, the biggest vendor was Sun. It was first to market with RISC computers in 1987 and was cleaning up. Both of these were in the SVR4 camp.

The rest of the vendors (not AT&T, Xenix, or Sun) joined the OSF/1 camp, including leaders like IBM, DEC, and HP.

Sun had released it’s first multi-CPU computers in 1991 using the older SunOS, but that kernel didn’t really support them well, having just a single kernel lock. In other words, the kernel was single CPU only, while userspace code could run on multiple CPUs. The hardware wasn’t really effective until Solaris 2 was released in 1992.

OSF/1 (1992)

DEC was really the only vendor that truly adopted OSF/1 after the Unix wars. The rest of the Unix vendors ended up simply extending their Unix variants with POSIX support.

While DEC’s first OSF/1 systems didn’t support multiple CPUs, they were still based upon the Mach kernel, and hence, supported threads.

Windows NT (1993)

After the wild success of Windows 3.0 in 1990, Microsoft decided to move away from OS/2, which wasn’t compatible with Windows. They instead created a new operating system from scratch with the intent of being Windows-compatible.

In 1993, Microsoft released Windows NT 3.0, a full operating system. The weird version number “3.0” was to drive home the point that consumers should see it as just another version of Windows. It looked and felt a lot like the Windows 3.0 they were accustomed to, while being a completely different thing — a full instead of toy operating-system.

The core operating system was based upon OS/2, DEC VMS, and some Unix features. It had backwards compatible sub-systems for MS-DOS, OS/2 1.3, and POSIX. Most importantly, it could run Windows 3.0 apps, inside what they called a “Windows on Window” subsystem or “WoW”.

WinNT inherited the concepts of kernel threads from OS/2, but more than that, Microsoft aggressively used threads throughout the operating system, and encouraged programmers to use them. It was also aggressively multi-CPU, so would likely have invented a version of threads even if it hadn’t inherited them from OS/2.

POSIX threads (1995)

The enduring legacy of the 1988 Unix wars is the POSIX standardization of the API. While all these companies intended to create a unified operating-system that each would license, they ended up just continuing to support their legacy product, adding POSIX compatibility as necessary.

When IEEE POSIX 1003.1c was standardized in 1995, some vendors already had kernel threads, some didn’t. For those who had kernel threads, they were often not quite compatible with the POSIX API.

For example, Linux’s first version of threads was in 1996, though it didn’t fully support the POSIX standard until 2003.

The BSD’s were even later than Linux. FreeBSD didn’t get it right until 2006.

M:N vs. 1:1 threads

As explained at the top, userspace threads are different than kernel threads.

A lot of early thread support tried to mix them behind a single API, called M:N threads. This meant there could be more of these userspace threads than kernel threads. That meant having to do both cooperative multitasking in userspace and preemptive multitasking in the kernel at the same time.

This was a broken idea, and everyone eventually gave up on it.

The straightforward kernel threads model you know today is called 1:1, meaning that as far as the kernel is concerned, for every kernel thread, there is only one userspace thread. You can still subdivide that userspace thread into multiple, cooperative multitasking userspace threads, but it something you do on top of an API like POSIX threads, not as part of it.

That’s why FreeBSD didn’t support POSIX threads until 2006, because they spent many years prior to that trying to get M:N working. This caused problems making the simpler 1:1 POSIX model work properly.

They failed to learn the lessons of history. Solaris had initially added M:N threading in 1992, and by 2002, gave it up, and went to the simpler 1:1 threading model. But in 2003, FreeBSD first added the M:N threading model, before they, too, gave it up in 2006.

Engineers love the “more powerful is better” idea of programming, when the reality is that “simpler is better”.

The upshot is that you shouldn’t even have to learn this M:N concept because it’s been abandoned. But yet, any discussion of the history of threads is going to talk about it.

Conclusion

The history of userspace threads goes all the way back. The toy operating-systems of the 1980s, like MacOS and Windows used them heavily with cooperative multitasking.

But for kernel threads, the history really goes back to OS/2 in 1987 and the first commercial Mach release (NeXT) in 1989.

Solaris had the reputation of the leading, most advanced operating-system of the 1990s, but it’s the heritage of Windows and macOS that did kernel threads first — and did them right.

The creator of Linux, Linus Torvalds, is famous for his rants. This latest rant is linked below. As this is my bailiwick, where I am an expert, I thought I’d comment.

https://lore.kernel.org/lkml/CAHk-=wjLCqUUWd8DzG+xsOn-yVL0Q=O35U9D6j6=2DUWX52ghQ@mail.gmail.com/

tl;dr: He’s right on some things, and wrong on other things.

What I bring to this debate are:

1. actually looking at the code, browsing git commits

2. perspective of a very senior programmer who has been through these debates before

Let’s read the code

The pull request added two macros in a core Linux file <linux/include/wordpart.h>, which is in turn included within <linux/include/kernel.h>, one of the most central files within the Linux kernel. The added macros are highlighted in bold below:

#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define lower_32_bits(n) ((u32)((n) & 0xffffffff))
#define upper_16_bits(n) ((u16)((n) >> 16))
#define lower_16_bits(n) ((u16)((n) & 0xffff))
#define make_u32_from_two_u16(hi, lo)	(((u32)(hi) << 16) | (u32)(lo))
#define make_u64_from_two_u32(hi, lo)	(((u64)(hi) << 32) | (u32)(lo))
#define REPEAT_BYTE(x)	((~0ul / 0xff) * (x))
#define REPEAT_BYTE_U32(x)	lower_32_bits(REPEAT_BYTE(x))
#ifdef __LITTLE_ENDIAN
#  define aligned_byte_mask(n) ((1UL << 8*(n))-1)
#else
#  define aligned_byte_mask(n) (~0xffUL << (BITS_PER_LONG - 8 - 8*(n)))
#endif

The thing to note about this is that their definition of make_u32_from_two_u16() is wrong. It casts the low part to a (u32) instead of a (u16). If the original variable has high-order bits (such as by being signed and negative), it’ll incorrectly put those bits in the upper part. In other words, make_u32_from_two_u16(0,-1) will be 0xFFFFFFFF.

The definition should’ve looked like, making it (u16)lo.

#define make_u32_from_two_u16(hi, lo)	(((u32)(hi) << 16) | (u16)(lo))

This is a pernicious sort of bug. It’ll almost never show up, and is impossible to test for.

But when the bug does appear, it’s impossible to fix. That’s because you can’t be sure that some other code doesn’t depend upon this bad behavior combining two (u32) integers. Its behavior becomes cast in stone.

If you are going to add such a utility function that’s visible from the entire Linux kernel, then you’d damn well get it right. They got it wrong.

In any case, this macro was added purely to support the definitions in <include/linux/mailbox/riscv-rpmi-message.h>.

#define RPMI_VER_MAJOR(__ver)		upper_16_bits(__ver)
#define RPMI_VER_MINOR(__ver)		lower_16_bits(__ver)
#define RPMI_MKVER(__maj, __min)	make_u32_from_two_u16(__maj, __min)

This RPMI_MKVER() macro is then used in the file <drivers/clk/clk-rpmi.c>.

if (msg.attr.value < RPMI_MKVER(1, 0)) {
    return dev_err_probe(dev, -EINVAL,
     "msg protocol version mismatch, expected 0x%x, found 0x%x\n",
				     RPMI_MKVER(1, 0), msg.attr.value);
}

This sort sort of packing version numbers into an integer is incredibly common and you see the same coding pattern across thousands of drivers and libraries. A version number like v1.0 is encoded as 0x00010000. The above snippet of code is one of many common ways of dealing with the problem.

There are other common patterns. An alternate solution might be the following, getting rid of the whole hierarchy of macros.

#define RPMI_VERSION_1_0 0x00010000u
if (msg.attr.value < RPMI_VERSION_1_0) {
    return dev_err_probe(dev, -EINVAL,
     "msg protocol version mismatch, expected 0x%x, found 0x%x\n",
				     RPMI_VERSION_1_0, msg.attr.value);
}

(I like this solution better, by the way.)

What we see here is a common hubris among programmers. When they see there is no standard solution for a common problem, where programmers solve it differently, often poorly, they decide that other programmers need to be told how to do it in one standard manner. Hence, they put their macro in a global file included throughout the kernel instead of confining their changes to just their own code. It’s such a fine, beautiful thing, everyone should use it.

This is where technical debt comes from, namely, programmers designing for future code reuse. There is a toxic culture centered around reusing code, such as “DRY” or “Don’t Repeat Yourself”. Instead of simply solving the problem at hand, programmers try to create general solutions first, then use that generic solution to solve their specific problem.

The cost of this approach is demonstrated above, where a bug becomes cast in stone, where it cannot be fixed without knowing whether this introduces a new bug.

The better way of programming is to just solve the problem at hand as simply and narrowly as you can. That’s why I’d just create a RPMI_VERSION_1_0 constant. It’s even better at communicating intent, without introduce a whole lot of technical debt from multiple layers of macros.

Where Torvalds is wrong

Linus makes some strong points, which I expanded upon in the above section. But he’s wrong in a lot of other ways.

First of all, the use of such macros to hide details is personal preference. There is a constant ying-vs-yang in such situations where sometimes your preference is to see the raw details, and sometime your preference is to see the intent rather than details. The same programmer might have different preferences at different times.

I, too, have cursed programmers for unnecessarily obfuscating trivial details. If voodoo worked, I’d have a shelf of programmer dolls to stick pins into. But at the same time, when writing code, I’ve decided that it’s more appropriate to show intent than details, so am guilty of this myself. (I had a manager once complain about this in my code.)

Linus has more reason to complain about polluting the global namespace here. The <wordpart.h> file is included in pretty much every source file of the Linux kernel, so that’s a huge impact for something that’s only actually used in one place.

On the other hand, this isn’t strictly policed. As we’ll see below, it’s the evolution of the kernel. When people solve a common problem, they make a solution commonly available, and the amount of stuff included through <include/linux/kernel.h> grows over time. Linus should have a stricter police against people letting their shit taint the overall kernel, but he doesn’t.

One of Linus’s complaints is that the name of the macro doesn’t communicate which side is the high portion, and which is the low. Okay boomer. The macro isn’t defined using the parameters (a, b) as Linus suggests, but (hi, lo), clearly communicating what Linus says is missing. Modern editors show macro parameters, such as in the screenshot from VSCode:

What we see in this screenshot is how the editor pops up a display, where not only can we see the order of parameters, but also the implementation of the macro. Which is high and which is low is clear and unambiguous, just not in the name.

It’s unreasonable to expect code to be written as it was back in the days of the original Unix, which was largely written using paper teletypes and the ed program. We have modern IDEs and should use them.

Linus discusses the implementation of the macro, pointing out (as I did above) you need a (u16) cast to prevent “high bits from polluting the result”. But he describes the macro as:

(a << 16) + b

Don’t mix operator families like this, even with parentheses. The correct macro should be consistent. Either the addition + should be changed to an OR |, or the shift << should be changed to multiplication *.

(a << 16) | b
(a * 0x10000) + b

It’s a small thing, and one of personal preference, I guess. But there are a ton of bugs in code that come from mixing operator families. Programmers understand the order of precedent of related operators, but not of unrelated ones. Moreover, it’s a code purity thing here. What’s being done is manipulating bits, not performing arithmetic.

All this minor stuff is really just a warmup to the macro immediately preceding make_u32_from_two_u16() in <wordpart.h>:

#define lower_16_bits(n) ((u16)((n) & 0xffff))

This should blow your mind if you are a programmer. Combining two integers isn’t the most trivial macro imaginable, this extraction of the lower 16 bits is.

This is so incredibly unnecessary it’s hard to imagine it exists, or is even used. Yet, 15 files in the Linux kernel use this macro.

It was added in June of 2021 to <include/linux/kernel.h> in an otherwise simple pull request. Last year, this family of macros was moved to the newly created <wordpart.h> file, which <kernel.h> then included. We can see the evolution over time as each of these macros was added to the kernel, and then the uptake as people started using them.

If such trivial macros already exist, it’s not unreasonable to add more of the same type.

Combining smaller integers into a bigger one is an extraordinary common task. For example, Microsoft has a MAKEDLONG() macro that’s used extensively throughout Windows. As mentioned above, programmers can actually get it wrong by not masking off high bits. It’s not unreasonable for the Linux kernel to have a standard definition of this to reduce people getting it wrong.

Linus is correct that polluting the entire kernel with a bad make_u32_from_two_u16() is bad, and the programmers should feel bad. But at the same time, he’s wrong on other stuff.

Linus is very nice

Rather than substance what offends most people is his tone. Calling somebody’s code “garbage” and “get bent” certainly doesn’t sound like nice language.

But, in fact, he’s extremely nice.

Bluntness is a virtue. There is no polite way to communicate the importance of the issue.

The rudeness here was from programmers who were so impressed by the beauty of their own code that they decided to pollute the entire kernel with it, telling everybody else how they should solve the same problem in their own code.

It’s a hubris that infects all big engineering organizations and big open source projects. There are a lot of relatively junior programmers who want nothing more than to tell everyone else how to code. It’s a constant battle pursued by passive aggressive politeness.

The notable attribute of Linus’s bluntness is that it’s not actually personal. It certainly sounds personal the way he attacks their baby, but it’s not. He makes it clear that he welcomes their further contributions. This code is garbage, not the people who wrote it.

Conflicts in engineering organizations and large projects are usually personal. That Linus makes this not-personal is a big tribute to him.

His issues have substance. While I argue above that it’s more a personal preference, there’s actually a lot of history and complexity behind the issue.

The easiest programmers to deal with are those who are direct, on point, and who don’t hold personal grudges. Even if their language is sometimes offensive, it’s far better than the norm of passive aggression and personal grudges.

Linus Torvalds is God. His time is a lot more valuable than yours. That he takes the time to teach you to be a better programmer is incredibly gracious. If he had to take the time to figure out how to communicate the level of garbage in your code without using impolite terms like “garbage”, he wouldn’t take the time at all. Be thankful.

Conclusion

I find the debate here tedious.

People aren’t discussing the code itself, which is why I went through and linked the files so you can easily see the code and what’s really happening.

People take the story as given, without thinking in terms of the global context. There are larger debates here about coupling, technical debt, and programmers trying to promote their solutions.

For me, the fatal flaw is masking off the high bits, such as with (u16)lo. That’s just bad programming and technical debt that the original programmers should feel real bad.

But I’m easy-going on the rest of it. It’s a constant battle between programmers wanting to communicate intent rather than implementation details, and the desire of programmers (especially senior ones) to see more of the implementation details. There’s a common problem here, and it’s not unreasonable to want a common solution.

My blogpost debunks one of those forensics efforts, namely an analysis of the dumped “HRC_pass.zip” file described in following blogpost: https://theforensicator.wordpress.com/2019/05/27/guccifer2-used-thumb-drive-in-us-ctz/

Their blogpost claims forensics show that Guccifer 2.0’s computer is located in the Central timezone of the United States, and that files were transferred to that computer from a USB thumbdrive, proving Guccifer 2.0’s physical location was in the United States and not Russia. (EDIT: original version claimed this connects with the Seth Rich conspiracy theory, it’s been pointed out to me this is wrong, that this person doesn’t believe in that theory).

Transfer rates

The major claim of this blogpost is that timestamps in HRC_pass.zip show the files were transferred at a rate of 15 megabytes/second to the hacker’s computer. They claim this means a USB thumbdrive was used, which transfers files at that speed. The fact a USB thumbdrive was used means the person transferring the files was physically located next to the computer, instead of operating remotely from Russia.

The authors of that forensics blogpost prove USB transfer speeds by reproducing them. They transfer the files using their own USB drive and find it’s near the 15 MBps speed. They show the following screenshot produced by Windows when it’s doing long copies.

That this must have been a USB drive can be debunked by showing that other transfers happen around the same speeds.

For example, in the picture below, I copy the files from a local server (QNAP RAID6 NAS) to my local machine. As you can see, the transfer rate is ~15 MBps.

Another way of copying files is extracting them from a ZIP archive, which is slow even on fast computers because of decompression. I extracted the DNC_pass.zip archive on my computer, and found it also ran at roughly these speeds:

(This is significantly slower than extraction on my MacBook Air, I haven’t figured out why. I suspect one reason is that I’m using a rotating disk drive.)

This second item is a highly likely source of the forensicked transfer speed. It’s typical for hackers to store such things in archives — unencrypted. But DNC_pass.zip was encrypted. To encrypt files in an unencrypted archive, you have to first un-archive/extract them, then re-archive them with a password.

A typical scenario would be that Guccifer 2.0 already had HRC_pass in an archive (ZIP, RAR, 7zip), and that what we see here is the actions of extracting all the files at that ~15 MBps speed, and then re-archiving them to create HRC_pass.zip.

I’m not asserting this is what happened. I’m only asserting that it’s extremely typical. It’s as likely as any other scenario, such as copying from a USB drive.

USB FAT timestamps

Another thing that points to a USB drive, according to them, is that file timestamps are “even”. By “even timestamps” we mean timestamps, down to the second, have an even digit at the end, like 8:53:06 rather than 8:53:05.

Back in the 1980s, Microsoft designed a layout for their disk drives called “FAT”. As a consequence of using a 16-bit number, they couldn’t quite record all the seconds in the day. By rounding up to an even number, the reduce in half the number they need to represent, which fit within 16-bits (Namely, 16-bits have 65,536 combinations, which isn’t enough for all 84,600 seconds in a day, but enough for 42,300 rounded seconds).

This timestamp format changed by the mid-1990s for their newer Windows systems, but legacy lasts a long time. All flash USB drivers smaller than 32 gigabytes continued to use the FAT layout through 2016. (Larger thumbdrives use the update exFAT format, which not only allows larger drives, but also fixes the timestamp problem — something to remember if you try to reproduce this using your own thumbdrives).

The theory is that if you see a bunch of files that all have rounded timestamps, then you know that they were transferred via a USB drive. That’s what the “forensics” of many of the Guccifer 2.0 dumps show, such timestamps.

There are two major flaws with that theory.

The first is ZIP archives can also show this effect. ZIP was created in 1989 and used the same timestamp format as FAT. The ZIP standard has since been updated to support better timestamps, down to the millisecond, but some ZIP software still uses the old standard.

In particular, the “Compress to…” feature on Microsoft Windows (even still with Windows 11) uses this old format. If you compress files with it to create a ZIP archive, then extract all the files, all the odd timestamps will be rounded up a second to an even timestamp.

You can easily reproduce this on your Windows desktop.

The second major flaw in the forensics theory is that USB (or ZIP) copies didn’t need to happen just at the moment the files were stolen. It could’ve happened at any time in the lifecycle of the files. Employees often exchange files with USB drives and ZIP archives within an organization. If you hack into a victim and steal their files, there’s a good chance a lot will have these rounded timestamps.

When you see these rounded timestamps, you can’t be certain when and where rounding happened. It could’ve been by an employee exchanging files with another employee using USB. It could’ve been a remnote hacker who used Window’s built-in “Compress to…” feature.

This condition is certainly a valid clue, one that can guide investigation, but by itself, it’s not evidence of any sort. There’s no way one can look at this and say the most likely explanation is an insider stealing the data with a USB drive.

Central timezone

Guccifer 2.0 might’ve used a USB drive, but the key to this forensics blogpost is that the computer was in the Central timezone of the United States. The combination of a physical USB drive and physical location means Guccifer 2.0 must have been located in America and not in Russia as claimed by CrowdStrike or FBI’s analysis.

The claim of the Central timezone isn’t bad, but it’s not as reliable as you think. It’s common for computers, especially servers, to have the “wrong” timezone configured.

A server services users from all sorts of timezones. They are often managed by remote IT workers located in different timezones as well. Thus, the idea of of a timezone matching the physical location of a server is somewhat meaningless.

But it’s still a strong clue. It’s not really evidence that CrowdStrike/FBI are lying to you, but is still something that can’t be easily ignored.

But here’s the thing: the 2018 GRU indictment clearly describes how Fancy Bear exfiltrated the data to a computer in Illinois, which is located in the Central time zone. This forensics blogpost even mentions the Illinois computer — but without noting that it’s located in the Central timezone.

Far from disputing the narrative in the DoJ’s GRU indictment, this timezone clue confirms it.

Transfer Rates #2

Another section of that document makes the claim:

“Extremely Slow Acquisition Times, Run Counter to Claims in the GRU Indictment”

They are referring to the second set of timestamps. The one above is the “Creation Time” from which we can calculate how fast the files were transferred/extracted onto Guccifer 2.0’s machine. The second is “Last Modified” timestamp, which they claim is the timestamp where they copied the files off the DNC machines.

That problem is that files have a life cycle with many repeated copy events. These “Last Modified” timestamps could reflect a copy by employees, or one of several copy events during the acquisition of the files — rather than the specific acquisition step they are talking about

Fancy Bear reportedly broke in on April 18, this copy event has timestamps from April 26, and they published the files on June 21. It’s reasonable to expect that the hackers are somehow responsible for these timestamps.

But we can’t possibly imagine why.

It doesn’t run counter to the GRU indictment. Acquisition of files often involve multiple copying steps. There could’ve been a slow copying step to get files to where the hackers staged them, and then copied at a fast speed when exfiltrating. There is therefore no conflict with how the GRU indictment describes things.

The forensics discussions speculates that X-Tunnel is sensitive to high-latency, which can introduce delays when copying small files that look like slow transfer speeds. But that’s true of a lot of protocols. A good example is the SMB protocol, the one that’s used for Microsoft servers.

The point is that we can’t conclude anything. Despite this, that forensics blogpost insists on a conclusion (Conclusion #2). All that can be concluded is that this mass copy of files happened on April 16 and nothing else, especially not that it’s the copy event we are thinking of (like an exfiltration copy).

Conclusion

A typical scenario that explains all the forensics is that Fancy Bear created a compressed archive on the hacked computers at the DNC, exfiltrated that archive to the computer in Illinois over then Internet, then extracted the contents onto that drive. They then re-archived those files in an encrypted ZIP to produce DNC_pass.zip. It matches all the forensics evidence from that that blogpost and also all the evidence from CrowdStrike and that 2018 GRU indictment.

By “typical” I don’t assert any likelihood, other than it’s as least as likely anything the authors of that blogpost claim. I have no idea what actually happened.

There are a few reasons why these people got this so wrong.

The biggest reason is confirmation bias. Human nature is to jump to a conclusion, and then look for evidence that confirms the guess. It’s extremely common in computer forensics. The Durham report notes that biggest reason for that 2016 Russian collusion investigation by the FBI was due to extreme confirmation bias, it’s a common problem.

A related problem is the need for cognitive closure. That forensics document is attempting to pick the most likely scenario and defend it as the correct answer, rather than admitting they simply don’t have a clue as to the cause of what they see. They demand that we must have a cause.

Another reason is hubris, being impressed with your own cleverness uncovering secrets. It’ll also impress other techies, who’ll be proud of their ability to understand it. These “Last Modified” and “Creation Time” timestamps are actually pretty complicated, where “Last Modified” is often the time when the first copy of the file was created, and “Creation Time” when the last copy was made. (By “often” I mean “not always” — as I said, these things are complicated.)

A more relevant reason here is politics and conspiracy theory. They’ve decided that Russia didn’t hack the DNC, that CrowdStrike and that DoJ indictment are lying, and that an insider/whistleblower like Seth Rich is responsible. Even if they admit that it’s possible other things transfer at 15 MBps and ZIPs can cause rounded timestamps, they’ll still insist their scenarios are the most likely. I’ve had conversations with obstinate people on twitter who make this claim.

As a tech expert myself, I dissent. Whatever is visible to them is not visible to me. I can’t see any evidence or even clues here that calls the mainstream narrative into question.

Replication

The original Guccifer 2.0 dumping HDR_pass.zip is at this location:
https://guccifer2.wordpress.com/2016/06/21/hillary-clinton/

There are links at the bottom pointing to the file. Some are broken, I used this link. Be careful you don’t click on the wrong thing and infect yourself with malware.
https://www.mediafire.com/?79a6zy27q9ung

The file is encrypted. Use this as a password:
#GucCi2/0

Transfer speeds vary wildly. On my Windows machine, my USB drive is over twice as fast. Likewise, ZIP extraction is near instantaneous on my MacBook, so fast I can’t even measure it. Also, modern USB thumbdrives likely use exFAT and not FAT, so won’t show the timestamp effect. Other ZIP software than Windows maintain the correct timestamps.

The timestamp metadata is counterintuitive.

When you see a file on your disk drive, the “Creation Time” is when the file was created on this disk drive. It’s really the “Creation Time” of the directory entry, the entry that holds the metadata, rather than the file itself. When archiving files into a ZIP, RAR, or 7zip, this metadata is recorded.

When copying a bunch of files, one after the other, and then archiving them, the "Creation Time” timestamps will record the speed at which the files were copied to the local disk. In this case, they record the event when Guccifer 2.0 copied them or extracted them to the disk before creating HRC_pass.zip.

The ever great Katie Moussouris bravely writes in support of Chris Krebs, the former head of CISA who spoke up against Trump defending the integrity of the 2020 election, who is now suffering Trump's retribution. Trump has not only announced a criminal investigation into Krebs, but is also punishing his employer, SentinelOne.

I say "bravely" because this puts her and her company in danger of retribution from the Trump administration. Her revenue includes contracts with government entities. She's therefore at real personal risk here. In contrast, I'm not so brave because I'm not really at risk. I haven't had government contracts for 10 years, so I'm not nearly as exposed for speaking out.

Krebs is not a singular case. Trump is retaliating against all sorts of foes:

• universities like Harvard and Columbia, suspending all payments from the government (such as for scientific research) unless they change their free speech policies

• law firms (like Perkins Coie)

• legacy media (CBS, ABC, AP)

• new media (Facebook)

• purging FBI and DoJ of lawyers and agents involved in Jan 6, his documents theft case, and others

All of us citizens have to decide whether to stand against this. The question is particularly acute for those in specific areas, like law firms standing up in defense of other law firms, FBI agents and DoJ prosecutors resigning in protest of the purge of their fellow employees, and so on.

The question for cybersecurity is whether we risk retribution to stand in defense of our own, such as Chris Krebs and likely more in the future.

Sadly, the correct answer for most is to keep your head down and stay quiet. This is especially true when you're an officer of a public company, where your duty to your shareholders is more important than personal politics. Same with officers in the military, where your duty is to follow orders, to avoid disputing the politics of your Commander in Chief. That's what "duty" means, that you aren't free in this situation.

But for the majority of us, there is no real cost. You should consider history, such as Niemöller's famous statement about Nazism, abbreviated "First they came for the Jews, and I didn't speak out... and then they came for me, and there was nobody left to speak out." Despots will keep going until people stand up against them. Staying quiet now risks Trump's despotism getting worse to a point where you have no choice but to resist. The more we stand together now, the less chance we'll have to stand alone later.

Some of you readers are Trumpists who believe such retribution is justified. It's not, it never has been. You might point to the legal cases against Trump as justifications for Trump's own actions against enemies. Two wrongs don't make a right. Assuming actions against Trump were politically motivated corruption, then what you are implicitly agreeing is that Trump's actions are also political corruption. The response to corruption isn't saying "they did it so we can," but to double-down on "all such corruption is wrong." Allowing such corruption is evil, and you are evil if you support it. I mean, I empathize with you, Letitia James ran on a political platform, so at least her actions were political. But at the same time, Trump is so obviously guilty of so much fraud and crime that it's unreasonable to think it's all politics.

Some of you readers are Trumpists who believe that the 2020 election was stolen, and therefore, Krebs was "lying." Maybe, but none of you have pointed to any evidence that hasn't been debunked. All Krebs is guilty of saying is this fact, that no evidence has been produced by Trump or supporters that calls into question the integrity of the 2020 election. The Republicans are now in control of all three branches of government and they still aren't producing any evidence.

The only proven "liar" here is Trump himself, not Krebs. Trump is guilty of endless lies about the election, most notably supporting Lindell's lies about "Chinese hackers" or D'Souza's lies about "mules." I've written much in both cases about how these people are lying.

Trump's EO targeting Krebs is clear and unambiguous political corruption punishing his enemies. Any good person would oppose this, at least privately. If they would not suffer obvious consequences, they should do so publicly.

I stand in support of Krebs, though it's not really that meaningful. I'm a jerk who loves to stand against things, like wokeness or Trumpism. Indeed, I oppose Krebs's work on "misinformation" because it's gone too far in the direction of suppressing opposition to wokeness. I mean, I'm quite sincere here. I studied pre-war Germany in college, and have committed to opposition to tyrants decades ago. But at the same time, I'm regularly in opposition to things.

But for the rest of you, this is a time to actually stand up against something. Ten years from now you'll get to look back and say "I stood against tyrants" or "I didn't speak out."

If you can, speak out.

Update: Krebs resigned today from SentinelOne. Alex Stamos speaks up in support of Krebs.

The Ciscogate issue came up again recently. As an insider who saw it play out from the inside, I thought I'd write up what I witnessed.

What was the event?

The event is famous in the hacker community. In typical fashion, a cybersecurity researcher found vulnerabilities in a product, in this case, Cisco routers. He was set to disclose them in the normal, responsible manner, at a talk at the BlackHat cybersecurity conference.

But things went crazy. Cisco swooped in with a bunch of lawyers the night before the talk and forced BlackHat to remove all materials from the printed program, with a large team ripping out pages early into the morning.

Nonetheless, the researcher (Michael Lynn) gave the talk. It was pretty good, one of the most entertaining.

At ISS

The whole thing started months before BlackHat.

The researcher, Mike Lynn, worked for Internet Security Systems (ISS), one of the pioneering cybersecurity companies from the 1990s that was swallowed up by IBM a year later.

ISS had a research team called "X-Force", full of hacker types that went hunting for vulnerabilities. It was one of the best teams at the time.

I don't want to take away anything from Mike Lynn's discoveries, but it should be remembered that the entire team contributed to the work. It was an environment where people bounced ideas off each other, or where experts could explain things. I don't know if any other researcher deserved credit for big contributions to this research, but I do feel that the X-Force group as a whole deserves some credit.

Once the details of the vulnerability had been confirmed, ISS and Cisco started going through the normal disclosure process. It's a process whereby researchers (ISS X-Force here) disclose the bug to a vendor (Cisco in this case), and then go through the process of helping them fix it. Once a fix is available, the researcher(s) in question give a talk at a cybersecurity conference, fully disclosing the bug.

Part of this disclosure is researchers wanting fame for their brilliance. Finding a bug in Cisco was (and still is) a pretty big deal, so would have given Lynn a bunch of fame.

But disclosure is mainly about fixing bugs, both the immediate bug in question but also in general. So many bugs don't exist today because they've been thoroughly disclosed and discussed. Such disclosure is a good thing.

Things go off the rails

ISS had a dysfunctional corporate culture. I'm pretty sure this is not exceptional, that all companies do. It's just that here, Ciscogate is the direct consequence of ISS's problems.

We had a person in charge of disclosure discussions with other vendors. Let's call him "Pete". Pete was also our representative within various industry groups and forums. He wasn't a "hacker", he was completely non-technical. He was a former special operations soldier from the military.

Pete was extremely political, in terms of corporate politics. He could often be found spreading rumors about other employees, or schmoozing the bosses. He was pretty toxic, in regards to corporate politics.

But he was also well-liked. He was a fun guy to hang around with, and there was always promise of some sort of story from his special operations days, though often only alluded to, because of course, it was still secret, and he couldn't reveal the details. I liked Pete a lot — except for this political streak.

His corporate politics weren't simply about increasing his standing at ISS, but also within the various industry groups he participated in.

He saw this Cisco bug as an opportunity. As part of its products and services, ISS needed vulnerability information from other vendors. We had good relationships with most of the big vendors (like Microsoft), but we didn't get vuln info from Cisco.

Pete's plan was this: in exchange for a better relationship with Cisco, exchanging vulnerability information, he could promise them the vuln would be squashed. This was his own plan, he came up with by himself. He told me this plan early in the process.

At the time, there were rumors of leaked Cisco source-code in the hacker underground. Pete convinced Cisco that Mike Lynn's research was based upon this leaked source-code, and that Lynn was going to reveal important trade secrets in his BlackHat talks.

In other words, maybe the offer to suppress the vuln wasn't enough, so he increased it to suppressing trade secrets and source code.

I doubt he ever said this directly. He had a way of alluding to things that he never said, of convincing the listeners of things that he never explicitly said. This is how he pushed rumors in the corporation, or discussed his exploits in the military.

Trade secrets

Intellectual property is a difficult area. Enforcement of some things are optional. A company doesn't have to enforce copyrights or patents. There have been other cases of corporations suppressing researcher talks based upon copyrights/patents. In those cases, we know the corporation is evil, because they don't have to. They are using those as an excuse.

But trademarks and trade secrets are different. If a corporation doesn't enforce, they lose them. There are many cases of corporate lawyers seizing domain names that contain trademarks. But this doesn't mean they are evil — the companies don't want the bad publicity, it's just something that trademark law forces them to do.

The point here is that when Cisco lawyers learn that trade secrets were going to be revealed, they took control of the situation.

Corporate lawyers don't care about vulnerabilities. That's a PR, marketing problem, or engineering problem, not theirs.

But when it's trade secrets, they go scorched earth on the situation. Cisco's lawyers don't see the situation as we do, that of typical vuln disclosure at a BlackHat talk. They only saw the trade secrets issue.

Of course, it was false, there was no Cisco trade secret issue, but their primary contact at ISS had told them there was. They were techies, they didn't know. All they knew is that they had to fix it.

From this perspective, they were totally justified. Had ISS's representative been telling the truth, this is roughly how their lawyers should've responded. Remember: it's not optional according to the law. Cisco had marketing people who knew this would damage their reputation, and cybersecurity researchers of their own who knew this was an evil way of dealing with vuln disclosure. Level heads could not prevail because the lawyers were in charge, falsely believing it was trade secrets.

The meeting

ISS was getting pushback from Cisco, of course. Cisco was telling ISS that they needed to cancel the talk.

So shortly before BlackHat (the Friday before, I think), ISS had a meeting, pulling in all the principal players. I was "Chief Scientist", not directly part of the X-Force, but still, high up in the company.

The problem was that neither the ISS CEO nor Cisco CEO were available to discuss the problem. They were in Washington DC as members of President Bush's cybersecurity council.

There was this other communication channel — the CEOs talking directly to each other. Pete schmoozed the ISS CEO, who in turned passed on the information to the Cisco CEO, who then told his lawyers.

At the start of this meeting, the ISS CEO said that Pete was in charge of resolving the situation. This was very bad since it was Pete who had created the whole mess to begin with.

ISS couldn't actually cancel the talk. They couldn't call up BlackHat and have the talk canceled. Instead, the most they could do is threaten to fire Mike Lynn if he went through with the talk.

I tried to be the voice of reason. I tried to be on Mike Lynn's side. Canceling a talk like this was a Bad Thing, for ISS, for Cisco, for Mike Lynn. ISS shouldn't have been pressuring him to cancel the talk. They shouldn't have put him in that position.

But once he was in that position, his best personal course of action would've been to comply. He'd be the most famous hacker on the planet for a few months, but forgotten after. Getting a job in the field would be difficult after that, as nobody wanted an employee who would rage quit and dump trade secrets (even if, in my opinion, there really weren't any trade secrets). He did get a new job, at Cisco’s major competitor Juniper, but I don’t remember him giving another cybersecurity talk afterward.

All of us at the time, except for Pete, were caught in a web of misinformation. We didn't understand the forces that were causing the problem. Pete was in charge, and he was driving things to destruction.

Aftermath

It's after all this that the public story starts. Mike decided to give the talk anyway, Cisco lawyers came in and ripped pages from the conference book, and you know the rest.

From what I saw, this was not the case of a company suppressing vulnerability research. Instead, it was corporate politics gone awry, crushing an individual employee. I suspect there is a little of this in every such story: it's never really "corporations" making such decisions so much as individuals. How they make decisions is often flawed, such as pursuing their own corporate politics goals.

This recent (horrible) paper cites the Ciscogate scandal as everyone knows it, "thin skinned lawyers" suppressing a vulnerability. It's wrong, it was thick skinned lawyers who knew next to nothing about vulns but who were protecting trade secrets.

Uniqueness

SSNs were invented before computers (as we know them). My parents were issued their SSNs back in the 1940s (for example). While they were intended to be unique for each person, the lack of computers back then meant they functionally weren't. Millions of people had faulty SSNs back then — not just duplicates, but people having multiple SSNs. My great-grandmother apparently had two.

It wasn't really until Gen Z that the system became robust against common flaws. But still, it's the Greatest Generation and Boomers who are getting retirement checks from when the system wasn't computerized.

This isn't necessarily a flaw. Europe loves the centralized government control of a national ID, but America has long rejected this. We don't necessarily want the federal government to assign us a unique number.

In America, it's the state, or even local community, that is in charge of such things as citizenship and voting rights. We want it that way. As a country, we've long been hostile to the central government being involved in our affairs. The national "REAL ID" law has taken 20 years to finally come into force. Having state databases synchronized with federal databases is a potential privacy violation.

It's not just our hostility to the federal government; we have strong freedom and privacy traditions that reject any sort of government interference with our identity.

As a consequence, roughly 20 million people in the United States have no government ID, or even birth certificates where they can establish such ID. This includes many people in rural communities, including those living far from civilization in the Appalachian Mountains or Louisiana Bayou. Sure, they may be getting their Social Security check, but they may otherwise be disconnected from all other government institutions.

The original problems with SSNs are well documented. There are legendary problems, such as a wallet company that once printed a sample Social Security card to put in their wallets with the number 078-05-1120. At its peak in the 1940s, over 5,000 people were using that as their number.

What's not documented is how well they've cleaned this up.

We know they attempt to clean it up. When two people are accidentally assigned the same SSN, they do indeed try to fix it.

But we don't know the rate of success. Maybe they fix 100% of such problems, but maybe they only fix 99.9%. When those people are in the mountains or bayous with duplicate numbers, how do you contact them to fix the problem? It's much harder than you think. Sure, you can send them mail to their mailing address, or try to send federal agents to talk to them in person, but what if they just still keep using the wrong number?

I suspect that there are a few sticky accounts they just can't solve and "de-duplicate." Government famously must accommodate extreme edge cases — to treat everyone equally. For example, any fingerprint identification system must also handle the case where people have no fingers. Our instinct that such people don't exist or don't matter would be wrong.

Fraud

This entire discussion is separate from fraud, such as immigrants stealing people's SSNs, or identity theft. That's the problem Musk is really concerned about, not the fact of duplicate numbers.

And, I suspect, Musk is right that the Social Security Administration isn't doing enough to solve it.

But uniqueness really has nothing to do with fraud. For example, this past year the government got more diligent about dead people receiving Social Security checks and recovered $30 million in false payments. They can do this even when numbers aren't unique.

SQL databases

Musk goes on to talk about “SQL”. Here he grossly wrong.

"SQL" is the language we use to access (modern) databases.

The discussion pivots here because SQL arranges data around unique keys. If we know somebody's unique key, like an SSN, we can then look up all the data entries related to that number.

If a key isn't actually unique, things misbehave.

That's the real discussion, that it's inconceivable that an SSN wouldn't be a unique key for database lookups. Of course it would be! It's obvious to any programmer!

Except, well, it isn't necessarily. In fact, most of the tables in such big databases would not use the SSN as a unique key. For example, somewhere there is a table containing how much you've earned every year. It would be the combination of your SSN and year that would be the unique key, not just the SSN alone.

If there exist valid duplicate uses of SSNs, then they could likewise use a combo key, including such things as date of birth, last name, or even just an arbitrary number (1, 2, 3, etc.) as the unique key.

From a programmer's point of view, there's nothing really that I can conclude from "SSNs are not de-duplicated." My naive assumptions about how they might've structured their SQL databases probably don't match the reality.

Now, Musk might be right. Back in the 1970s, there were big mainframe databases that predate SQL. Big corporations and big government have a lot of apps written in Cobol and IBM’s old database-like technology. However, indexing such files and tables doesn’t change. You still make the decision of SSN uniqueness, regardless of the technology.

Part of my knowledge here is as a programmer, and the Falsehoods Programmers Believe about Names. The things we consider immutable and unique about a person, such as names, date/place of birth, address, and so on — simply aren't.

For example, both my parents have different names on different official government documents. These aren't radically different names, just slight differences. This is the norm. It's especially true with people from different cultures where multiple "names" are the norm.

The point is:

• There are lots of good historical reasons why Social Security entries weren't unique, and it's not clear whether they've all been solved

• None of this really has anything to do with SQL database construction

Conclusion

The point of this post is that there's no reason for outrage and partisanship here. Musk hasn't supplied enough information to tell us what's happening, only enough to develop assumptions and misconceptions.

If there's something actually wrong, then he could quickly write up a document and explain things in full, addressing the sorts of issues I bring up here.

The wise Lesley Carhart asks:

Her wise answer to all three is "it's complicated," that these are in fact not "easy questions." It's the complexity of the answer that makes it the correct answer — even if different techies disagree on the complex parts.

But I'm somebody who gives wrong answers. I'm going to insist that there are easy answers to these questions. Specifically, the easy answers are:

• No. The US power grid is too diverse for hackers to seriously threaten as a whole, though localized blackouts are a big threat. The solution isn't to focus on the power grid as a whole, but to diversify things, such as households having battery backups.

• No. There is no silver bullet, magic pill solution to security, and AI isn't close to becoming one. AI will change the job of human analysts, but at the same rate their job is always changing.

• No. Anti-Microsoft hate is pervasive in the community, but the security features of all major operating systems are roughly the same and can be hardened the same. The best business solution is the one that satisfies business needs, not cybersecurity needs.

The following is a more extensive discussion.

Power Grid

Hackers can't take down the US power grid — at least, as a national policy, it's not something we need to worry about. Solar mass ejections are far more important. That's because the US power grid is really diverse, with over 10,000 companies involved in generation and transmission. It's designed to quickly segment itself. We've addressed the cascade of failures that happened in the 2003 East Coast Blackout..

National policy should instead focus on local blackouts, such as that of a major city. Hackers can break into local grids, or simply bribe a local operator, and cause painful blackouts.

The reason I describe "national policy" is that the threat of a "Cyber 9/11" for the last 30 years, from the likes of Richard Clarke, often results in bad policy. For example, it's pushed "Einstein" intrusion detection sensors everywhere which themselves now become a threat. It's now a single target the hacker can attack that impacts the entire grid, where before the grid was too diverse to attack with a single thing.

The long term path to securing the grid, from hackers or solar mass ejections, is more diversity, such as more home battery storage and solar panels. Solar and wind produce an over-abundance of power for a few hours each day. The grid should just deliver that to households which recharge their backup batteries and cars. Households would have incentive to invest in such infrastructure with current smart-grid technologies that charge more or less depending upon the current production and load on the grid.

In other words, to resist hackers, thinking in terms of "too big to fail" is the wrong approach. We should be thinking in diversifying infrastructure, such that households and businesses have more continuity in the face of a grid attack.

AI Replacing Security Analysts

Well, part of this question has nothing to do with cybersecurity, but is about the current capabilities of AI right now.

AI is "trained" on existing things. It'll certainly become a useful defensive technology, able to easily recognize known attacks in ways that human analysts struggle to.

But on the other hand, cybersecurity always works this way. Defenders address known attacks, and hackers are constantly pivoting to new things.

The reason security analysts are human is so that they can likewise pivot and deal with novel attacks.

In other words, AI isn't going to be the "magic pill" that solves cybersecurity. Nor is it the "magic pill" that's going to automate hacker attacks. Magic pills are the wrong way to conceptualize problems.

Now, AI is certainly going to change the job of cybersecurity analyst, but then, technology has always been changing that job. There are a few chronic problems that remain the same, but most of the job completely changes every decade.

Operating System

There is no secure operating system. All operating systems have roughly the same sort of features that experts can use to harden them, such as removing admin access or removing app privileges.

Classically, there has always been operating-system partisanship in the tech community. Specifically, it's standard for techies to tell you how insecure and unreliable Windows is, because they hate the monopoly dominance Windows has on the desktop. But it's never been technically true — technical experts exploit the trust people have in their expertise in order to pursue partisan anti-Microsoft goals. A classic example is the "Monoculture" paper by Dan Geer — technically vacuous but strokes the prejudices of those who hate Microsoft, so you'll get cheers from the crowd whenever you mention it at a cybersecurity conference.

I'd agree that most organizations are flawed in how they've deployed Microsoft solutions. But replacement is probably just a very different deployment of Microsoft solutions, not switching to macOS or (gasp) Linux on the desktop.

But the biggest flaw here is simply pretending that the best operating system is the most secure one. Even if there was one that was notably more secure, the best business operating system is still the one that best solves business needs, not which solves cybersecurity needs. We in cybersecurity believe that we are gods and everyone should listen to us, that the purpose of the organization is to solve our needs. The chief cause of burnout in the industry is that nobody seems to be listening to us when we stress the importance of cybersecurity. But cybersecurity isn't that important — the business is.

In short, it's our job to secure the business operating system, whichever one they choose. Cybersecurity needs should be considered, but only as one of many needs.

Conclusion

Of course, Lesley has the right answers: the only people you should trust are those who claim the answers are complicated.

But, as a contrarian, I'm still going to reach for the wrong answers. So that's this post.

AI disclaimer: AI will replace copy editors. I can’t help but make stupid mistakes that. I cannot see, so I’m now using AI to copy-edit my crap. Here’s the list of changes claude.ai made to this piece before I posted it. Also, the image at top was created by Grok.ai.

Somebody named “Spoonamoore” wrote a blogpost outlining a conspiracy by Musk (among others) to steal the 2024 election (for Trump). This has gotten a lot of attention in social-media. As a long time election conspiracy-theory debunker, people keep asking my opinion about it. I wrote a blogpost debunking the core point, about “bullet ballots”, but apparently that wasn’t enough. In this post, I analyze the entire thing.

Spoonamore’s conspiracy-theory has four elements:

1. Evidence (claimed) of fraudulent ballots, namely a statistically high number of “bullet ballots” containing only a vote for President.

2. Musk’s “free-speech petition” harvesting voter records that fake ballots can be tied to.

3. Remote access to “pollbooks” to make the count of “votes” and “voters” match.

4. Change the tabulator results, inserting votes votes at precincts to match how many voters the pollbooks claimed voted.

Step #1 bullet ballots

The core of the conspiracy-theory, the only part that might contain evidence, is the claim that there are an unusual number of “bullet ballots” (votes only for President) or “downvotes” (missing only some races other than President).

While he claims such evidence exists, he doesn’t cite it. He’s vague where he’s getting his information from. Neither his data sources or methods are disclosed.

In fact, “bullet ballots” wouldn’t be public. Few states provide enough information to count them. Georgia does, and once I get a hold of the ballot images after the 2024 election, I’m going to count this. But for right now, he cites no source for such information.

I can, however, partly reproduce his results by counting down votes instead. These are ballots that have a vote for President, but where some other races are missing votes. The number of downotes must therefore exceed the number of bullet ballots (all bullet ballots are also a downvote).

The downvotes are public. We can compare the vote counts for the President race against other races, most easily against statewide races like Senator and Governor.

For example, in the Arizona 2024 race, we can tell there were 3,389,405 votes for President and 3,348,017 votes for Senate, a difference of 1.2%. It’s not precise, we’d have to look at the actual ballots to know for certain, but it’s close enough.

In another blogpost, I do the work and show that number of such ballots is perfectly normal — at least, those where there’s a senate race. Remember from high-school civics class that 1/3rd of the states won’t have a Senate race this year. My home swing state of Georgia has neither a Senator nor Governor race in 2024.

In that blogpost, I document data sources and methods so that anybody can reproduce my work easily. That’s how you can identify conspiracy-theories: they don’t document their sources or methods.

Step #2 registration harvesting

The next step in the conspiracy-theory is Musk’s so-called “free-speech petition”. Musk created a website where people could sign up to promise to defend the First Amendment. He then gave away $1 million each day to the people who had signed up — on the condition that they were also registered to vote.

Musk’s idea was silly, of course. For one thing, it wasn’t a “petition” but a “promise”. For another thing, it’s unclear whether people were signing up to defend the actual First Amendment or Musk’s twisted interpretation of it. Thirdly, it’s not really recording those committed to free-speech so much as those wanting to win $1 million.

Critics see more in this effort. Offering a $1 million lottery to encourage people to register to vote is an illegal inducement, and Democrats are taking him to court over this. (As I read the law, he’s not actually guilty, but it’s close).

Spoonamore sees this as part of the larger conspiracy. In order to stuff ballot boxes with illegal votes, those ballots still need to be tied to legitimate registered voters.

At the end of the election, the number of registered voters who voted needs to match the number of votes cast. It’s a common conspiracy-theory that elections end up with “more votes than voters”, but in fact, this never really happens. It’s such a well-known cliche that it’s always checked after every election.

Thus, Spoonamore theorizes that by harvesting the street addresses of registered voters, Musk (or coconspirators) can stuff fraudulent votes matched to real voters.

It’s a nonsense theory, though, because voter registrations are already public records. In my state of Georgia, it costs $250 to get a list of all registered voters. Something similar is the case in all other states. Musk didn’t need a nefarious plan to harvest lists of eligible voters, he could simply get them with a public records request.

Spoonamore claims the scheme requires more than the public record of the registered voter, that knowledge of Trump support was needed. If registered Democrats suddenly voted for Trump in large numbers, this would look suspicious. But this sort of anomaly isn’t detectable. If it were, the anomaly of suspiciously high turnout among Republicans would be even more noticeable.

Step #3 pollbooks.

The pollbook is the list of eligible voters at a precinct. When you go on election day, they check your name against this list, to see if you are allowed. After you vote, they check off your name, so you an’t vote again.

Historically, this was a paper printout. On the morning of the election, the computers that hold the voter registration database would print out a list of voters for each precinct, including whether they’ve already voted (such as by mail-in ballot).

The theory here is that pollbook numbers need updating. As the count of stuff votes go up, you need to increment the count of voters. You need to mark off specific voter names who you claim have voted.

Spoonamore claims that “ePollBook data is nearly always linked to the internet”, but the opposite is true. It’s usually paper or a tablet not connected to the Internet.

That’s changing. States want to make it easy for people to vote at any precinct, so that if there is a line at one, they can just drive down the road to vote at another. This requires electronic pollbooks that are synchronized via the Internet during the election.

Arizona does this. They use Internet connected e-pollbooks. They have robust cybersecurity measures (such as using VPNs) that make Internet hacking implausible, but technically, they are connected to the Internet. Indeed, this was a problem during the 2024 primaries, early voting in Arizona was disrupted by a global IT outage. The electronic pollbooks could not connect to the server, and hence, the polls couldn’t open.

In the 2024 Presidential election, some precincts in rural counties had their pollbooks connected via Starlink, Musk’s satellite Internet service.

This again brings Musk into election conspiracy theories, but it’s incidental. Internet communication is encrypted on the ends, doubly so when using VPNs, so there’s nothing a router can do to spy on or intercept such traffic. Routers, even Musk’s Starlink routers, can’t do anything to interfere with this, other than refusing to route traffic. It’s a fundamental principle of the modern Internet that routers can only see the metadata (addresses of the packet destination) and not the payloads.

The reason Starlink appears here is because Musk re-invented satellite Internet, converting it from something impractical for most purposes to something as easy to setup as WiFi. It’s by far the easiest way to connect rural locations to the Internet, everywhere in the world, from the heart of the Amazon, to airplanes flying over the Atlantic, to Antarctica, to the election office in the middle of rural Cochise county in Arizona.

Musk is going to appear in a lot of conspiracy-theories in the future. Starlink is a revolution in ubiquitous Internet. It’s showing up everywhere — and will appear again in the next round of conspiracy-theories. But since it’s merely a router, it’s immaterial.

Step #4 tabulator stuffing

The last step is the ballot stuffing.

The problem is that a pure computer hack cannot create paper ballots, so we aren’t stuffing actual ballots, such messing with numbers inside the computer to increase vote count.

Paper ballots are used almost everywhere.

In Arizona, ballots are printed out and hand marked by humans, then fed into the tabulators. In Georgia, precincts use ballot marking devices that help a voter fill out the ballot, which is then printed out. The printout is then fed into a tabulator as a separate step. Sure, a computer is used to vote in Georgia, but there’s s till a paper ballot.

In theory, it wouldn’t be too hard hard for a malicious election worker to create fraudulent ballots and stuff them with the other paper ballots, but there are so many surveillance cameras, election watchers, and other workers that somebody would likely get caught. The conspiracy here is that it all happened via computers. Physical actions can be seen, computer actions are invisible.

Spoonamore’s theory is that the tabulator is hacked to add numbers to Trump’s total, creating more votes going out than paper ballots coming in.

That’s why hacking the pollbook and tabulator together is important, so that the number of votes and voters match. The idea is that at the end of the day, right before polls closed, the hacked pollbook will mark of registered voters that have not yet voted, to make the numbers even.

We can’t detect the hack by simply looking at the tabulator or pollbook records of the number of votes/voters, but we can detect this by counting the number of paper ballots. That’s why they suggest a hand recount of the President race.

The simplest way to guard against this is to count the paper ballots at the precinct at the end of the election day and verify the number of ballots tabulated match the number of paper ballots. This was a requirement proposed by the (Republican controlled) Georgia Election Board, but was struck down by the courts. Democrats claimed this was some sort of evil conspiracy by the Republicans, but is actually quite reasonable.

Tabulators are not connected to the Internet as Spoonamore claims, only some pollbooks. Thus, the reality is that local precinct workers would need to be part of the conspiracy. This is implausible — it demands a conspiracy to big that it’s unlikely to go undetected. Spoonamore claims that such workers can tricked into bad actions, but really, election procedures are more secure than that.

The one conspiracy-theory type thing I might agree with is that local election workers can plausibly tamper with tabulators. In 2021, Republican election conspiracy-theorists raided Coffee County in Georgia and stole all the code for Dominion’s computers, including tabulators. They’ve had 3 years to study that code looking for ways to subvert it.

There’s no evidence of such subversion, but it’s the one part of this entire conspiracy that’s plausible. I would demand evidence before I’d believe it did happen, but I cannot rule it out. In other words, it’s not as crazy as the rest of the conspiracy-theory.

Other nonsense

Spoonamore claims this hack is plausible because it’s a lot less complex than the Hamas pager hack.

It’s a silly comparison, such as claiming since we can land an austronaut on the moon that we can land an astronaut on the sun.

His overstates the complexity of the pager attack. It was not actually a “hack”, no “hackers” were involved. It was straightforward spycraft, inserting explosives into devices the enemy used. It’s impressive as heck, but not a hack.

Hacking pollbooks and tabulators is far harder than you think. The number of people involved would be absurdly high — no conspiracy that big could go long without some whistleblower revealing it.

No rational argument

You don’t need to do the work or be an expert to know this that this bunk. You can simply read the shape of his letter.

• He lists a lot of vague qualifications, like being the CTO or cofounder of companies that he doesn’t name. Rational people list their specific qualifications, if they were a CTO of a company, they name the company.

• He expects to be believed by the strength of such qualifications, not the strength of his arguments. For example, he doesn’t document where he gets his “bullet ballot” numbers from, so nobody can directly check them.

• He makes grandiose statements, like “duty to warn” or “defend the Constitution against all enemies foreign and domestic”. He’s got visions of grandeur.

The point is that he thinks he’s increasing his credibility by such things. He evidently is — among the conspiracy-theory crowd that keeps repeating his letter. But among rational people, all of this decreases his credibility.

Specifically, rational people demand evidence, and he’s provided none of that, only a theory. Has hasn’t even provided evidence of the one element where evidence would be available, the number of “bullet ballots” or “undervotes”.

This is why it’s right to call this a “conspiracy-theory”: he theorizes a very large conspiracy based upon no evidence.

Conclusion

This is an overly long blogpost, and I’m sorry if you had to slog through it, but my goal is to address all the points. Apparently, simply debunking his “bullet ballots” wasn’t enough, people wanted me to look at the rest of the claims.

One of them is this blogpost claiming there was a statistically impossible number of “bullet ballots”, those just containing votes for the Presidential race, and not any for the other ~20 races on the same ballot (such as US Senate, state legislature, or county dog catcher).

That blogpost is debunked in two ways:

1. The numbers are bad. I use the vote counts reported in the media (NBC News) to show that there’s no anomaly.

2. The idea is bad. It’s a logical fallacy to claim an unexplained anomaly can only be explained by a conspiracy. They distort the numbers to make them unexplainable instead of looking for explainations.

Doing the math

First, let’s do the calculations ourselves.

The source of my information is the https://nbcnews.com website’s “live results” pages showing the latest reported nubmers as of November 20, 2024. These aren’t the official numbers, but are close enough for our purposes.

My algorithm is to compare the President race numbers to the Senate race numbers. This isn’t “bullet ballots” precisely, but close enough. I’m not sure how actual “bullet ballots” can be counted (except in odd cases like Georgia that makes ballot images available).

We see that there were 3,389,405 votes for President and 3,348,017 votes for Senate, a differences of 41,388 votes — or 1.2%. This is the normal range we see across the country this election and in past elections.

Thus, we have disproved the claims of abnormal “bullet votes” in Arizona, that there were 123k such votes that was 7.2% of Trump’s total.

Conversely, we can do the same thing for Utah. This is not a swing state, Trump easily won it, and there was presumably no reason to cheat. That post claimed it had only 0.01% suspicious votes.

Doing the math, we see that this non-swing state had 2.2% such votes, where there was a vote for President but not for Senate. This is in fact much higher than Arizona’s 1.2%.

The other swing states with Senate elections break down as follows:

• Arizona = 1.2%

• Michigan = 1.6%

• Minnesota = 1.7%

• Nevada = 1.4%

• Pennsylvania = 1.0%

• Wisconsin = 0.9%

Some other non-swing states:

• Utah = 2.2%

• New Mexico = 2.2%

• Texas = 0.8%

• California = 3.1%

Thus, I’ve completely debunked that claim. There are no anomalies here.

Unexplained anomalies

The core principle of conspiracy-theories is that anything unexplained is proof of the conspiracy. The “conspiracy” is a universal explaination that explains everything, so becomes the null hypothesis if something cannot otherwise be explained.

In other words, it’s not positive proof that something happened, but a sort of negative proof based on believing nothing else could’ve happened.

And it doesn’t work out. It’s inconceivable that if somebody were trying to steal an election that they wouldn’t also insert votes for Senate and House races. The conspiracy-theory doesn’t actually explain the anomaly, it doesn’t explain why only one race was stolen and not others.

This flawed logic means they hunt for anomalies. When they see a data anomaly, they aren’t interested in explaining it. They instead seek to distort it to become even more anamolous, to make it even more unexplainable.

That’s what we saw in the above blogpost. There was no attempt to figure out what was actually going on, but only an attempt to exagerate the anomaly.

For example, in that post, it usually talks about the number of such votes cast, >1% of all votes. But when it gets around to talking about Arizon, it counts them as 7.3% of Trump votes cast, effectively doubling the number in comparison, to make it look twice as suspicious. Comparing to all votes would’ve netted a value of only 3.7%.

Likewise, their number of >1% is suspiciously off, as my count of Utah above shows, with 2.2% suspicious votes.

The point is that the anomally was largely the work of a creative manipulation of numbers rather than an actual anomaly.

Conclusion

I’ve debunked this in two ways. First, I shows that the numbers are wrong. The latest reported results show no anomalies among the swing states, especially not in Arizona and Nevada that were singled out.

Second, I show that the logic is wrong. The numbers were twisted to heighten anomalies, which then used the flawed premise that any unpexplained anomaly is proof of the conspiracy.

The following image shows a “bullet ballot” from the 2024 State of Georgia election. It contains only a vote for the President, while all the other races were ignored.

The latest is a claim that Dominion’s “EMS” servers have a backdoor password of “dvscorp08!”, hardcoded in every server they’ve sold for over a decade.

The fact are (probably, uncontroversially) true. The characterization is not.

Dominion’s cyberscurity rests upon preventing physical access to the machines, keeping them air gapped from the Internet.

Anyone with physical access to the machines can subvert them, with or without backdoor passwords. Every scenario shown above can already be done without those backdoor passwords.

Even if you fixed that problem, Dominion has a bunch of other holes. In other words, Dominion is going to fix this by using Windows operating-system accounts instead of MS-SQL accounts, but this is just a band-aid that doesn’t actually change the security of the system. The password hashes will just be Windows password hashes instead of MS-SQL password hashes.

The ultimate fix is to re-architect things from scratch, using such technologies as TPMs and DPAPI, to make the security architecture work more like how your phone works, where somebody with physical access can’t subvert the security. But this only marginally improves security — hostile election workers would still be able to subvert elections.

Details

It’s incredibly irritating that these people don’t actually go into details about what the heck they are talking about.

Consider Supreme Court decisions: they go in detail explaining every step to the point that even non-lawyers can understand. It’s tedious as heck, but you can read the exact reasoning the justices used to come to the decision.

These conspiracy-theorists do the opposite, skipping steps and important information such that even experts aren’t certain what, precisely, is going on. We have to make educated guess about what they might mean. This process is deliberate — they are creating an argument to entice those with little knowledge of the system, not building a case that can withstand skeptical experts.

Simply saying “backdoor password” doesn’t tell us what’s going on. The first question is “for what?”. The three common contexts are:

• BIOS passwords

• Windows account passwords

• Database passwords

In this case, “dvscorp08!” is the password for the a database (MS-SQL) account named “SAdmin”, on the “Dominion EMS” server. This is an administrative account that has full control over the database, meaning it can change values (like the number of votes) and even change some code called “stored procedure calls” that return different totals of votes, without changing the votes themselves.

The problem with Dominion’s EMS server is that it doesn’t really have security against hostile insiders. If you have physical access to the machine, you can already do such things as change votes and source code. The reason is that if you were a hostile insider, you could subvert the election in other ways, without needing to do the hacks described here.

Dominion EMS server

The system in question is Dominion’s “Election Management System” or “EMS”.

This is the central component of Dominion’s “Democracy Suite” of products which includes a lot of other things, such as ballot marking devices (BMDs) and tabulators.

Each county runs their own election, and will have one EMS, usually locked away in a room. The EMS is air gapped from the Internet, though will usually be part of a small local network connected to other devices.

The larger the county, the more the components of EMS will be split among various servers and workstations on the air gapped network. For a small county, like Mesa County in Colorado, most components are located on a single server.

The system manages many stages of the election, such as shown in the graphic:

Elections are much more complex than you think. In your mind, you simply get a paper ballot, mark which President you want, it then gets counted, and that’s the result.

But in America, we typically overload the Presidential election with all the other races. It’ll usually contain state elections, such as your local legislature representative. It’s also contain local county offices, like the water commissioner. It may contain municipal elections, like the local mayor.

The following is a ballot image from Douglas County in Georgia form 2020:

The races that can change throughout the county are the U.S Representative, the State Senator, the State Representative, and the Education District.

So before the election, a worker sits down to use the Dominion “Election Event Designer” app to design all these apps. This can include such tasks as creating audio files for use in ballot marking devices to help those with vision problems. It’s a lot of work.

Among the lack of details is where this dbo.Choice stored procedure comes from. It comes from the Designer app that designs the next election event. It’ll eventually be used after the election during Tallying of results. To really understand this, we need a full description of how all this works. Note that the screenshot in the thread only lists the Presidential race, not any of the numerous other races happening at the same time. I’m not sure if this is real information of an actual race — there’s no information where the hacker got this.

During and after the election comes the “tabulation” of votes. Dominion’s devices are called “ImageCast”, because among the things they do is to record an image of the ballot (such as the image above) before counting the votes. These devices can be connected via the air gapped network to the EMS, or more often, they record everything to a flash drive, which is then manually transported to the EMS and read in.

A worker then uses a “Results Tally and Reporting” app to then collect and report the votes.

Services

The EMS is built using standard modern technologies. In this case, they use the technology suite from Microsoft. The web server is running IIS server-side components written in C# that most then access the separate MS-SQL database. “IIS” is the name of Microsoft’s web server that comes with Windows, “C#” is the programming language most often used to write these components, and “MS-SQL” is Microsoft’s database software.

The specific problem here is hardcoded passwords in the C# code and within the database. This is common practice, because such servers are physically isolated from the world, but it’s not best practice.

It’s common because one component needs to connect to another, and that requires an account. Code running within the web server needs to log into the database just like any other user.

The Microsoft solution is “integrated” accounts, so that instead of the MS-SQL server maintaining it’s own login accounts (like “SAdmin” using the “dvscorp08!” password, that it instead trusts a Windows account.

In other words, Windows maintains a list of login accounts to log into Windows, and the database maintains login accounts to log into the database, and these are separate. But they don’t have to be: you can tell the database to trust Windows. You’ll still see something like “SAdmin” here in the database, but instead of password, you’ll see which Windows account that maps to.

It’s much like how a lot of websites allow you to login using your Apple, Google, or Facebook credentials without having to create a password specially for that website.

Security

To fix Dominion’s security would be to rearchitect the system to work more like an iPhone, where any “hard coded passwords” are stored in a tamper-proof chip.

Dominion’s security rests upon physical access, preventing outsiders from accessing the machines. Their machines, or local networks, are air gapped from the rest of the Internet, so hackers can’t get in. There is little to no protection against hostile insiders, as we’ve seen in Mesa County (Colorado) and Coffee County (Georgia), where Republican election officials allowed conspiracy-theorists to access the machines after the 2020 election.

The point is that getting rid of these machines will not secure elections. Malicious election workers can already subvert the elections without the machines.

Having computers tabulate ballots is certainly a problem because people don’t trust computers, but the reality is that if we got rid of computers and hand-counted all elections, reliability and trustworthiness of the elections would go down, not up.

Now with computers, we do have the opportunity to improve security and also guard against hostile insiders, but this would require a complete redesign of Dominion’s architecture.

But the security improvements would be small. You still have to trust administrators. All that you’d be doing is securing the system against hostile election workers.

How to do with without a backdoor password

The disk is not encrypted. Somebody could access the disk physically and change all this. (You can’t meaningfully encrypt the disk without a hardware TPM chip like what you have in your phone).

Anybody who can login to access the server, such as uploading votes to tally, can execute database commands to change values in the database. That’s what they are doing when uploading new votes. This can be done at any workstation with any user account.

All these things can be subverted outside of the EMS before sending the data to be tallied.

Without computers, ballots can be tampered with using the same techniques they could always have been tampered with. There are a lot of security controls to make this harder, but dedicated malicious elections workers can still cause problems.

How to discover such a hack

Among many other things, the timestamps would change. The hack shown here changing the stored procedure would be trivially detected.

After an election, the entire database is archived and can be audited to discover such things.

Conclusion

Details matter. The post above doesn’t contain enough details for experts to really know what’s going on. It only lists suggestive things, hoping to convince non-experts.

I try to go into more details in this post, but it would really require an entire textbook to explain it all.

In short, the scenarios described don’t need this “backdoor password”. It’s relying on the fact that people are scared by the concept of a “backdoor password”.

Dominion has crap security. It’s really only designed to keep outsiders out, and doesn’t do a good job protecting against malicious insiders. But malicious insiders have plenty of opportunity to subvert the system even without touching Dominion’s servers.

Sure, Dominion is going to “patch” this problem, but it’s not a real fix. The real fix is to redesign the security from scratch, such as using TPMs to encrypt the hard-drive.