History: what really happened with Ciscogate

an inside-insider's view

The Ciscogate issue came up again recently. As an insider who saw it play out from the inside, I thought I'd write up what I witnessed.

What was the event?

The event is famous in the hacker community. In typical fashion, a cybersecurity researcher found vulnerabilities in a product, in this case, Cisco routers. He was set to disclose them in the normal, responsible manner, at a talk at the BlackHat cybersecurity conference.

But things went crazy. Cisco swooped in with a bunch of lawyers the night before the talk and forced BlackHat to remove all materials from the printed program, with a large team ripping out pages early into the morning.

Nonetheless, the researcher (Michael Lynn) gave the talk. It was pretty good, one of the most entertaining.

At ISS

The whole thing started months before BlackHat.

The researcher, Mike Lynn, worked for Internet Security Systems (ISS), one of the pioneering cybersecurity companies from the 1990s that was swallowed up by IBM a year later.

ISS had a research team called "X-Force", full of hacker types that went hunting for vulnerabilities. It was one of the best teams at the time.

I don't want to take away anything from Mike Lynn's discoveries, but it should be remembered that the entire team contributed to the work. It was an environment where people bounced ideas off each other, or where experts could explain things. I don't know if any other researcher deserved credit for big contributions to this research, but I do feel that the X-Force group as a whole deserves some credit.

Once the details of the vulnerability had been confirmed, ISS and Cisco started going through the normal disclosure process. It's a process whereby researchers (ISS X-Force here) disclose the bug to a vendor (Cisco in this case), and then go through the process of helping them fix it. Once a fix is available, the researcher(s) in question give a talk at a cybersecurity conference, fully disclosing the bug.

Part of this disclosure is researchers wanting fame for their brilliance. Finding a bug in Cisco was (and still is) a pretty big deal, so would have given Lynn a bunch of fame.

But disclosure is mainly about fixing bugs, both the immediate bug in question but also in general. So many bugs don't exist today because they've been thoroughly disclosed and discussed. Such disclosure is a good thing.

Things go off the rails

ISS had a dysfunctional corporate culture. I'm pretty sure this is not exceptional, that all companies do. It's just that here, Ciscogate is the direct consequence of ISS's problems.

We had a person in charge of disclosure discussions with other vendors. Let's call him "Pete". Pete was also our representative within various industry groups and forums. He wasn't a "hacker", he was completely non-technical. He was a former special operations soldier from the military.

Pete was extremely political, in terms of corporate politics. He could often be found spreading rumors about other employees, or schmoozing the bosses. He was pretty toxic, in regards to corporate politics.

But he was also well-liked. He was a fun guy to hang around with, and there was always promise of some sort of story from his special operations days, though often only alluded to, because of course, it was still secret, and he couldn't reveal the details. I liked Pete a lot — except for this political streak.

His corporate politics weren't simply about increasing his standing at ISS, but also within the various industry groups he participated in.

He saw this Cisco bug as an opportunity. As part of its products and services, ISS needed vulnerability information from other vendors. We had good relationships with most of the big vendors (like Microsoft), but we didn't get vuln info from Cisco.

Pete's plan was this: in exchange for a better relationship with Cisco, exchanging vulnerability information, he could promise them the vuln would be squashed. This was his own plan, he came up with by himself. He told me this plan early in the process.

At the time, there were rumors of leaked Cisco source-code in the hacker underground. Pete convinced Cisco that Mike Lynn's research was based upon this leaked source-code, and that Lynn was going to reveal important trade secrets in his BlackHat talks.

In other words, maybe the offer to suppress the vuln wasn't enough, so he increased it to suppressing trade secrets and source code.

I doubt he ever said this directly. He had a way of alluding to things that he never said, of convincing the listeners of things that he never explicitly said. This is how he pushed rumors in the corporation, or discussed his exploits in the military.

Trade secrets

Intellectual property is a difficult area. Enforcement of some things are optional. A company doesn't have to enforce copyrights or patents. There have been other cases of corporations suppressing researcher talks based upon copyrights/patents. In those cases, we know the corporation is evil, because they don't have to. They are using those as an excuse.

But trademarks and trade secrets are different. If a corporation doesn't enforce, they lose them. There are many cases of corporate lawyers seizing domain names that contain trademarks. But this doesn't mean they are evil — the companies don't want the bad publicity, it's just something that trademark law forces them to do.

The point here is that when Cisco lawyers learn that trade secrets were going to be revealed, they took control of the situation.

Corporate lawyers don't care about vulnerabilities. That's a PR, marketing problem, or engineering problem, not theirs.

But when it's trade secrets, they go scorched earth on the situation. Cisco's lawyers don't see the situation as we do, that of typical vuln disclosure at a BlackHat talk. They only saw the trade secrets issue.

Of course, it was false, there was no Cisco trade secret issue, but their primary contact at ISS had told them there was. They were techies, they didn't know. All they knew is that they had to fix it.

From this perspective, they were totally justified. Had ISS's representative been telling the truth, this is roughly how their lawyers should've responded. Remember: it's not optional according to the law. Cisco had marketing people who knew this would damage their reputation, and cybersecurity researchers of their own who knew this was an evil way of dealing with vuln disclosure. Level heads could not prevail because the lawyers were in charge, falsely believing it was trade secrets.

The meeting

ISS was getting pushback from Cisco, of course. Cisco was telling ISS that they needed to cancel the talk.

So shortly before BlackHat (the Friday before, I think), ISS had a meeting, pulling in all the principal players. I was "Chief Scientist", not directly part of the X-Force, but still, high up in the company.

The problem was that neither the ISS CEO nor Cisco CEO were available to discuss the problem. They were in Washington DC as members of President Bush's cybersecurity council.

There was this other communication channel — the CEOs talking directly to each other. Pete schmoozed the ISS CEO, who in turned passed on the information to the Cisco CEO, who then told his lawyers.

At the start of this meeting, the ISS CEO said that Pete was in charge of resolving the situation. This was very bad since it was Pete who had created the whole mess to begin with.

ISS couldn't actually cancel the talk. They couldn't call up BlackHat and have the talk canceled. Instead, the most they could do is threaten to fire Mike Lynn if he went through with the talk.

I tried to be the voice of reason. I tried to be on Mike Lynn's side. Canceling a talk like this was a Bad Thing, for ISS, for Cisco, for Mike Lynn. ISS shouldn't have been pressuring him to cancel the talk. They shouldn't have put him in that position.

But once he was in that position, his best personal course of action would've been to comply. He'd be the most famous hacker on the planet for a few months, but forgotten after. Getting a job in the field would be difficult after that, as nobody wanted an employee who would rage quit and dump trade secrets (even if, in my opinion, there really weren't any trade secrets). He did get a new job, at Cisco’s major competitor Juniper, but I don’t remember him giving another cybersecurity talk afterward.

All of us at the time, except for Pete, were caught in a web of misinformation. We didn't understand the forces that were causing the problem. Pete was in charge, and he was driving things to destruction.

Aftermath

It's after all this that the public story starts. Mike decided to give the talk anyway, Cisco lawyers came in and ripped pages from the conference book, and you know the rest.

From what I saw, this was not the case of a company suppressing vulnerability research. Instead, it was corporate politics gone awry, crushing an individual employee. I suspect there is a little of this in every such story: it's never really "corporations" making such decisions so much as individuals. How they make decisions is often flawed, such as pursuing their own corporate politics goals.

This recent (horrible) paper cites the Ciscogate scandal as everyone knows it, "thin skinned lawyers" suppressing a vulnerability. It's wrong, it was thick skinned lawyers who knew next to nothing about vulns but who were protecting trade secrets.