Emergencies are an inherent part of cybersecurity. We need to budget for some number of emergencies, where employees are expected to work nights and weekends. On the other hand, organizations will often exploit this, whereby employees regularly work nights and weekends in a chronic state of emergency.
I though I’d write up some notes on this. Last week, the Internet experienced its worst cybersecurity threat in years, the “log4j” vulnerability. This was a legitimate emergency, requiring employees to work overtime, especially as we worked through multiple patches. At some point, though, somebody is going to need to declare the emergency over. It doesn’t mean the tasks are done, that log4j isn’t still a threat, but only that it’s no longer an emergency.
I’m thinking of some sort of “green→yellow→red” chart of cyber threat levels, similar to that DHS terrorist threat chart after 9/11.. My hypothetical chart would look something like the following:
GREEN - there are many pressing issues, but they’ve been around for a while, so no reason to believe this week is significantly different than last week, which is 98% fo the time
YELLOW - there’s an recent issue that’ll be at the top of your priority list to deal with during work hours, lower priority tasks can be delayed, which is 2% of the time
RED - there’s a recent issue that means employees are working overtime and weekends, which happens about once a year
We jumped to “red” on December 10th, where people agreed log4j was a major problem. It should’ve dropped down to “yellow” in few days, meaning, employees should’ve stopped staying after hours to work on the problem.
What should happen is the reprioritizing of tasks. Lower priority tasks can be pushed off into the future so that employees can work on this urgent task now. The lowest priority items can be dropped entirely. Thus, after a few days of 12 hour work days, employees should be going back to the regular 8 hour work days.
But it’s rare for that to happen. Instead, organizations continue business as usual. Nothing has been reshuffled, and all the work this event requires is still being done as something extra, with employees still continuing to work late.
That organizations fail at this isn’t because they are evil, but because they are lazy. If employees seem willing to stay late to work on things, then the organization just lets them. If fellow employees are volunteering their time to stay after work, then it’s hard to be the lone holdout. Their managers may not even notice that it’s happening. If employees are staying late, then a manager isn’t going to stand up and tell them to go home.
Going quickly back to “green” is the key point. That DHS Threat Level chart created after 9/11 had 5 levels, but never went bellow level 3. Even though they got rid of the chart in 2011, the fact remains that the temporarily “elevated” procedures still exist. Twenty years later, we are still under the state of emergency declared after the 9/11 attack.
It takes less political effort to move toward red, to becoming more concerned about an exceptional event. It takes a lot of political work to move back the other direction, to return to a situation of normalcy. That’s why the government fails at it, no politician is willing to look weak in the face of threat.
That’s the point of this post. Nobody should still be at “red” level, and most organizations should be at “green”. This log4j bug will continue to be a threat, and there will likely continue to be required patches. But this needs to be integrated to daily operations rather than continued to be treated as an exception. A year from now, log4j hacks will still be happening — but you should not be still under a state of emergency.
As a side note, I wanted to address this:
Actually, I can’t think of a better system than watching Twitter come to consensus on an emergency. The unofficial system works at least as quickly and reliably as any official system. I’d hate if people relied upon my judgement alone, or any small cabal.
If any anointed group existed, their responsibility should also include declaring the emergency over. In truth, that would be different for different organizations. But there should still be a matching declaration that applies to the bulk of the industry.