CrowdStrike outage demonstrates how industry may respond to a Cyber Pearl Harbor

So many have spoken of a Cyber Pearl Harbor. What would it look like? How likely is it? Could it be right around the corner?

While specific industries, organizations and agencies have had their share of significant incidents, few have had the broad-based impact of CrowdStrike's somewhat self-induced catastrophe. Along with it, we have learned if we are prepared.

While there were significant outages, disruption and loss of business, and an overall lack of system resilience, I am very encouraged overall how everyone responded. I have seen CISOs from all regions and industries collaborating real-time to help one another.

This is the kind of response we all hope for and one that is necessary when so many are impacted. They broke down the silos, used several forms of protected communications to share their team’s insight and experiences from the onset and over hours and hours of challenges and trouble shooting. It is clear practitioners have done their job exercising the response muscles they need for such an incident, but when it "hits the fan," it's great to be able to reach out to colleagues that are facing the same problems and having them respond in such supportive and helpful ways. What we are confirming is that most every security implementation has so many variations that the workarounds provided as guidance likely did not work the same for all. There were twists and turns along the way as organizations found their way through the blue screen of death. We saw real-time trouble shooting and trial and error at its best.

I am never a fan of doing a rollout midweek when the markets are open, but there may have been a reason and or need for it. It was refreshing to see leadership at CrowdStrike standing up, taking responsibility, and working relentlessly to help. I imagine this led to a better-than-expected result in what looked like a very dark few days. I expect to see much more scrutiny in updates going forward to include verification of testing, requests in advance for documented disaster recovery plans and greater justification for timing of these events.

I do think we have to expect these types of issues when so many are so reliant on any one industry leader. This will not get better as consolidation continues to accelerate and the major players with the greatest market share swallow more and more companies taking on more and more responsibility and therefore risk.

What made this even more eventful was that so much of the fix was manual. Response teams struggled with bitlocker recovery and the automated BSOD workaround was not working. Many of these organizations have tens of thousands of machines impacted. Massively manual -- thus it took an extraordinary effort to resolve.

To complicate things further, threat actors were taking advantage of the moment. In real-time critical processes had to be established to monitor for known malicious domains. Everything was moving so fast and it was important for individuals, teams and organizations to ensure they were heeding guidance from legitimate sources as there was so much noise out there on social channels.

It was amazing to see so many keeping a positive stride and pushing though. Humor was honestly a welcome respite for many that were running on adrenaline.

Days after, we find many lessons learned. Those in the trenches were empowered by three guiding principles.