Right now, much of the world has ground to a halt thanks to a botched update from cybersecurity provider CrowdStrike. Banks, hospitals, 911 call centers — our own Daniel Golson is even stranded in Schiphol Airport as we speak. Yet, judging by CrowdStrike’s response, the folks worst hit by the outage might be the IT workers who actually have to implement the company’s fix.
CrowdStrike issued a tech alert for the problem earlier today, which traced the error back to a single driver file. The company listed the steps for a workaround: Just boot your machine into Safe Mode, delete the file, restart, and everything will be good as new. Simple, right? Well, it would be, if not for the sheer number of machines affected — including many that can’t even be rebooted in this way.
Booting into Safe Mode isn’t a particularly scalable process — IT admins with massive networks of affected systems will have to address each issue manually, spending potentially hundreds of hours fixing CrowdStrike’s mistake. Yet these admins, somehow, can still count themselves lucky. For many, whose Windows machines are virtualized through providers like AWS, Safe Mode simply doesn’t exist.
For those admins, CrowdStrike has a solution: Disconnect your server’s storage, spin up a new server, connect the old storage to the new server without booting from it, remove the affected file, and then put everything back together. This process is a hassle, but not impossible to do — unless, of course, your security-minded CrowdStrike-customer company encrypts its storage with BitLocker and stores the encryption keys on a server that’s also affected by the outage.
The issues here hit all the parts of computing that most users never have to think about, but for IT admins the CrowdStrike fiasco is a near-perfect storm of problems that interact with each other in inconvenient ways. Systems will be fixed, the world will return to normal, but don’t expect it to happen today. Be patient, wait it out, and trust that eventually everything will come back online. And, next time you see a sysadmin at a bar, buy them a drink. They deserve it.