A Billion Blue Screens as Windows is Down Worldwide

by

A billion computer s worldwide needs to be rebooted in fail mode and have a driver manually. This was a mistake by Crowdstrike, where a bad update of Windows was performed.

The fix is to boot in safe mode and remove the bad csagent.sys or C-00000291*.sys drivers and then restart.

You will need to have admin access and the drive has to not be encrypted.

To fix the CrowdStrike bad driver problem, you can follow these steps:

Without Encryption:

Boot your Windows system into safe mode or the Windows Recovery Environment (winRE) by first powering up your machine until a manufacturer’s logo is visible.
Press the power button again to power up and the machine will boot to the winRE.
From the option menu, select Troubleshoot >> Advanced Options >> Startup Settings >> Restart.
Select option 5 or press F5 to restart the machine in safe mode with Networking.
Wait for the machine to boot into the safe mode desktop.
Open the File Manager and navigate to C:\Windows\System32\drivers\CrowdStrike
Look for and delete any files that match the pattern “C-00000291*.sys”
Reboot as normal.

With Encryption (BitLocker):

If your system is encrypted with BitLocker, you will need to enter the recovery key to proceed with the startup process. Once you have entered the recovery key, you can follow the steps above to delete the problematic driver file.

Remember to be cautious when deleting system files and ensure you have a backup of your important data before making any changes to your system.


If your drive is encrypted then this will be far more difficult.

CrowdStrike is a major cybersecurity technology company based in Austin, Texas. They provides cloud workload protection and endpoint security, threat intelligence, and cyberattack response service. They caused a billion blue screens of death with a faulty CrowdStrike software update. Affected machines were forced into a bootloop, making them unusable. The global outage is grounding commercial airline flights, temporarily taking Sky News offline, and even impacting 911 emergency call centers.

The blue screen of death is reported with a stop code of PAGE_FAULT_IN_NONPAGED_AREA from the csagent.sys driver.

Linux operating system (open source and more secure) supporters are all over twitter and other social media telling windows users. “I told you so”.

Crowdstrike Identified as Overvalued Before the Incident

The company, Crowdstrike, is being sold off and Microsoft has also lost about 2%.

Wall Street Bets had some people buy puts against Crowdstrike or short sell Crowdstrike. This was about 12 hours before the outage was caused.

Thesis: Crowdstrike is not worth 93 billion dollars (at time of writing).

Fear: CrowdStrike is an enterprise-grade employee spying app masquerading as a cloud application observability dashboard.

OBSERVATIONS

The 75th percentile retail investor has a tenuous grasp on “Cloud”, “Software Engineering”, and “Cyber Security”.

The median “Cyber Security Analyst” has a tenuous grasp on “Cyber Security”

The median “Software Engineer” has a tenuous grasp on “Cyber Security” and “Cloud”

The median retail investor has a tenuous grasp on “markets” and “liquidity pools”

CRITIQUES

Corporations could buy CrowdStrike to spy on their own employees.

CrowdStrike’s utility is limited- they simply collect all of their customer’s data and display it on a dashboard.

CrowdStrike is dangerous in that they have root access to every device(i.e. endpoint) across thousands of firms.

CrowdStrike customers sign up to get their firm’s data added to a bank which CrowdStrike then has license to use for “correlation”

CrowdStrike is a sitting-duck datamine for the FBI/NSA to subpoena.

CrowdStrike could potentially behave as a propaganda arm of the US government by creating “fake hacking stories” which are un-disprovable.They are able to do this due to information asymmetries in society.

Properly built “cloud applications” have security baked in by virtue of separation of concerns in the “software supply chain”. (e.g. containerization engine developer is different than the OS developer is different than the Cloud Infrastructure Provider).

CrowdStrike’s Falcon product contradicts their own guiding principle of “Zero-Trust Security”.

29 thoughts on “A Billion Blue Screens as Windows is Down Worldwide”

  1. Massive financial impacts (billions). What’s the remedy? Financial recourse? Probably none. Would not be real surprised if legislation emerges allowing litigation, regardless of the fine print in the software agreements. Products not fit for intended function due to screw-up.

    Microsoft’s deep pockets will likely be under siege. Bonanza for the legal profession. Seems fair to me.

  2. Certainly A Teachable Moment.
    The Hallmark of a successful, enduring, and epic Civilization is to be resilient, robust, and Backed-Up (data-wise, financially, and via essential infrastructure — personal, family, business, and region).
    The Touted, but not Really, epic Civilizations of Rome, Russia (sort of), China, Greece, something in South/Central America, etc., were aggressive and ambitious but ultimately fragile, simplistic, and of a single (leader, culture, infrastructure, etc.) — easy to collapse, confuse, and disintegrate – if over a generation rather than a weekend or a couple of ‘pandemic’ seasons.
    That’s the big difference with modern western culture – individual resilience, multiple energy and communications systems, a culture of savings and ownership, a certain skepticism with authority and The Commons — over the various other cultures differing in time and/or space.
    We need to seek out our fragilities – financial network (why cashless?), energy network, data and operating monoculture, supply chains, and some aspects of health and transportation — and vary, upsize, and back-them-up. It’s not about reducing risk and pulling back, as much, but allowing choice and over-engineered-ness. Not quite about dooms-prepping as much as choosing long-term quality and system back-up.
    It’s actually quite a fascinating thing to Google How Could Western Civilization Collapse.

    • Seems like a Trump-Vance-Musk ‘America First’ with a Libertarian, pro-localized supply chain policy framework could accomplish the undoing of many of those fragilities in the coming Quadrennial.

  3. In many ways, this is a narrative of “our times”, where millions of people (in this case IT peeps) become either entranced with, or required by management to fully invest in some “thing” that is to make systems “better”. Nothing bad seems to happen, and so it goes. The licenses are purchased for frequent auto-updates, and the system hums along. Then, it goes wonky.

    The problem is that C47100003_$.SYS is a black freaking box. Like the thousands and thousands of other black box files on every computer out there. (Don’t be too smug, Apple and Linux denizens … you’ve got thousands of black-box files too.). Something at-best innocent goes amiss and the update crashes THIS computer. Multiply by a million and you’ve got pandemonium. Who knows what the black box file was supposed to do? No one. It doesn’t matter. Its inscrutible computer gaga.

    Feels like the world of Necromancer. Gibson novel. Everything technology wise had in its dystopian future been reduced to black boxes. NO one knew how they really worked … but rather that they did, when hooked together the magical way. This is rapidly becoming our future, if not already here. Our automobiles have (now) hundreds of microprocessors. Somehow, in the 1970s, they had none. They have sensors to sense when other sensors are not sensing when yet more sensors should have alerted that an insensible thing has just happened and the car should be bricked. Right?

    Its our future.

    • The thing that’s making software vendors go gaga is “security in the kernel”.

      In the case of Linux, drivers are supposed to be open source and available for anyone to review.

      Very few do, of course, but usually any new drivers were under the scrutiny of Torvalds and his clique, who used to apply verbal abuse and shaming for any clear security violations.

      Not anymore. There are too many drivers, and the new younger developers don’t handle harsh words very well, so he more or less stopped.

      But surprisingly, there still are relatively few blunders due to upstream kernel drivers.

      Nevertheless we now have the nebulous threat of “security attacks” and “in the cloud” and everyone panics and accepts practices that would make Torvalds livid and perform a full-n Gordon Ramsay scene.

      This CrowdStrike thing is pushing kernel drivers into production systems, for supposedly finding malicious software, but it’s also that: a driver in kernel space that can easily bring down a system and depends entirely on the good will of its publisher to validate it before release.

      Something they clearly failed to do. Oh, but did you know they also did the same “error” with Red Hat Linux OSes not long ago?

      Sauce:
      https://access.redhat.com/solutions/7068083

  4. I was already up and running when thus happened, but we did feel it a little at work. I do mostly Linux, though. But critical infrastructure utilizing MS/CrowdStrike, bad, bad deal. Talk about goofing. It also feels, to me, kind of like a “all eggs in one basket” situation.

  5. This is probably the worst IT disaster in history, and it’s not even over yet. It’s hard to believe a single product – not even part of the MS OS – can wreck such deep level havoc. Most people who are responsible for these systems won’t have the expertise to follow the instructions, and the relative expertise of IT people who encrypted their drives will be matched by the added complexity of redoing their installs – all while being yelled at by irate customers and their bosses who are losing millions every hour. I remember staying up to 2am doing an emergency server switchover when the current server wouldn’t stop crashing. My whole department was down and I had to draft my bosses to update all the PCs looking for the old server. This is that to the 1,000th power, worldwide. Some shops will never recover. There will be Congressional investigations, promises to break up monopolies (as if having 10 Crowdstrikes instead of 1 will solve the fundamental over-reliance on technology). A few cutting edge AI systems that interact with Windows may complicate things, especially if the people who rely on AI have forgotten or never known, how to code.
    This is awful. I’m glad I’m out of that game. Whatever happened to Novell’s server OS? We never had these problems. Microsoft Windows should never have been used as a server OS. It’s always been hackable, buggy, and unreliable. Nothing has changed fundamentally since I left the IT world in 2006.

    • Meh. Y2K all over again. Public over-panic. Behind-the-scenes heroism, barely reported. All good.

      • This is what people thought Y2K would be, not what it actually was. Mass crashes of electronics rendering almost all computers and anything that relies on them unusable. Non of this came to pass because the bug was almost universally fixed before 2000. In our case, billions of computers have become unusable and require a hefty repair process before they can be used again. Fortunately, the cloud, which mostly runs on Linux, is fine, but hundreds, if not thousands, of companies are not functional right now due to their bricked devices.

  6. Imagine a future where Neuralink users get a bad update.

    “Will cell phones exist in the future? If you trust Elon Musk, founder of the brain implant company Neuralink, you might be inclined to think this technology will become obsolete.

    Just when we thought 2024 couldn’t get weirder, in a recent post on X, Musk suggested and responded to an X follower, stating, “In the future, there will be no phones, just Neuralinks.””

    See:

    https://thedebrief.org/the-future-is-going-to-be-weird-elon-musk-predicts-brain-chips-will-eventually-replace-phones/

  7. This is a crowdstrike problem. Faulty agent update pushed out to windows. Mac and Linux were unaffected. Yes, it impacted the windows platform with the falcon agent installed, but this was not a Microsoft code change or update.

    I somehow find myself in the strange position of defending Microsoft, but it is what it is.

  8. How is crowdstrike stock only down 12% on this? This should be a fatal error for a company tasked with security protection of critical computers.

  9. One does not simply reboot in to safe mode if your hard drive is encrypted with MS bitlocker. You must unlock your HD to boot in safe mode by looking up your bitlocker recovery key (using the internet).

    • This will be a showstopper for many.

      It requires to log into a Microsoft page, and know where exactly look for de key, then apply it.

      And that’s if your account has Admin access to the system. If not and you don’t have the Admin access, you’re toast.

    • Yup. bitlocker is the X drive. Rebooting into safe mode doesn’t help. It’ll be interesting to see what the solution is.

  10. Billions of computers bricked. Since you have to manually boot up the computer to fix, I could imagine the fallout from this lasting years, with many computers never being fixed.

    • I expect you could probably pre-package a cheap USB drive with the fix, boot off it once and you’re good, for people who are tech clueless. Then you need to deal with the inevitable distribution of infected fix-sticks…

      Happily not involved in this, myself.

      • It would have to be a usb boot thing. Laptops and desktops don’t come with floppy’s or CDs anymore. now in aging and org that blocked USBs…

        This is going to b painful.

  11. I got this last night but thought it was my old laptop failing and went to sleep. Woke up to realize it was a worldwide mess. Paralyzing companies and services in many countries.

    If this is a legit failure to check the update before releasing it, it’s a blunder worthy of lawsuits and bankruptcy for the involved.

    Happily, the fix is relatively easy (just boot in safe mode and delete a file, then reboot as normal). But you need to deal with bitlocker if you have a encrypted file system, thing that is extremely common on corporate builds and many commercial store-bought systems.

    If you don’t have things properly setup or ar beffudled by the bitlocker thingie, that means a visit to your local overwhelmed IT/store/repair worker.

    • I switched to the apple ecosystem years ago. I was mostly unaffected. Of course when I went to the gas station to fill up the truck, it was cask only because their Point of Sale system was down….

    • Somehow ties bsod, antivax and anti climate change all in the same rambling blog. I’ll stick with Brian.

  13. Cloudstrike separated the sheep from the goats, and the first world from the third world. Professionals run Linux. Australia down under did not make the cut.

    • Something fundamentally wrong if a single bad update from one company can wreak such havoc globally. Imagine what would happen if Microsoft update got hacked and pushed out something malicious? Catastrophic failure would ensue. We need more robust infrastructure. Let’s start with one simple step of never rolling out updates to millions of computers simultaneously, especially those running critical infrastructure. These things should be rolled out in small batches over time.

Comments are closed.