The CrowdStrike bug showed the world that one weak link in the chain can lead to chaos in a heartbeat

HardwareThe CrowdStrike bug showed the world that one weak link in the chain can lead to chaos in a heartbeatWhen you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

(Image credit: Diego Radames/Anadolu via Getty Images)

MADRID, SPAIN - JULY 19: A general view from Madrid-Barajas International Airport as passengers gather and wait due to the global communications outage caused by CrowdStrike, which provides cyber security services to US technology company Microsoft, on July 19, 2024 in Madrid, Spain. (Photo by Diego Radames/Anadolu via Getty Images)

Jump to:What happened?A weak link in the chainInterconnectivity and the butterfly effect

Occasionally a tech story makes its way out of our nerdy little bubble and into the big wide world. Outages, bugs, and cybersecurity incidents happen in the tech industry, but the damage usually isn’t large enough to warrant the sustained interest of the general public at large.

The CrowdStrike outagewas different. Health services, flights, global banking operations, significantly affected. All-too-familiar blue screens of death cropping up on billboards in Times Square. And a million headlines, many of which name-dropped Microsoft and borked Windows PCs as the root cause of the issue.

Of course, that’s not wrong, to a certain extent. It was indeed Microsoft Windows machines that unceremoniously fell over en masse. However, it was quickly revealed that, while Microsoft certainly had its part to play, another, lesser-known company was the cause of the issue behind the issue: CrowdStrike.

Previously a name not particularly well-known to the general population (at least, not to the same extent as Microsoft), CrowdStrike,a Texas-based cybersecurity firm, found itself embroiled in a global disaster. Now the dust has settled, more information has been revealed as to exactly what happened and why.

And while we certainly have more answers, they prompt some troubling questions. What does this sort of outage say about the stability of our ever-connected world? How could such a disastrous bug make its way into so many systems at once? And what’s to prevent events like it, perhaps on an even larger scale, happening in the future?

What happened?

This particular update was what the company refers to as a Rapid Response Content update. These updates are delivered as “Template Instances” which are themselves instantiations of a Template Type. Template Instances map to specific behaviours for sensors to observe and detect, and are a crucial part of Falcon’s cybersecurity protection capabilities.

The biggest gaming news, reviews and hardware deals

Keeping them updated, at a speed that keeps up with ongoing cybersecurity developments, is no easy task, especially without error. Still, these updates normally go through an extensive testing procedure. At the end of that testing procedure is a Content Validator, that performs validation checks on the content before publication.

This is a classic example of the cascading impact a simple upgrade can cause to multiple business sectors and in this case some critical infrastructure providers

According to CrowdStrike, it was this Content Validator that failed in its duties.

“Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data. Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.”

For a more detailed breakdown, CrowdStrike has since released anexternal technical root cause analysis(PDF) of the exact causes. Ultimately, however, its testing system let through an erroneous file, which was then pushed right to the heart of many machines at once.

(Image credit: Microsoft)

A generic blue screen of death in windows 10 os

Worse still, these machines were then stuck in a boot loop, in which the machine crashes, restarts, and then crashes again. For some this was a mild irritation, and a bad day in the office. For others, however, the stakes were much higher.

As Microsoft systems began crashing around the world,some 911 dispatchers were reduced to working on pen and paper. In Alaska, emergency calls went unanswered for hours, with similar issues affecting multiple emergency services worldwide. Doctors appointments and medical procedures were cancelled, as the systems behind them failed. Some public transport systems ground to a halt. Flights, banks, and media services went down with them.

Microsoft and CrowdStrike pulled the update before it could spread any further, with the latter releasing an update file without the error—but the damage was done, and chaos was well underway.

For a time, quick fix solutions were suggested. Microsoft’s Azure status page suggested that users repeatedly reboot their affected machines, with the suggestion thatsome of Microsoft’s customers had rebooted their systems up to 15 timesbefore the system was able to grab a non-broken update. Other alternatives included booting affected machines into Safe Mode and manually deleting the erroneous update file, or attaching known-working virtual disks to repair VMs.

(Image credit: Future)

Windows 11 emergency restart

CrowdStrike’s CEO issued a public apology. IT workers dug in for the weekend, setting to work fixing boot-looping machines. Eventually, Microsoft released a recovery tool, along witha statement estimating that 8.5 million Windows devices were affected, and that it had deployed hundreds of engineers and experts to restore affected services.

So what’s to prevent such a disaster from happening again? Well, from CrowdStrike’s perspective, its testing procedures are under review. The company has pledged to improve its Rapid Response Content testing, and add additional validation checks to “guard against this type of problematic content from being deployed in the future”.

But there’s a wider problem here, and it’s partly to do with The Cloud.

A weak link in the chain

In the vast, interconnected world we currently live in, it’s become increasingly impractical for huge service providers, such as Microsoft, to handle something as vital as cybersecurity and cybersecurity updates across all of its networks in-house. This necessitates the need for third party providers, and those third party providershave to be able to update their services, deep within the system, en masse and at speed, to keep up with the latest threats.

However, this is essentially like handing someone the keys to your house to come and check the locks while you’re away, and hoping they don’t knock over your Ming vase in the process. With the best will in the world, mistakes will be made, and without overseeing things yourself (or in Microsoft’s case, itself), responsibility is left to the third party to ensure that nothing is broken in the process.

However, if that third party fails, it’s you that ultimately takes the blame. Or in this case, Microsoft, at least in terms of public perception. It might have been CrowdStrike’s name in the headlines, but right next to it was Microsoft, along with images of blue screens the world over—an image tied so distinctively to the perception of Windows instability that it’s become representative of the term ‘system error’ in many people’s minds.

There were a lot of smug Linux folk out there crowing about how happy they were to not be using Microsoft’s ubiquitous OS, even thoughCrowdStrike had bricked a bunch of Linux-based systems just a month earlier. But because of the pervasiveness of the Windows ecosystem, that failure didn’t have the widespread institutional damage or media attention that the Microsoft-related error carried.

How we did this in the old days:When I was on Windows, this was the type of thing that greeted you every morning. Every. Single. Morning.You see, we all had a secondary “debug” PC, and each night we’d run NTStress on all of them, and all the lab machines. NTStress would… pic.twitter.com/rZkvpujbcrJuly 20, 2024

Interconnectivity and the butterfly effect

And then there’s the problem of interconnectivity as a whole. Many vital systems arenow so wholly dependent on cloud providersand online updates that, despite staging and rigorous testing procedures, a small mistake can magnify itself quickly. In this case, so quickly that it was able to take down millions of machines in one fell swoop, and many of those machines were crucial for maintaining a vast network of others.

Not only that, but in a world of increasing cyberattacks and more and more third party providers attempting to defeat them,speed is of the essence. In the ongoing game of cat and mouse between cybercrime and cybersecurity providers, those who snooze will inevitably lose. It’s telling that, at the heart of the issue here, was a “Rapid Response” update.

AsMuttukrishnan Rajarajan, Professor of Security Engineering and Director of the Institute for Cyber Security at City University London, puts it:

“As cyber threats are evolving at a rapid phase these companies are also under a lot of pressure to upgrade their systems. However, they have limited resources to scale at the level they need to manage such upgrades carefully as there are a lot of interdependencies in the supply chain.

“This is a classic example of the cascading impact a simple upgrade can cause to multiple business sectors and in this case some critical infrastructure providers.”

(Image credit: David Paul Morris/Bloomberg via Getty Images)

The Microsoft Corp. Windows Recovery screens displayed at George Bush Intercontinental Airport, in Houston, Texas, US, on Saturday, July 20, 2024. Airlines around the world experienced disruption on an unprecedented scale after a widespread global computer outage grounded planes and created chaos at airports. Photographer: David Paul Morris/Bloomberg via Getty Images

In adigital monoculture, single vulnerabilities across an interconnected set of systems can create a butterfly effect that ripples through infrastructure worldwide. Currently,15 companies worldwideare estimated to account for 62 percent of the market in cybersecurity services. That’s a lot of eggs in relatively few baskets.

Here, that potent combination resulted in a broken update that spread too quickly, to too many systems—and chaos was the result. Move fast, break stuff, goes the adage. And in this case, a whole lot was broken here.

TOPICSHardware

TOPICS

More about hardwareIf you’re trying to convince me your ‘companionship’ robot is ‘lifelike’, maybe don’t rip her face off in the demo videoIs the new RTX 5070 really as fast as Nvidia’s previous flagship RTX 4090 GPU? Turns out the answer is yes. Kinda.LatestToday’s Wordle answer for Monday, January 13See more latest►

More about hardware

If you’re trying to convince me your ‘companionship’ robot is ‘lifelike’, maybe don’t rip her face off in the demo videoIs the new RTX 5070 really as fast as Nvidia’s previous flagship RTX 4090 GPU? Turns out the answer is yes. Kinda.

A robot having its face pulled off

If you’re trying to convince me your ‘companionship’ robot is ‘lifelike’, maybe don’t rip her face off in the demo video

Nvidia RTX 5070 graphics card

Is the new RTX 5070 really as fast as Nvidia’s previous flagship RTX 4090 GPU? Turns out the answer is yes. Kinda.

LatestToday’s Wordle answer for Monday, January 13See more latest►

Latest

Today’s Wordle answer for Monday, January 13

Wordle today being played on a phone

Today’s Wordle answer for Monday, January 13

See more latest►

Most PopularDragonflight got WoW back on its feet, now we get to see if Blizzard can make the Worldsoul Saga runThis bizarre roguelike has a new take on the Vampire Survivors formula: letting you build your own custom weapons out of brains, eyeballs, and chimpanzee spines18 games the PC Gamer team can’t wait to play in 2025The Witcher 3’s now 2-year-old bonus quest is our first taste of the ‘vibe’ CD Projekt is going for in The Witcher 42024 was the year updates for old games beat out all the new ones for meTrain like you game with this adventure-inspired workout’It’s simply impossible to make a difficulty level that’s just right for all players': How Final Fantasy 14’s lead battle designer has been playing a precarious balancing game for Dawntrail’s dungeons and raidsPlease join me in getting super excited for all the cool looking survival games coming in 2025 (and beyond)Competitive shooters are at a crucial crossroads in 2025: ‘sweaty’ teamplay vs. casual funCall of Duty’s $28 Squid Game skins are the perfect crossover for our capitalist dystopia, and Activision knows exactly what it’s doingThese are the 14 biggest upcoming RPGs of 2025—get ready for another amazing year for the genre

Most Popular

Dragonflight got WoW back on its feet, now we get to see if Blizzard can make the Worldsoul Saga run

This bizarre roguelike has a new take on the Vampire Survivors formula: letting you build your own custom weapons out of brains, eyeballs, and chimpanzee spines

18 games the PC Gamer team can’t wait to play in 2025

The Witcher 3’s now 2-year-old bonus quest is our first taste of the ‘vibe’ CD Projekt is going for in The Witcher 4

2024 was the year updates for old games beat out all the new ones for me

Train like you game with this adventure-inspired workout

‘It’s simply impossible to make a difficulty level that’s just right for all players’: How Final Fantasy 14’s lead battle designer has been playing a precarious balancing game for Dawntrail’s dungeons and raids

Please join me in getting super excited for all the cool looking survival games coming in 2025 (and beyond)

Competitive shooters are at a crucial crossroads in 2025: ‘sweaty’ teamplay vs. casual fun

Call of Duty’s $28 Squid Game skins are the perfect crossover for our capitalist dystopia, and Activision knows exactly what it’s doing

These are the 14 biggest upcoming RPGs of 2025—get ready for another amazing year for the genre

HARDWARE BUYING GUIDESLATEST GAME REVIEWS1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop3Best mini PCs in 2025: The compact computers I love the most4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested1Thank Goodness You’re Here! review2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review3WD Black SN850X 8 TB NVMe SSD review4Ikea Utespelare desk review5Asus ROG Harpe Ace Mini wireless mouse review

HARDWARE BUYING GUIDESLATEST GAME REVIEWS

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

A bunch of the best Steam Deck accessories on a blue background.

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

3Best mini PCs in 2025: The compact computers I love the most

Best mini PCs in 2025: The compact computers I love the most

4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

1Thank Goodness You’re Here! review

Thank Goodness You’re Here review

1Thank Goodness You’re Here! review

Thank Goodness You’re Here! review

2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

3WD Black SN850X 8 TB NVMe SSD review

WD Black SN850X 8 TB NVMe SSD review

4Ikea Utespelare desk review

Ikea Utespelare desk review

5Asus ROG Harpe Ace Mini wireless mouse review

Asus ROG Harpe Ace Mini wireless mouse review

What happened?#

The biggest gaming news, reviews and hardware deals#

A weak link in the chain#

Interconnectivity and the butterfly effect#

What happened?

The biggest gaming news, reviews and hardware deals

A weak link in the chain

Interconnectivity and the butterfly effect