The antivirus update that broke the internet

Many people may have heard or seen the events regarding the blue screen of death on the majority of windows based systems at airports (over 200 indigo flights were canceled and they temporarily had to issue handwritten boarding pass), hospitals, banks, organizations, retail stores, transportation systems, and even NY times square advertised “The Blue Screen of Death” of windows.

At first, it seemed like a cyberattack, others blamed Microsoft directly, not knowing what really happened. But it was all because of a faulty automatic update.

The incident revolves around CrowdStrike, which is a cybersecurity company that specializes in endpoint and cloud protection. They offer a wide range of security solutions designed to prevent, detect, and respond to cyber threats.

But the issue faced by the world of the internet was caused by a faulty update of CrowdStrike Falcon, which is a cloud-native platform that provides endpoint protection. It uses AI and ML to identify and stop threats in real-time, even those that have never been seen before using behavioral analysis.

Why CrowdStrike in Microsoft? 

CrowdStrike Falcon, is integrated with Microsoft Defender, Microsoft's built-in endpoint security solution. This integration provides customers with several benefits including simplified security management, enhanced protection among others.

The general daily windows users didn't face this outage because this falcon software is primarily designed for enterprise and business environments. It's not typically available for individual home PCs or laptops as a standalone product.

The focus of CrowdStrike Falcon is on protecting large networks of devices and sensitive data within organizations. Such as managed threat hunting and advanced threat intelligence, are one of the specific needs of businesses and enterprises.

You might encounter CrowdStrike Falcon on a personal device, If you use a company-issued laptop or PC for work, it's possible that your organization has deployed CrowdStrike Falcon on it to protect their network and data.

What exactly happened in the incident?

The CrowdStrike Falcon failure that occurred on July 19, 2024, was caused by a defect in a Falcon sensor configuration update for Windows systems that was released by CrowdStrike. 

This update triggered a logic error in a specific “configuration file” (Channel File 291) that was part of the behavioral protection mechanisms used by the Falcon sensor (Falcon sensor is the core agent for the Falcon platform installed on all endpoints (desktops, laptops, servers, etc.))

And since this sensor operates on kernel level this leads to an entire system crash.

How the Falcon sensor works?

• Data Collection:

  • It continuously monitors the devices for any suspicious activity, collecting data on processes, network connections, file changes, and other events. 

  • The data is then sent to the CrowdStrike cloud for analysis.

• Threat Detection:

  • In the cloud, the collected data is further analyzed using ML algorithms and Cyber threat intelligence to identify potential threats, such as malware, exploits, and suspicious behavior.

• Threat Prevention and Response:

  • If a threat is detected, the Falcon sensor can take action to prevent or contain it.

• Blocking malicious processes

• Quarantining suspicious files

• And, Isolating the infected device from the network

• Also providing detailed information about the threat to security teams for further investigation and response on a single dashboard.

Here's a summary regarding the failure:

• Cause: A faulty content update for Windows hosts, not a cyberattack. 

• Impact: Primarily affected Windows devices running Falcon sensor version 7.11 and above that were online between 04:09 UTC and 05:27 UTC on July 19, 2024. Mac and Linux hosts were not affected. 

• Resolution: CrowdStrike identified and isolated the issue quickly, deploying a fix within hours. The problematic configuration file was reverted, and systems that were brought online after the fix was deployed were not impacted.

• Manual Recovery: For systems that didn't recover automatically, CrowdStrike provided instructions for manual removal of the problematic file.

A basic overview of the steps they recommended:

1. Boot Windows into Safe Mode or the Windows Recovery Environment

2. Navigate to the crowdstrike directory, by default it would be “C:\” such as “windows\system32\drivers\crowdstrike”

3. Locate the file matching “C-00000291*.sys” configuration file and delete it.

4. Boot the host normally.

Warning:

For proper guided instruction, you should follow their blog post where they explained everything from “what, why, how” of the incident. And the “FIX” as well.

Originally posted on LinkedIn LINK