CrowdStrike Confirms Root Cause of Falcon Sensor Outage; Healthcare Losses Anticipated to be $1.94B
As promised, CrowdStrike has published the root cause analysis of the faulty Falcon Sensor software update that caused Windows devices around the world to crash. CrowdStrike had previously published the preliminary findings from its investigation, which confirmed that this was a Channel File 2971 incident caused by a faulty update involving a new Template Type. The purpose of the update was to improve visibility into novel attack types.
The new Template Type had previously been used without incident; however, on July 19, 2024, despite passing multiple levels of testing, the update triggered an out-of-bounds memory read issue, causing Windows devices to get caught in a loop and display the Blue Screen of Death. CrowdStrike has now confirmed that several shortcomings have been identified that led to the crash, the most significant of which was a parameter mismatch in its rapid response content update. Falcon Sensor was expecting to receive 20 input fields but instead received 21, triggering an out-of-bounds memory read. The update on July 19, 2024, was the first IPC Template Type to make use of the new 21 input parameter field.
CrowdStrike has made several updates to prevent similar issues in the future, including increased testing during Template Type development, conducting additional checks of its Content Validator to prevent mismatching, further development layers and acceptance checks have been added, customers have been given greater control over the delivery of Rapid Response Content, and future updates will be rolled out gradually to reduce any impacts from any future faulty updates. CrowdStrike has also engaged two independent third-party software security vendors to conduct a review of the Falcon Sensor code for security and quality assurance.
July 26, 2024: CrowdStrike Issues Update on Outage Cause; Healthcare Hit Hardest with Estimated $1.94B in Losses
Earlier this week, CrowdStrike provided an update on its review of how a defective Falcon sensor software update caused around 8.5 million Windows computer systems to crash. CrowdStrike said the problem occurred with its Rapid Response Content, which is used to perform behavioral pattern-matching operations on the sensor. Rapid Response Content is delivered as template instances, which add new capabilities to the sensor that enable new telemetry and detection. Each template maps to specific behaviors for the sensor to observe, detect, or prevent.
Get The FREE
HIPAA Compliance Checklist
Immediate Delivery of Checklist Link To Your Email Address
Please Enter Correct Email Address
Your Privacy Respected
HIPAA Journal Privacy Policy
In February 2024, CrowdStrike issued a new InterProcessCommunications (IPC) template and conducted a stress test in March on various operating systems, then released a further three IPC template instances in April without any problems. A further two IPC templates were released in July, and both passed Crowdstrike’s validation processes; however, one of those templates contained defective content. The defective content passed the validation checks due to a bug in the Content Validator. The defective content resulted in an out-of-bounds memory read, which triggered an exception that resulted in a Windows operating system crash – the blue screen of death (BSOD). CrowdStrike has promised to release a full root cause analysis into the outage.
CrowdStrike said it will be improving its testing, conducting additional validation checks, and implementing a staggered deployment strategy for Rapid Response Content, starting with a canary deployment. The updates will be released internally initially, and then gradually rolled out out to larger portions of the sensor base. CrowdStrike will also be conducting multiple third-party security code reviews and independent reviews of end-to-end processes, from development through deployment. CrowdStrike said a significant number of the computers affected by the faulty update are now back in operation. Lawmakers in the US House of Representatives have called for CrowdStrike CEO George Kurtz to testify to Congress about the role of the company in the global IT outage.
Fortune 500 Firms Estimated to Have Lost $5.4 Billion
While fewer than 1% of Windows devices were affected by the update, it caused massive disruption globally. Around half of Fortune 500 firms use the Falcon platform, although only half of those firms appear to have experienced disruption. Parametrix, a provider of cloud outage analytics and insurance solutions estimates that 125 U.S. Fortune 500 firms experienced disruptions due to the faulty update and estimates that the collective direct losses are likely to be around $5.4 billion. Parametrix did not include Microsoft in its calculations.
Parametrix said insurance policies are likely to be triggered by the outage, as policies generally cover system failures from non-malicious acts, including human error. However, large risk retentions and policy limits are likely to limit coverage to around 10% to 20% of losses. Parametrix estimates insurance payouts of between $540 million and $1.08 billion.
The biggest losses are thought to have been experienced by the healthcare industry. Healthcare is estimated to have suffered direct losses of $1.94 billion, with an average estimated loss of $64.6 million per company, with the banking sector also experiencing high losses of $1.15 billion and an average loss of $71.84 million per company. These two sectors will have to absorb more than half of the total financial losses caused by the outage. While the overall losses at airlines are lower at an estimated $860 million, airlines are estimated to suffer the highest direct losses per company at $143.48 million.
