Crowdstrike: The Aftermath – PSW #836

(Note: The link above is just a placeholder) - The following articles were super interesting and worth a read, though extremely technical and in some cases describing new attack techniques:

• https://www.elastic.co/security-labs/false-file-immutability
• https://srcincite.io/blog/2024/07/21/jndi-injection-rce-via-path-manipulation-in-memoryuserdatabasefactory.html
• https://intezer.com/blog/incident-response/how-to-analyze-malicious-msi-installer-files/
• https://blog.doyensec.com/2024/07/18/custom-actions.html

This incident highlights just how fragile our critical infrastructure is today. I am gravely concerned about the future if we don't start making our infrastructure more resilient, it's just too fragile. I did pull on a thread about monopolies and monocultures, it's really a dotted line as we do not live in a world with true monopolies (for the most part, and this is a good thing). However, we do have these sub-monocultures as I call them. Large numbers of computers share largely the same technology stack, which leaves us vulnerable. While Crowdstrike may only have an 18% market share, the impact of the bad update affected 8.5 million computers and took out critical infrastructure across the globe. That number represents a mere 1% of Windows systems, yet it causes chaos. China is looking at this going "Heh, just wait until you see what we have planned". Below are some other interesting articles and takes on this event:

• CrowdStrike file update bricks Windows machines around the world - "Go to C:WindowsSystem32driversCrowdStrike and Locate and delete file matching "C-00000291.sys"* - Does this work for all or most of Crowdstrike users? What do you do if this doesn't work? Just wait until the machine pulls the new update. Did rebooting 11 times also provide a fix?
• CrowdStrike Outage: Microsoft Points Finger at EU Agreement - The EU agreement, to prevent unfair competition, says that MS must open the APIs that provide kernel access. However, Apple and Google are not in the same boat: "Apple even notified developers that kernel-level access would no longer be provided, which did affect some Mac software but improved system stability and security." This brings up an interesting point of control and freedom, versus a closed environment that is locked down but perhaps provides a more stable and secure environment because not everyone can access the kernel in a way that leads to instability.
• CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes - Does this indicate that testing procedures have been broken for some time and Linux kernel panics were an early warning system? "Linux kernel panics and Windows Blue Screens of Death are broadly comparable. The occurrence of kernel panics mere weeks before CrowdStrike broke many Windows implementations therefore hints at wider issues at the security vendor."
• systemd Talks Up Automatic Boot Assessment In Light Of The Crowdstrike-Microsoft Outage - I do believe this is a really nice feature, even though it is in systems that open up more attack surface and tries to do everything and be everything, which could be a huge downside. It can detect a kernel panic and then load a previous kernel. My question is if the issue is in a kernel module, the previous kernel will still load the bad kernel module, does it have enough intelligence to exclude a kernel module that is causing the panic?
• When the Crowd Strikes Back - Did Azure cause the update file to be corrupt?
• How did a CrowdStrike config file crash millions of Windows computers? We take a closer look at the code
• CrowdStrike blames a test software bug for that giant global mess it made
• Avoid another CrowdStrike takedown: Two approaches to replacing Windows

Basically, the driver tried to access a memory region that did not exist. Great insights from Omkar from OpenSSF: "So normally, if you've got a driver that's acting kind of buggy and causes a failure like this, Windows can auto resume by simply not loading the driver the next time. But if it is set as boot-start, which is supposed to be reserved for critical drivers, like one for your hard drive, Windows will not eliminate that from the startup sequence and will continue to fail over and over and over and over again, which is what we saw with the CrowdStrike failure."

Making assumptions in your testing leads to very bad results: "Whatever the Validator does or is supposed to do, it did not prevent the release of the July 19 Template Instance, despite it being a dud. That happened because CrowdStrike assumed that tests that passed the IPC Template Type delivered in March, and subsequent related IPC Template Instances, meant the July 19 release would be OK."

I do not believe replacing Windows with Linux is the answer with respect to the Crowdstrike incident. Linux is just as vulnerable to this type of flaw. Yes, we dodged a bullet on this one, but changing tech stacks is just shifting the problem. The key to solving this problem is to be more resilient. Did anyone have "Bad Crowdstrike update" on there incident response table top bingo card? I bet you will now...

I want to point out that true 0-Day vulnerabilities do not end up on the CISA KEV, only those that there is a patch or in some cases, clear remediation, end up on the KEV (sometimes the patch is to remove the device from the network, e.g. IoT routers that have no patches). US federal agencies are required to fix vulnerabilities on the KEV, so if there is no patch, they can't do that. This article is just trying to get attention. Please tune in next week to the Below The Surface podcast where I interview Tod Beardsley from CISA who is one of the lead people working on KEV.

"Dragos researchers wrote that they believe the hackers in control of the FrostyGoop malware first gained access to the targeted municipal energy company’s network by exploiting a vulnerability in an internet-exposed MikroTik router. The researchers said the router was not “adequately segmented” along with other servers and controllers, including one made by ENCO, a Chinese company." - We patch software, but somehow we just seem to forget that devices have firmware, which is software, that also needs to be updated. Also: "The FrostyGoop malware is designed to interact with industrial control devices (ICS) over Modbus" - which allowed attackers to essentially turn off the heat in January by: "The adversaries did not attempt to destroy the controllers. Instead, the adversaries caused the controllers to report inaccurate measurements, resulting in the incorrect operation of the system and the loss of heating to customers,"

The honeypot detection and non-functional exploits part is interesting here: "The goal of this exploit is to find vulnerable machines. The "double obfuscation" is likely supposed to bypass some filters and better discriminate against honeypots. I have seen "non functional" exploits used to detect honeypots by attempting to fingerprint the error message returned. Maybe a pattern to add to our honeypots after lunch."

An interesting walkthrough of the code may be useful to find other vulnerabilities in other similar firmware.

I'm a huge fan of EMBA! Recently I've been creating some custom scan profiles for certain situations. I will be releasing these next week on the Eclypsium blog, so please check back!

Matt was not a fan of the device he was using and did not recommend it. If you are doing professional work with reading SPI flash chips and/or eMMC go with a Dediprog (at least that's what my friends tell me).

This has nothing to do with Sonicwall, but this vulnerability just happened to come out in the same timeframe as the Crowdstrike thing. Most people do not apply firmware updates to their appliances and IoT devices. If they do, typically lower severity vulnerabilities, like DoS, don't get patched. If an adversary were to collect DoS vulnerabilities they hit strategic targets and exploit them all at once, it could be a bad day. So, my PSA is patch, patch, and patch more.

"A vulnerability in the authentication system of Cisco Smart Software Manager On-Prem (SSM On-Prem) could allow an unauthenticated, remote attacker to change the password of any user, including administrative users. This vulnerability is due to improper implementation of the password-change process. An attacker could exploit this vulnerability by sending crafted HTTP requests to an affected device. A successful exploit could allow an attacker to access the web UI or API with the privileges of the compromised user." - While the Smart Manager will default to HTTPS, my guess is that it will respond to HTTP requests as well, allowing this attack to occur. This attack is bad, essentially it is a global authentication bypass as it allows an attacker to change the password of any user and access the device. I feel like we need to give this its own CWE.

Another from Cisco: "This vulnerability is due to improper handling of email attachments when file analysis and content filters are enabled. A successful exploit could allow the attacker to replace any file on the underlying file system" - There has been several of these types of vulnerabilities in SEGs in the past, I believe Barracuda had one last year as well.

Similar to TunnelVision, this is a decent explanation: "Port shadowing allows an attacker to intercept and redirect encrypted traffic and learn more about the origins and connections operating across the VPN itself. It is an old issue that was written about several years ago and deals with a flaw that identifies which applications are communicating through a VPN by their network port. Port shadowing depends on Network Address Translation (NAT) and how the VPN software consumes NAT resources to initiate connection requests, allocates IP addresses, and sets up network routes. If exploited, it could be used to conduct man-in-the-middle attacks. As with Tunnelvision, exploited users have no indication that their sessions have been compromised; other than that, the two vulnerabilities use different aspects of the VPN infrastructure and operate differently." Source: https://www.csoonline.com/article/3476176/port-shadow-a-new-vpn-weakness.html

Interesting usage of shared object injection, typically used to exploit SUID/SGID binaries, in this case they exploited a set of binaries: one for user interaction with low privileges, and one with root privileges that used IPC and DBUS to communicate, ultimately: "The shared object injection vulnerability came into play again, allowing us to intercept the library calls we make by passing them through our shared library and injecting our code into the application. " - Also, the memes in this post are ON POINT, and a great write-up too. I learned something.

One of the best breakdowns of container escapes, complete with a really easy-to-read description, on a high level, of how containers work some of the Linux primitives that are involved in the setup of a container, and how attackers leverage certain techniques to break out. Well done!

PAUL: I had this one too, turning off the heat in January is a psyop for sure.

PAUL: This is amazing hacking!

PAUL: Storing the SSID and encryption key on an AWS server is not a feature Linksys, its a potential disaster for your user's security and privacy.

PAUL: Had they not begun loading malware I wonder how long they could have worked there and done other things that are not as obvious (like exfiltrate IP).

The exploit has been dubbed "EvilVideo." The vulnerability itself is due to a default setting that results in Telegram automatically downloading media files to Android devices. Telegram has released a fix for the vulnerability. CVE-2024-7014 - Weaponized exploit code is publicly available. Telegram issued an update.

A corrupt sensor configuration update for CrowdStrike’s breach-prevention Falcon platform caused massive outages around the world on Friday, July 19. While users can configure sensor update policies for Falcon, in this case, the corrupted file was a configuration or signature update, which is applied upon release. The faulty update triggered a logic error that caused systems to crash. The outages affected an estimated 8.5 million Windows devices.

Consider this a bad malware definition update, remember those are pushed as rapidly as possible to protect us from harm. In this case, a newly observed C2 frameworks which utilized named pipes. At this point, I'm predicting CrowdStrike will have an industry leading QA process to insure this never happens again.

The Remediation and Guidance Hub includes a statement from CrowdStrike CEO George Kurtz; technical details about the outage; guidance for identifying and remediating impacted hosts, recovering Bitlocker keys, and cloud-based environments; and third party vendor information.

If you're a CrowdStrike shop, odds are your helpdesk had a line with laptop recoveries while other parts of your IT staff likely spent the weekend returning servers to service. Be careful to check any "quick fixes" as adversaries are pushing out alternative solutions which are not what they appear. CrowdStrike also offered an option, if requested, to mark the bad update for your enterprise which would allow systems which were online briefly to possibly self-remove the offending file. You're going to want to leverage all the legitimate options.

Microsoft has released a USB tool to help users recover from outages caused by the corrupted CrowdStrike Falcon sensor configuration update. The tool requires a Windows 64-bit client with at least 8GB of free space. The outages have affected an estimated 8.5 million Windows devices.

The tool provides two recovery mechanisms, either WinPE or safe mode. WinPE doesn't require an admin account but may require manual entry of BitLocker recovery key. Safe mode works if you're using TPM only BitLocker, but if you're using TPM+PIN you need the PIN or Bitlocker recovery key. You'll also need an account with administrator rights to access the system. If you're using FDE, other than BitLocker, you're going to need to use that providers recovery mechanism.

Speaking at Aspen Security Forum on Friday, July 19, US Deputy National Security Advisor Anne Neuberger and Secretary of State Anthony Blinken observed that the CrowdStrike outages underscored the risks of consolidating the technology we rely on among a relatively small pool of companies. Neuberger said, “We need to really think about our digital resilience, not just in the systems we run, but in the globally connected security systems. The risks of consolidation, how we deal with that consolidation, and how we ensure that if an incident does occur, it can be contained, and we can recover quickly.” Blinken emphasized the need for organizations to protect systems through resilience and redundancy, and cautioned against relying on systems that introduce a “single point of failure.”

Concentration risk is a concern. The question is which is more concentrated, the use of a common EDR (CrowdStrike), or the use of a single OS (Windows?) It may not be practical to change your stance on OS use, you should evaluate the tradeoffs and understand the risks. While CrowdStrike impacted about 8.5 million Windows endpoints, alternate EDR such as Carbon Black, Trend Micro, or even AV solutions have similar numbers of endpoints. Unlike the old model of deploying more than one protection on the endpoint to improve coverage, the risk here is that your protection takes out your system. Many have forgotten that about 25 years ago an AV product marked Microsoft Office as malware and quarantined it. While a smaller scale, the net is the same. Consider that a provider who has been through a scenario like this may be better prepared to both prevent and respond to a future event.

A ransomware attack has shut down the computer system of the largest trial court in the country, officials with the Superior Court of Los Angeles County said. The court expected to re-open July 23rd.

Fortunately the court was able to detect and respond to contain and minimize this incident. Consider a tabletop where you're not only seeing a ransomware attack but also a secondary incident which also is disrupting services. The goal is to not be distracted by one big event thereby missing another.

A recent report from the US Department of Homeland Security’s Office of Inspector General (DHS OIG) says the Cybersecurity and Infrastructure Security Agency (CISA) and the Federal Law Enforcement Training Centers (FLETC) disregarded a direct order from DHS CISO to stop working with a contractor deemed “high risk.” DHS OIG notes that the organizations failed to adequately protect sensitive data, including personally identifiable information and law enforcement training curricula.

The DHS CISO revoked the ATO for the training system, which was deemed unable to protect PII, immediately followed by an approved exception to continue operations, without addressing the identified risks. Revoking the approval to operate, effectively turning off, a business system is very impactful, and difficult to enforce, ask me how I know this, even so, that process must include a statement about what must be done, to resume operations as pressure to resume operations is intense, and you're going to need support from high levels no matter how it plays out.

Police in the UK have arrested a teenager suspected of blackmail and violations of the Computer Misuse Act for his alleged involvement with a cybercrime group known as Scattered Spider. The group’s alleged leader was arrested in Spain in June. Scattered Spider is believed to be responsible for a cyberattack against MGM resorts in Las Vegas last summer.

At least two other members of the Scattered Spider ransomware gang, including the suspected leader, were arrested a month ago in Spain. In addition to the MGM attack a year ago, Scattered Spider also hit about 100 other organizations, many of which paid the ransom. Here is where involving the FBI in any ransomware attack you face can help in the long run. Make sure that you not only have their local office number, but that you've met their agents and they know who you are.

The US Treasury Department has imposed sanctions on two individuals for their alleged involvement in cyberattacks against elements of US critical infrastructure. Yuliya Pankratova allegedly oversaw operations for a hacking group with ties to Russia’s government. Denis Degtyarenko allegedly carried out compromise of a US energy company and developed training materials for compromising supervisory control and data acquisition (SCADA) systems.

These two are the Russian hacktivist group Cyber Army of Russia Reborn (CARR) leader and primary hacker. CARR initially focused on DDoS attacks directed at the Ukraine and has more recently started focusing on US and European critical infrastructure. While the attacks are categorized as unsophistocated, they are still succesful. This means you need to check the basics on your ICS/SCADA systems. Don't expose them directly to the Internet, employ segmentation, monitor for unsolicited activity, make sure you're on current best practices to protect these systems.

In a filing with the UDS Securities and Exchange Commission (SEC), safety equipment company Cadre Holdings disclosed “a cybersecurity incident in which an unauthorized third party gained access to certain technology systems of the company.” When Cadre detected unauthorized access to their network on July 15, they shut down some of their systems, disrupting operations.

While the attack was discovered July 15th, Cadre Holdings is still in the midst of the investigation and isn't disclosing the extent or impact of the event. In fact, the only reference, from Cadre Holdings to the event is the Form 8-K SEC filing. At the point you're filing a notice with your regulators, you shouild also be preparring information for your staff, members, etc.