Our incident response (IR) team has been busy! News of Chinese state-sponsored hacking group Hafnium’s exploitation of four Microsoft Exchange Server vulnerabilities set off frantic activity in SOCs and IT departments around the world. Numerous threat groups piggybacked on Hafnium’s attacks, and not all efforts to remove the malicious web shells used to create backdoors in “tens of thousands” of email servers were successful.

Microsoft issued patches for the vulnerabilities back on March 2nd, but they didn’t protect systems that had already been compromised. In an unprecedented move—and with the approval of a Texas court—the FBI accessed hundreds of servers in the U.S. to remotely delete remaining web shells, an operation that the agency is calling a success.

Although the FBI removed malware (without victim organizations’ permission), they did not actively fix the underlying vulnerabilities. Affected systems can be reinfected if action is not taken to protect them.

CBI’s IR team continues to address evolving Hafnium-related threats to protect customers across a variety of industries. Here are our top takeaways from recent engagements, and recommendations for the future.

Lessons Learned

1. Inadequate log file preservation: We struggled with log data in many customer environments. They either did not have it, it was not backed up, or they weren’t keeping the data long enough. Log data rolls over every time it hits a certain size threshold, and those thresholds were often not set up to be as large as they should have been. This impacted our ability to validate exploitation and determine what the adversaries did next: Did they move laterally? Did they exfiltrate data? Without adequate log data, it is hard to answer those questions.
2. Poor business continuity and disaster recovery (BC/DR): We noticed that the time to recover from known good backups was not optimal. This boiled down to the fact that, in our experience, many organizations don’t actively test their BC/DR plans. One client took nearly three days to get the backup from a business-critical email server downloaded and working. Had this been part of business testing, several days of outage would have been avoided.
3. Limited lateral movement and privilege escalation awareness capabilities. Attackers are going to pivot once they get in. In several instances, without the tools and technology we had brought in to facilitate incident response, there would have been little to no visibility into lateral movement and privilege escalation. An endpoint protection application would have allowed additional logging in the environment while limiting the threat actor’s ability to move laterally and elevate privileges.
4. Lack of retainers: It was like rush hour at Atlanta International Airport when a tornado hits, and we were the air traffic controllers prioritizing emergency landings. The customers who had retainers rose to the top of the list. From an incident response perspective, having a retainer in place is like having a FastPass when you take the kids to Disney World.
5. The danger of running on–premises legacy technology: A number of customers had kept Exchange data on-premises. Some law firms, for example, have client expectations that compelled them to hold sensitive data locally. Other organizations had a misguided belief in the cloud not being secure when in fact, 0365 is more secure than on-premises Exchange servers.
6. Absence of a playbook: We often have a snapshot of compromised servers and wanted to bring them back online in a segmented, isolated, quarantined state. However, many companies did not have a game plan for creating an isolated network that could be used for analysis in an IR situation.

Recommendations

The actions of Hafnium and our experience responding to attacks make it clear that organizations are at risk. In a déjà vu-inducing announcement on April 13th, The NSA warned of four additional vulnerabilities in Microsoft Exchange Server that involve a “low” attack complexity and can lead to remote code execution. There is no evidence that these vulnerabilities are being exploited in the wild. Still, as attackers continue to ramp up their capabilities, it is more important than ever for defenders to act.

• Improve your cyber hygiene. Patch management is a critical part of the vulnerability management process that you cannot afford to neglect.
• Increase visibility into your network. You can’t fight what you can’t see; raise log retention and push logs to off-site storage and into a write-only format to diminish the ability of attackers to cover their tracks. Ensure effective event log correlation and endpoint security with security information and event management (SIEM) and endpoint detection and response (EDR).
• Test your disaster recovery plan. Inadequate testing leads to a false sense of security that backup and recovery will work. Run test failovers annually, and retest when changes are made.

• Operationalize network isolation and micro-segmentation. Create virtual local area network (VLAN) silos that separate assets, so your network doesn’t become a virtual playground for attackers.
• Get an IR retainer. Quick access to knowledgeable incident responders is critical to reducing the impact of security incidents.

Reduce Response Time and Incident Impact

If your organization has been affected by recent attacks or you would like to learn more about improving your incident response capabilities, please contact us or visit our
Incident Response and Ransomware Resource Center.