Lessons from the Incident Response Trenches - Investigating and Eradicating Kwampirs

Kroll has deployed CB Response (CbR) during hundreds of cyber investigations because it can provide insights throughout each stage of the incident response (IR) process (see graphic). One of Kroll’s recent investigations, which involved the Kwampirs malware, illustrates how CbR helps uncover critical behavioral characteristics that aid in response strategies, particularly when working to identify, contain and eradicate malware.

Preparation

In this case, the preparation phase came to an abrupt end when law enforcement reached out with bad news: a threat actor known as Orangeworm was active in the client’s network and was clearly poised to carry out their objectives. Some indicators of compromise (IOCs) were provided, but the organization lacked the endpoint visibility necessary to make sense of the disconcerting results that firewall log searches returned. Kroll was engaged to help fill in the gaps.

The process of deploying an endpoint sensor such as CbR during an incident like this drives home a reason why CIS Critical Control #1 – a “basic control” – is to maintain a comprehensive IT hardware asset inventory. Basic, yes; easy to accomplish, no. The organization assembled resources and deployed the sensor as quickly as possible, but even several weeks into the incident, it was still discovering assets which lacked centralized management from IT.

Identification

We devote a significant amount of time and effort in the identification phrase because so much of what follows in IR depends on a thorough understanding of the threat(s) in the environment. 

• Situational Awareness

Kroll quickly began to identify signs of anomalous executables named in a way to resemble a legitimate Windows executable, wmiprvse.exe. However, these executables were in unexpected locations or they used simple variations on spelling, such as wmiaprvse.exe. These types of variations are relatively easy to spot in CbR with basic searches such as:

• process_name:wmi*.exe -process_name:wmiprvse.exe
• process_name:wmi*.exe -path:wbem

These executables were also observed to spawn child processes of rundll32.exe with two unusual features:  command line parameters that mentioned additional DLL files using the “Wmi” naming theme, and the command line parameter of “ControlTrace”:

• process_name:rundll32.exe cmdline:ControlTrace

These attributes can be summed up in a single process tree view, which shows a malicious executable installed as a Windows service, spawning the backdoor DLL by way of rundll32.exe, and allowing arbitrary commands to be executed by the threat actor.

Kroll was also able to validate that all the actor’s executables and DLLs were using some common metadata, including a company name of “Indiana Software Foundation”.

This kind of (likely accidental) consistency in malware is very helpful for incident responders, especially when it comes time to selectively eradicate files without fear of removing similarly named, legitimate Windows operating system files.

• Looking Back
  

A weakness of many EDR solutions during an IR situation is that they provide monitoring only from the date of installation, going forward. But getting a glimpse into what was happening on endpoints a week, month or even years ago can be vital to an investigation. Kroll has developed several scripts that harness the CB Live Response (CBLR) API to gather key forensic artifacts en masse from endpoints, such as specific event types from Windows Event logging, Shimcache, Autoruns and more. 

These artifacts facilitated a historical look-back and timelining capability that showed the threat actor had been present in the environment for multiple years prior to our doing any “dead box” forensics.

Containment

Many contemporary malware families, including not just Kwampirs, but also a long list of threats such as Emotet, Trickbot, NotPetya and others, have a spreading or lateral movement component that utilizes the Windows RPC and SMB protocols to remotely spread and install malicious services on peer systems. Organizations can greatly reduce this opportunity for lateral movement by implementing a host-based firewall block on inbound RPC and SMB requests to client systems. Although no single containment measure is foolproof, preventing continued spread over RPC/SMB is a great supplement to other CbR features, such as host isolation and hash banning. 

Eradication

During Kroll’s investigation, the organization’s existing anti-virus software vendor was able to provide updated signatures that started to detect and remove Kwampirs executables on some systems. However, one of Kwampirs’ evasion tactics is that each infected system received a unique hash for the malicious EXE and DLL files. Kroll leveraged the CBLR API once again to deploy a customized eradication PowerShell script to eradicate the services, executables and other remnants of Kwampirs not yet detected or removed by anti-virus. 

Recovery

After Kroll eradicated all known artifacts of the malware and its persistence mechanism from the environment, continued monitoring with CbR was critical in keeping an eye out for any signs of resurgent threat actor activity. As unmanaged systems were identified and installed with CbR, further eradication was required. Custom watchlists based on all the threat intelligence gained during the identification phase were a key aspect of enabling the organization to regain confidence in its network while it restored systems and resumed standard business operations.

Lessons Learned

The findings from this case echo the learnings – and associated best practices – that have emerged from hundreds of similar investigations Kroll has conducted with the help of CbR:

• You can’t protect what you don’t know about. The ability to account for every authorized device that connects to your network is fundamental to ultimately eradicating a malware attack and preventing a reinfection.
  
• Shut down lateral movement. Disabling inbound RPC and SMB traffic to clients helps prevent the spread of malicious services, a key aspect of many malware families today.  
  
• Look at the bigger picture. Active monitoring is great, but not always sufficient. Gathering forensic artifacts selectively can help to fill in the rest of the story and enable a more customized and successful response. 
  

A version of this article was originally published on carbonblack.com