Planning for Modern Application Security Forensics and Exploration

When a crime occurs on TV, special forensics investigators swarm to the site, cordoned off the crime scene with yellow tape, don gloves, and scout for fingerprints, body fluids, hair, and even traces of insect activity. All these actions help them recreate the who, what, where, when, and how of what happened.

Their success in solving crimes has given rise to a wide range of forensic science divisions for fields as diverse as accounting, elections, and treaty compliance. There are at least 34 forensic science divisions on Wikipedia — but not a single division specific to modern cybercrime, which may be the fastest-growing type of criminal activity in both quantity and financial impact.

Sure, digital forensics exists for recovering data from electronic and mobile devices. There is also computational forensics that guides the development of algorithms to study and solve forensic problems. Trace evidence analysis, using techniques like microspectrophotometry, is useful for finding particles left behind.

Thus far, however, cyber forensics, distributed application forensics, and microservice orchestration forensics are still a work in progress. This begs several questions:

• Where does one drag the yellow tape when cordoning off the scene of these modern crimes that may span multiple physical servers, containerized microservices, ephemeral serverless code, and third-party SaaS applications?
• What kind of digital fingerprints should be preserved to identify not only who the criminal is, but what precisely they did?
• How can we analyze the incident to mitigate and unwind the damage caused by the attack with the least impact on the business, customers, employees, and community?

It is still early days, but some common patterns are starting to emerge, at least for some of the more common kinds of attacks, such as the OWASP Top 10 API attacks.

Forensic analysis takes cybersecurity to a deeper level, owing to the requirement to manage evidence safely and securely. And despite the best efforts, most forensic investigations do a better job at preventing future attacks rather than prosecution. A 2019 UK study found that only 65 out of 17,900 hacking attempts led to prosecution. Successful forensic investigations typically involve high-profile efforts led by government teams such as the arrest of Ross Ulbricht for running the Silk Road Underground, Roman Seleznev for $169 million in credit card fraud, and Hamza Bendelladj for stealing over $400 million from American banks.

Planning for Forensic Analysis

Forensic science evolved to provide legal evidence in criminal cases. The core process consists of collecting, preserving, and analyzing evidence of the crime. Modern application developers and security teams should plan how they apply a similar process as part of best DevSecOps practices. Even if an investigation never leads to court, it can still help teams understand how the attack occurred, what happened, and how to fix application vulnerabilities.

Recent laws like the European Union’s General Data Protection Regulation and the California Consumer Privacy Act also mandate that companies identify and notify individuals whose personal information has been compromised in an attack. Enterprises must demonstrate they have captured an accurate record of the incident as part of the compliance process. Outside of this, forensic analysis can help to recreate a system of record and pinpoint how hackers succeeded.

Meeting regulatory requirements is becoming ever more difficult with the rapid pace of changes in modern application infrastructure. Traditional servers and mainframe applications had a relatively straightforward architecture to log and analyze. The rise of IaaS virtual machines (VMs) running in the cloud added a bit more complexity but also made it easier to record snapshots of the VMs as required. But the rise of microservices, containers, serverless infrastructure, and cloud services creates a much more complicated forensic trail to investigate.

Capturing Evidence

The modern equivalent of cordoning off the scene needs to start by thinking about what kind of evidence might be left behind so that it can be captured. One good practice is to create a sufficient monitoring infrastructure for collecting evidence about standard application behavior for applications that may span multiple systems. The U.S. National Institute of Standards and Technology provides essential guidance for security monitoring of complex microservice architectures that includes:

• Monitoring should be performed both at the gateway and service level to detect, alert, and respond to inappropriate behavior.
• Input validation errors and extra parameter error crashes and core dumps must be logged.
• Real-time event detection, analysis, and response capabilities should be implemented in the gateway, service mesh, and microservice.
• A centralized dashboard should display a comprehensive overview of the current security status.
• Teams need to collect a baseline of normal behavior that includes the outcome of business logic decisions, contact attempts, and other behavior to detect deviations.

Preserving Evidence

Modern microservice applications are ephemeral by nature, making it harder to assess what occurred long after an attack. Recreating the scene of the virtual crime requires finding a way to preserve not only the data but any metadata that can help the investigator recreate the context across multiple applications, application programming interfaces (APIs), systems, and logs.

In a physical investigation, teams cordon off the area to minimize the risk of other fingerprints, hair follicles, or materials contaminating the evidence. An investigator is not just concerned with the knife but also whose fingerprints are on it.

The digital equivalent also must consider how applications can affect the evidence. For example, in traditional digital forensics, an investigator would use tools to capture and analyze the data using dedicated forensics tools separate from the regular application. This would allow investigators to analyze images without changing the metadata about when it was captured or last accessed.

Modern distributed application forensics also needs to capture all metadata relevant to an incident such as which credentials were used as part of a chain of transactions, which container images were involved, the configuration settings, and what IP addresses were employed to access an API. This kind of data might normally be collected by modern observability tools and then quickly disposed of when not relevant to standard analytics. However, teams might consider preserving more of this metadata in the way of a potential security incident.

Analyzing the Evidence

Once all the evidence has been collected, teams need to think about organizing it for a single view. Characteristics such as container configuration settings and remote IP addresses are not often combined for a deeper analysis of application performance. But in forensic analysis, every trace of evidence might prove vital.

Once all data has been brought to the lab, researchers need to consider what kinds of patterns might help them connect the dots. In this case, techniques like computational analysis can help teams identify the correlations between seemingly disparate events. Finally, the modern equivalent of trace evidence analysis needs to analyze the digital particles that may have been left behind using techniques baked into distributed tracing tools.

This kind of analysis is getting easier to perform as researchers piece together the typical signatures of common new types of attacks, such as the OWASP API Security Top 10. Over time, an experienced investigator cultivates an informed insight of where to look for fingerprints and other evidence. In the same way, modern observability tools are getting much better at quickly finding, preserving, and analyzing evidence to solve the most common types of cybercrime.

First Steps

The best practice is to hire a dedicated cybersecurity forensics team, particularly for less common attack patterns that require a deep forensic investigation. For example, Solar Winds partnered with multiple leading cybersecurity experts to perform a deep forensic analysis of how the recent Sunburst attack was able to successfully penetrate enterprises and government agencies around the world. This initial forensic research helped other teams identify the types of evidence to hunt for.

Further research identified various anti-forensic techniques employed by hackers to evade detection for so long. These techniques included changing file names, the IP addresses used, and launch processes for each target. The attackers also changed time stamps to confuse analysis. Once the basic techniques were worked out, other forensics experts could improve the detection and prevention of similar problems within their organizations.

About the Author

George Lawton is a technology writer and regular contributor to The Inside Trace.