Forensic Investigation in Docker Environments: Unraveling the Secrets of Containers

“Criminals always leave evidence in their actions.”
Edmond Locard

image by midjourney/@wicasame

Start
The story of Noah's Ark, widely known as a biblical account, can be related to the context of Docker technology.

Just as Noah was instructed to build the Ark to house his family and divide it into compartments to accommodate all species of animals, similarly, Docker provides a container platform that allows you to create and isolate environments for applications and services using a specific configuration, such as the Dockerfile, and organize these containers into separate networks and volumes.

As the Ark protected its occupants during the flood, Docker provides isolation and security for applications and services running in containers. Each container is encapsulated, ensuring that an application's dependencies and configurations do not affect other containers running in the same environment.

The Ark represented salvation and hope in the midst of destruction; Docker represents a modern solution for deploying and managing applications in complex environments. With Docker, you can quickly build and deploy consistent environments across different operating systems and infrastructures, making it easier for development and operations teams to collaborate.

The benefits offered by Docker in the current technological context teach us about the importance of protection, cooperation, and preservation.

Introduction:

Docker has become a popular container platform, revolutionizing application development and management. However, this innovation also brought challenges in the area of forensic investigation. Docker containers offer flexibility and efficiency in packaging and deploying applications, but it's important to recognize that they can also be vulnerable to security incidents, just like any other computing environment.

In this article, we'll explore the fascinating world of forensics in Docker environments, revealing the essential techniques and tools to unlock the secrets hidden in containers. We'll cover analyzing Docker containers, detecting malicious activity, gathering evidence, and investigating incidents.

Disclaimer
This article is a comprehensive source of tips and tricks for Docker system protection and forensics. In it, we explore the ins and outs of this technology, offering valuable insights and scripts for asset resolution, optimization, and protection.

While it would be ideal to divide the article into three parts to address all forensic analysis possibilities, we are aware that time is limited. However, our intention is to provide the most relevant information to facilitate forensic analysis of the Docker environment, covering a wide range of possibilities.

Feel free to explore the techniques and tools shared in this article, and we hope you find useful resources for improving security and forensics in your Docker projects.

That's all folks!

I. What is Docker and why is it important for digital forensics?

Docker is an open-source platform that allows you to build, distribute and run applications in isolated containers. These containers are self-contained, lightweight, portable units that encapsulate all of the components needed to run an application, including code, libraries, and dependencies. Docker technology has revolutionized software development and deployment, providing greater efficiency, scalability, and consistency.

However, the ephemeral nature of Docker containers presents specific challenges for digital forensics. Unlike traditional virtual machines, Docker containers do not have a complete operating system, which makes evidence collection and forensics difficult. Furthermore, the dynamics of containers, such as rapid creation, modification, and destruction, add to the complexity of investigations.

II. Challenges of digital forensics in Docker environments:

Evidence Gathering: The transient nature of Docker containers makes evidence gathering a challenge. It is crucial to correctly identify and preserve relevant artifacts, such as container images, metadata, logs and associated file systems.

Data analysis: The lack of a complete operating system in Docker containers requires different forensic approaches. Data analysis must take into account the particularities of layered file systems, the interaction between containers, and the isolation techniques used.

Event reconstruction: The rapid creation and destruction of containers makes it difficult to accurately reconstruct events. It is necessary to track and correlate information dispersed in different containers and in distributed logs.

III. Digital forensics techniques in Docker environments:

Gathering Evidence: Gathering evidence in Docker environments involves identifying and preserving relevant images, containers, volumes, networks, and metadata. This can be done through tools like the Docker CLI, Docker API, and image registries.

Data analysis: Forensics on Docker containers requires understanding layered file systems and extracting key information. Tools such as DFF (Docker Forensic Framework) and Autopsy can help analyze Docker containers.

Understanding Docker Forensics challenges:

The comprehensive nature of Docker containers is ephemeral and can be easily destroyed, making evidence gathering a challenge. The rapid scalability and distribution of containers demand tailored forensic approaches, difficult to hamper the transmission of access to internal data.

Network Complexity: Docker has a complex set of networking features that can be challenging to configure and debug, especially in distributed environments. Setting up advanced networks like load balancers or multi-host networks can be tricky.

In this diagram, we have a simplified representation of network complexity in Docker. Docker has a set of networking features that enable communication between containers and the outside world.

At the top of the diagram, we have the "Docker Network", which is the virtual network layer of Docker. This virtual network allows the creation of isolated networks for containers, ensuring that they can communicate with each other without interfering with other containers or external networks.

Next, we have "Docker Containers", which are the individual containers running on Docker. Each container can have its own "Container Network", which is a virtual network interface internal to the container. These internal networks allow containers to communicate with each other within the same virtual network.

Within each container is "Container Networks", which represent the internal virtual network interfaces for each container. These interfaces allow communication between processes and services within the container.

In the diagram, we can see multiple containers connected to their respective internal networks. This represents the complexity that can arise when there are multiple containers communicating over a Docker network.

Diagram Description:

Docker Network: The "Docker Network" represents the overall networking environment of Docker, which consists of various container networks and other network-related resources.

Containers: In the diagram, there are three distinct containers (Container 1, Container 2, and Container 3), each running a separate service or application. Each container has its own virtual network interest.

Bridge Network: The "Bridge Network" is the default network automatically created when Docker is installed. Every container is connected to this network by default, allowing them to communicate with each other.

Overlay Network: The "Overlay Network" is a virtual network that allows containers on different hosts to communicate with each other, even if they are on different machines. This makes it possible to create distributed clusters and communicate between containers on different Docker Swarm nodes.

Collecting Evidence in Docker Containers:

Using specialized tools, such as the Docker Forensics Toolkit, to gather information about running containers.

This command displays a detailed list of all running containers, including information such as ID, name, image, status, exposed port, and more. This information is essential for forensic investigation in Docker environments, allowing investigators to identify relevant containers for analysis and evidence collection.
In addition, the Docker Forensics Toolkit offers a variety of other commands and features to aid in forensic investigation in Docker environments, such as forensically acquiring container images, analyzing logs and event logs, and more. These tools are designed to address the specific challenges of forensic investigation in Docker environments and provide valuable information for incident analysis.

Here are some of the main Docker Forensics Toolkit commands:

docker-forensics containers list

docker-forensics containers info <ID of the container>

docker-forensics images list

docker-forensics images info <ID of the container>

docker-forensics networks list

docker-forensics volumes list

docker-forensics container inspect <ID of the container>

docker-forensics container logs <ID of the container>

docker-forensics memory dump <ID of the container>

docker-forensics diff <ID of the container>

Forensic acquisition of container images for further analysis.

Identify the target container image: Get the ID or name of the container image you want to acquire for forensics.

Create a temporary container: Create a new container based on the target image, but with a mounted volume directory to store the acquired image.

docker run -v /path/to/acquisition:/mnt --name acquisition-container <ID or image name>

Copy Image to Acquisition Directory: Copy the container image to the acquisition directory mounted on the temporary container.

Remove temporary container: Remove the temporary container created for acquisition.

Now that we have the acquired container image in the specified acquire directory, this image can be used later for forensics, preserving the state of the container and allowing investigation of events and activities that occurred during the run.

Exploring Docker logs and event logs to reconstruct suspicious activity.

Identify the target container: Determine the ID or name of the specific container you want to explore the event logs and logs for.

Access container logs: Use the docker logs command to access the target container's logs.

Explore specific logs: Use additional docker logs command options to explore specific logs, such as filtering by date or searching by keyword.

docker logs --since <initial_date> --until <final_date> <ID or container name>

Export logs to file: For further analysis, you can export container logs to a file.

Use analysis tools: Use log analysis tools such as Elasticsearch, Logstash, and Kibana (ELK) to explore and visualize logs in a more advanced way.

We can then send the container logs to Elasticsearch using Logstash, and view and search the logs using Kibana.

By exploring Docker logs and event logs, you can reconstruct suspicious activity, identify potential security holes, and perform in-depth forensic analysis for investigation.

Container and Artifact Analysis:

Identify Target Container: Determine the ID or name of the specific container you want to analyze.

Create a temporary container: Create a new container based on the same image as the target container, but with a mounted volume directory for analysis.

docker run -v /path/to/analysis:/mnt --name analysis-container <container id or name>

Access the temporary container shell: Run the temporary container shell to gain access to your environment.

Analyze the file system: Explore the temporary container's file system to identify and examine relevant artifacts.

Extract files and data: Select the specific files or data you want to extract from the temporary container and copy them to the host.

Perform additional analysis: Perform additional analysis on the extracted files and data using forensic tools such as malware analysis, data recovery or information extraction.

Remove temporary container: Remove the temporary container created for analysis.

This will remove the "analysis-container" container from the system, enabling a detailed analysis of the containers and artifacts, exploring the file system, extracting relevant information and performing deeper forensic analysis.

Extraction of metadata and configuration information from containers.

Identify the target container: Determine the ID or name of the specific container from which you want to extract metadata and configuration information.

Access container information: Use the docker inspect command to get detailed information about the target container.

This will return a JSON containing the metadata and configuration information for the container.

Extract specific metadata: Parse the JSON returned by the docker inspect command to extract the desired specific metadata and configuration information.

Metadata example:

🛥️Container ID

🛥️Container name

🛥️Image used

🛥️Environment Variables

🛥️Mounted directories

🛥️Exposed doors

🛥️ Associated networks

Export information to file: To facilitate further analysis, you can export the extracted information to a file.

Recovery of important files and records inside containers.

Identify Target Container: Determine the ID or name of the specific container from which you want to retrieve important files or logs.

Create a temporary container: Create a new container based on the same image as the target container but with a mounted volume directory to retrieve the files.

docker run -v /path/to/recovery:/mnt --name recovery-container <container ID or name>

Access the temporary container shell: Run the temporary container shell to gain access to your environment.

Retrieve important files: Explore the temporary container's file system and copy important files to the mounted recovery directory.

Copy files to host: Copy retrieved files from temporary container to host.

Remove temporary container: Remove the temporary container created for recovery.

This way you will be able to recover important files and records from inside the containers, allowing the analysis and further investigation of incidents or suspicious activities.

Analysis of network traffic between containers and with the host to identify malicious activity.

Identify target containers: Determine the IDs or names of specific containers between which you want to analyze network traffic.

Run a container for network analysis: Create a new container based on a specialized image for network traffic analysis, such as Wireshark, which has the necessary tools.

Capture and analyze traffic: Within the network analysis container, use tools like Wireshark to capture and analyze network traffic.

Filter traffic: Apply filters in Wireshark to focus on analyzing specific traffic of interest, such as filtering for suspicious IP addresses, ports, protocols, or traffic patterns.

     • Use a traffic filtering tool such as a firewall or IDS/IPS (Intrusion Detection/Prevention System) to implement filtering rules.

     • Analyze network traffic in real time or use traffic logs to identify suspicious or unwanted patterns.

     • Based on collected information, define filtering rules to allow or block certain types of traffic.

     • Configure filtering rules according to the specific needs of your environment, taking into account network protocols, ports, and IP addresses, among other criteria.

• Deploy filtering rules on the firewall or IDS/IPS, ensuring they are up and running correctly.

• Continuously monitor network traffic and verify that filtering rules are being applied as expected.
• Adjust filtering rules, if necessary, based on newly identified threats or changing needs in the environment.
• Document the applied filtering rules, including the filtering criteria and the reasons for its implementation.

Analyze captured traffic: Analyze captured packets to identify malicious activity, such as suspicious communications, transfer of sensitive data or unauthorized traffic patterns.

• Traffic Capture: Use a traffic capture tool, such as Wireshark, to capture network traffic at a particular point on the network.
• View packets: Open the capture file in Wireshark to view captured packets. Packets are displayed in a list, showing information, such as source and destination IP addresses, ports, protocols and packet contents.

Protocol analysis: Identify the different protocols present in the captured packets, such as TCP, UDP, HTTP, DNS, among others. Review the specific information for each protocol to better understand the nature of the traffic.

Packet Filtering: Use Wireshark’s filtering features to filter packets based on specific criteria, such as IP addresses, ports, protocols or data patterns. This allows you to focus on analyzing the packages relevant to your goal.

Analysis of patterns and behaviors: Observe traffic patterns, such as frequency of communications, packet sizes, response times, and more. Look for suspicious or anomalous behavior that could indicate malicious activity.

Identification of important events: Search for important events in the captured packets, such as failed authentication attempts, requests to access restricted resources, communications with unknown IPs, among others. These events can provide valuable insights into potential threats.

Content Analysis: Analyze the content of captured packets, such as HTTP request headers, transmitted data, and server responses. This analysis can reveal relevant information about activities within the traffic.

We will now be able to perform a detailed analysis of network traffic between containers and with the host, allowing the identification of malicious activities and helping to detect and respond to security incidents in Docker environments.

Apply detection techniques: Use threat detection techniques, such as malware signatures, behavior analysis, detection rules, or machine learning algorithms, to identify suspicious activity based on collected logs.

Investigate suspicious events: Review specific events that are considered suspicious and investigate them in detail by examining additional records, collecting evidence and correlating relevant information.

Analyze Artifacts: Analyze artifacts within containers, such as executable files, scripts, libraries, or configurations, for signs of malicious activity.

Consult External Information: Consult external sources, such as lists of known IPs or malicious domains, threat databases, or security forums, for signs of suspicious activity associated with IPs, URLs, or other elements found in the logs.

Take Corrective Actions: Based on the analysis of the suspicious activities, take appropriate corrective measures to mitigate potential threats or incidents, such as blocking malicious IPs, disabling compromised containers, or updating security settings.

Use of memory analysis techniques to detect malicious processes in containers.

Get Container Image: Download the image of the container you want to analyze from memory. You can use the docker pull command to get the desired image.

Replace <image_name> with the name of the container image you want to download. For example, if you want to download the Ubuntu image, the command would be:

Docker will start the process of getting the container image from Docker Hub or a specified image registry, displaying the download progress and showing the image size and throughput.

Running the container in read-only mode: Run the container in read-only mode to avoid memory changes during analysis. You can use the docker run command with the --read-only option for this.

Replace <image_name> with the name of the container image you want to run in read-only mode. For example:

Docker will start running the container in read mode, which means the container will only have read permissions to the file system.

The container will start and you can see the output in the terminal depending on what is running in the container.

Note that in read-only mode, the container will not be able to make changes to the host's file system or its own internal files. This is useful for forensic analysis purposes or to prevent unwanted modifications to the container during runtime.

Container memory access: Use memory analysis tools, such as Volatility or LiME (Linux Memory Extractor), to access container memory.

Open a terminal or command prompt on your operating system.

Make sure you have a memory analysis tool installed, such as Volatility or LiME (Linux Memory Extractor), available in your environment.

Run the following command to list the running containers:

Identify the ID or name of the container for which you want to access memory.

Use the following command to run an interactive session inside the container:

Replace <container_name_or_id_of_container> with the name or ID of the container you want to access.

Within the container's interactive session, you can use memory analysis tool commands to extract information or analyze the container's memory.

For example, if you are using Volatility, you can run the following command to list the processes running inside the container:

Replace <profile> with the appropriate profile for the container's operating system.

Use the functionality of the memory analysis tool to access and analyze container memory data, such as running processes, network connections, logs or other relevant artifacts.

Keep in mind that memory analysis in containers can be complex, and it is important to have a good understanding of the memory analysis tool used and the proper techniques to obtain accurate and reliable results.

Crash Dump Extraction: Extract the crash dump from the container using the selected tool. This will allow you to analyze the container's memory in a controlled environment.

Make sure you have an installed memory analysis tool like Volatility available in your environment.

Run the following command to list the running containers:

Identify the ID or name of the container for which you want to extract the memory dump.

Use the following command to extract the container memory dump:

Replace <container_name_or_id_of_container> with the name or ID of the container you want to extract the memory dump from. The above command will create a file called "memory_dump", which will contain the memory dump.

Use a memory analysis tool, such as Volatility, to analyze the extracted memory dump file.

For example, if you are using Volatility, you can run the following command to list the processes running inside the memory dump:

Replace <profile> with the appropriate profile for the container's operating system.

Explore the functionality of the memory analysis tool to examine the data extracted from the memory dump and obtain relevant information, such as running processes, network connections, logs or other artifacts of interest.

Keep in mind that extracting and analyzing a container's memory dump can be complex, and it is important to have a good understanding of the memory analysis tool used and the proper techniques to obtain accurate and reliable results.

Choose an appropriate memory analysis tool command to extract specific information from the memory dump. Some examples of common commands include:

• imageinfo: displays general memory dump information, such as operating system version and image size.
• pslist: lists the processes running in the memory dump.
• netscan: analyzes the network connections present in the memory dump.
• filescan: identifies files and directories present in the memory dump.
• dlllist: lists the DLLs (dynamic link libraries) loaded in memory.

Run your chosen command, replacing <command> with the name of the command you want to run.

The memory analysis tool will display the results of the analysis, providing relevant information based on the command used.

Identification of malicious processes: During analysis, look for unusual behavior, such as suspicious processes, suspicious network activity, unexpected file modifications, or the presence of malicious artifacts in the container's memory.

Make sure you have a process analysis tool installed, such as Sysinternals Process Explorer (on Windows) or ps (on Linux), available in your environment.

Run the Process Analysis tool to view a list of processes running on the system.

Scan the list of processes for suspicious behavior, such as strange names, excessive resource usage, unusual network connections, or other indicators of malicious activity.

Identify suspicious processes and write down their IDs or names.

Use additional tools, such as an antivirus or an intrusion detection system, to perform more detailed scans on suspicious processes, if available.

If necessary, terminate malicious processes identified through appropriate process analysis tool commands or through additional security tools.

Further investigation: In case of detection of malicious activity, conduct further investigation to identify the nature and extent of the attack. This may involve analyzing other memory-related artifacts, such as configuration files, event logs, or network data.

After identifying a suspicious process or malicious activity, it is important to perform further investigation to gather more information and better understand the nature of the problem.

Write down the suspect process name, PID (Process Identifier) or other relevant information for reference during the investigation.

Use forensic analysis tools, such as Volatility or other specialized tools, to gather additional information about the suspicious process. Run specific forensic analysis tool commands to extract details about the suspicious process, such as memory information, network connections, open files, and more.

Analyze investigation results to identify patterns, suspicious behavior, unusual network connections, malicious files, or other evidence that could indicate the presence of a threat.

If necessary, consult threat databases, such as malware signature lists or indicators of compromise (IOCs), to verify that the suspected process matches any known threats.

Document all steps of the investigation, including tools used, commands executed, results obtained, and any relevant findings.

While Docker is a widely used and trusted technology, it is important to recognize that, like any software platform, it also has its vulnerabilities.

Here are some of the top vulnerabilities associated with Docker:

midjourney/@wicasame

Privilege Escaping: Under certain circumstances, a compromised container can be used to gain privileged access to the host system, resulting in a possible security compromise.
Host Kernel Vulnerabilities: Docker relies on the host system's kernel to provide container isolation. If the kernel is outdated or vulnerable, it could allow an attacker to compromise system security.

Docker Daemon Vulnerabilities: The Docker Daemon is responsible for controlling and managing the containers. Vulnerabilities in the Docker Daemon could allow malicious code execution or unauthorized access.

Container Image Vulnerabilities: Container images may contain outdated or vulnerable software. If a container is based on a compromised image, this could lead to known vulnerabilities being exploited.

Network Misconfiguration: Improper network configurations can allow a container to access sensitive resources or communicate with other containers in an unauthorized manner.

Exposing Unnecessary Ports: If container ports are exposed unnecessarily or incorrectly, this can create entry points for malicious attacks.

Data Volume Vulnerabilities: If container data volumes are not properly secured, there could be risks of confidential information being leaked or unauthorized access to the data.

Importantly, these vulnerabilities are not unique to Docker and are found in many other technologies. The key to mitigating these risks is adopting proper security practices, such as keeping Docker and the host system up to date, using reliable and secure images, applying configuration best practices, and regularly monitoring the security of containers, as per the tutorial mentioned in this article.

In addition, it is recommended to use security tools such as vulnerability scanners and container monitoring solutions to identify and mitigate potential threats. By taking a comprehensive approach to security, you can minimize the risks associated with using Docker and ensure a more secure container environment.

Tips and tricks

Docker API: The Docker API allows you to automate tasks and get detailed information about containers, images and other elements of the Docker environment. This interface can be exploited to obtain relevant metadata and records during an investigation.

DFF (Docker Forensic Framework): DFF is a specific tool for digital forensics in Docker environments. It provides capabilities for gathering evidence, analyzing layered file systems, and retrieving critical information for the investigation.

Autopsy: Autopsy is a widely used digital forensics platform that also supports analysis of Docker containers. It offers advanced features, such as file system visualization, log analysis, metadata extraction and deleted data recovery.

Distributed logs: In Docker environments, logs can be distributed across multiple containers. It is essential to use log aggregation and analysis tools, such as the ELK Stack (Elasticsearch, Logstash and Kibana), to centralize and correlate relevant information.

Communities and forums: Digital forensics in Docker environments is an ever-evolving field. Participating in Docker and digital forensics communities, forums, and discussion groups can provide you with valuable insights, share experiences, and learn from experts in the field.

URLs that provide information about Docker Forensics Toolkit

                   midjourney/@wicasame

🛥️ https://github.com/docker-forensics-toolkit/toolkit

🛥️ https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html

🛥️ https://github.com/docker/docker-bench-security
🛥️ https://github.com/docker/labs/tree/master/security

🛥️ https://github.com/aquasecurity/trivy

🛥️ https://github.com/slimtoolkit/slim

🛥️ https://github.com/docker/docker-bench-security

🛥️ https://github.com/CISOfy/lynis

🛥️ https://github.com/cr0hn/dockerscan

These URLs can be resources for detailed information about the Docker Forensics Toolkit and how to perform forensics on Docker containers. Remember to verify the reliability and authenticity of assets before using them in a production environment to ensure accuracy and effectiveness of forensics.

Final considerations

Digital forensics in Docker environments presents unique challenges due to the ephemeral and isolated nature of containers. However, with the proper techniques and the use of specialized tools, it is possible to perform accurate and effective investigations. Collaboration between the security community and developers is critical to advancing the area of digital forensics in Docker containers, improving existing techniques and developing new approaches.

Forensic investigation in Docker environments is a growing field that requires specialized knowledge and adapted tools. With the right techniques and tools, investigators can unlock vital evidence and contribute to cybersecurity and the fight against digital crime. As Docker technology continues to evolve, it is essential for security professionals to keep pace with these changes to meet the challenges ahead. Forensic investigation in Docker environments is an exciting and challenging area, opening up new possibilities for investigations in an increasingly cloud-based world. However, there are challenges to be overcome in forensic investigation in Docker environments. The lack of formal standards and guidelines for container forensic investigation, ensuring the legality of collected evidence and its admissibility in a legal context, and privacy and data protection issues in accessing containers are some of the existing challenges.

Warning: All information available in this article are for educational purposes only. Use these at your own discretion, the author of the article and the magazine owners cannot be held responsible for any damages caused.

 About the Author

Wilson Mendes - I am a cryptanalyst and work with Cyber Security Engineer | Python | FreeBSD | Red Team | Ethical Hacker Artificial intelligence.
http://www.wicasame.com/
https://www.linkedin.com/in/wilsoncsmendes/