Container Forensics: Investigating Kubernetes Compromises Like a Pro

Three in the morning, Falco alert: a pod in the payments namespace spawned /bin/sh after opening a reverse socket to a foreign IP. The on-call team cordoned the node, but the CISO's first question was brutal: do you have evidence that survives kubectl delete pod?. In most clusters Basilisk audits, the honest answer is no. Container forensics in Kubernetes is not a flavor of classic host forensics, it is its own discipline: the runtime recycles layers, cgroups, and namespaces faster than any analyst can launch Wireshark. This guide lays out a reproducible collection chain that begins before the incident and ends with signed evidence handover.

Why container forensics is different#

A container is not a tiny server, it is a process in isolated namespaces sharing one kernel. Its root filesystem is an overlay of read-only image layers plus a thin writable top layer (the upperdir) that is discarded when the pod dies. There is no persistent /var/log, no stable uptime context, and rarely a journald inside the container. If you do not collect evidence where it lives, you lose it: the scheduler can move the pod to another node, a CrashLoopBackOff resets state, and a plain kubectl rollout restart wipes the volatile context entirely. Forensic maturity in a cluster is not measured in tools, it is measured in the time between alert and the first immutable copy.

Threat model: vectors inside the cluster#

Before collecting, prioritize. The most common vectors are: a vulnerable application container with RCE, a stolen or over-privileged ServiceAccount token from /var/run/secrets, a container breakout via privileged: true, hostPath mounts, or a kernel bug, and a compromised supply chain in the image itself. Each vector leaves traces on a different layer: application vectors in pod memory, token abuse in the kube-apiserver audit log, breakouts on the node and its process tree, supply-chain compromise in the image layers. The threat model decides which layer you freeze first, because time is limited and overlay layers vanish first.

Pre-incident readiness#

The most effective forensics is the prepared kind. Turn on the kube-apiserver audit policy at least at Metadata level, at RequestResponse for sensitive verbs (exec, attach, portforward, create of RBAC objects), and ship the logs to an external, immutable sink. Schedule hourly etcd snapshots (etcdctl snapshot save) so you can reconstruct RBAC state before and after. Keep a forensic toolkit image ready (static binaries of busybox, lsof, ss, tcpdump, avml) in a separate registry, and write an RBAC role scoped only for incident responders. Without this groundwork, every collection is improvised and challengeable in court.

Step by step: live triage on the pod#

The first evidence layer lives inside the pod and is volatile by design. Order matters: first tag the node with a custom NoExecute taint so the scheduler evicts nothing, but do not cordon immediately, because cordon alone stops no running process. Trigger a disk snapshot via CSI (on EKS an EBS snapshot, on GKE gcloud compute disks snapshot), and only then enter the target container with kubectl debug using an ephemeral image carrying static binaries. Capture /proc/[pid]/exe, /proc/[pid]/maps, /proc/[pid]/environ, open sockets via ss -tanp, and the contents of /tmp and /dev/shm before any restart. Whoever has run DFIR on Linux: Live Triage with UAC and Velociraptor on traditional VMs must shift mindset: here the top layer vanishes the moment the pod dies.

Container memory acquisition#

Container memory is the real treasure because fileless malware and injected code exist only there. With containerd or CRI-O runtimes, the in-pod process PID is visible on the host (via crictl inspect or ctr task ls), so you run avml --pid <host_pid> or use LiME compiled against the exact node kernel to produce a full dump. That dump feeds directly into the workflow from Memory Forensics with Volatility 3: Analyzing Dumps in a Reproducible Lab, where linux.pslist, linux.malfind, and linux.check_syscall reveal injections the host EDR missed because they were confined to the container namespace. Every SHA-256 hash lands on a chain-of-custody sheet signed with Sigstore, tying back to the discipline in Supply Chain Security: Sigstore Signing and Real SBOMs in CI/CD.

Control plane and audit logs#

Moving up the stack, the investigation gets truly revealing. The kube-apiserver with audit at RequestResponse level records every exec, attach, and portforward as structured JSON, including user, sourceIPs, userAgent, and objectRef. In a real 2025 case, we recovered the smoking gun: the attacker created a ServiceAccount named monitoring-helper with cluster-admin via a kubectl apply -f - heredoc; the audit log showed user-agent kubectl/v1.29.2 from a residential IP at 3:47am. Cross-reference hourly etcd snapshots to reconstruct RBAC state before and after. This timeline work echoes Timeline Forensics on Windows: Plaso, Log2Timeline and KAPE in Practice, only applied to declarative resources.

Runtime forensics with eBPF#

Runtime forensics requires specific tooling. Deploy Tetragon or keep Falco with custom rules exporting to an external SIEM, never inside the same compromised cluster. For live capture, Aqua's tracee-ebpf records syscalls with container_id context, and sysdig inspect reads .scap files as if they were kernel pcaps. Beware false negatives: if the attacker adapted direct-syscall techniques from EDR Evasion for Research: Direct Syscalls Explained Without the Hype for Linux, or invokes syscalls bypassing libc, a pure userspace hook sees nothing. An eBPF sensor at the kernel tracepoint, by contrast, sees the actual syscall. Combined with Sigma hunting in the spirit of Threat Hunting with Sigma and Elastic: From Indicator to Detection Rule, your odds of catching the pivot multiply.

Network forensics in the CNI#

Network forensics in Kubernetes differs from traditional networking because the CNI does NAT and encapsulation and the pod IP is recycled after death. Capture traffic at the pod's veth pair level with tcpdump -i any on the host, filtered by the pod IP assigned by IPAM. If the cluster runs Cilium, hubble observe --pod payments/checkout-7f4 shows decoded L7 flows with process context. For persistent C2 channels, compare against IOCs from your Building C2 Infra with Sliver in an Isolated Lab for Defensive Research lab and inspect DNS at CoreDNS via query log. Always export pcaps to write-once storage (WORM or object-lock), because lawyers love to challenge integrity.

Dissecting compromised images#

Compromised images deserve their own chapter. Before destroying anything, docker save (or skopeo copy) the suspect image into an isolated forensic registry, then run dive and trivy fs to map files added at runtime via kubectl cp or docker exec against the original layers. In 60% of cases we have seen, the attacker did not modify the original image; they dropped binaries in /tmp or /dev/shm betting nobody would snapshot the overlay. When upstream compromise is suspected, ship layers to an ELF-adapted Malware Analysis in an Isolated Lab: Safe Setup with FlareVM and REMnux with Remnux running on an air-gapped VM.

Chain of custody and anti-forensics#

Evidentiary value rises and falls with the collection chain. Document for every artifact: what, when, from which node/pod, with which command, who collected it, SHA-256 before and after transfer, and store everything immutably. Expect anti-forensics: attackers wipe /tmp, manipulate container time via libfaketime, hide processes with an LD_PRELOAD rootkit, or use memfd_create for fileless execution that never touches disk. That is precisely why the memory dump is non-negotiable: a process with no file on disk is still visible in RAM. Sign every artifact with Sigstore/cosign and keep the private key off the cluster.

Common pitfalls#

The most expensive mistakes are procedural, not technical. kubectl delete pod before the snapshot destroys the overlay layer irreversibly. kubectl exec into the live container leaves your own traces and alters timestamps. Pulling tools over the network onto the compromised node tells the attacker you are there. Trusting the host EDR as your only source misses everything in the container namespace. And storing evidence in the same cluster the attacker controls is naive. Rehearse the order before the real thing arrives.

Checklist#

Short version for the runbook wall: (1) Taint the node NoExecute, freeze scheduling. (2) Trigger a CSI disk snapshot. (3) Memory dump via avml/LiME against the node kernel. (4) Capture pod volatiles: /proc/[pid]/*, sockets, /tmp, /dev/shm. (5) Export the kube-apiserver audit log and an etcd snapshot. (6) Record a network pcap at the veth. (7) Save the suspect image via skopeo copy, run trivy/dive. (8) Hash everything, sign with Sigstore, store in WORM. (9) Write the chain of custody. (10) Only then contain (isolate/delete).

FAQ#

Can I freeze a running container without killing it?#

Yes. Use kubectl debug with an ephemeral container instead of exec, since it shares the process namespace without touching the target entrypoint. At the host level you can pause the process with SIGSTOP (via the host PID), pull a memory dump, and then decide whether to resume or contain. Important: document the SIGSTOP, because it freezes state and the defense will want to see that in the report.

Is host EDR enough for container incidents?#

No. A host EDR sees processes and syscalls, but without namespace awareness it does not attribute them cleanly to the right pod and it misses fileless injections in container RAM. Always add an eBPF sensor with container_id context and a memory dump. Never rely on a single layer.

Conclusion#

Practical takeaway: rehearse the procedure before the incident. Spin up a kind or k3d cluster, simulate an attacker pod that opens a reverse shell, and time how long your team takes from alert to signed memory dump. If it exceeds 20 minutes, automate it with an operator that reacts to Falco events by applying the taint, triggering the CSI snapshot, and launching a collection job. Container forensics is not about exotic tools, it is about choreography rehearsed before the stage catches fire.