DFIR on Linux: Live Triage with UAC and Velociraptor

Three in the morning, a Debian 12 box exposes a suspicious cron rewriting /etc/ld.so.preload every 90 seconds. The wrong reflex is to power off the machine; the right one is to open the triage runbook and capture live state before the attacker notices the lights coming on. At Basilisk this scenario shows up monthly in simulation labs, and the rule is simple: order of volatility first, hypotheses later. Memory, connections, processes, open file descriptors and kernel modules are top priority, because a reboot or a misplaced kill -9 destroys 80% of what matters.

Order of volatility as the first law#

The order of volatility (RFC 3227) ranks evidence by how fast it evaporates: CPU registers and cache vanish in nanoseconds, RAM on power loss, network connections in seconds, running processes until the next reboot, disk the slowest. The practical consequence: you collect from volatile to persistent, never the reverse. A reboot to stop the cron wipes exactly the memory pages that hold the unpacked payload and the C2 address.

Concretely: first ss -tunap for active sockets, ps auxww and /proc/[pid]/exe for running processes (even deleted binaries can be reconstructed from /proc), lsof -n for open descriptors, cat /proc/modules for loaded kernel modules. Every output is written with a UTC timestamp and hostname to a read-only destination, never to the victim disk. Only once volatile state is secured do you think about isolation.

UAC for the one-shot triage#

Our default toolbox starts with UAC (Unix-like Artifacts Collector) by Tulpa Security and Velociraptor by Rapid7. UAC is perfect for offline one-shot triage: drop a ~6MB tar.gz, run ./uac -p ir_triage /mnt/evidence, and in 8 to 15 minutes you have hashes, /proc dumps, journald, every user bash_history, cron listings and SSH configuration compressed.

UAC is deliberately dependency-light: pure shell, runs on a host where you are not allowed to install anything, and respects order of volatility in its built-in profile. The key operational note: run UAC from a read-only mounted medium and write output to a separate encrypted target, so you do not alter a single atime on the victim. If you have not built the base lab yet, review Web Pentesting From Scratch: Building a Safe Lab with DVWA, Juice Shop and Burp Suite before simulating real incidents.

Memory acquisition with AVML#

Memory acquisition remains the most fragile step on Linux. On kernel 6.x, AVML from Microsoft Research is the most consistent option when LiME fails to compile against missing headers. Real command: avml --compress /evidence/memdump.lime. Reserve at least twice the RAM on disk and never write to the victim disk; mount a read-write NFS at /mnt/triage or a LUKS encrypted USB.

The reason for AVML over LiME is practical: LiME is a kernel module compiled against the exact kernel headers that are often missing on the victim, while AVML is a statically linked userspace binary that reads through /proc/kcore or /dev/crash. After capture, analysis moves to Volatility 3 with auto-detected profile, which is where the pipeline meets Memory Forensics with Volatility 3: Analyzing Dumps in a Reproducible Lab. Without a memory dump any LD_PRELOAD rootkit becomes urban legend in the report.

Analysis in Volatility 3#

With the dump in hand you build the process hierarchy in Volatility 3 with linux.pstree, list network connections with linux.sockstat, and hunt hidden or unlinked processes with linux.psscan. The classic finding on an LD_PRELOAD rootkit is a discrepancy: ps on the live system shows fewer processes than linux.psscan in the dump, because the rootkit manipulates the userspace view but not the kernel structures.

For modern kernels the symbol table (ISF, Intermediate Symbol Format) is decisive; without matching symbols Volatility returns incomplete results. Generate the ISF file from the vmlinux with debug symbols for the victim's exact kernel version. That is why it pays to maintain a symbol repository for your fleet's kernel versions ahead of time, so you do not lose hours sourcing symbols in the middle of an incident.

Hunting persistence: never a single place#

Persistence on Linux rarely lives in a single place. In recent Kinsing miner cases we found four simultaneous vectors: a systemd unit at /etc/systemd/system/.cache.service, a root crontab entry, a modified /etc/rc.local and a SUID wrapper at /usr/local/sbin/ssh. UAC catches all of that with the ir_triage profile, but it pays to also run find / -newermt '2026-06-01' -type f -mtime -8 2>/dev/null to close gaps.

Do not forget the quiet corners: ~/.bashrc and /etc/profile.d/, an authorized_keys with a slipped-in key, udev rules, PAM modules, and cron-at jobs. A single missed vector hands the host back to the attacker within hours. To correlate with equivalent Windows tradecraft, the team consults Windows Persistence: 10 Documented Techniques and Their Countermeasures, because opportunistic attackers reuse patterns across platforms.

Velociraptor: from one host to a fleet#

Velociraptor changes the game once we move from a single host to a fleet. We stand up a server on t3.medium, generate Linux clients with velociraptor config client, push them via Ansible, and in 20 minutes we have visibility. The most useful hunts in our playbook are Linux.Network.NetstatEnriched for active sessions, Linux.Sys.SUID for suspicious binaries, and a compiled Yara rule with signatures for Pupy, Sliver and Merlin.

The power of VQL is that you phrase a hypothesis as a query and fire it synchronously against 120 hosts: show me every process whose binary lives in /tmp or /dev/shm and has an outbound connection. When an IOC hits, we export a signed collection zip and pivot to static analysis in an isolated sandbox, as described in Malware Analysis in an Isolated Lab: Safe Setup with FlareVM and REMnux. The trick is never skipping SHA-256 hashing before moving artifacts.

Super-timeline with plaso#

Once volatile evidence is secured, a super-timeline turns scattered artifacts into a narrative. With log2timeline.py (plaso) you parse filesystem timestamps, journald, bash_history, cron logs, and webserver access into a single time-sorted view, then filter to the incident window with psort.py. At a glance you see the SSH login at 02:14 UTC preceded the payload curl at 02:15 and the systemd unit write at 02:16.

The most common mistake is mixing time zones: work consistently in UTC and record the host's offset, otherwise you build a causal chain that is off by hours. Watch for timestomping (mtime reset via touch -t); the ctime, harder for an attacker to forge, reveals the real modification time and exposes the tampering.

Chain of custody and hashing#

Every artifact gets a SHA-256 hash at collection time, and that hash is recomputed and compared on receipt at the analysis station. If they do not match, the evidence is worthless for any formal use. We document every decision in a chain of custody with UTC timestamp, hash and operator, because even in a lab the discipline builds the muscle needed for real engagements.

Practically that means an append-only logbook (ideally on a separate system) recording every command, every copy operation, and every isolation decision with a timestamp. Whoever learns this discipline only during a real incident loses half the night reconstructing what they themselves did. Automate the logging as much as possible so it is not forgotten under pressure.

After triage: detection and hardening#

Post-triage threat hunting closes the loop. We turn detections into Sigma rules and push them into Elastic, converting the incident into permanent defensive capability, a flow detailed in Threat Hunting with Sigma and Elastic: From Indicator to Detection Rule. A single ld.so.preload finding becomes a fleet-wide detection that flags the next attempt in seconds.

For hosts that survive the incident, we apply hardening following Linux Server Hardening: Applying CIS Benchmark Without Breaking Production and review SSH per SSH Hardening 2026: Algorithms, Certificates and Bastion Hosts before returning them to simulated production. A restored host without the initial access vector closed gets re-compromised within days.

Checklist for live triage#

Before you touch anything: (1) read-only toolkit mounted, output target encrypted and separate; (2) volatile state captured (sockets, processes, modules, /proc) with UTC timestamps; (3) memory dump via AVML with at least 2x RAM space reserved; (4) UAC ir_triage plus a manual find sweep for fresh files; (5) all four persistence classes (systemd, cron, rc.local/profile, SUID/authorized_keys) checked; (6) every artifact hashed and entered in the custody log; (7) isolation only after collection is complete; (8) detection played back as a Sigma rule.

FAQ#

Should I take the machine off the network immediately? It depends on what you are protecting. With active data exfiltration, yes, but ideally via an upstream firewall rule rather than pulling the cable at the host, so memory state and the C2 connection survive for capture. A hard disconnect can trigger in-progress encryption or a dead-man switch.

Is disk forensics enough without a memory dump? No, not against memory-resident malware and LD_PRELOAD rootkits. A lot of modern tooling barely touches disk. Without a memory dump you see half the story and write guesses into the report.

UAC or Velociraptor: which first? For a single isolated host, UAC is faster to set up and needs no server. As soon as you have more than a handful of hosts or repeated hunts, Velociraptor wins through central orchestration and VQL. In practice we combine both: Velociraptor for the fleet-wide sweep, UAC for deep collection on the hosts flagged as compromised. It is not either-or, it is a sequence.

Practical takeaway: build a LUKS pendrive today with UAC, AVML and a preconfigured Velociraptor client pointing at your hunt server. Test it on a clean Debian VM, time it, tune the profile and version the tar.gz in your internal repository. When the pager rings at 3 AM you do not want to be reading documentation, you want to be collecting evidence. Live triage is not art, it is a disciplined checklist executed under pressure.