Adversary Emulation with Caldera and MITRE ATT&CK in a Corporate Lab

When a client calls saying they bought the top-shelf EDR and want to know if 'it is any good', the honest answer is never a PowerPoint report. It is an adversary emulation engagement with a written scenario, ATT&CK mapping and cold metrics: how many steps went silent, how many produced an alert, how many escalated into human response within 60 minutes. At Basilisk OffSec we standardised this around Caldera 5.x, Atomic Red Team and a Windows domain lab with 8 hosts that mirrors the client environment without ever touching their production. This piece describes the stack, the flow and the mistakes we charged ourselves for so we would not repeat them.

Emulation is not a pentest: what adversary emulation actually measures

A pentest hunts vulnerabilities and proves exploitability. Adversary emulation asks a different question: if you reproduce a concrete threat step by step, does the defense see and stop it in time? The deliverable is not a list of CVEs but a detection scorecard per technique. That is why the output is always a table of ATT&CK technique, outcome (silent, alerted, blocked) and time to response. Anyone confusing this with a pentest buys the wrong product and ends up measuring the wrong thing.

The value comes from repeatability. The same emulation, run monthly, shows whether new Sigma rules and EDR tuning genuinely improve detection rate or whether only the feeling improved. Without a baseline, any claim of 'better security' is marketing.

The reproducible lab: eight hosts, Proxmox, and Terraform

The base lab is simple and reproducible: one Windows Server 2022 as DC, two Server 2019 boxes (file and SQL), three Windows 11 Enterprise workstations running Defender for Endpoint in block mode, and two Ubuntu 24.04 hosts with auditd and a Wazuh agent. Everything is orchestrated on Proxmox via Terraform with snapshots named per scenario phase. Anyone bootstrapping the Windows side from scratch should start with Active Directory Pentest: Step-by-Step Kerberoasting in a GOAD Lab, which already ships intentionally messy ACLs, and then plug in the pivoting setup from Pivoting with Chisel and Ligolo-ng: Segmented Networks in a Pentest Lab to model realistic VLAN segmentation. Without that segmentation, any lateral-movement detection metric is biased from the start.

Terraform is not decoration here. It guarantees the lab boots deterministically from the same state after every operation, so two runs stay comparable. Named per-phase snapshots let you jump back to the exact moment before lateral movement whenever a detection question stays open.

Caldera as the brain: adversary YAML and ATT&CK mapping

Caldera becomes the brain of the operation. We run the server on an isolated Debian box, load the 'atomic' and 'stockpile' plugins plus our internal fork 'basilisk-ttps' which packages region-specific behaviour like AnyDesk abuse in SMBs. Each Caldera adversary is a YAML file that cites ATT&CK technique IDs: T1059.001 for PowerShell, T1021.006 for WinRM, T1003.001 for LSASS dumping. We never run off-the-shelf adversaries unreviewed; they are useful as benchmarks, but the actual value comes from writing scenarios that reproduce the client threat model. A retail with an exposed POS does not deserve the same adversary as a fintech with federated Azure AD and strong MFA.

An adversary profile is an ordered sequence of abilities, each with an ATT&CK reference, an executor and a cleanup command. That cleanup line is not optional: it ensures the host returns to a known state after the run, or the first test contaminates the second. We version every YAML in git and review it like production code.

The four-phase execution loop

Execution follows a four-phase loop we learned to respect. First, simulated initial access on a controlled endpoint, usually following the playbook from Simulated Initial Access: Macros, LNK and ISO in an Isolated Windows 11 Lab with a payload signed by an internal CA. Second, execution and discovery leaning on LOLBins, where the material in Hunting Living-off-the-Land Binaries on Windows with KQL helps the blue team have KQL ready before the exercise starts. Third, lateral movement over SMB and WinRM abusing weak service accounts. Fourth, actions on objectives, which can mean SQL data exfiltration or simulated share encryption.

Each phase has a clear stop criterion: if the EDR kills the process in under 90 seconds, we log it as detected and try another path. That hard criterion prevents the most common self-deception pattern, where the operator keeps tweaking until something slips through and then claims the chain ran end to end. Detected is detected; that is a good result for the client, not a failure for the team.

Blue side: instrumentation, telemetry, and the three metrics

The boring but decisive part is blue-side instrumentation. Without comparable telemetry, adversary emulation turns into theatre. We deploy Sysmon with Olaf Hartong config, ship to an Elastic 8.14 cluster and apply converted Sigma rules, a workflow we covered in Threat Hunting with Sigma and Elastic: From Indicator to Detection Rule. Every event Caldera triggers gets a custom header x-caldera-op-id, so in Kibana we can query which technique produced which events and how long the analyst took to respond.

The three metrics we always report: MTTD per technique, detection rate by ATT&CK tactic and number of new Sigma rules authored as a direct result. Without those three numbers, the client thinks they bought a pentest and walks away frustrated. With them, they get a maturity curve they can show the board and defend across quarters.

A concrete run: from ability to detection gap

A typical run: Caldera delivers a Sandcat agent via a signed LNK, executes T1059.001 (PowerShell discovery), then T1003.001 (LSASS access through a LOLBin instead of Mimikatz to vary the signature), and pivots via T1021.006 (WinRM) to the file server. In a real client run the LSASS access went silent because the existing Sigma rule only targeted the process name mimikatz.exe rather than the suspicious handle access to lsass.exe. That exact gap is the product: a new, behavior-based rule that lands in the report.

For each ability we record the timestamp of the Caldera event and the timestamp of the first correlated alert in Elastic. The difference is the MTTD of that technique. Aggregated over the tactic it yields the detection rate we track quarter over quarter against the previous runs.

Ethical and operational traps

There are ethical and operational traps worth naming. Adversary emulation is not a license to run EDR evasion against someone else's production; anything involving direct syscalls or AMSI patching stays inside the lab, and whoever wants the reasoning can read EDR Evasion for Research: Direct Syscalls Explained Without the Hype and AMSI and ETW Bypass for Defensive Research: What Blue Teams Should Know. We also never share C2 infrastructure across clients; each engagement gets a dedicated Sliver teamserver built per Building C2 Infra with Sliver in an Isolated Lab for Defensive Research.

Every operation stands on a written authorization with a clear scope and rules of engagement. A destructive step, such as simulated share encryption, only runs against dedicated lab data and never against client production. That line is non-negotiable, not even to 'just quickly try it'.

Reporting and the purple team feedback loop

The final report always ties each TTP to a concrete countermeasure, feeding the feedback loop from Purple Team in Practice: Building a Red vs Blue Feedback Loop and keeping the exercise out of the PDF graveyard. For every undetected technique we deliver the proposed detection logic, not just the observation that a gap exists. That is the difference between a report that gets filed and one that measurably raises the detection rate on the next run.

The loop closes when the rules proposed in the report are tested in the next cycle. That is exactly why a monthly cadence beats one big annual test: only repetition proves the defense actually learned something.

FAQ: How is Caldera different from Atomic Red Team?

Atomic Red Team is a library of single test snippets per technique, ideal for spot-checking whether a detection for T1003.001 exists. Caldera orchestrates those building blocks into a coherent chain with an agent, planning and cleanup, so a whole attack story plays out. In practice we use Atomic for quick single validation and Caldera for the full, scripted operation. They do not compete, they complement each other.

FAQ: Do you need an expensive EDR for this to be worthwhile?

No. The whole point of emulation is to measure what your existing telemetry delivers. Sysmon plus Elastic plus converted Sigma rules is a completely free detection base you can emulate against meaningfully. An expensive EDR often improves the raw data, but without tested detection logic and a trained blue team it stays an expensive log archive.

Practical takeaway: start small, iterate monthly

Practical takeaway: start small. Spin up Caldera, write one adversary with five ATT&CK techniques aligned to your real threat model, run it against three monitored endpoints and measure MTTD per technique. Repeat monthly, swapping one technique each cycle. In six months you will have a defensible maturity curve to show executives, without buying any new expensive tool, and your SOC will stop complaining that 'nothing ever happens' during training.