Linux Server Hardening: Applying CIS Benchmark Without Breaking Production

Applying the full CIS Benchmark in one shot to a Debian 12 box serving 40k requests per minute is the fastest way to turn a Friday into a sev 1. The Basilisk team has watched more than one shop run the ansible-lockdown playbook straight against production and discover at 3 a.m. that control 5.2.16 disabled the service account orchestrating the Postgres backup. Serious hardening is not copying 380 controls from a PDF: it is picking the 60 that are worth the risk, replaying them in staging under the same load, and wiring telemetry so you know how many minutes you have before customer impact forces a rollback. This post is the runbook we actually use on real engagements.

Why <em>all at once</em> fails

A CIS Benchmark PDF lists Level 1 (conservative) and Level 2 (aggressive) separately, but most teams flatten both into a single to-do list and apply them in one configuration-management run. The problem is not any individual control, it is the combinatorics: 380 simultaneous changes mean that when a regression appears you can no longer bisect. You know something broke, but not which of the 380 lines did it. That is exactly how you end up at 3 a.m. running git revert on the whole playbook instead of applying a surgical fix.

The second trap is the idempotency fallacy. Many CIS remediations are not truly idempotent when the starting state differs from what the task assumes. A task that rewrites /etc/pam.d/common-auth assumes a specific default stack. If the host already has an SSSD or Kerberos module, the new line lands in the wrong order and you lock out every federated login. The fix is always the same: small batches, validation between each batch, and a documented rollback path before the first line is written.

The four buckets

The right starting point is splitting the controls into four buckets before you touch a single server. Bucket 1: kernel and boot (sysctl, GRUB, modules), low blast radius and high payoff. Bucket 2: network and firewall (nftables, IPv6, ICMP), medium risk if you have not mapped every port. Bucket 3: authentication, PAM, and SSH, where most post-hardening incidents live. Bucket 4: auditd, syslog, and AIDE integrity, essentially zero operational risk.

The order is deliberately counterintuitive: bucket 4 first, then 1, then 2, and leave SSH and PAM for last. You start where nothing can break (auditing), gather telemetry about normal behavior, then harden the kernel, then the network, and only touch the auth layer once you have verified a second access channel (console, out-of-band management, a second SSH key on a separate port). If you are already touching SSH, read SSH Hardening 2026: Algorithms, Certificates and Bastion Hosts before editing sshd_config because the algorithm story shifted again in 2026.

Inventory the current state with OpenSCAP

To inventory the current state, run openscap-scanner with the xccdf_org.ssgproject.content_profile_cis_level2_server profile from the ComplianceAsCode project. A real invocation looks like oscap xccdf eval --profile cis_level2_server --results scan.xml --report scan.html /usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xml. On a clean Ubuntu 24.04 you will see 110 to 140 controls in fail state, and that is expected, not a reason to panic.

Export the HTML report, import it into Jira as one epic per bucket, and estimate effort in risk points rather than hours. The trick is never applying a remediation without reading the rationale, because half of CIS Level 2 breaks modern workloads. Disabling usb-storage makes sense on a bastion, not on a host that writes encrypted dumps to a hardware key for an air-gapped regulator handoff. Treat the scan as a living document: re-run it after every batch and track progress as a curve, not a snapshot.

Kernel and sysctl layer

The kernel layer pays off enormously with low risk if you understand what you are tuning. kernel.kptr_restrict=2, kernel.dmesg_restrict=1, kernel.yama.ptrace_scope=2, and fs.protected_hardlinks=1 are free wins against local info leaks and symlink races. Add net.ipv4.conf.all.rp_filter=1, net.ipv4.tcp_syncookies=1, and kernel.randomize_va_space=2, and drop everything into a versioned /etc/sysctl.d/60-cis.conf rather than the main sysctl.conf so a package upgrade does not clobber your changes.

kernel.unprivileged_userns_clone=0, on the other hand, kills rootless Docker, Podman, Bubblewrap, and every application sandbox you might rely on, so if you run containers or use the techniques from Linux Application Sandboxing with Bubblewrap, Firejail and Flatpak, keep it at 1 and formally document the deviation. At boot, add a password-protected GRUB (CIS 1.4.x), Secure Boot with signed modules, and blacklist unused filesystem modules (cramfs, udf, usb-storage). For high-exposure services, push SELinux into enforcing with a custom targeted policy, exactly as we walk through in SELinux Without Fear: Custom Policies for Critical Services using an internet-facing Nginx as the worked example.

Network and firewall with nftables

Before you write a single firewall rule, map every listening port with ss -tulpen and reconcile it against the expected service. A CIS-aligned nftables ruleset works with default drop on input and forward plus an explicit allowlist. The most common self-lockout happens when you forget loopback traffic: without iif lo accept, local sockets, health checks, and the database loopback collapse, and the app throws opaque timeouts.

IPv6 is the silent trap. Many teams harden IPv4 cleanly and leave the entire IPv6 stack open, assuming it is not routed. Either you disable IPv6 consistently (net.ipv6.conf.all.disable_ipv6=1 plus the bootloader) or you mirror every IPv4 rule onto ip6tables/nftables inet family. Half coverage is worse than none because it creates a false sense of security. Test every rule change with a second open session as a lifeline.

Auditd without the log flood

Auditd almost always becomes a bottleneck if you copy the CIS ruleset blindly. Default rules generate 8 to 15 thousand events per minute on an average host, fill /var/log in six hours, and force journald to start dropping. The recipe we use at Basilisk is trimming execve rules for known service users (postgres, nginx, app) and keeping aggressive monitoring only for uid 0, sudo, and interactive shells.

Set -b 8192 for the backlog, --backlog_wait_time 0 against kernel stalls, and ship it through audisp-remote or an auditd plugin into a Sigma pipeline as we describe in Threat Hunting with Sigma and Elastic: From Indicator to Detection Rule. Otherwise you are generating expensive noise with zero actionable detection on the consuming end. Set -e 2 (immutable rules) only at the very end, because it locks every rule change until the next reboot.

An often-overlooked point: auditd and full execve logging cost measurable CPU on syscall-heavy services. On a reverse proxy making dozens of connect and openat calls per request, an overly broad ruleset can add 5 to 10 percent p95 latency. Measure that explicitly in the staging load test and treat audit rules as part of the performance budget, not a free add-on. The right metric is not the number of rules, it is events per second under normal operation.

Validation under real load

Validation is where most teams cut corners. Spin up a mirrored environment in LXD or Proxmox with the same kernel, the same glibc, and the same service versions, then replay 30 minutes of real traffic captured via tcpdump using k6 or wrk. Apply controls in batches of ten, rerun the test, and compare p95 latency and error rate. If regression exceeds 3% you bisect to identify which control caused it.

Also run atomic-red-team with techniques mapped to MITRE ATT&CK to confirm that hardening actually shrinks the attack surface: the logic is the same as in Adversary Emulation with Caldera and MITRE ATT&CK in a Corporate Lab, but aimed at post-exploitation on the hardened host. If a technique still succeeds after hardening, you have a measurable gap instead of a guess.

Disk crypto and physical access

For servers that touch sensitive data or that you cannot physically reach, pair hardening with coercion-resistant disk encryption and verified backups, following the playbook in Disk Crypto and Backups: VeraCrypt, LUKS and a Resilient 3-2-1 Strategy. LUKS2 with Argon2id, a TPM2-sealed key bound to PCR0+PCR7, and an encrypted offsite snapshot solve the physical theft scenario at the datacenter.

Combined with Secure Boot, signed modules, and a password-protected GRUB (CIS 1.4.x controls), you raise the cost of a hands-on attack to something only worth it against very specific targets. One warning: a TPM-bound key without a tested recovery passphrase is a time bomb. The day a firmware update changes the PCR values, the machine no longer boots, and without a stored passphrase the disk is data garbage.

Practical checklist

Before you declare a hardened host done: (1) OpenSCAP score documented before and after; (2) a second access path (console/OOB) verified before SSH was touched; (3) all sysctl changes versioned in /etc/sysctl.d/; (4) nftables with iif lo accept and tested IPv6 behavior; (5) auditd under 3000 events per minute at rest; (6) staging load test with under 3% p95 regression; (7) an atomic-red-team run that proves the closed techniques; (8) a rollback path documented and rehearsed once.

FAQ: Level 1 or Level 2?

For internet-facing servers without a strict compliance mandate, full Level 1 plus selected Level 2 controls from buckets 1 and 4 is the best cost-benefit point. Full Level 2 makes sense for regulated workloads (PCI-DSS, government mandates), but only with a documented exception register for the controls that break your specific workload. Blind Level 2 on a container host is an outage with a certificate attached.

FAQ: How often to re-scan? On every major OS release (Debian point release, LTS upgrade) and at least quarterly as an automated cron scan whose result flows into the same dashboards as your metrics. Drift is inevitable because package upgrades reset sysctl defaults and new services open new ports.

FAQ: Ansible, Bash script, or image baking? For a fleet, a hardened golden image (Packer plus the reviewed control list, then rolled out immutably) is more robust than an Ansible run against live hosts, because drift essentially disappears. Ansible remains good for the first iterative cycle and for hosts you cannot rebuild. A pure Bash script without idempotency is the worst option: it can neither be replayed cleanly nor reliably rolled back. Whatever you pick, the control list itself must be versioned and reviewable, not buried in a runbook wiki.

Practical takeaway: build a spreadsheet listing every CIS control you applied, the package version at the time, the openscap result before and after, and the latency delta measured in staging. Re-run that spreadsheet on every major OS release because sysctl defaults shift and auditd adds new fields. Hardening is not a project, it is a process: if six months from now you cannot prove via automated scan that those 60 controls remain in effect on the host, you do not have hardening, you have faith.