Supply Chain Security: Sigstore Signing and Real SBOMs in CI/CD

When the XZ Utils backdoor (CVE-2024-3094) nearly reached stable Linux distros in March 2024, it made one thing painfully clear: signing a GitHub tag no longer counts as supply chain security. The software supply chain has become priority real estate for attackers because a single compromised maintainer pushes malicious code to millions of hosts within hours. At Basilisk OffSec we sit on both sides: we simulate attacks against client pipelines and help teams close the doors we walk through. This post consolidates what we actually deploy in production CI/CD using Sigstore cosign, SLSA level 3, and CycloneDX SBOMs, with bias toward pragmatism over compliance theater.

Why the supply chain became the target

An attacker has two roads to your production: the front door (attack your exposed services) or the back door (compromise something you already trust). The back door scales better. The SolarWinds incident showed that a tampered build process hits 18,000 organizations at once. XZ showed that a patient attacker earns social trust over two years, gains maintainer rights, and plants a backdoor that only fires under very specific conditions.

The common denominator: the weakest link is never the cryptography, it is the human and procedural chain around the build. That is why the answer is not a single technology but a chain of signing (who built it), provenance (how and where it was built), and inventory (what is inside). Miss one link and you are flying blind. The 2021 CodeCov incident, where a leaked credential tampered with a bash uploader script, was the wake-up call for the signing layer.

Cosign keyless: Fulcio and Rekor

Step one is abandoning the long-lived private signing key sitting in a secret manager. cosign keyless uses OIDC from GitHub Actions, GitLab CI, or Buildkite to mint an ephemeral certificate via Fulcio, valid for 10 minutes, and records the signature in Rekor, an append-only transparency log. That eliminates a whole category of signing key leaks. A typical workflow has three jobs: reproducible build, sign with cosign sign --yes, and attest with the SLSA Provenance v1.0 predicate.

The key insight of the transparency log: even if an attacker briefly obtains a valid OIDC identity, every signature is publicly and immutably recorded. You can retroactively ask which artifacts were signed with which identity at which time, and hunt for anomalies. On client projects we measured a 90 percent reduction in credential rotation time after migrating from PGP keys to keyless, and that ties directly into what we discuss in AppSec Shift-Left: SAST, SCA and Secrets Scanning Without Slowing the Team.

SBOM with Syft and CycloneDX

An SBOM is not an XML file no one reads. We generate CycloneDX 1.6 with syft at build time, including SHA-256 hashes for every component, SPDX license, and supplier. A real invocation: syft packages dir:. -o cyclonedx-json=sbom.json. The file travels next to the OCI image as a referenced artifact, bound to the image via cosign attest --predicate sbom.json --type cyclonedx, not buried in an S3 bucket no one will find in six months.

The value of an SBOM shows up on the day of the next zero-day. When a critical CVE surfaces in a transitive dependency, the question is not abstract but operational: which of our 200 running images contain the vulnerable version? Without an SBOM that is multi-day archaeology. With a queryable SBOM inventory it is a query of minutes. That is exactly why you bind the SBOM to the artifact and not to a wiki page.

Scanning with Grype and Policy-as-Code

To validate, we run grype on the pull request and again in the background against the registry, because fresh CVEs land after merge. An image that was clean yesterday can carry a critical vulnerability today without a single line of code changing. We pair this with policy-as-code through Kyverno or Conftest, blocking deploys when the SBOM carries a dependency above CVSS 7.0 without a documented exception.

The exception mechanism matters. A hard policy with no documented exception path gets bypassed the moment it blocks an urgent deploy, and then it is dead. We use a VEX file (Vulnerability Exploitability eXchange) to declare that a given CVE is not exploitable in the specific context, with a rationale and an expiry date. The same policy pattern reappears when we discuss Linux Server Hardening: Applying CIS Benchmark Without Breaking Production on production servers.

Understanding the SLSA levels

SLSA (Supply-chain Levels for Software Artifacts) became the reference framework after Google and OpenSSF standardized v1.0 in 2023. Level 1 only requires a versioned build script. Level 2 demands a hosted builder with signed provenance. Level 3, where we want to land for sensitive projects, requires build isolation, non-forgeable provenance, and hermetic builds where the build has no uncontrolled network access.

The decisive jump is between Level 2 and 3: provenance the builder itself generates and signs cannot be forged by a compromised developer, because they do not control the builder process. For teams already wrestling with Dependency Confusion and Typosquatting: Practical Defense for Dev Teams, SLSA closes the publisher side, while namespace reservation and registry pinning cover the consumer side.

Provenance with slsa-github-generator

In practice we use GitHub Actions with the reusable workflow slsa-framework/slsa-github-generator, which produces an in-toto attestation signed by the runner itself via OIDC. That attestation cryptographically documents the source commit, the build parameters, and the toolchain version. An attacker who pwns a maintainer still has to break the GitHub builder, raising attack cost by orders of magnitude.

The most common break point here is non-reproducible builds. Timestamps embedded in Go binaries, absolute paths, and embedded build IDs make two builds of the same commit produce different hashes. Set -trimpath, pin SOURCE_DATE_EPOCH, and lock the toolchain in a lockfile, otherwise your provenance is signed but not verifiably reproducible.

Consumer-side verification

Consumer-side verification matters as much as producer-side signing. On Kubernetes clusters we deploy the Sigstore policy-controller admission webhook with a ClusterImagePolicy that requires every image in production namespaces to carry a valid cosign signature, OIDC identity from basilisk.example, and SLSA attestation from the expected builder. We configure soft-fail in staging to avoid blocking emergency deploys, and hard-fail in prod.

We also run cosign verify-blob on CLI binaries pulled to analyst workstations, wired into the workflow described in OPSEC for Security Researchers: Building a Personal Threat Model, because offensive teams download a lot of tooling from sketchy origins and that becomes an obvious vector. The principle: sign at the origin, verify at the consumption point, and never trust an artifact just because it sits in your registry.

Limits: social takeover and Scorecard

Where reality breaks: non-reproducible builds, timestamps embedded in Go binaries, transitive dependencies that drift between git pull and CI, and maintainers who refuse 2FA. The pre-2.0 XZ project showed social takeover signals that no signature catches: a new contributor building trust suspiciously fast, pressure on the original maintainer, and commits with obfuscated test data.

That is why we recommend combining cosign+SLSA+SBOM with periodic critical-maintainer review, OpenSSF Scorecard running weekly across the top 50 dependencies, and red team exercises that simulate maintainer compromise, similar to engagements described in Adversary Emulation with Caldera and MITRE ATT&CK in a Corporate Lab and Purple Team in Practice: Building a Red vs Blue Feedback Loop. Technology without a maintenance process turns into expensive theater.

Registry pinning and immutable digests

Signing and provenance buy you little if your deploys reference a moving tag like :latest. A :latest verified yesterday can point at a different image today. In production, always pin to the immutable digest (image@sha256:...), not a tag. The digest is the cryptographic fingerprint of the exact image content; a pinned digest plus a verified signature gives an unbroken chain from source commit to running container.

Complement that with a mirrored internal registry (Harbor, Artifactory) running a pull-through cache with immutability rules, so a deleted or overwritten upstream tag does not break your builds and an attacker cannot re-bind an already-verified tag. Combined with an allowlist of permitted registries in the admission controller, you close the gap dependency-confusion attacks slip through.

30-day rollout

This week: add cosign sign keyless to a single pipeline, generate a CycloneDX SBOM with syft, and publish it as an OCI artifact next to the image. Next week: turn on grype in the PR with a warn-only policy to calibrate your baseline CVE noise. Week three: switch on mandatory cosign verify in one staging namespace with soft-fail. Week four: flip to hard-fail in prod for a single, well-understood service and expand from there.

FAQ

Does keyless require internet access at build time? Yes, Fulcio and Rekor are online services. For air-gapped environments you run a private Sigstore instance (Fulcio, Rekor, Trillian) internally or fall back to key-based cosign with a key held in a KMS/HSM. Keyless is the default recommendation, not a dogma.

Does an SBOM replace vulnerability scanning? No. The SBOM is the inventory, the scanner is the analysis against it. An SBOM without continuous rescanning against current CVE feeds ages in days. Together they give a queryable, current picture of your attack surface; alone they are half measures.

How do you sell the cost to management? Not with compliance arguments but with the response-time math. During the last major OpenSSL CVE, teams without an SBOM inventory took days on average just to figure out which systems were affected. Teams with a bound SBOM answered the same question in minutes and started patching immediately. That difference translates directly into outage risk and overtime, and that is exactly what a budget owner understands.

Practical takeaway: start small and measurable. Within 30 days you have leverage to require SLSA L2 from vendors and evidence to respond when the next XZ lands, because another one is coming. The implementation cost on an existing pipeline runs about 2 to 4 engineer hours; the cost of skipping it is fielding a Sunday call asking whether you signed the container currently mining Monero in production.