Skip to content

Datadog's Turnstile checks signed evidence before every deploy

By Quang Rau

Published
Reading time
7 min read

Datadog’s Turnstile is a useful pattern for supply chain security. It signs a record for each step an artifact goes through, then checks those records against a policy before the container reaches production. I read Datadog’s design post in full. The main lesson is simple: a signed image proves the image did not change after signing. It does not prove what went into the image.

The gap that image signing leaves

Image signing answers two questions. Is this the same image that was signed? Did a trusted key sign it? Both answers can be yes, even when the image is bad.

Datadog lists three ways this happens. A malicious change can land in the source before the build. A poisoned dependency can end up in the image. A compromised build runner can swap the base image. In each case, the signing step runs normally and the signature is valid.

The rest of Datadog’s design follows from that gap.

One signed claim per stage

Turnstile uses a modified version of the in-toto attestation framework, as the post explains. An attestation is a signed statement. It names a subject, such as a commit hash or an image digest, and it makes one claim about that subject. The claim also names who made it.

The post lists seven claim types:

  • Commit safety: a scan found the commit benign, and a trusted key signed it.
  • Source provenance: the repository and its protection settings at commit time.
  • Pull request: who reviewed and approved the change.
  • Build provenance: the CI pipeline, the job, and the image signature.
  • SBOM: a software bill of materials, which is the full list of packages in the image.
  • Vulnerability metadata: known package problems at build time.
  • Deployment: the cluster and namespace the image went to.

Each attestation carries a key that links it to the next stage. That lets an investigator rebuild the path from commit to running container. The post’s example ties a commit, a pull request with two approvals, and an SBOM to one production namespace.

Signing without stored keys

Long-lived signing keys are hard to manage at this size. Every CI job, scanner and approval system would need its own key. Datadog’s client library, dd-attest, avoids that with keyless signing. Keyless here means no signing key is kept for the long term. The library signs with a short-lived key tied to the workload’s identity: a CI job, a Kubernetes service account, or an OpenID Connect (OIDC) identity. OIDC is a standard way for a system to prove who it is.

The trust question changes. Instead of “who holds this key?”, the check becomes “which identity made this claim, and was it allowed to?” If a CI job is compromised, Datadog can invalidate every attestation that job produced in the bad window. A policy can also require a claim from one specific producer. A claim signed by any other identity does not count.

Policies in Rego, rolled out in audit mode

Policies are written in Rego and evaluated by Open Policy Agent (OPA). The post gives examples. One requires an SBOM from the approved scanner. Another requires a merged, approved pull request on a protected branch. A third blocks any SBOM that lists a package with a known critical vulnerability.

Datadog does not turn on every policy at once. New policies run in audit mode first. The team looks for gaps and false positives, which are valid deploys the policy would block by mistake. Then some policies warn and others block.

The deploy-time check

The deploy workflow calls a library called policy-core. The flow looks like this:

Deployment workflowpolicy-core gets imagedigestFetch attestations fromAMSVerify signatures andproducersRun Rego policy forenvironmentWrite signed VSA to AMSAudit, warn or block

AMS is the Attestation Metadata Store, Datadog’s DynamoDB backend for attestations. The VSA is a Verification Summary Attestation. The SLSA spec for VSAs says a consumer can trust one verifier’s decision without seeing every input attestation. Datadog’s version carries more verification details.

The part I like most is the reuse. A deploy UI, a compliance report and an incident review can all read the stored VSA. None of them has to rerun the policy.

Protecting the policies

Policies are a target too. A modified policy could let bad images pass, or turn off a check. Datadog distributes policy bundles, and its own client libraries, with The Update Framework (TUF). The TUF project says it protects updates even when an attacker compromises the repository or the signing keys. Datadog’s post says its top-level root keys are offline, and clients check signed metadata against a trusted root before use.

The bundles ship as Open Container Initiative (OCI) artifacts, which means they come from the same registry as the images. That avoids one more runtime dependency. The post does not say how often bundles change or how a bad bundle gets rolled back.

Latency and caching

Turnstile checks every container image deployed at Datadog. The post gives these scale numbers: tens of thousands of deployments and images a day, millions of attestations, and tens of thousands of policy evaluations. Policy checks sit on the deploy path, so latency matters.

Datadog chose DynamoDB because the lookups are simple key reads. The team did not need joins or a relational model. The VSA results go into a per-process LRU cache, which drops the least recently used entry when it is full. The cache key includes the artifact digest, the policy ID, software versions, the environment and some input labels.

Across about 1 million evaluation traces in August 2026, the post reports:

  • A 92.8% policy cache hit rate.
  • A 75% cache hit rate for per-artifact evaluations.
  • Evaluation latency of 36 ms at P50 (the median) and 481 ms at P99 (the slowest 1 in 100).

The post names two hit rates but does not define how they differ or how they relate. The 36 ms median is fast. The 481 ms P99 is the number I would watch.

The cache explains the hit rates. The post says artifacts are often deployed several times to the same service, and that is why it added the cache. A team that ships each artifact once would get far less from the same cache.

What the post does not say

I could not run Turnstile. The numbers cover one month, August 2026, and the post describes internal tools, so this is a reading of the design, not a test. The post also leaves out:

  • Failure mode. The post does not say what happens when policy-core or AMS is down. Does a deploy fail closed, meaning it is blocked because the check could not run? Or does it fail open, meaning it goes through? This is the first question I would ask before a check can block anything.
  • Cache invalidation (how old results get cleared). The cache key lists a policy ID, but the post does not say how a policy change makes old cached results stale.
  • Audit results. There is no false-positive rate, no policy count and no before-and-after data from the audit period.
  • Cost and setup. There is no hardware detail, no cost figure and no comparison with the old flow.

My take

I would not copy the whole stack. TUF, OPA, DynamoDB and a custom VSA format are a lot for a small team. The ideas that carry over are smaller. Keep one stored verdict per artifact, so other tools read one answer. Name the producers you trust, so you can cut off one that is compromised. Let a new rule report before it can block a deploy.

If you run a deploy gate, the next step is small. Pick one deploy path you own. Write down whether it fails open or fails closed when its check is down. Only then add a rule that can block.

Sources

  1. Beyond container image signing: How we enforce software supply chain policies at Datadog (opens in new tab)
  2. SLSA: Verification Summary Attestation (VSA) (opens in new tab)
  3. in-toto Attestation Framework specification (opens in new tab)
  4. The Update Framework (TUF) (opens in new tab)