Safe Container Releases: Immutable Builds, Progressive Delivery, and Rollback

Published

Jul 14, 2026

Updated

Jul 15, 2026

Reading time

15 min read

Containers make application packaging consistent, but they do not make a deployment safe by themselves. A mutable image tag, startup migration, missing readiness check, or unmeasured traffic shift can turn a routine rollout into an incident. Safe container releases make risk visible and keep recovery options open, not by adding process for its own sake but by making each stage of a rollout something a human or a controller can actually verify.

That discipline matters even more once the service sits behind a managed edge layer rather than being reached directly. A release that changes a cache key, an auth check, or a response shape can look perfectly healthy in kubectl get pods while it is quietly breaking checkout for users several hops away at the edge. Orchestrating releases safely across many services and providers is exactly the kind of consistency problem a managed edge orchestration layer exists to catch, because it is watching the same rollout from the customer's side of the network, not just the cluster's.

The Twelve-Factor separation of build, release, and run is a useful baseline. Build creates an artifact. Release associates it with approved configuration and migration decisions. Run starts that exact release. Preserve those boundaries so engineers can answer what code ran, where, with which configuration, and how to reverse course.

Build immutable, traceable artifacts

Immutable container builds are the foundation everything else in this guide depends on: build a single image from a pinned base image and a locked dependency graph, tag it for human readability, but deploy by digest so the runtime receives the exact bytes that were tested. Record the source revision, build time, dependency metadata, and provenance in your artifact system.

Use a multi-stage build to keep compilers, package caches, and test tools out of the runtime image. Run as a non-root user where the application permits it, use a minimal runtime base, and avoid adding shell tools solely for production debugging. Smaller images generally pull faster during scale events, but minimality must not prevent necessary certificate, timezone, or diagnostic behavior.

Scan base images and dependencies continuously, then prioritize findings by exploitability and runtime exposure rather than treating every CVE as equally urgent. Sign artifacts with a supply-chain tool such as Sigstore's cosign and record SLSA-style build provenance, then verify signatures in the deployment path rather than only at build time. A signed vulnerable image is still vulnerable; signatures establish provenance, not safety.

Signing only pays off if something enforces it. An admission controller such as Kyverno's image-verification policies, or Sigstore's own policy-controller, can reject any pod spec whose image digest lacks a valid signature or attestation before it is ever scheduled, which closes the gap between "we sign images in CI" and "an unsigned or hand-edited manifest can still reach the cluster." Start these policies in audit or warn mode, confirm every legitimate release path still passes, and only then switch to enforcing so a misconfigured policy cannot block an incident rollback.

Make the release a reviewed unit

Create a release record that links image digest, deployment manifest or chart revision, configuration version, migration version, owner, and change reason. Promote the same tested artifact across environments. Rebuilding from the same source for production can pick up different dependency or base-image content and breaks the evidence chain.

Keep environment-specific configuration outside the image. Validate it before the traffic shift, including hostnames, certificates, secret references, resource requests, and feature flags. Do not use a container startup hook to fetch unpinned application code or run an unreviewed database mutation.

Keep database changes compatible

Database changes are often the real rollback constraint. Use the expand-contract pattern:

1. Add a backward-compatible schema, index, or field.
2. Release code that can read old and new representations.
3. Backfill or migrate data in bounded, observable batches.
4. Switch reads and writes after verification.
5. Remove the old path only after all supported releases no longer need it.

Avoid destructive or long-running migrations in an application pod startup path. Multiple replicas may race, a failed rollout may leave partial data, and a migration can exceed deployment timeouts. Run migrations as an owned, idempotent operation with a lock where required, progress monitoring, a pause plan, and an explicit rollback or forward-fix decision.

Messages and APIs need the same compatibility discipline. Consumers may lag producers during a rollout, and clients at the edge may retain cached assets or open connections for longer than a deployment. Version schemas, tolerate unknown fields where appropriate, and deprecate deliberately.

Shift traffic with progressive delivery

Start with the smallest safe exposure: internal validation, one replica, a canary slice, or a selected region. Compare the candidate with the current release using a defined observation window and automatic or operator-owned promotion criteria. Useful signals include successful request rate, p95 and p99 latency, resource saturation, dependency errors, queue age, cache hit ratio, and business journey completion.

Choose a rollout method that fits the risk:

• Rolling update: practical for compatible, stateless services with good readiness and drain behavior.
• Canary: limits blast radius and supports comparison under real traffic.
• Blue-green: provides a quick traffic reversal but requires data and dependency compatibility across both versions.
• Feature flag: separates code delivery from behavior exposure, but flags need ownership, expiry, and secure targeting.

Manually editing route weights, as in the Acme Shop example later in this guide, works and is easy to audit, but it does not scale past a handful of services. Progressive delivery controllers such as Argo Rollouts or Flagger automate the same idea: they own a canary's weight steps, query a metrics provider (Prometheus, Datadog, CloudWatch, and others are commonly supported) against an analysis template you define, and promote or abort automatically when the query crosses your threshold. Adopting one does not remove the need for good signals or a compatible migration; it only removes the human step of deciding to advance, so a bad analysis template will automate a bad decision just as confidently as a good one automates a good one.

Kubernetes maxUnavailable and maxSurge should reflect real capacity. A service that normally runs at 80% of its dependency or CPU budget has little room for a surge. Pod disruption budgets protect against voluntary disruptions only — a node drain, a cluster upgrade, anything that goes through the Eviction API — and do not stop an involuntary loss such as a node failure, an OOM kill, or the rollout's own replacement of old pods; they do not compensate for inadequate replica count, bad readiness, or a broken application version. If the workload is a StatefulSet rather than a Deployment (a stateful cache tier or a search index, for example), rolling updates default to OrderedReady and support a partition field for a manually staged rollout; do not assume Deployment-oriented canary tooling applies unmodified.

Readiness alone conflates two different questions: is the container still starting, and is it currently unable to serve traffic. A startupProbe answers the first one: while it is unsuccessful, Kubernetes suppresses both liveness and readiness checks, so a slow cache warm, migration client, or JVM class-load does not trip a liveness restart before the application has finished coming up. Configure a generous startupProbe for anything with unpredictable cold-start time, keep readinessProbe strict enough to actually gate traffic, and keep livenessProbe narrow enough that it only fires on a genuine deadlock rather than a slow dependency.

Protect latency through the release path

Release risk includes performance regressions. A new serialization library, cache-key change, connection-pool default, or logging call can affect tail latency before error rate rises. Set performance acceptance thresholds for critical routes and compare them with a baseline. Include cold starts, cache misses, and geographically distant users in tests where they represent production traffic.

If public content is delivered through an edge cache, deploy cache behavior and origin code as a coordinated change. Validate cache-control directives, vary dimensions, invalidation, origin authentication, and rollback behavior. Do not purge globally by default for every application release: a cold cache can overload an origin and increase worldwide latency. Use targeted invalidation and capacity planning.

Keep direct origin access restricted so traffic cannot bypass the intended edge controls during or after a release. Health checks must test the correct version and route. A load balancer that marks a pod healthy based only on a TCP connection can send users to an application that has not completed initialization.

Observe, stop, and recover

Instrument each release with an image digest or release identifier in metrics, logs, and trace resource attributes. Use OpenTelemetry propagation to connect edge requests, ingress, application spans, and dependency calls. Never attach secrets or customer payloads to telemetry.

Define stop conditions before rollout: a sustained error-rate increase, p99 latency regression, failed synthetic transaction, depleted error budget, queue-age rise, or elevated dependency saturation. Alerting should distinguish expected rollout churn from user impact, and the on-call owner must be able to pause promotion.

Rollback means returning traffic and behavior to a known-good state, not just applying an earlier manifest. Confirm the old image remains available, configuration remains compatible, database migrations permit it, and caches will not serve an inconsistent response. When rollback is unsafe, use a prepared forward fix with the same staged controls.

A safe container releases checklist

The pattern behind safe container releases is repeatable, so treat it as a checklist rather than something re-derived from memory each time. Before production, confirm the image digest is pinned and traceable; the dependency and base-image posture is understood, and an admission policy would accept the signature; configuration and secret references are validated; migrations are compatible and owned; startup, readiness, and shutdown probes are configured and tested, not just present; capacity includes rollout surge; release signals and stop conditions exist, whether operator-owned or driven by an analysis template; and rollback has been rehearsed for code, configuration, and data.

After release, keep watching beyond the controller's success message. Some defects appear only after cache expiry, token rotation, autoscaling, a regional traffic pattern, or a scheduled job. Record the result and improve the release template rather than relying on memory for the next change.

Apply it: Acme Shop catalog release

Acme Shop promotes one tested catalog-api digest with a backwards-compatible products.display_name addition. The migration runs as an idempotent, observable job before traffic changes. A reviewed route sends 95% of matching requests to the stable Service and 5% to the canary Service; a canary replica count supplies capacity, not percentage exposure. Both versions can read the old and new schema until the rollback window closes.

Digest-pinned canary manifest

This sandbox example makes the candidate bytes, Service selection, and traffic split reviewable. It requires an existing acme-shop-gateway listener whose controller supports Gateway API HTTPRoute weighted backend references. The digest and hostname are intentionally fake; never substitute a mutable tag or an unreviewed route for a production approval.

apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog-api-canary, namespace: acme-shop-sandbox }
spec:
  replicas: 1
  selector: { matchLabels: { app: catalog-api, track: canary } }
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  template:
    metadata: { labels: { app: catalog-api, track: canary, release: "2026-07-14.2" } }
    spec:
      containers:
        - name: catalog-api
          image: registry.example.invalid/acme/catalog-api@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
          readinessProbe: { httpGet: { path: /readyz, port: 8080 } }
---
apiVersion: v1
kind: Service
metadata: { name: catalog-api-stable, namespace: acme-shop-sandbox }
spec:
  selector: { app: catalog-api, track: stable }
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: catalog-api-canary, namespace: acme-shop-sandbox }
spec:
  selector: { app: catalog-api, track: canary }
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: catalog-api, namespace: acme-shop-sandbox }
spec:
  parentRefs: [{ name: acme-shop-gateway, sectionName: https }]
  hostnames: ["catalog.sandbox.acme.example.invalid"]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /v1/catalog } }]
      backendRefs:
        - name: catalog-api-stable
          port: 80
          weight: 95
        - name: catalog-api-canary
          port: 80
          weight: 5

release=2026-07-14.2 traffic=5% error_rate=0.08% p95_ms=184 synthetic_catalog=pass
gate=hold observation_window=10m stable_release=2026-07-07.4

Migration, validation, and rollback behavior

For products.display_name, first add the nullable column, deploy code that reads display_name ?? name, and backfill in rate-limited batches. Rollback before the contract step means returning traffic to the stable digest and stopping the backfill; the added nullable column remains harmless. Do not drop name until every supported release and asynchronous consumer has moved. If a migration is not backwards compatible, do not claim code rollback is available: pause and use the pre-approved forward fix or tested data restore path.

Apply the sandbox manifest only with server-side validation: kubectl apply --dry-run=server -f catalog-api-canary.yaml. Before sending traffic, verify the Gateway controller has accepted the route and resolved both backend references; a replicas: 1 deployment alone does not establish 5% exposure. Send a synthetic catalog request through the routed hostname, then observe the 5% cohort for ten minutes. Promote only if error rate, p95, and the synthetic journey stay inside the pre-approved gate. On failure, apply the reviewed stable-only HTTPRoute with catalog-api-stable at weight 100 and the canary omitted, confirm the route is accepted, keep the stable digest serving, stop the migration job if safe, and record whether a forward fix is required before retrying.

Related guides

• Modern Twelve-Factor Apps
• Cloud-Native Configuration and Secrets
• Stateless Services on Kubernetes
• Dependency Resilience

Authoritative references

• The Twelve-Factor App: Build, release, run
• Kubernetes: Deployments
• Kubernetes: Configure liveness, readiness, and startup probes
• Gateway API: HTTP traffic splitting
• Sigstore: Policy Controller overview
• Argo Rollouts: Analysis and progressive delivery
• Google SRE Workbook: Canarying releases
• OpenTelemetry documentation