Kubernetes SRE guide

Kubernetes SRE SLOs: Define, Measure, Operate

Turn customer journeys and service dependencies into explicit reliability and latency decisions, rather than a dashboard of disconnected Kubernetes metrics.

Published: Jul 14, 2026
Updated: Jul 14, 2026
Reading time: 14 min read

On this page

Kubernetes makes it easy to measure pods, nodes, and API requests. Those signals are essential, but they are not a service level objective (SLO). An SLO is an agreed target for a user-visible outcome over a stated window, paired with a service level indicator (SLI) that can show whether the target is being met.

An SLO is a product and engineering agreement

Do not adopt a percentile, availability number, or window because another organization publishes it. Select an objective from the user journey, the harm caused by failure or delay, historical behavior, and the team's ability to act on the result.

Overview

Outcome and prerequisites

Outcome: Acme Shop can measure checkout availability from valid customer requests, make a release decision from a 30-day error budget, and page only for sustained customer impact. Prerequisites: a non-production Acme Shop namespace, a route-level request counter and duration histogram, Prometheus rule deployment access, and agreement from checkout and platform owners on valid and successful outcomes.

Running scenario: Acme Shop checkout

Acme Shop operates checkout-api behind an ingress. A valid checkout request is one that reaches the route and receives a 2xx, 3xx, or 5xx response; malformed or intentionally rejected 4xx requests are tracked separately and are not silently reclassified after an incident. A good valid event is a 2xx or 3xx response. The first objective is 99.9% availability over 30 days, with a separate latency objective for successful requests.

Acme Shop checkout SLO measurement path

Shopper and edge
The edge adds trace context and records delivery outcome.
Ingress
Route-level status and duration identify requests that reach checkout.
checkout-api
Application metrics and traces connect the result to release and dependency evidence.
Prometheus rules
Recording rules calculate valid, good, and error-budget burn rates.
SLO decision
The dashboard and alert drive a scoped release or incident response.

Figure 1. Availability is measured from the checkout journey boundary, then correlated with ingress, application, and dependency evidence.

Define the indicator and budget

For availability, use the correct direction of the ratio:

availability = good valid events / total valid events

For 1,000,000 valid checkout requests at a 99.9% target, Acme Shop may have 1,000 bad events in the 30-day window. If 250 requests are bad, availability is 999,750 / 1,000,000 = 99.975% and 250 / 1,000 = 25% of the error budget is spent. A 1% bad-event rate is ten times the allowed 0.1% bad-event rate, so it is a 10x burn rate. This calculation is meaningful only while the valid-event definition remains stable and traffic is sufficient to interpret it.

Use a threshold-based good-event ratio for the availability SLI. For latency, define a second SLI such as the proportion of successful, valid requests completing within an agreed threshold. Do not use CPU, restart count, or pod readiness as a substitute for either user-visible indicator; they are diagnostic signals.

Record the SLI before alerting

The following Prometheus rule assumes http_requests_total is a monotonically increasing counter with bounded service, route, and code labels. It deliberately excludes 4xx responses from this availability population. Validate route and status semantics against the application before applying the rule through the normal review process.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: acme-shop-checkout-slo
  namespace: acme-shop-sandbox
spec:
  groups:
    - name: acme-shop.checkout-slo
      rules:
        - record: acme_shop:checkout_valid_requests:rate5m
          expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3..|5.."}[5m]))
        - record: acme_shop:checkout_good_requests:rate5m
          expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3.."}[5m]))
        - alert: AcmeShopCheckoutFastBurn
          expr: (1 - (acme_shop:checkout_good_requests:rate5m / acme_shop:checkout_valid_requests:rate5m)) / 0.001 > 14.4
          for: 10m
          labels:
            severity: page
          annotations:
            summary: "Acme Shop checkout is consuming its 99.9% availability budget quickly"
            runbook: "https://runbooks.example.invalid/acme-shop/checkout-slo"

The alert expression is a 5-minute error ratio divided by the 0.1% error-budget fraction. A rate above 14.4x, sustained for ten minutes, is a fast-burn signal; pair it with a longer-window condition and minimum-valid-request guard before using it for paging in production. Alerting policy should reflect customer impact and traffic volume, not a single failed health check.

Representative output: Acme Shop checkout SLO query

window=30d valid_requests=1000000 good_requests=999750
availability=99.975% target=99.900% allowed_bad=1000 actual_bad=250
budget_spent=25% current_error_rate=0.025% current_burn=0.25x
release_decision=continue_with_normal_review

Operate the objective

Maintain a long compliance window and shorter burn windows. Normal consumption can permit routine releases. Elevated burn should trigger release review and diagnosis; exhausted budget should trigger reliability work or an explicit decision by accountable owners. A security patch or urgent customer fix can still be appropriate, but the risk should be recorded rather than hidden behind an automatic gate.

Instrument the complete request path with bounded metrics, W3C trace context, logs, and Kubernetes events. Segment dashboards by route, region, release version, and cache state where those dimensions materially change the outcome. A green node dashboard is not evidence that checkout recovered.

Validation

Validation, rollback, and failure behavior

In a sandbox, send a known number of synthetic checkout requests that return both 2xx and controlled 5xx outcomes. Confirm the valid and good recording rules match the expected counts, then inspect a trace for each cohort. If a rule labels valid traffic incorrectly, remove only the unapproved PrometheusRule from the sandbox or restore its last reviewed revision; do not change status handling or suppress an alert globally. If the metric pipeline is unavailable or valid-request volume is too low, mark the SLO measurement unavailable, use the incident runbook and supporting telemetry, and repair measurement before relying on the budget for a release decision.

Troubleshooting

Symptom	Likely cause	Safe check	Recovery
Availability is above 100% or missing	Good and valid queries use different label filters or a zero denominator	Query both recording rules with identical route and service scope	Correct the reviewed rule and wait for fresh samples; do not clamp the result to hide a query defect.
Budget falls after a client validation change	`4xx` responses entered the valid population unexpectedly	Compare status-class counts by release version	Restore the prior application or metric-label behavior, then document whether the SLI contract should change.
Fast-burn alert fires during tiny traffic	No minimum-volume guard or low-volume service	Inspect valid requests and raw errors for the window	Route to a ticket or combine with a longer window until traffic supports paging.
SLO is green but customers report slowness	Availability is healthy while the latency SLI is absent or too broad	Compare route-level duration histogram and trace cohorts	Add or correct the latency SLI; do not redefine availability to conceal latency.
Dashboard cannot identify the affected region	Region or release labels are absent from correlated telemetry	Inspect one sanitized trace, log, and metric series	Restore the telemetry contract and rerun the controlled validation.

Authoritative references

Make delivery latency part of the reliability conversation

Talk to Optimi about measuring and designing edge and delivery paths alongside your application reliability objectives.

Discuss performance architecture