Kubernetes SRE guide
Kubernetes SLOs for SRE Teams: Define, Measure, Operate
Turn customer journeys and service dependencies into explicit reliability and latency decisions, rather than a dashboard of disconnected Kubernetes metrics.
On this page
Kubernetes makes it easy to measure pods, nodes, and API requests. Those signals are essential, but they are not a service level objective (SLO). An SLO is an agreed target for a user-visible outcome over a stated window, paired with a service level indicator (SLI) that can show whether the target is being met.
At the scale a managed edge-orchestration layer operates — many services and providers in front of one origin — a green cluster dashboard stops being a reliable proxy for customer experience long before anyone notices from inside the cluster. Kubernetes SLOs and their error budgets are what let a platform team agree on when a release is safe.
An SLO is a product and engineering agreement
Do not adopt a percentile, availability number, or window because another organization publishes it. Select an objective from the user journey, the harm caused by failure or delay, historical behavior, and the team's ability to act on the result.
Overview
Outcome and prerequisites
Outcome: Acme Shop can measure checkout availability and latency, make a release decision from a 30-day error budget, and page only for sustained impact using multi-window, multi-burn-rate SRE alerting. Prerequisites: a non-production namespace, a route-level request counter and duration histogram, Prometheus rule deployment access, and agreement on valid, good, and fast-enough outcomes.
Running scenario: Acme Shop checkout
Acme Shop operates checkout-api behind an ingress. A valid checkout request is one that reaches the route and receives a 2xx, 3xx, or 5xx response; malformed or intentionally rejected 4xx requests are tracked separately and are not silently reclassified after an incident. A good valid event is a 2xx or 3xx response. The first objective is 99.9% availability over 30 days. The second objective covers latency for successful requests: 95% of good checkout requests complete within 300 ms, and 99% complete within 900 ms, both measured at the ingress rather than inside the application.
- Shopper and edge
The edge adds trace context and records delivery outcome.
- Ingress
Route-level status and duration identify requests that reach checkout.
- checkout-api
Application metrics and traces connect the result to release and dependency evidence.
- Prometheus rules
Recording rules calculate valid, good, latency, and multi-window error-budget burn rates.
- SLO decision
The dashboard, burn-rate alerts, and release gate drive a scoped response, not a single health check.
Figure 1. Availability and latency are measured from the checkout journey boundary, then correlated with ingress, application, and dependency evidence.
Define Kubernetes SLOs: indicators and error budgets
Good Kubernetes SLOs start from a ratio the whole team can audit, not from whichever percentile a dashboard happens to plot by default. For availability, use the correct direction of the ratio:
availability = good valid events / total valid events
For 1,000,000 valid checkout requests at a 99.9% target, Acme Shop may have 1,000 bad events in the 30-day window. If 250 requests are bad, availability is 999,750 / 1,000,000 = 99.975% and 250 / 1,000 = 25% of the error budget is spent. A 1% bad-event rate is ten times the allowed 0.1% bad-event rate, so it is a 10x burn rate. This calculation is meaningful only while the valid-event definition remains stable and traffic is sufficient to interpret it.
Use a threshold-based good-event ratio for the availability SLI. For latency, prefer several percentile thresholds over one average: a mean hides the slow tail that drives complaints, and a single high percentile can look fine while ordinary requests quietly regress. Acme Shop treats a request as good for the tight objective within 300 ms, and separately tracks a looser 900 ms bound at the 99th percentile, so a tail-only regression surfaces before it reaches the 95th percentile too. CPU, restart count, and pod readiness are diagnostic signals, not substitutes for either indicator.
A rolling 30-day window ages incidents out gradually and suits release decisions; a calendar-quarter window reports more cleanly to leadership. Acme Shop uses both, fed by the same recording rules.
Record the SLI before alerting
The following Prometheus rule assumes http_requests_total is a monotonically increasing counter with bounded service, route, and code labels, and that a http_request_duration_seconds_bucket histogram is available for the same route. It deliberately excludes 4xx responses from the availability population. Validate route, status, and bucket-boundary semantics against the application before applying the rule through the normal review process.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: acme-shop-checkout-slo
namespace: acme-shop-sandbox
spec:
groups:
- name: acme-shop.checkout-slo
rules:
- record: acme_shop:checkout_valid_requests:rate5m
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3..|5.."}[5m]))
- record: acme_shop:checkout_good_requests:rate5m
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3.."}[5m]))
- record: acme_shop:checkout_valid_requests:rate1h
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3..|5.."}[1h]))
- record: acme_shop:checkout_good_requests:rate1h
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3.."}[1h]))
- record: acme_shop:checkout_valid_requests:rate6h
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3..|5.."}[6h]))
- record: acme_shop:checkout_good_requests:rate6h
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3.."}[6h]))
- record: acme_shop:checkout_valid_requests:rate3d
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3..|5.."}[3d]))
- record: acme_shop:checkout_good_requests:rate3d
expr: sum(rate(http_requests_total{service="checkout-api",route="/checkout",code=~"2..|3.."}[3d]))
- record: acme_shop:checkout_good_fast_requests:rate5m
expr: sum(rate(http_request_duration_seconds_bucket{service="checkout-api",route="/checkout",code=~"2..|3..",le="0.3"}[5m]))
- record: acme_shop:checkout_good_bounded_requests:rate5m
expr: sum(rate(http_request_duration_seconds_bucket{service="checkout-api",route="/checkout",code=~"2..|3..",le="0.9"}[5m]))
- alert: AcmeShopCheckoutFastBurn
expr: >-
(1 - (acme_shop:checkout_good_requests:rate1h / acme_shop:checkout_valid_requests:rate1h)) / 0.001 > 14.4
and (1 - (acme_shop:checkout_good_requests:rate5m / acme_shop:checkout_valid_requests:rate5m)) / 0.001 > 14.4
and acme_shop:checkout_valid_requests:rate1h > 1
for: 2m
labels:
severity: page
annotations:
summary: "Acme Shop checkout is burning its 99.9% availability budget at 14.4x"
runbook: "https://runbooks.example.invalid/acme-shop/checkout-slo"
- alert: AcmeShopCheckoutBudgetTicket
expr: >-
(1 - (acme_shop:checkout_good_requests:rate3d / acme_shop:checkout_valid_requests:rate3d)) / 0.001 > 1
and (1 - (acme_shop:checkout_good_requests:rate6h / acme_shop:checkout_valid_requests:rate6h)) / 0.001 > 1
and acme_shop:checkout_valid_requests:rate3d > 1
for: 1h
labels:
severity: ticket
annotations:
summary: "Acme Shop checkout has burned 10% of its 30-day availability budget"
runbook: "https://runbooks.example.invalid/acme-shop/checkout-slo"
Each recording rule is scoped to one window so alert expressions stay short; the two latency rules reuse the duration histogram at the 300 ms and 900 ms bucket boundaries. AcmeShopCheckoutFastBurn and AcmeShopCheckoutBudgetTicket implement the page and ticket tiers below; the intermediate slow-burn tier follows the same pattern with a rate30m recording rule.
window=30d valid_requests=1000000 good_requests=999750
availability=99.975% target=99.900% allowed_bad=1000 actual_bad=250
budget_spent=25% current_error_rate=0.025%
burn_5m=0.25x burn_30m=0.28x burn_1h=0.30x burn_6h=0.40x burn_3d=0.50x
latency_p95_target=300ms latency_p95_actual=240ms latency_p99_target=900ms latency_p99_actual=610ms
release_decision=continue_with_normal_reviewSRE alerting: multi-window, multi-burn-rate rules
A single fast-burn condition over one short window is not enough for production SRE alerting: a brief spike can cross a high burn-rate threshold without threatening the budget, and a slow leak can stay under a short-window threshold while consuming most of it. Google's SRE workbook fixes this by requiring two windows to agree — a long window that proves the burn is real, a short one that proves it is still happening.
| Alert tier | Long window / short window | Burn rate | Budget consumed | Acme Shop response |
|---|---|---|---|---|
| Page (fast) | 1h / 5m | 14.4x | ~2% per hour | Page on-call immediately; treat as an active incident. |
| Page (slow) | 6h / 30m | 6x | ~5% over six hours | Page on-call; diagnose before the trend compounds. |
| Ticket | 3d / 6h | 1x | ~10% over three days | File a ticket; resolve before the next scheduled release. |
Because an alert only stays firing while both windows are bad, it clears within minutes of a real recovery instead of staying lit for the rest of the long window. Pair every burn-rate condition with a minimum-valid-request guard, or a low-traffic route can cross a threshold on a handful of requests with no statistical weight; a newly launched route requires at least 1,000 valid requests in its shortest window before it can page, and falls back to a ticket until traffic clears that floor. The same two-window logic applies to the latency SLI — checkout can be fully available and still fail its latency objective.
Two windows must agree before escalation, reducing alert noise while retaining distinct responses for rapid, slow, and budget-consuming reliability loss.
One error budget, watched from one place
When checkout depends on several delivery and security providers sitting in front of the same origin, a burn-rate alert is only actionable if someone is correlating it against provider-side events in real time. This is the layer MYO is built for: it lines up Kubernetes-side SLO burn with edge, DNS, and security provider signals so a page tells the team where to look first, rather than only that the budget is burning.
Operate the error budget
Maintain a long compliance window for the release decision and shorter burn windows for detection, and write the error-budget policy down rather than relying on shared memory mid-incident. Acme Shop's policy: below 50% consumed, releases proceed under normal review; between 50% and 90%, a release needs a second reviewer and a rollback plan; above 90%, only reliability fixes and pre-approved security patches ship until the budget resets or the leads sign off on an exception. The policy names roles, not individuals, and lives next to the runbook.
Instrument the complete request path with bounded metrics, trace context, logs, and Kubernetes events. Segment dashboards by route, region, release, and cache state where those dimensions change the outcome. A green node dashboard is not evidence checkout recovered, and a green availability panel is not evidence latency recovered — review both SLIs, and any dependency SLOs the journey composes, before closing an incident.
Validation
Validation, rollback, and failure behavior
In a sandbox, send a known number of synthetic checkout requests that return both 2xx and controlled 5xx outcomes, plus a batch with injected delay past the 300 ms and 900 ms latency thresholds. Confirm the valid, good, and latency recording rules match the expected counts, then inspect a trace for each cohort, and drive a short outage that recovers within minutes to confirm the fast-burn alert clears once the short window does, not merely once it is present in the YAML. If a rule labels valid traffic incorrectly, remove only the unapproved PrometheusRule or restore its last reviewed revision; do not change status handling or suppress an alert globally. If the metric pipeline is unavailable or request volume is too low, mark the SLO measurement unavailable and repair it before relying on the budget for a release decision.
Troubleshooting
Troubleshooting
| Symptom | Likely cause | Safe check | Recovery |
|---|---|---|---|
| Availability is above 100% or missing | Good and valid queries use different label filters or a zero denominator | Query both recording rules with identical route and service scope | Correct the reviewed rule and wait for fresh samples; do not clamp the result to hide a query defect. |
| Budget falls after a client validation change | 4xx responses entered the valid population unexpectedly | Compare status-class counts by release version | Restore the prior application or metric-label behavior, then document whether the SLI contract should change. |
| Fast-burn alert fires during tiny traffic | No minimum-volume guard, or the guard is a guess rather than real traffic | Inspect valid requests for the window and compare against the guard | Recalibrate the request-volume floor from real traffic; route low-volume routes to a ticket until traffic supports paging. |
| SLO is green but customers report slowness | Availability is healthy while the latency SLI is absent, too broad, or measuring only one percentile | Compare route-level duration histogram and trace cohorts across both latency thresholds | Add or correct the latency SLI at more than one percentile; do not redefine availability to conceal latency. |
| Dashboard cannot identify the affected region | Region or release labels are absent from correlated telemetry | Inspect one sanitized trace, log, and metric series | Restore the telemetry contract and rerun the controlled validation. |
Related guides
- Kubernetes Observability
- Kubernetes Autoscaling
- Distributed Systems Latency Budgets
- Dependency Resilience
Authoritative references
- Google SRE: Service Level Objectives
- Google SRE: Alerting on SLOs
- Kubernetes: Metrics for Kubernetes System Components
- Prometheus Operator: API Reference
- OpenTelemetry Documentation
Extend your Kubernetes SLOs to the edge
Talk to Optimi about correlating error budgets and burn-rate alerts across your CDN, DNS, and security providers, so MYO gives Performance, Security, and Visibility one shared view of the objectives that matter to customers.
Discuss reliability and visibility