Kubernetes SRE guide

Kubernetes Observability: Metrics, Logs, Traces

Design telemetry around the questions operators must answer: who is affected, where the request slowed or failed, and what changed.

Published: Jul 14, 2026
Updated: Jul 15, 2026
Reading time: 9 min read

On this page

Kubernetes observability is not a collection of dashboards for every resource. It is the ability to explain service behavior across the client, delivery path, ingress, workload, cluster, and dependencies. A pod restart, high CPU, or failed readiness probe is useful evidence, but customers experience a completed request, delayed page, failed API call, or missing asynchronous result.

That chain matters most at the scale a managed edge-orchestration layer operates. Optimi sits in front of origins across several delivery providers, where an unexplained cluster signal is a gap in the evidence, not a curiosity. Kubernetes observability earns its name once cluster telemetry joins, request by request, to what happened at the edge.

Telemetry without correlation creates slower incidents

Standardize service identity, environment, region, and release version first. Instrumentation added without stable identifiers often produces more data but less evidence. MYO applies the same discipline across delivery-path signals, so a cluster trace and an edge log resolve to one incident timeline.

Overview

Outcome and prerequisites

Outcome: Acme Shop can trace a slow checkout from edge to dependency, query its related logs without exposing customer data, and distinguish a release regression from a cluster constraint. Prerequisites: a sandbox cluster, an OpenTelemetry Collector endpoint, Prometheus-compatible metrics storage, structured log access, and approved data-classification rules.

Kubernetes observability in practice: the Acme Shop checkout scenario

Acme Shop receives a report that EU checkout is slow after a canary release. The team needs one route-level view that connects the edge outcome, ingress duration, checkout-api release, database-pool wait, and sanitized trace ID. It does not place customer IDs, email addresses, authorization headers, full URLs, or request bodies in metric labels or trace attributes.

Acme Shop correlated checkout telemetry

Edge and ingress
Record route, region, cache status, response class, and W3C trace context.
checkout-api
Emit bounded HTTP metrics, structured logs, and child spans with release version.
Dependencies
Trace database-pool wait and payment calls without recording credentials or payloads.
Collector and stores
Apply resource limits, authentication, sampling, and retention policy.
Incident view
Use a dashboard, trace, logs, and deployment event to establish scope.

Figure 1. Stable identifiers join delivery, application, and dependency evidence without turning personal or secret data into telemetry.

Publish a small telemetry contract

Every Acme Shop signal includes service.name, service.namespace, deployment.environment.name, cloud.region where applicable, and release version. HTTP metrics use a route template and response class, not raw paths. Logs contain timestamp, level, service identity, release version, trace ID, and event-specific outcome. Sampling is explicit: slow and failed traces are retained according to a documented policy, and a collector outage has a defined loss policy.

This Deployment annotation is a narrow sandbox example for a Prometheus scraper. It exposes only a metrics port; actual scrape authorization, network policy, and service discovery remain platform responsibilities. The application must emit bounded labels such as route and code.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: acme-shop-sandbox
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
        app.kubernetes.io/version: "2026.07.14-canary"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9464"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: checkout-api
          image: registry.example.invalid/acme/checkout-api:2026.07.14-canary
          ports:
            - name: metrics
              containerPort: 9464
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi

Stable identifiers work best when the platform attaches them, rather than trusting every service to set them correctly. OpenTelemetry's Kubernetes semantic conventions define resource attributes such as k8s.namespace.name, k8s.pod.name, k8s.pod.uid, k8s.deployment.name, and k8s.node.name for this purpose. The Collector's k8sattributes processor resolves the pod behind a signal — by default by matching the connection's source IP to a cached pod IP — and attaches those attributes automatically, so checkout-api never needs to know its own node or deployment identity.

processors:
  k8sattributes:
    auth_type: serviceAccount
    pod_association:
      - sources:
          - from: connection
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.node.name

This default is a common failure mode: it assumes the Collector sees the pod's real source IP, which a mesh sidecar can hide. Production collectors pin association to k8s.pod.uid instead.

Sampling policy is a tradeoff, not a toggle. Head-based sampling decides before it knows whether a trace was slow or failed, so it can discard the request that mattered. Tail-based sampling defers that decision, buffering every span in the tail_sampling processor until the trace completes, then keeping or dropping it against a latency or error policy — at the cost of memory held open until each decision closes, so a concurrency spike can exhaust the Collector before the network does. Acme Shop bounds this with a memory_limiter and fails open to head sampling rather than letting the Collector fall over.

Add cluster monitoring beneath correlated metrics, logs, and traces

Acme Shop's checkout trace explains one request. Cluster monitoring explains what the platform was doing underneath it, and Kubernetes ships three narrower tools for that job.

metrics-server aggregates CPU and memory from each kubelet's /metrics/resource endpoint to feed the HorizontalPodAutoscaler and kubectl top. It holds no history: unwatched during the canary, that evidence is gone.
kube-state-metrics watches the API server, not the kubelets, turning object state — replica counts, node conditions, restart counts — into metrics that confirm whether the rollout itself behaved.
cAdvisor, exposed by the kubelet at /metrics/cadvisor, reports per-container CPU, memory, and network counters. Scraped into Prometheus, it becomes the historical timeline lined up against trace timestamps; unscraped, it is as transient as metrics-server.

None replaces the correlated telemetry above — they answer "was the platform healthy," where the trace answers "was this request healthy." Acme Shop keeps kube-state-metrics labels narrow: every optional label multiplies cardinality like an unbounded HTTP label.

Ask service questions, not resource-only questions

Dashboards should answer whether valid requests succeed and meet their objective; which route, region, release, or cache state changed; where time was spent; and whether replicas, scheduling, node resources, connection pools, or an upstream quota is constraining work. Kubernetes events explain image pulls, scheduling, eviction, and autoscaler actions, but their retention varies, so export the events required for operations and test that path under stress.

Occasionally the constraint is the control plane itself — a slow admission webhook, scheduler contention, or etcd latency — and pod-level dashboards stay quiet while requests queue at admission. Since v1.22, the kube-apiserver can export its own spans over OTLP behind the alpha APIServerTracing feature gate. Names are not guaranteed stable while alpha, so Acme Shop leaves it off by default and enables it deliberately when a control-plane cause cannot be confirmed any other way.

For public traffic, add synthetic critical journeys from selected locations and real-user measurements where privacy and consent permit. A fast cached asset does not prove an authenticated route is healthy. Segment by route, region, client type where material, origin, and delivery configuration rather than relying on a global latency average.

Representative output: correlated checkout trace

trace_id=4bf92f7a route=POST /checkout region=eu-west release=2026.07.14-canary status=503 total=812ms
edge.cache_status=BYPASS ingress.upstream_time=784ms
checkout-api.db_pool_wait=421ms db.query=96ms payment.attempt=0
log.level=warn event=checkout_deadline_exceeded correlation_id=req-7e1f
diagnosis=database_pool_wait increased in the canary cohort; no customer or credential fields recorded

Validation

Validation, rollback, and failure behavior

In the sandbox, establish a baseline from a disposable checkout test, then inject a bounded database-pool delay into the test dependency. Confirm the trace retains the slow request, the log can be found by trace ID, the dashboard isolates the sandbox and release, and no secret or personal field appears in any signal. Confirm too that k8s.* attributes match the serving pod, and that tail sampling kept the delayed trace. Remove the injected delay and verify latency, collector queue, and sample volume return to baseline. If a collector configuration or canary adds excessive cardinality, stop the canary, restore the last reviewed collector or application revision, and retain only sanitized evidence. If telemetry is unavailable during an incident, declare that gap, use service and dependency health signals, and repair the pipeline rather than disabling data protections.

Troubleshooting

Symptom	Likely cause	Safe check	Recovery
Metrics storage slows during load	Unbounded label such as raw URL or request ID	Inspect highest-cardinality series in the sandbox	Remove the label from metrics and keep detail in sampled logs or traces.
Trace and log cannot be joined	Trace context is not propagated through ingress or worker	Send one sandbox request and compare headers and trace IDs	Restore W3C propagation and test the affected hop before promotion.
Collector drops traces unexpectedly	Queue, memory limit, or exporter throughput is insufficient	Review collector accepted, refused, and queue metrics	Apply the reviewed loss policy or increase tested capacity; do not remove limits blindly.
Dashboard says healthy while EU users fail	Global aggregation hides the affected route or region	Filter synthetic and request metrics by route and region	Add the missing segment and alert on the customer journey.
Logs expose sensitive fields	Middleware logs headers, payloads, or identifiers	Review one sanitized sandbox record and logging configuration	Disable or redact the field, rotate exposed credentials if any, and follow the security process.
metrics-server and kube-state-metrics show healthy after the fact	Neither tool retains history	Check whether scraped cAdvisor metrics cover the window	Treat both as live-state only; use the scraped historical layer for postmortems.
Spans and logs are missing `k8s.*` attributes	Pod association does not match behind a mesh sidecar	Compare the Collector's source IP with the pod IP	Switch association to a value the SDK sets, such as `k8s.pod.uid`.

Authoritative references

Correlate cluster telemetry with the whole delivery path

Talk to Optimi about joining Kubernetes metrics, logs, and traces with edge signals in MYO, so Performance, Security, and Visibility incidents share one timeline.

Review delivery observability