Kubernetes SRE guide

Kubernetes Capacity Planning for Reliable Growth

Forecast the full request path, validate the real bottleneck, and reserve enough time and headroom for Kubernetes to respond safely.

Published: Jul 14, 2026
Updated: Jul 15, 2026
Reading time: 10 min read

On this page

Capacity planning for Kubernetes is an operating discipline, not a one-time estimate of node count. It connects expected demand to the resources and dependencies that complete useful work: application replicas, scheduler capacity, nodes, network addresses, storage, databases, queues, identity systems, third-party APIs, and the delivery path in front of the cluster.

For a business running one intelligent entry point in front of an origin, this discipline compounds rather than disappears: an orchestrated edge can absorb a spike, but it cannot manufacture database connections or ready pods the origin never planned for — it just hands the origin's real limit back faster. Treat the cluster's plan and the edge's routing and caching plan as one measured system, not two teams guessing about each other's headroom.

Capacity is a service property, not a CPU percentage

A cluster can show spare CPU while a database connection limit, network address pool, ingress controller, or external API quota is the actual limit. Plan from the customer transaction backward.

Overview

Outcome and prerequisites

Outcome: Acme Shop can approve a seasonal catalog campaign with enough origin and dependency capacity to absorb a measured burst, a node-pool loss, and a bounded regional traffic shift. Prerequisites: route-level demand history, representative load-test data, current requests and limits, node allocatable capacity, dependency quotas, and owners who can approve traffic and procurement decisions.

Running scenario: Acme Shop seasonal catalog campaign

Acme Shop forecasts 600 catalog requests per second at normal campaign peak and a 900 requests-per-second, 15-minute burst after an email launch. A tested catalog-api pod safely serves 75 requests per second at the target p95 latency. The cluster needs 12 serving replicas for the burst (900 / 75 = 12). Acme plans 15 replicas with 25% workload headroom (12 x 1.25 = 15) before accounting for a node failure. Each pod requests 500m CPU and 512Mi memory, so those 15 replicas request 7.5 CPU and 7.5Gi memory. The database, ingress, IP allocation, image registry, and cache-miss path must independently support the same scenario.

Acme Shop capacity model from shopper to dependency

Demand forecast
Separate public catalog reads, authenticated traffic, writes, and background work.
Edge and ingress
Measure cache-hit behavior, connection capacity, and the miss surge after invalidation.
Kubernetes workload
Translate measured per-pod throughput into replicas, requests, startup time, and node space.
Shared dependencies
Validate database connections, queues, storage, identity, and third-party quotas.
Failure role
Recalculate for node loss or regional traffic transfer before calling capacity sufficient.

Figure 1. A capacity plan follows the customer transaction through every shared limit rather than stopping at node CPU.

Kubernetes capacity planning: convert demand into resource commitments

Kubernetes capacity planning starts with production-like tests that identify useful throughput, not only maximum synthetic throughput. Record request mix, payload size, cache state, regional distribution, concurrency, and dependency behavior. Daily averages hide bursts and recovery surges. Include a regional evacuation, node-pool loss, or dependency outage, because retries and queue accumulation can make recovery demand larger than the original peak.

The ResourceQuota below is an intentional sandbox ceiling, not a production recommendation. It makes planned capacity visible and prevents an unrelated test from consuming namespace resources. Because the namespace enforces quota, every pod must declare requests — pair it with a LimitRange default, or pod creation is rejected outright.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: catalog-campaign-guardrail
  namespace: acme-shop-sandbox
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 10Gi
    limits.memory: 12Gi
    pods: "20"

Autoscaling changes the shape of the plan; it does not remove it. Measure time-to-capacity end to end: detection, autoscaler decision, node provisioning, pod scheduling, image pull, initialization, readiness, and traffic admission. Compare this time with the fastest credible demand increase. If it is too slow, maintain measured warm capacity, reduce initialization work, schedule predictable batch work away from the peak, or use a product-approved admission policy. Do not assume a cache ratio or geographic routing rule survives a release, purge, or regional incident.

Which node autoscaler is in play changes that lead time: Cluster Autoscaler only adds nodes from existing node groups, while Karpenter can provision new shapes on demand — both still take real minutes, and neither replaces a warm-capacity buffer if demand outruns it. Workload autoscaling (HPA/VPA) is its own topic, covered in Kubernetes Autoscaling.

Representative output: Acme Shop campaign capacity check

forecast_peak=900rps tested_pod_throughput=75rps replicas_required=12
workload_headroom=25% planned_replicas=15
pod_request_cpu=500m pod_request_memory=512Mi
planned_requests_cpu=7.5 planned_requests_memory=7.5Gi
node_loss_simulation=1 node remaining_allocatable_cpu=9.0 result=PASS
database_connections=60/100 ingress_active_connections=1840/5000 decision=approve_sandbox_ramp

Resource requests and headroom: what requests actually buy you

Requests and headroom are two different levers. A request is what the scheduler reserves; it decides which node a pod fits on. A limit is what the kubelet enforces once running, and CPU and memory limits fail differently: CPU throttles under the kernel's CFS quota, so an overloaded container slows down, while memory is enforced by OOM kill, so it is terminated. That asymmetry is why Acme keeps catalog-api's memory request and limit close together, but leaves more room on CPU — a slow response recovers, a killed pod does not.

Headroom is the separate, deliberate gap between tested-necessary and provisioned capacity — the 25% in Acme's model is a risk decision, not a Kubernetes setting. Under-sized requests make headroom meaningless, since the scheduler cannot pack the planned replicas onto available nodes; over-sized requests waste the capacity the headroom calculation assumes is free.

Two mechanisms change how this plays out. In-place Pod resize (stable in Kubernetes v1.35) lets the kubelet apply a new request or limit without recreating the pod — but it still cannot exceed node allocatable, and some memory decreases still force a restart. Vertical Pod Autoscaler output is a useful input for setting the request baseline from measured p95/p99 usage, but Acme treats it as a proposal to review, not an authority to auto-apply mid-campaign.

Cluster load testing: prove the model under stress, including degradation

Cluster load testing turns the arithmetic above into a validated plan. Increase representative load gradually, then introduce one bounded burst. Record service indicators, saturation, scaling events, pending-pod reasons, node behavior, cache-hit ratio, and dependency response. Stop before the test risks uncontrolled customer impact. Then test a controlled node-pool loss, slowed database read path, throttled external API, exhausted connection pool, delayed image registry, or telemetry-export failure. A healthy HTTP listener is not enough if a database, identity provider, or queue cannot complete the intended transaction.

A single burst is not sufficient cluster load testing on its own. Run three shapes and treat them as different questions:

Ramp — ease from zero to forecast peak over several minutes; confirms the 75rps per-pod figure still holds after a change.
Spike — jump straight to the 900rps burst with no ramp, the shape a real email launch produces; this stresses autoscaler decision-and-provisioning lead time, not steady-state throughput.
Soak — hold peak well past any cache TTL, connection-pool recycle, or certificate-rotation window, where slow memory growth and connection leaks surface that a 15-minute burst won't catch.

The tool (k6, Locust, or similar) matters less than the request mix: mirror the real ratio of cache-hit reads, cache-miss reads, authenticated calls, and writes, from a network profile close to real users, so generator-side saturation isn't mistaken for a cluster limit. For internet-facing services, measure client-visible latency, edge response, and origin saturation together — a public cache reduces steady origin load, but cold-cache and purge tests must prove the origin can handle the miss load.

Ramp, spike, and soak tests expose different capacity limits

A capacity plan needs all three test shapes: steady demand, sudden demand, and time-dependent failure modes each reveal different constraints.

Download:PNG SVG

One observability plane across the edge and the cluster

A load test is only as trustworthy as the signals behind it. Acme runs its cluster load testing with MYO watching the same window end to end — edge cache-hit ratio and ingress connections alongside pod readiness and node headroom — so a saturation point is attributed to a layer, not argued about afterward. Optimi doesn't replace the load-testing tool or the cluster's own metrics; it gives both teams one shared view of the same burst.

Validation

Validation, rollback, and failure behavior

In a non-production environment, begin with baseline demand, ramp to 600 requests per second, then run the approved 900 requests-per-second burst for a bounded interval. Validate p95 latency, error rate, ready replicas, pending-pod reasons, node allocatable headroom, database connections, and cache misses. Repeat the same evidence collection while one test node is unavailable. If a guardrail is breached, stop the load generator, return only the tested traffic policy and workload revision to the last reviewed state, and allow queues and caches to settle before analyzing results. Do not remove quotas, globally raise timeouts, or assume a failed scale-out will recover by adding more traffic.

Be precise about what node-loss simulation tests. A kubectl drain respects Pod Disruption Budgets and pauses rather than violating minAvailable — useful for rehearsing maintenance, but not a real failure. An involuntary loss (hardware fault, spot reclaim) never consults the PDB; the pods are simply gone, and replacements must schedule and become ready before capacity recovers.

Troubleshooting

Symptom	Likely cause	Safe check	Recovery
Nodes have spare CPU but requests queue	Database pool, ingress, IP allocation, or external quota is constrained	Compare journey latency with each shared saturation signal	Protect the constrained dependency and revise the model; do not add nodes blindly.
HPA requests replicas that never become ready	Startup, image pull, scheduling, or node provisioning exceeds demand ramp	Measure every interval from scale decision to readiness	Add measured headroom or reduce startup work before changing the forecast.
Campaign cache miss rate collapses	Purge, cache-key change, or routing shift bypasses the expected cache	Compare cache status and origin concurrency against baseline	Pause the change, restore the reviewed cache policy, and warm only safe public content.
Node-pool loss exhausts capacity	Plan counted all nodes as serving capacity	Simulate loss in the approved environment and inspect allocatable resources	Reserve failure headroom or reduce admissible load before launch.
Forecast is met but checkout fails	Read-heavy catalog forecast omitted write-path or identity limits	Separate transactions and inspect dependency quotas by journey	Add the omitted workload and dependency constraint to the next capacity review.
Pods restart repeatedly under load, no scheduling errors	Memory limit is undersized and the kernel is OOM-killing the container	Compare per-pod memory usage against request and limit	Raise memory request and limit together and re-run the ramp test.

Authoritative references

Bring Kubernetes capacity planning into the same view as the edge

Talk to Optimi about pairing your cluster's resource requests and headroom, and its cluster load testing results, with the Performance, Security, and Visibility of the delivery path in front of it — measured together in MYO ahead of the next campaign.

Review delivery and origin capacity