Kubernetes SRE guide
Kubernetes Capacity Planning for Reliable Growth
Forecast the full request path, validate the real bottleneck, and reserve enough time and headroom for Kubernetes to respond safely.
On this page
Capacity planning for Kubernetes is an operating discipline, not a one-time estimate of node count. It connects expected demand to the resources and dependencies that complete useful work: application replicas, scheduler capacity, nodes, network addresses, storage, databases, queues, identity systems, third-party APIs, and the delivery path in front of the cluster.
Capacity is a service property, not a CPU percentage
A cluster can show spare CPU while a database connection limit, network address pool, ingress controller, or external API quota is the actual limit. Plan from the customer transaction backward.
Overview
Outcome and prerequisites
Outcome: Acme Shop can approve a seasonal catalog campaign with enough origin and dependency capacity to absorb a measured burst, a node-pool loss, and a bounded regional traffic shift. Prerequisites: route-level demand history, representative load-test data, current requests and limits, node allocatable capacity, dependency quotas, and owners who can approve traffic and procurement decisions.
Running scenario: Acme Shop seasonal catalog campaign
Acme Shop forecasts 600 catalog requests per second at normal campaign peak and a 900 requests-per-second, 15-minute burst after an email launch. A tested catalog-api pod safely serves 75 requests per second at the target p95 latency. The cluster needs 12 serving replicas for the burst (900 / 75 = 12). Acme plans 15 replicas with 25% workload headroom (12 x 1.25 = 15) before accounting for a node failure. Each pod requests 500m CPU and 512Mi memory, so those 15 replicas request 7.5 CPU and 7.5Gi memory. The database, ingress, IP allocation, image registry, and cache-miss path must independently support the same scenario.
- Demand forecast
Separate public catalog reads, authenticated traffic, writes, and background work.
- Edge and ingress
Measure cache-hit behavior, connection capacity, and the miss surge after invalidation.
- Kubernetes workload
Translate measured per-pod throughput into replicas, requests, startup time, and node space.
- Shared dependencies
Validate database connections, queues, storage, identity, and third-party quotas.
- Failure role
Recalculate for node loss or regional traffic transfer before calling capacity sufficient.
Figure 1. A capacity plan follows the customer transaction through every shared limit rather than stopping at node CPU.
Convert demand into resource commitments
Use production-like tests to identify useful throughput, not only maximum synthetic throughput. Record request mix, payload size, authentication, cache state, regional distribution, concurrency, and dependency behavior. Daily averages hide bursts and recovery surges. Include a regional evacuation, node-pool loss, deployment rollback, or dependency outage because retries and queue accumulation can make recovery demand larger than the original peak.
The ResourceQuota below is an intentional sandbox ceiling, not a production recommendation. It makes the planned capacity visible and prevents an unrelated test from consuming all namespace resources. Its values must leave room for platform overhead and other workloads when translated to node capacity.
apiVersion: v1
kind: ResourceQuota
metadata:
name: catalog-campaign-guardrail
namespace: acme-shop-sandbox
spec:
hard:
requests.cpu: "10"
requests.memory: 10Gi
limits.memory: 12Gi
pods: "20"
Autoscaling changes the shape of the plan; it does not remove it. Measure time-to-capacity end to end: detection, autoscaler decision, node provisioning, pod scheduling, image pull, initialization, readiness, and traffic admission. Compare this time with the fastest credible demand increase. If it is too slow, maintain measured warm capacity, reduce initialization work, schedule predictable batch work away from the peak, or use a product-approved admission policy. Do not assume a cache ratio or geographic routing rule survives a release, purge, or regional incident.
forecast_peak=900rps tested_pod_throughput=75rps replicas_required=12
workload_headroom=25% planned_replicas=15
pod_request_cpu=500m pod_request_memory=512Mi
planned_requests_cpu=7.5 planned_requests_memory=7.5Gi
node_loss_simulation=1 node remaining_allocatable_cpu=9.0 result=PASS
database_connections=60/100 ingress_active_connections=1840/5000 decision=approve_sandbox_rampTest the model, including degradation
Increase representative load gradually, then introduce one bounded burst. Record service indicators, saturation, scaling events, pending-pod reasons, node behavior, cache-hit ratio, and dependency response. Stop before the test risks uncontrolled customer impact. Then test a controlled node-pool loss, slowed database read path, throttled external API, exhausted connection pool, delayed image registry, or telemetry-export failure. A healthy HTTP listener is not enough if a database, identity provider, or queue cannot complete the intended transaction.
For internet-facing services, measure client-visible latency, edge response, ingress time, origin saturation, and dependency behavior together. A public cache can reduce steady origin load, but cold-cache, purge, and failover tests must prove the origin and receiving region can handle the miss load. Private and write paths retain their own consistency, authorization, and capacity requirements.
Validation
Validation, rollback, and failure behavior
In a non-production environment, begin with baseline demand, ramp to 600 requests per second, then run the approved 900 requests-per-second burst for a bounded interval. Validate p95 latency, error rate, ready replicas, pending-pod reasons, node allocatable headroom, database connections, and cache misses. Repeat the same evidence collection while one test node is unavailable. If a guardrail is breached, stop the load generator, return only the tested traffic policy and workload revision to the last reviewed state, and allow queues and caches to settle before analyzing results. Do not remove quotas, globally raise timeouts, or assume a failed scale-out will recover by adding more traffic.
Troubleshooting
Troubleshooting
| Symptom | Likely cause | Safe check | Recovery |
|---|---|---|---|
| Nodes have spare CPU but requests queue | Database pool, ingress, IP allocation, or external quota is constrained | Compare journey latency with each shared saturation signal | Protect the constrained dependency and revise the model; do not add nodes blindly. |
| HPA requests replicas that never become ready | Startup, image pull, scheduling, or node provisioning exceeds demand ramp | Measure every interval from scale decision to readiness | Add measured headroom or reduce startup work before changing the forecast. |
| Campaign cache miss rate collapses | Purge, cache-key change, or routing shift bypasses the expected cache | Compare cache status and origin concurrency against baseline | Pause the change, restore the reviewed cache policy, and warm only safe public content. |
| Node-pool loss exhausts capacity | Plan counted all nodes as serving capacity | Simulate loss in the approved environment and inspect allocatable resources | Reserve failure headroom or reduce admissible load before launch. |
| Forecast is met but checkout fails | Read-heavy catalog forecast omitted write-path or identity limits | Separate transactions and inspect dependency quotas by journey | Add the omitted workload and dependency constraint to the next capacity review. |
Related guides
Authoritative references
- Kubernetes: Resource Management for Pods and Containers
- Kubernetes: Node Autoscaling
- Google SRE: Capacity Planning
- Kubernetes: Resource Quotas
Plan the path between users and Kubernetes
Talk to Optimi about edge and delivery performance architecture that can be evaluated alongside origin capacity, resilience, and regional demand.
Review delivery capacity