Architecture Bottleneck Lessons

Server CPU Saturation

Problem

Two API servers, each provisioned for 2,000 QPS, receive 2,500 QPS apiece from a load balancer handling 5,000 QPS of total traffic. Neither server has any headroom — both run at 125% at the same time. Each completes 2,000 requests per second and sheds the other 500, while its request queue grows without bound and latency spikes across the entire user base.

Symptoms in Production

• CPU pegged at 95–100% across all instances simultaneously
• Latency climbs from ~50 ms without limit as the M/M/1 queue grows (the simulator shows ∞ once a server reaches 100%)
• Throughput plateaus even as traffic continues to climb
• Load balancer health checks start failing, causing instance flapping
• HTTP 5xx error rates rise as servers begin rejecting or timing out requests

Root Cause

Each server handles 2,500 QPS against a ceiling of 2,000 — 125% utilization. It can complete only 2,000 per second, so 500 per second (20%) are shed; together with the configured 2% errors, 21.6% of requests fail. Under M/M/1 queueing theory, utilization at or above 100% means the queue never drains, so latency is unbounded. With both servers saturated and no autoscaling, there is no path to recovery without intervention.

Mitigations

• Horizontal scale: add server instances until aggregate capacity exceeds peak QPS by ≥ 30% — set Instances to 2 and each server drops to 62.5%
• Autoscaling: configure CPU-based HPA in Kubernetes to scale out before utilization hits 80%
• Request shedding: return HTTP 429 at ~75% utilization to signal backpressure upstream
• Async offloading: push expensive synchronous work into a background queue so the hot path stays lean
• Profiling: identify hot code paths — often a single expensive operation (N+1 query, JSON serialization) drives most CPU usage

Database Write Bottleneck

Problem

An application with an 80% write workload routes 2,000 QPS to a PostgreSQL instance configured with a maxWriteQPS of 400. The read path has comfortable headroom, but the write path faces 1,600 write requests per second against a ceiling of 400 — a 4× overload that causes transaction timeouts and connection pool exhaustion.

Symptoms in Production

• Write latency spikes dramatically while read latency stays stable
• Connection pool exhausted — applications log "too many connections" errors
• Replication lag grows on read replicas as the primary falls behind
• Disk I/O saturation on WAL (Write-Ahead Log) files
• DB CPU high due to lock contention and fsync pressure

Root Cause

The write capacity (maxWriteQPS: 400) is sized for a read-heavy workload assumption that does not match reality. With 80% of 2,000 incoming QPS being writes, the write path faces 1,600 QPS — 400% of its limit — while the read path is comfortable at 50% (400 reads/s against 800). Read replicas would not help: they add read capacity only, and every write still goes to the primary.

Mitigations

• Write queuing: buffer writes in Kafka or SQS, letting the DB consume at its own pace
• Batch writes: consolidate many small writes into fewer large transactions to reduce per-operation overhead
• CQRS: separate write and read models — scale the write store independently from read replicas
• Sharding: partition data horizontally so each shard absorbs a fraction of the write load
• Right-size the instance: upgrade to an instance class with higher IOPS and more write capacity

Cache Miss Storm

Problem

A Redis cache layer sits between the API servers and the database, but its effective hit rate has collapsed to 5% — caused by a cache flush, a cold restart, or a poorly chosen TTL. 95% of 8,000 QPS (7,600 requests/second) falls through to a database provisioned for 1,500 read QPS, causing immediate saturation.

Symptoms in Production

• Database CPU spikes suddenly to 100% following a cache flush or deployment
• p99 read latency jumps from 1 ms (cache) to 15 ms+ (DB) for all users
• Cache hit rate metric drops — visible in Redis INFO stats or CloudWatch
• DB connection pool exhaustion as threads pile up waiting for reads
• Thundering herd: all cache keys expire simultaneously after a mass invalidation

Root Cause

With a 5% cache hit rate, only 400 of 8,000 requests are served by Redis. The remaining 7,600 reach the database: 7,220 reads/s against a read ceiling of 1,500 (481%) and 380 writes/s against 300 (127%). The cache was silently absorbing 95% of traffic — at a 95% hit rate the same database sits at 25% — and its absence is catastrophic.

Mitigations

• Cache warming: pre-populate the cache before going live or after a flush, especially for hot keys
• Staggered TTLs: add jitter to expiry times to prevent synchronized mass expiry (thundering herd)
• Circuit breaker: detect cache miss rate spikes and shed load before the DB saturates
• Read replicas: add database read replicas so that cache misses are absorbed by multiple nodes — 4 replicas cover the 7,220 reads/s, but the 380 writes/s still overload the primary
• Local in-process cache: add a small L1 cache (Caffeine, node-lru-cache) to absorb hotspot requests even when Redis misses

Load Balancer Saturation

Problem

A load balancer rated for 6,000 QPS is placed in front of two well-provisioned servers capable of handling the full 10,000 QPS load. Despite having sufficient compute capacity downstream, all traffic is throttled at the ingress tier — the load balancer becomes the sole bottleneck, wasting the server headroom behind it.

Symptoms in Production

• High connection queue depth at the LB while downstream servers are idle
• Uniform latency spike across all endpoints regardless of server load
• LB CPU or connection-table saturation visible in cloud provider metrics
• Error rate rises at the LB (502/504s) while server error rate stays low
• Adding more server instances does not improve performance at all

Root Cause

The load balancer's maxQPS ceiling of 6,000 is 60% of actual traffic (10,000 QPS). It runs at 167%, sheds 4,000 requests per second (40% errors) and passes on only 6,000 — so each server idles at 37.5% (3,000 of 8,000). Unlike server saturation, this bottleneck cannot be resolved by scaling downstream nodes — the constraint sits at the entry point. LB saturation is frequently overlooked because teams focus on server and database metrics.

Mitigations

• Upgrade or scale the load balancer tier: use a higher-capacity SKU or provision multiple LB instances
• DNS-based load balancing: distribute traffic across multiple LBs via weighted DNS or Anycast routing
• Connection keep-alive: reduce LB connection churn by enabling HTTP/2 or persistent TCP connections
• Offload TLS termination: move TLS to dedicated hardware or CDN to free LB capacity for routing
• Layer 4 vs Layer 7: use L4 (TCP) load balancing for high-throughput paths; reserve L7 (HTTP) only where header inspection is required

Queue Consumer Lag

Problem

A Kafka queue is receiving 8,000 messages per second, but only 2 consumer instances are running, each capable of processing 2,000 messages/second. Total consumer throughput is 4,000 msg/s — half of the ingestion rate. The queue depth grows at 4,000 msg/s, accumulating hours of backlog within minutes.

Symptoms in Production

• Consumer lag metric grows continuously — visible in Kafka consumer group offsets
• End-to-end processing latency measured from produce to consume increases steadily
• Alert fires on consumer lag threshold (e.g., > 100k messages behind)
• Downstream systems receive stale data as freshness degrades with lag
• Memory pressure on brokers as unread partitions accumulate

Root Cause

Queue drain rate is maxThroughput × consumers = 2,000 × 2 = 4,000 msg/s. At 8,000 msgs/s ingestion, consumers process only 50% of incoming load: the queue runs at 200%. This is not a spike — it is a sustained 2× overload. Each second, 4,000 messages accumulate as backlog (a queue buffers rather than failing), and the lag compounds faster than it can be recovered during off-peak hours. The consumer servers behind the queue see only the drained 2,000 msg/s each, so they look healthy while the lag grows.

Mitigations

• Scale consumers horizontally: add more consumer instances up to the partition count (e.g., 8 consumers for 8 partitions) — 4 consumers only match ingestion (100%), so plan for 5 or more, and scale the consumer servers with them
• Increase partition count: more partitions enable more parallel consumers — plan partition count for peak load at deployment
• Consumer batching: process messages in micro-batches to amortize per-message overhead
• Separate fast and slow consumers: use dedicated consumer groups for latency-sensitive vs. bulk processing
• Backpressure from producer: implement producer-side rate limiting if downstream can never catch up

Retry Storm

Problem

A PostgreSQL database with a 20% error rate (connection timeouts under load) runs at 92% — 4,900 QPS against a capacity of 5,333 — behind API servers configured with retryCount: 3. Each failed call is retried up to 3 times, so every request costs 1 + 0.2 + 0.2² + 0.2³ ≈ 1.25 attempts. That pushes the database past 100%; once it is overloaded it sheds load, its error rate climbs, and every new error triggers more retries in a reinforcing feedback loop.

Symptoms in Production

• Database QPS in monitoring is significantly higher than expected given client traffic
• Error rate stays elevated even after the initial failure event passes
• CPU and connection count on the database oscillate in a sawtooth pattern as retries cause new failures
• Client-side timeout metrics show high retry counts — often hidden in SDK/HTTP client logs
• Cascading failures: upstream services begin timing out as latency climbs, triggering their own retries

Root Cause

Expected attempts per request are 1 + e + e² + e³, where e is the database's total error rate — configured plus overload. At e = 20% that is 1.25×, which takes the database to 115%. Past 100% it sheds load, e rises, attempts rise, and the loop settles at about 10,700 QPS: 2.2× the 4,900 base load, 200% utilization, 60% of attempts failing and unbounded latency. Without retries the same database runs at 92%. Retries recover some failed requests, but they triple the database's error rate. The system is in a positive feedback loop where more retries → more load → more errors → more retries.

Mitigations

• Exponential backoff with jitter: never retry with a fixed interval — use randomized exponential delays to desynchronize clients
• Retry budget: cap total retries per request at the service boundary, not per-hop (e.g., 3 retries total, not 3 per microservice)
• Circuit breaker: stop sending requests (and retries) when error rate exceeds a threshold — fail fast and recover
• Idempotency: ensure operations are safe to retry by design; use idempotency keys for write operations
• Retry only on transient errors: distinguish 503 (retry) from 422 (do not retry) — never retry on non-transient errors