DynamoDB Failed 5 Times. Then It Hit 89M req/sec.
The 5 generations of engineering that turned hot partitions into a solved problem — from consistent hashing to global admission control.
You have a database on one server. A few hundred users, everything's fast. Then your app takes off. The server's maxed out. So you add more servers — but now which server stores which data?
The obvious answer: hash(key) % num_servers. Clean. Simple. Until you add a fourth server and nearly every key maps to a different machine. Catastrophe.
The Hash Ring
Instead of a line, imagine a circle. Each server gets hashed to a position on this ring. Each key also gets hashed to a position. To find which server owns a key, walk clockwise until you hit a server.
When you add a new server, it lands somewhere on the ring, and only the keys between it and its predecessor need to move. Everything else stays put. Instead of reshuffling 90% of your data, you move maybe 10%.
But with just 3 servers, the ring divides unevenly. The fix? Virtual nodes — place each server at 150 positions scattered around the ring. The deviation drops from ~40% to ~5%.
Inside DynamoDB: The Request Path
Every request follows the same path:
- Request router — stateless, handles auth, hashes your partition key
- Cache lookup — 99.75% hit rate, nearly instant
- Storage node — B-tree for queries, WAL for durability, leader serializes writes
- Paxos replication — every partition replicated 3x across availability zones, write acknowledged when 2/3 confirm
This is how DynamoDB serves 89 million requests per second at single-digit millisecond latency.
The Enemy: Hot Partitions
A celebrity tweets a link to your app. Millions of requests slam one partition key. That partition has a limit of 3,000 reads/sec. It's instantly overwhelmed while every other partition sits idle.
DynamoDB spent 10 years and 5 generations solving this:
| Generation | Approach | Limitation |
|---|---|---|
| 1 | Uniform allocation | One hot key = throttled |
| 2 | Burst capacity (5 min buffer) | Useless against sustained traffic |
| 3 | Adaptive capacity | Limited by physical node capacity |
| 4 | Split for heat | Hot partition divides itself in two |
| 5 | Global admission control | Throughput decoupled from partitions entirely |
The meta twist: the fleet of token bucket servers managing global admission control is itself organized using consistent hashing. It's consistent hashing all the way down.
The Plot Twist
DynamoDB is named after Amazon's Dynamo paper (2007). But the two systems are almost nothing alike.
Original Dynamo: peer-to-peer, no leader, vector clocks, application-side conflict resolution.
DynamoDB: single leader per partition, no conflicts, no vector clocks. Amazon kept the name and rebuilt everything else.
One of DynamoDB's most elegant innovations — log replicas — didn't come from the paper at all. When a storage node fails, instead of copying a full 10GB B-tree (minutes), DynamoDB spins up a lightweight replica storing only the recent write-ahead log (hundreds of MB). Write quorum restored in seconds.
DynamoDB didn't become the most widely used NoSQL database by following a paper. It got there by solving problems the paper never imagined.
Watch the full animated breakdown: DynamoDB Failed 5 Times. Then It Hit 89M req/sec.