Distributed cache clusters: sharding a cache across nodes

A single cache instance is bounded by that one machine's memory — scaling vertically (a bigger machine) has a ceiling, and eventually a real, large working set (every product in a large catalog, every active user's session) exceeds what fits. The fix is the same one System Design's own database-sharding lesson applied to databases: split the data across multiple nodes, each holding a fraction of the total key space.

A simple modulo-based approach picks a node by hashing the key and taking the remainder against the current node count. This works until the node count changes — adding or removing even one node changes number_of_nodes, which changes the result of hash(key) % number_of_nodes for nearly every key, not just the keys that logically needed to move. The practical effect: resizing a cache cluster from 4 nodes to 5 can invalidate the vast majority of the cache all at once, recreating exactly the cold-start stampede problem lessons 4 and 6 both covered, just triggered by routine cluster scaling instead of a deploy or a TTL.

Consistent hashing (introduced conceptually in System Design's own lesson on the topic, for the general case of distributing keys across nodes) maps both nodes and keys onto the same point on a conceptual ring, and a key belongs to whichever node is the next one clockwise from that key's position on the ring.

Adding a new node only takes over the portion of the ring between itself and the next node counter-clockwise from it — every other key, on every other part of the ring, stays mapped to exactly the same node it was before. Removing a node has the same property in reverse: only the keys that belonged to the removed node need to move, redistributed to its immediate neighbor on the ring.

A plain consistent-hashing ring with only a few real nodes can still distribute keys unevenly — if two nodes happen to land close together on the ring, one node ends up responsible for a much larger arc (and thus many more keys) than another. Real implementations assign each physical node multiple positions on the ring (virtual nodes, often dozens or hundreds per physical node) — the more virtual nodes, the more evenly keys tend to spread across physical nodes in practice, since an uneven placement of a few real nodes gets smoothed out by a much larger number of smaller, more evenly-scattered virtual segments.

Some systems (Redis Cluster) handle this routing on the client side — the client library itself computes which node owns a key and connects directly. Others (Twemproxy in front of plain Redis instances) put a proxy in between that does the routing, so individual Redis nodes and application clients both stay simpler, at the cost of the proxy itself becoming a new piece of infrastructure to run and a potential bottleneck if it's not itself scaled out.