Caching — cache-aside, and the two hard problems

The cache-aside pattern in full, and why "there are only two hard things in computer science: cache invalidation and naming things" isn't just a joke.

Intermediate

3 min read

What a cache is actually for

A cache trades memory for speed by keeping a copy of expensive-to-compute or slow-to-fetch data somewhere faster to read. It only pays off for data that's read far more often than it changes — caching something that's written on every request and read once provides no benefit and adds real complexity for nothing.

The cache-aside pattern

The most common pattern in ordinary application code — the application itself manages the cache, not the database:

def get_user(user_id):
    cached = cache.get(f"user:{user_id}")
    if cached is not None:
        return cached                      # cache hit — skip the database entirely
 
    user = db.query("SELECT * FROM users WHERE id = %s", user_id)
    cache.set(f"user:{user_id}", user, ttl=300)  # cache miss — fetch, then store for next time
    return user

On a cache hit, the database is never touched. On a miss, the application fetches from the database and populates the cache for the next request — the first request after a miss pays the full cost, every request after that until the entry expires or is invalidated is fast. The ttl (time-to-live) bounds how stale the cached value can get even if nothing explicitly invalidates it.

The first hard problem: invalidation

The cache above will serve a stale user for up to 5 minutes after that user's row changes in the database — the cache has no way to know the underlying data changed unless something tells it. Two real strategies:

Invalidate on write — when the user is updated, explicitly delete (or update) cache["user:{id}"] in the same code path that writes to the database. Correct, but only works if every code path that writes to that data remembers to invalidate — miss one, and that path serves stale data indefinitely.
Rely on TTL alone — accept that data can be stale for up to the TTL window, and pick a TTL short enough that the staleness is acceptable for that specific data. Simpler, but it's a deliberate trade of correctness for simplicity, not a fix.

Most real systems combine both: TTL as a safety net, explicit invalidation on write as the primary mechanism for anything where staleness is actually noticeable to a user.

The second hard problem, briefly: naming things

The often-quoted rest of the joke — "...and off-by-one errors" — is really about cache keys, which are a naming problem: user:{id} above is a simple, unambiguous key, but real systems often need keys that encode several dimensions (a user's data for a specific locale, for a specific API version) — get the key scheme wrong and you either get incorrect cache hits (serving French content for a German request because the key didn't include locale) or an explosion of near-duplicate cache entries that defeats the purpose of caching in the first place.

Where caching actually goes, layered

Each layer catches what the layer before it missed, and each layer is progressively more expensive to hit: a browser cache hit costs nothing, a CDN hit costs one network round-trip to a nearby edge server, an application-cache hit costs a round-trip to Redis, and a database hit is the most expensive of all — real disk/index work, not just a lookup. Putting a cache in front of read-heavy, rarely-changing data (product catalogs, user profile data, computed aggregates) is one of the highest-leverage, lowest-risk performance changes available in a real system.