There isn't one kind of 'memory' in a computer — there's a whole ladder of them, each faster and smaller than the one below it, and almost every performance trick in computing is really about keeping data one rung higher on that ladder.
3 min read
A CPU register is roughly 100 times faster to access than RAM, and RAM is roughly 1,000 times faster than an SSD — and closer to 100,000 times faster than a spinning hard disk. This isn't a small difference to optimize around at the margins — it's large enough that where your data physically sits is often a bigger factor in real performance than the algorithm operating on it.
Each level acts as a cache for the level below it: frequently-accessed RAM contents get copied into the CPU cache; frequently-accessed disk contents get cached in RAM by the operating system automatically. The virtual memory and paging lessons earlier in this domain already covered exactly this idea one level of the hierarchy up — a page fault is what happens when something needed isn't in RAM and has to be pulled up from disk, the slowest possible case in this same hierarchy.
This whole hierarchy only pays off because of a genuine, measurable pattern called locality of reference: programs tend to access the same data (or data physically near it) repeatedly in a short window of time — a loop re-reading the same variable thousands of times, or code processing an array element by element. Because of that pattern, keeping recently- or soon-to-be-used data in a faster, smaller layer works far better in practice than it might sound like it should from the size numbers alone.
"Why is this loop so much slower than I expected, even though the number of operations is the same" is very often a memory-hierarchy question in disguise — how the data is laid out and accessed matters as much as what's computed. Recognizing that registers, cache, RAM, and disk aren't just different technologies but one continuous trade-off between speed and size is what makes performance intuitions like "keep hot data close" make sense as a single underlying idea, rather than a list of unrelated tips.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. Why does the memory hierarchy exist as multiple distinct levels instead of one uniform kind of memory?
2. Why does iterating an array in order tend to be faster than accessing its elements in random order?
3. Why does a dataset that fits entirely in RAM tend to perform dramatically better than one that doesn't?
Operating Systems Fundamentals