A model can score perfectly on the data it trained on and still be useless in practice — memorizing instead of generalizing. Splitting data into three separate pools, never mixed, is the actual discipline that catches this before it reaches production.
3 min read
A model with enough weights can, in principle, memorize its entire training set — achieving near-perfect accuracy on exactly the examples it saw, while learning nothing that generalizes to new, unseen inputs. This failure mode is called overfitting, and it's genuinely one of the most common, most dangerous ways an ML project goes wrong, precisely because the warning sign (bad performance) never shows up if you only ever check performance against the training data itself.
The discipline that catches this is splitting available data into three separate, non-overlapping pools before training even begins:
It's tempting to think checking against a validation set solves everything — but if you repeatedly tweak the model based on validation performance (which is the entire point of having one), you're effectively, indirectly fitting the model to the validation set too, just less directly than fitting to training data. This is exactly why a third, completely untouched test set exists — it's the one number that hasn't been used, even indirectly, to make any decision about the model, and is the only honest signal for how it'll perform on genuinely new data.
A model can also be too simple to capture the real pattern at all — underfitting — performing poorly on both training and test data alike. Overfitting gets more attention because it's the sneakier failure (looks great until you check the right thing), but underfitting is just as real a failure mode, usually meaning the model needs more capacity, better features, or simply more relevant training data.
Nearly everyone building on top of ML today calls an already-trained model's API rather than training one from scratch — but this discipline still matters directly: it's exactly why you should be skeptical of any AI vendor's benchmark numbers reported only on their own curated evaluation set, and why evaluating an LLM-powered feature you built (the last lesson in this domain covers this directly) should always be done against realistic examples your actual users will send — not just the handful of examples that happened to work well while you were building it, which is the exact same overfitting trap, one level up.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. What is overfitting?
2. Why is a validation set alone not sufficient — why is a separate test set needed?
3. What is underfitting, and how is it different from overfitting?
AI & ML Fundamentals