What is Overfitting? — one analogy

1The analogy

Think of someone who memorized a practice test book, answers and all. Inside that book they are unstoppable. Question three is B, question seven is D, no reading required. Each pass through it, the number of mistakes drops closer to zero and the confidence climbs right along with it.

Then the real exam arrives, one condition in a familiar problem is different, and they cannot start. They memorized the pairing of question and answer instead of the reasoning that produces the answer. Inside the book, memorizing and understanding look exactly alike. Only a problem the book never contained tells them apart.

Overfitting in an AI is precisely this state. Near-perfect on the data used for practice, and a sharp drop the moment it faces something it has never seen. The unsettling part is the confidence: nothing in the practice scores warns you that the collapse is coming.

2In detail

A practice book, a pop quiz, and an exam

When a model is trained, the available data is never used as one lump. It is cut roughly eight to one to one. Training data is the practice book the model actually studies. Validation data is the pop quiz taken during study, and its score drives the decisions: when to stop, how many layers to stack, how fast to learn. Test data is the exam pulled out exactly once, at the very end.

Why bother splitting at all? Grading with training data is grading with the answer key open on the desk. A model that memorized scores full marks there too. But tweak settings fifty times against the validation set and the developer has now, in effect, memorized the validation set as well. So one exam is kept sealed and never looked at until the work is finished.

The moment the two scores separate

Plot training loss and validation loss side by side as training runs, and overfitting becomes visible. Loss is a measure of how wrong the model is. Early on, both curves fall together, which means the model is learning real regularities.

Then, at some point, they part. Training loss keeps heading for the floor while validation loss quietly turns and climbs. From that point on the model is no longer learning rules. It is learning accidents that exist only in the training data. The width of the gap between the two curves is the size of the overfitting.

Why models drift toward memorizing

The first reason is a model too complex for its data. When there are millions of adjustable values and only a few hundred examples, storing the examples one by one is a faster route to a low loss than finding the rule they share.

The second is data that is scarce or lopsided. If every training photo of a cat was taken on a sofa, the model files the sofa pattern away as part of what makes a cat. Show it a cat on grass and the confidence evaporates.

The third is looping over the same data too long. The first few passes bring real gains. After that, the model starts accepting smudges on the photos and quirks of the lighting as rules worth learning.

Four ways to stop it

Early stopping halts training the moment the validation score stops improving. The point where the two curves separate is exactly the place to stop. Regularization penalizes any value that grows too large, which keeps the model from leaning its whole weight on a single clue.

Dropout switches off a random share of neurons on every pass. Since no neuron knows whether it will be present, none of them can hoard an answer. Augmentation flips, rotates and rebrightens the images to enlarge the practice book itself. The surest fix, of course, is collecting more real data.

The opposite mistake, and the actual goal

You can fall the other way. A model too simple to get even the training data right is underfitting. That one is easy to spot, since the training score is poor from the start. Make it bigger, or train it longer.

A good model sits somewhere between the two, where the training score and the validation score are both high and close together. What anyone wants is not skill at the practice book but skill at problems never seen before. That skill is called generalization.

3More precisely

Overfitting is a state where a model has learned the noise in its training data, leaving training error low while error on unseen data rises. The distance between the two is the generalization gap. A high training score is not itself a flaw; the widening gap is the problem. Measuring that gap requires data never used in training, which is the whole reason for holding a slice back.

The comparison breaks down in one place. A student decides to memorize, and knows they are doing it. A model decides nothing. It follows one instruction, lower the loss, and under certain conditions memorizing is simply the fastest way to lower it, so that is where it drifts, with no more intent than water running downhill.

There is also a wrinkle at the largest scale. In very large models, pushing the parameter count higher can make validation performance get worse and then improve again, a pattern known as double descent. Overfitting is not a curse that falls automatically on big models. It is a balance among model size, data volume and training time, and any of the three can be the one out of place.

4Try it yourself

5Common misconceptions

It's easy to think the higher the training accuracy the better the model, but actually the wider the gap between training and validation scores, the more overfitting you should suspect.
It's easy to think longer training always makes a model smarter, but actually past a certain point it memorizes noise and performance on new data slides back down.
It's easy to think more data always fixes it, but actually data that closely resembles what you already had adds nothing. Variety has to grow along with volume.

7One-line summary

In shortOverfitting is an AI that memorized the practice answers, and the cure is testing it on unseen problems along the way so you know when to stop.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02