Model Collapse

AI output is creeping into the training material

Large models learn by scraping up the text and images piled onto the internet. But the share of that material actually made by AI is climbing fast. When the next generation of models is trained, some of what was made by the previous generation ends up mixed into the pile.

The trouble is that it is hard to sort out. Tools built to spot AI-made content only give a probability, and anything a person has touched up gets even harder to call. Whoever is collecting the data usually has no idea which pieces are which.

That makes this less a single company's mistake than a case of the whole shared pool getting slightly tainted. It is also why material from the era before AI output was common is becoming more valuable.

The rare cases disappear first

A model imitates the distribution it learned from, but it imitates the common parts well and the rare parts poorly. So its output already contains less of the rare stuff than the original had. Train the next model on that output, and the rare share shrinks again.

This shaving repeats every generation. Regional dialects, unusual writing styles, uncommon compositions and color choices — the things sitting at the edges — go first. The middle holds up for a long time while the edges erode.

Surface quality stays fine for a while. Sentences read smoothly, images look clean. But underneath, everything is quietly drifting toward looking the same no matter what you ask for.

It only takes a few generations

Small-scale experiments show the pattern clearly. Train a model only on the previous generation's output, repeat that a few times, and the spread of results narrows noticeably within a handful of generations — and the collapse speeds up as it goes.

Errors stack up here. There is the gap from having only a limited sample, the gap from a model never fully capturing the true distribution, and the gap from training never converging perfectly. Each generation adds its own version of these, and the previous generation's approximation gets treated as ground truth by the next.

Real-world conditions are messier than the experiments, though. Human-made material keeps arriving, and there is a filtering effect too, since only the generated output people actually chose to keep survives. So collapse is less a foregone conclusion than a risk that shows up when the material goes unmanaged.

It is a different problem from mode collapse

The similar names make this easy to mix up. Mode collapse happens during a single model's training run — the model settles for a handful of outputs that work and gives up on the rest. It is a problem that starts and ends inside that one training process.

Model collapse is a problem between generations. One model's output becomes the next model's textbook, and that model's output becomes the textbook after that, narrowing the material itself across several generations. Training a single model well does nothing to stop it.

The fix is managing the material

First, keep bringing in human-made material. New human records keep refilling the edges, and archiving old original data carefully belongs here too.

Second, control the mix ratio. Rather than cutting out generated data entirely, it is more realistic to add it on top of human-made data while keeping its share low. Studies show this substantially slows the narrowing.

Third, mark and filter. Embedding an invisible marker at generation time and attaching a history to files lets later data collection screen some of it out. Using only material a person has reviewed and chosen helps too.