Model collapse is the degradation that occurs when models are trained on data generated by earlier models, and it eats the tails of the distribution first. Rare events, minority patterns and unusual phrasings disappear before average performance moves, so the damage is well advanced before any headline metric notices.
Shumailov and colleagues published the mechanism in Nature in 2024, demonstrating that indiscriminate training on generated content causes irreversible defects in which the tails of the original content distribution vanish.
The word doing the work there is tails. Collapse is not a uniform blurring. It is a systematic narrowing, and narrowing looks fine on any metric that reports an average.
At a glance
- Rare and minority patterns are lost first, while average metrics stay stable
- Each generation compounds the loss of the one before it
- Synthetic data is not inherently harmful; replacing real data with it is
- Accumulating real plus synthetic behaves very differently from substituting
- Provenance tracking is the practical defence
- Perplexity and average accuracy are poor detectors
Why do the tails go first?
Because sampling underrepresents rare events, and training on samples makes that underrepresentation permanent.
Any finite sample from a distribution contains proportionally fewer rare cases than the distribution itself. Train on that sample and the model learns the slightly narrower version. Generate from that model, sample again, and the narrowing compounds.
Figure 1 — Two ways to use synthetic data
Replacement
Each generation trains primarily on the previous generation’s output, with original data discarded or diluted away.
Errors compound. The distribution narrows generation over generation, and the loss is not recoverable from the degraded data.
Collapse pathway
Accumulation
Original human data is retained permanently, with synthetic data added alongside rather than substituted for it.
The real distribution stays anchored in the training set, which substantially mitigates the compounding effect.
Mitigated
This distinction is the practical takeaway. The question is not whether to use synthetic data, which is often useful and sometimes necessary. It is whether real data survives in the mix, permanently and in meaningful proportion.
What does collapse look like in practice?
Rarely like obvious breakage, which is what makes it dangerous.
- Reduced output diversity. Prompts that once produced varied responses converge on a house style.
- Loss of rare vocabulary. Specialist terms and proper nouns get replaced with common near-synonyms.
- Minority pattern erasure. Dialects, low-resource languages and uncommon formats degrade first.
- Overconfident averaging. The model produces the most typical answer more often, including where the typical answer is wrong.
- Stable headline metrics. Average accuracy and perplexity hold up while all of the above is happening.
Watch for this
Standard evaluation actively hides collapse. Benchmarks measure average performance on typical cases, which is precisely the region that survives longest. A model can lose substantial capability on rare inputs while its benchmark scores barely move. Detecting collapse requires measuring diversity and tail performance specifically, as separate metrics.
How do you detect it?
Measure the things averages conceal.
- Output diversity. Generate many completions for one prompt and measure distinctness between them. Falling diversity across model versions is the earliest signal available.
- Tail-slice accuracy. Build evaluation slices consisting entirely of rare cases, and report them separately from the aggregate. Never let them be averaged in.
- Vocabulary coverage. Track the breadth of distinct tokens and named entities the model produces across a fixed prompt set.
- Distributional comparison. Compare the statistical properties of generated text against a held-out sample of genuine human text, not against previous generations.
Point 4 carries the discipline the whole problem demands: always compare against real data, never against the previous model. Comparing generations to each other is how gradual drift becomes invisible.
Does this mean synthetic data is bad?
No, and treating the finding that way would be a misreading worth correcting.
Synthetic data is valuable and sometimes the only option: for privacy-constrained domains, for rare-event augmentation, for bootstrapping tasks with no labelled corpus. The QLoRA work is a reminder that carefully constructed small datasets can outperform much larger scraped ones.
The failure mode is specific. It is recursive training where generated data progressively replaces real data across generations, without provenance tracking and without anyone measuring diversity.
| Practice | Collapse risk | Why |
|---|---|---|
| Synthetic augmentation alongside real data | Low | Real distribution stays anchored |
| Human-filtered synthetic data | Low | Human judgment reintroduces signal from outside the loop |
| Distillation from a stronger teacher | Low to moderate | Single hop, no recursion, teacher trained on real data |
| Self-training on own output | High | Directly recursive with no external anchor |
| Scraping a web increasingly full of generated text | High and hard to control | Provenance is unknown, so recursion is invisible |
That last row is the industry-level version of the problem, and the reason provenance metadata matters well beyond any single team’s pipeline.
What prevents it?
Four practices, in rough order of how much protection they deliver per unit of effort.
- Track provenance on every training example. Human-authored, model-generated, or model-generated-and-human-verified. Without this label you cannot reason about the problem at all.
- Never discard original human data. Retain it permanently and keep it in every training mix rather than letting it dilute across versions.
- Put humans in the loop on synthetic data. Filtering and correction reintroduce information from outside the model’s distribution, which is exactly what recursion removes.
- Cap the synthetic proportion explicitly. Make it a stated parameter someone owns, not an emergent property of whatever data happened to be available.
What do experienced teams do differently?
They treat diversity as a first-class metric with an owner and an alert, rather than something inspected when output looks strange.
Because collapse is gradual and average metrics stay flat, the only reliable defence is a metric that moves early and gets watched. Teams that add diversity tracking after noticing a problem have usually already lost several generations of capability.
They also keep a permanently frozen sample of genuinely human data as the comparison baseline. As the open web fills with generated text, the ability to say what real data looked like at a known point becomes progressively harder to reconstruct and progressively more valuable.
A short glossary
- Model collapse
- Degenerative degradation in models trained on data produced by earlier models, characterised by loss of distribution tails.
- Distribution tails
- Rare events and uncommon patterns at the edges of a data distribution, lost first under recursive training.
- Accumulation
- Adding synthetic data alongside retained real data, rather than substituting it, which substantially mitigates collapse.
- Provenance
- Recorded origin of each training example, distinguishing human-authored from model-generated content.
- Self-training
- Training a model on its own generated output, the most directly recursive and highest-risk configuration.
Key takeaways
- Model collapse degrades the tails of a distribution first, so rare and minority patterns vanish before averages move.
- Shumailov et al. published the mechanism in Nature in 2024, describing irreversible defects from indiscriminate training on generated data.
- Accumulating synthetic data alongside retained real data behaves very differently from replacing real data with it.
- Standard benchmarks hide collapse, because they measure the typical cases that survive longest.
- Detection requires diversity metrics and tail-specific evaluation slices reported separately from aggregates.
- Always compare against held-out real data, never against the previous model generation.
- Provenance tracking on every training example is the precondition for managing any of this.
Frequently asked questions
What is model collapse?
A degenerative process where models trained on data generated by earlier models progressively lose the tails of the original distribution. Shumailov and colleagues documented it in Nature in 2024, showing that indiscriminate training on generated content causes irreversible defects in which rare events disappear from the learned distribution.
Does using synthetic data always cause collapse?
No. The risk comes from recursion where generated data progressively replaces real data across model generations. Synthetic data added alongside permanently retained human data, particularly when human-filtered, behaves very differently and is a legitimate and often valuable technique.
Why do rare cases degrade before common ones?
Because any finite sample underrepresents rare events relative to the true distribution. A model trained on that sample learns a slightly narrower distribution, and generating then resampling compounds the narrowing at each step. Common cases are reinforced while the edges thin out.
Can model collapse be reversed?
Not from the degraded data itself, since the information about the lost tails is no longer present in it. Recovery requires reintroducing genuine human data containing those rare patterns, which is why permanently retaining original datasets matters so much.
How do I detect collapse in my own models?
Measure output diversity across generations for fixed prompts, build evaluation slices made entirely of rare cases and report them separately, track vocabulary and entity coverage, and compare generated text against held-out human data rather than against your previous model version.
Is distillation the same as recursive training?
Not quite. Distillation is typically a single hop from a stronger teacher trained on real data, rather than an iterated loop feeding a model its own output. The risk is lower, though repeated distillation across several generations moves it toward the same territory.
Does web-scraped training data now carry this risk?
Increasingly, and it is difficult to control. As generated text accumulates online, scraped corpora contain an unknown and growing proportion of model output with no provenance labelling. This makes recursion invisible at the point of training, which is why provenance metadata matters beyond any single team.
Will benchmark scores show collapse?
Usually not until it is advanced. Benchmarks measure average performance on typical cases, which is the region that survives longest. Substantial capability can be lost on rare inputs while headline scores barely move, so diversity and tail metrics must be tracked separately.
References
- Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. AI models collapse when trained on recursively generated data. Nature, 2024.
- Shumailov, I., et al. The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493.
- Gerstgrasser, M., et al. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv:2404.01413.
- Dettmers, T., et al. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. Source for the finding that small high-quality datasets can outperform larger ones.
