You cannot eliminate hallucination, so build detection instead. The sharper question is not “how do I stop the model inventing things” but “how do I catch an invented claim before a user acts on it.” Grounded and ungrounded errors need entirely different tools, and conflating them is why most detection efforts underperform.
Most teams ask how to make the model stop hallucinating. That framing has no answer, because generation is probabilistic and a fluent wrong answer is indistinguishable from a fluent right one at the token level.
The tractable question is narrower. Given an output, how confident are you that each factual claim in it is supported? That question has good answers, several of which are cheap.
Which kind of hallucination are you dealing with?
Two categories, and almost every failed detection project applied a tool built for one to a problem of the other.
Figure 1 — Two problems that look identical in the output
Grounded hallucination
You supplied source documents. The model asserted something those documents do not support.
Verifiable by machine: compare each claim against the supplied text. No external knowledge needed.
Tractable
Ungrounded hallucination
No source was supplied. The model answered from parametric knowledge and got it wrong.
Detecting this requires knowing the truth, which is the problem you were trying to solve.
Much harder
The strategic implication: the cheapest hallucination fix is architectural. Supply sources, and you convert an intractable verification problem into a tractable one. Retrieval is a detection strategy as much as a knowledge strategy.
The ten techniques
| Technique | Catches | Cost | Limits |
|---|---|---|---|
| Citation requirement | Unsupported claims in grounded tasks | Very low | Model can cite the wrong span confidently |
| Span verification | Citations that do not support the claim | Low | Needs exact-match or entailment checking |
| Claim decomposition | Partially-true compound statements | Moderate | Extra generation pass per response |
| Natural language inference | Contradictions with source text | Moderate | Struggles with implicit and numeric entailment |
| Self-consistency sampling | Low-confidence fabrication | High: n generations | Confidently consistent errors survive |
| Token log-probability | Uncertain spans | Very low | Weak correlation with factual correctness |
| Structured extraction | Invented entities and figures | Low | Only covers extractable fields |
| Cross-model check | Family-specific fabrication | Moderate | Shared training data yields shared errors |
| Deterministic validators | Broken links, bad dates, failed lookups | Very low | Only checks machine-verifiable claims |
| Human review of flagged output | Everything | Highest | Does not scale without good triage |
Start with the two techniques at the bottom of the cost column: deterministic validators and citation requirements. Together they catch a surprising share of real incidents for almost no engineering effort, and neither needs a model.
Why is self-consistency oversold?
Because its failure mode is exactly the case you most need to catch.
Self-consistency generates the same answer several times and flags disagreement as a fabrication signal. When a model is genuinely uncertain, samples diverge and the technique works.
But a model that has confidently learned something wrong produces the same wrong answer every time. Consistency is high, the detector reports no problem, and the error passes through.
Watch for this
Self-consistency measures the model’s internal agreement, not truth. It reliably catches uncertainty and reliably misses confident error. Because confident errors are the ones that cause harm, a detection strategy built primarily on self-consistency inverts your risk profile while producing reassuring dashboards.
How do you build a claim-level pipeline?
Sentence-level verification is too coarse. A sentence containing three facts, one wrong, is neither supported nor unsupported.
- Decompose the output into atomic claims. Each should assert exactly one checkable thing.
- Route each claim by type. Numbers, dates and identifiers go to deterministic checks. Prose assertions go to entailment checking against sources.
- Verify each against the retrieved context, not against general knowledge, so the check stays tractable.
- Aggregate conservatively. One unsupported claim makes the response unsupported. Averaging support scores hides exactly the failure you are hunting.
- Route by severity, not by score. An unsupported dosage or price needs blocking; an unsupported adjective does not.
Step 5 is what makes the pipeline shippable. Treating every unsupported claim identically produces either an unusable false-positive rate or a threshold so loose it catches nothing.
Which signals are weaker than they look?
Token probability deserves particular scepticism.
The intuition is appealing: low-probability tokens should indicate uncertainty, and uncertainty should indicate fabrication. In practice the correlation is weak, because fluency and factuality are separate properties. A model can be highly confident about the shape of a sentence while its content is invented.
Use log-probability as a cheap triage signal to prioritise what gets expensive verification. Do not use it as a verdict.
Where do teams go wrong?
Trying to prevent rather than detect
Prompt instructions such as “only state facts you are certain of” produce hedging, not accuracy. The model has no reliable introspective access to its own certainty, so the instruction changes tone while leaving the error rate roughly intact.
Measuring on clean data
A detector evaluated only on outputs that happen to be correct cannot demonstrate that it discriminates. Build an evaluation set containing known fabrications, ideally harvested from real production failures, and report catch rate on that subset specifically.
Blocking on a single noisy signal
Any one detector produces false positives at a rate that will frustrate users if it gates output alone. Combine a cheap high-recall filter with an expensive high-precision check on whatever the first stage flags.
What do experienced teams do differently?
They make the system able to say it does not know, and they reward that behaviour in evaluation.
If your eval scores only answer quality, abstention always loses to a confident guess, and the system learns to guess. Scoring “correctly declined” as a success changes what the pipeline optimises toward.
They also log which retrieved chunk supported each claim, not merely which chunks were retrieved. When a false claim surfaces, that mapping distinguishes a retrieval failure from a generation failure in seconds, and those need entirely different fixes.
A short glossary
- Grounded hallucination
- A claim unsupported by the source documents supplied to the model, verifiable without external knowledge.
- Atomic claim
- A decomposed statement asserting exactly one checkable fact, used as the unit of verification.
- Natural language inference
- Determining whether one text entails, contradicts or is neutral toward another; the basis of automated claim checking.
- Self-consistency
- Sampling several generations and treating disagreement between them as an uncertainty signal.
- Abstention
- A system declining to answer when support is insufficient, which should be scored as a success rather than a failure.
Where to go deeper
Three directions repay the effort, in this order.
Start with the entailment literature, since claim verification is fundamentally a natural language inference task and that field has decades of work behind it. Then look at attribution evaluation, which formalises whether a generated statement is genuinely supported by a cited source. Finally, read the calibration literature, which addresses whether a model’s expressed confidence corresponds to its accuracy — the question underneath every uncertainty-based detector.
The unifying insight across all three: detection is a measurement problem, and measurement problems reward precise definitions far more than clever models.
Key takeaways
- Hallucination cannot be eliminated, so the practical goal is detection before a user acts on the output.
- Grounded errors are machine-verifiable against supplied sources; ungrounded errors require knowing the truth independently.
- Supplying sources is itself a detection strategy, converting an intractable check into a tractable one.
- Deterministic validators and citation requirements are the cheapest techniques and catch a large share of real incidents.
- Self-consistency catches uncertainty and misses confident error, which inverts your risk profile.
- Token log-probability correlates weakly with factuality; use it for triage, never as a verdict.
- Verify at claim level, aggregate conservatively, and route by severity rather than by score.
Frequently asked questions
Can hallucination be eliminated entirely?
No. Generation is probabilistic, and a fluent incorrect statement is structurally indistinguishable from a fluent correct one at the token level. The achievable goal is reducing frequency through grounding and catching the remainder through verification before output reaches a user.
Does retrieval-augmented generation stop hallucination?
It reduces it and, more usefully, changes its character. With sources supplied, an unsupported claim becomes checkable against those sources without external knowledge. Models still assert things their sources do not support, so retrieval makes verification tractable rather than making it unnecessary.
Is self-consistency a reliable detector?
Only for uncertainty. Sampling several generations catches cases where the model wavers, but a confidently wrong model produces the same wrong answer every time and passes the check. Since confident errors cause the most harm, self-consistency should never be your primary detector.
Do token probabilities indicate hallucination?
Weakly. Fluency and factuality are separate properties, so a model can be highly confident about phrasing while the content is invented. Log-probability works as a cheap triage signal for prioritising expensive verification, but produces poor verdicts on its own.
What is claim-level verification?
Decomposing an output into atomic statements that each assert one checkable fact, then verifying each independently against sources. It is more precise than sentence-level checking, because a single sentence containing several facts with one error is neither wholly supported nor wholly unsupported.
Should I use a second model to check the first?
It helps, with a caveat. Models trained on overlapping data share failure modes, so a checker from the same family may confirm the same fabrication. Use a different model family where possible, and prefer asking it to verify against supplied text rather than from its own knowledge.
How do I build an evaluation set for hallucination detection?
Harvest real fabrications from production failures rather than generating synthetic ones, which tend to be more obvious than reality. Include both supported and unsupported outputs, and report catch rate on the unsupported subset specifically, since overall accuracy on a mostly-correct set is uninformative.
Does prompting the model to be careful reduce hallucination?
Barely. Instructions to state only what it is certain of mainly produce hedged phrasing, because the model lacks reliable introspective access to its own certainty. The output sounds more cautious while the underlying error rate stays broadly unchanged.
References
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
- Ji, Z., et al. Survey of Hallucination in Natural Language Generation. arXiv:2202.03629.
- Rashkin, H., et al. Measuring Attribution in Natural Language Generation Models. arXiv:2112.12870.
- Dettmers, T., et al. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. Source for the caution on trusting benchmark-based evaluation of model output.
