Hallucination
When an AI invents an answer that sounds right but is not true
- An AI does not look facts up. It picks the most plausible next words and keeps going, which means it does not stop at the edge of what it knows.
- Gaps show up most around rare information, anything that changed after training ended, and questions built on a false premise.
- The inventions come in kinds. A year that slips by one is annoying; a source conjured out of nothing is far more dangerous.
- Retrieval, alignment training, demands for citations and a low temperature setting all cut the rate, but none of them reach zero.
- Checking names, numbers, dates and quotes against the original is still the most reliable defense.
Contents
1The analogy
Picture the last question on a final exam, the long written one. You cannot remember the answer, and a blank space is guaranteed to score nothing. So you weave the words you do recognize into five confident lines. The handwriting is neat, the sentences flow, the tone leaves no room for doubt. Until the grader opens the answer key and works through it line by line, that page is indistinguishable from the page of a student who actually knew. If anything, the better the writing, the more correct it looks.
An invented AI answer is exactly that page. Fluency is evidence that the page was written well. It is not evidence that the content is right. The trouble is that fluency is the only signal most readers have, and it points the wrong way.
2In detail
There is no line on the scoresheet for truth
A language model does not consult anything while writing. It calculates which piece is likely to follow the text so far, picks one, attaches it, and calculates again. After "the capital of France is," the piece "Paris" scores overwhelmingly high, so the right answer comes out. Not because the model knows a fact, but because it has seen those words standing side by side a very large number of times.
The problem arrives when the training text barely covers the question. What remains is a field of mediocre candidates, and "stop here" is not offered as a default option. The model takes the best of what is left and finishes the sentence. A wrong sentence comes out of the same calculation as a right one. That is why tone alone can never separate them.
The gaps appear in predictable places
Three situations account for most of it. The first is rare information: a little-known person, a small company, the schedule of a neighborhood festival, anything the world has only written a few lines about. The second is anything that changed after training stopped. A model's knowledge is frozen at some date, and it has no sense that its own knowledge is frozen.
The third is the loaded question. Ask "what year did that incident happen?" about an incident that never happened, and the model will not challenge the premise. It fills in an answer that fits the premise, because a plausible continuation is what it was built to produce. Wording a question as though something is settled is one of the fastest ways to be handed an invention.
The inventions come in kinds
The most common kind is a fact that drifts. A year lands one off, two people swap roles. Irritating, but relatively easy to spot. Harder is a source invented whole: a paper with a believable title, believable authors and a real-sounding journal, which turns out not to exist. Third is straying from material you supplied, where you paste in a document, ask for a summary, and sentences appear in the summary that are nowhere in the document. The first two clash with the world. The last clashes with the page in front of you.
The countermeasures already run several deep
The most widely used is RAG (Retrieval-Augmented Generation). Relevant documents are fetched before the answer is written and handed over alongside the question, so the model answers from material in front of it rather than from memory. Alignment training rewards saying "I don't know," which wears down the habit of filling every gap.
Asking in the prompt for evidence attached to each claim makes unsupported sentences stand out. Lowering a setting called temperature narrows the choice to high-probability candidates, so fewer odd sentences slip through. None of it changes the underlying mechanism: the next word is still chosen by probability.
The reader's habits are the last layer
You do not have to doubt every sentence. Spend the effort where being wrong would hurt. Six things deserve it: names, numbers, dates, quotations, legal or policy claims, and links. When an answer contains any of them, verify that part against the original.
If a link is attached, open it and confirm the sentence is actually there. Asking the same question again, worded a little differently, is also worth the minute it takes. The parts that change between the two answers are usually the parts that were invented.
3More precisely
Hallucination is the generation of content presented as fact that is supported neither by the training data nor by any evidence provided. Some researchers split it in two: faithfulness errors, which contradict the supplied source documents, and factuality errors, which contradict the world. The word has nothing to do with human hallucination. It was an early label that stuck.
The exam comparison breaks down at one point. A student staring at a question knows they do not know. A language model has no such awareness. What it handles is a probability distribution over the next token, and a low probability does not automatically convert into a signal that says "this is wrong." Confidence and correctness are computed nowhere near each other.
There is a second gap. The model has already been graded. During the tuning that shapes it around human preference, decisive phrasing tended to earn better scores than hedging, so the model leans toward sounding certain rather than sounding careful. That habit is what makes the invented page so convincing, and it is also why a model that hedged honestly would feel worse to use even while being more useful.
4Try it yourself
- LM Arena (compare two models) ailearn.space Ask two models the same question about an obscure person or a small local festival, and watch one state an invented fact with total confidence
- Fill in the Blank (what a language model learned) ailearn.space The habit of filling a gap with the most plausible words, on full display. That habit is where hallucination starts
- Perplexity (AI search) ailearn.space Get an answer with source links, then open one and check whether that sentence is really there
5Common misconceptions
It's easy to think the AI is lying on purpose, but actually there is no intent to deceive. True and false sentences come from the same calculation.
It's easy to think a confident answer is a correct one, but actually confidence of tone and accuracy of content move independently. Better writing can make a wrong answer more convincing.
It's easy to think attaching search fixes it, but actually the wrong document can be retrieved, and sentences absent from the retrieved document can still work their way into the answer.
7One-line summary
In shortHallucination is an AI filling in the page for a question it cannot answer, and a smooth sentence is no evidence that the sentence is true.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02