Why language models hallucinate — and why the way we grade them keeps it that way

Why models guess, and why we taught them to

Ask a large language model for a stranger’s birthday and it may answer “March 7” with the steady confidence of someone reading it off a card — and be wrong, three times running, with three different dates. The authors give exactly this kind of example: leading models asked a plain factual question — a person’s birthday, or what an obscure acronym stands for — each confidently inventing a different answer, none of them correct. The industry’s word for this is hallucination, which makes it sound like a fault in perception. The paper’s first move is to take the mystery out of it.

Start with how a model is built. In its first and largest training stage it learns, in effect, what fluent language looks like by reading an enormous amount of text. Now take a fact with no pattern behind it — one particular person’s birthday. If that date appeared in the training text once, or never, there is nothing for a pattern-learner to grab onto: the answer is, from the model’s point of view, arbitrary. The authors make this precise by borrowing an old idea (Alan Turing’s, from a different problem): if one in five birthdays shows up only once in the data, a model should be expected to get at least one in five of them wrong — not because it is broken, but because there was never anything there to learn. (By the same logic, models almost never get a country’s capital wrong: those appear constantly.) They argue, with some care, that telling a true statement from a plausible false one is itself a hard problem, and that producing only true statements is at least as hard as that. A floor of error is baked in.

That explains where hallucinations come from. It does not explain why they survive — why models, after all the later training meant to make them helpful and honest, still bluff rather than admit doubt. Here the paper’s analogy is almost uncomfortably apt. Picture a student in an exam who does not know an answer. If a blank scores zero and a guess might score one, the grade-maximising move is to guess — confidently, specifically, never “I’m not sure.” Students learn this. So, it turns out, do models — because we grade them the same way. The authors went through the benchmarks the field actually competes on, the leaderboards models are tuned to climb, and found that nearly all of them give “I don’t know” exactly the same score as a wrong answer: zero. Under that rule, a model that always guesses will beat an otherwise-identical model that honestly flags its uncertainty. We are, in a fairly literal sense, scoring them into it.

This is the part worth keeping, because it runs against the usual headline. Hallucination is often sold as an inevitable, almost mystical limit of the technology. The paper disputes that on both counts. The pretraining floor is not a mystery — it is ordinary statistical error, the kind machine learning has understood for decades. And the persistence is not inevitable — it is, in part, an incentive we built and could change. A system that simply declined to answer when unsure would not hallucinate at all; the reason deployed models don’t behave that way is that our scoreboards punish the refusal.

What the authors did

The paper has three parts. First, a mathematical argument that some hallucination is statistically forced during pretraining, by showing that “generate only valid text” is at least as hard as a binary “is this statement valid?” classification problem. Second, an argument — backed by a survey of ten influential benchmarks — that mainstream accuracy-style metrics reward guessing over abstaining. Third, a proposed fix and a case study testing it: open-rubric evaluations, where the scoring is stated inside the question itself (for example, “a correct answer scores 1, a wrong one −1, so abstain if you are less than 50% sure”), so that a model can tell when honesty is being rewarded. They try this on four frontier models — Google’s Gemini 3 Pro, OpenAI’s GPT-5, xAI’s Grok 4 and Anthropic’s Claude Opus 4.5 — using SimpleQA’s 4,326 factual questions. They are explicit that the case study is illustrative, “not a controlled evaluation across models” (default settings, no tuning, no cost normalisation).

What they found

• Pretraining forces some error. The rate at which a model emits confident falsehoods is bounded from below by (roughly twice) the error rate of the best “is this statement valid?” classifier built from it. For facts with no learnable pattern, that floor is at least the singleton rate — the fraction of facts that appear exactly once in training. Some hallucination is unavoidable even with perfectly clean data.
• Grading rewards guessing — concretely. Under ordinary correct/incorrect scoring, never abstaining is the optimal strategy, and the authors’ survey finds the vast majority of popular benchmarks score “I don’t know” as simply wrong. A vivid example from their own side: on the SimpleQA test, raw accuracy slightly favours OpenAI’s o4-mini — which answers almost everything and is wrong more than three-quarters of the time — over GPT-5-mini, which makes far fewer mistakes because it abstains when unsure. The more reckless model looks better on the scoreboard.
• Open rubrics flip the incentive (in their case study). They test a simple hallucination mitigation (have the model answer twice and abstain if the two answers disagree). Under standard accuracy, the mitigation cuts errors but also cuts accuracy — so the metric discourages adopting it. Under open rubrics, the same mitigation comes out ahead for all four models across a range of penalties; and GPT-5-mini — which raw accuracy had penalised for abstaining when unsure — comes out ahead of o4-mini once the scoring is stated openly (n = 4,326 questions per model).

What this probably means

Cutting hallucination is mostly not a matter of inventing more hallucination-specific tests. It is a matter of changing how the mainstream benchmarks score uncertainty, so that admitting “I don’t know” is no longer punished. Until the scoreboard changes, reducing hallucination will keep costing models accuracy points and so keep being discouraged — which is why the authors frame the problem as “socio-technical”: part better metric, part getting the influential leaderboards to adopt it.

How strong is the evidence?

• The core is mathematics — formal lower bounds, not measurements. As a theoretical argument it is sound on its own terms.
• It rests on deliberately simplified models of the problem; the authors themselves flag the “false trichotomy” of treating every response as correct, incorrect, or “I don’t know,” and the idealised “arbitrary facts” setting used for the cleanest bound.
• The benchmark survey is a small, curated sample — ten influential evaluations, not an exhaustive audit.
• The case study is real but limited: four frontier models, a single mitigation, SimpleQA only, default settings, explicitly “not a controlled evaluation.” It is a proof of concept for the incentive argument, not a benchmark result.
• Worth naming the vantage point: three of the four authors are or were employed at OpenAI, and the paper argues the field should change how it evaluates models. That is a well-argued position from an interested party, not a neutral outside review — to weigh, not to dismiss. (To its credit, the paper points the critique at its own models, o4-mini and GPT-5-mini, as readily as at others.)

Why it matters

It reframes a heavily hyped problem. “Hallucination” tends to be sold either as a spooky defect or as an immovable wall; this paper makes it ordinary and partly self-inflicted — a statistical floor we can actually understand, sitting on top of an incentive we chose. The wider lesson is quieter and more useful: further progress on reliability may depend as much on what we measure as on what we build.

Clean summary

Confident false answers from language models come from two places. The first is statistical: when a fact has no pattern to learn, a model trained to imitate language will sometimes get it wrong, and that floor can be estimated (for instance, from how many facts appear only once in training). The second is incentives: almost every benchmark that models are ranked on scores “I don’t know” the same as a wrong answer, so guessing always wins — to the point that a model wrong three-quarters of the time can outscore a more honest one that abstains. The authors’ proposal is not another hallucination test but “open rubrics”: state the scoring inside the question. In a case study on four frontier models, that flips the incentive so a hallucination-reducing method is rewarded rather than penalised. It is a theory-plus-survey paper with a small, explicitly uncontrolled experiment, peer-reviewed in Nature; the fix is promising but not yet shown to work at scale, and hallucinations are argued to be neither mysterious nor strictly inevitable.