A new research paper from OpenAI asks why large language models like GPT-5 and chatbots like ChatGPT still hallucinate, and whether anything can be done to reduce those hallucinations. In a blog post summarizing the paper, OpenAI defines hallucinations as plausible but false statements generated by language models. It acknowledges that despite improvements, hallucinations remain a fundamental challenge for all large language models, one that will never be completely eliminated.
To illustrate the point, researchers say that when they asked a widely used chatbot about the title of Adam Tauman Kalai’s Ph.D. dissertation, they got three different answers, all of them wrong. Kalai is one of the paper’s authors. They then asked about his birthday and received three different dates. Once again, all of them were wrong.
How can a chatbot be so wrong and sound so confident in its wrongness? The researchers suggest that hallucinations arise in part because of a pretraining process that focuses on getting models to correctly predict the next word, without true or false labels attached to the training statements. The models see only positive examples of fluent language and must approximate the overall distribution.
Spelling and parentheses follow consistent patterns, so errors there disappear with scale. But arbitrary low-frequency facts, like a pet’s birthday, cannot be predicted from patterns alone and hence lead to hallucinations.
The paper’s proposed solution focuses less on the initial pretraining process and more on how large language models are evaluated. It argues that the current evaluation models do not cause hallucinations themselves, but they set the wrong incentives.
The researchers compare these evaluations to the kind of multiple choice tests where random guessing makes sense because you might get lucky and be right, while leaving the answer blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say I don’t know.
The proposed solution is similar to tests like the SAT that include negative scoring for wrong answers or partial credit for leaving questions blank to discourage blind guessing. Similarly, OpenAI says model evaluations need to penalize confident errors more than you penalize uncertainty and give partial credit for appropriate expressions of uncertainty.
The researchers argue that it is not enough to introduce a few new uncertainty-aware tests on the side. Instead, the widely used, accuracy-based evaluations need to be updated so that their scoring discourages guessing. If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.