Hallucination
Output nothing actually supports: invented facts, fake citations, a function that does not exist. Confident or hedged, it's still made up.
See it
What it is
A hallucination is output nothing supports: a fact, citation or API that the sources do not contain and the world does not contain either. It arrives confident more often than not, but a hedged 'I believe the method is called flushBuffer()' is just as invented. It is not the model lying, and it is not a bug in the usual sense: an LLM samples likely continuations, and nothing in that process consults reality. A fabricated citation comes off the same machinery as a real one. The classic sightings are invented API methods and npm packages that do not exist, and the 2023 Mata v. Avianca filing where a lawyer submitted six court cases the model had conjured whole.
You reduce it by removing the guesswork rather than by asking for accuracy. In rough order of leverage: put the source text in the context and require quoted spans (grounding, RAG), give it a tool for anything it cannot know, make 'I don't know' an explicitly allowed and praised answer, then check the result with evals or a second-pass verifier. Structured output with nullable fields helps too, because a required field is an instruction to produce something.
Gotcha: fluency reads as evidence and the model's stated confidence is not calibrated, so 'I am certain' costs it nothing. Turning temperature down definitely makes output more repeatable; whether it makes factual errors rarer depends on the model and the task, so measure it on your own cases instead of assuming either way. The expensive failures are rarely the wild ones; they are the answers that are 90 percent right with one wrong version number buried in the middle.
Ask AI for it
Rewrite this feature so answers are grounded instead of guessed. Put the retrieved source documents in the prompt with stable ids, and instruct the model: answer only from the sources below; every claim must be followed by the id of the source it came from; quote the exact supporting sentence; if the sources do not cover the question, reply with 'not found in the provided sources' and nothing else; never fill a gap from general knowledge. Return structured output with an answer field, a citations array of source ids and quoted spans, and a nullable unsupported_claims field. Then add a verification pass that checks each quoted span actually appears in the cited source, and fails the response if it does not.