Hallucination mitigation and grounding

"Just use RAG" is not a complete answer to hallucination — retrieval can still fail, and a model can still ignore or misread what it retrieved. The real toolkit is several partial mitigations layered together, with an honest acknowledgment that none of them gets to zero.

Advanced

3 min read

Why hallucination isn't a bug in the usual sense

A model generates the statistically plausible next token given everything before it — it has no built-in mechanism that distinguishes "this is a fact I'm certain of" from "this is a fluent continuation that sounds right." A confident, well-formed, entirely fabricated citation and a confident, well-formed, correct one are produced by the same underlying process. This is why hallucination can't be patched away the way a normal software bug can — there's no flag to flip that makes the model "know when it doesn't know," because the model was never tracking that distinction to begin with.

Grounding: forcing the answer to trace back to something real

Retrieval (covered in its own lesson) is the main grounding mechanism — giving the model real, relevant source material to draw from instead of relying on parametric memory. But retrieval alone doesn't force the model to use what was retrieved; it can still answer from its own prior beliefs even with correct source material sitting right there in context. Two mechanisms that push further:

Required citations:
  system prompt/schema requires every factual claim to reference
  a specific retrieved source. A claim with no traceable source
  is a structural signal something's wrong, even before checking
  whether the claim is actually true.

Forced tool use for factual questions:
  the system prompt/routing logic requires a lookup tool to be
  called before answering certain categories of question at all,
  rather than trusting the model to decide on its own when it
  needs to check something instead of just answering.

Neither eliminates hallucination — a model can still cite a source that doesn't actually say what the claim asserts — but both convert a fully free-floating claim into one that's at least checkable, either by an eval harness or by a human spot-check.

Structured refusal as a deliberately engineered outcome

The strongest practical mitigation for out-of-scope or ungrounded questions isn't a better answer — it's a correct refusal, treated as a first-class expected output rather than a failure mode. A system prompt that explicitly states "if you don't have a specific source for this, say so rather than answering" gives the model an easy, sanctioned path to declining, which measurably reduces confident fabrication compared to a prompt that implicitly rewards always producing an answer. The evals lesson elsewhere in this domain covers detecting exactly this: distinguishing a correct decline from a confident, ungrounded answer with nothing backing it up.

Lower-level mechanisms that help at the margins

Lower sampling temperature reduces variance in wording but does not meaningfully reduce fabrication of facts — a common misconception, since a hallucinated fact can be generated just as confidently at temperature 0 as at temperature 1. Asking the model to quote the exact supporting text alongside a claim, rather than just naming a source, makes fabrication somewhat harder because the quote itself is falsifiable against the source. Chain-of-thought or explicit "state your confidence" prompting produces plausible-sounding confidence scores, but that confidence score is itself generated text, not a calibrated internal signal — treat it as a mild heuristic, not a reliable filter.

The honest limit

No combination of these techniques reduces hallucination to zero, and treating any single one as sufficient is itself a risk. For genuinely high-stakes outputs — anything feeding a financial, legal, or medical decision — the realistic mitigation is architectural: keep a human in the loop before consequential action is taken, the same propose-then-confirm boundary covered earlier in this domain, rather than trusting any prompt-level technique to fully close the gap.