How a large language model predicts the next word

Strip away every impressive-looking capability, and a large language model does exactly one thing: given a sequence of tokens so far, it outputs a probability distribution over every token in its vocabulary — literally a number for every possible next token, representing how likely each one is to come next — picks one, appends it to the sequence, and repeats the entire process to generate the one after that.

Text generation, code completion, answering questions, "reasoning" through a problem step by step — every one of these is this exact same next-token prediction loop running repeatedly. There is no separate "understanding" module distinct from this prediction process; the appearance of understanding is an emergent property of a large enough model having learned, from enormous amounts of text, extremely good statistics about what plausibly follows what.

The transformer architecture (what "GPT," and nearly every modern LLM, is built on) predicts each next token using an attention mechanism that lets the model weigh which earlier tokens in the context matter most for predicting the current one — dynamically, per prediction, rather than treating every earlier word equally. Predicting the word after "The trophy didn't fit in the suitcase because it was too" needs to weigh "trophy" heavily (it's the sensible referent for "it"), and largely ignore "suitcase" and "fit" — attention is the mechanism that lets the model learn to do exactly that kind of weighting, for arbitrarily complex relationships between distant tokens.

The model doesn't have to pick the single highest-probability token every time — sampling introduces controlled randomness into that choice, most directly controlled by a temperature parameter:

This directly explains why calling the same LLM API with the same prompt twice can produce genuinely different responses — the mechanism is deliberately stochastic by design, not a bug, and it's a real, tunable trade-off available in the API parameters of every major provider.

Because next-token prediction is fundamentally about plausibility, not verified truth, a model will confidently produce a fluent, statistically plausible-sounding token sequence even when it has no factual basis for the specific claim — it was never checking a fact, it was predicting what token most plausibly continues the sequence so far. This single insight — the model optimizes for "what comes next, statistically" rather than "what's true" — is the direct mechanistic root of the hallucination behavior the next lesson covers in depth.