Context windows and why they're not infinite memory

This is the single most consequential fact about how LLM APIs actually work, and it surprises people used to thinking of "a conversation" as persistent: every API call is completely stateless. The model has no memory of a previous call — a "conversation" only appears continuous because the calling application resends the entire prior conversation as part of the context on every single new request.

This is architecturally the same statelessness principle covered in the Networking Fundamentals domain's HTTP lesson — no server-side session persists automatically; the client is responsible for resending whatever state matters.

The context window is the maximum number of tokens (input plus output combined, for most models) that can fit in a single request — a fixed architectural limit of the specific model, commonly ranging from tens of thousands to over a million tokens depending on the model, but always finite. Once a conversation's accumulated history exceeds that limit, something has to be cut, summarized, or dropped — the model cannot simply "remember more" beyond its window, no matter how the request is structured.

A larger context window sounds like it should solve everything, but it comes with real costs that matter in practice: processing more tokens costs more (pricing is per-token, from the tokens lesson), takes longer (more tokens to process before a response starts), and — a subtler, genuinely researched problem — very long contexts can suffer from "lost in the middle" behavior, where information buried in the middle of a huge context gets attended to less reliably than information near the beginning or end. A bigger window is a real capability, but not a substitute for actually managing what gets included.

Real applications manage context deliberately rather than just resending everything indefinitely:

Any real AI-powered feature has to actively manage what goes into the context window — deciding what conversation history to keep, what documents to retrieve and include, how much of the token budget to reserve for the model's own response. This is exactly the kind of context-management logic the AI Agents & LLM Integration domain's production-feature lessons build on top of directly.