Conversation memory and summarization strategies

A context window is finite, and a long-running conversation eventually hits its limit. What happens next — silently truncating, erroring out, or actually managing memory — is a real design decision, not something to discover in production.

Intermediate

3 min read

The context window is a hard ceiling, not a soft guideline

Every model has a maximum number of tokens it can accept in a single request — the system prompt, tool catalog, and full conversation history combined. Since every request in a stateless API resends the entire conversation so far, a long-running chat doesn't just get slower as it grows, it eventually stops being possible to send at all. A support conversation that runs 80 turns, or an agent session that racks up dozens of tool calls and their results, is a realistic way to actually hit this ceiling — not an edge case reserved for pathological inputs.

The naive failure mode

Turn 1..N: conversation grows, still fits, works fine
Turn N+1:  conversation + new message exceeds the context window
           -> API returns an error, mid-conversation, with no warning
           -> user sees a broken agent for no reason they can identify

Doing nothing isn't a neutral choice — it's choosing "fail abruptly and unpredictably once a session happens to get long enough," which is a genuinely bad experience precisely because it's invisible until it happens.

Strategy 1: sliding window (simplest, has a real cost)

Keep only the most recent N messages, dropping older ones as new ones arrive. Cheap to implement, cheap to run, and it silently discards anything the model said or was told outside the window — a user who established an important constraint 50 turns ago ("I'm allergic to shellfish") gets no guarantee the model still has it once that turn scrolls out of the window. Fine for conversations where old context genuinely stops mattering; risky for anything where an early detail needs to stay authoritative for the whole session.

Strategy 2: summarization (keeps meaning, loses some fidelity)

Periodically replace the oldest chunk of conversation with a model-generated summary of it, then keep appending new turns after the summary instead of the original messages.

Before: [msg 1] [msg 2] ... [msg 40] [msg 41] [msg 42]  (approaching limit)

After:  [summary of msgs 1-35] [msg 36] ... [msg 42]     (well under limit)

This preserves the gist of everything earlier — the shellfish allergy likely survives into the summary — while still shrinking the token count substantially. The real cost: summarization is itself a model call (more latency, more spend), and any detail the summarization step judged unimportant is now genuinely gone, not just deprioritized. A summary is lossy compression, and what gets lost is exactly the thing you find out about later.

Strategy 3: explicit memory as structured data, not raw transcript

The most reliable approach for anything that actually matters long-term (user preferences, account facts, decisions made) is pulling it out of the conversation entirely and into a real, structured store — a user_facts table, not a hope that it survives inside a summary. The agent's context then includes "known facts about this user" as a small, explicit, always-included block, separate from and much smaller than the raw conversation history. This is more engineering work upfront but doesn't degrade as the conversation grows, and it's auditable in a way a summary buried in message history isn't.

These strategies compose, they don't compete

A real system typically layers these: recent turns kept verbatim (sliding window) for immediate coherence, older turns compressed (summarization) for continuity, and durable facts extracted into structured storage (explicit memory) for anything that must never silently drop. Picking exactly one and assuming it covers every case is usually where this goes wrong.