Cost and token budgeting for LLM features

LLM pricing is per-token, which sounds simple until an agent's actual per-request token count turns out to be five times what anyone estimated. Knowing what actually drives cost before shipping beats finding out from a surprise bill.

Beginner

3 min read

Tokens, not requests, are what gets billed

LLM APIs charge per token — roughly, per word-fragment — split into input tokens (everything sent: system prompt, tool catalog, conversation history, the new message) and output tokens (what the model generates back). Output tokens are typically priced several times higher than input tokens, because generating text is more computationally expensive than reading it. A single "request" isn't one price — its cost depends entirely on how much text went in and came out.

What actually drives cost in a real agent, ranked by how often it surprises people

1. Resent context, every single turn
   The system prompt, tool catalog, and full conversation history
   get sent again on every turn of a multi-turn conversation
   (covered in the prompt-caching lesson) — this is usually the
   single biggest line item, and the easiest to underestimate
   because it's invisible in a quick manual test with 2-3 messages.

2. Tool catalog size
   Every tool's name, description, and full parameter schema is
   sent on every request, whether or not that tool gets called.
   A catalog of 20 verbose tool definitions is real, recurring
   token weight before the conversation even starts.

3. Output length
   Priced higher per token than input, and less directly under
   your control — a model asked an open-ended question can
   generate a much longer answer than one asked a scoped question.

4. Images and other non-text input
   An image is converted into a number of tokens based on its
   resolution — an unresized multi-megabyte phone photo can cost
   far more than the text of an entire conversation around it.

Estimating cost before shipping, not after

The workable approach is a back-of-envelope calculation done deliberately, before launch: take a realistic system prompt + tool catalog token count, multiply by an expected number of turns per conversation, add expected output length, multiply by the per-token price, multiply by expected daily volume. This is rough by design — the point isn't precision, it's catching an order-of-magnitude problem (a tool catalog so large it dominates every request) before it's a production bill, not after.

The traps that turn a reasonable estimate into a surprising bill

A step-count cap that's too generous lets a single confused conversation loop through far more tool-call round-trips than intended, each one a full request. A model tier chosen for capability without checking its price tier — flagship models can cost several times more per token than a smaller model that would have been entirely sufficient for a narrow, well-scoped task. And retries: an endpoint that silently retries a failed call on error can turn one logical request into two or three billed ones without that multiplier ever showing up in a naive per-request estimate.

Levers, roughly in order of easiest to apply

Prompt caching for the stable prefix (often the single biggest win, covered in its own lesson). A smaller model for narrow, well-defined sub-tasks instead of routing everything through the most capable model available. A tighter step-count cap. Trimming a tool catalog to what's actually used rather than "every tool that might someday be useful." None of these require redesigning the feature — they're tuning knobs on a design that's already correct.