Tokens — how text becomes numbers

A neural network only computes on numbers, never on words directly. Tokenization is the step that breaks text into pieces and maps each one to a number — and it's the direct, mechanical reason LLM pricing is per-token, context windows have hard limits, and certain prompts behave in surprising ways.

Intermediate

3 min read

Neural networks compute on numbers — text has to become numbers first

Every lesson so far described a neural network operating purely on numeric inputs. Text obviously isn't numbers — tokenization is the step that converts it, breaking a string into a sequence of tokens (not necessarily whole words) and mapping each one to a numeric ID from the model's fixed vocabulary.

"unbelievable" -> ["un", "believ", "able"]  (illustrative — real splits vary by tokenizer)
-> [1234, 5678, 91011]  (numeric IDs from the model's vocabulary)

Why tokens aren't words

Splitting on whole words sounds simpler, but breaks badly for languages without spaces, made-up words, typos, and the sheer number of possible words in any language (a vocabulary would need to be enormous to cover every real and possible word directly). Modern tokenizers instead learn a vocabulary of common subword pieces from massive amounts of text — common whole words often do end up as single tokens, but rarer or unfamiliar words get split into smaller, more common fragments the model has actually seen often enough to have learned something about.

This directly explains several things that otherwise look arbitrary:

Different languages tokenize at different densities — languages underrepresented in a model's training data often split into noticeably more tokens per word, meaning the same sentence can cost more (and eat more of the context window) in one language than another.
Uncommon words and typos split into more, smaller tokens — a rare technical term, a name, or a typo often becomes several small fragments rather than one clean token, since the tokenizer never learned it as a common whole piece.

Why LLM pricing and limits are measured in tokens, not characters or words

Every practical constraint of an LLM API is expressed in tokens because tokens are the model's actual unit of computation and memory — not an arbitrary billing choice:

API pricing is per-token (input tokens and output tokens, often priced differently), because processing each token costs real compute during inference.
Context window limits (the next lesson) are measured in tokens, because that's literally the sequence length the model's architecture can attend to at once.
Output length limits are token-based for the same underlying reason.

A concrete, practical consequence: estimating cost and fit before you call the API

Because pricing and context limits are token-based, any real system calling an LLM API needs to estimate token count before sending a request — to predict cost, and to make sure a prompt plus its expected response will actually fit within the model's context window. Every major LLM provider publishes an official tokenizer library specifically so applications can count tokens locally, without an API call, before committing to one:

# Illustrative — the exact library differs per model provider
tokens = tokenizer.encode("How many tokens is this sentence?")
print(len(tokens))  # a real integer, used for cost/fit estimation before calling the API

Tokens — how text becomes numbers

Intermediate

3 min read

Neural networks compute on numbers — text has to become numbers first

"unbelievable" -> ["un", "believ", "able"]  (illustrative — real splits vary by tokenizer)
-> [1234, 5678, 91011]  (numeric IDs from the model's vocabulary)

Why tokens aren't words

This directly explains several things that otherwise look arbitrary:

Different languages tokenize at different densities — languages underrepresented in a model's training data often split into noticeably more tokens per word, meaning the same sentence can cost more (and eat more of the context window) in one language than another.
Uncommon words and typos split into more, smaller tokens — a rare technical term, a name, or a typo often becomes several small fragments rather than one clean token, since the tokenizer never learned it as a common whole piece.

Why LLM pricing and limits are measured in tokens, not characters or words

Every practical constraint of an LLM API is expressed in tokens because tokens are the model's actual unit of computation and memory — not an arbitrary billing choice:

API pricing is per-token (input tokens and output tokens, often priced differently), because processing each token costs real compute during inference.
Context window limits (the next lesson) are measured in tokens, because that's literally the sequence length the model's architecture can attend to at once.
Output length limits are token-based for the same underlying reason.

A concrete, practical consequence: estimating cost and fit before you call the API

# Illustrative — the exact library differs per model provider
tokens = tokenizer.encode("How many tokens is this sentence?")
print(len(tokens))  # a real integer, used for cost/fit estimation before calling the API

Tokens — how text becomes numbers

Neural networks compute on numbers — text has to become numbers first

Why tokens aren't words

Why LLM pricing and limits are measured in tokens, not characters or words

A concrete, practical consequence: estimating cost and fit before you call the API

Further reading

Tokens — how text becomes numbers

Neural networks compute on numbers — text has to become numbers first

Why tokens aren't words

Why LLM pricing and limits are measured in tokens, not characters or words

A concrete, practical consequence: estimating cost and fit before you call the API

Further reading