Embeddings — turning meaning into numbers

The previous lesson mapped each token to an arbitrary numeric ID — but that ID carries no meaning by itself. Token ID 1234 being numerically close to 1235 says nothing about whether the two tokens are related in meaning; IDs are just lookup keys into a vocabulary list, assigned in whatever order the vocabulary happened to be built. Something else has to turn a token into a representation that actually captures what it means.

An embedding is that representation — a list of numbers (a vector, commonly hundreds or thousands of numbers long) assigned to each token, learned during training such that tokens with related meanings end up numerically close to each other in that multi-dimensional space, and unrelated ones end up far apart.

Nobody manually assigns these numbers — they're learned during training, the same weight-adjustment process from earlier lessons, purely from how tokens actually co-occur across enormous amounts of text. "dog" and "puppy" end up close together because they appear in similar contexts across billions of sentences, not because anyone told the model they're related.

Because embeddings place similar meanings close together, you can turn "find documents about a similar topic" into a genuinely mathematical operation: embed a search query into the same numeric space, then find which stored documents' embeddings are numerically closest to it (using a distance measure like cosine similarity) — semantic search, which finds conceptually related results even when the exact words don't match at all. This is a completely different mechanism from traditional keyword search, and it's the specific capability that makes RAG (Retrieval-Augmented Generation, covered two lessons ahead) possible at all: retrieving relevant context by meaning, not by requiring exact keyword overlap.

Once you have millions of document embeddings, finding the closest ones to a query embedding by brute-force comparison becomes too slow to do naively. A vector database (Pinecone, pgvector, Weaviate, and others) is built specifically to index and search embeddings efficiently at scale — the infrastructure layer that makes semantic search and RAG practical in a real, latency-sensitive production system rather than only a research demo.

The same idea extends beyond single tokens: an entire sentence, paragraph, or document can be embedded as one vector representing its overall meaning, which is exactly the granularity RAG systems typically operate at — embedding whole chunks of a document, not individual words, so retrieval finds contextually relevant passages rather than isolated word matches.