Skip to content

Plain-language explainer

Embeddings, explained visually

An embedding is a list of numbers that represents a piece of text, an image or another item as a point in space, placed so that similar meanings sit close together. Language models learn one for every token, and search and RAG systems embed whole passages so they can find text by meaning instead of by matching words.

What is an embedding, and how does it capture meaning?

An embedding turns a piece of text into a list of numbers, a vector, positioned so that similar meanings sit close together. Meaning becomes geometry. Words and sentences that are used in similar ways end up near each other, even when they share no exact words. That is what lets a search box find the right help article from a different phrasing, and what lets retrieval pull the relevant document for an AI answer. The model learns these positions from how language is actually used.

Last reviewed

Do not just read it. Operate the mechanism yourself in a short interactive lesson.

See it work: Embeddings: meaning as coordinates →

Free, no code, no signup.

How text becomes an embedding

Inside a language model, the first step is a lookup table. Text is split into tokens, each token has an id, and each id points to one row of numbers. That row is the token's embedding. GPT-2 had a table of 50,257 rows with 768 numbers each, about 38.6 million parameters. Llama 3 8B has 128,256 rows of 4,096 numbers, about 525 million parameters, roughly 6.5% of the whole model.

Nobody fills in those numbers by hand. They start random and get adjusted during training, every time the model predicts a word. Words that show up in similar places, like 'dog' and 'puppy', get pushed toward similar numbers because that helps the predictions. Meaning ends up as position.

Search and RAG use a second kind: text embedding models. These read a whole sentence or paragraph and return one vector for all of it. Common sizes are 384 numbers for the small open model all-MiniLM-L6-v2, 1,536 for OpenAI's text-embedding-3-small and 3,072 for text-embedding-3-large.

A worked example: measuring similarity

Closeness between embeddings is usually measured with cosine similarity: how closely two vectors point in the same direction. It runs from 1 for the same direction, through 0 for unrelated, to minus 1 for opposite.

Here are three made-up embeddings with just three numbers each: dog [0.8, 0.3, 0.1], puppy [0.7, 0.4, 0.1] and invoice [0.1, 0.2, 0.9]. To compare dog and puppy, multiply matching numbers and add them: 0.56 + 0.12 + 0.01 = 0.69. Divide by the two lengths, 0.86 and 0.81, and you get 0.99. Dog and invoice come out at 0.29.

So 'dog' and 'puppy' point almost the same way, while 'invoice' points somewhere else, and the spelling had nothing to do with it. Real embeddings have hundreds or thousands of numbers, but the arithmetic is exactly this.

Cosine similarity between three toy embeddings. Closer to 1 means closer in meaning.
PairDot productLengthsCosine similarity
dog and puppy0.690.86 × 0.810.99, very close
dog and invoice0.230.86 × 0.930.29, far apart
puppy and invoice0.240.81 × 0.930.32, far apart

Why embeddings matter in real products

Retrieval-augmented generation, or RAG, runs on embeddings. Your documents are cut into chunks, each chunk is embedded once and stored. When a question arrives, it is embedded too, and the system fetches the chunks whose vectors sit closest. Those chunks go into the prompt, so the model answers from your documents.

Size turns into storage. One million chunks at 1,536 numbers each, stored as 4-byte floats, take about 6 GB. The same million at 384 numbers take about 1.5 GB. That is why teams often pick a smaller embedding model on purpose, and why OpenAI's newer embedding models let you cut vectors shorter while keeping most of their quality.

Embeddings also show how much structure training finds. In 2013, word2vec showed that the vector for 'king', minus 'man', plus 'woman', lands near 'queen'. Nobody programmed that relationship. It fell out of predicting words, and the same idea now sorts search results, recommendations and duplicate detection.

What people get wrong

  • Embeddings store the definition of a word. They store a position learned from usage, not a dictionary entry.
  • Matching means sharing keywords. Embeddings match on meaning, so 'car' and 'automobile' land close without overlapping letters.
  • One embedding is the true meaning of a word. Meaning shifts with context, which is what attention later adjusts.

Where you see it in real products

  • Semantic search and site search rank by vector closeness.
  • RAG uses embeddings to fetch the documents most related to a question.
  • Recommendations and dedup group items that sit near each other in vector space.

Common questions

What is the difference between a token and an embedding?
A token is a chunk of text with an id number, a symbol. An embedding is the list of numbers the model learned to attach to that id: a position in a space of meaning. The token says which word, the embedding carries what the model knows about it.
How many numbers are in one embedding?
Hundreds to a few thousand. Small retrieval models use 384 or 768 dimensions, large ones several thousand. More dimensions can hold finer distinctions and cost more to store and compare, which is why retrieval systems often pick a smaller model on purpose.
Can I compare embeddings from two different models?
No. Each model learns its own space, so the same sentence lands somewhere unrelated in another model's coordinates. If you switch embedding models, everything you had stored has to be embedded again.
What is cosine similarity?
Cosine similarity measures how closely two vectors point in the same direction, ignoring their length. It is 1 when they point the same way, 0 when they are unrelated and minus 1 when they are opposite. It is the most common way to decide which embeddings are close in meaning.
Are embeddings the same as a vector database?
No. An embedding is one vector that represents one piece of content. A vector database is where you store millions of them and search for the nearest ones quickly. The embedding model makes the vectors, the database finds the closest matches.

Related explainers

More in Foundations

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.