A real embedding has 768 numbers
A flat 2-D map only has room for one nearest neighbour per word.
The idea inside
A real embedding is a long list of numbers (768 in GPT-2 small), not two; similarity is the same dot product, now hundreds of cells wide.
After this lesson
You can explain that an embedding is a long list of numbers and that similarity is the same dot product, scaled up to hundreds of dimensions.
Where it leads
A word is a long list of numbers. But how does raw text become the words we embed in the first place?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
A real embedding is a long list of numbers (768 in GPT-2 small), not two; similarity is the same dot product, now hundreds of cells wide.
The question it opens with
A flat 2-D map only has room for one nearest neighbour per word.
The walkthrough, in the lesson's own words
- This is king the way the model really stores it. Not a dot, a list.
- Lit cells are where the two lists agree. That count is the score.
- Whose list is more like king's? Pick one, then watch them line up.
- Pick any two words. Where do their lists overlap?
- Frontier models go wider still: several thousand numbers per token.
- A score below zero means the two lists point in opposite directions, like 1.3.
- In 1.3 you multiplied two coordinates and added: x₁·x₂ + y₁·y₂. Here you do the exact same multiply-and-add, just over all
- Each direction in this space can carry one concept: gender, royalty, water, money. With hundreds of directions, a word can be near many words at once, each along its own axis.
- One honest caveat: in a real model no single cell means royalty or gender on its own. Each feature is smeared across many cells at once (researchers call it superposition), so tidy one-concept regions like these are a teaching simplification.
- A word like “Apple” can mean the company or the fruit. How can one word vector sit near tech terms and fruit terms at the same time?
- An embedding is a list of hundreds of numbers, not 2, so a word can sit near tech terms along one set of cells and near fruit along another. With that many directions, one word can be close to very different neighbours at the same time.
Key takeaway
High dimensions are just a long list; the extra room lets a word sit near different words along different directions.
What you can do after this lesson
You can explain that an embedding is a long list of numbers and that similarity is the same dot product, scaled up to hundreds of dimensions.
Check yourself: Why does a real embedding use hundreds of numbers instead of two?
- So a word can be near different words along different directions(correct)
- To make the model run faster
- Because letters need one number each
- To store the word's spelling
A real embedding is a long list (768 in GPT-2 small), so one word can sit near different words along different directions, something a flat 2-D map can't do.
Where it leads: A word is a long list of numbers. But how does raw text become the words we embed in the first place?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.