A real embedding has 768 numbers
A real embedding is a long list of numbers (768 in GPT-2 small), not two; similarity is the same dot product, now hundreds of cells wide.
1This is king the way the model really stores it. Not a dot, a list.your turn
On the map, king was one dot. Here is the same word, unpacked:
king
the ideaAn embedding is a long list of numbers, one per cell. The 2-D dot was only the first two. There is no magic, just a long list.
How long is the real list?
This strip shows 64 cells so you can read it. A real embedding in GPT-2 small is 768 numbers long. These cells are illustrative, not the model's actual values, but they behave like the real thing: similar words get similar lists.
→ continue← backR replay
A word is a long list of numbers. But how does raw text become the words we embed in the first place?
1.4 How AI chops text into tokensCommon questions
What is "A real embedding has 768 numbers" about?
A real embedding is a long list of numbers (768 in GPT-2 small), not two; similarity is the same dot product, now hundreds of cells wide.
What problem does it solve?
A flat 2-D map only has room for one nearest neighbour per word.
What will I be able to do after this lesson?
You can explain that an embedding is a long list of numbers and that similarity is the same dot product, scaled up to hundreds of dimensions.
What comes next?
A word is a long list of numbers. But how does raw text become the words we embed in the first place?