How AI chops text into tokens
Before meaning, text is split into tokens, common chunks from a fixed learned vocabulary.
1You see “strawberry” as 10 letters. How many pieces does the model see?your turn
what you see
strawberry
Tap how many pieces you think it becomes.
→ continue← backR replay
Tokens become vectors, but what about images and sound?
1.5 Beyond text: images become tokens tooBuilds on0.1How does an AI think?
Common questions
What is "How AI chops text into tokens" about?
Before meaning, text is split into tokens, common chunks from a fixed learned vocabulary.
What problem does it solve?
Why does an AI fumble 'how many r's in strawberry?', it isn't reading letters.
What will I be able to do after this lesson?
You can explain tokenization (BPE): text → tokens → ids, and why it explains spelling quirks and token costs.
What comes next?
Tokens become vectors, but what about images and sound?