How AI chops text into tokens

What this lesson shows

Before meaning, text is split into tokens, common chunks from a fixed learned vocabulary.

The question it opens with

Why does an AI fumble 'how many r's in strawberry?', it isn't reading letters.

The walkthrough, in the lesson's own words

• Slice “strawberry” into the chunks you think the model stores.
• Here’s where the model actually cuts. Replay it slowly to watch.
• Your turn. Type anything and watch it split into tokens.
• Tokens, not letters, are what the model counts and what you pay for.
• Toy vocabulary and ids; real tokenizers cut differently.
• One idea to notice: a leading space or a capital letter is part of the token. So “strawberry”, “ strawberry”, and “Strawberry” each get a different id.
• Try açaí: non-English letters (accents, other scripts) often split into their own tokens, so the same meaning can cost more tokens in another language.
• Two things follow. The model can’t easily count letters hidden inside a token, the famous “count the r’s in strawberry” fail. Frontier models mostly get that word right now (the fail is so famous it’s in their training data), but the blindness remains: try a rare word and the count wobbles again. And since a request is billed per token, roughly 1 token ≈ ¾ of an English word, tokens, not characters, are the unit of cost.
• Where do these chunks come from? Before any model is trained, an algorithm called BPE reads mountains of text and keeps the chunks that show up most. Frequent words end up as one token; rare words get broken into pieces. That fixed list of chunks is the
• Yes. Most tokenizers learned their chunks from mostly-English text, so an everyday English word often lands as one clean token, while the same idea in Spanish, Hindi, or Chinese tends to shatter into more pieces. Roughly (real splits vary by tokenizer):
• Same meaning, more tokens. So the identical message can cost noticeably more, and eat more of the context budget, in Spanish than in English.
• The gap is narrowing: newer vocabularies (the o200k / Llama 3 class, 100K-200K chunks) learned from far more multilingual text, so non-English words shatter less than they used to. English still tends to be cheapest, just by a smaller margin.
• An AI charges per token and has a token limit, not a word limit. Why does the same 100 words cost more in code or another language than in plain English?
• Tokenization splits text into word-pieces, and code, rare words, and many non-English languages break into more tokens per word. More tokens means more cost and more of the context budget used, which is why the same idea can be cheaper or pricier depending on how it tokenizes.
• Slice the word, then see which pieces are real tokens.
• Every piece is a real token, the model could store this.
• One slice per letter? A single letter isn’t a token here.
• Amber pieces aren’t real tokens, the model only keeps chunks from its vocabulary.
• Each chunk becomes a number, its token id, which is all the model ever sees.

Key takeaway

The model's real alphabet is tokens, not letters, which explains spelling slips and why tokens are the unit of cost.

What you can do after this lesson

You can explain tokenization (BPE): text → tokens → ids, and why it explains spelling quirks and token costs.

Prefer reading first? This lesson has a companion explainer: Tokenization

Where it leads: Tokens become vectors, but what about images and sound?

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.