Beyond text: images become tokens too
If a model only does math on token vectors, how can it 'see' an image or 'hear' audio?
The idea inside
Images and audio become vectors in the same space as text tokens.
After this lesson
You can explain how multimodal models turn images and audio into vectors in the same space as text, so the model compares them the same way it compares words.
Where it leads
Everything is vectors now. Prove you can place words yourself, then we predict the next word.
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Images and audio become vectors in the same space as text tokens.
The question it opens with
If a model only does math on token vectors, how can it 'see' an image or 'hear' audio?
The walkthrough, in the lesson's own words
- Tap the glowing patch to send it into the vector space. Then tap the rest.
- Find the matching patch with the dot product from 1.3.
- They land in the same space as the word vectors.
- A model needs a separate vision system bolted on to handle images.
- Image patches become vectors in the very same space as words, so one machine handles both.
- Words, image patches, audio frames, once they're all vectors, the model can compare any of them with the
- You paste a screenshot into a chatbot and ask a question about it in words, and it answers as if it read both together. How does one model handle a picture and text at once?
- The image is cut into patches and each becomes a vector in the same shared space as your words, so the model treats pixels and text as one stream of tokens. Nothing special joins them, they were always the same kind of vector.
- The whole question gets squeezed into one arrow too. How several word-vectors combine into one is Act 3's job.
- Invented scores that show the comparison, not measurements from a real model.
- No, this is just the comparison step. The real model runs attention + next-token prediction on top of these vectors to produce its answer.
- Scored. The cat patch wins with the highest dot-product score, words and pixels compare in one space. This is just the comparison step; the real model runs attention and next-token prediction to produce its answer.
Key takeaway
Multimodal models reuse the same trick: everything becomes vectors in one shared space.
What you can do after this lesson
You can explain how multimodal models turn images and audio into vectors in the same space as text, so the model compares them the same way it compares words.
Check yourself: How does a multimodal model 'see' an image?
- Patches of the image become vectors in the same space as text tokens(correct)
- It runs OCR and only reads text
- It guesses from the file name
- It can't, models are text-only
The image is cut into patches that become vectors in the same space as text tokens, so the model processes pixels and words with one machine.
Prefer reading first? This lesson has a companion explainer: Multimodal AI
Where it leads: Everything is vectors now. Prove you can place words yourself, then we predict the next word.
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.