How does a model see an image?

The image is cut into patches, each patch becomes a vector, and those vectors join the text tokens in the same sequence. From there attention treats them alike, which is how a model answers a question about a picture without a separate vision program.

Is it one model, or several stitched together?

Increasingly one. Early products bolted a captioner onto a text model. Current ones are trained on interleaved text and images from the start, which is what lets them reason about details in the picture rather than about a caption of it.

Can it read text inside an image?

Usually, and not always accurately. Clear headings and screenshots work well. Dense tables, handwriting and small print are where errors appear, so check anything you would otherwise have run through a real OCR tool.

Plain-language explainer

Multimodal AI, explained

How do images, audio, and documents become something a model can reason about?

Multimodal models turn every input, text, image, audio, or a screenshot, into vectors in one shared space, then reason over all of them together. An image is cut into patches and each patch becomes a vector, the same kind of vector a word becomes. Because they live in the same space, the model can compare a picture and a caption, answer a question about a chart, or describe what is on a screen. It is the same machinery as text, pointed at more kinds of input.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: Embeddings: meaning as coordinates →

Free, no code, no signup.

Then go deeper: Beyond text: images become tokens too Locked

What people get wrong

The model 'sees' like an eye. It converts pixels into vectors and reasons over those, not raw images.
Vision is a separate bolt-on model. Modern multimodal models share one representation across inputs.
It reads any image perfectly. Fine print, dense charts, and odd layouts still trip it up.

Where you see it in real products

Assistants answer questions about photos, screenshots, and PDFs.
Document AI extracts data from scans and forms.
Voice agents and computer-use agents build on multimodal understanding.

Common questions

How does a model see an image?: The image is cut into patches, each patch becomes a vector, and those vectors join the text tokens in the same sequence. From there attention treats them alike, which is how a model answers a question about a picture without a separate vision program.
Is it one model, or several stitched together?: Increasingly one. Early products bolted a captioner onto a text model. Current ones are trained on interleaved text and images from the start, which is what lets them reason about details in the picture rather than about a caption of it.
Can it read text inside an image?: Usually, and not always accurately. Clear headings and screenshots work well. Dense tables, handwriting and small print are where errors appear, so check anything you would otherwise have run through a real OCR tool.

Related explainers

Embeddings How LLMs work Computer-use agents Diffusion models Video and world models

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.