Video and world models

What this lesson shows

Video is the same denoising trick run over patches of space and time, so consistency is learned, never enforced. A world model goes further: it predicts the next state of a world conditioned on your actions, not the next word.

The question it opens with

AI video looks real for two seconds, then a hand grows a finger and the coffee refills itself. Why does it hold together, then drift?

The walkthrough, in the lesson's own words

• Drag noise down. Watch all four frames resolve at once.
• Scrub the clip. Which frame breaks the object apart?
• Pick an action. The world model predicts the next state.
• What changed when actions, not words, drove the prediction.
• A toy clip. A real video model denoises tens of thousands of little pixel blocks, not a handful of dots.
• One denoiser, run over frames at once. That is AI video.
• At full noise it is static. Pull the noise out.
• This is the denoising trick from the last lesson, run over little blocks of pixels that also span a few frames. Those blocks are called spacetime patches. Each frame walks from noise to a picture, and the model was trained to make the frames match. Nothing outside the model forces them to.
• A single image is chopped into small square patches. A clip adds a time axis, so a patch is a little block of pixels that also spans a few frames. The model denoises all of these patches together, which is why the frames move as one instead of flickering independently.
• You have seen these clips from Sora or Veo: perfect for two seconds, then physics slips.
• The clip looks smooth. Which frame quietly breaks the hand?
• By frame 4 the hand has grown a sixth finger. No rule in the model says a hand keeps five fingers, or that a cup that emptied stays empty. Consistency is only what it learned to imitate, so over enough frames it drifts.
• The model has no built-in idea of a hand, a cup, or a rule that objects persist. It only predicts patches that look like its training clips. Short clips drift little because the training clips were consistent; the longer it runs, the more small errors pile up.
• A text model predicts the next word. A world model predicts the next state of a world, given your action.
• A toy world with three actions. A real world model predicts full images or 3D scenes, one step ahead of your control.
• You reached the coin by choosing actions, and the model predicted where each one lands. Steerable prediction like this is what robots and game engines need: try an action in your head, see the result, then act for real.
• The prediction moved because YOU chose the action, not because a sentence came before it. That action-conditioning is the whole difference from a text model.
• A robot needs to imagine what happens if it moves an arm before it moves it. A world model lets it rehearse an action and see the predicted next state, so it can plan and avoid a mistake in the real world. Same idea powers playable, generated game worlds.
• You spotted why video drifts (nothing enforces object permanence between frames) and saw what changes when actions, not words, drive the prediction.
• AI video understands objects and physics, so it keeps them consistent.
• It denoises spacetime patches to imitate real clips. Consistency is learned, not enforced, so it drifts. A world model adds action-conditioning: it predicts the next state given what you do.
• A demo clip of someone juggling looks perfect for two seconds, then a ball vanishes and a hand warps. Using this lesson, explain why, and what a world model would add.
• The video is denoised patches of space and time trained to look like real clips, but nothing in the model enforces that a ball keeps existing or a hand keeps five fingers, so small errors accumulate and it drifts. A world model would go further and predict the next state conditioned on an action, so you could steer the scene and rehearse what happens next, which is why robotics and interactive worlds rely on it.
• Zoom back out to the 2026 model map, and you can now name the trick behind every family on it: next-token prediction for text, denoising for images and video, and action-conditioned next-state prediction for worlds.
• Start state. Pick an action to see the predicted next state.

Key takeaway

You spotted why video drifts, no rule holds objects together between frames, and saw what changes when actions, not words, drive the prediction.

What you can do after this lesson

You can explain how AI video is generated (denoising spacetime patches), why it drifts, and what a world model is: a next-state predictor conditioned on actions, which is why robotics cares.

Prefer reading first? This lesson has a companion explainer: Video and world models

Where it leads: Zoom back out to the 2026 model map, and you can now name the trick behind every family on it.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.