The data flywheel

What this lesson shows

Production traffic, logged and curated, feeds evals and training, so your data becomes the moat.

The question it opens with

Two teams use the same base model. Why does one keep pulling ahead?

The walkthrough, in the lesson's own words

• Two teams start on the same model. Turn the crank: send your data once around the loop.
• Keep turning. Watch quality rise and the dataset compound.
• Now skip the labeling step and try to turn it again.
• The loop is the moat: your production data, compounding.
• Six stations, one loop. Turn the crank to send your data around once.
• The labeled failures become fine-tuning examples and new test cases, that is what turns logged mistakes into a better model.
• Invented quality numbers; real gains per loop are smaller and slower, but the compounding shape is real.
• Real usage is the fuel: the team that logs its traffic, mines hard cases, and feeds them back gets a loop the other can't buy, the flywheel.
• Each turn grows more: the better model attracts more usage, mining more hard cases, the loop compounds with every user.
• No labeled ground truth means nothing to add to evals or training, the gap can't be bridged, so the crank just jolts and stops.
• Drop any one station (no observability, no curation, no labels) and the whole flywheel seizes. It only compounds when every step turns, the puck can't skip a gap.
• Not all training data is logged from real users. Models also learn from synthetic data the team generates on purpose: a strong model writes examples, solves problems, or rewrites weak answers, and those become training data for the next model. Paired with the flywheel, it fills gaps real traffic never covers, rare cases, new features, hard edge cases, and lets you scale data without scaling users. The risk is a loop that feeds on itself: train only on a model's own output and small errors compound, so teams filter hard and keep real human data and evals in the mix.
• A team has almost no real traffic yet but needs training and eval data for a brand-new feature. What can they do besides wait for users?
• Generate synthetic data: have a strong model produce realistic inputs and gold answers, then have humans or trusted evals filter and verify a sample. It bootstraps the flywheel before real traffic exists, as long as you check it against real cases so the model isn't just learning its own quirks.
• Break the ring at one station and the loop can't close.
• The amber puck is your growing dataset, riding the loop.

Key takeaway

Data, not the base model, is the durable edge, and it compounds with every user.

What you can do after this lesson

You can explain the data flywheel and why production data is now the key advantage.

Prefer reading first? This lesson has a companion explainer: Data flywheel

Where it leads: Powerful agents in production also create a new way to get attacked.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.