Mixture of Experts
How to add parameters without paying for them every token?
The idea inside
A router sends each token to a few expert FFNs; sparsity decouples cost.
After this lesson
You can explain Mixture of Experts: route to a few experts, all knowledge, low cost.
Where it leads
The model is too big to fit or serve…
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
A router sends each token to a few expert FFNs; sparsity decouples cost.
The question it opens with
How to add parameters without paying for them every token?
The walkthrough, in the lesson's own words
- Drag top-k down. Predict: how low can the compute go before knowledge drops?
- The router scores each expert, then wakes only your top-k.
- Grow the shelf to 16 experts. Does the bill per word follow?
- Push to 1 expert and answers do get worse in real models. They settle around 2 of many, the sweet spot you found.
- The router is a receptionist: it scores how well each expert fits this word (a dot product, like attention's scores in 3.3), then wakes only the top-k highest scorers. The scores here are illustrative toy numbers, roughly what routing looks like, not measured.
- Each word only needs a couple of specialties at a time, grammar here, cooking there. The other experts hold knowledge THIS word does not need right now.
- You doubled the shelf: twice the knowledge. But each word still wakes only its best 2, so the bill per word didn't move.
- Cost is paid per word, and a word still wakes just 2 experts. Growing 8 experts to 16 took you from 2-of-8 to 2-of-16, twice the knowledge on the shelf, the same 2 doing the work.
- They push this shelf much further: fine-grained MoE means hundreds of small experts with only a handful woken per token, plus one shared expert that stays on for the common patterns every word needs. DeepSeek V3 wakes 8 of 256 experts plus 1 shared one; most frontier models today are MoE inside.
- Spec sheets advertise both numbers: total parameters, the whole shelf, and active parameters, what one word wakes. "1T total, 32B active" means exactly this.
- The experts are subject-matter specialists, a committee of chatbots: one for law, one for medicine, one for code.
- Routing happens inside one model, and experts split up token patterns (punctuation, numbers, syntax) that no human syllabus would recognize.
- Two models are advertised with the same huge parameter count, yet one is far cheaper and faster to run. How can a giant model still be cheap per word?
- A Mixture of Experts model routes each word to only its top few experts, so most of the parameters stay asleep on any given word. The total knowledge on the shelf is huge, but the compute paid per word is small, which is why an MoE can hold frontier-scale knowledge at a fraction of a dense model's run cost.
Key takeaway
More knowledge, same per-token cost.
What you can do after this lesson
You can explain Mixture of Experts: route to a few experts, all knowledge, low cost.
Prefer reading first? This lesson has a companion explainer: Mixture of Experts
Where it leads: The model is too big to fit or serve…
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.