Mixture of Experts: route a word to its experts

1. 1Drag top-k down. Predict how few experts a word can wake before answers get worse.
2. 2Watch the router score every expert and wake only the top-k highest scorers.
3. 3Grow the shelf from 8 experts to 16. Twice the knowledge, and the bill per word does not move.

A Mixture of Experts model replaces some of its layers with a shelf of parallel sub-networks, the experts, plus a small router. For each word, the router scores how well every expert fits, a dot product like the ones in attention, and wakes only the top few. The other experts stay asleep. They hold knowledge this word does not need right now.

The widget lets you turn the two knobs that matter. Top-k is how many experts a word wakes. Push it to one and answers get worse in real models; they settle around two of many, which is the sweet spot you will find yourself. The shelf size is how many experts exist. Grow it from 8 to 16 and you have doubled the knowledge on the shelf, but each word still wakes its best two, so the cost per word stays flat.

That decoupling is the whole trick. Spec sheets advertise both numbers: total parameters, the whole shelf, and active parameters, what one word wakes. A line like one trillion total, thirty-two billion active means exactly what you just did on the stage.

Most frontier models today are MoE inside. Fine-grained versions push the shelf much further: hundreds of small experts with a handful woken per token, plus one shared expert that stays on for the common patterns every word needs. DeepSeek V3 wakes 8 of 256 experts plus a shared one. It is how a model can hold frontier-scale knowledge while running at a fraction of a dense model's cost per word.

Paste this into any HTML page, course platform, wiki or slide tool that accepts an iframe. It runs with no account, sets no cookies of its own, and links back here.

Attribution is built in. If you write about it, a link to this page is all we ask.

Are the experts specialists in topics?

No. Routing is learned per token and tends to track patterns such as punctuation, numbers or syntax, not human subjects. The committee-of-chatbots picture is the most common misconception about MoE.

Does an MoE model use all its parameters on every request?

No. Only the experts the router picks for each token actually run. That is why the total parameter count and the active parameter count are different numbers, and why the active one is what sets the cost per word.

Can I embed this MoE widget on my site?

Yes. The iframe snippet on this page works in any HTML page, course platform or slide tool that accepts an iframe. It needs no account and links back to the full lesson.