Does a mixture-of-experts model know more?

It can hold more parameters for the same running cost, which usually means more capacity for knowledge. But only a fraction of them run for any given token, so a sparse model with a very large total is not equivalent to a dense model of the same headline size.

Are the experts specialists in topics?

Not in the tidy way the name suggests. Routing is learned, not assigned, so what an expert ends up handling tends to be statistical patterns rather than the medicine expert. Some topical clustering does appear, but you cannot pick an expert by subject.

Why do labs quote two parameter counts?

One is the total parameters stored, the other is the active parameters used per token. Memory and hosting cost follow the total, compute per token follows the active count. Quoting only the larger number is how a sparse model gets to sound bigger than it runs.

Plain-language explainer

Mixture of Experts, explained

What is a Mixture of Experts model, and why do labs use it?

A Mixture of Experts model replaces some layers with many parallel sub-networks, the experts, plus a small router that picks a few of them for each token. The model can hold a huge number of parameters, but only the chosen experts run, so each token costs a fraction of the compute. That is the trick: the capacity of a giant model at something closer to the price of a small one. The router's choices are learned, not programmed.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: From the internet to your answer →

Free, no code, no signup.

Then go deeper: Mixture of Experts Locked

What people get wrong

The experts are subject specialists, one for law, one for medicine. Routing is learned per token and tends to track patterns like punctuation or syntax, not human topics.
An MoE model uses all its parameters on every request. Only the few experts the router picks actually run for each token.
More experts always means a better model. Balancing the router so experts stay evenly used is a real engineering problem.

Where you see it in real products

Model names like Mixtral 8x7B advertise the expert count right in the name.
Headlines about surprisingly cheap frontier models often trace back to MoE efficiency.
Pricing pages where a very large model costs less than its size suggests usually mean few parameters are active per token.

Common questions

Does a mixture-of-experts model know more?: It can hold more parameters for the same running cost, which usually means more capacity for knowledge. But only a fraction of them run for any given token, so a sparse model with a very large total is not equivalent to a dense model of the same headline size.
Are the experts specialists in topics?: Not in the tidy way the name suggests. Routing is learned, not assigned, so what an expert ends up handling tends to be statistical patterns rather than the medicine expert. Some topical clustering does appear, but you cannot pick an expert by subject.
Why do labs quote two parameter counts?: One is the total parameters stored, the other is the active parameters used per token. Memory and hosting cost follow the total, compute per token follows the active count. Quoting only the larger number is how a sparse model gets to sound bigger than it runs.

Related explainers

Model routing Quantization How LLMs work

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.