Model routing, explained

What exactly is a model router?

Every AI product that offers more than one model answers a small question thousands of times a day: which model should take this request? A model router is the piece that answers it. It sits in front of a family of models, inspects each incoming request, and picks which model serves it. The user sees one assistant. Behind the curtain, an easy question can go to a small fast model while a hard one goes to a frontier model.

The router itself is usually boring on purpose. It can be a handful of rules, a small trained classifier, or, in the newest products, the model family judging the difficulty of the request itself. Whatever the form, it has one job: make a good enough decision in milliseconds, for a tiny fraction of the cost of the call it is about to place. A router that costs as much as the calls it saves is not a router, it is overhead.

Two shapes cover most real systems. A router decides once, up front, and commits. A cascade tries the cheap model first, checks the answer, and escalates to a stronger model only when the check fails. Routing is faster because nothing is retried. Cascades are safer because a wrong guess gets caught. Mature systems often route first and keep a cascade as the safety net.

Why not just use the best model for everything?

Because you would pay frontier prices for work that does not need a frontier model. Real traffic is lopsided. A support bot sees password resets far more often than legal edge cases. A coding assistant renames variables more often than it designs systems. When most requests are easy, sending everything to the biggest model means overpaying on most of what your product does.

The price gap between tiers is what makes routing worth engineering. Inside each major provider's lineup, the small tier is roughly 5 to 10 times cheaper than the frontier tier, and industry write-ups in 2026 put the spread between a cheap workhorse and a state-of-the-art model at 10 to 20 times. Reasoning modes widen the gap further: a thinking model also generates many more output tokens per answer, and output tokens are the expensive ones.

Latency tiers the same way. Small models answer in well under a second, while a frontier model in thinking mode can take tens of seconds. For easy requests, routing down is not a sacrifice: you get a better experience, sooner, at a tenth of the price. The interactive lesson on this site lets you play the router yourself, triaging a stream of requests across tiers and watching cost, latency, and quality move together. The table shows the tier gaps as of mid-2026.

What routing strategies do teams actually use?

Four patterns cover the field, in rough order of sophistication.

• Rules. If the message is short and matches a known easy intent, use the small model. Rules are transparent, free to run, and capture the obvious wins. Most teams should start here.
• Learned routers. A small classifier trained on past traffic predicts whether the cheap model would be good enough for this request. RouteLLM, an open research project from the LMSYS group, reported cost cuts of up to roughly 85 percent on its benchmarks while keeping about 95 percent of frontier-model quality, by sending only the hard minority of queries upward.
• Cascades with fallback. Run the cheap model, verify the answer, escalate on failure. FrugalGPT, an early Stanford project, reported matching frontier quality on some tasks at a few percent of the cost this way. The catch is latency: an escalated request pays for two calls in a row, so the check has to be strict enough to fire rarely.
• Self-routing. The newest products let the model family judge its own difficulty. An auto mode reads the request and chooses between an instant variant and a thinking variant. That is a router wearing the model's own badge.

Where do you meet routing in products you already use?

ChatGPT is the loudest example. As of mid-2026 its default Auto mode is a router: each message is classified and served by a fast instant model or escalated to a thinking model, and auto-escalated messages have not even counted against the weekly quota for thinking. Getting there was rough. When GPT-5 launched with routing in 2025, misrouted requests made the product feel weaker than its parts, and user backlash pushed OpenAI to restore manual model pickers alongside Auto. Routing quality is product quality.

Other assistants ship the same idea as a fast-versus-thinking toggle or an effort setting, and coding tools quietly route by surface: autocomplete comes from a small low-latency model while the chat panel uses a large one.

Gateways turn routing into a product of its own. OpenRouter exposes hundreds of models behind one API and offers an auto option that picks a model per prompt, powered by the commercial router Not Diamond and billed at the chosen model's normal rate. Amazon Bedrock sells prompt routing between models of the same family, reporting cost cuts of up to about 30 percent in its tests. Most gateways also add the reliability flavor of routing: if a provider errors or times out, the request fails over to a backup.

How does a router decide, and what can go wrong?

Routers read cheap signals: how long the prompt is, what task it looks like (translate, summarize, debug, plan), whether tools or images are involved, how much conversation history rides along, and sometimes who is asking, since free tiers often route lower than paid ones. Learned routers add the strongest signal of all, the track record: on past traffic that looked like this, was the small model good enough?

The failure modes are predictable. The worst is the misrouted hard question: a request that reads as simple, gets the small model, and comes back as a confident wrong answer nobody flags. Second is inconsistency: different models have different voices and habits, so a routed product can feel like a different assistant from one message to the next. Third is the eval blind spot: if you evaluate models one at a time but production serves a mix, your dashboards describe a system no user actually experiences.

The defenses are unglamorous. Log which model served every request. Keep an escalation path the user can trigger, like a retry that forces the big model. And run your evals against the routed mix, not against the best model in isolation.

Should your team build a router or buy one?

Buy the plumbing first. A gateway that gives you one API over many models, per-request model choice, failover, and cost tracking removes most of the boring work. Whether to also buy the decision, letting a vendor's auto mode pick models for you, is a separate call: the vendor's definition of good enough was tuned on general traffic and its own cost structure, not on yours.

Build the decision when three things are true. Your traffic shows a clear easy-hard split. Your bill is large enough that a 40 percent cut pays for the engineering: routing a $50 monthly spend is a hobby, routing a $50,000 one is a job. And you own an eval set, because a router without evals is a cost cut you cannot bound.

Either way, start with rules over logged traffic and measure how often they misfire before training anything. A page of rules usually captures a large share of the win, and it builds the logging and eval habits a learned router needs anyway.

What people get wrong

Where you see it in real products

Common questions

Related explainers

More in Speed, cost and control