LLM Routing: Model Cascades, RouteLLM and FrugalGPT (2026)

Most production LLM applications send every request to one model, and that model is almost always the one that performed best in the demo. It is a comfortable default and an expensive one. A large share of real traffic, such as classification, extraction, short rewrites and FAQ-style questions, is answered equally well by a model that costs a fraction as much. You are paying a frontier premium on queries that never needed it.

LLM routing is the discipline of deciding, per request, which model should answer. It sits between your application and a pool of models, and it turns a fixed per-token bill into a quantity you can tune against a quality target. Two research lines anchor the field: FrugalGPT, which popularized the cascade, and RouteLLM, which trains routers from human preference data. Gateways such as LiteLLM, OpenRouter and Portkey now ship routing as a product feature.

This post separates what the papers actually show from what vendors imply. You will leave with a taxonomy, worked cost arithmetic, the quality signals that make deferral decisions possible, and a runnable Python harness.

What this covers: why one model per app wastes money, the four routing families, the cost-quality Pareto frontier, router training, quality estimation, latency, gateways, evaluation, and failure modes.

Context and Background

Per-token pricing for hosted models spans orders of magnitude. The FrugalGPT authors, Lingjiao Chen, Matei Zaharia and James Zou, opened their 2023 paper with exactly this observation: fees for popular APIs of the time, including GPT-4, ChatGPT and J1-Jumbo, differed by up to two orders of magnitude. That gap has not closed. Every major provider now sells a tiered family, a large flagship, a mid-size workhorse and a small, fast model, and the price ratio between the top and bottom tier is routinely an order of magnitude or more. Check current price sheets before you size anything, because they change often.

If quality were uniform across a workload, you would simply pick the cheapest tier that passes. It is not uniform. Query difficulty is heavy-tailed: most prompts are easy, a minority are hard, and the hard minority is where the strong model earns its price. The economic opportunity is the gap between the average difficulty your traffic presents and the maximum difficulty you provisioned for.

Three strategies exist for exploiting that gap, and the FrugalGPT paper names them cleanly. Prompt adaptation shortens or restructures prompts to reduce tokens. LLM approximation replaces calls with cheaper substitutes, such as completion caches or distilled models. LLM cascades try models in sequence and stop when an answer looks good enough. Routing is the generalization of the third idea: instead of always starting at the cheapest model, a router may predict up front which model to use.

Routing is not the only lever, and it composes with the others. If your traffic repeats, a semantic cache for LLM responses removes calls entirely, which beats any routing decision. If your prompts share long prefixes, prompt caching economics lowers the price of the calls you do make. Routing handles what is left: unique prompts of varying difficulty.

The research baseline matters because gateways describe their features loosely. For the primary sources, read the RouteLLM paper on arXiv and the FrugalGPT paper. The rest of this post cites them where numbers appear, and labels everything else as illustrative.

A Taxonomy of LLM Routing: Rules, Learned Routers, Cascades and LLM-as-Router

LLM routing falls into four families that differ in when the decision is made and what evidence it uses: rules decide from request metadata before any model runs, learned routers predict difficulty from the prompt text, cascades decide after seeing a cheap answer, and LLM-as-router asks a model to choose. Most mature systems combine two or more of these.

Figure 1: A gateway hosts one or more routing policies in front of a model pool, and every path feeds the same cost ledger.

The diagram shows the gateway as the single enforcement point. Rule-based routing sends traffic to a cheap model directly, a learned router picks from a pool, and a cascade starts cheap and defers to a stronger model when a verifier rejects the answer. All paths write to one log, which is what makes later evaluation possible.

Rule-based routing

Rules are the baseline you should build first. Route by tenant plan, endpoint, task type, prompt length, language, or a regular expression on the input. Portkey’s conditional routing is a clean example: a gateway config declares strategy.mode as conditional, each condition has a query over metadata.<key>, params.<key> or url.pathname with operators like $eq, $in, $gt and $regex, and a then naming the target. A default target catches everything else, and conditions are checked in order, so specific rules go first. Per Portkey’s documentation, a condition referencing a missing key evaluates to false rather than raising an error.

Rules cost nothing at runtime, have no failure mode beyond misconfiguration, and are fully explainable. Their weakness is that they know the label of a request, not its difficulty. A “summarize” endpoint may receive a two-line note and a 60-page contract. Rules also rot: every new use case needs a human to write a new clause.

Learned routers

A learned router predicts, from the prompt alone, whether a weak model will be good enough. RouteLLM is the reference design. It frames the problem as binary: a strong model and a weak model, with a router that outputs a score for how much the strong model is needed. A threshold converts the score into a decision, and moving the threshold slides you along the cost-quality curve.

The decision is made before any generation, so a learned router adds one small inference to the critical path and no wasted generation for hard queries. The price is that it must predict difficulty from text alone, which is intrinsically noisy. Section four covers how these are trained.

Cascades

A cascade, the FrugalGPT approach, calls the cheapest model first and inspects the result. If a scorer judges the answer reliable, it is returned. Otherwise the query is deferred to the next model. FrugalGPT’s cascade learns which models to chain and what acceptance thresholds to apply per stage. The paper’s headline claim is that the learned cascade can match GPT-4’s performance with up to 98% lower cost, or improve accuracy by 4% at the same cost, on the benchmarks the authors tested. Treat 98% as a best case on specific datasets and a 2023 price list, not a planning number.

Cascades see evidence a pre-generation router cannot: the actual answer. The cost is that deferred queries pay twice, and the user waits through both stages. AutoMix, from Aggarwal and colleagues, is a closely related design: the small model verifies its own answer few-shot, and a POMDP-based router decides whether to escalate, with the authors reporting cost reductions of over 50% at comparable performance across five models and five datasets.

LLM-as-router

The fourth family uses a language model to classify the request, either a small model producing a label or a larger one reasoning about which specialist to call. LiteLLM’s auto routing documentation lists an LLM classifier among its options, next to heuristics, lexical or semantic keyword rules and custom classifier plugins. It is flexible and can use rich instructions, but it adds a full generation to the critical path and inherits all the biases of LLM judgment. Use it when the routing taxonomy is semantic and changes often, for example selecting among domain specialists, and avoid it for pure cost control where a trained classifier is cheaper.

Which family when

Family	Decision time	Evidence	Added latency	Added cost	Best for
Rules	Before call	Metadata	~0	~0	Known task types, tenant tiers
Learned router	Before call	Prompt text	Small classifier pass	Tiny	Mixed traffic, strong-vs-weak choice
Cascade	After cheap call	Answer and signals	Full cheap call plus verify	Cheap call plus verifier on every query	Tasks with checkable answers
LLM-as-router	Before call	Prompt plus instructions	A full small generation	Small-model call per request	Semantic, evolving taxonomies

The matrix is qualitative on purpose. Real numbers depend on your models, your prompts and your traffic, which is why the evaluation section insists on measuring your own frontier.

The Cost-Quality Pareto Frontier, With Worked Arithmetic

Every routing policy is a point, or a curve, in a plane with cost on one axis and quality on the other. A policy is useful only if it lies on or near the Pareto frontier: no other configuration is both cheaper and better. The two endpoints are fixed by your models: all-weak sits at low cost and low quality, all-strong at high cost and high quality. Routing buys the region between them that a straight line would not give you.

The key insight is concavity. If the router is better than random, the curve bulges above the straight line between the endpoints, meaning you retain more quality than the fraction of strong-model calls would suggest. RouteLLM’s evaluation uses exactly this view: the x-axis is the percentage of calls sent to the strong model, the y-axis is benchmark performance, and a router is judged by how far it sits above the random-routing diagonal.

Worked arithmetic (illustrative prices)

The following numbers are illustrative, not quotes from any provider. Assume a strong model at a blended $10 per million tokens and a weak model at $0.50 per million. Assume an average request uses 1,000 tokens, so the strong model costs $0.010 per request and the weak model $0.0005. At 1 million requests a month:

All-strong: 1,000,000 x $0.010 = $10,000.
All-weak: 1,000,000 x $0.0005 = $500.

Router case. A learned router sends fraction p = 0.30 of traffic to the strong model. Expected cost per request is 0.30 x 0.010 + 0.70 x 0.0005 = 0.003 + 0.00035 = $0.00335, so $3,350 per million requests. That is a 66.5% reduction from all-strong. If the router itself costs $0.00002 per request (a small embedding or classifier pass), add $20, and the total is $3,370.

Cascade case. Every request calls the weak model, a verifier costs $0.0001 per request, and a fraction e = 0.30 escalates to the strong model. Expected cost is 0.0005 + 0.0001 + 0.30 x 0.010 = $0.0036, or $3,600 per million. The cascade costs slightly more than the router here because deferred queries pay for the cheap call and the verifier on top of the strong call, and every query pays the verifier.

The quality side. Suppose the strong model scores 0.90 on your eval and the weak model 0.70. The random-routing baseline at p = 0.30 gives 0.30 x 0.90 + 0.70 x 0.70 = 0.76. A good router that sends the queries that most need the strong model might reach 0.86, which is 95.6% of the strong model’s score at 33.5% of its cost. Whether you can actually reach 0.86 is an empirical question about your data. The point of the arithmetic is the shape: cost falls linearly in p, while quality falls slower than linearly if the router has signal.

The break-even rule for cascades

A cascade beats always-strong only if the average cost of the cheap path plus expected escalations is lower. Writing c_w for the weak cost, c_v for the verifier cost, c_s for the strong cost and e for the escalation rate, the condition is c_w + c_v + e x c_s < c_s, which rearranges to e < 1 – (c_w + c_v) / c_s. With the numbers above, e must stay below 1 – 0.0006 / 0.010 = 0.94. That is generous. The binding constraint is not cost but latency and quality: as e approaches 50% or more, you are making users wait through two calls on half the traffic, and the case for a router that avoids the wasted call strengthens.

Compare that to the router break-even. A router that costs c_r per request and sends fraction p to the strong model beats always-strong when c_r + p x c_s + (1 – p) x c_w < c_s. Because c_r is tiny relative to c_s, almost any p below roughly 0.9 saves money. The real question is always quality retention, not cost.

Reading published numbers honestly

Published savings are measured against a baseline, and the baseline changes the headline. RouteLLM’s LMSYS write-up reports, at 95% of GPT-4 performance, that on MT Bench a matrix-factorization router trained on Arena data needed 26% GPT-4 calls, roughly a 48% saving over a random router, and that adding LLM-judge-labeled augmentation lowered that to 14% GPT-4 calls, about 75% over random. The headline figures of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K are measured against always using GPT-4 and use the augmented routers. On MMLU, the authors note that routers trained only on Arena data performed near random, which they attribute to out-of-distribution questions, and that a small augmentation (about 1,500 golden-label samples, under 2% of training data) fixed it. The lesson is not that routers are magic. It is that routers generalize only as far as their training distribution, a point we return to in the failure modes section.

Training a Router From Preference Data

RouteLLM trains routers on human preference data of the kind Chatbot Arena collects: a prompt, two anonymous model responses, and a human vote of win, loss or tie. The trick is that this data was never collected to train routers, yet it contains exactly the signal a router needs, namely which prompts a stronger model wins on and which ones a weaker model holds its own on.

Figure 2: Preference pairs become win-rate labels, a router learns to score “needs the strong model”, and a calibration set converts that score into a threshold for a target strong-model share.

The pipeline in the figure has three stages. First, preference pairs are converted into labels for a particular strong-weak pair. Second, a model learns to predict the probability that the strong model wins on a given prompt. Third, a threshold is chosen on a calibration set so that a desired fraction of traffic goes to the strong model. Splitting these stages matters operationally: retraining the scorer and recalibrating the threshold are separate jobs.

The four router architectures

The RouteLLM release describes four trained routers plus a random baseline. The LMSYS write-up lists similarity-weighted ranking, matrix factorization, a BERT classifier and a causal-LLM classifier. The repository exposes them as sw_ranking, mf, bert and causal_llm, with mf marked as recommended.

Similarity-weighted ranking embeds the incoming prompt, finds similar prompts in the Arena data, and computes a weighted win rate from the neighbors. It is a nonparametric method, simple to reason about and expensive in proportion to the dataset size at inference. Matrix factorization learns a latent space where each model and each prompt have vectors, and a bilinear score predicts model strength on that prompt. It is cheap at inference and is the repository’s recommended default. The BERT classifier fine-tunes an encoder to predict the label directly. The causal-LLM router fine-tunes a small decoder on the same task, which gives the most capacity and the highest inference cost of the four.

All of this assumes a pair. RouteLLM’s design is two-model; multi-model routing is a different problem, the one RouterBench addresses with a dataset of over 405,000 inference outcomes from a range of LLMs and a framework for comparing multi-LLM routers. If your pool has five models, you either cascade pairwise routers or train a classifier over model identity, and you should read RouterBench before inventing your own evaluation.

Augmentation, and why it matters more than architecture

The write-up reports that routers trained on Arena data alone did poorly on MMLU, performing close to random. Adding labeled data from the target distribution fixed most of the gap. For MT Bench the authors used an LLM judge to label additional samples. For MMLU they used about 1,500 golden-label samples from the validation split. In plain terms, the router needs to have seen something like your traffic.

This is the single most actionable finding for practitioners. Do not expect an off-the-shelf router trained on public chat preferences to know that your legal-clause extraction task is hard for a small model. Collect your own labels. The cheapest way is to run both models on a sample of your traffic, grade with an LLM judge or ground truth where it exists, and train on the resulting win/loss labels. The companion post on LLM-as-a-judge evaluation, bias and calibration covers how to make those labels trustworthy.

Threshold calibration

A router outputs a score, not a decision. RouteLLM includes a calibration command: python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml computes the threshold that sends a requested share of a calibration set to the strong model, and the repository’s example output reads “For 50.0% strong model calls for mf, threshold = 0.11593”. You then address the router through the model name, as in router-mf-0.11593. The documentation page showed inconsistent threshold values between sections when fetched, which is a small reminder to recompute your own on your own data instead of copying an example.

Calibrate on data that looks like production. A threshold tuned on an Arena-like distribution will yield a different strong-model share on a traffic mix dominated by short transactional prompts. Re-check the realized share weekly and treat a drift in that share as an alert, because it is the earliest visible sign of distribution shift.

Generalizing across model pairs

A notable claim from the paper is transfer: routers trained for one strong-weak pair kept working when the pair was swapped at test time. The LMSYS write-up evaluates this by testing Claude 3 Opus as the strong model and Llama 3 8B as the weak model without retraining, reporting that results remained strong on MT Bench. The generalization test covers one benchmark, so read it as encouraging evidence, not a guarantee. The intuition is that the router learns prompt difficulty, a property that is partly model-independent. It will break down when the new weak model has a very different capability profile, such as a code specialist.

Quality Estimation Signals for Cascades and Deferral

A cascade lives or dies on one component: the function that looks at a cheap answer and decides whether to trust it. There are four families of signal, with very different cost and reliability.

Figure 3: In a cascade the gateway calls the weak tier, scores the answer with a verifier, and either returns it or defers to the strong tier.

Token log probabilities

Many APIs return per-token log probabilities. The mean or minimum token probability, or the entropy of the first few tokens, is a free confidence signal. For short structured outputs such as classification labels, the probability of the chosen label is a strong, well-behaved signal. For long free-form text it is weak: fluent text can be confidently wrong, and calibration varies by model and by whether post-training has flattened the distribution. Availability is also uneven, since not every provider exposes logprobs for every model, so verify this against the API you actually use.

Self-consistency

Sample the cheap model several times and measure agreement. If five samples return the same answer, confidence is high; if they scatter, defer. This works well for tasks with a discrete answer, like math or extraction, and poorly for open-ended generation where there is no single answer to agree on. The cost is real: n samples multiply the cheap-tier spend, so the cheap tier must stay far cheaper than the strong tier even after multiplication.

Self-verification and judge models

AutoMix uses few-shot self-verification, where the small model estimates whether its own answer is supported by the context, and then handles the noise in that estimate with a POMDP-based router. The paper reports its approach outperforming strong baselines and cutting cost by more than 50% at comparable performance across the five models and five datasets it studied. A separate judge model can do the same job with a rubric. Judges inherit the bias catalog described in the judge post: position bias in pairwise settings, verbosity preference, and self-preference when the judge shares a family with the answerer.

Task-specific verifiers

Where you can, replace learned signals with hard checks. Does the JSON parse and match the schema? Do generated unit tests pass? Does the SQL execute and return rows? Does the extracted date appear in the source text? A deterministic verifier is cheaper and far more trustworthy than any probabilistic one, and it is the reason cascades shine in structured workloads. Order your checks from cheapest to most expensive and short-circuit.

Learning the deferral rule

FrugalGPT’s contribution beyond the cascade idea is a learned scorer and per-stage thresholds that are optimized jointly for cost and accuracy. The practical translation: do not hand-set “defer if confidence below 0.8”. Collect (answer, correctness) pairs from your traffic, fit a small calibrated classifier on your chosen signals, and choose the threshold from a cost-quality curve. Hand-set thresholds are the most common source of silently degraded quality.

Latency, Caching and the Gateway Layer

Cost is the headline benefit, but latency decides whether a cascade is acceptable to users.

Latency arithmetic for cascades

All numbers here are illustrative. Suppose the weak model takes 1.0 s to answer, the verifier 0.4 s, and the strong model 3.0 s. Strong-only gives a flat 3.0 s. A cascade with a 30% escalation rate has a mean latency of 1.0 + 0.4 + 0.30 x 3.0 = 2.3 s, a mean improvement, since most users get a fast answer. But the escalated 30% experience 1.0 + 0.4 + 3.0 = 4.4 s, so the p95 and p99 are worse than strong-only. If your SLO is on tail latency, a cascade can violate it while improving the mean.

Mitigations exist. Run the verifier concurrently with the start of the strong call (speculative escalation) and cancel the strong call if the verifier accepts, which trades extra cost for tail latency. Stream the weak answer to the user while verification runs and only replace it when deferral fires, which only suits UIs that tolerate a visible revision. Or use a pre-generation router for latency-critical paths and reserve cascades for batch and asynchronous workloads where the extra second does not matter.

Streaming and tool calls

Cascades are awkward with streaming, because you cannot verify a response you have not finished generating. Routers do not have this problem since the decision precedes generation. Tool-calling agents add another wrinkle: switching models mid-conversation changes tool-call formats and behavior, so sticky routing within a session is usually wise. OpenRouter’s auto router documents this idea directly as session stickiness: it prefers the model a conversation already used, as long as that model still ranks well for the new prompt.

Interplay with caching

Order of operations matters. Put the exact-match and semantic caches in front of the router, since a cache hit costs nothing and needs no decision. Prompt caching changes the economics of the router itself: switching models in a conversation discards the provider-side prefix cache, which can erase the savings of a cheaper model if the prefix is large. Count cache-miss cost when you compute the benefit of a switch. Our post on prompt caching economics walks through the break-even math.

Gateways: what the products say

The three gateways named in this post take different approaches. The facts below come from each vendor’s documentation as read for this article; check the docs for current behavior since these features are changing quickly.

LiteLLM documents an Auto Routing feature, marked beta, described as one router for complexity, semantic and adaptive routing. Each request is classified, using heuristics, an LLM classifier, lexical or semantic keyword rules or a custom classifier plugin, and each tier can route to a pinned model, a random pool or a Thompson-sampled pool. The documentation warns that config keys and defaults may change between releases.

OpenRouter takes a market-signal approach. Its Auto Router classifies a prompt into one of about 30 task types, ranks models by recent community spend for that task type over a trailing seven-day window, applies a cost_tier setting from low to max, and returns a primary model plus fallbacks. It supports allowed_models and excluded_models patterns. You pay the selected model’s standard rate with no router fee, according to its documentation. Note what this does not do: it optimizes for what other users chose, not for quality measured on your task.

Portkey provides deterministic conditional routing over request metadata and parameters, as described earlier. It is the best fit when your routing logic is business logic: paid versus free tenants, regions, or model allow-lists.

A reasonable architecture layers them: deterministic rules for policy, a learned router or cascade for difficulty, and a fallback chain for provider errors. Keep availability fallback separate from quality routing in your mental model and your metrics, because they fail differently.

A Runnable Router and Cascade Harness

Theory is cheap; a harness you can poke at is better. The script below simulates a weak and a strong model on traffic with heavy-tailed difficulty, then sweeps a router threshold and a cascade confidence threshold. It needs only the Python standard library. Every number it prints is a property of the simulation, not a benchmark of any real model, and the simulation exists to show the shape of the trade-off and to give you a skeleton to wire to real APIs.

"""Minimal router + cascade harness with a simulated model pool (no API keys needed)."""
import random
from dataclasses import dataclass

random.seed(7)

@dataclass
class Model:
    name: str
    cost: float      # dollars per request (illustrative)
    skill: float     # ability on a 0..1 difficulty scale

    def answer(self, difficulty):
        # P(correct) falls as difficulty exceeds skill
        p = 1.0 if difficulty <= self.skill else max(0.0, 1 - 3 * (difficulty - self.skill))
        ok = random.random() < p
        conf = min(1.0, max(0.0, p + random.gauss(0, 0.12)))  # noisy confidence signal
        return ok, conf

WEAK = Model("weak", 0.0005, 0.55)
STRONG = Model("strong", 0.0100, 0.95)

def make_traffic(n):
    # heavy-tailed difficulty: most easy, few hard
    return [min(1.0, random.betavariate(1.6, 3.0)) for _ in range(n)]

def noisy_predictor(d):
    return d + random.gauss(0, 0.15)  # router sees difficulty with noise

def run_router(traffic, threshold, router_cost=0.00002):
    cost = ok_n = strong_n = 0
    for d in traffic:
        use_strong = noisy_predictor(d) > threshold
        m = STRONG if use_strong else WEAK
        ok, _ = m.answer(d)
        cost += m.cost + router_cost
        ok_n += ok
        strong_n += use_strong
    return cost, ok_n / len(traffic), strong_n / len(traffic)

def run_cascade(traffic, conf_threshold, verifier_cost=0.0001):
    cost = ok_n = esc = 0
    for d in traffic:
        ok, conf = WEAK.answer(d)
        cost += WEAK.cost + verifier_cost
        if conf < conf_threshold:
            ok, _ = STRONG.answer(d)
            cost += STRONG.cost
            esc += 1
        ok_n += ok
    return cost, ok_n / len(traffic), esc / len(traffic)

if __name__ == "__main__":
    traffic = make_traffic(20000)
    base_cost = len(traffic) * STRONG.cost
    base_q = sum(STRONG.answer(d)[0] for d in traffic) / len(traffic)
    print(f"all-strong: cost={base_cost:.1f} quality={base_q:.3f}")
    print("router sweep")
    for t in (0.2, 0.3, 0.4, 0.5, 0.6):
        c, q, s = run_router(traffic, t)
        print(f" t={t:.1f} strong_share={s:.2f} cost_vs_strong={c/base_cost:.2f} quality_retention={q/base_q:.3f}")
    print("cascade sweep")
    for t in (0.5, 0.7, 0.85, 0.95):
        c, q, e = run_cascade(traffic, t)
        print(f" t={t:.2f} escalation={e:.2f} cost_vs_strong={c/base_cost:.2f} quality_retention={q/base_q:.3f}")

Running it with the fixed seed produced the following on 20,000 simulated requests. These are simulation outputs, illustrative only.

Policy	Setting	Strong-call share	Cost vs all-strong	Quality retention
Router	threshold 0.3	55%	0.57	1.000
Router	threshold 0.5	27%	0.31	0.995
Router	threshold 0.6	16%	0.21	0.986
Cascade	confidence 0.70	10% escalated	0.16	0.987
Cascade	confidence 0.85	22% escalated	0.28	0.995

Three lessons transfer to real systems even though the numbers do not. First, the quality curve is flat for a long stretch and then falls off, so there is a sweet spot just before the cliff. Second, the cascade looked cheaper at the same retention only because the simulated confidence signal is informative; with a weak verifier the cascade degrades toward paying twice for nothing. Third, the harness makes the knob explicit: one threshold, one curve, one decision you can defend with data.

To adapt it, replace Model.answer with real API calls, replace noisy_predictor with a trained router such as a RouteLLM controller, and replace the simulated confidence with logprobs, a schema check or a judge. Log every decision with its inputs, because that log becomes your next training set.

Evaluating a Router: Savings at 95% Quality Retention

The standard acceptance test for a router is the one RouteLLM popularized: how much can you cut cost while retaining some fraction, typically 95%, of the strong model’s quality? Phrasing the question this way forces both axes into the conversation and prevents the common failure of declaring victory on cost alone.

Figure 4: A continuous evaluation loop samples production traffic, obtains strong-model reference answers, judges pairs, and gates threshold changes on quality retention.

Offline evaluation

Build a held-out set from real traffic, stratified by task type and difficulty. Run the strong model, the weak model and the router on it. Grade with ground truth where available and an LLM judge elsewhere. Then plot quality against strong-model share by sweeping the threshold. Report the share needed to hit 95% retention, the area between your router’s curve and the random-routing diagonal, and confidence intervals, because a few hundred prompts give wide error bars at the 95% mark.

Be careful about the baseline you quote. As the RouteLLM write-up demonstrates, a saving “over random routing” and a saving “over always-strong” are different numbers for the same router. Both are legitimate, but only the second maps to your invoice. RouterBench exists partly to standardize this comparison for multi-model routers, with its dataset of over 405,000 inference outcomes.

Online evaluation

Offline sets go stale. In production, sample a small percentage of routed requests, 1% to 5% is typical, and send the same prompt to the strong model in the background. Judge the routed answer against the strong answer pairwise, with order randomized to control position bias. The ratio of wins and ties is an online estimate of quality retention. Gate threshold changes on it, and canary them to a slice of traffic first.

The shadow-call cost is a tax on the savings: at a 3% sample rate with a strong model costing 20x the weak one, shadowing adds noticeable spend relative to a very aggressive router, so size the sample to the precision you need and no more. Use a cheaper strong-tier judge call where the judge only needs to compare, not regenerate.

Metrics worth tracking

Track realized strong-model share against target, cost per successful task rather than cost per request, quality retention with intervals, escalation rate for cascades, router latency overhead at p50 and p99, and per-segment breakdowns by tenant, language and task type. The per-segment view matters most, because aggregate retention can hold at 96% while one segment collapses to 70%.

Trade-offs, Gotchas, and What Goes Wrong

Routing fails in recognizable ways, and most are silent. Here are the ones that bite in practice.

Distribution shift. A router learns the world of its training data. The MMLU result is the cautionary tale: routers trained only on Arena preferences performed near random on MMLU questions until augmented with in-distribution labels. When you launch a new feature, or users start pasting code into a bot designed for FAQs, the router is suddenly out of distribution and still confidently returns scores. Monitor the input embedding distribution and the realized strong-model share, and retrain on a schedule.

Router bias. Routers learn correlates of difficulty, such as prompt length, vocabulary or language. A router trained mostly on English chat may route non-English prompts to the weak model because the weak model scored badly on a few of them, or may do the reverse. Because quality retention is an average, the harm concentrates on the groups the router mishandles. Evaluate per segment and treat disparities as defects, not noise.

Tail queries. The cost savings come from the easy 70%; the quality risk lives in the rare hard prompt that looks easy. A short question like “Is this contract clause enforceable?” is brief but demanding. Short prompts are the classic false negative for text-based difficulty predictors. Add rules that force the strong model for high-stakes categories no matter what the router says.

Safety routing. A cheaper model may have weaker safety behavior or weaker refusal calibration than the flagship. If routing moves a sensitive prompt to a model with different guardrails, you have changed your risk profile without a policy review. Pin sensitive categories, such as medical, legal and self-harm related content, to models you have red-teamed, and run safety classifiers independent of the router.

Cascade pathologies. A miscalibrated verifier either accepts bad answers (quality loss) or rejects good ones (cost with no benefit). Self-verification by a small model is noisy, as AutoMix’s authors acknowledge when they add a POMDP layer to cope with it. A cascade also compounds latency, as computed earlier, and can double-bill output tokens on retries. And an adversary who can influence the cheap model’s confidence can steer traffic.

Silent model drift. Providers update and deprecate models. A router calibrated against last quarter’s weak model faces a different model today. Pin model versions where the provider allows, and re-run calibration whenever a version changes.

Operational complexity. Every routing layer is one more thing to debug. When a user complains about a bad answer, you now ask which model answered, why it was chosen, and what the alternative would have said. If you do not log the routing decision, its score and the model version, you cannot answer. Treat the decision log as a first-class product, not a debug aid.

Economics that vanish. If your traffic is small, a router’s engineering and evaluation cost can exceed its savings. At 10,000 requests a month with the earlier illustrative prices, the all-strong bill is $100. No routing project pays for itself there.

Practical Recommendations

Start with measurement, not machinery. Log every request with its task type, token counts and the model used, then estimate what fraction of your spend sits in tasks a smaller model could plausibly handle. If the potential saving is small, stop there.

Then climb the ladder in order of effort and risk. Step one is deterministic rules: send obviously simple endpoints, such as classification, extraction and short rewrites, to the cheap tier, and keep high-stakes endpoints on the strong tier. Step two is caching, which beats routing for repetitive traffic. Step three is a cascade for tasks with deterministic verifiers, such as schema validation and test execution, where the deferral signal is trustworthy. Step four is a learned router, trained on your own traffic labeled by ground truth or a calibrated judge, for mixed-difficulty free-form tasks.

Throughout, define the acceptance criterion up front: for example, at least 95% quality retention on the held-out set and no segment below 90%, with cost per successful task as the objective. Gate every threshold change behind a canary.

A short checklist:

Log decision, score, model version and cost for every request.
Stratify evaluation by task, language and difficulty, not just in aggregate.
Calibrate thresholds on production-like data and recheck the realized strong-model share weekly.
Pin sensitive and high-stakes categories to the strong, red-teamed tier.
Keep availability fallbacks separate from quality routing in code and in dashboards.
Sample 1 to 5 percent of routed traffic for shadow comparison against the strong model.
Re-run calibration on every provider model version change.
Revisit the whole design when price sheets change, since the strong-to-weak price ratio sets the size of the prize.

Frequently Asked Questions

What is LLM routing?

LLM routing is the practice of choosing, for each request, which language model should answer it. A router or gateway sits in front of a pool of models and applies rules, a trained classifier, a cascade with a verifier, or another model to decide. The goal is to spend frontier-model money only where quality demands it, reducing cost while holding quality near a target such as 95% of the strongest model.

What is the difference between a router and a model cascade?

A router decides before any model answers, using only the prompt and metadata, so it adds one small inference and avoids wasted generations. A model cascade calls a cheap model first, evaluates its answer, and defers to a stronger model only when the answer looks unreliable. Cascades use richer evidence but make escalated queries pay twice in cost and latency. Many systems combine both.

How much can RouteLLM actually save?

The RouteLLM authors report over 2x cost reductions in some cases without hurting response quality, and the LMSYS write-up cites over 85% on MT Bench, 45% on MMLU and 35% on GSM8K versus always using GPT-4, while keeping 95% of its performance. Those results depend on augmented training data, a specific strong-weak model pair and the benchmark. Measure on your own traffic before planning around them.

Does LLM routing hurt answer quality?

It can, but you control the trade-off. A threshold slides you along a quality-cost curve, and a well-trained router retains most of the strong model’s quality at a fraction of the calls. Risk concentrates in out-of-distribution prompts, rare hard queries that look easy and under-served segments such as other languages. Evaluate per segment, sample shadow comparisons, and pin high-stakes categories to the strong model.

Should I use LiteLLM, OpenRouter or Portkey for routing?

They solve different problems. Portkey’s conditional routing is deterministic, based on request metadata and parameters, which suits business rules. OpenRouter’s Auto Router picks models using aggregate community spend per task type, so it is convenient but not tuned to your task. LiteLLM offers beta auto routing with several classifier options plus self-hosting. Verify current features in each product’s documentation, as they change quickly.

When is routing not worth building?

Skip it when volume is low, when nearly every request is genuinely hard, when you cannot measure quality, or when caching and prompt trimming already capture most of the saving. At 10,000 monthly requests the whole bill may be too small to justify the engineering and evaluation effort. Routing pays off at scale with mixed-difficulty traffic and a large price ratio between your model tiers.