Model Routing for Agents: A Three-Tier Router Built on Complexity, Latency and Cost

A lightweight orchestrator that picks a model per request by complexity, escalates on sustained failure and stops runaway loops, with runnable Python and the numbers to check it against.

TL;DR

  • Classify each request with cheap rules first and a small semantic model second, then send it to the cheapest tier that can handle it.
  • Escalate on sustained failure, not on the first bad turn, and cap model and tool calls per request.
  • Tag every call with the tier that served it, so cost per tier is a query, not a guess.
  • Budget tens of milliseconds for the router and measure whole workflows, not tokens per second.

One model for every agent turn is the expensive default

An agent that converts a currency and an agent that refactors a billing module make the same API call if you wire them to one model. You pay frontier prices for the currency lookup.

Iternal's selection guide puts numbers on it. Route simple tasks to a Haiku-class model, medium ones to Sonnet-class and only the hardest 15% to Opus-class, and output tokens cost a blended $10.50 per million against $25 for Opus on everything, a 58% saving. Your mix will differ, but the saving comes from the same place: most turns do not need the big model.

Picking "the best model" does not rescue you either. This month BenchLM's agentic leaderboard puts Claude Opus 5.5 first with a BenchAlign agentic score of 87.9, while WhatLLM's Agentic Index puts Claude Fable 5.1 first at 57.9. The two use different scales and name different leaders. Vendors have stopped pretending one model fits: Microsoft's Copilot Cowork offers Sonnet 5 as "the efficient choice for everyday tasks" and GPT 6 Astra as "the latest model for tough problems", and leaves the user to choose. A router makes that choice for them.

The usual objection is the "router tax", the latency the extra hop adds to every request. The measurements put it at tens of milliseconds, which is smaller than most people assume.

A three-tier router: classify, escalate, stop

The orchestrator does three jobs and nothing else.

  1. Classify. Regex and keyword rules catch the obvious cases for free. Anything they miss goes to a small embedding model that scores it against labelled examples. The output is a starting tier.
  2. Escalate. Start low and move up only when the task keeps failing. NVIDIA's Switchyard escalation router works this way: it starts each conversation with a lower-cost model, and an LLM judge moves the session to a more capable one "when it detects sustained difficulty". One bad turn is noise; two in a row is a signal.
  3. Stop. Azure's orchestration guidance says: "To guard against infinite tool-call loops, set iteration limits." A cheap model stuck in a loop can outspend a frontier model that finishes.

Here is the whole loop with the model calls stubbed out, so it runs as is:

import re
 
TIERS = ["small", "mid", "frontier"]  # cheapest first
RULES = [  # L1: regex rules settle the obvious cases
    (re.compile(r"\b(refactor|migrate|debug)\b", re.I), 2),
    (re.compile(r"\b(summari[sz]e|draft|compare)\b", re.I), 1),
    (re.compile(r"\b(convert|look up)\b", re.I), 0),
]
MAX_CALLS = 8       # iteration limit per request
ESCALATE_AFTER = 2  # consecutive failed turns before moving up a tier
 
def classify(prompt):
    for pattern, tier in RULES:
        if pattern.search(prompt):
            return tier
    return 1  # no rule matched: an L2 semantic model decides; start mid here
 
def run(prompt, turns):
    tier = start = classify(prompt)
    failures = 0
    for calls, ok in enumerate(turns, 1):
        if calls > MAX_CALLS:
            return {"prompt": prompt, "status": "stopped: iteration limit"}
        failures = 0 if ok else failures + 1
        if failures >= ESCALATE_AFTER and tier < len(TIERS) - 1:
            tier, failures = tier + 1, 0
    return {"prompt": prompt, "start": TIERS[start],
            "end": TIERS[tier], "cost_code": f"route/{TIERS[tier]}"}
 
print(run("Convert 30 EUR to USD", [True]))
print(run("Summarise this incident thread", [True, False, False, True]))
print(run("Summarise the logs", [False] * 9))
{'prompt': 'Convert 30 EUR to USD', 'start': 'small', 'end': 'small', 'cost_code': 'route/small'}
{'prompt': 'Summarise this incident thread', 'start': 'mid', 'end': 'frontier', 'cost_code': 'route/frontier'}
{'prompt': 'Summarise the logs', 'status': 'stopped: iteration limit'}

In production each ok comes from a tool result or a judge, and cost_code goes on the usage record. That tag is what lets finance see spend per tier.

What the router costs you in latency

The open-source llm-router project is a working reference for the classify step. Its fast path is "Deterministic Regex/Keyword matching (O(1) latency)"; its slow path is semantic routing with all-MiniLM-L6-v2 at about 30 ms, inside a stated budget of under 50 ms of routing overhead. It runs Llama-3.2-3B for reasoning and TinyLlama-1.1B for speed on consumer hardware, and exposes the threshold at launch:

python src/api/main.py router=semantic router.threshold=0.85

Thirty milliseconds is small next to a model call, and the hop itself can be free. Opper benchmarked 200 GPT-4.1 calls and found OpenRouter "70ms faster than OpenAI direct on time to first token (0.640s vs 0.712s)". OpenRouter is a hosted gateway, not your classifier, but it shows the extra hop is not where the time goes. Their conclusion: the blanket router tax "doesn't hold"; each router has its own profile.

The latency that does hurt is escalation. A request that fails twice on the small tier pays for those turns before it reaches the one that works. WhatLLM's advice applies: "Measure the full workflow, including tool latency, retries, and failures." If most of your traffic escalates, the router is making things slower and you should start that class higher.

Before you trust it: measure on your own traffic

Leaderboards get you a shortlist. Your own logs set the thresholds.

  • Label a sample of real requests by the cheapest tier that completed them, and replay them through the classifier.
  • Count mis-routes both ways. Routing a hard task to a weak model is the failure people watch for; the designpattern.fyi write-up lists "Classifier errors might route complex tasks to weaker models." Routing easy work to the frontier tier is the failure that erases your savings without anyone noticing.
  • Shadow before you switch. Run the router alongside the single-model path and compare completion rate, whole-workflow latency and cost per tier, using the cost codes.
  • Alert on escalation rate. A jump means the small model regressed or your traffic changed. Either way, the thresholds need another look.

Do not route if nearly every request is hard. With nothing to send down a tier, the router is pure overhead.

Sources

aiarchitecturellmroutingcost

All writing