Agent Native
Loading...

Architecture Pattern

Model Routing Cascade Pattern for Agents (2026)

Where to set routing thresholds in a cheap-model-first cascade: which agent turns escalate to frontier models, which confidence signals to trust, and per-route cost accounting. Grounded in a real $4,660 API bill cut 70% with routing, plus RouteLLM and FrugalGPT numbers.

Pattern: Cheap-model-first routing cascade with tiered escalation•Updated 2026-07-25

Problem

Agent loops default every turn — trivial or hard — to the most capable model configured, and the bill scales with the worst-case model, not the average task. We measured this directly: one month of running OpenClaw with everything pinned to Claude Opus at $75 per million output tokens produced a $4,660.87 Anthropic invoice, documented in our ClawRouter write-up — while roughly 80% of the requests were autocomplete-grade work a far cheaper model handles identically. The price spread a cascade can arbitrage in mid-2026 is wide: Claude Sonnet 5 is $2/$10 per million input/output tokens on introductory pricing through August 31, 2026 (then $3/$15), Claude Haiku 4.5 is $1/$5, Claude Opus is $5/$25 (sources: anthropic.com/news/claude-sonnet-5, platform.claude.com pricing docs), and MiniMax-M2.5 lists at $0.30/$1.20 on MiniMax's own API pricing page while scoring 80.2% on SWE-bench Verified (sources: platform.minimax.io pricing docs, minimax.io/news/minimax-m25). That is a 20x+ output-price gap between the cascade's floor and ceiling. The hard part is not wanting a cascade — it is deciding where the escalation thresholds sit and which signals are trustworthy enough to gate them.

Use the cascade when your agent runs many turns of mixed difficulty and the LLM line item is material — coding agents, support agents, long-horizon automation. Who should choose what: solo developers and small teams on OpenClaw or similar stacks should start with an off-the-shelf router (ClawRouter's local heuristic scoring cut the bill above by ~70% with sub-1ms routing decisions and zero added API latency); teams with labeled traffic and eval infrastructure should train or tune a learned router (the RouteLLM approach — lm-sys reports up to 85% cost reduction while retaining 95% of GPT-4 performance on MT-Bench, and the paper measures up to 3.66x cost savings; arxiv.org/abs/2406.18665); and teams whose entire workload is uniform frontier-grade reasoning should skip the cascade — a router that escalates 90% of turns is pure overhead, and honestly, at that point a single cheap-but-strong model like MiniMax-M2.5 flat may beat both the cascade and the frontier default on cost per solved task. Do not use a cascade where a wrong cheap-model answer has irreversible consequences before quality checks can catch it.

Components

  • Route classifier in front of the model call: heuristic (token count, code presence, tool-call complexity, keyword tiers), learned (a trained router in the RouteLLM style), or hybrid — running locally so routing adds no network hop
  • Model tier ladder with explicit rungs, e.g. local/ultra-cheap (MiniMax-M2.5 at $0.30/$1.20) → mid (Haiku 4.5 at $1/$5 or Sonnet 5 at $2/$10 intro) → frontier (Opus at $5/$25), each rung with a per-million price recorded in config
  • Escalation gate that decides when a cheap-tier answer is insufficient: task-class rules first, then calibrated uncertainty signals, then hard triggers (tool-call validation failure, schema-parse failure, repeated loop turns)
  • Sticky-context policy governing which conversation state travels up the cascade on escalation, since re-sending full agent context to the frontier tier is where escalation cost hides
  • Per-route cost ledger logging model, token counts, computed cost, and counterfactual frontier cost for every call — the ClawRouter implementation logs JSON lines per request precisely so savings are auditable rather than estimated
  • Threshold review loop: a recurring eval that replays sampled routed traffic against the frontier model to measure quality drift and re-tune thresholds

Flow

  1. 1Classify the turn before calling any model. Cheap signals go a long way: prompt length, presence of code blocks, whether the turn is a tool-result summarization vs a planning step, and explicit task tags from the agent harness. ClawRouter demonstrates this can be a local sub-1ms scoring decision with no API call.
  2. 2Route task classes, not just prompts. In an agent loop, turn type is the strongest prior: tool-output summarization, commit-message writing, and format conversion route to the floor tier; multi-file planning, architecture decisions, and first-attempt debugging route mid; only turns that failed mid-tier checks or carry explicit high-stakes tags open at the frontier.
  3. 3Set initial thresholds from the price ladder, not from intuition: escalation from a $1.20/M-output floor to a $25/M-output frontier is a ~20x step, so a turn should escalate only when the expected quality lift is worth 20x the tokens. FrugalGPT's cascade results (matching the best individual LLM's performance with up to 98% cost reduction on some workloads; arxiv.org/abs/2305.05176) came from exactly this scoring-then-escalating structure.
  4. 4Gate escalation on signals in order of trustworthiness. Hard signals first: failed JSON/schema parse, tool-call rejection, test failure, agent loop detection — these are objective and free. Then calibrated uncertainty: routing research (e.g. the Self-REF ICML work) finds learned confidence probes outperform both raw token logprobs and verbalized self-confidence, and that verbalized 'I am not sure' statements are the least calibrated signal available — do not build the gate on asking the model how confident it is.
  5. 5On escalation, forward a distilled context: the task, the cheap model's attempt, and the failure evidence — not the full transcript. This turns the frontier call into a review-and-correct pass and keeps the 20x rung from also being a 20x-token call.
  6. 6Record every routing decision in the cost ledger with the counterfactual: what this call cost vs what all-frontier would have cost. Our measured outcome from doing this with real traffic was the $4,660.87/month bill dropping ~70%, and per-route logs are what made that number a measurement instead of a marketing claim.
  7. 7Replay a sample of floor-tier and mid-tier turns against the frontier model weekly. If disagreement on a task class rises above your tolerance, move that class up one rung; if a frontier-routed class shows near-zero disagreement, move it down. Thresholds are living config, not launch-day constants.
  8. 8Re-check the price ladder itself on a calendar: Sonnet 5's $2/$10 is introductory and lapses to $3/$15 after August 31, 2026, which alone shifts the optimal mid-tier escalation math by 50% — pin pricing in config with a review date, not in code.

Tradeoffs

Heuristic vs learned routing

Heuristic routers (ClawRouter-style local scoring) are transparent, fast, and free to run, but plateau: they judge the prompt, not the model's actual competence on it. Learned routers close that gap — RouteLLM's routers reach up to 85% cost reduction at 95% GPT-4-level quality on MT-Bench per the lm-sys results — but require preference/label data and an eval harness to train and maintain. This is an honest third-party win: if you have the traffic data, the open-source RouteLLM framework will likely beat anything you hand-roll, and hosted routers (OpenRouter's auto-routing, NotDiamond, Martian) outsource the problem entirely at the cost of adding a vendor in your hot path.

Cascade complexity vs one good cheap model

A cascade is worth it only if tiers meaningfully differ on your workload. With MiniMax-M2.5 posting 80.2% on SWE-bench Verified at $0.30/$1.20 — a score in frontier territory at roughly 95% lower output price than Opus — some coding-agent workloads now get better cost-per-solved-task from a single strong cheap model plus retry than from a three-rung cascade with routing logic to maintain. Run the single-model baseline before building the cascade; sometimes the competitor's cheap model wins outright.

Escalation latency vs first-call quality

A cascade converts some frontier-grade turns into two sequential calls (cheap attempt, then frontier redo), adding a full round trip on exactly the turns that were hardest. FrugalGPT's own framing accepts this: savings come from the majority of turns that never escalate. If your agent is user-interactive and p95 latency matters more than cost, route those task classes directly to the frontier tier rather than letting them fail upward.

Confidence-signal trust

The gate is only as good as its signals, and the research is unflattering: verbalized model confidence is poorly calibrated, raw logprobs are decoding-sensitive and uncalibrated for correctness, and only tuned/learned signals (calibrated thresholds, confidence probes like Self-REF) route reliably. Budget for threshold tuning on held-out traffic as a standing cost of the pattern — an untuned confidence gate silently degrades into either an expensive always-escalate or a quality-leaking never-escalate.

Quality drift accountability

Cost savings are visible on the invoice immediately; quality loss is invisible until users complain. The replay-eval loop is not optional overhead — it is the only mechanism that tells you the 70% saving did not quietly cost you task success rate. Any cascade deployment without a drift eval is reporting only the numerator of its ROI.

Failure Modes

  • Ping-pong escalation: the cheap model fails, the frontier model succeeds, but the agent's next turn routes back to the floor tier which promptly fails on the follow-up — burning both tiers every cycle. Escalation needs stickiness: once a task escalates, its remaining turns stay at the winning tier until the task closes.
  • Verbalized-confidence gating: routing on the model saying it is confident. Calibration research consistently ranks self-reported confidence as the worst escalation signal; models are confidently wrong at the floor tier, which is precisely the failure the cascade exists to catch.
  • Counting only per-token savings: escalated turns pay the cheap attempt plus the frontier redo plus re-sent context. Cascades with high escalation rates can exceed all-frontier cost — which is why the ledger must log realized cost per solved task, not per call.
  • Routing on prompt length alone: short prompts ('why does this deadlock?') can be the hardest turns, and long tool outputs the easiest. Length-only heuristics systematically under-route short high-stakes reasoning turns.
  • Stale price ladder: thresholds tuned against Sonnet 5's $2/$10 introductory pricing silently become wrong when standard $3/$15 pricing starts September 1, 2026, or when a provider reprices. Treat pricing as config with an expiry date.
  • Silent floor-tier quality decay after a cheap-model provider ships a quantized or updated endpoint — without the weekly frontier-replay eval, the first detection is a customer-visible regression, not a dashboard.

Implementation Notes

  • Start with task-class routing and hard failure signals only — no confidence gating — and measure for a week. Our ClawRouter deployment reached its ~70% reduction on the $4,660.87 baseline with local heuristic scoring and per-request JSON-line cost logs; that is the achievable floor before any learned routing sophistication.
  • Encode the ladder with live prices and dates: MiniMax-M2.5 $0.30/$1.20 (MiniMax API docs; note MiniMax's launch announcement quoted $0.15 input — trust the API pricing page, and note the fast tier is listed as M2.5-highspeed at $0.60/$2.40), Haiku 4.5 $1/$5, Sonnet 5 $2/$10 until 2026-08-31 then $3/$15, Opus $5/$25 (Anthropic pricing docs). A cascade's economics are exactly these numbers; hardcoding them stale is a real money bug.
  • If you want a learned router, evaluate the open-source RouteLLM framework (github.com/lm-sys/routellm) before building: its matrix-factorization and BERT-based routers ship pretrained, and the paper (arXiv 2406.18665) reports the routers transfer across model pairs without retraining — the fastest path to beating heuristics.
  • Escalate with a structured handoff, not the raw transcript: task statement, cheap-tier attempt, objective failure evidence (failed test, schema error, tool rejection). This caps escalation token cost and gives the frontier model a review task, which is cheaper and more reliable than a cold restart.
  • Log the counterfactual on every call (cost_actual, cost_if_frontier, tier, escalation_reason). The difference between 'we think routing saves money' and 'routing cut a $4,660 bill by 70%' is exactly this ledger.
  • Wire the weekly replay eval before launch, not after: sample 1-5% of non-frontier turns, replay against the frontier model, score agreement per task class, and make threshold moves from that report. This is the AgentOps loop that keeps the cascade honest as models, prices, and your workload drift.