Melaya Blog

AI Model Routing: How It Works and How to Set It Up

AI model routing sends each request to the right LLM by cost, quality and latency. Learn the strategies, the failure modes, and how to pair routing with tools.

AI model routing is a decision layer that picks which model answers each request, instead of sending everything to one general-purpose model. Done well, it cuts cost and adds fallback. Routing only decides who thinks, though. If your agents also have to act in browsers, on Android devices and in business apps, you need a second layer that decides who acts, and that layer should not change when the model does. Melaya is that layer: you bring the model (Claude, GPT, Gemini, DeepSeek, Mistral, Qwen, Ollama, LM Studio), and Melaya supplies the tools, permissions and approvals.

TL;DR

• A router evaluates each request and selects a model by price, quality, latency or another stated goal, per IBM Research.
• Start with explicit rules and bounded fallback. Move to learned routing only after you test on representative traffic.
• Count the full cost: classifier overhead, retries, user rework and, for self-hosted models, hardware and staffing.
• Model routing chooses who thinks. Tool routing chooses who acts. An agent needs both.
• In Melaya, the model is yours to choose and swap, and every consequential action (publishing, sending, paying, writing to connected apps) runs behind human approval.

What AI model routing is

A router sits between your application and several models. For each request it decides where the request goes. IBM Research defines the router as a decision layer that selects among models by price, performance, inference speed or another objective you set, rather than sending every query to one general-purpose model.

The pool can be broad. Oracle's guide says routing can combine models from one provider, several vendors, private deployments and open-source models, using signals such as complexity, expected cost, latency, data sensitivity and required quality.

The search term "LLM routing" means the same thing. The reasoning is simple: a short classification task and a multi-step contract analysis do not need the same model, and paying frontier prices for both wastes budget.

Routing is distinct from fallback. LatentEval separates routing, which picks the first model before the call, from fallback chains, which act after a failure, refusal or timeout. Production systems generally need both.

How a router decides

A production router follows a repeatable sequence. Braintrust describes it as five steps:

1. Define the candidate pool.
2. Filter by eligibility: context length, modalities, tool calling, structured output, region and budget.
3. Evaluate a policy over the eligible candidates.
4. Execute the chosen call.
5. Apply fallback on retryable failures.

Eligibility comes first for a reason. Cloudflare's AI Gateway Auto Router first removes models that cannot support the request, credentials, billing, spend limits or health requirements. Only then does it classify the conversation, score expected quality against cost, and fall back to another eligible model. The lesson: capability and governance constraints filter candidates before cost or quality ranking.

Models differ on more than price. API7.ai lists language quality, context limits, modalities, latency, throughput, tool use, structured output, residency, logging boundaries, hosted prices and self-hosted operating costs. Each one is a possible routing signal or a hard filter.

Routing strategies compared

Redis's 2026 architecture guide groups strategies into three families: rule-based, semantic-similarity and predictive routing. Here is how they compare in practice.

IBM Research draws the line on cascades: cascading routing tries smaller models first and escalates, while simultaneous audition runs several models and therefore adds cost and delay.

If you use an open-source gateway, LiteLLM's documentation lists weighted selection, rate-limit-aware, least-busy, latency-based and cost-based strategies, plus cooldowns, timeouts, retries and cross-provider fallbacks. Those are the building blocks most teams combine.

A sensible starting policy

Start small:

1. Define two or three request classes, for example extraction, drafting and multi-step reasoning.
2. Assign each class a primary model and one fallback.
3. Set hard filters first: context length, tool calling, region.
4. Add a spend cap per route.
5. Log every decision.

API7.ai makes the same recommendation: begin with deterministic routing and bounded fallback, then measure quality, latency, token use, errors and fallback behavior on every route.

What routing does to cost, quality, latency and reliability

The gains are real and bounded. Per Vercel's production guide, routing can lower cost by sending requests to cheaper models, cut latency by choosing faster deployments and improve availability through fallback. It can also add classification overhead, retries and cascade calls, and aggressive routing to cheap models can hurt quality.

The research gives a sense of scale. On 5,001 held-out queries from VS Code Chat traffic, the HyDRA-Multi router retained 95.2 quality with 17.1% cost savings in conservative mode, and 92.3 quality with 38.0% savings in aggressive mode. Treat those as one benchmark, not a promise for your traffic. The EMNLP survey frames routing as a per-query balance of expected quality against inference cost, which means you optimize a workload-specific frontier, not the lowest token price.

Reliability is policy design

Different failures need different responses. Redis recommends retrying after a 429 using the provider's Retry-After guidance, shifting traffic when latency rises, and not retrying content-filter refusals as though they were infrastructure faults. Vercel separates three tools: retries work within a model group, fallbacks cross providers, and circuit breakers stop attempts against an unhealthy provider.

Session affinity

Cloudflare documents a trade-off many teams miss. Without a session ID, consecutive turns may land on different models and lose prompt-cache reuse, and the router may keep a model when switching would cost more cache value than it gains. For multi-turn agents, pin a session to a model unless there is a reason to move.

Count the whole bill

A router can become net-negative when miscalibration produces poor answers, user retries, excess escalation, or classifier overhead close to model latency, according to Tian Pan's production analysis. For self-hosted models, API7.ai adds that inference is an operated service with hardware, availability, security and staffing costs, so calculate savings on fully loaded cost.

How to test a router before it takes traffic

A router is a policy, so test it like one. A workable plan:

1. Build a golden set. Future AGI recommends a stratified set of 200 to 1,000 queries, refreshed weekly.
2. Score four things separately: route correctness, realized savings versus theory, quality preservation under substitution, and fallback correctness. Each gets its own dataset and gate.
3. Shadow route. Run the router on real traffic without exposing users to its output.
4. Gate learned routers harder. Metrum AI's case study requires 2,000 representative operator-owned requests with a disjoint holdout, cost and quality gates, real latency tests at intended concurrency and a 24-hour staging shadow before promotion.
5. Canary the rollout and version both the router and its thresholds.
6. Monitor in production: escalation rate, quality delta between routes and retry amplification. A rising escalation rate can signal traffic drift or a degraded classifier, as Tian Pan notes.

For agent workloads, add deterministic checks on schemas and tool arguments, plus behavioral checks for completion, correction or human takeover, as the Agent Lab Journal suggests.

Governance: routing is a data-flow decision

Every route is a decision about where your data goes. Vercel's secure gateway guide names four controls: provider selection, path blocking, retention enforcement and routing audit. For regulated sectors, Tigera lists data residency and classification policies, immutable audit logs, and policy-as-code that defines who can invoke which models and under what circumstances.

The NHIMG governance guidance adds practical habits: provider allowlists, classifying prompts and retrieval data before routing, logging every provider hop including retries, and being able to revoke a provider quickly.

One more rule from API7.ai: a fallback is not automatically behavior-compatible. Test schemas, tool calls, safety behavior, latency and residency on every candidate before it enters the pool.

Model routing vs. tool routing

Most routing content stops at the model. That covers half of what an agent needs. ToolRouter's documentation puts it plainly: model routing selects which language model answers a prompt and is billed in tokens, while tool routing selects which capability acts on the world and is usually billed per successful tool call. Model routing decides who thinks, tool routing decides who acts. Swapping the model does not give an agent a web crawler, and adding a crawler does not make the model cheaper.

This is where the tool layer matters. If your tools are wired to one provider's function-calling format, every routing change becomes a rebuild. If tools sit in a provider-independent layer, the router can change models freely and the workflows keep running.

Where Melaya fits: the tool layer under any router

Melaya is bring-your-own-model. It works with any cloud or local AI model, including Claude, GPT, Gemini, DeepSeek, Mistral, Qwen, Ollama and LM Studio. It does not sell or require its own model for agents. You pick the brain, and Melaya is the controlled pair of hands.

What that means for a routing setup:

• One tool layer, many models. The Agent Builder combines models, tools, memory, schedules, approvals, triggers and multi-agent pipelines in traceable, replayable runs. Teams can change models or use local infrastructure without rebuilding their tools and workflows.
• Real work on real surfaces. Browser Control, Mobile Agents and Computer Control let approved models operate websites, Android applications and desktop software. Mobile control is Android only, and Computer Control runs through the Melaya desktop app.
• Your existing client stays. The Melaya MCP Server brings these capabilities to ChatGPT, Claude, Cursor, Codex and other MCP-compatible clients.
• Approval where it counts. Every consequential action (publishing, sending, paying, writing to connected apps) runs behind human approval. Granular permissions and execution traces let you reconstruct what each run did.

That last point pairs with routing governance. A router log tells you which model answered. A Melaya execution trace tells you what the agent then did with it. To see how wide the model pool can be, browse the 51 AI model providers and LLM APIs Melaya covers, or read how browser automation works with any AI model, even local ones. If your pool includes open-weight models on your own hardware, the self-hosted AI agents guide covers that setup.

Example: one workflow, three models

Take a weekly competitor-pricing check. A small local model running in Ollama classifies incoming pages. A mid-tier cloud model extracts the prices into a schema. A stronger model drafts the summary for your team. Browser Control does the page visits, and the write to your connected app waits for a human approval. You can change any of the three models through your router without touching the browser step or the approval step. That separation is the point.

Perspective

Routing debates focus on tokens. In our experience building an automation platform, the harder question is what happens after the model answers.

Our view: your AI is the brain, Melaya is the controlled pair of hands. A router should be free to change the brain on cost, latency or data-residency grounds. The hands, meaning permissions, approvals, traces and device access, should stay constant, so a routing change never alters what an agent is allowed to do.

That is why we built Melaya provider-independent and bring-your-own-model. Granular permissions apply the same way whichever model is behind a run. Human approval gates the consequential steps. Execution traces are replayable, so when a route changes you can compare runs side by side.

Our advice: route the model by policy, keep the tool layer fixed, and review the traces weekly.

Melaya Team

Plans for the tool layer

If you are adding a tool layer under your router, start with the single capability you need. These are the published prices, read from the Melaya pricing page:

For the standalone products, the agent pricing breakdown explains what each one covers.

Get your agents acting, whichever model answers

Routing gives you the right model per request. Melaya gives that model hands: browsers, Android devices, desktop software and business apps, with permissions, approvals and replayable traces. Start with Browser Agents at $9.99 per month or try the Sandbox plan at $0 per month, and keep your router and your models exactly as they are.

If browser work is your first use case, see the AI browser agent product page.

FAQ

What is the difference between AI model routing and LLM routing?

They describe the same practice. Both mean a decision layer that picks which model handles each request by price, quality, latency or another objective. "AI model routing" is the broader label and may include non-language models, while "LLM routing" is the common term for text models.

Should I start with a learned router or rules?

Start with rules. Rule-based routing is predictable and easy to debug, and it gives you the baseline traffic data a learned router would need. Move to learned routing only after testing on a representative request set with a disjoint holdout and a staging shadow run.

How much can routing save?

It depends on your traffic. One published benchmark on 5,001 VS Code Chat queries showed 17.1% savings at 95.2 quality retention in conservative mode and 38.0% at 92.3 in aggressive mode. Measure your own savings on fully loaded cost, including retries, classifier overhead and rework.

Does routing replace fallback?

No. Routing picks the first model before the call. Fallback acts after a failure, refusal or timeout. Production systems generally need both, with different responses for rate limits, outages and safety refusals.

Can agents that use tools be routed across models too?

Yes, as long as every candidate supports the tools and schemas the agent uses, so test each one first. Keeping the tools in a provider-independent layer means a routing change does not require rebuilding workflows. In Melaya, consequential actions stay behind human approval whichever model is selected.

Sources

• LLM routing for quality, low-cost responses, IBM Research
• What Is LLM Routing?, Oracle
• What is model routing?, Braintrust
• Auto Router, Cloudflare AI Gateway docs
• Multi-LLM Routing with Qwen and Cloud Models, API7.ai
• LLM router architecture: best practices for 2026, Redis
• Router - Load Balancing, LiteLLM
• 6 LLM routing strategies for teams running multi-model, Vercel
• A Systematic Survey of Large Language Model Routing, EMNLP
• HyDRA: Hybrid Dynamic Routing Architecture
• Model Routing in Production, Tian Pan
• Evaluating LLM Routing Policies, Future AGI
• Case study: learned upstream selection, Metrum AI Router
• Cutting Agent Costs with Trace-Based Model Routing, Agent Lab Journal
• A Guide to Secure AI Gateway, Vercel
• LLM Gateway: Key Functions, Tigera
• How should enterprises govern LLM routing, NHIMG
• Tool routing vs model routing, ToolRouter
• Model router, LatentEval

Recommended

• How to Set Up an Android AI Agent, Step by Step
• 51 AI Model Providers and LLM APIs
• Browser Automation With Any AI Model, Even Local
• AI Agent Builder: Visual, No-Code, Free to Start