scriptRun Beta

LLM Router: How Model Routing Works and When Your App Needs One

AnnaAnna
November 11, 2026
Business
AI WorkflowsAutomation
LLM Router: How Model Routing Works and When You Need One

Quick Answer: An LLM router sits between your app and your model providers and chooses which model answers each request. Simple prompts go to fast, cheap models, hard ones go to stronger models, and failed calls go to a backup provider. You need one when you use several models, hit outages or rate limits, or want to cut costs.

  • Main strategies: rule-based, semantic, cost/complexity-based, learned (e.g. RouteLLM) and fallback routing.

  • Start simple: fallback plus a few rules covers most apps. Add smarter routing when your data shows a gain.

  • Trade-offs: a little added latency, one more dependency, and data passing through a third party.

  • Measure it: cost per 1,000 requests, p95 latency, fallback rate and quality spot-checks.

  • Fastest setup: an OpenAI-compatible API. Change the base URL and key, and keep your code.

Most teams start with one model for everything. It works until the bill grows, a provider has an outage, or you notice that "What are your opening hours?" is going to the same expensive model as "Review this 40-page contract."

No single model is the best choice for every request on cost, speed and quality at once.

This guide explains what an LLM router does, how routing decisions are made, the main strategies, what a router costs you, and how to tell whether your app needs one yet.

What is an LLM router

An LLM router is software that sits between your application and one or more model providers. Your app sends a request to the router once, and the router chooses which model and which provider should answer it. Your code no longer hard-codes "always call model X."

The core function of an LLM router comes down to four jobs:

  • Model selection. Pick the model that fits the request: a small, fast model for a short classification, a strong reasoning model for multi-step analysis, a long-context model for a large document.

  • Fallback and failover. When a provider returns an error, hits a rate limit or slows down, retry or send the request somewhere else instead of failing the user.

  • Load balancing. Spread traffic across keys, regions or providers so no single quota becomes the bottleneck.

  • Cost control. Keep expensive models for the requests that actually need them.

Some teams call this an AI model router or a model router.

💡 The idea is the same: the decision about which model to use moves out of your application code and into one place you can change.

LLM router vs LLM gateway

The two terms overlap, and many products do both. The simplest way to separate them: a router decides where a request goes; a gateway manages everything around the request.

LLM routerLLM gateway
Main question it answersWhich model should handle this request?How do all requests reach providers safely and consistently?
Typical featuresModel selection, routing rules, fallback, load balancingOne API for many providers, API keys, quotas, rate limits, logging, billing
Can exist without the other?Yes, as a few lines of code in your appYes, as a proxy that only forwards requests
In practiceUsually one feature inside a gatewayUsually includes routing

If you're comparing products, look at both halves. A router with smart rules but no failover, logging or spend limits still leaves you building the rest yourself.

How LLM routing works

LLM routing is the decision process a router runs on every request. Implementations differ, but most follow the same six steps:

  1. The request arrives. Your app sends a prompt to one endpoint, often in the OpenAI request format, with optional hints such as a task name, a user tier or a maximum cost.

  2. The router classifies it. It reads the signals it has: prompt length, task type, keywords, an embedding of the prompt, or a score from a small trained model that estimates how hard the request is.

  3. It picks a target. Based on that classification and your rules, it selects a model and a provider, for example a small model for a short FAQ answer or a frontier model for a coding task.

  4. It calls the provider. The request is translated into that provider's format if needed and sent.

  5. It handles failure. If the call returns a server error, a 429 rate-limit error or takes too long, the router retries with backoff or moves to the next model on a fallback list.

  6. It records the result. Model used, tokens, cost and latency are logged, so you can see whether your routing rules are doing what you expect.

LLM router workflow diagram showing request classification, model selection, usage logging, and fallback to a backup provider.
An LLM router selects a model for each request, tracks usage, and switches to a backup provider when errors or timeouts occur.

One request goes in; the router chooses the model, switches to a backup when a call fails, and logs what each request cost.

Your application only sees steps 1 and 6: one request in, one response out. That separation is the main reason teams adopt model routing.

You can change models, add providers or tighten cost rules without shipping new application code.

LLM routing strategies

There are five common ways a router decides where a request goes. Most production setups combine two or three: simple rules for the obvious cases, a smarter method for the rest, and fallback underneath everything.

StrategyHow it decidesBest forMain drawback
Rule-basedFixed conditions: task name, user tier, prompt lengthClear, predictable workloadsRules grow messy and miss edge cases
SemanticCompares the prompt's embedding to example prompts for each routeMixed traffic with recognizable intentsNeeds good examples and a tuned similarity threshold
Cost / complexityEstimates difficulty, sends easy prompts to cheaper modelsCutting spend on high-volume appsA wrong estimate sends a hard prompt to a weak model
LearnedA trained classifier predicts which model will answer well enoughLarge volumes with quality data to train onTraining data, evaluation and drift monitoring
Fallback / load balancingOrder of preference plus health checks, quotas and latencyUptime and rate-limit protectionDoesn't choose by quality, only by availability

Rule-based routing

The simplest LLM routing technique:
✅ If the task is "summarize_ticket", use model A;
✅ If the user is on the free plan, use model B;
✅ If the prompt is over 50,000 tokens, use a long-context model.
Rules are fast, cheap to run and easy to debug. Start here. Most apps get a large share of the benefit from a handful of rules.

Semantic routing

The router turns each incoming prompt into an embedding and compares it with example prompts for each route.

"What's my balance?" and "How much money do I have?" land on the same route even though they share no keywords.

Semantic routing works well for chatbots and support assistants where users phrase the same intent many ways.

You need a set of examples per route and a similarity threshold, plus a default route for prompts that match nothing.

Cost and complexity-based routing

Here the router estimates how hard a request is and sends it to the cheapest model likely to handle it.

Greetings, classifications and short factual answers go to a small model; multi-step reasoning, code and long analysis go to a frontier model.

This is the strategy behind most "cut your LLM bill" claims, because in many apps the majority of requests are simple.

Learned routing (RouteLLM)

Learned routers train a small model to predict, for each prompt, whether a cheaper model will answer well enough. The best-known open-source example is RouteLLM from LMSYS, which trained four router types, including a matrix factorization model and a BERT classifier.

Routing between GPT-4 Turbo and Mixtral 8x7B, LMSYS reported cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K while keeping 95% of GPT-4's performance.

Those are benchmark results from 2024 models, so treat them as evidence the approach works, not as a forecast for your traffic.

Fallback and load-balancing routing

This strategy doesn't judge the prompt at all. It keeps a preferred order of models or providers and moves down the list when a call fails: a 5xx error, a 429 rate-limit response, a timeout or a stream that drops mid-answer.

Load balancing spreads requests across several keys or providers so no single quota runs out. It's the least clever strategy and often the most valuable, because it keeps your app answering during an outage.

If you regularly hit 429s, see our guide to OpenAI API rate limits.

Do you need an LLM router

Not every app does. A router pays off when you have choices to make between models, or failures to absorb. If you tick two or more of these, it's worth setting one up:

  • You already call two or more models or providers, or plan to.

  • You've hit rate limits, 429 errors or provider outages in production.

  • Your traffic mixes simple requests (FAQs, tagging, short replies) with hard ones (reasoning, code, long documents).

  • Model spend is a line item someone asks about every month.

  • You want to test a new model on part of your traffic without a code release.

  • You're tired of managing separate keys, SDKs and invoices for each provider.

You can probably wait if you use one model, your volume is low, outages wouldn't hurt anyone, and the bill isn't a concern.

In that case, a single provider SDK is simpler, and adding a router early just adds a moving part.

Revisit the question when any of the conditions above changes.

There's also a middle path. Pointing your app at an OpenAI-compatible endpoint that supports many models costs you almost nothing today, because you keep the same SDK.

When you do need routing or failover later, it's a configuration change rather than a migration.

What an LLM router costs you

A router solves real problems, but it isn't free. Most vendor pages skip this part, so here it is plainly.

  • Added latency. Every request takes an extra hop, and some strategies add work before the call: computing an embedding or running a classifier. Rule-based routing adds very little; learned and semantic routing add more. Vendors quote very different overhead figures, so measure it yourself with your own prompts and region. For background on what drives response time, see our guide to API latency.

  • Another dependency. If the router goes down, everything behind it goes down too. Check its uptime record, status page and what happens to in-flight requests during an incident.

  • Data passing through a third party. With a hosted router, your prompts and responses pass through one more company. Read its data retention policy and check where data is processed, especially for regulated data.

  • Misroutes. A complexity estimate will sometimes send a hard prompt to a weak model. Users notice a bad answer more than a slow one, so start conservative and loosen the rules as you gain data.

  • Inconsistent outputs. Different models format answers differently. If your app parses JSON or expects a fixed structure, use structured outputs or validate responses, whichever model answered.

How to measure whether routing works

A router you can't measure is just a guess with extra steps. Before you turn it on, record a baseline from your current setup, then compare the same numbers after.

MetricWhat it tells youWatch for
Cost per 1,000 requestsWhether routing is saving moneySavings that come with a quality drop
p95 latencyThe slow-tail experience users actually feelRouter overhead or slow fallback models
Share of traffic per modelWhether the rules behave as intendedAlmost everything still going to the expensive model
Fallback rateHow often the primary choice failsA rising rate means a provider or quota problem
Quality spot-checksWhether cheaper models answer well enoughLower ratings or more retries on routed requests

For quality, sample a few dozen routed requests each week and review them, or run them through the strong model as well and compare.

Thumbs-up rates, retry rates and support tickets are useful signals too.

Roll out in steps. Begin with fallback only, add two or three simple rules, and move to semantic or learned routing only when your numbers show where the remaining savings are.

Types of LLM routers

When people search for the best LLM router, they're usually choosing between four kinds of solutions. The right one depends on how much you want to build and run yourself.

TypeExamplesYou getYou take on
Build your ownA few rules in your codeFull control, no extra vendorEvery provider SDK, retries, logging and keys
Open-source routerLiteLLM, RouteLLM, NVIDIA LLM RouterReady-made routing logic you can self-hostHosting, scaling, upgrades and monitoring
Hosted gatewayOpenRouter, Portkey, Azure AI Foundry model routerMany models behind one API, no servers to runA vendor in your request path
OpenAI-compatible API with routingScriptRunRouting, failover and one bill without changing your SDKA vendor in your request path

Building your own makes sense for one or two models and simple rules.

Open source suits teams with infrastructure experience and strict data requirements.

A hosted option suits teams that want routing working this week rather than this quarter.

For a side-by-side look at hosted options, see our comparison of OpenRouter alternatives.

Routing with an OpenAI-compatible API

The fastest way to add routing to an existing app is to keep the SDK you already use and change where it points.

Because most models can now be reached through the OpenAI request format, an OpenAI-compatible API lets you switch or route between models without rewriting your integration.

ScriptRun's OpenAI-compatible API works this way. You keep the OpenAI SDK, change the base_url and the key, and get:

  • Multiple model providers behind one interface, with no separate accounts or KYC for each provider

  • Smart routing that picks a model by quality, cost and context size

  • Automatic failover between providers during outages

  • Retries with backoff and request-level idempotency, so a retried call isn't processed twice

  • Per-project keys, limits and quotas, plus a single bill

Here's what the change looks like in Python:

from openai import OpenAI

client = OpenAI(
    base_url="SCRIPTRUN_BASE_URL",   # from your ScriptRun dashboard
    api_key="YOUR_SCRIPTRUN_KEY",
)

response = client.chat.completions.create(
    model="MODEL_ID",               # any model available in ScriptRun
    messages=[{"role": "user", "content": "Summarize this support ticket in two lines."}],
)
print(response.choices[0].message.content)

The rest of your code, including streaming, tools and embeddings, stays the same.

If you outgrow simple routing later, ScriptRun's Workflow API adds branching, conditions, timeouts and retries for multi-step AI tasks.

Get an API key to try it on your own traffic.


Helpful?
Share this article!

Build AI workflows in minutes

Create automations, connect APIs, and access 50+ AI models from a single platform.

Visual workflow builder
OpenAI-compatible API
Workflow marketplace & templates
Start free, plans from $5

FAQs about LLM Router

What is LLM routing?

LLM routing is the process of choosing which large language model should answer each request, based on rules, the prompt's meaning, its difficulty, cost targets or provider health. The software that makes this choice is called an LLM router.

Does an LLM router add latency?

Yes, a little. Every router adds a network hop, and semantic or learned routing also adds a classification step. Rule-based routing adds the least. In practice, routing simple prompts to faster models can lower average response time overall, but measure p95 latency on your own traffic before and after.

What's the difference between an LLM router and an LLM gateway?

A router decides which model handles a request. A gateway is the wider layer that gives you one API for many providers and handles keys, quotas, logging and billing. Most gateways include a router.

How does model routing reduce LLM costs?

It sends requests that don't need a frontier model to cheaper ones. If a large share of your traffic is simple, such as greetings, tagging or short answers, routing those requests to a small model cuts the bill without changing answers to the hard ones.

Is an LLM router better than calling models directly?

For one model with low traffic, calling it directly is simpler. Once you use several models, need uptime during provider outages or want control over spend, a router saves you from building fallback, retries and model selection into your own code.

Read more from our blog

Related posts

AI WorkflowsAutomation
Cheapest LLM API 2026: Verified Prices per 1M Tokens
BusinessNovember 20, 2026

Cheapest LLM API 2026: Verified Prices per 1M Tokens

Which is the cheapest LLM API? Verified prices for OpenAI, Claude, Gemini, DeepSeek, Qwen, Mistral and Groq, plus real workload costs and hidden fees.

1734
AnnaAnna
AI MarketplaceAI WorkflowsAutomation
AI Agent Marketplace: 10 Best Platforms Compared (2026)
BusinessNovember 19, 2026

AI Agent Marketplace: 10 Best Places to Find, Run and Sell AI Agents in 2026

What an AI agent marketplace is, 10 top platforms compared on cost to try and who can sell, a safety checklist, and how to list your own agent in 2026.

1886
AnnaAnna
AI WorkflowsAutomation
Free LLM API: 2026 Providers, Limits and Real Catches
BusinessNovember 18, 2026

Free LLM API: Which Providers Are Still Free in 2026 (Limits, Keys and Catches)

Every free LLM API worth using in 2026, checked against official docs: free tiers vs credits, limits, data terms, how to get a key, and fallback code.

1905
AnnaAnna
AI WorkflowsAutomation
OpenAI Compatible API: How It Works and What Breaks
BusinessNovember 17, 2026

OpenAI Compatible API: What It Is, Who Supports It and How to Switch Safely

An OpenAI compatible API guide: base URLs for 15 providers, what breaks when you switch, a copy-paste smoke test and a safe migration checklist.

1709
AnnaAnna
© 2024-2026 PIXEL LORIS IT SERVICES L.L.C. All rights reserved.