Intelligent Routing
Choosing the right path automatically
TL;DR
Choosing the right path automatically
> Overview
Not every user request needs the same treatment. Routing classifies what the user wants and sends it to the best-fit handler: a cheaper model for simple questions, a specialized one for complex tasks. This is the single biggest lever for cost optimization in AI products and directly impacts your unit economics.
> Why This Matters for Your Product
A simple FAQ lookup does not need a $0.03/request frontier model. A $0.001 model works fine. But a complex multi-step analysis does need the expensive model. Smart routing can cut AI costs by 60–80% while maintaining quality where it matters. This module teaches PMs how to define routing categories, set cost tiers, and monitor misrouting rates.
> Interactive & tools
Pricing (per 1K tokens)
Approximate cost per 1K tokens (early 2026)
| Model | Input ($) | Output ($) |
|---|---|---|
| Claude 3.5 Haiku | 0.0008 | 0.004 |
| Claude Sonnet | 0.003 | 0.015 |
| Claude Opus | 0.015 | 0.075 |
| GPT-4o mini | 0.00015 | 0.0006 |
| GPT-4o | 0.0025 | 0.01 |
| Gemini Flash | 0.000075 | 0.0003 |
With smart routing, 100K requests/day can save $50K–$200K annually vs. sending everything to a frontier model.
Prices change frequently; check provider pricing pages.
Fallback chain
Fallback chain
If the first model’s output doesn’t meet the bar, automatically try a more capable model. Not the same as reflection (same model retrying).
Related Engineering Patterns
These are the technical patterns your engineering team will implement. Understanding them helps you have better conversations.
Key Product Decisions
- [01]What categories of requests does your product handle?
- [02]Which categories justify expensive models vs. cheap ones?
- [03]What is the acceptable misrouting rate before user experience degrades?
- [04]Should you expose the routing decision to users (e.g., Using advanced analysis)?
Ask Your Engineering Team
- →What is our classification accuracy for routing decisions?
- →What is the cost difference between our model tiers?
- →Can we A/B test routing strategies to optimize cost vs. quality?
- →How do we handle fallback when the cheap model underperforms?
Unlock the decision framework
Free account — no credit card required. Sign up to see the full decision checklist and the questions to ask your engineering team.
Sign Up FreePlay the interactive Intelligent Routing game
Practice the decisions from this module in an interactive game. Sign up free to play and save your progress.
Sign Up Free to PlaySee the full decision framework
Sign up free to see this module's Key Decisions, the questions to ask your engineering team, and play the interactive Intelligent Routing game.
Sign Up Free