ml-llm-model-routing

Intelligent LLM routing with RouteLLM, semantic routing, and cost optimization

LLM Model Routing

Scope: Cost-effective LLM routing, RouteLLM framework, semantic routing, rule-based strategies, observability Lines: ~450 Last Updated: 2025-10-26

When to Use This Skill

Activate this skill when:

• Reducing LLM costs while maintaining quality (up to 85% reduction)
• Building applications with mixed complexity queries
• Implementing multi-model architectures for different use cases
• Optimizing response latency and quality tradeoffs
• Routing based on query intent, complexity, or domain
• Instrumenting routing decisions with observability tools
• Deploying production LLM systems with cost constraints

Core Concepts

The Routing Problem

Challenge: Strong models (GPT-4o, Claude 3.5 Sonnet) are expensive but not always necessary. Weak models (GPT-4o mini, Gemini Flash) are cheap but limited.

Solution: Route queries dynamically to appropriate models based on complexity, intent, or requirements.

Benefits:

• 85%+ cost reduction on benchmarks (MT Bench)
• 45% cost reduction on MMLU
• 35% cost reduction on GSM8K
• Maintained quality equivalent to always using strong models
• Improved average latency for simple queries

Routing Strategies

1. Learning-Based Routing (RouteLLM):

• Trained on preference data (human/GPT-4 judgments)
• Predicts which model pair will satisfy quality thresholds
• Generalizes across different strong/weak model pairs

2. Semantic Routing:

• Uses embeddings to match queries to specialized models
• Routes based on semantic similarity to reference prompts
• Enables domain-specific model assignment

3. Rule-Based Routing:

• Deterministic rules based on query metadata
• Simple heuristics (length, keywords, patterns)
• Fastest routing decision time

4. Hybrid Routing:

• Combines multiple strategies for robustness
• Fallback chains for reliability
• Context-aware routing logic

RouteLLM Framework

Architecture: Published at ICLR 2025, RouteLLM is an open-source framework from UC Berkeley and Anyscale for cost-effective LLM routing based on preference data.

Key Components:

• Router Models: Trained classifiers that predict model selection
• Preference Data: Training data from Chatbot Arena, human feedback
• Model Pairs: Strong model (GPT-4) + weak model (Mixtral, GPT-3.5)
• Threshold Tuning: Adjustable quality/cost tradeoff

Router Types:

1. sw_ranking (Elo-based):

• Weighted Elo calculation for routing decisions
• Fast, interpretable, no training required
• Good baseline performance

1. bert (BERT classifier):

• BERT encoder trained on preference data
• Balances speed and accuracy
• Recommended for production use

1. causal_llm (LLM classifier):

• Fine-tuned LLM as router (e.g., Llama)
• Highest accuracy but slower
• Best for quality-critical applications

1. matrix_factorization:

• Low-rank factorization of preference matrix
• Memory efficient
• Suitable for large-scale deployments

Performance:

• Matches commercial routers (Martian, Not Diamond)
• 40%+ cheaper than commercial alternatives
• Generalizes to new model pairs without retraining

Semantic Routing Architecture

Components:

• Encoder: Embedding model (BERT, Sentence Transformers, OpenAI)
• Route Definitions: Named routes with example prompts
• Similarity Matching: Cosine similarity to route examples
• Threshold: Minimum similarity for route assignment

Modern Implementations (2025):

1. vLLM Semantic Router:

• Rust-based, high-concurrency routing
• ModernBERT classifier for intent detection
• Zero-copy inference with Hugging Face Candle
• Mixture-of-Models (MoM) architecture

1. Red Hat Semantic Processor:

• Envoy ExtProc integration
• Hybrid Rust + Golang implementation
• Semantic cache with response reuse
• ~90% accuracy without LLM inference

1. Aurelio Labs Semantic Router:

• Multi-modal support (text, images)
• Easy encoder integration (Cohere, OpenAI, HuggingFace)
• Layer-based decision making

Advantages:

• Sub-millisecond routing decisions
• No LLM call required for routing
• Handles multi-domain applications
• Supports caching and reuse

---

Implementation Patterns

RouteLLM: Basic Setup

from routellm.controller import Controller

# Initialize router with model pair
router = Controller(
    routers=["bert"],  # or ["sw_ranking", "causal_llm", "mf"]
    strong_model="gpt-4-1106-preview",
    weak_model="mixtral-8x7b-instruct-v0.1",
    api_base="http://localhost:8000/v1",  # Optional: custom endpoint
)

# Route a query
query = "Explain quantum entanglement"
response = router.route(
    model="router-bert-0.5",  # router-{type}-{threshold}
    messages=[{"role": "user", "content": query}]
)

print(f"Routed to: {response.model}")
print(f"Response: {response.choices[0].message.content}")

Hanzo SDK Integration

Automatic Model Routing with Hanzo

Instead of manual RouteLLM implementation, use Hanzo SDK's built-in intelligent routing:

from hanzo import Hanzo

# Initialize with hybrid mode (local + cloud with automatic routing)
hanzo = Hanzo(
    inference_mode='hybrid',  # Combines local + cloud models
    auto_route=True,          # Enable intelligent routing
    cost_optimize=True        # Optimize for cost
)

# Automatic routing based on query complexity
response = hanzo.chat.completions.create(
    messages=[{"role": "user", "content": query}],
    # No need to specify model - Hanzo routes automatically
)

# Hanzo automatically:
# 1. Analyzes query complexity
# 2. Checks local models (Hanzo Node) first for privacy
# 3. Routes to cloud only if needed
# 4. Caches results for repeated queries
# 5. Tracks costs and latency
print(f"Routed to: {response.model}")
print(f"Cost: ${response.usage.cost:.4f}")
print(f"Latency: {response.latency_ms}ms")

Hanzo Routing Strategies

# 1. Privacy-First Routing (local whenever possible)
hanzo = Hanzo(
    inference_mode='hybrid',
    prefer_local=True,        # Use Hanzo Node first
    node_url='http://localhost:8080'
)

# Simple queries stay local (privacy + no cost)
response = hanzo.chat.completions.create(
    messages=[{"role": "user", "content": "What is 2+2?"}]
)
# Routed to: local-llama-3-8b

# Complex queries go to cloud
response = hanzo.chat.completions.create(
    messages=[{"role": "user", "content": "Analyze quantum cryptography"}]
)
# Routed to: claude-3.5-sonnet

# 2. Cost-Optimized Routing
hanzo = Hanzo(
    inference_mode='cloud',
    cost_limit_per_request=0.01,  # Max $0.01 per request
    fallback_on_limit=True         # Use cheaper model if exceeded
)

# 3. Latency-Optimized Routing
hanzo = Hanzo(
    inference_mode='hybrid',
    max_latency_ms=500,      # Sub-500ms required
    prefer_local=True        # Local models are fastest
)

# 4. Quality-Focused Routing
hanzo = Hanzo(
    inference_mode='cloud',
    quality_tier='high',      # Only top-tier models
    models=['gpt-4', 'claude-3.5-sonnet']
)

Hanzo MCP Tool for Agents

Expose Hanzo routing as MCP tool for agentic workflows:

import { MCPServer, Tool } from '@hanzo/mcp'
import { Hanzo } from 'hanzo'  // Python SDK

const hanzo = new Hanzo({
  inference_mode: 'hybrid',
  auto_route: true
})

const hanzoRouteTool: Tool = {
  name: 'hanzo_route_and_infer',
  description: 'Automatically route query to best model and generate response',
  parameters: {
    prompt: { type: 'string', required: true },
    strategy: {
      type: 'string',
      enum: ['cost', 'latency', 'quality', 'privacy'],
      default: 'cost'
    }
  },
  async execute({ prompt, strategy }) {
    const response = await hanzo.chat.completions.create({
      messages: [{ role: 'user', content: prompt }],
      routing_strategy: strategy
    })
    
    return {
      text: response.choices[0].message.content,
      model: response.model,
      cost: response.usage.cost,
      latency_ms: response.latency_ms
    }
  }
}

Hanzo vs Manual RouteLLM

| Feature | Manual RouteLLM | Hanzo SDK | |---------|-----------------|-----------| | Setup | 50+ lines config | 3 lines | | Local Models | Not supported | Built-in (Hanzo Node) | | Privacy | Manual logic | Automatic (local-first) | | Cost Tracking | Custom implementation | Built-in | | Caching | Manual | Automatic | | MCP Integration | Custom wrapper | Native support | | Fallback | Manual chains | Automatic | | Monitoring | Custom Prometheus | Built-in dashboard |

When to Use What:

• Hanzo SDK: Most use cases (simpler, more features)
• Manual RouteLLM: Research, custom training, non-Hanzo stack

RouteLLM: Threshold Tuning

# Threshold controls quality/cost tradeoff
# Higher threshold = more weak model usage = lower cost
# Lower threshold = more strong model usage = higher quality

thresholds = [0.3, 0.5, 0.7, 0.9]

for threshold in thresholds:
    router_name = f"router-bert-{threshold}"
    response = router.route(
        model=router_name,
        messages=[{"role": "user", "content": "Simple math: 2+2"}]
    )
    print(f"Threshold {threshold}: {response.model}")

# Results:
# 0.3 -> gpt-4 (conservative, high quality)
# 0.5 -> mixtral (balanced)
# 0.7 -> mixtral (aggressive cost savings)
# 0.9 -> mixtral (maximum savings)

RouteLLM: Evaluation Mode

import pandas as pd
from routellm.routers import ROUTER_CLS

# Load router for evaluation
router_cls = ROUTER_CLS["bert"]
router = router_cls(
    strong_model_name="gpt-4-1106-preview",
    weak_model_name="mixtral-8x7b-instruct-v0.1"
)

# Evaluate on dataset
test_queries = [
    "What is 2+2?",
    "Explain the implications of quantum computing on cryptography",
    "Name the capital of France",
    "Analyze the socioeconomic factors in the French Revolution"
]

decisions = []
for query in test_queries:
    # Get routing decision (0-1 score)
    score = router.route(query)
    selected = "strong" if score < 0.5 else "weak"
    decisions.append({
        "query": query[:50],
        "score": score,
        "model": selected
    })

df = pd.DataFrame(decisions)
print(df)

# Calculate cost savings
weak_ratio = (df["model"] == "weak").mean()
print(f"Weak model usage: {weak_ratio:.1%}")
print(f"Estimated savings: {weak_ratio * 0.95:.1%}")  # 95% cheaper

Semantic Routing with Aurelio

from semantic_router import Route, RouteLayer
from semantic_router.encoders import OpenAIEncoder

# Define routes with examples
code_route = Route(
    name="code",
    utterances=[
        "How do I write a function in Python?",
        "Debug this code snippet",
        "Explain this algorithm",
        "Write a sorting function"
    ]
)

general_route = Route(
    name="general",
    utterances=[
        "What is the weather today?",
        "Tell me a joke",
        "What's the capital of France?",
        "Who won the World Series?"
    ]
)

reasoning_route = Route(
    name="reasoning",
    utterances=[
        "Solve this complex math problem",
        "Analyze the implications of this policy",
        "Compare and contrast these approaches",
        "Provide a detailed ethical analysis"
    ]
)

# Initialize encoder and layer
encoder = OpenAIEncoder()
route_layer = RouteLayer(
    encoder=encoder,
    routes=[code_route, general_route, reasoning_route]
)

# Route queries
queries = [
    "How do I reverse a string in JavaScript?",
    "What's 2+2?",
    "Explain the philosophical implications of AI consciousness"
]

for query in queries:
    route = route_layer(query)
    print(f"Query: {query[:50]}")
    print(f"Route: {route.name if route else 'fallback'}\n")

# Map routes to models
model_map = {
    "code": "claude-3.5-sonnet",  # Best at coding
    "general": "gpt-4o-mini",  # Fast and cheap
    "reasoning": "gpt-4o",  # Strong reasoning
    None: "gpt-4o-mini"  # Default fallback
}

def route_and_call(query: str):
    route = route_layer(query)
    model = model_map.get(route.name if route else None)
    print(f"Routing to: {model}")
    # Call selected model...

Rule-Based Routing

import re
from typing import Literal

ModelType = Literal["strong", "weak", "specialized"]

class RuleBasedRouter:
    def __init__(
        self,
        strong_model: str = "gpt-4o",
        weak_model: str = "gpt-4o-mini",
        code_model: str = "claude-3.5-sonnet"
    ):
        self.strong = strong_model
        self.weak = weak_model
        self.code = code_model

    def route(self, query: str, context: dict = None) -> str:
        """Route based on heuristics"""
        query_lower = query.lower()

        # Rule 1: Code-related queries -> specialized model
        code_keywords = ["code", "function", "debug", "implement", "algorithm"]
        if any(kw in query_lower for kw in code_keywords):
            return self.code

        # Rule 2: Simple factual queries -> weak model
        simple_patterns = [
            r"^what is \w+\?$",
            r"^who is \w+\?$",
            r"^when did \w+",
            r"^\d+[\+\-\*/]\d+",  # Simple math
        ]
        if any(re.match(p, query_lower) for p in simple_patterns):
            return self.weak

        # Rule 3: Long, complex queries -> strong model
        if len(query) > 500:
            return self.strong

        # Rule 4: Reasoning keywords -> strong model
        reasoning_keywords = [
            "analyze", "implications", "compare", "evaluate",
            "critique", "synthesize", "ethical", "philosophical"
        ]
        if any(kw in query_lower for kw in reasoning_keywords):
            return self.strong

        # Rule 5: Context indicates multi-turn conversation -> strong model
        if context and context.get("turn_count", 0) > 3:
            return self.strong

        # Default: weak model
        return self.weak

# Usage
router = RuleBasedRouter()

queries = [
    "Write a binary search function",  # -> code model
    "What is the capital of France?",  # -> weak
    "2+2",  # -> weak
    "Analyze the ethical implications of AI in healthcare"  # -> strong
]

for q in queries:
    model = router.route(q)
    print(f"{q[:50]:50} -> {model}")

Hybrid Routing with Fallback

from typing import Optional
import openai

class HybridRouter:
    def __init__(self):
        self.semantic_router = route_layer  # From earlier example
        self.rule_router = RuleBasedRouter()

    def route(
        self,
        query: str,
        use_semantic: bool = True,
        use_rules: bool = True
    ) -> str:
        """Hybrid routing with fallback chain"""

        # Try semantic routing first (most accurate)
        if use_semantic:
            route = self.semantic_router(query)
            if route and route.name:
                return model_map.get(route.name)

        # Fallback to rule-based routing
        if use_rules:
            return self.rule_router.route(query)

        # Final fallback
        return "gpt-4o-mini"

    def route_with_retry(self, query: str, max_retries: int = 2):
        """Route with fallback on errors"""
        models = [
            self.route(query),  # Primary
            "gpt-4o",  # Fallback 1
            "claude-3.5-sonnet"  # Fallback 2
        ]

        for i, model in enumerate(models[:max_retries + 1]):
            try:
                response = openai.ChatCompletion.create(
                    model=model,
                    messages=[{"role": "user", "content": query}],
                    timeout=30
                )
                return {
                    "model": model,
                    "response": response.choices[0].message.content,
                    "attempt": i + 1
                }
            except Exception as e:
                print(f"Model {model} failed: {e}")
                if i == len(models) - 1:
                    raise
                continue

# Usage
hybrid = HybridRouter()
result = hybrid.route_with_retry(
    "Explain quantum entanglement to a 10-year-old"
)
print(f"Used model: {result['model']} (attempt {result['attempt']})")

Arize Phoenix Observability

import phoenix as px
from phoenix.trace import using_project
from opentelemetry import trace

# Start Phoenix
px.launch_app()

# Instrument routing
tracer = trace.get_tracer(__name__)

@using_project("llm-routing")
def route_and_call(query: str):
    with tracer.start_as_current_span("routing_decision") as span:
        # Route
        router = HybridRouter()
        selected_model = router.route(query)

        # Log routing decision
        span.set_attribute("query", query)
        span.set_attribute("selected_model", selected_model)
        span.set_attribute("routing_strategy", "hybrid")

        # Call model
        with tracer.start_as_current_span("llm_call") as call_span:
            call_span.set_attribute("model", selected_model)

            response = openai.ChatCompletion.create(
                model=selected_model,
                messages=[{"role": "user", "content": query}]
            )

            # Log metrics
            call_span.set_attribute("tokens_used", response.usage.total_tokens)
            call_span.set_attribute("cost_usd", calculate_cost(response, selected_model))

        return response

# View routing patterns in Phoenix UI
# Navigate to http://localhost:6006

---

Quick Reference

Model Strengths by Category (2025)

Category      | Best Models              | Use Case
--------------|--------------------------|----------------------------------
Reasoning     | GPT-4o, Claude 3.5/4    | Complex analysis, multi-step logic
Coding        | Claude 3.5 Sonnet       | Code generation, debugging
Speed         | Gemini 2.5 Flash        | 370 tok/s, real-time applications
Cost          | GPT-4o mini, Gemini Flash| Simple queries, high volume
Multimodal    | GPT-4o, Gemini 2.5      | Vision + text tasks
Long Context  | Claude 3.5 (200k)       | Document analysis, large codebases

Cost Comparison (per million tokens)

Model                    | Input    | Output   | Use Case
-------------------------|----------|----------|------------------
GPT-4o                   | $5.00    | $20.00   | General strong
GPT-4o mini              | $0.15    | $0.60    | General weak
Claude 3.5 Sonnet        | $3.00    | $15.00   | Coding strong
Claude 3 Haiku           | $0.25    | $1.25    | Fast weak
Gemini 2.5 Flash         | $0.10    | $0.40    | Cheapest strong
DeepSeek V3              | $0.30    | $1.00    | Open-source alt

Routing Decision Tree

Query characteristics -> Recommended strategy
────────────────────────────────────────────────
High volume, mixed complexity -> RouteLLM
Domain-specific (code, legal)  -> Semantic routing
Latency critical              -> Rule-based
Quality critical              -> Always strong model
Budget constrained            -> RouteLLM threshold 0.7+
Multi-turn conversations      -> Session-aware routing

---

Anti-Patterns

❌ Always using the strongest model: Wastes 10-50x cost on simple queries ✅ Route based on complexity; use weak models for 60-80% of queries

❌ No observability on routing decisions: Can't optimize or debug ✅ Use Arize Phoenix to trace routing logic and model performance

❌ Static routing rules that don't adapt: Performance degrades over time ✅ Continuously evaluate routing accuracy; retrain or adjust thresholds

❌ Ignoring latency in routing overhead: Routing adds 100ms+ delay ✅ Use cached semantic embeddings; pre-compute route assignments

❌ Single point of failure in routing: Entire system fails if router breaks ✅ Implement fallback chains; default to mid-tier model on router errors

❌ Not tracking cost savings: Can't justify routing complexity ✅ Log model selection, token usage, and calculate cost deltas

❌ Over-engineering for simple use cases: Rule-based is fine for 2-3 models ✅ Start simple; add RouteLLM/semantic routing when scaling to 5+ models

❌ Training routers on outdated model pairs: New models shift capabilities ✅ Regularly re-evaluate with current model benchmarks (GPT-4o, Claude 3.5)

---

Related Skills

• llm-model-selection.md - Choosing models for specific capabilities
• multi-model-orchestration.md - Coordinating multiple models in pipelines
• modal-web-endpoints.md - Deploying routing logic as FastAPI endpoints
• api-rate-limiting.md - Rate limiting across multiple model providers
• observability-distributed-tracing.md - Tracing multi-model systems
• api-error-handling.md - Handling provider failures and fallbacks

---

Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)