hanzo-llm-gateway

Hanzo LLM Gateway is a unified proxy that routes requests to 100+ LLM providers through a single OpenAI-compatible API.

Hanzo LLM Gateway - Unified AI Proxy for 100+ Providers

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-chat.md, hanzo/python-sdk.md, hanzo/hanzo-console.md

Overview

Hanzo LLM Gateway is a unified proxy that routes requests to 100+ LLM providers through a single OpenAI-compatible API. Fork of LiteLLM with Hanzo routing, cost tracking, and provider fallback. Powers api.hanzo.ai/v1.

Why Hanzo LLM Gateway?

One endpoint, all models: OpenAI, Anthropic, Google, Meta, Mistral, Zen + 100 more
Smart routing: Load balancing, fallback chains, cost optimization
Cost tracking: Per-request attribution via Hanzo Console
Self-hostable: Run your own gateway with custom provider keys
Rate limit handling: Automatic retry with provider rotation

OSS Base

Fork of LiteLLM proxy. Repo: github.com/hanzoai/llm.

When to use

Running a centralized LLM proxy for your team
Routing between multiple AI providers
Cost tracking and budget enforcement
Self-hosting LLM access behind your firewall
Adding custom models or providers

Hard requirements

At least one provider API key (OpenAI, Anthropic, etc.)
Port 4000 available (default)
PostgreSQL for logging (optional)

Quick reference

| Item | Value | |------|-------| | Public endpoint | https://api.hanzo.ai/v1 | | Internal endpoint | http://llm.hanzo.svc.cluster.local:4000/v1 | | Port | 4000 | | Config | config.yaml or env vars | | Dashboard | https://llm.hanzo.ai (Cloud UI, NOT LLM endpoint) | | Repo | github.com/hanzoai/llm |

One-file quickstart

Docker

docker run -d --name hanzo-llm \
 -p 4000:4000 \
 -e OPENAI_API_KEY="${OPENAI_API_KEY}" \
 -e ANTHROPIC_API_KEY="${ANTHROPIC_API_KEY}" \
 ghcr.io/hanzoai/llm:latest

Config file

# config.yaml
model_list:
 - model_name: zen-70b
 llm_params:
 model: together_ai/Qwen/Qwen3-235B-A22B
 api_key: os.environ/TOGETHER_API_KEY

 - model_name: gpt-4o
 llm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

 - model_name: claude-sonnet
 llm_params:
 model: anthropic/claude-sonnet-4-20250514
 api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
 routing_strategy: least-busy
 num_retries: 3
 fallbacks:
 - zen-70b: [gpt-4o, claude-sonnet]

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL

Make commands

# Clone from github.com/hanzoai first
cd <project>
make dev # Start dev server (port 4000)
make up # Docker compose up
docker compose up -d # Alternative

Test request

curl http://localhost:4000/v1/chat/completions \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -H "Content-Type: application/json" \
 -d '{
 "model": "zen-70b",
 "messages": [{"role": "user", "content": "Hello!"}]
 }'

Core Concepts

Provider Routing

Client Request
 │
 ▼
┌──────────────────┐
│ LLM Gateway │
│ (port 4000) │
├──────────────────┤
│ Router: │
│ ├─ least-busy │──▶ OpenAI
│ ├─ fallback │──▶ Anthropic
│ └─ cost-based │──▶ Together AI
├──────────────────┤ ▲
│ Logging: │ │
│ └─ PostgreSQL │ │
│ └─ Console │────┘ (cost tracking)
└──────────────────┘

Fallback Chains

When primary provider fails (429, 500, timeout), gateway automatically routes to fallback:

fallbacks:
 - zen-70b: [gpt-4o, claude-sonnet] # Try zen → GPT-4o → Claude
 - gpt-4o: [claude-sonnet] # Try GPT-4o → Claude

Budget & Rate Limits

general_settings:
 max_budget: 100.00 # USD per month
 budget_duration: 1m # Reset monthly
 max_parallel_requests: 100 # Concurrent limit

Zen Model Mapping

Zen models map to upstream providers (private config in `github.com/hanzoai/zengateway/config.yaml`):

BRAND POLICY: Never reference upstream model names in public-facing contexts. Zen models are presented as Hanzo's own architecture: Zen MoDE (Mixture of Diverse Experts).

Production Deployment

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
 name: hanzo-llm
spec:
 replicas: 3
 template:
 spec:
 containers:
 - name: llm
 image: ghcr.io/hanzoai/llm:latest
 ports:
 - containerPort: 4000
 env:
 - name: LITELLM_MASTER_KEY
 valueFrom:
 secretKeyRef:
 name: llm-secrets
 key: master-key
 - name: DATABASE_URL
 valueFrom:
 secretKeyRef:
 name: llm-secrets
 key: database-url

Troubleshooting

| Issue | Cause | Solution | |-------|-------|----------| | 401 on requests | Wrong master key | Check LITELLM_MASTER_KEY | | Provider timeout | Upstream provider slow | Increase timeout or add fallback | | Cost not tracking | No DATABASE_URL | Add PostgreSQL connection | | Model not found | Not in config | Add to config.yaml |

Related Skills

hanzo/hanzo-chat.md - Chat API (uses this gateway)
hanzo/hanzo-console.md - Observability (receives cost data)
hanzo/python-sdk.md - Client library
hanzo/zenlm.md - Zen model family

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: llm, proxy, litellm, gateway, routing Prerequisites: LLM API concepts, Docker