HIP-510: Enso — Learned Model Routing and Flywheel. Status Final. Hanzo architectural specification.
This proposal specifies Enso, Hanzo's learned router: a layer that selects the best model for every request across any enabled provider and any hosted model, then improves itself from feedback. Enso makes model choice per-request and makes cost a first-class, transparent axis — it optimizes quality − λ·cost − μ·latency over the whole pool. The routing decision costs microseconds on a CPU (measured 300 ns heuristic → 12 µs learned), never a GPU. Enso tiers cleanly from a transparent rule router (cold start) to a learned policy (xᵀWp, closed-form ridge fit + online per-user LinUCB) that takes over once eval data exists and falls back to the rules whenever it is unsure. Every routed request produces a content-free training tuple that trains both the router and — recursively — the next model. This HIP defines the routing contract, the per-org and global training loops, the reward sources (in-app feedback + an LLM-as-judge quality signal), the metering, and the economics.
Frontier models trade the lead benchmark-to-benchmark: the best coder is not the best reasoner is not the cheapest capable chat model. Pinning one model for all traffic leaves both quality and money on the table, and it couples a product to a single vendor's availability, pricing, and policy. A router that picks per-request recovers the leading capability on each task and makes the cost of that capability explicit and controllable — without changing the caller's integration (model=auto over an OpenAI-compatible API).
Three properties make this a distinct layer rather than a wrapper:
learned matrix–vector product — six orders of magnitude below the model call it precedes. No GPU is needed to route; only the selected model needs one.
one; a learned policy takes over per-task as eval data accrues and defers to the rules under uncertainty.
quality score become rewards; a content-free ledger (features + score, never prompt text) trains the policy online.
A request for the virtual model id auto (alias zen-router) is resolved to a concrete servable model before provider, pricing, and billing resolution, so the entire existing path bills and reports the model that actually served. The gateway sets X-Routed-Model on the response for transparency; the response model field echoes the same id.
(code, math, reasoning, creative, vision, long_context, cheap_chat, default) by media flag, approximate length, and keyword signal.
precedence **org/project > org > * (shared base) > conf**; the first servable model wins. The known predicate restricts every level to models the caller can actually serve, so a narrower scope can only narrow, never escalate. Access-gated SKUs additionally require an explicit grant.
X-Max-Cost per 1k tokens, X-Max-Latency-Ms) gate the choice; a policy cost ceiling fills an unset budget.
The learned policy replaces the ordered preference with a bilinear utility xᵀW p_m over a feature vector x (task one-hot, hashed n-grams, length, media) and a per-model profile p_m, argmax subject to the SLO and access gates. It falls back to the rule policy when confidence is low or W is unfit.
A routing decision is recorded as a content-free RoutingEvent (task, requested→routed model, confidence, source, feature vector, tokens, cost, latency — never prompt text). A reward in [0,1] is attached, keyed by the response/usage request id, from either:
regenerate) posted to POST /v1/add-routing-reward.
response against a task rubric; only the numeric score is stored.
Enso trains both a shared base and every org's own policy from one ledger read:
*). Fit from the rewards of orgs that opted in** (TrainingContribution = enabled) plus reserved internal traffic. Consent-gated: a customer contributes to the shared base only by choice.
its own data, gated against its own incumbent, deployed to its own policy row. An org needs no consent to train on its own data; the fold ensures its learned policy wins for that org while the base serves everyone else.
Each cycle is fit → gate → deploy → publish:
(offline closed-form ridge for W; online LinUCB per user for adaptation).
on a held-out reward measure (or a benchmark eval). Otherwise the incumbent is kept and training continues on more data — promote-or-keep.
prefers it on the next request with no reload (a sub-microsecond swap).
recorded and surfaced for observability.
Every routed request yields a labeled tuple (features, winning model, quality score). Beyond training the router, this is a preference/distillation dataset for model training: the winning responses are supervised-fine-tuning targets, the judge scores are a reward-model signal, and the discovered task boundaries inform pretraining. A better model becomes a better routing target, attracts more traffic, and generates more preference data — the router is the data flywheel for the next model. Model training remains GPU work; the router and its data collection do not.
For callers who prefer not to manage a pool, Enso is also offered as a managed family (enso, enso-flash, enso-ultra) over the same OpenAI-compatible API. enso-ultra is an adaptive fan-out: it probes one task-appropriate arm and escalates to a small panel with verify-then-select only when the probe is low-confidence, so a confident request bills one arm, not the panel.
Every routed request is metered on one plane, so per-request cost is a real, auditable number (not a model-card estimate). Two models:
LLM spend**. It is bounded above by the value it creates — routing away from over-provisioned models typically saves a large fraction of spend — and above what it costs to run (a microsecond CPU decision plus a sampled judge call). The fee is incentive-aligned: it grows only as the customer's routed spend grows, and the customer nets the savings minus the fee.
$/MTok per SKU, each rung billed strictlyabove its upstream cost, at the rung that served.
its pool, sets a cost ceiling, and sends model=auto over api.hanzo.ai/v1 — usable from any OpenAI-compatible client (CLIs, IDE assistants, agent harnesses).
for data, privacy, compliance, or organizational reasons.
routing data, and training on the shared base is opt-in.
native flag engine (/v1/flags), org-scoped, so the surface ships dark and enables per-org without a release.
The learned policy is deliberately a small feature vector and a bilinear form rather than an embedding classifier: it avoids an embedding call (which would dominate the decision at milliseconds) while still capturing task and domain signal, keeping the whole decision on CPU in microseconds. Gating before promotion — rather than continuously overwriting weights — is what makes the loop safe to run autonomously: a regression cannot ship, and the system keeps training until it has a real improvement. Per-org policies over one shared table (via the precedence fold) give every org a router tuned to its own workloads without a per-org serving stack.
auto is additive: a deployment without a router configuration treats it as any other unknown id, and every other model id resolves unchanged. Callers integrate once (OpenAI-compatible) and opt into routing by requesting auto.
The routing ledger is content-free by construction (features and reward only). Reward attachment and the training exports are gated on one predicate, not two kinds of credential: org admin for an org's own data, SuperAdmin for the platform-wide base — membership of the reserved admin org (HIP-0118). A machine doing the export presents its own IAM identity and is subject to the same predicate; there is no token type that substitutes for it. Cross-org isolation is enforced by the precedence fold and the known predicate: a scope can only narrow to models it can serve, never escalate to another tenant's.
hanzoai/engine (hanzo-router, enso).hanzoai/ai.hanzoai/enso.hanzoai/enso-bench.hanzoai/papers/enso).This document is placed in the public domain.