--- hip: 1311 title: Experiment — Arms, Assignment, a Verdict author: Hanzo AI type: Standards Track category: Interface capability: experiment status: Final implementation-go: shipped created: 2026-08-20 requires: HIP-1190, HIP-0026, HIP-0106, HIP-0139 --- # HIP-1311: Experiment — Arms, Assignment, a Verdict ## Abstract `/v1/experiment` is where a comparison is registered, run and settled: two or more arms, a deterministic assignment, an outcome measured against the control, a decision that locks the winner. The implementation is `hanzoai/cloud` `apps/experiment`. It is a **composition, not a fourth engine**. It owns the experiment registry and nothing else — assignment comes from `flags`, outcomes from the analytics warehouse (HIP-1190), evidence from the research record (HIP-1145). Each of those already has an owner, and none is copied here. ## Motivation This capability shared HIP-0063 with `flags` — two specifications in one file, which HIP-0139 §6 allows only for a merge in flight. The store boundary is where the two separate: flags owns definitions in a per-(org, project) file and evaluates them as a pure function; experiment owns a per-org registry of comparisons and composes that evaluation. The boundary is load-bearing rather than tidy. Flags is the primitive every other plane composes — admission, stage gating (HIP-0139 §8.2), traffic splits in subsystems that never touch this surface. An experiment is one caller of it, and filing the caller's specification inside the primitive's makes the primitive read as though it existed to serve A/B tests, which is the wrong way round. ## Specification The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as in RFC 2119. ### §1 The addresses Seven operations, every one under `/v1/experiment` (`manifest/apps.go:411`, `plugin/experiment/openapi.json`), all typed, none declared: - `GET|POST /v1/experiment` — list; create, which writes the multivariate flag definition the arms are served from - `GET /v1/experiment/{id}` — read, with results - `GET /v1/experiment/{id}/assign` — the arm for a subject - `POST /v1/experiment/{id}/analyze` — run the analysis over the evidence - `POST /v1/experiment/{id}/decide` — stop, and lock the winner by rewriting the flag definition's weights to 100% - `GET /v1/experiment/health` Create and decide both write a flag definition, deliberately: the arm a request is actually served comes from the flag, so an experiment holding its own copy of the split would be a second answer to one question, free to disagree with the answer serving traffic. ### §2 Assignment is the flags hash Assignment MUST be the same deterministic rollout hash the flags surface evaluates — `flags.Assign`, in process, no network hop — so a subject's arm is one value whether read here, evaluated at the flags endpoint, or composed by another subsystem in the binary (`apps/campaign` splits traffic the same way). Because the hash is a pure function of subject, key and seed, there is no assignment store: stickiness is a property of the arithmetic rather than of a row somebody has to keep. Changing the seed reshuffles the arms on purpose; nothing else does. ### §3 The outcome, and the analysis that ships Outcomes MUST be read from the one analytics plane rather than a second exposure topic: the analyze fold reads each subject's outcome from the org-scoped event query and joins it to the arm by subject id (`apps/experiment/analyze.go`). A serving surface that records an exposure records an ordinary analytics event through the same capture endpoint as every other event (HIP-1190) — no separate stream, no emitter in the request path. The analysis is one method: a two-proportion z-test against the control arm over the append-only evidence rows, computed with stdlib `math.Erfc` and no dependency (`apps/experiment/analyze.go:83,152-166`). Degenerate input — an arm with no subjects, a control with no conversions — MUST be answered as degenerate rather than scored, because a p-value printed over an empty arm is the one output of this surface a reader would act on and should not. Richer methods are open design; any that lands MUST evaluate through §2 rather than a second bucketing. ### §4 Tenant, store, price, emission, stage, upstream The org is the validated principal's — `principal.Org`, and `principal.Project` where the flag definition is scoped (HIP-0026) — never a client-supplied header. The credential is the org's ordinary bearer, there is no key family specific to this surface, and an unauthenticated caller is refused. It owns one store: the experiment registry, a per-org SQLite file (`{DataDir}/orgs/{slug}/experiments.db`, `apps/experiment/store.go:3-9`). Isolation is physical, so a cross-tenant read is not a predicate that can be wrong. It owns no assignment store (§2), no definition store (flags'), no event store (analytics') and no evidence store (research's). It is free, in those words: `Price: cloud.Free` (`plugin/experiment/main.go:22`). It publishes no events on the bus, so a customer's webhooks (HIP-1310) receive nothing from it — exposure reaches the warehouse from the surfaces that serve the traffic. It emits nothing to observability beyond the request span every route gets. The stage is `beta` (HIP-0139 §8): the manifest row declares it, so an org reaches the surface by the `experiment` flag and it is in no generated client, tool list or public page. `flags` itself stays `ga` — it is the mechanism stage-gating rides, so it cannot sit behind a flag. It derives from no OSS upstream: the significance test is stdlib arithmetic and the evaluator it composes is `flags`. ## Rationale The alternative is an engine of its own — its own bucketing, its own exposure topic, its own metric store. Each is a second copy of something that has an owner, and the copies fail in one direction: the arm a dashboard reports is not the arm the request was served, because two hashes drifted or two topics were configured apart. Composition costs a function call and removes the class. The other alternative — no registry, experiments expressed purely as multivariate flags — loses what makes a comparison settleable: which arm is the control, what the primary outcome is, and when it was decided. That is what the registry holds, and why §4's store boundary exists at all. ## Security Considerations The wrong implementation writes another tenant's traffic split. Decide rewrites a flag definition to 100% of one arm, so a cross-tenant write here is a stranger choosing which model, price or copy another org's customers are served — and it would read as an ordinary experiment concluding. The per-org file in §4 is what makes that unreachable rather than merely refused. The read side is a roadmap: an org's experiment roster names what it is about to ship and to whom. The exposure record carries the subject identifier and the arm and MUST NOT carry request or response content — outcome values are aggregates, so no prompt, completion or document reaches this plane. ## References - HIP-1190 — Analytics Event Standard - HIP-0026 — Identity and Access Management - HIP-0063 — Feature Flags Standard - HIP-0106 — Hanzo Plugin Contract - HIP-0139 — Capability - HIP-1145 — Research — The Experiment Record ## Copyright Released under CC0 1.0 Universal Public Domain Dedication.