HIP-121: BYO Compute Fleet & Metered Billing. Status Draft. Hanzo's own standard — read this before implementing against it.
This proposal defines the unified compute fleet: one per-org registry of attached compute — managed Hanzo clusters, bring-your-own (BYO) Kubernetes clusters, BYO GPU / bare-metal boxes, and bring-your-own-cloud (BYOC) accounts — exposed through a single /v1/clusters surface, and a three-tier metered billing model that charges each source by how it connects:
AWS / Azure / GCP / DigitalOcean / Nebius), pulled from that cloud's own cost API.
metal, a workstation, a desktop-linked box).
hanzo.network.Anyone brings compute → sees it in their fleet → schedules work on it → is billed fairly, on one invoice, through the canonical commerce/metering path — no manual step. Tenant isolation is the org boundary, enforced end to end: every credential and kubeconfig is sealed per-org in KMS (the MPC nodes see only ciphertext), and every fleet call is scoped by the gateway-validated X-Org-Id / X-Project-Id, never a client field.
The fleet is the hanzoai/cloud clients/fleet registry consumed by clients/visor (the /v1/clusters surface) and clients/ml (workload federation), per HIP-0106. Visor (HIP-0053) is the machine/cluster control plane. This HIP is the fleet + billing contract that sits on top of both.
A fleet is one org's set of compute sources. Each source has a kind, is owned by exactly one org (narrowed by project), and is served through the ONE /v1/clusters surface. There is never a second cluster surface.
| Source kind | How it attaches | Where it lives | |---|---|---| | Managed cluster | Hanzo provisions k8s (Kubernetes today; AWS/Azure/GCP via Visor provider backends) | Visor — merged into GET /v1/clusters | | BYO k8s cluster | Org pastes a kubeconfig → POST /v1/clusters | fleet.Registry (KMS-sealed) | | BYO GPU / bare metal | Box becomes a cluster via k3s + device plugins, OR a single box via hanzo gpu connect / desktop-link | fleet.Registry (cluster) / FleetWorker (box) | | Cloud account (BYOC) | Org connects THEIR AWS/Azure/GCP/DO/Nebius credentials; Visor provisions + manages in their account | Visor Provider (per-owner, sealed) |
The first three are compute targets the fleet can schedule onto; the fourth is a provider the fleet provisions against. All four are billed by the one model in "Three-tier billing" below.
/v1/clusters APIOrg-scoped; the org is the gateway-validated owner (tenant(c)), never a client field. Reference: cloud/clients/visor/byo.go.
| Method | Path | Purpose | |---|---|---| | GET | /v1/clusters | The org+project fleet: managed clusters (from Visor) merged with BYO clusters (from the registry), one list | | POST | /v1/clusters | Attach. A kubeconfig body ⇒ BYO attach (validate → seal → index). A provider+spec body ⇒ managed provisioning (Visor) — the sibling variant | | DELETE | /v1/clusters/:id | Detach a BYO cluster (index + sealed kubeconfig + cached client) |
POST /v1/clusters body:
{ "name": "spark-k8s", "kubeconfig": "<yaml>", "provider": "k3s", "default": true }Attach is a fail-closed pipeline:
tenant(c) → org, or 403 (X-Org-Id required).name (lowercased) and kubeconfig non-empty.503 (never store a kubeconfig inplaintext).
s.bill.Gate) — the customer brings the compute; Hanzo meters the management plane. Fee = ResourceFeeCents("CLOUD_COMPUTE_FEE_CENTS", "byo-cluster"). Fail-closed on an unfunded org.
s.fleet.Register(ctx, org, project, name, kubeconfig, provider, default) —reach the cluster, seal the kubeconfig, index the metadata.
s.bill.Meter(...) — record the nominal fee against the paying org, then 201 with the cluster view.
Reference: cloud/clients/fleet/fleet.go. The registry is the single source of truth consumed by both the /v1/clusters surface and ML serving. Its whole state is in the org's KMS; a nil KMS makes it a graceful, fail-closed no-op (Enabled() is false; Register errors, List/DynForOrg return empty).
func (r *Registry) Register(ctx, org, project, name, kubeconfig, provider string, isDefault bool) (Cluster, error)
func (r *Registry) List(org, project string) ([]Cluster, error)
func (r *Registry) Deregister(org, project, name string) (bool, error)
func (r *Registry) DynForOrg(org, project string) dynamic.Interface // the ONE federation seamRegister builds a dynamic client from the kubeconfig, lists nodes, and reads nvidia.com/gpu + amd.com/gpu from each node's status.allocatable. An unreachable cluster or unusable kubeconfig is rejected before anything is stored. The metadata (Cluster{Nodes, NvidiaGPU, AmdGPU, ...}) is derived, never client-asserted.
only non-secret metadata is indexed. The kubeconfig is never returned by any read path.
name within the (org, project) shard (upsert).DynForOrg returns the org+project's default cluster's client (KMS-loaded, cached) or nil — in which case the caller falls back to the home in-cluster client. This is how clients/ml places a workload on the org's chosen cluster with zero branching.
KMS key layout (the one place the project segment is added or omitted):
scopeRef(org, project) = org # default project (legacy, byte-identical)
= org + "/" + project # non-default project shard
index ref: <scopeRef>/fleet/clusters
config ref: <scopeRef>/fleet/clusters/<name>/kubeconfig # the sealed kubeconfigThe KMS client is mpcseal — an MPC/threshold store; the nodes hold only ciphertext. It is opened once from CLOUD_KMS_NODES + CLOUD_KMS_PASSPHRASE (the one bootstrap env shared with the rest of cloud); absent ⇒ disabled, never plaintext.
The org boundary is enforced once, at the principal's owner, per HIP-0134 / HIP-0112:
X-Org-Id / X-Project-Id, then mints them from the validated JWT: the owner claim → X-Org-Id; the token's project scope → X-Project-Id (default "default", minted only for a non-default project). These are ANDed authorization inputs, never identity authority on their own.
principal.Project(c) / principal.IsDefaultProject and the org via tenant(c). X-Org-Id and X-Project-Id propagate over the ZAP interconnect to visor / cloud / ml.
dimension (a distinct fleet shard, a distinct metering Actor), never a billing destination. So z@hanzo.ai bills only the hanzo org, and one project's fleet is a distinct shard within it.
Connected compute is metered by HOW it connects — all through the ONE commerce/metering path, so a customer sees one invoice however their compute is attached. The tier is a property of the source, resolved at connect. Rates are config-surfaced (the pricing service is the source of truth); the values below are the canonical defaults.
The org runs compute on its own AWS / Azure / GCP / DO / Nebius account. Hanzo reads that account's real spend and meters 1%.
service/cloud_cost.go, one CloudCostReader seam, one readerper cloud):
| Cloud | Cost source | Read | |---|---|---| | AWS | Cost Explorer | ce:GetCostAndUsage, UnblendedCost, month-to-date (us-east-1) | | DigitalOcean | Customer balance | GET /v2/customers/my/balance → month_to_date_usage | | Azure | Cost Management | scope-level cost query | | GCP | BigQuery billing export | SELECT SUM(cost) FROM <export> WHERE invoice.month = 'YYYYMM' |
AdvanceCostCursor(owner, provider, month, currentMTDCents) advances a per-(owner, provider, month) watermark and returns the newly-billable increment; the collector meters 1% of the increment, so across a month it totals 1% of that month's spend, exactly once. Month rollover is implicit in the cursor PK. The daily unit is serialized cluster-wide by ClaimBillingUnit("byoc:<owner>:<provider>:<YYYYMMDD>").
ErrCostUnavailable; the collector skips that account — no fee — rather than inventing a figure. No spend → no fee. Requires cost-read scope on the stored creds (documented per cloud).
A bare-metal, GPU, or desktop-linked box that costs the customer nothing per hour. ConnectFleetWorker (via hanzo gpu connect / desktop link) arms a FleetWorker{Kind: "byo-hardware", DeviceCount, GpuModel} keyed by (Owner, Name) with Project as the attribution dimension. The monthly meter reads GetConnectedFleetWorkers (cross-tenant, org recovered per row) and debits 100 cents × DeviceCount per connected box, once per month, under ClaimBillingUnit("device:<owner>:<worker>:<YYYYMM>"). DisconnectFleetWorker stops the fee (the row is retained so the closing month still bills its last device count).
If the connected box's validator address is in the hanzo.network mainnet validator set, it is natively available to the cloud at no fee — it already earns validator economics and secures the chain. chain.ValidatorExemption(ctx, address) is the gate, and it never fabricates an on-chain fact:
| State | Result | Billing | |---|---|---| | Wired + address in set | (true, "exempt") | free | | Wired + address not in set | (false, "billed") | billed as a device (Tier B) | | Lookup unwired (HANZO_VALIDATOR_INDEXER_URL empty) | (false, "pending-chain-wiring") | billed, flagged | | Wired + lookup errored | (false, "lookup-error", err) | billed (fail-safe), flagged |
The exemption is granted only on a positive, verified membership read (indexer → read; the chain is authoritative and read-only). It activates automatically the moment the operator wires the indexer; nothing is ever exempted on an unverified claim.
attach / connect ──► tier resolved (property of the source)
│
├─ BYO cluster attach ─► nominal mgmt-fee Gate + Meter (WIRED, live)
├─ resell machine up ─► hourly debit, MeterLease-guarded (WIRED, live)
├─ BYOC account ─► daily: 1% × cost increment (primitives shipped; loop = roadmap)
└─ BYO device ─► monthly: $1 × DeviceCount − validators (primitives shipped; loop = roadmap)
│
▼
commerce/metering (one client per org: NewMeteringClient → metering.Record)
│ RequestID = the billing-unit key; OTel mirror via telemetry.CountMetered
▼
commerce ── per-org ledger ──► ONE invoiceExactly-once is enforced outside the RequestID. Commerce does not dedup the withdraw on requestId, and visor runs replicas: 2 with no leader election, so every billable unit is guarded by an insert-once-wins lease:
MeterLease (ClaimMeterHour, PK = YYYYMMDDHH) — the hourly resell-computesweep. Lives on the shared linearizable engine (never per-org SQLite — two pods with separate files would each "win" and double-debit).
BillingLease (ClaimBillingUnit, PK = the unit string) — the daily BYOC andmonthly device units. A DB error yields "not claimed" ⇒ fail safe = do not bill (a missed unit is reconciled later; a double debit is not).
PriceToCents is the one price→cents rule (Ceil, with a 1e-9 epsilon so 0.07 × 100 does not round to 8¢; a $0 price is free).
| Surface | Audience | Scope | Backing | |---|---|---|---| | console.hanzo.ai | Any org (self-service) | Its OWN fleet, usage, schedules, billing | /v1/clusters + /vm/v1/machines, org-scoped | | admin.hanzo.ai | SuperAdmin (owner == "admin") | ALL orgs' fleets + usage + revenue, one board | /v1/admin/* cross-tenant aggregates |
Per the SuperAdmin standard: the unified all-org overview is gated on membership of the reserved admin org (owner == "admin"), the only cross-tenant scope. A brand-org admin (e.g. z@hanzo.ai, an org admin of hanzo) sees only the hanzo fleet on console — never the cross-tenant board.
The single cloud binary (HIP-0106) + white-label-by-hostname make one codebase serve three product modes with no forks:
unified, per-org billing.
compute (the fleet). Billed the nominal management fee + the three-tier model.
enterprise's own cloud + K8s, white-labeled to their brand, via the GCP/AWS/Azure Marketplace or the operator/Helm. Billed by license + entitlement metering, not per-org SaaS billing.
Everything in this HIP stays mode-agnostic: never hardcode hanzo.ai, always resolve brand + endpoints from config, never assume Hanzo's KMS/commerce is reachable.
Why one /v1/clusters for every org. The fleet is inherently cross-tenant (every org's compute in one control plane). One surface with one registry is the only way ML serving, the console, and the admin board see the same list. A parallel /v1/ml/clusters was built and deliberately deleted: the moment there are two lists they drift, and dynForOrg cannot know which cluster the org actually chose. One registry, two consumers (clients/visor, clients/ml), never a second surface. This is the "one and only one way" rule applied to compute.
Why 1% / $1 / free. Each rate matches what the compute actually costs the customer, so the fee is never hostile:
thin, usage-aligned management fee that scales to zero when they are idle. It is pulled from the cloud's own cost API, so it is auditable against the customer's own bill.
of, so a flat, trivially-predictable per-device fee is the honest unit. It arms on connect and stops on disconnect.
the chain; contributing spare cycles to the cloud is part of that contract. Charging it would be double-dipping.
One commerce/metering path renders all three as ledger lines, so a customer with a validator, two GPU boxes, and an AWS account sees one invoice — not three billing systems.
Why KMS-sealed, per-org isolation. A kubeconfig or a cloud credential is a full grant of authority over the customer's infrastructure. The blast radius of a leak is the customer's entire cluster or account. Sealing every secret in the org's own KMS project (the MPC nodes see only ciphertext), keying it under a per-org(+project) ref, and taking the org from the gateway-validated JWT — never a client field — makes cross-tenant read structurally impossible: there is no code path where org A's request resolves to org B's key. A disabled KMS fails the whole feature closed rather than degrading to plaintext.
Why watermark + lease instead of trusting request IDs. Commerce does not dedup withdrawals on requestId, and visor runs multiple replicas. Idempotency therefore cannot live in the client key; it must be a cluster-wide, insert-once-wins claim, with a DB error resolving to do not bill. Money safety biases every failure toward under-billing (reconcilable) over double-billing (a refund and a trust hit).
Everything below is real code in hanzoai/cloud and hanzoai/visor. The fleet surface and registry are shipped and live (cloud ≥ v1.786.90); the three-tier billing primitives are shipped, build, and pass tests; the daily and monthly billing orchestrator loops are the remaining roadmap — called out explicitly below. No stubs are claimed as shipped.
cloud/clients/fleet/fleet.go — the per-org KMS-sealed Registry (Register / List / Deregister / DynForOrg, scopeRef project sharding, kubeconfig sealed via mpcseal, GPU inventory via node list). Builds clean.
cloud/clients/visor/byo.go — attachCluster (POST /v1/clusters), detachCluster (DELETE /v1/clusters/:id), byoClusters (the GET merge), with the nominal CLOUD_COMPUTE_FEE_CENTS Gate/Meter. Builds clean.
cloud/resource_billing.go — ResourceMeter.Gate / Meter / MeterUsage, ResourceFeeCents (the wired nominal-fee path).
nvidia.com/gpu: 1. Requires the nvidia device-plugin ≥ v0.18.0 (earlier versions PANIC on the GB10's unified memory: nvmlDeviceGetMemoryInfo "Not Supported"; v0.18 logs and ignores it).
amd.com/gpu: 1, rocm/k8s-device-plugin.
kserve-crd + kserve-resources), RawDeployment mode (no Knative/Istio; warm serving).
X-Project-Id; principal.Project / principal.IsDefaultProject; every keyed surface (fleet ref, ml-<org> → ml-<org>-<project> namespaces, metering Actor) carries org+project. Live.
issuer, hanzo-org:<org> DO tags; per-owner cloud Provider connectors for DO / AWS / Azure / GCP / Hetzner / Lightsail (+ Nebius added this cycle); the hourly resell-compute meter (MeterRunningMachines + MeterLease) driven by the ticker.
Verified: go test ./object/... ./service/... → ok (visor); go build clean for cloud/clients/{fleet,visor} and visor/chain.
service/cloud_cost.go + service/cost_{aws,do,azure,gcp}.go — the CloudCostReader seam and one real cost reader per cloud (Tier A input).
object/cost_cursor.go — CostCursor + AdvanceCostCursor (thebill-the-increment watermark that makes 1% idempotent across a month).
object/fleet_worker.go — FleetWorker model + ConnectFleetWorker / DisconnectFleetWorker / GetConnectedFleetWorkers (the Tier B / C registry).
object/billing_lease.go — BillingLease + ClaimBillingUnit (daily byoc:… and monthly device:… exactly-once).
object/meter_lease.go — MeterLease + ClaimMeterHour (hourly sweepexactly-once).
service/metering.go — the one commerce path (NewMeteringClient, PriceToCents, MeterRunningMachines).
chain/validator.go — ValidatorExemption (Tier C, fail-safe honestycontract).
Every primitive exists and tests pass, but no production ticker yet calls AdvanceCostCursor / GetConnectedFleetWorkers to emit the debits. The orchestrator is: enumerate active providers/workers → read cost / count devices → apply the rate → metering.Record under a BillingLease. The nominal per-attach management fee is wired; the usage-aligned tiers are not.
hanzo gpu connect / desktop-link controller route that calls ConnectFleetWorker — the model and lifecycle exist; the HTTP handler does not.
HANZO_VALIDATOR_INDEXER_URL) — unwired by default, so Tier C reports pending-chain-wiring until the operator provisions it.
dynForOrg federation wired into the serving handlers (create/list/get/del/predict) — staged; returns the home client for non-BYO orgs, so it is a safe drop-in.
pattern exists; the combined fleet+revenue overview is a gap to close.
HIP-0302 / HIP-0107) — staged consolidation wave; visor Base mode is safe under replicas: 1 today (the MeterLease leader election needs a shared linearizable store).
The org is the isolation primitive, taken from the gateway-validated JWT owner claim and propagated as X-Org-Id, never from a client field. Every fleet method (Register / List / Deregister / DynForOrg) and every /v1/clusters handler takes (org, project) as resolved by tenant(c) / principal.Project. There is no code path where one org's request resolves another org's clusters: lux sees only lux's fleet, zoo only zoo's, a customer only their own, and within an org each project is a distinct shard.
under a per-org(+project) ref; the mpcseal nodes hold only ciphertext.
(list, get, view) ever returns it; only derived non-secret metadata (node count, GPU counts, endpoint) is surfaced.
Enabled() false; Register errors; attach returns 503). Plaintext persistence is structurally impossible — there is no non-KMS store for the secret.
or hostile kubeconfig is rejected up front; the derived inventory cannot be spoofed by the client.
ErrCostUnavailable and areskipped when creds lack scope; the validator exemption is granted only on a positive, verified on-chain read. No spend and no verified fact is ever invented.
lease (MeterLease / BillingLease); a DB error resolves to not billing. Under-billing is reconcilable; double-billing is not.
own server-side tag / owner column (e.g. the hanzo-org:<org> DO tag injected at launch), never from a client parameter. Billing keys on the org; the project refines attribution without changing the debit destination.
issuer (HIP-0111); a sibling brand's token is rejected, while every org within the brand is accepted — multi-tenant, not multi-trust.
The unified all-org fleet, usage, and revenue overview is gated on membership of the reserved admin org (owner == "admin") — the only cross-tenant scope, per the SuperAdmin standard. Per-org admins are confined to their own org's console. Trusting a per-org isAdmin for the platform board would be a privilege- escalation bug; the predicate is owner == "admin", checked the same way everywhere.
owner claimclients/fleet, clients/visor, clients/ml in one binarycloud/clients/fleet/fleet.go, cloud/clients/visor/byo.go, cloud/resource_billing.go — reference implementationvisor/service/{cloud_cost,cost_aws,cost_do,cost_azure,cost_gcp,metering}.go, visor/object/{cost_cursor,fleet_worker,billing_lease,meter_lease}.go, visor/chain/validator.go — billing primitiveshanzoai/commerce/metering — the canonical metering client and per-org ledgerCopyright and related rights waived via CC0.