hanzo-universe

Hanzo Universe (github.com/hanzoai/universe) is the single source of truth for all production infrastructure across the Hanzo ecosystem.

Hanzo Universe - Production Kubernetes Infrastructure

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-operator.md, hanzo/hanzo-kms.md, hanzo/hanzo-platform.md, hanzo/hanzo-stack.md, hanzo/hanzo-ingress.md

Overview

Hanzo Universe (github.com/hanzoai/universe) is the single source of truth for all production infrastructure across the Hanzo ecosystem. It contains Kustomize manifests for 50+ services, Terraform configs for DigitalOcean provisioning, Helm charts, GitHub Actions CI/CD workflows, Docker Compose stacks, and end-to-end tests. Everything runs on DOKS (DigitalOcean Kubernetes Service).

Repo: github.com/hanzoai/universe (private) Local path: ~/work/hanzo/universe/

Design Principles

When to Use

Cluster Topology

hanzo-k8s (24.199.76.156)

Primary cluster. Runs all core Hanzo services. DigitalOcean SFO3 region. 4-node worker pool with hostNetwork ingress.

lux-k8s (24.144.69.101)

Lux blockchain cluster. Runs validators (mainnet/testnet/devnet), gateway, explorer, bridge, exchange, and bootnode. Applied separately: kubectl apply -k lux/ --context do-sfo3-lux-k8s.

Deployment

DO App Platform is decommissioned (Feb 2026). All infrastructure is K8s-native.

Repository Structure

universe/
 infra/
 k8s/ # Production Kustomize manifests (main entry point)
 kustomization.yaml # Root -- includes all services below
 Makefile # diff, apply, validate, build, deploy-%
 namespace.yaml # hanzo namespace
 cluster-issuer.yaml # Let's Encrypt ClusterIssuer (cert-manager)
 platform-ingress.yaml # Platform-level ingress rules
 base/ # Shared deployment/ingress templates
 rbac/ # Cluster-wide RBAC (readonly, operator, ci roles)
 ci/ # CI deployer ServiceAccount
 ingress/ # Hanzo Ingress (Traefik-based L7 proxy, 4 replicas)
 # --- Operators ---
 hanzo-operator/ # HanzoService/Datastore/Gateway/MPC/Network/Ingress/Platform CRDs
 kms-operator/ # KMSSecret/KMSDynamicSecret/KMSPushSecret CRDs
 nchain/ # Blockchain node operator (nchain-system namespace)
 # --- Data ---
 sql/ # PostgreSQL 16 + ZAP (StatefulSet, 20Gi PVC)
 kv/ # Valkey + ZAP (StatefulSet, 2Gi PVC)
 docdb/ # DocDB -- FerretDB (MongoDB wire protocol over PostgreSQL)
 storage/ # Hanzo S3 (s3.hanzo.ai)
 search/ # Meilisearch (search.hanzo.ai, backup CronJob)
 vector/ # Qdrant (vector.hanzo.ai, backup CronJob)
 nats/ # NATS core messaging
 stream/ # Kafka wire protocol gateway over PubSub
 cloud-sql/ # Serverless PostgreSQL (Neon-based: pageserver, safekeeper, compute)
 # --- Identity & Security ---
 iam/ # Hanzo IAM IAM (hanzo.id, lux.id, zoo.id, pars.id)
 kms/ # KMS / Hanzo KMS (kms.hanzo.ai)
 login/ # White-label login UI
 mpc/ # Multi-party computation (threshold signing, StatefulSet)
 cloudflare/ # Cloudflare API credentials
 # --- Zero Trust ---
 zt/ # ZeroTier overlay mesh (controller, routers, MCP gateway)
 # --- AI & Models ---
 zen/ # Zen LM Gateway (identity injection + Console tracking)
 models/ # Model registry API (models.hanzo.ai)
 pricing/ # Pricing API + CronJob (pricing.hanzo.ai)
 rag-api/ # RAG API for chat
 crawl/ # crawl4ai web crawler
 # --- Applications ---
 chat/ # Hanzo Chat (LibreChat + DocDB + search)
 bot/ # Hanzo Bot (gateway, hub, docs, agents control plane, playground)
 cloud/ # Cloud API + agent + site
 commerce/ # Commerce API (HPA, PDB, service)
 console/ # Developer console + worker
 billing/ # Billing center
 gateway/ # API gateway (api.hanzo.ai)
 flow/ # Visual AI workflow builder + site
 auto/ # Visual workflow automation
 visor/ # Agent orchestration
 vm/ # Virtual machine management
 vmd/ # VM daemon
 # --- Serverless ---
 openfaas/ # OpenFaaS function control plane (controller, gateway, queue-worker)
 # --- Platform ---
 paas/ # PaaS v2 (platform.hanzo.ai, RBAC, PDB)
 gitops/ # GitOps sync engine
 registry/ # Container registry + PVC
 argocd/ # ArgoCD bootstrap (applications, RBAC, ingress)
 # --- Observability ---
 analytics/ # Event analytics
 insights/ # Product analytics
 monitoring/ # Prometheus ServiceMonitors, alerts, Grafana dashboards, OTel collector
 logging/ # Loki + Promtail log pipeline
 o11y/ # OTel collector config
 status/ # Status pages (hanzo, lux, zoo, pars, adnexus)
 sentry/ # Sentry error tracking
 # --- Sites & Misc ---
 app/ # hanzo.ai main app
 base/ # Database admin UI
 bootnode/ # bootnode.hanzo.ai (API + web)
 bootnode-whitelabel/ # White-label bootnode deployments
 captable/ # Cap table management
 dataroom/ # Data room
 dns/ # DNS service (ConfigMap, PDB)
 lux-build/ # Lux build service
 preview/ # PR preview environments (namespace, certs, pull secrets)
 sign/ # Document signing
 team/ # Project management (own namespace: team)
 # --- External Stacks (separate clusters) ---
 lux/ # Lux blockchain (lux-k8s cluster)
 operator/ # Lux operator
 validators/ # 15 validators across 3 networks (mainnet/testnet/devnet)
 gateway/ # KrakenD API gateway
 explorer/ # Blockscout explorer + frontend
 bridge/ # Cross-chain bridge
 exchange/ # DEX
 bootnode-lux/ # Lux bootnode
 agnost/ # Agnost deployments
 adnexus/ # AdNexus advertising (adnexus-k8s cluster)
 terraform/
 main.tf # DigitalOcean VPC, Droplets, LB, Firewall, DNS
 autoscaler.tf # Docker Swarm autoscaler config
 terraform.tfvars.example # Variable template
 do/
 autoscaler.py # DigitalOcean autoscaler script
 compose.autoscaler.yml # Autoscaler compose
 scripts/
 manager-init.sh # Swarm manager bootstrap
 worker-init.sh # Swarm worker bootstrap
 deploy.sh # One-shot infra deploy (Terraform + Swarm + Universe)
 k8s-deploy.sh # K8s-only deploy script
 README.md # Infra documentation
 helm/
 identity/ # IAM Helm chart (Hanzo IAM)
 Chart.yaml
 values.yaml
 values-production.yaml
 templates/ # deployment, service, ingress, configmap, secrets, pdb
 hanzo-stack/
 values.production.yaml # Production values for full stack chart
 deploy/ # Docker Compose production deployment
 compose.production.yml
 deploy.sh # Production deploy script
 kms.sh # KMS bootstrap
 dns-setup.sh # DNS configuration
 staging/ # Staging environment
 services/ # Docker Compose service definitions
 compose.yml # Main services compose
 compose-hanzo-*.yml # Hanzo service variants (complete, dev, minimal, services)
 compose-infra.yml # Infrastructure compose (postgres, redis)
 compose-lux.yml # Lux services compose
 compose-nodes.yml # Node compose
 config/ # Router configs, IAM configs, MCP servers
 .github/
 workflows/ # 17 CI/CD workflows
 actions/ # Reusable actions (KMS auth, KMS action)
 compose.yml # Root development compose (29k lines)
 compose.production.yml # Production compose
 compose.single-node.yml # Single-node compose
 Makefile # Root Makefile (up, down, status, logs, health, db-backup)
 Cargo.toml # Rust workspace root
 e2e/ # End-to-end tests
 docs/ # Documentation
 scripts/ # Utility scripts

K8s Deployment Commands

# From infra/k8s/:

# Preview changes (dry-run diff)
make diff

# Apply all resources idempotently
make apply

# Validate manifests without applying
make validate

# Render manifests to stdout
make build

# Deploy a single service
make deploy-chat
make deploy-iam
make deploy-kms

# Apply Lux cluster separately
kubectl apply -k lux/ --context do-sfo3-lux-k8s

Core Data Services

| Service | Image | Type | Port | Storage | |---------|-------|------|------|---------| | sql | ghcr.io/hanzoai/sql:latest | StatefulSet | 5432 (PG) + 9651 (ZAP) | 20Gi do-block-storage | | kv | ghcr.io/hanzoai/kv:latest | StatefulSet | 6379 (Valkey) + 9651 (ZAP) | 2Gi | | docdb | DocDB | StatefulSet | MongoDB wire protocol | Via PostgreSQL | | storage | Hanzo S3 | Deployment | S3 API | PVC | | search | Meilisearch | Deployment | HTTP | PVC + backup CronJob | | vector | Qdrant | Deployment | gRPC + HTTP | PVC + backup CronJob | | nats | NATS | - | 4222 | - | | stream | Kafka protocol | Deployment | Kafka wire | - | | cloud-sql | Neon-based | Multi-component | 5432 | S3-backed |

The sql and kv images are unified containers: PostgreSQL 16 + embedded ZAP server, and Valkey + embedded ZAP server respectively. Both expose dual ports. A backward-compat postgres Service alias exists for sql.

Secret Management

All secrets flow through KMS (kms.hanzo.ai) via the KMS Operator.

CRDs (API group: secrets.lux.network/v1alpha1)

| CRD | Purpose | |-----|---------| | KMSSecret | Pulls secrets from KMS and syncs to K8s Secret | | KMSDynamicSecret | Dynamic secrets with TTL and lease management | | KMSPushSecret | Pushes K8s Secrets back to KMS |

How KMSSecret Works

apiVersion: secrets.lux.network/v1alpha1
kind: KMSSecret
metadata:
 name: bot-kms-sync
 namespace: hanzo
spec:
 hostAPI: http://kms.hanzo.svc.cluster.local/api
 resyncInterval: 60
 authentication:
 universalAuth:
 credentialsRef:
 secretName: universal-auth-credentials
 secretNamespace: hanzo
 secretsScope:
 projectSlug: secrets-639-c
 envSlug: prod
 secretsPath: /bot
 managedSecretReference:
 secretName: bot-secrets
 secretNamespace: hanzo
 secretType: Opaque

Authentication methods: Universal Auth (credentialsRef), Kubernetes Auth (serviceAccountRef + identityId), Service Token (legacy).

The universal-auth-credentials secret (clientId/clientSecret) is created out-of-band via kubectl. Never committed to git.

Services with kms-secrets.yaml: bot, bootnode, captable, chat, cloud, commerce, insights, o11y, search, sign, sentry, storage, team, vector, zen, zt.

Services with direct secret.yaml/secrets.yaml (legacy or bootstrap): analytics, app, auto, cloudflare, flow, gateway, kv, mpc, paas, registry, sql.

Ingress Architecture

The ingress layer is a Traefik-based L7 reverse proxy (ghcr.io/hanzoai/ingress:latest):

Cloudflare DNS wildcards (*.hanzo.ai) round-robin across all 4 pod node IPs.

Hanzo Operator CRDs

The Hanzo Operator (ghcr.io/hanzoai/operator:latest) runs in hanzo-operator-system namespace and manages 7 CRDs under hanzo.ai/v1alpha1:

| CRD | Description | |-----|-------------| | HanzoService | Standard service (Deployment, Service, Ingress, HPA, PDB, NetworkPolicy, KMSSecret) | | HanzoDatastore | Stateful data service (StatefulSet, headless Service, PVC, CronJob backup) | | HanzoGateway | API gateway (KrakenD-based, ConfigMap routing) | | HanzoMPC | Multi-party computation (StatefulSet, dashboard, cache) | | HanzoNetwork | Blockchain validator network (StatefulSet, bootnode, indexer, explorer, bridge) | | HanzoIngress | Multi-host Ingress with cert-manager TLS | | HanzoPlatform | Meta-CRD composing all the above |

34 sample CRs in infra/k8s/hanzo-operator/crs/ cover every deployed service. PodDisruptionBudgets in pdbs/ for critical services (cloud-api, console, gateway, hanzo-app, hanzo-login, iam, kms, models, paas, pricing).

Lux Blockchain Stack (lux-k8s)

Applied separately via kubectl apply -k lux/ --context do-sfo3-lux-k8s:

Monitoring and Observability

Prometheus ServiceMonitors

17 ServiceMonitors scrape metrics from: IAM, commerce, gateway, bot, cloud, KMS, chat, analytics, console, PaaS, insights, auto, flow, base, zen, plus OTel collector.

Grafana Dashboards

6 ConfigMap-based dashboards: Payment monitoring, Platform health, Web analytics, Product usage, AI operations, Per-organization metrics.

Alerts (PrometheusRule)

Critical alerts: PaymentFailureRateHigh (>5% failure over 5m), ServiceDown (any hanzo namespace service unreachable >2m), PodCrashLooping (>5 restarts in 15m), HighErrorRate5xx (5xx error rate threshold).

Logging

Loki + Promtail pipeline with Grafana log exploration dashboard.

Tracing

OpenTelemetry Collector (otel-collector.hanzo.svc:4317) receives traces from ingress and application services.

RBAC Model

Three cluster-wide roles in rbac/:

| Role | Scope | Purpose | |------|-------|---------| | readonly | Cluster | Read-only access for developers | | operator | Cluster | Full management for operators | | ci | Cluster + namespace | Automated deployments from CI |

CI/CD Workflows

17 GitHub Actions workflows in .github/workflows/:

| Workflow | Trigger | Purpose | |----------|---------|---------| | ci.yml | Push/PR | Lint, test, validate manifests | | deploy.yml | Manual | Deploy specific services | | deploy-production.yml | Push to main | Full production deploy | | auto-deploy.yml | Various | Auto-deploy on changes | | e2e-production.yml | Post-deploy | End-to-end production tests | | nightly-backup.yml | Cron | Database backups | | pr-preview.yml | PR | Preview environments | | release.yml | Tag | Release artifacts | | build-zap-images.yml | Manual | Build sql/kv unified images | | reusable-deploy-service.yml | Called | Reusable service deploy | | reusable-deploy-cf-pages.yml | Called | Cloudflare Pages deploy | | reusable-deploy-static.yml | Called | Static site deploy | | reusable-go.yml | Called | Go build/test pipeline | | reusable-python.yml | Called | Python build/test pipeline | | reusable-rust.yml | Called | Rust build/test pipeline | | reusable-typescript.yml | Called | TypeScript build/test pipeline | | reusable-kms-oidc.yml | Called | KMS OIDC auth for CI |

Custom GitHub Actions in .github/actions/: kms-action (fetch secrets), kms-auth (authenticate to KMS).

Terraform Infrastructure

infra/terraform/main.tf provisions on DigitalOcean:

Docker Compose (Local Development)

Root Makefile orchestrates three compose files:

make setup # Create network, start postgres + redis + ClickHouse
make up # Start all services (services + datastore + IAM)
make down # Stop everything
make status # Show service status across all compose files
make health # Health check all services
make db-backup # Backup PostgreSQL + ClickHouse

Individual service targets: make traefik, make iam, make cloud, make analytics, make datastore, make platform, make infra.

Key Domain Mapping

| Domain | Service | Cluster | |--------|---------|---------| | hanzo.id, lux.id, zoo.id, pars.id, id.ad.nexus, id.bootno.de | IAM (Hanzo IAM) | hanzo-k8s | | kms.hanzo.ai | KMS (Hanzo KMS) | hanzo-k8s | | api.hanzo.ai | API Gateway | hanzo-k8s | | console.hanzo.ai | Developer Console | hanzo-k8s | | cloud.hanzo.ai | Cloud Dashboard | hanzo-k8s | | chat.hanzo.ai, hanzo.chat | Hanzo Chat | hanzo-k8s | | platform.hanzo.ai | PaaS | hanzo-k8s | | flow.hanzo.ai | Visual Workflow | hanzo-k8s | | auto.hanzo.ai | Automation | hanzo-k8s | | commerce.hanzo.ai | Commerce API | hanzo-k8s | | analytics.hanzo.ai | Event Analytics | hanzo-k8s | | search.hanzo.ai | Full-text Search | hanzo-k8s | | vector.hanzo.ai | Vector DB | hanzo-k8s | | s3.hanzo.ai | Object Storage | hanzo-k8s | | status.hanzo.ai | Status Page | hanzo-k8s | | hanzo.team | Project Management | hanzo-k8s | | api.lux.network | Lux Gateway (KrakenD) | lux-k8s | | explorer.lux.network | Blockscout Explorer | lux-k8s |

Database Strategy

All databases are in-cluster PostgreSQL (no managed DBaaS). The sql StatefulSet serves multiple databases for different services via init-db.sql. Valkey (kv) handles caching and queues. DocDB provides MongoDB wire protocol compatibility over PostgreSQL via FerretDB.

Cloud SQL (Neon-based) provides serverless PostgreSQL with pageserver, safekeeper, storage-broker, and compute nodes.

Common Operations

# Deploy everything to hanzo-k8s
cd ~/work/hanzo/universe/infra/k8s && make apply

# Deploy single service
cd ~/work/hanzo/universe/infra/k8s && make deploy-chat

# Preview changes
cd ~/work/hanzo/universe/infra/k8s && make diff

# Check KMS secret sync status
kubectl get kmssecrets -n hanzo

# View operator logs
kubectl logs -n hanzo-operator-system deploy/hanzo-operator-controller-manager

# View ingress logs
kubectl logs -n hanzo -l app=hanzo-ingress --tail=100

# Apply Lux stack
kubectl apply -k ~/work/hanzo/universe/infra/k8s/lux/ --context do-sfo3-lux-k8s

Related Skills

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: kubernetes, k8s, infrastructure, kustomize, terraform, production, deployment, ingress, operators, secrets, monitoring Prerequisites: Kubernetes, kubectl, Kustomize, Terraform (for provisioning), Docker Compose (for local dev)