hanzo-deploy

Hanzo deploys all services via Kubernetes manifests stored in the universe repo (hanzoai/universe).

Hanzo Deploy - Deployment Guide

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-k8s.md, hanzo/hanzo-platform.md, hanzo/hanzo-kms.md

Overview

Hanzo deploys all services via Kubernetes manifests stored in the universe repo (hanzoai/universe). Universe is the single source of truth for all infrastructure. Branch main is production, branch dev is staging. All images are built by CI/CD on self-hosted runners and pushed to GHCR. Never build images locally unless explicitly requested.

When to use

Hard requirements

  1. Universe is the source of truth: All K8s manifests live in ~/work/hanzo/universe/infra/k8s/
  2. main = production, dev = staging: Never push untested changes directly to main
  3. Self-hosted runners only: Never use GitHub Actions billing. All builds run on the 7-node runner pool in hanzo-k8s
  4. GHCR for images: ghcr.io/hanzoai/<service>:<tag>, always --platform linux/amd64
  5. KMS for secrets: All secrets via KMSSecret CRDs synced from kms.hanzo.ai. Never hardcode secrets in manifests
  6. No local builds: CI/CD builds for all architectures. Do not build images on dev machines
  7. Kustomize, not Helm: kubectl kustomize . | kubectl apply -f - for idempotent deploy
  8. No nginx, no caddy: Hanzo Ingress (Traefik) or the static plugin handles serving

Quick reference

| Item | Value | |------|-------| | Universe repo | github.com/hanzoai/universe (private) | | Local path | ~/work/hanzo/universe/infra/k8s/ | | Prod cluster | do-sfo3-hanzo-k8s (22 nodes) | | Staging cluster | do-sfo3-hanzo-dev-k8s | | Registry | ghcr.io/hanzoai/* | | DO registry | registry.digitalocean.com/hanzo (5/5 repo limit, avoid) | | Runner pool | 7x32GB nodes in hanzo-k8s | | KMS | kms.hanzo.ai |

Deployment workflow

Standard service deploy

# 1. Edit manifests in universe
cd ~/work/hanzo/universe/infra/k8s/<service>

# 2. Apply to production
kubectl --context do-sfo3-hanzo-k8s kustomize . | kubectl apply -f -

# 3. Verify
kubectl --context do-sfo3-hanzo-k8s -n hanzo get pods -l app=<service>
kubectl --context do-sfo3-hanzo-k8s -n hanzo logs -l app=<service> --tail=50

New service checklist

  1. Create directory in universe/infra/k8s/<service>/
  2. Write kustomization.yaml, deployment.yaml, service.yaml
  3. Add KMSSecret CRD in secrets.yaml for any secrets
  4. Add Ingress resource with ingressClassName: hanzo
  5. Add Cloudflare DNS record pointing to cluster LB (165.232.146.176 for hanzo-k8s)
  6. Set up CI/CD workflow in service repo to build+push image to GHCR
  7. Apply manifests: kubectl kustomize . | kubectl apply -f -

Image build (CI/CD)

All images are built in CI/CD pipelines, never locally.

# Typical .github/workflows/build.yml
name: Build and Push
on:
 push:
 branches: [main, develop]
jobs:
 build:
 runs-on: self-hosted # MUST use self-hosted runners
 steps:
 - uses: actions/checkout@v4
 - name: Login to GHCR
 run: echo "${{ secrets.GHCR_TOKEN }}" | docker login ghcr.io -u hanzoai --password-stdin
 - name: Build and push
 run: |
 docker buildx build \
 --platform linux/amd64 \
 --push \
 -t ghcr.io/hanzoai/<service>:latest \
 -t ghcr.io/hanzoai/<service>:${{ github.sha }} \
 .

Secret management

# secrets.yaml -- KMSSecret CRD
apiVersion: kms.hanzo.ai/v1
kind: KMSSecret
metadata:
 name: my-service-secrets
 namespace: hanzo
spec:
 project: my-service
 environment: production
 syncInterval: 5m
 secretRef:
 name: my-service-secrets
 secrets:
 - DATABASE_URL
 - API_KEY
 - JWT_SECRET

Universe directory structure

universe/infra/k8s/
 namespace.yaml # hanzo namespace
 cluster-issuer.yaml # Let's Encrypt issuer
 platform-ingress.yaml # Platform-level ingress
 rbac/ # RBAC for service accounts
 base/ # Shared base manifests
 # --- Services ---
 app/ # hanzo.app
 billing/ # billing.hanzo.ai
 bot/ # bot gateway
 chat/ # chat.hanzo.ai
 cloud/ # cloud.hanzo.ai (Casibase)
 commerce/ # commerce API
 console/ # console.hanzo.ai (Langfuse)
 dns/ # CoreDNS
 flow/ # workflow builder
 gateway/ # API gateway (KrakenD)
 iam/ # hanzo.id (Hanzo IAM)
 kms/ # kms.hanzo.ai (Hanzo KMS)
 monitoring/ # Prometheus/Grafana
 o11y/ # Hanzo o11y observability
 paas/ # platform.hanzo.ai
 registry/ # container registry
 search/ # search service
 sql/ # PostgreSQL + ZAP sidecar
 s3/ # Hanzo S3 object storage
 team/ # hanzo.team
 vector/ # vector DB
 zen/ # Zen model serving
 zt/ # zero-trust (OpenZiti)
 # ... 40+ service directories

Cluster topology

hanzo-k8s (production)

lux-k8s (Lux blockchain)

Gotchas

  1. GHA billing is blocked for hanzoai org. All CI/CD must use self-hosted runners.
  2. Runner pool waste: 7x32GB nodes at 17-26% utilization ($1,764/mo waste). Plan to scale to 2.
  3. DO registry limit: 5/5 repo limit on registry.digitalocean.com/hanzo. Use GHCR exclusively.
  4. Local TLS errors: Pushing to GHCR from local can fail with TLS errors. Use the buildx K8s driver.
  5. DO App Platform is DECOMMISSIONED: All services are K8s-native as of Feb 2026. No managed databases.
  6. In-cluster PostgreSQL: postgres.hanzo.svc -- no managed DB services.

Related Skills

Last Updated: 2026-03-23 Category: Hanzo Ecosystem Related: deployment, ci-cd, kubernetes, universe, ghcr Prerequisites: kubectl, K8s cluster access, GHCR credentials