HIP-36: CI/CD Build System Standard. Status Draft. Hanzo's own standard — read this before implementing against it.
This proposal defines the CI/CD build system standard for the Hanzo ecosystem. All 260+ repositories under the hanzoai GitHub organization MUST follow this specification for automated testing, building, releasing, and deploying software artifacts.
Hanzo Build provides standardized GitHub Actions workflows, reusable composite actions, and deployment patterns that enforce consistent quality gates across every project -- from the IAM identity provider to the LLM Gateway, from Rust ML frameworks to React frontends.
Repository: github.com/hanzoai/build Secret Management: kms.hanzo.ai (Hanzo KMS, HIP-0033) Primary Registry: ghcr.io/hanzoai
Managing CI/CD for 260+ repositories creates compounding problems:
npm test, Team B uses pnpm test, Team C uses yarn test. Multiply this by every build step and you get 260 slightly different pipelines that nobody fully understands.A single source of truth -- github.com/hanzoai/build -- that provides:
This section explains the why behind each architectural decision. CI/CD is foundational infrastructure -- the wrong choice here multiplies across every repository and every deploy.
260 repos x 1 manual update = 260 manual secret rotationsWith Hanzo KMS (at kms.hanzo.ai, powered by luxfi/kms MPC):
1 KMS update = all 260 repos pick up the new token on next buildThe secret injection flow works as follows:
GitHub Actions workflow starts
|
v
POST kms.hanzo.ai/api/v1/auth/universal-auth/login
- Sends KMS_CLIENT_ID + KMS_CLIENT_SECRET (these two ARE GitHub Secrets)
- Receives short-lived ACCESS_TOKEN
|
v
GET kms.hanzo.ai/api/v3/secrets/raw/{SECRET_NAME}
- Sends ACCESS_TOKEN as Bearer token
- Fetches DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, DIGITALOCEAN_ACCESS_TOKEN, etc.
|
v
Secrets injected as step outputs -> consumed by subsequent stepsHanzo infrastructure runs on two architectures:
| Environment | Architecture | Examples | |-------------|-------------|----------| | Production K8s | AMD64 | DigitalOcean droplets, Kubernetes cluster | | Developer machines | ARM64 | Apple Silicon MacBooks (M1/M2/M3/M4) | | CI runners | AMD64 | GitHub-hosted ubuntu-latest |
Without multi-arch images, developers running docker pull ghcr.io/hanzoai/iam:latest on Apple Silicon get an AMD64 image that runs under Rosetta emulation -- 2-5x slower and with subtle behavior differences.
Multi-arch manifests solve this. A single image tag (ghcr.io/hanzoai/iam:latest) contains both architectures. Docker automatically selects the native one.
The build uses docker/setup-qemu-action for cross-compilation and docker/setup-buildx-action for multi-platform builds:
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v5
with:
platforms: linux/amd64,linux/arm64Trade-off acknowledged: Multi-arch builds take 2-3x longer than single-arch builds because each platform compiles separately. We accept this because builds are not in the critical path for developer iteration (developers build locally) and the production correctness guarantee is worth the extra CI minutes.
# GHCR: must succeed (the build fails if this fails)
- name: Push to GHCR
uses: docker/build-push-action@v5
with:
push: true
tags: ghcr.io/hanzoai/${{ env.IMAGE }}:latest
# Docker Hub: best-effort (the build succeeds even if this fails)
- name: Push to Docker Hub
continue-on-error: true
uses: docker/build-push-action@v5
with:
push: true
tags: hanzoai/${{ env.IMAGE }}:latestRationale:
| Factor | GHCR | Docker Hub | |--------|------|------------| | Pull rate limits | None for authenticated GitHub users | 100/6hr anonymous, 200/6hr authenticated | | Cost for private repos | Free (included with GitHub plan) | $5-$9/month per private repo | | Auth for K8s pulls | GITHUB_TOKEN (already available) | Separate imagePullSecret required | | Public discoverability | Lower (GitHub Packages UI) | Higher (hub.docker.com search) | | Uptime/reliability | GitHub SLA | Has had outages affecting pulls |
GHCR is the source of truth. Docker Hub is a convenience mirror for public consumers. If Docker Hub is down or rate-limited, our builds and deployments are unaffected.
Our current deployment model is imperative:
CI builds image -> CI runs kubectl set image -> K8s rolls out new podsArgoCD and Flux provide declarative GitOps: a Git repository defines the desired state, and a controller in the cluster continuously reconciles toward it. This is superior in theory but premature for our current scale:
We will adopt ArgoCD when we exceed 5 clusters or need multi-environment promotion (dev -> staging -> production) with drift detection. This HIP does not preclude that migration; the standardized workflow templates can be updated to push manifests to a GitOps repo instead of running kubectl directly.
Every Hanzo repository MUST include workflows from the following template set. Not all templates are required for every repo -- a pure library needs only build.yml and test.yml, while a deployable service needs all four.
build.yml -- Build and TestTriggered on every push to main/master and on every pull request.
name: Build
on:
push:
branches: [main, master]
pull_request:
jobs:
test:
name: Tests
runs-on: ubuntu-latest
# Language-specific service containers (SQL, KV, etc.)
services:
postgres:
image: ghcr.io/hanzoai/sql:latest
env:
POSTGRES_USER: hanzo
POSTGRES_PASSWORD: hanzo123
POSTGRES_DB: test_db
ports: ["5432:5432"]
options: >-
--health-cmd="pg_isready -U hanzo"
--health-interval=10s
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@v4
# Go projects
- uses: actions/setup-go@v4
with:
go-version: '1.23'
cache-dependency-path: ./go.mod
- run: go test -v -race ./...
# Node.js projects
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'pnpm' # or 'yarn' for legacy repos
- run: pnpm install && pnpm test
# Python projects
- uses: astral-sh/setup-uv@v4
- run: uv sync --all-extras && uv run pytest -v
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Go: gofumpt via golangci-lint
# Node: eslint via pnpm lint
# Python: ruff via uv run ruff check .
# Rust: clippy via cargo clippy
build:
name: Build
runs-on: ubuntu-latest
needs: [test]
steps:
- uses: actions/checkout@v4
# Language-specific build with race detection (Go),
# production bundle (Node), or release build (Rust)release.yml -- Semantic Versioning and ReleaseTriggered only on push to main/master, after all build jobs succeed.
tag-release:
name: Create Tag
runs-on: ubuntu-latest
if: github.repository == 'hanzoai/${{ env.REPO }}' && github.event_name == 'push'
needs: [test, lint, build]
outputs:
new-release-published: ${{ steps.semantic.outputs.new_release_published }}
new-release-version: ${{ steps.semantic.outputs.new_release_version }}
steps:
- uses: actions/checkout@v4
- uses: cycjimmy/semantic-release-action@v4
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}Semantic release reads conventional commit messages to determine version bumps:
fix: -> patch (1.0.0 -> 1.0.1)feat: -> minor (1.0.0 -> 1.1.0)BREAKING CHANGE: -> major (1.0.0 -> 2.0.0)Note: Go packages MUST NOT be bumped above v1.x.x per Hanzo convention. The .releaserc in Go repos caps at minor/patch.
docker.yml -- Docker Build and Registry PushTriggered after a successful release tag. This is the core of the containerized build pipeline.
docker-release:
name: Docker Release
runs-on: ubuntu-latest
permissions:
contents: write
packages: write
needs: [tag-release]
if: needs.tag-release.outputs.new-release-published == 'true'
steps:
- uses: actions/checkout@v4
# -- Secret injection from KMS --
- name: Fetch CI secrets from Hanzo KMS
id: kms
env:
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
run: |
set -euo pipefail
KMS_URL="${KMS_URL:-https://kms.hanzo.ai}"
ACCESS_TOKEN="$(
curl -fsS -X POST "${KMS_URL}/api/v1/auth/universal-auth/login" \
-H "Content-Type: application/json" \
-d "$(jq -nc --arg cid "$KMS_CLIENT_ID" \
--arg cs "$KMS_CLIENT_SECRET" \
'{clientId: $cid, clientSecret: $cs}')" \
| jq -r '.accessToken'
)"
fetch_secret() {
curl -fsS \
"${KMS_URL}/api/v3/secrets/raw/${1}?\
workspaceSlug=gitops&environment=prod&\
secretPath=/ci&viewSecretValue=true&\
include_imports=true" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
| jq -r '.secret.secretValue'
}
for name in DOCKERHUB_USERNAME DOCKERHUB_TOKEN; do
val="$(fetch_secret "$name")"
echo "::add-mask::${val}"
echo "${name}=${val}" >> "$GITHUB_OUTPUT"
done
# -- Multi-arch build setup --
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
# -- Registry authentication --
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
id: dockerhub-login
continue-on-error: true
uses: docker/login-action@v3
with:
registry: docker.io
username: ${{ steps.kms.outputs.DOCKERHUB_USERNAME }}
password: ${{ steps.kms.outputs.DOCKERHUB_TOKEN }}
# -- Image metadata --
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/hanzoai/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}},value=${{ needs.tag-release.outputs.new-release-version }}
type=raw,value=latest,enable={{is_default_branch}}
type=sha,prefix={{branch}}-
# -- Build and push (GHCR: required) --
- name: Build and push to GHCR
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# -- Mirror to Docker Hub (best-effort) --
- name: Push to Docker Hub
if: steps.dockerhub-login.outcome == 'success'
continue-on-error: true
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: hanzoai/${{ env.IMAGE_NAME }}:latest
cache-from: type=ghadeploy.yml -- DeploymentTriggered after a successful Docker build on the default branch. Supports two deployment targets.
deploy:
name: Deploy
needs: [docker-release]
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
# -- Fetch deploy secrets from KMS --
- name: Fetch deploy secrets
id: kms
env:
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
run: |
# Same KMS login pattern as docker.yml
# Fetches: DIGITALOCEAN_ACCESS_TOKEN
# -- Kubernetes deployment (preferred) --
- name: Install doctl
uses: digitalocean/action-doctl@v2
with:
token: ${{ steps.kms.outputs.do_token }}
- name: Deploy to K8s
run: |
doctl kubernetes cluster kubeconfig save the cluster
kubectl set image deployment/$SERVICE \
$SERVICE=ghcr.io/hanzoai/$SERVICE:latest
kubectl rollout status deployment/$SERVICE \
--timeout=300s
- name: Health check
run: |
kubectl wait --for=condition=available \
deployment/$SERVICE --timeout=120sBuilds MUST use GitHub Actions cache to avoid redundant downloads.
| Language | Cache Mechanism | Configuration | |----------|----------------|---------------| | Go | actions/setup-go built-in | cache-dependency-path: ./go.mod | | Node.js | actions/setup-node built-in | cache: 'pnpm' (or 'yarn') | | Python | astral-sh/setup-uv built-in | Automatic uv cache | | Rust | actions/cache manual | ~/.cargo/registry, target/ | | Docker | GitHub Actions cache backend | cache-from: type=gha, cache-to: type=gha,mode=max |
Docker layer caching deserves special attention. The type=gha cache backend stores Docker layers in the GitHub Actions cache (10 GB per repo). This means a build that changes only the application layer reuses the base image, dependency install, and compilation layers from cache. For a Go project like IAM, this reduces build time from ~8 minutes to ~2 minutes.
All Docker images MUST use the following tag scheme:
ghcr.io/hanzoai/{service}:{tag}
Tags:
latest - Latest build from default branch
{semver} - Semantic version (e.g., 1.5.2)
{branch}-{sha} - Branch name + short commit SHA (e.g., main-a1b2c3d)Examples:
ghcr.io/hanzoai/iam:latest
ghcr.io/hanzoai/iam:1.5.2
ghcr.io/hanzoai/iam:main-a1b2c3d
ghcr.io/hanzoai/llm-gateway:latest
ghcr.io/hanzoai/llm-gateway:2.1.0CI workflows that require databases or caches MUST use Hanzo-maintained service images:
| Service | Image | Notes | |---------|-------|-------| | PostgreSQL | ghcr.io/hanzoai/sql:latest | PostgreSQL with extensions | | Redis | ghcr.io/hanzoai/kv:latest | Redis-compatible KV store | | DocumentDB | mongo:7 | Upstream (no Hanzo fork needed) | | MinIO | minio/minio:latest | S3-compatible object storage |
All repositories MUST configure branch protection on main/master:
build.yml must passAll commit messages MUST follow the Conventional Commits specification:
<type>[optional scope]: <description>
Types:
feat: New feature (triggers minor version bump)
fix: Bug fix (triggers patch version bump)
docs: Documentation only
style: Formatting, no code change
refactor: Code change that neither fixes nor adds
perf: Performance improvement
test: Adding or fixing tests
ci: CI/CD changes
chore: Build process or auxiliary tool changesSemantic-release reads these to determine the next version number automatically. No manual version bumping.
Every Hanzo repository with CI/CD follows this structure:
.github/
workflows/
build.yml # Test + lint + build (PR and push)
docker-deploy.yml # Docker build + registry push + deploy (push only)
sync.yml # Optional: sync fork with upstream
CODEOWNERS # Required reviewers per path
Dockerfile # Multi-stage, multi-target
compose.yml # Local development (NOT docker-compose.yml)
Makefile # Developer-facing commands
version.txt # Current version (read by CI)
.releaserc # Semantic-release configurationSecrets in Hanzo KMS are organized by workspace and path:
Workspace: gitops
Environment: prod
Path: /ci
Secrets:
DOCKERHUB_USERNAME # Docker Hub service account
DOCKERHUB_TOKEN # Docker Hub access token
DIGITALOCEAN_ACCESS_TOKEN # DO API token for K8s deploy
SLACK_WEBHOOK # Build notification webhook
DEPLOY_SSH_KEY # SSH key for Docker Compose targetsEach repository's KMS Universal Auth identity is scoped to read only from /ci. A repository that also needs application-level secrets (e.g., database URLs for E2E tests) gets additional path access as needed.
Hanzo services deploy to one of two target types:
The preferred deployment target. The CI workflow:
doctl using a KMS-sourced tokenkubectl for the the cluster clusterkubectl set image deployment/SERVICEkubectl rollout statuskubectl wait --for=condition=availableServices on the cluster (24.199.76.156):
IAM, KMS, Platform, Cloud, Console, Gateway,
Commerce, hanzo-app, web3, registry, bootnode-api
SQL, KV, DocumentDB, MinIOFor services not yet migrated to K8s. The CI workflow:
docker pull ghcr.io/hanzoai/SERVICE:latestdocker compose up -d SERVICEThis target is being phased out in favor of K8s. New services MUST deploy to K8s.
The github.com/hanzoai/build repository provides reusable workflows that individual repos call:
# In any Hanzo repo: .github/workflows/docker-deploy.yml
name: Docker Build and Deploy
on:
push:
branches: [main, master]
jobs:
build-and-deploy:
uses: hanzoai/build/.github/workflows/docker-k8s.yml@main
with:
image-name: my-service
k8s-namespace: hanzo
k8s-deployment: my-service
secrets:
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}This reduces per-repo workflow files from 150+ lines to ~15 lines while maintaining full customizability via with: inputs.
| Threat | Mitigation | |--------|-----------| | Leaked secrets in git history | All secrets fetched from KMS at runtime; never written to files or env that persists | | Compromised GitHub Actions runner | KMS access tokens expire in 15 minutes; runner has no persistent credentials | | Supply chain attack via Actions | Pin action versions to SHA, not mutable tags (actions/checkout@abc123 not @v4) | | Malicious PR running CI | PRs from forks do not have access to secrets; workflows use pull_request_target carefully | | Container image tampering | GHCR images are content-addressed by digest; Kubernetes can pin to digest | | Privilege escalation in deploy | KMS identities follow least privilege; CI identity cannot read production database credentials |
.env files. All secrets come from KMS.KMS_CLIENT_ID and KMS_CLIENT_SECRET. Everything else is fetched dynamically.::add-mask:: before use.All Docker images SHOULD be scanned before push using Trivy:
- name: Scan image
uses: aquasecurity/trivy-action@master
with:
image-ref: ghcr.io/hanzoai/${{ env.IMAGE }}:${{ env.VERSION }}
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload scan results
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'trivy-results.sarif'Critical vulnerabilities SHOULD block the release. High vulnerabilities SHOULD be reviewed within 7 days.
provenance: false currently (buildx provenance attestations cause issues with some registries). We will enable SLSA Level 2 provenance when registry support stabilizes.CI workflows emit metrics to the Hanzo observability stack:
| Metric | Description | Alert Threshold | |--------|-------------|-----------------| | ci.build.duration_seconds | Total workflow duration | > 15 minutes | | ci.test.duration_seconds | Test job duration | > 10 minutes | | ci.docker.build_seconds | Docker build + push duration | > 8 minutes | | ci.deploy.duration_seconds | Deploy + health check duration | > 5 minutes | | ci.build.failure_rate | Rolling 24h failure rate | > 20% | | ci.cache.hit_rate | Docker/dependency cache hit rate | < 50% |
Build results are posted to Slack via webhook:
[IAM] Deploy succeeded
Version: 1.5.2
Commit: a1b2c3d "feat: apply application.DefaultGroup for OAuth signups"
Duration: 3m 42s
Cluster: the clusterFailed builds include the failure step and a link to the workflow run.
gh repo create hanzoai/my-new-service \
--template hanzoai/template \
--publichanzoai/deploy) as deployment source of truthCopyright and related rights waived via CC0.