hanzo-k8s

Hanzo runs all production services on DOKS (DigitalOcean Kubernetes Service) clusters.

Hanzo K8s - Kubernetes Infrastructure

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-deploy.md, hanzo/hanzo-ingress.md, hanzo/hanzo-kms.md

Overview

Hanzo runs all production services on DOKS (DigitalOcean Kubernetes Service) clusters. Two primary clusters: hanzo-k8s for all Hanzo services and lux-k8s for Lux blockchain. All manifests live in the universe repo. Kustomize for manifest management, KMSSecret CRDs for secret sync, Hanzo Ingress (Traefik) for routing.

When to use

Managing or debugging K8s workloads on hanzo-k8s or lux-k8s
Adding new services to the cluster
Troubleshooting pod failures, networking, or ingress issues
Understanding the cluster topology

Hard requirements

Kustomize, not Helm: kubectl kustomize . | kubectl apply -f -
KMS for all secrets: KMSSecret CRDs, never hardcode in manifests
In-cluster PostgreSQL: postgres.hanzo.svc, no managed DB services
Hanzo IngressClass: ingressClassName: hanzo on all Ingress resources
GHCR images: ghcr.io/hanzoai/<service>:<tag>, always --platform linux/amd64
No DO App Platform: Fully decommissioned as of Feb 2026

Quick reference

hanzo-k8s (production)

| Item | Value | |------|-------| | Provider | DigitalOcean Kubernetes (DOKS) | | Location | SFO3 | | Context | do-sfo3-hanzo-k8s | | Nodes | 22 (15 workers + 7 runners) | | LB IP | 165.232.146.176 | | Namespace | hanzo (primary) | | IngressClass | hanzo (Traefik) | | PostgreSQL | postgres.hanzo.svc:5432 | | Redis/Valkey | redis.hanzo.svc:6379 | | MongoDB | mongodb.hanzo.svc:27017 | | Hanzo S3 | s3.hanzo.svc:9000 |

lux-k8s (blockchain)

| Item | Value | |------|-------| | Provider | DigitalOcean Kubernetes (DOKS) | | Location | SFO3 | | Context | do-sfo3-lux-k8s | | LB IP | 24.144.69.101 | | Services | 15 validators, gateway, markets, lux-cloud-web | | PostgreSQL | postgres.hanzo.svc:5432 |

Cluster services map

hanzo-k8s domains and services

| Domain | Backend Service | Port | |--------|-----------------|------| | hanzo.ai | hanzo-app | 3000 | | api.hanzo.ai | hanzo-gateway (KrakenD) | 8000 | | llm.hanzo.ai | hanzo-gateway | 8000 | | hanzo.id, lux.id, zoo.id, pars.id | IAM (Hanzo IAM) | 8000 | | kms.hanzo.ai | KMS (Hanzo KMS) | 8080 | | platform.hanzo.ai | Platform (Dokploy) | 3000 | | console.hanzo.ai | Console (Langfuse) | 3000 | | cloud.hanzo.ai | Cloud (Casibase) | 8000 | | chat.hanzo.ai | Chat (LibreChat) | 3080 | | hanzo.team | Team (Huly fork) | 8087 | | billing.hanzo.ai | Billing portal | 80 | | o11y.hanzo.ai | Hanzo o11y | 3301 |

lux-k8s domains and services

| Domain | Backend Service | |--------|-----------------| | api.lux.network | KrakenD Gateway | | cloud.lux.network | Lux Cloud | | markets.lux.network | Markets |

Database topology

hanzo-k8s databases

postgres.hanzo.svc:5432
 |- iam (Hanzo IAM)
 |- cloud (Casibase)
 |- console (Langfuse)
 |- hanzo_cloud (Cloud API)
 |- kms (Hanzo KMS)
 |- platform (Dokploy)

lux-k8s databases

postgres.hanzo.svc:5432
 |- cloud
 |- commerce
 |- console
 |- gateway
 |- hanzo
 |- kms

Common operations

Check service status

# List all pods
kubectl --context do-sfo3-hanzo-k8s -n hanzo get pods

# Check specific service
kubectl --context do-sfo3-hanzo-k8s -n hanzo get pods -l app=<service>

# View logs
kubectl --context do-sfo3-hanzo-k8s -n hanzo logs -l app=<service> --tail=100 -f

# Describe pod (for events/errors)
kubectl --context do-sfo3-hanzo-k8s -n hanzo describe pod <pod-name>

Deploy from universe

cd ~/work/hanzo/universe/infra/k8s/<service>
kubectl --context do-sfo3-hanzo-k8s kustomize . | kubectl apply -f -

Port forward for debugging

# Access a service locally
kubectl --context do-sfo3-hanzo-k8s -n hanzo port-forward svc/<service> 8080:8080

# Access PostgreSQL
kubectl --context do-sfo3-hanzo-k8s -n hanzo port-forward svc/postgres 5432:5432

Scale a deployment

kubectl --context do-sfo3-hanzo-k8s -n hanzo scale deployment/<service> --replicas=3

Manifest patterns

Standard deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-service
 namespace: hanzo
spec:
 replicas: 2
 selector:
 matchLabels:
 app: my-service
 template:
 metadata:
 labels:
 app: my-service
 spec:
 containers:
 - name: my-service
 image: ghcr.io/hanzoai/my-service:latest
 ports:
 - containerPort: 8080
 envFrom:
 - secretRef:
 name: my-service-secrets # Synced from KMS
 resources:
 requests:
 cpu: 100m
 memory: 128Mi
 limits:
 cpu: 500m
 memory: 512Mi
 livenessProbe:
 httpGet:
 path: /health
 port: 8080
 readinessProbe:
 httpGet:
 path: /ready
 port: 8080

Standard ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: my-service
 namespace: hanzo
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt
spec:
 ingressClassName: hanzo
 tls:
 - hosts:
 - my-service.hanzo.ai
 secretName: my-service-tls
 rules:
 - host: my-service.hanzo.ai
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: my-service
 port:
 number: 8080

KMSSecret for secret sync

apiVersion: kms.hanzo.ai/v1
kind: KMSSecret
metadata:
 name: my-service-secrets
 namespace: hanzo
spec:
 project: my-service
 environment: production
 syncInterval: 5m
 secretRef:
 name: my-service-secrets
 secrets:
 - DATABASE_URL
 - API_KEY

Node pools

| Pool | Count | Size | Purpose | |------|-------|------|---------| | worker-pool | 15 | 8vCPU/32GB | Application workloads | | runner-pool | 7 | 8vCPU/32GB | CI/CD self-hosted runners |

Runner pool is oversized (17-26% utilization). Plan to scale to 2 nodes.

Troubleshooting

| Issue | Cause | Solution | |-------|-------|----------| | Pod CrashLoopBackOff | Missing secrets | Check KMSSecret CRD is synced: kubectl get kmssecrets | | Ingress 404 | Wrong IngressClass | Ensure ingressClassName: hanzo | | DNS not resolving | Missing Cloudflare record | Add A record pointing to 165.232.146.176 | | Can't pull image | GHCR auth expired | Check imagePullSecrets on service account | | DB connection refused | Wrong host | Use postgres.hanzo.svc (in-cluster) not external URL | | Network policy blocking | Missing ingress rule | Add NetworkPolicy allowing ingress from Ingress controller |

Related Skills

hanzo/hanzo-deploy.md -- Deployment workflow
hanzo/hanzo-ingress.md -- Ingress controller
hanzo/hanzo-kms.md -- Secret management
hanzo/hanzo-o11y.md -- Observability

Last Updated: 2026-03-23 Category: Hanzo Ecosystem Related: kubernetes, doks, infrastructure, cluster, k8s Prerequisites: kubectl, K8s cluster access, Cloudflare DNS