HIP-68: Ingress Standard. Status Active. Hanzo's own standard — read this before implementing against it.
This proposal defines the Ingress standard for the Hanzo ecosystem. Hanzo Ingress is the Layer 7 reverse proxy and load balancer that sits at the edge of every Hanzo Kubernetes cluster. It handles host-based routing, TLS termination (via Cloudflare proxy mode), load balancing, health checks, and middleware chains for all inbound traffic.
Hanzo Ingress is a maintained fork of Traefik v3.6. It watches Kubernetes Ingress resources and IngressRoute CRDs natively -- no config generation, no external sync loops, no sidecar injection. A new Service or Ingress resource appears in the cluster; Ingress picks it up within seconds and starts routing traffic.
This HIP is explicitly distinct from HIP-44 (API Gateway), which is the application-level gateway handling authentication, rate limiting, and request transformation for api.hanzo.ai. Ingress operates one layer below: it routes api.hanzo.ai traffic to the API Gateway, hanzo.ai traffic to the marketing site, platform.hanzo.ai traffic to Dokploy, and so on. The API Gateway is one of many backends behind Ingress.
Repository: github.com/hanzoai/ingress Entrypoints: web (8080), websecure (8443) Image: ghcr.io/hanzoai/ingress:latest Providers: KubernetesIngress, KubernetesCRD
Hanzo operates two Kubernetes clusters (the cluster and lux-k8s) serving 30+ domains across multiple services. Each domain needs:
api.hanzo.ai goes to the API Gateway, platform.hanzo.ai goes to Dokploy, kms.hanzo.ai goes to KMS. Each domain is a separate routing decision at the edge.The NGINX Ingress Controller is the Kubernetes default. It has three problems that make it unsuitable for Hanzo:
nginx.conf file, validates it, and performs a graceful reload. This takes 5-30 seconds. During that window, new services are unreachable and deleted services still receive traffic.nginx.ingress.kubernetes.io/rewrite-target, nginx.ingress.kubernetes.io/ssl-redirect) for advanced configuration. Annotations are untyped strings with no schema validation. A typo silently does nothing. Traefik's IngressRoute CRDs are typed, validated at apply time, and self-documenting.| Factor | nginx-ingress | Hanzo Ingress (Traefik) | |--------|--------------|------------------------| | Config update | 5-30s (generate + reload) | < 1s (watch + apply) | | Advanced routing | Annotations (untyped) | IngressRoute CRDs (typed) | | PaaS integration | Translation layer needed | Native (Dokploy speaks Traefik) | | Middleware | Annotation hacks | First-class Middleware CRD | | TCP/UDP routing | Limited | Full TCP/UDP routing support | | Dashboard | None (third-party) | Built-in (disabled by default) |
The tradeoff: NGINX has marginally higher raw throughput for static content. For a dynamic Kubernetes environment where routing changes frequently and services come and go, Traefik's native Kubernetes integration is decisive.
Internet
|
+----------+----------+
| DigitalOcean LB |
| (TCP passthrough) |
+----------+----------+
|
+----------+----------+
| Hanzo Ingress |
| (Traefik v3.6) |
| :8080 / :8443 |
+----------+----------+
|
+--------------------+--------------------+
| | | | |
+----+----+ +--+---+ +---+--+ +----+----+ +---+---+
| API | | PaaS | | IAM | | KMS | | Chat |
| Gateway | |(Dokp)| | | | | | |
| :8080 | |:3000 | |:8000 | | :8080 | |:3081 |
+---------+ +------+ +------+ +---------+ +-------+Ingress is the first process to handle an inbound connection after the cloud load balancer. It terminates TLS (or relies on Cloudflare to do so), inspects the Host header, matches the request against its routing table, applies any configured middleware, and forwards the request to the correct backend service.
| Name | Port | Protocol | Purpose | |------|------|----------|---------| | web | 8080 | HTTP | Plaintext HTTP; redirects to websecure in production | | websecure | 8443 | HTTPS | TLS-terminated HTTPS traffic | | traefik | 9090 | HTTP | Dashboard and API (disabled by default) | | metrics | 9100 | HTTP | Prometheus metrics endpoint |
In production with Cloudflare proxy mode, the DigitalOcean LoadBalancer forwards TCP traffic on ports 80/443 to Ingress ports 8080/8443. Cloudflare terminates the external TLS connection and re-encrypts to the origin using Cloudflare's edge certificates.
Hanzo Ingress watches two Kubernetes providers simultaneously:
1. KubernetesIngress Provider
Watches standard networking.k8s.io/v1 Ingress resources. Any service that creates an Ingress resource with ingressClassName: hanzo-ingress is automatically routed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-gateway
namespace: hanzo
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
ingressClassName: hanzo-ingress
rules:
- host: api.hanzo.ai
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 80802. KubernetesCRD Provider
Watches Traefik-specific CRDs for advanced routing. IngressRoute resources support weighted routing, header-based matching, middleware chains, and TCP/UDP routing that standard Ingress resources cannot express.
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: platform-route
namespace: hanzo
spec:
entryPoints:
- websecure
routes:
- match: Host(`platform.hanzo.ai`)
kind: Rule
services:
- name: dokploy
port: 3000
middlewares:
- name: rate-limit
- name: security-headers| Host | Backend Service | Port | Cluster | |------|----------------|------|---------| | api.hanzo.ai | API Gateway (HIP-44) | 8080 | the cluster | | llm.hanzo.ai | LLM Gateway (HIP-4) | 4000 | the cluster | | hanzo.id | Hanzo IAM | 8000 | the cluster | | lux.id | Hanzo IAM | 8000 | the cluster | | zoo.id | Hanzo IAM | 8000 | the cluster | | pars.id | Hanzo IAM | 8000 | the cluster | | kms.hanzo.ai | KMS (Hanzo KMS) | 8080 | the cluster | | platform.hanzo.ai | Platform (Dokploy) | 3000 | the cluster | | console.hanzo.ai | Console | 3001 | the cluster | | cloud.hanzo.ai | Cloud | 3002 | the cluster | | hanzo.app | Main App | 3000 | the cluster | | api.lux.network | Lux Gateway (KrakenD) | 8080 | lux-k8s | | cloud.lux.network | Lux Cloud | 3000 | lux-k8s | | markets.lux.network | Markets | 3000 | lux-k8s |
Primary mode: Cloudflare Proxy (Flexible SSL)
All Hanzo domains use Cloudflare as DNS proxy. Cloudflare terminates TLS at the edge and forwards traffic to the origin. In this mode, Ingress receives plaintext HTTP from Cloudflare on port 8080. No certificate management is required on the origin.
Client --[TLS]--> Cloudflare --[HTTP]--> DO LB --[HTTP]--> Ingress :8080Secondary mode: Full (Strict) SSL with CertManager
For domains where end-to-end encryption is required (e.g., kms.hanzo.ai handling secrets), CertManager provisions Let's Encrypt certificates and stores them as Kubernetes Secrets. Ingress reads these secrets and terminates TLS at port 8443.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kms-tls
namespace: hanzo
spec:
secretName: kms-tls-secret
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- kms.hanzo.aiapiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
name: default
namespace: hanzo
spec:
minVersion: VersionTLS12
preferServerCipherSuites: true
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_AES_256_GCM_SHA384
- TLS_AES_128_GCM_SHA256
- TLS_CHACHA20_POLY1305_SHA256Middleware CRDs are applied per-route via IngressRoute references. Each middleware is a standalone Kubernetes resource that can be shared across routes.
Rate Limiting
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: rate-limit
namespace: hanzo
spec:
rateLimit:
average: 100
burst: 200
period: 1m
sourceCriterion:
ipStrategy:
depth: 1Security Headers
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: security-headers
namespace: hanzo
spec:
headers:
frameDeny: true
sslRedirect: true
browserXssFilter: true
contentTypeNosniff: true
stsIncludeSubdomains: true
stsPreload: true
stsSeconds: 31536000
customResponseHeaders:
X-Powered-By: ""
Server: ""Retry
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: retry
namespace: hanzo
spec:
retry:
attempts: 3
initialInterval: 100msCircuit Breaker
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: circuit-breaker
namespace: hanzo
spec:
circuitBreaker:
expression: "LatencyAtQuantileMS(50.0) > 1000 || NetworkErrorRatio() > 0.30"IP Whitelist (Admin Services)
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: admin-whitelist
namespace: hanzo
spec:
ipAllowList:
sourceRange:
- "127.0.0.1/8"
- "127.0.0.1/12"Ingress performs active health checks on backend services:
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: default
namespace: hanzo
spec:
serverName: ""
insecureSkipVerify: false
maxIdleConnsPerHost: 200
forwardingTimeouts:
dialTimeout: 5s
responseHeaderTimeout: 30s
idleConnTimeout: 90sKubernetes readiness probes are the primary health signal. When a pod's readiness probe fails, Kubernetes removes it from the Endpoints object. Ingress watches Endpoints and stops routing to that pod within one watch cycle (typically < 1 second). No additional health check configuration is needed for most services.
Default algorithm is weighted round-robin. Sticky sessions are available via cookie-based affinity for stateful services:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: stateful-service
namespace: hanzo
spec:
routes:
- match: Host(`app.hanzo.ai`)
services:
- name: stateful-backend
port: 8080
sticky:
cookie:
name: hanzo_affinity
secure: true
httpOnly: trueAccess Logs
Structured JSON on stdout, one line per request:
{
"time": "2026-02-24T12:00:00Z",
"level": "info",
"msg": "",
"ClientAddr": "127.0.0.1:54321",
"ClientHost": "127.0.0.1",
"Duration": 12345678,
"DownstreamStatus": 200,
"RequestHost": "api.hanzo.ai",
"RequestMethod": "POST",
"RequestPath": "/v1/chat/completions",
"RequestProtocol": "HTTP/1.1",
"RouterName": "api-gateway@kubernetes",
"ServiceName": "api-gateway-hanzo@kubernetes",
"entryPointName": "websecure"
}Prometheus Metrics
Exported on port 9100 with namespace hanzo_ingress:
| Metric | Type | Description | |--------|------|-------------| | hanzo_ingress_entrypoint_requests_total | Counter | Total requests by entrypoint, method, protocol, code | | hanzo_ingress_entrypoint_request_duration_seconds | Histogram | Request duration by entrypoint | | hanzo_ingress_entrypoint_open_connections | Gauge | Open connections by entrypoint, method, protocol | | hanzo_ingress_service_requests_total | Counter | Total requests by service, method, protocol, code | | hanzo_ingress_service_request_duration_seconds | Histogram | Request duration by service | | hanzo_ingress_service_open_connections | Gauge | Open connections by service | | hanzo_ingress_service_retries_total | Counter | Retry count by service | | hanzo_ingress_service_server_up | Gauge | Backend server health (1=up, 0=down) | | hanzo_ingress_tls_certs_not_after | Gauge | TLS certificate expiry timestamp | | hanzo_ingress_config_reloads_total | Counter | Configuration reload count | | hanzo_ingress_config_last_reload_success | Gauge | Last reload success (1/0) |
Tracing
OpenTelemetry traces are emitted for every request, propagating traceparent and tracestate headers to backend services. Traces are exported to the cluster's OTLP collector for correlation with backend spans.
Ingress requires the following Kubernetes RBAC permissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: hanzo-ingress
rules:
- apiGroups: [""]
resources: ["services", "endpoints", "secrets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "ingressclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses/status"]
verbs: ["update"]
- apiGroups: ["traefik.io"]
resources: ["ingressroutes", "ingressroutetcps", "ingressrouteudps",
"middlewares", "middlewaretcps", "tlsoptions", "tlsstores",
"traefikservices", "serverstransports", "serverstransporttcps"]
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions", "networking.k8s.io"]
resources: ["ingresses", "ingressclasses"]
verbs: ["get", "list", "watch"]The ClusterRole is bound to a dedicated ServiceAccount (hanzo-ingress) via ClusterRoleBinding.
Two replicas with rolling updates (maxSurge: 1, maxUnavailable: 0). Resource limits: 200m-1000m CPU, 128Mi-512Mi memory. Pod anti-affinity ensures replicas run on different nodes for availability.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hanzo-ingress
namespace: hanzo
spec:
replicas: 2
selector:
matchLabels:
app: hanzo-ingress
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: hanzo-ingress
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9100"
spec:
serviceAccountName: hanzo-ingress
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values: ["hanzo-ingress"]
topologyKey: kubernetes.io/hostname
containers:
- name: ingress
image: ghcr.io/hanzoai/ingress:latest
args:
- --entrypoints.web.address=:8080
- --entrypoints.websecure.address=:8443
- --entrypoints.metrics.address=:9100
- --providers.kubernetesingress=true
- --providers.kubernetesingress.ingressclass=hanzo-ingress
- --providers.kubernetescrd=true
- --metrics.prometheus=true
- --metrics.prometheus.entrypoint=metrics
- --accesslog=true
- --accesslog.format=json
- --ping=true
- --api.dashboard=false
- --api.insecure=false
ports:
- name: web
containerPort: 8080
- name: websecure
containerPort: 8443
- name: metrics
containerPort: 9100
resources:
requests: { cpu: "200m", memory: "128Mi" }
limits: { cpu: "1000m", memory: "512Mi" }
readinessProbe:
httpGet:
path: /ping
port: 9100
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /ping
port: 9100
initialDelaySeconds: 10
periodSeconds: 15
---
apiVersion: v1
kind: Service
metadata:
name: hanzo-ingress
namespace: hanzo
annotations:
service.beta.kubernetes.io/do-loadbalancer-protocol: "tcp"
service.beta.kubernetes.io/do-loadbalancer-size-slug: "lb-small"
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
type: LoadBalancer
selector:
app: hanzo-ingress
ports:
- name: web
port: 80
targetPort: 8080
- name: websecure
port: 443
targetPort: 8443
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: hanzo-ingress
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: traefik.io/ingress-controller# compose.yml
services:
ingress:
image: ghcr.io/hanzoai/ingress:latest
command:
- --entrypoints.web.address=:8080
- --entrypoints.websecure.address=:8443
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --api.dashboard=true
- --api.insecure=true
- --accesslog=true
- --accesslog.format=json
ports:
- "80:8080"
- "443:8443"
- "9090:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:roIn development mode, the Docker provider watches container labels for routing configuration. The dashboard is enabled for debugging.
Hanzo Ingress registers itself as the default IngressClass (hanzo-ingress). Any Ingress resource without an explicit ingressClassName is handled by Hanzo Ingress. Services that need a different ingress controller (e.g., for testing) can specify an alternate IngressClass.
Each brand runs the identical topology with its own values:
| Concern | Per-brand value | |---------|-----------------| | Identity origin | iam.hanzo.ai / lux.id / zoo.id / id.bootno.de / pars.id | | API host | api.<brand-domain> | | Apps | brand-scoped <org>-<app> client IDs (HIP-0111) | | Secrets | KMS project per brand | | Container registry | ghcr.io/hanzoai/ (Hanzo), ghcr.io/luxfi/ (Lux), ghcr.io/zooai/* (Zoo) |
Nothing in the topology is brand-special-cased. A brand is a set of values plugged into the same shape.
Ingress is configured with ingress.kubernetes.io/ annotations ONLY. A traefik. annotation is silently ignored, which leaves a route believing it has a rate limit it does not have — the failure is invisible until load arrives.
| HIP | Relationship | |-----|-------------| | HIP-4 (LLM Gateway) | Ingress routes llm.hanzo.ai traffic to LLM Gateway. Separate processes at different layers. | | HIP-14 (Application Deployment) | Dokploy generates IngressRoute resources that Ingress watches and routes. | | HIP-26 (IAM) | Ingress routes hanzo.id, lux.id, zoo.id, pars.id to IAM. No auth at the ingress layer; auth is handled by the API Gateway or services themselves. | | HIP-27 (KMS) | Ingress routes kms.hanzo.ai to KMS. TLS certificates for strict SSL mode are stored as K8s Secrets provisioned by CertManager. | | HIP-31 (Observability) | Ingress exports Prometheus metrics consumed by Grafana dashboards. Access logs feed into the log aggregation pipeline. | | HIP-106 (Cloud) | Ingress routes cloud.hanzo.ai and cloud.lux.network to their respective cloud services. | | HIP-44 (API Gateway) | The API Gateway (KrakenD) is a backend behind Ingress. Ingress handles L7 host routing; the API Gateway handles application-level concerns (auth, rate limiting, circuit breaking). | | HIP-49 (DNS) | DNS records point domains to the DigitalOcean LoadBalancer IP. Cloudflare proxies these records for DDoS protection and edge TLS. |
Internet
--> Cloudflare (DDoS protection, edge TLS, CDN caching)
--> DigitalOcean LoadBalancer (TCP passthrough, health checks)
--> Hanzo Ingress [HIP-68] (host routing, middleware, backend selection)
--> API Gateway [HIP-44] (auth, rate limiting, request transformation)
--> Backend Services (LLM Gateway, IAM, KMS, ...)Each layer has a single responsibility. Ingress does not authenticate requests. The API Gateway does not route by hostname. Cloudflare does not know about backend services. This separation means each layer can be replaced, scaled, or debugged independently.
localhost). No traffic leaves the cluster for internal routing.kubectl port-forward.includeSubdomains and preload on all HTTPS responses.Copyright and related rights waived via CC0.