HIP-28: Key-Value Store Standard. Status Active. Hanzo's own standard — read this before implementing against it.
This proposal defines the standard for Hanzo KV, the high-performance key-value store that serves as the shared caching, session, pub/sub, and streaming backbone for all services in the Hanzo ecosystem. Hanzo KV is built on KV 8.1, the Linux Foundation fork of Redis, and is distributed as ghcr.io/hanzoai/kv:latest. It exposes the RESP3 wire protocol on port 6379 and is a drop-in replacement for any KV client.
Repository: github.com/hanzoai/kv Port: 6379 Docker: ghcr.io/hanzoai/kv:latest and docker.io/hanzoai/kv:latest License: BSD-3-Clause
Every service in the Hanzo ecosystem -- IAM, LLM Gateway, Cloud, Chat, Commerce, Bot, Analytics, Zen -- needs a fast shared store for at least one of the following:
Previously, the Hanzo infrastructure relied on the Bitnami KV Helm chart deployed via helm install redis bitnami/redis. This worked, but introduced three problems:
in March 2024. Both licenses restrict how cloud providers and SaaS platforms can distribute KV. For an infrastructure company like Hanzo that ships managed services, this is a direct legal exposure.
init containers, and Sentinel by default. When any of these sidecars fail (e.g., the exporter cannot authenticate to a password-protected instance), the entire pod enters CrashLoopBackOff and the root cause is obscured.
KV image is ~12MB. In a cluster with rolling updates, smaller images mean faster pulls and shorter disruption windows.
Hanzo KV solves all three by replacing the entire Bitnami stack with a single, purpose-built container image based on KV.
This section explains every major design decision and why the alternatives were rejected. Infrastructure choices compound -- a wrong call here propagates to every service that touches KV. Each heading below addresses one decision.
Hanzo KV currently runs as a single-instance StatefulSet with 2Gi of PVC storage and a 2Gi memory limit. This is a deliberate choice, not a shortcut.
Scale math: Our current production dataset (sessions, rate-limit counters, cache entries across all services) occupies approximately 400MB of memory. Even with 10x growth, we stay under 4GB. A single KV instance on modern hardware can saturate a 10Gbps NIC at ~1.2 million ops/sec. Our peak observed throughput is approximately 8,000 ops/sec. We are three orders of magnitude below the single-node ceiling.
Cluster complexity: KV Cluster (and by extension KV Cluster) introduces hash slots, cross-slot restrictions on multi-key operations, MOVED/ASK redirects, and cluster bus gossip traffic. Every KV client library must understand cluster topology. Some operations (MULTI/EXEC across slots, Lua scripts touching multiple keys on different slots) simply do not work. This complexity buys horizontal scaling we do not need.
Failure modes: A single instance has exactly one failure mode -- the pod dies and restarts. With AOF persistence, data loss on restart is bounded to the last fsync interval (1 second by default). A cluster has N failure modes: split-brain during network partition, slot migration failures, gossip protocol desynchronization, and partial availability when a master is down and its replica has not yet been promoted.
Vertical ceiling: Kubernetes nodes support up to 64GB of memory. We can scale the KV StatefulSet to 32GB before even considering cluster mode. When we reach that point (which would imply ~80x current load), we will revisit with a separate HIP.
The Bitnami KV chart ships with a redis-exporter sidecar that scrapes INFO output and exposes Prometheus metrics on port 9121. When we migrated to Hanzo KV with password authentication, the exporter sidecar could not authenticate because it expected the password in a different environment variable format than our secret layout provided.
Rather than debug the exporter's authentication logic and add another secret reference, we removed the sidecar entirely. The reasoning:
INFO command already provides all metrics (memory, connections,keyspace, replication, persistence) in a machine-parseable format.
kubectl exec into the pod and running kv-cli INFO issufficient for debugging.
oliver006/redis_exporteras a separate Deployment (not a sidecar) with its own authentication config, decoupled from the KV pod lifecycle.
Principle: a database pod should contain exactly one process -- the database. Every sidecar is a potential crash-loop vector that takes the database down with it.
Hanzo KV implements RESP3 (REdis Serialization Protocol version 3) as defined by the KV protocol specification. All commands from the KV 7.2 command set are supported. Any client library that speaks RESP2 or RESP3 is compatible.
host: localhost
port: 6379
password: <from K8s secret "redis", key "redis-password">
db: 0 # default database
protocol: resp3 # RESP3 preferred, RESP2 accepted
tls: false # intra-cluster, TLS not requiredServices should construct their connection URL as:
redis://:${REDIS_PASSWORD}@redis-master:6379/0Or for explicit host within the hanzo namespace:
redis://:${REDIS_PASSWORD}@localhost:6379/0The production kv.conf (mounted from ConfigMap):
# Persistence: AOF only, no RDB snapshots
appendonly yes
save ""
# Eviction: LRU when memory limit is reached
maxmemory-policy allkeys-lru
# Safety: disable destructive bulk operations
rename-command FLUSHDB ""
rename-command FLUSHALL ""Additional settings applied via container command-line arguments:
--requirepass $(REDIS_PASSWORD) # authentication
--dir /data # persistence directory
--bind 0.0.0.0 # accept connections on all interfaces
--maxmemory-policy allkeys-lru # eviction policy (also in kv.conf for safety)
--protected-mode no # allow non-loopback connections (K8s networking)Readiness probe (is the instance ready to accept commands?):
exec:
command: ["sh", "-c", "kv-cli -a \"$REDIS_PASSWORD\" ping | grep -q PONG"]
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3Liveness probe (is the instance alive and not deadlocked?):
exec:
command: ["sh", "-c", "kv-cli -a \"$REDIS_PASSWORD\" ping | grep -q PONG"]
initialDelaySeconds: 15
periodSeconds: 30
failureThreshold: 5The liveness probe has a longer initialDelaySeconds and failureThreshold to avoid killing a pod that is replaying a large AOF on startup.
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 1000m
memory: 2GiThe memory limit (2Gi) acts as a hard ceiling. Combined with allkeys-lru, KV will evict the least-recently-used keys when approaching this limit rather than crashing with an OOM error.
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 2GiThe PVC stores the AOF file. With our current workload, the AOF (after automatic compaction) stays under 100MB. The 2Gi allocation provides 20x headroom.
The Dockerfile is minimal by design:
ARG KV_VERSION=8.1
FROM valkey/valkey:${KV_VERSION}-alpine AS base
FROM base
LABEL maintainer="dev@hanzo.ai"
LABEL org.opencontainers.image.source="https://github.com/hanzoai/kv"
LABEL org.opencontainers.image.description="Hanzo KV - High-performance key-value store"
LABEL org.opencontainers.image.vendor="Hanzo AI"
# Install Hanzo KV CLI tools
# Primary names are kv-* ; legacy valkey-* names remain as symlinks
RUN cp /usr/local/bin/valkey-server /usr/local/bin/kv-server \
&& cp /usr/local/bin/valkey-cli /usr/local/bin/kv-cli \
&& ln -sf /usr/local/bin/kv-cli /usr/local/bin/kv \
&& cp /usr/local/bin/valkey-sentinel /usr/local/bin/kv-sentinel 2>/dev/null; \
cp /usr/local/bin/valkey-benchmark /usr/local/bin/kv-benchmark 2>/dev/null; \
cp /usr/local/bin/valkey-check-aof /usr/local/bin/kv-check-aof 2>/dev/null; \
cp /usr/local/bin/valkey-check-rdb /usr/local/bin/kv-check-rdb 2>/dev/null; \
true
EXPOSE 6379
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
CMD kv ping | grep -q PONG || exit 1
ENTRYPOINT ["kv-server"]
CMD ["--bind", "0.0.0.0", "--dir", "/data", \
"--maxmemory-policy", "allkeys-lru", "--protected-mode", "no"]Key points:
valkey/valkey:8.1-alpine (~12MB compressed)kv-* names. The original valkey-* names remain as the originals. This gives operators a clean Hanzo-branded CLI while maintaining compatibility with scripts that reference valkey-cli.
would create a maintenance burden and diverge from upstream security fixes.
| Command | Description | |---------|-------------| | kv | Interactive CLI (symlink to kv-cli) | | kv-server | Start KV server | | kv-cli | Command-line client | | kv-sentinel | High-availability sentinel | | kv-benchmark | Performance benchmarking tool | | kv-check-aof | AOF file integrity checker | | kv-check-rdb | RDB file integrity checker |
The deploy workflow (.github/workflows/deploy.yml) has two stages:
Stage 1: Build
github.com/hanzoai/kvlinux/amd64, linux/arm64) via Docker Buildxghcr.io/hanzoai/kv) with tags: latest, git SHA, semverdocker.io/hanzoai/kv) as fallback (continue-on-error)Stage 2: Deploy (main branch only)
kubectl for the cluster cluster via doctlkubectl set image statefulset/redis-master kv=ghcr.io/hanzoai/kv:latestkubectl rollout status statefulset/redis-master --timeout=120sTrigger conditions: push to main, tag push (v*), or manual workflow_dispatch.
All manifests live in universe/infra/k8s/kv/ and are aggregated via Kustomize:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- statefulset.yaml
- service.yaml
- secret.yaml
- configmap.yamlThe migration from the Bitnami KV Helm chart to Hanzo KV was performed as follows:
helm uninstall redis removes the Deployment and Service but preserves the PVC (Helm default resourcePolicy: keep).
redis-data), same Service name (redis-master), and same secret name (redis). This means the new pod attaches to the existing PVC with all data intact.
kv-cli -a "$REDIS_PASSWORD" DBSIZE confirms key count matchespre-migration.
The migration is zero-downtime because the Service name and selector labels are preserved. Client connections fail for the ~30 seconds between the old pod terminating and the new pod passing its readiness probe, which is within the retry tolerance of all Hanzo services.
| Language | Package | Install | |----------|---------|---------| | Python | hanzo-kv | pip install hanzo-kv | | Go | hanzo/kv-go | go get github.com/hanzoai/kv-go | | Node.js | @hanzo/kv | npm install @hanzo/kv |
All three are thin wrappers around standard KV client libraries (redis-py, go-redis, ioredis) with Hanzo-specific defaults (connection URL construction, KMS secret resolution, structured logging). Any vanilla KV client works equally well.
All connections require a password. The password is stored in a K8s Secret:
apiVersion: v1
kind: Secret
metadata:
name: redis
namespace: hanzo
type: Opaque
stringData:
redis-password: "<generated-value>"Services receive the password via environment variable injection from this secret. The secret name (redis) and key (redis-password) match the Bitnami convention to avoid changing every service deployment manifest.
In production, this secret is synced from Hanzo KMS (kms.hanzo.ai) via the KMS Operator. The plaintext value in the manifest is a bootstrap default that gets overwritten on first KMS sync.
hanzo namespace (or with appropriate NetworkPolicy) can reachport 6379
--protected-mode no flag is safe because the pod is never exposed outside thecluster. Protected mode is a KV safety net for instances accidentally exposed to the internet without a password; our instance has both network isolation and a password.
As specified in the Configuration section, FLUSHDB and FLUSHALL are renamed to empty strings (disabled). Additional commands to consider disabling in future:
DEBUG -- can crash the server or dump memoryCONFIG -- can change runtime settings (e.g., disable authentication)SHUTDOWN -- can stop the serverThese are not currently disabled because they are useful for debugging in a cluster environment where only operators have kubectl exec access.
TLS is available in KV 8.1 but not enabled for intra-cluster communication. The reasoning:
kubectl accessand can read secrets directly
If we add external replication (e.g., cross-cluster) or expose KV outside the VPC, TLS will be enabled via --tls-port 6380 --tls-cert-file --tls-key-file --tls-ca-cert-file.
The 2Gi memory limit prevents a runaway client from consuming all node memory and triggering the Linux OOM killer (which would kill the KV process and potentially other pods on the same node). With allkeys-lru, KV gracefully evicts cold keys instead of refusing writes or crashing.
Services in the Hanzo ecosystem that connect to KV:
| Service | Use Case | Key Pattern | |---------|----------|-------------| | IAM (hanzo.id) | Session tokens, OAuth state | iam:session:, iam:oauth: | | LLM Gateway | Rate limiting, response cache | llm:rate:, llm:cache: | | Cloud | Job queues, inference state | cloud:job:, cloud:inf: | | Console | Session cache | console:session: | | Chat | Conversation state, pub/sub | chat:conv:, chat:stream: | | Bot | Command state, cooldowns | bot:state:, bot:cd: | | Analytics | Event buffering | analytics:buf: | | Zen | Model routing cache | zen:route: | | Commerce | Cart state, rate limits | commerce:cart: |
All keys SHOULD be prefixed with <service>:<category>:<id>. This enables:
kv-cli --stat or SCAN with pattern matchingKV's INFO command provides comprehensive metrics without any sidecar:
# Memory usage
kv-cli -a "$REDIS_PASSWORD" INFO memory
# Client connections
kv-cli -a "$REDIS_PASSWORD" INFO clients
# Keyspace statistics
kv-cli -a "$REDIS_PASSWORD" INFO keyspace
# Persistence status
kv-cli -a "$REDIS_PASSWORD" INFO persistence
# All metrics
kv-cli -a "$REDIS_PASSWORD" INFO all| Metric | Warning Threshold | Critical Threshold | |--------|------------------|--------------------| | used_memory | > 1.5Gi (75% of limit) | > 1.8Gi (90%) | | connected_clients | > 100 | > 500 | | evicted_keys | > 0 (indicates memory pressure) | > 1000/min | | rejected_connections | > 0 | > 10/min | | aof_last_bgrewrite_status | err | - | | instantaneous_ops_per_sec | > 50,000 | > 100,000 |
When continuous monitoring is needed, deploy oliver006/redis_exporter as a standalone Deployment in the hanzo namespace:
apiVersion: apps/v1
kind: Deployment
metadata:
name: kv-exporter
namespace: hanzo
spec:
replicas: 1
template:
spec:
containers:
- name: exporter
image: oliver006/redis_exporter:latest
env:
- name: REDIS_ADDR
value: redis-master:6379
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis
key: redis-password
ports:
- containerPort: 9121This runs as a separate pod, not a sidecar. If the exporter crashes, KV is unaffected.
Repository: github.com/hanzoai/kv
Key Files:
Dockerfile -- Multi-arch container image based on KV 8.1 Alpine.github/workflows/deploy.yml -- CI/CD: build, push to GHCR/Docker Hub, deploy to K8s.github/workflows/ci.yml -- Upstream KV test suitevalkey.conf -- Full reference configuration (upstream defaults)sentinel.conf -- Sentinel configuration for HA deploymentsK8s Manifests (universe/infra/k8s/kv/):
statefulset.yaml -- StatefulSet redis-master with PVC and health checksservice.yaml -- ClusterIP Service on port 6379configmap.yaml -- kv.conf (AOF, eviction policy, disabled commands)secret.yaml -- KV-compatible password secretkustomization.yaml -- Kustomize aggregationStatus: Implemented and running in production on the cluster (24.199.76.156)
Copyright and related rights waived via CC0.