hanzo-storage

Hanzo Storage is a high-performance, S3-compatible object storage server for AI workloads.

Hanzo Storage - S3-Compatible Object Storage

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-platform.md, hanzo/hanzo-kms.md, hanzo/hanzo-universe.md

Overview

Hanzo Storage is a high-performance, S3-compatible object storage server for AI workloads. Derived from SeaweedFS and optimized for the Hanzo ecosystem. Written in Go. Live in production on Hanzo K8s clusters.

Why Hanzo Storage?

S3 API Compatible: Drop-in replacement for Amazon S3; works with any S3 client or SDK
Built for AI: Optimized for large model artifacts, training datasets, and data pipelines
Erasure Coding: Configurable data redundancy across drives and nodes
Server-Side Encryption: SSE-S3 and SSE-KMS for data at rest and in transit
Multi-Tenancy: Isolated namespaces and access boundaries for teams and services
CLI: the s3 binary's own shell — 153 commands covering fs, cluster, ec, volume, and mq

Tech Stack

Language: Go (module: github.com/hanzoai/s3)
Go Version: 1.26.5
License: Apache-2.0

OSS Base

Derived from SeaweedFS (Apache-2.0), attribution retained in LICENSE. Repo: hanzoai/s3. go.mod names zero MinIO modules; the checksum libraries are our Apache-2.0 forks under hanzos3/ (crc64nvme, md5-simd, go).

There is no separate CLI or console repo. mc and the MinIO console are not part of this stack — use the s3 shell for administration.

When to use

Storing model artifacts, checkpoints, and training datasets
Serving static assets and large files via S3 API
Providing S3-compatible object storage for Hanzo services
Replacing Amazon S3 in self-hosted or hybrid deployments
Bucket lifecycle management (expiration, tiering)

Hard requirements

Go 1.26+ to build from source
Docker for container deployment
NVMe/SSD storage recommended for performance

Quick reference

| Item | Value | |------|-------| | Repo | github.com/hanzoai/s3 | | Module | github.com/hanzoai/s3 | | Go Version | 1.26.5 | | License | Apache-2.0 (SeaweedFS-derived) | | S3 API Port | 9000 | | Filer / Master / Volume | 8888 / 9333 / 8080 | | Docker Image | ghcr.io/hanzoai/s3:v1.0.14 (pin a semver; never :latest) | | CLI | the s3 shell (in-binary) |

One-file quickstart

Docker

docker run -p 9000:9000 -v s3-data:/data ghcr.io/hanzoai/s3:v1.0.14 \
  server -dir=/data -filer -s3 -s3.port=9000

Build from source

git clone https://github.com/hanzoai/s3.git
cd s3 && go build -o s3 .
./s3 server -dir=/data -filer -s3 -s3.port=9000

Administer with the `s3` shell

The shell is the CLI. There is no mc.

s3 shell
> s3.bucket.create -name my-bucket
> s3.bucket.list
> s3.accesskey.create -user alice
> fs.ls /buckets/my-bucket
> cluster.status

fs. walks the filer namespace, s3. manages buckets and access keys, cluster. and ec. cover topology and erasure coding.

Docker Compose

# compose.yml
services:
  s3:
    image: ghcr.io/hanzoai/s3:v1.0.14
    ports:
      - "9000:9000"
    command: server -dir=/data -ip.bind=0.0.0.0 -filer -s3 -s3.port=9000
    environment:
      AWS_ACCESS_KEY_ID: "${S3_ACCESS_KEY}"
      AWS_SECRET_ACCESS_KEY: "${S3_SECRET_KEY}"
    volumes:
      - s3-data:/data

volumes:
  s3-data:

Core Concepts

Architecture

S3 clients (any SDK) ──> s3 gateway (:9000)
                              │
                          filer (:8888) ──> master (:9333) ──> volume (:8080)

Master tracks topology, volume stores chunks, filer holds the namespace, and the S3 gateway maps buckets onto filer paths.

SDK Compatibility

Standard AWS SDKs (aws-sdk-go, boto3, @aws-sdk/client-s3) work without modification — prefer them. For Go, github.com/hanzos3/go is our Apache-2.0 client.

Key Internal Packages

s3/
 main.go # Entry point (calls cmd.Main)
 cmd/ # CLI commands and server logic
 internal/
 auth/ # Authentication
 bucket/ # Bucket management
 config/ # Configuration
 crypto/ # Encryption (SSE-S3, SSE-KMS)
 event/ # Event notification
 grid/ # Internal RPC grid
 hash/ # Content hashing
 http/ # HTTP server
 jwt/ # JWT handling
 kms/ # KMS integration
 s3select/ # S3 Select queries
 store/ # Object store backend
 helm/ # Helm charts
 workers/ # Background workers
 buildscripts/ # Build automation
 dockerscripts/ # Docker entry scripts

Environment Variables

# S3 credentials (standard AWS names)
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

# Admin identities
S3_ADMIN_USER=...
S3_ADMIN_PASSWORD=...
S3_ADMIN_READONLY_USER=...
S3_ADMIN_READONLY_PASSWORD=...

# Optional
S3_BUCKET=...
S3_EXTERNAL_URL=...

Credentials come from KMS, never from literals in a manifest.

Key Features

Erasure Coding: ec.encode / ec.rebuild / ec.balance / ec.scrub
Bucket Policies: s3.bucket.access, s3.bucket.owner, s3.anonymous.*
Object Lifecycle: s3.bucket.lifecycle (expiration and tiering rules)
Quotas and Locking: s3.bucket.quota, s3.bucket.lock
Versioning: Object versioning with delete markers
Remote Tiering: remote.mount / remote.cache against a backing store

Troubleshooting

| Issue | Cause | Solution | |-------|-------|----------| | Reaching for mc | Habit from the MinIO era | Use s3 shell; mc is not part of this stack | | Build fails | Go version mismatch | Requires Go 1.26+ | | S3 API refuses a bucket | Bucket has no filer path yet | s3.bucket.create -name <bucket> in the shell | | Volume full | volumeSizeLimitMB reached | Raise -master.volumeSizeLimitMB or add volume servers |

Related Skills

hanzo/hanzo-platform.md - PaaS deployment platform
hanzo/hanzo-kms.md - Secret management (SSE-KMS integration)
hanzo/hanzo-universe.md - Production K8s infrastructure
hanzo/hanzo-pubsub.md - Event notification targets (NATS)

Category: Hanzo Ecosystem Related: s3, storage, object-storage, seaweedfs, ai-data Prerequisites: Go 1.26+, Docker