hanzo-sentry

Hanzo Sentry is a self-hosted Sentry 24.2.0 deployment for error tracking and performance monitoring across all Hanzo, Lux, Zoo, and Pars services.

Hanzo Sentry - Error Tracking & Performance Monitoring

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-o11y.md, hanzo/hanzo-kms.md, hanzo/hanzo-id.md, hanzo/hanzo-universe.md

Overview

Hanzo Sentry is a self-hosted Sentry 24.2.0 deployment for error tracking and performance monitoring across all Hanzo, Lux, Zoo, and Pars services. It is a fork of getsentry/sentry with a custom Dockerfile branded for Hanzo, configured for OIDC SSO via hanzo.id, and deployed to the hanzo-k8s cluster.

Why Hanzo Sentry?

Self-hosted: Full control over data, no SaaS dependency
Multi-tenant: Single instance serving four domains (hanzo, lux, pars, zoo)
IAM integrated: OIDC SSO login via hanzo.id (app-sentry client)
Secrets via KMS: All credentials synced from kms.hanzo.ai
OTEL instrumented: Exports traces to the shared OTEL collector
Shared infrastructure: Uses hanzo-sql (PostgreSQL) and shared Redis

Tech Stack

Backend: Python 3.11 (Django 5.x, Celery, DRF)
Frontend: React + TypeScript (Webpack, Emotion CSS)
Queue: Celery via Redis (or RabbitMQ if configured)
Database: PostgreSQL via hanzo-sql.hanzo.svc
Cache: Redis via redis.hanzo.svc
Image: ghcr.io/hanzoai/sentry:latest
Base version: Sentry 24.2.0 (FSL-1.0-Apache-2.0 license)

OSS Base

Fork of getsentry/sentry. Local clone at ~/work/hanzo/sentry/. The GitHub repo hanzoai/sentry does not yet exist as a public fork -- currently origin points to getsentry/sentry and Hanzo customizations live in the self-hosted/ directory and K8s manifests in universe/infra/k8s/sentry/.

When to use

Instrumenting Hanzo services with error tracking
Monitoring performance of API endpoints, workers, and frontends
Viewing crash reports, stack traces, and breadcrumbs
Setting up alerting rules for error spikes
Tracking release health across deployments

Hard requirements

PostgreSQL via hanzo-sql.hanzo.svc (database sentry)
Redis via redis.hanzo.svc (cache, queues, rate limits, buffers)
KMS at kms.hanzo.ai for secret sync (secret-key, db-password, oidc-client-secret)
IAM at hanzo.id with app-sentry OIDC client for SSO

Quick reference

| Item | Value | |------|-------| | UI | https://sentry.hanzo.ai | | Alt domains | sentry.lux.network, sentry.pars.network, sentry.zoo.network | | Port | 9000 (web), internal K8s service | | Image | ghcr.io/hanzoai/sentry:latest | | Base version | Sentry 24.2.0 | | Language | Python 3.11 + TypeScript | | K8s namespace | hanzo | | Deployments | sentry-web (2 replicas), sentry-worker (2 replicas), sentry-cron (1 replica) | | Database | hanzo-sql.hanzo.svc:5432/sentry | | Redis | redis.hanzo.svc:6379 | | Auth | OIDC SSO via hanzo.id (app-sentry) | | KMS project | hanzo-k8s-epiq, path /sentry | | Local clone | ~/work/hanzo/sentry/ | | K8s manifests | ~/work/hanzo/universe/infra/k8s/sentry/ | | Health check | GET /_health/ |

One-file quickstart

Instrument a Python service

import sentry_sdk

sentry_sdk.init(
 dsn="https://<key>@sentry.hanzo.ai/<project-id>",
 traces_sample_rate=0.1,
 environment="production",
 release="[email protected]",
)

Instrument a Node.js service

const Sentry = require("@sentry/node");

Sentry.init({
 dsn: "https://<key>@sentry.hanzo.ai/<project-id>",
 tracesSampleRate: 0.1,
 environment: "production",
 release: "[email protected]",
});

Instrument a Go service

import "github.com/getsentry/sentry-go"

func main() {
 sentry.Init(sentry.ClientOptions{
 Dsn: "https://<key>@sentry.hanzo.ai/<project-id>",
 TracesSampleRate: 0.1,
 Environment: "production",
 Release: "[email protected]",
 })
 defer sentry.Flush(2 * time.Second)
}

Docker Compose (local development)

# compose.yml
services:
 sentry-web:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "web"]
 ports:
 - "9000:9000"
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 sentry-worker:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "worker"]
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 sentry-cron:
 image: ghcr.io/hanzoai/sentry:latest
 command: ["sentry", "run", "cron"]
 environment:
 SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
 SENTRY_POSTGRES_HOST: postgres
 SENTRY_DB_NAME: sentry
 SENTRY_DB_USER: sentry
 SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
 SENTRY_REDIS_HOST: redis

 postgres:
 image: postgres:16
 environment:
 POSTGRES_DB: sentry
 POSTGRES_USER: sentry
 POSTGRES_PASSWORD: "${SENTRY_DB_PASSWORD}"

 redis:
 image: redis:7-alpine

Core Concepts

Architecture

 ┌──────────────────────────────────┐
 │ K8s Ingress │
 │ sentry.hanzo.ai │
 │ sentry.lux.network │
 │ sentry.pars.network │
 │ sentry.zoo.network │
 └───────────────┬──────────────────┘
 │
 ┌───────────────▼──────────────────┐
 │ sentry-web (2 replicas) │
 │ Django + uWSGI on port 9000 │
 └──┬────────────┬─────────────┬────┘
 │ │ │
 ┌──────▼──┐ ┌─────▼─────┐ ┌───▼────────┐
 │ Postgres │ │ Redis │ │ OTEL │
 │ hanzo-sql│ │ (shared) │ │ Collector │
 └─────────┘ └───────────┘ └────────────┘
 │ │
 ┌──────▼──┐ ┌─────▼──────────────┐
 │ sentry- │ │ sentry-worker │
 │ cron │ │ (2 replicas, Celery) │
 └─────────┘ └─────────────────────┘

Three Deployments

| Component | Command | Replicas | Purpose | |-----------|---------|----------|---------| | sentry-web | sentry run web | 2 | Django web server (uWSGI) | | sentry-worker | sentry run worker | 2 | Celery async task processing | | sentry-cron | sentry run cron | 1 | Scheduled tasks (digests, cleanup) |

Multi-Tenant Domains

A single Sentry instance serves four domains via K8s Ingress. Each domain maps to a separate Sentry organization, allowing per-org project isolation:

sentry.hanzo.ai -- Hanzo AI services
sentry.lux.network -- Lux blockchain nodes
sentry.pars.network -- Pars network services
sentry.zoo.network -- Zoo Labs experiments

Auth Flow

SSO is via OIDC against hanzo.id:

User visits sentry.hanzo.ai and clicks "SSO Login"
Redirected to hanzo.id for authentication (client app-sentry)
OIDC callback returns to Sentry with identity token
Sentry creates/links user account and grants org membership

Dockerfile Customizations

The self-hosted/Dockerfile in the local clone contains Hanzo-specific changes:

OCI labels with [email protected] maintainer and Hanzo branding
References ghcr.io/hanzoai/sentry as image source
Copies self-hosted/sentry.conf.py and self-hosted/config.yml into /etc/sentry/
Entrypoint via tini + gosu for proper signal handling

KMS Secret Sync

Secrets are managed via the KMSSecret CRD that syncs from kms.hanzo.ai:

# kms-secrets.yaml
spec:
 hostAPI: https://kms.hanzo.ai/api
 managedSecretReference:
 secretName: sentry-secrets
 secretNamespace: hanzo
 authentication:
 universalAuth:
 secretsScope:
 projectSlug: hanzo-k8s-epiq
 envSlug: prod
 secretsPath: /sentry

Required secrets: secret-key, db-password, oidc-client-secret, commerce-api-key (optional).

Environment Variables

# Database (PostgreSQL)
SENTRY_POSTGRES_HOST=hanzo-sql.hanzo.svc
SENTRY_DB_NAME=sentry
SENTRY_DB_USER=sentry
SENTRY_DB_PASSWORD=<from KMS>
SENTRY_DB_PORT=5432

# Redis
SENTRY_REDIS_HOST=redis.hanzo.svc
SENTRY_REDIS_PORT=6379

# Core
SENTRY_SECRET_KEY=<from KMS>

# OIDC SSO
SENTRY_OIDC_ENABLED=true
SENTRY_OIDC_ISSUER=https://hanzo.id
SENTRY_OIDC_CLIENT_ID=app-sentry
SENTRY_OIDC_CLIENT_SECRET=<from KMS>

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.hanzo.svc:4318
OTEL_SERVICE_NAME=sentry

# Optional: Commerce billing
HANZO_COMMERCE_URL=https://commerce.hanzo.ai
HANZO_COMMERCE_API_KEY=<from KMS>

Local Development

cd ~/work/hanzo/sentry

# Python setup (uses direnv + pyenv, not uv — upstream pattern)
make develop

# Run dependent services (Postgres, Redis, Kafka, etc.)
make run-dependent-services

# Apply database migrations
make apply-migrations

# Start web server
sentry run web

# In another terminal: start worker
sentry run worker

# Run Python tests
make test-python-ci

# Run JavaScript tests
make test-js

Troubleshooting

| Issue | Cause | Solution | |-------|-------|----------| | OIDC login fails | app-sentry not configured in IAM | Create OIDC client at hanzo.id with redirect URI https://sentry.hanzo.ai/auth/sso/ | | Redis connection error | Missing SENTRY_REDIS_HOST | Ensure redis.hanzo.svc is reachable in cluster | | SENTRY_SECRET_KEY is undefined | KMS sync not running | Verify sentry-secrets K8s secret exists in hanzo namespace | | Migration errors | DB not created | Run sentry upgrade to create tables and apply migrations | | Worker not processing | Celery not connected to broker | Check Redis connectivity or RabbitMQ config | | Health check failing | Sentry not fully started | Wait for initialDelaySeconds (60s); check logs for startup errors | | npm ci fails | Peer dep conflicts | Use npm install with .npmrc containing legacy-peer-deps=true |

Related Skills

hanzo/hanzo-o11y.md - Observability stack (OTEL, Prometheus, Grafana)
hanzo/hanzo-id.md - IAM and OIDC SSO configuration
hanzo/hanzo-kms.md - Secret management via KMSSecret CRDs
hanzo/hanzo-universe.md - Production K8s infrastructure and manifests
hanzo/hanzo-database.md - Shared PostgreSQL (hanzo-sql)

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: sentry, error-tracking, monitoring, observability, self-hosted Prerequisites: Kubernetes, PostgreSQL, Redis, OIDC basics