hanzo-kensho

Kensho is a 17B parameter image generation foundation model using a Mixture of Experts (MoE) Diffusion Transformer architecture.

Hanzo Kensho - Image Generation Foundation Model

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-ml.md, hanzo/hanzo-engine.md, hanzo/hanzo-candle.md

Overview

Kensho is a 17B parameter image generation foundation model using a Mixture of Experts (MoE) Diffusion Transformer architecture. It generates high-quality images from text prompts at multiple resolutions, achieving state-of-the-art scores on DPG-Bench (85.89) and GenEval benchmarks.

The architecture combines a DiT (Diffusion Transformer) backbone with a MoE feed-forward network (4 routed experts, 2 activated per token) and uses Llama 3.1-8B-Instruct as a text encoder for deep prompt understanding. Generation uses flow matching schedulers for fast, high-fidelity output.

Zen MoDE Architecture

Kensho implements the Zen MoDE (Mixture of Diverse Experts) methodology applied to image diffusion:

• MoE gating with softmax scoring and top-k expert selection
• Load balancing loss for uniform expert utilization across distributed training
• SwiGLU feed-forward blocks within both joint and single transformer layers
• Flash Attention for memory-efficient cross-modal attention between image patches and text embeddings

When to use

• Text-to-image generation at production quality
• High-resolution image synthesis (up to 1360x768, multiple aspect ratios)
• Applications requiring deep prompt understanding (complex scenes, text rendering)
• Research into MoE diffusion architectures

Hard requirements

1. Python 3.10+ with PyTorch 2.5+
2. CUDA 12.4+ with Flash Attention installed
3. GPU VRAM: ~40GB (full model in bfloat16)
4. Llama 3.1-8B-Instruct access (HuggingFace gated model)

Quick reference

| Item | Value | |------|-------| | Repo | github.com/hanzoai/kensho | | Branch | main | | Language | Python | | Parameters | 17B | | Architecture | MoE Diffusion Transformer | | Precision | bfloat16 | | Install | pip install -r requirements.txt && pip install -U flash-attn --no-build-isolation | | Inference | python inference.py --model_type full | | License | See repo |

Model Variants

| Variant | Inference Steps | Guidance Scale | Flow Shift | Scheduler | |---------|----------------|----------------|------------|-----------| | Full | 50 | 5.0 | 3.0 | FlowUniPCMultistep | | Dev | 28 | 0.0 | 6.0 | FlashFlowMatchEuler | | Fast | 16 | 0.0 | 3.0 | FlashFlowMatchEuler |

Supported Resolutions

| Resolution | Aspect | |------------|--------| | 1024 x 1024 | Square | | 768 x 1360 | Portrait | | 1360 x 768 | Landscape | | 880 x 1168 | Portrait | | 1168 x 880 | Landscape | | 1248 x 832 | Landscape | | 832 x 1248 | Portrait |

Architecture Details

Text Prompt
 |
 v
┌──────────────────────────────┐
│ Llama 3.1-8B-Instruct │ (frozen text encoder)
│ + CLIP/T5 text encoders │
└──────────┬───────────────────┘
 |
 v
┌──────────────────────────────┐
│ Kensho DiT (17B params) │
│ │
│ PatchEmbed(4ch -> 1024dim) │
│ + RoPE positional encoding │
│ + Timestep embedding │
│ │
│ Joint Transformer Blocks: │
│ - Cross-attention (img+txt)│
│ - adaLN modulation │
│ - MoE FFN (SwiGLU) │
│ 4 experts, top-2 routing│
│ + load balancing loss │
│ │
│ Single Transformer Blocks: │
│ - Self-attention (img) │
│ - adaLN modulation │
│ - MoE FFN (SwiGLU) │
│ │
│ Output projection │
└──────────────────────────────┘
 |
 v
 Generated Image

MoE Gate Implementation

The expert routing uses a softmax-based gating mechanism:

• Input projected through learned weight matrix to num_routed_experts logits
• Softmax scoring with top-k selection (k=2 by default)
• Auxiliary load balancing loss (alpha=0.01) for training stability
• All-gather across distributed ranks for balanced expert utilization

One-file quickstart

import torch
from hi_diffusers import HiDreamImagePipeline, HiDreamImageTransformer2DModel
from hi_diffusers.schedulers.fm_solvers_unipc import FlowUniPCMultistepScheduler
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

# Load text encoder
tokenizer = PreTrainedTokenizerFast.from_pretrained(
 "meta-llama/Meta-Llama-3.1-8B-Instruct", use_fast=False
)
text_encoder = LlamaForCausalLM.from_pretrained(
 "meta-llama/Meta-Llama-3.1-8B-Instruct",
 output_hidden_states=True,
 output_attentions=True,
 torch_dtype=torch.bfloat16,
).to("cuda")

# Load pipeline
scheduler = FlowUniPCMultistepScheduler(
 num_train_timesteps=1000, shift=3.0, use_dynamic_shifting=False
)
transformer = HiDreamImageTransformer2DModel.from_pretrained(
 "HiDream-ai/HiDream-I1-Full",
 subfolder="transformer",
 torch_dtype=torch.bfloat16,
).to("cuda")

pipe = HiDreamImagePipeline.from_pretrained(
 "HiDream-ai/HiDream-I1-Full",
 scheduler=scheduler,
 tokenizer_4=tokenizer,
 text_encoder_4=text_encoder,
 torch_dtype=torch.bfloat16,
).to("cuda", torch.bfloat16)
pipe.transformer = transformer

# Generate
image = pipe(
 "A landscape painting of mountains at sunset",
 height=1024, width=1024,
 guidance_scale=5.0,
 num_inference_steps=50,
 generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("output.png")

Dependencies

torch>=2.5.1
torchvision>=0.20.1
diffusers>=0.32.1
transformers>=4.47.1
einops>=0.7.0
accelerate>=1.2.1
flash-attn (manual install)

Project Structure

kensho/
├── hi_diffusers/
│ ├── __init__.py
│ ├── models/
│ │ ├── attention.py # HiDreamAttention + FeedForwardSwiGLU
│ │ ├── attention_processor.py # Flash Attention processor
│ │ ├── embeddings.py # PatchEmbed, RoPE, TimestepEmbed
│ │ ├── moe.py # MoEGate + MOEFeedForwardSwiGLU
│ │ └── transformers/
│ │ └── transformer_hidream_image.py # Main DiT model
│ ├── pipelines/
│ │ └── hidream_image/
│ │ ├── pipeline_hidream_image.py # Inference pipeline
│ │ └── pipeline_output.py
│ └── schedulers/
│ ├── flash_flow_match.py # FlashFlowMatchEulerDiscreteScheduler
│ └── fm_solvers_unipc.py # FlowUniPCMultistepScheduler
├── inference.py # CLI inference script
├── gradio_demo.py # Gradio web UI
├── requirements.txt
└── assets/

Benchmark Results (DPG-Bench)

| Model | Overall | Global | Entity | Attribute | Relation | Other | |-------|---------|--------|--------|-----------|----------|-------| | DALL-E 3 | 83.50 | 90.97 | 89.61 | 88.39 | 90.58 | 89.83 | | Flux.1-dev | 83.79 | 85.80 | 86.79 | 89.98 | 90.04 | 89.90 | | SD3-Medium | 84.08 | 87.90 | 91.01 | 88.83 | 80.70 | 88.68 | | Kensho | 85.89 | 76.44 | 90.22 | 89.48 | 93.74 | 91.83 |

Related Skills

• hanzo/hanzo-ml.md - Rust ML framework for inference optimization
• hanzo/hanzo-engine.md - Model serving infrastructure
• hanzo/hanzo-candle.md - Candle tensor operations
• hanzo/hanzo-sho.md - Text diffusion (complementary modality)

---

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: image-generation, diffusion, moe, transformer, text-to-image Prerequisites: Python, PyTorch, CUDA, diffusion model concepts