hanzo-extract

Hanzo Extract is a Rust crate for extracting clean text from web pages, PDFs, and Claude Code conversation logs.

Hanzo Extract - Content Extraction & Sanitization

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-aci.md, hanzo/hanzo-guard.md

Overview

Hanzo Extract is a Rust crate for extracting clean text from web pages, PDFs, and Claude Code conversation logs. Designed to produce LLM-ready content with optional PII sanitization via hanzo-guard. Ships two CLI binaries (extract-web, extract-conversations) and a library API with async trait-based extractors.

Why Hanzo Extract?

• Web extraction: Fetches pages, strips scripts/nav/footer, extracts main content area
• PDF extraction: Text extraction from PDF documents via lopdf
• Conversation export: Turns Claude Code JSONL session logs into training datasets with quality scoring and train/val/test splits
• Sanitization: PII redaction pipeline via hanzo-guard (feature-gated)
• Feature flags: Compile only what you need (web, pdf, conversations, sanitize)

Tech Stack

• Language: Rust (edition 2021)
• Crate: hanzo-extract v0.1.0
• Async runtime: Tokio
• Web: reqwest + scraper (HTML parsing/CSS selectors)
• PDF: lopdf
• Conversations: walkdir, sha2, chrono, rand (anonymization + splits)
• Error handling: thiserror
• Serialization: serde + serde_json
• CI: GitHub Actions (ci.yml, pages.yml for docs)

Repo: github.com/hanzoai/extract

When to use

• Extracting clean text from web pages for LLM context/RAG
• Extracting text from PDF documents
• Exporting Claude Code conversations for fine-tuning datasets
• Building content pipelines that need PII redaction

Quick reference

| Item | Value | |------|-------| | Crate | hanzo-extract | | Repo | github.com/hanzoai/extract | | Version | 0.1.0 | | License | MIT OR Apache-2.0 | | Docs | docs.rs/hanzo-extract | | Default features | web, pdf |

Installation

cargo add hanzo-extract

# Cargo.toml - pick what you need
[dependencies]
hanzo-extract = "0.1" # web + pdf (default)
hanzo-extract = { version = "0.1", features = ["full"] } # everything
hanzo-extract = { version = "0.1", features = ["conversations"] } # dataset export

Usage

Web Extraction

use hanzo_extract::{WebExtractor, ExtractorConfig, Extractor};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
 let extractor = WebExtractor::new(ExtractorConfig::default());
 let result = extractor.extract("https://example.com").await?;

 println!("Title: {:?}", result.title);
 println!("Text: {}", result.text);
 println!("Words: {}", result.word_count);
 Ok(())
}

PDF Extraction

use hanzo_extract::{PdfExtractor, Extractor};

let extractor = PdfExtractor::default();
let result = extractor.extract("document.pdf").await?;
println!("{}", result.text);

Conversation Export (training datasets)

use hanzo_extract::conversations::{ConversationExporter, ExporterConfig};
use std::path::Path;

let mut exporter = ConversationExporter::new();
exporter.export(
 Path::new("~/.claude/projects"),
 Path::new("./training-data"),
)?;

Output structure:

./training-data/
 conversations_20260313.jsonl # Full conversation data
 training_20260313.jsonl # Instruction/response pairs
 splits/
 train_20260313.jsonl # 80%
 val_20260313.jsonl # 10%
 test_20260313.jsonl # 10%

CLI Binaries

# Web extraction
cargo install hanzo-extract --features web
extract-web https://example.com
extract-web https://example.com --json

# Conversation export
cargo install hanzo-extract --features conversations
extract-conversations --source ~/.claude/projects --output ./conversations

Configuration

use hanzo_extract::ExtractorConfig;

let config = ExtractorConfig::default()
 .with_max_length(200_000) // Max chars (default: 100,000)
 .with_timeout(60) // Request timeout secs (default: 30)
 .with_clean_text(true); // Strip HTML/scripts (default: true)

// With sanitization (requires `sanitize` feature)
let config = config
 .with_sanitize(true)
 .with_redact_pii(true)
 .with_detect_injection(true);

Feature Flags

| Feature | Default | Description | |---------|---------|-------------| | web | Yes | Web page extraction (reqwest + scraper) | | pdf | Yes | PDF text extraction (lopdf) | | conversations | No | Claude Code session export | | sanitize | No | PII redaction via hanzo-guard | | full | No | All features |

Key Files

| File | Purpose | |------|---------| | src/lib.rs | Crate root, Extractor trait, re-exports | | src/web.rs | WebExtractor with HTML parsing and content area detection | | src/pdf.rs | PdfExtractor for PDF text extraction | | src/conversations.rs | ConversationExporter with quality scoring and splits | | src/config.rs | ExtractorConfig with builder pattern | | src/sanitize.rs | Sanitization pipeline (hanzo-guard integration) | | src/error.rs | ExtractError enum (thiserror) | | src/result.rs | ExtractResult struct |

Quality Scoring (Conversations)

Conversations are scored 0.0-1.0 based on:

• Thinking/reasoning presence (+0.2)
• Tool usage (+0.15)
• Agentic tools like Task/dispatch (+0.1)
• Opus/Sonnet model (+0.1/+0.05)
• Response length (+0.1)

Related Skills

• hanzo/hanzo-aci.md - Agent Computer Interface (uses extraction for document conversion)
• hanzo/hanzo-guard.md - LLM I/O sanitization (powers the sanitize feature)

---

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: extraction, pdf, web, sanitization, llm, training-data, rust Prerequisites: Rust toolchain, Tokio runtime