training-free-grpo

Training-Free GRPO (Group Relative Policy Optimization) is Zoo Gym's default RLHF method that eliminates value networks entirely, achieving 40-60% memory reduction and 2× speedup compared to PPO.

Training-Free GRPO - RLHF Without Value Networks

Category: Zoo Gym Training Methods Skill Level: Advanced Prerequisites: Understanding of RLHF, PPO, policy optimization, reward modeling Related Skills: bitdelta.md, deltasoup.md, ../hanzo/hanzo-gym.md Zoo Gym Default: This is the default RLHF approach used in Zoo Gym

Overview

Training-Free GRPO (Group Relative Policy Optimization) is Zoo Gym's default RLHF method that eliminates value networks entirely, achieving 40-60% memory reduction and 2× speedup compared to PPO. It's based on DeepSeek's GRPO innovation and distinct from BitDelta/DeltaSoup which focus on per-user personalization and community learning.

Core Innovation: Group sampling + relative advantage estimation within groups = no value network needed = massive memory savings + faster training.

Why Training-Free GRPO?

The Problem: Traditional RLHF (PPO) is Expensive

PPO Requirements:

• Policy model: The LLM being trained (~10GB for 4B model)
• Value network: Separate network to estimate value function (~10GB)
• Reference model: Frozen copy for KL penalty (~10GB)
• Total: 30GB+ GPU memory for 4B model

PPO Training:

• Compute advantage: A = Q(s,a) - V(s)
• Requires value network V(s) trained simultaneously
• Value network training adds 40-60% overhead
• Memory bottleneck limits batch sizes

The Training-Free GRPO Solution

GRPO Requirements:

• Policy model: The LLM being trained (~10GB)
• Reference model: Frozen copy for KL penalty (~10GB)
• Value network: NONE ❌
• Total: 20GB GPU memory (40% less than PPO)

GRPO Training:

• Group sampling: Generate K responses per query
• Relative advantage: A = R_i - mean(R_group)
• No value network needed!
• 40-60% memory savings, 2× speedup

How Training-Free GRPO Works

1. Group Sampling

Generate multiple responses (K=5-8) for each query:

# Traditional PPO: 1 response per query
query = "Solve: 2x + 3 = 11"
response = model.generate(query)
reward = reward_model(query, response)

# Training-Free GRPO: K responses per query
K = 5
responses = [model.generate(query) for _ in range(K)]
rewards = [reward_model(query, r) for r in responses]

# Group:
# responses[0]: "x = 4" (correct) → reward = 1.0
# responses[1]: "x = 3" (wrong) → reward = 0.0
# responses[2]: "x = 4" (correct) → reward = 1.0
# responses[3]: "x = 5" (wrong) → reward = 0.0
# responses[4]: "x = 4" (correct) → reward = 1.0

2. Relative Advantage Estimation

Compute advantages relative to group mean:

# PPO: Absolute advantage (requires value network)
# A = Q(s,a) - V(s)
# V(s) = value_network(state) ← REQUIRES VALUE NETWORK

# GRPO: Relative advantage (no value network)
mean_reward = sum(rewards) / K  # 0.6
advantages = [r - mean_reward for r in rewards]

# Advantages:
# [1.0 - 0.6 = 0.4]  # Good response
# [0.0 - 0.6 = -0.6] # Bad response
# [1.0 - 0.6 = 0.4]  # Good response
# [0.0 - 0.6 = -0.6] # Bad response
# [1.0 - 0.6 = 0.4]  # Good response

3. Policy Optimization

Update policy using relative advantages:

# Compute policy loss (same as PPO, but advantages are relative)
for response, advantage in zip(responses, advantages):
    # Log probability under current policy
    log_prob = model.log_prob(query, response)

    # Log probability under reference policy (for KL penalty)
    ref_log_prob = ref_model.log_prob(query, response)

    # Ratio
    ratio = torch.exp(log_prob - ref_log_prob)

    # Clipped surrogate loss
    loss = -torch.min(
        ratio * advantage,
        torch.clamp(ratio, 1 - clip_range, 1 + clip_range) * advantage
    )

    # Backprop and update
    loss.backward()
    optimizer.step()

4. Why This Works

Key insight: Relative advantages are sufficient for policy optimization

# PPO (absolute advantage):
# A = Q(s,a) - V(s)
# "How much better is this action than average?"

# GRPO (relative advantage):
# A = R(s,a) - mean(R_group)
# "How much better is this response than other responses in the group?"

# Both capture the same signal!
# GRPO doesn't need V(s) because it's comparing within groups

Memory Comparison

| Component | PPO | GRPO | Savings | |-----------|-----|------|---------| | Policy Model | 10GB | 10GB | 0% | | Value Network | 10GB | 0GB | 100% | | Reference Model | 10GB | 10GB | 0% | | Total | 30GB | 20GB | 40% |

Training Speed Comparison (Qwen3-4B on 4× A100)

| Method | Training Time | Tokens/sec | Memory | |--------|---------------|------------|--------| | Full Fine-Tune | 12 hours | 500 | 40GB | | PPO | 8 hours | 800 | 30GB | | GRPO | 4 hours | 1600 | 20GB | | LoRA | 2 hours | 2000 | 15GB |

Note: LoRA is faster but doesn't do RLHF (only supervised fine-tuning).

Cost Savings: Training-Free GRPO vs Traditional Methods

Training Cost Comparison

| Method | Model | Training Samples | Training Cost | Performance (AIME24) | |--------|-------|-----------------|---------------|---------------------| | Training-Free GRPO | DeepSeek-V3.1-Terminus (671B) | 100 | ~$18 | 82.7% | | ReTool (Standard RL) | Qwen2.5-32B-Instruct | 1000s | ~$10,000 | 67.0% | | LoRA Fine-Tuning | Qwen2.5-32B-Instruct | 1000s | ~$20,000 | ~60-70% |

Key Findings:

• 500× cost reduction: $18 vs $10,000+ for superior performance
• 100× fewer samples: 100 vs 1000s of training examples needed
• Better results: 82.7% vs 67.0% on AIME24 benchmark
• No parameter updates: Zero gradient computation, instant deployment
• Larger model advantage: 671B frozen model outperforms fine-tuned 32B

Infrastructure Cost Breakdown

Traditional LoRA/RL Training (Example: ReTool on Qwen2.5-32B):

Training Cost:
- GPU hours: 20,000 × $0.5/hour = $10,000
- Data collection: $2,000-5,000
- Total: $12,000-15,000

Deployment Cost (Dedicated GPU):
- 4× GPUs at $0.5/GPU-hour = $2/hour = $1,440/month
- Inference: ~$0.005 per problem (400 problems/hour)
- Fixed infrastructure: Always running, even with low usage

Result: $10K+ training + $1.4K/month deployment

Training-Free GRPO (Example: DeepSeek-V3.1-Terminus):

Training Cost:
- API calls: 38M input tokens + 6.6M output tokens
- With cache hit pricing: ~$18 for 100 samples
- 3 training steps over 6 hours
- Total: $18

Deployment Cost (Pay-as-you-go API):
- ~$0.02 per problem (60K input + 8K output tokens)
- No fixed infrastructure
- Only pay for actual usage
- Automatic scaling, no GPU management

Result: $18 training + pay-per-use inference

Real-World Cost Example

Scenario: Medical diagnosis assistant (low-traffic, specialized domain)

Traditional LoRA Fine-Tuning:

• Training: $15,000 (data collection + GPU time)
• Deployment: $1,440/month (dedicated GPU)
• First year: $15,000 + $17,280 = $32,280
• Rigid specialization: Cannot adapt to new medical domains without retraining

Training-Free GRPO:

• Training: $50 (200 samples across 3 medical subdomains)
• Deployment: $200/month (100 diagnoses/day @ $0.02 each, 100 days/month)
• First year: $50 + $2,400 = $2,450
• Flexible: Switch medical domains by plugging in different experiences
• Savings: $29,830 (92% cost reduction)

Installation

# Zoo Gym includes Training-Free GRPO by default
git clone https://github.com/zooai/gym.git
cd gym
pip install -e .

# Or via pip
pip install zoo-gym

Usage Examples

Basic Training-Free GRPO

from zoo.gym import GRPOTrainer, GRPOConfig

# Configure GRPO
config = GRPOConfig(
    group_size=5,              # 5 rollouts per query
    max_epochs=3,              # 3 training epochs
    beta=0.1,                  # KL penalty coefficient
    clip_range=0.2,            # PPO-style clipping
    normalize_advantages=True, # Normalize advantages
    temperature_train=0.7,     # Training temperature
    temperature_eval=0.3       # Evaluation temperature
)

# Create trainer (no value network!)
trainer = GRPOTrainer(
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    config=config,
    reward_model=reward_fn
)

# Train with minimal samples
trainer.train(
    train_dataset=train_dataset,  # Only 100-1000 samples needed
    eval_dataset=eval_dataset
)

# Save trained model
trainer.save_model("./trained_model")

Training-Free GRPO with Experience Injection

from zoo.gym import TrainingFreeGRPO, GRPOConfig

# Configure Training-Free GRPO
config = GRPOConfig(
    group_size=5,
    max_epochs=3,
    temperature_train=0.7,
    temperature_eval=0.3
)

# Create trainer (frozen base model)
trainer = TrainingFreeGRPO(
    base_model="deepseek/v3.1-terminus",  # Frozen 671B model
    config=config
)

# Train with minimal samples (100 is enough!)
experiences = trainer.train(
    dataset=math_problems[:100],
    reward_model=reward_fn
)

# Deploy instantly - no parameter updates needed
response = trainer.infer(
    query="Solve: 2x + 3 = 11",
    experiences=experiences  # Plug in learned experiences
)
print(response)  # "x = 4"

CLI Training with GRPO

# Train with Training-Free GRPO (default in Zoo Gym)
llamafactory-cli train \
    --stage grpo \
    --do_train \
    --model_name_or_path Qwen/Qwen3-4B \
    --dataset your_preference_dataset \
    --template qwen3 \
    --finetuning_type lora \
    --output_dir ./model-grpo \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --num_train_epochs 1 \
    --learning_rate 1e-5 \
    --grpo_group_size 8 \
    --grpo_beta 0.1 \
    --grpo_clip_range 0.2

Custom Reward Function

def custom_reward_fn(query: str, response: str) -> float:
    """Custom reward function for math problems"""
    # Parse expected answer
    expected = parse_answer(query)

    # Parse model answer
    predicted = parse_answer(response)

    # Binary reward
    if predicted == expected:
        return 1.0
    else:
        return 0.0

# Use custom reward
trainer = GRPOTrainer(
    model=model,
    ref_model=ref_model,
    config=config,
    reward_model=custom_reward_fn  # Custom reward
)

Advanced Features

1. Group Size Tuning

# Larger groups = more stable but slower
config = GRPOConfig(group_size=10)  # 10 responses per query

# Smaller groups = faster but less stable
config = GRPOConfig(group_size=3)  # 3 responses per query

# Recommended: 5-8 for most tasks
config = GRPOConfig(group_size=5)  # Default

2. Temperature Scheduling

# Use higher temperature during training (exploration)
# Use lower temperature during evaluation (exploitation)
config = GRPOConfig(
    temperature_train=0.9,  # High exploration
    temperature_eval=0.1    # Low exploitation
)

3. KL Penalty Tuning

# Beta controls KL penalty (how much to stay close to reference model)
config = GRPOConfig(
    beta=0.01  # Weak penalty, more exploration
)

config = GRPOConfig(
    beta=0.5   # Strong penalty, stay close to reference
)

4. Integration with BitDelta

from zoo.gym import GRPOTrainer, BitDeltaConfig

# Train with GRPO + BitDelta compression
bitdelta_config = BitDeltaConfig(
    bits=1,
    group_size=128,
    safety_threshold=0.6
)

grpo_config = GRPOConfig(group_size=5, max_epochs=3)

# Train (GRPO) + Compress (BitDelta) + Serve
trainer = GRPOTrainer(model, ref_model, grpo_config)
trainer.train(train_dataset)
trainer.compress_deltas(bitdelta_config)  # 10× compression
trainer.save_model("./compressed_model")

Performance Benchmarks

AIME24 Math Benchmark

| Method | Model | Training Samples | Accuracy | Cost | |--------|-------|-----------------|----------|------| | Training-Free GRPO | DeepSeek-V3.1 (671B) | 100 | 82.7% | $18 | | ReTool (Standard RL) | Qwen2.5 (32B) | 1000s | 67.0% | $10,000 | | LoRA Fine-Tuning | Qwen2.5 (32B) | 1000s | ~65% | $20,000 | | Base Model (zero-shot) | Qwen2.5 (32B) | 0 | 45% | $0 |

Memory Efficiency (Training Qwen3-4B)

| Method | GPU Memory | Max Batch Size | |--------|------------|----------------| | Full Fine-Tune | 40GB | 1 | | PPO (with value network) | 30GB | 2 | | GRPO (no value network) | 20GB | 4 | | LoRA | 15GB | 8 |

Training Speed (4× A100 GPUs)

| Method | Tokens/sec | Training Time | Cost | |--------|------------|---------------|------| | Full Fine-Tune | 500 | 12 hours | $240 | | PPO | 800 | 8 hours | $160 | | GRPO | 1600 | 4 hours | $80 | | LoRA | 2000 | 2 hours | $40 |

GRPO vs PPO vs LoRA vs BitDelta

| Method | Purpose | Memory | Speed | Use Case | |--------|---------|--------|-------|----------| | Training-Free GRPO | RLHF without value network | -40% vs PPO | 2× vs PPO | Default for RLHF in Zoo Gym | | PPO | Traditional RLHF | Baseline | Baseline | Research, legacy systems | | LoRA | Supervised fine-tuning | -60% vs full | 4× vs full | Instruction tuning (not RLHF) | | BitDelta | Per-user personalization | 10× vs LoRA | Similar | Millions of user variants | | DeltaSoup | Community learning | Efficient | Async | Aggregate user improvements |

Key Distinctions:

• Training-Free GRPO: For RLHF (no value network, default in Zoo Gym)
• PPO: For RLHF (with value network, traditional approach)
• LoRA: For supervised fine-tuning (not RLHF)
• BitDelta: For per-user personalization (works on deltas, not RLHF)
• DeltaSoup: For community aggregation (Byzantine-robust, not RLHF)

Integration with Hanzo Ecosystem

Python SDK

from hanzo import Hanzo
from zoo.gym import GRPOConfig

hanzo = Hanzo(inference_mode='local')

# Train with GRPO
hanzo.train_grpo(
    base_model="qwen3-4b",
    dataset=preference_dataset,
    config=GRPOConfig(group_size=5, max_epochs=3)
)

# Serve trained model
response = hanzo.chat.completions.create(
    model="qwen3-4b-grpo",
    messages=[{"role": "user", "content": "Hello!"}]
)

Go SDK

package main

import (
    "context"
    "github.com/hanzoai/go-sdk"
    "github.com/hanzoai/go-sdk/option"
)

func main() {
    client := hanzoai.NewClient(
        option.WithInferenceMode("local"),
    )

    // Train with GRPO
    job, _ := client.Training.GRPO.Create(
        context.Background(),
        hanzoai.GRPOTrainingParams{
            BaseModel: hanzoai.F("qwen3-4b"),
            Dataset: hanzoai.F(preference_dataset),
            GroupSize: hanzoai.F(5),
            MaxEpochs: hanzoai.F(3),
        },
    )

    println("Training job:", job.ID)
}

Hanzo Node

# Start Hanzo Node with GRPO support
hanzo-node start --enable-grpo

# Train with GRPO
hanzo-node train grpo \
    --base-model qwen3-4b \
    --dataset ./preference_data.jsonl \
    --group-size 5 \
    --max-epochs 3 \
    --output ./trained_model

# Serve trained model
hanzo-node serve --model ./trained_model

Hanzo Engine

Training-Free GRPO is optimized in Hanzo Engine (Rust):

• Native Rust implementation
• Zero-copy experience injection
• Memory-efficient group sampling
• Multi-query parallel GRPO

Hanzo Dev

# Train with CLI
dev "train GRPO on math dataset with 100 samples, group size 5"

# Export trained model
dev "export GRPO model to GGUF format"

# Deploy to local node
dev "deploy GRPO model to hanzo node"

Hanzo Desktop

• Train GRPO on MacBook (M1/M2/M3)
• Mine $AI tokens while training
• ZenLM models (zen-eco, zen-agent) with GRPO
• Local RLHF for privacy

Best Practices

1. Group Size Selection

# Small datasets (< 100 samples): group_size=3-5
# Medium datasets (100-1000): group_size=5-8 (default)
# Large datasets (> 1000): group_size=8-10

2. Temperature Tuning

# Classification tasks: temperature_train=0.5, temperature_eval=0.1
# Generation tasks: temperature_train=0.9, temperature_eval=0.7
# Math/reasoning: temperature_train=0.7, temperature_eval=0.3 (default)

3. Sample Efficiency

# Training-Free GRPO works with 100-1000 samples
# More samples = better but diminishing returns
# 100 samples: ~80% of max performance
# 1000 samples: ~95% of max performance

Limitations and Considerations

1. Group Size Trade-off

• Larger groups = more stable but slower
• Smaller groups = faster but less stable
• Recommended: 5-8 for most tasks

2. Sample Efficiency

• Training-Free GRPO needs 100-1000 samples
• Still orders of magnitude fewer than PPO/LoRA
• Quality > Quantity for GRPO

3. Base Model Quality

• Training-Free GRPO relies on frozen base model
• Larger base models generally work better
• Example: 671B DeepSeek outperforms fine-tuned 32B

Related Skills

• bitdelta.md - Per-user personalization (10× compression, distinct from GRPO)
• deltasoup.md - Community-driven learning (Byzantine-robust aggregation)
• ../hanzo/hanzo-gym.md - Comprehensive Zoo Gym training guide
• ../hanzo/python-sdk.md - Python SDK integration
• ../hanzo/go-sdk.md - Go SDK integration
• ../hanzo/hanzo-node.md - Local inference infrastructure
• ../hanzo/hanzo-engine.md - Native Rust inference engine

Additional Resources

• GitHub: https://github.com/zooai/gym
• Paper: https://arxiv.org/abs/2502.01155 (DeepSeek GRPO)
• Zoo Labs Foundation: https://zoo.ngo (501(c)(3) wildlife conservation)

---

Remember: Training-Free GRPO is Zoo Gym's default RLHF method with 40% memory reduction, 2× speedup, and 500× cost savings compared to traditional RL - perfect for RLHF with minimal samples and no value network overhead.