dspy-evaluation

Evaluating DSPy programs with metrics, Evaluate class, and performance analysis

DSPy Evaluation

Scope: Metrics, Evaluate class, performance measurement, A/B testing, debugging Lines: ~380 Last Updated: 2025-10-25

When to Use This Skill

Activate this skill when:

• Measuring DSPy program accuracy and performance
• Comparing different modules or optimizations
• Setting up automated evaluation pipelines
• Debugging module failures
• Running A/B tests on prompt variations
• Validating program quality before production

Core Concepts

Evaluation in DSPy

Purpose: Measure how well your program performs

• Quantitative: Accuracy, F1, BLEU scores
• Qualitative: Inspect specific predictions
• Comparative: Before vs after optimization
• Continuous: Monitor production performance

Key insight: Good evaluation drives good optimization

Metrics

Definition: Functions that score predictions

Types:

1. Binary: True/False (correct/incorrect)
2. Continuous: 0.0 to 1.0 (partial credit)
3. Multi-metric: Combine multiple factors

Requirements:

def metric(example, pred, trace=None) -> bool | float:
    """
    Args:
        example: Ground truth example with .answer, etc.
        pred: Prediction object with .answer, .reasoning, etc.
        trace: Optional execution trace

    Returns:
        bool: True if correct, False otherwise
        OR
        float: Score between 0.0 and 1.0
    """
    pass

Evaluate Class

Purpose: Run systematic evaluation on datasets

Features:

• Parallel execution (multi-threading)
• Progress tracking
• Error handling
• Statistical aggregation
• Display modes (table, summary)

---

Patterns

Pattern 1: Simple Accuracy Metric

import dspy

# Binary metric: exact match
def exact_match(example, pred, trace=None):
    """Return True if prediction exactly matches answer."""
    return example.answer.lower().strip() == pred.answer.lower().strip()

# Usage
program = dspy.Predict("question -> answer")

# Manual evaluation
result = program(question="What is 2+2?")
example = dspy.Example(question="What is 2+2?", answer="4")

is_correct = exact_match(example, result)
print(f"Correct: {is_correct}")

When to use:

• Classification tasks
• QA with definite answers
• Quick validation

Pattern 2: Partial Credit Metric

import dspy

def fuzzy_match(example, pred, trace=None):
    """Return score based on answer overlap."""
    answer = pred.answer.lower()
    gold = example.answer.lower()

    # Exact match
    if answer == gold:
        return 1.0

    # Partial match: gold answer appears in prediction
    if gold in answer:
        return 0.7

    # Weak match: any word overlap
    answer_words = set(answer.split())
    gold_words = set(gold.split())
    overlap = len(answer_words & gold_words)

    if overlap > 0:
        return 0.3 * (overlap / len(gold_words))

    return 0.0

# Use with optimizer
optimizer = dspy.BootstrapFewShot(metric=fuzzy_match, metric_threshold=0.7)

When to use:

• Open-ended text generation
• Summarization
• Tasks where partial correctness matters

Pattern 3: Using Evaluate Class

import dspy
from dspy.evaluate import Evaluate

# Define program
program = dspy.ChainOfThought("question -> answer")

# Prepare test set
testset = [
    dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"),
    dspy.Example(question="Capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="Who wrote Hamlet?", answer="Shakespeare").with_inputs("question"),
]

# Define metric
def accuracy(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

# Create evaluator
evaluator = Evaluate(
    devset=testset,
    metric=accuracy,
    num_threads=4,  # Parallel evaluation
    display_progress=True,
    display_table=5,  # Show first 5 results
)

# Run evaluation
score = evaluator(program)
print(f"Accuracy: {score:.1%}")

Output example:

Question                    | Predicted Answer | Correct
----------------------------|------------------|--------
What is 2+2?               | 4                | ✓
Capital of France?         | Paris            | ✓
Who wrote Hamlet?          | William Shakespeare | ✓

Accuracy: 100.0%

Pattern 4: Multi-Metric Evaluation

import dspy
from dspy.evaluate import Evaluate

# Multiple metrics for comprehensive evaluation
def accuracy(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

def has_reasoning(example, pred, trace=None):
    """Check if reasoning is provided and non-trivial."""
    if not hasattr(pred, 'reasoning'):
        return 0.0
    return 1.0 if len(pred.reasoning.split()) > 5 else 0.0

def confidence_calibration(example, pred, trace=None):
    """Check if confidence matches correctness."""
    is_correct = accuracy(example, pred)
    if hasattr(pred, 'confidence'):
        conf = float(pred.confidence) if pred.confidence else 0.5
        # High confidence + correct = good
        # Low confidence + incorrect = good
        # Mismatch = bad
        if is_correct and conf > 0.7:
            return 1.0
        if not is_correct and conf < 0.3:
            return 1.0
        return 0.5
    return 0.5

# Evaluate each metric separately
eval_accuracy = Evaluate(devset=testset, metric=accuracy)
eval_reasoning = Evaluate(devset=testset, metric=has_reasoning)
eval_calibration = Evaluate(devset=testset, metric=confidence_calibration)

# Run all evaluations
acc_score = eval_accuracy(program)
reasoning_score = eval_reasoning(program)
calib_score = eval_calibration(program)

print(f"Accuracy: {acc_score:.1%}")
print(f"Reasoning Quality: {reasoning_score:.1%}")
print(f"Calibration: {calib_score:.1%}")

# Composite score
overall = 0.6 * acc_score + 0.2 * reasoning_score + 0.2 * calib_score
print(f"Overall Score: {overall:.1%}")

When to use:

• Production systems with multiple quality factors
• Need comprehensive quality assessment
• Balancing multiple objectives

Pattern 5: A/B Testing Programs

import dspy
from dspy.evaluate import Evaluate

# Program A: Simple Predict
program_a = dspy.Predict("question -> answer")

# Program B: ChainOfThought
program_b = dspy.ChainOfThought("question -> answer")

# Prepare test set
testset = [...]

# Define metric
def accuracy(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

# Create evaluator
evaluator = Evaluate(devset=testset, metric=accuracy, num_threads=4)

# Evaluate both programs
score_a = evaluator(program_a)
score_b = evaluator(program_b)

# Compare results
print(f"Program A (Predict): {score_a:.1%}")
print(f"Program B (ChainOfThought): {score_b:.1%}")
print(f"Winner: {'B' if score_b > score_a else 'A'}")
print(f"Improvement: {abs(score_b - score_a):.1%}")

When to use:

• Comparing module types (Predict vs ChainOfThought)
• Comparing optimizations (before vs after)
• Choosing between LMs (GPT-4 vs Claude)

Pattern 6: Error Analysis

import dspy
from dspy.evaluate import Evaluate

program = dspy.ChainOfThought("question -> answer")
testset = [...]

# Track errors for analysis
errors = []

def accuracy_with_logging(example, pred, trace=None):
    """Metric that logs errors for analysis."""
    correct = example.answer.lower() in pred.answer.lower()

    if not correct:
        errors.append({
            'question': example.question,
            'expected': example.answer,
            'predicted': pred.answer,
            'reasoning': getattr(pred, 'reasoning', 'N/A'),
        })

    return correct

# Run evaluation
evaluator = Evaluate(devset=testset, metric=accuracy_with_logging)
score = evaluator(program)

# Analyze errors
print(f"\nAccuracy: {score:.1%}")
print(f"\nErrors ({len(errors)}):")
for i, err in enumerate(errors[:5], 1):  # Show first 5 errors
    print(f"\n{i}. Question: {err['question']}")
    print(f"   Expected: {err['expected']}")
    print(f"   Predicted: {err['predicted']}")
    print(f"   Reasoning: {err['reasoning']}")

Benefits:

• Identify common failure patterns
• Guide optimization efforts
• Improve training data

Pattern 7: LM-as-Judge Metric

import dspy

# Use LM to evaluate predictions
judge = dspy.ChainOfThought("question, answer, predicted_answer -> score: float, explanation")

def llm_judge_metric(example, pred, trace=None):
    """Use language model to evaluate answer quality."""
    result = judge(
        question=example.question,
        answer=example.answer,
        predicted_answer=pred.answer
    )

    try:
        score = float(result.score)
        return max(0.0, min(1.0, score))  # Clamp to [0, 1]
    except:
        # Fallback to simple match if LM fails
        return 1.0 if example.answer.lower() in pred.answer.lower() else 0.0

# Use LM-as-judge for nuanced evaluation
evaluator = Evaluate(devset=testset, metric=llm_judge_metric, num_threads=1)  # Sequential for LM calls
score = evaluator(program)

When to use:

• Open-ended generation tasks
• Nuanced quality assessment
• When exact match is too strict

Caution:

• Slower (requires LM calls)
• More expensive
• LM judge can be wrong

Pattern 8: Production Monitoring

import dspy
from dspy.evaluate import Evaluate
import time

# Production program
program = dspy.ChainOfThought("question -> answer")

# Metrics for monitoring
def response_quality(example, pred, trace=None):
    """Monitor prediction quality."""
    return example.answer.lower() in pred.answer.lower()

def response_time(example, pred, trace=None):
    """Monitor latency (tracked separately)."""
    # This is illustrative; actual timing would be external
    return 1.0  # Placeholder

# Sample production traffic for evaluation
sample_size = 100
production_samples = [...]  # Collect from production logs

# Periodic evaluation
evaluator = Evaluate(
    devset=production_samples,
    metric=response_quality,
    num_threads=4,
)

# Run evaluation
score = evaluator(program)

# Alert if quality drops
if score < 0.85:
    print(f"⚠️ Quality degradation detected: {score:.1%}")
    # Send alert, trigger investigation
else:
    print(f"✓ Quality maintained: {score:.1%}")

When to use:

• Production monitoring
• Detecting model drift
• Continuous quality assurance

---

Quick Reference

Metric Function Template

def my_metric(example, pred, trace=None):
    """
    Evaluate prediction quality.

    Args:
        example: dspy.Example with ground truth
        pred: Prediction object from module
        trace: Optional execution trace

    Returns:
        bool: True/False for binary metrics
        float: 0.0-1.0 for continuous metrics
    """
    # Implement evaluation logic
    return score

Common Metrics

# Exact match
def exact_match(ex, pred, trace=None):
    return ex.answer == pred.answer

# Contains
def contains(ex, pred, trace=None):
    return ex.answer.lower() in pred.answer.lower()

# Fuzzy match
def fuzzy(ex, pred, trace=None):
    from difflib import SequenceMatcher
    ratio = SequenceMatcher(None, ex.answer, pred.answer).ratio()
    return ratio

# F1 score
def f1_score(ex, pred, trace=None):
    ex_words = set(ex.answer.lower().split())
    pred_words = set(pred.answer.lower().split())
    overlap = len(ex_words & pred_words)
    precision = overlap / len(pred_words) if pred_words else 0
    recall = overlap / len(ex_words) if ex_words else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Evaluation Checklist

✅ DO: Use separate test set (never seen during training)
✅ DO: Evaluate baseline before optimization
✅ DO: Use multiple metrics for comprehensive view
✅ DO: Run error analysis on failures
✅ DO: Report confidence intervals if dataset is small

❌ DON'T: Evaluate on training data (overfitting!)
❌ DON'T: Rely on single metric
❌ DON'T: Ignore errors (analyze them!)
❌ DON'T: Skip baseline comparison

---

Anti-Patterns

❌ Evaluating on training data: Overfitting, invalid results

# Bad
evaluator = Evaluate(devset=trainset, metric=accuracy)  # WRONG!

✅ Use held-out test set:

# Good
evaluator = Evaluate(devset=testset, metric=accuracy)

❌ Single metric only: Misses important quality factors

# Bad - only accuracy
score = evaluate(program, accuracy)

✅ Use multiple metrics:

# Good - comprehensive evaluation
acc = evaluate(program, accuracy)
f1 = evaluate(program, f1_score)
reasoning = evaluate(program, has_reasoning)

❌ No error analysis: Don't learn from failures

# Bad - just get score, don't investigate
score = evaluator(program)
print(score)  # Done, but learned nothing

✅ Analyze errors:

# Good - investigate failures
score = evaluator(program)
errors = [ex for ex in testset if not metric(ex, program(ex.question))]
analyze_common_patterns(errors)

---

Related Skills

• dspy-optimizers.md - Using metrics for optimization
• dspy-modules.md - Programs to evaluate
• dspy-assertions.md - Runtime validation
• llm-dataset-preparation.md - Creating evaluation datasets

---

Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)