Comprehensive guide to LLM-as-judge evaluation patterns including Prometheus 2 models, G-Eval framework, pairwise/pointwise/reference-guided methods, bias mitigation, and uncertainty quantification
Last Updated: 2025-10-26
Use LLM-as-judge evaluation when:
Anti-pattern: Using LLM-as-judge for tasks with objective ground truth (use exact match, BLEU, etc. instead). Never rely solely on LLM judges without validation against human judgments.
1. Pointwise Evaluation
2. Pairwise Evaluation
3. Reference-Guided Evaluation
Prometheus 2 (Fine-tuned Evaluators)
G-Eval (GPT-based)
GPT-4 / Claude 3 Opus (General-purpose)
Llama-3-70B-Instruct / Mixtral-8x7B
Position Bias: Judge favors first/second position in pairwise comparisons
Verbosity Bias: Judge favors longer responses
Self-Enhancement Bias: Judge favors outputs from same model family
Scale Inconsistency: Judge uses rating scale inconsistently
Self-Consistency: Sample multiple judgments and measure agreement Confidence Scoring: Ask judge to provide confidence level Multi-Judge Ensemble: Use multiple judges and measure consensus
When to use: Fine-tuned evaluator with explicit rubrics
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
class PrometheusEvaluator:
"""Prometheus 2 evaluator for pointwise assessment."""
def __init__(self, model_name="prometheus-eval/prometheus-7b-v2.0"):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
)
def evaluate(
self,
instruction: str,
response: str,
reference_answer: str = None,
rubric: str = None,
) -> dict:
"""
Evaluate response using Prometheus format.
Args:
instruction: The task instruction
response: The response to evaluate
reference_answer: Optional reference answer
rubric: Evaluation rubric (5-point scale by default)
Returns:
Dict with score and feedback
"""
# Default rubric
if rubric is None:
rubric = """
1: The response is completely incorrect or irrelevant.
2: The response has major errors or misses key points.
3: The response is acceptable but has some mistakes or lacks detail.
4: The response is good with minor issues.
5: The response is excellent, accurate, and comprehensive.
"""
# Prometheus prompt format
if reference_answer:
prompt = f"""###Task Description:
An instruction (might include an Input inside it), a response to evaluate, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"
###The instruction to evaluate:
{instruction}
###Response to evaluate:
{response}
###Reference Answer (Score 5):
{reference_answer}
###Score Rubrics:
{rubric}
###Feedback:"""
else:
prompt = f"""###Task Description:
An instruction (might include an Input inside it), a response to evaluate, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"
###The instruction to evaluate:
{instruction}
###Response to evaluate:
{response}
###Score Rubrics:
{rubric}
###Feedback:"""
# Generate evaluation
inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
with torch.no_grad():
outputs = self.model.generate(
**inputs,
max_new_tokens=512,
temperature=0.0,
do_sample=False,
)
evaluation = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
# Extract score and feedback
feedback = evaluation.split("###Feedback:")[-1].strip()
# Parse score
import re
score_match = re.search(r'\[RESULT\]\s*(\d+)', feedback)
score = int(score_match.group(1)) if score_match else None
return {
"score": score,
"feedback": feedback.split("[RESULT]")[0].strip() if score else feedback,
"raw_output": evaluation,
}
# Usage
evaluator = PrometheusEvaluator()
result = evaluator.evaluate(
instruction="Explain photosynthesis in simple terms.",
response="Photosynthesis is the process where plants use sunlight to make food from water and carbon dioxide.",
reference_answer="Photosynthesis is the process by which plants convert light energy into chemical energy. Plants use sunlight, water, and carbon dioxide to produce glucose (sugar) and oxygen.",
)
print(f"Score: {result['score']}/5")
print(f"Feedback: {result['feedback']}")When to use: GPT-4 based evaluation with chain-of-thought
from openai import OpenAI
import json
class GEvalEvaluator:
"""G-Eval framework using GPT-4 with chain-of-thought."""
def __init__(self, model="gpt-4-turbo-preview"):
self.client = OpenAI()
self.model = model
def evaluate(
self,
task_description: str,
criteria: str,
response: str,
context: str = None,
min_score: int = 1,
max_score: int = 5,
) -> dict:
"""
Evaluate response using G-Eval with chain-of-thought.
Args:
task_description: What the task is
criteria: What to evaluate (coherence, fluency, etc.)
response: The response to evaluate
context: Optional context (e.g., prompt, reference)
min_score: Minimum score
max_score: Maximum score
Returns:
Dict with score, reasoning, and confidence
"""
# G-Eval prompt with chain-of-thought
prompt = f"""You will be given a task description and evaluation criteria. Your job is to evaluate the given response based on the criteria.
Task Description:
{task_description}
Evaluation Criteria:
{criteria}
{f"Context: {context}" if context else ""}
Response to Evaluate:
{response}
Please evaluate step-by-step:
1. First, identify the key aspects of the evaluation criteria
2. Analyze how well the response meets each aspect
3. Provide a final score from {min_score} to {max_score}
Your response MUST be valid JSON in this format:
{{
"reasoning": "Detailed step-by-step analysis",
"score": <integer from {min_score} to {max_score}>,
"confidence": <float from 0.0 to 1.0>
}}
"""
# Call GPT-4 with JSON mode
response = self.client.chat.completions.create(
model=self.model,
messages=[
{
"role": "system",
"content": "You are an expert evaluator. Provide thoughtful, unbiased assessments.",
},
{"role": "user", "content": prompt},
],
response_format={"type": "json_object"},
temperature=0.0,
)
result = json.loads(response.choices[0].message.content)
return result
def evaluate_with_uncertainty(
self,
task_description: str,
criteria: str,
response: str,
num_samples: int = 5,
) -> dict:
"""
Evaluate with uncertainty quantification via self-consistency.
Args:
task_description: Task description
criteria: Evaluation criteria
response: Response to evaluate
num_samples: Number of evaluations to sample
Returns:
Dict with mean score, std, and all samples
"""
scores = []
reasonings = []
for _ in range(num_samples):
result = self.evaluate(task_description, criteria, response)
scores.append(result["score"])
reasonings.append(result["reasoning"])
import numpy as np
return {
"mean_score": np.mean(scores),
"std_score": np.std(scores),
"min_score": np.min(scores),
"max_score": np.max(scores),
"all_scores": scores,
"sample_reasonings": reasonings,
"confidence": 1.0 - (np.std(scores) / np.mean(scores)) if np.mean(scores) > 0 else 0.0,
}
# Usage
evaluator = GEvalEvaluator()
# Single evaluation
result = evaluator.evaluate(
task_description="Summarize a scientific article for general audience",
criteria="Evaluate based on: 1) Accuracy of scientific content, 2) Clarity for non-experts, 3) Conciseness",
response="Researchers found that exercise increases brain health by promoting neurogenesis. Regular physical activity can improve memory and reduce dementia risk.",
context="Original article discusses BDNF protein and hippocampal neurogenesis in exercise studies.",
)
print(f"Score: {result['score']}/5")
print(f"Reasoning: {result['reasoning']}")
# Evaluation with uncertainty
uncertain_result = evaluator.evaluate_with_uncertainty(
task_description="Summarize a scientific article for general audience",
criteria="Evaluate based on accuracy, clarity, and conciseness",
response="Researchers found that exercise increases brain health...",
num_samples=5,
)
print(f"Mean score: {uncertain_result['mean_score']:.2f} ± {uncertain_result['std_score']:.2f}")
print(f"Confidence: {uncertain_result['confidence']:.2f}")When to use: A/B testing with position bias correction
from openai import OpenAI
import json
class PairwiseJudge:
"""Pairwise comparison with position bias mitigation."""
def __init__(self, model="gpt-4-turbo-preview"):
self.client = OpenAI()
self.model = model
def compare(
self,
instruction: str,
response_a: str,
response_b: str,
criteria: str = "overall quality",
) -> dict:
"""
Compare two responses with bias mitigation.
Args:
instruction: The task instruction
response_a: First response
response_b: Second response
criteria: What to compare on
Returns:
Dict with winner and reasoning
"""
# First comparison (A first, B second)
result_ab = self._single_comparison(
instruction, response_a, response_b, criteria, "A", "B"
)
# Second comparison (B first, A second) to mitigate position bias
result_ba = self._single_comparison(
instruction, response_b, response_a, criteria, "B", "A"
)
# Aggregate results
if result_ab["winner"] == result_ba["winner"]:
# Consistent winner
winner = result_ab["winner"]
confidence = "high"
elif result_ab["winner"] == "tie" or result_ba["winner"] == "tie":
# One or both say tie
winner = "tie"
confidence = "medium"
else:
# Disagreement - position bias detected
winner = "tie"
confidence = "low"
return {
"winner": winner,
"confidence": confidence,
"reasoning_ab": result_ab["reasoning"],
"reasoning_ba": result_ba["reasoning"],
"position_bias_detected": result_ab["winner"] != result_ba["winner"] and "tie" not in [result_ab["winner"], result_ba["winner"]],
}
def _single_comparison(
self,
instruction: str,
response_first: str,
response_second: str,
criteria: str,
first_label: str,
second_label: str,
) -> dict:
"""Single pairwise comparison."""
prompt = f"""Compare the following two responses to the given instruction based on {criteria}.
Instruction:
{instruction}
Response {first_label}:
{response_first}
Response {second_label}:
{response_second}
Which response is better? Provide your answer as JSON:
{{
"winner": "{first_label}" or "{second_label}" or "tie",
"reasoning": "Detailed explanation of why"
}}
"""
response = self.client.chat.completions.create(
model=self.model,
messages=[
{
"role": "system",
"content": "You are an objective evaluator. Provide fair, unbiased comparisons.",
},
{"role": "user", "content": prompt},
],
response_format={"type": "json_object"},
temperature=0.0,
)
result = json.loads(response.choices[0].message.content)
return result
# Usage
judge = PairwiseJudge()
result = judge.compare(
instruction="Write a haiku about spring",
response_a="Cherry blossoms bloom\nGentle breeze carries petals\nSpring has come again",
response_b="Flowers are blooming\nIt is springtime outside now\nNature is pretty",
criteria="poetic quality and imagery",
)
print(f"Winner: {result['winner']}")
print(f"Confidence: {result['confidence']}")
print(f"Position bias detected: {result['position_bias_detected']}")
print(f"Reasoning (A-B): {result['reasoning_ab']}")When to use: Assessing multiple quality aspects simultaneously
from typing import List, Dict
from openai import OpenAI
import json
class MultiDimensionalJudge:
"""Evaluate response across multiple dimensions."""
def __init__(self, model="gpt-4-turbo-preview"):
self.client = OpenAI()
self.model = model
def evaluate(
self,
instruction: str,
response: str,
dimensions: List[Dict[str, str]],
) -> dict:
"""
Evaluate response across multiple dimensions.
Args:
instruction: Task instruction
response: Response to evaluate
dimensions: List of {name, description, min_score, max_score} dicts
Returns:
Dict with scores per dimension and overall
"""
# Build dimension descriptions
dimension_descriptions = []
for dim in dimensions:
dimension_descriptions.append(
f"- {dim['name']}: {dim['description']} (scale: {dim.get('min_score', 1)}-{dim.get('max_score', 5)})"
)
prompt = f"""Evaluate the following response across multiple dimensions.
Instruction:
{instruction}
Response:
{response}
Evaluation Dimensions:
{chr(10).join(dimension_descriptions)}
For each dimension:
1. Provide a score within the specified range
2. Explain your reasoning
Return your evaluation as JSON:
{{
"dimensions": [
{{
"name": "dimension_name",
"score": <score>,
"reasoning": "explanation"
}},
...
],
"overall_score": <average score>,
"summary": "overall assessment"
}}
"""
response_eval = self.client.chat.completions.create(
model=self.model,
messages=[
{
"role": "system",
"content": "You are an expert evaluator. Assess each dimension independently.",
},
{"role": "user", "content": prompt},
],
response_format={"type": "json_object"},
temperature=0.0,
)
result = json.loads(response_eval.choices[0].message.content)
# Calculate weighted overall score if weights provided
if any("weight" in dim for dim in dimensions):
weighted_score = sum(
dim_result["score"] * next((d["weight"] for d in dimensions if d["name"] == dim_result["name"]), 1.0)
for dim_result in result["dimensions"]
) / sum(dim.get("weight", 1.0) for dim in dimensions)
result["weighted_overall_score"] = weighted_score
return result
# Usage
judge = MultiDimensionalJudge()
result = judge.evaluate(
instruction="Explain quantum entanglement to a high school student",
response="Quantum entanglement is when two particles are connected in a weird way. If you measure one particle, it instantly affects the other, even if they're far apart. Einstein called it 'spooky action at a distance.' It's like having two magic coins that always land on opposite sides.",
dimensions=[
{
"name": "accuracy",
"description": "Scientific accuracy of the explanation",
"min_score": 1,
"max_score": 5,
"weight": 2.0, # Most important
},
{
"name": "clarity",
"description": "How clear and understandable the explanation is",
"min_score": 1,
"max_score": 5,
"weight": 1.5,
},
{
"name": "engagement",
"description": "How engaging and interesting the explanation is",
"min_score": 1,
"max_score": 5,
"weight": 1.0,
},
{
"name": "appropriate_level",
"description": "Whether the explanation is appropriate for high school students",
"min_score": 1,
"max_score": 5,
"weight": 1.5,
},
],
)
print("Dimension scores:")
for dim in result["dimensions"]:
print(f" {dim['name']}: {dim['score']}/5 - {dim['reasoning']}")
print(f"\nOverall score: {result['overall_score']:.2f}")
print(f"Weighted score: {result.get('weighted_overall_score', 'N/A'):.2f}")
print(f"Summary: {result['summary']}")When to use: Validating LLM judge against human experts
import pandas as pd
from sklearn.metrics import cohen_kappa_score
from typing import List, Dict
import json
class JudgeValidator:
"""Validate LLM judge against human annotations."""
def __init__(self, llm_judge):
"""
Args:
llm_judge: Any judge class with evaluate() method
"""
self.llm_judge = llm_judge
def validate_against_humans(
self,
test_cases: List[Dict],
human_annotations: List[int],
) -> dict:
"""
Validate LLM judge against human expert annotations.
Args:
test_cases: List of {instruction, response} dicts
human_annotations: List of human scores (same order as test_cases)
Returns:
Validation metrics
"""
llm_scores = []
for case in test_cases:
result = self.llm_judge.evaluate(
instruction=case["instruction"],
response=case["response"],
)
llm_scores.append(result["score"])
# Calculate agreement metrics
from scipy.stats import pearsonr, spearmanr
# Pearson correlation (linear relationship)
pearson_corr, pearson_p = pearsonr(llm_scores, human_annotations)
# Spearman correlation (rank-order relationship)
spearman_corr, spearman_p = spearmanr(llm_scores, human_annotations)
# Cohen's Kappa (categorical agreement)
kappa = cohen_kappa_score(human_annotations, llm_scores)
# Mean Absolute Error
mae = sum(abs(h - l) for h, l in zip(human_annotations, llm_scores)) / len(human_annotations)
# Identify disagreements
disagreements = [
{
"case": test_cases[i],
"human_score": human_annotations[i],
"llm_score": llm_scores[i],
"difference": abs(human_annotations[i] - llm_scores[i]),
}
for i in range(len(test_cases))
if abs(human_annotations[i] - llm_scores[i]) >= 2 # Significant disagreement
]
return {
"pearson_correlation": pearson_corr,
"spearman_correlation": spearman_corr,
"cohens_kappa": kappa,
"mean_absolute_error": mae,
"num_significant_disagreements": len(disagreements),
"disagreement_rate": len(disagreements) / len(test_cases),
"sample_disagreements": disagreements[:5], # Top 5 for review
}
def calibrate_judge(
self,
calibration_set: List[Dict],
human_scores: List[int],
target_correlation: float = 0.8,
) -> str:
"""
Generate calibration recommendations based on validation.
Args:
calibration_set: Test cases for calibration
human_scores: Human expert scores
target_correlation: Desired correlation threshold
Returns:
Calibration recommendations
"""
validation = self.validate_against_humans(calibration_set, human_scores)
recommendations = []
if validation["pearson_correlation"] < target_correlation:
recommendations.append(
f"Low correlation ({validation['pearson_correlation']:.2f}). Consider:\n"
" - Using a more capable judge model (e.g., GPT-4 instead of GPT-3.5)\n"
" - Refining evaluation criteria/rubrics\n"
" - Adding more examples to the prompt"
)
if validation["mean_absolute_error"] > 1.0:
recommendations.append(
f"High MAE ({validation['mean_absolute_error']:.2f}). The judge is often off by >1 point.\n"
" - Review sample disagreements\n"
" - Clarify scoring rubric\n"
" - Use reference-guided evaluation if possible"
)
if validation["disagreement_rate"] > 0.2:
recommendations.append(
f"High disagreement rate ({validation['disagreement_rate']:.1%}).\n"
" - Review cases where judge disagrees significantly\n"
" - Add calibration examples to prompt\n"
" - Consider ensemble of multiple judges"
)
if not recommendations:
recommendations.append(
"Judge performance is acceptable. Continue monitoring on production data."
)
return "\n\n".join(recommendations)
# Usage
from geval_evaluator import GEvalEvaluator # From Pattern 2
judge = GEvalEvaluator()
validator = JudgeValidator(judge)
# Test cases with human expert scores
test_cases = [
{
"instruction": "Explain photosynthesis",
"response": "Photosynthesis is how plants make food using sunlight...",
},
# ... more cases
]
human_scores = [4, 5, 3, 2, 4, 5, 3, 4] # From expert annotations
# Validate
validation_results = validator.validate_against_humans(test_cases, human_scores)
print(f"Pearson correlation: {validation_results['pearson_correlation']:.3f}")
print(f"Spearman correlation: {validation_results['spearman_correlation']:.3f}")
print(f"Cohen's Kappa: {validation_results['cohens_kappa']:.3f}")
print(f"MAE: {validation_results['mean_absolute_error']:.2f}")
# Get calibration recommendations
recommendations = validator.calibrate_judge(test_cases, human_scores)
print(f"\nRecommendations:\n{recommendations}")from typing import List, Dict, Optional
from dataclasses import dataclass
import json
from datetime import datetime
@dataclass
class JudgmentResult:
"""Result from LLM judge."""
score: float
reasoning: str
confidence: float
judge_model: str
timestamp: str
metadata: Dict = None
class ProductionJudgePipeline:
"""Production-ready LLM-as-judge pipeline with logging and monitoring."""
def __init__(
self,
judge_model: str = "gpt-4-turbo-preview",
enable_uncertainty: bool = True,
enable_bias_mitigation: bool = True,
log_path: str = "judge_logs.jsonl",
):
self.judge = GEvalEvaluator(model=judge_model)
self.pairwise_judge = PairwiseJudge(model=judge_model)
self.enable_uncertainty = enable_uncertainty
self.enable_bias_mitigation = enable_bias_mitigation
self.log_path = log_path
def judge_single(
self,
task_description: str,
criteria: str,
response: str,
context: Optional[str] = None,
) -> JudgmentResult:
"""Judge single response with optional uncertainty quantification."""
if self.enable_uncertainty:
result = self.judge.evaluate_with_uncertainty(
task_description=task_description,
criteria=criteria,
response=response,
num_samples=3,
)
judgment = JudgmentResult(
score=result["mean_score"],
reasoning=result["sample_reasonings"][0], # First reasoning
confidence=result["confidence"],
judge_model=self.judge.model,
timestamp=datetime.now().isoformat(),
metadata={
"std_score": result["std_score"],
"all_scores": result["all_scores"],
},
)
else:
result = self.judge.evaluate(
task_description=task_description,
criteria=criteria,
response=response,
context=context,
)
judgment = JudgmentResult(
score=result["score"],
reasoning=result["reasoning"],
confidence=result.get("confidence", 1.0),
judge_model=self.judge.model,
timestamp=datetime.now().isoformat(),
)
# Log judgment
self._log_judgment(judgment, task_description, response)
return judgment
def judge_pairwise(
self,
instruction: str,
response_a: str,
response_b: str,
criteria: str = "overall quality",
) -> Dict:
"""Judge pairwise comparison with bias mitigation."""
result = self.pairwise_judge.compare(
instruction=instruction,
response_a=response_a,
response_b=response_b,
criteria=criteria,
)
# Log comparison
self._log_comparison(result, instruction, response_a, response_b)
return result
def batch_evaluate(
self,
examples: List[Dict],
task_description: str,
criteria: str,
) -> List[JudgmentResult]:
"""Evaluate batch of examples."""
results = []
for i, example in enumerate(examples):
judgment = self.judge_single(
task_description=task_description,
criteria=criteria,
response=example["response"],
context=example.get("context"),
)
results.append(judgment)
# Progress
if (i + 1) % 10 == 0:
print(f"Evaluated {i + 1}/{len(examples)} examples")
# Calculate statistics
scores = [r.score for r in results]
avg_score = sum(scores) / len(scores)
avg_confidence = sum(r.confidence for r in results) / len(results)
print(f"\nBatch statistics:")
print(f" Average score: {avg_score:.2f}")
print(f" Average confidence: {avg_confidence:.2f}")
print(f" Score range: {min(scores):.2f} - {max(scores):.2f}")
return results
def _log_judgment(self, judgment: JudgmentResult, task: str, response: str):
"""Log judgment to file."""
log_entry = {
"type": "single_judgment",
"task": task,
"response": response,
"score": judgment.score,
"confidence": judgment.confidence,
"judge_model": judgment.judge_model,
"timestamp": judgment.timestamp,
}
with open(self.log_path, "a") as f:
f.write(json.dumps(log_entry) + "\n")
def _log_comparison(self, result: Dict, instruction: str, response_a: str, response_b: str):
"""Log pairwise comparison."""
log_entry = {
"type": "pairwise_comparison",
"instruction": instruction,
"response_a": response_a,
"response_b": response_b,
"winner": result["winner"],
"confidence": result["confidence"],
"position_bias_detected": result["position_bias_detected"],
"timestamp": datetime.now().isoformat(),
}
with open(self.log_path, "a") as f:
f.write(json.dumps(log_entry) + "\n")
# Usage
pipeline = ProductionJudgePipeline(
judge_model="gpt-4-turbo-preview",
enable_uncertainty=True,
enable_bias_mitigation=True,
)
# Single judgment
judgment = pipeline.judge_single(
task_description="Summarize scientific article",
criteria="Accuracy, clarity, and conciseness",
response="Researchers found exercise promotes brain health...",
)
print(f"Score: {judgment.score:.2f} (confidence: {judgment.confidence:.2f})")
# Batch evaluation
examples = [
{"response": "Response 1..."},
{"response": "Response 2..."},
# ... more
]
batch_results = pipeline.batch_evaluate(
examples=examples,
task_description="Question answering",
criteria="Correctness and completeness",
)Wrong: Using judge for tasks with clear right/wrong answers
# BAD: LLM judge for exact match tasks
judgment = judge.evaluate(
task="What is 2+2?",
response="4",
) # Expensive and unnecessaryRight: Use deterministic metrics
# GOOD: Exact match for objective tasks
def evaluate_math(response, expected):
return {"score": 1.0 if response.strip() == expected.strip() else 0.0}Wrong: Single pairwise comparison
# BAD: Position bias not addressed
winner = judge.compare(response_a, response_b) # May favor first positionRight: Swap positions and aggregate
# GOOD: Mitigate position bias
result_ab = judge.compare(response_a, response_b)
result_ba = judge.compare(response_b, response_a)
if result_ab["winner"] == result_ba["winner"]:
winner = result_ab["winner"] # Consistent
else:
winner = "tie" # Disagreement = bias detectedWrong: Blind trust in LLM judge
# BAD: No validation
scores = [judge.evaluate(case) for case in test_cases]
# Are these scores reliable? Unknown!Right: Validate on calibration set
# GOOD: Validate against human experts
validator = JudgeValidator(judge)
validation = validator.validate_against_humans(test_cases, human_scores)
if validation["pearson_correlation"] < 0.7:
print("WARNING: Low correlation with humans. Review calibration.")llm-benchmarks-evaluation.md: Standard benchmarks for objective capability testingllm-evaluation-frameworks.md: Arize Phoenix, Braintrust for production evaluationrag-evaluation-metrics.md: RAGAS metrics combining LLM-as-judge with retrieval evaluationcustom-llm-evaluation.md: Domain-specific evaluation metrics and continuous evaluationdspy-evaluation.md: DSPy metric functions and prompt optimizationLLM-as-judge provides scalable, flexible evaluation for subjective quality metrics:
Key Takeaways:
Best Practices:
When to combine with other skills:
llm-benchmarks-evaluation.md for objective capability testing (MMLU, HumanEval)llm-evaluation-frameworks.md to integrate judges with Phoenix/Braintrust pipelinesrag-evaluation-metrics.md for RAG-specific metrics with LLM-as-judge componentscustom-llm-evaluation.md for domain-specific rubrics and safety evaluation