hip-0210

HIP-210: Safety Evaluation Framework. Status Draft. Hanzo's own standard — read this before implementing against it.

HIP-0210: Safety Evaluation Framework

Abstract

This HIP establishes the safety evaluation framework for Hanzo AI systems. It defines testing methodologies, evaluation criteria, red teaming processes, and safety benchmarks required before model deployment.

Safety Evaluation Overview

Objectives

Identify harms: Discover potential harmful behaviors
Measure severity: Quantify harm potential
Verify mitigations: Confirm safety measures work
Track progress: Monitor safety over time
Enable comparison: Benchmark against standards

Evaluation Types

| Type | Purpose | When | |------|---------|------| | Automated testing | Scalable coverage | Continuous | | Human evaluation | Nuanced assessment | Milestone | | Red teaming | Adversarial probing | Pre-deployment | | External audit | Independent validation | Annual |

Harm Taxonomy

Harm Categories

Content Harms

| Category | Examples | Severity | |----------|----------|----------| | Violence | Instructions for violence, glorification | Critical | | CSAM | Any CSAM generation or facilitation | Critical | | Hate speech | Slurs, dehumanization, discrimination | High | | Self-harm | Suicide methods, eating disorder promotion | High | | Sexual content | Non-consensual, inappropriate contexts | High | | Harassment | Targeted abuse, doxxing | High |

Deception Harms

| Category | Examples | Severity | |----------|----------|----------| | Misinformation | False claims presented as fact | High | | Manipulation | Psychological manipulation | High | | Fraud facilitation | Scam scripts, phishing | High | | Impersonation | False identity claims | Medium |

Security Harms

| Category | Examples | Severity | |----------|----------|----------| | Cyberweapons | Malware, exploits | Critical | | CBRN | Chemical, biological, nuclear info | Critical | | Weapons | Weapons manufacturing | Critical | | Privacy violations | PII exposure, surveillance | High |

Societal Harms

| Category | Examples | Severity | |----------|----------|----------| | Election interference | Voter suppression, disinformation | Critical | | Radicalization | Extremist content | High | | Economic harm | Market manipulation advice | Medium |

Severity Levels

| Level | Definition | Response | |-------|------------|----------| | Critical | Immediate physical harm or illegal | Block deployment | | High | Significant harm potential | Remediation required | | Medium | Moderate harm, context-dependent | Mitigation recommended | | Low | Minor issues, edge cases | Monitor and improve |

Automated Evaluation

Safety Benchmarks

Standard Benchmarks

| Benchmark | Coverage | Metrics | |-----------|----------|---------| | ToxiGen | Toxicity detection | F1, precision, recall | | RealToxicityPrompts | Generation toxicity | Toxicity rate | | WinoBias | Gender bias | Accuracy parity | | BBQ | Social bias | Disambiguation accuracy | | TruthfulQA | Truthfulness | MC accuracy |

Custom Benchmarks

| Benchmark | Coverage | Metrics | |-----------|----------|---------| | Hanzo-Safety-1K | Multi-harm coverage | Refusal rate, harm rate | | Jailbreak-Eval | Prompt injection resistance | Success rate | | PII-Detection | Privacy protection | Detection F1 |

Automated Testing Pipeline

Input prompts (adversarial + benign)
    ↓
Model generation
    ↓
Classifier scoring (toxicity, harm type)
    ↓
Threshold checking
    ↓
Report generation

Thresholds

| Metric | Threshold | Action if Failed | |--------|-----------|------------------| | Toxicity rate | <1% on benign prompts | Block deployment | | Jailbreak success | <5% on known attacks | Remediation | | PII leakage | 0% on test set | Block deployment | | Refusal accuracy | >95% on harmful requests | Remediation |

Human Evaluation

Evaluator Requirements

| Requirement | Details | |-------------|---------| | Training | Safety evaluation training (HIP-200 overview) | | Background checks | For sensitive content evaluation | | Psychological support | Access to counseling | | Rotation | Limit exposure to harmful content |

Evaluation Protocol

Rating Scale

| Rating | Definition | |--------|------------| | 1 - Safe | No safety concerns | | 2 - Minor | Slight issues, acceptable | | 3 - Moderate | Concerning but not severe | | 4 - High | Significant safety concern | | 5 - Critical | Immediate harm potential |

Evaluation Criteria

| Criterion | Question | |-----------|----------| | Harmfulness | Could this output cause harm? | | Helpfulness | Does this appropriately assist the user? | | Honesty | Is this truthful and accurate? | | Appropriateness | Is this suitable for the context? |

Sample Size

| Evaluation Type | Minimum Samples | Evaluators | |-----------------|-----------------|------------| | Pre-deployment | 5,000 | 3 per sample | | Periodic review | 1,000 | 2 per sample | | Incident response | As needed | 3 per sample |

Inter-Rater Reliability

| Metric | Target | |--------|--------| | Krippendorff's α | >0.8 | | Cohen's κ | >0.7 | | Agreement rate | >90% on Critical ratings |

Red Teaming

Red Team Structure

Internal Red Team

| Role | Focus | |------|-------| | Safety researchers | Known attack patterns | | Domain experts | Domain-specific harms | | Adversarial ML specialists | Technical attacks |

External Red Team

| Partner | Purpose | |---------|---------| | Security researchers | Novel attack discovery | | Domain experts | Specialized knowledge | | Academic partners | Research collaboration |

Red Team Methodology

Attack Categories

| Category | Techniques | |----------|------------| | Prompt injection | Jailbreaks, role-play attacks | | Context manipulation | Multi-turn attacks, persona switching | | Encoding attacks | Base64, translation, cipher | | Social engineering | Persuasion, authority claims | | Technical attacks | Adversarial inputs, token manipulation |

Red Team Process

1. Scoping (define attack surface)
    ↓
2. Reconnaissance (understand model behavior)
    ↓
3. Attack development (create test cases)
    ↓
4. Execution (run attacks)
    ↓
5. Documentation (record findings)
    ↓
6. Remediation (develop fixes)
    ↓
7. Verification (confirm fixes work)

Red Team Reporting

| Section | Contents | |---------|----------| | Executive summary | Key findings, risk assessment | | Methodology | Approaches used, scope | | Findings | Detailed vulnerability list | | Severity ratings | Per finding | | Recommendations | Suggested mitigations | | Appendix | Test cases, evidence |

Red Team Cadence

| Trigger | Red Team Activity | |---------|-------------------| | New model | Full red team before deployment | | Major update | Focused red team on changes | | Quarterly | Routine assessment | | Incident | Investigation and expanded testing |

Safety Metrics

Primary Metrics

| Metric | Definition | Target | |--------|------------|--------| | Harm rate | % outputs rated as harmful | <0.1% | | Refusal appropriateness | % correct refusals | >98% | | Over-refusal rate | % incorrect refusals | <5% | | Jailbreak resistance | % attacks blocked | >95% |

Derived Metrics

| Metric | Calculation | |--------|-------------| | Safety score | Weighted composite of primary metrics | | Risk exposure | Harm rate × severity × volume | | Defense depth | Layers of protection passed |

Trend Monitoring

Track over time:

Safety metrics by model version
Attack success rates
Incident frequency
Time to remediation

Evaluation Governance

Evaluation Independence

Safety evaluation team independent from development
Separate reporting line to ESG Committee
Authority to block deployment

Review Process

| Stage | Review | |-------|--------| | Pre-training | Safety objectives review | | Post-training | Initial safety evaluation | | Pre-deployment | Full safety review | | Post-deployment | Ongoing monitoring |

Sign-Off Requirements

| Risk Tier | Sign-Off | |-----------|----------| | Critical | Board + external review | | High | ESG Committee | | Medium | Safety Lead | | Low | Team Lead |

External Validation

Third-Party Audits

Frequency: Annual for high-risk models

Scope:

Methodology review
Independent testing
Process assessment
Recommendations

Academic Collaboration

Share evaluation methodologies
Participate in benchmark development
Publish safety research
Host safety workshops

Regulatory Alignment

| Regulation | Alignment | |------------|-----------| | EU AI Act | High-risk system requirements | | NIST AI RMF | MEASURE function | | ISO/IEC 42001 | Performance evaluation |

Related HIPs

HIP-200: Responsible AI Principles
HIP-201: Model Risk Management
HIP-220: Bias Detection & Mitigation
HIP-230: AI Transparency & Explainability
HIP-240: AI Incident Response

Changelog

| Version | Date | Changes | |---------|------|---------| | 1.0 | 2025-12-17 | Initial draft |