engineering-incident-response

Comprehensive incident response practices including lifecycle management, severity classification, war rooms, runbooks, blameless postmortems, and SRE metrics (MTTR, MTTD, MTTA, MTBF)

Incident Response

Scope: Complete incident lifecycle from detection through postmortem, including IC role, communication protocols, escalation policies, metrics-driven improvement, and blameless culture development

Lines: ~700

Last Updated: 2025-10-27

Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Don't use this skill for:

Core Concepts

Concept 1: Incident Lifecycle

Definition: Structured progression from detection through resolution and learning

Lifecycle Stages:

Detection → Response → Mitigation → Resolution → Postmortem
    ↓         ↓           ↓            ↓            ↓
  Alert   Assemble   Implement    Verify      Document
           Team       Fix          Normal      Lessons

Key Principles:

  1. Customer First: Restore service before finding root cause
  2. Clear Communication: Keep stakeholders informed
  3. Document Everything: Timeline, decisions, actions
  4. Learn and Improve: Every incident is learning opportunity
  5. Blameless: Focus on systems, not individuals

Stage Objectives:

Concept 2: Severity Classification

Definition: Categorize incidents by impact to determine response urgency

Severity Levels:

SEV-1 (Critical):
├─ Complete outage or severe degradation
├─ >50% users affected
├─ Business-critical functions unavailable
└─ Response: Immediate, 24/7, all hands

SEV-2 (High):
├─ Significant degradation
├─ 10-50% users affected
├─ Major features impaired
└─ Response: Urgent, specialized team

SEV-3 (Medium):
├─ Minor degradation
├─ 1-10% users affected
├─ Workarounds available
└─ Response: Standard on-call

SEV-4 (Low):
├─ Minimal/no user impact
├─ Internal tools affected
├─ Cosmetic issues
└─ Response: Business hours

Classification Matrix: | Users Affected | Revenue Impact | SLA Risk | Severity | |----------------|----------------|----------|----------| | >50% | >$10K/hr | Yes | SEV-1 | | 10-50% | $1-10K/hr | Maybe | SEV-2 | | 1-10% | <$1K/hr | No | SEV-3 | | <1% | None | No | SEV-4 |

Escalation/De-escalation:

Concept 3: Incident Commander (IC) Role

Definition: Single person responsible for coordinating incident response

IC Responsibilities:

Coordinate: Direct technical responders
Decide: Final call on mitigation strategies
Communicate: Ensure stakeholders informed
Document: Maintain timeline
Drive: Keep team focused on restoration
Declare: Determine when resolved

IC is NOT:

IC Decision Framework:

Rollback? → Recent deployment + High confidence issue? → YES
          → Unsure or no deployment? → Investigate 15 min → Still bad? → ROLLBACK

Escalate? → Need expertise not on-call? → Page specialist
          → Need management decision? → Page manager
          → Need vendor help? → Open critical ticket

Resolve? → All criteria met + 15+ min stable? → YES
         → Any criteria failed? → Continue monitoring

Patterns

Pattern 1: War Room Coordination

Problem: Need structured communication during chaotic incidents

War Room Structure:

Channel: #incident-YYYYMMDD-description
├─ IC: Coordinates, makes decisions
├─ Tech Lead: Hands on keyboard, implements
├─ Comms Lead: External updates
├─ Scribe: Documents timeline
└─ SMEs: Domain experts as needed

Communication Format:
[HH:MM] [@person] [TYPE] Description

Types:
- ACTION: Something done
- DECISION: Choice made
- OBSERVATION: Data/status
- QUESTION: Need info
- ANSWER: Response

Example War Room Flow:

[14:30] @ic [DECISION] Declaring SEV-1, creating war room
[14:32] @tech-lead [OBSERVATION] Error rate 35%, started 14:20
[14:35] @tech-lead [OBSERVATION] Correlates with deployment v2.3.1
[14:37] @ic [DECISION] Rollback deployment
[14:40] @tech-lead [ACTION] Rollback complete
[14:45] @tech-lead [OBSERVATION] Error rate declining to 8%
[14:50] @ic [DECISION] Monitoring 15 min before resolving

Benefits:

Pattern 2: Blameless Postmortem

Problem: Need to learn from incidents without creating fear

Blameless Principles:

Focus on:
✓ Systems and processes
✓ "The monitoring didn't..."
✓ "The process allowed..."
✓ Why system permitted issue

Avoid:
✗ Individual blame
✗ "Alice deployed bad code"
✗ "Bob didn't notice"
✗ Who made mistake

Postmortem Structure:

# Postmortem: [Title]

## Executive Summary
3-sentence summary: what happened, impact, resolution

## Impact
- User impact metrics
- Business impact
- Technical metrics

## Timeline
| Time | Event |
|------|-------|
| ...  | ...   |

## Root Cause Analysis (Five Whys)
1. Why did X fail? → Answer
2. Why did that happen? → Answer
3. Why did that happen? → Answer
4. Why did that happen? → Answer
5. Why did that happen? → Answer

## What Went Well
- Positives to reinforce

## What Went Poorly
- Areas for improvement (blameless!)

## Action Items
| Action | Owner | Deadline | Category |
|--------|-------|----------|----------|
| ...    | ...   | ...      | Prevent  |

## Lessons Learned
Key takeaways for organization

Action Item Categories:

Pattern 3: Metrics-Driven Improvement

Problem: Need objective measures of incident response effectiveness

The Four Golden Metrics:

MTTR (Mean Time To Repair):
├─ Time from incident start to resolution
├─ Target: <1 hour SEV-1, <4 hours SEV-2
└─ Improve: Better runbooks, automation, rollback

MTTD (Mean Time To Detect):
├─ Time from issue start to detection
├─ Target: <5 minutes
└─ Improve: Synthetic monitoring, better alerts

MTTA (Mean Time To Acknowledge):
├─ Time from alert to acknowledgment
├─ Target: <5 min SEV-1, <15 min SEV-2
└─ Improve: Clear on-call, alert quality

MTBF (Mean Time Between Failures):
├─ Time between incidents
├─ Target: Increasing over time
└─ Improve: Action item completion, testing

Measurement Example:

# Calculate MTTR
incidents = load_resolved_incidents()
durations = [
    (inc.resolved_at - inc.started_at).total_seconds() / 60
    for inc in incidents
]
mttr_minutes = mean(durations)

# By severity
sev1_mttr = mean([d for inc, d in zip(incidents, durations)
                   if inc.severity == 'SEV-1'])

SLO and Error Budget:

service: api
slo:
  availability: 99.9%  # Max 43.8 min downtime/month
  error_rate: 0.1%

error_budget:
  monthly_downtime: 43.8 minutes
  used_this_month: 135 minutes  # 308% - EXHAUSTED

actions_when_exhausted:
  - Freeze non-critical deployments
  - Focus on reliability
  - Postmortem all incidents
  - Mandatory chaos engineering

Checklist

Incident Response Checklist

Initial Response (0-5 min):

Mitigation (5-60 min):

Resolution:

Postmortem (24-72 hours):

Anti-Patterns

Process Anti-Patterns:

❌ No clear IC → Multiple people making conflicting decisions
❌ Skip severity classification → Wrong response urgency
❌ Investigate before mitigating → Customers suffer longer
❌ No communication updates → Stakeholders anxious, duplicate work
❌ Resolve too early → Issue recurs immediately
❌ Skip postmortem → No learning, same issue repeats

Communication Anti-Patterns:

❌ Blame individuals → Fear, information hiding
❌ Vague updates → Stakeholders don't understand status
❌ Irregular updates → Anxiety and escalations
❌ Over-technical → Non-engineers confused
❌ No final notification → Unclear if resolved

Metrics Anti-Patterns:

❌ Only measure MTTR → Ignore detection and acknowledgment
❌ Don't track by severity → Can't prioritize improvements
❌ No trend analysis → Miss systemic issues
❌ Ignore action item completion → Same incidents repeat
❌ Celebrate "hero" fixes → Discourage prevention

Recovery

When Incident Gets Worse:

1. STOP current mitigation if making things worse
2. ROLLBACK to last known good state
3. RE-ASSESS severity (may need to escalate)
4. PAGE additional help (specialists, manager)
5. COMMUNICATE change in status
6. CONSIDER maintenance mode to stop damage

When IC Needs to Hand Off:

1. Brief new IC on:
   - Current status and impact
   - What's been tried
   - Current strategy
   - Key people and roles
   - Pending decisions

2. New IC announces role in war room
3. Old IC stays available for questions
4. Document handoff in timeline

When Postmortem Becomes Blame:

1. Facilitator intervenes immediately
2. Reframe: "What allowed this?" not "Who did this?"
3. Focus on systems: monitoring, processes, tools
4. If continues, pause meeting
5. Reset expectations: blameless or don't continue

Level 3: Resources

Extended Documentation: REFERENCE.md (3,200+ lines)

Scripts: Production-ready tools in resources/scripts/

Examples: Production-ready examples in resources/examples/

All scripts include:

Usage:

# Create incident with tracking
./create_incident.py create --title "High API error rate" \
  --severity SEV-1 --impact "50% of users affected" --service api

# Analyze incident metrics
./analyze_mttr.py --period 30 --verbose --compare-services

# Generate postmortem from incident
./generate_postmortem.py --incident-id INC-123 \
  --authors alice,bob --template deployment --output postmortem.md

# PagerDuty integration
python pagerduty-integration.py --create --service-id PXXXXXX

# Slack war room
python slack-incident-bot.py create-war-room \
  --incident-id INC-123 --title "API errors" --severity SEV-1

Related Skills