sev1-response-workflow

This document provides a complete walkthrough of responding to a SEV-1 (critical) incident from detection to postmortem.

SEV-1 Incident Response Workflow

This document provides a complete walkthrough of responding to a SEV-1 (critical) incident from detection to postmortem.

Scenario

Incident: High error rate (35%) in payment processing API Detection: Automated monitoring alert at 14:25 UTC Impact: 50% of payment transactions failing, affecting thousands of users

Phase 1: Detection and Declaration (14:25 - 14:35 UTC)

14:25 - Alert Fires

PagerDuty Alert:

CRITICAL: High Error Rate in Payment API
Error rate: 35% (threshold: 5%)
Dashboard: https://grafana.example.com/d/payments
Runbook: https://runbooks.example.com/payments/high-error-rate

On-call engineer (Alice) receives page

14:27 - Initial Assessment

Alice acknowledges alert and checks dashboard:

# Quick checks
curl https://api.example.com/health
# Returns: {"status": "degraded", "payment_api": "unhealthy"}

# Check recent deployments
kubectl rollout history deployment/payment-service | tail -5
# Shows: deployment v2.3.1 at 14:15 UTC (10 minutes ago)

# Check error logs
kubectl logs -l app=payment-service --since=15m | grep ERROR | head -20
# Shows: database connection errors

Assessment:

14:30 - Declare SEV-1

Alice determines this is SEV-1:

Actions:

# 1. Create incident in tracking system
./create_incident.py create \
  --title "High error rate in payment API" \
  --severity SEV-1 \
  --impact "50% of payment transactions failing" \
  --service payment-api \
  --detected-by monitoring

# Output: Created incident: INC-20251027-ABC123

# 2. Create PagerDuty incident
python pagerduty-integration.py create \
  --incident-id INC-20251027-ABC123 \
  --service-id PXXXXXX \
  --title "SEV-1: Payment API High Error Rate"

# 3. Create Slack war room
python slack-incident-bot.py create-war-room \
  --incident-id INC-20251027-ABC123 \
  --title "High error rate in payment API" \
  --severity SEV-1 \
  --ic-user-id U123ABC  # Alice

# Output: Created war room: #incident-20251027-abc123

14:32 - Initial War Room Message

Alice posts in #incident-20251027-abc123:

@here SEV-1 Incident Declared

INC-20251027-ABC123: High error rate in payment API

IMPACT:
- 35% error rate in payment transactions
- Affecting 50% of users attempting payments
- Started: ~14:20 UTC (correlates with deployment v2.3.1)

CURRENT STATUS: Investigating

TEAM:
- IC: @alice
- On-call: @alice (same person, need backup)

FIRST ACTIONS:
- Assessing rollback vs. fix
- Need database team to check connection pool
- Communications: notify support team NOW

Next update in 15 minutes.

14:33 - Page Additional Team

# Page database specialist
pagerduty incident create \
  --service database-service \
  --title "DB connection errors - Payment incident" \
  --urgency high

# Page engineering manager for IC support
pagerduty incident create \
  --escalation-policy eng-management \
  --title "SEV-1 needs IC support"

# Invite to war room
python slack-incident-bot.py invite \
  --channel incident-20251027-abc123 \
  --users U234BCD,U345CDE  # Bob (DB), Charlie (Mgr)

14:35 - Initial Customer Notification

Communications Lead (auto-assigned or manually):

Subject: [SEV-1] Payment Processing Issue

We are currently experiencing elevated error rates in payment processing.

IMPACT:
- Some payment transactions may fail
- Users may see error messages during checkout
- Started at: 14:20 UTC

STATUS:
- Issue identified and team actively investigating
- Working on mitigation

NEXT UPDATE:
- Will provide update in 30 minutes
- Status page: https://status.example.com

We apologize for the inconvenience.

Phase 2: Mitigation (14:35 - 14:50 UTC)

14:36 - Team Assembled

War room participants:

Charlie (Mgr) posts:

[14:36] @charlie Taking formal IC role. @alice moving to tech lead.

Roles:
- IC: @charlie
- Tech Lead: @alice
- Database SME: @bob
- Communications: @diana
- Scribe: @eve

@alice, what's your recommendation? Rollback or investigate?

14:37 - Rollback Decision

Timeline in war room:

[14:37] @alice [OBSERVATION] Deployment v2.3.1 correlates exactly with error spike
[14:38] @alice [OBSERVATION] Logs show "connection pool exhausted" errors
[14:38] @bob [OBSERVATION] DB connection count at 95/100 (near max)
[14:39] @bob [OBSERVATION] Queries look normal, but higher volume than usual
[14:40] @charlie [DECISION] Rollback deployment v2.3.1 to v2.3.0
[14:40] @alice [ACTION] Starting rollback now

Rollback execution:

# 1. Rollback deployment
kubectl rollout undo deployment/payment-service

# 2. Verify rollback
kubectl rollout status deployment/payment-service
# Output: successfully rolled out

# 3. Monitor metrics
watch -n 5 'curl -s https://api.example.com/metrics | grep payment_error_rate'

14:42 - Rollback Complete

[14:42] @alice [ACTION] Rollback complete, v2.3.0 now running
[14:43] @alice [OBSERVATION] Error rate still at 30%
[14:43] @charlie [DECISION] Rollback didn't help immediately, monitor for 5 minutes

14:45 - Mitigation Working

[14:45] @alice [OBSERVATION] Error rate declining: 22%
[14:46] @bob [OBSERVATION] DB connection count dropping: 75/100
[14:47] @alice [OBSERVATION] Error rate at 10%
[14:48] @alice [OBSERVATION] Error rate at 3%

14:48 - Additional Mitigation

[14:48] @bob [RECOMMENDATION] Increase connection pool to prevent recurrence
[14:49] @charlie [DECISION] Proceed with pool increase
[14:49] @bob [ACTION] Increasing pool from 20 to 50
# Increase connection pool
kubectl set env deployment/payment-service DB_POOL_SIZE=50

# Verify
kubectl get deployment payment-service -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="DB_POOL_SIZE")].value}'
# Output: 50

14:50 - Error Rate Normalized

[14:50] @alice [OBSERVATION] Error rate back to baseline: 0.5%
[14:50] @charlie [DECISION] Monitoring for 15 minutes before declaring resolved

14:50 - Update Customers

Subject: [SEV-1] Update #1 - Payment Processing Issue

UPDATE #1 - 14:50 UTC

CURRENT STATUS: Mitigated

ACTIONS TAKEN:
- 14:40 UTC: Rolled back recent deployment
- 14:49 UTC: Increased database connection pool capacity
- Error rate decreased from 35% to 0.5% (baseline)

CURRENT EFFORTS:
- Monitoring for stability before declaring resolved
- Investigating root cause

NEXT UPDATE: 15:20 UTC (or when resolved)

Phase 3: Monitoring and Resolution (14:50 - 15:15 UTC)

14:50 - 15:05: Monitoring Period

[14:52] @alice [OBSERVATION] All metrics stable for 2 minutes
[14:55] @alice [OBSERVATION] No new errors in logs
[14:58] @bob [OBSERVATION] DB connection pool at healthy 15/50 utilization
[15:00] @alice [OBSERVATION] Payment success rate back to 99.5% (normal)
[15:02] @alice [OBSERVATION] No customer complaints in last 10 minutes
[15:05] @alice [OBSERVATION] 15 minutes stable, all criteria met

15:05 - Pre-Resolution Checklist

Charlie (IC) verifies resolution criteria:

Resolution Criteria:
✓ Error rate back to baseline (<0.5%)
✓ Latency within SLO (p95 < 500ms)
✓ Throughput recovered to normal
✓ All health checks passing
✓ No active alerts
✓ User-facing functionality verified
✓ Support ticket volume normal
✓ No customer reports for 15+ minutes
✓ System stable for 15+ minutes
✓ Related systems healthy
✓ Dependencies verified operational

ALL CRITERIA MET

15:07 - Declare Resolved

[15:07] @charlie [DECISION] Declaring incident RESOLVED
[15:07] @charlie [NOTIFICATION] Incident INC-20251027-ABC123 is RESOLVED
[15:07] @charlie [ACTION] @diana send resolution notification
[15:08] @charlie [ACTION] @eve schedule postmortem for tomorrow 10:00 UTC

Update systems:

# Update incident status
./create_incident.py update INC-20251027-ABC123 \
  --status resolved \
  --event "Incident resolved" \
  --author charlie

# Resolve PagerDuty incident
python pagerduty-integration.py resolve \
  --incident-id PXXXXXX \
  --resolution "Rolled back deployment and increased connection pool"

# Update Slack
python slack-incident-bot.py post-resolution \
  --channel incident-20251027-abc123 \
  --incident-id INC-20251027-ABC123 \
  --duration "45 minutes" \
  --resolution "Rolled back deployment v2.3.1 and increased DB connection pool to 50"

15:10 - Final Customer Notification

Subject: [RESOLVED] Payment Processing Issue Resolved

The incident affecting payment processing has been RESOLVED.

SUMMARY:
- Started: 14:20 UTC
- Resolved: 15:07 UTC
- Duration: 47 minutes
- Impact: Elevated error rates in payment transactions

RESOLUTION:
- Rolled back deployment v2.3.1
- Increased database connection pool capacity
- Service is now operating normally
- Monitoring for continued stability

ROOT CAUSE:
- Preliminary: Deployment introduced query changes that exhausted DB connection pool
- Full postmortem will be published within 48 hours

NO ACTION REQUIRED from users. All pending payments will be automatically retried.

We apologize for any inconvenience this may have caused.

15:12 - Thank Team

[15:12] @charlie Thank you team for excellent response:
- @alice: Quick diagnosis and rollback
- @bob: Fast DB analysis and mitigation
- @diana: Clear customer communications
- @eve: Great timeline documentation

MTTR: 47 minutes from detection to resolution

Postmortem: Tomorrow 10:00 UTC in this channel
This war room will stay open for documentation.

Phase 4: Post-Incident (Next 48 hours)

Hour 1: Immediate Follow-up

Actions:

# Compile timeline
python slack-incident-bot.py export-timeline \
  --channel incident-20251027-abc123 \
  --output timeline.json

# Gather metrics
curl "https://grafana.example.com/api/dashboards/uid/payments" > metrics.json

# Take screenshots of key graphs
# (manual or automated)

# Begin postmortem draft
./generate_postmortem.py \
  --incident-id INC-20251027-ABC123 \
  --authors alice,bob,charlie \
  --template deployment \
  --output postmortems/INC-20251027-ABC123.md

Hour 24: Postmortem Draft

Draft sections completed:

Key findings:

  1. Immediate cause: Deployment v2.3.1 introduced N+1 query pattern
  2. Contributing factors: Connection pool too small, no load testing
  3. Latent conditions: No automated pool sizing, no query performance monitoring

Hour 26: Postmortem Meeting

Agenda (1 hour):

10:00-10:05: Read executive summary
10:05-10:15: Walk through timeline
10:15-10:25: What went well
10:25-10:40: What went poorly (blameless!)
10:40-10:55: Action items
10:55-11:00: Assign owners and deadlines

Action Items Defined:

Prevent Recurrence:

Improve Detection:

Improve Response:

Improve Processes:

Hour 48: Publish Postmortem

Published to:

Follow-up scheduled:

Metrics Captured

Response Metrics

Impact Metrics

Response Quality

Lessons Learned

What Went Well

  1. ✅ Fast detection: Monitoring alerted within 5 minutes
  2. ✅ Quick rollback: Recognized deployment correlation and acted fast
  3. ✅ Clear communication: Regular updates kept stakeholders informed
  4. ✅ Team coordination: War room structure kept everyone aligned
  5. ✅ Good documentation: Timeline made postmortem easy to write

What Went Poorly

  1. ❌ Deployment caused issue: Should have caught in testing
  2. ❌ No load testing: Last load test was 3 months old
  3. ❌ Missing alerts: No alert for connection pool utilization
  4. ❌ Code review gap: N+1 pattern not caught in review
  5. ❌ Manual pool sizing: Should be automated based on traffic

Key Takeaways

  1. Load testing must be automated and run on every deploy
  2. Connection pool needs monitoring with alerts before exhaustion
  3. Code review needs performance checklist for database queries
  4. Rollback is the safe default when deployment correlates with issues
  5. War room structure works - clear roles, regular updates, good documentation

Template Sections for Future Incidents

This workflow can be adapted for other SEV-1 incidents by:

  1. Replacing payment API scenario with actual service
  2. Adjusting mitigation steps based on root cause
  3. Keeping same structure: Detection → Mitigation → Resolution → Postmortem
  4. Using same communication templates
  5. Following same metrics collection