on-call-runbook

Quick reference for on-call engineers supporting the Payment API.

On-Call Runbook: Payment API

Quick reference for on-call engineers supporting the Payment API.

Service Overview

• Service: payment-api
• Team: Payments (@payments-team)
• Slack: #payments-sre
• PagerDuty: https://company.pagerduty.com/services/payment-api
• Dashboard: https://grafana.company.com/d/payment-api

Emergency Contacts

• Primary on-call: Check PagerDuty schedule
• Secondary backup: Check PagerDuty escalation
• Manager: @diana (Slack, +1-555-0100)
• Tech Lead: @alice (Slack, +1-555-0101)
• Executive escalation: VP Eng @frank (+1-555-0200)

Common Alerts

High Error Rate (> 1%)

Severity: Critical | Page: Yes

Symptoms: Users experiencing 500 errors, failed transactions

Quick Checks:

1. Dashboard: https://grafana.company.com/d/payment-api
2. Recent deployments: kubectl rollout history deployment/payment-api
3. Dependency health: Check auth-service, database
4. Traffic spike: Compare current vs normal load

Common Causes & Fixes:

• Recent deployment: Rollback via kubectl rollout undo deployment/payment-api
• Database issue: Check connections, consider failover
• Dependency down: Enable circuit breaker, use fallback
• Traffic spike: Scale horizontally: kubectl scale deployment/payment-api --replicas=10

Escalation: If not resolved in 30min, escalate to @tech-lead

High Latency (p95 > 500ms)

Severity: Warning | Page: Yes

Quick Checks:

1. Traces: https://jaeger.company.com/search?service=payment-api
2. Identify slow endpoints
3. Check database query performance
4. Review external API latency

Common Causes:

• Slow database queries: Check query plans, add indexes
• External API slow: Check circuit breaker, enable timeout
• Resource constrained: Check CPU/memory, scale if needed

Database Connection Pool Exhausted

Severity: Critical | Page: Yes

Quick Checks:

1. Connection pool metrics: Check Grafana
2. Long-running queries: SELECT * FROM pg_stat_activity WHERE state = 'active'
3. Connection leaks: Check application logs

Immediate Actions:

1. Increase pool size (temp): Update config, redeploy
2. Kill long queries: Use pg_terminate_backend(pid)
3. Restart app (last resort): kubectl rollout restart deployment/payment-api

Runbook Links

• High Error Rate: https://wiki.company.com/runbooks/high-error-rate
• Database Issues: https://wiki.company.com/runbooks/database
• Deployment Rollback: https://wiki.company.com/runbooks/rollback
• Scaling Guide: https://wiki.company.com/runbooks/scaling