engineering-deployment-strategies

Production deployment strategies including blue-green, canary, rolling updates, zero-downtime deployments, and database migrations

Deployment Strategies

Scope: Comprehensive deployment patterns, progressive delivery, feature flags, rollback procedures, and database migration strategies across Kubernetes, cloud platforms, and containerized environments

Lines: ~650

Last Updated: 2025-10-27

Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Designing deployment pipelines for production systems
Implementing zero-downtime deployments
Setting up progressive delivery (canary, blue-green)
Planning database schema migrations
Configuring rollback procedures
Implementing feature flags for gradual rollouts
Designing traffic shifting strategies
Setting up deployment validation and health checks

Don't use this skill for:

Basic CI/CD pipeline setup (see continuous-integration.md)
Container orchestration basics (see kubernetes-deployment)
Infrastructure provisioning (see terraform-best-practices)

Core Concepts

Concept 1: Deployment Strategy Selection

Definition: Choose deployment approach based on risk tolerance, downtime requirements, and infrastructure capabilities

Decision Matrix:

Requirement          | Recommended Strategy
---------------------|--------------------
Zero downtime        | Blue-Green, Canary, Rolling
Instant rollback     | Blue-Green
Gradual validation   | Canary
Limited resources    | Rolling, Recreate
Complex migrations   | Blue-Green + Feature Flags
High risk changes    | Canary + A/B Testing

Key Principles:

Understand blast radius of failures
Plan for rollback scenarios
Validate before full deployment
Monitor metrics during rollout
Maintain backward compatibility

Benefits:

Reduced deployment risk
Faster time to production
Improved reliability
Better user experience

Concept 2: Progressive Delivery

Definition: Gradually release changes with continuous validation and control

Components:

Code → Deploy → Feature Flag → Observe → Decide
         ↓           ↓            ↓         ↓
    Containers   Control      Metrics   Expand/
                 Exposure               Rollback

Traffic Management Patterns:

Percentage-based: Route X% of traffic to new version
User-based: Route specific users/groups to canary
Geography-based: Route by region for localized testing
Header-based: Route based on request headers (internal testing)

Validation Strategy:

Deploy to canary environment
Route small traffic percentage
Monitor error rates, latency, business metrics
Gradually increase traffic
Promote or rollback based on metrics

Concept 3: Database Migration Strategies

Definition: Evolve database schema without breaking application deployments

Expand-Contract Pattern:

Phase 1: EXPAND
  ├─ Add new schema alongside old
  ├─ Deploy app with dual writes
  └─ Both versions work

Phase 2: MIGRATE
  ├─ Backfill new schema
  └─ Verify data integrity

Phase 3: CONTRACT
  ├─ Deploy app using only new schema
  ├─ Remove old schema
  └─ Cleanup

Backward Compatibility Rules:

Add columns as nullable first
Never drop columns during deployment
Use views for compatibility layers
Dual write during transitions
Batch backfill operations

Rollback Considerations:

Keep old schema during rollback window
Test rollback procedures
Document migration dependencies
Use transactions where possible

Patterns

Pattern 1: Blue-Green Deployment

Problem: Need instant rollback capability with zero downtime

Kubernetes Implementation:

# Two identical deployments
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
---
# Service switches between blue and green
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: blue  # Change to 'green' to switch
  ports:
  - port: 80
    targetPort: 8080

Switch Procedure:

# 1. Deploy green environment
kubectl apply -f deployment-green.yaml

# 2. Wait for green to be ready
kubectl wait --for=condition=available deployment/myapp-green

# 3. Test green environment
curl http://myapp-green-preview/health

# 4. Switch traffic to green
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'

# 5. Monitor for issues (keep blue running)
sleep 600

# 6. Scale down blue (after validation)
kubectl scale deployment myapp-blue --replicas=0

Benefits: Instant rollback, full testing before switch Trade-offs: Requires 2x infrastructure, database complexity

Pattern 2: Canary Deployment with Flagger

Problem: Need gradual validation with automatic rollback

Flagger Canary Resource:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  provider: istio
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m

Progressive Rollout:

5% → Monitor → 15% → Monitor → 25% → Monitor → 50% → Complete
     ↓              ↓              ↓              ↓
   Healthy?      Healthy?       Healthy?       Healthy?
     ↓              ↓              ↓              ↓
   Continue      Continue       Continue       Promote
     or             or             or             or
   Rollback      Rollback       Rollback       Rollback

Benefits: Minimal blast radius, data-driven decisions Trade-offs: Slower deployment, complex traffic management

Pattern 3: Rolling Update with Health Checks

Problem: Need zero downtime with limited resources

Kubernetes Configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Max 1 pod down at a time
      maxSurge: 2        # Allow 2 extra pods during update
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: myapp
        image: myapp:v2.0.0
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

Update Process:

Create 1 new pod (within maxSurge)
Wait for readiness probe to pass
Terminate 1 old pod
Wait for graceful shutdown
Repeat until all pods updated

Benefits: No extra infrastructure, gradual rollout Trade-offs: Slower than blue-green, mixed versions during deployment

Pattern 4: Database Expand-Contract Migration

Problem: Need to change database schema without breaking deployments

Implementation:

-- PHASE 1: EXPAND (v1.1 app deployment)
ALTER TABLE orders ADD COLUMN status_name VARCHAR(50) NULL;

CREATE TRIGGER sync_status
BEFORE INSERT OR UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION sync_order_status();

-- PHASE 2: MIGRATE (background job)
UPDATE orders
SET status_name = CASE status
  WHEN 0 THEN 'pending'
  WHEN 1 THEN 'processing'
  WHEN 2 THEN 'completed'
END
WHERE status_name IS NULL;

-- PHASE 3: CONTRACT (v1.2 app deployment)
ALTER TABLE orders ALTER COLUMN status_name SET NOT NULL;
ALTER TABLE orders DROP COLUMN status;

Application Evolution:

v1.0: Reads/writes old column
v1.1: Reads old, writes both (dual write)
v1.2: Reads/writes new column only
v1.3: Old column removed

Benefits: Zero downtime, rollback-safe Trade-offs: Multi-phase deployment, temporary overhead

Pattern 5: Feature Flag Controlled Rollout

Problem: Decouple deployment from release

Implementation:

from launchdarkly import LDClient

ld_client = LDClient("sdk-key")

def get_pricing_page(user_id):
    user = {
        "key": user_id,
        "custom": {
            "deployment_version": os.getenv("APP_VERSION"),
            "canary_group": os.getenv("CANARY_GROUP")
        }
    }

    # Feature flag controls which version user sees
    use_new_pricing = ld_client.variation("new-pricing", user, False)

    if use_new_pricing:
        return render_template("pricing_v2.html")
    else:
        return render_template("pricing_v1.html")

Targeting Rules:

{
  "flagKey": "new-pricing",
  "targeting": [
    {
      "variation": 1,
      "values": ["canary"],
      "attribute": "canary_group"
    },
    {
      "variation": 1,
      "percentage": {
        "variations": [
          {"variation": 0, "weight": 95000},
          {"variation": 1, "weight": 5000}
        ]
      }
    }
  ]
}

Benefits: Deploy code safely, control feature exposure, instant rollback Trade-offs: Code complexity, flag management overhead

Anti-Patterns

Anti-Pattern 1: No Rollback Plan

Problem: Deploying without tested rollback procedures

Why It's Bad:

Panic during incidents
Extended downtime
Data loss risks
Manual recovery errors

Solution:

# Document and test rollback
# Blue-Green: Switch back
kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'

# Rolling: Rollback deployment
kubectl rollout undo deployment/myapp

# Database: Keep old schema during rollback window
# Feature Flags: Kill switch

Anti-Pattern 2: Testing in Production Only

Problem: No pre-deployment validation

Why It's Bad:

Discover issues after user impact
No safety net
Higher risk

Solution:

# Smoke tests before full rollout
- name: Test green environment
  run: |
    GREEN_URL=$(get_green_url)
    curl -f $GREEN_URL/health
    curl -f $GREEN_URL/api/test
    ./load-test.sh $GREEN_URL

Anti-Pattern 3: Incompatible Schema Changes

Problem: Breaking old application version with schema changes

Why It's Bad:

Forces all-or-nothing deployment
Can't rollback
Downtime required

Solution: Use expand-contract pattern, maintain backward compatibility

Anti-Pattern 4: No Health Checks

Problem: Routing traffic to unhealthy instances

Why It's Bad:

User-facing errors
Failed deployments go undetected
No automatic recovery

Solution:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 3

Anti-Pattern 5: Manual Deployment Steps

Problem: Undocumented manual interventions

Why It's Bad:

Not reproducible
Error-prone
Knowledge silos
Can't automate

Solution: Automate everything, document runbooks, use GitOps

Quick Reference

Deployment Strategy Comparison

| Strategy | Downtime | Rollback | Cost | Complexity | |----------|----------|----------|------|------------| | Recreate | High | Slow | Low | Low | | Rolling | None | Medium | Low | Medium | | Blue-Green | None | Instant | High (2x) | Medium | | Canary | None | Fast | Medium | High |

Health Check Best Practices

# Liveness: Is app alive?
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

# Readiness: Can app serve traffic?
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 2

Graceful Shutdown

import signal
import sys

def handle_shutdown(signum, frame):
    print("Shutting down gracefully...")
    # Stop accepting new requests
    server.close()
    # Wait for existing requests
    time.sleep(10)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_shutdown)

Database Migration Checklist

[ ] Schema changes are backward compatible
[ ] Dual write implemented during transition
[ ] Backfill plan for existing data
[ ] Rollback procedure documented
[ ] Migration tested in staging
[ ] Monitoring for data consistency

Level 3: Resources

Extended documentation, production-ready examples, and automation tools

REFERENCE.md

Location: resources/REFERENCE.md (4,033 lines)

Comprehensive reference covering:

Deployment strategy deep-dive (blue-green, canary, rolling, recreate)
Progressive delivery patterns and feature flags
Rollback procedures and automation
Zero-downtime deployment techniques
Database migration strategies (expand-contract, shadow tables)
Kubernetes deployment patterns with Istio/Flagger
Cloud provider deployment strategies (AWS, GCP, Azure)
Traffic management and load balancing
Monitoring and observability during deployments
Testing strategies and validation
Complete code examples for all patterns

Scripts

validate_deployment.py (600+ lines) Validates deployment configuration files across multiple platforms:

# Validate Kubernetes manifests
./validate_deployment.py --file deployment.yaml

# Check entire directory
./validate_deployment.py --directory k8s/ --type kubernetes

# JSON output for CI/CD
./validate_deployment.py --file deployment.yaml --json

execute_canary.py (550+ lines) Executes automated canary deployments with progressive traffic shifting:

# Kubernetes canary deployment
./execute_canary.py --platform kubernetes --service myapp --version v2.0

# AWS ALB canary
./execute_canary.py --platform aws-alb --service myapp --version v2.0

# Custom configuration
./execute_canary.py --config canary-config.yaml --json

test_deployment.sh (400+ lines) Tests deployment processes and measures downtime:

# Monitor deployment and measure downtime
./test_deployment.sh --url https://myapp.com --duration 300

# Test specific strategy
./test_deployment.sh --url https://myapp.com --strategy blue-green --json

Production Examples

Blue-Green Deployments:

examples/blue-green/kubernetes-blue-green.yaml: Complete K8s blue-green setup with health checks, PDB, and HPA
examples/docker/docker-compose-blue-green.yml: Docker Compose blue-green with NGINX load balancer

Canary Deployments:

examples/canary/flagger-canary.yaml: Flagger + Istio progressive canary with Prometheus metrics
examples/github-actions/canary-deployment-workflow.yml: GitHub Actions automated canary pipeline

Rolling Updates:

examples/rolling/kubernetes-rolling-update.yaml: Production-ready rolling update with proper health checks and resource limits

Database Migrations:

examples/database-migration/expand-contract-migration.sql: Complete expand-contract pattern with PostgreSQL

All examples include:

Production-ready configurations
Comprehensive comments
Health check implementations
Rollback procedures
Monitoring integration
Security best practices

Related Skills

continuous-integration.md - CI pipeline design
test-driven-development.md - Testing strategies
technical-debt.md - Managing deployment complexity
cloud-kubernetes-deployment - Kubernetes specifics
terraform-best-practices - Infrastructure provisioning

Last Updated: 2025-10-27 Maintainer: Skills Team Validation: CI-validated, production-tested patterns