engineering-capacity-planning

Comprehensive capacity planning including forecasting, resource modeling, load testing, scaling strategies, cost optimization, and disaster recovery planning for production systems

Capacity Planning

Scope: Forecasting methods, resource modeling (CPU/memory/disk/network), load testing, scaling strategies (vertical/horizontal/auto-scaling), cost optimization, cloud resource planning, database capacity planning, traffic analysis, disaster recovery capacity

Lines: ~850

Last Updated: 2025-10-27

Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Don't use this skill for:

Core Concepts

Concept 1: Capacity Planning Fundamentals

Definition: Proactive planning to ensure systems have sufficient resources to meet current and future demand

Key Principles:

Measure → Forecast → Plan → Provision → Monitor
   ↓         ↓        ↓        ↓          ↓
Current   Future   Resource  Deploy    Validate
 Usage    Demand   Sizing    Changes   Results

Planning Horizons:

Immediate (Days-Weeks):
├─ Handle current load spikes
├─ Address urgent capacity constraints
└─ Emergency scaling

Short-term (1-3 Months):
├─ Known launches or campaigns
├─ Seasonal patterns
└─ Planned migrations

Long-term (6-12+ Months):
├─ Business growth projections
├─ Architecture changes
└─ Strategic planning

Resource Types:

Concept 2: Forecasting Methods

Definition: Predict future resource usage based on historical data and growth patterns

Linear Forecasting:

# Simple linear regression
# Usage = baseline + growth_rate * time
# Best for: Steady, predictable growth

future_usage = current_usage + (growth_rate * time_periods)

# Example: 100 GB today, growing 10 GB/month
# In 6 months: 100 + (10 * 6) = 160 GB

Exponential Forecasting:

# Exponential growth
# Usage = baseline * (1 + growth_rate) ^ time
# Best for: Viral growth, compound growth

future_usage = current_usage * ((1 + growth_rate) ** time_periods)

# Example: 1000 users, growing 20%/month
# In 6 months: 1000 * (1.2^6) = 2,986 users

Seasonal Forecasting (Prophet):

# Facebook Prophet for seasonal patterns
# Best for: Weekly/monthly patterns, holidays

from prophet import Prophet

df = pd.DataFrame({
    'ds': dates,      # Date column
    'y': usage        # Usage metric
})

model = Prophet(
    yearly_seasonality=True,
    weekly_seasonality=True,
    daily_seasonality=False
)
model.fit(df)

# Forecast 90 days
future = model.make_future_dataframe(periods=90)
forecast = model.predict(future)

Time-Series Analysis (ARIMA):

# ARIMA for complex patterns
# Best for: Multiple trends, autocorrelation

from statsmodels.tsa.arima.model import ARIMA

model = ARIMA(usage_data, order=(p, d, q))
fitted = model.fit()

# Forecast next 30 days
forecast = fitted.forecast(steps=30)

Machine Learning (LSTM):

# Neural networks for complex patterns
# Best for: Non-linear relationships, multiple features

from tensorflow.keras import Sequential
from tensorflow.keras.layers import LSTM, Dense

model = Sequential([
    LSTM(50, return_sequences=True, input_shape=(lookback, features)),
    LSTM(50),
    Dense(1)
])

model.compile(optimizer='adam', loss='mse')
model.fit(X_train, y_train, epochs=50)

Concept 3: Resource Modeling

Definition: Model resource consumption based on workload characteristics

CPU Modeling:

CPU_needed = (requests_per_second * cpu_per_request) / cores_per_instance

Example:
- 1000 req/sec
- 10ms CPU per request
- 4 vCPU per instance

CPU_usage = (1000 * 0.010) / 4 = 2.5 cores (62.5% utilization)

Add headroom: 2.5 / 0.7 = 3.6 cores → Need 1 instance

Memory Modeling:

Memory_needed = base_memory + (connections * memory_per_connection)

Example:
- Base: 500 MB
- 1000 concurrent connections
- 2 MB per connection

Memory = 500 + (1000 * 2) = 2500 MB = 2.5 GB

Add headroom: 2.5 / 0.8 = 3.1 GB → Need 4 GB instance

Storage Modeling:

Storage_growth = current_size + (daily_growth * days) + retention

Example:
- Current: 1 TB
- Growth: 10 GB/day
- Forecast: 180 days
- Retention: 90 days

Storage = 1000 + (10 * 180) + (10 * 90) = 2800 GB = 2.8 TB

Add safety margin (20%): 2.8 * 1.2 = 3.4 TB

Network Modeling:

Bandwidth = (requests_per_second * avg_response_size) / (1024 * 1024)

Example:
- 5000 req/sec
- 50 KB avg response

Bandwidth = (5000 * 50) / 1024 = 244 MB/sec ≈ 2 Gbps

Peak traffic (3x): 6 Gbps required

Patterns

Pattern 1: Headroom and Safety Margins

Problem: Systems fail when running at 100% capacity

Headroom Strategy:

Resource Type    | Target Utilization | Safety Margin
-----------------|-------------------|---------------
CPU              | 70%               | 30%
Memory           | 80%               | 20%
Disk Space       | 80%               | 20%
IOPS             | 75%               | 25%
Network          | 60%               | 40%
Connection Pools | 75%               | 25%

Why Headroom Matters:

Without Headroom (100% target):
├─ No room for traffic spikes
├─ Deployment requires downtime
├─ Single failure cascades
└─ Performance degradation

With Headroom (70% target):
├─ Handles 43% traffic increase
├─ Rolling deployments safe
├─ Failure tolerance
└─ Consistent performance

Calculating Headroom:

def calculate_headroom(current_usage, capacity, target_util=0.7):
    """Calculate remaining headroom."""
    current_util = current_usage / capacity
    remaining = (target_util - current_util) * capacity
    return {
        'current_utilization': current_util,
        'remaining_headroom': remaining,
        'time_to_capacity': estimate_time_to_capacity(current_usage, remaining)
    }

Pattern 2: Load Testing Strategy

Problem: Need to validate capacity under realistic load

Load Testing Pyramid:

                  ┌─────────────────┐
                  │  Chaos Testing  │  (Rare, extreme scenarios)
                  └─────────────────┘
                ┌───────────────────────┐
                │   Stress Testing      │  (Beyond limits)
                └───────────────────────┘
            ┌───────────────────────────────┐
            │     Load Testing              │  (Expected peak)
            └───────────────────────────────┘
        ┌───────────────────────────────────────┐
        │      Baseline Testing                 │  (Normal load)
        └───────────────────────────────────────┘

Load Test Types:

Baseline Test:
  duration: 10 minutes
  load: Normal traffic (e.g., 100 req/sec)
  goal: Establish performance baseline
  metrics: P50, P95, P99 latency, error rate

Load Test:
  duration: 30 minutes
  load: Expected peak (e.g., 500 req/sec)
  goal: Verify capacity for known peaks
  metrics: Latency, throughput, resource usage

Stress Test:
  duration: 15 minutes
  load: Beyond peak (e.g., 1000 req/sec)
  goal: Find breaking point
  metrics: When does it fail? How does it fail?

Soak Test:
  duration: 4-24 hours
  load: Sustained normal-high load
  goal: Find memory leaks, resource exhaustion
  metrics: Memory growth, connection leaks

Spike Test:
  duration: 2 minutes
  load: Sudden 10x increase
  goal: Validate auto-scaling response
  metrics: Scale-up time, recovery time

Locust Load Test Example:

from locust import HttpUser, task, between

class APIUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def get_items(self):
        self.client.get("/api/items")

    @task(1)
    def create_item(self):
        self.client.post("/api/items", json={
            "name": "test",
            "value": 42
        })

# Run: locust -f loadtest.py --users 1000 --spawn-rate 10

Pattern 3: Auto-Scaling Strategy

Problem: Manual scaling is slow and error-prone

Horizontal Pod Autoscaler (Kubernetes):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    # CPU-based scaling
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

    # Memory-based scaling
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

    # Custom metric (RPS)
    - type: Pods
      pods:
        metric:
          name: requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scale down
      policies:
      - type: Percent
        value: 50                       # Remove max 50% of pods
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0     # Scale up immediately
      policies:
      - type: Percent
        value: 100                      # Double pods if needed
        periodSeconds: 30
      - type: Pods
        value: 5                        # Add max 5 pods
        periodSeconds: 30

AWS Auto Scaling:

{
  "ServiceNamespace": "ecs",
  "ScalableDimension": "ecs:service:DesiredCount",
  "PolicyType": "TargetTrackingScaling",
  "TargetTrackingScalingPolicyConfiguration": {
    "TargetValue": 70.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleOutCooldown": 60,
    "ScaleInCooldown": 300
  }
}

Scaling Triggers:

Metric Type       | Scale Up Threshold | Scale Down Threshold
------------------|--------------------|-----------------------
CPU               | >70% for 2 min    | <30% for 10 min
Memory            | >80% for 2 min    | <40% for 10 min
Request Rate      | >80% capacity     | <40% capacity
Response Time     | P95 > 500ms       | P95 < 200ms
Queue Depth       | >100 messages     | <10 messages
Error Rate        | >1%               | <0.1%

Pattern 4: Cost Optimization

Problem: Over-provisioning wastes money, under-provisioning hurts users

Cost Optimization Strategies:

1. Right-sizing:
   ├─ Analyze actual usage vs provisioned
   ├─ Downsize over-provisioned resources
   └─ Use smaller instance types where possible

2. Reserved/Committed Capacity:
   ├─ 1-year commitment: ~30% discount
   ├─ 3-year commitment: ~50% discount
   └─ Use for baseline capacity

3. Spot/Preemptible Instances:
   ├─ 70-90% discount
   ├─ Use for batch workloads
   └─ Fault-tolerant services

4. Auto-scaling:
   ├─ Scale down during off-peak
   ├─ Match capacity to demand
   └─ Avoid idle resources

5. Storage Tiering:
   ├─ Hot data: SSD
   ├─ Warm data: HDD
   └─ Cold data: Archive (S3 Glacier)

Cost Analysis:

def analyze_cost_optimization(resources):
    """Identify cost optimization opportunities."""

    opportunities = []

    for resource in resources:
        utilization = resource.avg_utilization
        cost = resource.monthly_cost

        # Under-utilized (< 30% for 30 days)
        if utilization < 0.30:
            savings = cost * 0.5  # Estimate 50% savings
            opportunities.append({
                'resource': resource.name,
                'action': 'Downsize or terminate',
                'current_util': utilization,
                'potential_savings': savings
            })

        # No reservation (stable workload)
        if not resource.is_reserved and resource.age_days > 90:
            savings = cost * 0.35  # 35% with 1-year RI
            opportunities.append({
                'resource': resource.name,
                'action': 'Purchase reserved capacity',
                'potential_savings': savings
            })

    return opportunities

Checklist

Capacity Planning Checklist

Data Collection:

Forecasting:

Capacity Planning:

Testing:

Implementation:

Monitoring:

Anti-Patterns

Planning Anti-Patterns:

❌ No historical data → Guessing instead of forecasting
❌ Short history (< 30 days) → Missing patterns
❌ Ignore seasonality → Under-capacity during peaks
❌ No headroom → Systems at 100%, no room for spikes
❌ Only plan for average → Fail during peak load
❌ Plan for 1 year out → Inaccurate, wasted effort

Testing Anti-Patterns:

❌ Test in production → Risk customer impact
❌ No load testing → Discover limits during incidents
❌ Test with synthetic data → Doesn't match real usage
❌ Single load test → Miss edge cases
❌ No stress testing → Don't know breaking point
❌ Ignore test failures → Launch without confidence

Scaling Anti-Patterns:

❌ Manual scaling only → Slow response to load changes
❌ Aggressive scale-down → Flapping, instability
❌ No scale-up delay → Over-react to spikes
❌ Scale on CPU only → Miss memory constraints
❌ No max replicas → Runaway scaling, cost explosion
❌ No monitoring → Don't know if scaling works

Cost Anti-Patterns:

❌ Over-provision "to be safe" → Wasted money
❌ No reservation strategy → Pay full price
❌ Ignore right-sizing → Pay for unused resources
❌ No cost monitoring → Surprise bills
❌ No auto-scaling → Pay for idle resources
❌ No storage lifecycle → Pay for old data

Recovery

When Forecasts Are Wrong:

1. MEASURE actual vs predicted variance
2. IDENTIFY root cause (unexpected growth, bad model, missing data)
3. ADJUST forecast model or parameters
4. UPDATE capacity plan with new forecast
5. COMMUNICATE changes to stakeholders
6. DOCUMENT lessons learned

When Load Tests Reveal Issues:

1. DOCUMENT the issue and load level
2. DETERMINE impact (hard limit or degradation?)
3. IDENTIFY bottleneck (CPU, memory, database, network)
4. CALCULATE capacity needed to pass
5. IMPLEMENT fixes (optimize or add capacity)
6. RE-TEST to validate
7. UPDATE capacity plan

When Scaling Fails:

1. REVERT to manual scaling if safe
2. DIAGNOSE root cause (metrics, limits, quotas)
3. TEST fixes in non-production
4. DEPLOY fix with monitoring
5. VALIDATE auto-scaling behavior
6. DOCUMENT failure and resolution

Level 3: Resources

Extended Documentation: REFERENCE.md (2,800+ lines)

Scripts: Production-ready tools in resources/scripts/

Examples: Production-ready examples in resources/examples/

All scripts include:

Usage:

# Forecast capacity for next 90 days
./forecast_capacity.py --metric cpu_usage --period 90 \
  --method prophet --output forecast.json --visualize

# Analyze resource usage patterns
./analyze_resource_usage.py --days 30 --resources cpu,memory,disk \
  --detect-anomalies --json

# Test scaling behavior
./test_scaling.py --target api-service --duration 30 \
  --max-rps 1000 --test-autoscaling --report scaling_report.html

# Generate capacity forecast
python examples/forecasting/prophet_forecast.py \
  --input metrics.csv --forecast-days 60

# Analyze costs
python examples/cost-optimization/analyze_costs.py \
  --provider aws --region us-east-1 --optimize

Related Skills

Last Updated: 2025-10-27 Format Version: 1.0 (Atomic) Level 3 Resources: Available