engineering-rfc-technical-design

Architecture proposals, technical diagrams, API design, trade-off analysis, and migration strategies for RFCs

RFC Technical Design

Scope: Techniques for designing architecture, evaluating alternatives, creating technical diagrams, and planning migrations Lines: ~340 Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Core Concepts

Concept 1: Architecture Design Levels (C4 Model)

Level 1: System Context

Level 2: Container Diagram

Level 3: Component Diagram

Level 4: Code Diagram

Concept 2: Trade-Off Analysis Framework

Evaluation Dimensions:

Quantification: Always include numbers where possible

Concept 3: API Design Principles

Versioning Strategies:

Backwards Compatibility:

Error Handling:

Patterns

Pattern 1: Architecture Diagram (C4 Level 2 - Container)

When to use:

## System Architecture: Real-Time Analytics Platform

### Container Diagram (C4 Level 2)

┌─────────────────────────────────────────────────────────────────┐ │ User Clients │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Web │ │ Mobile │ │ API │ │ │ │ Client │ │ App │ │ Client │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ └─────────┼─────────────────────┼──────────────────────┼───────────┘ │ │ │ │ HTTPS │ HTTPS │ HTTPS │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Load Balancer (NGINX) │ └────────────────────────────┬────────────────────────────────────┘ │ ┌──────────────────┼──────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ API Server │ │ API Server │ │ API Server │ │ (Node.js) │ │ (Node.js) │ │ (Node.js) │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ ├──────────────────┴──────────────────┤ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Redis │ │ RabbitMQ │ │ Cache │ │ Queue │ │ │ │ │ └─────────────┘ └──────┬──────┘ │ │ ▼ ┌──────────────┐ │ Worker │ │ Processes │ │ (Python) │ └──────┬───────┘ │ ┌─────────────────────────────────────┤ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ PostgreSQL │ │ S3 Bucket │ │ Database │ │ (Logs) │ │ │ │ │ └─────────────┘ └─────────────┘


### Component Responsibilities

**Load Balancer (NGINX)**:
- Distributes traffic across API servers (round-robin)
- Terminates SSL/TLS
- Rate limiting (1000 req/min per IP)

**API Server (Node.js + Express)**:
- Handles HTTP requests
- Authentication/authorization (JWT)
- Business logic
- Publishes jobs to RabbitMQ

**Redis Cache**:
- Session storage (JWT tokens)
- Query result caching (TTL: 5 min)
- Rate limit counters

**RabbitMQ Queue**:
- Async job processing (reports, exports)
- Decouples API from heavy workloads

**Worker Processes (Python)**:
- Consumes jobs from RabbitMQ
- Generates reports, processes data
- Uploads results to S3

**PostgreSQL Database**:
- Primary data store (users, projects, analytics)
- ACID transactions
- Read replicas for analytics queries

**S3 Bucket**:
- Stores generated reports, exports
- CloudFront CDN for fast delivery

Pattern 2: Sequence Diagram for Complex Workflows

Use case: Illustrating multi-step interactions

## Sequence Diagram: User Signup Flow

Client API Server Database SendGrid │ │ │ │ │ POST /signup │ │ │ ├────────────────>│ │ │ │ │ │ │ │ │ Validate │ │ │ │ email/password│ │ │ │ │ │ │ │ Check if │ │ │ │ email exists │ │ │ ├──────────────>│ │ │ │<──────────────┤ │ │ │ (not found) │ │ │ │ │ │ │ │ Create user │ │ │ ├──────────────>│ │ │ │<──────────────┤ │ │ │ (user_id) │ │ │ │ │ │ │ │ Generate │ │ │ │ verify token │ │ │ │ │ │ │ │ Send verification email │ │ ├─────────────────────────────>│ │ │<─────────────────────────────┤ │ │ (200 OK) │ │ │ │ │ │ 201 Created │ │ │ │<────────────────┤ │ │ │ {user_id, ...} │ │ │ │ │ │ │ │ │ │ Email │ │ │ │ delivered │ │ │ │ to user │ │ │ │ │


### Error Scenario: Email Already Exists

Client API Server Database │ │ │ │ POST /signup │ │ ├────────────────>│ │ │ │ Check email │ │ ├──────────────>│ │ │<──────────────┤ │ │ (found!) │ │ │ │ │ 400 Bad Req │ │ │<────────────────┤ │ │ {error_code: │ │ │ EMAIL_IN_USE} │ │

Pattern 3: Trade-Off Matrix for Alternatives

Use case: Comparing technical approaches objectively

## Trade-Off Analysis: Caching Strategy

### Options Evaluated

| Criterion | Redis (In-Memory) | Postgres (DB Cache) | Memcached | CDN (CloudFront) |
|-----------|-------------------|---------------------|-----------|------------------|
| **Latency** | <1ms (local) | ~10ms (network) | <1ms (local) | ~50ms (edge) |
| **Scalability** | Horizontal (sharding) | Vertical (limited) | Horizontal | Infinite |
| **Persistence** | Optional (RDB/AOF) | Yes (durable) | No (memory only) | No (edge cache) |
| **Complexity** | Medium (setup cluster) | Low (already have DB) | Low (simple) | Medium (CDN config) |
| **Cost** | $50/mo (10GB instance) | $0 (included) | $30/mo (5GB) | $0.02/GB (traffic) |
| **Use Case Fit** | Session + query cache | Small datasets | Session only | Static assets |
| **Max TTL** | Custom (hours) | Custom (hours) | 30 days | 1 year |

### Decision: Redis

**Chosen**: Redis (in-memory cache with persistence)

**Rationale**:
- **Latency**: <1ms meets requirement (<10ms)
- **Scalability**: Can shard by cache key if needed (10k→100k users)
- **Persistence**: RDB snapshots prevent cache stampede on restart
- **Cost**: $50/mo acceptable for performance gain (vs $0 DB cache, but 10x slower)
- **Flexibility**: Supports sessions, query cache, rate limiting, pub/sub

**Runner-Up**: Postgres materialized views (for read-heavy analytics only)

**Rejected**:
- **Memcached**: No persistence (unacceptable for rate limiting data)
- **CDN**: Only for static assets (API responses are dynamic)

Pattern 4: API Design with Versioning

Use case: Designing RESTful API with backwards compatibility

## API Design: User Management

### Base URL
- Production: `https://api.example.com`
- Staging: `https://api-staging.example.com`

### Versioning Strategy
- URI versioning: `/v1/users`, `/v2/users`
- Current version: `v1`
- Deprecation policy: 12 months notice, 6 months grace period

### Endpoints

#### `POST /v1/users` - Create User
**Request**:

{ "email": "user@example.com", "password": "SecurePass1!", "display_name": "John Doe" // optional }


**Response (201 Created)**:

{ "user_id": "usr_abc123", "email": "user@example.com", "display_name": "John Doe", "created_at": "2025-10-25T14:30:00Z" }


**Response (400 Bad Request)**:

{ "error_code": "VALIDATION_ERROR", "error_message": "Invalid input", "request_id": "req_xyz789", "field_errors": { "email": "Email already in use", "password": "Password must include a number" } }


**Rate Limiting**:
- 5 signups per IP per hour
- Headers: `X-RateLimit-Limit: 5`, `X-RateLimit-Remaining: 3`

#### `GET /v1/users/:id` - Get User Profile
**Authentication**: Required (JWT in `Authorization: Bearer <token>`)

**Response (200 OK)**:

{ "user_id": "usr_abc123", "email": "user@example.com", "display_name": "John Doe", "profile_photo_url": "https://cdn.example.com/photos/abc123.jpg", "created_at": "2025-10-25T14:30:00Z", "preferences": { "theme": "dark", "notifications": true } }


**Response (404 Not Found)**:

{ "error_code": "USER_NOT_FOUND", "error_message": "User with ID usr_abc123 not found", "request_id": "req_xyz789" }


### Breaking Change Example (v1 → v2)

**v1**: Returns `created_at` as Unix timestamp (integer)

{"created_at": 1698249600}


**v2**: Returns `created_at` as ISO 8601 string (breaking change)

{"created_at": "2025-10-25T14:30:00Z"}


**Migration Plan**:
1. Announce deprecation: 2025-11-01 (email, docs, changelog)
2. Grace period: v1 supported until 2026-05-01 (6 months)
3. Sunset: v1 returns 410 Gone after 2026-05-01

Pattern 5: Database Schema Design with Migrations

Use case: Planning schema changes with rollback strategy

## Database Schema: User Authentication

### Current Schema (v1)

CREATE TABLE users ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), email VARCHAR(255) UNIQUE NOT NULL, password_hash VARCHAR(255) NOT NULL, created_at TIMESTAMP DEFAULT NOW() );


### Proposed Schema (v2) - Add Email Verification

-- Migration: Add email verification columns ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE, ADD COLUMN email_verify_token VARCHAR(255), ADD COLUMN email_verify_sent_at TIMESTAMP;

-- Create index for fast token lookup CREATE INDEX idx_users_verify_token ON users(email_verify_token) WHERE email_verify_token IS NOT NULL;

-- Backfill existing users (mark as verified) UPDATE users SET email_verified = TRUE WHERE created_at < NOW();


### Migration Plan

**Step 1: Deploy Schema Change** (Zero Downtime)
- Run `ALTER TABLE` during low-traffic window
- PostgreSQL locks table briefly (<1s for 100k rows)
- Backfill existing users as verified (no behavior change)

**Step 2: Deploy Code Changes**
- New signups: Set `email_verified = FALSE`, send verification email
- Login: Check `email_verified`, block if false
- Feature flag: `REQUIRE_EMAIL_VERIFICATION` (gradual rollout)

**Step 3: Monitor**
- Track verification rate (target: 80% verify within 24 hours)
- Monitor support tickets (expect questions)
- Alert if verification email delivery rate <95%

**Rollback Plan**:
- Revert code: Disable `REQUIRE_EMAIL_VERIFICATION` flag
- Keep schema changes (no need to drop columns, just stop using)
- If catastrophic: Drop columns (loses verification data)

Pattern 6: Migration Strategy (Old System → New System)

Use case: Transitioning between systems with zero downtime

## Migration: Monolith → Microservices (Auth Service)

### Current State
- Monolithic Rails app handles everything (auth, users, projects)
- Auth logic mixed with business logic (tight coupling)
- 100k active users, 10k requests/min peak

### Target State
- Standalone Auth Service (Node.js + JWT)
- Microservices for users, projects (separate from auth)
- Same 100k users, seamless migration

### Migration Strategy: Strangler Fig Pattern

**Phase 1: Build Auth Service (Parallel Run)**
- Build new Auth Service (JWT-based)
- Dual-write: Monolith writes to both old session DB and new Auth Service
- Read from old system only (no behavior change)
- Duration: 2 weeks

**Phase 2: Shadow Traffic**
- Route 1% of login requests to Auth Service (shadow mode)
- Compare results with monolith (validate correctness)
- Fix discrepancies, increase to 10%, then 50%
- Duration: 2 weeks

**Phase 3: Gradual Cutover**
- Route 10% of users to Auth Service (read + write)
- Monitor latency, error rates, user complaints
- Increase to 50%, then 100% over 4 weeks
- Duration: 4 weeks

**Phase 4: Monolith Deprecation**
- All auth traffic goes to Auth Service
- Remove auth code from monolith (dead code)
- Decommission old session database
- Duration: 2 weeks

### Rollback Plan
- Feature flag: `USE_AUTH_SERVICE` (toggle per user)
- If Auth Service fails, revert flag → monolith
- Monitoring: Alert if Auth Service latency >100ms or error rate >1%

### Metrics
- **Latency**: Auth Service <50ms (vs monolith 80ms)
- **Error Rate**: <0.1% (vs monolith 0.2%)
- **User Impact**: Zero failed logins during migration

Pattern 7: Performance Benchmarking

Use case: Quantifying trade-offs with real data

## Performance Benchmark: Caching Strategy

### Test Setup
- Dataset: 1M user records
- Query: `GET /users/:id/profile` (fetch user + projects)
- Load: 1,000 requests/sec (sustained)
- Tool: Apache Bench (`ab -n 10000 -c 100`)

### Results

| Approach | p50 Latency | p95 Latency | p99 Latency | Throughput | CPU Usage |
|----------|-------------|-------------|-------------|------------|-----------|
| **No Cache** (baseline) | 120ms | 350ms | 850ms | 800 req/s | 80% |
| **Redis Cache** | 8ms | 15ms | 45ms | 8,000 req/s | 20% |
| **Postgres Materialized View** | 25ms | 60ms | 150ms | 3,000 req/s | 40% |
| **In-Memory Map** (Node.js) | 2ms | 5ms | 12ms | 10,000 req/s | 30% |

### Analysis

**Winner**: Redis Cache (chosen)
- **Latency**: 15x faster than no cache (p95: 15ms vs 350ms) ✅
- **Throughput**: 10x higher (8k vs 800 req/s) ✅
- **Cost**: $50/mo for Redis instance (acceptable)
- **Risk**: Single point of failure (mitigated with Redis Sentinel)

**Runner-Up**: In-Memory Map
- **Faster**: 2ms latency, 10k req/s
- **Problem**: No persistence (lost on restart), no sharing across servers
- **Verdict**: Only viable for single-server deployment

**Rejected**: Postgres Materialized View
- **Reason**: Still 25ms latency (vs <10ms requirement)
- **Use Case**: Good for analytics, not real-time API

Pattern 8: Failure Mode Analysis

Use case: Anticipating what can go wrong

## Failure Mode Analysis: Payment Processing

### Failure Scenarios

#### 1. Stripe API Down
**Probability**: Low (99.99% uptime SLA)
**Impact**: High (no payments processed)
**Mitigation**:
- Detect: Health check every 30s, alert if 3 failures
- Respond: Show maintenance page, queue payment attempts
- Recover: Retry queued payments when Stripe recovers

#### 2. Database Connection Pool Exhausted
**Probability**: Medium (under high load)
**Impact**: High (API returns 500 errors)
**Mitigation**:
- Prevent: Set connection pool limit (50 connections)
- Detect: Monitor active connections (alert if >80%)
- Respond: Reject new requests with 503 (retry-after header)
- Recover: Scale database read replicas, optimize queries

#### 3. Redis Cache Eviction
**Probability**: Medium (if cache fills up)
**Impact**: Low (fallback to database, slower but works)
**Mitigation**:
- Prevent: Set max memory limit, use LRU eviction policy
- Detect: Monitor cache hit rate (alert if <80%)
- Respond: Cache miss → fetch from DB, repopulate cache
- Recover: Increase Redis memory, optimize cache keys

#### 4. Webhook Delivery Failure
**Probability**: High (network issues, client downtime)
**Impact**: Medium (events not processed by client)
**Mitigation**:
- Detect: HTTP status != 200 or timeout (>30s)
- Respond: Exponential backoff retry (1s, 2s, 4s, 8s, 16s)
- Recover: Dead letter queue after 5 failures, alert for manual review

### Circuit Breaker Pattern

// Protect against cascading failures class CircuitBreaker { constructor(failureThreshold = 5, timeout = 60000) { this.failureCount = 0; this.failureThreshold = failureThreshold; this.timeout = timeout; this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN }

async call(fn) { if (this.state === 'OPEN') { throw new Error('Circuit breaker is OPEN'); }

try { const result = await fn(); this.onSuccess(); return result; } catch (error) { this.onFailure(); throw error; } }

onSuccess() { this.failureCount = 0; this.state = 'CLOSED'; }

onFailure() { this.failureCount++; if (this.failureCount >= this.failureThreshold) { this.state = 'OPEN'; setTimeout(() => { this.state = 'HALF_OPEN'; }, this.timeout); } } }

// Usage: Protect Stripe API calls const stripeBreaker = new CircuitBreaker(5, 60000); const charge = await stripeBreaker.call(() => stripe.charges.create({...}));

Quick Reference

Architecture Diagram Types

Diagram Type | Use Case | Audience
-------------|----------|----------
System Context (C4-1) | High-level overview | Leadership, PM
Container (C4-2) | Major components | Architects, senior eng
Component (C4-3) | Internal structure | Engineers
Sequence | Multi-step workflows | Engineers debugging
Data Flow | Data movement | Data engineers

Trade-Off Evaluation Checklist

Dimension | Question to Ask
----------|----------------
Performance | What is p95 latency? Throughput?
Scalability | How does it scale? (horizontal/vertical)
Complexity | How long to implement? Maintain?
Cost | $ per month at 10k users? At 100k?
Reliability | What's uptime SLA? Failure modes?
Maintainability | How easy to debug? Extend?

API Versioning Strategies

Strategy | Example | Pros | Cons
---------|---------|------|-----
URI | /v1/users | Simple, explicit | URL proliferation
Header | Accept: v=1 | Clean URLs | Less visible
Query Param | /users?v=1 | Easy to test | Ugly URLs

Key Guidelines

✅ DO: Quantify trade-offs (numbers, not adjectives)
✅ DO: Create diagrams for complex architectures
✅ DO: Benchmark alternatives with real data
✅ DO: Plan failure modes and mitigations
✅ DO: Design APIs with backwards compatibility

❌ DON'T: Pick solutions without evaluating alternatives
❌ DON'T: Use vague terms ("fast", "scalable") without metrics
❌ DON'T: Ignore migration complexity
❌ DON'T: Skip performance benchmarking
❌ DON'T: Design APIs without versioning strategy

Anti-Patterns

Critical Violations

❌ No Alternatives Analysis: Presenting one solution without comparison

# ❌ NEVER:
We'll use MongoDB for the database.

# ✅ CORRECT:
## Database Options
1. **PostgreSQL**: ACID, relations, complex queries (chosen)
2. **MongoDB**: Flexible schema, horizontal scaling
   - Rejected: No ACID for financial transactions (deal-breaker)
3. **DynamoDB**: Managed, infinite scale
   - Rejected: Vendor lock-in, limited query flexibility

**Decision**: Postgres for ACID guarantees and query richness.

❌ Premature Optimization: Optimizing without measuring ✅ Correct approach: Benchmark current performance, identify bottleneck, optimize

Common Mistakes

❌ Vague Performance Claims: "Redis is faster than Postgres"

# ❌ Don't:
Redis is faster, so we'll use it for caching.

# ✅ Correct:
## Benchmark Results
- Redis: 8ms p95 latency, 8k req/s throughput
- Postgres: 120ms p95 latency, 800 req/s throughput
- **Decision**: Redis for 15x latency improvement

❌ No Failure Analysis: Assuming happy path only ✅ Better: Document failure modes, circuit breakers, fallbacks

❌ API Without Versioning: No plan for breaking changes ✅ Better: URI versioning, deprecation policy, migration guide

❌ Big Bang Migration: Switching all users at once ✅ Better: Strangler fig pattern (gradual cutover with rollback plan)

Related Skills

Last Updated: 2025-10-25 Format Version: 1.0 (Atomic)