observability-distributed-tracing

Debugging latency issues in microservices

Distributed Tracing

Scope: OpenTelemetry, spans, trace context, sampling, Jaeger, Zipkin, trace propagation

Lines: 381

Last Updated: 2025-10-18

---

When to Use This Skill

Use this skill when:

• Debugging latency issues in microservices
• Understanding request flows across multiple services
• Identifying bottlenecks in distributed systems
• Tracking dependencies between services
• Measuring end-to-end latency percentiles
• Debugging intermittent errors in complex workflows
• Implementing service mesh observability
• Analyzing cascading failures

Don't use for:

• Simple monolithic applications (logs + metrics sufficient)
• Non-request-based workloads (batch jobs, cron tasks)
• Systems with <3 services (overhead not justified)

---

Core Concepts

Trace, Span, Context

Trace: Complete journey of a request through the system

• Unique Trace ID (e.g., a1b2c3d4e5f6g7h8)
• Contains multiple spans
• Represents end-to-end flow

Span: Single operation within a trace

• Unique Span ID
• Parent Span ID (for hierarchy)
• Start time, end time, duration
• Operation name (e.g., GET /api/users)
• Attributes (metadata: http.method, db.statement)
• Events (logs within span)
• Status (OK, ERROR)

Context: Propagated metadata

• Trace ID, Span ID, Parent Span ID
• Sampling decision
• Baggage (custom key-value pairs)

Trace Hierarchy

Trace: User purchases product
├─ Span 1: API Gateway (100ms)
│  ├─ Span 2: Auth Service (20ms)
│  │  └─ Span 3: Database query (5ms)
│  ├─ Span 4: Product Service (50ms)
│  │  ├─ Span 5: Database query (10ms)
│  │  └─ Span 6: Cache lookup (5ms)
│  └─ Span 7: Payment Service (30ms)
│     └─ Span 8: External API call (25ms)

Sampling

Why: Reduce data volume and cost (storing 100% traces is expensive)

Strategies:

1. Head-based sampling (decision at trace start):

• Probabilistic: 1% of traces
• Rate limiting: Max 100 traces/second
• Always sample errors

1. Tail-based sampling (decision after trace complete):

• Sample slow requests (>1s)
• Sample errors
• Sample rare endpoints

OpenTelemetry (OTel)

Why OpenTelemetry: Vendor-neutral, unified API for traces, metrics, logs

Components:

• SDK: Instrumentation in your code
• API: Interface for creating spans
• Exporter: Send data to backend (Jaeger, Zipkin, Tempo)
• Collector: Centralized agent for receiving/processing/exporting

---

Patterns

Pattern 1: OpenTelemetry Setup (Python)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from fastapi import FastAPI

# Initialize tracer provider
provider = TracerProvider()
trace.set_tracer_provider(provider)

# Configure exporter (send to OTel Collector or Jaeger)
otlp_exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # gRPC endpoint
    insecure=True
)

# Add span processor
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))

# Get tracer
tracer = trace.get_tracer(__name__)

# Auto-instrument FastAPI
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

# Auto-instrument HTTP requests
RequestsInstrumentor().instrument()

# Auto-instrument SQLAlchemy
SQLAlchemyInstrumentor().instrument()

# Manual span creation
@app.get("/api/users/{user_id}")
async def get_user(user_id: str):
    # Parent span created automatically by FastAPI instrumentation

    # Create child span for business logic
    with tracer.start_as_current_span("get_user_logic") as span:
        span.set_attribute("user_id", user_id)
        span.set_attribute("service", "user-service")

        try:
            # Database call (auto-instrumented)
            user = await db.get_user(user_id)

            # Add event
            span.add_event("user_fetched", {"user_id": user_id})

            # Create nested span for external API
            with tracer.start_as_current_span("fetch_user_preferences"):
                prefs = await external_api.get_preferences(user_id)

            span.set_status(trace.Status(trace.StatusCode.OK))
            return {"user": user, "preferences": prefs}

        except Exception as e:
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise

Pattern 2: OpenTelemetry Setup (Go)

package main

import (
    "context"
    "log"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
    "go.opentelemetry.io/otel/trace"
)

func initTracer() (*sdktrace.TracerProvider, error) {
    // Create OTLP exporter
    exporter, err := otlptracegrpc.New(
        context.Background(),
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    // Create resource (service metadata)
    res, err := resource.New(
        context.Background(),
        resource.WithAttributes(
            semconv.ServiceNameKey.String("api-service"),
            semconv.ServiceVersionKey.String("1.0.0"),
            semconv.DeploymentEnvironmentKey.String("production"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Create tracer provider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)

    return tp, nil
}

func main() {
    tp, err := initTracer()
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Fatal(err)
        }
    }()

    // Get tracer
    tracer := otel.Tracer("api-service")

    // Create root span
    ctx, span := tracer.Start(context.Background(), "handle_request")
    defer span.End()

    span.SetAttributes(
        attribute.String("http.method", "GET"),
        attribute.String("http.route", "/api/users/:id"),
    )

    // Create child span
    processUser(ctx, tracer, "user_123")

    span.SetStatus(trace.StatusCode(200), "OK")
}

func processUser(ctx context.Context, tracer trace.Tracer, userID string) {
    ctx, span := tracer.Start(ctx, "process_user")
    defer span.End()

    span.SetAttributes(attribute.String("user_id", userID))

    // Simulate work
    time.Sleep(50 * time.Millisecond)

    // Nested span
    fetchFromDB(ctx, tracer, userID)
}

func fetchFromDB(ctx context.Context, tracer trace.Tracer, userID string) {
    ctx, span := tracer.Start(ctx, "database_query")
    defer span.End()

    span.SetAttributes(
        attribute.String("db.system", "postgresql"),
        attribute.String("db.statement", "SELECT * FROM users WHERE id = ?"),
    )

    // Simulate query
    time.Sleep(10 * time.Millisecond)
}

Pattern 3: Trace Context Propagation (HTTP)

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
import requests

tracer = trace.get_tracer(__name__)

# Service A: Outgoing request
def call_service_b():
    with tracer.start_as_current_span("call_service_b") as span:
        headers = {}

        # Inject trace context into HTTP headers
        inject(headers)

        # Headers now contain:
        # traceparent: 00-<trace_id>-<span_id>-01
        # tracestate: ...

        response = requests.get(
            "http://service-b/api/data",
            headers=headers
        )

        return response.json()

# Service B: Incoming request
from fastapi import FastAPI, Request

app = FastAPI()

@app.get("/api/data")
async def get_data(request: Request):
    # Extract trace context from incoming headers
    ctx = extract(request.headers)

    # Continue trace from Service A
    with tracer.start_as_current_span("get_data", context=ctx) as span:
        span.set_attribute("service", "service-b")

        data = await fetch_data()
        return {"data": data}

Pattern 4: Custom Span Attributes

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

@app.post("/api/orders")
async def create_order(order: Order):
    with tracer.start_as_current_span("create_order") as span:
        # Standard semantic conventions
        span.set_attribute("http.method", "POST")
        span.set_attribute("http.route", "/api/orders")
        span.set_attribute("http.status_code", 201)

        # Custom business attributes
        span.set_attribute("order.id", order.id)
        span.set_attribute("order.total", order.total)
        span.set_attribute("order.items_count", len(order.items))
        span.set_attribute("customer.id", order.customer_id)
        span.set_attribute("payment.method", order.payment_method)

        # Add events (logs within span)
        span.add_event("order_validated", {
            "validation_duration_ms": 10
        })

        try:
            result = await process_order(order)

            span.add_event("payment_processed", {
                "transaction_id": result.transaction_id
            })

            span.set_status(Status(StatusCode.OK))
            return result

        except PaymentError as e:
            span.set_status(Status(StatusCode.ERROR, "Payment failed"))
            span.record_exception(e)

            # Add error attributes
            span.set_attribute("error", True)
            span.set_attribute("error.type", "PaymentError")
            span.set_attribute("error.message", str(e))

            raise

Pattern 5: Sampling Configuration

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import (
    TraceIdRatioBased,
    ParentBased,
    ALWAYS_ON,
    ALWAYS_OFF
)

# 1. Probabilistic sampling (1% of traces)
sampler = TraceIdRatioBased(0.01)

# 2. Parent-based sampling (respect parent's decision)
sampler = ParentBased(root=TraceIdRatioBased(0.1))

# 3. Always sample (development)
sampler = ALWAYS_ON

# 4. Never sample (disabled)
sampler = ALWAYS_OFF

# 5. Custom sampling (sample errors and slow requests)
class CustomSampler:
    def should_sample(self, context, trace_id, name, attributes):
        # Always sample if error
        if attributes.get("http.status_code", 0) >= 500:
            return ALWAYS_ON.should_sample(context, trace_id, name, attributes)

        # Always sample if slow
        if attributes.get("duration_ms", 0) > 1000:
            return ALWAYS_ON.should_sample(context, trace_id, name, attributes)

        # Otherwise 1% sampling
        return TraceIdRatioBased(0.01).should_sample(
            context, trace_id, name, attributes
        )

provider = TracerProvider(sampler=CustomSampler())

Pattern 6: Database Tracing

from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from sqlalchemy import create_engine

# Auto-instrument SQLAlchemy
engine = create_engine("postgresql://localhost/mydb")
SQLAlchemyInstrumentor().instrument(engine=engine)

# Manual span for complex queries
with tracer.start_as_current_span("complex_query") as span:
    span.set_attribute("db.system", "postgresql")
    span.set_attribute("db.name", "mydb")
    span.set_attribute("db.operation", "SELECT")
    span.set_attribute("db.statement", "SELECT * FROM users WHERE status = ?")

    results = session.execute(query)

    span.set_attribute("db.rows_returned", len(results))

---

Quick Reference

OpenTelemetry Libraries

# Python
pip install opentelemetry-api opentelemetry-sdk
pip install opentelemetry-exporter-otlp
pip install opentelemetry-instrumentation-fastapi
pip install opentelemetry-instrumentation-requests
pip install opentelemetry-instrumentation-sqlalchemy

# Go
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

# Node.js
npm install @opentelemetry/api @opentelemetry/sdk-node
npm install @opentelemetry/exporter-trace-otlp-grpc
npm install @opentelemetry/auto-instrumentations-node

Trace Backend Options

| Backend | Description | Use Case | |---------|-------------|----------| | Jaeger | CNCF project, full-featured | Self-hosted, open source | | Zipkin | Twitter origin, simple | Lightweight, easy setup | | Tempo | Grafana Labs, S3-compatible | Cost-effective, high volume | | Datadog APM | Commercial SaaS | Enterprise, full stack | | New Relic | Commercial SaaS | Enterprise, full stack |

Span Semantic Conventions

# HTTP
span.set_attribute("http.method", "GET")
span.set_attribute("http.url", "https://api.example.com/users")
span.set_attribute("http.status_code", 200)
span.set_attribute("http.route", "/users/:id")

# Database
span.set_attribute("db.system", "postgresql")
span.set_attribute("db.name", "mydb")
span.set_attribute("db.statement", "SELECT * FROM users")
span.set_attribute("db.operation", "SELECT")

# RPC
span.set_attribute("rpc.system", "grpc")
span.set_attribute("rpc.service", "UserService")
span.set_attribute("rpc.method", "GetUser")

# Messaging
span.set_attribute("messaging.system", "rabbitmq")
span.set_attribute("messaging.destination", "orders_queue")
span.set_attribute("messaging.operation", "publish")

Trace Context Headers

# W3C Trace Context (standard)
traceparent: 00-<trace_id>-<span_id>-<flags>
tracestate: vendor1=value1,vendor2=value2

# Example
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

---

Anti-Patterns

❌ Creating Too Many Spans

# WRONG: Span for every function call
with tracer.start_as_current_span("add_numbers"):  # Too granular!
    result = a + b

# CORRECT: Span for meaningful operations
with tracer.start_as_current_span("calculate_order_total"):
    total = sum(item.price for item in order.items)
    total += calculate_tax(total)
    total += calculate_shipping(order)

❌ Missing Trace Context Propagation

# WRONG: No trace context in HTTP call
requests.get("http://service-b/api/data")  # Breaks trace!

# CORRECT: Inject trace context
headers = {}
inject(headers)
requests.get("http://service-b/api/data", headers=headers)

❌ Forgetting to Close Spans

# WRONG: Span never closed
span = tracer.start_span("operation")
do_work()
# span.end() missing!

# CORRECT: Use context manager
with tracer.start_as_current_span("operation"):
    do_work()

❌ High-Cardinality Attributes

# WRONG: User ID in span name (high cardinality)
span = tracer.start_as_current_span(f"get_user_{user_id}")

# CORRECT: User ID as attribute
span = tracer.start_as_current_span("get_user")
span.set_attribute("user_id", user_id)

❌ Sampling Before Instrumentation

# WRONG: Sample before seeing attributes
sampler = TraceIdRatioBased(0.01)  # Can't see if request is slow/error

# CORRECT: Use tail-based sampling or custom sampler
class TailBasedSampler:
    def should_sample(self, context, trace_id, name, attributes):
        if attributes.get("http.status_code", 0) >= 500:
            return ALWAYS_ON
        return TraceIdRatioBased(0.01).should_sample(...)

❌ Blocking Operations in Spans

# WRONG: Synchronous sleep in async span
with tracer.start_as_current_span("wait"):
    time.sleep(10)  # Blocks event loop!

# CORRECT: Async sleep
with tracer.start_as_current_span("wait"):
    await asyncio.sleep(10)

---

Level 3: Resources

Location: skills/observability/distributed-tracing/resources/

Comprehensive Reference

REFERENCE.md (800+ lines): Deep-dive reference covering:

• Distributed tracing fundamentals (traces, spans, context propagation)
• OpenTelemetry specification and architecture (API, SDK, Collector, OTLP)
• Trace context propagation formats (W3C Trace Context, B3, Jaeger, X-Ray)
• Sampling strategies (head-based, tail-based, adaptive sampling)
• Tracing backends comparison (Jaeger, Zipkin, Tempo, Honeycomb, Datadog)
• Instrumentation patterns (auto-instrumentation vs manual spans)
• Performance overhead analysis and optimization techniques
• Correlation with logs and metrics (exemplars, unified queries)
• Production patterns (service mesh integration, multi-tenant tracing, error tracking)
• Common anti-patterns and troubleshooting guide

Executable Scripts

scripts/analyze_traces.py

• Parse and analyze trace data from JSON files (OTLP, Jaeger, Zipkin formats)
• Identify slow spans exceeding configurable threshold
• Detect error spans and analyze failure patterns
• Calculate critical path and total duration
• Service breakdown and span statistics
• Supports filtering by service, trace ID
• Output as human-readable report or JSON for automation

scripts/visualize_trace.py

• Generate HTML Gantt chart visualizations of traces
• Create SVG flame graphs for performance analysis
• Interactive visualizations with zoom, pan, and tooltips
• Show span hierarchy and timing relationships
• Color-coded by span status (OK, ERROR, UNSET)
• Support for multiple trace formats

scripts/test_propagation.sh

• Test trace context propagation across HTTP services
• Validate W3C Trace Context, B3, Jaeger, and X-Ray header formats
• Multi-hop propagation testing (service A → B → C)
• Verify trace ID preservation across boundaries
• JSON output for CI/CD integration
• Configurable propagation format and target URLs

Working Examples

examples/python/otel_fastapi_tracing.py

• Complete FastAPI service with OpenTelemetry instrumentation
• Auto-instrumentation for HTTP, database, and external calls
• Manual span creation for business logic
• Error handling and exception recording
• Span attributes, events, and status management
• OTLP exporter configuration

examples/python/otel_manual_spans.py

• Comprehensive manual span creation patterns
• Nested spans and parent-child relationships
• Different span kinds (CLIENT, SERVER, PRODUCER, CONSUMER, INTERNAL)
• Async context propagation with asyncio
• Span links for cross-trace references
• Batch processing patterns
• Error handling and retry logic

examples/typescript/otel_express_tracing.ts

• Express.js service with OpenTelemetry auto-instrumentation
• Manual span creation for complex operations
• Trace context propagation to downstream services
• Multi-step operations (order fulfillment workflow)
• Event tracking within spans
• Error handling middleware with tracing

examples/go/otel_http_tracing.go

• Go HTTP server with OpenTelemetry instrumentation
• otelhttp middleware for automatic span creation
• Manual span creation for business logic
• OTLP gRPC exporter configuration
• Span attributes using semantic conventions
• Producer spans for message publishing

examples/docker/jaeger-compose.yml

• Complete Docker Compose setup for distributed tracing
• Jaeger all-in-one (collector + query + UI)
• OpenTelemetry Collector with advanced processing
• Example microservices with tracing enabled
• Network configuration for inter-service communication

examples/docker/otel-collector-config.yaml

• Production-ready OpenTelemetry Collector configuration
• OTLP and Jaeger receivers
• Tail sampling processor for smart sampling
• Memory limiter and batch processing
• Resource detection and attribute processing
• Multiple exporters (Jaeger, OTLP, logging, Prometheus)
• Health check and diagnostics extensions

Usage Examples

# Analyze traces for slow operations
./scripts/analyze_traces.py --file traces.json --slow-threshold 500

# Generate interactive trace visualization
./scripts/visualize_trace.py --file traces.json --interactive --output trace.html

# Test trace propagation
./scripts/test_propagation.sh --url http://localhost:8000 --format w3c

# Run example services
python examples/python/otel_fastapi_tracing.py
ts-node examples/typescript/otel_express_tracing.ts
go run examples/go/otel_http_tracing.go

# Start tracing infrastructure
docker-compose -f examples/docker/jaeger-compose.yml up -d
# Access Jaeger UI: http://localhost:16686

---

Related Skills

• structured-logging.md - Correlate logs with trace IDs
• metrics-instrumentation.md - Complementary metrics (RED method)
• dashboard-design.md - Visualize traces in Grafana
• alerting-strategy.md - Alert on trace error rates and latency

---

Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)