observability-prometheus-monitoring

Production-ready Prometheus monitoring and alerting

Prometheus Monitoring

Scope: Prometheus, PromQL, recording rules, alerting rules, exporters, service discovery, federation

Lines: 650

Last Updated: 2025-10-27

---

When to Use This Skill

Use this skill when:

• Setting up Prometheus for infrastructure or application monitoring
• Writing PromQL queries for metrics analysis
• Configuring recording rules for query optimization
• Creating alerting rules for proactive monitoring
• Implementing custom exporters for application metrics
• Configuring service discovery for dynamic environments
• Setting up Prometheus federation for multi-cluster monitoring
• Optimizing Prometheus performance and storage
• Debugging high cardinality issues
• Integrating Prometheus with Grafana or Alertmanager

Don't use for:

• Log aggregation (use Loki or ELK instead)
• Distributed tracing (use Jaeger or Tempo instead)
• APM with transaction profiling (use dedicated APM tools)

---

Core Concepts

Prometheus Architecture

Pull-Based Model: Prometheus scrapes metrics from targets

• Targets expose metrics at HTTP endpoints (typically /metrics)
• Prometheus pulls metrics at configured intervals (default: 15s)
• Targets discovered via static config or service discovery
• Pushgateway available for batch jobs that can't be scraped

Components:

• Prometheus Server: Time-series database + query engine
• Exporters: Expose metrics from third-party systems
• Alertmanager: Handle alerts (deduplication, grouping, routing)
• Pushgateway: Receive metrics from short-lived jobs
• Client Libraries: Instrument applications

Data Model

Metrics: Named time series with key-value labels

http_requests_total{method="GET", endpoint="/api/users", status="200"} 1234
|                  |                                                    |
metric name        labels (dimensions)                               value

Metric Types:

1. Counter: Monotonically increasing (requests, errors, bytes)
2. Gauge: Can go up/down (temperature, memory, active connections)
3. Histogram: Distribution in buckets (latency, request size)
4. Summary: Similar to histogram, client-side quantiles

PromQL Basics

Selectors:

# Instant vector (one value per series at query time)
http_requests_total

# Filter by labels
http_requests_total{status="200", method="GET"}

# Regex matching
http_requests_total{status=~"2.."}

# Negative matching
http_requests_total{status!="200"}

# Range vector (values over time window)
http_requests_total[5m]

Functions:

# Rate: per-second average rate over time window
rate(http_requests_total[5m])

# Increase: total increase over time window
increase(http_requests_total[1h])

# Sum: aggregate across labels
sum(http_requests_total)

# Sum by label (keeps label)
sum by (status) (http_requests_total)

# Histogram quantile (percentile from histogram)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

---

Patterns

Pattern 1: Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    region: 'us-east-1'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

# Load rules
rule_files:
  - "rules/*.yml"
  - "alerts/*.yml"

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter (system metrics)
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):\d+'
        target_label: instance
        replacement: '${1}'

  # Kubernetes service discovery
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

  # Consul service discovery
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: job

  # File-based service discovery
  - job_name: 'file-sd'
    file_sd_configs:
      - files:
          - 'targets/*.json'
          - 'targets/*.yml'
        refresh_interval: 30s

Pattern 2: Recording Rules

# rules/recording-rules.yml
groups:
  - name: api_performance
    interval: 30s
    rules:
      # Precompute request rate per endpoint
      - record: api:http_requests:rate5m
        expr: |
          sum by (job, endpoint, method) (
            rate(http_requests_total[5m])
          )

      # Precompute error rate percentage
      - record: api:http_errors:rate5m
        expr: |
          sum by (job, endpoint) (
            rate(http_requests_total{status=~"5.."}[5m])
          ) / sum by (job, endpoint) (
            rate(http_requests_total[5m])
          ) * 100

      # Precompute latency percentiles
      - record: api:http_latency:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, endpoint, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

      - record: api:http_latency:p99
        expr: |
          histogram_quantile(0.99,
            sum by (job, endpoint, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

  - name: resource_utilization
    interval: 30s
    rules:
      # CPU utilization percentage
      - record: node:cpu:utilization
        expr: |
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          ) * 100)

      # Memory utilization percentage
      - record: node:memory:utilization
        expr: |
          100 * (1 - (
            node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
          ))

      # Disk utilization percentage
      - record: node:disk:utilization
        expr: |
          100 - (
            node_filesystem_avail_bytes / node_filesystem_size_bytes * 100
          )

Pattern 3: Alerting Rules

# alerts/alerting-rules.yml
groups:
  - name: service_health
    interval: 15s
    rules:
      # Service down
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

      # High error rate
      - alert: HighErrorRate
        expr: |
          sum by (job, endpoint) (
            rate(http_requests_total{status=~"5.."}[5m])
          ) / sum by (job, endpoint) (
            rate(http_requests_total[5m])
          ) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate on {{ $labels.job }}/{{ $labels.endpoint }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum by (job, endpoint, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          ) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High p95 latency on {{ $labels.job }}/{{ $labels.endpoint }}"
          description: "p95 latency is {{ $value }}s"

  - name: resource_alerts
    interval: 15s
    rules:
      # High CPU usage
      - alert: HighCPU
        expr: |
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          ) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value }}%"

      # High memory usage
      - alert: HighMemory
        expr: |
          100 * (1 - (
            node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
          )) > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High memory on {{ $labels.instance }}"
          description: "Memory usage is {{ $value }}%"

      # Disk space low
      - alert: DiskSpaceLow
        expr: |
          100 - (
            node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.*"}
            / node_filesystem_size_bytes * 100
          ) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk {{ $labels.device }} is {{ $value }}% full"

Pattern 4: Custom Exporter (Python)

#!/usr/bin/env python3
from prometheus_client import start_http_server, Gauge, Counter, Histogram
import time
import random

# Define metrics
app_status = Gauge('app_status', 'Application status', ['app_name'])
requests_total = Counter('requests_total', 'Total requests', ['endpoint', 'status'])
request_duration = Histogram(
    'request_duration_seconds',
    'Request duration',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0]
)
queue_size = Gauge('queue_size', 'Current queue size', ['queue_name'])
active_connections = Gauge('active_connections', 'Active connections')

def collect_metrics():
    """Collect application metrics."""
    # Update status metrics
    app_status.labels(app_name='api-server').set(1)  # 1 = healthy

    # Simulate request metrics
    endpoints = ['/api/users', '/api/orders', '/api/products']
    statuses = ['200', '404', '500']

    for endpoint in endpoints:
        status = random.choice(statuses)
        requests_total.labels(endpoint=endpoint, status=status).inc()

        duration = random.uniform(0.01, 1.0)
        request_duration.labels(endpoint=endpoint).observe(duration)

    # Update gauge metrics
    queue_size.labels(queue_name='tasks').set(random.randint(0, 100))
    active_connections.set(random.randint(10, 500))

if __name__ == '__main__':
    # Start HTTP server for metrics endpoint
    start_http_server(8080)
    print("Exporter running on :8080/metrics")

    # Collect metrics periodically
    while True:
        collect_metrics()
        time.sleep(5)

Pattern 5: Custom Exporter (Go)

package main

import (
    "log"
    "math/rand"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    appStatus = promauto.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "app_status",
            Help: "Application status",
        },
        []string{"app_name"},
    )

    requestsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "requests_total",
            Help: "Total requests",
        },
        []string{"endpoint", "status"},
    )

    requestDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "request_duration_seconds",
            Help:    "Request duration",
            Buckets: []float64{0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0},
        },
        []string{"endpoint"},
    )

    queueSize = promauto.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "queue_size",
            Help: "Current queue size",
        },
        []string{"queue_name"},
    )

    activeConnections = promauto.NewGauge(
        prometheus.GaugeOpts{
            Name: "active_connections",
            Help: "Active connections",
        },
    )
)

func collectMetrics() {
    endpoints := []string{"/api/users", "/api/orders", "/api/products"}
    statuses := []string{"200", "404", "500"}

    for {
        // Update status
        appStatus.WithLabelValues("api-server").Set(1)

        // Simulate requests
        for _, endpoint := range endpoints {
            status := statuses[rand.Intn(len(statuses))]
            requestsTotal.WithLabelValues(endpoint, status).Inc()

            duration := rand.Float64()
            requestDuration.WithLabelValues(endpoint).Observe(duration)
        }

        // Update gauges
        queueSize.WithLabelValues("tasks").Set(float64(rand.Intn(100)))
        activeConnections.Set(float64(rand.Intn(490) + 10))

        time.Sleep(5 * time.Second)
    }
}

func main() {
    // Start metrics collector
    go collectMetrics()

    // Expose metrics endpoint
    http.Handle("/metrics", promhttp.Handler())
    log.Println("Exporter running on :8080/metrics")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Pattern 6: Federation

# prometheus.yml (central Prometheus)
scrape_configs:
  # Federate from regional Prometheus instances
  - job_name: 'federate-us-east'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="api-server"}'
        - '{__name__=~"api:.*"}'  # Aggregated metrics
    static_configs:
      - targets:
          - 'prometheus-us-east:9090'

  - job_name: 'federate-eu-west'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="api-server"}'
        - '{__name__=~"api:.*"}'
    static_configs:
      - targets:
          - 'prometheus-eu-west:9090'

---

Quick Reference

PromQL Operators

# Arithmetic
metric1 + metric2
metric1 - metric2
metric1 * metric2
metric1 / metric2
metric1 % metric2
metric1 ^ metric2

# Comparison (returns 0 or 1)
metric1 == metric2
metric1 != metric2
metric1 > metric2
metric1 < metric2
metric1 >= metric2
metric1 <= metric2

# Logical (on boolean results)
metric1 and metric2
metric1 or metric2
metric1 unless metric2

Aggregation Functions

sum(metric) by (label)           # Sum
avg(metric) by (label)           # Average
min(metric) by (label)           # Minimum
max(metric) by (label)           # Maximum
count(metric) by (label)         # Count
stddev(metric) by (label)        # Standard deviation
stdvar(metric) by (label)        # Variance
topk(5, metric)                  # Top 5 values
bottomk(5, metric)               # Bottom 5 values
count_values("label", metric)    # Count unique values

Time Functions

rate(metric[5m])                 # Per-second rate
irate(metric[5m])                # Instant rate
increase(metric[1h])             # Total increase
delta(metric[1h])                # Difference (for gauges)
idelta(metric[5m])               # Instant delta
deriv(metric[5m])                # Per-second derivative
predict_linear(metric[1h], 3600) # Predict 1h ahead

Label Manipulation

# Keep only specific labels
label_replace(metric, "new", "$1", "old", "(.*)")

# Join metrics
metric1 * on(label) metric2

# Group left/right
metric1 * on(label) group_left(extra) metric2

---

Anti-Patterns

High Cardinality Labels

# WRONG: User ID creates millions of time series
http_requests{user_id="user_12345"}

# CORRECT: Use low-cardinality labels
http_requests{endpoint="/api/users", status="200"}

Unbounded Label Values

# WRONG: URL path with parameters
http_requests{path="/api/users/12345/orders/67890"}

# CORRECT: Template the path
http_requests{endpoint="/api/users/:id/orders/:order_id"}

Missing Recording Rules

# WRONG: Complex query repeated in every dashboard
histogram_quantile(0.95,
  sum by (job, endpoint, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# CORRECT: Create recording rule
- record: api:http_latency:p95
  expr: histogram_quantile(0.95, ...)

Inefficient Queries

# WRONG: Aggregation after filtering
sum(rate(metric[5m])) by (label1, label2) > 0

# CORRECT: Filter before aggregation
sum(rate(metric[5m])) by (label1, label2) > 0

---

Level 3: Resources

This skill has Level 3 Resources available with comprehensive reference material, production-ready scripts, and runnable examples.

Resource Structure

prometheus-monitoring/resources/
├── REFERENCE.md                    # Comprehensive reference (3,200 lines)
│   ├── Prometheus architecture and components
│   ├── Complete PromQL reference with examples
│   ├── Recording rules patterns and best practices
│   ├── Alerting rules with severity levels
│   ├── Service discovery configurations
│   ├── Federation and high availability
│   ├── Storage configuration and retention
│   ├── Cardinality management
│   ├── Performance optimization
│   └── Production troubleshooting
│
├── scripts/                        # Production-ready tools
│   ├── analyze_metrics.py          # Metrics and config analyzer
│   ├── validate_promql.py          # PromQL query validator
│   └── test_exporter.sh            # Exporter testing tool
│
└── examples/                       # Runnable examples
    ├── config/
    │   └── prometheus.yml          # Complete Prometheus config
    ├── rules/
    │   ├── recording-rules.yml     # Recording rules examples
    │   └── alert-rules.yml         # Alerting rules examples
    ├── exporters/
    │   ├── custom-exporter.py      # Python custom exporter
    │   └── custom-exporter.go      # Go custom exporter
    ├── queries/
    │   └── promql-examples.txt     # Common PromQL patterns
    ├── docker/
    │   └── docker-compose.yml      # Full monitoring stack
    └── python/
        └── metrics-client.py       # Application instrumentation

Key Resources

REFERENCE.md (3,200 lines): Comprehensive guide covering:

• Prometheus architecture and data model
• Complete PromQL reference with 100+ query examples
• Recording rules for performance optimization
• Alerting rules with best practices
• All service discovery mechanisms (Kubernetes, Consul, EC2, file)
• Federation for multi-cluster setups
• Remote write and remote read configurations
• Storage optimization and retention policies
• Cardinality management techniques
• Performance tuning and troubleshooting
• Integration with Grafana and Alertmanager
• Production deployment patterns

analyze_metrics.py: Production-ready analyzer

• Parses prometheus.yml configuration
• Analyzes metric cardinality from Prometheus API
• Detects high-cardinality labels
• Identifies anti-patterns (naming, labeling)
• Validates recording and alerting rules
• Provides optimization recommendations
• Example: analyze_metrics.py --config prometheus.yml --json

validate_promql.py: PromQL validator

• Validates PromQL syntax
• Tests queries against Prometheus API
• Checks for common anti-patterns
• Measures query performance
• Suggests optimizations
• Example: validate_promql.py --query 'rate(http_requests_total[5m])' --json

test_exporter.sh: Exporter testing tool

• Tests metrics endpoints
• Validates Prometheus text format
• Checks metric naming conventions
• Identifies cardinality issues
• Validates HELP and TYPE annotations
• Example: test_exporter.sh --endpoint http://localhost:9090/metrics --json

Runnable Examples:

• Complete Prometheus configuration with all features
• Recording rules and alerting rules
• Custom exporters in Python and Go
• 50+ PromQL query examples
• Docker Compose stack (Prometheus + Grafana + Alertmanager + Node Exporter)
• Application instrumentation example

Usage

# Access comprehensive reference
cat prometheus-monitoring/resources/REFERENCE.md

# Analyze Prometheus configuration and metrics
./scripts/analyze_metrics.py --config prometheus.yml --metrics-endpoint http://localhost:9090

# Validate PromQL queries
./scripts/validate_promql.py --query 'sum(rate(http_requests_total[5m]))' --json

# Test custom exporter
./scripts/test_exporter.sh --endpoint http://localhost:8080/metrics

# Run full monitoring stack
cd examples/docker
docker-compose up -d
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000 (admin/admin)
# Alertmanager: http://localhost:9093

# Run custom exporter
cd examples/exporters
python3 custom-exporter.py
# Metrics at http://localhost:8080/metrics

When to Use Level 3 Resources

Use these resources when:

• Setting up Prometheus for production environments
• Writing complex PromQL queries
• Optimizing Prometheus performance
• Debugging high cardinality issues
• Creating recording and alerting rules
• Implementing custom exporters
• Configuring service discovery
• Setting up federation or remote storage
• Training team on Prometheus best practices
• Troubleshooting Prometheus issues

---

Related Skills

• metrics-instrumentation.md - Instrumenting applications with Prometheus
• distributed-tracing.md - Complementary distributed tracing
• alerting-strategy.md - Advanced alerting patterns
• dashboard-design.md - Visualizing Prometheus data in Grafana

---

Last Updated: 2025-10-27 Format Version: 1.0 (Atomic) Level 3 Resources: Available