observability-dashboard-design

Building Grafana dashboards for service monitoring

Dashboard Design

Scope: Grafana, visualization types, SLO dashboards, troubleshooting views, panel best practices

Lines: 376

Last Updated: 2025-10-18

---

When to Use This Skill

Use this skill when:

• Building Grafana dashboards for service monitoring
• Designing SLO/SLI dashboards for stakeholders
• Creating troubleshooting views for on-call engineers
• Visualizing metrics from Prometheus, Loki, or other data sources
• Implementing RED (Rate, Errors, Duration) dashboards
• Building executive dashboards showing business metrics
• Designing real-time operational dashboards
• Creating custom alerting views

Don't use for:

• Static reports (use BI tools like Tableau, Looker)
• Ad-hoc data exploration (use Jupyter notebooks)
• Long-term trend analysis (use dedicated analytics platforms)

---

Core Concepts

Dashboard Types

1. Service Dashboard (Operational)

• Target audience: Engineers, on-call
• Focus: System health, errors, latency
• Update frequency: Real-time (5s-30s)
• Metrics: RED (Rate, Errors, Duration)

2. SLO Dashboard (Business)

• Target audience: Managers, stakeholders
• Focus: Reliability targets, error budgets
• Update frequency: Hourly/daily
• Metrics: Availability, latency percentiles, error budget burn

3. Troubleshooting Dashboard (Diagnostic)

• Target audience: On-call engineers
• Focus: Detailed system state, dependencies
• Update frequency: Real-time (1s-5s)
• Metrics: Resource usage, request flow, errors by endpoint

4. Executive Dashboard (Strategic)

• Target audience: Leadership
• Focus: High-level KPIs, trends
• Update frequency: Daily/weekly
• Metrics: Uptime, revenue, active users

Visualization Types

Time Series Graph: Trends over time

• Use for: Latency, throughput, error rate
• Best for: Spotting patterns, anomalies

Gauge: Single value with thresholds

• Use for: Current status (CPU, memory, disk)
• Best for: At-a-glance health checks

Stat Panel: Single metric value

• Use for: Totals, percentages, counts
• Best for: KPIs, summary metrics

Heatmap: Distribution over time

• Use for: Latency distribution, bucket histograms
• Best for: Identifying percentile outliers

Table: Multi-dimensional data

• Use for: Service list, error breakdown
• Best for: Detailed comparisons

Bar Chart: Compare categories

• Use for: Error counts by endpoint, requests by region
• Best for: Top-N comparisons

RED Method

Rate: Requests per second

rate(http_requests_total[5m])

Errors: Error percentage

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

Duration: Latency percentiles

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

USE Method (Resources)

Utilization: % time resource busy

cpu_usage_percent
memory_usage_percent

Saturation: Degree of queuing/waiting

queue_depth
connection_pool_waiting

Errors: Error count/rate

rate(disk_errors_total[5m])
rate(network_errors_total[5m])

---

Patterns

Pattern 1: Service Overview Dashboard (Grafana JSON)

{
  "dashboard": {
    "title": "API Service Overview",
    "tags": ["service", "api"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=\"api\"}[5m]))",
            "legendFormat": "Total Requests/sec"
          }
        ],
        "yaxes": [
          {"format": "reqps", "label": "Requests/sec"}
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=\"api\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"api\"}[5m])) * 100",
            "legendFormat": "Error Rate %"
          }
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": {"type": "gt", "params": [1]},
              "operator": {"type": "and"},
              "query": {"params": ["A", "5m", "now"]},
              "type": "query"
            }
          ]
        }
      },
      {
        "title": "Latency (P50, P95, P99)",
        "type": "graph",
        "gridPos": {"x": 0, "y": 8, "w": 24, "h": 8},
        "targets": [
          {
            "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=\"api\"}[5m])) by (le))",
            "legendFormat": "P50"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=\"api\"}[5m])) by (le))",
            "legendFormat": "P95"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"api\"}[5m])) by (le))",
            "legendFormat": "P99"
          }
        ],
        "yaxes": [
          {"format": "s", "label": "Latency"}
        ]
      }
    ]
  }
}

Pattern 2: SLO Dashboard

{
  "dashboard": {
    "title": "SLO Dashboard - 99.9% Availability",
    "panels": [
      {
        "title": "Availability (30d)",
        "type": "stat",
        "gridPos": {"x": 0, "y": 0, "w": 6, "h": 4},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status!~\"5..\"}[30d])) / sum(rate(http_requests_total[30d])) * 100"
          }
        ],
        "options": {
          "colorMode": "background",
          "graphMode": "none",
          "textMode": "value_and_name"
        },
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {"value": 0, "color": "red"},
                {"value": 99, "color": "yellow"},
                {"value": 99.9, "color": "green"}
              ]
            }
          }
        }
      },
      {
        "title": "Error Budget Remaining (30d)",
        "type": "gauge",
        "gridPos": {"x": 6, "y": 0, "w": 6, "h": 4},
        "targets": [
          {
            "expr": "(0.001 - (1 - sum(rate(http_requests_total{status!~\"5..\"}[30d])) / sum(rate(http_requests_total[30d])))) / 0.001 * 100"
          }
        ],
        "options": {
          "showThresholdLabels": true,
          "showThresholdMarkers": true
        },
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "min": 0,
            "max": 100,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {"value": 0, "color": "red"},
                {"value": 25, "color": "yellow"},
                {"value": 50, "color": "green"}
              ]
            }
          }
        }
      },
      {
        "title": "Error Budget Burn Rate",
        "type": "graph",
        "gridPos": {"x": 0, "y": 4, "w": 24, "h": 8},
        "targets": [
          {
            "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[1h])) / sum(rate(http_requests_total[1h]))) / 0.001",
            "legendFormat": "1h burn rate (1.0 = on track)"
          },
          {
            "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[6h])) / sum(rate(http_requests_total[6h]))) / 0.001",
            "legendFormat": "6h burn rate"
          }
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": {"type": "gt", "params": [10]},
              "query": {"params": ["A", "5m", "now"]}
            }
          ],
          "name": "Error budget burning 10x faster"
        }
      }
    ]
  }
}

Pattern 3: Troubleshooting Dashboard

{
  "dashboard": {
    "title": "Troubleshooting - API Service",
    "panels": [
      {
        "title": "Requests by Endpoint (Top 10)",
        "type": "table",
        "targets": [
          {
            "expr": "topk(10, sum by (endpoint) (rate(http_requests_total{service=\"api\"}[5m])))",
            "format": "table",
            "instant": true
          }
        ]
      },
      {
        "title": "Errors by Endpoint",
        "type": "bargauge",
        "targets": [
          {
            "expr": "sum by (endpoint) (rate(http_requests_total{service=\"api\",status=~\"5..\"}[5m]))"
          }
        ],
        "options": {
          "orientation": "horizontal",
          "displayMode": "gradient"
        }
      },
      {
        "title": "Latency Heatmap",
        "type": "heatmap",
        "targets": [
          {
            "expr": "sum(rate(http_request_duration_seconds_bucket{service=\"api\"}[5m])) by (le)",
            "format": "heatmap"
          }
        ],
        "heatmap": {
          "yAxis": {"format": "s", "decimals": 2}
        }
      },
      {
        "title": "Database Connection Pool",
        "type": "graph",
        "targets": [
          {
            "expr": "database_connections{state=\"active\"}",
            "legendFormat": "Active"
          },
          {
            "expr": "database_connections{state=\"idle\"}",
            "legendFormat": "Idle"
          },
          {
            "expr": "database_connections_max",
            "legendFormat": "Max"
          }
        ]
      }
    ]
  }
}

Pattern 4: Resource Usage Dashboard

{
  "dashboard": {
    "title": "Resource Usage",
    "panels": [
      {
        "title": "CPU Usage by Instance",
        "type": "graph",
        "targets": [
          {
            "expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
            "legendFormat": "{{ instance }}"
          }
        ],
        "yaxes": [
          {"format": "percent", "max": 100}
        ]
      },
      {
        "title": "Memory Usage by Instance",
        "type": "graph",
        "targets": [
          {
            "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100",
            "legendFormat": "{{ instance }}"
          }
        ],
        "yaxes": [
          {"format": "percent", "max": 100}
        ]
      },
      {
        "title": "Disk I/O",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(node_disk_read_bytes_total[5m])",
            "legendFormat": "Read - {{ device }}"
          },
          {
            "expr": "rate(node_disk_written_bytes_total[5m])",
            "legendFormat": "Write - {{ device }}"
          }
        ],
        "yaxes": [
          {"format": "Bps"}
        ]
      }
    ]
  }
}

Pattern 5: Business Metrics Dashboard

{
  "dashboard": {
    "title": "Business Metrics",
    "panels": [
      {
        "title": "Revenue (Last 24h)",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(increase(revenue_total_dollars[24h]))"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "currencyUSD",
            "decimals": 2
          }
        }
      },
      {
        "title": "Orders by Product Category",
        "type": "piechart",
        "targets": [
          {
            "expr": "sum by (product_category) (increase(orders_total[24h]))"
          }
        ]
      },
      {
        "title": "Active Users (Last 30 days)",
        "type": "graph",
        "targets": [
          {
            "expr": "count(count by (user_id) (increase(user_activity_total[24h])))"
          }
        ]
      },
      {
        "title": "Cart Abandonment Rate",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(increase(cart_abandonment_total[24h])) / sum(increase(cart_created_total[24h])) * 100"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "thresholds": {
              "steps": [
                {"value": 0, "color": "green"},
                {"value": 50, "color": "yellow"},
                {"value": 70, "color": "red"}
              ]
            }
          }
        }
      }
    ]
  }
}

---

Quick Reference

Dashboard Design Principles

1. Most important metrics first (top-left)
2. Logical grouping (related panels together)
3. Consistent time ranges (use dashboard-level variable)
4. Clear titles (action-oriented: "Check latency" not "Latency")
5. Appropriate colors (red for errors, green for success)
6. Thresholds (visual indicators for good/bad states)
7. Legends (concise, meaningful labels)

Panel Layout Guidelines

+------------------------------------------+
|  Critical KPIs (Stats/Gauges)            |  ← Row 1: 4 panels
+------------------------------------------+
|  Trends (Time Series)                    |  ← Row 2: 2-3 panels
+------------------------------------------+
|  Detailed Breakdown (Tables/Bar Charts)  |  ← Row 3: 1-2 panels
+------------------------------------------+
|  Resource Usage (Graphs)                 |  ← Row 4: 2-3 panels
+------------------------------------------+

Common PromQL Patterns

# Rate (requests/sec)
rate(metric_total[5m])

# Error rate (%)
sum(rate(errors[5m])) / sum(rate(requests[5m])) * 100

# Percentiles
histogram_quantile(0.95, rate(metric_bucket[5m]))

# Top N
topk(10, sum by (label) (rate(metric[5m])))

# Aggregation
sum by (service) (rate(metric[5m]))
avg by (instance) (metric)
max by (region) (metric)

# Math
metric1 / metric2  # Ratio
metric1 - metric2  # Difference
metric1 > 100      # Filter

Grafana Variables

# Service selector
Name: service
Query: label_values(http_requests_total, service)
Usage: {service="$service"}

# Time range
Name: range
Type: Interval
Values: 5m,15m,1h,6h,24h
Usage: rate(metric[$range])

# Instance selector
Name: instance
Query: label_values(up{service="$service"}, instance)
Usage: {instance=~"$instance"}

---

Anti-Patterns

❌ Too Many Panels

# WRONG: 50 panels on one dashboard (overwhelming)
# CORRECT: 8-12 panels max, use drill-down links

❌ No Thresholds

// WRONG: Plain graph with no visual indicators
{"type": "graph"}

// CORRECT: Color-coded thresholds
{
  "type": "graph",
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "steps": [
          {"value": 0, "color": "green"},
          {"value": 80, "color": "yellow"},
          {"value": 90, "color": "red"}
        ]
      }
    }
  }
}

❌ Inconsistent Time Ranges

# WRONG: Panel 1 uses [5m], Panel 2 uses [1h]
# CORRECT: Use dashboard variable $range

❌ Raw Counters Instead of Rates

# WRONG: Counter value (meaningless)
http_requests_total

# CORRECT: Rate (requests/sec)
rate(http_requests_total[5m])

❌ Missing Context in Titles

# WRONG: "Requests"
# CORRECT: "API Requests per Second (5m avg)"

❌ Too Much Data in Tables

# WRONG: 1000 rows in table
topk(1000, metric)

# CORRECT: Top 10
topk(10, metric)

---

Related Skills

• metrics-instrumentation.md - Define metrics for dashboards
• structured-logging.md - Link logs from dashboard panels
• distributed-tracing.md - Drill down to traces
• alerting-strategy.md - Dashboard-based alert status

---

Last Updated: 2025-10-18 Format Version: 1.0 (Atomic)