TickYouOff
Sign in
Back
📊

Monitoring and Alerting Setup

Hard 17 items · 2 hours
testuser's avatar
testuser Published 1 month ago
Technical

This checklist guides engineers and SREs through a practical observability setup: metrics, logs, tracing, uptime checks, alert tuning, and on-call practices. Follow these steps to build reliable alerts and incident workflows for production systems.

Progress
0 / 17
  1. Define monitoring objectives and SLOs — List critical services, SLO targets, and acceptable error/latency budgets.
  2. Inventory telemetry sources — Document services, hosts, third-party deps, and available metrics/logs/traces.
  3. Configure metrics collection (Prometheus/CloudWatch) — Set exporters, scrape jobs or CloudWatch metrics, labels, and retention policies.
  4. Instrument application code for metrics and tracing — Add counters, histograms, meaningful labels, and OpenTelemetry spans where needed.
  5. Set up log aggregation (Loki/CloudWatch Logs) — Centralize logs, enforce structured logging, and configure retention/indices.
  6. Configure distributed tracing backend — Install collector (Tempo/Jaeger/OTel) and set sampling and span forwarding.
  7. Create dashboards for key SLOs and signals — Build dashboards showing latency, error rate, throughput, and resource saturation.
  8. Implement uptime and synthetic monitoring — Add external checks, heartbeat metrics, and synthetic tests for critical endpoints.
  9. Define alerting rules and thresholds — Translate SLO violations into alert rules with aggregation windows and severity.
  10. Tune alert noise, deduplication, and silencing policies — Add grouping, inhibition, and temporary silences to reduce alert fatigue.
  11. Integrate alerts with PagerDuty and Slack — Create notification channels, escalation paths, and template payloads for alerts.
  12. Configure PagerDuty integration — Create service, escalation policy, and routing key; restrict who can modify it.
  13. Configure Slack integration — Set up channels, webhook or app, and message formatting for actionable alerts.
  14. Send test alerts and verify escalation — Trigger test incidents to confirm delivery, escalation, and on-call handoffs.
  15. Attach runbook links and remediation steps to alerts — Include concise runbook URLs, commands, and rollback steps in alert payloads.
  16. Set up on-call rotation and playbooks — Define shifts, backups, escalation layers, and handover procedures.
  17. Run incident drills and validate postmortem process — Conduct a fire drill, verify notifications, and practice writing postmortems.
Sign in to save
📝 My Notes