PromQL Queries Behind Our Most Useful Alerts

Dashboards are nice; alerts are what page someone. Here are the queries backing ours.

Error rate over 5 minutes (RED method)

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.02

p99 latency per route

histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
)

Pods stuck restarting

increase(kube_pod_container_status_restarts_total[15m]) > 3

Node disk about to fill up

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

Saturated CPU across a deployment

avg(rate(container_cpu_usage_seconds_total{namespace="prod"}[5m])) by (pod)
/
avg(kube_pod_container_resource_limits{resource="cpu"}) by (pod) > 0.9

Quick ad-hoc check with the HTTP API

curl -s 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=up{job="checkout"}' | jq '.data.result'

predict_linear is the one people forget — it's what lets you page on "disk fills up in 4 hours" instead of "disk is full."