📊 Monitoring & Alerting with the kube-prometheus-stack

The kube-prometheus-stack Helm chart gives you production monitoring in one command: Prometheus, Alertmanager, Grafana, node/kube metrics, and dozens of ready-made dashboards and alerts. This guide installs it, scrapes your own app, and ships a real alert to Slack.

Prerequisites

• A Kubernetes cluster with kubectl + helm.
• An app that exposes Prometheus metrics at /metrics (or one you can add a client library to).
• A Slack incoming webhook URL for the alerting section.

What's in the box

• Prometheus — scrapes metrics over HTTP and stores them as time series; you query with PromQL.
• Alertmanager — takes firing alerts, deduplicates/groups them, and routes to Slack, email, PagerDuty.
• Grafana — dashboards over Prometheus data.
• Prometheus Operator — configure all of the above with Kubernetes CRDs (ServiceMonitor, PrometheusRule) instead of editing config files.
• node-exporter + kube-state-metrics — expose host and Kubernetes object metrics.

Install it

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Port-forward Grafana and you already have cluster, node, and pod dashboards:

kubectl -n monitoring port-forward svc/kps-grafana 3000:80
# login: admin / (kubectl get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d)

Scrape your own app the operator way

Expose /metrics in your app, front it with a Service, then add a ServiceMonitor — no Prometheus config edits:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: monitoring
  labels: {release: kps}          # MUST match the stack's serviceMonitorSelector
spec:
  namespaceSelector: {matchNames: [default]}
  selector:
    matchLabels: {app: checkout}  # matches your Service's labels
  endpoints:
    - port: http                  # the Service port NAME, not number
      path: /metrics
      interval: 15s

Alert on symptoms, not causes

The golden rule: page on what users feel. "Error rate > 2% for 5 minutes" is a good alert; "CPU > 80%" usually isn't (high CPU with happy users is fine). Write it as a PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: {name: checkout-slo, namespace: monitoring, labels: {release: kps}}
spec:
  groups:
    - name: checkout
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout"}[5m])) > 0.02
          for: 5m
          labels: {severity: page}
          annotations:
            summary: "Checkout 5xx above 2% for 5m"
            runbook_url: "https://wiki/runbooks/checkout"

for: 5m avoids flapping; runbook_url turns a 3am page into a checklist.

Route alerts to Slack

Configure Alertmanager (via the stack's values or an AlertmanagerConfig CRD):

route:
  receiver: slack-warnings
  group_by: [alertname, namespace]
  routes:
    - matchers: [severity="page"]
      receiver: slack-pages
receivers:
  - name: slack-pages
    slack_configs:
      - api_url: <your-webhook>
        channel: "#pages"
        title: '{{ .CommonAnnotations.summary }}'

Group and inhibit aggressively — alert fatigue is a system failure; if everything pages, nothing does.

The PromQL you actually need

You don't need to master PromQL to be effective — five patterns cover almost everything.

• Rate of a counter — counters only ever increase, so you always wrap them in rate() over a window: rate(http_requests_total[5m]) gives per-second request rate.
• Error ratio — divide the bad rate by the total rate (the alert above): errors as a fraction.
• A percentile from a histogram — histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) gives p95 latency. Percentiles, not averages — an average hides the slow tail that users feel.
• Aggregate away noise — sum by (service) (...) collapses per-pod series into per-service ones so a dashboard stays readable.
• Saturation — resource used ÷ resource available, e.g. memory working set over the limit.

Recording rules pre-compute expensive expressions (like that p95) on a schedule, so dashboards and alerts read a cheap pre-aggregated series instead of crunching raw data every time — the standard fix when Grafana panels load slowly.

Cardinality is the silent killer. Every unique combination of label values is a separate stored series. Put a user ID, email, or full URL in a label and you create millions of series that blow up Prometheus's memory. Keep labels bounded (route templates, not raw paths; status classes, not every code). If Prometheus is eating RAM, high cardinality is almost always why.

Grafana dashboards (and dashboards-as-code)

The stack ships with excellent built-in dashboards, but you'll want your own per-service views. Two rules keep them useful:

• Start from the golden signals (below) — a service dashboard's top row should be latency, traffic, errors, and saturation, so anyone can assess health in five seconds.
• Design for the incident, not the demo. The question a dashboard answers at 3am is "what changed and where does it hurt?" — so put rate/error/latency side by side, annotate deploys, and link to the runbook.

Crucially, treat dashboards as code. Clicking together panels that live only in Grafana's database is fragile; instead define dashboards as JSON (or with the Grafana Terraform provider / Jsonnet + Grafonnet) and version them in git. The Prometheus Operator can even load dashboards from ConfigMaps, so your monitoring is reproducible and reviewable like any other code.

Alertmanager: routing, grouping, silences & on-call

Alertmanager is the difference between "we get paged for real problems" and "everyone muted the channel." Its job is to take raw firing alerts and make them humane:

• Grouping collapses related alerts into one notification — if a whole service is down, you get one page, not fifty.
• Routing sends the right alert to the right place by label: severity: page to PagerDuty/Slack on-call, severity: warning to a low-noise email, per-team routing by a team label.
• Inhibition suppresses downstream alerts when an upstream one is firing (don't page for "high latency" when the alert above already says "service down").
• Silences let on-call mute a known issue during maintenance so it doesn't nag — time-boxed, so it auto-expires.

Good on-call practice sits on top: every paging alert links a runbook, alerts are actionable (if there's nothing to do, it shouldn't page), and you review noisy alerts regularly. Alert fatigue is a system failure — tune relentlessly.

SLOs & burn-rate alerting

Symptom alerts are good; SLO-based alerting is better. You define a Service Level Objective — say 99.9% of requests succeed over 30 days — which gives you an error budget (0.1% may fail). Then you alert on the burn rate: how fast you're consuming that budget.

        - alert: ErrorBudgetBurnFast
          expr: |
            (
              sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="checkout"}[5m]))
            ) > (14.4 * 0.001)          # 14.4x burn of a 99.9% SLO → budget gone in ~2 days
          for: 5m
          labels: {severity: page}

Multi-window burn-rate alerts (a fast window for sharp outages, a slow window for gradual erosion) page you proportionally to how much trouble you're actually in — the modern, low-noise way to alert. It also reframes reliability as a budget you spend on shipping features, aligning engineering and product.

Long-term storage & scaling Prometheus

A single Prometheus keeps a few weeks of data on local disk and doesn't replicate — fine to start, a limit at scale. When you need long retention, global views across clusters, or high availability, add a layer via remote write: Thanos, Grafana Mimir, or VictoriaMetrics. They ship Prometheus data to cheap object storage (S3/GCS), deduplicate multiple Prometheus replicas, and give one query endpoint across many clusters — unlimited retention without bloating each Prometheus. You rarely need this on day one, but knowing the path exists stops you from painting yourself into a corner.

Instrumenting your application

The built-in dashboards cover infrastructure; the real value is metrics from your own code. Add a Prometheus client library and expose /metrics. You mainly use three metric types:

• Counter — a value that only goes up (requests, errors). Always query it with rate().
• Gauge — a value that goes up and down (queue depth, in-flight requests, temperature).
• Histogram — bucketed observations (request duration) that let you compute percentiles server-side.

from prometheus_client import Counter, Histogram
REQS = Counter("http_requests_total", "Requests", ["route", "code"])
LAT  = Histogram("http_request_duration_seconds", "Latency", ["route"])

# in the handler:
with LAT.labels(route).time():
    ...
REQS.labels(route, code).inc()

That's enough to compute all four golden signals for the service. Label discipline matters: label by route template (/users/:id), not the raw path, or you recreate the cardinality explosion. Instrumenting your own code is what turns monitoring from "the node is up" into "checkout's p95 latency just doubled."

Exporters: monitoring things you can't modify

You can't add code to Postgres, Redis, or a network switch — so Prometheus uses exporters: small sidecar processes that translate a system's stats into Prometheus metrics. There's an exporter for almost everything: node_exporter (host CPU/memory/disk), postgres_exporter, mysqld_exporter, redis_exporter, blackbox_exporter (probe a URL/port from outside for uptime and TLS expiry). You run the exporter, point a ServiceMonitor at it, and its metrics flow in alongside your app's. Between your own instrumentation and exporters, you can cover an entire stack — app, databases, infrastructure, external endpoints — in one Prometheus.

Federation & scaling across clusters

A single Prometheus per cluster is the norm, but when you run many clusters you want a global view. Two approaches: federation (a central Prometheus scrapes aggregated metrics from each cluster's Prometheus — simple, good for high-level rollups) and, more scalably, the remote-write systems from earlier (Thanos/Mimir/VictoriaMetrics) that give one query layer across everything with long retention. Start with one Prometheus per cluster and a Grafana pointed at all of them; reach for federation or a global store only when you actually need cross-cluster queries or longer history than local disk holds.

Blackbox monitoring & synthetic checks

Everything so far is whitebox monitoring — metrics from inside your systems. But "all my internal metrics look fine" and "users can't reach the site" can both be true at once (a DNS, TLS, or load-balancer problem sits outside your app). Blackbox monitoring closes that gap by probing your service the way a user would, from the outside:

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata: {name: uptime, labels: {release: kps}}
spec:
  prober: {url: blackbox-exporter:9115}
  targets:
    staticConfig: {static: ["https://mysite.com/health"]}

The blackbox_exporter performs the probe (HTTP, TCP, DNS, ICMP) and reports whether it succeeded, the response time, and — very usefully — the TLS certificate expiry, so you get paged before a cert lapses and takes the site down. Run these probes from outside the cluster where possible, so they catch the ingress and network path too. The combination is the goal: whitebox tells you why something is wrong, blackbox confirms that users are actually affected — and alerting on the blackbox check is the closest proxy you have for "is the service actually up for real people?"

Pull vs push, and the Pushgateway

Prometheus is a pull system: it scrapes targets on a schedule. That's deliberate — Prometheus controls timing, can tell when a target is down (the scrape fails), and there's no client-side buffering to lose. Long-running services should always be scraped. But some workloads don't fit pull: a batch/cron job may finish and exit before any scrape happens. For those, push the metric to a Pushgateway as the job ends, and Prometheus scrapes the gateway. Use it only for short-lived jobs — pushing long-running service metrics through it defeats up/down detection and creates stale data. Rule: pull by default; push only for batch jobs that would otherwise never be scraped.

The four things to always watch (golden signals)

Latency, traffic, errors, saturation. Dashboard those four per service and you'll catch most problems before users complain.

Verify it worked

1. Prometheus UI → Status → Targets: your checkout target is UP.
2. Run the alert's PromQL in Graph — it evaluates without error.
3. Force errors in the app; after for: 5m the alert moves Pending → Firing (Prometheus → Alerts) and a message lands in Slack.
4. Grafana shows your golden-signals panels moving.

Troubleshooting

• Target missing / DOWN → the ServiceMonitor's labels don't match the stack's selector (must include release: kps), or the port is a number instead of the Service port name, or the namespaceSelector is wrong.
• PromQL returns nothing → the metric name/labels differ from your app's. Check /metrics directly and Prometheus Status → Targets → labels.
• Alert never fires → the expression is false, or for: hasn't elapsed. Test the query first; lower for: temporarily.
• No Slack message → bad webhook, or the route didn't match your alert's labels. Check Alertmanager logs and the Status page.
• Alert storm on deploy → add for:, group by alertname, and use inhibition rules.

Where to go next

Our free course installs the stack, wires a demo app via ServiceMonitor, builds a golden-signals dashboard, and ships a symptom-based alert to Slack — the exact loop real SRE teams run, guided in a lab.