Alertmanager

Last reviewed Aug 26, 2026 · 787 words

Alertmanager exists so that one dead switch produces one notification instead of 40. Prometheus decides that a rule is firing; Alertmanager deduplicates the stream, groups related alerts by label, waits 30 seconds by default for the rest of the storm to arrive, and sends a single message to whichever receiver the routing tree picks. It does this on 64 MB of RAM. The cost is a YAML file that is easy to get subtly wrong, and the defence against that is one alert that is designed never to stop firing.

Where it sits, and what it is not

Alertmanager is not a monitoring tool. It has no idea whether your disk is full; it only knows that Prometheus sent it an alert with some labels. It is the last stage of the Prometheus pipeline, listening on port 9093, and Prometheus is told where it lives:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Grafana has its own alerting engine and can also forward to Alertmanager, so a stack with both usually settles on Alertmanager as the single place where routing and silencing live. The Grafana vs Prometheus comparison covers how the two divide the work.

A routing tree small enough to trust

Most homelab configurations need fewer than 40 lines. The shape is one root route with sensible grouping, a receiver for everyday noise, and a child route that catches critical alerts and sends them somewhere louder:

route:
  receiver: ntfy
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ["severity=critical"]
      receiver: ntfy-critical
receivers:
  - name: ntfy
    webhook_configs:
      - url: http://ntfy:80/homelab
  - name: ntfy-critical
    webhook_configs:
      - url: http://ntfy:80/homelab-critical

The three timers matter more than anything else in the file. group_wait is how long a new group waits before the first notification; group_interval is how long before a group with new alerts sends again; repeat_interval is how often an unchanged, still-firing group nags you. The defaults (30s, 5m, 4h) are right for most people; the one change I make on a homelab is repeat_interval: 12h, because a still-broken disk does not need to tell me six times a day. Routes are matched top to bottom and the first match wins unless continue: true is set, which is the single most common source of "why did this go to the wrong channel".

A plain webhook receiver is the pragmatic choice in a homelab because ntfy and Gotify both accept one, and the phone app does the rest. Slack, email, PagerDuty and Opsgenie are built in if you need them.

The Watchdog alert is not optional

A silent Alertmanager and a healthy homelab look identical. The fix is a Prometheus rule that fires unconditionally:

- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none

Route it to a receiver that expects a heartbeat, such as a healthchecks-style ping or a dedicated ntfy topic you check on, with a short repeat_interval. When the heartbeat stops, you know the pipeline broke somewhere between Prometheus and your phone. Every serious Prometheus deployment does this; every homelab should. It costs one rule and about 5 minutes, and it is the only alert in the file whose absence you will never notice until it matters.

Silences and inhibition save your weekends

Rebooting a host on Saturday should not page you 12 times. amtool silence add instance="nas:9100" --duration 2h --comment "reboot" does it from the shell; the web UI on 9093 does the same with a form. Inhibition is the automatic version: an inhibit_rules entry that says "when NodeDown fires for an instance, suppress everything else from that instance" removes the cascade of disk, memory and service alerts that follow a machine going away. Write that one rule on day one.

When you do not need it

If you have 6 services and no Prometheus, do not add Prometheus to get Alertmanager. Uptime Kuma sends its own notifications, and the homelab monitoring post explains where the line between uptime checks and metrics actually falls. Alertmanager earns its place once you have enough rules that they fire together, which is the point where naive notifications become noise you start ignoring.

What I'd do

Run the official image alongside Prometheus, one root route to ntfy with the default timers, a critical child route to a separate topic with sound on, one inhibit rule for host-down, and the Watchdog alert wired to a heartbeat from day one. Keep the whole file under 60 lines and validate it with amtool check-config alertmanager.yml before every reload. That setup has stayed unchanged in my rack for years; the rules change, the routing does not.