Prometheus

Last reviewed Aug 24, 2026 · 662 words

Prometheus is the metrics database under half the dashboards you've admired, and the homelab version of it is much smaller than its cloud-native reputation suggests: one container, one YAML file listing what to scrape, and a couple of exporters. What you get for that is history — not "is it up right now" (that's Uptime Kuma's job) but "what has disk usage done since March", which is the question that predicts failures instead of announcing them.

The pull model in one paragraph

Everything in the Prometheus world exposes a /metrics HTTP page of plain-text numbers, and Prometheus visits ("scrapes") each one on an interval, storing every value as a timestamped series. That inversion — the server fetches, agents just serve text — is why the ecosystem is so composable: anything that can serve HTTP can be monitored, and you can curl any exporter to see exactly what Prometheus sees. The scrape config is the whole setup:

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["nas:9100", "minipc:9100", "pi:9100"]
  - job_name: cadvisor
    static_configs:
      - targets: ["minipc:8080"]

The three exporters that cover a homelab

node_exporter (port 9100) on every Linux box is 90% of the value: CPU, memory, disk space and IO, network, temperatures, systemd units. cAdvisor adds per-container CPU/memory, answering "which container is eating the box". smartctl_exporter (or scrutiny, if you prefer it packaged) surfaces disk SMART data — reallocated sectors trending upward is the single most valuable early warning a homelab can have, arriving weeks before the disk dies and the backups get their exam. Beyond those, grep your existing stack before adding anything: many self-hosted apps (Gitea, Caddy, Home Assistant) already expose /metrics and just need a scrape entry.

Retention and sizing: smaller than you fear

Default retention is 15 days — raise it, that's too short for the trends that make Prometheus worthwhile: --storage.tsdb.retention.time=180d costs surprisingly little, because samples compress to roughly 1–2 bytes each. A homelab scraping three nodes and thirty containers every 30 seconds runs around 2–5 GB of disk for six months of history and idles under 512 MB of RAM. Two operational notes from the catalogue that deserve repeating: persist the data volume, and know that Prometheus ships no authentication — keep port 9090 LAN/Tailscale-only or put your proxy's auth in front, because the admin API can delete data.

Enough PromQL to be dangerous

Four patterns cover most homelab questions, typed into the built-in graph page or a Grafana panel:

• Disk full when? predict_linear(node_filesystem_avail_bytes[7d], 86400*30) < 0 — linear-extrapolates a month ahead.
• CPU per machine: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
• Memory actually available: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
• Container memory hogs: topk(5, container_memory_working_set_bytes)

rate(...[5m]) — "per-second average over five minutes" — is the one idiom worth internalising; nearly every counter query wraps it. Resist learning more until a question forces you.

Where alerting should live

Prometheus's own alerting rules feed a separate Alertmanager component, with routing, grouping, and silences — powerful, and more machinery than most homelabs should adopt. The pragmatic split from the monitoring stack guide: Kuma alerts on down, Grafana's built-in alerting evaluates the handful of PromQL conditions worth waking for (disk trend, SMART, backup-job staleness), and Alertmanager waits until you're running things other people depend on. Prometheus's job in a homelab is memory, not sirens.

What I'd do

The compose file with 180-day retention, node_exporter everywhere via the same compose files that run everything else, cAdvisor and smartctl_exporter on the main box, port 9090 tailnet-only, Grafana's dashboard 1860 on top, and the four queries above saved. Total footprint: under a gigabyte of RAM across the stack, a few gigabytes of disk, and the quiet superpower of answering "when did that start?" with a graph instead of a guess.