Pushgateway

Last reviewed Sep 13, 2026 · 870 words

Most people who install Pushgateway do not need it. A cron job on a host you already scrape can write a file for node_exporter's textfile collector, and a batch job that runs longer than a scrape interval can expose /metrics itself. Pushgateway exists for the narrow case that remains: a short-lived job, on no particular host, whose last result you need to see in Prometheus. For that case it is one Go binary, 32 MB of RAM, and about 15 minutes to wire up correctly, provided you understand the two things it deliberately refuses to do.

It remembers forever and it never adds anything up

Pushgateway is a cache of metric groups, keyed by job and whatever labels you put in the URL. A push replaces the whole group. That gives you two behaviours that surprise people. First, there is no expiry: a job that pushed once in March and was then deleted still reports its March values in September, and Prometheus will happily graph them. You delete stale groups by hand with an HTTP DELETE or through the web UI on port 9091. Second, there is no aggregation. Ten parallel workers pushing to the same group overwrite each other; each needs its own instance label, and if you want a total, you sum in PromQL. Anyone who wants a counter that many clients increment wants a StatsD-style aggregator, not this.

The scrape config line everyone forgets

Prometheus treats every target's job and instance as its own to assign, and overwrites them on scrape. For Pushgateway that is wrong, because the whole point is the job label the pushing script chose. The fix is one line:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ["pushgateway:9091"]

Without honor_labels: true, every pushed metric shows up with job="pushgateway" and your alerts on job="backup" silently match nothing. This is the single most common mistake with the tool and the docs put it in bold for a reason.

Pushing from a shell script is a curl one-liner

The URL encodes the grouping labels, and the body is plain exposition format:

cat <<METRICS | curl --data-binary @- http://pushgateway:9091/metrics/job/backup/instance/nas
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
# TYPE backup_duration_seconds gauge
backup_duration_seconds 842
# TYPE backup_bytes gauge
backup_bytes 53687091200
METRICS

Push gauges that describe the last run: when it finished, how long it took, what it produced, and an exit status. Do not push counters you increment across runs, because a replaced group loses history and a restarted job resets to zero without Prometheus seeing a reset. The official client libraries in Go, Python, and Java have a push_to_gateway helper if the job is not a shell script.

Alert on staleness, not on values

Because groups never expire, the useful alert is "this job has not reported recently," and Pushgateway gives you the metric to write it. Every group carries push_time_seconds, the timestamp of its last push:

- alert: BackupNotRun
  expr: time() - push_time_seconds{job="backup"} > 90000
  for: 10m

That fires if the nightly backup missed a day, whether it crashed, was never started, or the host it runs from died. Pair it with an alert on your own exit-status gauge for failures that did report in. Route both through Alertmanager like everything else; nothing about Pushgateway changes how alerting works downstream.

Persist it, and accept that it is a single point of failure

By default the state lives in memory and a container restart empties it, which turns every staleness alert on at once. Run it with --persistence.file=/data/pushgateway.data and a volume, and it writes to disk every 5 minutes by default. The catalogue is right that it can become a single point of failure: every batch job in the fleet depends on this one process being up, and there is no clustering. The mitigation is to keep it dumb, keep it on the same box as Prometheus so they fail together, and keep the number of jobs that need it small.

Two alternatives that are usually better

The node_exporter textfile collector handles any job that runs on a machine you already scrape: the job writes a .prom file into --collector.textfile.directory, and node-exporter serves it with the host metrics. That is most cron jobs. For long-running batch work, expose /metrics and scrape it directly. If you are already deploying Grafana Alloy, its pipeline can accept and forward pushed metrics too, which removes one binary from the fleet. Pushgateway is for the remainder: containers that start, run for 40 seconds, push, and exit, on a scheduler that gives them no fixed address.

What I'd do

Textfile collector for every cron job on a real host. Pushgateway only for jobs on Kubernetes CronJobs or CI runners, with honor_labels: true, persistence on a volume, an instance label per parallel worker, and a push_time_seconds staleness alert for each job the day it starts pushing. Delete groups from retired jobs immediately. Run it next to Prometheus in the monitoring stack, never as a shared service the whole company pushes to; that is when the 32 MB binary becomes the outage everyone remembers.