Kafka Exporter

Last reviewed Sep 13, 2026 · 826 words

There is one Kafka metric that pages people at 3 am, and it is kafka_consumergroup_lag. Kafka Exporter exists to produce it. The 2,535-star Go binary connects to your brokers as an ordinary client, asks for topic offsets and consumer group offsets, subtracts one from the other, and serves the result on port 9308 for Prometheus to scrape. It uses about 64 MB of RAM, needs no JMX agent inside the broker, and it is Apache-2.0. It is also not a Kafka monitoring solution on its own, and understanding what it does not measure is half of using it well.

What it measures, and what it never will

Kafka Exporter looks at Kafka from the outside, using the same protocol a consumer would. That gets you kafka_brokers, kafka_topic_partitions, kafka_topic_partition_current_offset, kafka_topic_partition_oldest_offset, kafka_topic_partition_in_sync_replica, kafka_topic_partition_under_replicated_partition, kafka_consumergroup_current_offset, and the lag metrics per group, topic and partition. Together these answer the operational questions: is any consumer falling behind, is any partition under-replicated, is any topic growing without bound.

What it cannot see is inside the JVM. Broker heap, GC pauses, request queue times, network thread utilisation and disk write latency all live in JMX, and the way to get them is the Prometheus JMX exporter loaded as a Java agent on each broker. The two exporters are complementary, not rivals. Kafka Exporter alone tells you that consumers are lagging; the JMX exporter tells you whether that is because the broker is drowning. A serious deployment runs both, plus Node Exporter on each broker host for disk and network.

A self-hoster's Kafka stack in practice

Most people reading this are running 1 to 3 brokers for a home data pipeline, an event-sourced side project, or a small company's integration bus, and the stack looks like this: brokers, Kafka Exporter as a sidecar, Prometheus scraping it, Grafana with the community dashboard 7589 ("Kafka Exporter Overview") imported, and Alertmanager for the lag alert. That whole monitoring layer fits in under 1 GB of RAM and is what the hosted tools in the Datadog Kafka alternatives list charge per host for.

The container is one line of configuration:

services:
  kafka-exporter:
    image: danielqsj/kafka-exporter:latest
    command:
      - --kafka.server=kafka-1:9092
      - --kafka.server=kafka-2:9092
    ports:
      - "9308:9308"
    restart: unless-stopped

Add --sasl.enabled, --sasl.username, --sasl.password and --tls.enabled for secured clusters, and --topic.filter and --group.filter with regular expressions if a busy cluster makes the metric count too large. One exporter instance covers one cluster; bootstrap it with every broker listed so it survives a broker being down. In Kubernetes it deploys as a single-replica Deployment with a ServiceMonitor, which is the environment most of its users run.

The alert that earns its keep

Absolute lag is a poor alert. A topic doing 50,000 messages a second will always show thousands of messages of lag, and a topic doing 3 a minute will show 0 until something breaks. Alert on lag that is growing over time, per group:

- alert: KafkaConsumerLagGrowing
  expr: sum by (consumergroup, topic) (delta(kafka_consumergroup_lag[10m])) > 0
    and sum by (consumergroup, topic) (kafka_consumergroup_lag) > 1000
  for: 15m

Pair it with kafka_topic_partition_under_replicated_partition > 0 for 5 minutes, which is the single most reliable early warning of a broker in trouble, and with a check that kafka_brokers equals the number you expect. Three alerts cover 90 percent of what goes wrong in a small cluster.

One caveat on the lag numbers themselves: the exporter reports lag as of its last poll, and consumer groups that use offset commits infrequently will show sawtooth lag that is not real. Match the scrape interval to the commit interval, or accept the jitter and alert on trends as above.

Sizing and the scrape cost

Every scrape triggers metadata and offset requests to the brokers, so a 30-second interval on a cluster with 500 topics and 200 consumer groups produces a noticeable trickle of admin traffic. That is fine for the brokers and heavy for Prometheus: the metric cardinality is topics times partitions times groups. Use the filters to drop internal topics like __consumer_offsets and any groups you do not care about, and bump the scrape interval to 60 seconds on large clusters. For a home cluster with 20 topics none of this matters and the default settings are right.

What I'd do

Deploy Kafka Exporter beside the brokers on day one of any Kafka install, before there is a consumer to lag, so the baseline exists. Import dashboard 7589, write the three alerts above, and route them through the same Alertmanager as the rest of the monitoring stack. Add the JMX exporter only when a real incident leaves you wanting broker internals, which for a home cluster may be never. Kafka Exporter is a small, boring, well-maintained component that does one job, and that is exactly what a monitoring tool should be.