mtail

Last reviewed Sep 13, 2026 · 833 words

An nginx access log, a file containing this:

counter http_requests_total by status, method

/^\S+ \S+ \S+ \[[^\]]+\] "(?P<method>\S+) \S+ \S+" (?P<status>\d+)/ {
  http_requests_total[$status][$method]++
}

and the command mtail --progs /etc/mtail --logs /var/log/nginx/access.log gives you a per-status, per-method request counter on http://host:3903/metrics that Prometheus can scrape 20 seconds later. That is the entire pitch. mtail exists for the software that will never expose a /metrics endpoint, and it gets there without shipping a single log line anywhere.

Where it sits in a monitoring stack

mtail is a sidecar, not a platform. It tails files (or reads from a named pipe), runs each line through the programs you wrote, and updates in-memory counters, gauges and histograms. It stores no logs and no history; restart it and the counters reset to zero, which Prometheus handles fine because rate() is built for counter resets. The typical deployment is one mtail per host, or one per pod in Kubernetes, next to whatever legacy thing writes the log: Postfix, HAProxy, a Java app with log4j and no Micrometer, a vendor appliance that only exports syslog. At 64 MB of RAM and a Go binary with no dependencies it costs almost nothing to run alongside.

What it is not is a log pipeline. If you also want to search the logs, Loki with LogQL's metric queries, or Vector with a log_to_metric transform, can derive the same counters and keep the raw lines. Both are much bigger projects with more operational surface, and for anyone already running Loki they are the safer place to put this logic. mtail wins when you want metrics and nothing else, on a host where installing an agent that ships logs off-box is unwelcome or pointless.

The program language is small but not obvious

Programs are .mtail files. Each declares metrics at the top, then lists conditions (usually regexes) with actions in braces. The pieces people trip on:

Named capture groups are the way to pull labels out of a line, and the metric must declare those labels with by before you can index into it. Forgetting the by status declaration and then writing metric[$status]++ is the most common compile error I see.

Timestamps matter more than you expect. mtail will parse a timestamp from the line via strptime if you tell it to; if you do not, it uses wall-clock time at ingestion, which is fine for live tailing and wrong for replaying an old file.

Histograms need explicit buckets: histogram latency_seconds buckets 0.01, 0.05, 0.1, 0.5, 1, 5. And gauge versus counter is a real distinction: a counter can only go up, and assigning to one is an error.

Test programs with mtail --compile_only --progs ./dir before deploying, and keep a sample log next to each program. There is also a --one_shot mode that processes a file to completion and dumps the metrics, which doubles as a unit test.

Regex on every line is the cost model

Every line runs through every condition in every loaded program. On a quiet host this is invisible. On a box logging 5,000 lines per second with five programs of unanchored, backtracking-prone regexes, mtail becomes the top CPU consumer. Anchor patterns with ^, put the cheapest condition first, use const to share sub-patterns, and split rarely-matching programs so their regexes stay simple. Also point --logs at specific files rather than a glob over a directory of rotating logs unless you need it; mtail handles rotation, but each extra file is another goroutine and another set of reads.

A working deployment is a systemd unit

[Unit]
Description=mtail
After=network.target

[Service]
ExecStart=/usr/local/bin/mtail --progs /etc/mtail --logs /var/log/nginx/access.log --port 3903
Restart=always
User=mtail
SupplementaryGroups=adm

The adm group is what lets it read /var/log on Debian-family systems. Add a scrape job for port 3903 to Prometheus, and the Grafana panel is sum by (status) (rate(http_requests_total[5m])). Alert on the 5xx series and you have turned a log that nobody read into a pager rule in under an hour. Docker works too, but bind-mounting a host log directory read-only into a container adds a layer for no gain when the binary already has nothing to install.

What I'd do

Use mtail for exactly the case above: one or two log-only services per host that need a handful of counters and an alert, with the programs kept in git next to a sample log and a --compile_only check in CI. Keep programs under 20 lines each and anchored. If you find yourself writing a fourth program, or you want to actually read the lines behind a spike, that is the signal to move the job into Loki and let mtail go. Everything else that belongs in this stack is under monitoring.