Fluentd

Last reviewed Aug 26, 2026 · 781 words

Run Fluent Bit unless you need one of three things: a plugin that only exists for Fluentd, heavy per-record rewriting and routing, or a central aggregator that receives from many lightweight agents. Fluent Bit is written in C, idles in a few megabytes, and covers the "tail these files and ship them to Loki" case completely. Fluentd is the Ruby original: heavier (the catalogue's 128 MB minimum is realistic and a busy instance wants more), slower to start, and backed by more than 1,000 plugins and a CNCF graduation that means it is not going anywhere. Same project family, different jobs.

Source, filter, match: the whole config model

Every Fluentd config is a pipeline of three block types. <source> blocks accept events and tag them, <filter> blocks modify events whose tags match, and <match> blocks send them somewhere. A config that receives Docker logs, parses JSON lines, and ships to Loki looks like this:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter docker.**>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type json
  </parse>
</filter>

<match docker.**>
  @type loki
  url http://loki:3100
  <label>
    container $.container_name
  </label>
  <buffer>
    @type file
    path /fluentd/buffer/loki
    flush_interval 5s
    retry_forever true
  </buffer>
</match>

Swap the match block for @type opensearch with host opensearch and port 9200 to land in OpenSearch instead, or @type file to write to disk while you decide. Tags are what route events, and the glob is the thing to learn first: docker. matches docker.nginx, docker.app.web, and so on.

Docker's fluentd log driver is the quickest win

Docker can send container output straight to port 24224 with no file tailing at all:

services:
  nginx:
    image: nginx:stable
    logging:
      driver: fluentd
      options:
        fluentd-address: 127.0.0.1:24224
        fluentd-async: "true"
        tag: docker.{{.Name}}

fluentd-async: "true" is not optional in practice. Without it, a container using this driver refuses to start if Fluentd is down, which turns a logging outage into an application outage during a reboot when start order is unpredictable. With it, Docker buffers locally and reconnects. The trade is that docker logs no longer works for that container, because the output went to Fluentd rather than to Docker's own json-file store.

Buffering is the feature you are paying the RAM for

The <buffer> block above is the reason to run Fluentd at all. With @type file, events are written to disk before delivery; if Loki or OpenSearch is unreachable, Fluentd retries with backoff and nothing is lost until the buffer path fills. Set total_limit_size (for example 2g) to cap it, and flush_thread_count 2 if the destination can take parallel writes. retry_forever true is right for a homelab where the destination will come back; on a shared host prefer retry_max_times 20 so a dead destination eventually stops filling the disk. Fluent Bit has file buffering too, but Fluentd's is more configurable and its plugins expose more of it.

Plugins live in an image you build yourself

The official fluent/fluentd image ships with almost no output plugins. Loki, OpenSearch, and most other destinations need a gem installed, which means a small Dockerfile:

FROM fluent/fluentd:edge
USER root
RUN gem install fluent-plugin-grafana-loki fluent-plugin-opensearch --no-document
USER fluent

Pin the base tag to a specific release once it works. The plugin ecosystem is the trade-off in both directions: there is a plugin for nearly every destination, and each one is a separate project with its own release cadence and occasional breakage against a new Fluentd minor.

Fluentd against the two tools people confuse it with

The pattern that scales is Fluent Bit on every host forwarding to one Fluentd aggregator that does the parsing, enrichment, and fan-out. For a single homelab box that is more than you need.

What I'd do

One box, a handful of containers: Fluent Bit or Grafana Alloy straight to Loki, and skip Fluentd. Several hosts, a legacy app that only logs to syslog, and a destination that needs a plugin: Fluent Bit at the edge, one Fluentd aggregator with file buffering and a 2 GB cap, Loki as the store. Either way the monitoring category is where the visualisation half of this lives; Fluentd only moves the logs, it does not show them to you.