Apache NiFi

Last reviewed Sep 13, 2026 · 849 words

Apache NiFi is the wrong tool for 9 out of 10 automation jobs a self-hoster has in mind, and the right one for the tenth in a way nothing else in the automation category can match. The 2 GB RAM floor and the "Hard" rating are not exaggerations; the JVM alone wants more memory than most of your other containers combined. Where it earns that cost is moving large volumes of data between systems with guaranteed delivery, backpressure, and a provenance record showing exactly what happened to every single file. If your job is "when X happens, call Y's API", run n8n and keep the 2 GB.

The word that separates NiFi from the workflow tools is "flowfile"

n8n and Node-RED pass JSON messages between nodes. NiFi passes flowfiles: a content blob of any size plus attributes, stored on disk in a content repository, with every transition recorded in a provenance repository. That design means a 4 GB video, a 10 million-row CSV, or a stream of 5,000 syslog lines per second all flow through the same processors without loading into memory. It also means a crash mid-flow loses nothing, because the queue state is on disk. That durability is the reason organisations moving data between a hospital's systems, or shipping logs from 200 hosts into a lake, pick NiFi. It is also why it is overkill for polling an RSS feed.

Running it: 2.x made the defaults safe and the setup stricter

Since the 2.0 line NiFi requires Java 21, listens on HTTPS only, and refuses to start with a weak single-user password. The container generates credentials if you do not supply them, and they appear once in the log, which surprises people who expect a plain web UI on first boot.

services:
  nifi:
    image: apache/nifi:latest
    ports:
      - "8443:8443"
    environment:
      - SINGLE_USER_CREDENTIALS_USERNAME=admin
      - SINGLE_USER_CREDENTIALS_PASSWORD=a-passphrase-of-twelve-or-more
      - NIFI_WEB_PROXY_HOST=nifi.example.com:443
      - NIFI_JVM_HEAP_INIT=1g
      - NIFI_JVM_HEAP_MAX=2g
    volumes:
      - ./nifi/database_repository:/opt/nifi/nifi-current/database_repository
      - ./nifi/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi/content_repository:/opt/nifi/nifi-current/content_repository
      - ./nifi/provenance_repository:/opt/nifi/nifi-current/provenance_repository
      - ./nifi/conf:/opt/nifi/nifi-current/conf
    restart: unless-stopped

The password must be at least 12 characters or startup fails. NIFI_WEB_PROXY_HOST has to match whatever hostname you reach it through, or the UI rejects the request as a proxy mismatch; that one error costs more first-time users an evening than anything else. And mount all four repositories, because the content repository fills disk fast under real load and you want it on a volume you can size.

Backpressure is the feature you will not appreciate until it saves you

Every connection between two processors has a queue with a size threshold, 10,000 flowfiles or 1 GB by default. When a downstream system slows, the queue fills, the upstream processor stops being scheduled, and pressure propagates back to the source without dropping anything. Prioritisers on the same queue let urgent data jump the line. Workflow tools tend to have none of this; they fire, retry a few times, and give up. When you are draining a Kafka topic into a database that just went into maintenance, the difference between the two behaviours is a Monday-morning incident.

Provenance answers "where did this record go"

Right-click any processor, open data provenance, and NiFi shows each flowfile's lineage as a graph: received here, attributes modified there, content replaced, routed on this relationship, sent to that endpoint, with timestamps and the content at each step available for download. It is the single feature that regulated industries buy NiFi for, and it is fully present in the open-source Apache-2.0 build. The cost is disk: provenance repositories on a busy instance grow by GBs a day, and the retention settings in nifi.properties are the first thing to tune.

Who should stay away

Anyone whose automations are event-driven glue between SaaS APIs. Anyone on a box with less than 4 GB total, because NiFi at 2 GB plus the rest of a homelab will swap. Anyone who wants a scheduler for batch jobs with dependencies, which is Apache Airflow territory. And anyone who expects a quick win: the processor catalogue has hundreds of entries, the UI is a canvas rather than a wizard, and a competent first flow is a weekend, not an hour.

What I'd do

Reach for NiFi only when the problem is stated in bytes per second or files per hour and the words "must not lose any" appear in the requirement. Give it a dedicated VM or a 4 GB LXC, set the heap to 2 GB, put the content repository on real disk, and front it with a reverse proxy that sets the proxy host correctly. For everything else that looks like automation in a homelab, n8n for API glue and Node-RED for anything involving MQTT or sensors will do the job on a tenth of the memory.