Apache Airflow

Last reviewed Aug 26, 2026 · 969 words

Most people who type "self-hosted Airflow" into a search box want cron with a web UI and retries, and Airflow is the wrong answer to that question. It is an orchestrator for Python data pipelines, it needs 4 GB of RAM before you have written a single task, and the official quick-start compose file launches 7 containers. Self-host it when you already write DAGs at work and want the same tool at home, or when a Postgres-backed job history with dependency graphs is the actual requirement. For everything else, n8n or Kestra get you to a scheduled job in a tenth of the time.

The 4 GB minimum is real, and it is mostly Python

The catalogue lists 4096 MB as the floor, and that is not padding. A running Airflow 3 is a scheduler process, a DAG processor, a web/API server, a triggerer for deferred tasks, and a metadata database, each a separate Python interpreter with the full Airflow import tree loaded. Add the Celery worker, Redis, and Flower from the stock compose and you are past 3 GB idle on a machine doing nothing. On a 4 GB VM it runs; on an 8 GB one it stops swapping during DAG parsing. I would not put it on the same box as a media server.

Drop Celery: LocalExecutor fits a single machine

The quick-start compose exists to demonstrate the distributed shape, not to recommend it. For one box, LocalExecutor runs tasks as subprocesses of the scheduler and removes Redis, the worker, and Flower entirely. The DAG processor stays, because Airflow 3 made it a required standalone service. That is the compose I actually run:

x-airflow-common: &airflow-common
  image: apache/airflow:latest
  environment:
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    - postgres

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - ./pgdata:/var/lib/postgresql/data
  airflow-api-server:
    <<: *airflow-common
    command: api-server
    ports:
      - "8080:8080"
  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
  airflow-dag-processor:
    <<: *airflow-common
    command: dag-processor

Set AIRFLOW_UID=$(id -u) in a .env file first, or the containers cannot write to ./logs and the scheduler dies with a permissions error rather than a useful one. Run airflow db migrate once against this stack before starting the long-lived services, then create your first login the way your auth manager expects: Airflow 3 defaults to the simple auth manager (initial credentials land in the logs), while the older FAB manager wants airflow users create. LOAD_EXAMPLES=false matters more than it looks: the 40-odd example DAGs otherwise get parsed every 30 seconds forever.

Postgres, not SQLite, from the first minute

Airflow supports SQLite for local experiments and nothing else. It serialises every scheduler write, forbids parallel task execution, and the migration path to Postgres later is a database dump you will not enjoy. Start on Postgres, and treat the metadata database as the one thing worth backing up: DAG code lives in git, but run history, XComs, variables, and connections live only in that database. My Postgres for everything note covers the nightly pg_dump pattern that fits here.

Top-level DAG code runs every 30 seconds

The gotcha that catches every newcomer is parse-time versus run-time. The DAG processor imports every file in dags/ on a loop (the min_file_process_interval default is 30 seconds), so any code at module level runs on that cadence. A requests.get() or a database query at the top of a DAG file becomes a request every 30 seconds, times however many parsing processes you run. Keep module level to imports and DAG definition; put the real work inside task functions. The airflow dags list-import-errors command shows what the parser is choking on when a DAG silently fails to appear.

Two settings to write explicitly on every DAG: catchup=False, because older Airflow defaults would backfill every missed interval since start_date the moment you unpause a DAG, and a static start_date in the past, because a dynamic datetime.now() breaks scheduling in ways the UI does not explain.

Where the difficulty rating comes from

The "Hard" label is not about installation. It is that Airflow has an opinion about time that takes weeks to internalise (a run with a logical date of Monday executes on Tuesday, after the interval closes), an operator ecosystem where the provider package you need may be pinned to a dependency that conflicts with your code, and a UI that surfaces state across DAG, run, and task-instance levels that all mean slightly different things. The official docs are thorough on all of it, but they assume you will read them.

If your jobs are "back up this folder at 02:00" or "ping this API and post to a channel", none of that complexity buys you anything. The automation category has tools that were built for exactly that shape, and the people who evaluate Airflow as an AWS Step Functions replacement are usually the ones who genuinely need it.

What I'd do

If you build data pipelines professionally: the LocalExecutor compose above on a dedicated 8 GB VM, Postgres 16 with a nightly dump, DAGs in a git repo mounted read-only, examples disabled, catchup=False in a shared default-args dict. Upgrade Airflow on a deliberate schedule after reading the release notes, because provider packages and the core move independently. If you do not build data pipelines professionally: install n8n instead, and come back to Airflow when a job has more than 5 dependent steps and a history you need to query.