Mage

Last reviewed Aug 26, 2026 · 904 words

Mage sits in the gap between "a cron job that runs a Python script" and "we operate an Airflow cluster", and for a self-hoster with 2 to 20 data pipelines it fills that gap better than either. A pipeline is a folder of blocks (load, transform, export), each block is a Python or SQL file you edit in a notebook-like browser UI on port 6789, and the scheduler, run history and retries live in the same container. Apache Airflow gives you more; it also asks for a metadata database, a scheduler, a webserver and a worker before it will run hello world.

Where it sits in a homelab stack

Mage is not a service you use; it is a service that moves data between the services you use. The pattern that makes sense at home is a small warehouse, usually Postgres because it is already running for everything else, fed by Mage pipelines and read by Grafana or a notebook. Sources that people actually wire up: the electricity supplier's API pulled nightly, bank CSV exports dropped into a folder, Plausible or Umami analytics exported for long-term retention, a Home Assistant history dump, a broker's activity export headed for Ghostfolio. Destinations are Postgres, DuckDB files, ClickHouse if you have gone that far, or S3-compatible storage.

What it is not is a glue tool. If the job is "when a form is submitted, post to Matrix and create a ticket", that is n8n, and the two coexist happily: n8n for events and integrations, Mage for tabular data on a schedule. The automation category mixes both kinds, and picking the wrong one for the job is the most common mistake in it.

One container, a project folder, two variables

services:
  mage:
    image: mageai/mageai:latest
    command: mage start homelab
    ports:
      - "6789:6789"
    environment:
      MAGE_DATABASE_CONNECTION_URL: postgresql+psycopg2://mage:change-me@postgres:5432/mage
      REQUIRE_USER_AUTHENTICATION: "1"
    volumes:
      - ./mage:/home/src
    restart: unless-stopped

mage start homelab creates a project named homelab in the mounted folder on first run; every pipeline, block and trigger you create in the UI becomes a file under it, which means the whole thing is a git repository and can be diffed and restored. The metadata store defaults to SQLite inside the project, which is fine for one person; pointing MAGE_DATABASE_CONNECTION_URL at Postgres is what you do before relying on the run history. REQUIRE_USER_AUTHENTICATION turns on the login screen, and the default owner credentials must be changed on first login because without the variable the UI is wide open.

The 2 GB minimum is real. The container carries pandas, a Jupyter-style kernel per pipeline while you are editing, and whatever connectors you install. On 1 GB it starts and then falls over the first time a block loads a real dataset.

The in-memory trap

Blocks hand data to each other as DataFrames in memory. That makes the notebook experience pleasant, since you can see the output of each step as a table, and it means a pipeline that loads a 5 GB table into the first block needs more than 5 GB of RAM to get to the second one. The fixes are the ones data engineers already know: push the filtering into SQL so the loader returns only what the transform needs, process in date-bounded chunks with a backfill trigger, or write intermediate results to the warehouse and read them back. Mage supports all three; it just does not stop you from doing the naive thing on a homelab box that cannot afford it.

Triggers are where the scheduler lives: cron-style schedules, API triggers for "run when this webhook fires", and event triggers. Backfills re-run a pipeline over a date range, which is the feature you will be grateful for after a source API was down for 3 days. The data integration pipeline type wraps Singer taps and targets, so common SaaS sources come as configuration rather than code.

Mage against Airflow, Dagster and n8n

Airflow is the answer when a team already knows it or the pipeline count is in the hundreds. Dagster is the answer when you want tests and typed data assets and are happy in Python. Mage is the answer when one person wants to build 10 pipelines this month and see the data as they go, which is most homelabs and most small teams leaving Azure Data Factory.

What I'd do

Run the compose above next to the Postgres you already have, set the metadata URL to Postgres from day one, and commit the project folder to git. Build the first pipeline as a scheduled nightly load of one API into one table, watch the block outputs in the UI, then add a Grafana panel on top. Keep n8n for anything that starts with a webhook. If the pipeline count crosses about 50 or a second team wants in, revisit Airflow or Dagster; below that, Mage is the least operations for the most data.