Luigi

Last reviewed Aug 26, 2026 · 807 words

Luigi is 14 years old, was written at Spotify to chain Hadoop jobs, and remains the smallest way to turn a handful of Python scripts into a dependency-aware batch pipeline that does not re-run what already finished. You should pick it when your problem is "run B after A, only if A's output exists, and tell me which step failed", and you should skip it when you need a scheduler, because Luigi does not have one. That gap is not a bug. Luigi assumes cron, or whatever else triggers you, and it concentrates on the part cron is bad at: knowing what is done.

A pipeline is three methods per task

Every Luigi task is a class with requires() for what must come first, output() for the file or table it produces, and run() for the work. The output existing is the definition of done; on the next run, tasks whose targets exist are skipped without executing.

import datetime
import subprocess
import luigi


class DumpDatabase(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())

    def output(self):
        return luigi.LocalTarget(f"/srv/backups/db-{self.date}.sql.gz")

    def run(self):
        with self.output().temporary_path() as tmp:
            subprocess.run(f"pg_dump app | gzip > {tmp}", shell=True, check=True)


class ShipOffsite(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())

    def requires(self):
        return DumpDatabase(self.date)

    def output(self):
        return luigi.LocalTarget(f"/srv/backups/db-{self.date}.shipped")

    def run(self):
        subprocess.run(["rclone", "copy", self.input().path, "b2:backups/"], check=True)
        with self.output().open("w") as f:
            f.write("ok\n")

luigi --module backup ShipOffsite --local-scheduler runs both in order. Run it again and nothing happens, because both targets exist. temporary_path() is the detail that makes it safe: the output only appears once run() finishes, so a crash mid-dump does not leave a half-file that the next run mistakes for success.

The central scheduler is a small web UI you can actually run

Passing --local-scheduler is fine for a single cron job. For anything with several concurrent invocations, run luigid, the central scheduler, as a service; it listens on port 8082 with a web UI that shows the dependency graph, running tasks and failures, and it prevents two workers running the same task at once. It holds state in memory with an optional state file, needs on the order of 100 MB, and is one pip install luigi plus a systemd unit away. It still does not trigger anything. A homelab pipeline is therefore cron at 02:00 calling luigi --module ..., luigid on the side so you can see what happened, and a failure notification configured through the [email] section of luigi.cfg or, more usefully, a ping to a dead-man's switch such as Healthchecks from the last task.

Where it sits against Airflow, Prefect and Dagster

Apache Airflow has a real scheduler, a metadata database, a large UI and a memory footprint to match; it is the right choice when dozens of DAGs run on their own calendars. Prefect and Dagster are the modern Python takes with richer observability and asset-oriented models, and both run a server component you host. Luigi wins on exactly one axis: it is a library, not a platform. The 256 MB figure is for luigid; the pipeline itself is your own Python process. For a self-hoster with three to ten batch jobs that already live in cron, that is the whole reason to choose it. For a data team with SLAs, it is the reason not to.

What it does not do, stated plainly

No scheduling, no retries beyond a configurable retry count on the central scheduler, no backfill UI, no notion of streaming, and a development pace that is accurately described as maintenance rather than growth: releases still arrive, but the project is not chasing features. The contrib modules for Postgres, S3, Hive and Spark work but expect the client library versions that were current when they were written. If you find yourself writing a task that polls, or a task that waits for a time, you have outgrown it.

What I'd do

Use Luigi for the batch chores that already exist as scripts: database dumps, media transcodes, report generation, archive rotation. One module per pipeline, LocalTarget outputs with temporary_path(), luigid under systemd on port 8082 bound to the LAN, and cron doing the triggering. Add a Healthchecks ping to the final task so silence is an alert. The moment you want a calendar of schedules, backfills or per-run dashboards, move to Prefect or Dagster and keep the Luigi task bodies as the migration path; the run() methods port almost unchanged. Until then, 200 lines of Luigi beat a platform you have to operate.