Prefect

Last reviewed Aug 26, 2026 · 798 words

Prefect is the orchestrator for people who already have Python scripts on cron and want to know why last night's run failed without ssh-ing into the box. The whole pitch fits in 2 decorators: put @flow on a function, @task on the functions it calls, and you get retries, logging, a run history and a dashboard on port 4200 with no DAG files, no scheduler daemon, and no XCom. The open-source server is Apache-2.0, 23,686 stars, and idles happily in 1 GB of RAM. The catch is not the software; it is deciding whether you need an orchestrator at all.

Your existing script becomes a flow in 4 lines

Here is the entire conversion for a nightly backup-and-report script:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch(url: str) -> bytes:
    ...

@flow(log_prints=True)
def nightly():
    data = fetch("https://example.com/export.csv")
    print(f"got {len(data)} bytes")

if __name__ == "__main__":
    nightly()

Run it with python nightly.py and it executes like any script, except every task run, retry and print statement is now recorded. Because flows are plain Python, branching and loops are ordinary if and for, and a flow can call other flows. That dynamic model is the thing Prefect does better than Airflow, where a DAG's shape is supposed to be static.

The server is small; the workers do the work

pip install prefect then prefect server start gives you the API and UI on http://127.0.0.1:4200 backed by SQLite. That is fine for trying it and wrong for production, where you should point PREFECT_API_DATABASE_CONNECTION_URL at Postgres before the first real run; SQLite under concurrent flow runs is a well-worn way to get locked-database errors. If you already run Postgres for other services, the Postgres for everything pattern applies here without caveats.

The part that confuses newcomers is that the server does not run your code. It stores schedules and run state; a worker polls a work pool and executes flows. So a minimal production layout is 3 pieces: the server container, a Postgres database, and at least 1 worker started with prefect worker start --pool my-pool on a machine with PREFECT_API_URL=http://server:4200/api set. A flow is "deployed" by registering it against a pool with a schedule, after which the worker picks it up. Keep the worker on the same host as the data it touches; the server can live anywhere. The official prefecthq/prefect image covers both roles, and a Helm chart exists for the Kubernetes crowd.

What stays in Prefect Cloud

The listed con is "some features tied to Prefect Cloud", and the boundary is worth knowing before you commit. The open-source server gives you scheduling, work pools, the full UI, and, since Prefect 3, events and automations (run this flow when that one fails, alert on a stuck run). What remains Cloud-only is the organisational layer: SSO, role-based access control, audit logs, and the managed push work pools where Prefect runs your infrastructure for you. For a homelab or a single data team, none of that bites. For a company that needs per-user permissions on who can trigger production flows, you either accept a single shared login behind your own identity provider or pay for Cloud.

Airflow, Dagster, or Prefect

Apache Airflow is the incumbent, has the largest provider ecosystem, and is the right answer when you hire people who already know it or need its hundreds of prebuilt operators. It is also heavier to run and clumsier for dynamic workflows. Dagster is the asset-oriented alternative: excellent when your pipelines are really "materialise these tables in order", opinionated in ways that take a week to internalise. Prefect wins when the mental model you want is "my Python, scheduled and observed", which is most self-hosters and most small data teams. And if the jobs are not Python at all, a general automation tool like n8n or Windmill may be the better fit than any of the three.

What I'd do

Postgres plus the Prefect server in one compose file on a VPS with 2 GB of RAM, one worker on the box where the data lives, and TLS from a reverse proxy in front of port 4200. Convert one nagging cron job first, the one that fails silently, and leave the rest alone until that one has run for a fortnight. Move the others only as they earn it. If you find yourself wanting SSO and per-user roles, that is your signal that the project has grown past homelab scale and the Cloud tier is priced for exactly that moment.