Dagster

Last reviewed Aug 26, 2026 · 800 words

Dagster charges its learning cost up front. You spend the first afternoon unlearning "tasks that run on a schedule" and relearning "assets that get materialised when stale", and about a third of people bounce off at that point. Those who stay tend not to go back to Airflow. Self-hosting the open-source edition is 4 processes plus a Postgres, fits in the listed 2 GB minimum for small pipelines, and gives you the same web UI the paid product uses.

An asset is a thing, not a step

In Airflow you describe steps; in Dagster you describe the tables, files and models your pipeline produces, and the dependency graph falls out of function arguments:

from dagster import asset

@asset
def raw_events():
    return fetch_from_api()

@asset
def daily_summary(raw_events):
    return summarise(raw_events)

Because daily_summary takes raw_events as a parameter, Dagster knows the lineage, shows it in the UI, and can answer "which assets are stale and why" without you writing a DAG by hand. Schedules and sensors then say when to refresh which assets. The payoff is that partial reruns, backfills and "rebuild everything downstream of this table" are first-class operations rather than scripts you write at 2 a.m. The cost is exactly the catalogue's one listed drawback: the model takes real time to internalise.

Four processes make a deployment

dagster dev runs everything in one process for a laptop. A real deployment splits into the webserver (dagster-webserver, port 3000 by default), the daemon (dagster-daemon run, which owns schedules, sensors and the run queue, and without which nothing fires automatically), one gRPC code server per code location holding your pipeline code, and Postgres for run, event and schedule storage configured in dagster.yaml. workspace.yaml tells the webserver and daemon where the code servers are. The split matters because it lets you redeploy pipeline code without restarting the control plane, which is the operational win over monolithic schedulers.

The project publishes Docker images and an official Helm chart; on a single homelab node, Compose with the 4 services above plus Postgres is the sane path. Launched runs default to subprocesses of the code server, or to fresh containers if you configure the Docker run launcher, which keeps a memory-hungry pipeline from taking the UI down with it.

What it costs to run on a homelab

2 GB of RAM is the floor for the control plane alone; the pipelines themselves are Python processes and cost whatever your pandas or dbt work costs. Dagster is Apache-2.0, 16,061 GitHub stars, Python throughout, and the OSS edition is the same core the hosted product runs. It is not a tool for "back up my photos nightly"; for that a cron line or n8n wins. It earns its footprint when you have 10 or more interdependent datasets and someone will ask "why is this number wrong" at least weekly.

Dagster against the field

Airflow is more widely deployed and has more integrations; Dagster has the better local development story, built-in typing and asset checks that catch bad data before it propagates, and a UI that explains state instead of just listing runs. Prefect is easier to start with and less opinionated, which is either a feature or the absence of one depending on your team.

The paid line is a control plane, not features

Dagster+ is the company's hosted product: serverless or hybrid execution, branch deployments per pull request, alerting, insights and single sign-on. None of that is needed to run pipelines; the OSS webserver, daemon and code servers are the complete system. The things you lose self-hosting are convenience for pull-request previews and someone else patching the control plane, and the automation category has nothing else that combines lineage and testing this well for free.

What I'd do

Prototype with dagster dev on a laptop until the asset model clicks, which took me about 2 days of real use. Then deploy with Compose: webserver, daemon, one code server, Postgres on its own volume, and the Docker run launcher so runs are isolated. Put it on a box with 4 GB free rather than the 2 GB minimum, back up Postgres nightly, and pin versions because the project releases weekly. If the pipelines are fewer than 5 and nobody asks about lineage, skip all of it and use Prefect or cron; Dagster's value is proportional to how tangled the data already is.