Apache DolphinScheduler

Last reviewed Aug 26, 2026 · 760 words

Apache DolphinScheduler needs 2 GB of RAM before it runs a single job, is rated Hard in the catalogue, and is the correct choice for roughly 1 self-hoster in 50: the one running nightly Spark, Hive, Flink or multi-hour SQL pipelines across several machines. Everyone else wanting scheduled workflows should look at Kestra or n8n and keep their evening. That said, for the data-engineering case it is a serious, Apache-governed, 14,443-star project with a visual DAG designer that non-programmers can operate, and there is a way to try it that does not involve installing ZooKeeper.

Standalone mode is the only sane first install

Production DolphinScheduler is 4 Java services (master, worker, API server, alert server) plus a registry (ZooKeeper by default) plus a metadata database (MySQL or PostgreSQL). That is a week of homework. The project ships a standalone-server build that bundles all of it with an embedded H2 database and an in-process registry, and it is one container:

docker run -d --name dolphinscheduler \
  -p 12345:12345 -p 25333:25333 \
  apache/dolphinscheduler-standalone-server:latest

The UI is at http://host:12345/dolphinscheduler/ui and the default login is admin with password dolphinscheduler123. Change it in the first 5 minutes. Standalone mode loses your definitions when the container is recreated unless you point it at an external database, so treat it as a trial. Once you like it, the same image family runs the split services with a real Postgres and ZooKeeper; the compose file in the repo's deploy/docker directory is the template.

The visual DAG designer is the reason it exists

Most orchestrators are Python files. DolphinScheduler is a canvas: drag a Shell task, an SQL task, a Python task, a Spark task or an HTTP task onto the board, wire the dependencies, set the cron, save. Task types run to several dozen plugins (DataX, Sqoop, Flink, Kubernetes, Jupyter, MLflow, SageMaker, conditional and switch nodes, sub-workflows). Parameters flow between tasks, and a failed run can be rerun from the failed node onward rather than from scratch.

The feature nobody mentions until they need it is "complement data": backfill a workflow for a date range, and the scheduler runs one instance per day, serially or in parallel, with the date injected as a parameter. If you have ever written that loop by hand in cron, that alone justifies the evaluation.

Tenants and worker groups are Linux users and Linux boxes

Two concepts trip up new operators. A tenant maps to a Linux user on the worker machine; tasks execute as that user, so the user must exist and have the permissions and environment (Java, Python, Spark client) the task needs. A worker group is a named set of worker servers; you assign tasks to groups so that heavy Spark jobs land on the 64 GB node and the curl task does not. Both are why the Docker route gets awkward for anything beyond trials: the workers want real hosts with real toolchains, which is what "bare-metal" in the deployment list is politely telling you.

Where it loses to the alternatives

Against Airflow: DolphinScheduler is friendlier for people who do not want to write DAGs as code and has better multi-tenant isolation, but Apache Airflow has a far larger provider ecosystem and 10 years of Stack Overflow answers. Against Kestra: Kestra is YAML-declared, lighter (a single JVM plus Postgres), and a nicer fit for a homelab or a small team. Against AWS Step Functions or Azure Data Factory, which the catalogue lists as the tools it replaces: you trade a per-execution bill for a Java stack you must patch, and the trade only pays once you have enough runs. The catalogue's single con, "heavy Java-based stack", is the whole story; the JVM heap tuning in bin/env/dolphinscheduler_env.sh is where you will spend your first bad afternoon.

What I'd do

Real ETL across more than 1 machine with mixed Spark and SQL jobs: run the standalone container for 1 week, build 2 workflows, then deploy the split services on 2 Linux hosts with PostgreSQL 15 and a 3-node ZooKeeper, budget 4 GB for the master and API server and size workers to the jobs. Everything else in the automation category is a smaller hammer, and for a single-box homelab, Kestra gets you 80 percent of this with a quarter of the memory and no ZooKeeper.