Celery

Last reviewed Aug 26, 2026 · 824 words

The Celery bug that eats the most weekends is a number: 3600. With Redis as the broker, any task that runs longer than the 1-hour visibility timeout is handed to a second worker while the first is still running it, and you find out as a duplicated email, a doubled invoice, or a file processed twice. Celery is 17 years old, BSD-licensed, and the default background-job system for the Python world, and none of that shows you the operational surface. This guide is that surface, because a task queue is not an app you install; it is a component you run inside one.

You are running 3 processes, not 1

A Celery deployment is a broker, one or more workers, and optionally a scheduler and a result store. The broker is Redis or RabbitMQ; the documentation treats RabbitMQ as the reference transport, and nearly everyone uses Redis because it is already there. Workers are your application code started in worker mode:

celery -A myproject worker --loglevel=info --concurrency=4
celery -A myproject beat --loglevel=info

The prefork pool defaults to one process per CPU core, and for I/O-heavy tasks you can run far more. A result backend (Redis again, or the database) is only needed if callers wait for return values; fire-and-forget tasks can skip it and save the round trips. A minimal setup is about 256 MB of RAM for a worker plus whatever the broker needs, which is why it fits on the same box as the app it serves.

The four settings to change before the first production task

The defaults favour throughput over safety, and the fix is a short block in the Celery config:

broker_transport_options = {"visibility_timeout": 43200}  # 12 h, longer than any task
task_acks_late = True
task_reject_on_worker_lost = True
worker_prefetch_multiplier = 1
task_time_limit = 3600

acks_late means a task is acknowledged after it finishes rather than when it is received, so a worker that dies mid-task hands it back instead of losing it. That only helps if the task is idempotent, so write tasks that can run twice: check before you send, upsert instead of insert. prefetch_multiplier=1 stops a busy worker hoarding 16 tasks while its neighbours sit idle. --max-tasks-per-child=500 on the worker command line recycles processes and is the standard cure for slow memory growth in long-lived workers.

Beat is cron with exactly one allowed copy

Celery Beat is the scheduler for periodic tasks, and it has a single rule: run one instance. Two Beat processes, which happens the first time a deploy scales a container to 2 replicas, means every scheduled job fires twice. Keep it as its own service with replicas: 1, and if the schedule needs editing at runtime, the django-celery-beat package moves it into the database with an admin UI. The schedule itself is plain Python, crontab syntax included, so crontab(hour=3, minute=0) runs nightly at 03:00 in the worker's configured timezone, which you should set explicitly rather than trusting the container default.

Watching it: queue length is the only metric that matters

Flower is the standard dashboard, a separate process on port 5555 that shows workers, task history and rates; put it behind auth, because it can revoke and retry tasks. The single number to alert on is queue depth: redis-cli llen celery for the default queue. A queue that grows for 10 minutes means workers are dead, stuck, or under-provisioned, and every other symptom follows from that. A Celery exporter for Prometheus exists if you already graph everything; for a homelab, a cron job that checks llen and pings you is enough.

Where you already run it, and where you should not

If you host Paperless-ngx, you run Celery: its document consumption pipeline is Celery workers over a Redis broker, which is the whole reason the compose file has a Redis container. Netbox, Sentry and a long list of Django applications do the same, so understanding the worker and broker relationship pays off across your stack. What Celery is not is a general automation tool. If the jobs are not Python code inside your own application, n8n for glue between services, Windmill for scripts with a UI, or Temporal for long-running workflows that must survive restarts all fit better than bolting a task queue onto a project that has no app to attach it to.

What I'd do

Redis as broker, one worker service sized to CPU cores, one Beat service pinned to a single replica, and the five-line safety config above committed before the first real task ships. Make every task idempotent and set the visibility timeout above the longest one. Alert on queue length, ignore most of the rest. Celery run this way is boring for years, and boring is the whole point of a mature queue.