Multi-worker Celery¶
What this is¶
How PiHerder runs backup, Move, and the exclusive jobs (patch, checks, stack, templates, host reboot) on Celery: pool slots, optional multiple worker containers, and a per-server Redis mutex so one host is never rsync’d twice at once. Exclusive jobs do not take that mutex.
Why it exists¶
Large fleets want parallel backups; a single host’s destination tree must stay consistent. Mutex + concurrency knobs give both without inventing a second queue system.
Backups run in parallel across different hosts. The same host never has two active backups (Redis mutex piherder:server_lock:backup:{server_id}). Move takes that same mutex on both source and dest (lower server id first) so a backup cannot overlap a copy.
| Concept | Meaning |
|---|---|
| Node | One Celery worker process/container |
| Pool slots | Prefork children (CELERY_CONCURRENCY) |
Default: 1 node · 2 pool slots. Prefer raising CELERY_CONCURRENCY before scaling containers.
Jobs records the Celery nodename (hostname on the task request) when a worker claims the row, including while it is still pending on SSH or the per-host lock. The Jobs list, job detail, and JobHold show that name. A queued row no worker has taken says not claimed. Scaled containers must use distinct nodenames (default celery@<container hostname>, nmap worker nmap@%h) so the UI can tell them apart. A redelivery that fails a job already running keeps the name of the worker that started it.
| Knob | Default | Notes |
|---|---|---|
CELERY_CONCURRENCY | 2 | Pool slots per Celery node |
PIHERDER_SERVER_LOCK_TTL | 7200 | Lock TTL if worker dies mid-rsync. Not refreshed during a long copy — a Move/backup longer than this can lose the mutex. Raise the TTL for huge datasets. |
| Task hard limit | 7200 | Host backup, Move, and exclusive jobs. A Drive copy may run for 7 days and is acknowledged when the worker receives it. |
| Redis visibility | 10800 | An unacked task is handed out again after 3 hours, which is longer than the 2-hour hard limit. |
| Shared volumes | required | Same /backups (and herder/data mounts) on web + celery-worker |
| Cancel | — | Revoke via celery_task_id; mutex released in finally |
Optional multi-container:
# remove fixed container_name from celery-worker in compose, then:
docker compose up -d --scale celery-worker=N
Status shows N nodes and sum of pool slots.
Auth rate limiting¶
Login / 2FA attempt limits are in-process memory (per web process). With the default single Uvicorn worker this is fine. If you run multiple web replicas, each process has its own counter — prefer a reverse-proxy rate limit or a future Redis-backed limiter for HA.
Exclusive jobs on the default queue¶
OS patch, container patch, OS/container/stack update checks, compose stack mutations, template deploy / redeploy / drift check, and host facts run as app.tasks.exclusive_job on the default Celery queue (the same celery-worker as backups). They do not take Move’s dual-host backup mutex. retention and the herder’s own backup run as app.tasks.housekeeping_job on that same queue and do not take a host exclusive slot. nmap stays on celery-worker-nmap (-Q nmap). host_bootstrap is app.tasks.host_bootstrap_job on the default queue. It acks on receipt, does not retry, and the message expires after 120 seconds. A worker crash, an expired message, or a host that already has a bootstrap running fails that job. The sudo password is not redelivered.
Recycling web does not fail these exclusive jobs. Recycling celery-worker while one is running fails it (a patch or compose action is not resumed mid-flight). If the host’s SSH is down when the worker picks the job up, the row stays pending, the worker probes again every 30 seconds, and the exclusive slot stays held. The wait ends when SSH works or after the max wait. Default 30 minutes. Change it under Settings → General → Jobs. PIHERDER_EXCLUSIVE_HOST_WAIT_SEC locks that field when it is set (minimum 30 seconds, maximum 86400). Uptime Kuma showing the host down, or last_seen older than 15 minutes, can label the job waiting on host. Only a successful SSH probe resumes it. Neither signal fails the job.
| Job family | Execution | Parallelism rule |
|---|---|---|
backup | Celery | Many hosts in parallel; one backup per host (Redis mutex) |
service_migrate | Celery | Dual-host backup mutex + DB exclusive with stack mutation on both ids. Recycle web is safe; recycle worker fails a running Move |
service_migrate_undo | Celery | Same mutex. Only a failed post-flip Move. compose stop on dest, never down -v. Recycle worker fails a running undo and leaves the dest tree |
service_migrate_dest_recover | Celery | Same mutex. Only a Move that died during dest_up. Inspect dest, then compose stop and start source. DNS and NPM stay. Recycle worker fails a running recover |
os_patch / container_patch | Celery default queue | One active job of that type per host (DB exclusive). No backup mutex. Refused while host_reboot is active |
host_reboot | Celery default queue | One reboot per host. Also refused while os_patch, container_patch, or backup is pending or running on that host |
os_update_check / container_update_check / docker_stack_check | Celery default queue | One active check of that type per host. SSH down keeps the job pending |
docker_stack_deploy / _stop / _start / _restart / _down / _remove / template deploy and redeploy | Celery default queue | One active stack mutation per host (shared lane) |
template_drift_check | Celery default queue | One drift check per host. Not a stack write |
host_facts | Celery default queue | One active snapshot per host. SSH down keeps it pending |
retention | Celery default queue | No host exclusive slot. A web recycle does not fail it |
herder_backup | Celery default queue | No host. One PiHerder backup at a time. A web recycle does not fail it |
Raising CELERY_CONCURRENCY or adding Celery nodes does not cause a single container patch to run twice. Double-triggers from the UI or bulk queue attach to the existing job instead (HTTP 409 on the API). A single-host patch does not block a backup or Move on a different host.