Jobs, audit & notifications¶

What this is¶

Three related systems — do not confuse them:

Open notifications via the bell icon (no separate Alerts nav link).

Why they exist¶

Long SSH work must not block the browser (jobs). Homelab and multi-operator setups need accountability (audit). Noise should not bury failures (notifications + optional Web Push). Together they replace “did that cron fire?” shell archaeology. About keeps a short count. View schedule groups jobs by type and lists each host with its status, the latest matching job, and the next time it will run. A saved per-host docker_prune is one of those types. A restart does not clear that time. A missed slot is not a run.

End-to-end: follow one backup through all three¶

1. Start a backup from a server Backups page.
2. Jobs → row goes pending → running → success / failed; open detail for the log tail. A host that sends files straight out names the destination in that detail: Google Drive or OneDrive and the folder, or the LAN host, share, and path, plus each source folder on that destination. The sign-in is not in the row.
3. Audit → filter that server / backup actions; see request → queued → running → complete phases and client IP. The completed row for a straight-out host names the same destination.
4. If it failed earlier, a notification may open; on success, related open alerts can auto-resolve (optional push: Resolved: …).
5. Dismiss the inbox item when you have acted.

Jobs¶

Where: nav Jobs (/jobs) · compact panel on each server detail.

Job types (examples)¶

Statuses: pending → running → success / failed.

A host-facts row still pending or running after its task is gone is failed on the fleet tick (30 minutes for host facts, 2 hours for the other exclusive types and for backups). The host leaves refreshing, and a failed refresh keeps the previous OS and kernel.

A running job whose Celery worker is gone is failed with Interrupted. The Celery worker is no longer alive. The check runs when a worker starts and on the host-facts fleet tick. If workers answer but the task id is missing, the job waits 60 seconds. If the worker check itself fails, running jobs are left alone. If no worker answers, the job fails immediately. Web startup does not run this check. This reconcile ships in 1.13.0. docker_build_cache_prune is an audit action for the build-cache prune, not a job type. host_bootstrap is an audit action for the SSH Bootstrap missing tools run. It records the actor, the OS id and codename, and whether rsync or docker_ce was installed. The one-time sudo password is Fernet-encrypted in the broker only until a worker picks it up, acked on receipt, expires after 120 seconds, is never retried or redelivered, and is never logged. A worker crash marks the job failed and the operator reruns with a fresh password. If no worker takes the message, the pending job fails with expired before a worker picked it up, rerun with a fresh password. A second bootstrap while one already holds the host fails at once with another job holds this host, rerun when it finishes, and the password is not queued. The audit row does not record that password.

Web restart: retention, the herder’s own backup, host facts, OS/container patch, host reboot, update checks, stack jobs, template jobs, backups, nmap, and Move are Celery — a web recycle does not fail them. Recycle celery-worker while one of those is running fails that job (it is not resumed mid-flight). If SSH is down at the start of a patch or stack job, the row stays pending and is probed again until the host answers or the wait limit (Settings → General → Jobs, default 30 minutes). A Kuma “host down” alert or a last_seen older than 15 minutes can show waiting on host on that pending job. Those are labels. The job resumes only when SSH works, and they do not fail it. See Multi-worker.

Exclusive jobs (one per type per host)¶

These types do not stack on the same server while already pending or running:

• os_patch, container_patch, host_reboot (also waits for a patch or backup, and those wait for a reboot)
• os_update_check, container_update_check
• Stack lifecycle + template deploy/redeploy (shared stack mutation lane on the host)
• service_migrate, service_migrate_undo, and service_migrate_dest_recover — exclusive with backup and stack mutation on both source and dest
• template_drift_check (one drift job at a time per host; not a stack write)
• host_facts (one snapshot job at a time per host). The scheduler queues one about every 15 minutes per host. Each run writes a Jobs row and an Audit row. The default Jobs list and the default Audit list leave a successful snapshot out. A failed snapshot stays on both lists. Show routine host snapshots on Jobs, and Show routine rows on Audit, bring the successful rows back.

A second start reuses the existing job (UI follows it; REST 409 with already_active / existing job). Backups use a separate rule: per-host Redis mutex + Celery (see Multi-worker). Move a service also refuses a migrate while either host is busy.

Why exclusive: two apt upgrades at once on one host is a recipe for lock conflicts and opaque failure.

Fleet Jobs UI¶

• Ops hero at the top: dual-line pulse (running / queue / ok / fail + type chips) and app timezone caption
• Filters: server, status, type, date range with 7d / 30d / 90d / Clear presets, per-page 10 / 20 / 50 / 100 (cookie ph_per_page shared with Servers / Docker / discovery)
• Date presets use the Settings timezone calendar day (not the browser’s local midnight)
• Active only — pending + running
• Row → detail modal (summary, log tail, scheduled flag)
• Worker on each row and in the detail modal: the Celery nodename (celery@…, nmap@…) once a worker has claimed the job, not claimed while it is still queued, web for demo simulation. Two workers show two different names. JobHold’s status line includes the same name
• Cancel works from list and modal (where applicable)
• Link to Audit for historical trail

Live progress¶

JobHold / progress modals poll status and log lines for OS/container patch, nmap scans (including vuln DB update), and similar work. If a job was already active, the modal notes that and tracks the existing job_id. When a job finishes, the modal stays open with a done banner (no forced full-page reload) so you can read the log.

Stale data cleanup is configured under Settings → General — separate from per-server backup retention jobs.

Bulk fleet queue¶

From the Servers list, multi-select hosts and queue the same job type across eligible servers (feature flags apply). See Updates & patching — Bulk actions.

Audit¶

Where: nav Audit (/audit). Also from:

• Server detail footer: All logs, Backup logs, Docker logs, OS audit logs (filtered by host)
• Notifications → Audit log
• Jobs panel links

Actors may be:

• Session user (display name + email)
• API token name + id (automation)
• system / scheduler for cron jobs

Client IP (v0.5.0): every request-driven audit row stores the source IP.

Also audited with IP: login / login failed / 2FA, and API token create/update/rotate/revoke. A read call from an API token or from hosted /mcp writes an audit row only when Settings, Security policy, API and MCP read audit is on. The row stores the token id and name, and the client IP. It does not store the token. Those rows stay off the default Audit list until Show routine rows is on, and they keep their own retention (1-90 days, default 7). Turning the flag on or off writes its own audit row. Writes stay audited either way. A restore from a backup writes one audit row, action backup_restore (label Backup restore). It is not its own job type. Free-text search matches IPs. Detail modal shows IP.

Filter by user, server, token, action, status, date range (same 7d / 30d / 90d presets as Jobs — app timezone), or free-text (includes IP). Per-page is the same 10 / 20 / 50 / 100 cookie as Jobs.

The Audit page uses the same ops-hero pattern: status bars, top action-type chips (as filter links), and the active timezone in the subtitle. On dense filter layouts the pulse can be collapsed (Hide pulse) so the filter row stays primary; detail rows open a branded detail modal (summary + snippet).

Console command transcripts (v1.3)¶

When Settings → Console → Command audit is on, each finished web shell can attach a command timeline (Fernet-encrypted, not in Audit search). Open the ssh_console_close event: operator+ can expand the timeline, open /audit/console/{id}, or download .txt. Viewers see counts only (“transcript hidden”). Demo never stores bodies. Retention drops the ciphertext and keeps the fact that a transcript existed. See Web SSH console.

Backup lifecycle events¶

Each backup job writes append-only phases:

Completed backups show a summary line with source count and total size (e.g. 2 sources · 1.5 MB), duration, and a detail modal with per-source sizes. A host that sends files straight out adds the destination on that line, for example 2 sources · LAN NAS / SMB · 192.0.2.42/share/Backups. The detail keeps the same place on each source. The sign-in is not in the row. Queued and running phase rows are noise: Hide incomplete runs hides them, and they are left out of the Audit active pulse. Older rows that still say running are treated the same way.

Timezone display¶

All event times are stored in UTC and rendered in the app timezone from Settings → timezone (e.g. Africa/Johannesburg / SAST). The Audit header shows the active zone. Jobs and Notifications use the same rule.

Notifications¶

• Bell icon → open / dismiss
• Deep links into the relevant server or page
• Optional Web Push for new open notifications — PWA & Web Push
• When an alert auto-resolves (e.g. backup succeeds, updates clear), a push may fire as Resolved: … using the same type preference as the original alert
• Dismiss is idempotent if already closed

Why an inbox separate from Audit: Audit is forever; the inbox is a short “todo” list for open problems.

Job, nmap-run, and console-open history is aggregated on Reports (backups, OS patches, LAN live, Docker deploys, console sessions).

API¶

Automation can list and trigger jobs with Bearer tokens — API tokens. Hosted MCP uses that same list and the same 409 when a job is already active — Agents (MCP). Compose stack actions and Move stay in the browser.