Jobs, audit & notifications¶

What this is¶

Three related systems — do not confuse them:

Open notifications via the bell icon (no separate Alerts nav link).

Why they exist¶

Long SSH work must not block the browser (jobs). Homelab and multi-operator setups need accountability (audit). Noise should not bury failures (notifications + optional Web Push). Together they replace “did that cron fire?” shell archaeology.

End-to-end: follow one backup through all three¶

1. Start a backup from a server Backups page.
2. Jobs → row goes pending → running → success / failed; open detail for log tail.
3. Audit → filter that server / backup actions; see request → queued → running → complete phases and client IP.
4. If it failed earlier, a notification may open; on success, related open alerts can auto-resolve (optional push: Resolved: …).
5. Dismiss the inbox item when you have acted.

Jobs¶

Where: nav Jobs (/jobs) · compact panel on each server detail.

Job types (examples)¶

Statuses: pending → running → success / failed.

Web restart: OS/container patch and other web-process jobs (not Celery backups / nmap) that are still pending or running are failed on startup. They cannot still be executing after uvicorn exits; leaving them blocked new exclusive work on that host. Bulk Upgrade OS runs on a shared patch thread pool (several hosts at once), not one sequential BackgroundTasks chain.

Exclusive jobs (one per type per host)¶

These types do not stack on the same server while already pending or running:

• os_patch, container_patch
• os_update_check, container_update_check
• Stack lifecycle + template deploy/redeploy (shared stack mutation lane on the host)
• service_migrate — exclusive with backup and stack mutation on both source and dest
• template_drift_check (one drift job at a time per host; not a stack write)

A second start reuses the existing job (UI follows it; REST 409 with already_active / existing job). Backups use a separate rule: per-host Redis mutex + Celery (see Multi-worker). Move a service also refuses a migrate while either host is busy.

Why exclusive: two apt upgrades at once on one host is a recipe for lock conflicts and opaque failure.

Fleet Jobs UI¶

• Ops hero at the top: dual-line pulse (running / queue / ok / fail + type chips) and app timezone caption
• Filters: server, status, type, date range with 7d / 30d / 90d / Clear presets, per-page 10 / 20 / 50 / 100 (cookie ph_per_page shared with Servers / Docker / discovery)
• Date presets use the Settings timezone calendar day (not the browser’s local midnight)
• Active only — pending + running
• Row → detail modal (summary, log tail, scheduled flag)
• Cancel works from list and modal (where applicable)
• Link to Audit for historical trail

Live progress¶

JobHold / progress modals poll status and log lines for OS/container patch, nmap scans (including vuln DB update), and similar work. If a job was already active, the modal notes that and tracks the existing job_id. When a job finishes, the modal stays open with a done banner (no forced full-page reload) so you can read the log.

Stale data cleanup is configured under Settings → General — separate from per-server backup retention jobs.

Bulk fleet queue¶

From the Servers list, multi-select hosts and queue the same job type across eligible servers (feature flags apply). See Updates & patching — Bulk actions.

Audit¶

Where: nav Audit (/audit). Also from:

• Server detail footer: All logs, Backup logs, Docker logs, OS audit logs (filtered by host)
• Notifications → Audit log
• Jobs panel links

Actors may be:

• Session user (display name + email)
• API token name + id (automation)
• system / scheduler for cron jobs

Client IP (v0.5.0): every request-driven audit row stores the source IP.

Also audited with IP: login / login failed / 2FA, and API token create/update/rotate/revoke. Free-text search matches IPs. Detail modal shows IP.

Filter by user, server, token, action, status, date range (same 7d / 30d / 90d presets as Jobs — app timezone), or free-text (includes IP). Per-page is the same 10 / 20 / 50 / 100 cookie as Jobs.

The Audit page uses the same ops-hero pattern: status bars, top action-type chips (as filter links), and the active timezone in the subtitle. On dense filter layouts the pulse can be collapsed (Hide pulse) so the filter row stays primary; detail rows open a branded detail modal (summary + snippet).

Console command transcripts (v1.3)¶

When Settings → Console → Command audit is on, each finished web shell can attach a command timeline (Fernet-encrypted, not in Audit search). Open the ssh_console_close event: operator+ can expand the timeline, open /audit/console/{id}, or download .txt. Viewers see counts only (“transcript hidden”). Demo never stores bodies. Retention drops the ciphertext and keeps the fact that a transcript existed. See Web SSH console.

Backup lifecycle events¶

Each backup job writes append-only phases:

Completed backups show a summary line with source count and total size (e.g. 2 sources · 1.5 MB), duration, and a detail modal with per-source sizes. Incomplete/running noise can be hidden with Hide incomplete runs.

Timezone display¶

All event times are stored in UTC and rendered in the app timezone from Settings → timezone (e.g. Africa/Johannesburg / SAST). The Audit header shows the active zone. Jobs and Notifications use the same rule.

Notifications¶

• Bell icon → open / dismiss
• Deep links into the relevant server or page
• Optional Web Push for new open notifications — PWA & Web Push
• When an alert auto-resolves (e.g. backup succeeds, updates clear), a push may fire as Resolved: … using the same type preference as the original alert
• Dismiss is idempotent if already closed

Why an inbox separate from Audit: Audit is forever; the inbox is a short “todo” list for open problems.

Job, nmap-run, and console-open history is aggregated on Reports (backups, OS patches, LAN live, Docker deploys, console sessions).

API¶

Automation can list/trigger jobs with Bearer tokens — API tokens.