Troubleshooting¶
What this is¶
A symptom → page map when something fails. Start here, then open Jobs, Audit, and Settings → Status before deep host dives.
Why this section¶
Most failures cluster around SSH path, Celery/backups, push TLS, or template/Docker paths — the linked pages list concrete checks.
| Symptom | Page |
|---|---|
| SSH / key / deps / docker group | SSH, rsync & dependencies |
| SSH “host key changed” / mismatch after rebuild | SSH troubleshooting — reset the pinned key on SSH access, then Test connection |
Web container exits immediately / weak SECRET_KEY | Set a long random SECRET_KEY. 1.2 refuses boot on compose defaults unless PIHERDER_ALLOW_INSECURE (env) |
HAOS: no ha, versions ?, disk empty | HAOS hosts · System info Refresh; SSH add-on + ha on PATH; rsync for backups |
| HAOS OS check says unsupported / apt | Set host profile HAOS or re-run check to auto-mark; rebuild web if image stale |
| Backup failed / stuck pending | Backups |
| No push on phone / PWA | Push / PWA |
| Template deploy / Docker editor | Templates & Docker |
| From-host missing config sidecar / host labels | From host · Templates troubleshooting |
| Reboot hangs / UI stuck after reboot | Updates — Reboot |
| Same patch job appears twice | Jobs — Exclusive jobs · Multi-worker |
| Bulk Upgrade OS stuck pending while one host stays running | One request used to run patches sequentially. Recreate web on v1.4.0 — orphan os_patch rows fail on startup so the exclusive lock clears; bulk now uses the patch pool (hosts in parallel). Updates |
| Full editor link does nothing | Compose edit — use ⋯ Full editor… or deployment Open host file editor |
| Drift after intentional host edit (keep change) | Deploy — Accept host as desired |
| Fleet Services empty | Dashboard & Services — bind Kuma monitors |
| Reports empty / history shorter than expected | Reports — needs finished Jobs / nmap runs / console Audit; Cleanup can trim rows |
| Files button missing / 404 | Flag PIHERDER_HOST_FILES (default off). Viewer 403. Host Files |
| Move to another host missing / 404 | Flag PIHERDER_SERVICE_MIGRATE (default off). Recreate web. Viewer 403. Locked / HAOS refused. Demo never copies. Move a service |
| Move dest picker looks stuck | Preflight SSHs both hosts — wait modal + “Checking destination…”. Recreate web if the local image is stale |
| Move job fails “active service_migrate job #N” on itself | Fixed on this freeze branch — job preflight no longer treats its own row as busy. Failed job is idle; start Move again after rebuild |
Move rsync change_dir "/home/…\#342\#200\#246" | Truncated docker ps mount (Unicode ellipsis). Rebuild this freeze branch; inspect fills full paths. Source may be stopped — Start all if you need it up before retry |
Move job up -d failed with no compose text | Older build; current train logs docker compose pull/up output on the job. Source is stopped; dest may have files. SSH dest cd … && docker compose up -d if you need the stack now |
| Move preflight blocks dest | Same page — remap dest host port or project name/folder, disk, DNS name, NPM poll cache, busy backup/stack/migrate job |
Move dest up address already in use | Recheck now uses live dest ss listen + docker. Remap dest host port. Leftover dest folder from a failed Move is overwrite, not a block. If the source stack uses host network, remap does not apply — dest must have that port free |
Move rsync change_dir "/var/run/docker.sock" / Not a directory | Host socket (Uptime Kuma Docker monitor). Rebuild this freeze branch — Move no longer rsyncs docker.sock; dest binds dest’s own socket |
Move dest project owned root:root / fleet SSH user | Rebuild this freeze branch — Move chowns dest project to dest docker root owner (bjorn for /home/bjorn/docker), not piherder / root |
| Recheck destination looks like a no-op | Wait modal should show for dest pick, port change, and Recheck. Rebuild web |
| Move job overlay vanishes with no success/fail | Overlay now stays with Succeeded/Failed until Close. Rebuild web |
| Move job TLS / Kuma red | No auto-rollback. Dest may already be up with names flipped. Fix dest. Staging kept under /backups/_migrate/{job_id} |
| Move succeeded but NPM still points at the old Pi | Job skipped PUT when there was no fabric DNS row. Current train PUTs from the proxy-host binding. Rebuild web, poll NPM, or Move again. Optional Adopt into fabric puts the name on the DNS list without rewriting Pi-hole |
| Move copy / dest-up failed; source is stopped | JobHold Start source stack (same as Docker ⋯ Start all). Dest names were not flipped |
| Grafana chip missing on dest after Move | Only Containers (project/container) binds follow dest. Host metrics/logs stay. Rebuild web if dest Docker still shows the old host |
| Map stack mode has no container after Move | Stack reads dest Docker inventory. Job #1100 failed at rebind (duplicate Grafana bind) after dest was already up — dest snapshot stayed empty. Refresh Docker on dest (or rebuild web: dest inventory refreshes after up). Duplicate dest Grafana chips are dropped, not copied |
| Move leftover wipe did too little / too much | Default leaves source stopped. Remove source only deletes the jailed project dir + copied named volumes — dest never, extra binds outside the project folder stay |
| Files upload fails / too large | Default 512 MiB; Settings → Files (ceiling 32 GiB) or lock with PIHERDER_HOST_FILES_MAX_BYTES; raise any extra reverse-proxy body cap / timeout |
| Files download stuck ~12 MiB | Dedicated SFTP + Caddy must not gzip application/octet-stream (flush_interval -1). Rebuild web + Caddy. Host Files |
.env / PEM won’t open | Listing is allowed; open/download needs Passkey/TOTP (same grant as privileged Files). Host Files |
| chmod/chown permission denied | ⋯ → Privileged. If that user is not root, add NOPASSWD for chmod/chown. Host Files |
| Files editor save fails / Permission denied | Root-owned files: privileged identity retries with sudo -n tee. The editor bar shows the reason. Add NOPASSWD tee or use a root privileged identity. Host Files |
| Files list tiny on a phone | Toolbar expand (or ⋯ → Maximize) hides the hero so the list fills the screen — same idea as console Maximize / Hosts map full screen. Restore brings the hero back. Host Files |
| Image preview ‹ › feels stuck | Peek is fast; the picture is a second SFTP pull. A loading overlay stays until the image arrives. Host Files |
| Docker mounts greyed out | Host path is outside the current jail — ⋯ → Privileged (named volumes live under /var/lib/docker/volumes). Bind mounts under the project folder work on fleet. |
| Zip failed on the host | Zip runs on the Pi (zip or python3). Install one of those, or check disk space. The tree is not pulled through PiHerder. |
| Network map hosts not linked / cloud wrong | Network maps — set LAN/gateway/public IP; hard-refresh after rebuild |
| Hosts map focus won’t clear on second click | Hard-refresh for latest fabric-mesh.js; click same node again or Clear focus |
| Stack deps float away when Discovered is off | Hard-refresh fabric-stack-expand.js; fan re-anchors to compact layout |
| Server page slow / clicks lag 5–10s | Fixed: no live Pi-hole probe on every server detail paint; Backups no longer recursive du on load — rebuild/restart web |
| Device edit closes to Integrations instead of server | Open from the LAN chip on that host (includes return=server:…); Cancel/Save returns to the server |
| Layout stuck after phone rotate (esp. Network) | Hard-reload once after deploy; maps should reflow without leaving the page (Appearance) |
| Dashboard NPM hosts ≠ NPM proxy count | Dashboard uses poll proxy_host_count; poll NPM integration if stale |
| Cert deploy / renew failed | Certificates · Jobs + Audit |
sudo: I'm sorry piherder… / post-deploy denied | Cert sudo denied — add NOPASSWD; match post-deploy exactly |
| Drift after host edit (detect / revert) | Deploy — Check drift / Apply last known |
| Stack unhealthy after upgrade | Status · Upgrades |
| After host reboot, UI down until I start a container | web used to have no restart policy (others already unless-stopped). Confirm docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' piherder-web is unless-stopped, then docker compose up -d. Also systemctl is-enabled docker. Install |
| Account page messy / huge cards | Compact hub + Edit sheets (same as Settings → Security). Rebuild/restart web. |
| Password strength still says min 10 after policy change | Meter now reads live Settings. Hard-refresh. Admin create / reset / Account / forgot-password share one JS. Users |
| Unlink SSO did nothing / no confirm | Account → Connected accounts → Edit → Unlink… opens a confirmation sheet. Hard-refresh after rebuild. SSO |
| Unlink dumped onto a useless 2FA screen (“Authenticator or backup code required.”) | Fixed: step-up stays on the unlink (or password) sheet with Confirm passkey / 6-digit. Rebuild/restart web + hard-refresh. |
| Account Link Authentik does nothing (login SSO still works) | CSP form-action 'self' blocked a form POST 303 to the IdP. Link now hops /auth/oidc/link (GET) after step-up. Rebuild/restart web. Confirm passkey or enter a code, then Link. |
Demo demo-maintain.sh stuck on Installing build dependencies | Docker bridge has no PyPI. Ctrl+C. Pull latest v1.3.0-dev, then ./scripts/demo-maintain.sh redeploy (host-network docker build -t piherder:demo). Do not compose build / Hub pull. DEMO_SITE |
| Cannot open Settings tabs / herder restore | Roles — control plane is admin only |
| First boot asks to register / no default password | Expected — First login |
| Sole admin forgot password / lost 2FA / locked out | Locked out / sole admin recovery — host CLI recover-admin |
| SSO / OIDC login fails or IdP down | SSO / OpenID Connect · Locked out for break-glass |
| Console “too many shells” / idle disconnect | Web SSH console · Settings → Console |
| Console Settings fields read-only | A PIHERDER_SSH_CONSOLE_* env var is set (lock). Unset it or env reference |
Always useful¶
- Jobs page for work unit status + log tail
- Audit for who/what / client IP
- Settings → Status for stack health (Settings)
- Server SSH access → Check dependencies (or Test connection) for remote tools
- Scenario index: Operator scenarios