Stale-backup alerts, per-server alert limits, and update runs from the Updates page

- Backups: a guest whose newest backup is older than its limit is an alert (backup kind),
  in the daily backup message, the digest and /metrics (homelab_backup_guests, ..._last_age_seconds)
- Per-server thresholds (servers.thresholds, migration 0030): offline minutes, disk percent,
  forecast window, backup age, and per-disk limits or "leave alone"; applied everywhere the
  global ones were; saving clears the alerts cache
- Update runs (update_runs, migration 0031): Update... on the Updates page puts the server under
  maintenance, runs a Semaphore playbook limited to it, waits for the host to report, and
  reports what is left and whether a restart is due; poller survives restarts
- Notification toggle for update-run results; tests and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 01:25:09 +02:00
1 parent 5dc6cea69c
commit 7a5b623444
42 files changed
+7561 -57

No files matched your search

+31 -3
View File
@@ -447,8 +447,20 @@ installed or refreshed — that one question is asked outside the agent's sandbo
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
said — an older agent, or another package manager — shows "not reported", never "up
to date"; run its install command again to update its agent (how, and how to
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply
updates, run a playbook.
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)).
**Update runs** (Updates → Update…, operators). Apply a server's waiting updates in
one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server
under maintenance (so nothing alerts while it works and restarts), runs the playbook
limited to that one server, follows it, waits for the server's agent to report again,
and then says how many updates are still waiting and whether a restart is due before
ending the window it made. The runs are listed on the Updates page and in the server's
history, survive a restart of this app, and end as *failed* (with the window ended, so
the alerts can speak) if the playbook fails, Semaphore stops answering, the server
doesn't report again within 30 minutes, or it takes more than three hours. "Stop
following" ends the window and stops watching; it can't stop a playbook already running
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
send*). It never restarts the server itself — the playbook does, if it is written to.
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
@@ -476,7 +488,23 @@ backup task that names the guest), and did that go well — Recent backup, Stale
than 1, 2, 3, 7 or 14 days, your choice), Failed, No backup, or Unknown (a job covers it
but nothing says when it last ran, typical when backups go somewhere other than a
Backup Server — never guessed at). Worst first, searchable, with CSV, and the weekly
digest lists what needs a look.
digest lists what needs a look. A guest whose newest backup is older than **Settings →
Notifications → "A guest's backup is too old after"** (48 hours by default; 0 is off) is
also an alert of the Backups kind — a warning, critical at three times the limit — with a
runbook link like any other, sent once a day with the other backup messages (under the
Proxmox backup toggle) and counted in `/metrics` (`homelab_backup_guests`,
`homelab_backup_last_age_seconds`). A server that is a guest can have its own limit
(a weekly backup is fine at eight days). Failed backups already raise their own alerts —
the latest failing run on a node, and snapshots that fail verification — so those aren't
repeated per guest. The page's own "Stale after" selector is only a view.
**Alert limits per server.** On a server's page an administrator can give it limits of
its own — offline after, disk usage percentage, the disk-forecast window, and (for a
Proxmox guest) how old its backup may be — and per disk, either a limit or "no alerts
for this disk". Whatever is left empty follows Settings, so changing the global limit
still reaches every server that hasn't said otherwise. They apply everywhere the global
ones did: the alerts, the notifications, the dashboard, the map, the forecast, the
digest and `/metrics`.
**Inventory** (Infrastructure → Inventory) keeps the hardware behind it all: name and
kind, maker, model, serial, where it sits (and rack unit), when it was bought and for