Stale-backup alerts, per-server alert limits, and update runs from the Updates page
- Backups: a guest whose newest backup is older than its limit is an alert (backup kind), in the daily backup message, the digest and /metrics (homelab_backup_guests, ..._last_age_seconds) - Per-server thresholds (servers.thresholds, migration 0030): offline minutes, disk percent, forecast window, backup age, and per-disk limits or "leave alone"; applied everywhere the global ones were; saving clears the alerts cache - Update runs (update_runs, migration 0031): Update... on the Updates page puts the server under maintenance, runs a Semaphore playbook limited to it, waits for the host to report, and reports what is left and whether a restart is due; poller survives restarts - Notification toggle for update-run results; tests and docs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5dc6cea69c
commit
7a5b623444
42 files changed
+7561
-57
No files matched your search
@@ -447,8 +447,20 @@ installed or refreshed — that one question is asked outside the agent's sandbo
|
||||
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
|
||||
said — an older agent, or another package manager — shows "not reported", never "up
|
||||
to date"; run its install command again to update its agent (how, and how to
|
||||
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply
|
||||
updates, run a playbook.
|
||||
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)).
|
||||
|
||||
**Update runs** (Updates → Update…, operators). Apply a server's waiting updates in
|
||||
one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server
|
||||
under maintenance (so nothing alerts while it works and restarts), runs the playbook
|
||||
limited to that one server, follows it, waits for the server's agent to report again,
|
||||
and then says how many updates are still waiting and whether a restart is due before
|
||||
ending the window it made. The runs are listed on the Updates page and in the server's
|
||||
history, survive a restart of this app, and end as *failed* (with the window ended, so
|
||||
the alerts can speak) if the playbook fails, Semaphore stops answering, the server
|
||||
doesn't report again within 30 minutes, or it takes more than three hours. "Stop
|
||||
following" ends the window and stops watching; it can't stop a playbook already running
|
||||
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
|
||||
send*). It never restarts the server itself — the playbook does, if it is written to.
|
||||
|
||||
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
|
||||
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
|
||||
@@ -476,7 +488,23 @@ backup task that names the guest), and did that go well — Recent backup, Stale
|
||||
than 1, 2, 3, 7 or 14 days, your choice), Failed, No backup, or Unknown (a job covers it
|
||||
but nothing says when it last ran, typical when backups go somewhere other than a
|
||||
Backup Server — never guessed at). Worst first, searchable, with CSV, and the weekly
|
||||
digest lists what needs a look.
|
||||
digest lists what needs a look. A guest whose newest backup is older than **Settings →
|
||||
Notifications → "A guest's backup is too old after"** (48 hours by default; 0 is off) is
|
||||
also an alert of the Backups kind — a warning, critical at three times the limit — with a
|
||||
runbook link like any other, sent once a day with the other backup messages (under the
|
||||
Proxmox backup toggle) and counted in `/metrics` (`homelab_backup_guests`,
|
||||
`homelab_backup_last_age_seconds`). A server that is a guest can have its own limit
|
||||
(a weekly backup is fine at eight days). Failed backups already raise their own alerts —
|
||||
the latest failing run on a node, and snapshots that fail verification — so those aren't
|
||||
repeated per guest. The page's own "Stale after" selector is only a view.
|
||||
|
||||
**Alert limits per server.** On a server's page an administrator can give it limits of
|
||||
its own — offline after, disk usage percentage, the disk-forecast window, and (for a
|
||||
Proxmox guest) how old its backup may be — and per disk, either a limit or "no alerts
|
||||
for this disk". Whatever is left empty follows Settings, so changing the global limit
|
||||
still reaches every server that hasn't said otherwise. They apply everywhere the global
|
||||
ones did: the alerts, the notifications, the dashboard, the map, the forecast, the
|
||||
digest and `/metrics`.
|
||||
|
||||
**Inventory** (Infrastructure → Inventory) keeps the hardware behind it all: name and
|
||||
kind, maker, model, serial, where it sits (and rack unit), when it was bought and for
|
||||
|
||||
Reference in new issue
Block a user