Acknowledge and hand over alerts, incident timelines, and built-in database backups

- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what
  needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics
- Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents,
  incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs,
  related alerts and notes; Incidents page and a digest section
- Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download,
  check, upload, delete, and restore staged and applied at start-up before the database opens
  (pre-restore copy kept); alerts and /metrics for stale or failing backups
- Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 02:21:08 +02:00
1 parent 7a5b623444
commit 2744a84719
41 files changed
+6573 -26

No files matched your search

+32
View File
@@ -449,6 +449,38 @@ said — an older agent, or another package manager — shows "not reported", ne
to date"; run its install command again to update its agent (how, and how to
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)).
**Acknowledging and handing over alerts.** On the Alerts page, **Acknowledge…** (operators) says
"somebody is on this": an optional note, who has it (an administrator or operator), and how long it should
be left alone (an hour to a week, or until the alert is gone). It changes nothing about the problem — the
alert is still raised, still counted by `/metrics` and still on the wall display — but the alert shows who
has it and why, the sidebar count, the dashboard strip and the Alerts cards count only what nobody is on,
and the list can be filtered to "nobody is on it" or "handed to me". An acknowledgement that runs out puts
the alert back among the unattended; it goes by itself once the alert has been gone for half an hour, so
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
server's history.
**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
began around the same time, and the notes people wrote (operators can add them; the author or an
administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look
every five minutes, so one that comes and goes between two looks may be missed. An alert is only called
resolved after it has been missing for a couple of minutes in a row — thirty when a check couldn't run, and
never in the first minutes after the app starts, when server alerts are held back on purpose. Resolved
incidents are kept for 400 days. The weekly digest has an *Incidents* section.
**Database backups** (Settings → Backup). A copy of Homelab Manager's own database is taken every day (03:30
in the notification time zone, the newest 14 kept; on unless you turn it off), checked as it is made, and
kept in `data/backups/`. The page lists them with **Download**, **Check**, **Restore…** and **Delete**,
makes one now (**Back up now**), takes one from somewhere else (**Upload a backup…**) and sets the schedule.
A restore is checked first, then staged, and takes effect the next time the app starts — restart it
(`docker compose restart homelab-manager`); the database that was in use is kept as a "before a restore"
backup, so a restore can be undone. A backup made by a *newer* version than the one running is refused.
If the newest backup gets more than two days old, or a scheduled one fails, it is an alert (and a message).
**A backup does not contain `CREDENTIALS_ENCRYPTION_KEY`**, which decrypts the credentials stored in it,
and it is still on the same machine until you copy it away: see [BACKUP.md](BACKUP.md).
**Update runs** (Updates → Update…, operators). Apply a server's waiting updates in
one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server
under maintenance (so nothing alerts while it works and restarts), runs the playbook