Acknowledge and hand over alerts, incident timelines, and built-in database backups
- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics - Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents, incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs, related alerts and notes; Incidents page and a digest section - Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download, check, upload, delete, and restore staged and applied at start-up before the database opens (pre-restore copy kept); alerts and /metrics for stale or failing backups - Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7a5b623444
commit
2744a84719
41 files changed
+6573
-26
No files matched your search
@@ -449,6 +449,38 @@ said — an older agent, or another package manager — shows "not reported", ne
|
||||
to date"; run its install command again to update its agent (how, and how to
|
||||
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)).
|
||||
|
||||
**Acknowledging and handing over alerts.** On the Alerts page, **Acknowledge…** (operators) says
|
||||
"somebody is on this": an optional note, who has it (an administrator or operator), and how long it should
|
||||
be left alone (an hour to a week, or until the alert is gone). It changes nothing about the problem — the
|
||||
alert is still raised, still counted by `/metrics` and still on the wall display — but the alert shows who
|
||||
has it and why, the sidebar count, the dashboard strip and the Alerts cards count only what nobody is on,
|
||||
and the list can be filtered to "nobody is on it" or "handed to me". An acknowledgement that runs out puts
|
||||
the alert back among the unattended; it goes by itself once the alert has been gone for half an hour, so
|
||||
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
|
||||
server's history.
|
||||
|
||||
**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an
|
||||
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
|
||||
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
|
||||
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
|
||||
began around the same time, and the notes people wrote (operators can add them; the author or an
|
||||
administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look
|
||||
every five minutes, so one that comes and goes between two looks may be missed. An alert is only called
|
||||
resolved after it has been missing for a couple of minutes in a row — thirty when a check couldn't run, and
|
||||
never in the first minutes after the app starts, when server alerts are held back on purpose. Resolved
|
||||
incidents are kept for 400 days. The weekly digest has an *Incidents* section.
|
||||
|
||||
**Database backups** (Settings → Backup). A copy of Homelab Manager's own database is taken every day (03:30
|
||||
in the notification time zone, the newest 14 kept; on unless you turn it off), checked as it is made, and
|
||||
kept in `data/backups/`. The page lists them with **Download**, **Check**, **Restore…** and **Delete**,
|
||||
makes one now (**Back up now**), takes one from somewhere else (**Upload a backup…**) and sets the schedule.
|
||||
A restore is checked first, then staged, and takes effect the next time the app starts — restart it
|
||||
(`docker compose restart homelab-manager`); the database that was in use is kept as a "before a restore"
|
||||
backup, so a restore can be undone. A backup made by a *newer* version than the one running is refused.
|
||||
If the newest backup gets more than two days old, or a scheduled one fails, it is an alert (and a message).
|
||||
**A backup does not contain `CREDENTIALS_ENCRYPTION_KEY`**, which decrypts the credentials stored in it,
|
||||
and it is still on the same machine until you copy it away: see [BACKUP.md](BACKUP.md).
|
||||
|
||||
**Update runs** (Updates → Update…, operators). Apply a server's waiting updates in
|
||||
one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server
|
||||
under maintenance (so nothing alerts while it works and restarts), runs the playbook
|
||||
|
||||
Reference in new issue
Block a user