Acknowledge and hand over alerts, incident timelines, and built-in database backups

- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what
  needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics
- Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents,
  incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs,
  related alerts and notes; Incidents page and a digest section
- Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download,
  check, upload, delete, and restore staged and applied at start-up before the database opens
  (pre-restore copy kept); alerts and /metrics for stale or failing backups
- Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 02:21:08 +02:00
1 parent 7a5b623444
commit 2744a84719
41 files changed
+6573 -26

No files matched your search

+40 -1
View File
@@ -4,6 +4,7 @@ Everything Homelab Manager knows is in **one SQLite file**, plus a handful of se
`.env`. A backup is those two things — and the part people get wrong is keeping them **apart**
(see [Keeping backups safe](#keeping-backups-safe)).
- [The built-in backups (Settings → Backup)](#the-built-in-backups-settings--backup)
- [What to back up](#what-to-back-up)
- [Taking a backup](#taking-a-backup)
- [Checking a backup](#checking-a-backup)
@@ -18,6 +19,34 @@ Everything Homelab Manager knows is in **one SQLite file**, plus a handful of se
The commands assume the standard Docker setup from the README: you run them in the folder that
holds `docker-compose.yml`, `.env` and `data/`, and the service is called `homelab-manager`.
## The built-in backups (Settings → Backup)
You don't need the snippets below for the everyday case: Homelab Manager takes a backup of its own
database **every day** (03:30 in the notification time zone, the newest 14 kept — both changeable, and it
can be switched off), checks each one as it makes it, and keeps them in `data/backups/` with the same
names the snippets use. **Settings → Backup** shows them and does the rest:
- **Back up now** makes one on the spot. **Download** takes a copy to your computer (the quickest way
off the machine). **Check** opens a backup and says whether SQLite finds it sound, how many users,
servers and integrations it holds, and whether it is from an older or newer version.
- **Upload a backup…** takes a file from somewhere else (another machine, the script below) and keeps it
if it checks out. A file that isn't a Homelab Manager database is refused and nothing is kept.
- **Restore…** checks the backup and *stages* it. Nothing changes until the app next starts — restart it
(`docker compose restart homelab-manager`) and, before anything opens the database, the staged file
takes the database's place. The database that was in use is **kept** in `data/backups/` as
`…-pre-restore.sqlite`, so a restore that wasn't what you wanted is undone by restoring that file. If
something stops it half way, the old database is put back. A backup from a **newer** version than the
one running is refused; an older one is fine (the start-up brings it up to date). **Cancel the restore**
until you restart if you change your mind.
- If the newest backup is more than two days old, or a scheduled one fails, there is an **alert** (and a
message through your channels), and `/metrics` has `homelab_selfbackup_last_success_timestamp_seconds`
for your own monitoring.
Three things it does **not** do, and the rest of this page is about: it does not hold
`CREDENTIALS_ENCRYPTION_KEY` (keep that separately — see [Keeping backups safe](#keeping-backups-safe)),
it does not copy anything off the machine on its own (Download, or copy `data/backups/` with your usual
tool), and it does not replace a restore drill. Access to all of it is for administrators only.
## What to back up
| What | Where | Why | If you lose it |
@@ -275,7 +304,17 @@ What you can get back, from least to most work:
## What has been tested, and what hasn't
Tested on a scratch database made with the real migrations:
**The built-in backups** were tested with the app's own tests (a backup holds what was there and not what came
after; two in the same second don't collide; garbage, someone else's database and a newer version's
database are refused; the newest N are kept; a staged restore is checked, can be cancelled and puts nothing
half-copied in place; a restore that can't complete leaves the database exactly as it was) and by a real
restore: a backup was taken, a user added, the restore staged, the actual built app (`node dist/index.js`)
restarted on the same data folder — the user added after the backup was gone, the previous database was
kept as `…-pre-restore.sqlite`, and the next daily backup was scheduled. **Not** tested: running it inside the
real Docker container, and a very large database (the copy is made on this machine's disk and takes about as
long as copying the file).
The snippet-based way below was tested on a scratch database made with the real migrations:
- The snapshot (`VACUUM INTO`) taken from a second process **while the app was writing**: it
completed in tens of milliseconds, passed the integrity check, held all migrations, and an