Acknowledge and hand over alerts, incident timelines, and built-in database backups

- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what
  needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics
- Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents,
  incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs,
  related alerts and notes; Incidents page and a digest section
- Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download,
  check, upload, delete, and restore staged and applied at start-up before the database opens
  (pre-restore copy kept); alerts and /metrics for stale or failing backups
- Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 02:21:08 +02:00
1 parent 7a5b623444
commit 2744a84719
41 files changed
+6573 -26

No files matched your search

+6 -1
View File
@@ -56,6 +56,11 @@ All gauges. Labels are escaped as Prometheus requires.
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. |
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
@@ -68,7 +73,7 @@ All gauges. Labels are escaped as Prometheus requires.
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`.
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`.
## Event webhooks