Acknowledge and hand over alerts, incident timelines, and built-in database backups
- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics - Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents, incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs, related alerts and notes; Incidents page and a digest section - Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download, check, upload, delete, and restore staged and applied at start-up before the database opens (pre-restore copy kept); alerts and /metrics for stale or failing backups - Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7a5b623444
commit
2744a84719
41 files changed
+6573
-26
No files matched your search
@@ -56,6 +56,11 @@ All gauges. Labels are escaped as Prometheus requires.
|
||||
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
|
||||
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
|
||||
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
|
||||
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. |
|
||||
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
|
||||
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
|
||||
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
|
||||
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
|
||||
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
|
||||
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
|
||||
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
|
||||
@@ -68,7 +73,7 @@ All gauges. Labels are escaped as Prometheus requires.
|
||||
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
|
||||
|
||||
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
|
||||
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`.
|
||||
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`.
|
||||
|
||||
## Event webhooks
|
||||
|
||||
|
||||
Reference in new issue
Block a user