Files
Homelab-manager/NOTIFICATIONS.md
T
bobbanandClaude Sonnet 5.5 7a5b623444 Stale-backup alerts, per-server alert limits, and update runs from the Updates page
- Backups: a guest whose newest backup is older than its limit is an alert (backup kind),
  in the daily backup message, the digest and /metrics (homelab_backup_guests, ..._last_age_seconds)
- Per-server thresholds (servers.thresholds, migration 0030): offline minutes, disk percent,
  forecast window, backup age, and per-disk limits or "leave alone"; applied everywhere the
  global ones were; saving clears the alerts cache
- Update runs (update_runs, migration 0031): Update... on the Updates page puts the server under
  maintenance, runs a Semaphore playbook limited to it, waits for the host to report, and
  reports what is left and whether a restart is due; poller survives restarts
- Notification toggle for update-run results; tests and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 01:25:09 +02:00

175 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Notifications
Every notification in Homelab Manager is sent through the same pipe
(`server/src/services/notify.ts`) to whichever channels are enabled under
**Settings → Notifications**: Gotify, ntfy, SMTP (email), and a generic
JSON webhook. All four fire for every notification below — there's no
per-event channel routing, only a per-event on/off toggle and, for the
daily ones, one shared time/timezone.
The settings page has three tabs, with one Save button for all of them (it
shows "Unsaved changes" until you save, and the browser asks before the page
is closed with changes unsaved): **Channels** (the four channels, each a row
you open to fill in and send a test), **What to send** (the event toggles
grouped by topic, each group's thresholds beside its switches) and **When**
(the time zone, the daily reminder time, quiet hours and the weekly digest).
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
Two settings apply to *every* notification regardless of what triggered
it:
- **Quiet hours** (Settings → Notifications → Quiet hours): while
enabled and inside the configured window, a notification is held
instead of sent immediately, then delivered as a single digest at the
window's end time. This matters most for the real-time alerts below —
the daily reminders already fire once at a time you pick, usually
outside the window anyway.
- **Maintenance windows** (the Maintenance page): opening one for a
server, integration, or DNS provider silences the alerts that target
it specifically (offline/disk-full for a server; storage, backup, and
automation-failure alerts for an integration; "integration down" for
that whole service type) — see the Maintenance page's own "What gets
silenced" panel for the exact list.
Where a trigger has a configurable threshold, that's noted in its row
below; several are fixed and can't be changed from the UI.
## Daily reminders
These six checks share one schedule: **Settings → Notifications → Daily
reminder time** (default 08:00) and **Timezone** (default UTC). Unlike
the state-based alerts further down, these **re-send every day the
condition is still true** — there's no "only once" de-duplication, so an
expired secret you haven't renewed yet will be mentioned again at the
next day's check, and the day after that.
Each one also runs once at server startup if it hasn't already run
today (e.g. after an upgrade or a period offline), so you're not waiting
until the next scheduled time to catch up.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
| Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send |
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
## State-based alerts
These two run on a fixed **15-minute interval** (matching the agent's
own default report interval) — not the daily reminder time above, and
not user-configurable. Unlike the daily reminders, these are
**state-based**: a problem is announced once when it first appears, and
once more when it clears — not repeated on every 15-minute pass while it
continues. A condition that can't currently be read (an integration
that's down, a server whose agent hasn't reported) is held exactly as it
was rather than cleared or re-announced, so a temporary read failure
can't fake a recovery. On first-ever run (or right after upgrading to
this feature), whatever's already failing is recorded silently as the
starting baseline rather than announced all at once.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) |
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
## Real-time alerts
These fire immediately, as the triggering action happens — not on any
schedule.
| Event | "Notify on" toggle | Fires when |
|---|---|---|
| DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
| DNS record updated | *DNS record updated* | An existing DNS record is edited. |
| DNS record deleted | *DNS record deleted* | A DNS record is deleted. |
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window. | Settings → Notifications → **Alert after** (default 3 consecutive failures) |
| Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
## Digest
| Notification | Fires when |
|---|---|
| "Notifications from quiet hours" | Once, at quiet hours' configured end time, **only if enabled and only if at least one notification was held** during the window. Bundles every held notification's title and message into one message, then clears the queue. |
## Weekly digest
| Notification | Fires when |
|---|---|
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
**what happened** (servers that stopped reporting and came back, services that stopped answering, maintenance);
**disks heading for full**; secrets and domains that are **expiring** or have expired (and hardware whose warranty ends within 60 days); guests that are **not backed up**, have a failed or stale backup; **updates waiting** (how many
servers, how many security fixes, who needs a restart, the longest wait); and **planned maintenance** in the next seven
days. A week with nothing in it says "A quiet week: nothing needs attention." On the settings page, *Preview* shows
what it would say right now and *Send it now* sends it at once (through the channels that are on, whatever the time).
## The Alerts page
**Operations → Alerts** shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and
export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs
in three ways:
- It ignores the "Notify on" toggles. Turning a notification off doesn't hide the problem from the page.
- Problems under a **maintenance window** are kept on the list, marked *silenced* and counted separately, instead of being dropped.
- It also lists things nothing notifies about: Uptime Kuma monitors that are down, osTicket tickets that are overdue, and any integration that's
failing its last few calls (before the threshold that triggers an "integration down" notification).
It runs the checks live when opened (a recent result is reused for a minute), and shows what it couldn't read at the top, so a missing section means
"couldn't check" and not "all clear". An alert that has an osTicket ticket (see the next section) shows its ticket number.
## Tickets in osTicket
Not a notification, but it acts on the same alerts: with **Settings → Alert tickets** switched on (off by default), a problem on the Alerts page opens a
ticket in osTicket. It runs every 15 minutes, starting about a minute and a half after the app starts.
| Rule | What it means |
|---|---|
| Which problems | The severities and kinds you choose — **critical and warning, every kind** by default. Info (updates waiting and the like) is off. |
| Must last | A problem must be there in **2 checks in a row** (15–30 minutes) before it opens a ticket, so a blip that fixes itself — one failed call to an integration, a monitor that flickers — never does. Can be set from 1 (at once) to 4. |
| One ticket per problem | However many checks see it, once. After a ticket is open, the problem isn't ticketed again while it lasts — even if someone closes the ticket in osTicket. |
| Comes back | A problem that clears and returns **within 60 minutes** is the same problem (no new ticket). After longer, it is a new one and gets a new ticket. |
| Many at once | **More than 5 new problems in one check** become a single summary ticket listing them all, so one outage doesn't fill the queue. Up to 5 get a ticket each. |
| Maintenance windows | A problem under one never opens a ticket. If it is still there after the window ends, it does. |
| Never about osTicket | Overdue tickets, and osTicket's own calls failing, never open a ticket — it would go to the thing that is down, or add to the pile. |
| Already there at switch-on | **Left alone by default**: the first check after switching on records what is wrong as the starting line, and only problems that start after it get a ticket. Can be set to ticket those too. Switching it off and on again starts a new line. |
| If osTicket can't be reached | Nothing is recorded as opened; the next check tries again. A refused key, a wrong address and similar are shown on the settings page and in the Diagnostic Log. |
| If something can't be read | When a check couldn't read everything (an integration was down), nothing is treated as gone, so a problem isn't called cleared just because its source couldn't be looked at. |
The ticket says what is wrong, the severity, the kind, where it comes from, when it was first seen, and links back to the problem in this app. osTicket's
API can't update or close a ticket, so it isn't touched again: close it in osTicket when the problem is dealt with. Every ticket opened (and every test
ticket) is in the Audit Log as `alert_ticket`, and changes to the settings as `settings` / `update_alert_tickets`. "Send a test ticket" and "Check now"
on the settings page use the saved settings.
## What does *not* send a notification
Worth calling out explicitly, since it's easy to assume everything in
the app alerts on something:
- **osTicket's own tickets** — the integration lists open tickets, but
there is no notification for it (e.g. nothing pings you about a
newly-overdue ticket); the page and the Alerts page show them. (The
other direction — an alert opening a ticket — is the section above.)
- **phpIPAM sync**, **Uptime Kuma monitor/maintenance import**, and any
other manual "Sync from X" action — these run when you click the
button and report their result on the page itself, not via a
notification.
- **Test buttons** (Settings → Notifications → each channel's "Send
test") send a one-off message through that one channel only, to
confirm it's wired up correctly — not a real event and not affected by
quiet hours.
- **Consistency reports**, **Diagnostic Log**, and **Audit Log** are all
read-only views you check yourself; none of them push a notification
on their own (the Diagnostic Log's failures are what feed the
integration-down alert above, but reading the log itself never
triggers anything).