Put alert rules, service levels, notifications and alert tickets on one Alert settings page, in four tabs
Old addresses (/alert-rules, /service-levels, /settings/notifications) lead to the right tab. Notifications and Alert tickets stay administrator-only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
07ce2b0305
commit
4269213275
40 files changed
+212
-188
No files matched your search
+24
-24
@@ -2,7 +2,7 @@
|
||||
|
||||
Every notification in Homelab Manager is sent through the same pipe
|
||||
(`server/src/services/notify.ts`) to whichever channels are enabled under
|
||||
**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
||||
**Alert settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
||||
JSON webhook, and push to phones and browsers. All five fire for every notification below unless a
|
||||
**routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the
|
||||
daily ones, one shared time/timezone.
|
||||
@@ -13,12 +13,12 @@ is closed with changes unsaved): **Channels** (the five channels, each a row
|
||||
you open to fill in and send a test), **What to send** (the event toggles
|
||||
grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When**
|
||||
(the time zone, the daily reminder time, quiet hours and the weekly digest).
|
||||
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
|
||||
The address keeps the tab, e.g. `/alert-settings?view=notifications&tab=when`.
|
||||
|
||||
Two settings apply to *every* notification regardless of what triggered
|
||||
it:
|
||||
|
||||
- **Quiet hours** (Settings → Notifications → Quiet hours): while
|
||||
- **Quiet hours** (Alert settings → Notifications → Quiet hours): while
|
||||
enabled and inside the configured window, a notification is held
|
||||
instead of sent immediately, then delivered as a single digest at the
|
||||
window's end time. This matters most for the real-time alerts below —
|
||||
@@ -26,7 +26,7 @@ it:
|
||||
outside the window anyway. *Still send critical alerts* (off by default)
|
||||
lets a **critical** one — a server down, a disk full, mains power lost —
|
||||
through at once, so only the warnings wait for the morning.
|
||||
- **Away** (Settings → Notifications → When, and a banner on every page while
|
||||
- **Away** (Alert settings → Notifications → When, and a banner on every page while
|
||||
it is on): "I'm away until Sunday". See [Away](#away).
|
||||
- **Maintenance windows** (the Maintenance page): opening one for a
|
||||
server, integration, or DNS provider silences the alerts that target
|
||||
@@ -40,7 +40,7 @@ below; several are fixed and can't be changed from the UI.
|
||||
|
||||
## Who hears what
|
||||
|
||||
Under **Settings → Notifications → Who hears what**.
|
||||
Under **Alert settings → Notifications → Who hears what**.
|
||||
|
||||
**Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of:
|
||||
the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how
|
||||
@@ -80,7 +80,7 @@ notification of its own — it is something to look at when you are looking at b
|
||||
|
||||
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
|
||||
this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under
|
||||
Settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
|
||||
Alert settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
|
||||
each person opens the same row and presses **Notify this browser**; the browser asks permission and the row lists their browsers,
|
||||
with a button to send a test and one to remove each.
|
||||
|
||||
@@ -98,7 +98,7 @@ with a button to send a test and one to remove each.
|
||||
|
||||
## Daily reminders
|
||||
|
||||
These six checks share one schedule: **Settings → Notifications → Daily
|
||||
These six checks share one schedule: **Alert settings → Notifications → Daily
|
||||
reminder time** (default 08:00) and **Timezone** (default UTC). Unlike
|
||||
the state-based alerts further down, these **re-send every day the
|
||||
condition is still true** — there's no "only once" de-duplication, so an
|
||||
@@ -112,12 +112,12 @@ until the next scheduled time to catch up.
|
||||
| Check | "Notify on" toggle | Fires when | Configured at |
|
||||
|---|---|---|---|
|
||||
| Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
|
||||
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
|
||||
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Alert settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
|
||||
| Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
|
||||
| Docker containers broken | *Warn when a Docker container is broken* | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (*Server offline, disk nearly full …*, `healthAlerts`), with a recovery message when it runs again. See [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken). | Its own switch; *Containers to leave alone* |
|
||||
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
|
||||
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
|
||||
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send |
|
||||
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Alert settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
|
||||
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Alert settings → Notifications → What to send |
|
||||
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
|
||||
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
|
||||
|
||||
@@ -137,9 +137,9 @@ starting baseline rather than announced all at once.
|
||||
|
||||
| Check | "Notify on" toggle | Fires when | Configured at |
|
||||
|---|---|---|---|
|
||||
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) |
|
||||
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Alert settings → Notifications → Health checks → **Server offline after** (default 60 min) |
|
||||
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
|
||||
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
|
||||
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Alert settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
|
||||
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
|
||||
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
|
||||
|
||||
@@ -153,7 +153,7 @@ schedule.
|
||||
| DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
|
||||
| DNS record updated | *DNS record updated* | An existing DNS record is edited. |
|
||||
| DNS record deleted | *DNS record deleted* | A DNS record is deleted. |
|
||||
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Settings → Notifications → **Alert after** (default 3 consecutive failures) |
|
||||
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Alert settings → Notifications → **Alert after** (default 3 consecutive failures) |
|
||||
| Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
|
||||
|
||||
## Digest
|
||||
@@ -170,7 +170,7 @@ A scheduled backup of the app's own database that fails sends one message throug
|
||||
|
||||
| Notification | Fires when |
|
||||
|---|---|
|
||||
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
||||
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Alert settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
||||
|
||||
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
|
||||
is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
|
||||
@@ -180,20 +180,20 @@ the December one. Both are the same report over three or twelve months, sent the
|
||||
|
||||
## Service levels
|
||||
|
||||
A service-level target (Monitoring → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used
|
||||
A service-level target (Alert settings → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used
|
||||
(a warning: "Gitea: 90% of the error budget for its 99.9% target is used up for October 2026 — down 40 min of 45 min allowed, 5 min left") and again when it is **overspent** (critical: "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed").
|
||||
It is *Monitoring* for routing, and it ends when the month turns over. Maintenance does not hold it back: planned downtime counts. Each target has its own switch.
|
||||
|
||||
## Network devices
|
||||
|
||||
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
|
||||
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Alert settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
|
||||
*Server offline, disk nearly full …* (`healthAlerts`): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as *Server down*; maintenance on the
|
||||
device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be
|
||||
read for twenty, its devices are held — neither announced nor cleared.
|
||||
|
||||
## Your own alert rules, and snoozing
|
||||
|
||||
An alert rule (Monitoring → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held
|
||||
An alert rule (Alert settings → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held
|
||||
back by maintenance on the server. It is checked on the health pass, every 15 minutes.
|
||||
|
||||
An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or recovered by the health and automation checks until the snooze ends. When it ends and the alert is still there, one **"Snooze ended"** message
|
||||
@@ -201,7 +201,7 @@ An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or r
|
||||
|
||||
## Buttons on notifications
|
||||
|
||||
Switched on under **Settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
|
||||
Switched on under **Alert settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
|
||||
that ended, and the escalation reminder — can be acted on from the message:
|
||||
|
||||
| Channel | One alert | Several |
|
||||
@@ -217,7 +217,7 @@ it came from. **Anyone who can read the notification can use it** for those 24 h
|
||||
|
||||
## Away
|
||||
|
||||
For a stretch rather than every night. **Settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
|
||||
For a stretch rather than every night. **Alert settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
|
||||
holiday ahead), a note to yourself and whether to **still send critical alerts**. While it lasts:
|
||||
|
||||
- notifications are **held** — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held;
|
||||
@@ -232,7 +232,7 @@ administrator's to set (it decides who is told what), everyone sees the banner,
|
||||
|
||||
## Flapping alerts
|
||||
|
||||
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
|
||||
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Alert settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
|
||||
|
||||
- one message, **"Flapping alert"**, says "↻ <the alert> — raised 3 times within an hour";
|
||||
- its raise and recovery messages are then **held back** (the alert stays on the Alerts page with a *flapping* badge and still counts for the menu badge and /metrics);
|
||||
@@ -250,13 +250,13 @@ carries the tags of the server, and maintenance on the server holds it back.
|
||||
|
||||
A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk
|
||||
nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under
|
||||
Settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
|
||||
Alert settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
|
||||
|
||||
## Weekly digest
|
||||
|
||||
| Notification | Fires when |
|
||||
|---|---|
|
||||
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
|
||||
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Alert settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
|
||||
|
||||
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
|
||||
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
|
||||
@@ -283,7 +283,7 @@ It runs the checks live when opened (a recent result is reused for a minute), an
|
||||
|
||||
## Tickets in osTicket
|
||||
|
||||
Not a notification, but it acts on the same alerts: with **Settings → Notifications → Alert tickets** switched on (off by default), a problem on the Alerts page opens a
|
||||
Not a notification, but it acts on the same alerts: with **Alert settings → Alert tickets** switched on (off by default), a problem on the Alerts page opens a
|
||||
ticket in osTicket. It runs every 15 minutes, starting about a minute and a half after the app starts. It never closes a ticket by itself; closing is something you choose, in the Close drawer on the Alerts page, when the integration has a separate database login for it (see INTEGRATIONS.md).
|
||||
|
||||
| Rule | What it means |
|
||||
@@ -323,7 +323,7 @@ the app alerts on something:
|
||||
other manual "Sync from X" action — these run when you click the
|
||||
button and report their result on the page itself, not via a
|
||||
notification.
|
||||
- **Test buttons** (Settings → Notifications → each channel's "Send
|
||||
- **Test buttons** (Alert settings → Notifications → each channel's "Send
|
||||
test") send a one-off message through that one channel only, to
|
||||
confirm it's wired up correctly — not a real event and not affected by
|
||||
quiet hours.
|
||||
|
||||
Reference in new issue
Block a user