Put alert rules, service levels, notifications and alert tickets on one Alert settings page, in four tabs

Old addresses (/alert-rules, /service-levels, /settings/notifications) lead to the right tab.
Notifications and Alert tickets stay administrator-only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-03 20:38:15 +01:00
1 parent 07ce2b0305
commit 4269213275
40 files changed
+212 -188

No files matched your search

+1 -1
View File
@@ -254,7 +254,7 @@ Each agent says which operating system it runs, in the report's `system.os`: `{
- **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart.
The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows
client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Settings → Notifications (*Flag an
client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Alert settings → Notifications (*Flag an
operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent
older than this version reports no OS and the server shows nothing for it.
+1 -1
View File
@@ -256,7 +256,7 @@ was taken (from your `.env` backup).
Symptoms: an integration or a DNS provider fails with *"Unsupported state or unable to
authenticate data"*, and notification channels that worked before don't. The database is fine;
only the stored credentials can't be read. If you can't find the old key, enter the credentials
again (Integrations → edit, DNS providers, Settings → Notifications). Nothing else is lost.
again (Integrations → edit, DNS providers, Alert settings → Notifications). Nothing else is lost.
**Don't** change `CREDENTIALS_ENCRYPTION_KEY` on a running system to "fix" this: it makes
everything stored so far unreadable.
+3 -3
View File
@@ -1207,7 +1207,7 @@ How many were streaming, and how many of those were transcoded, at a moment: one
#### `service_levels`
A service-level target (Monitoring → Service levels): what a thing is held to, measured over calendar months. No foreign keys: the thing measured is a plain reference, and a target whose subject is removed is shown as having nothing to measure.
A service-level target (Alert settings → Service levels): what a thing is held to, measured over calendar months. No foreign keys: the thing measured is a plain reference, and a target whose subject is removed is shown as having nothing to measure.
| Column | Type | Notes |
|---|---|---|
@@ -1268,7 +1268,7 @@ Each report of a backup, for its history: 400 days and at most 400 per job. `bac
#### `alert_rules`
Limits of the administrator's own (Monitoring → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule:<id>:server:<server>[:<disk>]`.
Limits of the administrator's own (Alert settings → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule:<id>:server:<server>[:<disk>]`.
| Column | Type | Notes |
|---|---|---|
@@ -1562,7 +1562,7 @@ that were never changed aren't stored; the app fills in defaults when reading.
| `display` | `dateFormat`, `timeFormat`, `pageSize`. |
| `providerColors`, `integrationColors` | Badge colours, by provider / integration type. |
| `consistency` | `excludedRanges` — addresses the Consistency and IP Addresses pages ignore. Default `["172.16.0.0/12"]` (Docker's networks). |
| `alertTickets` | Opening osTicket tickets for alerts: `enabled`, `integrationId`, `severities`, `categories`, `requesterName`/`requesterEmail`, `topicId`, `priorityId`, `notifyStaff`, `existingProblems`, `confirmChecks`, `summaryAbove`, `reopenAfterMinutes`. Edited under Settings → Notifications → Alert tickets. The API address and key are on the osTicket integration, not here. |
| `alertTickets` | Opening osTicket tickets for alerts: `enabled`, `integrationId`, `severities`, `categories`, `requesterName`/`requesterEmail`, `topicId`, `priorityId`, `notifyStaff`, `existingProblems`, `confirmChecks`, `summaryAbove`, `reopenAfterMinutes`. Edited under Alert settings → Alert tickets. The API address and key are on the osTicket integration, not here. |
| `nameGenerator` | `themes` — the Generator's server-name lists: `[{ "id", "label", "names": [...], "lastImport"? }]`. Edited under Settings → Names. |
Rows whose key starts with **`_internal:`** are scheduler bookkeeping, not
+6 -6
View File
@@ -70,7 +70,7 @@ It is for the readers written from documentation (see "Not verified" on each int
- **Container health:** on the 15-minute health pass the app also reads every container's state and status text and raises alerts for what is broken — see [Docker containers that are broken](docs/MONITORING.md#docker-containers-that-are-broken). A Docker host that Dockhand
cannot read (its Hawser agent is down) is an alert too, and its containers' alerts are held, not cleared. The Docker page shows each container's problem and has a **Leave alone** button.
- **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host — and so are **high** ones once a host has as many as a number you set (*Also warn when a Docker host has this many high vulnerabilities*, off by default) — (a setting turns the critical one off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*.
- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged.
- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Alert settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged.
### Uptime Kuma
@@ -169,7 +169,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by
PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot),
and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed
verification or a datastore couldn't be read at all — the "Notify on" section in Settings → Notifications has a
verification or a datastore couldn't be read at all — the "Notify on" section in Alert settings → Notifications has a
dedicated toggle for it, sharing the same daily reminder time as the other daily checks.
- **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the
`PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run
@@ -220,7 +220,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
a field looks wrong, tell us what happened and we'll adjust — this one has had less real-world exposure than
every other integration listed here.
**Optional: opening tickets for alerts** (Settings → Notifications → Alert tickets)
**Optional: opening tickets for alerts** (Alert settings → Alert tickets)
- **Config fields:** Ticket API address (optional), Ticket API key (optional, stored encrypted like the database
password), Allow self-signed certificate (ticket API). Leave them empty and the integration works as before.
@@ -229,7 +229,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
app's requests come from (behind Docker or a reverse proxy that is the host's or proxy's address, not the
container's — check osTicket's log if it refuses) and tick **Can create tickets**. osTicket refuses a key from any
other address, which is the most common reason a test ticket fails.
- **What is sent:** one ticket per problem — name and email you set under Settings → Notifications → Alert tickets, a subject, the
- **What is sent:** one ticket per problem — name and email you set under Alert settings → Alert tickets, a subject, the
problem's text and a link back to this app, optionally a help topic and priority id — with `autorespond` off (the
"user" is this app, so an acknowledgement would only bounce) and osTicket's own staff alert on or off as you choose.
- **What the API can't do:** close, reopen, or comment on a ticket — osTicket's API has no such endpoint. Closing one is
@@ -263,7 +263,7 @@ osTicket's API cannot close a ticket, so this is the one place in the app that *
*Closed from Homelab Manager by <name>* with the reason you gave; and a **"closed" event** is recorded so the ticket's history reads like any other. Nothing else is touched, and nothing is ever deleted.
- **What it checks first:** that the tables and columns it writes to are there (written for **osTicket 1.14–1.18**); a different osTicket is refused with the list of what is missing, before anything is written. It closes only a ticket that is **open**, and only if its status is still what it was when it was read — a ticket somebody
else closes at the same moment is left as they left it. A ticket that was opened as part of a **summary ticket** (many problems at once) is not closed from here, because it covers other alerts.
- **Test closing** (Settings → Notifications → Alert tickets, next to *Send a test ticket*; administrators): signs in with the login, checks the tables and columns, and then **tries every write a close makes inside a transaction that is rolled back** — so a missing grant is found there and not on the first real ticket. Nothing stays (a counter or two
- **Test closing** (Alert settings → Alert tickets, next to *Send a test ticket*; administrators): signs in with the login, checks the tables and columns, and then **tries every write a close makes inside a transaction that is rolled back** — so a missing grant is found there and not on the first real ticket. Nothing stays (a counter or two
inside the database moves on).
- **Closing several alerts at once:** pick them on the Alerts page and choose *Close*; when some of them have a ticket that can be closed from here, the drawer offers *Also close N osTicket tickets* (with their numbers), and it is closed one ticket after another, each written on its alert's incident and in the audit log as above. One that cannot be closed does not stop the others: the alerts are closed all the same and the message says how many tickets were closed, were already closed, or could not be (with the first reason). Summary tickets and an osTicket with no closing login are counted and left open. **Undo opens the alerts again but cannot reopen the tickets** — the message says so.
- **What the alert shows:** the ticket badge says *(closed)* once it has been closed from here, and the Close drawer says so instead of offering it again. If the ticket cannot be closed, the alert is **still closed** and you are told why; the reason is also written on the incident and in the audit log (`alert_ticket/close_ticket`, `close_ticket_failed`).
@@ -305,7 +305,7 @@ osTicket's API cannot close a ticket, so this is the one place in the app that *
It all appears on **Network → Network devices**.
- A device is a **node in the dependency map**, like a server: the *connected through* link is made into a dependency by itself (an access point needs its switch, the switch needs the gateway), and you can write
others by hand (a server needs the switch it is plugged into).
- A device the controller has not seen online for longer than **Settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what
- A device the controller has not seen online for longer than **Alert settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what
it is connected through if that is down too, is folded under it, like a guest under its host. It can be silenced with a maintenance window, and has runbooks like any target.
- **Servers plugged in:** the controller's list of what is connected is matched to servers by IP address (exact, and only when one server has that address). A server is then *connected through* the switch or access point
it is on — a dependency made by itself, shown on the server's page and on the Connections map. A key or account that may not read clients costs only this; devices, networks and Wi-Fi names are unaffected.
+24 -24
View File
@@ -2,7 +2,7 @@
Every notification in Homelab Manager is sent through the same pipe
(`server/src/services/notify.ts`) to whichever channels are enabled under
**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
**Alert settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
JSON webhook, and push to phones and browsers. All five fire for every notification below unless a
**routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the
daily ones, one shared time/timezone.
@@ -13,12 +13,12 @@ is closed with changes unsaved): **Channels** (the five channels, each a row
you open to fill in and send a test), **What to send** (the event toggles
grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When**
(the time zone, the daily reminder time, quiet hours and the weekly digest).
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
The address keeps the tab, e.g. `/alert-settings?view=notifications&tab=when`.
Two settings apply to *every* notification regardless of what triggered
it:
- **Quiet hours** (Settings → Notifications → Quiet hours): while
- **Quiet hours** (Alert settings → Notifications → Quiet hours): while
enabled and inside the configured window, a notification is held
instead of sent immediately, then delivered as a single digest at the
window's end time. This matters most for the real-time alerts below —
@@ -26,7 +26,7 @@ it:
outside the window anyway. *Still send critical alerts* (off by default)
lets a **critical** one — a server down, a disk full, mains power lost —
through at once, so only the warnings wait for the morning.
- **Away** (Settings → Notifications → When, and a banner on every page while
- **Away** (Alert settings → Notifications → When, and a banner on every page while
it is on): "I'm away until Sunday". See [Away](#away).
- **Maintenance windows** (the Maintenance page): opening one for a
server, integration, or DNS provider silences the alerts that target
@@ -40,7 +40,7 @@ below; several are fixed and can't be changed from the UI.
## Who hears what
Under **Settings → Notifications → Who hears what**.
Under **Alert settings → Notifications → Who hears what**.
**Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of:
the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how
@@ -80,7 +80,7 @@ notification of its own — it is something to look at when you are looking at b
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under
Settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
Alert settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
each person opens the same row and presses **Notify this browser**; the browser asks permission and the row lists their browsers,
with a button to send a test and one to remove each.
@@ -98,7 +98,7 @@ with a button to send a test and one to remove each.
## Daily reminders
These six checks share one schedule: **Settings → Notifications → Daily
These six checks share one schedule: **Alert settings → Notifications → Daily
reminder time** (default 08:00) and **Timezone** (default UTC). Unlike
the state-based alerts further down, these **re-send every day the
condition is still true** — there's no "only once" de-duplication, so an
@@ -112,12 +112,12 @@ until the next scheduled time to catch up.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Alert settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
| Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
| Docker containers broken | *Warn when a Docker container is broken* | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (*Server offline, disk nearly full …*, `healthAlerts`), with a recovery message when it runs again. See [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken). | Its own switch; *Containers to leave alone* |
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send |
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Alert settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Alert settings → Notifications → What to send |
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
@@ -137,9 +137,9 @@ starting baseline rather than announced all at once.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) |
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Alert settings → Notifications → Health checks → **Server offline after** (default 60 min) |
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Alert settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
@@ -153,7 +153,7 @@ schedule.
| DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
| DNS record updated | *DNS record updated* | An existing DNS record is edited. |
| DNS record deleted | *DNS record deleted* | A DNS record is deleted. |
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Settings → Notifications → **Alert after** (default 3 consecutive failures) |
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Alert settings → Notifications → **Alert after** (default 3 consecutive failures) |
| Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
## Digest
@@ -170,7 +170,7 @@ A scheduled backup of the app's own database that fails sends one message throug
| Notification | Fires when |
|---|---|
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Alert settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
@@ -180,20 +180,20 @@ the December one. Both are the same report over three or twelve months, sent the
## Service levels
A service-level target (Monitoring → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used
A service-level target (Alert settings → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used
(a warning: "Gitea: 90% of the error budget for its 99.9% target is used up for October 2026 — down 40 min of 45 min allowed, 5 min left") and again when it is **overspent** (critical: "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed").
It is *Monitoring* for routing, and it ends when the month turns over. Maintenance does not hold it back: planned downtime counts. Each target has its own switch.
## Network devices
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Alert settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
*Server offline, disk nearly full …* (`healthAlerts`): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as *Server down*; maintenance on the
device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be
read for twenty, its devices are held — neither announced nor cleared.
## Your own alert rules, and snoozing
An alert rule (Monitoring → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held
An alert rule (Alert settings → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held
back by maintenance on the server. It is checked on the health pass, every 15 minutes.
An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or recovered by the health and automation checks until the snooze ends. When it ends and the alert is still there, one **"Snooze ended"** message
@@ -201,7 +201,7 @@ An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or r
## Buttons on notifications
Switched on under **Settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
Switched on under **Alert settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
that ended, and the escalation reminder — can be acted on from the message:
| Channel | One alert | Several |
@@ -217,7 +217,7 @@ it came from. **Anyone who can read the notification can use it** for those 24 h
## Away
For a stretch rather than every night. **Settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
For a stretch rather than every night. **Alert settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
holiday ahead), a note to yourself and whether to **still send critical alerts**. While it lasts:
- notifications are **held** — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held;
@@ -232,7 +232,7 @@ administrator's to set (it decides who is told what), everyone sees the banner,
## Flapping alerts
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Alert settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
- one message, **"Flapping alert"**, says "↻ <the alert> — raised 3 times within an hour";
- its raise and recovery messages are then **held back** (the alert stays on the Alerts page with a *flapping* badge and still counts for the menu badge and /metrics);
@@ -250,13 +250,13 @@ carries the tags of the server, and maintenance on the server holds it back.
A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk
nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under
Settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
Alert settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
## Weekly digest
| Notification | Fires when |
|---|---|
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Alert settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
@@ -283,7 +283,7 @@ It runs the checks live when opened (a recent result is reused for a minute), an
## Tickets in osTicket
Not a notification, but it acts on the same alerts: with **Settings → Notifications → Alert tickets** switched on (off by default), a problem on the Alerts page opens a
Not a notification, but it acts on the same alerts: with **Alert settings → Alert tickets** switched on (off by default), a problem on the Alerts page opens a
ticket in osTicket. It runs every 15 minutes, starting about a minute and a half after the app starts. It never closes a ticket by itself; closing is something you choose, in the Close drawer on the Alerts page, when the integration has a separate database login for it (see INTEGRATIONS.md).
| Rule | What it means |
@@ -323,7 +323,7 @@ the app alerts on something:
other manual "Sync from X" action — these run when you click the
button and report their result on the page itself, not via a
notification.
- **Test buttons** (Settings → Notifications → each channel's "Send
- **Test buttons** (Alert settings → Notifications → each channel's "Send
test") send a one-off message through that one channel only, to
confirm it's wired up correctly — not a real event and not affected by
quiet hours.
+20 -19
View File
@@ -187,6 +187,7 @@ All modules from the original plan are built:
Backup page's job and run lists, the comparison table) have an Export CSV
too, which writes the table as it is on screen — times as exact timestamps,
action buttons left out.
- **Alert settings** (Monitoring → Alert settings) — everything that decides what is an alert and who hears of it, on one page with four tabs: *Alert rules*, *Service levels*, *Notifications* and *Alert tickets* (`/alert-settings?view=rules|levels|notifications|tickets`). Everyone signed in sees the first two; the last two are administrators' (they used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/notifications` still lead to the right tab). Maintenance stays on its own page.
- **Settings** (admin-only) — notification channels (Gotify, ntfy, SMTP,
generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS
record added/updated/deleted, daily secret-expiry reminder and daily
@@ -221,7 +222,7 @@ Server and storage health is watched every 15 minutes: a server whose agent
stops reporting, a server disk / Proxmox storage / Synology volume passing a
usage threshold, and a Synology volume or disk that's degraded or failing each
raise one notification when the problem starts and one when it clears (both
thresholds are set under Settings → Notifications). Active problems are
thresholds are set under Alert settings → Notifications). Active problems are
remembered across restarts, so a rebuild doesn't re-alert them.
**Tags** — servers can be tagged (prod, media, rack-1, …) from their detail
@@ -305,7 +306,7 @@ like `lab.example.se` resolves to the `example.se` registration that actually
expires); others can be added by hand. Each is looked up daily over RDAP where
the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what
makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap.
You're reminded daily from N days before expiry (Settings → Notifications,
You're reminded daily from N days before expiry (Alert settings → Notifications,
default 30) until it's renewed, and told when an expiry date couldn't be
refreshed for several days. Registries that don't publish an expiry (`.de`,
`.eu`) can be tracked but have no date to warn about. Private zones (`.lan`,
@@ -339,7 +340,7 @@ send the notifications — so the two can't disagree — but ignores the on/off
toggles, since it's for looking at rather than being interrupted by. Problems
under a maintenance window stay listed, marked silenced and counted separately.
It also looks ahead: from an agent server's own history (see its Trends card), a
disk that is on course to be full within 14 days (Settings → Notifications → "Warn
disk that is on course to be full within 14 days (Alert settings → Notifications → "Warn
when a disk will fill within"; 0 turns it off) is flagged with its rate of growth
and the days left — critical when that is three days or less. It needs at least
three days of steady growth to say anything, skips a disk that jumped once or
@@ -354,7 +355,7 @@ It checks live (a recent result is reused for a minute; "Check now" forces a
fresh one), and anything it couldn't read is called out at the top rather than
quietly treated as fine.
**Alert tickets** (Settings → Notifications → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the
**Alert tickets** (Alert settings → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the
problems on the Alerts page are looked at, and one that is new and has lasted two checks in a row gets one ticket — never a second for the same problem,
and never one for something under a maintenance window or about osTicket itself. You choose which severities and kinds count (critical and warning by
default), and the Alerts page shows the ticket number on each alert that has one. If many problems appear at once — a network drop that makes thirty
@@ -455,7 +456,7 @@ alerts always stand alone.
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also
A **failed** test is always an alert (critical) until a newer one passes. Under Alert settings → Notifications → "A restore should have been tested within" you can also
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
kept, `/metrics` has each subject's age and result, and everything is in the change log.
@@ -474,7 +475,7 @@ so it is folded only when every enabled instance of that kind is explained. A se
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
alerts of their own, so they aren't part of this.)
**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
**Who hears what** (Alert settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
@@ -762,7 +763,7 @@ alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n
status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md).
**Service levels** (Monitoring → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
**Service levels** (Alert settings → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
month allows about 45 minutes; 99 % about 7 hours). A target is set on an **endpoint check** (measured from its answers, day by day), a **server** or an **integration** (measured from when the audit log recorded it going away and
coming back, to the minute), and is measured over calendar months in the notification time zone. Each card shows the figure so far, the downtime used, what is allowed and what is left, how much of the budget is used (and where the
pace so far would end the month), and how each of the last six months went. When **80 %** of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over — switchable per target. Planned maintenance is
@@ -773,9 +774,9 @@ pace so far would end the month), and how each of the last six months went. When
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Alert settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
**Weekly digest** (Alert settings → Notifications, off until switched on) sends one message a
week through your notification channels: what is wrong now, what was down and for how
long, what happened, disks heading for full, things about to expire, updates waiting
and maintenance coming up — each topic only if it has something to say, and "a quiet
@@ -890,18 +891,18 @@ minimum in both themes. Motion is turned down for those who ask the system to.
**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that
nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up
(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Alert settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
**Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and
never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button;
the same list is under Settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
the same list is under Alert settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's
averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey
**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least
30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The
warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*).
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Alert settings → Notifications (*Servers and disks*).
**A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four
months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too
@@ -917,7 +918,7 @@ servers, alerts, incidents, inventory, network equipment, endpoint checks and ma
gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab —
its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices
join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Alert settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
(also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md.
**Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then
@@ -932,7 +933,7 @@ hours (maintenance), is handed to you or is still going is counted in the badge
**A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits),
switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section.
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Alert settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
(a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains
power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is
sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md.
@@ -942,7 +943,7 @@ all find something. Removing a dependency, an admin link, a runbook, a hand-made
what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first.
**Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same
alert has been **raised three times within an hour** (Settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
alert has been **raised three times within an hour** (Alert settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics.
Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks)
still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log.
@@ -959,7 +960,7 @@ what was there before (the earlier acknowledgement text, the earlier tags, the e
**Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a
badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows
(client and server). A server whose system is past its end, or ends within **Settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
(client and server). A server whose system is past its end, or ends within **Alert settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`.
**Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a
@@ -995,7 +996,7 @@ a battery to be replaced and a load of 90 % are warnings). The UPS is also a **n
folded under its alert ("also takes down …") instead of each being announced; the page lists what is on it in the **order to shut it down**, so nothing is switched off while something that needs it is still running. The app only
reads — it never sends a command to the UPS. Not tried against a real UPS; see INTEGRATIONS.md.
**Alert rules of your own.** **Monitoring → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*,
**Alert rules of your own.** **Alert settings → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*,
*a disk above 80%* (every disk, or the one you name), *a disk warmer than 55 °C*. A rule says what to watch, above or below what, for how long (a reading in the history every 10–15 minutes; every reading in that time has
to be past the limit, so a dip starts the count again), how serious it is and which servers it covers (all, one, or those with a tag). **Try it now** shows which servers it would be raised on right now before it is saved.
What a rule finds is an ordinary alert on the Alerts page, in the *Monitoring* category with the rule's own severity — announced, recovered, held back while flapping, silenced by maintenance on the server and
@@ -1005,7 +1006,7 @@ the disk's last report, so it is judged right now and has no time.
**Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's
storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside
each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window
(Settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
(Alert settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
`homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history.
**Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under
@@ -1024,7 +1025,7 @@ switch it is plugged into; something that needs it; a change that touches everyt
*close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a
change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API.
**Buttons on notifications.** With **Settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
**Buttons on notifications.** With **Alert settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as
buttons — three on an ntfy message, two on a phone push (the browser's own notification), with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons;
Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it
+10 -10
View File
@@ -53,13 +53,12 @@ page's own top-level link.
| Consistency | `/consistency` | ✅ | ✅ | ✅ |
| **Monitoring** | | | | |
| Endpoints | `/endpoints` | ✅ | ✅ | ✅ |
| Alert rules | `/alert-rules` | ✅ | ✅ | ✅ |
| Alert settings (tabs: Alert rules, Service levels; and for administrators only Notifications, Alert tickets) | `/alert-settings` | ✅ | ✅ | ✅ |
| Heartbeats | `/heartbeats` | ✅ | ✅ | ✅ |
| Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ |
| UPS | `/ups` | ✅ | ✅ | ✅ |
| Prometheus | `/prometheus` | ✅ | ✅ | ✅ |
| Speedtest | `/speedtest` | ✅ | ✅ | ✅ |
| Service levels | `/service-levels` | ✅ | ✅ | ✅ |
| **Automation** | | | | |
| Semaphore | `/semaphore` | ✅ | ✅ | ✅ |
| Gitea | `/gitea` | ✅ | ✅ | ✅ |
@@ -126,7 +125,7 @@ even further, to admin only:
- **Alert tickets**: everyone sees which alerts have an osTicket ticket (a
badge on the Alerts page, linking to the ticket in osTicket); switching the
feature on, choosing what opens a ticket, and sending a test ticket are
under Settings → Notifications → Alert tickets, which is admin-only. **Closing a ticket** along with an alert (the tick box in the Close drawer, for one alert or several picked together) is an operator's, like closing the alert; the login it writes with, and **Test closing**, are administrators'.
under Alert settings → Alert tickets, which is admin-only. **Closing a ticket** along with an alert (the tick box in the Close drawer, for one alert or several picked together) is an operator's, like closing the alert; the login it writes with, and **Test closing**, are administrators'.
- **Runbooks**: everyone reads them (on a server's page, an integration's card, the *What to do* link beside an alert for a whole kind of alert,
next to an alert); writing, editing and removing one needs operator.
- **Inventory**: everyone can see the hardware (and print its labels); adding, editing and removing needs
@@ -138,11 +137,11 @@ even further, to admin only:
**Settings search** is part of Settings, which is administrators'.
- **Network devices** (Network → Network devices): everyone can see the devices, networks and Wi-Fi names read from UniFi controllers; **Read now** and putting a device under maintenance need operator; adding the UniFi
integration itself (a key or login) is an administrator's job, like any integration.
- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Settings → Notifications (administrators).
- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Alert settings → Notifications (administrators).
- **Docker containers** (Docker page): everyone signed in sees each container and what is wrong with it; start, stop and restart need operator; **Leave alone** (no alerts for a container) and the list in Settings are administrators'.
- **Away mode** (Settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held.
- **Buttons on notifications** (Settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else.
- **Alert rules** (Monitoring → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is
- **Away mode** (Alert settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held.
- **Buttons on notifications** (Alert settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else.
- **Alert rules** (Alert settings → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is
woken up). **Snooze** an alert, wake it, **close** it and open it again need operator, like acknowledging. The storage forecast beside Proxmox storages and Synology volumes is for everyone who can see those pages.
- **What changed before this?** (on an incident's page, and `GET /api/v1/incidents/:id/before`): everyone signed in, like the incident record it is made from.
- **Dependencies**: everyone can see what needs what (including what needs the Internet); adding and removing one needs operator. So does adding or hiding a *suggested* one.
@@ -152,7 +151,7 @@ even further, to admin only:
- **Proxy hosts** (Network → Proxy hosts): everyone signed in sees what Nginx Proxy Manager serves; **Read now** needs operator; adding the integration (its login) is an administrator's job, and so is **making, changing, switching off and deleting a proxy host** — only where the integration was given a user that may manage.
- **Home Assistant** (Apps → Home Assistant): everyone signed in sees what is watched, its state and history; choosing what to watch, its limits, stopping and **Read now** need operator (the list of all
entities is read live, so only operators see it); adding the integration (its token) is an administrator's job.
- **Service levels** (Monitoring → Service levels): everyone signed in sees the targets and what is left of the budget; setting, changing and removing one is administrator-only, and each is in the audit log.
- **Service levels** (Alert settings → Service levels): everyone signed in sees the targets and what is left of the budget; setting, changing and removing one is administrator-only, and each is in the audit log.
- **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map.
- **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone;
*Check now* needs operator.
@@ -175,7 +174,7 @@ even further, to admin only:
- **Duplicati** (Backups → Duplicati): everyone signed in sees every backup and how it went; setting how often one is expected, muting it and forgetting it need operator; adding a machine (which makes an address that can write to this app) and removing one are administrators'. Reports arrive on a secret address with no sign-in: the token is the access.
- **Restore tests**: everyone sees them on the Backup coverage page; recording one needs operator; removing one is administrator-only.
- **Change log**: operators and administrators (it is made from the audit log, which has the same readers).
- **Push notifications**: an administrator switches the channel on (Settings → Notifications). Each person, whatever their role, adds
- **Push notifications**: an administrator switches the channel on (Alert settings → Notifications). Each person, whatever their role, adds
and removes their own browsers and can send a test to them; an administrator can remove anyone's.
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
are going; starting an update run (**Update…**) or stopping one needs operator.
@@ -199,7 +198,8 @@ even further, to admin only:
- **The delivery log of an event webhook** (what was sent, with the messages, and sending one again) and **Settings → Data → Storage** (how big the app's data is) are administrator-only; sending a message again is in the audit log.
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
- Everything under **Settings** (notification channels, badge colors,
- **Alert settings**: the *Alert rules* and *Service levels* tabs are for everyone to read (changing them is administrators'); the *Notifications* and *Alert tickets* tabs are only shown to administrators, as they were under Settings.
- Everything under **Settings** (badge colors,
display prefs, log retention, backup/restore) is admin-only, matching
the page itself being admin-only.
+28 -28
View File
@@ -72,7 +72,7 @@ Three ideas run through everything:
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications |
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Alert settings → Notifications |
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
@@ -84,7 +84,7 @@ After the app **restarts**, server-based checks are paused for **20 minutes** so
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Alert settings → Notifications → *Servers and disks*), or the server's own limit if it
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
@@ -129,22 +129,22 @@ Every threshold here is a default; where it can be changed, the place is named.
| Check | Raised when | Default | Changed in |
|---|---|---|---|
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk |
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Alert settings → Notifications; per server |
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Alert settings → Notifications; per server, per disk |
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Alert settings → Notifications; per server |
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Settings → Notifications |
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Settings → Notifications |
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Alert settings → Notifications |
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Alert settings → Notifications |
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Alert settings → Notifications |
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Alert settings → Notifications |
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Alert settings → Notifications; per server |
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Alert settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Alert settings → Notifications |
| **Docker updates** | A container image has a newer version | | Alert settings → Notifications |
| **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) |
| **Hetzner server off / traffic / rescue / backup** | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; [below](#hetzner-cloud) |
| **Sonarr / Radarr / Prowlarr** | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; [below](#media-managers-sonarr-radarr-prowlarr) |
@@ -161,19 +161,19 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) |
| **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) |
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
| **App behind its newest release** | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; [below](#app-versions) |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Alert settings → Notifications |
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
| **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) |
| **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) |
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
| **Your own rules** | [Below](#your-own-alert-rules) | | Alert settings → Alert rules |
## What is normal for the time
@@ -192,13 +192,13 @@ It is **quiet by design**. A reading is a warning only when **all** of these hol
The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health
condition (`baseline:cpu:server:<id>`, `baseline:mem:server:<id>`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for
a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Settings → Notifications → What to send → Servers and disks.
a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Alert settings → Notifications → What to send → Servers and disks.
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
## Service levels
**Monitoring → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the
**Alert settings → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the
downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and
where today's pace would end the month, once a day has passed), and how each of the last six months went.
@@ -217,7 +217,7 @@ and gone when the month turns over and the budget starts again. Each target can
## Your own alert rules
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
Where the built-in checks do not fit, **Alert settings → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
- **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
- **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the
@@ -475,14 +475,14 @@ The hard part is *stopped*: many containers are stopped on purpose (a stack that
that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already
long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone.
(Settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.)
(Alert settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.)
**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything.
**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Alert settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything.
Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a
Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`).
**Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other.
**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Alert settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
## Endpoint checks and heartbeats
@@ -527,7 +527,7 @@ Both are judged **every minute**, with the same remember-and-compare bookkeeping
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
- **An integration is "failing"** when its last **3 calls in a row** (Alert settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
integration's page.
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
@@ -542,7 +542,7 @@ A forecast says "this will be full in N days" from a **straight line through the
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
- It is shown as an alert when it falls **within the window** (Alert settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
@@ -579,7 +579,7 @@ browser's own wishes. **Quiet hours** hold messages and send one digest at the e
### Flapping
An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
An alert **raised three times within an hour** (Alert settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
### Escalation
+1 -1
View File
@@ -257,7 +257,7 @@ All gauges. The same table is checked by a test against what `/metrics` really s
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Alert settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
+3 -3
View File
@@ -6,7 +6,7 @@ here, and they are easy to mix up:
| | **Event webhooks** | **The notification webhook** |
|---|---|---|
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
| Where | **Settings → API and webhooks → Event webhooks** | **Alert settings → Notifications → Channels → Webhook** |
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
@@ -274,7 +274,7 @@ def hook():
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
**Alert settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
```
POST <your address>
@@ -294,7 +294,7 @@ X-Webhook-Secret: <the secret> (only if one is set)
## Buttons on notifications
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
it on under **Alert settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
The addresses behind them, for anything that wants to build on it:
+1 -1
View File
@@ -1788,7 +1788,7 @@ export const webhookEvents = ["alert.opened", "alert.resolved", "maintenance.sta
export type WebhookEvent = (typeof webhookEvents)[number];
// A URL that gets a signed JSON message for the events it subscribes to — for Home Assistant, n8n, a status board. Separate from the
// chat-style webhook under Settings → Notifications, which carries the same text a person would read.
// chat-style webhook under Alert settings → Notifications, which carries the same text a person would read.
export const eventWebhooks = sqliteTable("event_webhooks", {
id: integer("id").primaryKey({ autoIncrement: true }),
label: text("label").notNull(),
+1 -1
View File
@@ -224,7 +224,7 @@ export const INTEGRATION_FIELDS: Partial<Record<IntegrationType, IntegrationFiel
optional: true,
placeholder: "ost_ (osTicket's default, unless changed at install)",
},
// Only for opening tickets from alerts (Settings → Notifications → Alert tickets). The rest of this integration reads the database and never writes.
// Only for opening tickets from alerts (Alert settings → Alert tickets). The rest of this integration reads the database and never writes.
{
key: "apiUrl",
label: "Ticket API address (optional)",
+1 -1
View File
@@ -239,7 +239,7 @@ export function agendaFromData(data: AgendaData, now: Date, tz: string, days: nu
day(`renewal:${r.id}`, "renewal", r.nextRenewal, () => ({ title: `${r.name} is billed — ${money}`, link: "/costs" }));
}
// A restore test is due a set time after the last good one — only if a time limit is set (Settings → Notifications).
// A restore test is due a set time after the last good one — only if a time limit is set (Alert settings → Notifications).
if (data.restoreLimitDays > 0) {
for (const t of data.restoreTests) {
if ((t.result !== "ok" && t.result !== "partial") || !isYmd(t.testedAt)) continue;
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* The links behind buttons on a notification (see services/alertActions.ts for what they do): Acknowledge, Silence for an hour, Snooze until tomorrow morning — from the phone push, an ntfy message, a link in an email or a Gotify message,
* without signing in to the app. A switch an administrator turns on (Settings → Notifications → Who hears what); off, no link works.
* without signing in to the app. A switch an administrator turns on (Alert settings → Notifications → Who hears what); off, no link works.
*
* What makes it safe enough to have is how little each link can do:
*
+3 -3
View File
@@ -74,15 +74,15 @@ export function mostWithin(times: number[], windowMs: number): number {
/** What to try, from the kind of alert and the key it has. Plain advice; nothing it says is done for anyone. */
export function hintFor(category: string, alertId: string): string {
const key = keyOfAlertId(alertId, category);
if (key.startsWith("rule:")) return "A limit of your own (Monitoring → Alert rules): if it keeps coming back, raise the limit or make it hold for longer before it raises.";
if (key.startsWith("rule:")) return "A limit of your own (Alert settings → Alert rules): if it keeps coming back, raise the limit or make it hold for longer before it raises.";
if (key.startsWith("new-port:")) return "If this port should be listening, mark it as expected on the server's Ports card.";
if (key.startsWith("heartbeat")) return "If the job is just sometimes late, give its heartbeat more grace; if it fails, look at what it printed.";
if (key.startsWith("endpoint:")) return "Raise “Alert after … failures in a row” on the check, or its timeout or interval — or fix what is making it fail.";
if (key.startsWith("endpoint-cert:")) return "Renew the certificate, or lower how early the check warns.";
if (key.startsWith("failed-units:")) return "If a service is expected to fail, add it to “Services to ignore” on the server's Host health card; otherwise fix the service.";
if (key.startsWith("disk-forecast:") || category === "disk") return "Raise the limit for this disk (a server can have its own, per disk), or leave it alone on the server's page if it is meant to be full.";
if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Settings → Notifications, or give this server its own limit.";
if (category === "integration") return "Raise “Alert after … consecutive failures” under Settings → Notifications; the Call log (Integrations) shows what is failing.";
if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Alert settings → Notifications, or give this server its own limit.";
if (category === "integration") return "Raise “Alert after … consecutive failures” under Alert settings → Notifications; the Call log (Integrations) shows what is failing.";
if (category === "backup") return "A guest that is backed up late each time may need its “too old” limit raised on its server, or its job moved.";
return "Acknowledge it with a note, silence it with maintenance while you work on it, or fix what raises it.";
}
+1 -1
View File
@@ -360,7 +360,7 @@ export async function sendTestTicket(actor: CurrentUser): Promise<CreatedTicket>
topicId: settings.topicId,
priorityId: settings.priorityId,
subject: "[Homelab] Test ticket from Homelab Manager",
message: `This is a test. ${actor.name ?? actor.email ?? "An administrator"} asked Homelab Manager to open it from Settings → Notifications → Alert tickets, to check that tickets can be created here.\n\nNothing is wrong. You can close this ticket.\n\n${absolute("/settings/alert-tickets")}`,
message: `This is a test. ${actor.name ?? actor.email ?? "An administrator"} asked Homelab Manager to open it from Alert settings → Alert tickets, to check that tickets can be created here.\n\nNothing is wrong. You can close this ticket.\n\n${absolute("/settings/alert-tickets")}`,
});
await recordAudit({ actor, category: "alert_ticket", action: "test", targetType: "integration", targetId: settings.integrationId ?? undefined, detail: { ticket: created.number } });
return created;
+1 -1
View File
@@ -73,7 +73,7 @@ const SOURCES: Record<string, { label: string; link: string }> = {
npm: { label: "Nginx Proxy Manager", link: "/proxy-hosts" },
homeassistant: { label: "Home Assistant", link: "/home-assistant" },
ups: { label: "UPS", link: "/ups" },
slo: { label: "Service level", link: "/service-levels" },
slo: { label: "Service level", link: "/alert-settings?view=levels" },
duplicati: { label: "Duplicati", link: "/duplicati" },
jellyfin: { label: "Jellyfin", link: "/jellyfin" },
authentik: { label: "Authentik", link: "/authentik" },
+1 -1
View File
@@ -5,7 +5,7 @@
* Nothing here starts a scan: Dockhand scans when it is told to (by hand, after a pull, or for a whole host), and this only reads what it kept. So an image that was never scanned is
* not known to be clean — the page says how many of a host's images have a scan — and an environment whose summary could not be read is *unknown*, never *none*.
*
* What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Settings → Notifications). Dockhand can also notify
* What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Alert settings → Notifications). Dockhand can also notify
* about vulnerabilities itself; if both are switched on you hear twice.
*/
import { and, eq, notInArray } from "drizzle-orm";
+1 -1
View File
@@ -1,7 +1,7 @@
/**
* Event webhooks: tell another system when something happens. Each webhook is a URL with the events it wants; when one happens it is
* sent a JSON message, signed with the webhook's secret so the receiver can tell it came from here. This is separate from the chat-style
* webhook under Settings → Notifications, which carries the same sentence a person would read.
* webhook under Alert settings → Notifications, which carries the same sentence a person would read.
*
* alert.opened / alert.resolved a problem began or ended — on the same terms as a notification (a problem under maintenance,
* or folded under another, isn't announced until it stands alone)
+1 -1
View File
@@ -1,7 +1,7 @@
/**
* Restore tests: a backup nobody has ever restored is a hope, not a backup. Somebody records when they tried to restore a guest (or anything else
* — a share, a database) and how it went, and the app keeps the history, shows each guest's newest test beside its backups, and speaks up
* when a test failed (always) or when none has been done in too long (if a time limit is set under Settings → Notifications).
* when a test failed (always) or when none has been done in too long (if a time limit is set under Alert settings → Notifications).
*/
import { desc } from "drizzle-orm";
import { db } from "../db/client.js";
+1 -1
View File
@@ -1,5 +1,5 @@
/**
* Per-server alert thresholds. The ones in Settings → Notifications apply to every server; some machines are different — a NAS that sits at
* Per-server alert thresholds. The ones in Alert settings → Notifications apply to every server; some machines are different — a NAS that sits at
* 92% by design, a database host that should speak up at 80%, a laptop that is meant to be off at night. Each server may override the
* server-offline time, the disk-usage percentage, the disk forecast window and (for the Proxmox guest it is linked to) how old a backup may
* be, and may set a limit for one disk or say not to alert about that disk at all. Anything not overridden follows the global setting, so
+3 -3
View File
@@ -165,7 +165,7 @@ export interface MonthlyReportSettings {
time: string;
}
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Alert settings → Notifications. */
export interface DigestSettings {
enabled: boolean;
/** 0 = Sunday … 6 = Saturday, in notifications.timezone. */
@@ -235,7 +235,7 @@ export interface NameGeneratorSettings {
export type AlertTicketSeverity = "critical" | "warning" | "info";
/** Opening an osTicket ticket for an alert. Off until an admin sets it up under Settings → Notifications → Alert tickets. */
/** Opening an osTicket ticket for an alert. Off until an admin sets it up under Alert settings → Alert tickets. */
export interface AlertTicketSettings {
enabled: boolean;
/** The osTicket integration whose API (address and key) is used. */
@@ -397,7 +397,7 @@ function openValue(section: string, field: string, stored: unknown): unknown {
const name = `${section}.${field}`;
if (!warnedUnreadable.has(name)) {
warnedUnreadable.add(name);
console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Settings → Notifications.`);
console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Alert settings → Notifications.`);
}
return "";
}
+8 -8
View File
@@ -8,7 +8,7 @@ import Secrets from "./pages/Secrets";
import Ipam from "./pages/Ipam";
import NetworkDevices from "./pages/NetworkDevices";
import ProxyHosts from "./pages/ProxyHosts";
import AlertRules from "./pages/AlertRules";
import AlertSettings, { NotificationsRedirect } from "./pages/AlertSettings";
import HomeAssistant from "./pages/HomeAssistant";
import Ups from "./pages/Ups";
import Jellyfin from "./pages/Jellyfin";
@@ -27,7 +27,6 @@ import TechnitiumDns from "./pages/TechnitiumDns";
import Grafana from "./pages/Grafana";
import Speedtest from "./pages/Speedtest";
import Costs from "./pages/Costs";
import ServiceLevels from "./pages/ServiceLevels";
import Duplicati from "./pages/Duplicati";
import MyWorkPage from "./pages/MyWork";
import ApiTokens from "./pages/ApiTokens";
@@ -66,7 +65,7 @@ import BackupsHub from "./pages/BackupsHub";
import Consistency from "./pages/Consistency";
import Privacy from "./pages/Privacy";
import Settings from "./pages/Settings";
import { DataGroup, DisplayGroup, NotificationsGroup } from "./pages/settings/SettingsGroups";
import { DataGroup, DisplayGroup } from "./pages/settings/SettingsGroups";
import TagSettings from "./pages/settings/TagSettings";
import NameSettings from "./pages/settings/NameSettings";
import ShareSettings from "./pages/settings/ShareSettings";
@@ -171,7 +170,8 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/ipam" element={<Ipam user={user} />} />
<Route path="/network/devices" element={<NetworkDevices user={user} />} />
<Route path="/proxy-hosts" element={<ProxyHosts user={user} />} />
<Route path="/alert-rules" element={<AlertRules user={user} />} />
<Route path="/alert-settings" element={<AlertSettings user={user} />} />
<Route path="/alert-rules" element={<Navigate to="/alert-settings" replace />} />
<Route path="/home-assistant" element={<HomeAssistant user={user} />} />
<Route path="/ups" element={<Ups user={user} />} />
<Route path="/jellyfin" element={<Jellyfin />} />
@@ -191,7 +191,7 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/grafana" element={<Grafana />} />
<Route path="/speedtest" element={<Speedtest />} />
<Route path="/costs" element={<Costs user={user} />} />
<Route path="/service-levels" element={<ServiceLevels user={user} />} />
<Route path="/service-levels" element={<Navigate to="/alert-settings?view=levels" replace />} />
<Route path="/duplicati" element={<Duplicati user={user} />} />
<Route path="/ports" element={<Ports user={user} />} />
<Route path="/secrets" element={<Secrets user={user} />} />
@@ -248,8 +248,8 @@ function Shelled({ user }: { user: CurrentUser }) {
</RequireRole>
}
>
<Route index element={<Navigate to="notifications" replace />} />
<Route path="notifications" element={<NotificationsGroup />} />
<Route index element={<Navigate to="sign-in" replace />} />
<Route path="notifications" element={<NotificationsRedirect />} />
<Route path="sign-in" element={<SignInSettings />} />
<Route path="display" element={<DisplayGroup />} />
<Route path="names" element={<NameSettings />} />
@@ -259,7 +259,7 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="data" element={<DataGroup />} />
{/* Tabs that were gathered into others: old links and bookmarks still arrive. */}
<Route path="badges" element={<Navigate to="/settings/display?view=badges" replace />} />
<Route path="alert-tickets" element={<Navigate to="/settings/notifications?view=tickets" replace />} />
<Route path="alert-tickets" element={<Navigate to="/alert-settings?view=tickets" replace />} />
<Route path="backup" element={<Navigate to="/settings/data" replace />} />
<Route path="logs" element={<Navigate to="/settings/data?view=logs" replace />} />
<Route path="cache" element={<Navigate to="/settings/data?view=cache" replace />} />
+1 -1
View File
@@ -3156,7 +3156,7 @@ export interface DockhandVulnerabilityEnvironment {
}
export interface DockhandVulnerabilitiesResponse {
/** Whether critical findings are alerted (Settings → Notifications). */
/** Whether critical findings are alerted (Alert settings → Notifications). */
alerts: boolean;
/** A host with at least this many high findings is alerted too; 0 = never. */
highThreshold: number;
+1 -1
View File
@@ -77,7 +77,7 @@ export default function AlertNoise() {
<>
<p className="text-secondary">
The alerts that keep coming back. Each time one is raised and later goes away counts once; only those raised at least twice are listed. The last column says what usually helps — a longer grace time, a different threshold,
or fixing the thing itself. An alert raised and cleared often enough in an hour is called <strong>flapping</strong> and its messages are held back until it has been calm for an hour (set under Settings → Notifications).
or fixing the thing itself. An alert raised and cleared often enough in an hour is called <strong>flapping</strong> and its messages are held back until it has been calm for an hour (set under Alert settings → Notifications).
</p>
<ErrorNotice error={error} />
+1 -1
View File
@@ -206,7 +206,7 @@ export default function DatabaseBackups() {
Save
</button>
</div>
<div className="col-12 text-secondary small">The time is in the notification time zone (Settings → Notifications). An alert is raised if the newest backup gets more than two days old.</div>
<div className="col-12 text-secondary small">The time is in the notification time zone (Alert settings → Notifications). An alert is raised if the newest backup gets more than two days old.</div>
</form>
{data.snapshots.length === 0 ? (
+2 -2
View File
@@ -16,7 +16,7 @@ const FIELDS: { key: Field; label: string; unit: string; min: number; max: numbe
const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0);
/**
* This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every
* This server's own alert limits. Anything left empty follows Alert settings → Notifications, so changing the global limit still reaches every
* server that hasn't said otherwise. A disk can have a limit of its own, or be left alone altogether. Everyone can see what is set; only
* an administrator changes it.
*/
@@ -115,7 +115,7 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC
}}
>
<div className="card-body">
<p className="text-secondary small">Leave a box empty to follow Settings → Notifications (shown inside the box). Only what you fill in is different for this server.</p>
<p className="text-secondary small">Leave a box empty to follow Alert settings → Notifications (shown inside the box). Only what you fill in is different for this server.</p>
<div className="row g-3">
{FIELDS.filter((f) => !f.onlyForGuests || isGuest || own?.[f.key] !== undefined).map((f) => (
<div className="col-md-6" key={f.key}>
+1 -1
View File
@@ -389,7 +389,7 @@ export default function AppShell({ user, children }: { user: CurrentUser; childr
{away.away.state === "planned" ? `Away from ${formatDateTime(new Date(away.away.from!))} until ${formatDateTime(new Date(away.away.until))}` : `Away until ${formatDateTime(new Date(away.away.until))}`}
{away.away.note ? ` — ${away.away.note}` : ""}.{" "}
{away.away.state === "active" ? `Notifications are held${away.away.criticalThrough ? " (critical alerts still come through)" : ""}${away.held > 0 ? ` — ${away.held} so far` : ""}. ` : ""}
<Link to="/settings/notifications?tab=when">{user.role === "admin" ? "Manage" : "Details"}</Link>
<Link to="/alert-settings?view=notifications&tab=when">{user.role === "admin" ? "Manage" : "Details"}</Link>
</div>
</div>
)}
+1 -3
View File
@@ -8,7 +8,6 @@ import {
IconArrowsSplit2,
IconHome2,
IconBatteryCharging,
IconTargetArrow,
IconDatabaseExport,
IconAdjustmentsAlt,
IconChecklist,
@@ -151,14 +150,13 @@ export const NAV: NavEntry[] = [
icon: <IconHeartbeat size={20} />,
items: [
{ to: "/endpoints", label: "Endpoints", icon: <IconHeartbeat size={20} /> },
{ to: "/alert-rules", label: "Alert rules", icon: <IconAdjustmentsAlt size={20} /> },
{ to: "/alert-settings", label: "Alert settings", icon: <IconAdjustmentsAlt size={20} /> },
{ to: "/heartbeats", label: "Heartbeats", icon: <IconHeartRateMonitor size={20} /> },
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
{ to: "/ups", label: "UPS", icon: <IconBatteryCharging size={20} /> },
{ to: "/prometheus", label: "Prometheus", icon: <IconFlame size={20} /> },
{ to: "/grafana", label: "Grafana", icon: <IconChartAreaLine size={20} /> },
{ to: "/speedtest", label: "Speedtest", icon: <IconGauge size={20} /> },
{ to: "/service-levels", label: "Service levels", icon: <IconTargetArrow size={20} /> },
],
},
{
+2 -2
View File
@@ -221,7 +221,7 @@ function RuleDrawer({ initial, servers, tags, onClose, onSaved }: { initial: Ale
</select>
)}
<div className="text-secondary small mb-3">
Severity and kind decide who hears of it: these are <strong>Monitoring</strong> alerts, so the routing rules under Settings → Notifications apply to them like any other.
Severity and kind decide who hears of it: these are <strong>Monitoring</strong> alerts, so the routing rules under Alert settings → Notifications apply to them like any other.
</div>
{tried && (
<div className={`alert ${tried.length > 0 ? "alert-warning" : "alert-success"} py-2`}>
@@ -298,7 +298,7 @@ export default function AlertRules({ user }: { user: CurrentUser }) {
return (
<>
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
<h2 className="page-title mb-0">Alert rules</h2>
<h3 className="mb-0">Alert rules</h3>
{canEdit && (
<div className="ms-auto">
<button className="btn btn-primary" onClick={() => setEditing({ name: "", metric: "cpu", threshold: 90, forMinutes: 15, severity: "warning" })}>
+46
View File
@@ -0,0 +1,46 @@
import { Navigate, useLocation, useSearchParams } from "react-router-dom";
import type { CurrentUser } from "../api/client";
import ViewTabs from "../components/ViewTabs";
import AlertRules from "./AlertRules";
import ServiceLevels from "./ServiceLevels";
import NotificationSettings from "./settings/NotificationSettings";
import AlertTicketSettings from "./settings/AlertTicketSettings";
type View = "rules" | "levels" | "notifications" | "tickets";
/**
* Everything that decides what is an alert and who hears of it, on one page with a tab for each: **Alert rules** (limits of your own), **Service levels**
* (targets and error budgets), **Notifications** (channels, what is sent, who hears what, quiet hours, away mode) and **Alert tickets** (osTicket). The first two
* are for everyone to read; the last two are Settings, so only an administrator sees them. The tab is `?view=`; Notifications keeps its own `?tab=` and `?section=`.
*/
export default function AlertSettings({ user }: { user: CurrentUser }) {
const [params] = useSearchParams();
const isAdmin = user.role === "admin";
const asked = params.get("view");
const view: View = asked === "levels" ? "levels" : isAdmin && (asked === "notifications" || asked === "tickets") ? asked : "rules";
const tabs = [
{ to: "/alert-settings", label: "Alert rules", active: view === "rules" },
{ to: "/alert-settings?view=levels", label: "Service levels", active: view === "levels" },
...(isAdmin
? [
{ to: "/alert-settings?view=notifications", label: "Notifications", active: view === "notifications" },
{ to: "/alert-settings?view=tickets", label: "Alert tickets", active: view === "tickets" },
]
: []),
];
return (
<>
<h2 className="page-title mb-3">Alert settings</h2>
<ViewTabs label="Alert settings" tabs={tabs} />
{view === "levels" ? <ServiceLevels user={user} /> : view === "notifications" ? <NotificationSettings /> : view === "tickets" ? <AlertTicketSettings /> : <AlertRules user={user} />}
</>
);
}
/** `/settings/notifications` (and its `?tab=`, `?section=`, `?view=tickets`) as it was before the alert settings were one page: the same place on the new one. */
export function NotificationsRedirect() {
const { search } = useLocation();
const next = new URLSearchParams(search);
next.set("view", next.get("view") === "tickets" ? "tickets" : "notifications");
return <Navigate to={`/alert-settings?${next.toString()}`} replace />;
}
+1 -1
View File
@@ -379,7 +379,7 @@ export default function Backups({ user, tabs }: { user?: CurrentUser; /** The ta
</div>
<div className="card-body text-secondary small pb-0">
A backup nobody has restored is a hope, not a backup. Record each time you try one — for a guest above, or for anything else (a share, a database). A failed test is raised on the
Alerts page; under Settings → Notifications you can also ask to be reminded when a guest with backups hasn&rsquo;t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
Alerts page; under Alert settings → Notifications you can also ask to be reminded when a guest with backups hasn&rsquo;t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
</div>
{custom.length > 0 && (
<div className="card-body pb-0">
+1 -1
View File
@@ -753,7 +753,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
{ id: "server", title: "Add a server and install its agent", description: "The agent reports scheduled tasks, disks, memory and open ports, and lets the app tell when a server goes quiet.", to: "/servers", action: "Add a server", done: (serverSummary?.total ?? 0) > 0 },
{ id: "dns", title: "Connect a DNS provider", description: "See your zones and records, and have names matched to servers.", to: "/dns", action: "Connect", done: (dnsStats?.totalProviders ?? 0) > 0 },
{ id: "secret", title: "Track something that expires", description: "An API token, a certificate or a password — you're warned before it runs out.", to: "/secrets", action: "Add one", done: (secrets?.length ?? 0) > 0 },
{ id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/settings/notifications", action: "Set up", done: notifyConfigured === true },
{ id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/alert-settings?view=notifications", action: "Set up", done: notifyConfigured === true },
{ id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 },
];
+3 -3
View File
@@ -121,15 +121,15 @@ export default function DockerVulnerabilities({ integrationId }: { integrationId
<strong>Critical</strong> findings are an alert — see <Link to="/alerts">Alerts</Link>;{" "}
{data.highThreshold > 0 ? (
<>
so is a host with <strong>{data.highThreshold} or more high</strong> ones (<Link to="/settings/notifications">Settings → Notifications</Link>); medium and low are only shown here.
so is a host with <strong>{data.highThreshold} or more high</strong> ones (<Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>); medium and low are only shown here.
</>
) : (
<>high, medium and low are only shown here (high ones can be alerted from a number you choose under Settings → Notifications).</>
<>high, medium and low are only shown here (high ones can be alerted from a number you choose under Alert settings → Notifications).</>
)}
</>
) : (
<>
The alert for critical findings is switched off (<Link to="/settings/notifications">Settings → Notifications</Link>).
The alert for critical findings is switched off (<Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>).
</>
)}
</p>
+2 -2
View File
@@ -45,7 +45,7 @@ function ServiceLevelsSection({ rows, partial }: { rows: ServiceLevelResult[]; p
<div className="card-body">
<p>
<strong>{metCount}</strong> of {rows.length} {rows.length === 1 ? "target was" : "targets were"} met{partial ? " so far" : ""}. Each is judged on the downtime it allowed over the whole stretch (the error budget).{" "}
<Link to="/service-levels">Set targets</Link>
<Link to="/alert-settings?view=levels">Set targets</Link>
</p>
<div className="table-responsive">
<table className="table table-vcenter table-sm">
@@ -201,7 +201,7 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
{user.role === "admin" ? (
<>
{" "}
It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: <Link to="/settings/notifications">Settings → Notifications</Link>.
It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: <Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>.
</>
) : null}
</p>
+1 -1
View File
@@ -229,7 +229,7 @@ export default function Privacy() {
you configured, to read status and, when someone with permission asks, to act (start a VM, run a template…).
</li>
<li className="mb-2">
<strong>osTicket tickets</strong> — only if an admin has switched on Settings → Notifications → Alert tickets: for a problem that has lasted, the text of the alert
<strong>osTicket tickets</strong> — only if an admin has switched on Alert settings → Alert tickets: for a problem that has lasted, the text of the alert
(which server or integration, what is wrong), a link back to this app, and the requester name and email set there go to your osTicket's API.
</li>
<li className="mb-2">
+1 -1
View File
@@ -185,7 +185,7 @@ export default function ServiceLevels({ user }: { user: CurrentUser }) {
return (
<>
<div className="d-flex align-items-center mb-1">
<h2 className="page-title mb-0">Service levels</h2>
<h3 className="mb-0">Service levels</h3>
{isAdmin && !draft && (
<button className="btn btn-primary ms-auto" onClick={() => setDraft({ ...EMPTY })}>
Set a target
-1
View File
@@ -3,7 +3,6 @@ import { Link, NavLink, Outlet } from "react-router-dom";
import { searchSettings } from "../utils/settingsIndex";
const SUB_NAV = [
{ to: "/settings/notifications", label: "Notifications" },
{ to: "/settings/sign-in", label: "Sign-in" },
{ to: "/settings/display", label: "Display" },
{ to: "/settings/names", label: "Names" },
+1 -21
View File
@@ -1,7 +1,5 @@
import { useSearchParams } from "react-router-dom";
import ViewTabs from "../../components/ViewTabs";
import NotificationSettings from "./NotificationSettings";
import AlertTicketSettings from "./AlertTicketSettings";
import DisplaySettings from "./DisplaySettings";
import BadgeSettings from "./BadgeSettings";
import BackupSettings from "./BackupSettings";
@@ -9,27 +7,9 @@ import LogSettings from "./LogSettings";
import CacheSettings from "./CacheSettings";
import StorageSettings from "./StorageSettings";
// Settings used to have twelve tabs of its own. Three of them gather the ones that belong together; each part is the same page it always was,
// Settings used to have twelve tabs of its own. Two of them gather the ones that belong together; each part is the same page it always was,
// switched by `?view=` (so it can be linked to, and the old addresses redirect here).
/** Notifications, and — beside it — opening an osTicket ticket for an alert. */
export function NotificationsGroup() {
const [params] = useSearchParams();
const view = params.get("view") === "tickets" ? "tickets" : "notifications";
return (
<>
<ViewTabs
label="Notification settings"
tabs={[
{ to: "/settings/notifications", label: "Notifications", active: view === "notifications" },
{ to: "/settings/notifications?view=tickets", label: "Alert tickets", active: view === "tickets" },
]}
/>
{view === "tickets" ? <AlertTicketSettings /> : <NotificationSettings />}
</>
);
}
/** How things look: dates, times and table sizes, and the colours of the badges. */
export function DisplayGroup() {
const [params] = useSearchParams();
+26 -26
View File
@@ -11,35 +11,35 @@ export interface SettingsEntry {
words: string;
}
const N = "/settings/notifications";
const N = "/alert-settings?view=notifications";
export const SETTINGS_INDEX: SettingsEntry[] = [
// Notifications
{ title: "Gotify, ntfy, email and webhook channels", where: "Notifications › Channels", to: `${N}?tab=channels`, words: "gotify ntfy smtp email mail webhook push channel test send where notifications go" },
{ title: "Browser and phone push", where: "Notifications › Channels", to: `${N}?tab=channels`, words: "web push browser phone device subscribe pwa notify vapid" },
{ title: "Server offline after (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
{ title: "Disk usage alert at (percent)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk full storage percent threshold space" },
{ title: "Warn when a disk will fill within (days)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk forecast fill growth trend days" },
{ title: "Away mode: hold notifications while I'm away", where: "Notifications › When", to: `${N}?tab=when`, words: "away holiday vacation travel back hold digest pause mute do not disturb critical" },
{ title: "Let critical alerts through quiet hours", where: "Notifications › When", to: `${N}?tab=when`, words: "quiet hours critical night wake sleep through urgent" },
{ title: "Quarterly and annual reports", where: "Notifications › When", to: `${N}?tab=when`, words: "report quarterly annual yearly quarter year send message monthly" },
{ title: "Warn about critical vulnerabilities in Docker images", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events&section=backups`, words: "vulnerability vulnerabilities cve security scan critical dockhand grype trivy image docker" },
{ title: "Warn when a Docker container is broken, and containers to leave alone", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events&section=backups`, words: "docker container restarting restart loop unhealthy stopped exited created dockhand hawser ignore leave alone" },
{ title: "Warn when CPU or memory is unusually high for the time", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
{ title: "Warn when a server starts listening on a new port", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "new port baseline listening tcp monitoring" },
{ title: "Call an alert flapping after (raises in an hour)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
{ title: "Alert when a network device is offline for (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "unifi network device switch access point gateway offline" },
{ title: "Secret, Tailscale key and domain expiry reminders", where: "Notifications › What to send › Secrets, keys and domains", to: `${N}?tab=events&section=secrets`, words: "expiry expire secrets certificate tailscale key domain renew warn days" },
{ title: "Backups too old, and restore tests", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events&section=backups`, words: "backup stale old restore test guest proxmox updates docker image" },
{ title: "Integration, automation and ticket alerts", where: "Notifications › What to send › Integrations and automation", to: `${N}?tab=events&section=integrations`, words: "integration failing semaphore gitea automation failed osticket tickets uptime kuma" },
{ title: "DNS change notifications", where: "Notifications › What to send › DNS changes", to: `${N}?tab=events&section=dns`, words: "dns changes record added removed" },
{ title: "Routing rules: which channel gets which alert", where: "Notifications › Who hears what", to: `${N}?tab=routing`, words: "routing rules route channel tags severity kinds mute" },
{ title: "Escalation: tell more people if nobody acts", where: "Notifications › Who hears what", to: `${N}?tab=routing`, words: "escalation escalate unacknowledged steps neglected" },
{ title: "Time zone and daily reminders", where: "Notifications › When", to: `${N}?tab=when`, words: "timezone time zone daily reminder time of day" },
{ title: "Quiet hours", where: "Notifications › When", to: `${N}?tab=when`, words: "quiet hours night hold queue do not disturb" },
{ title: "Monthly report", where: "Notifications › When", to: `${N}?tab=when`, words: "monthly report day of month send availability" },
{ title: "Weekly digest", where: "Notifications › When", to: `${N}?tab=when`, words: "weekly digest summary email day" },
{ title: "Open osTicket tickets from alerts", where: "Notifications › Alert tickets", to: `${N}?view=tickets`, words: "osticket ticket alert open api key which problems" },
{ title: "Gotify, ntfy, email and webhook channels", where: "Alert settings › Notifications › Channels", to: `${N}&tab=channels`, words: "gotify ntfy smtp email mail webhook push channel test send where notifications go" },
{ title: "Browser and phone push", where: "Alert settings › Notifications › Channels", to: `${N}&tab=channels`, words: "web push browser phone device subscribe pwa notify vapid" },
{ title: "Server offline after (minutes)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
{ title: "Disk usage alert at (percent)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "disk full storage percent threshold space" },
{ title: "Warn when a disk will fill within (days)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "disk forecast fill growth trend days" },
{ title: "Away mode: hold notifications while I'm away", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "away holiday vacation travel back hold digest pause mute do not disturb critical" },
{ title: "Let critical alerts through quiet hours", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "quiet hours critical night wake sleep through urgent" },
{ title: "Quarterly and annual reports", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "report quarterly annual yearly quarter year send message monthly" },
{ title: "Warn about critical vulnerabilities in Docker images", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events&section=backups`, words: "vulnerability vulnerabilities cve security scan critical dockhand grype trivy image docker" },
{ title: "Warn when a Docker container is broken, and containers to leave alone", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events&section=backups`, words: "docker container restarting restart loop unhealthy stopped exited created dockhand hawser ignore leave alone" },
{ title: "Warn when CPU or memory is unusually high for the time", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
{ title: "Warn when a server starts listening on a new port", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "new port baseline listening tcp monitoring" },
{ title: "Call an alert flapping after (raises in an hour)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
{ title: "Alert when a network device is offline for (minutes)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "unifi network device switch access point gateway offline" },
{ title: "Secret, Tailscale key and domain expiry reminders", where: "Alert settings › Notifications › What to send › Secrets, keys and domains", to: `${N}&tab=events&section=secrets`, words: "expiry expire secrets certificate tailscale key domain renew warn days" },
{ title: "Backups too old, and restore tests", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events&section=backups`, words: "backup stale old restore test guest proxmox updates docker image" },
{ title: "Integration, automation and ticket alerts", where: "Alert settings › Notifications › What to send › Integrations and automation", to: `${N}&tab=events&section=integrations`, words: "integration failing semaphore gitea automation failed osticket tickets uptime kuma" },
{ title: "DNS change notifications", where: "Alert settings › Notifications › What to send › DNS changes", to: `${N}&tab=events&section=dns`, words: "dns changes record added removed" },
{ title: "Routing rules: which channel gets which alert", where: "Alert settings › Notifications › Who hears what", to: `${N}&tab=routing`, words: "routing rules route channel tags severity kinds mute" },
{ title: "Escalation: tell more people if nobody acts", where: "Alert settings › Notifications › Who hears what", to: `${N}&tab=routing`, words: "escalation escalate unacknowledged steps neglected" },
{ title: "Time zone and daily reminders", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "timezone time zone daily reminder time of day" },
{ title: "Quiet hours", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "quiet hours night hold queue do not disturb" },
{ title: "Monthly report", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "monthly report day of month send availability" },
{ title: "Weekly digest", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "weekly digest summary email day" },
{ title: "Open osTicket tickets from alerts", where: "Alert settings › Alert tickets", to: "/alert-settings?view=tickets", words: "osticket ticket alert open api key which problems" },
// Sign-in
{ title: "Roles from groups", where: "Sign-in", to: "/settings/sign-in", words: "roles groups authentik oidc admin operator viewer claim map" },
{ title: "Profile pictures (Gravatar)", where: "Sign-in", to: "/settings/sign-in", words: "profile picture avatar gravatar photo identity provider" },