diff --git a/AGENT_UPDATE.md b/AGENT_UPDATE.md index 7c552c5..98dac6e 100644 --- a/AGENT_UPDATE.md +++ b/AGENT_UPDATE.md @@ -254,7 +254,7 @@ Each agent says which operating system it runs, in the report's `system.os`: `{ - **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart. The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows -client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Settings → Notifications (*Flag an +client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Alert settings → Notifications (*Flag an operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent older than this version reports no OS and the server shows nothing for it. diff --git a/BACKUP.md b/BACKUP.md index 56379ba..e143fe7 100644 --- a/BACKUP.md +++ b/BACKUP.md @@ -256,7 +256,7 @@ was taken (from your `.env` backup). Symptoms: an integration or a DNS provider fails with *"Unsupported state or unable to authenticate data"*, and notification channels that worked before don't. The database is fine; only the stored credentials can't be read. If you can't find the old key, enter the credentials -again (Integrations → edit, DNS providers, Settings → Notifications). Nothing else is lost. +again (Integrations → edit, DNS providers, Alert settings → Notifications). Nothing else is lost. **Don't** change `CREDENTIALS_ENCRYPTION_KEY` on a running system to "fix" this: it makes everything stored so far unreadable. diff --git a/DATABASE.md b/DATABASE.md index cc79d83..9bb608c 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -1207,7 +1207,7 @@ How many were streaming, and how many of those were transcoded, at a moment: one #### `service_levels` -A service-level target (Monitoring → Service levels): what a thing is held to, measured over calendar months. No foreign keys: the thing measured is a plain reference, and a target whose subject is removed is shown as having nothing to measure. +A service-level target (Alert settings → Service levels): what a thing is held to, measured over calendar months. No foreign keys: the thing measured is a plain reference, and a target whose subject is removed is shown as having nothing to measure. | Column | Type | Notes | |---|---|---| @@ -1268,7 +1268,7 @@ Each report of a backup, for its history: 400 days and at most 400 per job. `bac #### `alert_rules` -Limits of the administrator's own (Monitoring → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule::server:[:]`. +Limits of the administrator's own (Alert settings → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule::server:[:]`. | Column | Type | Notes | |---|---|---| @@ -1562,7 +1562,7 @@ that were never changed aren't stored; the app fills in defaults when reading. | `display` | `dateFormat`, `timeFormat`, `pageSize`. | | `providerColors`, `integrationColors` | Badge colours, by provider / integration type. | | `consistency` | `excludedRanges` — addresses the Consistency and IP Addresses pages ignore. Default `["172.16.0.0/12"]` (Docker's networks). | -| `alertTickets` | Opening osTicket tickets for alerts: `enabled`, `integrationId`, `severities`, `categories`, `requesterName`/`requesterEmail`, `topicId`, `priorityId`, `notifyStaff`, `existingProblems`, `confirmChecks`, `summaryAbove`, `reopenAfterMinutes`. Edited under Settings → Notifications → Alert tickets. The API address and key are on the osTicket integration, not here. | +| `alertTickets` | Opening osTicket tickets for alerts: `enabled`, `integrationId`, `severities`, `categories`, `requesterName`/`requesterEmail`, `topicId`, `priorityId`, `notifyStaff`, `existingProblems`, `confirmChecks`, `summaryAbove`, `reopenAfterMinutes`. Edited under Alert settings → Alert tickets. The API address and key are on the osTicket integration, not here. | | `nameGenerator` | `themes` — the Generator's server-name lists: `[{ "id", "label", "names": [...], "lastImport"? }]`. Edited under Settings → Names. | Rows whose key starts with **`_internal:`** are scheduler bookkeeping, not diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md index 85049dd..bace810 100644 --- a/INTEGRATIONS.md +++ b/INTEGRATIONS.md @@ -70,7 +70,7 @@ It is for the readers written from documentation (see "Not verified" on each int - **Container health:** on the 15-minute health pass the app also reads every container's state and status text and raises alerts for what is broken — see [Docker containers that are broken](docs/MONITORING.md#docker-containers-that-are-broken). A Docker host that Dockhand cannot read (its Hawser agent is down) is an alert too, and its containers' alerts are held, not cleared. The Docker page shows each container's problem and has a **Leave alone** button. - **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host — and so are **high** ones once a host has as many as a number you set (*Also warn when a Docker host has this many high vulnerabilities*, off by default) — (a setting turns the critical one off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*. -- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged. +- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Alert settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged. ### Uptime Kuma @@ -169,7 +169,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot), and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed - verification or a datastore couldn't be read at all — the "Notify on" section in Settings → Notifications has a + verification or a datastore couldn't be read at all — the "Notify on" section in Alert settings → Notifications has a dedicated toggle for it, sharing the same daily reminder time as the other daily checks. - **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the `PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run @@ -220,7 +220,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi a field looks wrong, tell us what happened and we'll adjust — this one has had less real-world exposure than every other integration listed here. -**Optional: opening tickets for alerts** (Settings → Notifications → Alert tickets) +**Optional: opening tickets for alerts** (Alert settings → Alert tickets) - **Config fields:** Ticket API address (optional), Ticket API key (optional, stored encrypted like the database password), Allow self-signed certificate (ticket API). Leave them empty and the integration works as before. @@ -229,7 +229,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi app's requests come from (behind Docker or a reverse proxy that is the host's or proxy's address, not the container's — check osTicket's log if it refuses) and tick **Can create tickets**. osTicket refuses a key from any other address, which is the most common reason a test ticket fails. -- **What is sent:** one ticket per problem — name and email you set under Settings → Notifications → Alert tickets, a subject, the +- **What is sent:** one ticket per problem — name and email you set under Alert settings → Alert tickets, a subject, the problem's text and a link back to this app, optionally a help topic and priority id — with `autorespond` off (the "user" is this app, so an acknowledgement would only bounce) and osTicket's own staff alert on or off as you choose. - **What the API can't do:** close, reopen, or comment on a ticket — osTicket's API has no such endpoint. Closing one is @@ -263,7 +263,7 @@ osTicket's API cannot close a ticket, so this is the one place in the app that * *Closed from Homelab Manager by * with the reason you gave; and a **"closed" event** is recorded so the ticket's history reads like any other. Nothing else is touched, and nothing is ever deleted. - **What it checks first:** that the tables and columns it writes to are there (written for **osTicket 1.14–1.18**); a different osTicket is refused with the list of what is missing, before anything is written. It closes only a ticket that is **open**, and only if its status is still what it was when it was read — a ticket somebody else closes at the same moment is left as they left it. A ticket that was opened as part of a **summary ticket** (many problems at once) is not closed from here, because it covers other alerts. -- **Test closing** (Settings → Notifications → Alert tickets, next to *Send a test ticket*; administrators): signs in with the login, checks the tables and columns, and then **tries every write a close makes inside a transaction that is rolled back** — so a missing grant is found there and not on the first real ticket. Nothing stays (a counter or two +- **Test closing** (Alert settings → Alert tickets, next to *Send a test ticket*; administrators): signs in with the login, checks the tables and columns, and then **tries every write a close makes inside a transaction that is rolled back** — so a missing grant is found there and not on the first real ticket. Nothing stays (a counter or two inside the database moves on). - **Closing several alerts at once:** pick them on the Alerts page and choose *Close*; when some of them have a ticket that can be closed from here, the drawer offers *Also close N osTicket tickets* (with their numbers), and it is closed one ticket after another, each written on its alert's incident and in the audit log as above. One that cannot be closed does not stop the others: the alerts are closed all the same and the message says how many tickets were closed, were already closed, or could not be (with the first reason). Summary tickets and an osTicket with no closing login are counted and left open. **Undo opens the alerts again but cannot reopen the tickets** — the message says so. - **What the alert shows:** the ticket badge says *(closed)* once it has been closed from here, and the Close drawer says so instead of offering it again. If the ticket cannot be closed, the alert is **still closed** and you are told why; the reason is also written on the incident and in the audit log (`alert_ticket/close_ticket`, `close_ticket_failed`). @@ -305,7 +305,7 @@ osTicket's API cannot close a ticket, so this is the one place in the app that * It all appears on **Network → Network devices**. - A device is a **node in the dependency map**, like a server: the *connected through* link is made into a dependency by itself (an access point needs its switch, the switch needs the gateway), and you can write others by hand (a server needs the switch it is plugged into). - - A device the controller has not seen online for longer than **Settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what + - A device the controller has not seen online for longer than **Alert settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what it is connected through if that is down too, is folded under it, like a guest under its host. It can be silenced with a maintenance window, and has runbooks like any target. - **Servers plugged in:** the controller's list of what is connected is matched to servers by IP address (exact, and only when one server has that address). A server is then *connected through* the switch or access point it is on — a dependency made by itself, shown on the server's page and on the Connections map. A key or account that may not read clients costs only this; devices, networks and Wi-Fi names are unaffected. diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index 9625229..1c0dc2d 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -2,7 +2,7 @@ Every notification in Homelab Manager is sent through the same pipe (`server/src/services/notify.ts`) to whichever channels are enabled under -**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic +**Alert settings → Notifications**: Gotify, ntfy, SMTP (email), a generic JSON webhook, and push to phones and browsers. All five fire for every notification below unless a **routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the daily ones, one shared time/timezone. @@ -13,12 +13,12 @@ is closed with changes unsaved): **Channels** (the five channels, each a row you open to fill in and send a test), **What to send** (the event toggles grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When** (the time zone, the daily reminder time, quiet hours and the weekly digest). -The address keeps the tab, e.g. `/settings/notifications?tab=when`. +The address keeps the tab, e.g. `/alert-settings?view=notifications&tab=when`. Two settings apply to *every* notification regardless of what triggered it: -- **Quiet hours** (Settings → Notifications → Quiet hours): while +- **Quiet hours** (Alert settings → Notifications → Quiet hours): while enabled and inside the configured window, a notification is held instead of sent immediately, then delivered as a single digest at the window's end time. This matters most for the real-time alerts below — @@ -26,7 +26,7 @@ it: outside the window anyway. *Still send critical alerts* (off by default) lets a **critical** one — a server down, a disk full, mains power lost — through at once, so only the warnings wait for the morning. -- **Away** (Settings → Notifications → When, and a banner on every page while +- **Away** (Alert settings → Notifications → When, and a banner on every page while it is on): "I'm away until Sunday". See [Away](#away). - **Maintenance windows** (the Maintenance page): opening one for a server, integration, or DNS provider silences the alerts that target @@ -40,7 +40,7 @@ below; several are fixed and can't be changed from the UI. ## Who hears what -Under **Settings → Notifications → Who hears what**. +Under **Alert settings → Notifications → Who hears what**. **Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of: the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how @@ -80,7 +80,7 @@ notification of its own — it is something to look at when you are looking at b The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under -Settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then +Alert settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then each person opens the same row and presses **Notify this browser**; the browser asks permission and the row lists their browsers, with a button to send a test and one to remove each. @@ -98,7 +98,7 @@ with a button to send a test and one to remove each. ## Daily reminders -These six checks share one schedule: **Settings → Notifications → Daily +These six checks share one schedule: **Alert settings → Notifications → Daily reminder time** (default 08:00) and **Timezone** (default UTC). Unlike the state-based alerts further down, these **re-send every day the condition is still true** — there's no "only once" de-duplication, so an @@ -112,12 +112,12 @@ until the next scheduled time to catch up. | Check | "Notify on" toggle | Fires when | Configured at | |---|---|---|---| | Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret | -| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) | +| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Alert settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) | | Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days | | Docker containers broken | *Warn when a Docker container is broken* | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (*Server offline, disk nearly full …*, `healthAlerts`), with a recovery message when it runs again. See [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken). | Its own switch; *Containers to leave alone* | | Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) | -| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page | -| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send | +| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Alert settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page | +| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Alert settings → Notifications → What to send | | Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable | | Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable | @@ -137,9 +137,9 @@ starting baseline rather than announced all at once. | Check | "Notify on" toggle | Fires when | Configured at | |---|---|---|---| -| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) | +| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Alert settings → Notifications → Health checks → **Server offline after** (default 60 min) | | Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. | -| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) | +| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Alert settings → Notifications → Health checks → **Disk usage alert at** (default 90%) | | Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable | | Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable | @@ -153,7 +153,7 @@ schedule. | DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). | | DNS record updated | *DNS record updated* | An existing DNS record is edited. | | DNS record deleted | *DNS record deleted* | A DNS record is deleted. | -| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Settings → Notifications → **Alert after** (default 3 consecutive failures) | +| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Alert settings → Notifications → **Alert after** (default 3 consecutive failures) | | Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. | ## Digest @@ -170,7 +170,7 @@ A scheduled backup of the app's own database that fails sends one message throug | Notification | Fires when | |---|---| -| "Homelab monthly report — " | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. | +| "Homelab monthly report — " | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Alert settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. | Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page. @@ -180,20 +180,20 @@ the December one. Both are the same report over three or twelve months, sent the ## Service levels -A service-level target (Monitoring → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used +A service-level target (Alert settings → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used (a warning: "Gitea: 90% of the error budget for its 99.9% target is used up for October 2026 — down 40 min of 45 min allowed, 5 min left") and again when it is **overspent** (critical: "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed"). It is *Monitoring* for routing, and it ends when the month turns over. Maintenance does not hold it back: planned downtime counts. Each target has its own switch. ## Network devices -A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under +A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Alert settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under *Server offline, disk nearly full …* (`healthAlerts`): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as *Server down*; maintenance on the device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be read for twenty, its devices are held — neither announced nor cleared. ## Your own alert rules, and snoozing -An alert rule (Monitoring → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held +An alert rule (Alert settings → Alert rules) that is past its limit is announced like any other health alert (*Health Alert*, with a recovery message), counted as **Monitoring** for routing with the rule's own severity, and held back by maintenance on the server. It is checked on the health pass, every 15 minutes. An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or recovered by the health and automation checks until the snooze ends. When it ends and the alert is still there, one **"Snooze ended"** message @@ -201,7 +201,7 @@ An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or r ## Buttons on notifications -Switched on under **Settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze +Switched on under **Alert settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze that ended, and the escalation reminder — can be acted on from the message: | Channel | One alert | Several | @@ -217,7 +217,7 @@ it came from. **Anyone who can read the notification can use it** for those 24 h ## Away -For a stretch rather than every night. **Settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a +For a stretch rather than every night. **Alert settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a holiday ahead), a note to yourself and whether to **still send critical alerts**. While it lasts: - notifications are **held** — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held; @@ -232,7 +232,7 @@ administrator's to set (it decides who is told what), everyone sees the banner, ## Flapping alerts -A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*: +A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Alert settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*: - one message, **"Flapping alert"**, says "↻ — raised 3 times within an hour"; - its raise and recovery messages are then **held back** (the alert stays on the Alerts page with a *flapping* badge and still counts for the menu badge and /metrics); @@ -250,13 +250,13 @@ carries the tags of the server, and maintenance on the server holds it back. A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under -Settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce. +Alert settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce. ## Weekly digest | Notification | Fires when | |---|---| -| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. | +| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Alert settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. | It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say: what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage); @@ -283,7 +283,7 @@ It runs the checks live when opened (a recent result is reused for a minute), an ## Tickets in osTicket -Not a notification, but it acts on the same alerts: with **Settings → Notifications → Alert tickets** switched on (off by default), a problem on the Alerts page opens a +Not a notification, but it acts on the same alerts: with **Alert settings → Alert tickets** switched on (off by default), a problem on the Alerts page opens a ticket in osTicket. It runs every 15 minutes, starting about a minute and a half after the app starts. It never closes a ticket by itself; closing is something you choose, in the Close drawer on the Alerts page, when the integration has a separate database login for it (see INTEGRATIONS.md). | Rule | What it means | @@ -323,7 +323,7 @@ the app alerts on something: other manual "Sync from X" action — these run when you click the button and report their result on the page itself, not via a notification. -- **Test buttons** (Settings → Notifications → each channel's "Send +- **Test buttons** (Alert settings → Notifications → each channel's "Send test") send a one-off message through that one channel only, to confirm it's wired up correctly — not a real event and not affected by quiet hours. diff --git a/README.md b/README.md index da05888..4d92870 100644 --- a/README.md +++ b/README.md @@ -187,6 +187,7 @@ All modules from the original plan are built: Backup page's job and run lists, the comparison table) have an Export CSV too, which writes the table as it is on screen — times as exact timestamps, action buttons left out. +- **Alert settings** (Monitoring → Alert settings) — everything that decides what is an alert and who hears of it, on one page with four tabs: *Alert rules*, *Service levels*, *Notifications* and *Alert tickets* (`/alert-settings?view=rules|levels|notifications|tickets`). Everyone signed in sees the first two; the last two are administrators' (they used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/notifications` still lead to the right tab). Maintenance stays on its own page. - **Settings** (admin-only) — notification channels (Gotify, ntfy, SMTP, generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS record added/updated/deleted, daily secret-expiry reminder and daily @@ -221,7 +222,7 @@ Server and storage health is watched every 15 minutes: a server whose agent stops reporting, a server disk / Proxmox storage / Synology volume passing a usage threshold, and a Synology volume or disk that's degraded or failing each raise one notification when the problem starts and one when it clears (both -thresholds are set under Settings → Notifications). Active problems are +thresholds are set under Alert settings → Notifications). Active problems are remembered across restarts, so a rebuild doesn't re-alert them. **Tags** — servers can be tagged (prod, media, rack-1, …) from their detail @@ -305,7 +306,7 @@ like `lab.example.se` resolves to the `example.se` registration that actually expires); others can be added by hand. Each is looked up daily over RDAP where the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap. -You're reminded daily from N days before expiry (Settings → Notifications, +You're reminded daily from N days before expiry (Alert settings → Notifications, default 30) until it's renewed, and told when an expiry date couldn't be refreshed for several days. Registries that don't publish an expiry (`.de`, `.eu`) can be tracked but have no date to warn about. Private zones (`.lan`, @@ -339,7 +340,7 @@ send the notifications — so the two can't disagree — but ignores the on/off toggles, since it's for looking at rather than being interrupted by. Problems under a maintenance window stay listed, marked silenced and counted separately. It also looks ahead: from an agent server's own history (see its Trends card), a -disk that is on course to be full within 14 days (Settings → Notifications → "Warn +disk that is on course to be full within 14 days (Alert settings → Notifications → "Warn when a disk will fill within"; 0 turns it off) is flagged with its rate of growth and the days left — critical when that is three days or less. It needs at least three days of steady growth to say anything, skips a disk that jumped once or @@ -354,7 +355,7 @@ It checks live (a recent result is reused for a minute; "Check now" forces a fresh one), and anything it couldn't read is called out at the top rather than quietly treated as fine. -**Alert tickets** (Settings → Notifications → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the +**Alert tickets** (Alert settings → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the problems on the Alerts page are looked at, and one that is new and has lasted two checks in a row gets one ticket — never a second for the same problem, and never one for something under a maintenance window or about osTicket itself. You choose which severities and kinds count (critical and warning by default), and the Alerts page shows the ticket number on each alert that has one. If many problems appear at once — a network drop that makes thirty @@ -455,7 +456,7 @@ alerts always stand alone. **Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups. -A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also +A **failed** test is always an alert (critical) until a newer one passes. Under Alert settings → Notifications → "A restore should have been tested within" you can also ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track; 0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is kept, `/metrics` has each subject's age and result, and everything is in the change log. @@ -474,7 +475,7 @@ so it is folded only when every enabled instance of that kind is explained. A se is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no alerts of their own, so they aren't part of this.) -**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with +**Who hears what** (Alert settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the @@ -762,7 +763,7 @@ alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in [docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md). -**Service levels** (Monitoring → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day +**Service levels** (Alert settings → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day month allows about 45 minutes; 99 % about 7 hours). A target is set on an **endpoint check** (measured from its answers, day by day), a **server** or an **integration** (measured from when the audit log recorded it going away and coming back, to the minute), and is measured over calendar months in the notification time zone. Each card shows the figure so far, the downtime used, what is allowed and what is left, how much of the budget is used (and where the pace so far would end the month), and how each of the last six months went. When **80 %** of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over — switchable per target. Planned maintenance is @@ -773,9 +774,9 @@ pace so far would end the month), and how each of the last six months went. When ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone; the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them, -and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December. +and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Alert settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December. -**Weekly digest** (Settings → Notifications, off until switched on) sends one message a +**Weekly digest** (Alert settings → Notifications, off until switched on) sends one message a week through your notification channels: what is wrong now, what was down and for how long, what happened, disks heading for full, things about to expire, updates waiting and maintenance coming up — each topic only if it has something to say, and "a quiet @@ -890,18 +891,18 @@ minimum in both themes. Motion is turned down for those who ask the system to. **New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up (where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on -it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there. +it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Alert settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there. **Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button; -the same list is under Settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md. +the same list is under Alert settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md. **What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey **usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least 30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by -maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*). +maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Alert settings → Notifications (*Servers and disks*). **A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too @@ -917,7 +918,7 @@ servers, alerts, incidents, inventory, network equipment, endpoint checks and ma gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab — its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under -its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance +its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Alert settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance (also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md. **Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then @@ -932,7 +933,7 @@ hours (maintenance), is handed to you or is still going is counted in the badge **A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits), switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section. -**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts +**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Alert settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts (a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md. @@ -942,7 +943,7 @@ all find something. Removing a dependency, an admin link, a runbook, a hand-made what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first. **Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same -alert has been **raised three times within an hour** (Settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says +alert has been **raised three times within an hour** (Alert settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics. Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks) still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log. @@ -959,7 +960,7 @@ what was there before (the earlier acknowledgement text, the earlier tags, the e **Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows -(client and server). A server whose system is past its end, or ends within **Settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once +(client and server). A server whose system is past its end, or ends within **Alert settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`. **Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a @@ -995,7 +996,7 @@ a battery to be replaced and a load of 90 % are warnings). The UPS is also a **n folded under its alert ("also takes down …") instead of each being announced; the page lists what is on it in the **order to shut it down**, so nothing is switched off while something that needs it is still running. The app only reads — it never sends a command to the UPS. Not tried against a real UPS; see INTEGRATIONS.md. -**Alert rules of your own.** **Monitoring → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*, +**Alert rules of your own.** **Alert settings → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*, *a disk above 80%* (every disk, or the one you name), *a disk warmer than 55 °C*. A rule says what to watch, above or below what, for how long (a reading in the history every 10–15 minutes; every reading in that time has to be past the limit, so a dip starts the count again), how serious it is and which servers it covers (all, one, or those with a tag). **Try it now** shows which servers it would be raised on right now before it is saved. What a rule finds is an ordinary alert on the Alerts page, in the *Monitoring* category with the rule's own severity — announced, recovered, held back while flapping, silenced by maintenance on the server and @@ -1005,7 +1006,7 @@ the disk's last report, so it is judged right now and has no time. **Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window -(Settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is +(Alert settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is `homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history. **Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under @@ -1024,7 +1025,7 @@ switch it is plugged into; something that needs it; a change that touches everyt *close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API. -**Buttons on notifications.** With **Settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app +**Buttons on notifications.** With **Alert settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as buttons — three on an ntfy message, two on a phone push (the browser's own notification), with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons; Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it diff --git a/ROLES.md b/ROLES.md index a5f2524..5bd028c 100644 --- a/ROLES.md +++ b/ROLES.md @@ -53,13 +53,12 @@ page's own top-level link. | Consistency | `/consistency` | ✅ | ✅ | ✅ | | **Monitoring** | | | | | | Endpoints | `/endpoints` | ✅ | ✅ | ✅ | -| Alert rules | `/alert-rules` | ✅ | ✅ | ✅ | +| Alert settings (tabs: Alert rules, Service levels; and for administrators only Notifications, Alert tickets) | `/alert-settings` | ✅ | ✅ | ✅ | | Heartbeats | `/heartbeats` | ✅ | ✅ | ✅ | | Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ | | UPS | `/ups` | ✅ | ✅ | ✅ | | Prometheus | `/prometheus` | ✅ | ✅ | ✅ | | Speedtest | `/speedtest` | ✅ | ✅ | ✅ | -| Service levels | `/service-levels` | ✅ | ✅ | ✅ | | **Automation** | | | | | | Semaphore | `/semaphore` | ✅ | ✅ | ✅ | | Gitea | `/gitea` | ✅ | ✅ | ✅ | @@ -126,7 +125,7 @@ even further, to admin only: - **Alert tickets**: everyone sees which alerts have an osTicket ticket (a badge on the Alerts page, linking to the ticket in osTicket); switching the feature on, choosing what opens a ticket, and sending a test ticket are - under Settings → Notifications → Alert tickets, which is admin-only. **Closing a ticket** along with an alert (the tick box in the Close drawer, for one alert or several picked together) is an operator's, like closing the alert; the login it writes with, and **Test closing**, are administrators'. + under Alert settings → Alert tickets, which is admin-only. **Closing a ticket** along with an alert (the tick box in the Close drawer, for one alert or several picked together) is an operator's, like closing the alert; the login it writes with, and **Test closing**, are administrators'. - **Runbooks**: everyone reads them (on a server's page, an integration's card, the *What to do* link beside an alert for a whole kind of alert, next to an alert); writing, editing and removing one needs operator. - **Inventory**: everyone can see the hardware (and print its labels); adding, editing and removing needs @@ -138,11 +137,11 @@ even further, to admin only: **Settings search** is part of Settings, which is administrators'. - **Network devices** (Network → Network devices): everyone can see the devices, networks and Wi-Fi names read from UniFi controllers; **Read now** and putting a device under maintenance need operator; adding the UniFi integration itself (a key or login) is an administrator's job, like any integration. -- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Settings → Notifications (administrators). +- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Alert settings → Notifications (administrators). - **Docker containers** (Docker page): everyone signed in sees each container and what is wrong with it; start, stop and restart need operator; **Leave alone** (no alerts for a container) and the list in Settings are administrators'. -- **Away mode** (Settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held. -- **Buttons on notifications** (Settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else. -- **Alert rules** (Monitoring → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is +- **Away mode** (Alert settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held. +- **Buttons on notifications** (Alert settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else. +- **Alert rules** (Alert settings → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is woken up). **Snooze** an alert, wake it, **close** it and open it again need operator, like acknowledging. The storage forecast beside Proxmox storages and Synology volumes is for everyone who can see those pages. - **What changed before this?** (on an incident's page, and `GET /api/v1/incidents/:id/before`): everyone signed in, like the incident record it is made from. - **Dependencies**: everyone can see what needs what (including what needs the Internet); adding and removing one needs operator. So does adding or hiding a *suggested* one. @@ -152,7 +151,7 @@ even further, to admin only: - **Proxy hosts** (Network → Proxy hosts): everyone signed in sees what Nginx Proxy Manager serves; **Read now** needs operator; adding the integration (its login) is an administrator's job, and so is **making, changing, switching off and deleting a proxy host** — only where the integration was given a user that may manage. - **Home Assistant** (Apps → Home Assistant): everyone signed in sees what is watched, its state and history; choosing what to watch, its limits, stopping and **Read now** need operator (the list of all entities is read live, so only operators see it); adding the integration (its token) is an administrator's job. -- **Service levels** (Monitoring → Service levels): everyone signed in sees the targets and what is left of the budget; setting, changing and removing one is administrator-only, and each is in the audit log. +- **Service levels** (Alert settings → Service levels): everyone signed in sees the targets and what is left of the budget; setting, changing and removing one is administrator-only, and each is in the audit log. - **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map. - **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone; *Check now* needs operator. @@ -175,7 +174,7 @@ even further, to admin only: - **Duplicati** (Backups → Duplicati): everyone signed in sees every backup and how it went; setting how often one is expected, muting it and forgetting it need operator; adding a machine (which makes an address that can write to this app) and removing one are administrators'. Reports arrive on a secret address with no sign-in: the token is the access. - **Restore tests**: everyone sees them on the Backup coverage page; recording one needs operator; removing one is administrator-only. - **Change log**: operators and administrators (it is made from the audit log, which has the same readers). -- **Push notifications**: an administrator switches the channel on (Settings → Notifications). Each person, whatever their role, adds +- **Push notifications**: an administrator switches the channel on (Alert settings → Notifications). Each person, whatever their role, adds and removes their own browsers and can send a test to them; an administrator can remove anyone's. - **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs are going; starting an update run (**Update…**) or stopping one needs operator. @@ -199,7 +198,8 @@ even further, to admin only: - **The delivery log of an event webhook** (what was sent, with the messages, and sending one again) and **Settings → Data → Storage** (how big the app's data is) are administrator-only; sending a message again is in the audit log. - **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login. -- Everything under **Settings** (notification channels, badge colors, +- **Alert settings**: the *Alert rules* and *Service levels* tabs are for everyone to read (changing them is administrators'); the *Notifications* and *Alert tickets* tabs are only shown to administrators, as they were under Settings. +- Everything under **Settings** (badge colors, display prefs, log retention, backup/restore) is admin-only, matching the page itself being admin-only. diff --git a/docs/MONITORING.md b/docs/MONITORING.md index ef8ff76..ec83c1b 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -72,7 +72,7 @@ Three ideas run through everything: | **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** | | **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due | | **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** | -| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications | +| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Alert settings → Notifications | | **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time | | **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass | | **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** | @@ -84,7 +84,7 @@ After the app **restarts**, server-based checks are paused for **20 minutes** so The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md). -**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it +**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Alert settings → Notifications → *Servers and disks*), or the server's own limit if it has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts. While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all. @@ -129,22 +129,22 @@ Every threshold here is a default; where it can be changed, the place is named. | Check | Raised when | Default | Changed in | |---|---|---|---| -| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server | -| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk | -| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server | +| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Alert settings → Notifications; per server | +| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Alert settings → Notifications; per server, per disk | +| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Alert settings → Notifications; per server | | **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) | | **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable | | **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable | | **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable | -| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Settings → Notifications | -| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Settings → Notifications | -| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications | -| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications | +| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Alert settings → Notifications | +| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Alert settings → Notifications | +| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Alert settings → Notifications | +| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Alert settings → Notifications | | **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page | -| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server | -| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications | -| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications | -| **Docker updates** | A container image has a newer version | | Settings → Notifications | +| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Alert settings → Notifications; per server | +| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Alert settings → Notifications | +| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Alert settings → Notifications | +| **Docker updates** | A container image has a newer version | | Alert settings → Notifications | | **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) | | **Hetzner server off / traffic / rescue / backup** | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; [below](#hetzner-cloud) | | **Sonarr / Radarr / Prowlarr** | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; [below](#media-managers-sonarr-radarr-prowlarr) | @@ -161,19 +161,19 @@ Every threshold here is a default; where it can be changed, the place is named. | **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) | | **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) | | **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) | -| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) | -| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) | -| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Settings → Notifications; [below](#docker-containers-that-are-broken) | -| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Settings → Notifications; [below](#docker-containers-that-are-broken) | +| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) | +| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) | +| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) | +| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) | | **App behind its newest release** | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; [below](#app-versions) | | **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable | -| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications | +| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Alert settings → Notifications | | **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check | | **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat | | **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page | | **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) | | **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) | -| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules | +| **Your own rules** | [Below](#your-own-alert-rules) | | Alert settings → Alert rules | ## What is normal for the time @@ -192,13 +192,13 @@ It is **quiet by design**. A reading is a warning only when **all** of these hol The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health condition (`baseline:cpu:server:`, `baseline:mem:server:`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for -a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Settings → Notifications → What to send → Servers and disks. +a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Alert settings → Notifications → What to send → Servers and disks. The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure. ## Service levels -**Monitoring → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the +**Alert settings → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and where today's pace would end the month, once a day has passed), and how each of the last six months went. @@ -217,7 +217,7 @@ and gone when the month turns over and the budget starts again. Each target can ## Your own alert rules -Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers: +Where the built-in checks do not fit, **Alert settings → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers: - **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature. - **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the @@ -475,14 +475,14 @@ The hard part is *stopped*: many containers are stopped on purpose (a stack that that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone. -(Settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.) +(Alert settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.) -**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything. +**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Alert settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything. Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`). **Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other. -**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host. +**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Alert settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host. ## Endpoint checks and heartbeats @@ -527,7 +527,7 @@ Both are judged **every minute**, with the same remember-and-compare bookkeeping Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app, using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**. -- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a +- **An integration is "failing"** when its last **3 calls in a row** (Alert settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the integration's page. - **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check. @@ -542,7 +542,7 @@ A forecast says "this will be full in N days" from a **straight line through the wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat. - It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too. - A disk that is **already over its limit** is left to the ordinary "nearly full" alert. -- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as +- It is shown as an alert when it falls **within the window** (Alert settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as `homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears. - **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history. @@ -579,7 +579,7 @@ browser's own wishes. **Quiet hours** hold messages and send one digest at the e ### Flapping -An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After +An alert **raised three times within an hour** (Alert settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back. ### Escalation diff --git a/docs/PROMETHEUS-GRAFANA.md b/docs/PROMETHEUS-GRAFANA.md index 7c6610e..eb039ed 100644 --- a/docs/PROMETHEUS-GRAFANA.md +++ b/docs/PROMETHEUS-GRAFANA.md @@ -257,7 +257,7 @@ All gauges. The same table is checked by a test against what `/metrics` really s | Series | Labels | Meaning | |---|---|---| | `homelab_info` | `environment` | Always 1; says which installation this is. | -| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. | +| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Alert settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. | | `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. | | `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. | | `homelab_server_memory_used_percent` | `server` | Memory in use. | diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md index 233a5aa..a37697c 100644 --- a/docs/WEBHOOKS.md +++ b/docs/WEBHOOKS.md @@ -6,7 +6,7 @@ here, and they are easy to mix up: | | **Event webhooks** | **The notification webhook** | |---|---|---| | For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint | -| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** | +| Where | **Settings → API and webhooks → Event webhooks** | **Alert settings → Notifications → Channels → Webhook** | | Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) | | Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` | | Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) | @@ -274,7 +274,7 @@ def hook(): The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post. -**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret. +**Alert settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret. ``` POST @@ -294,7 +294,7 @@ X-Webhook-Secret: (only if one is set) ## Buttons on notifications A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches -it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications). +it on under **Alert settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications). The addresses behind them, for anything that wants to build on it: diff --git a/server/src/db/schema.ts b/server/src/db/schema.ts index 2009fc6..2b14dac 100644 --- a/server/src/db/schema.ts +++ b/server/src/db/schema.ts @@ -1788,7 +1788,7 @@ export const webhookEvents = ["alert.opened", "alert.resolved", "maintenance.sta export type WebhookEvent = (typeof webhookEvents)[number]; // A URL that gets a signed JSON message for the events it subscribes to — for Home Assistant, n8n, a status board. Separate from the -// chat-style webhook under Settings → Notifications, which carries the same text a person would read. +// chat-style webhook under Alert settings → Notifications, which carries the same text a person would read. export const eventWebhooks = sqliteTable("event_webhooks", { id: integer("id").primaryKey({ autoIncrement: true }), label: text("label").notNull(), diff --git a/server/src/integrations/fieldSchemas.ts b/server/src/integrations/fieldSchemas.ts index 574dea7..bffbc7b 100644 --- a/server/src/integrations/fieldSchemas.ts +++ b/server/src/integrations/fieldSchemas.ts @@ -224,7 +224,7 @@ export const INTEGRATION_FIELDS: Partial ({ title: `${r.name} is billed — ${money}`, link: "/costs" })); } - // A restore test is due a set time after the last good one — only if a time limit is set (Settings → Notifications). + // A restore test is due a set time after the last good one — only if a time limit is set (Alert settings → Notifications). if (data.restoreLimitDays > 0) { for (const t of data.restoreTests) { if ((t.result !== "ok" && t.result !== "partial") || !isYmd(t.testedAt)) continue; diff --git a/server/src/services/alertActionLinks.ts b/server/src/services/alertActionLinks.ts index 3e14fd4..f86013d 100644 --- a/server/src/services/alertActionLinks.ts +++ b/server/src/services/alertActionLinks.ts @@ -1,6 +1,6 @@ /** * The links behind buttons on a notification (see services/alertActions.ts for what they do): Acknowledge, Silence for an hour, Snooze until tomorrow morning — from the phone push, an ntfy message, a link in an email or a Gotify message, - * without signing in to the app. A switch an administrator turns on (Settings → Notifications → Who hears what); off, no link works. + * without signing in to the app. A switch an administrator turns on (Alert settings → Notifications → Who hears what); off, no link works. * * What makes it safe enough to have is how little each link can do: * diff --git a/server/src/services/alertNoise.ts b/server/src/services/alertNoise.ts index 4e1768b..2236412 100644 --- a/server/src/services/alertNoise.ts +++ b/server/src/services/alertNoise.ts @@ -74,15 +74,15 @@ export function mostWithin(times: number[], windowMs: number): number { /** What to try, from the kind of alert and the key it has. Plain advice; nothing it says is done for anyone. */ export function hintFor(category: string, alertId: string): string { const key = keyOfAlertId(alertId, category); - if (key.startsWith("rule:")) return "A limit of your own (Monitoring → Alert rules): if it keeps coming back, raise the limit or make it hold for longer before it raises."; + if (key.startsWith("rule:")) return "A limit of your own (Alert settings → Alert rules): if it keeps coming back, raise the limit or make it hold for longer before it raises."; if (key.startsWith("new-port:")) return "If this port should be listening, mark it as expected on the server's Ports card."; if (key.startsWith("heartbeat")) return "If the job is just sometimes late, give its heartbeat more grace; if it fails, look at what it printed."; if (key.startsWith("endpoint:")) return "Raise “Alert after … failures in a row” on the check, or its timeout or interval — or fix what is making it fail."; if (key.startsWith("endpoint-cert:")) return "Renew the certificate, or lower how early the check warns."; if (key.startsWith("failed-units:")) return "If a service is expected to fail, add it to “Services to ignore” on the server's Host health card; otherwise fix the service."; if (key.startsWith("disk-forecast:") || category === "disk") return "Raise the limit for this disk (a server can have its own, per disk), or leave it alone on the server's page if it is meant to be full."; - if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Settings → Notifications, or give this server its own limit."; - if (category === "integration") return "Raise “Alert after … consecutive failures” under Settings → Notifications; the Call log (Integrations) shows what is failing."; + if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Alert settings → Notifications, or give this server its own limit."; + if (category === "integration") return "Raise “Alert after … consecutive failures” under Alert settings → Notifications; the Call log (Integrations) shows what is failing."; if (category === "backup") return "A guest that is backed up late each time may need its “too old” limit raised on its server, or its job moved."; return "Acknowledge it with a note, silence it with maintenance while you work on it, or fix what raises it."; } diff --git a/server/src/services/alertTickets.ts b/server/src/services/alertTickets.ts index 1b0370e..4bab755 100644 --- a/server/src/services/alertTickets.ts +++ b/server/src/services/alertTickets.ts @@ -360,7 +360,7 @@ export async function sendTestTicket(actor: CurrentUser): Promise topicId: settings.topicId, priorityId: settings.priorityId, subject: "[Homelab] Test ticket from Homelab Manager", - message: `This is a test. ${actor.name ?? actor.email ?? "An administrator"} asked Homelab Manager to open it from Settings → Notifications → Alert tickets, to check that tickets can be created here.\n\nNothing is wrong. You can close this ticket.\n\n${absolute("/settings/alert-tickets")}`, + message: `This is a test. ${actor.name ?? actor.email ?? "An administrator"} asked Homelab Manager to open it from Alert settings → Alert tickets, to check that tickets can be created here.\n\nNothing is wrong. You can close this ticket.\n\n${absolute("/settings/alert-tickets")}`, }); await recordAudit({ actor, category: "alert_ticket", action: "test", targetType: "integration", targetId: settings.integrationId ?? undefined, detail: { ticket: created.number } }); return created; diff --git a/server/src/services/alerts.ts b/server/src/services/alerts.ts index 8aa71c1..34c3c6a 100644 --- a/server/src/services/alerts.ts +++ b/server/src/services/alerts.ts @@ -73,7 +73,7 @@ const SOURCES: Record = { npm: { label: "Nginx Proxy Manager", link: "/proxy-hosts" }, homeassistant: { label: "Home Assistant", link: "/home-assistant" }, ups: { label: "UPS", link: "/ups" }, - slo: { label: "Service level", link: "/service-levels" }, + slo: { label: "Service level", link: "/alert-settings?view=levels" }, duplicati: { label: "Duplicati", link: "/duplicati" }, jellyfin: { label: "Jellyfin", link: "/jellyfin" }, authentik: { label: "Authentik", link: "/authentik" }, diff --git a/server/src/services/dockerVulnSync.ts b/server/src/services/dockerVulnSync.ts index bd49c33..4436225 100644 --- a/server/src/services/dockerVulnSync.ts +++ b/server/src/services/dockerVulnSync.ts @@ -5,7 +5,7 @@ * Nothing here starts a scan: Dockhand scans when it is told to (by hand, after a pull, or for a whole host), and this only reads what it kept. So an image that was never scanned is * not known to be clean — the page says how many of a host's images have a scan — and an environment whose summary could not be read is *unknown*, never *none*. * - * What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Settings → Notifications). Dockhand can also notify + * What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Alert settings → Notifications). Dockhand can also notify * about vulnerabilities itself; if both are switched on you hear twice. */ import { and, eq, notInArray } from "drizzle-orm"; diff --git a/server/src/services/eventWebhooks.ts b/server/src/services/eventWebhooks.ts index aea5127..569895c 100644 --- a/server/src/services/eventWebhooks.ts +++ b/server/src/services/eventWebhooks.ts @@ -1,7 +1,7 @@ /** * Event webhooks: tell another system when something happens. Each webhook is a URL with the events it wants; when one happens it is * sent a JSON message, signed with the webhook's secret so the receiver can tell it came from here. This is separate from the chat-style - * webhook under Settings → Notifications, which carries the same sentence a person would read. + * webhook under Alert settings → Notifications, which carries the same sentence a person would read. * * alert.opened / alert.resolved a problem began or ended — on the same terms as a notification (a problem under maintenance, * or folded under another, isn't announced until it stands alone) diff --git a/server/src/services/restoreTests.ts b/server/src/services/restoreTests.ts index 15ce85c..d7faf27 100644 --- a/server/src/services/restoreTests.ts +++ b/server/src/services/restoreTests.ts @@ -1,7 +1,7 @@ /** * Restore tests: a backup nobody has ever restored is a hope, not a backup. Somebody records when they tried to restore a guest (or anything else * — a share, a database) and how it went, and the app keeps the history, shows each guest's newest test beside its backups, and speaks up - * when a test failed (always) or when none has been done in too long (if a time limit is set under Settings → Notifications). + * when a test failed (always) or when none has been done in too long (if a time limit is set under Alert settings → Notifications). */ import { desc } from "drizzle-orm"; import { db } from "../db/client.js"; diff --git a/server/src/services/serverThresholds.ts b/server/src/services/serverThresholds.ts index d55fa4c..c63bf94 100644 --- a/server/src/services/serverThresholds.ts +++ b/server/src/services/serverThresholds.ts @@ -1,5 +1,5 @@ /** - * Per-server alert thresholds. The ones in Settings → Notifications apply to every server; some machines are different — a NAS that sits at + * Per-server alert thresholds. The ones in Alert settings → Notifications apply to every server; some machines are different — a NAS that sits at * 92% by design, a database host that should speak up at 80%, a laptop that is meant to be off at night. Each server may override the * server-offline time, the disk-usage percentage, the disk forecast window and (for the Proxmox guest it is linked to) how old a backup may * be, and may set a limit for one disk or say not to alert about that disk at all. Anything not overridden follows the global setting, so diff --git a/server/src/services/settingsStore.ts b/server/src/services/settingsStore.ts index 346fb76..fe89940 100644 --- a/server/src/services/settingsStore.ts +++ b/server/src/services/settingsStore.ts @@ -165,7 +165,7 @@ export interface MonthlyReportSettings { time: string; } -/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */ +/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Alert settings → Notifications. */ export interface DigestSettings { enabled: boolean; /** 0 = Sunday … 6 = Saturday, in notifications.timezone. */ @@ -235,7 +235,7 @@ export interface NameGeneratorSettings { export type AlertTicketSeverity = "critical" | "warning" | "info"; -/** Opening an osTicket ticket for an alert. Off until an admin sets it up under Settings → Notifications → Alert tickets. */ +/** Opening an osTicket ticket for an alert. Off until an admin sets it up under Alert settings → Alert tickets. */ export interface AlertTicketSettings { enabled: boolean; /** The osTicket integration whose API (address and key) is used. */ @@ -397,7 +397,7 @@ function openValue(section: string, field: string, stored: unknown): unknown { const name = `${section}.${field}`; if (!warnedUnreadable.has(name)) { warnedUnreadable.add(name); - console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Settings → Notifications.`); + console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Alert settings → Notifications.`); } return ""; } diff --git a/web/src/App.tsx b/web/src/App.tsx index 85c41d7..2dd168b 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -8,7 +8,7 @@ import Secrets from "./pages/Secrets"; import Ipam from "./pages/Ipam"; import NetworkDevices from "./pages/NetworkDevices"; import ProxyHosts from "./pages/ProxyHosts"; -import AlertRules from "./pages/AlertRules"; +import AlertSettings, { NotificationsRedirect } from "./pages/AlertSettings"; import HomeAssistant from "./pages/HomeAssistant"; import Ups from "./pages/Ups"; import Jellyfin from "./pages/Jellyfin"; @@ -27,7 +27,6 @@ import TechnitiumDns from "./pages/TechnitiumDns"; import Grafana from "./pages/Grafana"; import Speedtest from "./pages/Speedtest"; import Costs from "./pages/Costs"; -import ServiceLevels from "./pages/ServiceLevels"; import Duplicati from "./pages/Duplicati"; import MyWorkPage from "./pages/MyWork"; import ApiTokens from "./pages/ApiTokens"; @@ -66,7 +65,7 @@ import BackupsHub from "./pages/BackupsHub"; import Consistency from "./pages/Consistency"; import Privacy from "./pages/Privacy"; import Settings from "./pages/Settings"; -import { DataGroup, DisplayGroup, NotificationsGroup } from "./pages/settings/SettingsGroups"; +import { DataGroup, DisplayGroup } from "./pages/settings/SettingsGroups"; import TagSettings from "./pages/settings/TagSettings"; import NameSettings from "./pages/settings/NameSettings"; import ShareSettings from "./pages/settings/ShareSettings"; @@ -171,7 +170,8 @@ function Shelled({ user }: { user: CurrentUser }) { } /> } /> } /> - } /> + } /> + } /> } /> } /> } /> @@ -191,7 +191,7 @@ function Shelled({ user }: { user: CurrentUser }) { } /> } /> } /> - } /> + } /> } /> } /> } /> @@ -248,8 +248,8 @@ function Shelled({ user }: { user: CurrentUser }) { } > - } /> - } /> + } /> + } /> } /> } /> } /> @@ -259,7 +259,7 @@ function Shelled({ user }: { user: CurrentUser }) { } /> {/* Tabs that were gathered into others: old links and bookmarks still arrive. */} } /> - } /> + } /> } /> } /> } /> diff --git a/web/src/api/client.ts b/web/src/api/client.ts index 5003123..72d49c9 100644 --- a/web/src/api/client.ts +++ b/web/src/api/client.ts @@ -3156,7 +3156,7 @@ export interface DockhandVulnerabilityEnvironment { } export interface DockhandVulnerabilitiesResponse { - /** Whether critical findings are alerted (Settings → Notifications). */ + /** Whether critical findings are alerted (Alert settings → Notifications). */ alerts: boolean; /** A host with at least this many high findings is alerted too; 0 = never. */ highThreshold: number; diff --git a/web/src/components/AlertNoise.tsx b/web/src/components/AlertNoise.tsx index 4b26426..f83eb77 100644 --- a/web/src/components/AlertNoise.tsx +++ b/web/src/components/AlertNoise.tsx @@ -77,7 +77,7 @@ export default function AlertNoise() { <>

The alerts that keep coming back. Each time one is raised and later goes away counts once; only those raised at least twice are listed. The last column says what usually helps — a longer grace time, a different threshold, - or fixing the thing itself. An alert raised and cleared often enough in an hour is called flapping and its messages are held back until it has been calm for an hour (set under Settings → Notifications). + or fixing the thing itself. An alert raised and cleared often enough in an hour is called flapping and its messages are held back until it has been calm for an hour (set under Alert settings → Notifications).

diff --git a/web/src/components/DatabaseBackups.tsx b/web/src/components/DatabaseBackups.tsx index a7200f5..b722479 100644 --- a/web/src/components/DatabaseBackups.tsx +++ b/web/src/components/DatabaseBackups.tsx @@ -206,7 +206,7 @@ export default function DatabaseBackups() { Save -
The time is in the notification time zone (Settings → Notifications). An alert is raised if the newest backup gets more than two days old.
+
The time is in the notification time zone (Alert settings → Notifications). An alert is raised if the newest backup gets more than two days old.
{data.snapshots.length === 0 ? ( diff --git a/web/src/components/ServerThresholds.tsx b/web/src/components/ServerThresholds.tsx index e7f5b0e..ac5d4cf 100644 --- a/web/src/components/ServerThresholds.tsx +++ b/web/src/components/ServerThresholds.tsx @@ -16,7 +16,7 @@ const FIELDS: { key: Field; label: string; unit: string; min: number; max: numbe const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0); /** - * This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every + * This server's own alert limits. Anything left empty follows Alert settings → Notifications, so changing the global limit still reaches every * server that hasn't said otherwise. A disk can have a limit of its own, or be left alone altogether. Everyone can see what is set; only * an administrator changes it. */ @@ -115,7 +115,7 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC }} >
-

Leave a box empty to follow Settings → Notifications (shown inside the box). Only what you fill in is different for this server.

+

Leave a box empty to follow Alert settings → Notifications (shown inside the box). Only what you fill in is different for this server.

{FIELDS.filter((f) => !f.onlyForGuests || isGuest || own?.[f.key] !== undefined).map((f) => (
diff --git a/web/src/layout/AppShell.tsx b/web/src/layout/AppShell.tsx index 78f496d..6517cd2 100644 --- a/web/src/layout/AppShell.tsx +++ b/web/src/layout/AppShell.tsx @@ -389,7 +389,7 @@ export default function AppShell({ user, children }: { user: CurrentUser; childr {away.away.state === "planned" ? `Away from ${formatDateTime(new Date(away.away.from!))} until ${formatDateTime(new Date(away.away.until))}` : `Away until ${formatDateTime(new Date(away.away.until))}`} {away.away.note ? ` — ${away.away.note}` : ""}.{" "} {away.away.state === "active" ? `Notifications are held${away.away.criticalThrough ? " (critical alerts still come through)" : ""}${away.held > 0 ? ` — ${away.held} so far` : ""}. ` : ""} - {user.role === "admin" ? "Manage" : "Details"} + {user.role === "admin" ? "Manage" : "Details"}
)} diff --git a/web/src/layout/nav.tsx b/web/src/layout/nav.tsx index 3442f44..779a818 100644 --- a/web/src/layout/nav.tsx +++ b/web/src/layout/nav.tsx @@ -8,7 +8,6 @@ import { IconArrowsSplit2, IconHome2, IconBatteryCharging, - IconTargetArrow, IconDatabaseExport, IconAdjustmentsAlt, IconChecklist, @@ -151,14 +150,13 @@ export const NAV: NavEntry[] = [ icon: , items: [ { to: "/endpoints", label: "Endpoints", icon: }, - { to: "/alert-rules", label: "Alert rules", icon: }, + { to: "/alert-settings", label: "Alert settings", icon: }, { to: "/heartbeats", label: "Heartbeats", icon: }, { to: "/uptime-kuma", label: "Uptime Kuma", icon: }, { to: "/ups", label: "UPS", icon: }, { to: "/prometheus", label: "Prometheus", icon: }, { to: "/grafana", label: "Grafana", icon: }, { to: "/speedtest", label: "Speedtest", icon: }, - { to: "/service-levels", label: "Service levels", icon: }, ], }, { diff --git a/web/src/pages/AlertRules.tsx b/web/src/pages/AlertRules.tsx index 97619b8..fc1ede1 100644 --- a/web/src/pages/AlertRules.tsx +++ b/web/src/pages/AlertRules.tsx @@ -221,7 +221,7 @@ function RuleDrawer({ initial, servers, tags, onClose, onSaved }: { initial: Ale )}
- Severity and kind decide who hears of it: these are Monitoring alerts, so the routing rules under Settings → Notifications apply to them like any other. + Severity and kind decide who hears of it: these are Monitoring alerts, so the routing rules under Alert settings → Notifications apply to them like any other.
{tried && (
0 ? "alert-warning" : "alert-success"} py-2`}> @@ -298,7 +298,7 @@ export default function AlertRules({ user }: { user: CurrentUser }) { return ( <>
-

Alert rules

+

Alert rules

{canEdit && (
A backup nobody has restored is a hope, not a backup. Record each time you try one — for a guest above, or for anything else (a share, a database). A failed test is raised on the - Alerts page; under Settings → Notifications you can also ask to be reminded when a guest with backups hasn’t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}. + Alerts page; under Alert settings → Notifications you can also ask to be reminded when a guest with backups hasn’t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
{custom.length > 0 && (
diff --git a/web/src/pages/Dashboard.tsx b/web/src/pages/Dashboard.tsx index b16598f..6c83f15 100644 --- a/web/src/pages/Dashboard.tsx +++ b/web/src/pages/Dashboard.tsx @@ -753,7 +753,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) { { id: "server", title: "Add a server and install its agent", description: "The agent reports scheduled tasks, disks, memory and open ports, and lets the app tell when a server goes quiet.", to: "/servers", action: "Add a server", done: (serverSummary?.total ?? 0) > 0 }, { id: "dns", title: "Connect a DNS provider", description: "See your zones and records, and have names matched to servers.", to: "/dns", action: "Connect", done: (dnsStats?.totalProviders ?? 0) > 0 }, { id: "secret", title: "Track something that expires", description: "An API token, a certificate or a password — you're warned before it runs out.", to: "/secrets", action: "Add one", done: (secrets?.length ?? 0) > 0 }, - { id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/settings/notifications", action: "Set up", done: notifyConfigured === true }, + { id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/alert-settings?view=notifications", action: "Set up", done: notifyConfigured === true }, { id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 }, ]; diff --git a/web/src/pages/DockerVulnerabilities.tsx b/web/src/pages/DockerVulnerabilities.tsx index 385b13b..0b2394c 100644 --- a/web/src/pages/DockerVulnerabilities.tsx +++ b/web/src/pages/DockerVulnerabilities.tsx @@ -121,15 +121,15 @@ export default function DockerVulnerabilities({ integrationId }: { integrationId Critical findings are an alert — see Alerts;{" "} {data.highThreshold > 0 ? ( <> - so is a host with {data.highThreshold} or more high ones (Settings → Notifications); medium and low are only shown here. + so is a host with {data.highThreshold} or more high ones (Alert settings → Notifications); medium and low are only shown here. ) : ( - <>high, medium and low are only shown here (high ones can be alerted from a number you choose under Settings → Notifications). + <>high, medium and low are only shown here (high ones can be alerted from a number you choose under Alert settings → Notifications). )} ) : ( <> - The alert for critical findings is switched off (Settings → Notifications). + The alert for critical findings is switched off (Alert settings → Notifications). )}

diff --git a/web/src/pages/MonthlyReport.tsx b/web/src/pages/MonthlyReport.tsx index 322d3ed..761a57c 100644 --- a/web/src/pages/MonthlyReport.tsx +++ b/web/src/pages/MonthlyReport.tsx @@ -45,7 +45,7 @@ function ServiceLevelsSection({ rows, partial }: { rows: ServiceLevelResult[]; p

{metCount} of {rows.length} {rows.length === 1 ? "target was" : "targets were"} met{partial ? " so far" : ""}. Each is judged on the downtime it allowed over the whole stretch (the error budget).{" "} - Set targets + Set targets

@@ -201,7 +201,7 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) { {user.role === "admin" ? ( <> {" "} - It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: Settings → Notifications. + It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: Alert settings → Notifications. ) : null}

diff --git a/web/src/pages/Privacy.tsx b/web/src/pages/Privacy.tsx index 9f4d4cf..5db1b97 100644 --- a/web/src/pages/Privacy.tsx +++ b/web/src/pages/Privacy.tsx @@ -229,7 +229,7 @@ export default function Privacy() { you configured, to read status and, when someone with permission asks, to act (start a VM, run a template…).
  • - osTicket tickets — only if an admin has switched on Settings → Notifications → Alert tickets: for a problem that has lasted, the text of the alert + osTicket tickets — only if an admin has switched on Alert settings → Alert tickets: for a problem that has lasted, the text of the alert (which server or integration, what is wrong), a link back to this app, and the requester name and email set there go to your osTicket's API.
  • diff --git a/web/src/pages/ServiceLevels.tsx b/web/src/pages/ServiceLevels.tsx index 109a76e..d943a82 100644 --- a/web/src/pages/ServiceLevels.tsx +++ b/web/src/pages/ServiceLevels.tsx @@ -185,7 +185,7 @@ export default function ServiceLevels({ user }: { user: CurrentUser }) { return ( <>
    -

    Service levels

    +

    Service levels

    {isAdmin && !draft && (