Move Notifications back to Settings and link to it from Alert settings
Alert settings keeps Alert rules, Service levels and Alert tickets; administrators get a link to Settings -> Notifications. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4269213275
commit
ccb863e7c3
32 files changed
+152
-150
No files matched your search
+1
-1
@@ -254,7 +254,7 @@ Each agent says which operating system it runs, in the report's `system.os`: `{
|
|||||||
- **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart.
|
- **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart.
|
||||||
|
|
||||||
The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows
|
The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows
|
||||||
client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Alert settings → Notifications (*Flag an
|
client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Settings → Notifications (*Flag an
|
||||||
operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent
|
operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent
|
||||||
older than this version reports no OS and the server shows nothing for it.
|
older than this version reports no OS and the server shows nothing for it.
|
||||||
|
|
||||||
|
|||||||
@@ -256,7 +256,7 @@ was taken (from your `.env` backup).
|
|||||||
Symptoms: an integration or a DNS provider fails with *"Unsupported state or unable to
|
Symptoms: an integration or a DNS provider fails with *"Unsupported state or unable to
|
||||||
authenticate data"*, and notification channels that worked before don't. The database is fine;
|
authenticate data"*, and notification channels that worked before don't. The database is fine;
|
||||||
only the stored credentials can't be read. If you can't find the old key, enter the credentials
|
only the stored credentials can't be read. If you can't find the old key, enter the credentials
|
||||||
again (Integrations → edit, DNS providers, Alert settings → Notifications). Nothing else is lost.
|
again (Integrations → edit, DNS providers, Settings → Notifications). Nothing else is lost.
|
||||||
**Don't** change `CREDENTIALS_ENCRYPTION_KEY` on a running system to "fix" this: it makes
|
**Don't** change `CREDENTIALS_ENCRYPTION_KEY` on a running system to "fix" this: it makes
|
||||||
everything stored so far unreadable.
|
everything stored so far unreadable.
|
||||||
|
|
||||||
|
|||||||
+3
-3
@@ -70,7 +70,7 @@ It is for the readers written from documentation (see "Not verified" on each int
|
|||||||
- **Container health:** on the 15-minute health pass the app also reads every container's state and status text and raises alerts for what is broken — see [Docker containers that are broken](docs/MONITORING.md#docker-containers-that-are-broken). A Docker host that Dockhand
|
- **Container health:** on the 15-minute health pass the app also reads every container's state and status text and raises alerts for what is broken — see [Docker containers that are broken](docs/MONITORING.md#docker-containers-that-are-broken). A Docker host that Dockhand
|
||||||
cannot read (its Hawser agent is down) is an alert too, and its containers' alerts are held, not cleared. The Docker page shows each container's problem and has a **Leave alone** button.
|
cannot read (its Hawser agent is down) is an alert too, and its containers' alerts are held, not cleared. The Docker page shows each container's problem and has a **Leave alone** button.
|
||||||
- **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host — and so are **high** ones once a host has as many as a number you set (*Also warn when a Docker host has this many high vulnerabilities*, off by default) — (a setting turns the critical one off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*.
|
- **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host — and so are **high** ones once a host has as many as a number you set (*Also warn when a Docker host has this many high vulnerabilities*, off by default) — (a setting turns the critical one off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*.
|
||||||
- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Alert settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged.
|
- **Image age:** the same pass reads each Docker host's images and containers (`GET /api/images?env=`, `GET /api/containers?env=&all=true`) and keeps, for every image a *running* container uses, when it was built (Docker's own `Created`). Matched by the name the container gives (Docker Hub's `docker.io/library/` and a missing `:latest` are not a difference) or by id. With *Warn about running Docker images that were built more than … days ago* set (Settings → Notifications; **off by default**), a host with such images is a warning — whatever the scans of them said. The Docker vulnerabilities view shows each host's oldest running images either way. **Written from Dockhand's documented list (`Id`, `RepoTags`, `Created`), not tried against a live Dockhand**; an image whose date cannot be found is not judged.
|
||||||
|
|
||||||
### Uptime Kuma
|
### Uptime Kuma
|
||||||
|
|
||||||
@@ -169,7 +169,7 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
|
|||||||
backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by
|
backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by
|
||||||
PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot),
|
PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot),
|
||||||
and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed
|
and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed
|
||||||
verification or a datastore couldn't be read at all — the "Notify on" section in Alert settings → Notifications has a
|
verification or a datastore couldn't be read at all — the "Notify on" section in Settings → Notifications has a
|
||||||
dedicated toggle for it, sharing the same daily reminder time as the other daily checks.
|
dedicated toggle for it, sharing the same daily reminder time as the other daily checks.
|
||||||
- **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the
|
- **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the
|
||||||
`PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run
|
`PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run
|
||||||
@@ -305,7 +305,7 @@ osTicket's API cannot close a ticket, so this is the one place in the app that *
|
|||||||
It all appears on **Network → Network devices**.
|
It all appears on **Network → Network devices**.
|
||||||
- A device is a **node in the dependency map**, like a server: the *connected through* link is made into a dependency by itself (an access point needs its switch, the switch needs the gateway), and you can write
|
- A device is a **node in the dependency map**, like a server: the *connected through* link is made into a dependency by itself (an access point needs its switch, the switch needs the gateway), and you can write
|
||||||
others by hand (a server needs the switch it is plugged into).
|
others by hand (a server needs the switch it is plugged into).
|
||||||
- A device the controller has not seen online for longer than **Alert settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what
|
- A device the controller has not seen online for longer than **Settings → Notifications → Alert when a network device is offline for** (10 minutes) is a critical **offline** alert about it. What needs it, and what
|
||||||
it is connected through if that is down too, is folded under it, like a guest under its host. It can be silenced with a maintenance window, and has runbooks like any target.
|
it is connected through if that is down too, is folded under it, like a guest under its host. It can be silenced with a maintenance window, and has runbooks like any target.
|
||||||
- **Servers plugged in:** the controller's list of what is connected is matched to servers by IP address (exact, and only when one server has that address). A server is then *connected through* the switch or access point
|
- **Servers plugged in:** the controller's list of what is connected is matched to servers by IP address (exact, and only when one server has that address). A server is then *connected through* the switch or access point
|
||||||
it is on — a dependency made by itself, shown on the server's page and on the Connections map. A key or account that may not read clients costs only this; devices, networks and Wi-Fi names are unaffected.
|
it is on — a dependency made by itself, shown on the server's page and on the Connections map. A key or account that may not read clients costs only this; devices, networks and Wi-Fi names are unaffected.
|
||||||
|
|||||||
+21
-21
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Every notification in Homelab Manager is sent through the same pipe
|
Every notification in Homelab Manager is sent through the same pipe
|
||||||
(`server/src/services/notify.ts`) to whichever channels are enabled under
|
(`server/src/services/notify.ts`) to whichever channels are enabled under
|
||||||
**Alert settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
||||||
JSON webhook, and push to phones and browsers. All five fire for every notification below unless a
|
JSON webhook, and push to phones and browsers. All five fire for every notification below unless a
|
||||||
**routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the
|
**routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the
|
||||||
daily ones, one shared time/timezone.
|
daily ones, one shared time/timezone.
|
||||||
@@ -13,12 +13,12 @@ is closed with changes unsaved): **Channels** (the five channels, each a row
|
|||||||
you open to fill in and send a test), **What to send** (the event toggles
|
you open to fill in and send a test), **What to send** (the event toggles
|
||||||
grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When**
|
grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When**
|
||||||
(the time zone, the daily reminder time, quiet hours and the weekly digest).
|
(the time zone, the daily reminder time, quiet hours and the weekly digest).
|
||||||
The address keeps the tab, e.g. `/alert-settings?view=notifications&tab=when`.
|
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
|
||||||
|
|
||||||
Two settings apply to *every* notification regardless of what triggered
|
Two settings apply to *every* notification regardless of what triggered
|
||||||
it:
|
it:
|
||||||
|
|
||||||
- **Quiet hours** (Alert settings → Notifications → Quiet hours): while
|
- **Quiet hours** (Settings → Notifications → Quiet hours): while
|
||||||
enabled and inside the configured window, a notification is held
|
enabled and inside the configured window, a notification is held
|
||||||
instead of sent immediately, then delivered as a single digest at the
|
instead of sent immediately, then delivered as a single digest at the
|
||||||
window's end time. This matters most for the real-time alerts below —
|
window's end time. This matters most for the real-time alerts below —
|
||||||
@@ -26,7 +26,7 @@ it:
|
|||||||
outside the window anyway. *Still send critical alerts* (off by default)
|
outside the window anyway. *Still send critical alerts* (off by default)
|
||||||
lets a **critical** one — a server down, a disk full, mains power lost —
|
lets a **critical** one — a server down, a disk full, mains power lost —
|
||||||
through at once, so only the warnings wait for the morning.
|
through at once, so only the warnings wait for the morning.
|
||||||
- **Away** (Alert settings → Notifications → When, and a banner on every page while
|
- **Away** (Settings → Notifications → When, and a banner on every page while
|
||||||
it is on): "I'm away until Sunday". See [Away](#away).
|
it is on): "I'm away until Sunday". See [Away](#away).
|
||||||
- **Maintenance windows** (the Maintenance page): opening one for a
|
- **Maintenance windows** (the Maintenance page): opening one for a
|
||||||
server, integration, or DNS provider silences the alerts that target
|
server, integration, or DNS provider silences the alerts that target
|
||||||
@@ -40,7 +40,7 @@ below; several are fixed and can't be changed from the UI.
|
|||||||
|
|
||||||
## Who hears what
|
## Who hears what
|
||||||
|
|
||||||
Under **Alert settings → Notifications → Who hears what**.
|
Under **Settings → Notifications → Who hears what**.
|
||||||
|
|
||||||
**Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of:
|
**Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of:
|
||||||
the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how
|
the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how
|
||||||
@@ -80,7 +80,7 @@ notification of its own — it is something to look at when you are looking at b
|
|||||||
|
|
||||||
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
|
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
|
||||||
this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under
|
this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under
|
||||||
Alert settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
|
Settings → Notifications → Channels (the optional *contact* is given to the browsers' push services, as the standard asks), and then
|
||||||
each person opens the same row and presses **Notify this browser**; the browser asks permission and the row lists their browsers,
|
each person opens the same row and presses **Notify this browser**; the browser asks permission and the row lists their browsers,
|
||||||
with a button to send a test and one to remove each.
|
with a button to send a test and one to remove each.
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@ with a button to send a test and one to remove each.
|
|||||||
|
|
||||||
## Daily reminders
|
## Daily reminders
|
||||||
|
|
||||||
These six checks share one schedule: **Alert settings → Notifications → Daily
|
These six checks share one schedule: **Settings → Notifications → Daily
|
||||||
reminder time** (default 08:00) and **Timezone** (default UTC). Unlike
|
reminder time** (default 08:00) and **Timezone** (default UTC). Unlike
|
||||||
the state-based alerts further down, these **re-send every day the
|
the state-based alerts further down, these **re-send every day the
|
||||||
condition is still true** — there's no "only once" de-duplication, so an
|
condition is still true** — there's no "only once" de-duplication, so an
|
||||||
@@ -112,12 +112,12 @@ until the next scheduled time to catch up.
|
|||||||
| Check | "Notify on" toggle | Fires when | Configured at |
|
| Check | "Notify on" toggle | Fires when | Configured at |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
|
| Secret expiry | *Secret expiry reminder* | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
|
||||||
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Alert settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
|
| Domain expiry | *Domain registration expiring or expired (daily reminder)* | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. `.de`, `.eu`) are tracked but never trigger this. | Settings → Notifications → Health checks → **Domain expiry warning** (default 30 days) |
|
||||||
| Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
|
| Tailscale key expiry | *Tailscale key expiry reminder* | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
|
||||||
| Docker containers broken | *Warn when a Docker container is broken* | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (*Server offline, disk nearly full …*, `healthAlerts`), with a recovery message when it runs again. See [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken). | Its own switch; *Containers to leave alone* |
|
| Docker containers broken | *Warn when a Docker container is broken* | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (*Server offline, disk nearly full …*, `healthAlerts`), with a recovery message when it runs again. See [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken). | Its own switch; *Containers to leave alone* |
|
||||||
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
|
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
|
||||||
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Alert settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
|
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
|
||||||
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Alert settings → Notifications → What to send |
|
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send |
|
||||||
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
|
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
|
||||||
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
|
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
|
||||||
|
|
||||||
@@ -137,9 +137,9 @@ starting baseline rather than announced all at once.
|
|||||||
|
|
||||||
| Check | "Notify on" toggle | Fires when | Configured at |
|
| Check | "Notify on" toggle | Fires when | Configured at |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Alert settings → Notifications → Health checks → **Server offline after** (default 60 min) |
|
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) |
|
||||||
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
|
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
|
||||||
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Alert settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
|
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
|
||||||
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
|
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
|
||||||
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
|
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
|
||||||
|
|
||||||
@@ -153,7 +153,7 @@ schedule.
|
|||||||
| DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
|
| DNS record added | *DNS record added* | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
|
||||||
| DNS record updated | *DNS record updated* | An existing DNS record is edited. |
|
| DNS record updated | *DNS record updated* | An existing DNS record is edited. |
|
||||||
| DNS record deleted | *DNS record deleted* | A DNS record is deleted. |
|
| DNS record deleted | *DNS record deleted* | A DNS record is deleted. |
|
||||||
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Alert settings → Notifications → **Alert after** (default 3 consecutive failures) |
|
| Integration/DNS provider down | *Integration/DNS provider failing repeatedly* | Any integration or DNS provider adapter's calls fail a configurable number of times **in a row**. Tracked per adapter *type* (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, **or if it is explained by something it needs that is down** (see *Dependencies* in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. | Settings → Notifications → **Alert after** (default 3 consecutive failures) |
|
||||||
| Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
|
| Integration/DNS provider recovered | *(same toggle)* | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
|
||||||
|
|
||||||
## Digest
|
## Digest
|
||||||
@@ -170,7 +170,7 @@ A scheduled backup of the app's own database that fails sends one message throug
|
|||||||
|
|
||||||
| Notification | Fires when |
|
| Notification | Fires when |
|
||||||
|---|---|
|
|---|---|
|
||||||
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Alert settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
||||||
|
|
||||||
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
|
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
|
||||||
is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
|
is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
|
||||||
@@ -186,7 +186,7 @@ It is *Monitoring* for routing, and it ends when the month turns over. Maintenan
|
|||||||
|
|
||||||
## Network devices
|
## Network devices
|
||||||
|
|
||||||
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Alert settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
|
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Settings → Notifications → *Servers and disks*, 10 minutes) is announced like a server that stopped reporting, under
|
||||||
*Server offline, disk nearly full …* (`healthAlerts`): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as *Server down*; maintenance on the
|
*Server offline, disk nearly full …* (`healthAlerts`): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as *Server down*; maintenance on the
|
||||||
device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be
|
device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be
|
||||||
read for twenty, its devices are held — neither announced nor cleared.
|
read for twenty, its devices are held — neither announced nor cleared.
|
||||||
@@ -201,7 +201,7 @@ An alert that is **snoozed** (Alerts page → *Snooze…*) is not announced or r
|
|||||||
|
|
||||||
## Buttons on notifications
|
## Buttons on notifications
|
||||||
|
|
||||||
Switched on under **Alert settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
|
Switched on under **Settings → Notifications → Who hears what → Buttons on notifications** (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze
|
||||||
that ended, and the escalation reminder — can be acted on from the message:
|
that ended, and the escalation reminder — can be acted on from the message:
|
||||||
|
|
||||||
| Channel | One alert | Several |
|
| Channel | One alert | Several |
|
||||||
@@ -217,7 +217,7 @@ it came from. **Anyone who can read the notification can use it** for those 24 h
|
|||||||
|
|
||||||
## Away
|
## Away
|
||||||
|
|
||||||
For a stretch rather than every night. **Alert settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
|
For a stretch rather than every night. **Settings → Notifications → When → Away** takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a
|
||||||
holiday ahead), a note to yourself and whether to **still send critical alerts**. While it lasts:
|
holiday ahead), a note to yourself and whether to **still send critical alerts**. While it lasts:
|
||||||
|
|
||||||
- notifications are **held** — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held;
|
- notifications are **held** — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held;
|
||||||
@@ -232,7 +232,7 @@ administrator's to set (it decides who is told what), everyone sees the banner,
|
|||||||
|
|
||||||
## Flapping alerts
|
## Flapping alerts
|
||||||
|
|
||||||
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Alert settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
|
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised **three times within an hour** (Settings → Notifications → *Servers and disks*; 0 = off) it is *flapping*:
|
||||||
|
|
||||||
- one message, **"Flapping alert"**, says "↻ <the alert> — raised 3 times within an hour";
|
- one message, **"Flapping alert"**, says "↻ <the alert> — raised 3 times within an hour";
|
||||||
- its raise and recovery messages are then **held back** (the alert stays on the Alerts page with a *flapping* badge and still counts for the menu badge and /metrics);
|
- its raise and recovery messages are then **held back** (the alert stays on the Alerts page with a *flapping* badge and still counts for the menu badge and /metrics);
|
||||||
@@ -250,13 +250,13 @@ carries the tags of the server, and maintenance on the server holds it back.
|
|||||||
|
|
||||||
A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk
|
A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk
|
||||||
nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under
|
nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under
|
||||||
Alert settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
|
Settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce.
|
||||||
|
|
||||||
## Weekly digest
|
## Weekly digest
|
||||||
|
|
||||||
| Notification | Fires when |
|
| Notification | Fires when |
|
||||||
|---|---|
|
|---|---|
|
||||||
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Alert settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
|
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
|
||||||
|
|
||||||
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
|
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
|
||||||
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
|
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
|
||||||
@@ -323,7 +323,7 @@ the app alerts on something:
|
|||||||
other manual "Sync from X" action — these run when you click the
|
other manual "Sync from X" action — these run when you click the
|
||||||
button and report their result on the page itself, not via a
|
button and report their result on the page itself, not via a
|
||||||
notification.
|
notification.
|
||||||
- **Test buttons** (Alert settings → Notifications → each channel's "Send
|
- **Test buttons** (Settings → Notifications → each channel's "Send
|
||||||
test") send a one-off message through that one channel only, to
|
test") send a one-off message through that one channel only, to
|
||||||
confirm it's wired up correctly — not a real event and not affected by
|
confirm it's wired up correctly — not a real event and not affected by
|
||||||
quiet hours.
|
quiet hours.
|
||||||
|
|||||||
@@ -187,7 +187,7 @@ All modules from the original plan are built:
|
|||||||
Backup page's job and run lists, the comparison table) have an Export CSV
|
Backup page's job and run lists, the comparison table) have an Export CSV
|
||||||
too, which writes the table as it is on screen — times as exact timestamps,
|
too, which writes the table as it is on screen — times as exact timestamps,
|
||||||
action buttons left out.
|
action buttons left out.
|
||||||
- **Alert settings** (Monitoring → Alert settings) — everything that decides what is an alert and who hears of it, on one page with four tabs: *Alert rules*, *Service levels*, *Notifications* and *Alert tickets* (`/alert-settings?view=rules|levels|notifications|tickets`). Everyone signed in sees the first two; the last two are administrators' (they used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/notifications` still lead to the right tab). Maintenance stays on its own page.
|
- **Alert settings** (Monitoring → Alert settings) — what decides what is an alert, on one page with three tabs: *Alert rules*, *Service levels* and *Alert tickets* (`/alert-settings?view=rules|levels|tickets`). Everyone signed in sees the first two; *Alert tickets* is administrators' (it used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/alert-tickets` still lead to the right tab). Notifications — channels, what is sent, who hears what, quiet hours — serve more than alerts, so they stay under **Settings → Notifications**, with a link to them at the top of this page for administrators. Maintenance stays on its own page.
|
||||||
- **Settings** (admin-only) — notification channels (Gotify, ntfy, SMTP,
|
- **Settings** (admin-only) — notification channels (Gotify, ntfy, SMTP,
|
||||||
generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS
|
generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS
|
||||||
record added/updated/deleted, daily secret-expiry reminder and daily
|
record added/updated/deleted, daily secret-expiry reminder and daily
|
||||||
@@ -222,7 +222,7 @@ Server and storage health is watched every 15 minutes: a server whose agent
|
|||||||
stops reporting, a server disk / Proxmox storage / Synology volume passing a
|
stops reporting, a server disk / Proxmox storage / Synology volume passing a
|
||||||
usage threshold, and a Synology volume or disk that's degraded or failing each
|
usage threshold, and a Synology volume or disk that's degraded or failing each
|
||||||
raise one notification when the problem starts and one when it clears (both
|
raise one notification when the problem starts and one when it clears (both
|
||||||
thresholds are set under Alert settings → Notifications). Active problems are
|
thresholds are set under Settings → Notifications). Active problems are
|
||||||
remembered across restarts, so a rebuild doesn't re-alert them.
|
remembered across restarts, so a rebuild doesn't re-alert them.
|
||||||
|
|
||||||
**Tags** — servers can be tagged (prod, media, rack-1, …) from their detail
|
**Tags** — servers can be tagged (prod, media, rack-1, …) from their detail
|
||||||
@@ -306,7 +306,7 @@ like `lab.example.se` resolves to the `example.se` registration that actually
|
|||||||
expires); others can be added by hand. Each is looked up daily over RDAP where
|
expires); others can be added by hand. Each is looked up daily over RDAP where
|
||||||
the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what
|
the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what
|
||||||
makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap.
|
makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap.
|
||||||
You're reminded daily from N days before expiry (Alert settings → Notifications,
|
You're reminded daily from N days before expiry (Settings → Notifications,
|
||||||
default 30) until it's renewed, and told when an expiry date couldn't be
|
default 30) until it's renewed, and told when an expiry date couldn't be
|
||||||
refreshed for several days. Registries that don't publish an expiry (`.de`,
|
refreshed for several days. Registries that don't publish an expiry (`.de`,
|
||||||
`.eu`) can be tracked but have no date to warn about. Private zones (`.lan`,
|
`.eu`) can be tracked but have no date to warn about. Private zones (`.lan`,
|
||||||
@@ -340,7 +340,7 @@ send the notifications — so the two can't disagree — but ignores the on/off
|
|||||||
toggles, since it's for looking at rather than being interrupted by. Problems
|
toggles, since it's for looking at rather than being interrupted by. Problems
|
||||||
under a maintenance window stay listed, marked silenced and counted separately.
|
under a maintenance window stay listed, marked silenced and counted separately.
|
||||||
It also looks ahead: from an agent server's own history (see its Trends card), a
|
It also looks ahead: from an agent server's own history (see its Trends card), a
|
||||||
disk that is on course to be full within 14 days (Alert settings → Notifications → "Warn
|
disk that is on course to be full within 14 days (Settings → Notifications → "Warn
|
||||||
when a disk will fill within"; 0 turns it off) is flagged with its rate of growth
|
when a disk will fill within"; 0 turns it off) is flagged with its rate of growth
|
||||||
and the days left — critical when that is three days or less. It needs at least
|
and the days left — critical when that is three days or less. It needs at least
|
||||||
three days of steady growth to say anything, skips a disk that jumped once or
|
three days of steady growth to say anything, skips a disk that jumped once or
|
||||||
@@ -456,7 +456,7 @@ alerts always stand alone.
|
|||||||
|
|
||||||
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
|
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
|
||||||
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
|
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
|
||||||
A **failed** test is always an alert (critical) until a newer one passes. Under Alert settings → Notifications → "A restore should have been tested within" you can also
|
A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also
|
||||||
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
|
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
|
||||||
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
|
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
|
||||||
kept, `/metrics` has each subject's age and result, and everything is in the change log.
|
kept, `/metrics` has each subject's age and result, and everything is in the change log.
|
||||||
@@ -475,7 +475,7 @@ so it is folded only when every enabled instance of that kind is explained. A se
|
|||||||
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
|
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
|
||||||
alerts of their own, so they aren't part of this.)
|
alerts of their own, so they aren't part of this.)
|
||||||
|
|
||||||
**Who hears what** (Alert settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
|
**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
|
||||||
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
|
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
|
||||||
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
|
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
|
||||||
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
|
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
|
||||||
@@ -774,9 +774,9 @@ pace so far would end the month), and how each of the last six months went. When
|
|||||||
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
|
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
|
||||||
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
|
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
|
||||||
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
|
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
|
||||||
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Alert settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
|
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
|
||||||
|
|
||||||
**Weekly digest** (Alert settings → Notifications, off until switched on) sends one message a
|
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
|
||||||
week through your notification channels: what is wrong now, what was down and for how
|
week through your notification channels: what is wrong now, what was down and for how
|
||||||
long, what happened, disks heading for full, things about to expire, updates waiting
|
long, what happened, disks heading for full, things about to expire, updates waiting
|
||||||
and maintenance coming up — each topic only if it has something to say, and "a quiet
|
and maintenance coming up — each topic only if it has something to say, and "a quiet
|
||||||
@@ -891,18 +891,18 @@ minimum in both themes. Motion is turned down for those who ask the system to.
|
|||||||
**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that
|
**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that
|
||||||
nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up
|
nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up
|
||||||
(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on
|
(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on
|
||||||
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Alert settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
|
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
|
||||||
|
|
||||||
**Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and
|
**Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and
|
||||||
never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button;
|
never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button;
|
||||||
the same list is under Alert settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
|
the same list is under Settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
|
||||||
|
|
||||||
**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's
|
**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's
|
||||||
averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey
|
averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey
|
||||||
**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least
|
**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least
|
||||||
30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The
|
30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The
|
||||||
warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by
|
warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by
|
||||||
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Alert settings → Notifications (*Servers and disks*).
|
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*).
|
||||||
|
|
||||||
**A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four
|
**A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four
|
||||||
months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too
|
months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too
|
||||||
@@ -918,7 +918,7 @@ servers, alerts, incidents, inventory, network equipment, endpoint checks and ma
|
|||||||
gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab —
|
gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab —
|
||||||
its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices
|
its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices
|
||||||
join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under
|
join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under
|
||||||
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Alert settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
|
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
|
||||||
(also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md.
|
(also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md.
|
||||||
|
|
||||||
**Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then
|
**Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then
|
||||||
@@ -933,7 +933,7 @@ hours (maintenance), is handed to you or is still going is counted in the badge
|
|||||||
**A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits),
|
**A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits),
|
||||||
switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section.
|
switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section.
|
||||||
|
|
||||||
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Alert settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
|
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
|
||||||
(a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains
|
(a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains
|
||||||
power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is
|
power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is
|
||||||
sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md.
|
sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md.
|
||||||
@@ -943,7 +943,7 @@ all find something. Removing a dependency, an admin link, a runbook, a hand-made
|
|||||||
what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first.
|
what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first.
|
||||||
|
|
||||||
**Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same
|
**Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same
|
||||||
alert has been **raised three times within an hour** (Alert settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
|
alert has been **raised three times within an hour** (Settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
|
||||||
so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics.
|
so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics.
|
||||||
Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks)
|
Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks)
|
||||||
still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log.
|
still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log.
|
||||||
@@ -960,7 +960,7 @@ what was there before (the earlier acknowledgement text, the earlier tags, the e
|
|||||||
|
|
||||||
**Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a
|
**Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a
|
||||||
badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows
|
badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows
|
||||||
(client and server). A server whose system is past its end, or ends within **Alert settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
|
(client and server). A server whose system is past its end, or ends within **Settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
|
||||||
support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`.
|
support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`.
|
||||||
|
|
||||||
**Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a
|
**Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a
|
||||||
@@ -1006,7 +1006,7 @@ the disk's last report, so it is judged right now and has no time.
|
|||||||
**Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's
|
**Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's
|
||||||
storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside
|
storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside
|
||||||
each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window
|
each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window
|
||||||
(Alert settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
|
(Settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
|
||||||
`homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history.
|
`homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history.
|
||||||
|
|
||||||
**Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under
|
**Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under
|
||||||
@@ -1025,7 +1025,7 @@ switch it is plugged into; something that needs it; a change that touches everyt
|
|||||||
*close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a
|
*close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a
|
||||||
change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API.
|
change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API.
|
||||||
|
|
||||||
**Buttons on notifications.** With **Alert settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
|
**Buttons on notifications.** With **Settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
|
||||||
or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as
|
or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as
|
||||||
buttons — three on an ntfy message, two on a phone push (the browser's own notification), with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons;
|
buttons — three on an ntfy message, two on a phone push (the browser's own notification), with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons;
|
||||||
Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it
|
Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ page's own top-level link.
|
|||||||
| Consistency | `/consistency` | ✅ | ✅ | ✅ |
|
| Consistency | `/consistency` | ✅ | ✅ | ✅ |
|
||||||
| **Monitoring** | | | | |
|
| **Monitoring** | | | | |
|
||||||
| Endpoints | `/endpoints` | ✅ | ✅ | ✅ |
|
| Endpoints | `/endpoints` | ✅ | ✅ | ✅ |
|
||||||
| Alert settings (tabs: Alert rules, Service levels; and for administrators only Notifications, Alert tickets) | `/alert-settings` | ✅ | ✅ | ✅ |
|
| Alert settings (tabs: Alert rules, Service levels; and for administrators only Alert tickets, plus a link to Settings → Notifications) | `/alert-settings` | ✅ | ✅ | ✅ |
|
||||||
| Heartbeats | `/heartbeats` | ✅ | ✅ | ✅ |
|
| Heartbeats | `/heartbeats` | ✅ | ✅ | ✅ |
|
||||||
| Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ |
|
| Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ |
|
||||||
| UPS | `/ups` | ✅ | ✅ | ✅ |
|
| UPS | `/ups` | ✅ | ✅ | ✅ |
|
||||||
@@ -137,10 +137,10 @@ even further, to admin only:
|
|||||||
**Settings search** is part of Settings, which is administrators'.
|
**Settings search** is part of Settings, which is administrators'.
|
||||||
- **Network devices** (Network → Network devices): everyone can see the devices, networks and Wi-Fi names read from UniFi controllers; **Read now** and putting a device under maintenance need operator; adding the UniFi
|
- **Network devices** (Network → Network devices): everyone can see the devices, networks and Wi-Fi names read from UniFi controllers; **Read now** and putting a device under maintenance need operator; adding the UniFi
|
||||||
integration itself (a key or login) is an administrator's job, like any integration.
|
integration itself (a key or login) is an administrator's job, like any integration.
|
||||||
- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Alert settings → Notifications (administrators).
|
- **Docker vulnerabilities** (Docker page): everyone signed in sees the counts and the critical findings from Dockhand's scans; nothing on the page starts a scan or changes anything. Switching the alert off is in Settings → Notifications (administrators).
|
||||||
- **Docker containers** (Docker page): everyone signed in sees each container and what is wrong with it; start, stop and restart need operator; **Leave alone** (no alerts for a container) and the list in Settings are administrators'.
|
- **Docker containers** (Docker page): everyone signed in sees each container and what is wrong with it; start, stop and restart need operator; **Leave alone** (no alerts for a container) and the list in Settings are administrators'.
|
||||||
- **Away mode** (Alert settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held.
|
- **Away mode** (Settings → Notifications → When): administrators start, plan, change and end it, since it decides who is told what; everyone signed in sees the banner that says someone is away and how many notifications are held.
|
||||||
- **Buttons on notifications** (Alert settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else.
|
- **Buttons on notifications** (Settings → Notifications → Who hears what): switching them on is administrator-only. A link in a notification needs **no role at all** — the signed token is the access, for the alerts in that message, for 24 hours, to acknowledge, silence for an hour or snooze, and nothing else.
|
||||||
- **Alert rules** (Alert settings → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is
|
- **Alert rules** (Alert settings → Alert rules): everyone signed in sees the rules and where each is raised now; making, changing, trying, switching and removing one is administrator-only (a rule decides who is
|
||||||
woken up). **Snooze** an alert, wake it, **close** it and open it again need operator, like acknowledging. The storage forecast beside Proxmox storages and Synology volumes is for everyone who can see those pages.
|
woken up). **Snooze** an alert, wake it, **close** it and open it again need operator, like acknowledging. The storage forecast beside Proxmox storages and Synology volumes is for everyone who can see those pages.
|
||||||
- **What changed before this?** (on an incident's page, and `GET /api/v1/incidents/:id/before`): everyone signed in, like the incident record it is made from.
|
- **What changed before this?** (on an incident's page, and `GET /api/v1/incidents/:id/before`): everyone signed in, like the incident record it is made from.
|
||||||
@@ -174,7 +174,7 @@ even further, to admin only:
|
|||||||
- **Duplicati** (Backups → Duplicati): everyone signed in sees every backup and how it went; setting how often one is expected, muting it and forgetting it need operator; adding a machine (which makes an address that can write to this app) and removing one are administrators'. Reports arrive on a secret address with no sign-in: the token is the access.
|
- **Duplicati** (Backups → Duplicati): everyone signed in sees every backup and how it went; setting how often one is expected, muting it and forgetting it need operator; adding a machine (which makes an address that can write to this app) and removing one are administrators'. Reports arrive on a secret address with no sign-in: the token is the access.
|
||||||
- **Restore tests**: everyone sees them on the Backup coverage page; recording one needs operator; removing one is administrator-only.
|
- **Restore tests**: everyone sees them on the Backup coverage page; recording one needs operator; removing one is administrator-only.
|
||||||
- **Change log**: operators and administrators (it is made from the audit log, which has the same readers).
|
- **Change log**: operators and administrators (it is made from the audit log, which has the same readers).
|
||||||
- **Push notifications**: an administrator switches the channel on (Alert settings → Notifications). Each person, whatever their role, adds
|
- **Push notifications**: an administrator switches the channel on (Settings → Notifications). Each person, whatever their role, adds
|
||||||
and removes their own browsers and can send a test to them; an administrator can remove anyone's.
|
and removes their own browsers and can send a test to them; an administrator can remove anyone's.
|
||||||
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
|
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
|
||||||
are going; starting an update run (**Update…**) or stopping one needs operator.
|
are going; starting an update run (**Update…**) or stopping one needs operator.
|
||||||
@@ -198,8 +198,8 @@ even further, to admin only:
|
|||||||
- **The delivery log of an event webhook** (what was sent, with the messages, and sending one again) and **Settings → Data → Storage** (how big the app's data is) are administrator-only; sending a message again is in the audit log.
|
- **The delivery log of an event webhook** (what was sent, with the messages, and sending one again) and **Settings → Data → Storage** (how big the app's data is) are administrator-only; sending a message again is in the audit log.
|
||||||
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
|
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
|
||||||
Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
|
Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
|
||||||
- **Alert settings**: the *Alert rules* and *Service levels* tabs are for everyone to read (changing them is administrators'); the *Notifications* and *Alert tickets* tabs are only shown to administrators, as they were under Settings.
|
- **Alert settings**: the *Alert rules* and *Service levels* tabs are for everyone to read (changing them is administrators'); the *Alert tickets* tab, and the link to Settings → Notifications, are only shown to administrators, as they were under Settings.
|
||||||
- Everything under **Settings** (badge colors,
|
- Everything under **Settings** (notification channels, badge colors,
|
||||||
display prefs, log retention, backup/restore) is admin-only, matching
|
display prefs, log retention, backup/restore) is admin-only, matching
|
||||||
the page itself being admin-only.
|
the page itself being admin-only.
|
||||||
|
|
||||||
|
|||||||
+25
-25
@@ -72,7 +72,7 @@ Three ideas run through everything:
|
|||||||
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
|
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
|
||||||
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
|
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
|
||||||
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
|
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
|
||||||
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Alert settings → Notifications |
|
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications |
|
||||||
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
|
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
|
||||||
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
|
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
|
||||||
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
|
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
|
||||||
@@ -84,7 +84,7 @@ After the app **restarts**, server-based checks are paused for **20 minutes** so
|
|||||||
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
|
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
|
||||||
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
|
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
|
||||||
|
|
||||||
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Alert settings → Notifications → *Servers and disks*), or the server's own limit if it
|
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it
|
||||||
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
|
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
|
||||||
|
|
||||||
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
|
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
|
||||||
@@ -129,22 +129,22 @@ Every threshold here is a default; where it can be changed, the place is named.
|
|||||||
|
|
||||||
| Check | Raised when | Default | Changed in |
|
| Check | Raised when | Default | Changed in |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Alert settings → Notifications; per server |
|
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
|
||||||
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Alert settings → Notifications; per server, per disk |
|
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk |
|
||||||
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Alert settings → Notifications; per server |
|
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
|
||||||
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
|
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
|
||||||
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
|
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
|
||||||
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
|
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
|
||||||
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
|
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
|
||||||
| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Alert settings → Notifications |
|
| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Settings → Notifications |
|
||||||
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Alert settings → Notifications |
|
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Settings → Notifications |
|
||||||
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Alert settings → Notifications |
|
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
|
||||||
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Alert settings → Notifications |
|
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
|
||||||
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
|
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
|
||||||
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Alert settings → Notifications; per server |
|
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
|
||||||
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Alert settings → Notifications |
|
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
|
||||||
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Alert settings → Notifications |
|
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
|
||||||
| **Docker updates** | A container image has a newer version | | Alert settings → Notifications |
|
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
|
||||||
| **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) |
|
| **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) |
|
||||||
| **Hetzner server off / traffic / rescue / backup** | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; [below](#hetzner-cloud) |
|
| **Hetzner server off / traffic / rescue / backup** | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; [below](#hetzner-cloud) |
|
||||||
| **Sonarr / Radarr / Prowlarr** | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; [below](#media-managers-sonarr-radarr-prowlarr) |
|
| **Sonarr / Radarr / Prowlarr** | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; [below](#media-managers-sonarr-radarr-prowlarr) |
|
||||||
@@ -161,13 +161,13 @@ Every threshold here is a default; where it can be changed, the place is named.
|
|||||||
| **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) |
|
| **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) |
|
||||||
| **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) |
|
| **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) |
|
||||||
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
|
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
|
||||||
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
|
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
|
||||||
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
|
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
|
||||||
| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
|
| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Settings → Notifications; [below](#docker-containers-that-are-broken) |
|
||||||
| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Alert settings → Notifications; [below](#docker-containers-that-are-broken) |
|
| **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Settings → Notifications; [below](#docker-containers-that-are-broken) |
|
||||||
| **App behind its newest release** | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; [below](#app-versions) |
|
| **App behind its newest release** | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; [below](#app-versions) |
|
||||||
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
|
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
|
||||||
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Alert settings → Notifications |
|
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
|
||||||
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
||||||
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
||||||
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
||||||
@@ -192,7 +192,7 @@ It is **quiet by design**. A reading is a warning only when **all** of these hol
|
|||||||
|
|
||||||
The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health
|
The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health
|
||||||
condition (`baseline:cpu:server:<id>`, `baseline:mem:server:<id>`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for
|
condition (`baseline:cpu:server:<id>`, `baseline:mem:server:<id>`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for
|
||||||
a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Alert settings → Notifications → What to send → Servers and disks.
|
a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Settings → Notifications → What to send → Servers and disks.
|
||||||
|
|
||||||
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
|
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
|
||||||
|
|
||||||
@@ -475,14 +475,14 @@ The hard part is *stopped*: many containers are stopped on purpose (a stack that
|
|||||||
that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already
|
that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already
|
||||||
long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone.
|
long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone.
|
||||||
|
|
||||||
(Alert settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.)
|
(Settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.)
|
||||||
|
|
||||||
**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Alert settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything.
|
**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything.
|
||||||
Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a
|
Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a
|
||||||
Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`).
|
Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`).
|
||||||
|
|
||||||
**Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other.
|
**Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other.
|
||||||
**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Alert settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
|
**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
|
||||||
|
|
||||||
## Endpoint checks and heartbeats
|
## Endpoint checks and heartbeats
|
||||||
|
|
||||||
@@ -527,7 +527,7 @@ Both are judged **every minute**, with the same remember-and-compare bookkeeping
|
|||||||
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
|
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
|
||||||
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
|
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
|
||||||
|
|
||||||
- **An integration is "failing"** when its last **3 calls in a row** (Alert settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
|
- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
|
||||||
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
|
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
|
||||||
integration's page.
|
integration's page.
|
||||||
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
|
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
|
||||||
@@ -542,7 +542,7 @@ A forecast says "this will be full in N days" from a **straight line through the
|
|||||||
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
|
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
|
||||||
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
|
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
|
||||||
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
|
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
|
||||||
- It is shown as an alert when it falls **within the window** (Alert settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
|
- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
|
||||||
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
|
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
|
||||||
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
|
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
|
||||||
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
|
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
|
||||||
@@ -579,7 +579,7 @@ browser's own wishes. **Quiet hours** hold messages and send one digest at the e
|
|||||||
|
|
||||||
### Flapping
|
### Flapping
|
||||||
|
|
||||||
An alert **raised three times within an hour** (Alert settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
|
An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
|
||||||
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
|
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
|
||||||
|
|
||||||
### Escalation
|
### Escalation
|
||||||
|
|||||||
@@ -257,7 +257,7 @@ All gauges. The same table is checked by a test against what `/metrics` really s
|
|||||||
| Series | Labels | Meaning |
|
| Series | Labels | Meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `homelab_info` | `environment` | Always 1; says which installation this is. |
|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
|
||||||
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Alert settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
|
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
|
||||||
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
|
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
|
||||||
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
|
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
|
||||||
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
|
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
|
||||||
|
|||||||
+3
-3
@@ -6,7 +6,7 @@ here, and they are easy to mix up:
|
|||||||
| | **Event webhooks** | **The notification webhook** |
|
| | **Event webhooks** | **The notification webhook** |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
|
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
|
||||||
| Where | **Settings → API and webhooks → Event webhooks** | **Alert settings → Notifications → Channels → Webhook** |
|
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
|
||||||
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
|
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
|
||||||
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
|
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
|
||||||
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
|
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
|
||||||
@@ -274,7 +274,7 @@ def hook():
|
|||||||
|
|
||||||
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
|
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
|
||||||
|
|
||||||
**Alert settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
|
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
|
||||||
|
|
||||||
```
|
```
|
||||||
POST <your address>
|
POST <your address>
|
||||||
@@ -294,7 +294,7 @@ X-Webhook-Secret: <the secret> (only if one is set)
|
|||||||
## Buttons on notifications
|
## Buttons on notifications
|
||||||
|
|
||||||
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
|
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
|
||||||
it on under **Alert settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
|
it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
|
||||||
|
|
||||||
The addresses behind them, for anything that wants to build on it:
|
The addresses behind them, for anything that wants to build on it:
|
||||||
|
|
||||||
|
|||||||
@@ -1788,7 +1788,7 @@ export const webhookEvents = ["alert.opened", "alert.resolved", "maintenance.sta
|
|||||||
export type WebhookEvent = (typeof webhookEvents)[number];
|
export type WebhookEvent = (typeof webhookEvents)[number];
|
||||||
|
|
||||||
// A URL that gets a signed JSON message for the events it subscribes to — for Home Assistant, n8n, a status board. Separate from the
|
// A URL that gets a signed JSON message for the events it subscribes to — for Home Assistant, n8n, a status board. Separate from the
|
||||||
// chat-style webhook under Alert settings → Notifications, which carries the same text a person would read.
|
// chat-style webhook under Settings → Notifications, which carries the same text a person would read.
|
||||||
export const eventWebhooks = sqliteTable("event_webhooks", {
|
export const eventWebhooks = sqliteTable("event_webhooks", {
|
||||||
id: integer("id").primaryKey({ autoIncrement: true }),
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
label: text("label").notNull(),
|
label: text("label").notNull(),
|
||||||
|
|||||||
@@ -239,7 +239,7 @@ export function agendaFromData(data: AgendaData, now: Date, tz: string, days: nu
|
|||||||
day(`renewal:${r.id}`, "renewal", r.nextRenewal, () => ({ title: `${r.name} is billed — ${money}`, link: "/costs" }));
|
day(`renewal:${r.id}`, "renewal", r.nextRenewal, () => ({ title: `${r.name} is billed — ${money}`, link: "/costs" }));
|
||||||
}
|
}
|
||||||
|
|
||||||
// A restore test is due a set time after the last good one — only if a time limit is set (Alert settings → Notifications).
|
// A restore test is due a set time after the last good one — only if a time limit is set (Settings → Notifications).
|
||||||
if (data.restoreLimitDays > 0) {
|
if (data.restoreLimitDays > 0) {
|
||||||
for (const t of data.restoreTests) {
|
for (const t of data.restoreTests) {
|
||||||
if ((t.result !== "ok" && t.result !== "partial") || !isYmd(t.testedAt)) continue;
|
if ((t.result !== "ok" && t.result !== "partial") || !isYmd(t.testedAt)) continue;
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
/**
|
/**
|
||||||
* The links behind buttons on a notification (see services/alertActions.ts for what they do): Acknowledge, Silence for an hour, Snooze until tomorrow morning — from the phone push, an ntfy message, a link in an email or a Gotify message,
|
* The links behind buttons on a notification (see services/alertActions.ts for what they do): Acknowledge, Silence for an hour, Snooze until tomorrow morning — from the phone push, an ntfy message, a link in an email or a Gotify message,
|
||||||
* without signing in to the app. A switch an administrator turns on (Alert settings → Notifications → Who hears what); off, no link works.
|
* without signing in to the app. A switch an administrator turns on (Settings → Notifications → Who hears what); off, no link works.
|
||||||
*
|
*
|
||||||
* What makes it safe enough to have is how little each link can do:
|
* What makes it safe enough to have is how little each link can do:
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -81,8 +81,8 @@ export function hintFor(category: string, alertId: string): string {
|
|||||||
if (key.startsWith("endpoint-cert:")) return "Renew the certificate, or lower how early the check warns.";
|
if (key.startsWith("endpoint-cert:")) return "Renew the certificate, or lower how early the check warns.";
|
||||||
if (key.startsWith("failed-units:")) return "If a service is expected to fail, add it to “Services to ignore” on the server's Host health card; otherwise fix the service.";
|
if (key.startsWith("failed-units:")) return "If a service is expected to fail, add it to “Services to ignore” on the server's Host health card; otherwise fix the service.";
|
||||||
if (key.startsWith("disk-forecast:") || category === "disk") return "Raise the limit for this disk (a server can have its own, per disk), or leave it alone on the server's page if it is meant to be full.";
|
if (key.startsWith("disk-forecast:") || category === "disk") return "Raise the limit for this disk (a server can have its own, per disk), or leave it alone on the server's page if it is meant to be full.";
|
||||||
if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Alert settings → Notifications, or give this server its own limit.";
|
if (category === "offline") return "If the server is just slow to report, raise “Server offline after” under Settings → Notifications, or give this server its own limit.";
|
||||||
if (category === "integration") return "Raise “Alert after … consecutive failures” under Alert settings → Notifications; the Call log (Integrations) shows what is failing.";
|
if (category === "integration") return "Raise “Alert after … consecutive failures” under Settings → Notifications; the Call log (Integrations) shows what is failing.";
|
||||||
if (category === "backup") return "A guest that is backed up late each time may need its “too old” limit raised on its server, or its job moved.";
|
if (category === "backup") return "A guest that is backed up late each time may need its “too old” limit raised on its server, or its job moved.";
|
||||||
return "Acknowledge it with a note, silence it with maintenance while you work on it, or fix what raises it.";
|
return "Acknowledge it with a note, silence it with maintenance while you work on it, or fix what raises it.";
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
* Nothing here starts a scan: Dockhand scans when it is told to (by hand, after a pull, or for a whole host), and this only reads what it kept. So an image that was never scanned is
|
* Nothing here starts a scan: Dockhand scans when it is told to (by hand, after a pull, or for a whole host), and this only reads what it kept. So an image that was never scanned is
|
||||||
* not known to be clean — the page says how many of a host's images have a scan — and an environment whose summary could not be read is *unknown*, never *none*.
|
* not known to be clean — the page says how many of a host's images have a scan — and an environment whose summary could not be read is *unknown*, never *none*.
|
||||||
*
|
*
|
||||||
* What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Alert settings → Notifications). Dockhand can also notify
|
* What is raised is **critical** findings only (a host with one or more is a warning), and only while the setting for it is on (Settings → Notifications). Dockhand can also notify
|
||||||
* about vulnerabilities itself; if both are switched on you hear twice.
|
* about vulnerabilities itself; if both are switched on you hear twice.
|
||||||
*/
|
*/
|
||||||
import { and, eq, notInArray } from "drizzle-orm";
|
import { and, eq, notInArray } from "drizzle-orm";
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
/**
|
/**
|
||||||
* Event webhooks: tell another system when something happens. Each webhook is a URL with the events it wants; when one happens it is
|
* Event webhooks: tell another system when something happens. Each webhook is a URL with the events it wants; when one happens it is
|
||||||
* sent a JSON message, signed with the webhook's secret so the receiver can tell it came from here. This is separate from the chat-style
|
* sent a JSON message, signed with the webhook's secret so the receiver can tell it came from here. This is separate from the chat-style
|
||||||
* webhook under Alert settings → Notifications, which carries the same sentence a person would read.
|
* webhook under Settings → Notifications, which carries the same sentence a person would read.
|
||||||
*
|
*
|
||||||
* alert.opened / alert.resolved a problem began or ended — on the same terms as a notification (a problem under maintenance,
|
* alert.opened / alert.resolved a problem began or ended — on the same terms as a notification (a problem under maintenance,
|
||||||
* or folded under another, isn't announced until it stands alone)
|
* or folded under another, isn't announced until it stands alone)
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
/**
|
/**
|
||||||
* Restore tests: a backup nobody has ever restored is a hope, not a backup. Somebody records when they tried to restore a guest (or anything else
|
* Restore tests: a backup nobody has ever restored is a hope, not a backup. Somebody records when they tried to restore a guest (or anything else
|
||||||
* — a share, a database) and how it went, and the app keeps the history, shows each guest's newest test beside its backups, and speaks up
|
* — a share, a database) and how it went, and the app keeps the history, shows each guest's newest test beside its backups, and speaks up
|
||||||
* when a test failed (always) or when none has been done in too long (if a time limit is set under Alert settings → Notifications).
|
* when a test failed (always) or when none has been done in too long (if a time limit is set under Settings → Notifications).
|
||||||
*/
|
*/
|
||||||
import { desc } from "drizzle-orm";
|
import { desc } from "drizzle-orm";
|
||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/**
|
/**
|
||||||
* Per-server alert thresholds. The ones in Alert settings → Notifications apply to every server; some machines are different — a NAS that sits at
|
* Per-server alert thresholds. The ones in Settings → Notifications apply to every server; some machines are different — a NAS that sits at
|
||||||
* 92% by design, a database host that should speak up at 80%, a laptop that is meant to be off at night. Each server may override the
|
* 92% by design, a database host that should speak up at 80%, a laptop that is meant to be off at night. Each server may override the
|
||||||
* server-offline time, the disk-usage percentage, the disk forecast window and (for the Proxmox guest it is linked to) how old a backup may
|
* server-offline time, the disk-usage percentage, the disk forecast window and (for the Proxmox guest it is linked to) how old a backup may
|
||||||
* be, and may set a limit for one disk or say not to alert about that disk at all. Anything not overridden follows the global setting, so
|
* be, and may set a limit for one disk or say not to alert about that disk at all. Anything not overridden follows the global setting, so
|
||||||
|
|||||||
@@ -165,7 +165,7 @@ export interface MonthlyReportSettings {
|
|||||||
time: string;
|
time: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Alert settings → Notifications. */
|
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */
|
||||||
export interface DigestSettings {
|
export interface DigestSettings {
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
/** 0 = Sunday … 6 = Saturday, in notifications.timezone. */
|
/** 0 = Sunday … 6 = Saturday, in notifications.timezone. */
|
||||||
@@ -397,7 +397,7 @@ function openValue(section: string, field: string, stored: unknown): unknown {
|
|||||||
const name = `${section}.${field}`;
|
const name = `${section}.${field}`;
|
||||||
if (!warnedUnreadable.has(name)) {
|
if (!warnedUnreadable.has(name)) {
|
||||||
warnedUnreadable.add(name);
|
warnedUnreadable.add(name);
|
||||||
console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Alert settings → Notifications.`);
|
console.warn(`Can't decrypt the stored ${name} — CREDENTIALS_ENCRYPTION_KEY has changed or is missing. Enter it again under Settings → Notifications.`);
|
||||||
}
|
}
|
||||||
return "";
|
return "";
|
||||||
}
|
}
|
||||||
|
|||||||
+3
-3
@@ -8,7 +8,7 @@ import Secrets from "./pages/Secrets";
|
|||||||
import Ipam from "./pages/Ipam";
|
import Ipam from "./pages/Ipam";
|
||||||
import NetworkDevices from "./pages/NetworkDevices";
|
import NetworkDevices from "./pages/NetworkDevices";
|
||||||
import ProxyHosts from "./pages/ProxyHosts";
|
import ProxyHosts from "./pages/ProxyHosts";
|
||||||
import AlertSettings, { NotificationsRedirect } from "./pages/AlertSettings";
|
import AlertSettings, { NotificationsPage } from "./pages/AlertSettings";
|
||||||
import HomeAssistant from "./pages/HomeAssistant";
|
import HomeAssistant from "./pages/HomeAssistant";
|
||||||
import Ups from "./pages/Ups";
|
import Ups from "./pages/Ups";
|
||||||
import Jellyfin from "./pages/Jellyfin";
|
import Jellyfin from "./pages/Jellyfin";
|
||||||
@@ -248,8 +248,8 @@ function Shelled({ user }: { user: CurrentUser }) {
|
|||||||
</RequireRole>
|
</RequireRole>
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
<Route index element={<Navigate to="sign-in" replace />} />
|
<Route index element={<Navigate to="notifications" replace />} />
|
||||||
<Route path="notifications" element={<NotificationsRedirect />} />
|
<Route path="notifications" element={<NotificationsPage />} />
|
||||||
<Route path="sign-in" element={<SignInSettings />} />
|
<Route path="sign-in" element={<SignInSettings />} />
|
||||||
<Route path="display" element={<DisplayGroup />} />
|
<Route path="display" element={<DisplayGroup />} />
|
||||||
<Route path="names" element={<NameSettings />} />
|
<Route path="names" element={<NameSettings />} />
|
||||||
|
|||||||
@@ -3156,7 +3156,7 @@ export interface DockhandVulnerabilityEnvironment {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export interface DockhandVulnerabilitiesResponse {
|
export interface DockhandVulnerabilitiesResponse {
|
||||||
/** Whether critical findings are alerted (Alert settings → Notifications). */
|
/** Whether critical findings are alerted (Settings → Notifications). */
|
||||||
alerts: boolean;
|
alerts: boolean;
|
||||||
/** A host with at least this many high findings is alerted too; 0 = never. */
|
/** A host with at least this many high findings is alerted too; 0 = never. */
|
||||||
highThreshold: number;
|
highThreshold: number;
|
||||||
|
|||||||
@@ -77,7 +77,7 @@ export default function AlertNoise() {
|
|||||||
<>
|
<>
|
||||||
<p className="text-secondary">
|
<p className="text-secondary">
|
||||||
The alerts that keep coming back. Each time one is raised and later goes away counts once; only those raised at least twice are listed. The last column says what usually helps — a longer grace time, a different threshold,
|
The alerts that keep coming back. Each time one is raised and later goes away counts once; only those raised at least twice are listed. The last column says what usually helps — a longer grace time, a different threshold,
|
||||||
or fixing the thing itself. An alert raised and cleared often enough in an hour is called <strong>flapping</strong> and its messages are held back until it has been calm for an hour (set under Alert settings → Notifications).
|
or fixing the thing itself. An alert raised and cleared often enough in an hour is called <strong>flapping</strong> and its messages are held back until it has been calm for an hour (set under Settings → Notifications).
|
||||||
</p>
|
</p>
|
||||||
<ErrorNotice error={error} />
|
<ErrorNotice error={error} />
|
||||||
|
|
||||||
|
|||||||
@@ -206,7 +206,7 @@ export default function DatabaseBackups() {
|
|||||||
Save
|
Save
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
<div className="col-12 text-secondary small">The time is in the notification time zone (Alert settings → Notifications). An alert is raised if the newest backup gets more than two days old.</div>
|
<div className="col-12 text-secondary small">The time is in the notification time zone (Settings → Notifications). An alert is raised if the newest backup gets more than two days old.</div>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
{data.snapshots.length === 0 ? (
|
{data.snapshots.length === 0 ? (
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ const FIELDS: { key: Field; label: string; unit: string; min: number; max: numbe
|
|||||||
const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0);
|
const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* This server's own alert limits. Anything left empty follows Alert settings → Notifications, so changing the global limit still reaches every
|
* This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every
|
||||||
* server that hasn't said otherwise. A disk can have a limit of its own, or be left alone altogether. Everyone can see what is set; only
|
* server that hasn't said otherwise. A disk can have a limit of its own, or be left alone altogether. Everyone can see what is set; only
|
||||||
* an administrator changes it.
|
* an administrator changes it.
|
||||||
*/
|
*/
|
||||||
@@ -115,7 +115,7 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC
|
|||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
<div className="card-body">
|
<div className="card-body">
|
||||||
<p className="text-secondary small">Leave a box empty to follow Alert settings → Notifications (shown inside the box). Only what you fill in is different for this server.</p>
|
<p className="text-secondary small">Leave a box empty to follow Settings → Notifications (shown inside the box). Only what you fill in is different for this server.</p>
|
||||||
<div className="row g-3">
|
<div className="row g-3">
|
||||||
{FIELDS.filter((f) => !f.onlyForGuests || isGuest || own?.[f.key] !== undefined).map((f) => (
|
{FIELDS.filter((f) => !f.onlyForGuests || isGuest || own?.[f.key] !== undefined).map((f) => (
|
||||||
<div className="col-md-6" key={f.key}>
|
<div className="col-md-6" key={f.key}>
|
||||||
|
|||||||
@@ -389,7 +389,7 @@ export default function AppShell({ user, children }: { user: CurrentUser; childr
|
|||||||
{away.away.state === "planned" ? `Away from ${formatDateTime(new Date(away.away.from!))} until ${formatDateTime(new Date(away.away.until))}` : `Away until ${formatDateTime(new Date(away.away.until))}`}
|
{away.away.state === "planned" ? `Away from ${formatDateTime(new Date(away.away.from!))} until ${formatDateTime(new Date(away.away.until))}` : `Away until ${formatDateTime(new Date(away.away.until))}`}
|
||||||
{away.away.note ? ` — ${away.away.note}` : ""}.{" "}
|
{away.away.note ? ` — ${away.away.note}` : ""}.{" "}
|
||||||
{away.away.state === "active" ? `Notifications are held${away.away.criticalThrough ? " (critical alerts still come through)" : ""}${away.held > 0 ? ` — ${away.held} so far` : ""}. ` : ""}
|
{away.away.state === "active" ? `Notifications are held${away.away.criticalThrough ? " (critical alerts still come through)" : ""}${away.held > 0 ? ` — ${away.held} so far` : ""}. ` : ""}
|
||||||
<Link to="/alert-settings?view=notifications&tab=when">{user.role === "admin" ? "Manage" : "Details"}</Link>
|
<Link to="/settings/notifications?tab=when">{user.role === "admin" ? "Manage" : "Details"}</Link>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -221,7 +221,7 @@ function RuleDrawer({ initial, servers, tags, onClose, onSaved }: { initial: Ale
|
|||||||
</select>
|
</select>
|
||||||
)}
|
)}
|
||||||
<div className="text-secondary small mb-3">
|
<div className="text-secondary small mb-3">
|
||||||
Severity and kind decide who hears of it: these are <strong>Monitoring</strong> alerts, so the routing rules under Alert settings → Notifications apply to them like any other.
|
Severity and kind decide who hears of it: these are <strong>Monitoring</strong> alerts, so the routing rules under Settings → Notifications apply to them like any other.
|
||||||
</div>
|
</div>
|
||||||
{tried && (
|
{tried && (
|
||||||
<div className={`alert ${tried.length > 0 ? "alert-warning" : "alert-success"} py-2`}>
|
<div className={`alert ${tried.length > 0 ? "alert-warning" : "alert-success"} py-2`}>
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { Navigate, useLocation, useSearchParams } from "react-router-dom";
|
import { Link, Navigate, useLocation, useSearchParams } from "react-router-dom";
|
||||||
import type { CurrentUser } from "../api/client";
|
import type { CurrentUser } from "../api/client";
|
||||||
import ViewTabs from "../components/ViewTabs";
|
import ViewTabs from "../components/ViewTabs";
|
||||||
import AlertRules from "./AlertRules";
|
import AlertRules from "./AlertRules";
|
||||||
@@ -6,41 +6,42 @@ import ServiceLevels from "./ServiceLevels";
|
|||||||
import NotificationSettings from "./settings/NotificationSettings";
|
import NotificationSettings from "./settings/NotificationSettings";
|
||||||
import AlertTicketSettings from "./settings/AlertTicketSettings";
|
import AlertTicketSettings from "./settings/AlertTicketSettings";
|
||||||
|
|
||||||
type View = "rules" | "levels" | "notifications" | "tickets";
|
type View = "rules" | "levels" | "tickets";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Everything that decides what is an alert and who hears of it, on one page with a tab for each: **Alert rules** (limits of your own), **Service levels**
|
* What decides what is an alert, on one page with a tab for each: **Alert rules** (limits of your own), **Service levels** (targets and error budgets) and **Alert tickets**
|
||||||
* (targets and error budgets), **Notifications** (channels, what is sent, who hears what, quiet hours, away mode) and **Alert tickets** (osTicket). The first two
|
* (osTicket). The first two are for everyone to read; the third is Settings, so only an administrator sees it. The tab is `?view=`. Who hears of an alert — the
|
||||||
* are for everyone to read; the last two are Settings, so only an administrator sees them. The tab is `?view=`; Notifications keeps its own `?tab=` and `?section=`.
|
* channels, what is sent, quiet hours — is not here: it serves more than alerts, so it stays under Settings → Notifications, and an administrator gets a link to it.
|
||||||
*/
|
*/
|
||||||
export default function AlertSettings({ user }: { user: CurrentUser }) {
|
export default function AlertSettings({ user }: { user: CurrentUser }) {
|
||||||
const [params] = useSearchParams();
|
const [params] = useSearchParams();
|
||||||
const isAdmin = user.role === "admin";
|
const isAdmin = user.role === "admin";
|
||||||
const asked = params.get("view");
|
const asked = params.get("view");
|
||||||
const view: View = asked === "levels" ? "levels" : isAdmin && (asked === "notifications" || asked === "tickets") ? asked : "rules";
|
const view: View = asked === "levels" ? "levels" : isAdmin && asked === "tickets" ? "tickets" : "rules";
|
||||||
const tabs = [
|
const tabs = [
|
||||||
{ to: "/alert-settings", label: "Alert rules", active: view === "rules" },
|
{ to: "/alert-settings", label: "Alert rules", active: view === "rules" },
|
||||||
{ to: "/alert-settings?view=levels", label: "Service levels", active: view === "levels" },
|
{ to: "/alert-settings?view=levels", label: "Service levels", active: view === "levels" },
|
||||||
...(isAdmin
|
...(isAdmin ? [{ to: "/alert-settings?view=tickets", label: "Alert tickets", active: view === "tickets" }] : []),
|
||||||
? [
|
|
||||||
{ to: "/alert-settings?view=notifications", label: "Notifications", active: view === "notifications" },
|
|
||||||
{ to: "/alert-settings?view=tickets", label: "Alert tickets", active: view === "tickets" },
|
|
||||||
]
|
|
||||||
: []),
|
|
||||||
];
|
];
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<h2 className="page-title mb-3">Alert settings</h2>
|
<div className="d-flex flex-wrap align-items-center gap-2 mb-3">
|
||||||
|
<h2 className="page-title mb-0">Alert settings</h2>
|
||||||
|
{isAdmin && (
|
||||||
|
<Link to="/settings/notifications" className="ms-auto">
|
||||||
|
Notification settings: channels, what is sent, quiet hours →
|
||||||
|
</Link>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
<ViewTabs label="Alert settings" tabs={tabs} />
|
<ViewTabs label="Alert settings" tabs={tabs} />
|
||||||
{view === "levels" ? <ServiceLevels user={user} /> : view === "notifications" ? <NotificationSettings /> : view === "tickets" ? <AlertTicketSettings /> : <AlertRules user={user} />}
|
{view === "levels" ? <ServiceLevels user={user} /> : view === "tickets" ? <AlertTicketSettings /> : <AlertRules user={user} />}
|
||||||
</>
|
</>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** `/settings/notifications` (and its `?tab=`, `?section=`, `?view=tickets`) as it was before the alert settings were one page: the same place on the new one. */
|
/** `/settings/notifications?view=tickets` is how Alert tickets was addressed when it was a tab beside Notifications: it is a tab of Alert settings now. Anything else is Notifications itself. */
|
||||||
export function NotificationsRedirect() {
|
export function NotificationsPage() {
|
||||||
const { search } = useLocation();
|
const { search } = useLocation();
|
||||||
const next = new URLSearchParams(search);
|
if (new URLSearchParams(search).get("view") === "tickets") return <Navigate to="/alert-settings?view=tickets" replace />;
|
||||||
next.set("view", next.get("view") === "tickets" ? "tickets" : "notifications");
|
return <NotificationSettings />;
|
||||||
return <Navigate to={`/alert-settings?${next.toString()}`} replace />;
|
|
||||||
}
|
}
|
||||||
@@ -379,7 +379,7 @@ export default function Backups({ user, tabs }: { user?: CurrentUser; /** The ta
|
|||||||
</div>
|
</div>
|
||||||
<div className="card-body text-secondary small pb-0">
|
<div className="card-body text-secondary small pb-0">
|
||||||
A backup nobody has restored is a hope, not a backup. Record each time you try one — for a guest above, or for anything else (a share, a database). A failed test is raised on the
|
A backup nobody has restored is a hope, not a backup. Record each time you try one — for a guest above, or for anything else (a share, a database). A failed test is raised on the
|
||||||
Alerts page; under Alert settings → Notifications you can also ask to be reminded when a guest with backups hasn’t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
|
Alerts page; under Settings → Notifications you can also ask to be reminded when a guest with backups hasn’t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
|
||||||
</div>
|
</div>
|
||||||
{custom.length > 0 && (
|
{custom.length > 0 && (
|
||||||
<div className="card-body pb-0">
|
<div className="card-body pb-0">
|
||||||
|
|||||||
@@ -753,7 +753,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
|||||||
{ id: "server", title: "Add a server and install its agent", description: "The agent reports scheduled tasks, disks, memory and open ports, and lets the app tell when a server goes quiet.", to: "/servers", action: "Add a server", done: (serverSummary?.total ?? 0) > 0 },
|
{ id: "server", title: "Add a server and install its agent", description: "The agent reports scheduled tasks, disks, memory and open ports, and lets the app tell when a server goes quiet.", to: "/servers", action: "Add a server", done: (serverSummary?.total ?? 0) > 0 },
|
||||||
{ id: "dns", title: "Connect a DNS provider", description: "See your zones and records, and have names matched to servers.", to: "/dns", action: "Connect", done: (dnsStats?.totalProviders ?? 0) > 0 },
|
{ id: "dns", title: "Connect a DNS provider", description: "See your zones and records, and have names matched to servers.", to: "/dns", action: "Connect", done: (dnsStats?.totalProviders ?? 0) > 0 },
|
||||||
{ id: "secret", title: "Track something that expires", description: "An API token, a certificate or a password — you're warned before it runs out.", to: "/secrets", action: "Add one", done: (secrets?.length ?? 0) > 0 },
|
{ id: "secret", title: "Track something that expires", description: "An API token, a certificate or a password — you're warned before it runs out.", to: "/secrets", action: "Add one", done: (secrets?.length ?? 0) > 0 },
|
||||||
{ id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/alert-settings?view=notifications", action: "Set up", done: notifyConfigured === true },
|
{ id: "notify", title: "Choose how you'll be told about problems", description: "Gotify, ntfy, email, a webhook or push to your phone, with the checks you want.", to: "/settings/notifications", action: "Set up", done: notifyConfigured === true },
|
||||||
{ id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 },
|
{ id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 },
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|||||||
@@ -121,15 +121,15 @@ export default function DockerVulnerabilities({ integrationId }: { integrationId
|
|||||||
<strong>Critical</strong> findings are an alert — see <Link to="/alerts">Alerts</Link>;{" "}
|
<strong>Critical</strong> findings are an alert — see <Link to="/alerts">Alerts</Link>;{" "}
|
||||||
{data.highThreshold > 0 ? (
|
{data.highThreshold > 0 ? (
|
||||||
<>
|
<>
|
||||||
so is a host with <strong>{data.highThreshold} or more high</strong> ones (<Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>); medium and low are only shown here.
|
so is a host with <strong>{data.highThreshold} or more high</strong> ones (<Link to="/settings/notifications">Settings → Notifications</Link>); medium and low are only shown here.
|
||||||
</>
|
</>
|
||||||
) : (
|
) : (
|
||||||
<>high, medium and low are only shown here (high ones can be alerted from a number you choose under Alert settings → Notifications).</>
|
<>high, medium and low are only shown here (high ones can be alerted from a number you choose under Settings → Notifications).</>
|
||||||
)}
|
)}
|
||||||
</>
|
</>
|
||||||
) : (
|
) : (
|
||||||
<>
|
<>
|
||||||
The alert for critical findings is switched off (<Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>).
|
The alert for critical findings is switched off (<Link to="/settings/notifications">Settings → Notifications</Link>).
|
||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
</p>
|
</p>
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
{user.role === "admin" ? (
|
{user.role === "admin" ? (
|
||||||
<>
|
<>
|
||||||
{" "}
|
{" "}
|
||||||
It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: <Link to="/alert-settings?view=notifications">Alert settings → Notifications</Link>.
|
It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: <Link to="/settings/notifications">Settings → Notifications</Link>.
|
||||||
</>
|
</>
|
||||||
) : null}
|
) : null}
|
||||||
</p>
|
</p>
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { Link, NavLink, Outlet } from "react-router-dom";
|
|||||||
import { searchSettings } from "../utils/settingsIndex";
|
import { searchSettings } from "../utils/settingsIndex";
|
||||||
|
|
||||||
const SUB_NAV = [
|
const SUB_NAV = [
|
||||||
|
{ to: "/settings/notifications", label: "Notifications" },
|
||||||
{ to: "/settings/sign-in", label: "Sign-in" },
|
{ to: "/settings/sign-in", label: "Sign-in" },
|
||||||
{ to: "/settings/display", label: "Display" },
|
{ to: "/settings/display", label: "Display" },
|
||||||
{ to: "/settings/names", label: "Names" },
|
{ to: "/settings/names", label: "Names" },
|
||||||
|
|||||||
@@ -11,34 +11,34 @@ export interface SettingsEntry {
|
|||||||
words: string;
|
words: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
const N = "/alert-settings?view=notifications";
|
const N = "/settings/notifications";
|
||||||
|
|
||||||
export const SETTINGS_INDEX: SettingsEntry[] = [
|
export const SETTINGS_INDEX: SettingsEntry[] = [
|
||||||
// Notifications
|
// Notifications
|
||||||
{ title: "Gotify, ntfy, email and webhook channels", where: "Alert settings › Notifications › Channels", to: `${N}&tab=channels`, words: "gotify ntfy smtp email mail webhook push channel test send where notifications go" },
|
{ title: "Gotify, ntfy, email and webhook channels", where: "Notifications › Channels", to: `${N}?tab=channels`, words: "gotify ntfy smtp email mail webhook push channel test send where notifications go" },
|
||||||
{ title: "Browser and phone push", where: "Alert settings › Notifications › Channels", to: `${N}&tab=channels`, words: "web push browser phone device subscribe pwa notify vapid" },
|
{ title: "Browser and phone push", where: "Notifications › Channels", to: `${N}?tab=channels`, words: "web push browser phone device subscribe pwa notify vapid" },
|
||||||
{ title: "Server offline after (minutes)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
|
{ title: "Server offline after (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
|
||||||
{ title: "Disk usage alert at (percent)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "disk full storage percent threshold space" },
|
{ title: "Disk usage alert at (percent)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk full storage percent threshold space" },
|
||||||
{ title: "Warn when a disk will fill within (days)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "disk forecast fill growth trend days" },
|
{ title: "Warn when a disk will fill within (days)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk forecast fill growth trend days" },
|
||||||
{ title: "Away mode: hold notifications while I'm away", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "away holiday vacation travel back hold digest pause mute do not disturb critical" },
|
{ title: "Away mode: hold notifications while I'm away", where: "Notifications › When", to: `${N}?tab=when`, words: "away holiday vacation travel back hold digest pause mute do not disturb critical" },
|
||||||
{ title: "Let critical alerts through quiet hours", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "quiet hours critical night wake sleep through urgent" },
|
{ title: "Let critical alerts through quiet hours", where: "Notifications › When", to: `${N}?tab=when`, words: "quiet hours critical night wake sleep through urgent" },
|
||||||
{ title: "Quarterly and annual reports", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "report quarterly annual yearly quarter year send message monthly" },
|
{ title: "Quarterly and annual reports", where: "Notifications › When", to: `${N}?tab=when`, words: "report quarterly annual yearly quarter year send message monthly" },
|
||||||
{ title: "Warn about critical vulnerabilities in Docker images", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events§ion=backups`, words: "vulnerability vulnerabilities cve security scan critical dockhand grype trivy image docker" },
|
{ title: "Warn about critical vulnerabilities in Docker images", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events§ion=backups`, words: "vulnerability vulnerabilities cve security scan critical dockhand grype trivy image docker" },
|
||||||
{ title: "Warn when a Docker container is broken, and containers to leave alone", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events§ion=backups`, words: "docker container restarting restart loop unhealthy stopped exited created dockhand hawser ignore leave alone" },
|
{ title: "Warn when a Docker container is broken, and containers to leave alone", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events§ion=backups`, words: "docker container restarting restart loop unhealthy stopped exited created dockhand hawser ignore leave alone" },
|
||||||
{ title: "Warn when CPU or memory is unusually high for the time", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
|
{ title: "Warn when CPU or memory is unusually high for the time", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
|
||||||
{ title: "Warn when a server starts listening on a new port", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "new port baseline listening tcp monitoring" },
|
{ title: "Warn when a server starts listening on a new port", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "new port baseline listening tcp monitoring" },
|
||||||
{ title: "Call an alert flapping after (raises in an hour)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
|
{ title: "Call an alert flapping after (raises in an hour)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
|
||||||
{ title: "Alert when a network device is offline for (minutes)", where: "Alert settings › Notifications › What to send › Servers and disks", to: `${N}&tab=events`, words: "unifi network device switch access point gateway offline" },
|
{ title: "Alert when a network device is offline for (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "unifi network device switch access point gateway offline" },
|
||||||
{ title: "Secret, Tailscale key and domain expiry reminders", where: "Alert settings › Notifications › What to send › Secrets, keys and domains", to: `${N}&tab=events§ion=secrets`, words: "expiry expire secrets certificate tailscale key domain renew warn days" },
|
{ title: "Secret, Tailscale key and domain expiry reminders", where: "Notifications › What to send › Secrets, keys and domains", to: `${N}?tab=events§ion=secrets`, words: "expiry expire secrets certificate tailscale key domain renew warn days" },
|
||||||
{ title: "Backups too old, and restore tests", where: "Alert settings › Notifications › What to send › Backups and updates", to: `${N}&tab=events§ion=backups`, words: "backup stale old restore test guest proxmox updates docker image" },
|
{ title: "Backups too old, and restore tests", where: "Notifications › What to send › Backups and updates", to: `${N}?tab=events§ion=backups`, words: "backup stale old restore test guest proxmox updates docker image" },
|
||||||
{ title: "Integration, automation and ticket alerts", where: "Alert settings › Notifications › What to send › Integrations and automation", to: `${N}&tab=events§ion=integrations`, words: "integration failing semaphore gitea automation failed osticket tickets uptime kuma" },
|
{ title: "Integration, automation and ticket alerts", where: "Notifications › What to send › Integrations and automation", to: `${N}?tab=events§ion=integrations`, words: "integration failing semaphore gitea automation failed osticket tickets uptime kuma" },
|
||||||
{ title: "DNS change notifications", where: "Alert settings › Notifications › What to send › DNS changes", to: `${N}&tab=events§ion=dns`, words: "dns changes record added removed" },
|
{ title: "DNS change notifications", where: "Notifications › What to send › DNS changes", to: `${N}?tab=events§ion=dns`, words: "dns changes record added removed" },
|
||||||
{ title: "Routing rules: which channel gets which alert", where: "Alert settings › Notifications › Who hears what", to: `${N}&tab=routing`, words: "routing rules route channel tags severity kinds mute" },
|
{ title: "Routing rules: which channel gets which alert", where: "Notifications › Who hears what", to: `${N}?tab=routing`, words: "routing rules route channel tags severity kinds mute" },
|
||||||
{ title: "Escalation: tell more people if nobody acts", where: "Alert settings › Notifications › Who hears what", to: `${N}&tab=routing`, words: "escalation escalate unacknowledged steps neglected" },
|
{ title: "Escalation: tell more people if nobody acts", where: "Notifications › Who hears what", to: `${N}?tab=routing`, words: "escalation escalate unacknowledged steps neglected" },
|
||||||
{ title: "Time zone and daily reminders", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "timezone time zone daily reminder time of day" },
|
{ title: "Time zone and daily reminders", where: "Notifications › When", to: `${N}?tab=when`, words: "timezone time zone daily reminder time of day" },
|
||||||
{ title: "Quiet hours", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "quiet hours night hold queue do not disturb" },
|
{ title: "Quiet hours", where: "Notifications › When", to: `${N}?tab=when`, words: "quiet hours night hold queue do not disturb" },
|
||||||
{ title: "Monthly report", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "monthly report day of month send availability" },
|
{ title: "Monthly report", where: "Notifications › When", to: `${N}?tab=when`, words: "monthly report day of month send availability" },
|
||||||
{ title: "Weekly digest", where: "Alert settings › Notifications › When", to: `${N}&tab=when`, words: "weekly digest summary email day" },
|
{ title: "Weekly digest", where: "Notifications › When", to: `${N}?tab=when`, words: "weekly digest summary email day" },
|
||||||
{ title: "Open osTicket tickets from alerts", where: "Alert settings › Alert tickets", to: "/alert-settings?view=tickets", words: "osticket ticket alert open api key which problems" },
|
{ title: "Open osTicket tickets from alerts", where: "Alert settings › Alert tickets", to: "/alert-settings?view=tickets", words: "osticket ticket alert open api key which problems" },
|
||||||
// Sign-in
|
// Sign-in
|
||||||
{ title: "Roles from groups", where: "Sign-in", to: "/settings/sign-in", words: "roles groups authentik oidc admin operator viewer claim map" },
|
{ title: "Roles from groups", where: "Sign-in", to: "/settings/sign-in", words: "roles groups authentik oidc admin operator viewer claim map" },
|
||||||
|
|||||||
Reference in new issue
Block a user