Put alert rules, service levels, notifications and alert tickets on one Alert settings page, in four tabs
Old addresses (/alert-rules, /service-levels, /settings/notifications) lead to the right tab. Notifications and Alert tickets stay administrator-only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
07ce2b0305
commit
4269213275
40 files changed
+212
-188
No files matched your search
@@ -187,6 +187,7 @@ All modules from the original plan are built:
|
||||
Backup page's job and run lists, the comparison table) have an Export CSV
|
||||
too, which writes the table as it is on screen — times as exact timestamps,
|
||||
action buttons left out.
|
||||
- **Alert settings** (Monitoring → Alert settings) — everything that decides what is an alert and who hears of it, on one page with four tabs: *Alert rules*, *Service levels*, *Notifications* and *Alert tickets* (`/alert-settings?view=rules|levels|notifications|tickets`). Everyone signed in sees the first two; the last two are administrators' (they used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/notifications` still lead to the right tab). Maintenance stays on its own page.
|
||||
- **Settings** (admin-only) — notification channels (Gotify, ntfy, SMTP,
|
||||
generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS
|
||||
record added/updated/deleted, daily secret-expiry reminder and daily
|
||||
@@ -221,7 +222,7 @@ Server and storage health is watched every 15 minutes: a server whose agent
|
||||
stops reporting, a server disk / Proxmox storage / Synology volume passing a
|
||||
usage threshold, and a Synology volume or disk that's degraded or failing each
|
||||
raise one notification when the problem starts and one when it clears (both
|
||||
thresholds are set under Settings → Notifications). Active problems are
|
||||
thresholds are set under Alert settings → Notifications). Active problems are
|
||||
remembered across restarts, so a rebuild doesn't re-alert them.
|
||||
|
||||
**Tags** — servers can be tagged (prod, media, rack-1, …) from their detail
|
||||
@@ -305,7 +306,7 @@ like `lab.example.se` resolves to the `example.se` registration that actually
|
||||
expires); others can be added by hand. Each is looked up daily over RDAP where
|
||||
the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what
|
||||
makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap.
|
||||
You're reminded daily from N days before expiry (Settings → Notifications,
|
||||
You're reminded daily from N days before expiry (Alert settings → Notifications,
|
||||
default 30) until it's renewed, and told when an expiry date couldn't be
|
||||
refreshed for several days. Registries that don't publish an expiry (`.de`,
|
||||
`.eu`) can be tracked but have no date to warn about. Private zones (`.lan`,
|
||||
@@ -339,7 +340,7 @@ send the notifications — so the two can't disagree — but ignores the on/off
|
||||
toggles, since it's for looking at rather than being interrupted by. Problems
|
||||
under a maintenance window stay listed, marked silenced and counted separately.
|
||||
It also looks ahead: from an agent server's own history (see its Trends card), a
|
||||
disk that is on course to be full within 14 days (Settings → Notifications → "Warn
|
||||
disk that is on course to be full within 14 days (Alert settings → Notifications → "Warn
|
||||
when a disk will fill within"; 0 turns it off) is flagged with its rate of growth
|
||||
and the days left — critical when that is three days or less. It needs at least
|
||||
three days of steady growth to say anything, skips a disk that jumped once or
|
||||
@@ -354,7 +355,7 @@ It checks live (a recent result is reused for a minute; "Check now" forces a
|
||||
fresh one), and anything it couldn't read is called out at the top rather than
|
||||
quietly treated as fine.
|
||||
|
||||
**Alert tickets** (Settings → Notifications → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the
|
||||
**Alert tickets** (Alert settings → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the
|
||||
problems on the Alerts page are looked at, and one that is new and has lasted two checks in a row gets one ticket — never a second for the same problem,
|
||||
and never one for something under a maintenance window or about osTicket itself. You choose which severities and kinds count (critical and warning by
|
||||
default), and the Alerts page shows the ticket number on each alert that has one. If many problems appear at once — a network drop that makes thirty
|
||||
@@ -455,7 +456,7 @@ alerts always stand alone.
|
||||
|
||||
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
|
||||
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
|
||||
A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also
|
||||
A **failed** test is always an alert (critical) until a newer one passes. Under Alert settings → Notifications → "A restore should have been tested within" you can also
|
||||
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
|
||||
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
|
||||
kept, `/metrics` has each subject's age and result, and everything is in the change log.
|
||||
@@ -474,7 +475,7 @@ so it is folded only when every enabled instance of that kind is explained. A se
|
||||
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
|
||||
alerts of their own, so they aren't part of this.)
|
||||
|
||||
**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
|
||||
**Who hears what** (Alert settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
|
||||
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
|
||||
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
|
||||
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
|
||||
@@ -762,7 +763,7 @@ alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n
|
||||
status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in
|
||||
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md).
|
||||
|
||||
**Service levels** (Monitoring → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
|
||||
**Service levels** (Alert settings → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
|
||||
month allows about 45 minutes; 99 % about 7 hours). A target is set on an **endpoint check** (measured from its answers, day by day), a **server** or an **integration** (measured from when the audit log recorded it going away and
|
||||
coming back, to the minute), and is measured over calendar months in the notification time zone. Each card shows the figure so far, the downtime used, what is allowed and what is left, how much of the budget is used (and where the
|
||||
pace so far would end the month), and how each of the last six months went. When **80 %** of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over — switchable per target. Planned maintenance is
|
||||
@@ -773,9 +774,9 @@ pace so far would end the month), and how each of the last six months went. When
|
||||
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
|
||||
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
|
||||
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
|
||||
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
|
||||
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Alert settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
|
||||
|
||||
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
|
||||
**Weekly digest** (Alert settings → Notifications, off until switched on) sends one message a
|
||||
week through your notification channels: what is wrong now, what was down and for how
|
||||
long, what happened, disks heading for full, things about to expire, updates waiting
|
||||
and maintenance coming up — each topic only if it has something to say, and "a quiet
|
||||
@@ -890,18 +891,18 @@ minimum in both themes. Motion is turned down for those who ask the system to.
|
||||
**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that
|
||||
nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up
|
||||
(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on
|
||||
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
|
||||
it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Alert settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
|
||||
|
||||
**Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and
|
||||
never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button;
|
||||
the same list is under Settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
|
||||
the same list is under Alert settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
|
||||
|
||||
**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's
|
||||
averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey
|
||||
**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least
|
||||
30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The
|
||||
warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by
|
||||
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*).
|
||||
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Alert settings → Notifications (*Servers and disks*).
|
||||
|
||||
**A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four
|
||||
months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too
|
||||
@@ -917,7 +918,7 @@ servers, alerts, incidents, inventory, network equipment, endpoint checks and ma
|
||||
gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab —
|
||||
its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices
|
||||
join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under
|
||||
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
|
||||
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Alert settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
|
||||
(also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md.
|
||||
|
||||
**Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then
|
||||
@@ -932,7 +933,7 @@ hours (maintenance), is handed to you or is still going is counted in the badge
|
||||
**A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits),
|
||||
switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section.
|
||||
|
||||
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
|
||||
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Alert settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
|
||||
(a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains
|
||||
power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is
|
||||
sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md.
|
||||
@@ -942,7 +943,7 @@ all find something. Removing a dependency, an admin link, a runbook, a hand-made
|
||||
what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first.
|
||||
|
||||
**Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same
|
||||
alert has been **raised three times within an hour** (Settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
|
||||
alert has been **raised three times within an hour** (Alert settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
|
||||
so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics.
|
||||
Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks)
|
||||
still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log.
|
||||
@@ -959,7 +960,7 @@ what was there before (the earlier acknowledgement text, the earlier tags, the e
|
||||
|
||||
**Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a
|
||||
badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows
|
||||
(client and server). A server whose system is past its end, or ends within **Settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
|
||||
(client and server). A server whose system is past its end, or ends within **Alert settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
|
||||
support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`.
|
||||
|
||||
**Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a
|
||||
@@ -995,7 +996,7 @@ a battery to be replaced and a load of 90 % are warnings). The UPS is also a **n
|
||||
folded under its alert ("also takes down …") instead of each being announced; the page lists what is on it in the **order to shut it down**, so nothing is switched off while something that needs it is still running. The app only
|
||||
reads — it never sends a command to the UPS. Not tried against a real UPS; see INTEGRATIONS.md.
|
||||
|
||||
**Alert rules of your own.** **Monitoring → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*,
|
||||
**Alert rules of your own.** **Alert settings → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*,
|
||||
*a disk above 80%* (every disk, or the one you name), *a disk warmer than 55 °C*. A rule says what to watch, above or below what, for how long (a reading in the history every 10–15 minutes; every reading in that time has
|
||||
to be past the limit, so a dip starts the count again), how serious it is and which servers it covers (all, one, or those with a tag). **Try it now** shows which servers it would be raised on right now before it is saved.
|
||||
What a rule finds is an ordinary alert on the Alerts page, in the *Monitoring* category with the rule's own severity — announced, recovered, held back while flapping, silenced by maintenance on the server and
|
||||
@@ -1005,7 +1006,7 @@ the disk's last report, so it is judged right now and has no time.
|
||||
**Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's
|
||||
storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside
|
||||
each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window
|
||||
(Settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
|
||||
(Alert settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
|
||||
`homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history.
|
||||
|
||||
**Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under
|
||||
@@ -1024,7 +1025,7 @@ switch it is plugged into; something that needs it; a change that touches everyt
|
||||
*close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a
|
||||
change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API.
|
||||
|
||||
**Buttons on notifications.** With **Settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
|
||||
**Buttons on notifications.** With **Alert settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
|
||||
or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as
|
||||
buttons — three on an ntfy message, two on a phone push (the browser's own notification), with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons;
|
||||
Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it
|
||||
|
||||
Reference in new issue
Block a user