Heartbeats, Coming up and suggested checks; Operations menu tidied

Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app
alerts when the call is late or the job says it failed. Coming up: one agenda of everything
dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests,
waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested
checks: web ports and certificates the app already knows about, added with a click.

Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group
(Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under
Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow
the real clock.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-19 21:42:34 +02:00
1 parent c05c847d17
commit 4e0c6927b2
44 files changed
+7092 -75

No files matched your search

+31 -5
View File
@@ -16,7 +16,7 @@ Authentik (OIDC), with local admin/operator/viewer roles.
All modules from the original plan are built:
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Monitoring, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
- Authentik OIDC login, roles (first user to sign in becomes admin), audit log
(changes made in the app, sign-ins and sign-outs with the IP they came from, new accounts, and the log's own
automatic trimming — attributed to "system"; settings changes show what changed, but never credentials)
@@ -95,7 +95,7 @@ All modules from the original plan are built:
start/stop/restart buttons right on the detail page. The detail page also has an **Admin Links** section
(operator/admin to add/edit/remove) for bookmarking that server's own
admin UIs — Dockge, Webmin, Cockpit, Portainer, or anything else reachable
by URL. **Operations → Admin Links** summarizes every server's admin links
by URL. **Infrastructure → Admin Links** summarizes every server's admin links
in one sortable, searchable table (server, label, URL, an "Open" link, and
the same add/edit/delete as the per-server section — adding one here just
asks which server it belongs to), so finding or managing one doesn't mean
@@ -220,7 +220,7 @@ offered as one-click suggestions when tagging), give any tag a colour of your ch
that already exists merges the two, and both rename and delete rewrite every server
that carries the tag.
**Name generator** — Operations → Generator suggests server names (and usernames and
**Name generator** — Administration → Generator suggests server names (and usernames and
passwords, which are made in your browser and never stored). Server names are picked from
name lists: Swedish girl and boy names, Disney and Pixar characters, Norse mythology and
Astrid Lindgren to start with, or "Mixed" for all of them together. A name already used
@@ -507,7 +507,7 @@ the alert back among the unattended; it goes by itself once the alert has been g
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
server's history.
**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an
**Incidents** (Operations → Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
@@ -561,7 +561,7 @@ text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a
of commands, backticks inline code; links become clickable, and nothing typed in a
note can run as HTML.
**Endpoint checks** (Network → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
**Endpoint checks** (Monitoring → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
also want a word in the page; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
@@ -571,6 +571,32 @@ server, so maintenance on it silences the check's alerts and the server's tags c
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
dashboard and in `/metrics`. Results are kept for two weeks.
**Suggested checks** (on the Endpoints page, administrators). From what the app already knows it offers checks you could add with a click: a **web port a server's agent
sees open** (80, 443, 8080, 8443 — when a server has both the plain and the secure one, the secure one, since it also reads the certificate), under the name DNS gives that
server if there is one (an A or AAAA record with one of its addresses, or an alias to it), otherwise its own hostname or address; and a **secret that already reads a
certificate**, since a check also hears when the service stops answering. Each is linked to its server. Something a check already covers is never offered, **Not for me**
stops one being offered (and they can be brought back), and nothing is asked until you add it. Uptime Kuma's monitors are not offered: they are already being watched, and
the adapter only keeps a host and port for them, not the address.
**Heartbeats** (Monitoring → Heartbeats). "Did my 3 a.m. job actually run?" An administrator makes a heartbeat for a job — how often it should report, and how late is
still fine — and gets an address with a long random token in it. The job calls it when it finishes (`curl -fsS -m 10 -o /dev/null <address>` after the backup in cron);
the app writes that down and shows when the next call is due, the run times and the last runs as a row of ticks. A call that does **not come in time** is a *warning*
alert and one notification (the same bookkeeping as everything else: said once, again as cleared when the call comes); a job that calls `<address>/fail` — it ran and went
wrong — is a *critical* alert at once, with what it said. `/start` before the job lets the finishing call say how long it took; anything sent with a call
(`--data-raw "3.2 GB copied"`, or `?msg=`) is kept as a note, up to 200 characters. A heartbeat can be linked to the server the job runs on, so maintenance there silences
it and the server's tags route it. Everyone can look; only administrators see the addresses, make, change and remove heartbeats, or give one a **new address** if it got out.
The address needs no sign-in (it is a token, like an agent's), and all it can do is say "I'm alive": it can't read anything. Calls are limited per address, and an
unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [API.md](API.md).
**Coming up** (Operations → Coming up, and a card on the dashboard). Everything with a date ahead in one list, nearest first: secrets, domains and certificates that
expire (including the ones endpoint checks see), Tailscale keys that are expiring, hardware warranties and end of support, planned and **recurring** maintenance, restore
tests that are due (when a time limit is set), and — as one line — updates that have been waiting. What has already passed in the last 30 days is shown as past, under
*Needs attention now*. Narrow it by kind, server and how far ahead (up to a year); the filters are in the address, so a view can be linked. A schedule that repeats weekly is one line (its next time) unless
you ask for every one. An administrator can make a **calendar feed** (Settings → Share links → *A calendar feed*): an address for a phone's calendar to subscribe to, refreshed
every few hours, with each day as an all-day event and maintenance with its start and end. A feed made with *kinds only* says only "A secret expires", with no names, so it is fine on a shared
calendar; *with names* also gives the name and a link back. Like any share link the address is shown once and can be removed. Tailscale keys are asked for at most every
ten minutes, and only those the integration reports as expiring soon are listed.
**Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox