Heartbeats, Coming up and suggested checks; Operations menu tidied
Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app alerts when the call is late or the job says it failed. Coming up: one agenda of everything dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests, waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested checks: web ports and certificates the app already knows about, added with a click. Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group (Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow the real clock. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c05c847d17
commit
4e0c6927b2
44 files changed
+7092
-75
No files matched your search
@@ -16,7 +16,7 @@ Authentik (OIDC), with local admin/operator/viewer roles.
|
||||
|
||||
All modules from the original plan are built:
|
||||
|
||||
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
|
||||
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Monitoring, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
|
||||
- Authentik OIDC login, roles (first user to sign in becomes admin), audit log
|
||||
(changes made in the app, sign-ins and sign-outs with the IP they came from, new accounts, and the log's own
|
||||
automatic trimming — attributed to "system"; settings changes show what changed, but never credentials)
|
||||
@@ -95,7 +95,7 @@ All modules from the original plan are built:
|
||||
start/stop/restart buttons right on the detail page. The detail page also has an **Admin Links** section
|
||||
(operator/admin to add/edit/remove) for bookmarking that server's own
|
||||
admin UIs — Dockge, Webmin, Cockpit, Portainer, or anything else reachable
|
||||
by URL. **Operations → Admin Links** summarizes every server's admin links
|
||||
by URL. **Infrastructure → Admin Links** summarizes every server's admin links
|
||||
in one sortable, searchable table (server, label, URL, an "Open" link, and
|
||||
the same add/edit/delete as the per-server section — adding one here just
|
||||
asks which server it belongs to), so finding or managing one doesn't mean
|
||||
@@ -220,7 +220,7 @@ offered as one-click suggestions when tagging), give any tag a colour of your ch
|
||||
that already exists merges the two, and both rename and delete rewrite every server
|
||||
that carries the tag.
|
||||
|
||||
**Name generator** — Operations → Generator suggests server names (and usernames and
|
||||
**Name generator** — Administration → Generator suggests server names (and usernames and
|
||||
passwords, which are made in your browser and never stored). Server names are picked from
|
||||
name lists: Swedish girl and boy names, Disney and Pixar characters, Norse mythology and
|
||||
Astrid Lindgren to start with, or "Mixed" for all of them together. A name already used
|
||||
@@ -507,7 +507,7 @@ the alert back among the unattended; it goes by itself once the alert has been g
|
||||
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
|
||||
server's history.
|
||||
|
||||
**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an
|
||||
**Incidents** (Operations → Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an
|
||||
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
|
||||
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
|
||||
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
|
||||
@@ -561,7 +561,7 @@ text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a
|
||||
of commands, backticks inline code; links become clickable, and nothing typed in a
|
||||
note can run as HTML.
|
||||
|
||||
**Endpoint checks** (Network → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
|
||||
**Endpoint checks** (Monitoring → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
|
||||
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
|
||||
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
|
||||
also want a word in the page; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
|
||||
@@ -571,6 +571,32 @@ server, so maintenance on it silences the check's alerts and the server's tags c
|
||||
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
|
||||
dashboard and in `/metrics`. Results are kept for two weeks.
|
||||
|
||||
**Suggested checks** (on the Endpoints page, administrators). From what the app already knows it offers checks you could add with a click: a **web port a server's agent
|
||||
sees open** (80, 443, 8080, 8443 — when a server has both the plain and the secure one, the secure one, since it also reads the certificate), under the name DNS gives that
|
||||
server if there is one (an A or AAAA record with one of its addresses, or an alias to it), otherwise its own hostname or address; and a **secret that already reads a
|
||||
certificate**, since a check also hears when the service stops answering. Each is linked to its server. Something a check already covers is never offered, **Not for me**
|
||||
stops one being offered (and they can be brought back), and nothing is asked until you add it. Uptime Kuma's monitors are not offered: they are already being watched, and
|
||||
the adapter only keeps a host and port for them, not the address.
|
||||
|
||||
**Heartbeats** (Monitoring → Heartbeats). "Did my 3 a.m. job actually run?" An administrator makes a heartbeat for a job — how often it should report, and how late is
|
||||
still fine — and gets an address with a long random token in it. The job calls it when it finishes (`curl -fsS -m 10 -o /dev/null <address>` after the backup in cron);
|
||||
the app writes that down and shows when the next call is due, the run times and the last runs as a row of ticks. A call that does **not come in time** is a *warning*
|
||||
alert and one notification (the same bookkeeping as everything else: said once, again as cleared when the call comes); a job that calls `<address>/fail` — it ran and went
|
||||
wrong — is a *critical* alert at once, with what it said. `/start` before the job lets the finishing call say how long it took; anything sent with a call
|
||||
(`--data-raw "3.2 GB copied"`, or `?msg=`) is kept as a note, up to 200 characters. A heartbeat can be linked to the server the job runs on, so maintenance there silences
|
||||
it and the server's tags route it. Everyone can look; only administrators see the addresses, make, change and remove heartbeats, or give one a **new address** if it got out.
|
||||
The address needs no sign-in (it is a token, like an agent's), and all it can do is say "I'm alive": it can't read anything. Calls are limited per address, and an
|
||||
unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [API.md](API.md).
|
||||
|
||||
**Coming up** (Operations → Coming up, and a card on the dashboard). Everything with a date ahead in one list, nearest first: secrets, domains and certificates that
|
||||
expire (including the ones endpoint checks see), Tailscale keys that are expiring, hardware warranties and end of support, planned and **recurring** maintenance, restore
|
||||
tests that are due (when a time limit is set), and — as one line — updates that have been waiting. What has already passed in the last 30 days is shown as past, under
|
||||
*Needs attention now*. Narrow it by kind, server and how far ahead (up to a year); the filters are in the address, so a view can be linked. A schedule that repeats weekly is one line (its next time) unless
|
||||
you ask for every one. An administrator can make a **calendar feed** (Settings → Share links → *A calendar feed*): an address for a phone's calendar to subscribe to, refreshed
|
||||
every few hours, with each day as an all-day event and maintenance with its start and end. A feed made with *kinds only* says only "A secret expires", with no names, so it is fine on a shared
|
||||
calendar; *with names* also gives the name and a link back. Like any share link the address is shown once and can be removed. Tailscale keys are asked for at most every
|
||||
ten minutes, and only those the integration reports as expiring soon are listed.
|
||||
|
||||
**Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed
|
||||
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
|
||||
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox
|
||||
|
||||
Reference in new issue
Block a user