Heartbeats, Coming up and suggested checks; Operations menu tidied
Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app alerts when the call is late or the job says it failed. Coming up: one agenda of everything dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests, waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested checks: web ports and certificates the app already knows about, added with a click. Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group (Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow the real clock. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c05c847d17
commit
4e0c6927b2
44 files changed
+7092
-75
No files matched your search
@@ -1,8 +1,10 @@
|
||||
# Metrics and webhooks
|
||||
# Metrics, webhooks, heartbeats and a calendar feed
|
||||
|
||||
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
|
||||
**Settings → API and webhooks**: a read-only `/metrics` address for Prometheus, and webhooks that are
|
||||
sent a signed message when something happens.
|
||||
Ways for other systems to work with Homelab Manager:
|
||||
|
||||
- a read-only **`/metrics`** address for Prometheus, and **event webhooks** that are sent a signed message when something happens — both set up by an administrator under **Settings → API and webhooks** (the first two sections below);
|
||||
- **heartbeats**: a cron job or script calls an address when it finishes, and the app speaks up when it doesn't (further down);
|
||||
- a **calendar feed** of what is coming up, for a phone's calendar to subscribe to (at the end).
|
||||
|
||||
## `/metrics` — for Prometheus
|
||||
|
||||
@@ -72,6 +74,9 @@ All gauges. Labels are escaped as Prometheus requires.
|
||||
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
|
||||
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
||||
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
||||
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
||||
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
|
||||
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
|
||||
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
|
||||
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
|
||||
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
|
||||
@@ -178,3 +183,47 @@ def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str)
|
||||
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
|
||||
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
|
||||
its address or secret.
|
||||
|
||||
## Heartbeats — for cron jobs and scripts
|
||||
|
||||
A heartbeat is made by an administrator under **Monitoring → Heartbeats**, and its address is shown there to administrators. The job calls it when it finishes:
|
||||
|
||||
```
|
||||
GET /api/heartbeat/<token> it worked
|
||||
GET /api/heartbeat/<token>/start it began (optional: lets the next call say how long it took)
|
||||
GET /api/heartbeat/<token>/fail it ran and went wrong: an alert at once
|
||||
```
|
||||
|
||||
```sh
|
||||
# in cron, after the job
|
||||
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx
|
||||
|
||||
# in a script, with a start, a failure and a note
|
||||
curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start
|
||||
./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \
|
||||
|| curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail
|
||||
```
|
||||
|
||||
- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work.
|
||||
Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once).
|
||||
- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters.
|
||||
- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address.
|
||||
- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes;
|
||||
a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`.
|
||||
- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them.
|
||||
|
||||
## Calendar feed
|
||||
|
||||
An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL").
|
||||
|
||||
```
|
||||
GET /api/public/calendar/<token>.ics
|
||||
```
|
||||
|
||||
- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time.
|
||||
A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address.
|
||||
- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and
|
||||
recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out.
|
||||
- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app.
|
||||
An event's `UID` is a hash of what it is and its date, the same on every fetch.
|
||||
- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does.
|
||||
Reference in new issue
Block a user