Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 68b41b5586 A year of history, one chart, and a read API with personal tokens
History: server load, memory and disks are also kept as hourly (four months) and
daily (over two years) averages with lowest and highest; endpoint answers as
hourly (four months) beside the daily figures. Charts ask for 24 hours to a year
and are drawn from whichever is fine enough. A disk that two weeks call flat is
looked at again over ninety days for the forecast. Built once from what is kept.

One chart: TimeChart replaces the sparkline and the multi-line chart: hover and
arrow-key readout, a lowest/highest band, gaps shown as gaps, the same ranges
everywhere and the choice remembered. Used for server trends, comparing servers,
a check's availability and answer times (Endpoints > History) and a server's
availability.

Read API: /api/v1 (servers, metrics, alerts, incidents, inventory, network,
endpoints and their history, maintenance) with personal read-only tokens made
under the user menu: shown once, hashed, 30/90/365 days or none, ten at most,
removed with their owner. GET only. Documented in API.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 20:22:58 +02:00

276 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Metrics, webhooks, heartbeats and a calendar feed
Ways for other systems to work with Homelab Manager:
- a read-only **`/metrics`** address for Prometheus, and **event webhooks** that are sent a signed message when something happens — both set up by an administrator under **Settings → API and webhooks** (the first two sections below);
- a **read API** (`/api/v1`) with a personal, read-only token for each person, for scripts, Home Assistant and dashboards (the section after `/metrics`);
- **heartbeats**: a cron job or script calls an address when it finishes, and the app speaks up when it doesn't (further down);
- a **calendar feed** of what is coming up, for a phone's calendar to subscribe to (at the end).
## `/metrics` — for Prometheus
```
GET /metrics
Authorization: Bearer hlm_m_…
```
- A **metrics token** is made on the settings page. It is shown once, only its SHA-256 is kept, and it can
be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
- The token goes in the `Authorization` header **only**. A token in the address (`?token=…`) is not
accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same
`401`.
- The answer is Prometheus's text format, `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
Requests are limited to 120 a minute from one address.
- It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few
database reads. (The alert counts are the app's own result, reused for up to a minute.)
```yaml
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
static_configs:
- targets: ["homelab.example.lan"]
```
### What it says
All gauges. Labels are escaped as Prometheus requires.
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. |
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`.
## The read API — `/api/v1`
JSON about servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance — for a script, Home Assistant, a dashboard. It only reads.
```
GET /api/v1/alerts
Authorization: Bearer hlm_p_…
```
- **A personal token**, made by whoever is going to use it under the user menu → **API tokens** (everyone signed in can, whatever their role). It is shown once, only its SHA-256 is kept, and it is that person's own: they see
and remove theirs, nobody else's. It can last 30, 90 or 365 days or until removed, a person may have ten, and it stops working at once when removed — and when the person is.
- The token goes in the `Authorization` header **only**. A token in the address is not accepted (addresses end up in logs). No token, a wrong one, an expired one and a removed one all get the same `401` with a JSON body.
- **Only `GET`** (and `HEAD`). Any other verb is `405`, with a token or without: there is nothing here that changes anything.
- A token sees what its owner sees on the pages — no secrets, no settings, no passwords, no tokens of servers. Requests are limited to 300 a minute from one address. Answers are `Cache-Control: no-store`.
- Dates are ISO 8601 in UTC, sizes are bytes, percentages are 0–100. An address inside the app (`url`) is a full address, built from `APP_BASE_URL`.
| Address | What it answers |
|---|---|
| `GET /api/v1` | The list of these addresses. |
| `GET /api/v1/me` | Whose token this is: the person's name and role, and the token's name and end. |
| `GET /api/v1/servers` | Every server: `status` (`online`, `offline`, `unknown`), `lastSeenAt`, `tags`, `ipAddresses`, `cpu`, `memory`, `disks` (with `usedPercent`), `updates`, `hardware`, `url`. Filter with `?tag=` and `?status=`. |
| `GET /api/v1/servers/:id` | One server, the same shape. |
| `GET /api/v1/servers/:id/metrics?days=` | Its load, memory and disks as lines of points, 1 to 365 days (7 by default). Points over more than 35 days are hours' or days' averages, with `lo` and `hi`. Disks also carry a `trend` and, when rising steadily, `daysToFull`. |
| `GET /api/v1/alerts` | What is wrong now: `needingAttention` (critical, warning, acknowledged), `counts`, `couldntCheck`, and `alerts` with `severity`, `category`, `message`, `silenced`, `flapping`, `target`, `causedBy`, `acknowledgement` (who, when, note, until) and `url`. Filter with `?severity=`, `?category=`, `?acknowledged=true|false`, `?silenced=true|false`. |
| `GET /api/v1/incidents?status=&limit=` | The record of alerts that began and ended: `open` or `resolved`, newest first, up to 500 (100 by default). |
| `GET /api/v1/incidents/:id` | One incident with its timeline. |
| `GET /api/v1/inventory` | The hardware: maker, model, serial, where, bought, warranty, end of support, power. |
| `GET /api/v1/network` | Switches, access points and gateways read from UniFi (state, address, uptime, firmware, uplink), the networks and Wi-Fi names, and which server is plugged into which device. |
| `GET /api/v1/endpoints` | The checks: `status` (`up`, `down`, `unknown`, `paused`), last answer, certificate end, 24-hour and 7-day uptime. |
| `GET /api/v1/endpoints/:id/history?days=` | A check's availability and answer times over 1 to 365 days, as points, from each answer (up to 2 days), the hourly figures (up to 120) or the daily ones. |
| `GET /api/v1/maintenance` | What is under maintenance now (`active`) and what is coming (`upcoming`). |
```json
{ "generatedAt": "2026-10-20T09:14:02.511Z",
"needingAttention": { "critical": 1, "warning": 0, "acknowledged": 1 },
"alerts": [
{ "id": "offline:offline:server:3", "severity": "critical", "category": "offline", "source": "Server",
"message": "anna hasn't reported for 9 h", "silenced": false, "flapping": false,
"target": { "type": "server", "id": 3, "name": "anna" }, "causedBy": null, "affects": [],
"acknowledgement": { "by": "Ada", "at": "2026-10-20T08:01:00.000Z", "note": "On it", "until": "2026-10-20T12:01:00.000Z", "assignee": "Ada" },
"url": "https://hm.example.lan/servers/3" } ] }
```
Errors are JSON too: `{ "error": "unauthorized" | "read_only" | "not_found" | "invalid_id", "message": … }`. In Home Assistant, a `rest` sensor with the token in `headers:` and `value_template: "{{ value_json.needingAttention.critical }}"` is enough for a count of what needs attention.
## A Grafana dashboard
Settings → API and webhooks → **Download the dashboard** (administrators) gives `homelab-manager-dashboard.json`; the same file is at
`GET /api/metrics-tokens/grafana-dashboard.json` for a signed-in administrator. In Grafana: **Dashboards → New → Import**, upload it, and
choose your Prometheus when asked (the file never names one, so it isn't tied to any installation). It is a dashboard with a **Server**
picker (taken from `homelab_server_up`) and five rows:
| Row | What it has |
|---|---|
| Needs attention | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
| Servers | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
| Disks | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
| Backups | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
| Secrets, power and the app itself | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. |
The dashboard is built by the app from the same code that serves `/metrics`, and a test checks that it only asks for series that are
really served — so after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later
and has been checked for structure (every panel on the grid, every query well-formed, every series real), **not** opened in a Grafana here.
## Event webhooks
A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is
separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person
would read.
| Event | Sent when |
|---|---|
| `alert.opened` | A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. |
| `alert.resolved` | A problem ends. |
| `maintenance.started` | A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). |
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails
(the same message, with the same `X-HLM-Delivery`), then the result is noted on the webhook in Settings
(a status code, or what went wrong). A request is given five seconds, and redirects are not followed.
### The message
```
POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
```
```json
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
```
- `alert.opened` data: `alert` with `key` (stable for the same problem), `source`, `severity` (`critical` or
`warning`) and `message`. `alert.resolved` has `alert.key` and `alert.message`.
- `maintenance.started` / `maintenance.ended` data: `window` with `id`, `targetType` (`server`,
`integration`, `dns_provider`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source`
(`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime
Kuma) and, on `ended`, `endedAt`.
- A **Test** button on the settings page sends `{"event": "test", …}` whatever the webhook is set to.
### Checking the signature
With a secret, `X-HLM-Signature` is `sha256=` and the hex HMAC-SHA256 of the **exact body bytes** with the
secret as the key. Compare in constant time, and reject a message whose signature doesn't match.
```js
import { createHmac, timingSafeEqual } from "node:crypto";
function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
```
```python
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
```
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
its address or secret.
## Heartbeats — for cron jobs and scripts
A heartbeat is made by an administrator under **Monitoring → Heartbeats**, and its address is shown there to administrators. The job calls it when it finishes:
```
GET /api/heartbeat/<token> it worked
GET /api/heartbeat/<token>/start it began (optional: lets the next call say how long it took)
GET /api/heartbeat/<token>/fail it ran and went wrong: an alert at once
```
```sh
# in cron, after the job
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx
# in a script, with a start, a failure and a note
curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start
./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \
|| curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail
```
- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work.
Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once).
- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters.
- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address.
- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes;
a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`.
- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them.
## Calendar feed
An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL").
```
GET /api/public/calendar/<token>.ics
```
- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time.
A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address.
- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and
recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out.
- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app.
An event's `UID` is a hash of what it is and its date, the same on every fetch.
- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does.