Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 5cc51c57de Add status pages (a share link kind) and a downloadable Grafana dashboard
- Share links can open a status page: an admin picks servers, integrations and Uptime Kuma
  monitors and names them; the public sees only those names, Working / Not working /
  Being worked on / Can't tell, an optional message and the 30-day figure (share_links.kind/
  title/message and status_items, migration 0033). A status address doesn't open the wall and
  the other way round; Uptime Kuma is asked at most every 30 seconds
- Grafana dashboard built from code (download in Settings -> API and webhooks); a test checks
  it only uses series /metrics serves
- Tests and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 15:38:08 +02:00

176 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Metrics and webhooks
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
**Settings → API and webhooks**: a read-only `/metrics` address for Prometheus, and webhooks that are
sent a signed message when something happens.
## `/metrics` — for Prometheus
```
GET /metrics
Authorization: Bearer hlm_m_…
```
- A **metrics token** is made on the settings page. It is shown once, only its SHA-256 is kept, and it can
be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
- The token goes in the `Authorization` header **only**. A token in the address (`?token=…`) is not
accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same
`401`.
- The answer is Prometheus's text format, `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
Requests are limited to 120 a minute from one address.
- It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few
database reads. (The alert counts are the app's own result, reused for up to a minute.)
```yaml
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
static_configs:
- targets: ["homelab.example.lan"]
```
### What it says
All gauges. Labels are escaped as Prometheus requires.
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. |
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`.
## A Grafana dashboard
Settings → API and webhooks → **Download the dashboard** (administrators) gives `homelab-manager-dashboard.json`; the same file is at
`GET /api/metrics-tokens/grafana-dashboard.json` for a signed-in administrator. In Grafana: **Dashboards → New → Import**, upload it, and
choose your Prometheus when asked (the file never names one, so it isn't tied to any installation). It is a dashboard with a **Server**
picker (taken from `homelab_server_up`) and five rows:
| Row | What it has |
|---|---|
| Needs attention | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
| Servers | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
| Disks | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
| Backups | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
| Secrets, power and the app itself | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. |
The dashboard is built by the app from the same code that serves `/metrics`, and a test checks that it only asks for series that are
really served — so after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later
and has been checked for structure (every panel on the grid, every query well-formed, every series real), **not** opened in a Grafana here.
## Event webhooks
A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is
separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person
would read.
| Event | Sent when |
|---|---|
| `alert.opened` | A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. |
| `alert.resolved` | A problem ends. |
| `maintenance.started` | A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). |
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails
(the same message, with the same `X-HLM-Delivery`), then the result is noted on the webhook in Settings
(a status code, or what went wrong). A request is given five seconds, and redirects are not followed.
### The message
```
POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
```
```json
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
```
- `alert.opened` data: `alert` with `key` (stable for the same problem), `source`, `severity` (`critical` or
`warning`) and `message`. `alert.resolved` has `alert.key` and `alert.message`.
- `maintenance.started` / `maintenance.ended` data: `window` with `id`, `targetType` (`server`,
`integration`, `dns_provider`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source`
(`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime
Kuma) and, on `ended`, `endedAt`.
- A **Test** button on the settings page sends `{"event": "test", …}` whatever the webhook is set to.
### Checking the signature
With a secret, `X-HLM-Signature` is `sha256=` and the hex HMAC-SHA256 of the **exact body bytes** with the
secret as the key. Compare in constant time, and reject a message whose signature doesn't match.
```js
import { createHmac, timingSafeEqual } from "node:crypto";
function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
```
```python
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
```
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
its address or secret.