Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 5dc6cea69c Host health from the Linux agent, agent self-update, and disk forecast in /metrics
- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates;
  SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox
- Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new
  "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series
- Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified
  download, atomic replace; GET /api/agent/report/update
- homelab_server_disk_days_to_full in /metrics
- Tests (server, real PowerShell against a fake server) and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 00:54:24 +02:00

150 lines
7.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Metrics and webhooks
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
**Settings → API and webhooks**: a read-only `/metrics` address for Prometheus, and webhooks that are
sent a signed message when something happens.
## `/metrics` — for Prometheus
```
GET /metrics
Authorization: Bearer hlm_m_…
```
- A **metrics token** is made on the settings page. It is shown once, only its SHA-256 is kept, and it can
be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
- The token goes in the `Authorization` header **only**. A token in the address (`?token=…`) is not
accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same
`401`.
- The answer is Prometheus's text format, `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
Requests are limited to 120 a minute from one address.
- It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few
database reads. (The alert counts are the app's own result, reused for up to a minute.)
```yaml
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
static_configs:
- targets: ["homelab.example.lan"]
```
### What it says
All gauges. Labels are escaped as Prometheus requires.
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`.
## Event webhooks
A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is
separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person
would read.
| Event | Sent when |
|---|---|
| `alert.opened` | A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. |
| `alert.resolved` | A problem ends. |
| `maintenance.started` | A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). |
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails
(the same message, with the same `X-HLM-Delivery`), then the result is noted on the webhook in Settings
(a status code, or what went wrong). A request is given five seconds, and redirects are not followed.
### The message
```
POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
```
```json
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
```
- `alert.opened` data: `alert` with `key` (stable for the same problem), `source`, `severity` (`critical` or
`warning`) and `message`. `alert.resolved` has `alert.key` and `alert.message`.
- `maintenance.started` / `maintenance.ended` data: `window` with `id`, `targetType` (`server`,
`integration`, `dns_provider`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source`
(`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime
Kuma) and, on `ended`, `endedAt`.
- A **Test** button on the settings page sends `{"event": "test", …}` whatever the webhook is set to.
### Checking the signature
With a secret, `X-HLM-Signature` is `sha256=` and the hex HMAC-SHA256 of the **exact body bytes** with the
secret as the key. Compare in constant time, and reject a message whose signature doesn't match.
```js
import { createHmac, timingSafeEqual } from "node:crypto";
function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
```
```python
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
```
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
its address or secret.