- Backups: a guest whose newest backup is older than its limit is an alert (backup kind), in the daily backup message, the digest and /metrics (homelab_backup_guests, ..._last_age_seconds) - Per-server thresholds (servers.thresholds, migration 0030): offline minutes, disk percent, forecast window, backup age, and per-disk limits or "leave alone"; applied everywhere the global ones were; saving clears the alerts cache - Update runs (update_runs, migration 0031): Update... on the Updates page puts the server under maintenance, runs a Semaphore playbook limited to it, waits for the host to report, and reports what is left and whether a restart is due; poller survives restarts - Notification toggle for update-run results; tests and docs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
152 lines
8.2 KiB
Markdown
152 lines
8.2 KiB
Markdown
# Metrics and webhooks
|
||
|
||
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
|
||
**Settings → API and webhooks**: a read-only `/metrics` address for Prometheus, and webhooks that are
|
||
sent a signed message when something happens.
|
||
|
||
## `/metrics` — for Prometheus
|
||
|
||
```
|
||
GET /metrics
|
||
Authorization: Bearer hlm_m_…
|
||
```
|
||
|
||
- A **metrics token** is made on the settings page. It is shown once, only its SHA-256 is kept, and it can
|
||
be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
|
||
- The token goes in the `Authorization` header **only**. A token in the address (`?token=…`) is not
|
||
accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same
|
||
`401`.
|
||
- The answer is Prometheus's text format, `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
|
||
Requests are limited to 120 a minute from one address.
|
||
- It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few
|
||
database reads. (The alert counts are the app's own result, reused for up to a minute.)
|
||
|
||
```yaml
|
||
scrape_configs:
|
||
- job_name: homelab-manager
|
||
scheme: https
|
||
metrics_path: /metrics
|
||
authorization:
|
||
credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
|
||
static_configs:
|
||
- targets: ["homelab.example.lan"]
|
||
```
|
||
|
||
### What it says
|
||
|
||
All gauges. Labels are escaped as Prometheus requires.
|
||
|
||
| Series | Labels | Meaning |
|
||
|---|---|---|
|
||
| `homelab_info` | `environment` | Always 1; says which installation this is. |
|
||
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
|
||
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
|
||
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
|
||
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
|
||
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
|
||
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
|
||
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
|
||
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
|
||
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
|
||
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
|
||
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
|
||
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
|
||
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
|
||
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
|
||
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
|
||
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
|
||
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
|
||
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
|
||
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
|
||
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
|
||
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
|
||
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
|
||
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
|
||
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
|
||
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
|
||
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
|
||
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
|
||
|
||
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
|
||
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`.
|
||
|
||
## Event webhooks
|
||
|
||
A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is
|
||
separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person
|
||
would read.
|
||
|
||
| Event | Sent when |
|
||
|---|---|
|
||
| `alert.opened` | A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. |
|
||
| `alert.resolved` | A problem ends. |
|
||
| `maintenance.started` | A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). |
|
||
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
|
||
|
||
Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails
|
||
(the same message, with the same `X-HLM-Delivery`), then the result is noted on the webhook in Settings
|
||
(a status code, or what went wrong). A request is given five seconds, and redirects are not followed.
|
||
|
||
### The message
|
||
|
||
```
|
||
POST <your url>
|
||
Content-Type: application/json
|
||
User-Agent: HomelabManager-Webhook
|
||
X-HLM-Event: alert.opened
|
||
X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message)
|
||
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
|
||
```
|
||
|
||
```json
|
||
{
|
||
"event": "alert.opened",
|
||
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
|
||
"createdAt": "2026-10-17T21:04:11.482Z",
|
||
"source": "homelab-manager",
|
||
"data": {
|
||
"alert": {
|
||
"key": "offline:server:3",
|
||
"source": "server",
|
||
"severity": "critical",
|
||
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
- `alert.opened` data: `alert` with `key` (stable for the same problem), `source`, `severity` (`critical` or
|
||
`warning`) and `message`. `alert.resolved` has `alert.key` and `alert.message`.
|
||
- `maintenance.started` / `maintenance.ended` data: `window` with `id`, `targetType` (`server`,
|
||
`integration`, `dns_provider`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source`
|
||
(`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime
|
||
Kuma) and, on `ended`, `endedAt`.
|
||
- A **Test** button on the settings page sends `{"event": "test", …}` whatever the webhook is set to.
|
||
|
||
### Checking the signature
|
||
|
||
With a secret, `X-HLM-Signature` is `sha256=` and the hex HMAC-SHA256 of the **exact body bytes** with the
|
||
secret as the key. Compare in constant time, and reject a message whose signature doesn't match.
|
||
|
||
```js
|
||
import { createHmac, timingSafeEqual } from "node:crypto";
|
||
|
||
function isFromHomelabManager(secret, rawBody, signatureHeader) {
|
||
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
|
||
const given = Buffer.from(signatureHeader ?? "");
|
||
return given.length === expected.length && timingSafeEqual(given, expected);
|
||
}
|
||
```
|
||
|
||
```python
|
||
import hmac, hashlib
|
||
|
||
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
|
||
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
|
||
return hmac.compare_digest(expected, signature_header or "")
|
||
```
|
||
|
||
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
|
||
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
|
||
its address or secret.
|