Documentation: guides in docs/, and the reference docs brought up to date

- New docs/ folder: SSO, API, Prometheus and Grafana, webhooks, heartbeats and calendar, how monitoring works, and an index; the old root API.md is split across them
- Read API: alerts carry snoozedUntil
- DATABASE.md: table and migration counts, indexes, relations, integration config, settings and internal flags, delete rules, retention
- ROLES.md: the sidebar table rebuilt from the menu
- NOTIFICATIONS.md, INTEGRATIONS.md, BACKUP.md, AGENT_UPDATE.md: what had drifted
- README: links to the guides; accounts cannot be deleted in the app

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-21 00:09:43 +02:00
1 parent 24ae8db380
commit c2e76262ca
17 files changed
+1773 -334

No files matched your search

+377
View File
@@ -0,0 +1,377 @@
# API
Homelab Manager has two kinds of programmatic interface:
| Interface | For | Who can use it | Section |
|---|---|---|---|
| **The read API** — `/api/v1` | Scripts, Home Assistant, dashboards that want JSON about servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance | Anyone signed in can make a personal, read-only token | [The read API](#the-read-api) |
| **Machine addresses** — agent reports, heartbeats, `/metrics`, share links | The agents, cron jobs, Prometheus, status pages and calendars | Each has its own token | [Other addresses for machines](#other-addresses-for-machines) |
**There is no API that changes anything.** The read API refuses every verb but `GET` and `HEAD`. Everything else the app does (acknowledge an alert, start maintenance, add a check, change a setting)
is done in the web interface, which is not a supported API: its addresses and shapes change with the pages. The read API's shapes are a deliberate, stable contract.
Examples below use `https://hm.example.lan` for your `APP_BASE_URL`. Related pages: [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) for `/metrics`, [WEBHOOKS.md](WEBHOOKS.md) for being *told* when something
happens instead of asking, [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md), [MONITORING.md](MONITORING.md) for what the alert fields mean.
## The read API
### Getting a token
1. Sign in, open the **user menu → API tokens**. This is open to everyone signed in, whatever their role.
2. Give the token a name (e.g. `Home Assistant`) and choose how long it lasts: **30, 90 or 365 days, or until removed**.
3. Copy the token (it starts with `hlm_p_`). **It is shown once**; only its SHA-256 is kept.
A token belongs to the person who made it. They see and remove their own, nobody else's. A person can have **ten**. A token **stops working at once** when it is removed, when it expires and when
its owner is removed. It sees what its owner sees on the pages: no secrets, no passwords, no server tokens, no settings.
### Making a request
```bash
curl -H "Authorization: Bearer hlm_p_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/api/v1/alerts
```
- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted**, because addresses end up in logs.
- **No token, a wrong one, an expired one and a removed one all get the same answer**: `401` with a JSON body and `WWW-Authenticate: Bearer realm="homelab-manager"`.
- **Only `GET` and `HEAD`.** Any other verb gets `405` (`{"error":"read_only"}`) with `Allow: GET, HEAD` — with a token or without.
- **Limits:** 300 requests a minute from one address (`429` beyond that). Answers carry `Cache-Control: no-store`.
- **Be gentle:** the data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes). Polling more often than once a minute gains nothing.
### Conventions
- Times are **ISO 8601 in UTC** (`2026-10-20T21:44:30.244Z`). A few older fields written by the database itself (`createdAt` on a server) are `YYYY-MM-DD HH:MM:SS` in UTC.
- Sizes are **bytes**. Percentages are **0–100** (not 0–1).
- `url` is a **full address** into the app, built from `APP_BASE_URL`, so a script does not need to know where the app lives.
- Missing information is `null`, never an invented value. A server whose agent has never said which OS it runs has `operatingSystem: null`.
- Errors are JSON: `{ "error": "<code>", "message": "…" }` with `error` one of `unauthorized`, `read_only`, `not_found`, `invalid_id`.
- Unknown addresses under `/api/v1` get `404` and point at `GET /api/v1`.
### Addresses
| Address | What it answers |
|---|---|
| `GET /api/v1` | The list of these addresses. |
| `GET /api/v1/me` | Whose token this is. |
| `GET /api/v1/servers` | Every server. Filter with `?tag=` and `?status=online\|offline\|unknown`. |
| `GET /api/v1/servers/:id` | One server, the same shape. |
| `GET /api/v1/servers/:id/metrics?days=` | Its load, memory and disks over time. `days` 1–365 (7 by default). |
| `GET /api/v1/alerts` | What is wrong now. Filter with `?severity=`, `?category=`, `?acknowledged=true\|false`, `?silenced=true\|false`. |
| `GET /api/v1/incidents` | The record of alerts that began and ended. `?status=open\|resolved`, `?limit=` (1–500, default 100). |
| `GET /api/v1/incidents/:id` | One incident with its timeline. |
| `GET /api/v1/inventory` | The hardware inventory. |
| `GET /api/v1/network` | Network equipment read from UniFi: devices, networks, Wi-Fi names, and which server is plugged into which device. |
| `GET /api/v1/endpoints` | The endpoint checks and how they are doing. |
| `GET /api/v1/endpoints/:id/history?days=` | One check's availability and answer times over time. |
| `GET /api/v1/maintenance` | Maintenance windows running now and coming up. |
### `GET /api/v1/me`
```json
{
"user": { "id": 1, "name": "Demo Admin", "email": "demo@example.lan", "role": "admin" },
"token": { "id": 1, "label": "Home Assistant", "expiresAt": "2027-01-18T21:45:43.390Z" }
}
```
`expiresAt` is `null` for a token with no end. A quick way to check a token works.
### `GET /api/v1/servers`
```json
{ "servers": [
{
"id": 1,
"name": "wiki-host",
"hostname": "wiki-host.lan",
"description": null,
"os": "linux",
"tags": ["prod"],
"ipAddresses": ["192.168.1.20"],
"status": "online",
"lastSeenAt": "2026-10-20T21:44:30.244Z",
"createdAt": "2026-01-01 00:00:00",
"agentVersion": 7,
"runsOnServerId": null,
"hardware": { "manufacturer": null, "model": null, "serial": null },
"operatingSystem": {
"id": "ubuntu", "name": "Ubuntu 24.04 LTS", "version": "24.04", "product": "Ubuntu 24.04 LTS",
"endsOn": "2029-05-31", "state": "ok", "daysLeft": 951
},
"cpu": { "model": "Intel(R) N100", "cores": 4, "loadPercent": 12.4 },
"memory": { "totalBytes": 16623411200, "usedBytes": 4210356224 },
"disks": [ { "mount": "/", "sizeBytes": 245107195904, "usedBytes": 61276798976, "usedPercent": 25 } ],
"updates": { "manager": "apt", "count": 5, "security": 2, "rebootRequired": false, "reportedAt": "2026-10-20T21:44:30.000Z" },
"health": { "failedUnits": 0, "reportedAt": "2026-10-20T21:44:30.000Z" },
"thresholds": null,
"url": "https://hm.example.lan/servers/1"
}
] }
```
| Field | Meaning |
|---|---|
| `status` | `online` while the agent has reported within the offline limit (60 minutes unless the server has its own), `offline` after, `unknown` if it never has. |
| `lastSeenAt` | When the agent last reported. |
| `agentVersion` | The version of the agent script, as it says; `null` until it has said. |
| `runsOnServerId` | The server this one runs on (a VM's Proxmox host, say), when one is set. |
| `operatingSystem` | What the agent says it runs, and when that release stops being supported. `state` is `ok`, `soon` (inside the warning window), `ended` or `unknown` (a release the app has no date for); `daysLeft` is negative once it has ended. `null` until the agent has said. |
| `cpu.loadPercent` | Load average as a share of the cores — an approximation, not instantaneous usage. |
| `disks[].usedPercent` | One decimal. |
| `updates` | What is waiting to be installed; `null` when the agent does not report it. `security` is `null` when the package manager cannot say. |
| `health.failedUnits` | The number of failed services (Linux), when the agent reports them. |
| `thresholds` | The server's own alert limits, if it has any that replace the global ones; otherwise `null`. |
### `GET /api/v1/servers/:id/metrics?days=`
```json
{ "days": 7, "series": [
{ "metric": "cpu", "kind": "cpu", "label": "CPU load", "latest": 12.4, "trend": null,
"points": [ { "t": "2026-10-20T20:59:30.572Z", "v": 12 }, { "t": "2026-10-20T21:14:30.572Z", "v": 13.5 } ] },
{ "metric": "disk:/", "kind": "disk", "label": "Disk /", "latest": 25,
"trend": { "perDay": 0.4, "fit": 0.97, "spanDays": 13.9, "daysToFull": 188 },
"points": [ … ] }
] }
```
- `kind` is `cpu`, `mem` or `disk`; a disk's `metric` is `disk:<mount>`. Values are percentages.
- The range decides how fine the points are: up to **35 days** they are the samples themselves (one every 10–15 minutes, thinned to at most about 100 points); beyond that they are **hourly (to 120 days) or daily
averages** and carry `lo` and `hi` (the lowest and highest in that hour or day).
- A disk's `trend` is a straight-line fit: `perDay` is points per day, `fit` is how straight (0–1), `daysToFull` is set only when the disk is rising steadily. `null` for CPU and memory, and for a disk
with too little history.
### `GET /api/v1/alerts`
```json
{
"generatedAt": "2026-10-20T21:45:53.522Z",
"needingAttention": { "critical": 1, "warning": 3, "acknowledged": 0 },
"counts": { "critical": 1, "warning": 3, "info": 0, "silenced": 0, "grouped": 0 },
"couldntCheck": [
{ "name": "Proxmox “Lab PVE”", "error": "connect ECONNREFUSED 127.0.0.1:3071", "affects": ["Proxmox backups", "Server and storage health"] }
],
"alerts": [
{
"id": "monitoring:rule:1:server:1",
"severity": "critical",
"category": "monitoring",
"source": "Server",
"message": "wiki-host: cpu load is 95%, above 90% for 30 min [CPU above 90% for 30 minutes]",
"url": "https://hm.example.lan/servers/1?view=history",
"silenced": false,
"snoozedUntil": null,
"flapping": false,
"target": { "type": "server", "id": 1, "name": "wiki-host" },
"causedBy": null,
"affects": [],
"acknowledgement": null
}
]
}
```
| Field | Meaning |
|---|---|
| `needingAttention` | What wants a look: critical and warning alerts that are not silenced and that nobody has acknowledged, and how many somebody has. This is the number on the menu badge. |
| `counts` | Everything active by severity, whether acknowledged or not. `silenced` are held back by maintenance (or snoozed); `grouped` are folded under the host that explains them. Neither is in `critical`/`warning`/`info`. |
| `couldntCheck` | Sources the app could not read this time, and which checks that leaves blind. **An empty `alerts` list with something here is not "all clear".** |
| `id` | Stable for the same problem: `<category>:<key>`. |
| `severity` | `critical`, `warning` or `info`. |
| `category` | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring` or `tickets`. |
| `source` | Where it comes from: `Server`, `Proxmox`, `Synology`, `UniFi`, `Endpoint`, `Heartbeat` … |
| `silenced` | Under a maintenance window, snoozed, or folded under a host that is down. Notifications for it are held back. |
| `snoozedUntil` | When a snooze on it ends; `null` otherwise. |
| `flapping` | It keeps being raised and cleared; its messages are being held back. |
| `target` | What a maintenance window would silence it with (`server`, `integration`, `dns_provider` or `device`), or `null`. |
| `causedBy` | The alert it is folded under, as `{ name, alertId? }`, or `null`. |
| `affects` | On a host's alert: the servers on it whose own alerts are folded under this one. |
| `acknowledgement` | `{ by, at, note, until, assignee }` while someone has said they are on it; otherwise `null`. |
### `GET /api/v1/incidents` and `/:id`
An *incident* is one time an alert was raised and later went away; the same alert coming back is a new incident.
```json
{ "incidents": [
{ "id": 4, "alertKey": "updates:os-support:1", "severity": "warning", "peakSeverity": "warning", "category": "updates",
"source": "Server", "message": "wiki-host runs Ubuntu 18.04 LTS, whose support ended on 2023-05-31 …",
"link": "/servers/1", "target": { "type": "server", "id": 1, "name": "wiki-host" }, "causedBy": null,
"openedAt": "2026-10-20T21:45:53.522Z", "resolvedAt": null, "durationMs": 127, "open": true, "notes": 0 }
] }
```
`GET /api/v1/incidents/:id` adds the **timeline** (when it was raised, acknowledged, handed over, what was done to the thing it is about in the hour around it, when it resolved) and the notes
people wrote. Open incidents are always listed; resolved ones newest first, up to `limit`.
### `GET /api/v1/inventory`
```json
{ "assets": [
{ "id": 3, "name": "NAS", "kind": "server", "manufacturer": "Synology", "model": "DS923+", "serial": "…", "location": "Rack A", "rackUnit": "U6-9",
"purchaseDate": "2024-03-02", "purchasePrice": 6500, "warrantyUntil": "2027-03-02", "endOfSupport": null, "replacementCost": 7200,
"powerWatts": 120, "notes": null, "server": { "id": 6, "name": "nas" }, "updatedAt": "2026-10-20T21:45:30.244Z" }
] }
```
`server` links a piece of hardware to the server it is, when it is one. `kind` is `server`, `storage`, `network`, `power`, `client` or `other`.
### `GET /api/v1/network`
Present once a UniFi integration has been read: `{ devices, networks, ssids, serverLinks }` — each device with its `state` (`online`, `offline`, `updating`, `pending`, `other`), `ip`, `mac`, `uptimeSeconds`,
firmware, what it is connected through and what depends on it; the networks (VLAN, address range); the Wi-Fi names (security, which network); and `serverLinks` saying which server is plugged into which
device (and port). Wi-Fi passphrases are never read, so they are never here.
### `GET /api/v1/endpoints` and `/:id/history`
```json
{ "endpoints": [
{ "id": 1, "name": "Wiki", "kind": "http", "target": "https://192.168.1.20/", "port": null, "enabled": true,
"status": "up", "lastCheckedAt": "2026-10-20T21:50:00.000Z", "lastResponseMs": 41, "lastMessage": null,
"certificateExpiresAt": "2027-01-04T12:00:00.000Z", "uptime24hPercent": 100, "uptime7dPercent": 99.9,
"averageResponseMs24h": 44, "serverId": 1 }
] }
```
`status` is `up`, `down`, `unknown` (never asked yet) or `paused` (switched off). `lastMessage` is set only while it is failing (why). `kind` is `http`, `tcp` or `tls`.
`/:id/history?days=` answers `{ check: {id, name}, … points … }`: the availability and answer times over 1–365 days, drawn from each answer (up to 2 days), the hourly figures (up to 120 days) or the daily
ones, whichever is fine enough and still kept.
### `GET /api/v1/maintenance`
```json
{ "active": [
{ "id": 1, "target": { "type": "server", "id": 3, "name": "photos", "kind": "Server" }, "reason": "Kernel update",
"startsAt": "2026-10-20T21:45:43.616Z", "endsAt": "2026-10-20T22:45:43.616Z", "createdBy": "Demo Admin" } ],
"upcoming": [] }
```
### Examples
**curl and `jq`: is anything critical?**
```bash
curl -fsS -H "Authorization: Bearer $HLM_TOKEN" https://hm.example.lan/api/v1/alerts \
| jq -r '.alerts[] | select(.severity=="critical" and (.silenced|not)) | "\(.source): \(.message)"'
```
**A shell check that fails when something needs attention** (for cron, or a Kuma "push" monitor):
```bash
n=$(curl -fsS -H "Authorization: Bearer $HLM_TOKEN" https://hm.example.lan/api/v1/alerts | jq '.needingAttention.critical')
[ "$n" -eq 0 ] || { echo "$n critical alerts"; exit 1; }
```
**Python:**
```python
import os, requests
BASE = "https://hm.example.lan/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['HLM_TOKEN']}"}
def get(path, **params):
r = requests.get(f"{BASE}{path}", headers=HEADERS, params=params, timeout=10)
r.raise_for_status()
return r.json()
for s in get("/servers", status="offline")["servers"]:
print(s["name"], "last seen", s["lastSeenAt"])
full = [(s["name"], d["mount"], d["usedPercent"]) for s in get("/servers")["servers"] for d in s["disks"] if d["usedPercent"] and d["usedPercent"] > 85]
print(full)
```
**Node.js (18+):**
```js
const res = await fetch("https://hm.example.lan/api/v1/endpoints", { headers: { Authorization: `Bearer ${process.env.HLM_TOKEN}` } });
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
const { endpoints } = await res.json();
console.log(endpoints.filter((e) => e.status === "down").map((e) => e.name));
```
**Home Assistant** — put the whole header value, including the word `Bearer`, in `secrets.yaml`:
```yaml
# secrets.yaml
hlm_auth: "Bearer hlm_p_xxxxxxxxxxxxxxxxxxxxxxxx"
```
```yaml
# configuration.yaml
rest:
- resource: https://hm.example.lan/api/v1/alerts
scan_interval: 60
headers:
Authorization: !secret hlm_auth
sensor:
- name: "Homelab critical alerts"
value_template: "{{ value_json.needingAttention.critical }}"
- name: "Homelab warnings"
value_template: "{{ value_json.needingAttention.warning }}"
binary_sensor:
- name: "Homelab needs attention"
value_template: "{{ value_json.needingAttention.critical > 0 }}"
device_class: problem
```
A similar sensor on `/api/v1/servers?status=offline` with `value_template: "{{ value_json.servers | length }}"` counts servers that have gone quiet.
## Other addresses for machines
These do not use a session and are not part of the read API. Each is protected by its own token and does one thing.
### Agent report — `POST /api/agent/report`
What the agents on the servers call, every 15 minutes by default. The token is the **server's own** (shown once when the server is added under **Servers → Add a server**, and put in the agent by the install
one-liner); it can only replace *that* server's report. You can call it yourself from anything that can make an HTTPS request — a router, a NAS, a script — to make a server appear and be monitored.
```bash
curl -fsS -X POST https://hm.example.lan/api/agent/report \
-H "Authorization: Bearer <the server's token>" -H "Content-Type: application/json" \
-d '{
"hostname": "nas",
"os_type": "linux",
"agent_version": 7,
"reported_at": "2026-10-20T21:50:00Z",
"system": {
"ip_addresses": ["192.168.1.30"],
"cpu": { "model": "Intel(R) N100", "cores": 4, "load_percent": 12.4 },
"memory": { "total_bytes": 16623411200, "used_bytes": 4210356224 },
"disks": [ { "mount": "/", "size_bytes": 245107195904, "used_bytes": 61276798976 } ],
"listening_ports": [ { "protocol": "tcp", "port": 22, "address": "0.0.0.0", "process": "sshd" } ],
"updates": { "manager": "apt", "count": 5, "security": 2, "reboot_required": false },
"health": {
"failed_units": [ { "unit": "foo.service", "description": "Foo" } ],
"smart": [ { "device": "/dev/sda", "model": "WD Red", "healthy": true, "temperature_c": 38, "wear_percent": null } ],
"zfs": [ { "name": "tank", "health": "ONLINE", "capacity_percent": 61 } ]
},
"hardware": { "manufacturer": "ASUS", "model": "PN42", "serial": "…" },
"os": { "id": "debian", "name": "Debian GNU/Linux 12", "version": "12" }
},
"tasks": [ { "schedule_type": "cron", "name": "backup", "command": "/usr/local/bin/backup.sh", "schedule_expression": "0 3 * * *", "source": "root" } ]
}'
```
- **Everything in `system` is optional**, and a part that is malformed is dropped rather than costing the report. Text is trimmed to a sane length. `tasks` may be `[]` (but is required).
- Answers: `202` with `{ "ok": true, "taskCount": 1 }` when the report is taken; `401` for no or a wrong token (`missing_token`, `invalid_token`); `400` (`invalid_body`) for a report that is not JSON of this shape. Reports may be up to **4 MB**.
- `schedule_type` is `cron`, `systemd_timer` or `windows_task`. `os_type` is recorded only if it is `linux` or `windows`. The same token replaces the previous report's tasks and figures.
- `GET /api/agent/report/update?version=<n>` (same token) answers `{ "update": bool, "version": …, "sha256": … }` for an agent that has been told to keep itself up to date; see [AGENT_UPDATE.md](../AGENT_UPDATE.md).
How the figures are used is in [MONITORING.md](MONITORING.md).
### Heartbeats, metrics, status pages and calendars
| Address | What | Token | Documented in |
|---|---|---|---|
| `GET /api/heartbeat/<token>[/start\|/fail]` | A cron job says it ran | in the address | [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) |
| `GET /metrics` | Prometheus text format | `Authorization: Bearer hlm_m_…` | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
| `GET /api/public/calendar/<token>.ics` | A calendar feed of what is coming up | in the address (a share link) | [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) |
| `GET /api/public/status/<token>`, `/api/public/wall/<token>` | The data behind a shared status page and wall display | in the address (a share link) | Settings → Share links |
| `GET /health` | Is the app up? | none | answers `{"ok":true}` |
## Reading the API safely
- Treat a token like a password: it can read everything its owner can. Give each script its own (so one can be removed alone), choose the shortest life that suits, and keep it in a secret store, not in a repository.
- Use HTTPS. The token is sent in a header with every request.
- A removed token stops working at once; if one leaks, remove it under **API tokens** and make another.
- Every token's last use is shown on the API tokens page, which makes an unused one easy to spot and remove.
+70
View File
@@ -0,0 +1,70 @@
# Heartbeats and the calendar feed
Two addresses that are not part of the [read API](API.md) and need no sign-in, because each carries its own token:
- **Heartbeats** — a cron job or script calls an address when it finishes, and Homelab Manager speaks up when it does not. This is how to monitor things the app cannot see by itself: a backup script,
a nightly sync, a certificate renewal.
- A **calendar feed** — what is coming up (expiries, warranties, maintenance), for a phone or desktop calendar to subscribe to.
How a missed heartbeat becomes an alert, a notification and an incident is in [MONITORING.md](MONITORING.md).
## Heartbeats — for cron jobs and scripts
### Setting one up
Administrators: **Monitoring → Heartbeats → Add a heartbeat**.
| Field | Notes |
|---|---|
| Name | what the job is, e.g. "Nightly backup" (up to 100 characters) |
| **Period** | how often the job should report, from 1 minute to 30 days |
| **Grace** | how much later than that is still fine, 0 minutes to a day (so a job that sometimes runs long is not a false alarm) |
| Server | optional: the server the job runs on. **Maintenance on that server silences the heartbeat's alerts**, and the alert links to it. |
| Notes | up to 500 characters |
Up to **100** heartbeats. Everyone signed in sees how each job is doing; only administrators see its **address** (with the token) — it is what lets anyone say "I'm alive" for the job. **New address** gives
a heartbeat a new token and the old address stops working at once.
### Calling it
The job calls its address when it finishes:
```
GET /api/heartbeat/<token> it worked
GET /api/heartbeat/<token>/start it began (optional: lets the next call say how long it took)
GET /api/heartbeat/<token>/fail it ran and went wrong: an alert at once
```
```sh
# in cron, after the job
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx
# in a script, with a start, a failure and a note
curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start
./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \
|| curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail
```
- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work.
Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once).
- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters.
- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address.
- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes;
a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`.
- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them.
## Calendar feed
An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL").
```
GET /api/public/calendar/<token>.ics
```
- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time.
A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address.
- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and
recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out.
- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app.
An event's `UID` is a hash of what it is and its date, the same on every fetch.
- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does.
+304
View File
@@ -0,0 +1,304 @@
# How monitoring works
This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the
"why did it do that?" page. Setting things up is in the other pages ([SSO](SSO.md), [API](API.md), [Prometheus and Grafana](PROMETHEUS-GRAFANA.md), [webhooks](WEBHOOKS.md),
[heartbeats](HEARTBEATS-AND-CALENDAR.md)); the list of every notification is [NOTIFICATIONS.md](../NOTIFICATIONS.md).
- [The big picture](#the-big-picture)
- [What is watched, and how often](#what-is-watched-and-how-often)
- [Servers and the agent](#servers-and-the-agent)
- [From a reading to an alert](#from-a-reading-to-an-alert)
- [The built-in checks](#the-built-in-checks)
- [Your own alert rules](#your-own-alert-rules)
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
- [Integrations](#integrations)
- [Forecasts](#forecasts)
- [Dependencies: one outage, one alert](#dependencies-one-outage-one-alert)
- [What happens to an alert](#what-happens-to-an-alert) — notify, acknowledge, silence, snooze, flapping, escalate
- [Incidents](#incidents)
- [History and how long it is kept](#history-and-how-long-it-is-kept)
- [Where to see it](#where-to-see-it)
- [Tuning, in questions and answers](#tuning-in-questions-and-answers)
- [What monitoring does not do](#what-monitoring-does-not-do)
## The big picture
```
agents ─────────┐
integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · silence
endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push
heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines)
your own rules ─┘ ├──► incidents (the record) and history
└──► /metrics and the read API
```
Three ideas run through everything:
1. **A check turns readings into *conditions*.** A condition is "this specific problem exists right now" and has a stable **key** (`offline:server:3`, `disk:server:3:/`) — the same problem always has the same key.
2. **Alerts are announced on a change, not on a schedule.** A problem is announced **once** when it appears and **once** when it ends. It is not repeated every time it is looked at.
3. **Silence is never "all clear".** When something cannot be *read* (an integration is down, a server's agent is quiet), what depended on it is **held exactly as it was** — neither raised nor cleared —
and the Alerts page lists it under *Couldn't check everything*. The app never turns "I couldn't look" into "nothing is wrong".
## What is watched, and how often
| What | How it gets to the app | How often |
|---|---|---|
| **Servers** (Linux, Windows) | The **agent** on each server **pushes** a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects *to* a server. | Every **15 minutes** by default (set when the agent is installed) |
| **Proxmox**, **Synology** | The health pass **reads** them: node and storage usage, volume and disk status. | Every **15 minutes** |
| **UniFi** | A sync **reads** devices, networks, Wi-Fi names and clients and keeps them. | Every **5 minutes** |
| **Home Assistant** | A sync **reads** the entities you chose to watch and keeps their state and a history. | Every **5 minutes** |
| **Nginx Proxy Manager** | **Read** when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute |
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications |
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
After the app **restarts**, server-based checks are paused for **20 minutes** so agents get a chance to report before any server is judged.
## Servers and the agent
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
**Per-server settings.** A server's *Settings* section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for
one disk or "never alert about this disk", and services never to count as failed (names or patterns such as `gupdate*`). Whatever it does not replace follows the global setting.
**What the report is used for:** the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support
check, pending updates, the dependency map, and `/metrics`.
## From a reading to an alert
There are two paths, designed so they cannot disagree:
**1. The state-based checks (the notifications).** Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the **list of conditions that exist right now**. The app compares that list with
the one it remembered from last time:
- **new** → announced (a notification, an event webhook, a row in the incident record);
- **gone** → announced as resolved;
- **still there** → nothing;
- **its source could not be read this time** → carried over untouched (not resolved, not repeated).
The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all.
**2. The Alerts page.** It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), **live**, and turns what they find into one flat list. It ignores the per-event on/off toggles —
the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a
20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under *Couldn't check everything*.
Each alert has:
| | |
|---|---|
| **Severity** | `critical`, `warning` or `info`. Things that are *gone* (offline, a failing SMART disk, a disk over 95%) are critical; things that are *coming* (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. |
| **Category** | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring`, `tickets` — what the filter on the page and the [routing rules](#who-hears-about-it) work on. |
| **Source** | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … |
| **Target** | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. |
| **Key / id** | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. |
## The built-in checks
Every threshold here is a default; where it can be changed, the place is named.
| Check | Raised when | Default | Changed in |
|---|---|---|---|
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk |
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card | on | Settings → Notifications |
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
## Your own alert rules
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
- **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
- **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the
history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time.
- **How serious:** warning or critical. **For which servers:** all of them, one, or those with a tag.
- **Try it now** shows which servers it would be raised on, from current data, before you save it.
A rule's finding is an **ordinary alert** with the key `rule:<rule id>:server:<server id>[:<disk>]`, in the *Monitoring* category: announced and resolved like the others, held back by maintenance on the
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is
offline or has not reported for 45 minutes is not judged.
## Endpoint checks and heartbeats
**Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds:
| Kind | What it asks |
|---|---|
| `http` | Fetches an `http(s)://` address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. `200-299, 301`), and — if you ask — the page contains a keyword. Follows up to 5 redirects. |
| `tcp` | Opens a connection to a host and port. |
| `tls` | Connects and reads the certificate; the answer is the days left. |
Per check: **interval** (1–1440 minutes, default 5), **timeout** (default 10 s), **failures before an alert** (default 2 in a row — one blip is not an alert), **certificate warning days** (default 14), an
optional **server it is linked to** (maintenance on that server silences it, and the alert links to it), and *accept a self-signed certificate*. Each answer — time, result, message — is kept, with hourly and
daily summaries for the charts and uptime figures.
- A check **fails an alert** (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A **certificate** within its warning days is its own alert (`endpoint-cert:…`).
- What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine.
- The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under *Endpoints → Suggested*, and proxied hosts Nginx Proxy Manager serves.
**Heartbeats** (Monitoring → Heartbeats) are the other direction: a cron job or script **calls an address when it finishes**, and the app alerts when it does *not*. A heartbeat has a **period** and a **grace**; it is
**late** when both have passed since the last call (a *warning*), and a call to `/fail` is **critical** at once. Details and examples: [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md).
Both are judged **every minute**, with the same remember-and-compare bookkeeping as the other checks.
## Integrations
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
integration's page.
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (**20 minutes**).
- **An integration can explain other alerts.** A service that stops answering because the integration it needs is failing is folded under it ([Dependencies](#dependencies-one-outage-one-alert)).
## Forecasts
A forecast says "this will be full in N days" from a **straight line through the recent history**, and is deliberately cautious:
- It needs **at least six readings covering at least three days**, and a **steady** climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
## Dependencies: one outage, one alert
If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a **graph of what needs what**:
- a server **runs on** a host (set by hand, or worked out from its Proxmox node);
- a UniFi device is **connected through** another, and a **server is plugged into** a switch or access point (matched by IP address);
- **dependencies you write** (*Map → Dependencies*: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out.
When something is down and what depends on it is down too, the dependants' alerts are **folded under** the first: shown nested, marked *caused by …*, not announced on their own, and the host's message says
"— also takes down web and db". If the host comes back and a dependant **stays** down, it is announced then, as new: **an outage is never swallowed.**
## What happens to an alert
Several things can quiet an alert. They are different, and it matters which you use:
| | Promise about the thing? | Hides it? | Notifications | Ends |
|---|---|---|---|---|
| **Acknowledge** ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in `/metrics`; leaves the *needs attention* count | Stops [escalation](#escalation) | When you choose (an hour … 30 days), or when the alert has been gone half an hour |
| **Snooze** ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; **one reminder** at the end if it is still there | At the time you set (up to 90 days), or earlier with *Wake now* |
| **Maintenance window** | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked *silenced* | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as **new** |
| **Folded under a host** | — automatic | Nested under its cause | Not announced on its own | When the cause ends |
| **Flapping** | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm |
### Who hears about it
Notifications go to the channels that are on (**Gotify, ntfy, email, a webhook, push to phones and browsers**), narrowed by **routing rules** (first match wins: by kind, by severity, by the server's tags) and each
browser's own wishes. **Quiet hours** hold messages and send one digest at the end of the window. Every message the app can send, and what toggles it, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
**Event webhooks** are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included ([WEBHOOKS.md](WEBHOOKS.md)).
### Flapping
An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
### Escalation
An alert raised for a while with **nobody on it** can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra
email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes.
### Maintenance
*Maintenance* (a window of 5 minutes to a week, started by hand, in bulk, from a **recurring schedule**, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and
its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under
maintenance with it. A silenced problem is **not remembered as known**, so one still there when the window ends **alerts then, as new**.
## Incidents
Every time an alert was raised and later went away is an **incident**: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its
timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts **every five minutes** whether or not anyone has
the page open; the same alert coming back is a **new** incident. Resolved incidents are kept **400 days**. They are on **Operations → Alerts → History**, in the read API (`/api/v1/incidents`) and in the
monthly report.
## History and how long it is kept
| What | Kept | Grain |
|---|---|---|
| A server's load, memory and each disk | **35 days** of samples (one every 10–15 minutes), **120 days** of hourly summaries, **~800 days** of daily summaries | summaries carry the lowest and highest |
| Endpoint check answers | **14 days** of every answer, hourly summaries for **120 days**, daily summaries for years | |
| Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day |
| Proxmox storage and Synology volume use | **400 days**, one reading an hour | |
| Home Assistant watched numbers | **35 days**, one reading every few minutes | |
| Incidents | **400 days** after they resolve | |
| The audit log and the Diagnostic Log | Until automatic purging is switched on under *Settings → Data* (then 90 days by default, configurable) | |
A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there.
## Where to see it
| | Shows |
|---|---|
| **Dashboard**, **My work**, the menu badge | What needs attention now; what is yours |
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **Proxy hosts** | Each source's own view |
| **Weekly digest** and **monthly report** | The week and the month in one message |
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
| **Read API** (`/api/v1`) | [API.md](API.md) |
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
| **Status page**, **wall display**, **calendar feed** | Read-only views for others; made under *Settings → Share links* |
## Tuning, in questions and answers
**I am about to reboot a server.** Start maintenance on it (Maintenance page, or *Silence* on an alert): its alerts are held and, if it is still down at the end, announced as new.
**An alert is noisy because it keeps coming and going.** Let it flap (it will be damped after three raises an hour), then look at *Alerts → Noise* — it says what usually helps: a longer grace, a different limit.
For your own rules, raise the limit or lengthen the time.
**One server legitimately runs at 92% disk.** Give that server (or that one disk) its own limit — or *never alert about this disk* — in the server's *Settings* section.
**I want to be reminded about something later, not silence it.** Snooze it.
**I do not want to be woken for warnings.** Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours.
**A job on a server has to run every night.** Make a heartbeat for it and call it from the end of the script.
**I want my own limit, such as "memory above 85% for an hour".** Add an alert rule.
**An integration's alerts disappeared.** Check *Couldn't check everything* on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why.
**The app restarted and nothing about servers is alerting.** For 20 minutes after a restart servers are not judged, so the agents can report first.
**I want these figures in my own monitoring.** `/metrics` for Prometheus, the read API for scripts, event webhooks to be told.
## What monitoring does not do
- **It does not fix anything.** The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.)
- **It does not see inside a server that has no agent** — only what an endpoint check, a heartbeat or an integration can say about it.
- **It is not a replacement for a second pair of eyes on the app itself.** If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on `up{job="homelab-manager"}`
([PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md#alerting-rules-to-start-from)), or a check from another tool against `GET /health`.
- **It is as fresh as its sources.** A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.
+337
View File
@@ -0,0 +1,337 @@
# Prometheus and Grafana
Homelab Manager can be scraped by **Prometheus** at `/metrics`, and ships a ready-made **Grafana dashboard** built on those series. This page sets up both, from nothing, and lists every series.
- [How it fits together](#how-it-fits-together)
- [1. Make a metrics token](#1-make-a-metrics-token)
- [2. Check it by hand](#2-check-it-by-hand)
- [3. Set up Prometheus](#3-set-up-prometheus)
- [4. Set up Grafana](#4-set-up-grafana)
- [A complete example with Docker Compose](#a-complete-example-with-docker-compose)
- [Alerting rules to start from](#alerting-rules-to-start-from)
- [Every series](#every-series)
- [Useful queries](#useful-queries)
- [Troubleshooting](#troubleshooting)
## How it fits together
```
agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana
(reads its own database; (stores the history,
calls nothing outside) draws the dashboard)
```
- `/metrics` is a **view of what the app already knows**. A scrape reads the database and the app's last alert result (reused for up to a minute); it **never calls out** — not to Proxmox, not to a server, not
to an integration. Scraping every 15 seconds costs a few database reads.
- The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is **not needed** to see trends in the app itself. Use Prometheus and Grafana when you
want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager.
- **Every series is a gauge.** Labels are escaped as Prometheus requires.
## 1. Make a metrics token
Administrators: **Settings → API and webhooks → Metrics for Prometheus → Make a token**.
- The token (it starts with `hlm_m_`) is **shown once**; only its SHA-256 is kept.
- Up to **ten** can exist — make one per Prometheus, so one can be removed without touching the others. A removed token **stops working at once**.
- Each shows when it was last used, so an unused one is easy to find.
- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted** (addresses end up in logs).
- No token, a wrong one and a removed one all get the same `401`. Requests are limited to **120 a minute** per address (`429` beyond).
- The answer is Prometheus's text format: `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
A metrics token can read `/metrics` and nothing else. It is a different thing from the *personal API tokens* used for the [read API](API.md), and neither can sign in.
## 2. Check it by hand
```bash
curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30
```
You should see something like this (trimmed; the `# HELP` lines are left out here):
```
# TYPE homelab_info gauge
homelab_info{environment=""} 1
# TYPE homelab_server_up gauge
homelab_server_up{server="wiki-host"} 1
# TYPE homelab_server_last_seen_timestamp_seconds gauge
homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670
# TYPE homelab_storage_days_to_full gauge
homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21
# TYPE homelab_alerts gauge
homelab_alerts{severity="critical"} 1
homelab_alerts{severity="warning"} 3
homelab_alerts{severity="info"} 0
```
Without the header you get `401` — that is the right answer:
```bash
curl -i https://hm.example.lan/metrics # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics"
```
## 3. Set up Prometheus
Put the token in a file of its own, readable only by Prometheus, with **no trailing newline**:
```bash
printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token
chmod 600 /etc/prometheus/homelab-manager.token
```
`prometheus.yml`:
```yaml
global:
scrape_interval: 30s
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/homelab-manager.token
static_configs:
- targets: ["hm.example.lan"] # host[:port] of APP_BASE_URL, without https://
# If the certificate comes from your own CA, point at it (rather than skipping verification):
# tls_config:
# ca_file: /etc/prometheus/my-ca.pem
```
- **30–60 seconds is plenty.** The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes).
- Prometheus accepts the token inline too (`credentials: hlm_m_…`), but a file keeps it out of the configuration you share.
- Behind a reverse proxy, use the address the proxy serves (`scheme: https`, port 443). Reaching the container directly is `scheme: http`, target `host:3000` — fine on a trusted network.
- Reload Prometheus (`kill -HUP`, or `POST /-/reload` if enabled) and open **Status → Targets**: `homelab-manager` should be **UP**.
- **Several installations** (production and a dev copy): the series `homelab_info{environment="…"}` carries the name set by `ENVIRONMENT_NAME`, so you can tell them apart; add an `instance`/`env` label in the scrape
config as well.
## 4. Set up Grafana
1. **Add Prometheus as a data source** — *Connections → Data sources → Add → Prometheus*, URL e.g. `http://prometheus:9090`, *Save & test*.
2. **Download the dashboard** — in Homelab Manager: **Settings → API and webhooks → Download the dashboard** (administrators). It is a file, `homelab-manager-dashboard.json`. (The same file is served to a signed-in
administrator at `GET /api/metrics-tokens/grafana-dashboard.json`.)
3. **Import it** — in Grafana: **Dashboards → New → Import**, upload the file, and **choose your Prometheus** when asked. The file never names a data source, so it is not tied to any installation.
The dashboard has a **Server** picker (filled from `homelab_server_up`) and five rows:
| Row | What it has |
|---|---|
| **Needs attention** | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
| **Servers** | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
| **Disks** | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
| **Backups** | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
| **Secrets, power and the app itself** | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. |
The dashboard is **built by the app from the same code that serves `/metrics`**, and a test checks that it only asks for series that are really served. So **after an upgrade that adds series, download it again**
to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), **but it has not been opened in a Grafana
by the author** — if a panel looks wrong, the series table below says what each query should be.
## A complete example with Docker Compose
Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder:
```
monitoring/
├─ docker-compose.yml
├─ prometheus.yml (as above, with credentials_file: /etc/prometheus/homelab-manager.token)
├─ homelab-manager.token (the token only, no newline)
└─ rules/
└─ homelab-manager.yml (the alerting rules below, optional)
```
```yaml
# docker-compose.yml
services:
prometheus:
image: prom/prometheus:latest
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=400d # a year and a bit: the dashboard's long ranges are then real
- --web.enable-lifecycle
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro
- ./rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports: ["9090:9090"]
grafana:
image: grafana/grafana:latest
restart: unless-stopped
volumes:
- grafana-data:/var/lib/grafana
ports: ["3001:3000"]
depends_on: [prometheus]
volumes:
prometheus-data:
grafana-data:
```
With the rules, add to `prometheus.yml`:
```yaml
rule_files:
- /etc/prometheus/rules/*.yml
```
In Grafana the data source URL is then `http://prometheus:9090` (the service name, since they share a compose network).
## Alerting rules to start from
Homelab Manager already sends its own notifications (see [MONITORING.md](MONITORING.md)). **Use these in addition only if you want the same things in Alertmanager**, otherwise you will be told twice.
They are the ones worth having *outside* the app — above all, the app itself going quiet.
```yaml
# rules/homelab-manager.yml
groups:
- name: homelab-manager
rules:
# The one thing the app cannot tell you about itself.
- alert: HomelabManagerDown
expr: up{job="homelab-manager"} == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "Homelab Manager cannot be scraped" }
- alert: HomelabCriticalAlerts
expr: homelab_alerts{severity="critical"} > 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" }
- alert: HomelabServerDown
expr: homelab_server_up == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $labels.server }} has stopped reporting" }
- alert: HomelabDiskFilling
expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7
for: 1h
labels: { severity: warning }
annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" }
- alert: HomelabDiskUsageHigh
expr: homelab_server_disk_used_percent > 90
for: 15m
labels: { severity: warning }
annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" }
- alert: HomelabSelfBackupStale
expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400
labels: { severity: warning }
annotations: { summary: "Homelab Manager's own backup is more than two days old" }
- alert: HomelabEndpointDown
expr: homelab_endpoint_up == 0
for: 5m
labels: { severity: warning }
annotations: { summary: "Endpoint check {{ $labels.name }} is failing" }
- alert: HomelabCertificateExpiring
expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14
labels: { severity: warning }
annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" }
- alert: HomelabHeartbeatMissed
expr: homelab_heartbeat_up == 0
labels: { severity: warning }
annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" }
- alert: HomelabIntegrationFailing
expr: homelab_integration_up == 0
for: 10m
labels: { severity: warning }
annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" }
```
A series that is **left out** is not zero: `homelab_integration_up`, `homelab_endpoint_up` and `homelab_selfbackup_last_attempt_ok` do not exist until something has been tried. That is why the rules above
compare against a condition rather than use `absent()`.
## Every series
All gauges. The same table is checked by a test against what `/metrics` really serves.
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_storage_days_to_full` | `pool` | Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb. |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts`. |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window or a snooze. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
## Useful queries
```promql
# Servers that have gone quiet
homelab_server_up == 0
# The five fullest disks
topk(5, homelab_server_disk_used_percent)
# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full)
sort(homelab_server_disk_days_to_full < 30)
# Endpoints slower than half a second
homelab_endpoint_response_milliseconds > 500
# How many guests are not well backed up
sum(homelab_backup_guests{status=~"stale|failed|none"})
# Updates waiting across all servers, and how many are security fixes
sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates)
# Is the dev copy up as well as production? (several installations: filter on the label you added)
homelab_info
```
## Troubleshooting
| Symptom | Cause |
|---|---|
| Target is **DOWN**, `401 Unauthorized` | The token is wrong, was removed, or has a trailing newline in its file (use `printf '%s'`). The same `401` is given for no token and a bad one. |
| Target is **DOWN**, `429 Too many requests` | More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval. |
| Target is **DOWN**, certificate error | Prometheus does not trust the certificate. Give it the CA with `tls_config.ca_file` (or `insecure_skip_verify: true` on a trusted LAN, knowing what that gives up). |
| Target **UP** but some series are missing | A series that has nothing to say is **left out** rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected. |
| A dashboard panel is empty | The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade. |
| Numbers differ from the Alerts page | `homelab_alerts` counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute. |
| Names with odd characters look wrong | Label values are escaped (`\"`, `\\`, `\n`) as the Prometheus format requires; Prometheus and Grafana unescape them. |
+41
View File
@@ -0,0 +1,41 @@
# Documentation
Everything about running and integrating with Homelab Manager is in Markdown files in this repository. This folder holds the **guides**; the **references** (who can do what, every notification, every table,
the credentials each integration needs) are next to the main [README](../README.md).
## Guides — this folder
| Guide | Read it when you want to … |
|---|---|
| **[SSO.md](SSO.md)** | Set up sign-in with Authentik: what is needed (HTTPS, reverse proxy, a signing key), the provider and `.env` settings, how roles can follow groups, and what to check when sign-in fails. |
| **[API.md](API.md)** | Call Homelab Manager from a script, Home Assistant or another tool: personal tokens, every `/api/v1` address with its answer, examples in curl, Python, Node and Home Assistant, and the other machine-facing addresses (agent reports, heartbeats, share links). |
| **[PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md)** | Scrape `/metrics` with Prometheus, import the Grafana dashboard, copy a working Docker Compose and alerting rules, and look up every series. |
| **[WEBHOOKS.md](WEBHOOKS.md)** | Be told when something happens: signed event webhooks (alert opened/resolved, maintenance started/ended) with receivers in Node and Python, and the chat-style notification webhook. |
| **[HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md)** | Monitor a cron job (it calls an address when it finishes) or subscribe a calendar to what is coming up. |
| **[MONITORING.md](MONITORING.md)** | Understand how monitoring works: what is watched and how often, how a reading becomes an alert, what acknowledge, snooze, maintenance and flapping each do, incidents, history, forecasts, and how to tune it. |
## References — next to the README
| File | What it is |
|---|---|
| [README.md](../README.md) | What the app is, every feature, installation (Docker, arm64, development). |
| [ROLES.md](../ROLES.md) | Viewer, operator, administrator: who sees which page and who can change what. |
| [NOTIFICATIONS.md](../NOTIFICATIONS.md) | Every notification the app sends, what triggers it, what toggles it, routing, quiet hours, escalation. |
| [INTEGRATIONS.md](../INTEGRATIONS.md) | What each integration and DNS provider needs (credentials, permissions) and what it does. |
| [DATABASE.md](../DATABASE.md) | The database tables, columns and relationships. |
| [BACKUP.md](../BACKUP.md) | What to back up, how (also while it runs), how to restore, and how to check a backup. |
| [AGENT_UPDATE.md](../AGENT_UPDATE.md) | Keeping the agents on the servers up to date. |
## Where to start
- **A new installation:** [README](../README.md) (install) → [SSO.md](SSO.md) (sign-in) → add servers and integrations in the app → [MONITORING.md](MONITORING.md) (what to expect) → [BACKUP.md](../BACKUP.md).
- **Getting figures into Grafana:** [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md).
- **Getting alerts into something else:** [WEBHOOKS.md](WEBHOOKS.md) for pushes, [API.md](API.md) for asking.
- **"Why did / didn't it alert?":** [MONITORING.md](MONITORING.md), then [NOTIFICATIONS.md](../NOTIFICATIONS.md).
## Conventions in these pages
- Examples use **`https://hm.example.lan`** for your own address (`APP_BASE_URL`), and tokens like `hlm_p_xxxxxxxx…` that you will replace.
- A **menu path** such as *Settings → API and webhooks* is in the web interface; who may open it is in [ROLES.md](../ROLES.md).
- Sample answers have the shapes the app really sends (several were captured from a test installation and trimmed); the values are only illustrative.
- Things that **have not been checked against the real thing** (a third-party system, a Grafana version, another identity provider) are said so where they come up.
+229
View File
@@ -0,0 +1,229 @@
# Single sign-on (SSO)
Homelab Manager has no passwords of its own. People sign in through an identity provider with **OpenID Connect (OIDC)**, and the app only ever sees who they are,
never a password. It is written for **[Authentik](https://goauthentik.io/)** and the settings are named after it; any provider that speaks standard OIDC *can* work
(see [Other identity providers](#other-identity-providers)), but only Authentik has been used.
This page covers what the setup needs, how to configure Authentik and the app, how roles are decided, and what to check when sign-in fails.
- [How sign-in works](#how-sign-in-works)
- [What you need before you start](#what-you-need-before-you-start)
- [1. Set up Authentik](#1-set-up-authentik)
- [2. Set up the app](#2-set-up-the-app)
- [3. The first sign-in](#3-the-first-sign-in)
- [Roles](#roles) — by hand, or from groups
- [Sessions and signing out](#sessions-and-signing-out)
- [What does *not* use SSO](#what-does-not-use-sso)
- [Troubleshooting](#troubleshooting)
- [If you are locked out](#if-you-are-locked-out)
- [Security notes](#security-notes)
## How sign-in works
1. A person opens the app and is sent to `/auth/login`. The app starts an *authorization code* flow with **PKCE** (a one-time code verifier and challenge) and a random `state`,
and redirects the browser to Authentik.
2. The person signs in at Authentik (with whatever Authentik asks for: password, passkey, two-factor …).
3. Authentik sends the browser back to **`<APP_BASE_URL>/auth/callback`** with a one-time code.
4. The app, from its own server, swaps the code for tokens at Authentik, checks the **ID token's signature** against the keys Authentik publishes (not only its contents), and
asks Authentik's *userinfo* address for the person's name, email, picture and groups.
5. The app finds the person by their **`sub`** (Authentik's stable id for them) or makes an account, decides their role ([below](#roles)), writes the sign-in to the audit log
and starts a session with a **new session id**.
The *client* is **confidential**: the app holds a client secret, and the code exchange happens server to server. The scopes asked for are `openid email profile`.
## What you need before you start
| You need | Why |
|---|---|
| **An Authentik instance** | The identity provider. |
| **The app reachable by browsers at a fixed address — `APP_BASE_URL`** — and **over HTTPS** in production | The redirect address Authentik sends people back to is built from it. The session cookie is marked `Secure` when the app runs in production (`NODE_ENV=production`, which the supplied Docker setup does), and browsers only keep a `Secure` cookie from an HTTPS page — so behind plain `http://` on a LAN address, sign-in appears to work and then sends you straight back to the login page. (`http://localhost` is the exception browsers allow.) Put the app behind a reverse proxy with a certificate. |
| **A reverse proxy that passes `X-Forwarded-For` and `X-Forwarded-Proto`** (one proxy in front) | The app trusts one proxy hop, so the sign-in log and the rate limit see each person's address instead of the proxy's. |
| **The app able to reach Authentik, server to server** | It calls Authentik's discovery, token, userinfo and key (JWKS) addresses itself, from wherever it runs — from inside the Docker network that may not be the name your browser uses. |
| **Authentik's certificate trusted by the app** | If Authentik uses a certificate from your own CA, give Node the CA with `NODE_EXTRA_CA_CERTS=/path/to/ca.pem`. Do **not** switch certificate checking off with `NODE_TLS_REJECT_UNAUTHORIZED=0`. |
| **Reasonably correct clocks** on both machines | Tokens have times in them; a minute of drift is tolerated, more is refused. |
| **Two secrets generated by you** | `SESSION_SECRET` (signs the session cookie) and `CREDENTIALS_ENCRYPTION_KEY` (encrypts stored integration credentials). Not part of SSO itself, but the app will not run in production without the first. |
Generate both with:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
## 1. Set up Authentik
The names of the menus differ a little between Authentik versions; what has to be true does not.
**Create the provider** — *Applications → Providers → Create → OAuth2/OpenID Provider*:
| Setting | Value |
|---|---|
| Name | anything, e.g. `Homelab Manager` |
| Authorization flow | your usual one (explicit or implicit consent, as you prefer) |
| **Client type** | **Confidential** |
| Client ID / Client secret | generated for you — you will copy both into the app's `.env` |
| **Redirect URIs** | `https://hm.example.lan/auth/callback` — your `APP_BASE_URL` followed by `/auth/callback`, as an **exact (strict) match** |
| **Signing key** | **choose a certificate (an RSA key), such as *authentik Self-signed Certificate*.** With no signing key Authentik signs ID tokens with the client secret (HS256) and publishes no keys, and the app refuses such a token. |
| Scopes | `openid`, `email`, `profile` (the defaults) |
| Subject mode | leave the default (*Based on the hashed User ID*) — see the warning below |
| Issuer mode | leave the default (*Each provider has a different issuer*) |
> **Do not change the subject mode later.** The app recognises a person by their `sub`. Changing how Authentik builds it makes every person a *new* person: a new account
> (a viewer, with none of the old account's role) and the old one left behind.
**Create the application** — *Applications → Applications → Create*: give it a name and a **slug** (e.g. `homelab-manager`), choose the provider above, and decide **who may use it**
(a policy or group binding on the application). Authentik decides who can *sign in*; the app's roles decide what they can *do* once in.
The **issuer URL** you need next is on the provider's page — normally `https://authentik.example.lan/application/o/<slug>/`, **with the trailing slash**.
## 2. Set up the app
Put these in the `.env` file next to `docker-compose.yml` (a template is in `.env.example`):
```ini
# Where people reach the app. HTTPS, no trailing slash needed. Used to build the redirect address.
APP_BASE_URL=https://hm.example.lan
# From the Authentik provider.
AUTHENTIK_ISSUER_URL=https://authentik.example.lan/application/o/homelab-manager/
AUTHENTIK_CLIENT_ID=…
AUTHENTIK_CLIENT_SECRET=…
# Generated by you (see above). Changing SESSION_SECRET signs everyone out.
SESSION_SECRET=…
CREDENTIALS_ENCRYPTION_KEY=…
```
| Variable | Meaning |
|---|---|
| `APP_BASE_URL` | The address people type. The redirect address is `APP_BASE_URL` + `/auth/callback`; it must match what the Authentik provider allows, character for character (scheme, host, port). Also the address used in links in notifications and in the read API. |
| `AUTHENTIK_ISSUER_URL` | The provider's issuer URL. The app *discovers* the rest from `<issuer>/.well-known/openid-configuration`, so a wrong or unreachable issuer is the usual reason sign-in fails. |
| `AUTHENTIK_CLIENT_ID`, `AUTHENTIK_CLIENT_SECRET` | From the provider. The secret is read from the environment only; it is not stored in the database. |
| `SESSION_SECRET` | Signs the session cookie. In production the app **refuses to start** without it, and warns if it is still the placeholder from `.env.example` or shorter than 32 characters. |
| `SESSION_DIR` | Where sessions are kept on disk (default `data/sessions`, inside the data volume). |
| `NODE_ENV` | `production` in the Docker image: makes the cookie `Secure` and requires `SESSION_SECRET`. |
If any of the three `AUTHENTIK_*` values is missing the app still starts — so the rest of it can be looked at — but `/auth/login` fails until they are set. Restart the container after
changing `.env`.
## 3. The first sign-in
**The first person to sign in becomes the administrator**, so sign in yourself first on a new installation. Everyone after that starts as a **viewer** until an administrator
changes their role under **Administration → Users** — or until their groups decide it ([below](#roles)).
Only people Authentik lets into the application can sign in at all, which is why the application's access policy matters: the first account is whoever you let in first.
## Roles
There are three: **viewer** (sees everything, changes nothing), **operator** (acknowledges and silences alerts, starts maintenance, runs updates, writes notes and jobs …) and
**admin** (everything, including settings, integrations and users). [ROLES.md](../ROLES.md) lists exactly who can see and do what.
### By hand (the default)
Administrators change roles on **Administration → Users**. The only administrator cannot be demoted — there is always at least one.
### From groups
Instead of setting roles by hand, let the groups Authentik reports decide them. In Authentik, the user's groups come in a **`groups`** claim that is part of the **`profile`** scope the app
already asks for, so there is nothing to add on the Authentik side except the groups themselves (e.g. `homelab-admins`, `homelab-ops`) and who is in them.
In the app: **Settings → Sign-in → Roles from groups**.
| Setting | Meaning |
|---|---|
| Set roles from groups at each sign-in | The switch. Off, roles are by hand. |
| Claim that holds the groups | `groups` for Authentik. |
| Admin groups | Names (one per line or comma separated; case does not matter). In any of them → **admin**. |
| Operator groups | In any of them (and in no admin group) → **operator**. |
| Someone in none of those groups | **becomes a viewer**, or **keeps the role they already have**. |
The rules are applied **at every sign-in**, so a change in Authentik shows up the next time the person signs in (or when their session next starts), not instantly.
Safeguards:
- If a sign-in carries **no groups claim at all** (a wrong claim name, a scope that is missing), roles are left exactly as they are — "we weren't told" is never treated as "in no groups".
- **The only administrator is never demoted** by a group rule.
- **The first person to sign in is still the administrator**, whatever the groups say.
- While roles follow groups, **roles cannot be set by hand** (a role set on the Users page would be undone at the next sign-in); the Users page says so.
- Every role change made from groups is written to the audit log (`user` / `role_from_groups`, and `role_from_groups_blocked` when the last administrator was protected).
Before switching it on, press **Check against the groups people had last time**: it changes nothing and shows what each person *would* become, from the groups they had at their last sign-in.
The groups each person had are also shown on the Users page, so you can see the names before relying on them.
### Profile pictures
Optional, off by default (**Settings → Sign-in → Profile pictures**): the picture Authentik gives, and/or a Gravatar. Either one means *the server* fetches the picture from an outside
address; people's browsers never do, and the app serves the pictures from itself.
## Sessions and signing out
- A session lasts up to **7 days**. The cookie is `HttpOnly`, `SameSite=Lax` and, in production, `Secure`.
- **Disabling someone in Authentik does not end an app session that is already running.** To cut someone off at once, end their session under **Users → Signed in now**
(administrators); it ends immediately, and they cannot start another once Authentik refuses them. Accounts themselves are not deleted in the app — a person who no longer signs in
simply has no session.
- **Signing out** of the app also ends the person's Authentik session when Authentik supports *RP-initiated logout* (it does), so the next visit asks for a password again.
Ending a session from *Signed in now* ends only the app's own session, not the one at Authentik.
- Sign-ins and sign-outs are in the audit log, with the address they came from.
## What does *not* use SSO
Some addresses are meant for machines and use their own tokens instead of a session. None of them can be used to sign in, and each stops working the moment its token is removed.
| Address | Used by | Token |
|---|---|---|
| `/api/agent/report` | the agent on each server | one per server, shown once when the server is added |
| `/metrics` | Prometheus | a *metrics token* (Settings → API and webhooks) — see [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
| `/api/v1/…` | scripts, Home Assistant | a *personal API token* (user menu → API tokens) — see [API.md](API.md) |
| `/api/heartbeat/<token>` | cron jobs | the heartbeat's own address — see [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) |
| `/api/public/…` | status pages, the wall display, the calendar feed | a *share link* (Settings → Share links) |
| `/health` | container health checks | none (answers `{"ok":true}`) |
## Troubleshooting
A failed sign-in shows a plain page ("Sign-in failed") and nothing more — deliberately, since the reason would tell an attacker what to adjust. **The reason is in the server log and in
the audit log** (*Change log → Raw audit log*, category `session`, action `login_failed`, with the address and a short reason).
| Reason in the log | What it usually means |
|---|---|
| `provider_unreachable` | The app could not reach Authentik to read its configuration: a wrong `AUTHENTIK_ISSUER_URL` (missing trailing slash, wrong slug, `http` instead of `https`), a name the *container* cannot resolve, a firewall, or a certificate the app does not trust (see `NODE_EXTRA_CA_CERTS`). It retries on the next sign-in; no restart is needed. |
| `provider_rejected` | Authentik answered but the exchange failed. Most often: **the redirect URI** in the provider is not exactly `APP_BASE_URL/auth/callback`; the **client secret** is wrong; the provider has **no signing key** (HS256 tokens are refused); a clock that is several minutes off; or the browser took too long and the one-time code ran out. |
| `no_identity` | Authentik signed the person in but sent no `sub`. Check the provider's scopes include `openid`. |
| *(not recorded)* `no_pending_login` | A page left open too long, a bookmarked callback address, or the session cookie was not kept. If it happens every time, the cookie is the problem — see the next row. |
Other symptoms:
| Symptom | Likely cause |
|---|---|
| You sign in at Authentik and land on the login page again, over and over | The session cookie is not being kept. In production it is `Secure`, so the site must be opened over **HTTPS**; behind a reverse proxy, check it sends `X-Forwarded-Proto`. Also check `APP_BASE_URL` is the address in the browser's address bar. |
| Everyone is signed out after a restart | `SESSION_SECRET` changed (or is generated fresh each start), or the `data/sessions` folder is not on the persistent volume. |
| New people are all viewers although roles from groups is on | The sign-in carried no groups: check the claim name, and that the Authentik user is in the groups. Look at the Users page — the groups seen at each person's last sign-in are shown there. |
| Roles from groups did nothing to an administrator | Working as intended if they are the only administrator (a group rule never removes the last one); the audit log says `role_from_groups_blocked`. |
| "Cannot remove the only admin" | Same safeguard, on the Users page. Make someone else an administrator first. |
| A person with a different login at Authentik (renamed, merged) shows up as a new person | The app matches on `sub`, not on name or email. See the subject-mode warning above. |
## If you are locked out
There is no local password, so if the identity provider is down or misconfigured nobody can sign in. People who are **already signed in** keep working until their session ends.
To get back in, fix the provider or the `AUTHENTIK_*` settings and restart the container.
If the only administrator's account was changed so that nobody is an administrator any more, promote someone directly in the database (stop the app, or accept that it is a
single quick write; **take a copy of the data folder first** — see [BACKUP.md](../BACKUP.md)):
```bash
sqlite3 data/homelab-manager.sqlite "UPDATE users SET role = 'admin' WHERE email = 'you@example.lan';"
```
## Security notes
- **PKCE and `state`** protect the code exchange; the callback is refused without the pending sign-in that the same browser started.
- **The ID token's signature is verified** against Authentik's published keys, so the login does not depend on the connection to Authentik alone.
- **A new session id is issued at sign-in**, so an id planted in the browser beforehand cannot be used.
- **Requests that change something are refused when they come from another origin** (the cookie is `SameSite=Lax`, which does not stop another app on the same parent domain), including `GET /auth/logout`.
- **Sign-in is rate-limited** to 20 attempts a minute per address, and each failure is audited.
- Every response carries `frame-ancestors 'none'`, `X-Frame-Options: DENY`, `nosniff` and a same-origin referrer policy.
## Other identity providers
The code uses standard OIDC discovery (`<issuer>/.well-known/openid-configuration`), the authorization code flow with PKCE (`S256`), a confidential client, **RS256-signed ID tokens
with a published JWKS**, and the `sub`, `email`, `name`/`preferred_username`, `picture` and a groups claim (name configurable) — nothing Authentik-specific. Keycloak, Authelia and
similar providers should therefore work by putting *their* issuer URL, client id and secret in the `AUTHENTIK_*` variables and setting the groups claim name under **Settings → Sign-in**.
This has **not been tried**; if you try one, make sure it signs with RS256 and that the redirect address is registered exactly.
+284
View File
@@ -0,0 +1,284 @@
# Webhooks
Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook
here, and they are easy to mix up:
| | **Event webhooks** | **The notification webhook** |
|---|---|---|
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
| Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) |
- [Event webhooks](#event-webhooks)
- [Setting one up](#setting-one-up)
- [Events](#events)
- [The message](#the-message)
- [Checking the signature](#checking-the-signature)
- [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to)
- [Alert keys](#alert-keys)
- [A small receiver](#a-small-receiver)
- [Home Assistant and n8n](#home-assistant-and-n8n)
- [The notification webhook](#the-notification-webhook)
- [Troubleshooting](#troubleshooting)
## Event webhooks
### Setting one up
Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**.
| Field | Notes |
|---|---|
| Name | up to 60 characters; what the list shows |
| Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. |
| Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. |
| Events | at least one of the four below |
- **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one.
- **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong).
- The list shows when the last message was sent and its result — a status code, `timed out`, or the error.
- Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own).
### Events
| Event | Sent when |
|---|---|
| `alert.opened` | A problem begins. |
| `alert.resolved` | A problem ends. |
| `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. |
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
**What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new
ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem
under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem
being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read
those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md).
### The message
```
POST <your address>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
```
```json
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
```
| `data` | Fields |
|---|---|
| `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` |
| `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) |
| `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) |
| `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward |
| `test` | `message` |
A captured test message, as a receiver sees it (headers lower-cased by the receiving server):
```json
{
"headers": {
"x-hlm-event": "test",
"x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61",
"x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57",
"user-agent": "HomelabManager-Webhook",
"content-type": "application/json"
},
"body": {
"event": "test",
"id": "72118185-66dd-49d8-8886-aed8bdc40d61",
"createdAt": "2026-10-20T21:46:21.349Z",
"source": "homelab-manager",
"data": { "message": "This is a test from Homelab Manager." }
}
}
```
### Checking the signature
With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and
re-serialised JSON), compare in **constant time**, and reject a message that does not match.
```js
import { createHmac, timingSafeEqual } from "node:crypto";
export function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
```
```python
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
```
```bash
# Quick check at a shell, with the body saved to a file:
printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')"
```
The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you
have handled and refuse repeats, and reject a `createdAt` that is far from your clock.
### Delivery: what to rely on and what not to
- Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the
result is written on the webhook and forgotten.
- A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address.
- **There is no queue and no replay.** A receiver that is down for longer than those two tries **misses the event**. Messages for different events can arrive in any order. If you need to be certain, treat the
webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now.
- Because a retry reuses the delivery id, **handle each id once** (idempotently).
- Answer quickly (`200` is enough) and do slow work afterwards.
### Alert keys
`alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes:
| Key | About |
|---|---|
| `offline:server:<id>` | A server has stopped reporting |
| `offline:device:<id>` | A UniFi device has been offline too long |
| `disk:server:<id>:<mount>` | A disk is nearly full |
| `disk:proxmox:<integration>:…`, `disk:synology:<integration>:<volume>` | A Proxmox storage or Synology volume is nearly full |
| `synology-volume:<integration>:<volume>`, `synology-disk:<integration>:<disk>` | A Synology volume or disk is not healthy |
| `failed-units:server:<id>` | Services have failed on a server |
| `smart:server:<id>:<device>`, `zfs:server:<id>:<pool>` | A disk says it is failing, a pool is not online |
| `new-port:server:<id>:<proto>:<port>` | A new TCP port started listening |
| `rule:<rule id>:server:<id>[:<disk>]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) |
| `endpoint:<check id>`, `endpoint-cert:<check id>` | An endpoint check is failing, or its certificate is running out |
| `heartbeat:<id>`, `heartbeat-fail:<id>` | A job reported late, or said it failed |
| `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed |
Treat unknown shapes as opaque; match on the whole key.
### A small receiver
Node with Express — the one thing to get right is to verify the signature against the **raw** body:
```js
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
const SECRET = process.env.HLM_WEBHOOK_SECRET;
const seen = new Set(); // use something persistent for real
const app = express();
app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => {
const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex"));
const given = Buffer.from(req.get("x-hlm-signature") ?? "");
if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401);
const msg = JSON.parse(req.body.toString("utf8"));
if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled
seen.add(msg.id);
if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`);
else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`);
res.sendStatus(200);
});
app.listen(8080);
```
Python with Flask:
```python
import hmac, hashlib, os
from flask import Flask, request, abort
SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode()
app = Flask(__name__)
@app.post("/hlm")
def hook():
raw = request.get_data() # the exact bytes
expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")):
abort(401)
msg = request.get_json()
if msg["event"] == "alert.opened":
print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"])
return "", 200
```
### Home Assistant and n8n
- **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/<id>`), which is the access control, so a
secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward.
```yaml
automation:
- alias: "Homelab Manager critical alert"
trigger:
- platform: webhook
webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts
allowed_methods: [POST]
local_only: true
condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}"
action:
- service: notify.mobile_app_phone
data:
title: "Homelab alert"
message: "{{ trigger.json.data.alert.message }}"
```
- **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC
in a Code node as above.
## The notification webhook
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
```
POST <your address>
Content-Type: application/json
X-Webhook-Secret: <the secret> (only if one is set)
{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" }
```
- The body is exactly `{ "content": "<title in bold>\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between).
- `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook.
- It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It
is delivered once with no retry; a failure is written to the server log.
- The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only.
- Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
## Troubleshooting
| Symptom | Cause |
|---|---|
| **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). |
| **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. |
| **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). |
| **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). |
| The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. |
| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. |
| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. |
| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |