diff --git a/AGENT_UPDATE.md b/AGENT_UPDATE.md index 4627d44..7c552c5 100644 --- a/AGENT_UPDATE.md +++ b/AGENT_UPDATE.md @@ -27,6 +27,7 @@ number when it is worth installing the agent again for. | 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. | | 4 (Windows) | Also reports failed services, disk health and Storage Spaces pools (below), and what the machine is (maker, model, serial number). | | 6 (Linux), 5 (Windows) | Also says **what it couldn't collect and why** (no `smartctl`, no systemd, a package manager that didn't answer, Windows Update that timed out …), shown on the server's page with what to do about it (below). | +| 7 (Linux), 6 (Windows) | Also says **which operating system it runs** (distribution and release, or Windows product and build), so the server's page can say when that release stops being supported (below). | | 5 (Linux) | Also says what the machine is — maker, model and serial number, read from the firmware — so the inventory can fill itself in (below). | Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field @@ -244,6 +245,21 @@ what is written there with what the agent says, when you choose to. To see exactly what a host finds, as root: `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` — it prints the update figures, the failed services, the SMART readings and the ZFS pools, says whether it would update itself, and changes nothing. +## What the machine runs (Linux version 7, Windows version 6) + +Each agent says which operating system it runs, in the report's `system.os`: `{ id, name, version }`. + +- **Linux** reads `/etc/os-release` — `ID` (lower case, e.g. `ubuntu`, `debian`, `rocky`), `VERSION_ID` and `PRETTY_NAME`. On a **Proxmox VE** host (where `pveversion` exists) it says `proxmox-ve` and the + release instead, since the Debian underneath is not what needs watching. +- **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart. + +The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows +client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Settings → Notifications (*Flag an +operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent +older than this version reports no OS and the server shows nothing for it. + +`sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` prints what it found. + ## How the update check works — and why a server may stay "not reported" The agent only *asks*; it never installs, downloads or refreshes anything. diff --git a/API.md b/API.md deleted file mode 100644 index b3c6010..0000000 --- a/API.md +++ /dev/null @@ -1,276 +0,0 @@ -# Metrics, webhooks, heartbeats and a calendar feed - -Ways for other systems to work with Homelab Manager: - -- a read-only **`/metrics`** address for Prometheus, and **event webhooks** that are sent a signed message when something happens — both set up by an administrator under **Settings → API and webhooks** (the first two sections below); -- a **read API** (`/api/v1`) with a personal, read-only token for each person, for scripts, Home Assistant and dashboards (the section after `/metrics`); -- **heartbeats**: a cron job or script calls an address when it finishes, and the app speaks up when it doesn't (further down); -- a **calendar feed** of what is coming up, for a phone's calendar to subscribe to (at the end). - -## `/metrics` — for Prometheus - -``` -GET /metrics -Authorization: Bearer hlm_m_… -``` - -- A **metrics token** is made on the settings page. It is shown once, only its SHA-256 is kept, and it can - be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus. -- The token goes in the `Authorization` header **only**. A token in the address (`?token=…`) is not - accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same - `401`. -- The answer is Prometheus's text format, `Content-Type: text/plain; version=0.0.4; charset=utf-8`. - Requests are limited to 120 a minute from one address. -- It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few - database reads. (The alert counts are the app's own result, reused for up to a minute.) - -```yaml -scrape_configs: - - job_name: homelab-manager - scheme: https - metrics_path: /metrics - authorization: - credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx - static_configs: - - targets: ["homelab.example.lan"] -``` - -### What it says - -All gauges. Labels are escaped as Prometheus requires. - -| Series | Labels | Meaning | -|---|---|---| -| `homelab_info` | `environment` | Always 1; says which installation this is. | -| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. | -| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. | -| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. | -| `homelab_server_memory_used_percent` | `server` | Memory in use. | -| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. | -| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). | -| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. | -| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. | -| `homelab_server_reboot_required` | `server` | 1 if a restart is due. | -| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. | -| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. | -| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. | -| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. | -| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. | -| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. | -| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). | -| `homelab_storage_days_to_full` | `pool` | Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb over the last days. | -| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). | -| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. | -| `homelab_incidents_open` | | Incidents that have begun and not yet ended. | -| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. | -| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. | -| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). | -| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. | -| `homelab_alerts_grouped` | | Alerts folded under a server that is down. | -| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. | -| `homelab_maintenance_active_windows` | | Maintenance windows running now. | -| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). | -| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. | -| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. | -| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. | -| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. | -| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. | -| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). | -| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. | -| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). | -| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). | -| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. | -| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. | -| `homelab_scrape_duration_seconds` | | How long building the answer took. | -| `homelab_scrape_timestamp_seconds` | | When the answer was built. | - -Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`, -`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`. - -## The read API — `/api/v1` - -JSON about servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance — for a script, Home Assistant, a dashboard. It only reads. - -``` -GET /api/v1/alerts -Authorization: Bearer hlm_p_… -``` - -- **A personal token**, made by whoever is going to use it under the user menu → **API tokens** (everyone signed in can, whatever their role). It is shown once, only its SHA-256 is kept, and it is that person's own: they see - and remove theirs, nobody else's. It can last 30, 90 or 365 days or until removed, a person may have ten, and it stops working at once when removed — and when the person is. -- The token goes in the `Authorization` header **only**. A token in the address is not accepted (addresses end up in logs). No token, a wrong one, an expired one and a removed one all get the same `401` with a JSON body. -- **Only `GET`** (and `HEAD`). Any other verb is `405`, with a token or without: there is nothing here that changes anything. -- A token sees what its owner sees on the pages — no secrets, no settings, no passwords, no tokens of servers. Requests are limited to 300 a minute from one address. Answers are `Cache-Control: no-store`. -- Dates are ISO 8601 in UTC, sizes are bytes, percentages are 0–100. An address inside the app (`url`) is a full address, built from `APP_BASE_URL`. - -| Address | What it answers | -|---|---| -| `GET /api/v1` | The list of these addresses. | -| `GET /api/v1/me` | Whose token this is: the person's name and role, and the token's name and end. | -| `GET /api/v1/servers` | Every server: `status` (`online`, `offline`, `unknown`), `lastSeenAt`, `tags`, `ipAddresses`, `cpu`, `memory`, `disks` (with `usedPercent`), `updates`, `hardware`, `operatingSystem` (`name`, `version`, `endsOn`, `state`: `ok`, `soon`, `ended` or `unknown`, `daysLeft`; null until the agent says), `url`. Filter with `?tag=` and `?status=`. | -| `GET /api/v1/servers/:id` | One server, the same shape. | -| `GET /api/v1/servers/:id/metrics?days=` | Its load, memory and disks as lines of points, 1 to 365 days (7 by default). Points over more than 35 days are hours' or days' averages, with `lo` and `hi`. Disks also carry a `trend` and, when rising steadily, `daysToFull`. | -| `GET /api/v1/alerts` | What is wrong now: `needingAttention` (critical, warning, acknowledged), `counts`, `couldntCheck`, and `alerts` with `severity`, `category`, `message`, `silenced`, `flapping`, `target`, `causedBy`, `acknowledgement` (who, when, note, until) and `url`. Filter with `?severity=`, `?category=`, `?acknowledged=true|false`, `?silenced=true|false`. | -| `GET /api/v1/incidents?status=&limit=` | The record of alerts that began and ended: `open` or `resolved`, newest first, up to 500 (100 by default). | -| `GET /api/v1/incidents/:id` | One incident with its timeline. | -| `GET /api/v1/inventory` | The hardware: maker, model, serial, where, bought, warranty, end of support, power. | -| `GET /api/v1/network` | Switches, access points and gateways read from UniFi (state, address, uptime, firmware, uplink), the networks and Wi-Fi names, and which server is plugged into which device. | -| `GET /api/v1/endpoints` | The checks: `status` (`up`, `down`, `unknown`, `paused`), last answer, certificate end, 24-hour and 7-day uptime. | -| `GET /api/v1/endpoints/:id/history?days=` | A check's availability and answer times over 1 to 365 days, as points, from each answer (up to 2 days), the hourly figures (up to 120) or the daily ones. | -| `GET /api/v1/maintenance` | What is under maintenance now (`active`) and what is coming (`upcoming`). | - -```json -{ "generatedAt": "2026-10-20T09:14:02.511Z", - "needingAttention": { "critical": 1, "warning": 0, "acknowledged": 1 }, - "alerts": [ - { "id": "offline:offline:server:3", "severity": "critical", "category": "offline", "source": "Server", - "message": "anna hasn't reported for 9 h", "silenced": false, "flapping": false, - "target": { "type": "server", "id": 3, "name": "anna" }, "causedBy": null, "affects": [], - "acknowledgement": { "by": "Ada", "at": "2026-10-20T08:01:00.000Z", "note": "On it", "until": "2026-10-20T12:01:00.000Z", "assignee": "Ada" }, - "url": "https://hm.example.lan/servers/3" } ] } -``` - -Errors are JSON too: `{ "error": "unauthorized" | "read_only" | "not_found" | "invalid_id", "message": … }`. In Home Assistant, a `rest` sensor with the token in `headers:` and `value_template: "{{ value_json.needingAttention.critical }}"` is enough for a count of what needs attention. - -## A Grafana dashboard - -Settings → API and webhooks → **Download the dashboard** (administrators) gives `homelab-manager-dashboard.json`; the same file is at -`GET /api/metrics-tokens/grafana-dashboard.json` for a signed-in administrator. In Grafana: **Dashboards → New → Import**, upload it, and -choose your Prometheus when asked (the file never names one, so it isn't tied to any installation). It is a dashboard with a **Server** -picker (taken from `homelab_server_up`) and five rows: - -| Row | What it has | -|---|---| -| Needs attention | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. | -| Servers | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. | -| Disks | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. | -| Backups | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. | -| Secrets, power and the app itself | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. | - -The dashboard is built by the app from the same code that serves `/metrics`, and a test checks that it only asks for series that are -really served — so after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later -and has been checked for structure (every panel on the grid, every query well-formed, every series real), **not** opened in a Grafana here. - -## Event webhooks - -A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is -separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person -would read. - -| Event | Sent when | -|---|---| -| `alert.opened` | A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. | -| `alert.resolved` | A problem ends. | -| `maintenance.started` | A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). | -| `maintenance.ended` | A maintenance window finishes, on its own or ended early. | - -Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails -(the same message, with the same `X-HLM-Delivery`), then the result is noted on the webhook in Settings -(a status code, or what went wrong). A request is given five seconds, and redirects are not followed. - -### The message - -``` -POST -Content-Type: application/json -User-Agent: HomelabManager-Webhook -X-HLM-Event: alert.opened -X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message) -X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret) -``` - -```json -{ - "event": "alert.opened", - "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c", - "createdAt": "2026-10-17T21:04:11.482Z", - "source": "homelab-manager", - "data": { - "alert": { - "key": "offline:server:3", - "source": "server", - "severity": "critical", - "message": "pve1 hasn't reported for 1 h — also takes down web and db" - } - } -} -``` - -- `alert.opened` data: `alert` with `key` (stable for the same problem), `source`, `severity` (`critical` or - `warning`) and `message`. `alert.resolved` has `alert.key` and `alert.message`. -- `maintenance.started` / `maintenance.ended` data: `window` with `id`, `targetType` (`server`, - `integration`, `dns_provider`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` - (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime - Kuma) and, on `ended`, `endedAt`. -- A **Test** button on the settings page sends `{"event": "test", …}` whatever the webhook is set to. - -### Checking the signature - -With a secret, `X-HLM-Signature` is `sha256=` and the hex HMAC-SHA256 of the **exact body bytes** with the -secret as the key. Compare in constant time, and reject a message whose signature doesn't match. - -```js -import { createHmac, timingSafeEqual } from "node:crypto"; - -function isFromHomelabManager(secret, rawBody, signatureHeader) { - const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`); - const given = Buffer.from(signatureHeader ?? ""); - return given.length === expected.length && timingSafeEqual(given, expected); -} -``` - -```python -import hmac, hashlib - -def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool: - expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest() - return hmac.compare_digest(expected, signature_header or "") -``` - -The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved. -Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never -its address or secret. - -## Heartbeats — for cron jobs and scripts - -A heartbeat is made by an administrator under **Monitoring → Heartbeats**, and its address is shown there to administrators. The job calls it when it finishes: - -``` -GET /api/heartbeat/ it worked -GET /api/heartbeat//start it began (optional: lets the next call say how long it took) -GET /api/heartbeat//fail it ran and went wrong: an alert at once -``` - -```sh -# in cron, after the job -0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx - -# in a script, with a start, a failure and a note -curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start -./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \ - || curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail -``` - -- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work. - Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once). -- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters. -- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address. -- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes; - a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`. -- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them. - -## Calendar feed - -An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL"). - -``` -GET /api/public/calendar/.ics -``` - -- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time. - A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address. -- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and - recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out. -- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app. - An event's `UID` is a hash of what it is and its date, the same on every fetch. -- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does. diff --git a/BACKUP.md b/BACKUP.md index 697f2aa..56379ba 100644 --- a/BACKUP.md +++ b/BACKUP.md @@ -23,7 +23,7 @@ holds `docker-compose.yml`, `.env` and `data/`, and the service is called `homel You don't need the snippets below for the everyday case: Homelab Manager takes a backup of its own database **every day** (03:30 in the notification time zone, the newest 14 kept — both changeable, and it -can be switched off), checks each one as it makes it, and keeps them in `data/backups/` with the same +can be switched off; backups you make by hand, upload or restore from are kept too: the newest ten, three and three), checks each one as it makes it, and keeps them in `data/backups/` with the same names the snippets use. **Settings → Data → Backup** shows them and does the rest: - **Back up now** makes one on the spot. **Download** takes a copy to your computer (the quickest way @@ -51,7 +51,7 @@ tool), and it does not replace a restore drill. Access to all of it is for admin | What | Where | Why | If you lose it | |---|---|---|---| -| **The database** | `data/homelab-manager.sqlite` | Everything: users and roles, servers and their agent tokens (as hashes), scheduled tasks, secrets, IP addresses, DNS caches, domains, ports, tags, maintenance windows, both logs, settings, the integrations and DNS providers (with their credentials, encrypted), the alert-ticket memory | You start from nothing — see [If there is no backup](#if-there-is-no-backup) | +| **The database** | `data/homelab-manager.sqlite` | Everything: users and roles, servers and their agent tokens (as hashes) with their history (load, memory, disks), scheduled tasks, secrets, IP addresses, DNS caches, domains, ports, tags, notes, to-do items, runbooks, the hardware inventory, endpoint checks and heartbeats with their results, your own alert rules, maintenance windows and snoozes, dependencies, incidents, what was read from UniFi, Proxmox and Synology storage, and Home Assistant, both logs, settings, the integrations and DNS providers (with their credentials, encrypted), the alert-ticket memory | You start from nothing — see [If there is no backup](#if-there-is-no-backup) | | **`CREDENTIALS_ENCRYPTION_KEY`** | `.env` | Decrypts every stored credential: integration tokens and passwords, DNS provider credentials, and the Gotify/ntfy/SMTP/webhook credentials | The database restores, but every credential in it is unreadable and has to be entered again | | **The rest of `.env`** | `.env` | `SESSION_SECRET`, `APP_BASE_URL`, the Authentik client id and secret | `SESSION_SECRET`: everyone has to sign in again, nothing more. The Authentik values can be copied from Authentik again | | Your `docker-compose.yml` and the image version | next to `.env` | So you can bring it back the way it was. Note which **version** each backup came from (the image tag, or the git commit) | Re-create it from the README | @@ -282,7 +282,9 @@ settings over the current ones and adds integrations and providers that don't ex by type and name; it never overwrites an existing one. It does **not** contain servers (and their agent tokens), tasks, secrets, IP addresses, domains, -ports, tags, maintenance windows, the users, or either log. So it is the right thing for moving +ports, tags, maintenance windows, the users, or either log — nor the other data the app keeps: endpoint +checks, heartbeats, alert rules, notes and to-do items, the inventory, dependencies, the Home Assistant +watch list, incidents or any history. So it is the right thing for moving your *configuration* to a new instance, or as a second copy of the part that is tedious to re-enter. It is **not** a backup of the system: use the database backup for that. diff --git a/DATABASE.md b/DATABASE.md index 5e2156a..bd8e70e 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -23,8 +23,8 @@ what the app actually creates. |---|---| | **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. | | **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. | -| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0033`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | -| **Tables** | 35 application tables, plus drizzle's own `__drizzle_migrations`. | +| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0051`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | +| **Tables** | 61 application tables, plus drizzle's own `__drizzle_migrations`. | **Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`. @@ -44,7 +44,7 @@ what the app actually creates. - Dates without a time (`secrets.expiry_date`, `domains.expires_at`) are `YYYY-MM-DD`. - **JSON columns** are `TEXT` holding JSON; see [what's inside](#whats-stored-inside-the-json-columns). - **Enumerations** (roles, types) are plain text — SQLite doesn't enforce the allowed values; the app does. -- **Indexes:** besides primary keys, the only indexes are the unique ones listed per table. There are no secondary indexes; the data sets here (hundreds to a few thousand rows) don't need them. +- **Indexes:** besides primary keys and the unique ones listed per table, a few plain indexes serve the tables that are read by range or by what they are about: `endpoint_results`, `heartbeat_pings`, `server_metrics`, `ha_readings` and `storage_samples` (by owner and time), `restore_tests`, `renewals`, `update_runs`, `patch_plan_runs`, `alert_incidents`, `incident_notes`, `alert_tickets`, `runbooks` and `status_items`. The other tables have a few thousand rows at most and need none. ## How the tables relate @@ -78,12 +78,18 @@ erDiagram servers ||--o{ port_forwards : "optional (set null)" integrations ||--o{ servers : "proxmox link (app clears it)" integrations ||--o{ alert_tickets : "osTicket used (set null)" + share_links ||--o{ status_items : "cascade" + users ||--o{ alert_acks : "assignee (set null)" + alert_incidents ||--o{ incident_notes : "cascade" + servers ||--o{ update_runs : "cascade" + patch_plans ||--o{ patch_plan_runs : "cascade" + endpoint_checks ||--o{ endpoint_daily : "cascade" ``` -Twenty tables stand alone, with no foreign keys: `settings`, `notes`, `alert_rules`, `alert_snoozes`, `secrets`, -`ipam_entries`, `domains`, `diag_log`, `notification_queue`, -`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`, `dependencies`, `endpoint_checks`, `restore_tests`, `heartbeats`. `maintenance_windows` and `maintenance_aliases` also have no -foreign key — see [soft references](#soft-references-not-foreign-keys). +These 22 tables are without a foreign key to another table, and — but for `integration_credentials`, which `integrations` and `dns_providers` point at — with none from another table to them; where they refer to something else it is by id or name, see +[soft references](#soft-references-not-foreign-keys): `settings`, `secrets`, `ipam_entries`, `domains`, `diag_log`, `notification_queue`, `consistency_ignores`, `tag_definitions`, +`maintenance_windows`, `maintenance_aliases`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `restore_tests`, `dependencies`, `runbooks`, `renewals`, `assets`, `notes`, +`alert_rules`, `alert_snoozes` and `integration_credentials`. ## Tables @@ -618,7 +624,7 @@ One row per endpoint check per UTC hour (`hour`, ISO): `total`, `ok`, `ms_sum`, #### `api_tokens` -A person's personal read-only API tokens (see API.md): `user_id` (→ `users.id`, **cascade**: they go with the person), `label`, `token_hash` (SHA-256, unique — the secret itself is shown once and not kept), `token_prefix` (its first twelve characters, to tell tokens apart), `created_at`, `last_used_at` (written at most once a minute) and `expires_at` (null = until removed). +A person's personal read-only API tokens (see [docs/API.md](docs/API.md)): `user_id` (→ `users.id`, **cascade**: they go with the person), `label`, `token_hash` (SHA-256, unique — the secret itself is shown once and not kept), `token_prefix` (its first twelve characters, to tell tokens apart), `created_at`, `last_used_at` (written at most once a minute) and `expires_at` (null = until removed). #### `renewals` @@ -641,8 +647,8 @@ A playbook run in Semaphore for one server, wrapped in a maintenance window and | `started_by` | text | | | `started_at`, `task_ended_at`, `host_reported_at`, `finished_at` | text | ISO. | | `updates_before`, `updates_after` | integer | The waiting count before, and from the report that came after. | -| `reboot_required` | bool | From that report. | -| `poll_errors` | integer | Consecutive times Semaphore didn't answer (ten ends the run). | +| `reboot_required` | bool NOT NULL, default 0 | From that report. | +| `poll_errors` | integer NOT NULL, default 0 | Consecutive times Semaphore didn't answer (ten ends the run). | #### `runbooks` @@ -1043,9 +1049,12 @@ What a status page shows, in order. | `synology` | `url`, `username`, `insecure` | `password` | | `semaphore`, `gitea`, `dockhand` | `url` | `token` | | `tailscale` | `tailnet` | `apiKey` | -| `uptimekuma` | `url`, `username` | `password` (an API key, or the password on old installs) | +| `uptimekuma` | `url`, `username`, `loginUsername`, `syncMaintenance` | `password` (an API key, or the password on old installs), `loginPassword` | | `phpipam` | `url`, `appId`, `insecure` | `token` | -| `osticket` | `host`, `port`, `database`, `username`, `tablePrefix` | `password` | +| `osticket` | `host`, `port`, `database`, `username`, `tablePrefix`, `apiUrl`, `apiInsecure` | `password`, `apiKey` | +| `unifi` | `url`, `username`, `site`, `selfHosted`, `insecure` | `apiKey` (or `password`, with `username`) | +| `npm` (Nginx Proxy Manager) | `url`, `email`, `insecure` | `password` | +| `homeassistant` | `url`, `insecure` | `token` | ## Settings keys @@ -1061,6 +1070,7 @@ that were never changed aren't stored; the app fills in defaults when reading. | `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). | | `quietHours` | `enabled`, `start`, `end`. | | `signIn` | Roles from groups: `roleFromGroups` (false), `groupsClaim` (`groups`), `adminGroups`, `operatorGroups`, `otherwise` (`viewer` or `keep`). | +| `profilePictures` | Where pictures of people come from: `identityProvider` (false) and `gravatar` (false). Either one means the server fetches pictures from an outside address; they are checked, kept for a day and served from this app. | | `selfBackup` | The daily backup of this database: `enabled` (true), `time` (`HH:MM`, default `03:30`, in the notification time zone) and `keep` (14 — how many daily ones are kept). | | `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. | | `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. | @@ -1084,18 +1094,25 @@ settings, and aren't part of the app's settings object: | `_internal:integration_history_since` | When the app first began writing integrations' failing / working-again moments; the uptime ribbons claim nothing earlier for integrations. | | `_internal:alertTicketsBaselineDone`, `_internal:alertTicketsLastRun` | Whether the alert-ticket starting line has been recorded since the feature was switched on, and JSON for the last check's result. | | `_internal:webPushVapid` | JSON `{publicKey, privateKey}`: the server's own key pair for web push, made the first time it is needed. The private half is encrypted like the other credentials. Replacing it (deleting the row) makes every browser subscribe again. | -| `_internal:healthActiveConditions`, `_internal:automationActiveConditions` | JSON: the problems currently active, so each is announced once and again when it clears, and survives a restart. | +| `_internal:healthActiveConditions`, `_internal:automationActiveConditions`, `_internal:endpointActiveConditions` | JSON: the problems currently active (health checks; automation runs; endpoint checks and heartbeats), so each is announced once and again when it clears, and survives a restart. | +| `_internal:flapState` | JSON, by alert key: the recent raises of each alert and whether it is flapping (see NOTIFICATIONS.md), so a restart remembers what is being held. | +| `_internal:unifiSync`, `_internal:homeAssistantSync` | JSON, by integration id: how the last read of each controller or Home Assistant went (`at` is the last *successful* read; one older than twenty minutes is not judged). | +| `_internal:selfBackupLastAttempt` | JSON: how the last attempt to back this database up went (time, whether it worked, why not). | +| `_internal:endpointSuggestionsDismissed`, `_internal:dependencySuggestionsDismissed` | JSON arrays of the suggestion keys somebody said "not that" to. | +| `_internal:metricsRollupBackfilled`, `_internal:endpointHourlyBackfilled` | The time the hourly and daily summaries were first built from the samples already kept (once, after they were added). | ## What happens on delete | Deleting… | Effect | |---|---| -| a **server** | Its `scheduled_tasks`, `server_links` and `server_ports` are deleted with it. Port forwards that pointed at it stay, with `server_id` cleared. | +| a **server** | Its `scheduled_tasks`, `server_links`, `server_ports`, `server_metrics`, `server_metrics_rollup`, `update_runs` and `network_server_links` are deleted with it. Port forwards that pointed at it stay, with `server_id` cleared. Things that refer to it by id only (notes, to-do items, endpoint checks, heartbeats, alert rules, dependencies, runbooks, maintenance windows) stay, pointing at nothing — see below. | | a **DNS provider** | Its cached zones and records are deleted. | +| an **integration** | What was read from it is deleted with it: UniFi devices, networks and server links, Home Assistant watched entities and readings, and the history of its storage pools (`storage_series`, `storage_samples`). | +| an **endpoint check**, a **heartbeat**, an **incident**, a **patch plan**, a **share link** | Their results and summaries (`endpoint_results`, `_hourly`, `_daily`), pings, notes, runs and status-page items. | | an **integration** or **DNS provider** | The credential row it used is deleted by the app (not by the database). | | an **integration** used for alert tickets | `alert_tickets.integration_id` is cleared; the history stays. | | an **integration credential** | The integration / provider using it keeps existing, with `credential_id` cleared. | -| a **user** | Their audit entries stay; `actor_user_id` is cleared and `actor_label` keeps the name. | +| a **user** | Their audit entries stay; `actor_user_id` is cleared and `actor_label` keeps the name. Their personal API tokens and push subscriptions are deleted with them; to-do items and acknowledgements handed to them stay, with the assignee cleared and the name kept. (The app has no way to delete a user: the rules apply to a removal made in the database.) | | a **Proxmox integration** that servers are linked to | Those servers keep existing and lose their Proxmox link (all four `proxmox_*` columns). The app does this before deleting; the database alone would refuse — see the next section. | ### Soft references (not foreign keys) @@ -1111,6 +1128,10 @@ a stale value is possible and the app treats it as "nothing there": - `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider) - `servers.tags` (→ `tag_definitions.name`) - `consistency_ignores.key` +- `notes.target_id` and `todos.target_id` (→ a server or a network device, by `target_type`) and `runbooks.target_id` +- `alert_rules.server_id` (→ a server; a rule for a server that is gone judges nothing) +- `alert_acks.alert_key`, `alert_snoozes.alert_id`, `alert_tickets.alert_key` and `alert_incidents.alert_key` (→ an alert, by its id on the Alerts page: the same problem has the same id) +- `network_devices` ↔ `dependencies` and `maintenance_windows` with `device` as their type (→ `network_devices.id`) ## The Proxmox link's foreign key @@ -1132,10 +1153,21 @@ a hand-written SQL statement, say — has to do the same first. ## Retention and backups -**Retention.** Only the two logs are trimmed: `audit_log` and `diag_log` entries -older than `logRetention.retentionDays` are deleted by the purge job (off by -default), and `notification_queue` is emptied each time the quiet-hours digest is -sent. Everything else stays until someone deletes it. +**Retention.** The two logs, `audit_log` and `diag_log`, are trimmed by the purge job once it is switched on (`logRetention`: off by default, then entries older than `retentionDays` +are deleted), and `notification_queue` is emptied each time the quiet-hours digest is sent. The history tables trim themselves: + +| Table | Kept | +|---|---| +| `server_metrics` | 35 days of samples | +| `server_metrics_rollup` | hourly rows 120 days, daily rows about 800 days | +| `endpoint_results` | 14 days of every answer | +| `endpoint_hourly`, `endpoint_daily` | 120 days, and 800 days | +| `storage_samples` | 400 days (a pool not reported for 30 days is deleted with its readings) | +| `ha_readings` | 35 days | +| `heartbeat_pings` | 14 days | +| `alert_incidents`, `incident_notes` | 400 days after the incident resolved | + +Everything else stays until someone deletes it. **Credentials at rest.** Integration and DNS provider credentials are encrypted in `integration_credentials` (above). The notification channels' diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md index 4ea149b..83860f8 100644 --- a/INTEGRATIONS.md +++ b/INTEGRATIONS.md @@ -59,7 +59,7 @@ the dashboard views themselves load fine. (UniFi, Nginx Proxy Manager and Home A ### Uptime Kuma -- **Config fields:** Uptime Kuma URL, username (leave blank — see below), API key +- **Config fields:** Uptime Kuma URL, username (leave blank — see below), API key, and the optional *Login username* and *Login password* (for mirroring maintenance windows) with a checkbox to keep maintenance windows in step with Uptime Kuma's schedule - **Auth:** HTTP Basic, with the API key as the password and the username left empty. Uptime Kuma has no conventional REST API — the dashboard talks to it over Socket.IO — so this integration reads its Prometheus-metrics endpoint (`GET /metrics`) instead and parses that. On installs from before the API-key diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index 052e9d0..be96c45 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -219,10 +219,11 @@ what it would say right now and *Send it now* sends it at once (through the chan **Operations → Alerts** (the *Now* view; *History* lists every incident that has come and gone) shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs -in three ways: +in a few ways: - It ignores the "Notify on" toggles. Turning a notification off doesn't hide the problem from the page. -- Problems under a **maintenance window** are kept on the list, marked *silenced* and counted separately, instead of being dropped. +- Problems under a **maintenance window** are kept on the list, marked *silenced* and counted separately, instead of being dropped. So are **snoozed** ones, marked with when they come back. +- It also lists what is on the Alerts page only: forecasts, operating systems out of support, a restore test that failed or is overdue — and, at the top, what could not be checked. - It also lists things nothing notifies about: Uptime Kuma monitors that are down, osTicket tickets that are overdue, and any integration that's failing its last few calls (before the threshold that triggers an "integration down" notification). @@ -254,12 +255,15 @@ on the settings page use the saved settings. ## What does *not* send a notification -- **Storage forecasts** (a Proxmox storage or Synology volume that will be full in a few days) are on the Alerts page, in the weekly digest and in /metrics, but are not sent as they happen — like the forecast for a - server's disk. - Worth calling out explicitly, since it's easy to assume everything in the app alerts on something: +- **Forecasts** — a server's disk, a Proxmox storage or a Synology volume that will be full within the forecast window — are on the Alerts page, in the weekly digest and in /metrics, + but are not sent as they happen. +- **Operating-system support** — a server whose OS has passed, or is about to pass, its end of support — is on the Alerts page (a warning once it has ended, a note while it is coming), not sent. +- **A snoozed alert**: its raise and recovery messages from the health and automation checks are held until the snooze ends (then one "Snooze ended" message, if it is still there). The summaries + other checks send — stale backups, expiring secrets, Docker updates — are not held. + - **osTicket's own tickets** — the integration lists open tickets, but there is no notification for it (e.g. nothing pings you about a newly-overdue ticket); the page and the Alerts page show them. (The diff --git a/README.md b/README.md index a47b3c2..e3b3acf 100644 --- a/README.md +++ b/README.md @@ -254,7 +254,7 @@ audit log (as `session` / `login_failed`, with the address and a short reason), Every response carries `frame-ancestors 'none'`, `X-Frame-Options`, `nosniff` and a no-referrer-to-other-sites policy. In production the app refuses to start without a `SESSION_SECRET`, and warns if it is still the placeholder from `.env.example`. Things to know: the first account to sign in becomes admin, so sign in yourself first on a new install. A session lasts up to 7 days after last use, -and disabling someone in Authentik does not end it — delete them on the Users page (or end the session under Users → *Signed in now*) and it ends at once. +and disabling someone in Authentik does not end it — end the session under Users → *Signed in now* and it ends at once (accounts themselves are not deleted in the app). If Authentik uses a certificate from an internal CA, point Node at the CA with `NODE_EXTRA_CA_CERTS=/path/to/ca.pem` rather than turning verification off with `NODE_TLS_REJECT_UNAUTHORIZED=0`. @@ -597,7 +597,7 @@ wrong — is a *critical* alert at once, with what it said. `/start` before the (`--data-raw "3.2 GB copied"`, or `?msg=`) is kept as a note, up to 200 characters. A heartbeat can be linked to the server the job runs on, so maintenance there silences it and the server's tags route it. Everyone can look; only administrators see the addresses, make, change and remove heartbeats, or give one a **new address** if it got out. The address needs no sign-in (it is a token, like an agent's), and all it can do is say "I'm alive": it can't read anything. Calls are limited per address, and an -unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [API.md](API.md). +unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [docs/HEARTBEATS-AND-CALENDAR.md](docs/HEARTBEATS-AND-CALENDAR.md). **Coming up** (Operations → Coming up, and a card on the dashboard). Everything with a date ahead in one list, nearest first: secrets, domains and certificates that expire (including the ones endpoint checks see), Tailscale keys that are expiring, hardware warranties and end of support, planned and **recurring** maintenance, restore @@ -684,7 +684,7 @@ expire — to a request with a metrics token (a bearer token made on that page, once, only its hash kept). Event webhooks send a signed JSON message to a URL when an alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n or a status board. Both are described, with the exact series and message formats, in -[API.md](API.md). +[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md). **Monthly report** (Operations → Monthly report; and, if switched on, a message on a day of the month). How a calendar month went, on one page that can be read, printed and copied as text: **availability** (servers, from when each stopped reporting and reported again; endpoints, from every check's answers, kept day by day for years; integrations), **incidents** (how many began and @@ -751,7 +751,7 @@ asked. It has rows for what needs attention, servers (with a server picker), dis filling up, SMART health, SSD wear, temperature, ZFS pools), backups (guests by state, age of each newest backup, the app's own backup) and secrets, power and the app itself. It is built from the same code as `/metrics`, and a test checks that it only asks for series that -are really served. See [API.md](API.md). +are really served. See [docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md). **Since you last looked.** The dashboard opens with a card on what happened since you last pressed "Got it" (the first time, the last day; never more than @@ -812,7 +812,7 @@ months, days for over two years); a check's answers are kept as hours for four m stands for an hour or a day, a gap in the data is a gap in the line, and the range you pick is remembered for the next chart. **Endpoints → History…** opens a check's charts; a server's **History** section has its availability (30 days to a year) above its trends. **API tokens.** The user menu → **API tokens** (for everyone signed in): make a read-only token for a script, Home Assistant or a dashboard — shown once, 30, 90 or 365 days or no end, ten at most, removed at once on request and with the person — and use it with `GET /api/v1/…` for -servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance. The addresses, what each answers and an example are in [API.md](API.md). +servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance. The addresses, what each answers and an example are in [docs/API.md](docs/API.md). **UniFi network equipment.** Add a UniFi controller under Integrations (an API key, or a read-only local account; see INTEGRATIONS.md) and **Network → Network devices** lists its switches, access points and gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab — @@ -929,7 +929,14 @@ needs the site to be served over HTTPS (browsers only offer install on secure origins; `localhost` also counts). The bundled service worker deliberately caches nothing, so an installed copy always shows the current build. -**More documentation**: [ROLES.md](ROLES.md) — who can see and do what; +**More documentation** — the guides are in [docs/](docs/README.md): +[docs/SSO.md](docs/SSO.md) — single sign-on with Authentik, what the setup needs, roles from groups, troubleshooting; +[docs/API.md](docs/API.md) — the read API (`/api/v1`) and the other machine-facing addresses; +[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) — `/metrics`, Prometheus, the Grafana dashboard and every series; +[docs/WEBHOOKS.md](docs/WEBHOOKS.md) — event webhooks (signed JSON) and the notification webhook; +[docs/HEARTBEATS-AND-CALENDAR.md](docs/HEARTBEATS-AND-CALENDAR.md) — cron-job heartbeats and the calendar feed; +[docs/MONITORING.md](docs/MONITORING.md) — how monitoring works, from a reading to an alert. +The references: [ROLES.md](ROLES.md) — who can see and do what; [NOTIFICATIONS.md](NOTIFICATIONS.md) — every notification the app sends and when; [INTEGRATIONS.md](INTEGRATIONS.md) — the credentials each integration needs; [DATABASE.md](DATABASE.md) — the database tables, columns and relationships; diff --git a/ROLES.md b/ROLES.md index 00936e1..a045760 100644 --- a/ROLES.md +++ b/ROLES.md @@ -19,49 +19,59 @@ page's own top-level link. | Menu item | Path | Viewer | Operator | Admin | |---|---|:---:|:---:|:---:| | Dashboard | `/` | ✅ | ✅ | ✅ | +| My work | `/my-work` | ❌ | ✅ | ✅ | | **Infrastructure** | | | | | | Servers | `/servers` | ✅ | ✅ | ✅ | | Proxmox | `/proxmox` | ✅ | ✅ | ✅ | | Synology | `/synology` | ✅ | ✅ | ✅ | -| Proxmox Backup | `/pbs` | ✅ | ✅ | ✅ | -| Backup coverage | `/backups` | ✅ | ✅ | ✅ | -| Inventory | `/inventory` | ✅ | ✅ | ✅ | -| Dependencies (Map → Dependencies) | `/map?view=dependencies` | ✅ | ✅ | ✅ | -| Endpoints | `/endpoints` | ✅ | ✅ | ✅ | -| Change log | `/changes` | — | ✅ | ✅ | | Docker | `/docker` | ✅ | ✅ | ✅ | | Tailscale | `/tailscale` | ✅ | ✅ | ✅ | +| Map (Connections, and Dependencies) | `/map` | ✅ | ✅ | ✅ | +| Inventory (and Racks, Labels, Lifecycle) | `/inventory` | ✅ | ✅ | ✅ | +| Admin Links | `/admin-links` | ✅ | ✅ | ✅ | +| **Backups** | | | | | +| Backup coverage | `/backups` | ✅ | ✅ | ✅ | +| Proxmox Backup | `/pbs` | ✅ | ✅ | ✅ | | **Network** | | | | | -| DNS | `/dns` | ✅ | ✅ | ✅ | -| Domains | `/domains` | ✅ | ✅ | ✅ | +| DNS (and Domains) | `/dns` | ✅ | ✅ | ✅ | +| Network devices | `/network/devices` | ✅ | ✅ | ✅ | +| Proxy hosts | `/proxy-hosts` | ✅ | ✅ | ✅ | | IP Addresses | `/ipam` | ✅ | ✅ | ✅ | | Ports | `/ports` | ✅ | ✅ | ✅ | | Consistency | `/consistency` | ✅ | ✅ | ✅ | +| **Monitoring** | | | | | +| Endpoints | `/endpoints` | ✅ | ✅ | ✅ | +| Alert rules | `/alert-rules` | ✅ | ✅ | ✅ | +| Heartbeats | `/heartbeats` | ✅ | ✅ | ✅ | +| Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ | +| Home Assistant | `/home-assistant` | ✅ | ✅ | ✅ | | **Automation** | | | | | | Semaphore | `/semaphore` | ✅ | ✅ | ✅ | | Gitea | `/gitea` | ✅ | ✅ | ✅ | -| Secrets | `/secrets` | ✅ | ✅ | ✅ | -| **Operations** | | | | | -| Alerts | `/alerts` | ✅ | ✅ | ✅ | -| Maintenance | `/maintenance` | ✅ | ✅ | ✅ | -| Updates | `/updates` | ✅ | ✅ | ✅ | -| Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ | | osTicket | `/osticket` | ✅ | ✅ | ✅ | -| Admin Links | `/admin-links` | ✅ | ✅ | ✅ | -| Generator | `/generator` | ✅ | ✅ | ✅ | +| Secrets (a top-level entry) | `/secrets` | ✅ | ✅ | ✅ | +| **Operations** | | | | | +| Alerts (Now, History, Noise) | `/alerts` | ✅ | ✅ | ✅ | +| Maintenance | `/maintenance` | ✅ | ✅ | ✅ | +| Updates (and Patch plans) | `/updates` | ✅ | ✅ | ✅ | +| Coming up | `/coming-up` | ✅ | ✅ | ✅ | +| To do | `/todo` | ✅ | ✅ | ✅ | +| Monthly report | `/reports` | ✅ | ✅ | ✅ | +| Change log (and the raw audit log, `?view=audit`) | `/changes` | ❌ | ✅ | ✅ | | **Administration** | | | | | | Integrations | `/integrations` | ✅ | ✅ | ✅ | +| Diagnostic Log (Integrations → Call log) | `/integrations?view=calls` | ❌ | ❌ | ✅ | +| Generator | `/generator` | ✅ | ✅ | ✅ | | Users | `/users` | ❌ | ❌ | ✅ | | Sessions (Users → Signed in now) | `/users?view=sessions` | ❌ | ❌ | ✅ | -| Audit Log (Change log → Raw audit log) | `/changes?view=audit` | ❌ | ✅ | ✅ | -| Diagnostic Log (Integrations → Call log) | `/integrations?view=calls` | ❌ | ❌ | ✅ | | Settings | `/settings` | ❌ | ❌ | ✅ | -| Privacy (footer link, not in a group) | `/privacy` | ✅ | ✅ | ✅ | +| **Outside the menu** | | | | | +| API tokens (user menu) | `/api-tokens` | ✅ | ✅ | ✅ | +| Privacy (footer link) | `/privacy` | ✅ | ✅ | ✅ | -So in practice: **every page is visible to every role except the five -under Administration** — Users (with Signed in now) and Settings need admin; the -Call log tab of Integrations needs admin (operators and viewers see Integrations without it); the raw audit log, a tab of the Change log, needs operator or admin. -Sessions, the Audit Log, the Diagnostic Log and Dependencies were pages of their own once; their old addresses redirect to the sections above. +So in practice: **every page is visible to every role except a few** — *My work* and the *Change log* (with the raw audit log) need operator; *Users* (with *Signed in now*) and *Settings* need +admin; and the *Call log* tab of Integrations needs admin (operators and viewers see Integrations without it). The sections that used to be pages of their own — Sessions, the audit log, the +Diagnostic Log, Dependencies, Domains — now live inside those pages, and their old addresses redirect there. ## Being able to see a page isn't the same as being able to change things diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..098450f --- /dev/null +++ b/docs/API.md @@ -0,0 +1,377 @@ +# API + +Homelab Manager has two kinds of programmatic interface: + +| Interface | For | Who can use it | Section | +|---|---|---|---| +| **The read API** — `/api/v1` | Scripts, Home Assistant, dashboards that want JSON about servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance | Anyone signed in can make a personal, read-only token | [The read API](#the-read-api) | +| **Machine addresses** — agent reports, heartbeats, `/metrics`, share links | The agents, cron jobs, Prometheus, status pages and calendars | Each has its own token | [Other addresses for machines](#other-addresses-for-machines) | + +**There is no API that changes anything.** The read API refuses every verb but `GET` and `HEAD`. Everything else the app does (acknowledge an alert, start maintenance, add a check, change a setting) +is done in the web interface, which is not a supported API: its addresses and shapes change with the pages. The read API's shapes are a deliberate, stable contract. + +Examples below use `https://hm.example.lan` for your `APP_BASE_URL`. Related pages: [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) for `/metrics`, [WEBHOOKS.md](WEBHOOKS.md) for being *told* when something +happens instead of asking, [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md), [MONITORING.md](MONITORING.md) for what the alert fields mean. + +## The read API + +### Getting a token + +1. Sign in, open the **user menu → API tokens**. This is open to everyone signed in, whatever their role. +2. Give the token a name (e.g. `Home Assistant`) and choose how long it lasts: **30, 90 or 365 days, or until removed**. +3. Copy the token (it starts with `hlm_p_`). **It is shown once**; only its SHA-256 is kept. + +A token belongs to the person who made it. They see and remove their own, nobody else's. A person can have **ten**. A token **stops working at once** when it is removed, when it expires and when +its owner is removed. It sees what its owner sees on the pages: no secrets, no passwords, no server tokens, no settings. + +### Making a request + +```bash +curl -H "Authorization: Bearer hlm_p_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/api/v1/alerts +``` + +- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted**, because addresses end up in logs. +- **No token, a wrong one, an expired one and a removed one all get the same answer**: `401` with a JSON body and `WWW-Authenticate: Bearer realm="homelab-manager"`. +- **Only `GET` and `HEAD`.** Any other verb gets `405` (`{"error":"read_only"}`) with `Allow: GET, HEAD` — with a token or without. +- **Limits:** 300 requests a minute from one address (`429` beyond that). Answers carry `Cache-Control: no-store`. +- **Be gentle:** the data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes). Polling more often than once a minute gains nothing. + +### Conventions + +- Times are **ISO 8601 in UTC** (`2026-10-20T21:44:30.244Z`). A few older fields written by the database itself (`createdAt` on a server) are `YYYY-MM-DD HH:MM:SS` in UTC. +- Sizes are **bytes**. Percentages are **0–100** (not 0–1). +- `url` is a **full address** into the app, built from `APP_BASE_URL`, so a script does not need to know where the app lives. +- Missing information is `null`, never an invented value. A server whose agent has never said which OS it runs has `operatingSystem: null`. +- Errors are JSON: `{ "error": "", "message": "…" }` with `error` one of `unauthorized`, `read_only`, `not_found`, `invalid_id`. +- Unknown addresses under `/api/v1` get `404` and point at `GET /api/v1`. + +### Addresses + +| Address | What it answers | +|---|---| +| `GET /api/v1` | The list of these addresses. | +| `GET /api/v1/me` | Whose token this is. | +| `GET /api/v1/servers` | Every server. Filter with `?tag=` and `?status=online\|offline\|unknown`. | +| `GET /api/v1/servers/:id` | One server, the same shape. | +| `GET /api/v1/servers/:id/metrics?days=` | Its load, memory and disks over time. `days` 1–365 (7 by default). | +| `GET /api/v1/alerts` | What is wrong now. Filter with `?severity=`, `?category=`, `?acknowledged=true\|false`, `?silenced=true\|false`. | +| `GET /api/v1/incidents` | The record of alerts that began and ended. `?status=open\|resolved`, `?limit=` (1–500, default 100). | +| `GET /api/v1/incidents/:id` | One incident with its timeline. | +| `GET /api/v1/inventory` | The hardware inventory. | +| `GET /api/v1/network` | Network equipment read from UniFi: devices, networks, Wi-Fi names, and which server is plugged into which device. | +| `GET /api/v1/endpoints` | The endpoint checks and how they are doing. | +| `GET /api/v1/endpoints/:id/history?days=` | One check's availability and answer times over time. | +| `GET /api/v1/maintenance` | Maintenance windows running now and coming up. | + +### `GET /api/v1/me` + +```json +{ + "user": { "id": 1, "name": "Demo Admin", "email": "demo@example.lan", "role": "admin" }, + "token": { "id": 1, "label": "Home Assistant", "expiresAt": "2027-01-18T21:45:43.390Z" } +} +``` + +`expiresAt` is `null` for a token with no end. A quick way to check a token works. + +### `GET /api/v1/servers` + +```json +{ "servers": [ + { + "id": 1, + "name": "wiki-host", + "hostname": "wiki-host.lan", + "description": null, + "os": "linux", + "tags": ["prod"], + "ipAddresses": ["192.168.1.20"], + "status": "online", + "lastSeenAt": "2026-10-20T21:44:30.244Z", + "createdAt": "2026-01-01 00:00:00", + "agentVersion": 7, + "runsOnServerId": null, + "hardware": { "manufacturer": null, "model": null, "serial": null }, + "operatingSystem": { + "id": "ubuntu", "name": "Ubuntu 24.04 LTS", "version": "24.04", "product": "Ubuntu 24.04 LTS", + "endsOn": "2029-05-31", "state": "ok", "daysLeft": 951 + }, + "cpu": { "model": "Intel(R) N100", "cores": 4, "loadPercent": 12.4 }, + "memory": { "totalBytes": 16623411200, "usedBytes": 4210356224 }, + "disks": [ { "mount": "/", "sizeBytes": 245107195904, "usedBytes": 61276798976, "usedPercent": 25 } ], + "updates": { "manager": "apt", "count": 5, "security": 2, "rebootRequired": false, "reportedAt": "2026-10-20T21:44:30.000Z" }, + "health": { "failedUnits": 0, "reportedAt": "2026-10-20T21:44:30.000Z" }, + "thresholds": null, + "url": "https://hm.example.lan/servers/1" + } +] } +``` + +| Field | Meaning | +|---|---| +| `status` | `online` while the agent has reported within the offline limit (60 minutes unless the server has its own), `offline` after, `unknown` if it never has. | +| `lastSeenAt` | When the agent last reported. | +| `agentVersion` | The version of the agent script, as it says; `null` until it has said. | +| `runsOnServerId` | The server this one runs on (a VM's Proxmox host, say), when one is set. | +| `operatingSystem` | What the agent says it runs, and when that release stops being supported. `state` is `ok`, `soon` (inside the warning window), `ended` or `unknown` (a release the app has no date for); `daysLeft` is negative once it has ended. `null` until the agent has said. | +| `cpu.loadPercent` | Load average as a share of the cores — an approximation, not instantaneous usage. | +| `disks[].usedPercent` | One decimal. | +| `updates` | What is waiting to be installed; `null` when the agent does not report it. `security` is `null` when the package manager cannot say. | +| `health.failedUnits` | The number of failed services (Linux), when the agent reports them. | +| `thresholds` | The server's own alert limits, if it has any that replace the global ones; otherwise `null`. | + +### `GET /api/v1/servers/:id/metrics?days=` + +```json +{ "days": 7, "series": [ + { "metric": "cpu", "kind": "cpu", "label": "CPU load", "latest": 12.4, "trend": null, + "points": [ { "t": "2026-10-20T20:59:30.572Z", "v": 12 }, { "t": "2026-10-20T21:14:30.572Z", "v": 13.5 } ] }, + { "metric": "disk:/", "kind": "disk", "label": "Disk /", "latest": 25, + "trend": { "perDay": 0.4, "fit": 0.97, "spanDays": 13.9, "daysToFull": 188 }, + "points": [ … ] } +] } +``` + +- `kind` is `cpu`, `mem` or `disk`; a disk's `metric` is `disk:`. Values are percentages. +- The range decides how fine the points are: up to **35 days** they are the samples themselves (one every 10–15 minutes, thinned to at most about 100 points); beyond that they are **hourly (to 120 days) or daily + averages** and carry `lo` and `hi` (the lowest and highest in that hour or day). +- A disk's `trend` is a straight-line fit: `perDay` is points per day, `fit` is how straight (0–1), `daysToFull` is set only when the disk is rising steadily. `null` for CPU and memory, and for a disk + with too little history. + +### `GET /api/v1/alerts` + +```json +{ + "generatedAt": "2026-10-20T21:45:53.522Z", + "needingAttention": { "critical": 1, "warning": 3, "acknowledged": 0 }, + "counts": { "critical": 1, "warning": 3, "info": 0, "silenced": 0, "grouped": 0 }, + "couldntCheck": [ + { "name": "Proxmox “Lab PVE”", "error": "connect ECONNREFUSED 127.0.0.1:3071", "affects": ["Proxmox backups", "Server and storage health"] } + ], + "alerts": [ + { + "id": "monitoring:rule:1:server:1", + "severity": "critical", + "category": "monitoring", + "source": "Server", + "message": "wiki-host: cpu load is 95%, above 90% for 30 min [CPU above 90% for 30 minutes]", + "url": "https://hm.example.lan/servers/1?view=history", + "silenced": false, + "snoozedUntil": null, + "flapping": false, + "target": { "type": "server", "id": 1, "name": "wiki-host" }, + "causedBy": null, + "affects": [], + "acknowledgement": null + } + ] +} +``` + +| Field | Meaning | +|---|---| +| `needingAttention` | What wants a look: critical and warning alerts that are not silenced and that nobody has acknowledged, and how many somebody has. This is the number on the menu badge. | +| `counts` | Everything active by severity, whether acknowledged or not. `silenced` are held back by maintenance (or snoozed); `grouped` are folded under the host that explains them. Neither is in `critical`/`warning`/`info`. | +| `couldntCheck` | Sources the app could not read this time, and which checks that leaves blind. **An empty `alerts` list with something here is not "all clear".** | +| `id` | Stable for the same problem: `:`. | +| `severity` | `critical`, `warning` or `info`. | +| `category` | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring` or `tickets`. | +| `source` | Where it comes from: `Server`, `Proxmox`, `Synology`, `UniFi`, `Endpoint`, `Heartbeat` … | +| `silenced` | Under a maintenance window, snoozed, or folded under a host that is down. Notifications for it are held back. | +| `snoozedUntil` | When a snooze on it ends; `null` otherwise. | +| `flapping` | It keeps being raised and cleared; its messages are being held back. | +| `target` | What a maintenance window would silence it with (`server`, `integration`, `dns_provider` or `device`), or `null`. | +| `causedBy` | The alert it is folded under, as `{ name, alertId? }`, or `null`. | +| `affects` | On a host's alert: the servers on it whose own alerts are folded under this one. | +| `acknowledgement` | `{ by, at, note, until, assignee }` while someone has said they are on it; otherwise `null`. | + +### `GET /api/v1/incidents` and `/:id` + +An *incident* is one time an alert was raised and later went away; the same alert coming back is a new incident. + +```json +{ "incidents": [ + { "id": 4, "alertKey": "updates:os-support:1", "severity": "warning", "peakSeverity": "warning", "category": "updates", + "source": "Server", "message": "wiki-host runs Ubuntu 18.04 LTS, whose support ended on 2023-05-31 …", + "link": "/servers/1", "target": { "type": "server", "id": 1, "name": "wiki-host" }, "causedBy": null, + "openedAt": "2026-10-20T21:45:53.522Z", "resolvedAt": null, "durationMs": 127, "open": true, "notes": 0 } +] } +``` + +`GET /api/v1/incidents/:id` adds the **timeline** (when it was raised, acknowledged, handed over, what was done to the thing it is about in the hour around it, when it resolved) and the notes +people wrote. Open incidents are always listed; resolved ones newest first, up to `limit`. + +### `GET /api/v1/inventory` + +```json +{ "assets": [ + { "id": 3, "name": "NAS", "kind": "server", "manufacturer": "Synology", "model": "DS923+", "serial": "…", "location": "Rack A", "rackUnit": "U6-9", + "purchaseDate": "2024-03-02", "purchasePrice": 6500, "warrantyUntil": "2027-03-02", "endOfSupport": null, "replacementCost": 7200, + "powerWatts": 120, "notes": null, "server": { "id": 6, "name": "nas" }, "updatedAt": "2026-10-20T21:45:30.244Z" } +] } +``` + +`server` links a piece of hardware to the server it is, when it is one. `kind` is `server`, `storage`, `network`, `power`, `client` or `other`. + +### `GET /api/v1/network` + +Present once a UniFi integration has been read: `{ devices, networks, ssids, serverLinks }` — each device with its `state` (`online`, `offline`, `updating`, `pending`, `other`), `ip`, `mac`, `uptimeSeconds`, +firmware, what it is connected through and what depends on it; the networks (VLAN, address range); the Wi-Fi names (security, which network); and `serverLinks` saying which server is plugged into which +device (and port). Wi-Fi passphrases are never read, so they are never here. + +### `GET /api/v1/endpoints` and `/:id/history` + +```json +{ "endpoints": [ + { "id": 1, "name": "Wiki", "kind": "http", "target": "https://192.168.1.20/", "port": null, "enabled": true, + "status": "up", "lastCheckedAt": "2026-10-20T21:50:00.000Z", "lastResponseMs": 41, "lastMessage": null, + "certificateExpiresAt": "2027-01-04T12:00:00.000Z", "uptime24hPercent": 100, "uptime7dPercent": 99.9, + "averageResponseMs24h": 44, "serverId": 1 } +] } +``` + +`status` is `up`, `down`, `unknown` (never asked yet) or `paused` (switched off). `lastMessage` is set only while it is failing (why). `kind` is `http`, `tcp` or `tls`. + +`/:id/history?days=` answers `{ check: {id, name}, … points … }`: the availability and answer times over 1–365 days, drawn from each answer (up to 2 days), the hourly figures (up to 120 days) or the daily +ones, whichever is fine enough and still kept. + +### `GET /api/v1/maintenance` + +```json +{ "active": [ + { "id": 1, "target": { "type": "server", "id": 3, "name": "photos", "kind": "Server" }, "reason": "Kernel update", + "startsAt": "2026-10-20T21:45:43.616Z", "endsAt": "2026-10-20T22:45:43.616Z", "createdBy": "Demo Admin" } ], + "upcoming": [] } +``` + +### Examples + +**curl and `jq`: is anything critical?** + +```bash +curl -fsS -H "Authorization: Bearer $HLM_TOKEN" https://hm.example.lan/api/v1/alerts \ + | jq -r '.alerts[] | select(.severity=="critical" and (.silenced|not)) | "\(.source): \(.message)"' +``` + +**A shell check that fails when something needs attention** (for cron, or a Kuma "push" monitor): + +```bash +n=$(curl -fsS -H "Authorization: Bearer $HLM_TOKEN" https://hm.example.lan/api/v1/alerts | jq '.needingAttention.critical') +[ "$n" -eq 0 ] || { echo "$n critical alerts"; exit 1; } +``` + +**Python:** + +```python +import os, requests + +BASE = "https://hm.example.lan/api/v1" +HEADERS = {"Authorization": f"Bearer {os.environ['HLM_TOKEN']}"} + +def get(path, **params): + r = requests.get(f"{BASE}{path}", headers=HEADERS, params=params, timeout=10) + r.raise_for_status() + return r.json() + +for s in get("/servers", status="offline")["servers"]: + print(s["name"], "last seen", s["lastSeenAt"]) + +full = [(s["name"], d["mount"], d["usedPercent"]) for s in get("/servers")["servers"] for d in s["disks"] if d["usedPercent"] and d["usedPercent"] > 85] +print(full) +``` + +**Node.js (18+):** + +```js +const res = await fetch("https://hm.example.lan/api/v1/endpoints", { headers: { Authorization: `Bearer ${process.env.HLM_TOKEN}` } }); +if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); +const { endpoints } = await res.json(); +console.log(endpoints.filter((e) => e.status === "down").map((e) => e.name)); +``` + +**Home Assistant** — put the whole header value, including the word `Bearer`, in `secrets.yaml`: + +```yaml +# secrets.yaml +hlm_auth: "Bearer hlm_p_xxxxxxxxxxxxxxxxxxxxxxxx" +``` + +```yaml +# configuration.yaml +rest: + - resource: https://hm.example.lan/api/v1/alerts + scan_interval: 60 + headers: + Authorization: !secret hlm_auth + sensor: + - name: "Homelab critical alerts" + value_template: "{{ value_json.needingAttention.critical }}" + - name: "Homelab warnings" + value_template: "{{ value_json.needingAttention.warning }}" + binary_sensor: + - name: "Homelab needs attention" + value_template: "{{ value_json.needingAttention.critical > 0 }}" + device_class: problem +``` + +A similar sensor on `/api/v1/servers?status=offline` with `value_template: "{{ value_json.servers | length }}"` counts servers that have gone quiet. + +## Other addresses for machines + +These do not use a session and are not part of the read API. Each is protected by its own token and does one thing. + +### Agent report — `POST /api/agent/report` + +What the agents on the servers call, every 15 minutes by default. The token is the **server's own** (shown once when the server is added under **Servers → Add a server**, and put in the agent by the install +one-liner); it can only replace *that* server's report. You can call it yourself from anything that can make an HTTPS request — a router, a NAS, a script — to make a server appear and be monitored. + +```bash +curl -fsS -X POST https://hm.example.lan/api/agent/report \ + -H "Authorization: Bearer " -H "Content-Type: application/json" \ + -d '{ + "hostname": "nas", + "os_type": "linux", + "agent_version": 7, + "reported_at": "2026-10-20T21:50:00Z", + "system": { + "ip_addresses": ["192.168.1.30"], + "cpu": { "model": "Intel(R) N100", "cores": 4, "load_percent": 12.4 }, + "memory": { "total_bytes": 16623411200, "used_bytes": 4210356224 }, + "disks": [ { "mount": "/", "size_bytes": 245107195904, "used_bytes": 61276798976 } ], + "listening_ports": [ { "protocol": "tcp", "port": 22, "address": "0.0.0.0", "process": "sshd" } ], + "updates": { "manager": "apt", "count": 5, "security": 2, "reboot_required": false }, + "health": { + "failed_units": [ { "unit": "foo.service", "description": "Foo" } ], + "smart": [ { "device": "/dev/sda", "model": "WD Red", "healthy": true, "temperature_c": 38, "wear_percent": null } ], + "zfs": [ { "name": "tank", "health": "ONLINE", "capacity_percent": 61 } ] + }, + "hardware": { "manufacturer": "ASUS", "model": "PN42", "serial": "…" }, + "os": { "id": "debian", "name": "Debian GNU/Linux 12", "version": "12" } + }, + "tasks": [ { "schedule_type": "cron", "name": "backup", "command": "/usr/local/bin/backup.sh", "schedule_expression": "0 3 * * *", "source": "root" } ] + }' +``` + +- **Everything in `system` is optional**, and a part that is malformed is dropped rather than costing the report. Text is trimmed to a sane length. `tasks` may be `[]` (but is required). +- Answers: `202` with `{ "ok": true, "taskCount": 1 }` when the report is taken; `401` for no or a wrong token (`missing_token`, `invalid_token`); `400` (`invalid_body`) for a report that is not JSON of this shape. Reports may be up to **4 MB**. +- `schedule_type` is `cron`, `systemd_timer` or `windows_task`. `os_type` is recorded only if it is `linux` or `windows`. The same token replaces the previous report's tasks and figures. +- `GET /api/agent/report/update?version=` (same token) answers `{ "update": bool, "version": …, "sha256": … }` for an agent that has been told to keep itself up to date; see [AGENT_UPDATE.md](../AGENT_UPDATE.md). + +How the figures are used is in [MONITORING.md](MONITORING.md). + +### Heartbeats, metrics, status pages and calendars + +| Address | What | Token | Documented in | +|---|---|---|---| +| `GET /api/heartbeat/[/start\|/fail]` | A cron job says it ran | in the address | [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) | +| `GET /metrics` | Prometheus text format | `Authorization: Bearer hlm_m_…` | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) | +| `GET /api/public/calendar/.ics` | A calendar feed of what is coming up | in the address (a share link) | [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) | +| `GET /api/public/status/`, `/api/public/wall/` | The data behind a shared status page and wall display | in the address (a share link) | Settings → Share links | +| `GET /health` | Is the app up? | none | answers `{"ok":true}` | + +## Reading the API safely + +- Treat a token like a password: it can read everything its owner can. Give each script its own (so one can be removed alone), choose the shortest life that suits, and keep it in a secret store, not in a repository. +- Use HTTPS. The token is sent in a header with every request. +- A removed token stops working at once; if one leaks, remove it under **API tokens** and make another. +- Every token's last use is shown on the API tokens page, which makes an unused one easy to spot and remove. diff --git a/docs/HEARTBEATS-AND-CALENDAR.md b/docs/HEARTBEATS-AND-CALENDAR.md new file mode 100644 index 0000000..35fd317 --- /dev/null +++ b/docs/HEARTBEATS-AND-CALENDAR.md @@ -0,0 +1,70 @@ +# Heartbeats and the calendar feed + +Two addresses that are not part of the [read API](API.md) and need no sign-in, because each carries its own token: + +- **Heartbeats** — a cron job or script calls an address when it finishes, and Homelab Manager speaks up when it does not. This is how to monitor things the app cannot see by itself: a backup script, + a nightly sync, a certificate renewal. +- A **calendar feed** — what is coming up (expiries, warranties, maintenance), for a phone or desktop calendar to subscribe to. + +How a missed heartbeat becomes an alert, a notification and an incident is in [MONITORING.md](MONITORING.md). + +## Heartbeats — for cron jobs and scripts + +### Setting one up + +Administrators: **Monitoring → Heartbeats → Add a heartbeat**. + +| Field | Notes | +|---|---| +| Name | what the job is, e.g. "Nightly backup" (up to 100 characters) | +| **Period** | how often the job should report, from 1 minute to 30 days | +| **Grace** | how much later than that is still fine, 0 minutes to a day (so a job that sometimes runs long is not a false alarm) | +| Server | optional: the server the job runs on. **Maintenance on that server silences the heartbeat's alerts**, and the alert links to it. | +| Notes | up to 500 characters | + +Up to **100** heartbeats. Everyone signed in sees how each job is doing; only administrators see its **address** (with the token) — it is what lets anyone say "I'm alive" for the job. **New address** gives +a heartbeat a new token and the old address stops working at once. + +### Calling it + +The job calls its address when it finishes: + +``` +GET /api/heartbeat/ it worked +GET /api/heartbeat//start it began (optional: lets the next call say how long it took) +GET /api/heartbeat//fail it ran and went wrong: an alert at once +``` + +```sh +# in cron, after the job +0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx + +# in a script, with a start, a failure and a note +curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start +./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \ + || curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail +``` + +- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work. + Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once). +- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters. +- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address. +- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes; + a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`. +- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them. + +## Calendar feed + +An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL"). + +``` +GET /api/public/calendar/.ics +``` + +- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time. + A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address. +- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and + recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out. +- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app. + An event's `UID` is a hash of what it is and its date, the same on every fetch. +- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does. diff --git a/docs/MONITORING.md b/docs/MONITORING.md new file mode 100644 index 0000000..9d33268 --- /dev/null +++ b/docs/MONITORING.md @@ -0,0 +1,304 @@ +# How monitoring works + +This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the +"why did it do that?" page. Setting things up is in the other pages ([SSO](SSO.md), [API](API.md), [Prometheus and Grafana](PROMETHEUS-GRAFANA.md), [webhooks](WEBHOOKS.md), +[heartbeats](HEARTBEATS-AND-CALENDAR.md)); the list of every notification is [NOTIFICATIONS.md](../NOTIFICATIONS.md). + +- [The big picture](#the-big-picture) +- [What is watched, and how often](#what-is-watched-and-how-often) +- [Servers and the agent](#servers-and-the-agent) +- [From a reading to an alert](#from-a-reading-to-an-alert) +- [The built-in checks](#the-built-in-checks) +- [Your own alert rules](#your-own-alert-rules) +- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats) +- [Integrations](#integrations) +- [Forecasts](#forecasts) +- [Dependencies: one outage, one alert](#dependencies-one-outage-one-alert) +- [What happens to an alert](#what-happens-to-an-alert) — notify, acknowledge, silence, snooze, flapping, escalate +- [Incidents](#incidents) +- [History and how long it is kept](#history-and-how-long-it-is-kept) +- [Where to see it](#where-to-see-it) +- [Tuning, in questions and answers](#tuning-in-questions-and-answers) +- [What monitoring does not do](#what-monitoring-does-not-do) + +## The big picture + +``` + agents ─────────┐ + integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · silence + endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push + heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines) + your own rules ─┘ ├──► incidents (the record) and history + └──► /metrics and the read API +``` + +Three ideas run through everything: + +1. **A check turns readings into *conditions*.** A condition is "this specific problem exists right now" and has a stable **key** (`offline:server:3`, `disk:server:3:/`) — the same problem always has the same key. +2. **Alerts are announced on a change, not on a schedule.** A problem is announced **once** when it appears and **once** when it ends. It is not repeated every time it is looked at. +3. **Silence is never "all clear".** When something cannot be *read* (an integration is down, a server's agent is quiet), what depended on it is **held exactly as it was** — neither raised nor cleared — + and the Alerts page lists it under *Couldn't check everything*. The app never turns "I couldn't look" into "nothing is wrong". + +## What is watched, and how often + +| What | How it gets to the app | How often | +|---|---|---| +| **Servers** (Linux, Windows) | The **agent** on each server **pushes** a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects *to* a server. | Every **15 minutes** by default (set when the agent is installed) | +| **Proxmox**, **Synology** | The health pass **reads** them: node and storage usage, volume and disk status. | Every **15 minutes** | +| **UniFi** | A sync **reads** devices, networks, Wi-Fi names and clients and keeps them. | Every **5 minutes** | +| **Home Assistant** | A sync **reads** the entities you chose to watch and keeps their state and a history. | Every **5 minutes** | +| **Nginx Proxy Manager** | **Read** when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute | +| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** | +| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due | +| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** | +| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications | +| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time | +| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass | +| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** | + +After the app **restarts**, server-based checks are paused for **20 minutes** so agents get a chance to report before any server is judged. + +## Servers and the agent + +The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app +replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md). + +**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it +has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts. + +While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all. + +**Per-server settings.** A server's *Settings* section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for +one disk or "never alert about this disk", and services never to count as failed (names or patterns such as `gupdate*`). Whatever it does not replace follows the global setting. + +**What the report is used for:** the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support +check, pending updates, the dependency map, and `/metrics`. + +## From a reading to an alert + +There are two paths, designed so they cannot disagree: + +**1. The state-based checks (the notifications).** Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the **list of conditions that exist right now**. The app compares that list with +the one it remembered from last time: + +- **new** → announced (a notification, an event webhook, a row in the incident record); +- **gone** → announced as resolved; +- **still there** → nothing; +- **its source could not be read this time** → carried over untouched (not resolved, not repeated). + +The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all. + +**2. The Alerts page.** It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), **live**, and turns what they find into one flat list. It ignores the per-event on/off toggles — +the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a +20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under *Couldn't check everything*. + +Each alert has: + +| | | +|---|---| +| **Severity** | `critical`, `warning` or `info`. Things that are *gone* (offline, a failing SMART disk, a disk over 95%) are critical; things that are *coming* (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. | +| **Category** | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring`, `tickets` — what the filter on the page and the [routing rules](#who-hears-about-it) work on. | +| **Source** | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … | +| **Target** | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. | +| **Key / id** | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. | + +## The built-in checks + +Every threshold here is a default; where it can be changed, the place is named. + +| Check | Raised when | Default | Changed in | +|---|---|---|---| +| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server | +| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk | +| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server | +| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) | +| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable | +| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable | +| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable | +| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card | on | Settings → Notifications | +| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications | +| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications | +| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page | +| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server | +| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications | +| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications | +| **Docker updates** | A container image has a newer version | | Settings → Notifications | +| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable | +| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications | +| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check | +| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat | +| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page | +| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules | + +## Your own alert rules + +Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers: + +- **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature. +- **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the + history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time. +- **How serious:** warning or critical. **For which servers:** all of them, one, or those with a tag. +- **Try it now** shows which servers it would be raised on, from current data, before you save it. + +A rule's finding is an **ordinary alert** with the key `rule::server:[:]`, in the *Monitoring* category: announced and resolved like the others, held back by maintenance on the +server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is +offline or has not reported for 45 minutes is not judged. + +## Endpoint checks and heartbeats + +**Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds: + +| Kind | What it asks | +|---|---| +| `http` | Fetches an `http(s)://` address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. `200-299, 301`), and — if you ask — the page contains a keyword. Follows up to 5 redirects. | +| `tcp` | Opens a connection to a host and port. | +| `tls` | Connects and reads the certificate; the answer is the days left. | + +Per check: **interval** (1–1440 minutes, default 5), **timeout** (default 10 s), **failures before an alert** (default 2 in a row — one blip is not an alert), **certificate warning days** (default 14), an +optional **server it is linked to** (maintenance on that server silences it, and the alert links to it), and *accept a self-signed certificate*. Each answer — time, result, message — is kept, with hourly and +daily summaries for the charts and uptime figures. + +- A check **fails an alert** (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A **certificate** within its warning days is its own alert (`endpoint-cert:…`). +- What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine. +- The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under *Endpoints → Suggested*, and proxied hosts Nginx Proxy Manager serves. + +**Heartbeats** (Monitoring → Heartbeats) are the other direction: a cron job or script **calls an address when it finishes**, and the app alerts when it does *not*. A heartbeat has a **period** and a **grace**; it is +**late** when both have passed since the last call (a *warning*), and a call to `/fail` is **critical** at once. Details and examples: [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md). + +Both are judged **every minute**, with the same remember-and-compare bookkeeping as the other checks. + +## Integrations + +Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app, +using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**. + +- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a + restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the + integration's page. +- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check. + The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (**20 minutes**). +- **An integration can explain other alerts.** A service that stops answering because the integration it needs is failing is folded under it ([Dependencies](#dependencies-one-outage-one-alert)). + +## Forecasts + +A forecast says "this will be full in N days" from a **straight line through the recent history**, and is deliberately cautious: + +- It needs **at least six readings covering at least three days**, and a **steady** climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that + wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat. +- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too. +- A disk that is **already over its limit** is left to the ordinary "nearly full" alert. +- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as + `homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears. +- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is + shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history. + +## Dependencies: one outage, one alert + +If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a **graph of what needs what**: + +- a server **runs on** a host (set by hand, or worked out from its Proxmox node); +- a UniFi device is **connected through** another, and a **server is plugged into** a switch or access point (matched by IP address); +- **dependencies you write** (*Map → Dependencies*: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out. + +When something is down and what depends on it is down too, the dependants' alerts are **folded under** the first: shown nested, marked *caused by …*, not announced on their own, and the host's message says +"— also takes down web and db". If the host comes back and a dependant **stays** down, it is announced then, as new: **an outage is never swallowed.** + +## What happens to an alert + +Several things can quiet an alert. They are different, and it matters which you use: + +| | Promise about the thing? | Hides it? | Notifications | Ends | +|---|---|---|---|---| +| **Acknowledge** ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in `/metrics`; leaves the *needs attention* count | Stops [escalation](#escalation) | When you choose (an hour … 30 days), or when the alert has been gone half an hour | +| **Snooze** ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; **one reminder** at the end if it is still there | At the time you set (up to 90 days), or earlier with *Wake now* | +| **Maintenance window** | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked *silenced* | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as **new** | +| **Folded under a host** | — automatic | Nested under its cause | Not announced on its own | When the cause ends | +| **Flapping** | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm | + +### Who hears about it + +Notifications go to the channels that are on (**Gotify, ntfy, email, a webhook, push to phones and browsers**), narrowed by **routing rules** (first match wins: by kind, by severity, by the server's tags) and each +browser's own wishes. **Quiet hours** hold messages and send one digest at the end of the window. Every message the app can send, and what toggles it, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md). +**Event webhooks** are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included ([WEBHOOKS.md](WEBHOOKS.md)). + +### Flapping + +An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After +an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back. + +### Escalation + +An alert raised for a while with **nobody on it** can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra +email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes. + +### Maintenance + +*Maintenance* (a window of 5 minutes to a week, started by hand, in bulk, from a **recurring schedule**, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and +its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under +maintenance with it. A silenced problem is **not remembered as known**, so one still there when the window ends **alerts then, as new**. + +## Incidents + +Every time an alert was raised and later went away is an **incident**: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its +timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts **every five minutes** whether or not anyone has +the page open; the same alert coming back is a **new** incident. Resolved incidents are kept **400 days**. They are on **Operations → Alerts → History**, in the read API (`/api/v1/incidents`) and in the +monthly report. + +## History and how long it is kept + +| What | Kept | Grain | +|---|---|---| +| A server's load, memory and each disk | **35 days** of samples (one every 10–15 minutes), **120 days** of hourly summaries, **~800 days** of daily summaries | summaries carry the lowest and highest | +| Endpoint check answers | **14 days** of every answer, hourly summaries for **120 days**, daily summaries for years | | +| Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day | +| Proxmox storage and Synology volume use | **400 days**, one reading an hour | | +| Home Assistant watched numbers | **35 days**, one reading every few minutes | | +| Incidents | **400 days** after they resolve | | +| The audit log and the Diagnostic Log | Until automatic purging is switched on under *Settings → Data* (then 90 days by default, configurable) | | + +A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there. + +## Where to see it + +| | Shows | +|---|---| +| **Dashboard**, **My work**, the menu badge | What needs attention now; what is yours | +| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back | +| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it | +| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **Proxy hosts** | Each source's own view | +| **Weekly digest** and **monthly report** | The week and the month in one message | +| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) | +| **Read API** (`/api/v1`) | [API.md](API.md) | +| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) | +| **Status page**, **wall display**, **calendar feed** | Read-only views for others; made under *Settings → Share links* | + +## Tuning, in questions and answers + +**I am about to reboot a server.** Start maintenance on it (Maintenance page, or *Silence* on an alert): its alerts are held and, if it is still down at the end, announced as new. + +**An alert is noisy because it keeps coming and going.** Let it flap (it will be damped after three raises an hour), then look at *Alerts → Noise* — it says what usually helps: a longer grace, a different limit. +For your own rules, raise the limit or lengthen the time. + +**One server legitimately runs at 92% disk.** Give that server (or that one disk) its own limit — or *never alert about this disk* — in the server's *Settings* section. + +**I want to be reminded about something later, not silence it.** Snooze it. + +**I do not want to be woken for warnings.** Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours. + +**A job on a server has to run every night.** Make a heartbeat for it and call it from the end of the script. + +**I want my own limit, such as "memory above 85% for an hour".** Add an alert rule. + +**An integration's alerts disappeared.** Check *Couldn't check everything* on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why. + +**The app restarted and nothing about servers is alerting.** For 20 minutes after a restart servers are not judged, so the agents can report first. + +**I want these figures in my own monitoring.** `/metrics` for Prometheus, the read API for scripts, event webhooks to be told. + +## What monitoring does not do + +- **It does not fix anything.** The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.) +- **It does not see inside a server that has no agent** — only what an endpoint check, a heartbeat or an integration can say about it. +- **It is not a replacement for a second pair of eyes on the app itself.** If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on `up{job="homelab-manager"}` + ([PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md#alerting-rules-to-start-from)), or a check from another tool against `GET /health`. +- **It is as fresh as its sources.** A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read. diff --git a/docs/PROMETHEUS-GRAFANA.md b/docs/PROMETHEUS-GRAFANA.md new file mode 100644 index 0000000..6136910 --- /dev/null +++ b/docs/PROMETHEUS-GRAFANA.md @@ -0,0 +1,337 @@ +# Prometheus and Grafana + +Homelab Manager can be scraped by **Prometheus** at `/metrics`, and ships a ready-made **Grafana dashboard** built on those series. This page sets up both, from nothing, and lists every series. + +- [How it fits together](#how-it-fits-together) +- [1. Make a metrics token](#1-make-a-metrics-token) +- [2. Check it by hand](#2-check-it-by-hand) +- [3. Set up Prometheus](#3-set-up-prometheus) +- [4. Set up Grafana](#4-set-up-grafana) +- [A complete example with Docker Compose](#a-complete-example-with-docker-compose) +- [Alerting rules to start from](#alerting-rules-to-start-from) +- [Every series](#every-series) +- [Useful queries](#useful-queries) +- [Troubleshooting](#troubleshooting) + +## How it fits together + +``` + agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana + (reads its own database; (stores the history, + calls nothing outside) draws the dashboard) +``` + +- `/metrics` is a **view of what the app already knows**. A scrape reads the database and the app's last alert result (reused for up to a minute); it **never calls out** — not to Proxmox, not to a server, not + to an integration. Scraping every 15 seconds costs a few database reads. +- The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is **not needed** to see trends in the app itself. Use Prometheus and Grafana when you + want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager. +- **Every series is a gauge.** Labels are escaped as Prometheus requires. + +## 1. Make a metrics token + +Administrators: **Settings → API and webhooks → Metrics for Prometheus → Make a token**. + +- The token (it starts with `hlm_m_`) is **shown once**; only its SHA-256 is kept. +- Up to **ten** can exist — make one per Prometheus, so one can be removed without touching the others. A removed token **stops working at once**. +- Each shows when it was last used, so an unused one is easy to find. +- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted** (addresses end up in logs). +- No token, a wrong one and a removed one all get the same `401`. Requests are limited to **120 a minute** per address (`429` beyond). +- The answer is Prometheus's text format: `Content-Type: text/plain; version=0.0.4; charset=utf-8`. + +A metrics token can read `/metrics` and nothing else. It is a different thing from the *personal API tokens* used for the [read API](API.md), and neither can sign in. + +## 2. Check it by hand + +```bash +curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30 +``` + +You should see something like this (trimmed; the `# HELP` lines are left out here): + +``` +# TYPE homelab_info gauge +homelab_info{environment=""} 1 +# TYPE homelab_server_up gauge +homelab_server_up{server="wiki-host"} 1 +# TYPE homelab_server_last_seen_timestamp_seconds gauge +homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670 +# TYPE homelab_storage_days_to_full gauge +homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21 +# TYPE homelab_alerts gauge +homelab_alerts{severity="critical"} 1 +homelab_alerts{severity="warning"} 3 +homelab_alerts{severity="info"} 0 +``` + +Without the header you get `401` — that is the right answer: + +```bash +curl -i https://hm.example.lan/metrics # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics" +``` + +## 3. Set up Prometheus + +Put the token in a file of its own, readable only by Prometheus, with **no trailing newline**: + +```bash +printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token +chmod 600 /etc/prometheus/homelab-manager.token +``` + +`prometheus.yml`: + +```yaml +global: + scrape_interval: 30s + +scrape_configs: + - job_name: homelab-manager + scheme: https + metrics_path: /metrics + authorization: + type: Bearer + credentials_file: /etc/prometheus/homelab-manager.token + static_configs: + - targets: ["hm.example.lan"] # host[:port] of APP_BASE_URL, without https:// + # If the certificate comes from your own CA, point at it (rather than skipping verification): + # tls_config: + # ca_file: /etc/prometheus/my-ca.pem +``` + +- **30–60 seconds is plenty.** The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes). +- Prometheus accepts the token inline too (`credentials: hlm_m_…`), but a file keeps it out of the configuration you share. +- Behind a reverse proxy, use the address the proxy serves (`scheme: https`, port 443). Reaching the container directly is `scheme: http`, target `host:3000` — fine on a trusted network. +- Reload Prometheus (`kill -HUP`, or `POST /-/reload` if enabled) and open **Status → Targets**: `homelab-manager` should be **UP**. +- **Several installations** (production and a dev copy): the series `homelab_info{environment="…"}` carries the name set by `ENVIRONMENT_NAME`, so you can tell them apart; add an `instance`/`env` label in the scrape + config as well. + +## 4. Set up Grafana + +1. **Add Prometheus as a data source** — *Connections → Data sources → Add → Prometheus*, URL e.g. `http://prometheus:9090`, *Save & test*. +2. **Download the dashboard** — in Homelab Manager: **Settings → API and webhooks → Download the dashboard** (administrators). It is a file, `homelab-manager-dashboard.json`. (The same file is served to a signed-in + administrator at `GET /api/metrics-tokens/grafana-dashboard.json`.) +3. **Import it** — in Grafana: **Dashboards → New → Import**, upload the file, and **choose your Prometheus** when asked. The file never names a data source, so it is not tied to any installation. + +The dashboard has a **Server** picker (filled from `homelab_server_up`) and five rows: + +| Row | What it has | +|---|---| +| **Needs attention** | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. | +| **Servers** | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. | +| **Disks** | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. | +| **Backups** | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. | +| **Secrets, power and the app itself** | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. | + +The dashboard is **built by the app from the same code that serves `/metrics`**, and a test checks that it only asks for series that are really served. So **after an upgrade that adds series, download it again** +to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), **but it has not been opened in a Grafana +by the author** — if a panel looks wrong, the series table below says what each query should be. + +## A complete example with Docker Compose + +Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder: + +``` +monitoring/ +├─ docker-compose.yml +├─ prometheus.yml (as above, with credentials_file: /etc/prometheus/homelab-manager.token) +├─ homelab-manager.token (the token only, no newline) +└─ rules/ + └─ homelab-manager.yml (the alerting rules below, optional) +``` + +```yaml +# docker-compose.yml +services: + prometheus: + image: prom/prometheus:latest + restart: unless-stopped + command: + - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.retention.time=400d # a year and a bit: the dashboard's long ranges are then real + - --web.enable-lifecycle + volumes: + - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro + - ./rules:/etc/prometheus/rules:ro + - prometheus-data:/prometheus + ports: ["9090:9090"] + + grafana: + image: grafana/grafana:latest + restart: unless-stopped + volumes: + - grafana-data:/var/lib/grafana + ports: ["3001:3000"] + depends_on: [prometheus] + +volumes: + prometheus-data: + grafana-data: +``` + +With the rules, add to `prometheus.yml`: + +```yaml +rule_files: + - /etc/prometheus/rules/*.yml +``` + +In Grafana the data source URL is then `http://prometheus:9090` (the service name, since they share a compose network). + +## Alerting rules to start from + +Homelab Manager already sends its own notifications (see [MONITORING.md](MONITORING.md)). **Use these in addition only if you want the same things in Alertmanager**, otherwise you will be told twice. +They are the ones worth having *outside* the app — above all, the app itself going quiet. + +```yaml +# rules/homelab-manager.yml +groups: + - name: homelab-manager + rules: + # The one thing the app cannot tell you about itself. + - alert: HomelabManagerDown + expr: up{job="homelab-manager"} == 0 + for: 5m + labels: { severity: critical } + annotations: { summary: "Homelab Manager cannot be scraped" } + + - alert: HomelabCriticalAlerts + expr: homelab_alerts{severity="critical"} > 0 + for: 5m + labels: { severity: critical } + annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" } + + - alert: HomelabServerDown + expr: homelab_server_up == 0 + for: 5m + labels: { severity: critical } + annotations: { summary: "{{ $labels.server }} has stopped reporting" } + + - alert: HomelabDiskFilling + expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7 + for: 1h + labels: { severity: warning } + annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" } + + - alert: HomelabDiskUsageHigh + expr: homelab_server_disk_used_percent > 90 + for: 15m + labels: { severity: warning } + annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" } + + - alert: HomelabSelfBackupStale + expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400 + labels: { severity: warning } + annotations: { summary: "Homelab Manager's own backup is more than two days old" } + + - alert: HomelabEndpointDown + expr: homelab_endpoint_up == 0 + for: 5m + labels: { severity: warning } + annotations: { summary: "Endpoint check {{ $labels.name }} is failing" } + + - alert: HomelabCertificateExpiring + expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14 + labels: { severity: warning } + annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" } + + - alert: HomelabHeartbeatMissed + expr: homelab_heartbeat_up == 0 + labels: { severity: warning } + annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" } + + - alert: HomelabIntegrationFailing + expr: homelab_integration_up == 0 + for: 10m + labels: { severity: warning } + annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" } +``` + +A series that is **left out** is not zero: `homelab_integration_up`, `homelab_endpoint_up` and `homelab_selfbackup_last_attempt_ok` do not exist until something has been tried. That is why the rules above +compare against a condition rather than use `absent()`. + +## Every series + +All gauges. The same table is checked by a test against what `/metrics` really serves. + +| Series | Labels | Meaning | +|---|---|---| +| `homelab_info` | `environment` | Always 1; says which installation this is. | +| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. | +| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. | +| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. | +| `homelab_server_memory_used_percent` | `server` | Memory in use. | +| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. | +| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). | +| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. | +| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. | +| `homelab_server_reboot_required` | `server` | 1 if a restart is due. | +| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. | +| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. | +| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. | +| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. | +| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. | +| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state. | +| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). | +| `homelab_storage_days_to_full` | `pool` | Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb. | +| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). | +| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts`. | +| `homelab_alerts_silenced` | | Alerts held back by a maintenance window or a snooze. | +| `homelab_alerts_grouped` | | Alerts folded under a server that is down. | +| `homelab_incidents_open` | | Incidents that have begun and not yet ended. | +| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. | +| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. | +| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). | +| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. | +| `homelab_maintenance_active_windows` | | Maintenance windows running now. | +| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). | +| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. | +| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. | +| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. | +| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. | +| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. | +| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). | +| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. | +| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). | +| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). | +| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. | +| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. | +| `homelab_scrape_duration_seconds` | | How long building the answer took. | +| `homelab_scrape_timestamp_seconds` | | When the answer was built. | + +## Useful queries + +```promql +# Servers that have gone quiet +homelab_server_up == 0 + +# The five fullest disks +topk(5, homelab_server_disk_used_percent) + +# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full) +sort(homelab_server_disk_days_to_full < 30) + +# Endpoints slower than half a second +homelab_endpoint_response_milliseconds > 500 + +# How many guests are not well backed up +sum(homelab_backup_guests{status=~"stale|failed|none"}) + +# Updates waiting across all servers, and how many are security fixes +sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates) + +# Is the dev copy up as well as production? (several installations: filter on the label you added) +homelab_info +``` + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| Target is **DOWN**, `401 Unauthorized` | The token is wrong, was removed, or has a trailing newline in its file (use `printf '%s'`). The same `401` is given for no token and a bad one. | +| Target is **DOWN**, `429 Too many requests` | More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval. | +| Target is **DOWN**, certificate error | Prometheus does not trust the certificate. Give it the CA with `tls_config.ca_file` (or `insecure_skip_verify: true` on a trusted LAN, knowing what that gives up). | +| Target **UP** but some series are missing | A series that has nothing to say is **left out** rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected. | +| A dashboard panel is empty | The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade. | +| Numbers differ from the Alerts page | `homelab_alerts` counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute. | +| Names with odd characters look wrong | Label values are escaped (`\"`, `\\`, `\n`) as the Prometheus format requires; Prometheus and Grafana unescape them. | diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..550b738 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,41 @@ +# Documentation + +Everything about running and integrating with Homelab Manager is in Markdown files in this repository. This folder holds the **guides**; the **references** (who can do what, every notification, every table, +the credentials each integration needs) are next to the main [README](../README.md). + +## Guides — this folder + +| Guide | Read it when you want to … | +|---|---| +| **[SSO.md](SSO.md)** | Set up sign-in with Authentik: what is needed (HTTPS, reverse proxy, a signing key), the provider and `.env` settings, how roles can follow groups, and what to check when sign-in fails. | +| **[API.md](API.md)** | Call Homelab Manager from a script, Home Assistant or another tool: personal tokens, every `/api/v1` address with its answer, examples in curl, Python, Node and Home Assistant, and the other machine-facing addresses (agent reports, heartbeats, share links). | +| **[PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md)** | Scrape `/metrics` with Prometheus, import the Grafana dashboard, copy a working Docker Compose and alerting rules, and look up every series. | +| **[WEBHOOKS.md](WEBHOOKS.md)** | Be told when something happens: signed event webhooks (alert opened/resolved, maintenance started/ended) with receivers in Node and Python, and the chat-style notification webhook. | +| **[HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md)** | Monitor a cron job (it calls an address when it finishes) or subscribe a calendar to what is coming up. | +| **[MONITORING.md](MONITORING.md)** | Understand how monitoring works: what is watched and how often, how a reading becomes an alert, what acknowledge, snooze, maintenance and flapping each do, incidents, history, forecasts, and how to tune it. | + +## References — next to the README + +| File | What it is | +|---|---| +| [README.md](../README.md) | What the app is, every feature, installation (Docker, arm64, development). | +| [ROLES.md](../ROLES.md) | Viewer, operator, administrator: who sees which page and who can change what. | +| [NOTIFICATIONS.md](../NOTIFICATIONS.md) | Every notification the app sends, what triggers it, what toggles it, routing, quiet hours, escalation. | +| [INTEGRATIONS.md](../INTEGRATIONS.md) | What each integration and DNS provider needs (credentials, permissions) and what it does. | +| [DATABASE.md](../DATABASE.md) | The database tables, columns and relationships. | +| [BACKUP.md](../BACKUP.md) | What to back up, how (also while it runs), how to restore, and how to check a backup. | +| [AGENT_UPDATE.md](../AGENT_UPDATE.md) | Keeping the agents on the servers up to date. | + +## Where to start + +- **A new installation:** [README](../README.md) (install) → [SSO.md](SSO.md) (sign-in) → add servers and integrations in the app → [MONITORING.md](MONITORING.md) (what to expect) → [BACKUP.md](../BACKUP.md). +- **Getting figures into Grafana:** [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md). +- **Getting alerts into something else:** [WEBHOOKS.md](WEBHOOKS.md) for pushes, [API.md](API.md) for asking. +- **"Why did / didn't it alert?":** [MONITORING.md](MONITORING.md), then [NOTIFICATIONS.md](../NOTIFICATIONS.md). + +## Conventions in these pages + +- Examples use **`https://hm.example.lan`** for your own address (`APP_BASE_URL`), and tokens like `hlm_p_xxxxxxxx…` that you will replace. +- A **menu path** such as *Settings → API and webhooks* is in the web interface; who may open it is in [ROLES.md](../ROLES.md). +- Sample answers have the shapes the app really sends (several were captured from a test installation and trimmed); the values are only illustrative. +- Things that **have not been checked against the real thing** (a third-party system, a Grafana version, another identity provider) are said so where they come up. diff --git a/docs/SSO.md b/docs/SSO.md new file mode 100644 index 0000000..007337a --- /dev/null +++ b/docs/SSO.md @@ -0,0 +1,229 @@ +# Single sign-on (SSO) + +Homelab Manager has no passwords of its own. People sign in through an identity provider with **OpenID Connect (OIDC)**, and the app only ever sees who they are, +never a password. It is written for **[Authentik](https://goauthentik.io/)** and the settings are named after it; any provider that speaks standard OIDC *can* work +(see [Other identity providers](#other-identity-providers)), but only Authentik has been used. + +This page covers what the setup needs, how to configure Authentik and the app, how roles are decided, and what to check when sign-in fails. + +- [How sign-in works](#how-sign-in-works) +- [What you need before you start](#what-you-need-before-you-start) +- [1. Set up Authentik](#1-set-up-authentik) +- [2. Set up the app](#2-set-up-the-app) +- [3. The first sign-in](#3-the-first-sign-in) +- [Roles](#roles) — by hand, or from groups +- [Sessions and signing out](#sessions-and-signing-out) +- [What does *not* use SSO](#what-does-not-use-sso) +- [Troubleshooting](#troubleshooting) +- [If you are locked out](#if-you-are-locked-out) +- [Security notes](#security-notes) + +## How sign-in works + +1. A person opens the app and is sent to `/auth/login`. The app starts an *authorization code* flow with **PKCE** (a one-time code verifier and challenge) and a random `state`, + and redirects the browser to Authentik. +2. The person signs in at Authentik (with whatever Authentik asks for: password, passkey, two-factor …). +3. Authentik sends the browser back to **`/auth/callback`** with a one-time code. +4. The app, from its own server, swaps the code for tokens at Authentik, checks the **ID token's signature** against the keys Authentik publishes (not only its contents), and + asks Authentik's *userinfo* address for the person's name, email, picture and groups. +5. The app finds the person by their **`sub`** (Authentik's stable id for them) or makes an account, decides their role ([below](#roles)), writes the sign-in to the audit log + and starts a session with a **new session id**. + +The *client* is **confidential**: the app holds a client secret, and the code exchange happens server to server. The scopes asked for are `openid email profile`. + +## What you need before you start + +| You need | Why | +|---|---| +| **An Authentik instance** | The identity provider. | +| **The app reachable by browsers at a fixed address — `APP_BASE_URL`** — and **over HTTPS** in production | The redirect address Authentik sends people back to is built from it. The session cookie is marked `Secure` when the app runs in production (`NODE_ENV=production`, which the supplied Docker setup does), and browsers only keep a `Secure` cookie from an HTTPS page — so behind plain `http://` on a LAN address, sign-in appears to work and then sends you straight back to the login page. (`http://localhost` is the exception browsers allow.) Put the app behind a reverse proxy with a certificate. | +| **A reverse proxy that passes `X-Forwarded-For` and `X-Forwarded-Proto`** (one proxy in front) | The app trusts one proxy hop, so the sign-in log and the rate limit see each person's address instead of the proxy's. | +| **The app able to reach Authentik, server to server** | It calls Authentik's discovery, token, userinfo and key (JWKS) addresses itself, from wherever it runs — from inside the Docker network that may not be the name your browser uses. | +| **Authentik's certificate trusted by the app** | If Authentik uses a certificate from your own CA, give Node the CA with `NODE_EXTRA_CA_CERTS=/path/to/ca.pem`. Do **not** switch certificate checking off with `NODE_TLS_REJECT_UNAUTHORIZED=0`. | +| **Reasonably correct clocks** on both machines | Tokens have times in them; a minute of drift is tolerated, more is refused. | +| **Two secrets generated by you** | `SESSION_SECRET` (signs the session cookie) and `CREDENTIALS_ENCRYPTION_KEY` (encrypts stored integration credentials). Not part of SSO itself, but the app will not run in production without the first. | + +Generate both with: + +```bash +node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" +``` + +## 1. Set up Authentik + +The names of the menus differ a little between Authentik versions; what has to be true does not. + +**Create the provider** — *Applications → Providers → Create → OAuth2/OpenID Provider*: + +| Setting | Value | +|---|---| +| Name | anything, e.g. `Homelab Manager` | +| Authorization flow | your usual one (explicit or implicit consent, as you prefer) | +| **Client type** | **Confidential** | +| Client ID / Client secret | generated for you — you will copy both into the app's `.env` | +| **Redirect URIs** | `https://hm.example.lan/auth/callback` — your `APP_BASE_URL` followed by `/auth/callback`, as an **exact (strict) match** | +| **Signing key** | **choose a certificate (an RSA key), such as *authentik Self-signed Certificate*.** With no signing key Authentik signs ID tokens with the client secret (HS256) and publishes no keys, and the app refuses such a token. | +| Scopes | `openid`, `email`, `profile` (the defaults) | +| Subject mode | leave the default (*Based on the hashed User ID*) — see the warning below | +| Issuer mode | leave the default (*Each provider has a different issuer*) | + +> **Do not change the subject mode later.** The app recognises a person by their `sub`. Changing how Authentik builds it makes every person a *new* person: a new account +> (a viewer, with none of the old account's role) and the old one left behind. + +**Create the application** — *Applications → Applications → Create*: give it a name and a **slug** (e.g. `homelab-manager`), choose the provider above, and decide **who may use it** +(a policy or group binding on the application). Authentik decides who can *sign in*; the app's roles decide what they can *do* once in. + +The **issuer URL** you need next is on the provider's page — normally `https://authentik.example.lan/application/o//`, **with the trailing slash**. + +## 2. Set up the app + +Put these in the `.env` file next to `docker-compose.yml` (a template is in `.env.example`): + +```ini +# Where people reach the app. HTTPS, no trailing slash needed. Used to build the redirect address. +APP_BASE_URL=https://hm.example.lan + +# From the Authentik provider. +AUTHENTIK_ISSUER_URL=https://authentik.example.lan/application/o/homelab-manager/ +AUTHENTIK_CLIENT_ID=… +AUTHENTIK_CLIENT_SECRET=… + +# Generated by you (see above). Changing SESSION_SECRET signs everyone out. +SESSION_SECRET=… +CREDENTIALS_ENCRYPTION_KEY=… +``` + +| Variable | Meaning | +|---|---| +| `APP_BASE_URL` | The address people type. The redirect address is `APP_BASE_URL` + `/auth/callback`; it must match what the Authentik provider allows, character for character (scheme, host, port). Also the address used in links in notifications and in the read API. | +| `AUTHENTIK_ISSUER_URL` | The provider's issuer URL. The app *discovers* the rest from `/.well-known/openid-configuration`, so a wrong or unreachable issuer is the usual reason sign-in fails. | +| `AUTHENTIK_CLIENT_ID`, `AUTHENTIK_CLIENT_SECRET` | From the provider. The secret is read from the environment only; it is not stored in the database. | +| `SESSION_SECRET` | Signs the session cookie. In production the app **refuses to start** without it, and warns if it is still the placeholder from `.env.example` or shorter than 32 characters. | +| `SESSION_DIR` | Where sessions are kept on disk (default `data/sessions`, inside the data volume). | +| `NODE_ENV` | `production` in the Docker image: makes the cookie `Secure` and requires `SESSION_SECRET`. | + +If any of the three `AUTHENTIK_*` values is missing the app still starts — so the rest of it can be looked at — but `/auth/login` fails until they are set. Restart the container after +changing `.env`. + +## 3. The first sign-in + +**The first person to sign in becomes the administrator**, so sign in yourself first on a new installation. Everyone after that starts as a **viewer** until an administrator +changes their role under **Administration → Users** — or until their groups decide it ([below](#roles)). + +Only people Authentik lets into the application can sign in at all, which is why the application's access policy matters: the first account is whoever you let in first. + +## Roles + +There are three: **viewer** (sees everything, changes nothing), **operator** (acknowledges and silences alerts, starts maintenance, runs updates, writes notes and jobs …) and +**admin** (everything, including settings, integrations and users). [ROLES.md](../ROLES.md) lists exactly who can see and do what. + +### By hand (the default) + +Administrators change roles on **Administration → Users**. The only administrator cannot be demoted — there is always at least one. + +### From groups + +Instead of setting roles by hand, let the groups Authentik reports decide them. In Authentik, the user's groups come in a **`groups`** claim that is part of the **`profile`** scope the app +already asks for, so there is nothing to add on the Authentik side except the groups themselves (e.g. `homelab-admins`, `homelab-ops`) and who is in them. + +In the app: **Settings → Sign-in → Roles from groups**. + +| Setting | Meaning | +|---|---| +| Set roles from groups at each sign-in | The switch. Off, roles are by hand. | +| Claim that holds the groups | `groups` for Authentik. | +| Admin groups | Names (one per line or comma separated; case does not matter). In any of them → **admin**. | +| Operator groups | In any of them (and in no admin group) → **operator**. | +| Someone in none of those groups | **becomes a viewer**, or **keeps the role they already have**. | + +The rules are applied **at every sign-in**, so a change in Authentik shows up the next time the person signs in (or when their session next starts), not instantly. + +Safeguards: + +- If a sign-in carries **no groups claim at all** (a wrong claim name, a scope that is missing), roles are left exactly as they are — "we weren't told" is never treated as "in no groups". +- **The only administrator is never demoted** by a group rule. +- **The first person to sign in is still the administrator**, whatever the groups say. +- While roles follow groups, **roles cannot be set by hand** (a role set on the Users page would be undone at the next sign-in); the Users page says so. +- Every role change made from groups is written to the audit log (`user` / `role_from_groups`, and `role_from_groups_blocked` when the last administrator was protected). + +Before switching it on, press **Check against the groups people had last time**: it changes nothing and shows what each person *would* become, from the groups they had at their last sign-in. +The groups each person had are also shown on the Users page, so you can see the names before relying on them. + +### Profile pictures + +Optional, off by default (**Settings → Sign-in → Profile pictures**): the picture Authentik gives, and/or a Gravatar. Either one means *the server* fetches the picture from an outside +address; people's browsers never do, and the app serves the pictures from itself. + +## Sessions and signing out + +- A session lasts up to **7 days**. The cookie is `HttpOnly`, `SameSite=Lax` and, in production, `Secure`. +- **Disabling someone in Authentik does not end an app session that is already running.** To cut someone off at once, end their session under **Users → Signed in now** + (administrators); it ends immediately, and they cannot start another once Authentik refuses them. Accounts themselves are not deleted in the app — a person who no longer signs in + simply has no session. +- **Signing out** of the app also ends the person's Authentik session when Authentik supports *RP-initiated logout* (it does), so the next visit asks for a password again. + Ending a session from *Signed in now* ends only the app's own session, not the one at Authentik. +- Sign-ins and sign-outs are in the audit log, with the address they came from. + +## What does *not* use SSO + +Some addresses are meant for machines and use their own tokens instead of a session. None of them can be used to sign in, and each stops working the moment its token is removed. + +| Address | Used by | Token | +|---|---|---| +| `/api/agent/report` | the agent on each server | one per server, shown once when the server is added | +| `/metrics` | Prometheus | a *metrics token* (Settings → API and webhooks) — see [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) | +| `/api/v1/…` | scripts, Home Assistant | a *personal API token* (user menu → API tokens) — see [API.md](API.md) | +| `/api/heartbeat/` | cron jobs | the heartbeat's own address — see [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md) | +| `/api/public/…` | status pages, the wall display, the calendar feed | a *share link* (Settings → Share links) | +| `/health` | container health checks | none (answers `{"ok":true}`) | + +## Troubleshooting + +A failed sign-in shows a plain page ("Sign-in failed") and nothing more — deliberately, since the reason would tell an attacker what to adjust. **The reason is in the server log and in +the audit log** (*Change log → Raw audit log*, category `session`, action `login_failed`, with the address and a short reason). + +| Reason in the log | What it usually means | +|---|---| +| `provider_unreachable` | The app could not reach Authentik to read its configuration: a wrong `AUTHENTIK_ISSUER_URL` (missing trailing slash, wrong slug, `http` instead of `https`), a name the *container* cannot resolve, a firewall, or a certificate the app does not trust (see `NODE_EXTRA_CA_CERTS`). It retries on the next sign-in; no restart is needed. | +| `provider_rejected` | Authentik answered but the exchange failed. Most often: **the redirect URI** in the provider is not exactly `APP_BASE_URL/auth/callback`; the **client secret** is wrong; the provider has **no signing key** (HS256 tokens are refused); a clock that is several minutes off; or the browser took too long and the one-time code ran out. | +| `no_identity` | Authentik signed the person in but sent no `sub`. Check the provider's scopes include `openid`. | +| *(not recorded)* `no_pending_login` | A page left open too long, a bookmarked callback address, or the session cookie was not kept. If it happens every time, the cookie is the problem — see the next row. | + +Other symptoms: + +| Symptom | Likely cause | +|---|---| +| You sign in at Authentik and land on the login page again, over and over | The session cookie is not being kept. In production it is `Secure`, so the site must be opened over **HTTPS**; behind a reverse proxy, check it sends `X-Forwarded-Proto`. Also check `APP_BASE_URL` is the address in the browser's address bar. | +| Everyone is signed out after a restart | `SESSION_SECRET` changed (or is generated fresh each start), or the `data/sessions` folder is not on the persistent volume. | +| New people are all viewers although roles from groups is on | The sign-in carried no groups: check the claim name, and that the Authentik user is in the groups. Look at the Users page — the groups seen at each person's last sign-in are shown there. | +| Roles from groups did nothing to an administrator | Working as intended if they are the only administrator (a group rule never removes the last one); the audit log says `role_from_groups_blocked`. | +| "Cannot remove the only admin" | Same safeguard, on the Users page. Make someone else an administrator first. | +| A person with a different login at Authentik (renamed, merged) shows up as a new person | The app matches on `sub`, not on name or email. See the subject-mode warning above. | + +## If you are locked out + +There is no local password, so if the identity provider is down or misconfigured nobody can sign in. People who are **already signed in** keep working until their session ends. +To get back in, fix the provider or the `AUTHENTIK_*` settings and restart the container. + +If the only administrator's account was changed so that nobody is an administrator any more, promote someone directly in the database (stop the app, or accept that it is a +single quick write; **take a copy of the data folder first** — see [BACKUP.md](../BACKUP.md)): + +```bash +sqlite3 data/homelab-manager.sqlite "UPDATE users SET role = 'admin' WHERE email = 'you@example.lan';" +``` + +## Security notes + +- **PKCE and `state`** protect the code exchange; the callback is refused without the pending sign-in that the same browser started. +- **The ID token's signature is verified** against Authentik's published keys, so the login does not depend on the connection to Authentik alone. +- **A new session id is issued at sign-in**, so an id planted in the browser beforehand cannot be used. +- **Requests that change something are refused when they come from another origin** (the cookie is `SameSite=Lax`, which does not stop another app on the same parent domain), including `GET /auth/logout`. +- **Sign-in is rate-limited** to 20 attempts a minute per address, and each failure is audited. +- Every response carries `frame-ancestors 'none'`, `X-Frame-Options: DENY`, `nosniff` and a same-origin referrer policy. + +## Other identity providers + +The code uses standard OIDC discovery (`/.well-known/openid-configuration`), the authorization code flow with PKCE (`S256`), a confidential client, **RS256-signed ID tokens +with a published JWKS**, and the `sub`, `email`, `name`/`preferred_username`, `picture` and a groups claim (name configurable) — nothing Authentik-specific. Keycloak, Authelia and +similar providers should therefore work by putting *their* issuer URL, client id and secret in the `AUTHENTIK_*` variables and setting the groups claim name under **Settings → Sign-in**. +This has **not been tried**; if you try one, make sure it signs with RS256 and that the redirect address is registered exactly. diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md new file mode 100644 index 0000000..2c12b98 --- /dev/null +++ b/docs/WEBHOOKS.md @@ -0,0 +1,284 @@ +# Webhooks + +Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook +here, and they are easy to mix up: + +| | **Event webhooks** | **The notification webhook** | +|---|---|---| +| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint | +| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** | +| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) | +| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` | +| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) | +| Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) | + +- [Event webhooks](#event-webhooks) + - [Setting one up](#setting-one-up) + - [Events](#events) + - [The message](#the-message) + - [Checking the signature](#checking-the-signature) + - [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to) + - [Alert keys](#alert-keys) + - [A small receiver](#a-small-receiver) + - [Home Assistant and n8n](#home-assistant-and-n8n) +- [The notification webhook](#the-notification-webhook) +- [Troubleshooting](#troubleshooting) + +## Event webhooks + +### Setting one up + +Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**. + +| Field | Notes | +|---|---| +| Name | up to 60 characters; what the list shows | +| Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. | +| Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. | +| Events | at least one of the four below | + +- **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one. +- **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong). +- The list shows when the last message was sent and its result — a status code, `timed out`, or the error. +- Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own). + +### Events + +| Event | Sent when | +|---|---| +| `alert.opened` | A problem begins. | +| `alert.resolved` | A problem ends. | +| `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. | +| `maintenance.ended` | A maintenance window finishes, on its own or ended early. | + +**What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new +ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem +under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem +being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read +those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md). + +### The message + +``` +POST +Content-Type: application/json +User-Agent: HomelabManager-Webhook +X-HLM-Event: alert.opened +X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message) +X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret) +``` + +```json +{ + "event": "alert.opened", + "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c", + "createdAt": "2026-10-17T21:04:11.482Z", + "source": "homelab-manager", + "data": { + "alert": { + "key": "offline:server:3", + "source": "server", + "severity": "critical", + "message": "pve1 hasn't reported for 1 h — also takes down web and db" + } + } +} +``` + +| `data` | Fields | +|---|---| +| `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` | +| `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) | +| `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) | +| `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward | +| `test` | `message` | + +A captured test message, as a receiver sees it (headers lower-cased by the receiving server): + +```json +{ + "headers": { + "x-hlm-event": "test", + "x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61", + "x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57", + "user-agent": "HomelabManager-Webhook", + "content-type": "application/json" + }, + "body": { + "event": "test", + "id": "72118185-66dd-49d8-8886-aed8bdc40d61", + "createdAt": "2026-10-20T21:46:21.349Z", + "source": "homelab-manager", + "data": { "message": "This is a test from Homelab Manager." } + } +} +``` + +### Checking the signature + +With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and +re-serialised JSON), compare in **constant time**, and reject a message that does not match. + +```js +import { createHmac, timingSafeEqual } from "node:crypto"; + +export function isFromHomelabManager(secret, rawBody, signatureHeader) { + const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`); + const given = Buffer.from(signatureHeader ?? ""); + return given.length === expected.length && timingSafeEqual(given, expected); +} +``` + +```python +import hmac, hashlib + +def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool: + expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest() + return hmac.compare_digest(expected, signature_header or "") +``` + +```bash +# Quick check at a shell, with the body saved to a file: +printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')" +``` + +The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you +have handled and refuse repeats, and reject a `createdAt` that is far from your clock. + +### Delivery: what to rely on and what not to + +- Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the + result is written on the webhook and forgotten. +- A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address. +- **There is no queue and no replay.** A receiver that is down for longer than those two tries **misses the event**. Messages for different events can arrive in any order. If you need to be certain, treat the + webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now. +- Because a retry reuses the delivery id, **handle each id once** (idempotently). +- Answer quickly (`200` is enough) and do slow work afterwards. + +### Alert keys + +`alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes: + +| Key | About | +|---|---| +| `offline:server:` | A server has stopped reporting | +| `offline:device:` | A UniFi device has been offline too long | +| `disk:server::` | A disk is nearly full | +| `disk:proxmox::…`, `disk:synology::` | A Proxmox storage or Synology volume is nearly full | +| `synology-volume::`, `synology-disk::` | A Synology volume or disk is not healthy | +| `failed-units:server:` | Services have failed on a server | +| `smart:server::`, `zfs:server::` | A disk says it is failing, a pool is not online | +| `new-port:server:::` | A new TCP port started listening | +| `rule::server:[:]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) | +| `endpoint:`, `endpoint-cert:` | An endpoint check is failing, or its certificate is running out | +| `heartbeat:`, `heartbeat-fail:` | A job reported late, or said it failed | +| `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed | + +Treat unknown shapes as opaque; match on the whole key. + +### A small receiver + +Node with Express — the one thing to get right is to verify the signature against the **raw** body: + +```js +import express from "express"; +import { createHmac, timingSafeEqual } from "node:crypto"; + +const SECRET = process.env.HLM_WEBHOOK_SECRET; +const seen = new Set(); // use something persistent for real + +const app = express(); +app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => { + const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex")); + const given = Buffer.from(req.get("x-hlm-signature") ?? ""); + if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401); + + const msg = JSON.parse(req.body.toString("utf8")); + if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled + seen.add(msg.id); + + if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`); + else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`); + res.sendStatus(200); +}); +app.listen(8080); +``` + +Python with Flask: + +```python +import hmac, hashlib, os +from flask import Flask, request, abort + +SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode() +app = Flask(__name__) + +@app.post("/hlm") +def hook(): + raw = request.get_data() # the exact bytes + expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest() + if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")): + abort(401) + msg = request.get_json() + if msg["event"] == "alert.opened": + print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"]) + return "", 200 +``` + +### Home Assistant and n8n + +- **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/`), which is the access control, so a + secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward. + + ```yaml + automation: + - alias: "Homelab Manager critical alert" + trigger: + - platform: webhook + webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts + allowed_methods: [POST] + local_only: true + condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}" + action: + - service: notify.mobile_app_phone + data: + title: "Homelab alert" + message: "{{ trigger.json.data.alert.message }}" + ``` + +- **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC + in a Code node as above. + +## The notification webhook + +The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post. + +**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret. + +``` +POST +Content-Type: application/json +X-Webhook-Secret: (only if one is set) + +{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" } +``` + +- The body is exactly `{ "content": "\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between). +- `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook. +- It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It + is delivered once with no retry; a failure is written to the server log. +- The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only. +- Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md). + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). | +| **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. | +| **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). | +| **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). | +| The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. | +| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. | +| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. | +| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. | diff --git a/server/src/services/publicApi.ts b/server/src/services/publicApi.ts index 50401d1..d613cd4 100644 --- a/server/src/services/publicApi.ts +++ b/server/src/services/publicApi.ts @@ -98,6 +98,8 @@ export async function alertsV1() { url: absoluteUrl(a.link), silenced: a.silenced, flapping: !!a.flapping, + // Snoozed alerts are also `silenced`; this says until when they are. + snoozedUntil: a.snoozed?.until ?? null, target: a.target, causedBy: a.causedBy ?? null, affects: a.affects ?? [], diff --git a/web/src/pages/ApiTokens.tsx b/web/src/pages/ApiTokens.tsx index 6b9dd86..f7d7260 100644 --- a/web/src/pages/ApiTokens.tsx +++ b/web/src/pages/ApiTokens.tsx @@ -213,7 +213,7 @@ export default function ApiTokens() { </li> ))} </ul> - <div className="text-secondary small">The full list, with an example of each answer, is in API.md.</div> + <div className="text-secondary small">The full list, with an example of each answer, is in docs/API.md.</div> </div> </div> </>