# Webhooks Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook here, and they are easy to mix up: | | **Event webhooks** | **The notification webhook** | |---|---|---| | For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint | | Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** | | Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) | | Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` | | Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) | | Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) | - [Event webhooks](#event-webhooks) - [Setting one up](#setting-one-up) - [Events](#events) - [The message](#the-message) - [Checking the signature](#checking-the-signature) - [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to) - [Alert keys](#alert-keys) - [A small receiver](#a-small-receiver) - [Home Assistant and n8n](#home-assistant-and-n8n) - [The notification webhook](#the-notification-webhook) - [Buttons on notifications](#buttons-on-notifications) - [Troubleshooting](#troubleshooting) ## Event webhooks ### Setting one up Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**. | Field | Notes | |---|---| | Name | up to 60 characters; what the list shows | | Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. | | Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. | | Events | at least one of the four below | - **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one. - **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong). - The list shows when the last message was sent and its result — a status code, `timed out`, or the error. - Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own). ### Events | Event | Sent when | |---|---| | `alert.opened` | A problem begins. | | `alert.resolved` | A problem ends. | | `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. | | `maintenance.ended` | A maintenance window finishes, on its own or ended early. | | `game_server.started` | A game server (Infrastructure → Game servers) is seen turned on — starting or running after being off. | | `game_server.stopped` | A game server is seen turned off — stopping or offline after being on. | **Game servers.** The accounts are read every five minutes, so `at` is to the nearest five minutes and a server that is switched off and on again between two reads is not seen at all. *Starting* counts as on and *stopping* as off, so a start is announced once, when it begins, not again when it has finished. A server read for the first time, one being installed or moved, and one whose state is unknown are not announced. Unlike the chat notification, these are **not** held back by the *Game servers turned on or off* switch in Settings → Notifications: a machine hears every change. A receiver that itself sent the power signal can tell its own doing by `signal`/`by`. **What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md). ### The message ``` POST Content-Type: application/json User-Agent: HomelabManager-Webhook X-HLM-Event: alert.opened X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message) X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret) ``` ```json { "event": "alert.opened", "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c", "createdAt": "2026-10-17T21:04:11.482Z", "source": "homelab-manager", "data": { "alert": { "key": "offline:server:3", "source": "server", "severity": "critical", "message": "pve1 hasn't reported for 1 h — also takes down web and db" } } } ``` | `data` | Fields | |---|---| | `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` | | `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) | | `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) | | `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward | | `game_server.started` / `game_server.stopped` | `gameServer`: `id` (the panel's id), `name`, `game`, `integrationId`, `integrationName`, `on` (`true` for started), `at` (when the read saw it), `signal` (`start`, `stop`, `restart` or `kill` when the change followed a power signal sent from this app, else `null`) and `by` (who sent that signal, else `null`) | | `test` | `message` | A captured test message, as a receiver sees it (headers lower-cased by the receiving server): ```json { "headers": { "x-hlm-event": "test", "x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61", "x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57", "user-agent": "HomelabManager-Webhook", "content-type": "application/json" }, "body": { "event": "test", "id": "72118185-66dd-49d8-8886-aed8bdc40d61", "createdAt": "2026-10-20T21:46:21.349Z", "source": "homelab-manager", "data": { "message": "This is a test from Homelab Manager." } } } ``` ### Checking the signature With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and re-serialised JSON), compare in **constant time**, and reject a message that does not match. ```js import { createHmac, timingSafeEqual } from "node:crypto"; export function isFromHomelabManager(secret, rawBody, signatureHeader) { const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`); const given = Buffer.from(signatureHeader ?? ""); return given.length === expected.length && timingSafeEqual(given, expected); } ``` ```python import hmac, hashlib def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool: expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, signature_header or "") ``` ```bash # Quick check at a shell, with the body saved to a file: printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')" ``` The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you have handled and refuse repeats, and reject a `createdAt` that is far from your clock. ### Delivery: what to rely on and what not to - Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the result is written on the webhook and in its [delivery log](#the-delivery-log-and-sending-again). - A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address. - **There is no queue and no automatic replay.** A receiver that is down for longer than those two tries **misses the event** — until someone sends it again from the delivery log. Messages for different events can arrive in any order. If you need to be certain, treat the webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now. - Because a retry reuses the delivery id, **handle each id once** (idempotently). - Answer quickly (`200` is enough) and do slow work afterwards. ### The delivery log and sending again Every message sent to a webhook is written down: when, which event, whether it got through, what came back (`200`, or what went wrong, such as `connect ECONNREFUSED` or `500`), whether it needed its second try, and the message itself. **Settings → API and webhooks → Event webhooks → Sent** opens the log for one webhook, newest first, and **Show the message** unfolds the JSON that was sent. **Send again** sends a message from the log **exactly as it was** — the same bytes, the same `id` and `X-HLM-Delivery`, a signature made over the same body — so a receiver that already handled it can recognise it by its id and ignore it, and one that missed it handles it now. The new attempt is logged as a delivery of its own, marked *sent again by hand*, and updates the webhook's *Last* result. Like the first send it is tried twice. It is for when the receiver was down or the fault was at its end; it does not repair a wrong address. - **The last 100 per webhook, and none older than 14 days,** are kept. Removing a webhook removes its log. - A message larger than **16 000 characters** is not kept in full, so it is not shown and cannot be sent again (`409 too_large`). The messages the app sends are far smaller. - **Administrators only** — the log holds what was sent, which names servers and says what is wrong with them. Each resend is in the audit log (*webhook › retry*) and in the **Change log**. ``` GET /api/event-webhooks//deliveries?limit=50 the log (limit 1–100) POST /api/event-webhooks//deliveries//retry send one again → { "ok": true, "result": "200" } ``` These are the app's own addresses (they need an administrator's sign-in, not an API token), listed for completeness. ### Alert keys `alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes: | Key | About | |---|---| | `offline:server:` | A server has stopped reporting | | `offline:device:` | A UniFi device has been offline too long | | `disk:server::` | A disk is nearly full | | `disk:proxmox::…`, `disk:synology::` | A Proxmox storage or Synology volume is nearly full | | `synology-volume::`, `synology-disk::` | A Synology volume or disk is not healthy | | `failed-units:server:` | Services have failed on a server | | `smart:server::`, `zfs:server::` | A disk says it is failing, a pool is not online | | `new-port:server:::` | A new TCP port started listening | | `rule::server:[:]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) | | `endpoint:`, `endpoint-cert:` | An endpoint check is failing, or its certificate is running out | | `heartbeat:`, `heartbeat-fail:` | A job reported late, or said it failed | | `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed | Treat unknown shapes as opaque; match on the whole key. ### A small receiver Node with Express — the one thing to get right is to verify the signature against the **raw** body: ```js import express from "express"; import { createHmac, timingSafeEqual } from "node:crypto"; const SECRET = process.env.HLM_WEBHOOK_SECRET; const seen = new Set(); // use something persistent for real const app = express(); app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => { const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex")); const given = Buffer.from(req.get("x-hlm-signature") ?? ""); if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401); const msg = JSON.parse(req.body.toString("utf8")); if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled seen.add(msg.id); if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`); else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`); res.sendStatus(200); }); app.listen(8080); ``` Python with Flask: ```python import hmac, hashlib, os from flask import Flask, request, abort SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode() app = Flask(__name__) @app.post("/hlm") def hook(): raw = request.get_data() # the exact bytes expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest() if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")): abort(401) msg = request.get_json() if msg["event"] == "alert.opened": print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"]) return "", 200 ``` ### Home Assistant and n8n - **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/`), which is the access control, so a secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward. ```yaml automation: - alias: "Homelab Manager critical alert" trigger: - platform: webhook webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts allowed_methods: [POST] local_only: true condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}" action: - service: notify.mobile_app_phone data: title: "Homelab alert" message: "{{ trigger.json.data.alert.message }}" ``` - **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC in a Code node as above. ## The notification webhook A notification channel (beside Gotify, ntfy, Telegram, Pushover, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post. **Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret. ``` POST Content-Type: application/json X-Webhook-Secret: (only if one is set) { "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" } ``` - The body is exactly `{ "content": "\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between). - `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook. - It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It is delivered once with no retry; a failure is written to the server log. - The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only. - Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md). ## Buttons on notifications A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications). The addresses behind them, for anything that wants to build on it: ``` GET /api/alert-actions/<token> a page listing the alerts with a button for each action — opening it changes nothing POST /api/alert-actions/<token>/ack acknowledge ("Acknowledged from a notification", until the alert is gone) POST /api/alert-actions/<token>/silence start an hour of maintenance on what the alert is about (only if it is about something a window can silence) POST /api/alert-actions/<token>/snooze snooze until tomorrow 09:00 in the notification time zone add ?key=<alert key> (or a form field "key") when the token is for more than one alert ``` ```json { "ok": true, "message": "Acknowledged." } ``` - **The token is the access.** It is signed (HMAC-SHA256, with a key derived from `SESSION_SECRET`), is for the **alerts named in it only** (the alert keys of that one message, up to 20), and **ends after 24 hours**. Nothing is stored. A forged, altered or ended link is `404`/`410`. A `key` that is not in the token is `400`; an alert that has cleared is `404` ("it has cleared"); a silence for something with nothing to silence it with is `409`. - **It can do these three things and nothing else** — no reading, no settings, no other alert. Everything it does is in the audit log, said to be done *from a notification link*, with the IP address it came from. - **A POST from a browser form gets a page back; anything else gets JSON.** `GET` never acts, so a link preview, a chat unfurl or a mail scanner cannot press a button. Limited to 60 requests a minute from one address. - **Anyone who can read the notification can use its buttons** for the 24 hours. That is the trade for not having to sign in: keep the ntfy topic private (a self-hosted ntfy, or a topic with a password), do not point this at a public topic, and remember the link is in an email's text. Changing `SESSION_SECRET` or switching the setting off ends every link at once. - The links are built from `APP_BASE_URL`, so it has to be an address your phone can open. ## Troubleshooting | Symptom | Cause | |---|---| | **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). | | **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. | | **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). | | **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). | | The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. | | An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. | | A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. | | I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |