Files
Homelab-manager/docs/WEBHOOKS.md
T
bobbanandClaude Sonnet 5.5 aebf3c6ce1 Buttons on notifications: acknowledge, silence and snooze from the message
- A switch (off by default) under Settings -> Notifications -> Who hears what
- ntfy gets buttons (three for one alert, one link for several); phone push gets two buttons handled by the service worker; Gotify, email and the chat webhook get the link as text; escalation messages carry them too
- Each link is a signed token for the alerts in that one message only, good for 24 hours, for three actions; opening it changes nothing, only a POST acts; everything is audited as "from a notification link"
- Docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 20:42:04 +02:00

313 lines
18 KiB
Markdown

# Webhooks
Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook
here, and they are easy to mix up:
| | **Event webhooks** | **The notification webhook** |
|---|---|---|
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
| Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) |
- [Event webhooks](#event-webhooks)
- [Setting one up](#setting-one-up)
- [Events](#events)
- [The message](#the-message)
- [Checking the signature](#checking-the-signature)
- [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to)
- [Alert keys](#alert-keys)
- [A small receiver](#a-small-receiver)
- [Home Assistant and n8n](#home-assistant-and-n8n)
- [The notification webhook](#the-notification-webhook)
- [Buttons on notifications](#buttons-on-notifications)
- [Troubleshooting](#troubleshooting)
## Event webhooks
### Setting one up
Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**.
| Field | Notes |
|---|---|
| Name | up to 60 characters; what the list shows |
| Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. |
| Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. |
| Events | at least one of the four below |
- **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one.
- **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong).
- The list shows when the last message was sent and its result — a status code, `timed out`, or the error.
- Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own).
### Events
| Event | Sent when |
|---|---|
| `alert.opened` | A problem begins. |
| `alert.resolved` | A problem ends. |
| `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. |
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
**What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new
ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem
under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem
being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read
those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md).
### The message
```
POST <your address>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
```
```json
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
```
| `data` | Fields |
|---|---|
| `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` |
| `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) |
| `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) |
| `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward |
| `test` | `message` |
A captured test message, as a receiver sees it (headers lower-cased by the receiving server):
```json
{
"headers": {
"x-hlm-event": "test",
"x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61",
"x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57",
"user-agent": "HomelabManager-Webhook",
"content-type": "application/json"
},
"body": {
"event": "test",
"id": "72118185-66dd-49d8-8886-aed8bdc40d61",
"createdAt": "2026-10-20T21:46:21.349Z",
"source": "homelab-manager",
"data": { "message": "This is a test from Homelab Manager." }
}
}
```
### Checking the signature
With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and
re-serialised JSON), compare in **constant time**, and reject a message that does not match.
```js
import { createHmac, timingSafeEqual } from "node:crypto";
export function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
```
```python
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
```
```bash
# Quick check at a shell, with the body saved to a file:
printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')"
```
The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you
have handled and refuse repeats, and reject a `createdAt` that is far from your clock.
### Delivery: what to rely on and what not to
- Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the
result is written on the webhook and forgotten.
- A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address.
- **There is no queue and no replay.** A receiver that is down for longer than those two tries **misses the event**. Messages for different events can arrive in any order. If you need to be certain, treat the
webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now.
- Because a retry reuses the delivery id, **handle each id once** (idempotently).
- Answer quickly (`200` is enough) and do slow work afterwards.
### Alert keys
`alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes:
| Key | About |
|---|---|
| `offline:server:<id>` | A server has stopped reporting |
| `offline:device:<id>` | A UniFi device has been offline too long |
| `disk:server:<id>:<mount>` | A disk is nearly full |
| `disk:proxmox:<integration>:…`, `disk:synology:<integration>:<volume>` | A Proxmox storage or Synology volume is nearly full |
| `synology-volume:<integration>:<volume>`, `synology-disk:<integration>:<disk>` | A Synology volume or disk is not healthy |
| `failed-units:server:<id>` | Services have failed on a server |
| `smart:server:<id>:<device>`, `zfs:server:<id>:<pool>` | A disk says it is failing, a pool is not online |
| `new-port:server:<id>:<proto>:<port>` | A new TCP port started listening |
| `rule:<rule id>:server:<id>[:<disk>]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) |
| `endpoint:<check id>`, `endpoint-cert:<check id>` | An endpoint check is failing, or its certificate is running out |
| `heartbeat:<id>`, `heartbeat-fail:<id>` | A job reported late, or said it failed |
| `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed |
Treat unknown shapes as opaque; match on the whole key.
### A small receiver
Node with Express — the one thing to get right is to verify the signature against the **raw** body:
```js
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
const SECRET = process.env.HLM_WEBHOOK_SECRET;
const seen = new Set(); // use something persistent for real
const app = express();
app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => {
const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex"));
const given = Buffer.from(req.get("x-hlm-signature") ?? "");
if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401);
const msg = JSON.parse(req.body.toString("utf8"));
if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled
seen.add(msg.id);
if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`);
else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`);
res.sendStatus(200);
});
app.listen(8080);
```
Python with Flask:
```python
import hmac, hashlib, os
from flask import Flask, request, abort
SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode()
app = Flask(__name__)
@app.post("/hlm")
def hook():
raw = request.get_data() # the exact bytes
expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")):
abort(401)
msg = request.get_json()
if msg["event"] == "alert.opened":
print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"])
return "", 200
```
### Home Assistant and n8n
- **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/<id>`), which is the access control, so a
secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward.
```yaml
automation:
- alias: "Homelab Manager critical alert"
trigger:
- platform: webhook
webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts
allowed_methods: [POST]
local_only: true
condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}"
action:
- service: notify.mobile_app_phone
data:
title: "Homelab alert"
message: "{{ trigger.json.data.alert.message }}"
```
- **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC
in a Code node as above.
## The notification webhook
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
```
POST <your address>
Content-Type: application/json
X-Webhook-Secret: <the secret> (only if one is set)
{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" }
```
- The body is exactly `{ "content": "<title in bold>\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between).
- `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook.
- It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It
is delivered once with no retry; a failure is written to the server log.
- The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only.
- Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
## Buttons on notifications
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
The addresses behind them, for anything that wants to build on it:
```
GET /api/alert-actions/<token> a page listing the alerts with a button for each action — opening it changes nothing
POST /api/alert-actions/<token>/ack acknowledge ("Acknowledged from a notification", until the alert is gone)
POST /api/alert-actions/<token>/silence start an hour of maintenance on what the alert is about (only if it is about something a window can silence)
POST /api/alert-actions/<token>/snooze snooze until tomorrow 09:00 in the notification time zone
add ?key=<alert key> (or a form field "key") when the token is for more than one alert
```
```json
{ "ok": true, "message": "Acknowledged." }
```
- **The token is the access.** It is signed (HMAC-SHA256, with a key derived from `SESSION_SECRET`), is for the **alerts named in it only** (the alert keys of that one message, up to 20), and **ends after 24 hours**. Nothing
is stored. A forged, altered or ended link is `404`/`410`. A `key` that is not in the token is `400`; an alert that has cleared is `404` ("it has cleared"); a silence for something with nothing to silence it with is `409`.
- **It can do these three things and nothing else** — no reading, no settings, no other alert. Everything it does is in the audit log, said to be done *from a notification link*, with the IP address it came from.
- **A POST from a browser form gets a page back; anything else gets JSON.** `GET` never acts, so a link preview, a chat unfurl or a mail scanner cannot press a button. Limited to 60 requests a minute from one address.
- **Anyone who can read the notification can use its buttons** for the 24 hours. That is the trade for not having to sign in: keep the ntfy topic private (a self-hosted ntfy, or a topic with a password), do not point this at a public
topic, and remember the link is in an email's text. Changing `SESSION_SECRET` or switching the setting off ends every link at once.
- The links are built from `APP_BASE_URL`, so it has to be an address your phone can open.
## Troubleshooting
| Symptom | Cause |
|---|---|
| **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). |
| **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. |
| **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). |
| **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). |
| The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. |
| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. |
| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. |
| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |