- A switch (off by default) under Settings -> Notifications -> Who hears what - ntfy gets buttons (three for one alert, one link for several); phone push gets two buttons handled by the service worker; Gotify, email and the chat webhook get the link as text; escalation messages carry them too - Each link is a signed token for the alerts in that one message only, good for 24 hours, for three actions; opening it changes nothing, only a POST acts; everything is audited as "from a notification link" - Docs updated Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
313 lines
18 KiB
Markdown
313 lines
18 KiB
Markdown
# Webhooks
|
|
|
|
Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook
|
|
here, and they are easy to mix up:
|
|
|
|
| | **Event webhooks** | **The notification webhook** |
|
|
|---|---|---|
|
|
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
|
|
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
|
|
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
|
|
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
|
|
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
|
|
| Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) |
|
|
|
|
- [Event webhooks](#event-webhooks)
|
|
- [Setting one up](#setting-one-up)
|
|
- [Events](#events)
|
|
- [The message](#the-message)
|
|
- [Checking the signature](#checking-the-signature)
|
|
- [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to)
|
|
- [Alert keys](#alert-keys)
|
|
- [A small receiver](#a-small-receiver)
|
|
- [Home Assistant and n8n](#home-assistant-and-n8n)
|
|
- [The notification webhook](#the-notification-webhook)
|
|
- [Buttons on notifications](#buttons-on-notifications)
|
|
- [Troubleshooting](#troubleshooting)
|
|
|
|
## Event webhooks
|
|
|
|
### Setting one up
|
|
|
|
Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**.
|
|
|
|
| Field | Notes |
|
|
|---|---|
|
|
| Name | up to 60 characters; what the list shows |
|
|
| Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. |
|
|
| Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. |
|
|
| Events | at least one of the four below |
|
|
|
|
- **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one.
|
|
- **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong).
|
|
- The list shows when the last message was sent and its result — a status code, `timed out`, or the error.
|
|
- Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own).
|
|
|
|
### Events
|
|
|
|
| Event | Sent when |
|
|
|---|---|
|
|
| `alert.opened` | A problem begins. |
|
|
| `alert.resolved` | A problem ends. |
|
|
| `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. |
|
|
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
|
|
|
|
**What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new
|
|
ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem
|
|
under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem
|
|
being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read
|
|
those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md).
|
|
|
|
### The message
|
|
|
|
```
|
|
POST <your address>
|
|
Content-Type: application/json
|
|
User-Agent: HomelabManager-Webhook
|
|
X-HLM-Event: alert.opened
|
|
X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message)
|
|
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
|
|
```
|
|
|
|
```json
|
|
{
|
|
"event": "alert.opened",
|
|
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
|
|
"createdAt": "2026-10-17T21:04:11.482Z",
|
|
"source": "homelab-manager",
|
|
"data": {
|
|
"alert": {
|
|
"key": "offline:server:3",
|
|
"source": "server",
|
|
"severity": "critical",
|
|
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
| `data` | Fields |
|
|
|---|---|
|
|
| `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` |
|
|
| `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) |
|
|
| `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) |
|
|
| `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward |
|
|
| `test` | `message` |
|
|
|
|
A captured test message, as a receiver sees it (headers lower-cased by the receiving server):
|
|
|
|
```json
|
|
{
|
|
"headers": {
|
|
"x-hlm-event": "test",
|
|
"x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61",
|
|
"x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57",
|
|
"user-agent": "HomelabManager-Webhook",
|
|
"content-type": "application/json"
|
|
},
|
|
"body": {
|
|
"event": "test",
|
|
"id": "72118185-66dd-49d8-8886-aed8bdc40d61",
|
|
"createdAt": "2026-10-20T21:46:21.349Z",
|
|
"source": "homelab-manager",
|
|
"data": { "message": "This is a test from Homelab Manager." }
|
|
}
|
|
}
|
|
```
|
|
|
|
### Checking the signature
|
|
|
|
With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and
|
|
re-serialised JSON), compare in **constant time**, and reject a message that does not match.
|
|
|
|
```js
|
|
import { createHmac, timingSafeEqual } from "node:crypto";
|
|
|
|
export function isFromHomelabManager(secret, rawBody, signatureHeader) {
|
|
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
|
|
const given = Buffer.from(signatureHeader ?? "");
|
|
return given.length === expected.length && timingSafeEqual(given, expected);
|
|
}
|
|
```
|
|
|
|
```python
|
|
import hmac, hashlib
|
|
|
|
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
|
|
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
|
|
return hmac.compare_digest(expected, signature_header or "")
|
|
```
|
|
|
|
```bash
|
|
# Quick check at a shell, with the body saved to a file:
|
|
printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')"
|
|
```
|
|
|
|
The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you
|
|
have handled and refuse repeats, and reject a `createdAt` that is far from your clock.
|
|
|
|
### Delivery: what to rely on and what not to
|
|
|
|
- Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the
|
|
result is written on the webhook and forgotten.
|
|
- A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address.
|
|
- **There is no queue and no replay.** A receiver that is down for longer than those two tries **misses the event**. Messages for different events can arrive in any order. If you need to be certain, treat the
|
|
webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now.
|
|
- Because a retry reuses the delivery id, **handle each id once** (idempotently).
|
|
- Answer quickly (`200` is enough) and do slow work afterwards.
|
|
|
|
### Alert keys
|
|
|
|
`alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes:
|
|
|
|
| Key | About |
|
|
|---|---|
|
|
| `offline:server:<id>` | A server has stopped reporting |
|
|
| `offline:device:<id>` | A UniFi device has been offline too long |
|
|
| `disk:server:<id>:<mount>` | A disk is nearly full |
|
|
| `disk:proxmox:<integration>:…`, `disk:synology:<integration>:<volume>` | A Proxmox storage or Synology volume is nearly full |
|
|
| `synology-volume:<integration>:<volume>`, `synology-disk:<integration>:<disk>` | A Synology volume or disk is not healthy |
|
|
| `failed-units:server:<id>` | Services have failed on a server |
|
|
| `smart:server:<id>:<device>`, `zfs:server:<id>:<pool>` | A disk says it is failing, a pool is not online |
|
|
| `new-port:server:<id>:<proto>:<port>` | A new TCP port started listening |
|
|
| `rule:<rule id>:server:<id>[:<disk>]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) |
|
|
| `endpoint:<check id>`, `endpoint-cert:<check id>` | An endpoint check is failing, or its certificate is running out |
|
|
| `heartbeat:<id>`, `heartbeat-fail:<id>` | A job reported late, or said it failed |
|
|
| `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed |
|
|
|
|
Treat unknown shapes as opaque; match on the whole key.
|
|
|
|
### A small receiver
|
|
|
|
Node with Express — the one thing to get right is to verify the signature against the **raw** body:
|
|
|
|
```js
|
|
import express from "express";
|
|
import { createHmac, timingSafeEqual } from "node:crypto";
|
|
|
|
const SECRET = process.env.HLM_WEBHOOK_SECRET;
|
|
const seen = new Set(); // use something persistent for real
|
|
|
|
const app = express();
|
|
app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => {
|
|
const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex"));
|
|
const given = Buffer.from(req.get("x-hlm-signature") ?? "");
|
|
if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401);
|
|
|
|
const msg = JSON.parse(req.body.toString("utf8"));
|
|
if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled
|
|
seen.add(msg.id);
|
|
|
|
if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`);
|
|
else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`);
|
|
res.sendStatus(200);
|
|
});
|
|
app.listen(8080);
|
|
```
|
|
|
|
Python with Flask:
|
|
|
|
```python
|
|
import hmac, hashlib, os
|
|
from flask import Flask, request, abort
|
|
|
|
SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode()
|
|
app = Flask(__name__)
|
|
|
|
@app.post("/hlm")
|
|
def hook():
|
|
raw = request.get_data() # the exact bytes
|
|
expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
|
|
if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")):
|
|
abort(401)
|
|
msg = request.get_json()
|
|
if msg["event"] == "alert.opened":
|
|
print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"])
|
|
return "", 200
|
|
```
|
|
|
|
### Home Assistant and n8n
|
|
|
|
- **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/<id>`), which is the access control, so a
|
|
secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward.
|
|
|
|
```yaml
|
|
automation:
|
|
- alias: "Homelab Manager critical alert"
|
|
trigger:
|
|
- platform: webhook
|
|
webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts
|
|
allowed_methods: [POST]
|
|
local_only: true
|
|
condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}"
|
|
action:
|
|
- service: notify.mobile_app_phone
|
|
data:
|
|
title: "Homelab alert"
|
|
message: "{{ trigger.json.data.alert.message }}"
|
|
```
|
|
|
|
- **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC
|
|
in a Code node as above.
|
|
|
|
## The notification webhook
|
|
|
|
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
|
|
|
|
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
|
|
|
|
```
|
|
POST <your address>
|
|
Content-Type: application/json
|
|
X-Webhook-Secret: <the secret> (only if one is set)
|
|
|
|
{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" }
|
|
```
|
|
|
|
- The body is exactly `{ "content": "<title in bold>\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between).
|
|
- `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook.
|
|
- It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It
|
|
is delivered once with no retry; a failure is written to the server log.
|
|
- The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only.
|
|
- Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
|
|
|
|
## Buttons on notifications
|
|
|
|
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
|
|
it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
|
|
|
|
The addresses behind them, for anything that wants to build on it:
|
|
|
|
```
|
|
GET /api/alert-actions/<token> a page listing the alerts with a button for each action — opening it changes nothing
|
|
POST /api/alert-actions/<token>/ack acknowledge ("Acknowledged from a notification", until the alert is gone)
|
|
POST /api/alert-actions/<token>/silence start an hour of maintenance on what the alert is about (only if it is about something a window can silence)
|
|
POST /api/alert-actions/<token>/snooze snooze until tomorrow 09:00 in the notification time zone
|
|
add ?key=<alert key> (or a form field "key") when the token is for more than one alert
|
|
```
|
|
|
|
```json
|
|
{ "ok": true, "message": "Acknowledged." }
|
|
```
|
|
|
|
- **The token is the access.** It is signed (HMAC-SHA256, with a key derived from `SESSION_SECRET`), is for the **alerts named in it only** (the alert keys of that one message, up to 20), and **ends after 24 hours**. Nothing
|
|
is stored. A forged, altered or ended link is `404`/`410`. A `key` that is not in the token is `400`; an alert that has cleared is `404` ("it has cleared"); a silence for something with nothing to silence it with is `409`.
|
|
- **It can do these three things and nothing else** — no reading, no settings, no other alert. Everything it does is in the audit log, said to be done *from a notification link*, with the IP address it came from.
|
|
- **A POST from a browser form gets a page back; anything else gets JSON.** `GET` never acts, so a link preview, a chat unfurl or a mail scanner cannot press a button. Limited to 60 requests a minute from one address.
|
|
- **Anyone who can read the notification can use its buttons** for the 24 hours. That is the trade for not having to sign in: keep the ntfy topic private (a self-hosted ntfy, or a topic with a password), do not point this at a public
|
|
topic, and remember the link is in an email's text. Changing `SESSION_SECRET` or switching the setting off ends every link at once.
|
|
- The links are built from `APP_BASE_URL`, so it has to be an address your phone can open.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Cause |
|
|
|---|---|
|
|
| **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). |
|
|
| **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. |
|
|
| **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). |
|
|
| **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). |
|
|
| The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. |
|
|
| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. |
|
|
| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. |
|
|
| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |
|