Settings -> Data -> Storage measures the database, backups and sessions and every table (size, rows, growth, how long it is kept, how far back it reaches), and flags histories that are not being trimmed or have no limit. Event webhooks now keep a delivery log (last 100 per webhook, 14 days): what was sent, how it went, the message itself, and Send again, which resends it exactly as it was (same id and signature). Resends are audited and in the Change log. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
333 lines
20 KiB
Markdown
333 lines
20 KiB
Markdown
# Webhooks
|
||
|
||
Homelab Manager can **tell other systems when something happens** — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook
|
||
here, and they are easy to mix up:
|
||
|
||
| | **Event webhooks** | **The notification webhook** |
|
||
|---|---|---|
|
||
| For | **Machines**: a structured, signed JSON message per event | **People**: the same sentence a human would read, posted to a chat-style endpoint |
|
||
| Where | **Settings → API and webhooks → Event webhooks** | **Settings → Notifications → Channels → Webhook** |
|
||
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
|
||
| Body | `{ event, id, createdAt, source, data }` | `{ "content": "**Title**\nmessage" }` |
|
||
| Authenticity | `X-HLM-Signature` — an HMAC-SHA256 of the body | a shared secret in `X-Webhook-Secret` (not a signature) |
|
||
| Held back by routing, quiet hours, flapping, snooze | **No** (it is a separate way of being told) | **Yes** (it is a notification like the others) |
|
||
|
||
- [Event webhooks](#event-webhooks)
|
||
- [Setting one up](#setting-one-up)
|
||
- [Events](#events)
|
||
- [The message](#the-message)
|
||
- [Checking the signature](#checking-the-signature)
|
||
- [Delivery: what to rely on and what not to](#delivery-what-to-rely-on-and-what-not-to)
|
||
- [Alert keys](#alert-keys)
|
||
- [A small receiver](#a-small-receiver)
|
||
- [Home Assistant and n8n](#home-assistant-and-n8n)
|
||
- [The notification webhook](#the-notification-webhook)
|
||
- [Buttons on notifications](#buttons-on-notifications)
|
||
- [Troubleshooting](#troubleshooting)
|
||
|
||
## Event webhooks
|
||
|
||
### Setting one up
|
||
|
||
Administrators: **Settings → API and webhooks → Event webhooks → Add a webhook**.
|
||
|
||
| Field | Notes |
|
||
|---|---|
|
||
| Name | up to 60 characters; what the list shows |
|
||
| Address | `http://` or `https://`, **no user name or password in it**. The address is not restricted to public hosts — a receiver on your LAN is the normal case. |
|
||
| Secret | optional, up to 200 characters. With one, every message is **signed**. Stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set; **never shown again** after saving. |
|
||
| Events | at least one of the four below |
|
||
|
||
- **Up to 20 webhooks.** Each can be **switched off** and on, **removed**, and **tested**. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one.
|
||
- **Test** sends `{"event": "test", …}` right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (`200`, or what went wrong).
|
||
- The list shows when the last message was sent and its result — a status code, `timed out`, or the error.
|
||
- Making, switching and removing a webhook are in the audit log with its **name and events — never its address or secret** (the address can carry a token of its own).
|
||
|
||
### Events
|
||
|
||
| Event | Sent when |
|
||
|---|---|
|
||
| `alert.opened` | A problem begins. |
|
||
| `alert.resolved` | A problem ends. |
|
||
| `maintenance.started` | A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. |
|
||
| `maintenance.ended` | A maintenance window finishes, on its own or ended early. |
|
||
|
||
**What counts as an alert here.** `alert.opened` and `alert.resolved` are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new
|
||
ports, network devices offline, your own [alert rules](MONITORING.md#your-own-alert-rules), endpoint checks, heartbeats, and automation failures — on the **same terms as a notification**: a problem
|
||
under a maintenance window, or folded under a server that is down, is **not announced until it stands alone**. Two things that *do* hold back a notification do **not** hold back the webhook: a problem
|
||
being **flapping**, and one that is **snoozed** — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read
|
||
those from the [read API](API.md) or [`/metrics`](PROMETHEUS-GRAFANA.md).
|
||
|
||
### The message
|
||
|
||
```
|
||
POST <your address>
|
||
Content-Type: application/json
|
||
User-Agent: HomelabManager-Webhook
|
||
X-HLM-Event: alert.opened
|
||
X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message)
|
||
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
|
||
```
|
||
|
||
```json
|
||
{
|
||
"event": "alert.opened",
|
||
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
|
||
"createdAt": "2026-10-17T21:04:11.482Z",
|
||
"source": "homelab-manager",
|
||
"data": {
|
||
"alert": {
|
||
"key": "offline:server:3",
|
||
"source": "server",
|
||
"severity": "critical",
|
||
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
| `data` | Fields |
|
||
|---|---|
|
||
| `alert.opened` | `alert.key` (stable for the same problem — see [Alert keys](#alert-keys)), `alert.source` (where it came from, e.g. `server`, `proxmox:3:pve1`, `endpoint`, `heartbeat`), `alert.severity` (`critical` or `warning`), `alert.message` |
|
||
| `alert.resolved` | `alert.key`, `alert.message` (what it said when it opened) |
|
||
| `maintenance.started` | `window`: `id`, `targetType` (`server`, `integration`, `dns_provider`, `device`), `targetId`, `targetName`, `reason`, `startedAt`, `endsAt`, `source` (`null` for one started by hand, `schedule:…` for a recurring one, `kuma:…` for one mirrored from Uptime Kuma) |
|
||
| `maintenance.ended` | the same `window`, plus `endedAt` — one ended early has its `endsAt` brought forward |
|
||
| `test` | `message` |
|
||
|
||
A captured test message, as a receiver sees it (headers lower-cased by the receiving server):
|
||
|
||
```json
|
||
{
|
||
"headers": {
|
||
"x-hlm-event": "test",
|
||
"x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61",
|
||
"x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57",
|
||
"user-agent": "HomelabManager-Webhook",
|
||
"content-type": "application/json"
|
||
},
|
||
"body": {
|
||
"event": "test",
|
||
"id": "72118185-66dd-49d8-8886-aed8bdc40d61",
|
||
"createdAt": "2026-10-20T21:46:21.349Z",
|
||
"source": "homelab-manager",
|
||
"data": { "message": "This is a test from Homelab Manager." }
|
||
}
|
||
}
|
||
```
|
||
|
||
### Checking the signature
|
||
|
||
With a secret, `X-HLM-Signature` is `sha256=` followed by the hex **HMAC-SHA256 of the exact body bytes**, with the secret as the key. Compute it over the **raw body as received** (not over parsed and
|
||
re-serialised JSON), compare in **constant time**, and reject a message that does not match.
|
||
|
||
```js
|
||
import { createHmac, timingSafeEqual } from "node:crypto";
|
||
|
||
export function isFromHomelabManager(secret, rawBody, signatureHeader) {
|
||
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
|
||
const given = Buffer.from(signatureHeader ?? "");
|
||
return given.length === expected.length && timingSafeEqual(given, expected);
|
||
}
|
||
```
|
||
|
||
```python
|
||
import hmac, hashlib
|
||
|
||
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
|
||
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
|
||
return hmac.compare_digest(expected, signature_header or "")
|
||
```
|
||
|
||
```bash
|
||
# Quick check at a shell, with the body saved to a file:
|
||
printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')"
|
||
```
|
||
|
||
The signature proves the message came from someone with the secret; it does **not** stop a captured message being sent again later. If that matters, remember the `id` (or `X-HLM-Delivery`) of what you
|
||
have handled and refuse repeats, and reject a `createdAt` that is far from your clock.
|
||
|
||
### Delivery: what to rely on and what not to
|
||
|
||
- Delivery is **best-effort and never holds anything up.** One try; if it does not get a `2xx` answer, **one more after two seconds** (the same message, the same `id` and `X-HLM-Delivery`); then the
|
||
result is written on the webhook and in its [delivery log](#the-delivery-log-and-sending-again).
|
||
- A request is given **five seconds**. **Redirects are not followed** (a redirect counts as a failure) — give the final address.
|
||
- **There is no queue and no automatic replay.** A receiver that is down for longer than those two tries **misses the event** — until someone sends it again from the delivery log. Messages for different events can arrive in any order. If you need to be certain, treat the
|
||
webhook as a *nudge* and reconcile with the [read API](API.md) (`GET /api/v1/alerts`, `/api/v1/maintenance`), which always says what is true now.
|
||
- Because a retry reuses the delivery id, **handle each id once** (idempotently).
|
||
- Answer quickly (`200` is enough) and do slow work afterwards.
|
||
|
||
### The delivery log and sending again
|
||
|
||
Every message sent to a webhook is written down: when, which event, whether it got through, what came back (`200`, or what went wrong, such as `connect ECONNREFUSED` or `500`), whether it needed its second try, and the
|
||
message itself. **Settings → API and webhooks → Event webhooks → Sent** opens the log for one webhook, newest first, and **Show the message** unfolds the JSON that was sent.
|
||
|
||
**Send again** sends a message from the log **exactly as it was** — the same bytes, the same `id` and `X-HLM-Delivery`, a signature made over the same body — so a receiver that already handled it can recognise it by its id and
|
||
ignore it, and one that missed it handles it now. The new attempt is logged as a delivery of its own, marked *sent again by hand*, and updates the webhook's *Last* result. Like the first send it is tried twice.
|
||
It is for when the receiver was down or the fault was at its end; it does not repair a wrong address.
|
||
|
||
- **The last 100 per webhook, and none older than 14 days,** are kept. Removing a webhook removes its log.
|
||
- A message larger than **16 000 characters** is not kept in full, so it is not shown and cannot be sent again (`409 too_large`). The messages the app sends are far smaller.
|
||
- **Administrators only** — the log holds what was sent, which names servers and says what is wrong with them. Each resend is in the audit log (*webhook › retry*) and in the **Change log**.
|
||
|
||
```
|
||
GET /api/event-webhooks/<id>/deliveries?limit=50 the log (limit 1–100)
|
||
POST /api/event-webhooks/<id>/deliveries/<delivery row id>/retry send one again → { "ok": true, "result": "200" }
|
||
```
|
||
|
||
These are the app's own addresses (they need an administrator's sign-in, not an API token), listed for completeness.
|
||
|
||
### Alert keys
|
||
|
||
`alert.key` is the stable identity of a problem: the same problem has the same key every time, so you can match an `opened` with its `resolved`. The shapes:
|
||
|
||
| Key | About |
|
||
|---|---|
|
||
| `offline:server:<id>` | A server has stopped reporting |
|
||
| `offline:device:<id>` | A UniFi device has been offline too long |
|
||
| `disk:server:<id>:<mount>` | A disk is nearly full |
|
||
| `disk:proxmox:<integration>:…`, `disk:synology:<integration>:<volume>` | A Proxmox storage or Synology volume is nearly full |
|
||
| `synology-volume:<integration>:<volume>`, `synology-disk:<integration>:<disk>` | A Synology volume or disk is not healthy |
|
||
| `failed-units:server:<id>` | Services have failed on a server |
|
||
| `smart:server:<id>:<device>`, `zfs:server:<id>:<pool>` | A disk says it is failing, a pool is not online |
|
||
| `new-port:server:<id>:<proto>:<port>` | A new TCP port started listening |
|
||
| `rule:<rule id>:server:<id>[:<disk>]` | One of your own [alert rules](MONITORING.md#your-own-alert-rules) |
|
||
| `endpoint:<check id>`, `endpoint-cert:<check id>` | An endpoint check is failing, or its certificate is running out |
|
||
| `heartbeat:<id>`, `heartbeat-fail:<id>` | A job reported late, or said it failed |
|
||
| `automation:semaphore:…`, `automation:gitea:…` | A Semaphore task or a Gitea Actions run failed |
|
||
|
||
Treat unknown shapes as opaque; match on the whole key.
|
||
|
||
### A small receiver
|
||
|
||
Node with Express — the one thing to get right is to verify the signature against the **raw** body:
|
||
|
||
```js
|
||
import express from "express";
|
||
import { createHmac, timingSafeEqual } from "node:crypto";
|
||
|
||
const SECRET = process.env.HLM_WEBHOOK_SECRET;
|
||
const seen = new Set(); // use something persistent for real
|
||
|
||
const app = express();
|
||
app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => {
|
||
const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex"));
|
||
const given = Buffer.from(req.get("x-hlm-signature") ?? "");
|
||
if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401);
|
||
|
||
const msg = JSON.parse(req.body.toString("utf8"));
|
||
if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled
|
||
seen.add(msg.id);
|
||
|
||
if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`);
|
||
else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`);
|
||
res.sendStatus(200);
|
||
});
|
||
app.listen(8080);
|
||
```
|
||
|
||
Python with Flask:
|
||
|
||
```python
|
||
import hmac, hashlib, os
|
||
from flask import Flask, request, abort
|
||
|
||
SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode()
|
||
app = Flask(__name__)
|
||
|
||
@app.post("/hlm")
|
||
def hook():
|
||
raw = request.get_data() # the exact bytes
|
||
expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
|
||
if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")):
|
||
abort(401)
|
||
msg = request.get_json()
|
||
if msg["event"] == "alert.opened":
|
||
print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"])
|
||
return "", 200
|
||
```
|
||
|
||
### Home Assistant and n8n
|
||
|
||
- **Home Assistant** — an automation with a *webhook trigger* can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (`/api/webhook/<id>`), which is the access control, so a
|
||
secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward.
|
||
|
||
```yaml
|
||
automation:
|
||
- alias: "Homelab Manager critical alert"
|
||
trigger:
|
||
- platform: webhook
|
||
webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts
|
||
allowed_methods: [POST]
|
||
local_only: true
|
||
condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}"
|
||
action:
|
||
- service: notify.mobile_app_phone
|
||
data:
|
||
title: "Homelab alert"
|
||
message: "{{ trigger.json.data.alert.message }}"
|
||
```
|
||
|
||
- **n8n** — a *Webhook* node (method POST) gives you the address; add a *Code* or *IF* node on `{{$json.body.event}}`. To verify the signature in n8n, enable the node's *Raw Body* option and compute the HMAC
|
||
in a Code node as above.
|
||
|
||
## The notification webhook
|
||
|
||
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
|
||
|
||
**Settings → Notifications → Channels → Webhook**: switch it on, give the address, and optionally a secret.
|
||
|
||
```
|
||
POST <your address>
|
||
Content-Type: application/json
|
||
X-Webhook-Secret: <the secret> (only if one is set)
|
||
|
||
{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" }
|
||
```
|
||
|
||
- The body is exactly `{ "content": "<title in bold>\n<message>" }` — the shape Discord's incoming webhooks take (Slack and Mattermost expect `text`, so they need a small adapter in between).
|
||
- `X-Webhook-Secret` is the secret **as given**, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook.
|
||
- It follows everything a notification follows: the **routing rules** (which channels hear what), **quiet hours**, **maintenance**, **flapping** and **snooze**, and the toggle for each kind of notification. It
|
||
is delivered once with no retry; a failure is written to the server log.
|
||
- The secret is stored encrypted when `CREDENTIALS_ENCRYPTION_KEY` is set. The **Send test** button on the channel sends a one-off message through that channel only.
|
||
- Which messages exist, and when each is sent, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
|
||
|
||
## Buttons on notifications
|
||
|
||
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: **Acknowledge**, **Silence for an hour** and **Snooze until tomorrow at 09:00**. An administrator switches
|
||
it on under **Settings → Notifications → Who hears what → Buttons on notifications**; it is off until then, and off means no link works. How each channel shows them is in [NOTIFICATIONS.md](../NOTIFICATIONS.md#buttons-on-notifications).
|
||
|
||
The addresses behind them, for anything that wants to build on it:
|
||
|
||
```
|
||
GET /api/alert-actions/<token> a page listing the alerts with a button for each action — opening it changes nothing
|
||
POST /api/alert-actions/<token>/ack acknowledge ("Acknowledged from a notification", until the alert is gone)
|
||
POST /api/alert-actions/<token>/silence start an hour of maintenance on what the alert is about (only if it is about something a window can silence)
|
||
POST /api/alert-actions/<token>/snooze snooze until tomorrow 09:00 in the notification time zone
|
||
add ?key=<alert key> (or a form field "key") when the token is for more than one alert
|
||
```
|
||
|
||
```json
|
||
{ "ok": true, "message": "Acknowledged." }
|
||
```
|
||
|
||
- **The token is the access.** It is signed (HMAC-SHA256, with a key derived from `SESSION_SECRET`), is for the **alerts named in it only** (the alert keys of that one message, up to 20), and **ends after 24 hours**. Nothing
|
||
is stored. A forged, altered or ended link is `404`/`410`. A `key` that is not in the token is `400`; an alert that has cleared is `404` ("it has cleared"); a silence for something with nothing to silence it with is `409`.
|
||
- **It can do these three things and nothing else** — no reading, no settings, no other alert. Everything it does is in the audit log, said to be done *from a notification link*, with the IP address it came from.
|
||
- **A POST from a browser form gets a page back; anything else gets JSON.** `GET` never acts, so a link preview, a chat unfurl or a mail scanner cannot press a button. Limited to 60 requests a minute from one address.
|
||
- **Anyone who can read the notification can use its buttons** for the 24 hours. That is the trade for not having to sign in: keep the ntfy topic private (a self-hosted ntfy, or a topic with a password), do not point this at a public
|
||
topic, and remember the link is in an email's text. Changing `SESSION_SECRET` or switching the setting off ends every link at once.
|
||
- The links are built from `APP_BASE_URL`, so it has to be an address your phone can open.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause |
|
||
|---|---|
|
||
| **Test** says `timed out` | The receiver did not answer in five seconds, or is not reachable *from the app* (containers: `localhost` is the container itself). |
|
||
| **Test** says `401`/`403` | Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. |
|
||
| **Test** says `301`/`302`/`308` | Redirects are not followed. Use the final address (`https://` instead of `http://`, the trailing slash). |
|
||
| **Test** works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). |
|
||
| The same alert arrives twice | A retry — the `id` is the same. Handle each `id` once. |
|
||
| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. |
|
||
| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from `/api/v1` or `/metrics`. |
|
||
| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |
|