Alert settings keeps Alert rules, Service levels and Alert tickets; administrators get a link to Settings -> Notifications. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
20 KiB
Webhooks
Homelab Manager can tell other systems when something happens — Home Assistant, n8n, a status board, your own script — instead of making them ask. There are two different things called a webhook here, and they are easy to mix up:
| Event webhooks | The notification webhook | |
|---|---|---|
| For | Machines: a structured, signed JSON message per event | People: the same sentence a human would read, posted to a chat-style endpoint |
| Where | Settings → API and webhooks → Event webhooks | Settings → Notifications → Channels → Webhook |
| Sent when | An alert opens or resolves, a maintenance window starts or ends | Whenever a notification is sent to a channel (it is one of the five channels) |
| Body | { event, id, createdAt, source, data } |
{ "content": "**Title**\nmessage" } |
| Authenticity | X-HLM-Signature — an HMAC-SHA256 of the body |
a shared secret in X-Webhook-Secret (not a signature) |
| Held back by routing, quiet hours, flapping, snooze | No (it is a separate way of being told) | Yes (it is a notification like the others) |
Event webhooks
Setting one up
Administrators: Settings → API and webhooks → Event webhooks → Add a webhook.
| Field | Notes |
|---|---|
| Name | up to 60 characters; what the list shows |
| Address | http:// or https://, no user name or password in it. The address is not restricted to public hosts — a receiver on your LAN is the normal case. |
| Secret | optional, up to 200 characters. With one, every message is signed. Stored encrypted when CREDENTIALS_ENCRYPTION_KEY is set; never shown again after saving. |
| Events | at least one of the four below |
- Up to 20 webhooks. Each can be switched off and on, removed, and tested. The secret cannot be edited afterwards: to change it, remove the webhook and make a new one.
- Test sends
{"event": "test", …}right away, signed like the real thing, whether or not the webhook is switched on or wants that event, and says how it went (200, or what went wrong). - The list shows when the last message was sent and its result — a status code,
timed out, or the error. - Making, switching and removing a webhook are in the audit log with its name and events — never its address or secret (the address can carry a token of its own).
Events
| Event | Sent when |
|---|---|
alert.opened |
A problem begins. |
alert.resolved |
A problem ends. |
maintenance.started |
A maintenance window begins — by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma. |
maintenance.ended |
A maintenance window finishes, on its own or ended early. |
What counts as an alert here. alert.opened and alert.resolved are sent for the alerts that go through the health-style bookkeeping — server offline, disks, failed services, SMART and ZFS problems, new
ports, network devices offline, your own alert rules, endpoint checks, heartbeats, and automation failures — on the same terms as a notification: a problem
under a maintenance window, or folded under a server that is down, is not announced until it stands alone. Two things that do hold back a notification do not hold back the webhook: a problem
being flapping, and one that is snoozed — a machine should hear every raise and clear. Summaries (an expiring secret, a stale backup, Docker updates, the weekly digest) are not events; read
those from the read API or /metrics.
The message
POST <your address>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
data |
Fields |
|---|---|
alert.opened |
alert.key (stable for the same problem — see Alert keys), alert.source (where it came from, e.g. server, proxmox:3:pve1, endpoint, heartbeat), alert.severity (critical or warning), alert.message |
alert.resolved |
alert.key, alert.message (what it said when it opened) |
maintenance.started |
window: id, targetType (server, integration, dns_provider, device), targetId, targetName, reason, startedAt, endsAt, source (null for one started by hand, schedule:… for a recurring one, kuma:… for one mirrored from Uptime Kuma) |
maintenance.ended |
the same window, plus endedAt — one ended early has its endsAt brought forward |
test |
message |
A captured test message, as a receiver sees it (headers lower-cased by the receiving server):
{
"headers": {
"x-hlm-event": "test",
"x-hlm-delivery": "72118185-66dd-49d8-8886-aed8bdc40d61",
"x-hlm-signature": "sha256=c10f409aac74524e3bc84c0c062f536aa44b98c3bede082f970dfa496fb81f57",
"user-agent": "HomelabManager-Webhook",
"content-type": "application/json"
},
"body": {
"event": "test",
"id": "72118185-66dd-49d8-8886-aed8bdc40d61",
"createdAt": "2026-10-20T21:46:21.349Z",
"source": "homelab-manager",
"data": { "message": "This is a test from Homelab Manager." }
}
}
Checking the signature
With a secret, X-HLM-Signature is sha256= followed by the hex HMAC-SHA256 of the exact body bytes, with the secret as the key. Compute it over the raw body as received (not over parsed and
re-serialised JSON), compare in constant time, and reject a message that does not match.
import { createHmac, timingSafeEqual } from "node:crypto";
export function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
# Quick check at a shell, with the body saved to a file:
printf 'sha256=%s\n' "$(openssl dgst -sha256 -hmac "$SECRET" -hex < body.json | sed 's/^.* //')"
The signature proves the message came from someone with the secret; it does not stop a captured message being sent again later. If that matters, remember the id (or X-HLM-Delivery) of what you
have handled and refuse repeats, and reject a createdAt that is far from your clock.
Delivery: what to rely on and what not to
- Delivery is best-effort and never holds anything up. One try; if it does not get a
2xxanswer, one more after two seconds (the same message, the sameidandX-HLM-Delivery); then the result is written on the webhook and in its delivery log. - A request is given five seconds. Redirects are not followed (a redirect counts as a failure) — give the final address.
- There is no queue and no automatic replay. A receiver that is down for longer than those two tries misses the event — until someone sends it again from the delivery log. Messages for different events can arrive in any order. If you need to be certain, treat the
webhook as a nudge and reconcile with the read API (
GET /api/v1/alerts,/api/v1/maintenance), which always says what is true now. - Because a retry reuses the delivery id, handle each id once (idempotently).
- Answer quickly (
200is enough) and do slow work afterwards.
The delivery log and sending again
Every message sent to a webhook is written down: when, which event, whether it got through, what came back (200, or what went wrong, such as connect ECONNREFUSED or 500), whether it needed its second try, and the
message itself. Settings → API and webhooks → Event webhooks → Sent opens the log for one webhook, newest first, and Show the message unfolds the JSON that was sent.
Send again sends a message from the log exactly as it was — the same bytes, the same id and X-HLM-Delivery, a signature made over the same body — so a receiver that already handled it can recognise it by its id and
ignore it, and one that missed it handles it now. The new attempt is logged as a delivery of its own, marked sent again by hand, and updates the webhook's Last result. Like the first send it is tried twice.
It is for when the receiver was down or the fault was at its end; it does not repair a wrong address.
- The last 100 per webhook, and none older than 14 days, are kept. Removing a webhook removes its log.
- A message larger than 16 000 characters is not kept in full, so it is not shown and cannot be sent again (
409 too_large). The messages the app sends are far smaller. - Administrators only — the log holds what was sent, which names servers and says what is wrong with them. Each resend is in the audit log (webhook › retry) and in the Change log.
GET /api/event-webhooks/<id>/deliveries?limit=50 the log (limit 1–100)
POST /api/event-webhooks/<id>/deliveries/<delivery row id>/retry send one again → { "ok": true, "result": "200" }
These are the app's own addresses (they need an administrator's sign-in, not an API token), listed for completeness.
Alert keys
alert.key is the stable identity of a problem: the same problem has the same key every time, so you can match an opened with its resolved. The shapes:
| Key | About |
|---|---|
offline:server:<id> |
A server has stopped reporting |
offline:device:<id> |
A UniFi device has been offline too long |
disk:server:<id>:<mount> |
A disk is nearly full |
disk:proxmox:<integration>:…, disk:synology:<integration>:<volume> |
A Proxmox storage or Synology volume is nearly full |
synology-volume:<integration>:<volume>, synology-disk:<integration>:<disk> |
A Synology volume or disk is not healthy |
failed-units:server:<id> |
Services have failed on a server |
smart:server:<id>:<device>, zfs:server:<id>:<pool> |
A disk says it is failing, a pool is not online |
new-port:server:<id>:<proto>:<port> |
A new TCP port started listening |
rule:<rule id>:server:<id>[:<disk>] |
One of your own alert rules |
endpoint:<check id>, endpoint-cert:<check id> |
An endpoint check is failing, or its certificate is running out |
heartbeat:<id>, heartbeat-fail:<id> |
A job reported late, or said it failed |
automation:semaphore:…, automation:gitea:… |
A Semaphore task or a Gitea Actions run failed |
Treat unknown shapes as opaque; match on the whole key.
A small receiver
Node with Express — the one thing to get right is to verify the signature against the raw body:
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
const SECRET = process.env.HLM_WEBHOOK_SECRET;
const seen = new Set(); // use something persistent for real
const app = express();
app.post("/hlm", express.raw({ type: "application/json" }), (req, res) => {
const expected = Buffer.from("sha256=" + createHmac("sha256", SECRET).update(req.body).digest("hex"));
const given = Buffer.from(req.get("x-hlm-signature") ?? "");
if (given.length !== expected.length || !timingSafeEqual(given, expected)) return res.sendStatus(401);
const msg = JSON.parse(req.body.toString("utf8"));
if (seen.has(msg.id)) return res.sendStatus(200); // a retry of something already handled
seen.add(msg.id);
if (msg.event === "alert.opened") console.log(`[${msg.data.alert.severity}] ${msg.data.alert.message}`);
else if (msg.event === "alert.resolved") console.log(`resolved: ${msg.data.alert.message}`);
res.sendStatus(200);
});
app.listen(8080);
Python with Flask:
import hmac, hashlib, os
from flask import Flask, request, abort
SECRET = os.environ["HLM_WEBHOOK_SECRET"].encode()
app = Flask(__name__)
@app.post("/hlm")
def hook():
raw = request.get_data() # the exact bytes
expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, request.headers.get("X-HLM-Signature", "")):
abort(401)
msg = request.get_json()
if msg["event"] == "alert.opened":
print(msg["data"]["alert"]["severity"], msg["data"]["alert"]["message"])
return "", 200
Home Assistant and n8n
-
Home Assistant — an automation with a webhook trigger can take the message as it comes. Home Assistant's webhook URL has an unguessable id in it (
/api/webhook/<id>), which is the access control, so a secret here is optional; if you want the signature, check it in a small receiver like the ones above and forward.automation: - alias: "Homelab Manager critical alert" trigger: - platform: webhook webhook_id: homelab_manager_alerts # then the address is https://ha.example.lan/api/webhook/homelab_manager_alerts allowed_methods: [POST] local_only: true condition: "{{ trigger.json.event == 'alert.opened' and trigger.json.data.alert.severity == 'critical' }}" action: - service: notify.mobile_app_phone data: title: "Homelab alert" message: "{{ trigger.json.data.alert.message }}" -
n8n — a Webhook node (method POST) gives you the address; add a Code or IF node on
{{$json.body.event}}. To verify the signature in n8n, enable the node's Raw Body option and compute the HMAC in a Code node as above.
The notification webhook
The fifth notification channel (beside Gotify, ntfy, email and push) posts each notification to a URL you give it, as the sentence a person would read — handy for chat tools that accept an incoming-webhook post.
Settings → Notifications → Channels → Webhook: switch it on, give the address, and optionally a secret.
POST <your address>
Content-Type: application/json
X-Webhook-Secret: <the secret> (only if one is set)
{ "content": "**Homelab Manager — Health Alert**\n1 new issue:\n\n⚠ pve1 hasn't reported for 1 h" }
- The body is exactly
{ "content": "<title in bold>\n<message>" }— the shape Discord's incoming webhooks take (Slack and Mattermost expecttext, so they need a small adapter in between). X-Webhook-Secretis the secret as given, not a signature — it only lets a receiver reject strangers. For something verifiable, use an event webhook.- It follows everything a notification follows: the routing rules (which channels hear what), quiet hours, maintenance, flapping and snooze, and the toggle for each kind of notification. It is delivered once with no retry; a failure is written to the server log.
- The secret is stored encrypted when
CREDENTIALS_ENCRYPTION_KEYis set. The Send test button on the channel sends a one-off message through that channel only. - Which messages exist, and when each is sent, is in NOTIFICATIONS.md.
Buttons on notifications
A notification can carry buttons that act on the alert it is about, from the phone's lock screen, without signing in: Acknowledge, Silence for an hour and Snooze until tomorrow at 09:00. An administrator switches it on under Settings → Notifications → Who hears what → Buttons on notifications; it is off until then, and off means no link works. How each channel shows them is in NOTIFICATIONS.md.
The addresses behind them, for anything that wants to build on it:
GET /api/alert-actions/<token> a page listing the alerts with a button for each action — opening it changes nothing
POST /api/alert-actions/<token>/ack acknowledge ("Acknowledged from a notification", until the alert is gone)
POST /api/alert-actions/<token>/silence start an hour of maintenance on what the alert is about (only if it is about something a window can silence)
POST /api/alert-actions/<token>/snooze snooze until tomorrow 09:00 in the notification time zone
add ?key=<alert key> (or a form field "key") when the token is for more than one alert
{ "ok": true, "message": "Acknowledged." }
- The token is the access. It is signed (HMAC-SHA256, with a key derived from
SESSION_SECRET), is for the alerts named in it only (the alert keys of that one message, up to 20), and ends after 24 hours. Nothing is stored. A forged, altered or ended link is404/410. Akeythat is not in the token is400; an alert that has cleared is404("it has cleared"); a silence for something with nothing to silence it with is409. - It can do these three things and nothing else — no reading, no settings, no other alert. Everything it does is in the audit log, said to be done from a notification link, with the IP address it came from.
- A POST from a browser form gets a page back; anything else gets JSON.
GETnever acts, so a link preview, a chat unfurl or a mail scanner cannot press a button. Limited to 60 requests a minute from one address. - Anyone who can read the notification can use its buttons for the 24 hours. That is the trade for not having to sign in: keep the ntfy topic private (a self-hosted ntfy, or a topic with a password), do not point this at a public
topic, and remember the link is in an email's text. Changing
SESSION_SECRETor switching the setting off ends every link at once. - The links are built from
APP_BASE_URL, so it has to be an address your phone can open.
Troubleshooting
| Symptom | Cause |
|---|---|
Test says timed out |
The receiver did not answer in five seconds, or is not reachable from the app (containers: localhost is the container itself). |
Test says 401/403 |
Your receiver rejected it — often a signature check against the wrong secret, or one computed over parsed rather than raw JSON. |
Test says 301/302/308 |
Redirects are not followed. Use the final address (https:// instead of http://, the trailing slash). |
| Test works but real events never arrive | The webhook is switched off, or does not have that event ticked, or the alert is under maintenance / folded under a host (not announced until it stands alone). |
| The same alert arrives twice | A retry — the id is the same. Handle each id once. |
| An alert I snoozed or that is flapping still arrived | By design: machines hear every raise and clear. |
| A secret about to expire / a stale backup did not produce an event | Summaries are not events. Read them from /api/v1 or /metrics. |
| I lost the secret | It cannot be shown again. Remove the webhook and make a new one. |