- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics - Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents, incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs, related alerts and notes; Incidents page and a digest section - Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download, check, upload, delete, and restore staged and applied at start-up before the database opens (pre-restore copy kept); alerts and /metrics for stale or failing backups - Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
8.8 KiB
Metrics and webhooks
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
Settings → API and webhooks: a read-only /metrics address for Prometheus, and webhooks that are
sent a signed message when something happens.
/metrics — for Prometheus
GET /metrics
Authorization: Bearer hlm_m_…
- A metrics token is made on the settings page. It is shown once, only its SHA-256 is kept, and it can be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
- The token goes in the
Authorizationheader only. A token in the address (?token=…) is not accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same401. - The answer is Prometheus's text format,
Content-Type: text/plain; version=0.0.4; charset=utf-8. Requests are limited to 120 a minute from one address. - It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few database reads. (The alert counts are the app's own result, reused for up to a minute.)
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
static_configs:
- targets: ["homelab.example.lan"]
What it says
All gauges. Labels are escaped as Prometheus requires.
| Series | Labels | Meaning |
|---|---|---|
homelab_info |
environment |
Always 1; says which installation this is. |
homelab_server_up |
server |
1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
homelab_server_last_seen_timestamp_seconds |
server |
When the agent last reported. |
homelab_server_cpu_load_percent |
server |
CPU load as the agent reports it. |
homelab_server_memory_used_percent |
server |
Memory in use. |
homelab_server_disk_used_percent |
server, mount |
How full each disk is. |
homelab_server_uptime_ratio_30d |
server |
Share of the last 30 days the server was up, 0–1 (from the uptime record). |
homelab_server_pending_updates |
server |
Updates waiting to be installed, when the agent reports them. |
homelab_server_pending_security_updates |
server |
Of those, the security fixes, when the package manager can say. |
homelab_server_reboot_required |
server |
1 if a restart is due. |
homelab_server_agent_outdated |
server |
1 if the agent is older than the one this installation would install now. |
homelab_server_failed_units |
server |
Services (systemd units) that have failed, when a Linux agent reports them. |
homelab_server_disk_healthy |
server, device |
1 if the disk's own SMART self-assessment passes, 0 if it fails. |
homelab_server_disk_wear_percent |
server, device |
How much of an SSD's rated life is used, where the disk says. |
homelab_server_disk_temperature_celsius |
server, device |
A disk's temperature. |
homelab_server_zfs_pool_online |
server, pool |
1 if a ZFS pool is ONLINE, 0 in any other state. |
homelab_server_disk_days_to_full |
server, mount |
Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
homelab_alerts |
severity |
Active alerts: critical, warning, info (not counting silenced ones). |
homelab_alerts_acknowledged |
Alerts somebody has acknowledged (and that aren't silenced). Still counted in homelab_alerts above. |
|
homelab_incidents_open |
Incidents that have begun and not yet ended. | |
homelab_selfbackup_snapshots |
How many backups of Homelab Manager's own database are kept. | |
homelab_selfbackup_last_success_timestamp_seconds |
When the newest of them was made, as a Unix time. | |
homelab_selfbackup_last_attempt_ok |
1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). | |
homelab_alerts_silenced |
Alerts held back by a maintenance window. | |
homelab_alerts_grouped |
Alerts folded under a server that is down. | |
homelab_integration_up |
integration, type |
1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
homelab_maintenance_active_windows |
Maintenance windows running now. | |
homelab_secret_expiry_days |
name |
Days until a secret expires (negative once it has). |
homelab_inventory_power_watts |
Typical draw of everything in the inventory. | |
homelab_backup_guests |
status |
Proxmox guests by how well they are backed up: ok, stale, failed, none, unknown. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
homelab_backup_last_age_seconds |
guest, vmid, integration |
How long ago a guest's newest backup was. |
homelab_scrape_duration_seconds |
How long building the answer took. | |
homelab_scrape_timestamp_seconds |
When the answer was built. |
Some alerting ideas: homelab_server_up == 0, homelab_server_disk_used_percent > 90,
homelab_secret_expiry_days < 14, homelab_server_disk_days_to_full < 7, homelab_alerts{severity="critical"} > 0, time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400.
Event webhooks
A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person would read.
| Event | Sent when |
|---|---|
alert.opened |
A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone. |
alert.resolved |
A problem ends. |
maintenance.started |
A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma). |
maintenance.ended |
A maintenance window finishes, on its own or ended early. |
Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails
(the same message, with the same X-HLM-Delivery), then the result is noted on the webhook in Settings
(a status code, or what went wrong). A request is given five seconds, and redirects are not followed.
The message
POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-… (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c… (only if the webhook has a secret)
{
"event": "alert.opened",
"id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
"createdAt": "2026-10-17T21:04:11.482Z",
"source": "homelab-manager",
"data": {
"alert": {
"key": "offline:server:3",
"source": "server",
"severity": "critical",
"message": "pve1 hasn't reported for 1 h — also takes down web and db"
}
}
}
alert.openeddata:alertwithkey(stable for the same problem),source,severity(criticalorwarning) andmessage.alert.resolvedhasalert.keyandalert.message.maintenance.started/maintenance.endeddata:windowwithid,targetType(server,integration,dns_provider),targetId,targetName,reason,startedAt,endsAt,source(nullfor one started by hand,schedule:…for a recurring one,kuma:…for one mirrored from Uptime Kuma) and, onended,endedAt.- A Test button on the settings page sends
{"event": "test", …}whatever the webhook is set to.
Checking the signature
With a secret, X-HLM-Signature is sha256= and the hex HMAC-SHA256 of the exact body bytes with the
secret as the key. Compare in constant time, and reject a message whose signature doesn't match.
import { createHmac, timingSafeEqual } from "node:crypto";
function isFromHomelabManager(secret, rawBody, signatureHeader) {
const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
const given = Buffer.from(signatureHeader ?? "");
return given.length === expected.length && timingSafeEqual(given, expected);
}
import hmac, hashlib
def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header or "")
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved. Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never its address or secret.