Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 92726568b7 Endpoint checks, restore tests, and a change log
- Endpoints: HTTP, TCP and TLS checks with answer time, uptime and certificate expiry; failures and expiring certificates become alerts and notifications; linked to a server for maintenance and tags
- Restore tests: record when a backup was restored and how it went; failed tests always alert, overdue ones with a limit set; shown on the Backup coverage page
- Change log: one filterable timeline made from the audit log and the incident record

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 23:03:56 +02:00

11 KiB
Raw Blame History

Metrics and webhooks

Two ways for other systems to work with Homelab Manager, both set up by an administrator under Settings → API and webhooks: a read-only /metrics address for Prometheus, and webhooks that are sent a signed message when something happens.

/metrics — for Prometheus

GET /metrics
Authorization: Bearer hlm_m_…
  • A metrics token is made on the settings page. It is shown once, only its SHA-256 is kept, and it can be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
  • The token goes in the Authorization header only. A token in the address (?token=…) is not accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same 401.
  • The answer is Prometheus's text format, Content-Type: text/plain; version=0.0.4; charset=utf-8. Requests are limited to 120 a minute from one address.
  • It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few database reads. (The alert counts are the app's own result, reused for up to a minute.)
scrape_configs:
  - job_name: homelab-manager
    scheme: https
    metrics_path: /metrics
    authorization:
      credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
    static_configs:
      - targets: ["homelab.example.lan"]

What it says

All gauges. Labels are escaped as Prometheus requires.

Series Labels Meaning
homelab_info environment Always 1; says which installation this is.
homelab_server_up server 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out.
homelab_server_last_seen_timestamp_seconds server When the agent last reported.
homelab_server_cpu_load_percent server CPU load as the agent reports it.
homelab_server_memory_used_percent server Memory in use.
homelab_server_disk_used_percent server, mount How full each disk is.
homelab_server_uptime_ratio_30d server Share of the last 30 days the server was up, 0–1 (from the uptime record).
homelab_server_pending_updates server Updates waiting to be installed, when the agent reports them.
homelab_server_pending_security_updates server Of those, the security fixes, when the package manager can say.
homelab_server_reboot_required server 1 if a restart is due.
homelab_server_agent_outdated server 1 if the agent is older than the one this installation would install now.
homelab_server_failed_units server Services (systemd units) that have failed, when a Linux agent reports them.
homelab_server_disk_healthy server, device 1 if the disk's own SMART self-assessment passes, 0 if it fails.
homelab_server_disk_wear_percent server, device How much of an SSD's rated life is used, where the disk says.
homelab_server_disk_temperature_celsius server, device A disk's temperature.
homelab_server_zfs_pool_online server, pool 1 if a ZFS pool is ONLINE, 0 in any other state.
homelab_server_disk_days_to_full server, mount Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest).
homelab_alerts severity Active alerts: critical, warning, info (not counting silenced ones).
homelab_alerts_acknowledged Alerts somebody has acknowledged (and that aren't silenced). Still counted in homelab_alerts above.
homelab_incidents_open Incidents that have begun and not yet ended.
homelab_selfbackup_snapshots How many backups of Homelab Manager's own database are kept.
homelab_selfbackup_last_success_timestamp_seconds When the newest of them was made, as a Unix time.
homelab_selfbackup_last_attempt_ok 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried).
homelab_alerts_silenced Alerts held back by a maintenance window.
homelab_alerts_grouped Alerts folded under a server that is down.
homelab_integration_up integration, type 1 if the integration's latest calls work, 0 if failing; left out until something has called it.
homelab_maintenance_active_windows Maintenance windows running now.
homelab_secret_expiry_days name Days until a secret expires (negative once it has).
homelab_inventory_power_watts Typical draw of everything in the inventory.
homelab_backup_guests status Proxmox guests by how well they are backed up: ok, stale, failed, none, unknown. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read.
homelab_backup_last_age_seconds guest, vmid, integration How long ago a guest's newest backup was.
homelab_endpoint_up name, kind 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked.
homelab_endpoint_response_milliseconds name, kind How long it took to answer the last time.
homelab_endpoint_certificate_days_left name, kind Days until the certificate it shows expires (negative once it has).
homelab_restore_test_age_days subject Days since a restore of this was last tested; things marked "no need to test" are left out.
homelab_restore_test_passed subject 1 if the last restore test passed (fully or partly), 0 if it failed.
homelab_scrape_duration_seconds How long building the answer took.
homelab_scrape_timestamp_seconds When the answer was built.

Some alerting ideas: homelab_server_up == 0, homelab_server_disk_used_percent > 90, homelab_secret_expiry_days < 14, homelab_server_disk_days_to_full < 7, homelab_alerts{severity="critical"} > 0, time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400.

A Grafana dashboard

Settings → API and webhooks → Download the dashboard (administrators) gives homelab-manager-dashboard.json; the same file is at GET /api/metrics-tokens/grafana-dashboard.json for a signed-in administrator. In Grafana: Dashboards → New → Import, upload it, and choose your Prometheus when asked (the file never names one, so it isn't tied to any installation). It is a dashboard with a Server picker (taken from homelab_server_up) and five rows:

Row What it has
Needs attention Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state.
Servers One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported.
Disks Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools.
Backups Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups.
Secrets, power and the app itself Secrets and certificates by days left, the inventory's power draw, how long /metrics takes to build.

The dashboard is built by the app from the same code that serves /metrics, and a test checks that it only asks for series that are really served — so after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well-formed, every series real), not opened in a Grafana here.

Event webhooks

A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person would read.

Event Sent when
alert.opened A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone.
alert.resolved A problem ends.
maintenance.started A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma).
maintenance.ended A maintenance window finishes, on its own or ended early.

Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails (the same message, with the same X-HLM-Delivery), then the result is noted on the webhook in Settings (a status code, or what went wrong). A request is given five seconds, and redirects are not followed.

The message

POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-…            (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c…         (only if the webhook has a secret)
{
  "event": "alert.opened",
  "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
  "createdAt": "2026-10-17T21:04:11.482Z",
  "source": "homelab-manager",
  "data": {
    "alert": {
      "key": "offline:server:3",
      "source": "server",
      "severity": "critical",
      "message": "pve1 hasn't reported for 1 h — also takes down web and db"
    }
  }
}
  • alert.opened data: alert with key (stable for the same problem), source, severity (critical or warning) and message. alert.resolved has alert.key and alert.message.
  • maintenance.started / maintenance.ended data: window with id, targetType (server, integration, dns_provider), targetId, targetName, reason, startedAt, endsAt, source (null for one started by hand, schedule:… for a recurring one, kuma:… for one mirrored from Uptime Kuma) and, on ended, endedAt.
  • A Test button on the settings page sends {"event": "test", …} whatever the webhook is set to.

Checking the signature

With a secret, X-HLM-Signature is sha256= and the hex HMAC-SHA256 of the exact body bytes with the secret as the key. Compare in constant time, and reject a message whose signature doesn't match.

import { createHmac, timingSafeEqual } from "node:crypto";

function isFromHomelabManager(secret, rawBody, signatureHeader) {
  const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
  const given = Buffer.from(signatureHeader ?? "");
  return given.length === expected.length && timingSafeEqual(given, expected);
}
import hmac, hashlib

def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header or "")

The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved. Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never its address or secret.