Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 5dc6cea69c Host health from the Linux agent, agent self-update, and disk forecast in /metrics
- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates;
  SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox
- Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new
  "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series
- Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified
  download, atomic replace; GET /api/agent/report/update
- homelab_server_disk_days_to_full in /metrics
- Tests (server, real PowerShell against a fake server) and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 00:54:24 +02:00

7.7 KiB
Raw Blame History

Metrics and webhooks

Two ways for other systems to work with Homelab Manager, both set up by an administrator under Settings → API and webhooks: a read-only /metrics address for Prometheus, and webhooks that are sent a signed message when something happens.

/metrics — for Prometheus

GET /metrics
Authorization: Bearer hlm_m_…
  • A metrics token is made on the settings page. It is shown once, only its SHA-256 is kept, and it can be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
  • The token goes in the Authorization header only. A token in the address (?token=…) is not accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same 401.
  • The answer is Prometheus's text format, Content-Type: text/plain; version=0.0.4; charset=utf-8. Requests are limited to 120 a minute from one address.
  • It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few database reads. (The alert counts are the app's own result, reused for up to a minute.)
scrape_configs:
  - job_name: homelab-manager
    scheme: https
    metrics_path: /metrics
    authorization:
      credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
    static_configs:
      - targets: ["homelab.example.lan"]

What it says

All gauges. Labels are escaped as Prometheus requires.

Series Labels Meaning
homelab_info environment Always 1; says which installation this is.
homelab_server_up server 1 if the agent has reported within the offline threshold (Settings → Notifications), else 0. Servers that never reported are left out.
homelab_server_last_seen_timestamp_seconds server When the agent last reported.
homelab_server_cpu_load_percent server CPU load as the agent reports it.
homelab_server_memory_used_percent server Memory in use.
homelab_server_disk_used_percent server, mount How full each disk is.
homelab_server_uptime_ratio_30d server Share of the last 30 days the server was up, 0–1 (from the uptime record).
homelab_server_pending_updates server Updates waiting to be installed, when the agent reports them.
homelab_server_pending_security_updates server Of those, the security fixes, when the package manager can say.
homelab_server_reboot_required server 1 if a restart is due.
homelab_server_agent_outdated server 1 if the agent is older than the one this installation would install now.
homelab_server_failed_units server Services (systemd units) that have failed, when a Linux agent reports them.
homelab_server_disk_healthy server, device 1 if the disk's own SMART self-assessment passes, 0 if it fails.
homelab_server_disk_wear_percent server, device How much of an SSD's rated life is used, where the disk says.
homelab_server_disk_temperature_celsius server, device A disk's temperature.
homelab_server_zfs_pool_online server, pool 1 if a ZFS pool is ONLINE, 0 in any other state.
homelab_server_disk_days_to_full server, mount Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest).
homelab_alerts severity Active alerts: critical, warning, info (not counting silenced ones).
homelab_alerts_silenced Alerts held back by a maintenance window.
homelab_alerts_grouped Alerts folded under a server that is down.
homelab_integration_up integration, type 1 if the integration's latest calls work, 0 if failing; left out until something has called it.
homelab_maintenance_active_windows Maintenance windows running now.
homelab_secret_expiry_days name Days until a secret expires (negative once it has).
homelab_inventory_power_watts Typical draw of everything in the inventory.
homelab_scrape_duration_seconds How long building the answer took.
homelab_scrape_timestamp_seconds When the answer was built.

Some alerting ideas: homelab_server_up == 0, homelab_server_disk_used_percent > 90, homelab_secret_expiry_days < 14, homelab_server_disk_days_to_full < 7, homelab_alerts{severity="critical"} > 0.

Event webhooks

A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person would read.

Event Sent when
alert.opened A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone.
alert.resolved A problem ends.
maintenance.started A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma).
maintenance.ended A maintenance window finishes, on its own or ended early.

Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails (the same message, with the same X-HLM-Delivery), then the result is noted on the webhook in Settings (a status code, or what went wrong). A request is given five seconds, and redirects are not followed.

The message

POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-…            (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c…         (only if the webhook has a secret)
{
  "event": "alert.opened",
  "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
  "createdAt": "2026-10-17T21:04:11.482Z",
  "source": "homelab-manager",
  "data": {
    "alert": {
      "key": "offline:server:3",
      "source": "server",
      "severity": "critical",
      "message": "pve1 hasn't reported for 1 h — also takes down web and db"
    }
  }
}
  • alert.opened data: alert with key (stable for the same problem), source, severity (critical or warning) and message. alert.resolved has alert.key and alert.message.
  • maintenance.started / maintenance.ended data: window with id, targetType (server, integration, dns_provider), targetId, targetName, reason, startedAt, endsAt, source (null for one started by hand, schedule:… for a recurring one, kuma:… for one mirrored from Uptime Kuma) and, on ended, endedAt.
  • A Test button on the settings page sends {"event": "test", …} whatever the webhook is set to.

Checking the signature

With a secret, X-HLM-Signature is sha256= and the hex HMAC-SHA256 of the exact body bytes with the secret as the key. Compare in constant time, and reject a message whose signature doesn't match.

import { createHmac, timingSafeEqual } from "node:crypto";

function isFromHomelabManager(secret, rawBody, signatureHeader) {
  const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
  const given = Buffer.from(signatureHeader ?? "");
  return given.length === expected.length && timingSafeEqual(given, expected);
}
import hmac, hashlib

def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header or "")

The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved. Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never its address or secret.