Files
Homelab-manager/API.md
T
bobbanandClaude Sonnet 5.5 18ceed62e8 Racks, notes and to-do, OS support, suggested dependencies, Nginx Proxy Manager and Home Assistant
- Racks page and a network diagram (UniFi tree with servers, printable)
- Notes on servers and devices; To do list shown in Coming up and My work
- Agents report the operating system (Linux 7, Windows 6); end-of-life badge and alert
- Suggested dependencies from where a service is reached
- Nginx Proxy Manager integration: proxy hosts page, certificate and host alerts, suggested checks
- Home Assistant integration: watch chosen entities, history chart, limit and unavailable alerts
- TimeChart gets a "value" scale that fits negatives; docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 21:47:56 +02:00

20 KiB
Raw Blame History

Metrics, webhooks, heartbeats and a calendar feed

Ways for other systems to work with Homelab Manager:

  • a read-only /metrics address for Prometheus, and event webhooks that are sent a signed message when something happens — both set up by an administrator under Settings → API and webhooks (the first two sections below);
  • a read API (/api/v1) with a personal, read-only token for each person, for scripts, Home Assistant and dashboards (the section after /metrics);
  • heartbeats: a cron job or script calls an address when it finishes, and the app speaks up when it doesn't (further down);
  • a calendar feed of what is coming up, for a phone's calendar to subscribe to (at the end).

/metrics — for Prometheus

GET /metrics
Authorization: Bearer hlm_m_…
  • A metrics token is made on the settings page. It is shown once, only its SHA-256 is kept, and it can be removed at any time (it stops working at once). Up to ten can exist, e.g. one per Prometheus.
  • The token goes in the Authorization header only. A token in the address (?token=…) is not accepted, because addresses end up in logs. No token, a wrong one and a removed one get the same 401.
  • The answer is Prometheus's text format, Content-Type: text/plain; version=0.0.4; charset=utf-8. Requests are limited to 120 a minute from one address.
  • It reads what the app already knows — it calls out to nothing — so scraping every 15 seconds costs a few database reads. (The alert counts are the app's own result, reused for up to a minute.)
scrape_configs:
  - job_name: homelab-manager
    scheme: https
    metrics_path: /metrics
    authorization:
      credentials: hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx
    static_configs:
      - targets: ["homelab.example.lan"]

What it says

All gauges. Labels are escaped as Prometheus requires.

Series Labels Meaning
homelab_info environment Always 1; says which installation this is.
homelab_server_up server 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out.
homelab_server_last_seen_timestamp_seconds server When the agent last reported.
homelab_server_cpu_load_percent server CPU load as the agent reports it.
homelab_server_memory_used_percent server Memory in use.
homelab_server_disk_used_percent server, mount How full each disk is.
homelab_server_uptime_ratio_30d server Share of the last 30 days the server was up, 0–1 (from the uptime record).
homelab_server_pending_updates server Updates waiting to be installed, when the agent reports them.
homelab_server_pending_security_updates server Of those, the security fixes, when the package manager can say.
homelab_server_reboot_required server 1 if a restart is due.
homelab_server_agent_outdated server 1 if the agent is older than the one this installation would install now.
homelab_server_failed_units server Services (systemd units) that have failed, when a Linux agent reports them.
homelab_server_disk_healthy server, device 1 if the disk's own SMART self-assessment passes, 0 if it fails.
homelab_server_disk_wear_percent server, device How much of an SSD's rated life is used, where the disk says.
homelab_server_disk_temperature_celsius server, device A disk's temperature.
homelab_server_zfs_pool_online server, pool 1 if a ZFS pool is ONLINE, 0 in any other state.
homelab_server_disk_days_to_full server, mount Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest).
homelab_alerts severity Active alerts: critical, warning, info (not counting silenced ones).
homelab_alerts_acknowledged Alerts somebody has acknowledged (and that aren't silenced). Still counted in homelab_alerts above.
homelab_incidents_open Incidents that have begun and not yet ended.
homelab_selfbackup_snapshots How many backups of Homelab Manager's own database are kept.
homelab_selfbackup_last_success_timestamp_seconds When the newest of them was made, as a Unix time.
homelab_selfbackup_last_attempt_ok 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried).
homelab_alerts_silenced Alerts held back by a maintenance window.
homelab_alerts_grouped Alerts folded under a server that is down.
homelab_integration_up integration, type 1 if the integration's latest calls work, 0 if failing; left out until something has called it.
homelab_maintenance_active_windows Maintenance windows running now.
homelab_secret_expiry_days name Days until a secret expires (negative once it has).
homelab_inventory_power_watts Typical draw of everything in the inventory.
homelab_backup_guests status Proxmox guests by how well they are backed up: ok, stale, failed, none, unknown. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read.
homelab_backup_last_age_seconds guest, vmid, integration How long ago a guest's newest backup was.
homelab_endpoint_up name, kind 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked.
homelab_endpoint_response_milliseconds name, kind How long it took to answer the last time.
homelab_endpoint_certificate_days_left name, kind Days until the certificate it shows expires (negative once it has).
homelab_heartbeat_up name 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out.
homelab_heartbeat_last_ping_timestamp_seconds name When the job last reported that it finished (Unix time).
homelab_heartbeat_last_duration_seconds name How long the job said its last run took (only when it sent /start).
homelab_restore_test_age_days subject Days since a restore of this was last tested; things marked "no need to test" are left out.
homelab_restore_test_passed subject 1 if the last restore test passed (fully or partly), 0 if it failed.
homelab_scrape_duration_seconds How long building the answer took.
homelab_scrape_timestamp_seconds When the answer was built.

Some alerting ideas: homelab_server_up == 0, homelab_server_disk_used_percent > 90, homelab_secret_expiry_days < 14, homelab_server_disk_days_to_full < 7, homelab_alerts{severity="critical"} > 0, time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400.

The read API — /api/v1

JSON about servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance — for a script, Home Assistant, a dashboard. It only reads.

GET /api/v1/alerts
Authorization: Bearer hlm_p_…
  • A personal token, made by whoever is going to use it under the user menu → API tokens (everyone signed in can, whatever their role). It is shown once, only its SHA-256 is kept, and it is that person's own: they see and remove theirs, nobody else's. It can last 30, 90 or 365 days or until removed, a person may have ten, and it stops working at once when removed — and when the person is.
  • The token goes in the Authorization header only. A token in the address is not accepted (addresses end up in logs). No token, a wrong one, an expired one and a removed one all get the same 401 with a JSON body.
  • Only GET (and HEAD). Any other verb is 405, with a token or without: there is nothing here that changes anything.
  • A token sees what its owner sees on the pages — no secrets, no settings, no passwords, no tokens of servers. Requests are limited to 300 a minute from one address. Answers are Cache-Control: no-store.
  • Dates are ISO 8601 in UTC, sizes are bytes, percentages are 0–100. An address inside the app (url) is a full address, built from APP_BASE_URL.
Address What it answers
GET /api/v1 The list of these addresses.
GET /api/v1/me Whose token this is: the person's name and role, and the token's name and end.
GET /api/v1/servers Every server: status (online, offline, unknown), lastSeenAt, tags, ipAddresses, cpu, memory, disks (with usedPercent), updates, hardware, operatingSystem (name, version, endsOn, state: ok, soon, ended or unknown, daysLeft; null until the agent says), url. Filter with ?tag= and ?status=.
GET /api/v1/servers/:id One server, the same shape.
GET /api/v1/servers/:id/metrics?days= Its load, memory and disks as lines of points, 1 to 365 days (7 by default). Points over more than 35 days are hours' or days' averages, with lo and hi. Disks also carry a trend and, when rising steadily, daysToFull.
GET /api/v1/alerts What is wrong now: needingAttention (critical, warning, acknowledged), counts, couldntCheck, and alerts with severity, category, message, silenced, flapping, target, causedBy, acknowledgement (who, when, note, until) and url. Filter with ?severity=, ?category=, `?acknowledged=true
GET /api/v1/incidents?status=&limit= The record of alerts that began and ended: open or resolved, newest first, up to 500 (100 by default).
GET /api/v1/incidents/:id One incident with its timeline.
GET /api/v1/inventory The hardware: maker, model, serial, where, bought, warranty, end of support, power.
GET /api/v1/network Switches, access points and gateways read from UniFi (state, address, uptime, firmware, uplink), the networks and Wi-Fi names, and which server is plugged into which device.
GET /api/v1/endpoints The checks: status (up, down, unknown, paused), last answer, certificate end, 24-hour and 7-day uptime.
GET /api/v1/endpoints/:id/history?days= A check's availability and answer times over 1 to 365 days, as points, from each answer (up to 2 days), the hourly figures (up to 120) or the daily ones.
GET /api/v1/maintenance What is under maintenance now (active) and what is coming (upcoming).
{ "generatedAt": "2026-10-20T09:14:02.511Z",
  "needingAttention": { "critical": 1, "warning": 0, "acknowledged": 1 },
  "alerts": [
    { "id": "offline:offline:server:3", "severity": "critical", "category": "offline", "source": "Server",
      "message": "anna hasn't reported for 9 h", "silenced": false, "flapping": false,
      "target": { "type": "server", "id": 3, "name": "anna" }, "causedBy": null, "affects": [],
      "acknowledgement": { "by": "Ada", "at": "2026-10-20T08:01:00.000Z", "note": "On it", "until": "2026-10-20T12:01:00.000Z", "assignee": "Ada" },
      "url": "https://hm.example.lan/servers/3" } ] }

Errors are JSON too: { "error": "unauthorized" | "read_only" | "not_found" | "invalid_id", "message": … }. In Home Assistant, a rest sensor with the token in headers: and value_template: "{{ value_json.needingAttention.critical }}" is enough for a count of what needs attention.

A Grafana dashboard

Settings → API and webhooks → Download the dashboard (administrators) gives homelab-manager-dashboard.json; the same file is at GET /api/metrics-tokens/grafana-dashboard.json for a signed-in administrator. In Grafana: Dashboards → New → Import, upload it, and choose your Prometheus when asked (the file never names one, so it isn't tied to any installation). It is a dashboard with a Server picker (taken from homelab_server_up) and five rows:

Row What it has
Needs attention Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state.
Servers One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported.
Disks Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools.
Backups Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups.
Secrets, power and the app itself Secrets and certificates by days left, the inventory's power draw, how long /metrics takes to build.

The dashboard is built by the app from the same code that serves /metrics, and a test checks that it only asks for series that are really served — so after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well-formed, every series real), not opened in a Grafana here.

Event webhooks

A webhook is a URL (http or https, no user name or password in it) with a list of events it wants. It is separate from the chat-style webhook under Settings → Notifications, which carries the sentence a person would read.

Event Sent when
alert.opened A problem begins. Announced on the same terms as a notification: a problem under a maintenance window, or folded under a server that is down, is not announced until it stands alone.
alert.resolved A problem ends.
maintenance.started A maintenance window begins, however it was made (by hand, in bulk, from a recurring schedule, mirrored from Uptime Kuma).
maintenance.ended A maintenance window finishes, on its own or ended early.

Delivery is best-effort and never holds anything up: one try, one more after two seconds if that fails (the same message, with the same X-HLM-Delivery), then the result is noted on the webhook in Settings (a status code, or what went wrong). A request is given five seconds, and redirects are not followed.

The message

POST <your url>
Content-Type: application/json
User-Agent: HomelabManager-Webhook
X-HLM-Event: alert.opened
X-HLM-Delivery: 5b0c3e0a-…            (the same for a retry of the same message)
X-HLM-Signature: sha256=9f2c…         (only if the webhook has a secret)
{
  "event": "alert.opened",
  "id": "5b0c3e0a-1f4e-4a55-9d0e-6c1c1c1c1c1c",
  "createdAt": "2026-10-17T21:04:11.482Z",
  "source": "homelab-manager",
  "data": {
    "alert": {
      "key": "offline:server:3",
      "source": "server",
      "severity": "critical",
      "message": "pve1 hasn't reported for 1 h — also takes down web and db"
    }
  }
}
  • alert.opened data: alert with key (stable for the same problem), source, severity (critical or warning) and message. alert.resolved has alert.key and alert.message.
  • maintenance.started / maintenance.ended data: window with id, targetType (server, integration, dns_provider), targetId, targetName, reason, startedAt, endsAt, source (null for one started by hand, schedule:… for a recurring one, kuma:… for one mirrored from Uptime Kuma) and, on ended, endedAt.
  • A Test button on the settings page sends {"event": "test", …} whatever the webhook is set to.

Checking the signature

With a secret, X-HLM-Signature is sha256= and the hex HMAC-SHA256 of the exact body bytes with the secret as the key. Compare in constant time, and reject a message whose signature doesn't match.

import { createHmac, timingSafeEqual } from "node:crypto";

function isFromHomelabManager(secret, rawBody, signatureHeader) {
  const expected = Buffer.from(`sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`);
  const given = Buffer.from(signatureHeader ?? "");
  return given.length === expected.length && timingSafeEqual(given, expected);
}
import hmac, hashlib

def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header or "")

The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved. Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never its address or secret.

Heartbeats — for cron jobs and scripts

A heartbeat is made by an administrator under Monitoring → Heartbeats, and its address is shown there to administrators. The job calls it when it finishes:

GET  /api/heartbeat/<token>          it worked
GET  /api/heartbeat/<token>/start    it began (optional: lets the next call say how long it took)
GET  /api/heartbeat/<token>/fail     it ran and went wrong: an alert at once
# in cron, after the job
0 3 * * *  /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx

# in a script, with a start, a failure and a note
curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start
./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \
           || curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail
  • The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work. Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once).
  • Anything sent with a call — the body of a POST, or ?msg=… — is kept as a note, cut to one line of at most 200 characters.
  • The answer is 200 and OK. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get 404 Not found in the same words. Calls are limited to 120 a minute from one address.
  • A job is late when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes; a call to /fail is critical and stays until a call that works. A heartbeat that is switched off still answers 200.
  • Both are alerts of the Automation category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them.

Calendar feed

An administrator makes a calendar feed under Settings → Share links: an address ending in .ics for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL").

GET /api/public/calendar/<token>.ics
  • Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time. A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets 404 in the same words. Limited to 60 requests a minute from one address.
  • It carries what Coming up lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out.
  • With kinds only, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With names it carries the name and a link back to the app. An event's UID is a hash of what it is and its date, the same on every fetch.
  • The feed asks to be refreshed every six hours (REFRESH-INTERVAL); a calendar app decides for itself how often it does.