New Authentik integration (read-only, every 5 minutes): version, workers, outposts and their health, certificates and API tokens that expire, failed sign-ins and configuration errors. Monitoring -> Authentik page. Alerts: an outpost that serves apps has gone quiet (critical), certificate or token about to run out (critical close to it), no worker, failed sign-ins, configuration errors. Held by maintenance, held when unreadable, in /metrics. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
21 KiB
Prometheus and Grafana
Homelab Manager can be scraped by Prometheus at /metrics, and ships a ready-made Grafana dashboard built on those series. This page sets up both, from nothing, and lists every series.
- How it fits together
- 1. Make a metrics token
- 2. Check it by hand
- 3. Set up Prometheus
- 4. Set up Grafana
- A complete example with Docker Compose
- Alerting rules to start from
- Every series
- Useful queries
- Troubleshooting
How it fits together
agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana
(reads its own database; (stores the history,
calls nothing outside) draws the dashboard)
/metricsis a view of what the app already knows. A scrape reads the database and the app's last alert result (reused for up to a minute); it never calls out — not to Proxmox, not to a server, not to an integration. Scraping every 15 seconds costs a few database reads.- The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is not needed to see trends in the app itself. Use Prometheus and Grafana when you want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager.
- Every series is a gauge. Labels are escaped as Prometheus requires.
1. Make a metrics token
Administrators: Settings → API and webhooks → Metrics for Prometheus → Make a token.
- The token (it starts with
hlm_m_) is shown once; only its SHA-256 is kept. - Up to ten can exist — make one per Prometheus, so one can be removed without touching the others. A removed token stops working at once.
- Each shows when it was last used, so an unused one is easy to find.
- The token goes in the
Authorization: Bearer …header only. A token in the address (?token=…) is not accepted (addresses end up in logs). - No token, a wrong one and a removed one all get the same
401. Requests are limited to 120 a minute per address (429beyond). - The answer is Prometheus's text format:
Content-Type: text/plain; version=0.0.4; charset=utf-8.
A metrics token can read /metrics and nothing else. It is a different thing from the personal API tokens used for the read API, and neither can sign in.
2. Check it by hand
curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30
You should see something like this (trimmed; the # HELP lines are left out here):
# TYPE homelab_info gauge
homelab_info{environment=""} 1
# TYPE homelab_server_up gauge
homelab_server_up{server="wiki-host"} 1
# TYPE homelab_server_last_seen_timestamp_seconds gauge
homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670
# TYPE homelab_storage_days_to_full gauge
homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21
# TYPE homelab_alerts gauge
homelab_alerts{severity="critical"} 1
homelab_alerts{severity="warning"} 3
homelab_alerts{severity="info"} 0
Without the header you get 401 — that is the right answer:
curl -i https://hm.example.lan/metrics # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics"
3. Set up Prometheus
Put the token in a file of its own, readable only by Prometheus, with no trailing newline:
printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token
chmod 600 /etc/prometheus/homelab-manager.token
prometheus.yml:
global:
scrape_interval: 30s
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/homelab-manager.token
static_configs:
- targets: ["hm.example.lan"] # host[:port] of APP_BASE_URL, without https://
# If the certificate comes from your own CA, point at it (rather than skipping verification):
# tls_config:
# ca_file: /etc/prometheus/my-ca.pem
- 30–60 seconds is plenty. The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes).
- Prometheus accepts the token inline too (
credentials: hlm_m_…), but a file keeps it out of the configuration you share. - Behind a reverse proxy, use the address the proxy serves (
scheme: https, port 443). Reaching the container directly isscheme: http, targethost:3000— fine on a trusted network. - Reload Prometheus (
kill -HUP, orPOST /-/reloadif enabled) and open Status → Targets:homelab-managershould be UP. - Several installations (production and a dev copy): the series
homelab_info{environment="…"}carries the name set byENVIRONMENT_NAME, so you can tell them apart; add aninstance/envlabel in the scrape config as well.
4. Set up Grafana
- Add Prometheus as a data source — Connections → Data sources → Add → Prometheus, URL e.g.
http://prometheus:9090, Save & test. - Download the dashboard — in Homelab Manager: Settings → API and webhooks → Download the dashboard (administrators). It is a file,
homelab-manager-dashboard.json. (The same file is served to a signed-in administrator atGET /api/metrics-tokens/grafana-dashboard.json.) - Import it — in Grafana: Dashboards → New → Import, upload the file, and choose your Prometheus when asked. The file never names a data source, so it is not tied to any installation.
The dashboard has a Server picker (filled from homelab_server_up) and five rows:
| Row | What it has |
|---|---|
| Needs attention | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
| Servers | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
| Disks | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
| Backups | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
| Secrets, power and the app itself | Secrets and certificates by days left, the inventory's power draw, how long /metrics takes to build. |
The dashboard is built by the app from the same code that serves /metrics, and a test checks that it only asks for series that are really served. So after an upgrade that adds series, download it again
to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), but it has not been opened in a Grafana
by the author — if a panel looks wrong, the series table below says what each query should be.
A complete example with Docker Compose
Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder:
monitoring/
├─ docker-compose.yml
├─ prometheus.yml (as above, with credentials_file: /etc/prometheus/homelab-manager.token)
├─ homelab-manager.token (the token only, no newline)
└─ rules/
└─ homelab-manager.yml (the alerting rules below, optional)
# docker-compose.yml
services:
prometheus:
image: prom/prometheus:latest
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=400d # a year and a bit: the dashboard's long ranges are then real
- --web.enable-lifecycle
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro
- ./rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports: ["9090:9090"]
grafana:
image: grafana/grafana:latest
restart: unless-stopped
volumes:
- grafana-data:/var/lib/grafana
ports: ["3001:3000"]
depends_on: [prometheus]
volumes:
prometheus-data:
grafana-data:
With the rules, add to prometheus.yml:
rule_files:
- /etc/prometheus/rules/*.yml
In Grafana the data source URL is then http://prometheus:9090 (the service name, since they share a compose network).
Alerting rules to start from
Homelab Manager already sends its own notifications (see MONITORING.md). Use these in addition only if you want the same things in Alertmanager, otherwise you will be told twice. They are the ones worth having outside the app — above all, the app itself going quiet.
# rules/homelab-manager.yml
groups:
- name: homelab-manager
rules:
# The one thing the app cannot tell you about itself.
- alert: HomelabManagerDown
expr: up{job="homelab-manager"} == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "Homelab Manager cannot be scraped" }
- alert: HomelabCriticalAlerts
expr: homelab_alerts{severity="critical"} > 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" }
- alert: HomelabServerDown
expr: homelab_server_up == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $labels.server }} has stopped reporting" }
- alert: HomelabDiskFilling
expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7
for: 1h
labels: { severity: warning }
annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" }
- alert: HomelabDiskUsageHigh
expr: homelab_server_disk_used_percent > 90
for: 15m
labels: { severity: warning }
annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" }
- alert: HomelabSelfBackupStale
expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400
labels: { severity: warning }
annotations: { summary: "Homelab Manager's own backup is more than two days old" }
- alert: HomelabEndpointDown
expr: homelab_endpoint_up == 0
for: 5m
labels: { severity: warning }
annotations: { summary: "Endpoint check {{ $labels.name }} is failing" }
- alert: HomelabCertificateExpiring
expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14
labels: { severity: warning }
annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" }
- alert: HomelabHeartbeatMissed
expr: homelab_heartbeat_up == 0
labels: { severity: warning }
annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" }
- alert: HomelabIntegrationFailing
expr: homelab_integration_up == 0
for: 10m
labels: { severity: warning }
annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" }
A series that is left out is not zero: homelab_integration_up, homelab_endpoint_up and homelab_selfbackup_last_attempt_ok do not exist until something has been tried. That is why the rules above
compare against a condition rather than use absent().
Every series
All gauges. The same table is checked by a test against what /metrics really serves.
| Series | Labels | Meaning |
|---|---|---|
homelab_info |
environment |
Always 1; says which installation this is. |
homelab_server_up |
server |
1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
homelab_server_last_seen_timestamp_seconds |
server |
When the agent last reported. |
homelab_server_cpu_load_percent |
server |
CPU load as the agent reports it. |
homelab_server_memory_used_percent |
server |
Memory in use. |
homelab_server_disk_used_percent |
server, mount |
How full each disk is. |
homelab_server_uptime_ratio_30d |
server |
Share of the last 30 days the server was up, 0–1 (from the uptime record). |
homelab_server_pending_updates |
server |
Updates waiting to be installed, when the agent reports them. |
homelab_server_pending_security_updates |
server |
Of those, the security fixes, when the package manager can say. |
homelab_server_reboot_required |
server |
1 if a restart is due. |
homelab_server_agent_outdated |
server |
1 if the agent is older than the one this installation would install now. |
homelab_server_failed_units |
server |
Services (systemd units) that have failed, when a Linux agent reports them. |
homelab_server_disk_healthy |
server, device |
1 if the disk's own SMART self-assessment passes, 0 if it fails. |
homelab_server_disk_wear_percent |
server, device |
How much of an SSD's rated life is used, where the disk says. |
homelab_server_disk_temperature_celsius |
server, device |
A disk's temperature. |
homelab_server_zfs_pool_online |
server, pool |
1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state. |
homelab_server_disk_days_to_full |
server, mount |
Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
homelab_storage_days_to_full |
pool |
Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb. |
homelab_alerts |
severity |
Active alerts: critical, warning, info (not counting silenced ones). |
homelab_alerts_acknowledged |
Alerts somebody has acknowledged (and that aren't silenced). Still counted in homelab_alerts. |
|
homelab_alerts_silenced |
Alerts held back by a maintenance window or a snooze. | |
homelab_alerts_grouped |
Alerts folded under a server that is down. | |
homelab_incidents_open |
Incidents that have begun and not yet ended. | |
homelab_selfbackup_snapshots |
How many backups of Homelab Manager's own database are kept. | |
homelab_selfbackup_last_success_timestamp_seconds |
When the newest of them was made, as a Unix time. | |
homelab_selfbackup_last_attempt_ok |
1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). | |
homelab_integration_up |
integration, type |
1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
homelab_maintenance_active_windows |
Maintenance windows running now. | |
homelab_secret_expiry_days |
name |
Days until a secret expires (negative once it has). |
homelab_inventory_power_watts |
Typical draw of everything in the inventory. | |
homelab_backup_guests |
status |
Proxmox guests by how well they are backed up: ok, stale, failed, none, unknown. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
homelab_backup_last_age_seconds |
guest, vmid, integration |
How long ago a guest's newest backup was. |
homelab_endpoint_up |
name, kind |
1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
homelab_endpoint_response_milliseconds |
name, kind |
How long it took to answer the last time. |
homelab_endpoint_value |
name, kind |
The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
homelab_endpoint_certificate_days_left |
name, kind |
Days until the certificate it shows expires (negative once it has). |
homelab_service_level_target_percent |
name, subject, type |
The availability a service-level target asks for. |
homelab_service_level_availability_percent, homelab_service_level_error_budget_remaining_minutes |
name, subject, type |
How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
homelab_duplicati_backup_ok |
backup, source |
1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
homelab_duplicati_last_success_age_seconds, homelab_duplicati_source_size_bytes |
backup, source |
Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
homelab_authentik_outpost_up |
name, outpost |
1 if the Authentik outpost has had an instance report in the last five minutes, else 0; left out for an Authentik not read in the last twenty minutes. |
homelab_authentik_workers, homelab_authentik_failed_logins_24h |
name |
How many Authentik workers are connected, and how many sign-ins failed in the last day (100 means at least that). |
homelab_authentik_certificate_expiry_days |
name, certificate |
Whole days until a certificate (one Authentik does not renew itself) runs out; negative once it has. |
homelab_jellyfin_streams, homelab_jellyfin_transcodes |
name |
How many are streaming from Jellyfin right now, and how many of those it is transcoding; left out for a Jellyfin not read in the last ten minutes. |
homelab_jellyfin_pending_restart |
name |
1 if Jellyfin is waiting to be restarted, else 0. |
homelab_ups_on_battery |
name |
1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
homelab_ups_battery_charge_percent, homelab_ups_runtime_seconds, homelab_ups_load_percent, homelab_ups_input_volts |
name |
How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
homelab_heartbeat_up |
name |
1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
homelab_heartbeat_last_ping_timestamp_seconds |
name |
When the job last reported that it finished (Unix time). |
homelab_heartbeat_last_duration_seconds |
name |
How long the job said its last run took (only when it sent /start). |
homelab_restore_test_age_days |
subject |
Days since a restore of this was last tested; things marked "no need to test" are left out. |
homelab_restore_test_passed |
subject |
1 if the last restore test passed (fully or partly), 0 if it failed. |
homelab_scrape_duration_seconds |
How long building the answer took. | |
homelab_scrape_timestamp_seconds |
When the answer was built. |
Useful queries
# Servers that have gone quiet
homelab_server_up == 0
# The five fullest disks
topk(5, homelab_server_disk_used_percent)
# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full)
sort(homelab_server_disk_days_to_full < 30)
# Endpoints slower than half a second
homelab_endpoint_response_milliseconds > 500
# How many guests are not well backed up
sum(homelab_backup_guests{status=~"stale|failed|none"})
# Updates waiting across all servers, and how many are security fixes
sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates)
# Is the dev copy up as well as production? (several installations: filter on the label you added)
homelab_info
Troubleshooting
| Symptom | Cause |
|---|---|
Target is DOWN, 401 Unauthorized |
The token is wrong, was removed, or has a trailing newline in its file (use printf '%s'). The same 401 is given for no token and a bad one. |
Target is DOWN, 429 Too many requests |
More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval. |
| Target is DOWN, certificate error | Prometheus does not trust the certificate. Give it the CA with tls_config.ca_file (or insecure_skip_verify: true on a trusted LAN, knowing what that gives up). |
| Target UP but some series are missing | A series that has nothing to say is left out rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected. |
| A dashboard panel is empty | The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade. |
| Numbers differ from the Alerts page | homelab_alerts counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute. |
| Names with odd characters look wrong | Label values are escaped (\", \\, \n) as the Prometheus format requires; Prometheus and Grafana unescape them. |