Files
Homelab-manager/docs/PROMETHEUS-GRAFANA.md
T
bobbanandClaude Sonnet 5.5 2f082ca5c2 Duplicati: know when a backup failed or has gone quiet
Duplicati reports each operation to a secret address (its own send-http-url option), so
nothing calls Duplicati and no password for it is kept. Each machine is a source; each job
is learned by name and shows its last result, duration, size, changes and a history.
A failed run is a warning (critical from the second in a row); one that has not succeeded
for longer than expected (per job, 2 days by default) is a warning, critical at three
times that. Maintenance on the machine's server silences them; /metrics series added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 21:36:13 +02:00

20 KiB
Raw Blame History

Prometheus and Grafana

Homelab Manager can be scraped by Prometheus at /metrics, and ships a ready-made Grafana dashboard built on those series. This page sets up both, from nothing, and lists every series.

How it fits together

 agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana
                                      (reads its own database;           (stores the history,
                                       calls nothing outside)             draws the dashboard)
  • /metrics is a view of what the app already knows. A scrape reads the database and the app's last alert result (reused for up to a minute); it never calls out — not to Proxmox, not to a server, not to an integration. Scraping every 15 seconds costs a few database reads.
  • The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is not needed to see trends in the app itself. Use Prometheus and Grafana when you want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager.
  • Every series is a gauge. Labels are escaped as Prometheus requires.

1. Make a metrics token

Administrators: Settings → API and webhooks → Metrics for Prometheus → Make a token.

  • The token (it starts with hlm_m_) is shown once; only its SHA-256 is kept.
  • Up to ten can exist — make one per Prometheus, so one can be removed without touching the others. A removed token stops working at once.
  • Each shows when it was last used, so an unused one is easy to find.
  • The token goes in the Authorization: Bearer … header only. A token in the address (?token=…) is not accepted (addresses end up in logs).
  • No token, a wrong one and a removed one all get the same 401. Requests are limited to 120 a minute per address (429 beyond).
  • The answer is Prometheus's text format: Content-Type: text/plain; version=0.0.4; charset=utf-8.

A metrics token can read /metrics and nothing else. It is a different thing from the personal API tokens used for the read API, and neither can sign in.

2. Check it by hand

curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30

You should see something like this (trimmed; the # HELP lines are left out here):

# TYPE homelab_info gauge
homelab_info{environment=""} 1
# TYPE homelab_server_up gauge
homelab_server_up{server="wiki-host"} 1
# TYPE homelab_server_last_seen_timestamp_seconds gauge
homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670
# TYPE homelab_storage_days_to_full gauge
homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21
# TYPE homelab_alerts gauge
homelab_alerts{severity="critical"} 1
homelab_alerts{severity="warning"} 3
homelab_alerts{severity="info"} 0

Without the header you get 401 — that is the right answer:

curl -i https://hm.example.lan/metrics        # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics"

3. Set up Prometheus

Put the token in a file of its own, readable only by Prometheus, with no trailing newline:

printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token
chmod 600 /etc/prometheus/homelab-manager.token

prometheus.yml:

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: homelab-manager
    scheme: https
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/homelab-manager.token
    static_configs:
      - targets: ["hm.example.lan"]          # host[:port] of APP_BASE_URL, without https://
    # If the certificate comes from your own CA, point at it (rather than skipping verification):
    # tls_config:
    #   ca_file: /etc/prometheus/my-ca.pem
  • 30–60 seconds is plenty. The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes).
  • Prometheus accepts the token inline too (credentials: hlm_m_…), but a file keeps it out of the configuration you share.
  • Behind a reverse proxy, use the address the proxy serves (scheme: https, port 443). Reaching the container directly is scheme: http, target host:3000 — fine on a trusted network.
  • Reload Prometheus (kill -HUP, or POST /-/reload if enabled) and open Status → Targets: homelab-manager should be UP.
  • Several installations (production and a dev copy): the series homelab_info{environment="…"} carries the name set by ENVIRONMENT_NAME, so you can tell them apart; add an instance/env label in the scrape config as well.

4. Set up Grafana

  1. Add Prometheus as a data source — Connections → Data sources → Add → Prometheus, URL e.g. http://prometheus:9090, Save & test.
  2. Download the dashboard — in Homelab Manager: Settings → API and webhooks → Download the dashboard (administrators). It is a file, homelab-manager-dashboard.json. (The same file is served to a signed-in administrator at GET /api/metrics-tokens/grafana-dashboard.json.)
  3. Import it — in Grafana: Dashboards → New → Import, upload the file, and choose your Prometheus when asked. The file never names a data source, so it is not tied to any installation.

The dashboard has a Server picker (filled from homelab_server_up) and five rows:

Row What it has
Needs attention Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state.
Servers One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported.
Disks Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools.
Backups Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups.
Secrets, power and the app itself Secrets and certificates by days left, the inventory's power draw, how long /metrics takes to build.

The dashboard is built by the app from the same code that serves /metrics, and a test checks that it only asks for series that are really served. So after an upgrade that adds series, download it again to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), but it has not been opened in a Grafana by the author — if a panel looks wrong, the series table below says what each query should be.

A complete example with Docker Compose

Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder:

monitoring/
├─ docker-compose.yml
├─ prometheus.yml            (as above, with credentials_file: /etc/prometheus/homelab-manager.token)
├─ homelab-manager.token     (the token only, no newline)
└─ rules/
   └─ homelab-manager.yml    (the alerting rules below, optional)
# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=400d       # a year and a bit: the dashboard's long ranges are then real
      - --web.enable-lifecycle
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro
      - ./rules:/etc/prometheus/rules:ro
      - prometheus-data:/prometheus
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana
    ports: ["3001:3000"]
    depends_on: [prometheus]

volumes:
  prometheus-data:
  grafana-data:

With the rules, add to prometheus.yml:

rule_files:
  - /etc/prometheus/rules/*.yml

In Grafana the data source URL is then http://prometheus:9090 (the service name, since they share a compose network).

Alerting rules to start from

Homelab Manager already sends its own notifications (see MONITORING.md). Use these in addition only if you want the same things in Alertmanager, otherwise you will be told twice. They are the ones worth having outside the app — above all, the app itself going quiet.

# rules/homelab-manager.yml
groups:
  - name: homelab-manager
    rules:
      # The one thing the app cannot tell you about itself.
      - alert: HomelabManagerDown
        expr: up{job="homelab-manager"} == 0
        for: 5m
        labels: { severity: critical }
        annotations: { summary: "Homelab Manager cannot be scraped" }

      - alert: HomelabCriticalAlerts
        expr: homelab_alerts{severity="critical"} > 0
        for: 5m
        labels: { severity: critical }
        annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" }

      - alert: HomelabServerDown
        expr: homelab_server_up == 0
        for: 5m
        labels: { severity: critical }
        annotations: { summary: "{{ $labels.server }} has stopped reporting" }

      - alert: HomelabDiskFilling
        expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7
        for: 1h
        labels: { severity: warning }
        annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" }

      - alert: HomelabDiskUsageHigh
        expr: homelab_server_disk_used_percent > 90
        for: 15m
        labels: { severity: warning }
        annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" }

      - alert: HomelabSelfBackupStale
        expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400
        labels: { severity: warning }
        annotations: { summary: "Homelab Manager's own backup is more than two days old" }

      - alert: HomelabEndpointDown
        expr: homelab_endpoint_up == 0
        for: 5m
        labels: { severity: warning }
        annotations: { summary: "Endpoint check {{ $labels.name }} is failing" }

      - alert: HomelabCertificateExpiring
        expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14
        labels: { severity: warning }
        annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" }

      - alert: HomelabHeartbeatMissed
        expr: homelab_heartbeat_up == 0
        labels: { severity: warning }
        annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" }

      - alert: HomelabIntegrationFailing
        expr: homelab_integration_up == 0
        for: 10m
        labels: { severity: warning }
        annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" }

A series that is left out is not zero: homelab_integration_up, homelab_endpoint_up and homelab_selfbackup_last_attempt_ok do not exist until something has been tried. That is why the rules above compare against a condition rather than use absent().

Every series

All gauges. The same table is checked by a test against what /metrics really serves.

Series Labels Meaning
homelab_info environment Always 1; says which installation this is.
homelab_server_up server 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out.
homelab_server_last_seen_timestamp_seconds server When the agent last reported.
homelab_server_cpu_load_percent server CPU load as the agent reports it.
homelab_server_memory_used_percent server Memory in use.
homelab_server_disk_used_percent server, mount How full each disk is.
homelab_server_uptime_ratio_30d server Share of the last 30 days the server was up, 0–1 (from the uptime record).
homelab_server_pending_updates server Updates waiting to be installed, when the agent reports them.
homelab_server_pending_security_updates server Of those, the security fixes, when the package manager can say.
homelab_server_reboot_required server 1 if a restart is due.
homelab_server_agent_outdated server 1 if the agent is older than the one this installation would install now.
homelab_server_failed_units server Services (systemd units) that have failed, when a Linux agent reports them.
homelab_server_disk_healthy server, device 1 if the disk's own SMART self-assessment passes, 0 if it fails.
homelab_server_disk_wear_percent server, device How much of an SSD's rated life is used, where the disk says.
homelab_server_disk_temperature_celsius server, device A disk's temperature.
homelab_server_zfs_pool_online server, pool 1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state.
homelab_server_disk_days_to_full server, mount Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest).
homelab_storage_days_to_full pool Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb.
homelab_alerts severity Active alerts: critical, warning, info (not counting silenced ones).
homelab_alerts_acknowledged Alerts somebody has acknowledged (and that aren't silenced). Still counted in homelab_alerts.
homelab_alerts_silenced Alerts held back by a maintenance window or a snooze.
homelab_alerts_grouped Alerts folded under a server that is down.
homelab_incidents_open Incidents that have begun and not yet ended.
homelab_selfbackup_snapshots How many backups of Homelab Manager's own database are kept.
homelab_selfbackup_last_success_timestamp_seconds When the newest of them was made, as a Unix time.
homelab_selfbackup_last_attempt_ok 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried).
homelab_integration_up integration, type 1 if the integration's latest calls work, 0 if failing; left out until something has called it.
homelab_maintenance_active_windows Maintenance windows running now.
homelab_secret_expiry_days name Days until a secret expires (negative once it has).
homelab_inventory_power_watts Typical draw of everything in the inventory.
homelab_backup_guests status Proxmox guests by how well they are backed up: ok, stale, failed, none, unknown. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read.
homelab_backup_last_age_seconds guest, vmid, integration How long ago a guest's newest backup was.
homelab_endpoint_up name, kind 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked.
homelab_endpoint_response_milliseconds name, kind How long it took to answer the last time.
homelab_endpoint_value name, kind The number a value check last read from the answer (checks that judge a value only, and only when it is a number).
homelab_endpoint_certificate_days_left name, kind Days until the certificate it shows expires (negative once it has).
homelab_service_level_target_percent name, subject, type The availability a service-level target asks for.
homelab_service_level_availability_percent, homelab_service_level_error_budget_remaining_minutes name, subject, type How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month.
homelab_duplicati_backup_ok backup, source 1 if the Duplicati backup's last run worked and it is not overdue, else 0.
homelab_duplicati_last_success_age_seconds, homelab_duplicati_source_size_bytes backup, source Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined.
homelab_ups_on_battery name 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes.
homelab_ups_battery_charge_percent, homelab_ups_runtime_seconds, homelab_ups_load_percent, homelab_ups_input_volts name How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say.
homelab_heartbeat_up name 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out.
homelab_heartbeat_last_ping_timestamp_seconds name When the job last reported that it finished (Unix time).
homelab_heartbeat_last_duration_seconds name How long the job said its last run took (only when it sent /start).
homelab_restore_test_age_days subject Days since a restore of this was last tested; things marked "no need to test" are left out.
homelab_restore_test_passed subject 1 if the last restore test passed (fully or partly), 0 if it failed.
homelab_scrape_duration_seconds How long building the answer took.
homelab_scrape_timestamp_seconds When the answer was built.

Useful queries

# Servers that have gone quiet
homelab_server_up == 0

# The five fullest disks
topk(5, homelab_server_disk_used_percent)

# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full)
sort(homelab_server_disk_days_to_full < 30)

# Endpoints slower than half a second
homelab_endpoint_response_milliseconds > 500

# How many guests are not well backed up
sum(homelab_backup_guests{status=~"stale|failed|none"})

# Updates waiting across all servers, and how many are security fixes
sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates)

# Is the dev copy up as well as production? (several installations: filter on the label you added)
homelab_info

Troubleshooting

Symptom Cause
Target is DOWN, 401 Unauthorized The token is wrong, was removed, or has a trailing newline in its file (use printf '%s'). The same 401 is given for no token and a bad one.
Target is DOWN, 429 Too many requests More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval.
Target is DOWN, certificate error Prometheus does not trust the certificate. Give it the CA with tls_config.ca_file (or insecure_skip_verify: true on a trusted LAN, knowing what that gives up).
Target UP but some series are missing A series that has nothing to say is left out rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected.
A dashboard panel is empty The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade.
Numbers differ from the Alerts page homelab_alerts counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute.
Names with odd characters look wrong Label values are escaped (\", \\, \n) as the Prometheus format requires; Prometheus and Grafana unescape them.