Files
Homelab-manager/docs/PROMETHEUS-GRAFANA.md
T
bobbanandClaude Sonnet 5.5 01acbaf831 Add Hetzner Robot: dedicated servers and storage boxes, with shutting-down, locked and rescue-left-on alerts
Read every 15 minutes (Robot allows 200 requests an hour; read-only): each server's product, addresses, status, rescue system and paid-until, and the storage boxes. Adapter, sync service, hetzner_robot_state table, /api/hetzner-robot, Infrastructure → Hetzner Robot page, metrics and docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-04 23:26:26 +01:00

401 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Prometheus and Grafana
Homelab Manager can be scraped by **Prometheus** at `/metrics`, and ships a ready-made **Grafana dashboard** built on those series. This page sets up both, from nothing, and lists every series.
- [How it fits together](#how-it-fits-together)
- [1. Make a metrics token](#1-make-a-metrics-token)
- [2. Check it by hand](#2-check-it-by-hand)
- [3. Set up Prometheus](#3-set-up-prometheus)
- [4. Set up Grafana](#4-set-up-grafana)
- [A complete example with Docker Compose](#a-complete-example-with-docker-compose)
- [Alerting rules to start from](#alerting-rules-to-start-from)
- [Every series](#every-series)
- [Useful queries](#useful-queries)
- [Troubleshooting](#troubleshooting)
## How it fits together
```
agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana
(reads its own database; (stores the history,
calls nothing outside) draws the dashboard)
```
- `/metrics` is a **view of what the app already knows**. A scrape reads the database and the app's last alert result (reused for up to a minute); it **never calls out** — not to Proxmox, not to a server, not
to an integration. Scraping every 15 seconds costs a few database reads.
- The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is **not needed** to see trends in the app itself. Use Prometheus and Grafana when you
want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager.
- **Every series is a gauge.** Labels are escaped as Prometheus requires.
## 1. Make a metrics token
Administrators: **Settings → API and webhooks → Metrics for Prometheus → Make a token**.
- The token (it starts with `hlm_m_`) is **shown once**; only its SHA-256 is kept.
- Up to **ten** can exist — make one per Prometheus, so one can be removed without touching the others. A removed token **stops working at once**.
- Each shows when it was last used, so an unused one is easy to find.
- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted** (addresses end up in logs).
- No token, a wrong one and a removed one all get the same `401`. Requests are limited to **120 a minute** per address (`429` beyond).
- The answer is Prometheus's text format: `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
A metrics token can read `/metrics` and nothing else. It is a different thing from the *personal API tokens* used for the [read API](API.md), and neither can sign in.
## 2. Check it by hand
```bash
curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30
```
You should see something like this (trimmed; the `# HELP` lines are left out here):
```
# TYPE homelab_info gauge
homelab_info{environment=""} 1
# TYPE homelab_server_up gauge
homelab_server_up{server="wiki-host"} 1
# TYPE homelab_server_last_seen_timestamp_seconds gauge
homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670
# TYPE homelab_storage_days_to_full gauge
homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21
# TYPE homelab_alerts gauge
homelab_alerts{severity="critical"} 1
homelab_alerts{severity="warning"} 3
homelab_alerts{severity="info"} 0
```
Without the header you get `401` — that is the right answer:
```bash
curl -i https://hm.example.lan/metrics # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics"
```
## 3. Set up Prometheus
Put the token in a file of its own, readable only by Prometheus, with **no trailing newline**:
```bash
printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token
chmod 600 /etc/prometheus/homelab-manager.token
```
`prometheus.yml`:
```yaml
global:
scrape_interval: 30s
scrape_configs:
- job_name: homelab-manager
scheme: https
metrics_path: /metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/homelab-manager.token
static_configs:
- targets: ["hm.example.lan"] # host[:port] of APP_BASE_URL, without https://
# If the certificate comes from your own CA, point at it (rather than skipping verification):
# tls_config:
# ca_file: /etc/prometheus/my-ca.pem
```
- **30–60 seconds is plenty.** The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes).
- Prometheus accepts the token inline too (`credentials: hlm_m_…`), but a file keeps it out of the configuration you share.
- Behind a reverse proxy, use the address the proxy serves (`scheme: https`, port 443). Reaching the container directly is `scheme: http`, target `host:3000` — fine on a trusted network.
- Reload Prometheus (`kill -HUP`, or `POST /-/reload` if enabled) and open **Status → Targets**: `homelab-manager` should be **UP**.
- **Several installations** (production and a dev copy): the series `homelab_info{environment="…"}` carries the name set by `ENVIRONMENT_NAME`, so you can tell them apart; add an `instance`/`env` label in the scrape
config as well.
## 4. Set up Grafana
1. **Add Prometheus as a data source** — *Connections → Data sources → Add → Prometheus*, URL e.g. `http://prometheus:9090`, *Save & test*.
2. **Download the dashboard** — in Homelab Manager: **Settings → API and webhooks → Download the dashboard** (administrators). It is a file, `homelab-manager-dashboard.json`. (The same file is served to a signed-in
administrator at `GET /api/metrics-tokens/grafana-dashboard.json`.)
3. **Import it** — in Grafana: **Dashboards → New → Import**, upload the file, and **choose your Prometheus** when asked. The file never names a data source, so it is not tied to any installation.
The dashboard has a **Server** picker (filled from `homelab_server_up`) and five rows:
| Row | What it has |
|---|---|
| **Needs attention** | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
| **Servers** | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
| **Disks** | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
| **Backups** | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
| **Secrets, power and the app itself** | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. |
The dashboard is **built by the app from the same code that serves `/metrics`**, and a test checks that it only asks for series that are really served. So **after an upgrade that adds series, download it again**
to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), **but it has not been opened in a Grafana
by the author** — if a panel looks wrong, the series table below says what each query should be.
## A complete example with Docker Compose
Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder:
```
monitoring/
├─ docker-compose.yml
├─ prometheus.yml (as above, with credentials_file: /etc/prometheus/homelab-manager.token)
├─ homelab-manager.token (the token only, no newline)
└─ rules/
└─ homelab-manager.yml (the alerting rules below, optional)
```
```yaml
# docker-compose.yml
services:
prometheus:
image: prom/prometheus:latest
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=400d # a year and a bit: the dashboard's long ranges are then real
- --web.enable-lifecycle
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro
- ./rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports: ["9090:9090"]
grafana:
image: grafana/grafana:latest
restart: unless-stopped
volumes:
- grafana-data:/var/lib/grafana
ports: ["3001:3000"]
depends_on: [prometheus]
volumes:
prometheus-data:
grafana-data:
```
With the rules, add to `prometheus.yml`:
```yaml
rule_files:
- /etc/prometheus/rules/*.yml
```
In Grafana the data source URL is then `http://prometheus:9090` (the service name, since they share a compose network).
## Alerting rules to start from
Homelab Manager already sends its own notifications (see [MONITORING.md](MONITORING.md)). **Use these in addition only if you want the same things in Alertmanager**, otherwise you will be told twice.
They are the ones worth having *outside* the app — above all, the app itself going quiet.
```yaml
# rules/homelab-manager.yml
groups:
- name: homelab-manager
rules:
# The one thing the app cannot tell you about itself.
- alert: HomelabManagerDown
expr: up{job="homelab-manager"} == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "Homelab Manager cannot be scraped" }
- alert: HomelabCriticalAlerts
expr: homelab_alerts{severity="critical"} > 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" }
- alert: HomelabServerDown
expr: homelab_server_up == 0
for: 5m
labels: { severity: critical }
annotations: { summary: "{{ $labels.server }} has stopped reporting" }
- alert: HomelabDiskFilling
expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7
for: 1h
labels: { severity: warning }
annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" }
- alert: HomelabDiskUsageHigh
expr: homelab_server_disk_used_percent > 90
for: 15m
labels: { severity: warning }
annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" }
- alert: HomelabSelfBackupStale
expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400
labels: { severity: warning }
annotations: { summary: "Homelab Manager's own backup is more than two days old" }
- alert: HomelabEndpointDown
expr: homelab_endpoint_up == 0
for: 5m
labels: { severity: warning }
annotations: { summary: "Endpoint check {{ $labels.name }} is failing" }
- alert: HomelabCertificateExpiring
expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14
labels: { severity: warning }
annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" }
- alert: HomelabHeartbeatMissed
expr: homelab_heartbeat_up == 0
labels: { severity: warning }
annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" }
- alert: HomelabIntegrationFailing
expr: homelab_integration_up == 0
for: 10m
labels: { severity: warning }
annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" }
```
A series that is **left out** is not zero: `homelab_integration_up`, `homelab_endpoint_up` and `homelab_selfbackup_last_attempt_ok` do not exist until something has been tried. That is why the rules above
compare against a condition rather than use `absent()`.
## Every series
All gauges. The same table is checked by a test against what `/metrics` really serves.
| Series | Labels | Meaning |
|---|---|---|
| `homelab_info` | `environment` | Always 1; says which installation this is. |
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_storage_days_to_full` | `pool` | Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb. |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts`. |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window or a snooze. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
| `homelab_endpoint_value` | `name`, `kind` | The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
| `homelab_service_level_target_percent` | `name`, `subject`, `type` | The availability a service-level target asks for. |
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
| `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
| `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
| `homelab_nextcloud_cron_ok` | `name` | 1 if Nextcloud's own check says its background jobs ran within the last hour, else 0; absent when that could not be asked (no login, or before Nextcloud 28). |
| `homelab_nextcloud_maintenance_mode`, `homelab_nextcloud_app_updates` | `name` | 1 if Nextcloud is in maintenance mode, else 0; and how many apps have an update waiting (left out when not asked). Left out for a Nextcloud not read in the last twenty minutes. |
| `homelab_nextcloud_free_bytes`, `homelab_nextcloud_active_users_24h` | `name` | Bytes free where the data is kept, and the users active in the last day (both need an administrator's app password). |
| `homelab_docker_vulnerabilities` | `integration`, `host`, `severity` | The vulnerabilities in the scanned images of a Docker host, by severity (`critical`, `high`, `medium`, `low`), from Dockhand's stored scans. Left out for a host whose count could not be read, and when the reading is older than 45 minutes. |
| `homelab_docker_images_scanned` | `integration`, `host` | How many of a Docker host's images have a scan in Dockhand (the rest are not known to be clean). |
| `homelab_speedtest_download_bits_per_second`, `homelab_speedtest_upload_bits_per_second` | `name` | The download and upload speed of the newest completed speed test of a Speedtest Tracker, in bits per second. Left out while the reading is older than 45 minutes. |
| `homelab_speedtest_ping_seconds` | `name` | The ping of that test, in seconds. |
| `homelab_hetzner_server_running` | `project`, `server` | 1 if the Hetzner Cloud server is running, else 0. |
| `homelab_hetzner_traffic_outgoing_bytes`, `homelab_hetzner_traffic_included_bytes` | `project`, `server` | This month's outgoing traffic of a Hetzner Cloud server, and what its type includes per month (bytes). |
| `homelab_hetzner_monthly_cost` | `project`, `currency` | The estimated monthly cost of a Hetzner Cloud project at list prices, including VAT (an estimate: it has the load balancers and floating IPs, not traffic over the quota). |
| `homelab_hetzner_traffic_forecast_bytes` | `project`, `server` | What a Hetzner Cloud server's outgoing traffic is forecast to come to by the end of the month; left out when there is too little to say. |
| `homelab_hetzner_load_balancer_unhealthy_targets` | `project`, `load_balancer` | How many targets of a Hetzner Cloud load balancer are unhealthy. |
| `homelab_hetzner_certificate_expiry_timestamp_seconds` | `project`, `certificate` | When a Hetzner Cloud certificate ends, in seconds since 1970. |
| `homelab_hetzner_robot_servers` | `name` | How many dedicated servers a Hetzner Robot account has. |
| `homelab_hetzner_robot_rescue_active` | `name`, `server`, `server_name` | 1 if a dedicated server's rescue system is on for the next boot, else 0; left out when it could not be asked. |
| `homelab_hetzner_robot_days_left` | `name`, `server`, `server_name` — or `storage_box`, `storage_box_name` | Calendar days left of a cancelled dedicated server or storage box: 0 on its last day, negative after. Only for cancelled ones. |
| `homelab_hetzner_robot_storage_box_locked` | `name`, `storage_box`, `storage_box_name` | 1 if a storage box is locked, else 0. |
| `homelab_arr_health_issues` | `app`, `name` | How many warnings or errors Sonarr, Radarr or Prowlarr reports about itself. |
| `homelab_arr_queue_items`, `homelab_arr_queue_stuck_items` | `app`, `name` | The downloads in the queue of Sonarr or Radarr, and how many are stuck (judged wrong, failed or blocked for two hours or more). |
| `homelab_arr_wanted_missing` | `app`, `name` | Wanted items Sonarr or Radarr has no file for. |
| `homelab_arr_disk_free_bytes` | `app`, `name`, `path` | Free space of a disk Sonarr, Radarr or Prowlarr uses. |
| `homelab_download_speed_bytes` | `client`, `name`, `direction` | How fast qBittorrent or SABnzbd is downloading (`down`) or uploading (`up`), in bytes per second; SABnzbd does not say `up`. |
| `homelab_download_items` | `client`, `name`, `state` | How many items the download client has in each state (downloading, queued, stalled, seeding …); a state with none is left out except downloading and queued. |
| `homelab_download_stuck_items` | `client`, `name` | How many items are stuck (in an error state, or stalled with nothing moving for a day). |
| `homelab_download_paused`, `homelab_download_connected` | `client`, `name` | 1 if the whole queue is paused (SABnzbd), and 1 if the client can reach the network, i.e. is not `disconnected` (qBittorrent only), else 0. |
| `homelab_download_failed_recent` | `client`, `name` | Downloads that failed in the last day (SABnzbd only). |
| `homelab_download_disk_free_bytes` | `client`, `name`, `folder` | Free space of a folder the download client uses. |
| `homelab_app_update_available` | `app`, `name`, `running`, `latest` | 1 if a newer release of the application is out than the one it runs, else 0. Only while Operations → App versions is switched on, and for an application whose newest release has been looked up. |
| `homelab_adguard_running`, `homelab_adguard_protection_enabled`, `homelab_adguard_dns_up` | `name` | 1 if AdGuard Home's DNS server runs, if its protection is on, and if it answered the app's test DNS question; else 0. |
| `homelab_adguard_dns_probe_seconds` | `name` | How long it took to answer that question; left out when it did not. |
| `homelab_adguard_queries`, `homelab_adguard_blocked`, `homelab_adguard_avg_processing_seconds` | `name` | The queries it answered in its statistics period, how many it blocked, and its average processing time. |
| `homelab_technitium_dns_up`, `homelab_technitium_blocking_enabled` | `name` | 1 if Technitium's DNS server answered the app's test DNS question, and if its blocking is on; else 0. |
| `homelab_technitium_dns_probe_seconds` | `name` | How long it took to answer that question; left out when it did not. |
| `homelab_technitium_uptime_seconds` | `name` | How long the DNS server has run, since it last started; left out when it did not say. |
| `homelab_technitium_block_lists_overdue` | `name` | 1 if the block lists' update was due more than six hours ago, else 0. Only where block lists are set up. |
| `homelab_technitium_queries`, `homelab_technitium_blocked`, `homelab_technitium_cached`, `homelab_technitium_recursive`, `homelab_technitium_server_failures`, `homelab_technitium_clients` | `name` | The last day's queries, how many were blocked, answered from the cache, resolved by the server and ended in a server failure, and how many different clients asked. Gauges over a rolling day, not counters. |
| `homelab_technitium_cache_entries` | `name` | Entries in its DNS cache now. |
| `homelab_technitium_zones`, `homelab_technitium_zone_problems` | `name` | The zones it holds (its internal ones not counted), and how many have expired or cannot sync. |
| `homelab_technitium_dhcp_leases` | `name` | The DHCP leases it holds. Left out when it has no DHCP server. |
| `homelab_grafana_rules`, `homelab_grafana_rules_firing`, `homelab_grafana_rules_error` | `name` | How many alert rules Grafana has, how many are firing and how many cannot be evaluated. |
| `homelab_grafana_datasources_failing` | `name` | How many of Grafana's data sources fail its health check. |
| `homelab_fragnet_server_running` | `account`, `server`, `game` | 1 if the Fragnet game server is running, else 0. |
| `homelab_fragnet_disk_used_bytes`, `homelab_fragnet_disk_limit_bytes` | `account`, `server`, `game` | The disk a Fragnet game server uses, and the disk its plan allows (left out when unlimited). |
| `homelab_fragnet_memory_used_bytes`, `homelab_fragnet_memory_limit_bytes` | `account`, `server`, `game` | The memory a Fragnet game server uses (0 while stopped), and its limit (left out when unlimited). |
| `homelab_fragnet_newest_backup_timestamp_seconds` | `account`, `server`, `game` | When the newest successful backup of a game server finished, in seconds since 1970; left out for one with no backup. |
| `homelab_netbox_workers`, `homelab_netbox_devices`, `homelab_netbox_ip_addresses` | `name` | The background workers running, and the devices and IP addresses NetBox holds; left out for a NetBox not read in the last twenty minutes. |
| `homelab_n8n_workflows`, `homelab_n8n_workflows_active`, `homelab_n8n_workflows_failing` | `name` | How many workflows n8n has, how many are switched on, and how many of those failed on their last run (absent when the runs could not be read). |
| `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes` | `name` | How many photos and videos Immich holds and what the library takes; left out for an Immich not read in the last twenty minutes. |
| `homelab_immich_disk_used_percent`, `homelab_immich_failed_jobs` | `name` | How full the disk under the library is, and how many background jobs have failed (all queues). |
| `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` | `name` | The documents Paperless-ngx holds, those waiting in its inbox, and the imports that failed in the last week and were not dismissed. |
| `homelab_paperless_component_ok` | `name`, `component` | 1 if that part of Paperless (database, redis, celery, index, classifier, sanity_check) is in order, else 0; absent when the system status could not be read. |
| `homelab_authentik_outpost_up` | `name`, `outpost` | 1 if the Authentik outpost has had an instance report in the last five minutes, else 0; left out for an Authentik not read in the last twenty minutes. |
| `homelab_authentik_workers`, `homelab_authentik_failed_logins_24h` | `name` | How many Authentik workers are connected, and how many sign-ins failed in the last day (100 means at least that). |
| `homelab_authentik_certificate_expiry_days` | `name`, `certificate` | Whole days until a certificate (one Authentik does not renew itself) runs out; negative once it has. |
| `homelab_jellyfin_streams`, `homelab_jellyfin_transcodes` | `name` | How many are streaming from Jellyfin right now, and how many of those it is transcoding; left out for a Jellyfin not read in the last ten minutes. |
| `homelab_jellyfin_pending_restart` | `name` | 1 if Jellyfin is waiting to be restarted, else 0. |
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
## Useful queries
```promql
# Servers that have gone quiet
homelab_server_up == 0
# The five fullest disks
topk(5, homelab_server_disk_used_percent)
# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full)
sort(homelab_server_disk_days_to_full < 30)
# Endpoints slower than half a second
homelab_endpoint_response_milliseconds > 500
# How many guests are not well backed up
sum(homelab_backup_guests{status=~"stale|failed|none"})
# Updates waiting across all servers, and how many are security fixes
sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates)
# Is the dev copy up as well as production? (several installations: filter on the label you added)
homelab_info
```
## Troubleshooting
| Symptom | Cause |
|---|---|
| Target is **DOWN**, `401 Unauthorized` | The token is wrong, was removed, or has a trailing newline in its file (use `printf '%s'`). The same `401` is given for no token and a bad one. |
| Target is **DOWN**, `429 Too many requests` | More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval. |
| Target is **DOWN**, certificate error | Prometheus does not trust the certificate. Give it the CA with `tls_config.ca_file` (or `insecure_skip_verify: true` on a trusted LAN, knowing what that gives up). |
| Target **UP** but some series are missing | A series that has nothing to say is **left out** rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected. |
| A dashboard panel is empty | The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade. |
| Numbers differ from the Alerts page | `homelab_alerts` counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute. |
| Names with odd characters look wrong | Label values are escaped (`\"`, `\\`, `\n`) as the Prometheus format requires; Prometheus and Grafana unescape them. |