Reads what Dockhand's scans have already stored (never starts one) every 15 minutes and raises a warning per Docker host with critical findings. Unknown counts and stale readings are held, not treated as clean. Settings -> Notifications has a switch (on by default); the Docker page shows counts, scan coverage and the critical findings; /metrics and docs updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
361 lines
24 KiB
Markdown
361 lines
24 KiB
Markdown
# Prometheus and Grafana
|
||
|
||
Homelab Manager can be scraped by **Prometheus** at `/metrics`, and ships a ready-made **Grafana dashboard** built on those series. This page sets up both, from nothing, and lists every series.
|
||
|
||
- [How it fits together](#how-it-fits-together)
|
||
- [1. Make a metrics token](#1-make-a-metrics-token)
|
||
- [2. Check it by hand](#2-check-it-by-hand)
|
||
- [3. Set up Prometheus](#3-set-up-prometheus)
|
||
- [4. Set up Grafana](#4-set-up-grafana)
|
||
- [A complete example with Docker Compose](#a-complete-example-with-docker-compose)
|
||
- [Alerting rules to start from](#alerting-rules-to-start-from)
|
||
- [Every series](#every-series)
|
||
- [Useful queries](#useful-queries)
|
||
- [Troubleshooting](#troubleshooting)
|
||
|
||
## How it fits together
|
||
|
||
```
|
||
agents, checks, integrations ──► Homelab Manager ──/metrics──► Prometheus ──► Grafana
|
||
(reads its own database; (stores the history,
|
||
calls nothing outside) draws the dashboard)
|
||
```
|
||
|
||
- `/metrics` is a **view of what the app already knows**. A scrape reads the database and the app's last alert result (reused for up to a minute); it **never calls out** — not to Proxmox, not to a server, not
|
||
to an integration. Scraping every 15 seconds costs a few database reads.
|
||
- The app keeps its own history (a year of server load and disks, endpoint answer times, uptime), so Prometheus is **not needed** to see trends in the app itself. Use Prometheus and Grafana when you
|
||
want these figures next to the rest of your monitoring, longer retention, or your own alert rules in Alertmanager.
|
||
- **Every series is a gauge.** Labels are escaped as Prometheus requires.
|
||
|
||
## 1. Make a metrics token
|
||
|
||
Administrators: **Settings → API and webhooks → Metrics for Prometheus → Make a token**.
|
||
|
||
- The token (it starts with `hlm_m_`) is **shown once**; only its SHA-256 is kept.
|
||
- Up to **ten** can exist — make one per Prometheus, so one can be removed without touching the others. A removed token **stops working at once**.
|
||
- Each shows when it was last used, so an unused one is easy to find.
|
||
- The token goes in the **`Authorization: Bearer …` header only**. A token in the address (`?token=…`) is **not accepted** (addresses end up in logs).
|
||
- No token, a wrong one and a removed one all get the same `401`. Requests are limited to **120 a minute** per address (`429` beyond).
|
||
- The answer is Prometheus's text format: `Content-Type: text/plain; version=0.0.4; charset=utf-8`.
|
||
|
||
A metrics token can read `/metrics` and nothing else. It is a different thing from the *personal API tokens* used for the [read API](API.md), and neither can sign in.
|
||
|
||
## 2. Check it by hand
|
||
|
||
```bash
|
||
curl -fsS -H "Authorization: Bearer hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx" https://hm.example.lan/metrics | head -30
|
||
```
|
||
|
||
You should see something like this (trimmed; the `# HELP` lines are left out here):
|
||
|
||
```
|
||
# TYPE homelab_info gauge
|
||
homelab_info{environment=""} 1
|
||
# TYPE homelab_server_up gauge
|
||
homelab_server_up{server="wiki-host"} 1
|
||
# TYPE homelab_server_last_seen_timestamp_seconds gauge
|
||
homelab_server_last_seen_timestamp_seconds{server="wiki-host"} 1792532670
|
||
# TYPE homelab_storage_days_to_full gauge
|
||
homelab_storage_days_to_full{pool="Lab PVE / pve1: storage \"tank\""} 21
|
||
# TYPE homelab_alerts gauge
|
||
homelab_alerts{severity="critical"} 1
|
||
homelab_alerts{severity="warning"} 3
|
||
homelab_alerts{severity="info"} 0
|
||
```
|
||
|
||
Without the header you get `401` — that is the right answer:
|
||
|
||
```bash
|
||
curl -i https://hm.example.lan/metrics # HTTP/1.1 401 Unauthorized, WWW-Authenticate: Bearer realm="metrics"
|
||
```
|
||
|
||
## 3. Set up Prometheus
|
||
|
||
Put the token in a file of its own, readable only by Prometheus, with **no trailing newline**:
|
||
|
||
```bash
|
||
printf '%s' 'hlm_m_xxxxxxxxxxxxxxxxxxxxxxxx' > /etc/prometheus/homelab-manager.token
|
||
chmod 600 /etc/prometheus/homelab-manager.token
|
||
```
|
||
|
||
`prometheus.yml`:
|
||
|
||
```yaml
|
||
global:
|
||
scrape_interval: 30s
|
||
|
||
scrape_configs:
|
||
- job_name: homelab-manager
|
||
scheme: https
|
||
metrics_path: /metrics
|
||
authorization:
|
||
type: Bearer
|
||
credentials_file: /etc/prometheus/homelab-manager.token
|
||
static_configs:
|
||
- targets: ["hm.example.lan"] # host[:port] of APP_BASE_URL, without https://
|
||
# If the certificate comes from your own CA, point at it (rather than skipping verification):
|
||
# tls_config:
|
||
# ca_file: /etc/prometheus/my-ca.pem
|
||
```
|
||
|
||
- **30–60 seconds is plenty.** The app's own data changes at the pace of the agents (every 15 minutes or so) and the checks (minutes).
|
||
- Prometheus accepts the token inline too (`credentials: hlm_m_…`), but a file keeps it out of the configuration you share.
|
||
- Behind a reverse proxy, use the address the proxy serves (`scheme: https`, port 443). Reaching the container directly is `scheme: http`, target `host:3000` — fine on a trusted network.
|
||
- Reload Prometheus (`kill -HUP`, or `POST /-/reload` if enabled) and open **Status → Targets**: `homelab-manager` should be **UP**.
|
||
- **Several installations** (production and a dev copy): the series `homelab_info{environment="…"}` carries the name set by `ENVIRONMENT_NAME`, so you can tell them apart; add an `instance`/`env` label in the scrape
|
||
config as well.
|
||
|
||
## 4. Set up Grafana
|
||
|
||
1. **Add Prometheus as a data source** — *Connections → Data sources → Add → Prometheus*, URL e.g. `http://prometheus:9090`, *Save & test*.
|
||
2. **Download the dashboard** — in Homelab Manager: **Settings → API and webhooks → Download the dashboard** (administrators). It is a file, `homelab-manager-dashboard.json`. (The same file is served to a signed-in
|
||
administrator at `GET /api/metrics-tokens/grafana-dashboard.json`.)
|
||
3. **Import it** — in Grafana: **Dashboards → New → Import**, upload the file, and **choose your Prometheus** when asked. The file never names a data source, so it is not tied to any installation.
|
||
|
||
The dashboard has a **Server** picker (filled from `homelab_server_up`) and five rows:
|
||
|
||
| Row | What it has |
|
||
|---|---|
|
||
| **Needs attention** | Critical alerts, warnings, acknowledged, open incidents, servers down and up, integrations failing, maintenance, stale and failed backups, failed services, updates waiting, restarts needed; alerts over time; each integration's state. |
|
||
| **Servers** | One table (up, CPU, memory, updates, security updates, failed services, agent behind), CPU and memory over time, 30-day uptime, time since each agent last reported. |
|
||
| **Disks** | Usage with a 90% line, disks filling up (days to full), SMART health, SSD life used, disk temperature, ZFS pools. |
|
||
| **Backups** | Guests by backup state, the age of each newest backup, and the age, last result and count of Homelab Manager's own backups. |
|
||
| **Secrets, power and the app itself** | Secrets and certificates by days left, the inventory's power draw, how long `/metrics` takes to build. |
|
||
|
||
The dashboard is **built by the app from the same code that serves `/metrics`**, and a test checks that it only asks for series that are really served. So **after an upgrade that adds series, download it again**
|
||
to get panels for them. It was written for Grafana 9 and later and has been checked for structure (every panel on the grid, every query well formed, every series real), **but it has not been opened in a Grafana
|
||
by the author** — if a panel looks wrong, the series table below says what each query should be.
|
||
|
||
## A complete example with Docker Compose
|
||
|
||
Prometheus and Grafana side by side, on the same host as Homelab Manager or elsewhere. Files in one folder:
|
||
|
||
```
|
||
monitoring/
|
||
├─ docker-compose.yml
|
||
├─ prometheus.yml (as above, with credentials_file: /etc/prometheus/homelab-manager.token)
|
||
├─ homelab-manager.token (the token only, no newline)
|
||
└─ rules/
|
||
└─ homelab-manager.yml (the alerting rules below, optional)
|
||
```
|
||
|
||
```yaml
|
||
# docker-compose.yml
|
||
services:
|
||
prometheus:
|
||
image: prom/prometheus:latest
|
||
restart: unless-stopped
|
||
command:
|
||
- --config.file=/etc/prometheus/prometheus.yml
|
||
- --storage.tsdb.retention.time=400d # a year and a bit: the dashboard's long ranges are then real
|
||
- --web.enable-lifecycle
|
||
volumes:
|
||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||
- ./homelab-manager.token:/etc/prometheus/homelab-manager.token:ro
|
||
- ./rules:/etc/prometheus/rules:ro
|
||
- prometheus-data:/prometheus
|
||
ports: ["9090:9090"]
|
||
|
||
grafana:
|
||
image: grafana/grafana:latest
|
||
restart: unless-stopped
|
||
volumes:
|
||
- grafana-data:/var/lib/grafana
|
||
ports: ["3001:3000"]
|
||
depends_on: [prometheus]
|
||
|
||
volumes:
|
||
prometheus-data:
|
||
grafana-data:
|
||
```
|
||
|
||
With the rules, add to `prometheus.yml`:
|
||
|
||
```yaml
|
||
rule_files:
|
||
- /etc/prometheus/rules/*.yml
|
||
```
|
||
|
||
In Grafana the data source URL is then `http://prometheus:9090` (the service name, since they share a compose network).
|
||
|
||
## Alerting rules to start from
|
||
|
||
Homelab Manager already sends its own notifications (see [MONITORING.md](MONITORING.md)). **Use these in addition only if you want the same things in Alertmanager**, otherwise you will be told twice.
|
||
They are the ones worth having *outside* the app — above all, the app itself going quiet.
|
||
|
||
```yaml
|
||
# rules/homelab-manager.yml
|
||
groups:
|
||
- name: homelab-manager
|
||
rules:
|
||
# The one thing the app cannot tell you about itself.
|
||
- alert: HomelabManagerDown
|
||
expr: up{job="homelab-manager"} == 0
|
||
for: 5m
|
||
labels: { severity: critical }
|
||
annotations: { summary: "Homelab Manager cannot be scraped" }
|
||
|
||
- alert: HomelabCriticalAlerts
|
||
expr: homelab_alerts{severity="critical"} > 0
|
||
for: 5m
|
||
labels: { severity: critical }
|
||
annotations: { summary: "{{ $value }} critical alert(s) in Homelab Manager" }
|
||
|
||
- alert: HomelabServerDown
|
||
expr: homelab_server_up == 0
|
||
for: 5m
|
||
labels: { severity: critical }
|
||
annotations: { summary: "{{ $labels.server }} has stopped reporting" }
|
||
|
||
- alert: HomelabDiskFilling
|
||
expr: homelab_server_disk_days_to_full < 7 or homelab_storage_days_to_full < 7
|
||
for: 1h
|
||
labels: { severity: warning }
|
||
annotations: { summary: "A disk or storage will be full within a week ({{ $labels.mount }}{{ $labels.pool }})" }
|
||
|
||
- alert: HomelabDiskUsageHigh
|
||
expr: homelab_server_disk_used_percent > 90
|
||
for: 15m
|
||
labels: { severity: warning }
|
||
annotations: { summary: "{{ $labels.server }}: {{ $labels.mount }} is {{ $value | printf \"%.0f\" }}% full" }
|
||
|
||
- alert: HomelabSelfBackupStale
|
||
expr: time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400
|
||
labels: { severity: warning }
|
||
annotations: { summary: "Homelab Manager's own backup is more than two days old" }
|
||
|
||
- alert: HomelabEndpointDown
|
||
expr: homelab_endpoint_up == 0
|
||
for: 5m
|
||
labels: { severity: warning }
|
||
annotations: { summary: "Endpoint check {{ $labels.name }} is failing" }
|
||
|
||
- alert: HomelabCertificateExpiring
|
||
expr: homelab_endpoint_certificate_days_left < 14 or homelab_secret_expiry_days < 14
|
||
labels: { severity: warning }
|
||
annotations: { summary: "A certificate or secret expires in under two weeks ({{ $labels.name }})" }
|
||
|
||
- alert: HomelabHeartbeatMissed
|
||
expr: homelab_heartbeat_up == 0
|
||
labels: { severity: warning }
|
||
annotations: { summary: "Job {{ $labels.name }} has not reported in time, or failed" }
|
||
|
||
- alert: HomelabIntegrationFailing
|
||
expr: homelab_integration_up == 0
|
||
for: 10m
|
||
labels: { severity: warning }
|
||
annotations: { summary: "Integration {{ $labels.integration }} ({{ $labels.type }}) keeps failing" }
|
||
```
|
||
|
||
A series that is **left out** is not zero: `homelab_integration_up`, `homelab_endpoint_up` and `homelab_selfbackup_last_attempt_ok` do not exist until something has been tried. That is why the rules above
|
||
compare against a condition rather than use `absent()`.
|
||
|
||
## Every series
|
||
|
||
All gauges. The same table is checked by a test against what `/metrics` really serves.
|
||
|
||
| Series | Labels | Meaning |
|
||
|---|---|---|
|
||
| `homelab_info` | `environment` | Always 1; says which installation this is. |
|
||
| `homelab_server_up` | `server` | 1 if the agent has reported within the offline threshold (Settings → Notifications, or the server's own limit if it has one), else 0. Servers that never reported are left out. |
|
||
| `homelab_server_last_seen_timestamp_seconds` | `server` | When the agent last reported. |
|
||
| `homelab_server_cpu_load_percent` | `server` | CPU load as the agent reports it. |
|
||
| `homelab_server_memory_used_percent` | `server` | Memory in use. |
|
||
| `homelab_server_disk_used_percent` | `server`, `mount` | How full each disk is. |
|
||
| `homelab_server_uptime_ratio_30d` | `server` | Share of the last 30 days the server was up, 0–1 (from the uptime record). |
|
||
| `homelab_server_pending_updates` | `server` | Updates waiting to be installed, when the agent reports them. |
|
||
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
|
||
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
|
||
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
|
||
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
|
||
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
|
||
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
|
||
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
|
||
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool (or Windows Storage Spaces pool) is ONLINE, 0 in any other state. |
|
||
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
|
||
| `homelab_storage_days_to_full` | `pool` | Days until a Proxmox storage or Synology volume is full at its present rate of filling; only for those with a steady climb. |
|
||
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
|
||
| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts`. |
|
||
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window or a snooze. |
|
||
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
|
||
| `homelab_incidents_open` | | Incidents that have begun and not yet ended. |
|
||
| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. |
|
||
| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. |
|
||
| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). |
|
||
| `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. |
|
||
| `homelab_maintenance_active_windows` | | Maintenance windows running now. |
|
||
| `homelab_secret_expiry_days` | `name` | Days until a secret expires (negative once it has). |
|
||
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
|
||
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
|
||
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
|
||
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
|
||
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
||
| `homelab_endpoint_value` | `name`, `kind` | The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
|
||
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
||
| `homelab_service_level_target_percent` | `name`, `subject`, `type` | The availability a service-level target asks for. |
|
||
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
|
||
| `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
|
||
| `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
|
||
| `homelab_nextcloud_cron_ok` | `name` | 1 if Nextcloud's own check says its background jobs ran within the last hour, else 0; absent when that could not be asked (no login, or before Nextcloud 28). |
|
||
| `homelab_nextcloud_maintenance_mode`, `homelab_nextcloud_app_updates` | `name` | 1 if Nextcloud is in maintenance mode, else 0; and how many apps have an update waiting (left out when not asked). Left out for a Nextcloud not read in the last twenty minutes. |
|
||
| `homelab_nextcloud_free_bytes`, `homelab_nextcloud_active_users_24h` | `name` | Bytes free where the data is kept, and the users active in the last day (both need an administrator's app password). |
|
||
| `homelab_docker_vulnerabilities` | `integration`, `host`, `severity` | The vulnerabilities in the scanned images of a Docker host, by severity (`critical`, `high`, `medium`, `low`), from Dockhand's stored scans. Left out for a host whose count could not be read, and when the reading is older than 45 minutes. |
|
||
| `homelab_docker_images_scanned` | `integration`, `host` | How many of a Docker host's images have a scan in Dockhand (the rest are not known to be clean). |
|
||
| `homelab_netbox_workers`, `homelab_netbox_devices`, `homelab_netbox_ip_addresses` | `name` | The background workers running, and the devices and IP addresses NetBox holds; left out for a NetBox not read in the last twenty minutes. |
|
||
| `homelab_n8n_workflows`, `homelab_n8n_workflows_active`, `homelab_n8n_workflows_failing` | `name` | How many workflows n8n has, how many are switched on, and how many of those failed on their last run (absent when the runs could not be read). |
|
||
| `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes` | `name` | How many photos and videos Immich holds and what the library takes; left out for an Immich not read in the last twenty minutes. |
|
||
| `homelab_immich_disk_used_percent`, `homelab_immich_failed_jobs` | `name` | How full the disk under the library is, and how many background jobs have failed (all queues). |
|
||
| `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` | `name` | The documents Paperless-ngx holds, those waiting in its inbox, and the imports that failed in the last week and were not dismissed. |
|
||
| `homelab_paperless_component_ok` | `name`, `component` | 1 if that part of Paperless (database, redis, celery, index, classifier, sanity_check) is in order, else 0; absent when the system status could not be read. |
|
||
| `homelab_authentik_outpost_up` | `name`, `outpost` | 1 if the Authentik outpost has had an instance report in the last five minutes, else 0; left out for an Authentik not read in the last twenty minutes. |
|
||
| `homelab_authentik_workers`, `homelab_authentik_failed_logins_24h` | `name` | How many Authentik workers are connected, and how many sign-ins failed in the last day (100 means at least that). |
|
||
| `homelab_authentik_certificate_expiry_days` | `name`, `certificate` | Whole days until a certificate (one Authentik does not renew itself) runs out; negative once it has. |
|
||
| `homelab_jellyfin_streams`, `homelab_jellyfin_transcodes` | `name` | How many are streaming from Jellyfin right now, and how many of those it is transcoding; left out for a Jellyfin not read in the last ten minutes. |
|
||
| `homelab_jellyfin_pending_restart` | `name` | 1 if Jellyfin is waiting to be restarted, else 0. |
|
||
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
|
||
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
|
||
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
||
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
|
||
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
|
||
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
|
||
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
|
||
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
|
||
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
|
||
|
||
## Useful queries
|
||
|
||
```promql
|
||
# Servers that have gone quiet
|
||
homelab_server_up == 0
|
||
|
||
# The five fullest disks
|
||
topk(5, homelab_server_disk_used_percent)
|
||
|
||
# Disks that will be full within a month, soonest first (and the same for storage: homelab_storage_days_to_full)
|
||
sort(homelab_server_disk_days_to_full < 30)
|
||
|
||
# Endpoints slower than half a second
|
||
homelab_endpoint_response_milliseconds > 500
|
||
|
||
# How many guests are not well backed up
|
||
sum(homelab_backup_guests{status=~"stale|failed|none"})
|
||
|
||
# Updates waiting across all servers, and how many are security fixes
|
||
sum(homelab_server_pending_updates), sum(homelab_server_pending_security_updates)
|
||
|
||
# Is the dev copy up as well as production? (several installations: filter on the label you added)
|
||
homelab_info
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause |
|
||
|---|---|
|
||
| Target is **DOWN**, `401 Unauthorized` | The token is wrong, was removed, or has a trailing newline in its file (use `printf '%s'`). The same `401` is given for no token and a bad one. |
|
||
| Target is **DOWN**, `429 Too many requests` | More than 120 requests a minute from one address — a second Prometheus, or a very short scrape interval. |
|
||
| Target is **DOWN**, certificate error | Prometheus does not trust the certificate. Give it the CA with `tls_config.ca_file` (or `insecure_skip_verify: true` on a trusted LAN, knowing what that gives up). |
|
||
| Target **UP** but some series are missing | A series that has nothing to say is **left out** rather than reported as 0 — an integration nobody has called yet, a check never asked, a server whose agent reports no SMART data. That is expected. |
|
||
| A dashboard panel is empty | The series it asks for is left out (as above), or the app is older than the dashboard file. Download the dashboard again after an upgrade. |
|
||
| Numbers differ from the Alerts page | `homelab_alerts` counts active alerts and leaves out silenced ones; the page shows both. The app's own result is reused for up to a minute. |
|
||
| Names with odd characters look wrong | Label values are escaped (`\"`, `\\`, `\n`) as the Prometheus format requires; Prometheus and Grafana unescape them. |
|