Prometheus, Immich and Paperless-ngx integrations

Three read-only integrations, each with a monitoring page, alerts, maintenance
silencing and held-when-unreadable handling.

Prometheus: scrape targets that stop answering (5 minutes, one alert per job)
and alerts that Prometheus's own rules fire (critical when labelled so).
Immich: disk under the library filling, failed background jobs, counts.
Paperless-ngx: worker/database/Redis in error, documents that failed to
import, index and sanity check, disk. Immich and Paperless also in /metrics.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-24 22:33:17 +02:00
1 parent aebe98bf15
commit 251a4a9d73
47 files changed
+24148 -7

No files matched your search

+44
View File
@@ -13,6 +13,9 @@ This page explains what Homelab Manager watches, how often, how a problem become
- [Service levels](#service-levels)
- [Your own alert rules](#your-own-alert-rules)
- [Duplicati backups](#duplicati-backups)
- [Immich](#immich)
- [Paperless-ngx](#paperless-ngx)
- [Prometheus](#prometheus)
- [Authentik](#authentik)
- [Jellyfin](#jellyfin)
- [Docker containers that are broken](#docker-containers-that-are-broken)
@@ -131,6 +134,9 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Immich disk filling / jobs failing** | The disk under the library is 90 % full (critical at 95 %), or 10 or more background jobs have failed | on, with the integration | Immich page; [below](#immich) |
| **Paperless-ngx needs a look** | The worker, database or Redis is in error (critical), documents failed to import, the index or sanity check is in error, or the disk is 90 % full | on, with the integration | Paperless page; [below](#paperless-ngx) |
| **Prometheus target down / alert firing** | A scrape target has not answered for five minutes, or one of Prometheus's own alert rules is firing | on, with the integration | Prometheus page; [below](#prometheus) |
| **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) |
| **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) |
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
@@ -215,6 +221,44 @@ and one that no longer exists can be **forgotten** (with its history; if Duplica
What is kept: the last result of each job, and each report for 400 days or 400 reports. `/metrics` has `homelab_duplicati_backup_ok`, `homelab_duplicati_last_success_age_seconds` and `homelab_duplicati_source_size_bytes` for each job. A job is only learned from its first report, so a job that has
*never* reported is not known to be missing; and because the reports are pushed, **the app being down when a backup finishes** loses that report (the backup shows as overdue only once it is).
## Immich
An integration of type *Immich* ([INTEGRATIONS.md](../INTEGRATIONS.md#immich)) is read every five minutes, and **Monitoring → Immich** shows the photo and video counts, what the library takes, the disk under it and each background job queue with how many failed. Photos cannot be made again, and Immich fails quietly:
| Alert | When | Severity |
|---|---|---|
| **Disk under the library filling** | 90 % full | warning; critical from 95 % |
| **Background jobs failed** | 10 or more failed jobs, all queues together; the three worst queues are named | warning |
Failed jobs stay counted until they are run again or cleared in Immich (Administration → Jobs), so that alert stays until you do. A queue with a backlog of *waiting* jobs is not an alert — a big import has one for days. Maintenance on the integration holds them. A reading older than 20 minutes is not believed.
## Paperless-ngx
An integration of type *Paperless-ngx* ([INTEGRATIONS.md](../INTEGRATIONS.md#paperless-ngx)) is read every five minutes, and **Monitoring → Paperless** shows the documents and the inbox, the disk, Paperless's own system status and the imports that failed in the last week.
| Alert | When | Severity |
|---|---|---|
| **Database, Redis or the worker in error** | The system status says `ERROR` — a scan dropped in the folder just sits there when the worker is down | critical |
| **Index or classifier in error** | The system status says `ERROR` | warning |
| **Sanity check found problems** | The system status says `WARNING` for the sanity check | warning |
| **Documents failed to import** | One or more unacknowledged failures in the last week; the files and the latest error are named | warning |
| **Disk filling** | 90 % full | warning; critical from 95 % |
A classifier that has not been trained yet reports a *warning* in Paperless and is not shown as a problem. The system status needs a superuser's token; with another, the page says it is unknown and only the failed imports and the counts are judged. Maintenance on the integration holds the alerts.
## Prometheus
An integration of type *Prometheus* ([INTEGRATIONS.md](../INTEGRATIONS.md#prometheus)) is read every two minutes, and **Monitoring → Prometheus** shows its targets by job (how many up, how many down), the ones not answering and the alerts its rules have firing. It makes what you already measure
there part of the same list, with the same notifications, silencing and incidents as the rest.
| Alert | When | Severity |
|---|---|---|
| **Targets not answering** | One or more targets of a job have been down for five minutes. One alert per job, naming up to three instances and the first scrape error | warning |
| **Alert firing** | One of Prometheus's rules is firing. One alert per alert name ("DiskFull (3 firing)"), with the first summary | critical if any has `severity="critical"` (or `page`), else warning |
Pending alerts, `info` / `none` severities and the heartbeats every pipeline carries (`Watchdog`, `DeadMansSwitch`, `InfoInhibitor`) are not shown as problems. Maintenance on the integration holds them. A reading older than ten minutes is not believed. Alerts that
Grafana manages itself are not in Prometheus's list; to see one here, write it as a Prometheus rule instead. Nothing from Prometheus is exported back to `/metrics`.
## Authentik
An integration of type *Authentik* ([INTEGRATIONS.md](../INTEGRATIONS.md#authentik)) is read every five minutes, and **Monitoring → Authentik** shows its version, workers, each outpost (and whether it is reporting), the certificates and API tokens that run out — soonest first — and the failed sign-ins of the last day.
+4
View File
@@ -296,6 +296,10 @@ All gauges. The same table is checked by a test against what `/metrics` really s
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
| `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
| `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
| `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes` | `name` | How many photos and videos Immich holds and what the library takes; left out for an Immich not read in the last twenty minutes. |
| `homelab_immich_disk_used_percent`, `homelab_immich_failed_jobs` | `name` | How full the disk under the library is, and how many background jobs have failed (all queues). |
| `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` | `name` | The documents Paperless-ngx holds, those waiting in its inbox, and the imports that failed in the last week and were not dismissed. |
| `homelab_paperless_component_ok` | `name`, `component` | 1 if that part of Paperless (database, redis, celery, index, classifier, sanity_check) is in order, else 0; absent when the system status could not be read. |
| `homelab_authentik_outpost_up` | `name`, `outpost` | 1 if the Authentik outpost has had an instance report in the last five minutes, else 0; left out for an Authentik not read in the last twenty minutes. |
| `homelab_authentik_workers`, `homelab_authentik_failed_logins_24h` | `name` | How many Authentik workers are connected, and how many sign-ins failed in the last day (100 means at least that). |
| `homelab_authentik_certificate_expiry_days` | `name`, `certificate` | Whole days until a certificate (one Authentik does not renew itself) runs out; negative once it has. |