Prometheus, Immich and Paperless-ngx integrations
Three read-only integrations, each with a monitoring page, alerts, maintenance silencing and held-when-unreadable handling. Prometheus: scrape targets that stop answering (5 minutes, one alert per job) and alerts that Prometheus's own rules fire (critical when labelled so). Immich: disk under the library filling, failed background jobs, counts. Paperless-ngx: worker/database/Redis in error, documents that failed to import, index and sanity check, disk. Immich and Paperless also in /metrics. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
aebe98bf15
commit
251a4a9d73
47 files changed
+24148
-7
No files matched your search
@@ -335,6 +335,45 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
|
||||
- **Not verified against a live Jellyfin** — the endpoints (`/System/Info`, `/Sessions`, `/ScheduledTasks`, `/Items/Counts`, `/System/ActivityLog/Entries`) and their fields are as Jellyfin publishes them and the reader is tried against a pretend server. Every field is read defensively, and a part that cannot be read (older versions
|
||||
differ) is left out without failing the rest; the **Diagnostic Log** has what was asked and answered.
|
||||
|
||||
### Immich
|
||||
|
||||
- **Config fields:** the **address** of Immich (`http://host:2283`), an **API key** (Immich → Account Settings → API Keys; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*.
|
||||
- **Required access:** read only. Give the key just these permissions: `server.about`, `server.storage`, `server.statistics` and `job.read` (`queue.read` on newer versions) — statistics and jobs need an administrator's key. This app only ever sends GET requests; **nothing is changed, and no job is started, paused or cleared**.
|
||||
If one of those is refused, that part is left out and the rest still works.
|
||||
- **What it does:** every **five minutes** Immich is read and stored — its version (and whether a newer one is out), how full the **disk under the library** is, how many **photos and videos** there are and what they take, and the **background job queues** with how many jobs **failed** in each.
|
||||
**Monitoring → Immich** shows it.
|
||||
- **Alerts** (source *Immich*; held back by maintenance on the integration): **the disk under the library is 90 % full** (warning; **95 %**, critical) and **10 or more background jobs have failed**, all queues together (warning; the worst queues are named — thumbnails or metadata that were never made.
|
||||
Failed jobs stay counted until you run them again or clear them in Immich, so the alert stays until you do).
|
||||
- If Immich cannot be read for 20 minutes its alerts are neither raised nor cleared, the Alerts page says it could not check, and the integration is raised as failing like any other. A part that could not be asked is *unknown*, never *none*.
|
||||
- `/metrics` has `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes`, `homelab_immich_disk_used_percent` and `homelab_immich_failed_jobs` for each Immich that was read lately.
|
||||
- **Not verified against a live Immich** — the endpoints (`/api/server/about`, `/server/storage`, `/server/statistics`, `/server/version-check`, `/api/jobs`, and `/api/queues` on newer versions) and their fields are as Immich publishes them, and the reader is tried against a pretend server that answers both ways. Immich renames
|
||||
these now and then; the **Diagnostic Log** has what was asked and answered if something comes back empty. Whether a *waiting* backlog is a problem is not judged: a big import has one for days.
|
||||
|
||||
### Paperless-ngx
|
||||
|
||||
- **Config fields:** the **address** of Paperless-ngx (`http://host:8000`), an **API token** (your profile → *API Auth Token*, or Auth Tokens in the admin pages; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*.
|
||||
- **Required access:** read only, and a **superuser's token** is best: the system status (the worker, Redis, the index, the disk) is only given to a superuser. With an ordinary user's token the document counts and the failed imports are still read, and the page says the status is unknown. This app only ever sends GET
|
||||
requests; **nothing is changed, and no failed task is dismissed**.
|
||||
- **What it does:** every **five minutes** Paperless is read and stored — its version (and whether a newer one is out), the status of the **database, Redis, the worker that consumes documents, the search index, the classifier and the sanity check**, the **disk**, the **document and inbox counts** and the **imports that failed**
|
||||
in the last week and that nobody has dismissed. **Monitoring → Paperless** shows it.
|
||||
- **Alerts** (source *Paperless-ngx*; held back by maintenance on the integration): **the database, Redis or the worker in error** (critical — nothing can be consumed or found), **the index or the classifier in error** (warning), **the sanity check found problems** (warning), **documents failed to import** (warning; the files are named and the
|
||||
latest error shown — it clears when you dismiss them in Paperless, or after a week), and **the disk under the documents is 90 % full** (warning; **95 %**, critical). A classifier that simply has not been trained yet (Paperless's *warning*) is not an alert.
|
||||
- If Paperless cannot be read for 20 minutes its alerts are neither raised nor cleared, the Alerts page says it could not check, and the integration is raised as failing like any other. A part that could not be asked is *unknown*, never *none*.
|
||||
- `/metrics` has `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` and `homelab_paperless_component_ok` for each Paperless that was read lately.
|
||||
- **Not verified against a live Paperless** — the endpoints (`/api/status/`, `/api/statistics/`, `/api/tasks/`, `/api/remote_version/`) and their fields are as Paperless-ngx publishes them and the reader is tried against a pretend server. The **Diagnostic Log** has what was asked and answered.
|
||||
|
||||
### Prometheus
|
||||
|
||||
- **Config fields:** the **address** of Prometheus (`http://host:9090`); optionally a **bearer token**, if something in front of Prometheus asks for one (stored encrypted); and, for an https address with a certificate you made yourself, *insecure*.
|
||||
- **Required access:** read only, and nothing needs to be installed or configured in Prometheus. This app only ever sends GET requests: `/api/v1/status/buildinfo`, `/api/v1/targets` and `/api/v1/alerts`.
|
||||
- **What it does:** every **two minutes** Prometheus is read and stored — its version, every active **scrape target** with whether its last scrape worked (and the error), and the **alerts its own rules have firing**. **Monitoring → Prometheus** shows targets by job, the ones not answering and the alerts. So whatever you already
|
||||
measure there — a node exporter on each host, a service's own metrics — is part of the one list of what is wrong.
|
||||
- **Alerts** (category *Monitoring*, source *Prometheus*; held back by maintenance on the integration): **targets not answering** — one alert for each *job*, naming the instances and the scrape error, and only once a target has been down for **five minutes** (one failed scrape is not news; the time is kept from one read to the next),
|
||||
and **an alert that Prometheus's rules are firing** — one for each alert name, **critical** if any of them has the label `severity="critical"` (or `page`), a warning otherwise. Pending alerts, `info` ones and the by-design heartbeats (`Watchdog`, `DeadMansSwitch`, `InfoInhibitor`) say nothing.
|
||||
- If Prometheus cannot be read for ten minutes its alerts are neither raised nor cleared, the Alerts page says it could not check, and the integration is raised as failing like any other. A list Prometheus did not answer (targets, or alerts) is *unknown*, never *none*.
|
||||
- Nothing is sent back to `/metrics`: this app already exposes its own (see [PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md)), and exporting what Prometheus said into Prometheus would only go round in a circle.
|
||||
- **Not verified against a live Prometheus** — the endpoints and fields are as Prometheus documents them and the reader is tried against a pretend server. Alerts that **Grafana manages itself** (Grafana-managed alert rules) are not in Prometheus's list and are not seen; alerts that Prometheus's own rules send to Alertmanager are, while they are active.
|
||||
|
||||
### Authentik
|
||||
|
||||
- **Config fields:** the **address** of Authentik (`https://auth.example.lan`), an **API token** (Authentik → Directory → Tokens and App passwords → *Create*, intent **API Token**, for a user who may read; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*.
|
||||
|
||||
Reference in new issue
Block a user