Nextcloud, NetBox and n8n integrations

Three read-only integrations, each with a monitoring page, alerts, maintenance
silencing and held-when-unreadable handling.

Nextcloud: database needing upgrade (critical), maintenance mode, and the data
disk running low; users, files and app updates with an admin app password.
NetBox: no background worker running, counts, and a list of the servers this
app manages that have no address in NetBox (a list, not an alert).
n8n: an active workflow whose last run failed, clearing when it next succeeds.

Disk messages for Immich, Paperless and Nextcloud now use the same units as
the pages that show them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-24 23:13:13 +02:00
1 parent a16996d975
commit 4ff0b44943
46 files changed
+10137 -10

No files matched your search

+42 -2
View File
@@ -23,8 +23,8 @@ what the app actually creates.
|---|---| |---|---|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. | | **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. | | **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0062`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | | **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0063`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 75 application tables, plus drizzle's own `__drizzle_migrations`. | | **Tables** | 78 application tables, plus drizzle's own `__drizzle_migrations`. |
**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`. **Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`.
@@ -72,6 +72,9 @@ erDiagram
integrations ||--o| authentik_state : "cascade" integrations ||--o| authentik_state : "cascade"
integrations ||--o| prometheus_state : "cascade" integrations ||--o| prometheus_state : "cascade"
integrations ||--o| immich_state : "cascade" integrations ||--o| immich_state : "cascade"
integrations ||--o| nextcloud_state : "cascade"
integrations ||--o| netbox_state : "cascade"
integrations ||--o| n8n_state : "cascade"
integrations ||--o| paperless_state : "cascade" integrations ||--o| paperless_state : "cascade"
integrations ||--o{ jellyfin_readings : "cascade" integrations ||--o{ jellyfin_readings : "cascade"
users ||--o{ todos : "assignee (set null)" users ||--o{ todos : "assignee (set null)"
@@ -868,6 +871,43 @@ A UPS at a moment, for the charts: one every five minutes, and every minute whil
| `runtime_seconds` | integer | | | `runtime_seconds` | integer | |
| `on_battery` | bool NOT NULL, default 0 | | | `on_battery` | bool NOT NULL, default 0 | |
#### `nextcloud_state`
How a Nextcloud (an integration of type `nextcloud`) is right now, one row each, replaced at every read (every five minutes). `integration_id` is the primary key and a foreign key with **cascade**. There is no history table.
| Column | Type | Notes |
|---|---|---|
| `integration_id` | integer PK | |
| `version` | text | |
| `maintenance`, `needs_upgrade` | bool NOT NULL, default 0 | From status.php. |
| `detail` | bool NOT NULL, default 0 | Whether the monitoring app answered (an administrator's app password is needed). |
| `free_bytes`, `users`, `files`, `active_24h`, `app_updates` | integer | Null when the monitoring app was not asked or did not say. |
| `updated_at` | text NOT NULL | ISO. |
#### `netbox_state`
How a NetBox (an integration of type `netbox`) is right now, one row each, replaced at every read (every five minutes). `integration_id` is the primary key and a foreign key with **cascade**.
| Column | Type | Notes |
|---|---|---|
| `integration_id` | integer PK | |
| `version` | text | |
| `workers_running` | integer | How many RQ workers; **null** when /status/ did not say (which is not the same as 0). |
| `devices`, `virtual_machines`, `prefixes`, `ip_addresses` | integer | Counts; null when not read. |
| `ips` | text | JSON array of the addresses NetBox holds, without the prefix length, at most 5000; null when they could not be listed. |
| `updated_at` | text NOT NULL | ISO. |
#### `n8n_state`
How an n8n (an integration of type `n8n`) is right now, one row each, replaced at every read (every five minutes). `integration_id` is the primary key and a foreign key with **cascade**.
| Column | Type | Notes |
|---|---|---|
| `integration_id` | integer PK | |
| `workflows` | text NOT NULL, default `[]` | JSON: id, name, active, lastStatus, lastAt, failed24h — the last outcome and the failures of the last day are worked out from the newest 250 runs. |
| `executions_known` | bool NOT NULL, default 0 | False when the runs could not be asked, so no last outcome is known. |
| `updated_at` | text NOT NULL | ISO. |
#### `immich_state` #### `immich_state`
How an Immich (an integration of type `immich`) is right now, one row each, replaced at every read (every five minutes). `integration_id` is the primary key and a foreign key with **cascade**. There is no history table. How an Immich (an integration of type `immich`) is right now, one row each, replaced at every read (every five minutes). `integration_id` is the primary key and a foreign key with **cascade**. There is no history table.
+37
View File
@@ -349,6 +349,43 @@ That works on every Duplicati version, needs no login handshake and leaves nothi
- **Not verified against a live Immich** — the endpoints (`/api/server/about`, `/server/storage`, `/server/statistics`, `/server/version-check`, `/api/jobs`, and `/api/queues` on newer versions) and their fields are as Immich publishes them, and the reader is tried against a pretend server that answers both ways. Immich renames - **Not verified against a live Immich** — the endpoints (`/api/server/about`, `/server/storage`, `/server/statistics`, `/server/version-check`, `/api/jobs`, and `/api/queues` on newer versions) and their fields are as Immich publishes them, and the reader is tried against a pretend server that answers both ways. Immich renames
these now and then; the **Diagnostic Log** has what was asked and answered if something comes back empty. Whether a *waiting* backlog is a problem is not judged: a big import has one for days. these now and then; the **Diagnostic Log** has what was asked and answered if something comes back empty. Whether a *waiting* backlog is a problem is not judged: a big import has one for days.
### Nextcloud
- **Config fields:** the **address** of Nextcloud (`https://cloud.example.lan`); optionally an **administrator's name and an app password** (Personal settings → Security → Devices & sessions → *Create new app password*; stored encrypted — never the account's own password); and, for an https address with a certificate you made
yourself, *insecure*.
- **Required access:** read only. Without a login only `/status.php` is read, which Nextcloud answers to anyone. With one, the *Monitoring* app (`serverinfo`, installed by default) is also asked, which needs an **administrator**; if it is refused or the app is off, the status is still read and the page says so. This
app only ever sends GET requests, and **nothing is changed**.
- **What it does:** every **five minutes** Nextcloud is read and stored — its version, whether it is in **maintenance mode**, whether it **needs its database upgrading**, and (with the login) the **room left where the data is kept**, the users, the files, who was active in the last day and how many **app updates** wait. **Monitoring → Nextcloud** shows it.
- **Alerts** (source *Nextcloud*; held back by maintenance on the integration): **the database needs upgrading** (critical — nobody can use it until `occ upgrade`), **maintenance mode is on** (warning — nobody can sign in), and **less than 20 GB is left for the data** (warning; under **5 GB**, critical).
- If Nextcloud cannot be read for 20 minutes its alerts are neither raised nor cleared, the Alerts page says it could not check, and the integration is raised as failing like any other.
- `/metrics` has `homelab_nextcloud_maintenance_mode`, `homelab_nextcloud_free_bytes`, `homelab_nextcloud_active_users_24h` and `homelab_nextcloud_app_updates` for each Nextcloud that was read lately.
- **Not verified against a live Nextcloud** — `/status.php` and the Monitoring app's `/ocs/v2.php/apps/serverinfo/api/v1/info` are as Nextcloud publishes them and the reader is tried against a pretend server. **Whether background jobs (cron) are running is not read**: the API used here does not say, so a cron that has
stopped is not noticed. With **Nextcloud AIO** the admin pages are the mastercontainer's, but the Nextcloud behind it is an ordinary one and is read the same way (use its own address, not the AIO admin address).
### NetBox
- **Config fields:** the **address** of NetBox (`https://netbox.example.lan`), an **API token** (your profile → API Tokens; stored encrypted; **write enabled off** is enough — both the older `Token` tokens and NetBox 4.5's `nbt_…` tokens work) and, for an https address with a certificate you made yourself, *insecure*.
- **Required access:** read only. `/api/status/`, and the counts and addresses of devices, virtual machines, prefixes and IP addresses. This app only ever sends GET requests; **nothing is changed**.
- **What it does:** every **five minutes** NetBox is read and stored — its version, **how many background workers are running** (they run its webhooks, scripts, jobs and reports; in the Docker image they are a separate container that can be down while the pages still load), how many devices, VMs, prefixes and addresses it holds, and the addresses
themselves (up to 5000). **Monitoring → NetBox** shows it.
- **Alert** (source *NetBox*; held back by maintenance on the integration): **no background worker is running** (warning).
- **Not in NetBox**: the page lists the servers this app manages **none of whose addresses are in NetBox** (an address that is the same on every host, like Docker's bridge, is left out as on the Consistency page; loopback and link-local addresses say nothing). That is a **list, not an alert** — documentation that is behind is not an outage. If NetBox
holds more than 5000 addresses the page says what is missing may be in the part that was not read.
- If NetBox cannot be read for 20 minutes its alert is neither raised nor cleared, and the Alerts page says it could not check.
- `/metrics` has `homelab_netbox_workers`, `homelab_netbox_devices` and `homelab_netbox_ip_addresses` for each NetBox that was read lately.
- **Not verified against a live NetBox** — the endpoints and fields are as NetBox publishes them and the reader is tried against a pretend server that answers the addresses in pages. The **Diagnostic Log** has what was asked and answered.
### n8n
- **Config fields:** the **address** of n8n (`http://host:5678`), an **API key** (Settings → n8n API → *Create an API key*; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*.
- **Required access:** read only. `GET /api/v1/workflows` and `GET /api/v1/executions` — the public API has to be switched on (it is on by default in current versions). This app only ever sends GET requests: **no workflow is started, changed or switched off**.
- **What it does:** every **five minutes** n8n is read and stored — the workflows and whether each is switched on, and, from the **newest 250 runs**, each workflow's **last outcome** and how many of its runs failed in the last day. **Monitoring → n8n** shows it.
- **Alert** (source *n8n*, category *Automation*; held back by maintenance on the integration): **a workflow that is switched on failed on its last run** (warning; one alert for each workflow, saying when and, if more than one run failed, how many in the last day). It **clears by itself** when the workflow next succeeds. A workflow that is switched off, or
whose failure is more than a week old, is not raised.
- If n8n cannot be read for 20 minutes its alerts are neither raised nor cleared, and the Alerts page says it could not check. If the runs could not be read (but the workflows could), the last outcomes are *unknown* and nothing is claimed.
- `/metrics` has `homelab_n8n_workflows`, `homelab_n8n_workflows_active` and `homelab_n8n_workflows_failing` for each n8n that was read lately.
- **Not verified against a live n8n** — the endpoints and fields are as n8n documents them and the reader is tried against a pretend server that pages with a cursor. A workflow that runs less often than 250 other runs go by may show no run (and so no alert); the page says it is judged from the newest 250.
### Paperless-ngx ### Paperless-ngx
- **Config fields:** the **address** of Paperless-ngx (`http://host:8000`), an **API token** (your profile → *API Auth Token*, or Auth Tokens in the admin pages; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*. - **Config fields:** the **address** of Paperless-ngx (`http://host:8000`), an **API token** (your profile → *API Auth Token*, or Auth Tokens in the admin pages; stored encrypted) and, for an https address with a certificate you made yourself, *insecure*.
+9
View File
@@ -526,6 +526,15 @@ meant to be kept and how far back it actually reaches. A history that holds far
address once, and paste three options into Duplicati's default options; no password for Duplicati is kept and the app never calls it. Every job then appears with when it last succeeded, how the last run went, how long it took, how big the source is and what changed, with a history; a failed run is a warning (critical from the address once, and paste three options into Duplicati's default options; no password for Duplicati is kept and the app never calls it. Every job then appears with when it last succeeded, how the last run went, how long it took, how big the source is and what changed, with a history; a failed run is a warning (critical from the
second in a row), an overdue one (2 days by default, per job) a warning (critical at three times that), and maintenance on the machine's server silences them. See INTEGRATIONS.md and docs/MONITORING.md. second in a row), an overdue one (2 days by default, per job) a warning (critical at three times that), and maintenance on the machine's server silences them. See INTEGRATIONS.md and docs/MONITORING.md.
**Nextcloud** (Monitoring → Nextcloud). Add a Nextcloud integration (its address; an administrator's app password adds more) and the app tells you when people **cannot use it**: **critical** when its database needs upgrading, a warning when it is stuck in **maintenance mode**, and a warning (critical under 5 GB) when the **data disk is low**.
The page also shows the users, the files and the app updates waiting. Whether its cron runs is not read. See INTEGRATIONS.md and docs/MONITORING.md.
**NetBox** (Monitoring → NetBox). Add a NetBox integration (its address and a read-only token): a warning when **no background worker** is running (they are a separate container in the Docker image, and can be down while the pages load), and a list on the page of **the servers you run that have no address in NetBox**. That list is
not an alert; documentation that is behind is not an outage. See INTEGRATIONS.md and docs/MONITORING.md.
**n8n** (Monitoring → n8n). Add an n8n integration (its address and an API key): a warning for each **workflow that is switched on and failed on its last run** — an automation that has quietly stopped — which clears when it next succeeds. The page lists the workflows with their last run and the failures of the last day. See
INTEGRATIONS.md and docs/MONITORING.md.
**Immich** (Monitoring → Immich). Add an Immich integration (its address and a read-only API key) and the app watches the thing you cannot recreate: it warns when the **disk under the photo library** is filling (critical at 95 %) and when **background jobs have failed** — the thumbnails and metadata that Immich **Immich** (Monitoring → Immich). Add an Immich integration (its address and a read-only API key) and the app watches the thing you cannot recreate: it warns when the **disk under the photo library** is filling (critical at 95 %) and when **background jobs have failed** — the thumbnails and metadata that Immich
quietly never made. The page shows the photo and video counts, what the library takes and each job queue. See INTEGRATIONS.md and docs/MONITORING.md. quietly never made. The page shows the photo and video counts, what the library takes and each job queue. See INTEGRATIONS.md and docs/MONITORING.md.
+4
View File
@@ -50,6 +50,9 @@ page's own top-level link.
| Authentik | `/authentik` | ✅ | ✅ | ✅ | | Authentik | `/authentik` | ✅ | ✅ | ✅ |
| Prometheus | `/prometheus` | ✅ | ✅ | ✅ | | Prometheus | `/prometheus` | ✅ | ✅ | ✅ |
| Immich | `/immich` | ✅ | ✅ | ✅ | | Immich | `/immich` | ✅ | ✅ | ✅ |
| Nextcloud | `/nextcloud` | ✅ | ✅ | ✅ |
| NetBox | `/netbox` | ✅ | ✅ | ✅ |
| n8n | `/n8n` | ✅ | ✅ | ✅ |
| Paperless-ngx | `/paperless` | ✅ | ✅ | ✅ | | Paperless-ngx | `/paperless` | ✅ | ✅ | ✅ |
| **Automation** | | | | | | **Automation** | | | | |
| Semaphore | `/semaphore` | ✅ | ✅ | ✅ | | Semaphore | `/semaphore` | ✅ | ✅ | ✅ |
@@ -137,6 +140,7 @@ even further, to admin only:
- **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map. - **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map.
- **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone; - **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone;
*Check now* needs operator. *Check now* needs operator.
- **Nextcloud, NetBox and n8n** (Monitoring → each): everyone signed in sees the status, the counts and what is wrong — for NetBox, the names and addresses of the servers that are not in it; for n8n, the names of the workflows; adding one (its address and a login, token or key) is an administrator's job, like any integration.
- **Immich** (Monitoring → Immich): everyone signed in sees the counts, the disk and the job queues; adding one (its address and API key) is an administrator's job, like any integration. - **Immich** (Monitoring → Immich): everyone signed in sees the counts, the disk and the job queues; adding one (its address and API key) is an administrator's job, like any integration.
- **Paperless-ngx** (Monitoring → Paperless): everyone signed in sees the counts, the system status and the names of the files that failed to import; adding one (its address and a superuser's token) is an administrator's job, like any integration. - **Paperless-ngx** (Monitoring → Paperless): everyone signed in sees the counts, the system status and the names of the files that failed to import; adding one (its address and a superuser's token) is an administrator's job, like any integration.
- **Prometheus** (Monitoring → Prometheus): everyone signed in sees the targets, what is down and the alerts firing; adding one (its address, and a token if it needs one) is an administrator's job, like any integration. - **Prometheus** (Monitoring → Prometheus): everyone signed in sees the targets, what is down and the alerts firing; adding one (its address, and a token if it needs one) is an administrator's job, like any integration.
+38
View File
@@ -13,6 +13,9 @@ This page explains what Homelab Manager watches, how often, how a problem become
- [Service levels](#service-levels) - [Service levels](#service-levels)
- [Your own alert rules](#your-own-alert-rules) - [Your own alert rules](#your-own-alert-rules)
- [Duplicati backups](#duplicati-backups) - [Duplicati backups](#duplicati-backups)
- [Nextcloud](#nextcloud)
- [NetBox](#netbox)
- [n8n](#n8n)
- [Immich](#immich) - [Immich](#immich)
- [Paperless-ngx](#paperless-ngx) - [Paperless-ngx](#paperless-ngx)
- [Prometheus](#prometheus) - [Prometheus](#prometheus)
@@ -134,6 +137,9 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications | | **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications | | **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications | | **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) |
| **NetBox has no worker** | No background worker is running | on, with the integration | NetBox page; [below](#netbox) |
| **n8n workflow failed** | A workflow that is switched on failed on its last run | on, with the integration | n8n page; [below](#n8n) |
| **Immich disk filling / jobs failing** | The disk under the library is 90 % full (critical at 95 %), or 10 or more background jobs have failed | on, with the integration | Immich page; [below](#immich) | | **Immich disk filling / jobs failing** | The disk under the library is 90 % full (critical at 95 %), or 10 or more background jobs have failed | on, with the integration | Immich page; [below](#immich) |
| **Paperless-ngx needs a look** | The worker, database or Redis is in error (critical), documents failed to import, the index or sanity check is in error, or the disk is 90 % full | on, with the integration | Paperless page; [below](#paperless-ngx) | | **Paperless-ngx needs a look** | The worker, database or Redis is in error (critical), documents failed to import, the index or sanity check is in error, or the disk is 90 % full | on, with the integration | Paperless page; [below](#paperless-ngx) |
| **Prometheus target down / alert firing** | A scrape target has not answered for five minutes, or one of Prometheus's own alert rules is firing | on, with the integration | Prometheus page; [below](#prometheus) | | **Prometheus target down / alert firing** | A scrape target has not answered for five minutes, or one of Prometheus's own alert rules is firing | on, with the integration | Prometheus page; [below](#prometheus) |
@@ -225,6 +231,38 @@ and one that no longer exists can be **forgotten** (with its history; if Duplica
What is kept: the last result of each job, and each report for 400 days or 400 reports. `/metrics` has `homelab_duplicati_backup_ok`, `homelab_duplicati_last_success_age_seconds` and `homelab_duplicati_source_size_bytes` for each job. A job is only learned from its first report, so a job that has What is kept: the last result of each job, and each report for 400 days or 400 reports. `/metrics` has `homelab_duplicati_backup_ok`, `homelab_duplicati_last_success_age_seconds` and `homelab_duplicati_source_size_bytes` for each job. A job is only learned from its first report, so a job that has
*never* reported is not known to be missing; and because the reports are pushed, **the app being down when a backup finishes** loses that report (the backup shows as overdue only once it is). *never* reported is not known to be missing; and because the reports are pushed, **the app being down when a backup finishes** loses that report (the backup shows as overdue only once it is).
## Nextcloud
An integration of type *Nextcloud* ([INTEGRATIONS.md](../INTEGRATIONS.md#nextcloud)) is read every five minutes, and **Monitoring → Nextcloud** shows its version, whether it is usable, and — with an administrator's app password — the room left for the data, the users, the files and the app updates waiting.
| Alert | When | Severity |
|---|---|---|
| **Database needs upgrading** | status.php says `needsDbUpgrade` — nobody can use Nextcloud until `occ upgrade` is run | critical |
| **Maintenance mode on** | status.php says `maintenance` — nobody can sign in | warning |
| **Data disk low** | Less than 20 GB free where the data is kept (needs the login) | warning; critical under 5 GB |
Maintenance on the integration holds them. A reading older than 20 minutes is not believed. Whether Nextcloud's **background jobs** run is not read (the API does not say), so a stopped cron goes unnoticed.
## NetBox
An integration of type *NetBox* ([INTEGRATIONS.md](../INTEGRATIONS.md#netbox)) is read every five minutes, and **Monitoring → NetBox** shows its workers, what it holds and — the other half — **which of your servers are not in it**.
| Alert | When | Severity |
|---|---|---|
| **No background worker** | `/api/status/` says no RQ worker is running — webhooks, scripts, jobs and reports are not run, although the pages load | warning |
The list of servers with no address in NetBox is not an alert: it is the app's servers (from what their agents report) against NetBox's addresses, leaving out Docker bridge addresses and the other ranges excluded on the Consistency page. A server counts as missing only if it has addresses and none is in NetBox.
## n8n
An integration of type *n8n* ([INTEGRATIONS.md](../INTEGRATIONS.md#n8n)) is read every five minutes, and **Monitoring → n8n** lists the workflows with their last run and the failures of the last day.
| Alert | When | Severity |
|---|---|---|
| **Workflow failed** | A workflow that is switched on failed (or crashed) on its last run, in the last week. One alert per workflow; it clears when the workflow next succeeds | warning |
A workflow that is switched off says nothing, and what each workflow last did is judged from n8n's **newest 250 runs**. If n8n gives its workflows but not its runs, the last outcomes are unknown and nothing is claimed. Maintenance on the integration holds the alerts.
## Immich ## Immich
An integration of type *Immich* ([INTEGRATIONS.md](../INTEGRATIONS.md#immich)) is read every five minutes, and **Monitoring → Immich** shows the photo and video counts, what the library takes, the disk under it and each background job queue with how many failed. Photos cannot be made again, and Immich fails quietly: An integration of type *Immich* ([INTEGRATIONS.md](../INTEGRATIONS.md#immich)) is read every five minutes, and **Monitoring → Immich** shows the photo and video counts, what the library takes, the disk under it and each background job queue with how many failed. Photos cannot be made again, and Immich fails quietly:
+4
View File
@@ -296,6 +296,10 @@ All gauges. The same table is checked by a test against what `/metrics` really s
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. | | `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
| `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. | | `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
| `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. | | `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
| `homelab_nextcloud_maintenance_mode`, `homelab_nextcloud_app_updates` | `name` | 1 if Nextcloud is in maintenance mode, else 0; and how many apps have an update waiting (left out when not asked). Left out for a Nextcloud not read in the last twenty minutes. |
| `homelab_nextcloud_free_bytes`, `homelab_nextcloud_active_users_24h` | `name` | Bytes free where the data is kept, and the users active in the last day (both need an administrator's app password). |
| `homelab_netbox_workers`, `homelab_netbox_devices`, `homelab_netbox_ip_addresses` | `name` | The background workers running, and the devices and IP addresses NetBox holds; left out for a NetBox not read in the last twenty minutes. |
| `homelab_n8n_workflows`, `homelab_n8n_workflows_active`, `homelab_n8n_workflows_failing` | `name` | How many workflows n8n has, how many are switched on, and how many of those failed on their last run (absent when the runs could not be read). |
| `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes` | `name` | How many photos and videos Immich holds and what the library takes; left out for an Immich not read in the last twenty minutes. | | `homelab_immich_photos`, `homelab_immich_videos`, `homelab_immich_library_bytes` | `name` | How many photos and videos Immich holds and what the library takes; left out for an Immich not read in the last twenty minutes. |
| `homelab_immich_disk_used_percent`, `homelab_immich_failed_jobs` | `name` | How full the disk under the library is, and how many background jobs have failed (all queues). | | `homelab_immich_disk_used_percent`, `homelab_immich_failed_jobs` | `name` | How full the disk under the library is, and how many background jobs have failed (all queues). |
| `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` | `name` | The documents Paperless-ngx holds, those waiting in its inbox, and the imports that failed in the last week and were not dismissed. | | `homelab_paperless_documents`, `homelab_paperless_inbox`, `homelab_paperless_failed_imports` | `name` | The documents Paperless-ngx holds, those waiting in its inbox, and the imports that failed in the last week and were not dismissed. |
+35
View File
@@ -0,0 +1,35 @@
CREATE TABLE `n8n_state` (
`integration_id` integer PRIMARY KEY NOT NULL,
`workflows` text DEFAULT '[]' NOT NULL,
`executions_known` integer DEFAULT false NOT NULL,
`updated_at` text NOT NULL,
FOREIGN KEY (`integration_id`) REFERENCES `integrations`(`id`) ON UPDATE no action ON DELETE cascade
);
--> statement-breakpoint
CREATE TABLE `netbox_state` (
`integration_id` integer PRIMARY KEY NOT NULL,
`version` text,
`workers_running` integer,
`devices` integer,
`virtual_machines` integer,
`prefixes` integer,
`ip_addresses` integer,
`ips` text,
`updated_at` text NOT NULL,
FOREIGN KEY (`integration_id`) REFERENCES `integrations`(`id`) ON UPDATE no action ON DELETE cascade
);
--> statement-breakpoint
CREATE TABLE `nextcloud_state` (
`integration_id` integer PRIMARY KEY NOT NULL,
`version` text,
`maintenance` integer DEFAULT false NOT NULL,
`needs_upgrade` integer DEFAULT false NOT NULL,
`detail` integer DEFAULT false NOT NULL,
`free_bytes` integer,
`users` integer,
`files` integer,
`active_24h` integer,
`app_updates` integer,
`updated_at` text NOT NULL,
FOREIGN KEY (`integration_id`) REFERENCES `integrations`(`id`) ON UPDATE no action ON DELETE cascade
);
File diff suppressed because it is too large. Load diff
+7
View File
@@ -442,6 +442,13 @@
"when": 1792874762673, "when": 1792874762673,
"tag": "0062_spotty_viper", "tag": "0062_spotty_viper",
"breakpoints": true "breakpoints": true
},
{
"idx": 63,
"version": "6",
"when": 1792875717002,
"tag": "0063_lucky_talkback",
"breakpoints": true
} }
] ]
} }
+6
View File
@@ -64,6 +64,9 @@ import { authentikRouter } from "./routes/authentik.js";
import { prometheusRouter } from "./routes/prometheus.js"; import { prometheusRouter } from "./routes/prometheus.js";
import { immichRouter } from "./routes/immich.js"; import { immichRouter } from "./routes/immich.js";
import { paperlessRouter } from "./routes/paperless.js"; import { paperlessRouter } from "./routes/paperless.js";
import { n8nRouter } from "./routes/n8n.js";
import { netboxRouter } from "./routes/netbox.js";
import { nextcloudRouter } from "./routes/nextcloud.js";
import { serviceLevelsRouter } from "./routes/serviceLevels.js"; import { serviceLevelsRouter } from "./routes/serviceLevels.js";
import { publicApiRouter } from "./routes/publicApi.js"; import { publicApiRouter } from "./routes/publicApi.js";
import { generatorRouter } from "./routes/generator.js"; import { generatorRouter } from "./routes/generator.js";
@@ -204,6 +207,9 @@ export function createApp() {
app.use("/api/prometheus", prometheusRouter); app.use("/api/prometheus", prometheusRouter);
app.use("/api/immich", immichRouter); app.use("/api/immich", immichRouter);
app.use("/api/paperless", paperlessRouter); app.use("/api/paperless", paperlessRouter);
app.use("/api/n8n", n8nRouter);
app.use("/api/netbox", netboxRouter);
app.use("/api/nextcloud", nextcloudRouter);
app.use("/api/service-levels", serviceLevelsRouter); app.use("/api/service-levels", serviceLevelsRouter);
app.use("/api/todos", todosRouter); app.use("/api/todos", todosRouter);
app.use("/api/notes", notesRouter); app.use("/api/notes", notesRouter);
+55
View File
@@ -558,6 +558,9 @@ export const integrationTypes = [
"prometheus", "prometheus",
"immich", "immich",
"paperless", "paperless",
"nextcloud",
"netbox",
"n8n",
] as const; ] as const;
export type IntegrationType = (typeof integrationTypes)[number]; export type IntegrationType = (typeof integrationTypes)[number];
@@ -1164,6 +1167,58 @@ export const haReadings = sqliteTable(
(t) => [index("ha_readings_lookup_idx").on(t.integrationId, t.entityId, t.at)], (t) => [index("ha_readings_lookup_idx").on(t.integrationId, t.entityId, t.at)],
); );
// ─── Nextcloud ──────────────────────────────────────────────────────────────
// How a Nextcloud (an integration of type "nextcloud") is right now, one row each, replaced at every read: its version, whether it is in maintenance mode or needs its database
// upgrading, and — with an administrator's app password — the room left for the data, the users and files, and the app updates waiting. A column that is null could not be read.
// Read by services/nextcloudSync.ts.
export const nextcloudState = sqliteTable("nextcloud_state", {
integrationId: integer("integration_id")
.primaryKey()
.references(() => integrations.id, { onDelete: "cascade" }),
version: text("version"),
maintenance: integer("maintenance", { mode: "boolean" }).notNull().default(false),
needsUpgrade: integer("needs_upgrade", { mode: "boolean" }).notNull().default(false),
detail: integer("detail", { mode: "boolean" }).notNull().default(false), // whether the monitoring app answered
freeBytes: integer("free_bytes"),
users: integer("users"),
files: integer("files"),
active24h: integer("active_24h"),
appUpdates: integer("app_updates"), // how many apps have an update waiting; null: not asked
updatedAt: text("updated_at").notNull(), // ISO
});
// ─── NetBox ─────────────────────────────────────────────────────────────────
// How a NetBox (an integration of type "netbox") is right now, one row each, replaced at every read: its version, how many background workers are running, the counts and the IP
// addresses it holds (kept so the servers this app manages that are not in it can be listed). A column that is null could not be read. Read by services/netboxSync.ts.
export const netboxState = sqliteTable("netbox_state", {
integrationId: integer("integration_id")
.primaryKey()
.references(() => integrations.id, { onDelete: "cascade" }),
version: text("version"),
workersRunning: integer("workers_running"), // null: /status/ did not say
devices: integer("devices"),
virtualMachines: integer("virtual_machines"),
prefixes: integer("prefixes"),
ipAddresses: integer("ip_addresses"),
ips: text("ips"), // JSON string[] of the addresses (without the prefix length), at most 5000; null: could not be listed
updatedAt: text("updated_at").notNull(), // ISO
});
// ─── n8n ────────────────────────────────────────────────────────────────────
// How an n8n (an integration of type "n8n") is right now, one row each, replaced at every read: its workflows, each with its last outcome and the failures of the last day (worked out
// from the latest 250 runs). Read by services/n8nSync.ts.
export const n8nState = sqliteTable("n8n_state", {
integrationId: integer("integration_id")
.primaryKey()
.references(() => integrations.id, { onDelete: "cascade" }),
workflows: text("workflows").notNull().default("[]"), // JSON: id, name, active, lastStatus, lastAt, failed24h
executionsKnown: integer("executions_known", { mode: "boolean" }).notNull().default(false), // false: the runs could not be asked, so no last outcome is known
updatedAt: text("updated_at").notNull(), // ISO
});
// ─── Paperless-ngx ────────────────────────────────────────────────────────── // ─── Paperless-ngx ──────────────────────────────────────────────────────────
// How a Paperless-ngx (an integration of type "paperless") is right now, one row each, replaced at every read: its version, the disk, how each component of its system status is, the // How a Paperless-ngx (an integration of type "paperless") is right now, one row each, replaced at every read: its version, the disk, how each component of its system status is, the
+6
View File
@@ -34,6 +34,9 @@ import { initAuthentikSync } from "./services/authentikSync.js";
import { initPrometheusSync } from "./services/prometheusSync.js"; import { initPrometheusSync } from "./services/prometheusSync.js";
import { initImmichSync } from "./services/immichSync.js"; import { initImmichSync } from "./services/immichSync.js";
import { initPaperlessSync } from "./services/paperlessSync.js"; import { initPaperlessSync } from "./services/paperlessSync.js";
import { initN8nSync } from "./services/n8nSync.js";
import { initNetBoxSync } from "./services/netboxSync.js";
import { initNextcloudSync } from "./services/nextcloudSync.js";
import { backfillMetricRollups } from "./services/serverMetrics.js"; import { backfillMetricRollups } from "./services/serverMetrics.js";
import { backfillEndpointHourly } from "./services/endpointChecks.js"; import { backfillEndpointHourly } from "./services/endpointChecks.js";
import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js"; import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js";
@@ -89,6 +92,9 @@ initAuthentikSync();
initPrometheusSync(); initPrometheusSync();
initImmichSync(); initImmichSync();
initPaperlessSync(); initPaperlessSync();
initN8nSync();
initNetBoxSync();
initNextcloudSync();
// The hourly and daily figures are built from what is already kept, once, the first time after they were added (in the background: nothing waits for it). // The hourly and daily figures are built from what is already kept, once, the first time after they were added (in the background: nothing waits for it).
void Promise.all([backfillMetricRollups(), backfillEndpointHourly()]).catch((err) => console.error("[history] couldn't build the hourly and daily figures:", err)); void Promise.all([backfillMetricRollups(), backfillEndpointHourly()]).catch((err) => console.error("[history] couldn't build the hourly and daily figures:", err));
+16
View File
@@ -99,6 +99,22 @@ export const INTEGRATION_FIELDS: Partial<Record<IntegrationType, IntegrationFiel
{ key: "token", label: "Long-lived access token", secret: true, type: "password", placeholder: "made on your profile page in Home Assistant" }, { key: "token", label: "Long-lived access token", secret: true, type: "password", placeholder: "made on your profile page in Home Assistant" },
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" }, { key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
], ],
nextcloud: [
{ key: "url", label: "Nextcloud URL", secret: false, placeholder: "https://cloud.example.lan" },
{ key: "username", label: "Administrator's name (optional)", secret: false, optional: true, placeholder: "for the disk, the users and the app updates" },
{ key: "appPassword", label: "App password (optional)", secret: true, optional: true, type: "password", placeholder: "Personal settings → Security → Devices & sessions" },
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
],
netbox: [
{ key: "url", label: "NetBox URL", secret: false, placeholder: "https://netbox.example.lan" },
{ key: "token", label: "API token", secret: true, type: "password", placeholder: "your profile → API Tokens; read-only is enough" },
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
],
n8n: [
{ key: "url", label: "n8n URL", secret: false, placeholder: "http://n8n.example.lan:5678" },
{ key: "apiKey", label: "API key", secret: true, type: "password", placeholder: "Settings → n8n API" },
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
],
paperless: [ paperless: [
{ key: "url", label: "Paperless-ngx URL", secret: false, placeholder: "http://paperless.example.lan:8000" }, { key: "url", label: "Paperless-ngx URL", secret: false, placeholder: "http://paperless.example.lan:8000" },
{ key: "token", label: "API token", secret: true, type: "password", placeholder: "your profile → API Auth Token; a superuser's, for the system status" }, { key: "token", label: "API token", secret: true, type: "password", placeholder: "your profile → API Auth Token; a superuser's, for the system status" },
+127
View File
@@ -0,0 +1,127 @@
/**
* n8n adapter. Config: url, apiKey (made in n8n under Settings → n8n API); optional: insecure.
*
* Reading only: the workflows (and which are switched on), and the most recent runs, from which each workflow's **last run** and how many runs failed in the last day are worked out —
* an automation that has been failing since Tuesday is what is worth hearing about. Nothing is sent to n8n but GET requests for these.
*
* Endpoints and field names are from n8n's published public API (`/api/v1/workflows`, `/api/v1/executions`) and have not been checked against a live server here; every field is
* read defensively. Only the newest 250 runs are looked at, so a workflow that runs more rarely than that many other runs may show no run.
*/
import { requestJson } from "../httpJson.js";
import { withDiagLogging } from "../../services/diagLog.js";
export interface N8nConfig {
url: string;
apiKey: string;
insecure?: boolean;
}
export interface N8nWorkflow {
id: string;
name: string;
active: boolean;
/** error, crashed, success, canceled, waiting, running, …; null when no run of it is among those looked at. */
lastStatus: string | null;
lastAt: string | null;
/** Failed (error or crashed) runs in the last 24 hours. */
failed24h: number;
}
export interface N8nRead {
workflows: N8nWorkflow[];
/** Whether the runs could be asked; false means the workflows' last runs are unknown, not fine. */
executionsKnown: boolean;
}
export interface N8nAdapter {
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
read(now?: Date): Promise<N8nRead>;
}
const obj = (v: unknown): Record<string, unknown> => (v !== null && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : {});
const str = (v: unknown): string | null => (typeof v === "string" && v.trim() !== "" ? v.trim() : v !== null && typeof v === "number" ? String(v) : null);
const iso = (v: unknown): string | null => {
const s = typeof v === "string" ? v : null;
return s && Number.isFinite(Date.parse(s)) ? new Date(s).toISOString() : null;
};
export const FAILED_STATUSES = new Set(["error", "crashed"]);
const PAGES = 4;
/** Pure: the workflows of one page. */
export function parseWorkflows(raw: unknown): { id: string; name: string; active: boolean }[] {
const out: { id: string; name: string; active: boolean }[] = [];
for (const item of Array.isArray(obj(raw).data) ? (obj(raw).data as unknown[]) : []) {
const w = obj(item);
const id = str(w.id);
if (!id) continue;
out.push({ id, name: str(w.name) ?? id, active: w.active === true });
}
return out;
}
/** Pure: each workflow's last run and its failures in the last day, from a list of runs. Runs come newest first, but nothing here depends on it. */
export function summariseRuns(workflows: { id: string; name: string; active: boolean }[], raw: unknown, now: Date): N8nWorkflow[] {
const since = now.getTime() - 86_400_000;
const last = new Map<string, { status: string; t: number; at: string }>();
const failed = new Map<string, number>();
for (const item of Array.isArray(obj(raw).data) ? (obj(raw).data as unknown[]) : []) {
const e = obj(item);
const wf = str(e.workflowId);
const status = str(e.status)?.toLowerCase() ?? (e.finished === true ? "success" : null);
const at = iso(e.stoppedAt) ?? iso(e.startedAt);
if (!wf || !status || !at) continue;
const t = Date.parse(at);
// A run that is still going is not an outcome: the one before it is.
if (status !== "running" && status !== "waiting" && status !== "new") {
const seen = last.get(wf);
if (!seen || t > seen.t) last.set(wf, { status, t, at });
if (FAILED_STATUSES.has(status) && t >= since) failed.set(wf, (failed.get(wf) ?? 0) + 1);
}
}
return workflows.map((w) => ({ ...w, lastStatus: last.get(w.id)?.status ?? null, lastAt: last.get(w.id)?.at ?? null, failed24h: failed.get(w.id) ?? 0 }));
}
export function createN8nAdapter(config: N8nConfig): N8nAdapter {
const base = String(config.url ?? "").replace(/\/+$/, "");
const insecure = config.insecure === true;
const headers = { "X-N8N-API-KEY": config.apiKey, Accept: "application/json" };
async function get(path: string): Promise<unknown> {
const res = await requestJson(`${base}/api/v1${path}`, { headers, insecure });
if (res.status === 401 || res.status === 403) throw new Error(`n8n did not accept the API key (HTTP ${res.status}) — make one under Settings → n8n API`);
if (res.status === 404) throw new Error("n8n has no public API at this address — check the address, and that the API is switched on");
if (res.status < 200 || res.status >= 300) throw new Error(`n8n error: HTTP ${res.status}`);
if (res.json === null) throw new Error("n8n answered with something that isn't JSON — is the address its own?");
return res.json;
}
async function read(now: Date = new Date()): Promise<N8nRead> {
// The workflows are the hard requirement: they prove the key is accepted.
const workflows: { id: string; name: string; active: boolean }[] = [];
let cursor: string | null = null;
for (let page = 0; page < PAGES; page++) {
const raw = await get(`/workflows?limit=250${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ""}`);
workflows.push(...parseWorkflows(raw));
cursor = str(obj(raw).nextCursor);
if (!cursor) break;
}
try {
return { workflows: summariseRuns(workflows, await get("/executions?limit=250"), now), executionsKnown: true };
} catch {
return { workflows: workflows.map((w) => ({ ...w, lastStatus: null, lastAt: null, failed24h: 0 })), executionsKnown: false };
}
}
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
const start = Date.now();
try {
await get("/workflows?limit=1");
return { ok: true, latencyMs: Date.now() - start };
} catch (err) {
return { ok: false, error: err instanceof Error ? err.message : String(err) };
}
}
return withDiagLogging("n8n", { ping, read });
}
+124
View File
@@ -0,0 +1,124 @@
/**
* NetBox adapter. Config: url, token (made in NetBox under your profile → API Tokens; read-only is enough — "Write enabled" off); optional: insecure.
*
* Reading only: NetBox's version, **how many background workers are running** (they run its webhooks, scripts, jobs and reports — and in the Docker image they are a separate
* container that can be down while the page still loads), how many devices, virtual machines, prefixes and IP addresses it holds, and the IP addresses themselves — so the app can say
* which of the servers it manages have no address in NetBox. Nothing is sent to NetBox but GET requests for these.
*
* Endpoints and field names are from NetBox's published REST API (`/api/status/`, `/api/dcim/devices/`, `/api/virtualization/virtual-machines/`, `/api/ipam/prefixes/`,
* `/api/ipam/ip-addresses/`) and have not been checked against a live server here; every field is read defensively.
*/
import { requestJson } from "../httpJson.js";
import { withDiagLogging } from "../../services/diagLog.js";
export interface NetBoxConfig {
url: string;
token: string;
insecure?: boolean;
}
export interface NetBoxRead {
version: string | null;
/** Null when /status/ did not say, which is not the same as none. */
workersRunning: number | null;
devices: number | null;
virtualMachines: number | null;
prefixes: number | null;
ipAddresses: number | null;
/** The addresses themselves, without their prefix length, up to MAX_IPS; null when they could not be asked. */
ips: string[] | null;
}
export interface NetBoxAdapter {
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
read(): Promise<NetBoxRead>;
}
export const MAX_IPS = 5000;
const PAGE = 1000;
const obj = (v: unknown): Record<string, unknown> => (v !== null && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : {});
const str = (v: unknown): string | null => (typeof v === "string" && v.trim() !== "" ? v.trim() : null);
const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
/** Pure: what /api/status/ says. */
export function parseStatus(raw: unknown): { version: string | null; workersRunning: number | null } {
const o = obj(raw);
return { version: str(o["netbox-version"]), workersRunning: num(o["rq-workers-running"]) };
}
/** Pure: how many things a list says there are (`count`); null when the answer has none. */
export function parseCount(raw: unknown): number | null {
return num(obj(raw).count);
}
/** Pure: the addresses of one page of /ipam/ip-addresses/, without the "/24". */
export function parseAddresses(raw: unknown): string[] {
const results = obj(raw).results;
const out: string[] = [];
for (const item of Array.isArray(results) ? results : []) {
const a = str(obj(item).address);
if (a) out.push(a.split("/")[0].toLowerCase());
}
return out;
}
/** NetBox 4.5's v2 tokens (nbt_…) go as a Bearer token; the older ones as "Token". */
export const authorization = (token: string) => (token.startsWith("nbt_") ? `Bearer ${token}` : `Token ${token}`);
export function createNetBoxAdapter(config: NetBoxConfig): NetBoxAdapter {
const base = String(config.url ?? "").replace(/\/+$/, "");
const insecure = config.insecure === true;
const headers = { Authorization: authorization(config.token), Accept: "application/json" };
async function get(path: string): Promise<unknown> {
const res = await requestJson(`${base}/api${path}`, { headers, insecure });
if (res.status === 401 || res.status === 403) throw new Error(`NetBox did not accept the token (HTTP ${res.status}) — check it, and that it has not expired`);
if (res.status === 404) throw new Error("NetBox has no API at this address — check the address (and that it is NetBox's own, not a proxy's)");
if (res.status < 200 || res.status >= 300) throw new Error(`NetBox error: HTTP ${res.status}`);
if (res.json === null) throw new Error("NetBox answered with something that isn't JSON — is the address its own?");
return res.json;
}
async function read(): Promise<NetBoxRead> {
const status = parseStatus(await get("/status/"));
const soft = async <T>(path: string, parse: (raw: unknown) => T): Promise<T | null> => {
try {
return parse(await get(path));
} catch {
return null;
}
};
const [devices, virtualMachines, prefixes, ipAddresses] = await Promise.all([
soft("/dcim/devices/?limit=1", parseCount),
soft("/virtualization/virtual-machines/?limit=1", parseCount),
soft("/ipam/prefixes/?limit=1", parseCount),
soft("/ipam/ip-addresses/?limit=1", parseCount),
]);
let ips: string[] | null = null;
if (ipAddresses !== null) {
const all: string[] = [];
try {
for (let offset = 0; offset < Math.min(ipAddresses, MAX_IPS); offset += PAGE) {
all.push(...parseAddresses(await get(`/ipam/ip-addresses/?brief=1&limit=${PAGE}&offset=${offset}`)));
}
ips = [...new Set(all)].slice(0, MAX_IPS);
} catch {
ips = null;
}
}
return { version: status.version, workersRunning: status.workersRunning, devices, virtualMachines, prefixes, ipAddresses, ips };
}
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
const start = Date.now();
try {
await get("/status/");
return { ok: true, latencyMs: Date.now() - start };
} catch (err) {
return { ok: false, error: err instanceof Error ? err.message : String(err) };
}
}
return withDiagLogging("netbox", { ping, read });
}
@@ -0,0 +1,112 @@
/**
* Nextcloud adapter. Config: url; optional: username and appPassword (an administrator's, made under Personal settings → Security → Devices & sessions, so the server's own
* password is never needed), insecure.
*
* Reading only. Without a login: /status.php, which Nextcloud answers to anyone — its version, whether it is in **maintenance mode** and whether it **needs its database
* upgrading** (either way nobody can sign in). With an administrator's app password the *Monitoring* ("serverinfo") app is also asked: how much room is left for the data, how many
* users and files, who was active, and how many app updates wait. Nothing is sent to Nextcloud but GET requests for these.
*
* Endpoints and field names are from Nextcloud's published API (`/status.php`, `/ocs/v2.php/apps/serverinfo/api/v1/info`) and have not been checked against a live server here;
* every field is read defensively. Whether background jobs (cron) are running is *not* read: the API this app uses does not say.
*/
import { requestJson } from "../httpJson.js";
import { withDiagLogging } from "../../services/diagLog.js";
export interface NextcloudConfig {
url: string;
username?: string;
appPassword?: string;
insecure?: boolean;
}
export interface NextcloudRead {
version: string | null;
maintenance: boolean;
needsUpgrade: boolean;
/** Whether the monitoring app answered; false without a login, or when it refused or is not installed. */
detail: boolean;
/** Bytes free where the data is kept. */
freeBytes: number | null;
users: number | null;
files: number | null;
/** Users active in the last 24 hours. */
active24h: number | null;
/** How many apps have an update waiting; null when not asked. */
appUpdates: number | null;
}
export interface NextcloudAdapter {
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
read(): Promise<NextcloudRead>;
}
const obj = (v: unknown): Record<string, unknown> => (v !== null && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : {});
const str = (v: unknown): string | null => (typeof v === "string" && v.trim() !== "" ? v.trim() : null);
const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
/** Nextcloud's own JSON gives booleans as booleans, but some answers say 1 or "1". */
const flag = (v: unknown): boolean => v === true || v === 1 || v === "1" || v === "true";
/** Pure: what /status.php says. */
export function parseStatus(raw: unknown): Pick<NextcloudRead, "version" | "maintenance" | "needsUpgrade"> & { installed: boolean } {
const o = obj(raw);
return { version: str(o.versionstring) ?? str(o.version), maintenance: flag(o.maintenance), needsUpgrade: flag(o.needsDbUpgrade), installed: o.installed === undefined ? true : flag(o.installed) };
}
/** Pure: what the monitoring app says (`ocs.data`). */
export function parseServerInfo(raw: unknown): Pick<NextcloudRead, "freeBytes" | "users" | "files" | "active24h" | "appUpdates"> | null {
const data = obj(obj(raw).ocs).data;
if (data === undefined || data === null || typeof data !== "object") return null;
const nc = obj(obj(data).nextcloud);
const system = obj(nc.system);
const apps = obj(system.apps);
const listed = Object.keys(obj(apps.app_updates)).length;
return {
freeBytes: num(system.freespace),
users: num(obj(nc.storage).num_users),
files: num(obj(nc.storage).num_files),
active24h: num(obj(obj(data).activeUsers).last24hours),
appUpdates: num(apps.num_updates_available) ?? (listed > 0 ? listed : null),
};
}
export function createNextcloudAdapter(config: NextcloudConfig): NextcloudAdapter {
const base = String(config.url ?? "").replace(/\/+$/, "");
const insecure = config.insecure === true;
const login = config.username && config.appPassword ? { Authorization: `Basic ${Buffer.from(`${config.username}:${config.appPassword}`).toString("base64")}`, "OCS-APIRequest": "true" } : null;
async function get(path: string, headers: Record<string, string> = {}): Promise<unknown> {
const res = await requestJson(`${base}${path}`, { headers: { Accept: "application/json", ...headers }, insecure });
if (res.status === 401 || res.status === 403) throw new Error(`Nextcloud did not accept the login (HTTP ${res.status}) — use an administrator's name and an app password`);
if (res.status === 404) throw new Error("Nextcloud has no such page at this address — check the address (and that it is Nextcloud's own, not a proxy's)");
if (res.status < 200 || res.status >= 300) throw new Error(`Nextcloud error: HTTP ${res.status}`);
if (res.json === null) throw new Error("Nextcloud answered with something that isn't JSON — is the address its own?");
return res.json;
}
async function read(): Promise<NextcloudRead> {
const status = parseStatus(await get("/status.php"));
if (!status.installed) throw new Error("That Nextcloud says it is not installed yet");
let detail: ReturnType<typeof parseServerInfo> = null;
if (login) {
try {
detail = parseServerInfo(await get("/ocs/v2.php/apps/serverinfo/api/v1/info?format=json", login));
} catch {
detail = null; // not an administrator, or the Monitoring app is off: the rest is still worth having
}
}
return { version: status.version, maintenance: status.maintenance, needsUpgrade: status.needsUpgrade, detail: detail !== null, freeBytes: detail?.freeBytes ?? null, users: detail?.users ?? null, files: detail?.files ?? null, active24h: detail?.active24h ?? null, appUpdates: detail?.appUpdates ?? null };
}
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
const start = Date.now();
try {
await get("/status.php");
if (login) await get("/ocs/v2.php/apps/serverinfo/api/v1/info?format=json&skipApps=true&skipUpdate=true", login);
return { ok: true, latencyMs: Date.now() - start };
} catch (err) {
return { ok: false, error: err instanceof Error ? err.message : String(err) };
}
}
return withDiagLogging("nextcloud", { ping, read });
}
+9
View File
@@ -19,6 +19,9 @@ import { createAuthentikAdapter } from "./authentik/adapter.js";
import { createPrometheusAdapter } from "./prometheus/adapter.js"; import { createPrometheusAdapter } from "./prometheus/adapter.js";
import { createImmichAdapter } from "./immich/adapter.js"; import { createImmichAdapter } from "./immich/adapter.js";
import { createPaperlessAdapter } from "./paperless/adapter.js"; import { createPaperlessAdapter } from "./paperless/adapter.js";
import { createN8nAdapter } from "./n8n/adapter.js";
import { createNetBoxAdapter } from "./netbox/adapter.js";
import { createNextcloudAdapter } from "./nextcloud/adapter.js";
export interface PingableAdapter { export interface PingableAdapter {
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>; ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
@@ -71,6 +74,12 @@ export function createIntegrationAdapter(type: IntegrationType, config: Integrat
return createImmichAdapter(config as any); return createImmichAdapter(config as any);
case "paperless": case "paperless":
return createPaperlessAdapter(config as any); return createPaperlessAdapter(config as any);
case "n8n":
return createN8nAdapter(config as any);
case "netbox":
return createNetBoxAdapter(config as any);
case "nextcloud":
return createNextcloudAdapter(config as any);
default: default:
throw new Error(`Integration type "${type}" is not implemented yet`); throw new Error(`Integration type "${type}" is not implemented yet`);
} }
+12
View File
@@ -34,6 +34,9 @@ import { forgetAuthentik, syncAuthentik } from "../services/authentikSync.js";
import { forgetPrometheus, syncPrometheus } from "../services/prometheusSync.js"; import { forgetPrometheus, syncPrometheus } from "../services/prometheusSync.js";
import { forgetImmich, syncImmich } from "../services/immichSync.js"; import { forgetImmich, syncImmich } from "../services/immichSync.js";
import { forgetPaperless, syncPaperless } from "../services/paperlessSync.js"; import { forgetPaperless, syncPaperless } from "../services/paperlessSync.js";
import { forgetN8n, syncN8n } from "../services/n8nSync.js";
import { forgetNetBox, syncNetBox } from "../services/netboxSync.js";
import { forgetNextcloud, syncNextcloud } from "../services/nextcloudSync.js";
import { invalidateAlertsCache } from "../services/alerts.js"; import { invalidateAlertsCache } from "../services/alerts.js";
import { getSettings, updateSettings } from "../services/settingsStore.js"; import { getSettings, updateSettings } from "../services/settingsStore.js";
import { asyncHandler } from "../utils/asyncHandler.js"; import { asyncHandler } from "../utils/asyncHandler.js";
@@ -141,6 +144,9 @@ integrationsRouter.post("/", requireRole("admin"), asyncHandler(async (req, res)
if (type === "prometheus") void syncPrometheus(created.id).then(() => invalidateAlertsCache()); if (type === "prometheus") void syncPrometheus(created.id).then(() => invalidateAlertsCache());
if (type === "immich") void syncImmich(created.id).then(() => invalidateAlertsCache()); if (type === "immich") void syncImmich(created.id).then(() => invalidateAlertsCache());
if (type === "paperless") void syncPaperless(created.id).then(() => invalidateAlertsCache()); if (type === "paperless") void syncPaperless(created.id).then(() => invalidateAlertsCache());
if (type === "n8n") void syncN8n(created.id).then(() => invalidateAlertsCache());
if (type === "netbox") void syncNetBox(created.id).then(() => invalidateAlertsCache());
if (type === "nextcloud") void syncNextcloud(created.id).then(() => invalidateAlertsCache());
res.status(201).json({ res.status(201).json({
integration: { integration: {
@@ -249,6 +255,9 @@ integrationsRouter.patch("/:id", requireRole("admin"), asyncHandler(async (req,
if (updated.type === "prometheus" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncPrometheus(updated.id).then(() => invalidateAlertsCache()); if (updated.type === "prometheus" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncPrometheus(updated.id).then(() => invalidateAlertsCache());
if (updated.type === "immich" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncImmich(updated.id).then(() => invalidateAlertsCache()); if (updated.type === "immich" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncImmich(updated.id).then(() => invalidateAlertsCache());
if (updated.type === "paperless" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncPaperless(updated.id).then(() => invalidateAlertsCache()); if (updated.type === "paperless" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncPaperless(updated.id).then(() => invalidateAlertsCache());
if (updated.type === "n8n" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncN8n(updated.id).then(() => invalidateAlertsCache());
if (updated.type === "netbox" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncNetBox(updated.id).then(() => invalidateAlertsCache());
if (updated.type === "nextcloud" && updated.enabled && (parsed.data.config || parsed.data.enabled === true)) void syncNextcloud(updated.id).then(() => invalidateAlertsCache());
res.json({ res.json({
integration: { integration: {
@@ -333,6 +342,9 @@ integrationsRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req,
if (existing.type === "prometheus") await forgetPrometheus(id); if (existing.type === "prometheus") await forgetPrometheus(id);
if (existing.type === "immich") await forgetImmich(id); if (existing.type === "immich") await forgetImmich(id);
if (existing.type === "paperless") await forgetPaperless(id); if (existing.type === "paperless") await forgetPaperless(id);
if (existing.type === "n8n") await forgetN8n(id);
if (existing.type === "netbox") await forgetNetBox(id);
if (existing.type === "nextcloud") await forgetNextcloud(id);
await db.delete(integrations).where(eq(integrations.id, id)); await db.delete(integrations).where(eq(integrations.id, id));
if (existing.credentialId) { if (existing.credentialId) {
await db.delete(integrationCredentials).where(eq(integrationCredentials.id, existing.credentialId)); await db.delete(integrationCredentials).where(eq(integrationCredentials.id, existing.credentialId));
+37
View File
@@ -0,0 +1,37 @@
import { Router } from "express";
import { eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, n8nState } from "../db/schema.js";
import { requireAuth } from "../auth/middleware.js";
import { isN8nFresh, n8nProblems, n8nSyncStatuses, workflowsOf } from "../services/n8nSync.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const n8nRouter = Router();
n8nRouter.use(requireAuth);
/** Every n8n, how it is, from what the last read stored: its workflows with their last runs, and what is wrong. Everyone signed in can see it, like the other pages made of stored readings. */
n8nRouter.get("/", asyncHandler(async (_req, res) => {
const [rows, states, statuses] = await Promise.all([
db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "n8n")).orderBy(integrations.name),
db.select().from(n8nState),
n8nSyncStatuses(),
]);
const byId = new Map(states.map((s) => [s.integrationId, s]));
const now = new Date();
res.json({
n8ns: rows.map((r) => {
const state = byId.get(r.id) ?? null;
const status = statuses[r.id] ?? null;
const fresh = isN8nFresh(status ?? undefined, now.getTime());
return {
id: r.id,
name: r.name,
enabled: r.enabled,
status,
fresh,
state: state ? { workflows: workflowsOf(state), executionsKnown: state.executionsKnown, updatedAt: state.updatedAt } : null,
problems: state && fresh ? n8nProblems(r.id, r.name, state, now) : [],
};
}),
});
}));
+68
View File
@@ -0,0 +1,68 @@
import { Router } from "express";
import { eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, netboxState, servers } from "../db/schema.js";
import { requireAuth } from "../auth/middleware.js";
import { makeExclusion } from "../services/consistency.js";
import { ipsOf, isNetBoxFresh, netboxProblems, netboxSyncStatuses, serversMissingFromNetBox } from "../services/netboxSync.js";
import { getSettings } from "../services/settingsStore.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const netboxRouter = Router();
netboxRouter.use(requireAuth);
const parseIps = (stored: string | null): string[] => {
if (!stored) return [];
try {
const v = JSON.parse(stored);
return Array.isArray(v) ? v.filter((x): x is string => typeof x === "string") : [];
} catch {
return [];
}
};
/** Every NetBox, how it is, from what the last read stored: its workers, the counts, and which of the servers this app manages have no address in it. Everyone signed in can see it. */
netboxRouter.get("/", asyncHandler(async (_req, res) => {
const [rows, states, statuses, serverRows, settings] = await Promise.all([
db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "netbox")).orderBy(integrations.name),
db.select().from(netboxState),
netboxSyncStatuses(),
db.select({ id: servers.id, name: servers.name, ips: servers.ipAddresses }).from(servers),
getSettings(),
]);
const byId = new Map(states.map((s) => [s.integrationId, s]));
const serverList = serverRows.map((s) => ({ id: s.id, name: s.name, ips: parseIps(s.ips) }));
const excluded = makeExclusion(settings.consistency.excludedRanges);
const now = Date.now();
res.json({
netboxes: rows.map((r) => {
const state = byId.get(r.id) ?? null;
const status = statuses[r.id] ?? null;
const fresh = isNetBoxFresh(status ?? undefined, now);
const ips = state ? ipsOf(state) : null;
return {
id: r.id,
name: r.name,
enabled: r.enabled,
status,
fresh,
state: state
? {
version: state.version,
workersRunning: state.workersRunning,
devices: state.devices,
virtualMachines: state.virtualMachines,
prefixes: state.prefixes,
ipAddresses: state.ipAddresses,
/** Null when NetBox's addresses could not be listed, so nothing can be said about what is missing from it. */
missing: ips ? serversMissingFromNetBox(serverList, ips, excluded) : null,
/** More addresses than were kept: what is missing may be in the part that was not read. */
truncated: ips !== null && state.ipAddresses !== null && state.ipAddresses > ips.length,
updatedAt: state.updatedAt,
}
: null,
problems: state && fresh ? netboxProblems(r.id, r.name, state) : [],
};
}),
});
}));
+50
View File
@@ -0,0 +1,50 @@
import { Router } from "express";
import { eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, nextcloudState } from "../db/schema.js";
import { requireAuth } from "../auth/middleware.js";
import { isNextcloudFresh, nextcloudProblems, nextcloudSyncStatuses } from "../services/nextcloudSync.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const nextcloudRouter = Router();
nextcloudRouter.use(requireAuth);
/** Every Nextcloud, how it is, from what the last read stored: maintenance mode, the data disk, the users, the app updates and what is wrong. Everyone signed in can see it, like the other pages made of stored readings. */
nextcloudRouter.get("/", asyncHandler(async (_req, res) => {
const [rows, states, statuses] = await Promise.all([
db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "nextcloud")).orderBy(integrations.name),
db.select().from(nextcloudState),
nextcloudSyncStatuses(),
]);
const byId = new Map(states.map((s) => [s.integrationId, s]));
const now = Date.now();
res.json({
nextclouds: rows.map((r) => {
const state = byId.get(r.id) ?? null;
const status = statuses[r.id] ?? null;
const fresh = isNextcloudFresh(status ?? undefined, now);
return {
id: r.id,
name: r.name,
enabled: r.enabled,
status,
fresh,
state: state
? {
version: state.version,
maintenance: state.maintenance,
needsUpgrade: state.needsUpgrade,
detail: state.detail,
freeBytes: state.freeBytes,
users: state.users,
files: state.files,
active24h: state.active24h,
appUpdates: state.appUpdates,
updatedAt: state.updatedAt,
}
: null,
problems: state && fresh ? nextcloudProblems(r.id, r.name, state) : [],
};
}),
});
}));
+11 -2
View File
@@ -80,6 +80,9 @@ const SOURCES: Record<string, { label: string; link: string }> = {
prometheus: { label: "Prometheus", link: "/prometheus" }, prometheus: { label: "Prometheus", link: "/prometheus" },
immich: { label: "Immich", link: "/immich" }, immich: { label: "Immich", link: "/immich" },
paperless: { label: "Paperless-ngx", link: "/paperless" }, paperless: { label: "Paperless-ngx", link: "/paperless" },
n8n: { label: "n8n", link: "/n8n" },
netbox: { label: "NetBox", link: "/netbox" },
nextcloud: { label: "Nextcloud", link: "/nextcloud" },
secrets: { label: "Secrets", link: "/secrets" }, secrets: { label: "Secrets", link: "/secrets" },
domains: { label: "Domains", link: "/dns?view=domains" }, domains: { label: "Domains", link: "/dns?view=domains" },
endpoint: { label: "Endpoint", link: "/endpoints" }, endpoint: { label: "Endpoint", link: "/endpoints" },
@@ -118,6 +121,12 @@ function originOfConditionKey(key: string): { type: string; id: number | null; c
if (m) return { type: "dockhand", id: Number(m[1]), category: "services" }; if (m) return { type: "dockhand", id: Number(m[1]), category: "services" };
m = /^duplicati-(?:failed|stale):(\d+)/.exec(key); m = /^duplicati-(?:failed|stale):(\d+)/.exec(key);
if (m) return { type: "duplicati", id: Number(m[1]), category: "backup" }; if (m) return { type: "duplicati", id: Number(m[1]), category: "backup" };
m = /^nextcloud-(?:maintenance|upgrade|disk):(\d+)/.exec(key);
if (m) return { type: "nextcloud", id: Number(m[1]), category: key.startsWith("nextcloud-disk") ? "disk" : "services" };
m = /^netbox-(?:workers):(\d+)/.exec(key);
if (m) return { type: "netbox", id: Number(m[1]), category: "services" };
m = /^n8n-(?:failed):(\d+)/.exec(key);
if (m) return { type: "n8n", id: Number(m[1]), category: "automation" };
m = /^paperless-(?:component|failed|disk):(\d+)/.exec(key); m = /^paperless-(?:component|failed|disk):(\d+)/.exec(key);
if (m) return { type: "paperless", id: Number(m[1]), category: key.startsWith("paperless-disk") ? "disk" : "services" }; if (m) return { type: "paperless", id: Number(m[1]), category: key.startsWith("paperless-disk") ? "disk" : "services" };
m = /^immich-(?:disk|jobs):(\d+)/.exec(key); m = /^immich-(?:disk|jobs):(\d+)/.exec(key);
@@ -250,7 +259,7 @@ export async function collectAlerts(): Promise<AlertsReport> {
type: origin.type, type: origin.type,
message: c.message, message: c.message,
key: c.key, key: c.key,
link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") || c.key.startsWith("baseline:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : origin.type === "slo" ? "/service-levels" : origin.type === "dockhand" ? "/docker" : origin.type === "duplicati" ? "/duplicati" : origin.type === "jellyfin" ? "/jellyfin" : origin.type === "authentik" ? "/authentik" : origin.type === "prometheus" ? "/prometheus" : origin.type === "immich" ? "/immich" : origin.type === "paperless" ? "/paperless" : undefined, link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") || c.key.startsWith("baseline:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : origin.type === "slo" ? "/service-levels" : origin.type === "dockhand" ? "/docker" : origin.type === "duplicati" ? "/duplicati" : origin.type === "jellyfin" ? "/jellyfin" : origin.type === "authentik" ? "/authentik" : origin.type === "prometheus" ? "/prometheus" : origin.type === "immich" ? "/immich" : origin.type === "paperless" ? "/paperless" : origin.type === "nextcloud" ? "/nextcloud" : origin.type === "netbox" ? "/netbox" : origin.type === "n8n" ? "/n8n" : undefined,
silenced: subject !== null && subjects.has(subject), target: targetOfSubject(subject), silenced: subject !== null && subjects.has(subject), target: targetOfSubject(subject),
}); });
} }
@@ -260,7 +269,7 @@ export async function collectAlerts(): Promise<AlertsReport> {
if (held.has("slo")) cannotCheck("Service-level targets", "couldn't be judged — see the server log", "Server and storage health"); if (held.has("slo")) cannotCheck("Service-level targets", "couldn't be judged — see the server log", "Server and storage health");
if (held.has("server:baseline")) cannotCheck("What is usual for the time", "couldn't be judged — see the server log", "Server and storage health"); if (held.has("server:baseline")) cannotCheck("What is usual for the time", "couldn't be judged — see the server log", "Server and storage health");
for (const h of held) { for (const h of held) {
const m = /^(proxmox|synology|unifi|ups|dockhand|jellyfin|authentik|prometheus|immich|paperless):(\d+)(?::(.+))?$/.exec(h); const m = /^(proxmox|synology|unifi|ups|dockhand|jellyfin|authentik|prometheus|immich|paperless|nextcloud|netbox|n8n):(\d+)(?::(.+))?$/.exec(h);
if (!m || (m[3] && held.has(`${m[1]}:${m[2]}`))) continue; if (!m || (m[3] && held.has(`${m[1]}:${m[2]}`))) continue;
const row = intById.get(Number(m[2])); const row = intById.get(Number(m[2]));
cannotCheck(`${SOURCES[m[1]].label} “${row?.name ?? `#${m[2]}`}”${m[3] ? ` (node ${m[3]})` : ""}`, "couldn't be read — see the Diagnostic Log", "Server and storage health"); cannotCheck(`${SOURCES[m[1]].label} “${row?.name ?? `#${m[2]}`}”${m[3] ? ` (node ${m[3]})` : ""}`, "couldn't be read — see the Diagnostic Log", "Server and storage health");
+73 -1
View File
@@ -1,6 +1,6 @@
import { and, eq } from "drizzle-orm"; import { and, eq } from "drizzle-orm";
import { db } from "../db/client.js"; import { db } from "../db/client.js";
import { authentikState, paperlessState, immichState, prometheusState, integrations, jellyfinState, networkDevices, serverPorts, servers, upsState } from "../db/schema.js"; import { authentikState, n8nState, netboxState, nextcloudState, paperlessState, immichState, prometheusState, integrations, jellyfinState, networkDevices, serverPorts, servers, upsState } from "../db/schema.js";
import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { loadIntegrationConfig } from "../integrations/loadIntegration.js";
import { createProxmoxAdapter, type ProxmoxNodeStats } from "../integrations/proxmox/adapter.js"; import { createProxmoxAdapter, type ProxmoxNodeStats } from "../integrations/proxmox/adapter.js";
import { createSynologyAdapter, type SynologyStorageInfo } from "../integrations/synology/adapter.js"; import { createSynologyAdapter, type SynologyStorageInfo } from "../integrations/synology/adapter.js";
@@ -25,6 +25,9 @@ import { authentikProblems, authentikSyncStatuses, isAuthentikFresh, type Authen
import { isPrometheusFresh, prometheusProblems, prometheusSyncStatuses, type PrometheusProblem } from "./prometheusSync.js"; import { isPrometheusFresh, prometheusProblems, prometheusSyncStatuses, type PrometheusProblem } from "./prometheusSync.js";
import { immichProblems, immichSyncStatuses, isImmichFresh, type ImmichProblem } from "./immichSync.js"; import { immichProblems, immichSyncStatuses, isImmichFresh, type ImmichProblem } from "./immichSync.js";
import { isPaperlessFresh, paperlessProblems, paperlessSyncStatuses, type PaperlessProblem } from "./paperlessSync.js"; import { isPaperlessFresh, paperlessProblems, paperlessSyncStatuses, type PaperlessProblem } from "./paperlessSync.js";
import { isN8nFresh, n8nProblems, n8nSyncStatuses, type N8nProblem } from "./n8nSync.js";
import { isNetBoxFresh, netboxProblems, netboxSyncStatuses, type NetBoxProblem } from "./netboxSync.js";
import { isNextcloudFresh, nextcloudProblems, nextcloudSyncStatuses, type NextcloudProblem } from "./nextcloudSync.js";
import { evaluateContainers, loadContainerMemory, saveContainerMemory, type ContainerMemory, type DockerSnapshot } from "./containerHealth.js"; import { evaluateContainers, loadContainerMemory, saveContainerMemory, type ContainerMemory, type DockerSnapshot } from "./containerHealth.js";
import { recordStorageSamples } from "./storageHistory.js"; import { recordStorageSamples } from "./storageHistory.js";
import { isUpsFresh, upsProblems, upsSyncStatuses, type UpsStateRow } from "./upsSync.js"; import { isUpsFresh, upsProblems, upsSyncStatuses, type UpsStateRow } from "./upsSync.js";
@@ -92,6 +95,12 @@ export interface HealthSnapshot {
baselines?: BaselineFinding[]; baselines?: BaselineFinding[];
/** Service-level targets whose error budget for the month is mostly or wholly used (services/serviceLevels.ts). */ /** Service-level targets whose error budget for the month is mostly or wholly used (services/serviceLevels.ts). */
levels?: LevelFinding[]; levels?: LevelFinding[];
/** What is wrong with each Nextcloud (a component in error, documents that failed to import, a full disk), only while that reading is recent (services/nextcloudSync.ts). */
nextcloud?: { integrationId: number; problems: NextcloudProblem[] }[];
/** What is wrong with each NetBox (a component in error, documents that failed to import, a full disk), only while that reading is recent (services/netboxSync.ts). */
netbox?: { integrationId: number; problems: NetBoxProblem[] }[];
/** What is wrong with each n8n (a component in error, documents that failed to import, a full disk), only while that reading is recent (services/n8nSync.ts). */
n8n?: { integrationId: number; problems: N8nProblem[] }[];
/** What is wrong with each Paperless-ngx (a component in error, documents that failed to import, a full disk), only while that reading is recent (services/paperlessSync.ts). */ /** What is wrong with each Paperless-ngx (a component in error, documents that failed to import, a full disk), only while that reading is recent (services/paperlessSync.ts). */
paperless?: { integrationId: number; problems: PaperlessProblem[] }[]; paperless?: { integrationId: number; problems: PaperlessProblem[] }[];
/** What is wrong with each Immich (a disk filling, background jobs failing), only while that reading is recent (services/immichSync.ts). */ /** What is wrong with each Immich (a disk filling, background jobs failing), only while that reading is recent (services/immichSync.ts). */
@@ -213,6 +222,15 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
} }
} }
// A Nextcloud whose worker has stopped, whose imports fail, or whose disk is filling.
for (const p of snapshot.nextcloud ?? []) for (const c of p.problems) out.push({ key: c.key, source: `nextcloud:${p.integrationId}`, message: c.message, severity: c.severity });
// A NetBox whose worker has stopped, whose imports fail, or whose disk is filling.
for (const p of snapshot.netbox ?? []) for (const c of p.problems) out.push({ key: c.key, source: `netbox:${p.integrationId}`, message: c.message, severity: c.severity });
// A N8n whose worker has stopped, whose imports fail, or whose disk is filling.
for (const p of snapshot.n8n ?? []) for (const c of p.problems) out.push({ key: c.key, source: `n8n:${p.integrationId}`, message: c.message, severity: c.severity });
// A Paperless whose worker has stopped, whose imports fail, or whose disk is filling. // A Paperless whose worker has stopped, whose imports fail, or whose disk is filling.
for (const p of snapshot.paperless ?? []) for (const c of p.problems) out.push({ key: c.key, source: `paperless:${p.integrationId}`, message: c.message, severity: c.severity }); for (const p of snapshot.paperless ?? []) for (const c of p.problems) out.push({ key: c.key, source: `paperless:${p.integrationId}`, message: c.message, severity: c.severity });
@@ -458,6 +476,60 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel
} }
} }
// Nextcloud: read by its own pass every few minutes; what is judged is what that pass stored, and only while it is recent.
try {
const rowsL = await db.select({ id: integrations.id, name: integrations.name }).from(integrations).where(and(eq(integrations.type, "nextcloud"), eq(integrations.enabled, true)));
if (rowsL.length > 0) {
const statuses = await nextcloudSyncStatuses();
for (const r of rowsL) {
const [state] = await db.select().from(nextcloudState).where(eq(nextcloudState.integrationId, r.id)).limit(1);
if (!state || !isNextcloudFresh(statuses[r.id], Date.now())) {
held.add(`nextcloud:${r.id}`);
continue;
}
(snapshot.nextcloud ??= []).push({ integrationId: r.id, problems: nextcloudProblems(r.id, r.name, state) });
}
}
} catch (err) {
console.error("[health] couldn't judge Nextcloud:", err instanceof Error ? err.message : err);
}
// NetBox: read by its own pass every few minutes; what is judged is what that pass stored, and only while it is recent.
try {
const rowsL = await db.select({ id: integrations.id, name: integrations.name }).from(integrations).where(and(eq(integrations.type, "netbox"), eq(integrations.enabled, true)));
if (rowsL.length > 0) {
const statuses = await netboxSyncStatuses();
for (const r of rowsL) {
const [state] = await db.select().from(netboxState).where(eq(netboxState.integrationId, r.id)).limit(1);
if (!state || !isNetBoxFresh(statuses[r.id], Date.now())) {
held.add(`netbox:${r.id}`);
continue;
}
(snapshot.netbox ??= []).push({ integrationId: r.id, problems: netboxProblems(r.id, r.name, state) });
}
}
} catch (err) {
console.error("[health] couldn't judge NetBox:", err instanceof Error ? err.message : err);
}
// N8n: read by its own pass every few minutes; what is judged is what that pass stored, and only while it is recent.
try {
const rowsL = await db.select({ id: integrations.id, name: integrations.name }).from(integrations).where(and(eq(integrations.type, "n8n"), eq(integrations.enabled, true)));
if (rowsL.length > 0) {
const statuses = await n8nSyncStatuses();
for (const r of rowsL) {
const [state] = await db.select().from(n8nState).where(eq(n8nState.integrationId, r.id)).limit(1);
if (!state || !isN8nFresh(statuses[r.id], Date.now())) {
held.add(`n8n:${r.id}`);
continue;
}
(snapshot.n8n ??= []).push({ integrationId: r.id, problems: n8nProblems(r.id, r.name, state) });
}
}
} catch (err) {
console.error("[health] couldn't judge N8n:", err instanceof Error ? err.message : err);
}
// Paperless: read by its own pass every few minutes; what is judged is what that pass stored, and only while it is recent. // Paperless: read by its own pass every few minutes; what is judged is what that pass stored, and only while it is recent.
try { try {
const rowsL = await db.select({ id: integrations.id, name: integrations.name }).from(integrations).where(and(eq(integrations.type, "paperless"), eq(integrations.enabled, true))); const rowsL = await db.select({ id: integrations.id, name: integrations.name }).from(integrations).where(and(eq(integrations.type, "paperless"), eq(integrations.enabled, true)));
+2 -1
View File
@@ -121,7 +121,8 @@ export interface ImmichProblem {
severity: "warning" | "critical"; severity: "warning" | "critical";
} }
const gb = (bytes: number) => `${(bytes / 1e9).toFixed(bytes >= 1e11 ? 0 : 1)} GB`; /** In the same units the pages show (binary, written GB), so an alert and the tile beside it agree. */
const gb = (bytes: number) => `${(bytes / 1024 ** 3).toFixed(bytes >= 100 * 1024 ** 3 ? 0 : 1)} GB`;
/** Pure: what is wrong with one Immich, from what is stored for it. */ /** Pure: what is wrong with one Immich, from what is stored for it. */
export function immichProblems(id: number, name: string, s: Pick<StateRow, "diskSize" | "diskUsed" | "diskPercent" | "queues">): ImmichProblem[] { export function immichProblems(id: number, name: string, s: Pick<StateRow, "diskSize" | "diskUsed" | "diskPercent" | "queues">): ImmichProblem[] {
+6
View File
@@ -33,6 +33,12 @@ export function subjectOfConditionKey(key: string): string | null {
if (m) return `integration:${m[1]}`; if (m) return `integration:${m[1]}`;
m = /^jellyfin-(?:restart|task|logins):(\d+)/.exec(key); m = /^jellyfin-(?:restart|task|logins):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`; if (m) return `integration:${m[1]}`;
m = /^nextcloud-(?:maintenance|upgrade|disk):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`;
m = /^netbox-(?:workers):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`;
m = /^n8n-(?:failed):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`;
m = /^paperless-(?:component|failed|disk):(\d+)/.exec(key); m = /^paperless-(?:component|failed|disk):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`; if (m) return `integration:${m[1]}`;
m = /^immich-(?:disk|jobs):(\d+)/.exec(key); m = /^immich-(?:disk|jobs):(\d+)/.exec(key);
+51 -1
View File
@@ -5,11 +5,14 @@
*/ */
import { eq } from "drizzle-orm"; import { eq } from "drizzle-orm";
import { db } from "../db/client.js"; import { db } from "../db/client.js";
import { assets, endpointChecks, heartbeats, integrations, maintenanceWindows, secrets, servers, upsState, jellyfinState, authentikState, immichState, paperlessState } from "../db/schema.js"; import { assets, endpointChecks, heartbeats, integrations, maintenanceWindows, secrets, servers, upsState, jellyfinState, authentikState, immichState, paperlessState, nextcloudState, netboxState, n8nState } from "../db/schema.js";
import { isUpsFresh, upsSyncStatuses } from "./upsSync.js"; import { isUpsFresh, upsSyncStatuses } from "./upsSync.js";
import { levelStatuses } from "./serviceLevels.js"; import { levelStatuses } from "./serviceLevels.js";
import { backupState, listAll as listDuplicati } from "./duplicati.js"; import { backupState, listAll as listDuplicati } from "./duplicati.js";
import { isJellyfinFresh, jellyfinSyncStatuses, streamsOf } from "./jellyfinSync.js"; import { isJellyfinFresh, jellyfinSyncStatuses, streamsOf } from "./jellyfinSync.js";
import { isNextcloudFresh, nextcloudSyncStatuses } from "./nextcloudSync.js";
import { isNetBoxFresh, netboxSyncStatuses } from "./netboxSync.js";
import { isN8nFresh, n8nSyncStatuses, workflowsOf as n8nWorkflowsOf } from "./n8nSync.js";
import { componentsOf as paperlessComponentsOf, failedOf as paperlessFailedOf, isPaperlessFresh, paperlessSyncStatuses } from "./paperlessSync.js"; import { componentsOf as paperlessComponentsOf, failedOf as paperlessFailedOf, isPaperlessFresh, paperlessSyncStatuses } from "./paperlessSync.js";
import { immichSyncStatuses, isImmichFresh, queuesOf as immichQueuesOf } from "./immichSync.js"; import { immichSyncStatuses, isImmichFresh, queuesOf as immichQueuesOf } from "./immichSync.js";
import { authentikSyncStatuses, certificatesOf, daysUntil, isAuthentikFresh, outpostsOf } from "./authentikSync.js"; import { authentikSyncStatuses, certificatesOf, daysUntil, isAuthentikFresh, outpostsOf } from "./authentikSync.js";
@@ -64,6 +67,50 @@ class Exposition {
} }
} }
/** Each Nextcloud, for /metrics: whether it is in maintenance mode, the room left for the data, who was active and the app updates waiting. Only while the reading is recent. */
async function addNextcloudMetrics(out: Exposition, now: Date) {
const statuses = await nextcloudSyncStatuses();
const names = new Map((await db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "nextcloud"))).map((r) => [r.id, r]));
for (const s of await db.select().from(nextcloudState)) {
const row = names.get(s.integrationId);
if (!row || !row.enabled || !isNextcloudFresh(statuses[s.integrationId], now.getTime())) continue;
const labels = { name: row.name };
out.add("homelab_nextcloud_maintenance_mode", "gauge", "1 if Nextcloud is in maintenance mode, else 0.", s.maintenance ? 1 : 0, labels);
if (s.freeBytes !== null) out.add("homelab_nextcloud_free_bytes", "gauge", "Bytes free where Nextcloud keeps its data.", s.freeBytes, labels);
if (s.active24h !== null) out.add("homelab_nextcloud_active_users_24h", "gauge", "Users active in the last 24 hours.", s.active24h, labels);
if (s.appUpdates !== null) out.add("homelab_nextcloud_app_updates", "gauge", "How many apps have an update waiting.", s.appUpdates, labels);
}
}
/** Each NetBox, for /metrics: the workers running and what it holds. Only while the reading is recent. */
async function addNetBoxMetrics(out: Exposition, now: Date) {
const statuses = await netboxSyncStatuses();
const names = new Map((await db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "netbox"))).map((r) => [r.id, r]));
for (const s of await db.select().from(netboxState)) {
const row = names.get(s.integrationId);
if (!row || !row.enabled || !isNetBoxFresh(statuses[s.integrationId], now.getTime())) continue;
const labels = { name: row.name };
if (s.workersRunning !== null) out.add("homelab_netbox_workers", "gauge", "How many background workers NetBox has running.", s.workersRunning, labels);
if (s.devices !== null) out.add("homelab_netbox_devices", "gauge", "How many devices NetBox holds.", s.devices, labels);
if (s.ipAddresses !== null) out.add("homelab_netbox_ip_addresses", "gauge", "How many IP addresses NetBox holds.", s.ipAddresses, labels);
}
}
/** Each n8n, for /metrics: how many workflows there are, are switched on and have failed on their last run. Only while the reading is recent. */
async function addN8nMetrics(out: Exposition, now: Date) {
const statuses = await n8nSyncStatuses();
const names = new Map((await db.select({ id: integrations.id, name: integrations.name, enabled: integrations.enabled }).from(integrations).where(eq(integrations.type, "n8n"))).map((r) => [r.id, r]));
for (const s of await db.select().from(n8nState)) {
const row = names.get(s.integrationId);
if (!row || !row.enabled || !isN8nFresh(statuses[s.integrationId], now.getTime())) continue;
const labels = { name: row.name };
const workflows = n8nWorkflowsOf(s);
out.add("homelab_n8n_workflows", "gauge", "How many workflows n8n has.", workflows.length, labels);
out.add("homelab_n8n_workflows_active", "gauge", "How many of them are switched on.", workflows.filter((w) => w.active).length, labels);
if (s.executionsKnown) out.add("homelab_n8n_workflows_failing", "gauge", "Active workflows whose last run failed.", workflows.filter((w) => w.active && (w.lastStatus === "error" || w.lastStatus === "crashed")).length, labels);
}
}
/** Each Paperless-ngx, for /metrics: the documents, the inbox, the imports that failed and whether each component of its status is in order. Only while the reading is recent. */ /** Each Paperless-ngx, for /metrics: the documents, the inbox, the imports that failed and whether each component of its status is in order. Only while the reading is recent. */
async function addPaperlessMetrics(out: Exposition, now: Date) { async function addPaperlessMetrics(out: Exposition, now: Date) {
const statuses = await paperlessSyncStatuses(); const statuses = await paperlessSyncStatuses();
@@ -337,6 +384,9 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
await addAuthentikMetrics(out, now); await addAuthentikMetrics(out, now);
await addImmichMetrics(out, now); await addImmichMetrics(out, now);
await addPaperlessMetrics(out, now); await addPaperlessMetrics(out, now);
await addN8nMetrics(out, now);
await addNetBoxMetrics(out, now);
await addNextcloudMetrics(out, now);
await addHeartbeatMetrics(out, now); await addHeartbeatMetrics(out, now);
return out.toString(); return out.toString();
} }
+133
View File
@@ -0,0 +1,133 @@
/**
* Keeps what n8n says up to date: every five minutes each n8n integration is read (its workflows, and from the latest runs each one's last outcome and failures in the last day)
* and the state is stored. Pages and alerts are built from what is stored, so none of them talks to n8n.
*
* What is raised is an **active workflow whose last run failed** — an automation that has quietly stopped doing its job — and it clears by itself when the workflow next succeeds.
*/
import { and, eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, n8nState } from "../db/schema.js";
import { createN8nAdapter, FAILED_STATUSES, type N8nRead, type N8nWorkflow } from "../integrations/n8n/adapter.js";
import { loadIntegrationConfig } from "../integrations/loadIntegration.js";
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
export const SYNC_INTERVAL_MS = 5 * 60_000;
/** After this long without a good read, what is stored is no longer believed. */
export const STALE_AFTER_MS = 20 * 60_000;
/** A failure older than this is no longer news (a workflow that ran once, failed, and is never run again). */
export const FAILED_WITHIN_MS = 7 * 86_400_000;
const STATUS_FLAG = "n8nSync";
export interface N8nSyncStatus {
/** The last read that worked (kept when a later one failed). */
at: string;
ok: boolean;
error: string | null;
}
async function readStatuses(): Promise<Record<string, N8nSyncStatus>> {
try {
const raw = await getInternalFlag(STATUS_FLAG);
return raw ? (JSON.parse(raw) as Record<string, N8nSyncStatus>) : {};
} catch {
return {};
}
}
export async function n8nSyncStatuses(): Promise<Record<number, N8nSyncStatus>> {
return Object.fromEntries(Object.entries(await readStatuses()).map(([k, v]) => [Number(k), v]));
}
export const isN8nFresh = (status: N8nSyncStatus | undefined, nowMs: number) => !!status && status.ok && nowMs - Date.parse(status.at) <= STALE_AFTER_MS;
export type StateRow = typeof n8nState.$inferSelect;
export const workflowsOf = (row: Pick<StateRow, "workflows">): N8nWorkflow[] => {
try {
const v = JSON.parse(row.workflows);
return Array.isArray(v) ? (v as N8nWorkflow[]) : [];
} catch {
return [];
}
};
/** Writes down one read, replacing the last. */
export async function storeRead(integrationId: number, r: N8nRead, now: Date = new Date()): Promise<void> {
const values = { workflows: JSON.stringify(r.workflows), executionsKnown: r.executionsKnown, updatedAt: now.toISOString() };
await db.insert(n8nState).values({ integrationId, ...values }).onConflictDoUpdate({ target: n8nState.integrationId, set: values });
}
/** Reads one n8n and stores it. Never throws: how it went is the answer, and what was stored is left alone when it failed. */
export async function syncN8n(integrationId: number, now: Date = new Date(), read?: (config: Record<string, unknown>) => Promise<N8nRead>): Promise<N8nSyncStatus> {
let status: N8nSyncStatus;
try {
const loaded = await loadIntegrationConfig(integrationId);
if (!loaded || loaded.integration.type !== "n8n") throw new Error("not an n8n integration");
const result = await (read ? read(loaded.config) : createN8nAdapter(loaded.config as never).read(now));
await storeRead(integrationId, result, now);
status = { at: now.toISOString(), ok: true, error: null };
} catch (err) {
const previous = (await readStatuses())[String(integrationId)];
status = { at: previous?.at ?? now.toISOString(), ok: false, error: err instanceof Error ? err.message : String(err) };
console.error(`[n8n] couldn't read integration ${integrationId}:`, status.error);
}
try {
const all = await readStatuses();
all[String(integrationId)] = status;
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
} catch (err) {
console.error("[n8n] couldn't record the result:", err);
}
return status;
}
export async function syncAllN8n(now: Date = new Date()): Promise<Record<number, N8nSyncStatus>> {
const rows = await db.select({ id: integrations.id }).from(integrations).where(and(eq(integrations.type, "n8n"), eq(integrations.enabled, true)));
const out: Record<number, N8nSyncStatus> = {};
for (const r of rows) out[r.id] = await syncN8n(r.id, now);
return out;
}
export async function forgetN8n(integrationId: number): Promise<void> {
const all = await readStatuses();
if (String(integrationId) in all) {
delete all[String(integrationId)];
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
}
}
// ─── What is wrong ──────────────────────────────────────────────────────────
export interface N8nProblem {
key: string;
message: string;
severity: "warning" | "critical";
}
const ago = (iso: string, now: Date) => {
const min = Math.max(0, Math.round((now.getTime() - Date.parse(iso)) / 60_000));
return min < 90 ? `${min} min ago` : min < 2 * 1440 ? `${Math.round(min / 60)} h ago` : `${Math.round(min / 1440)} days ago`;
};
/** Pure: the active workflows whose last run failed, within the last week. A workflow that is switched off is a person's choice and says nothing. */
export function n8nProblems(id: number, name: string, s: Pick<StateRow, "workflows" | "executionsKnown">, now: Date = new Date()): N8nProblem[] {
if (!s.executionsKnown) return [];
const out: N8nProblem[] = [];
for (const w of workflowsOf(s)) {
if (!w.active || !w.lastStatus || !w.lastAt || !FAILED_STATUSES.has(w.lastStatus)) continue;
if (now.getTime() - Date.parse(w.lastAt) > FAILED_WITHIN_MS) continue;
const many = w.failed24h > 1 ? `, and ${w.failed24h} runs failed in the last day` : "";
out.push({ key: `n8n-failed:${id}:${w.id}`, severity: "warning", message: `${name}: the workflow “${w.name}” failed on its last run (${ago(w.lastAt, now)}${w.lastStatus === "crashed" ? ", crashed" : ""})${many}` });
}
return out;
}
let timer: ReturnType<typeof setInterval> | null = null;
export function initN8nSync(): void {
if (timer) return;
const pass = () => void syncAllN8n().catch((err) => console.error("[n8n] pass failed:", err));
timer = setInterval(pass, SYNC_INTERVAL_MS);
setTimeout(pass, 25_000);
console.log(`[n8n] reading n8n every ${SYNC_INTERVAL_MS / 60000} minutes`);
}
+151
View File
@@ -0,0 +1,151 @@
/**
* Keeps what NetBox says up to date: every five minutes each NetBox integration is read (its version, how many background workers are running, the counts, and the IP addresses it
* holds) and the state is stored. Pages and alerts are built from what is stored, so none of them talks to NetBox.
*
* Two uses. **Is it working**: the workers (a separate container in the Docker image) run NetBox's webhooks, scripts and jobs, and can be down while the pages load. **Is it complete**:
* the servers this app manages whose addresses are nowhere in NetBox, which is a list on the page and not an alert — documentation that is behind is not an outage.
*/
import { and, eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, netboxState } from "../db/schema.js";
import { createNetBoxAdapter, type NetBoxRead } from "../integrations/netbox/adapter.js";
import { loadIntegrationConfig } from "../integrations/loadIntegration.js";
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
export const SYNC_INTERVAL_MS = 5 * 60_000;
/** After this long without a good read, what is stored is no longer believed. */
export const STALE_AFTER_MS = 20 * 60_000;
const STATUS_FLAG = "netboxSync";
export interface NetBoxSyncStatus {
/** The last read that worked (kept when a later one failed). */
at: string;
ok: boolean;
error: string | null;
}
async function readStatuses(): Promise<Record<string, NetBoxSyncStatus>> {
try {
const raw = await getInternalFlag(STATUS_FLAG);
return raw ? (JSON.parse(raw) as Record<string, NetBoxSyncStatus>) : {};
} catch {
return {};
}
}
export async function netboxSyncStatuses(): Promise<Record<number, NetBoxSyncStatus>> {
return Object.fromEntries(Object.entries(await readStatuses()).map(([k, v]) => [Number(k), v]));
}
export const isNetBoxFresh = (status: NetBoxSyncStatus | undefined, nowMs: number) => !!status && status.ok && nowMs - Date.parse(status.at) <= STALE_AFTER_MS;
export type StateRow = typeof netboxState.$inferSelect;
/** The addresses NetBox holds; null when they could not be asked. */
export const ipsOf = (row: Pick<StateRow, "ips">): string[] | null => {
if (row.ips === null) return null;
try {
const v = JSON.parse(row.ips);
return Array.isArray(v) ? (v as string[]) : null;
} catch {
return null;
}
};
/** Writes down one read, replacing the last. */
export async function storeRead(integrationId: number, r: NetBoxRead, now: Date = new Date()): Promise<void> {
const values = {
version: r.version,
workersRunning: r.workersRunning,
devices: r.devices,
virtualMachines: r.virtualMachines,
prefixes: r.prefixes,
ipAddresses: r.ipAddresses,
ips: r.ips && JSON.stringify(r.ips),
updatedAt: now.toISOString(),
};
await db.insert(netboxState).values({ integrationId, ...values }).onConflictDoUpdate({ target: netboxState.integrationId, set: values });
}
/** Reads one NetBox and stores it. Never throws: how it went is the answer, and what was stored is left alone when it failed. */
export async function syncNetBox(integrationId: number, now: Date = new Date(), read?: (config: Record<string, unknown>) => Promise<NetBoxRead>): Promise<NetBoxSyncStatus> {
let status: NetBoxSyncStatus;
try {
const loaded = await loadIntegrationConfig(integrationId);
if (!loaded || loaded.integration.type !== "netbox") throw new Error("not a NetBox integration");
const result = await (read ? read(loaded.config) : createNetBoxAdapter(loaded.config as never).read());
await storeRead(integrationId, result, now);
status = { at: now.toISOString(), ok: true, error: null };
} catch (err) {
const previous = (await readStatuses())[String(integrationId)];
status = { at: previous?.at ?? now.toISOString(), ok: false, error: err instanceof Error ? err.message : String(err) };
console.error(`[netbox] couldn't read integration ${integrationId}:`, status.error);
}
try {
const all = await readStatuses();
all[String(integrationId)] = status;
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
} catch (err) {
console.error("[netbox] couldn't record the result:", err);
}
return status;
}
export async function syncAllNetBox(now: Date = new Date()): Promise<Record<number, NetBoxSyncStatus>> {
const rows = await db.select({ id: integrations.id }).from(integrations).where(and(eq(integrations.type, "netbox"), eq(integrations.enabled, true)));
const out: Record<number, NetBoxSyncStatus> = {};
for (const r of rows) out[r.id] = await syncNetBox(r.id, now);
return out;
}
export async function forgetNetBox(integrationId: number): Promise<void> {
const all = await readStatuses();
if (String(integrationId) in all) {
delete all[String(integrationId)];
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
}
}
// ─── What is wrong ──────────────────────────────────────────────────────────
export interface NetBoxProblem {
key: string;
message: string;
severity: "warning" | "critical";
}
/** Pure: what is wrong with one NetBox, from what is stored for it. */
export function netboxProblems(id: number, name: string, s: Pick<StateRow, "workersRunning">): NetBoxProblem[] {
if (s.workersRunning === 0) return [{ key: `netbox-workers:${id}`, severity: "warning", message: `${name}: no background worker is running — its webhooks, scripts, jobs and reports are not being run` }];
return [];
}
// ─── Is it complete ─────────────────────────────────────────────────────────
/** An address that says nothing about whether a machine is documented: loopback, link-local, and the like. */
const skip = (ip: string) => ip.startsWith("127.") || ip === "::1" || ip.startsWith("169.254.") || ip.toLowerCase().startsWith("fe80:");
/**
* Pure: the servers none of whose addresses are in NetBox. A server with no usable address is not judged (nothing is known to look for); one is "missing" only when it has
* addresses and not one of them is in NetBox's list. `excluded` takes out addresses that are the same on every host (Docker's bridge) before they are weighed.
*/
export function serversMissingFromNetBox(servers: { id: number; name: string; ips: string[] }[], netboxIps: string[], excluded: (ip: string) => boolean = () => false): { id: number; name: string; ips: string[] }[] {
const known = new Set(netboxIps.map((i) => i.trim().toLowerCase()));
const out: { id: number; name: string; ips: string[] }[] = [];
for (const s of servers) {
const own = [...new Set(s.ips.map((i) => i.trim().toLowerCase()).filter((i) => i !== "" && !skip(i) && !excluded(i)))];
if (own.length === 0) continue;
if (!own.some((i) => known.has(i))) out.push({ id: s.id, name: s.name, ips: own });
}
return out.sort((a, b) => a.name.localeCompare(b.name));
}
let timer: ReturnType<typeof setInterval> | null = null;
export function initNetBoxSync(): void {
if (timer) return;
const pass = () => void syncAllNetBox().catch((err) => console.error("[netbox] pass failed:", err));
timer = setInterval(pass, SYNC_INTERVAL_MS);
setTimeout(pass, 23_000);
console.log(`[netbox] reading NetBox every ${SYNC_INTERVAL_MS / 60000} minutes`);
}
+133
View File
@@ -0,0 +1,133 @@
/**
* Keeps what Nextcloud says up to date: every five minutes each Nextcloud integration is read (whether it is in maintenance mode or needs its database upgrading, and — with an
* administrator's app password — how much room is left, the users and files, and the app updates waiting) and the state is stored. Pages and alerts are built from what is stored, so
* none of them talks to Nextcloud.
*
* What is raised is what stops anyone signing in (maintenance mode left on, a database that wants upgrading) and the data disk running out.
*/
import { and, eq } from "drizzle-orm";
import { db } from "../db/client.js";
import { integrations, nextcloudState } from "../db/schema.js";
import { createNextcloudAdapter, type NextcloudRead } from "../integrations/nextcloud/adapter.js";
import { loadIntegrationConfig } from "../integrations/loadIntegration.js";
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
export const SYNC_INTERVAL_MS = 5 * 60_000;
/** After this long without a good read, what is stored is no longer believed. */
export const STALE_AFTER_MS = 20 * 60_000;
/** Room left for the data: a warning below this, critical below the other. */
export const FREE_WARN_BYTES = 20e9;
export const FREE_CRITICAL_BYTES = 5e9;
const STATUS_FLAG = "nextcloudSync";
export interface NextcloudSyncStatus {
/** The last read that worked (kept when a later one failed). */
at: string;
ok: boolean;
error: string | null;
}
async function readStatuses(): Promise<Record<string, NextcloudSyncStatus>> {
try {
const raw = await getInternalFlag(STATUS_FLAG);
return raw ? (JSON.parse(raw) as Record<string, NextcloudSyncStatus>) : {};
} catch {
return {};
}
}
export async function nextcloudSyncStatuses(): Promise<Record<number, NextcloudSyncStatus>> {
return Object.fromEntries(Object.entries(await readStatuses()).map(([k, v]) => [Number(k), v]));
}
export const isNextcloudFresh = (status: NextcloudSyncStatus | undefined, nowMs: number) => !!status && status.ok && nowMs - Date.parse(status.at) <= STALE_AFTER_MS;
export type StateRow = typeof nextcloudState.$inferSelect;
/** Writes down one read, replacing the last. */
export async function storeRead(integrationId: number, r: NextcloudRead, now: Date = new Date()): Promise<void> {
const values = {
version: r.version,
maintenance: r.maintenance,
needsUpgrade: r.needsUpgrade,
detail: r.detail,
freeBytes: r.freeBytes,
users: r.users,
files: r.files,
active24h: r.active24h,
appUpdates: r.appUpdates,
updatedAt: now.toISOString(),
};
await db.insert(nextcloudState).values({ integrationId, ...values }).onConflictDoUpdate({ target: nextcloudState.integrationId, set: values });
}
/** Reads one Nextcloud and stores it. Never throws: how it went is the answer, and what was stored is left alone when it failed. */
export async function syncNextcloud(integrationId: number, now: Date = new Date(), read?: (config: Record<string, unknown>) => Promise<NextcloudRead>): Promise<NextcloudSyncStatus> {
let status: NextcloudSyncStatus;
try {
const loaded = await loadIntegrationConfig(integrationId);
if (!loaded || loaded.integration.type !== "nextcloud") throw new Error("not a Nextcloud integration");
const result = await (read ? read(loaded.config) : createNextcloudAdapter(loaded.config as never).read());
await storeRead(integrationId, result, now);
status = { at: now.toISOString(), ok: true, error: null };
} catch (err) {
const previous = (await readStatuses())[String(integrationId)];
status = { at: previous?.at ?? now.toISOString(), ok: false, error: err instanceof Error ? err.message : String(err) };
console.error(`[nextcloud] couldn't read integration ${integrationId}:`, status.error);
}
try {
const all = await readStatuses();
all[String(integrationId)] = status;
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
} catch (err) {
console.error("[nextcloud] couldn't record the result:", err);
}
return status;
}
export async function syncAllNextcloud(now: Date = new Date()): Promise<Record<number, NextcloudSyncStatus>> {
const rows = await db.select({ id: integrations.id }).from(integrations).where(and(eq(integrations.type, "nextcloud"), eq(integrations.enabled, true)));
const out: Record<number, NextcloudSyncStatus> = {};
for (const r of rows) out[r.id] = await syncNextcloud(r.id, now);
return out;
}
export async function forgetNextcloud(integrationId: number): Promise<void> {
const all = await readStatuses();
if (String(integrationId) in all) {
delete all[String(integrationId)];
await setInternalFlag(STATUS_FLAG, JSON.stringify(all));
}
}
// ─── What is wrong ──────────────────────────────────────────────────────────
export interface NextcloudProblem {
key: string;
message: string;
severity: "warning" | "critical";
}
/** In the same units the pages show (binary, written GB), so an alert and the tile beside it agree. */
const gb = (bytes: number) => `${(bytes / 1024 ** 3).toFixed(bytes >= 100 * 1024 ** 3 ? 0 : 1)} GB`;
/** Pure: what is wrong with one Nextcloud, from what is stored for it. */
export function nextcloudProblems(id: number, name: string, s: Pick<StateRow, "maintenance" | "needsUpgrade" | "freeBytes">): NextcloudProblem[] {
const out: NextcloudProblem[] = [];
if (s.needsUpgrade) out.push({ key: `nextcloud-upgrade:${id}`, severity: "critical", message: `${name}: its database needs upgrading — nobody can use it until "occ upgrade" has been run` });
if (s.maintenance) out.push({ key: `nextcloud-maintenance:${id}`, severity: "warning", message: `${name} is in maintenance mode — nobody can sign in ("occ maintenance:mode --off" ends it)` });
if (s.freeBytes !== null && s.freeBytes < FREE_WARN_BYTES) {
out.push({ key: `nextcloud-disk:${id}`, severity: s.freeBytes < FREE_CRITICAL_BYTES ? "critical" : "warning", message: `${name}: only ${gb(s.freeBytes)} is left where the data is kept` });
}
return out;
}
let timer: ReturnType<typeof setInterval> | null = null;
export function initNextcloudSync(): void {
if (timer) return;
const pass = () => void syncAllNextcloud().catch((err) => console.error("[nextcloud] pass failed:", err));
timer = setInterval(pass, SYNC_INTERVAL_MS);
setTimeout(pass, 21_000);
console.log(`[nextcloud] reading Nextcloud every ${SYNC_INTERVAL_MS / 60000} minutes`);
}
+2 -1
View File
@@ -122,7 +122,8 @@ export interface PaperlessProblem {
severity: "warning" | "critical"; severity: "warning" | "critical";
} }
const gb = (bytes: number) => `${(bytes / 1e9).toFixed(bytes >= 1e11 ? 0 : 1)} GB`; /** In the same units the pages show (binary, written GB), so an alert and the tile beside it agree. */
const gb = (bytes: number) => `${(bytes / 1024 ** 3).toFixed(bytes >= 100 * 1024 ** 3 ? 0 : 1)} GB`;
const WORDS: Record<string, string> = { database: "the database", redis: "Redis", celery: "the worker that consumes documents", index: "the search index", classifier: "the classifier", sanity_check: "the sanity check" }; const WORDS: Record<string, string> = { database: "the database", redis: "Redis", celery: "the worker that consumes documents", index: "the search index", classifier: "the classifier", sanity_check: "the sanity check" };
/** Pure: what is wrong with one Paperless, from what is stored for it. */ /** Pure: what is wrong with one Paperless, from what is stored for it. */
+1 -1
View File
@@ -159,7 +159,7 @@ describe("what is wrong", () => {
assert.deepEqual(keys(row({ diskPercent: 89.9 })), []); assert.deepEqual(keys(row({ diskPercent: 89.9 })), []);
assert.deepEqual(keys(row({ diskPercent: 90 })), ["immich-disk:1=warning"]); assert.deepEqual(keys(row({ diskPercent: 90 })), ["immich-disk:1=warning"]);
assert.deepEqual(keys(row({ diskPercent: 95 })), ["immich-disk:1=critical"]); assert.deepEqual(keys(row({ diskPercent: 95 })), ["immich-disk:1=critical"]);
assert.match(im.immichProblems(1, "Photos", row({ diskPercent: 96, diskUsed: 3.84e12 }))[0].message, /96 % full \(3840 GB of 4000 GB\)/); assert.match(im.immichProblems(1, "Photos", row({ diskPercent: 96, diskUsed: 3.84e12 }))[0].message, /96 % full \(3576 GB of 3725 GB\)/);
}); });
test("failed jobs are one alert when there are ten or more together, naming the worst queues", () => { test("failed jobs are one alert when there are ten or more together, naming the worst queues", () => {
+262
View File
@@ -0,0 +1,262 @@
/**
* n8n: reading what its API says (workflows in pages, the latest runs), working out each workflow's last outcome, what is stored, what is wrong, the health pass, the alerts, the
* page's route and /metrics.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer as createHttpServer, type Server } from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-n8n-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, integrations, n8nState } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const adapter = await import("../src/integrations/n8n/adapter.js");
const n8 = await import("../src/services/n8nSync.js");
const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js");
const { subjectOfConditionKey } = await import("../src/services/maintenance.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { n8nRouter } = await import("../src/routes/n8n.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { setInternalFlag } = await import("../src/services/settingsStore.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const MIN = 60_000;
const DAY = 86_400_000;
const NOW = new Date("2026-10-20T12:00:00Z");
const ago = (min: number, from: Date = NOW) => new Date(from.getTime() - min * MIN).toISOString();
type Read = import("../src/integrations/n8n/adapter.js").N8nRead;
type Workflow = import("../src/integrations/n8n/adapter.js").N8nWorkflow;
const wf = (id: string, name: string, extra: Partial<Workflow> = {}): Workflow => ({ id, name, active: true, lastStatus: "success", lastAt: ago(30), failed24h: 0, ...extra });
const read = (workflows: Workflow[] = [wf("1", "Backup notifier")], extra: Partial<Read> = {}): Read => ({ workflows, executionsKnown: true, ...extra });
describe("what n8n's API says", () => {
test("workflows: id, name and whether it is switched on; ones without an id are left out", () => {
const w = adapter.parseWorkflows({ data: [{ id: "a1", name: "Notify", active: true }, { id: 7, name: "Old", active: false }, { name: "no id" }] });
assert.deepEqual(w, [{ id: "a1", name: "Notify", active: true }, { id: "7", name: "Old", active: false }]);
assert.deepEqual(adapter.parseWorkflows(null), []);
});
test("each workflow's last outcome is the newest finished run; a run still going is not an outcome; failures are counted for the last day", () => {
const workflows = [{ id: "1", name: "A", active: true }, { id: "2", name: "B", active: true }, { id: "3", name: "C", active: false }];
const runs = {
data: [
{ workflowId: "1", status: "running", startedAt: ago(1) }, // still going: the one before it counts
{ workflowId: "1", status: "error", stoppedAt: ago(20) },
{ workflowId: "1", status: "success", stoppedAt: ago(200) },
{ workflowId: "1", status: "crashed", stoppedAt: ago(300) },
{ workflowId: "1", status: "error", stoppedAt: new Date(NOW.getTime() - 2 * DAY).toISOString() }, // older than a day
{ workflowId: "2", status: "success", stoppedAt: ago(50) },
{ workflowId: "9", status: "error", stoppedAt: ago(10) }, // a workflow that is not in the list
],
};
const s = adapter.summariseRuns(workflows, runs, NOW);
assert.deepEqual(s.map((x) => [x.id, x.lastStatus, x.failed24h]), [["1", "error", 2], ["2", "success", 0], ["3", null, 0]]);
assert.equal(s[0].lastAt, ago(20));
assert.equal(s[2].lastAt, null);
});
test("order does not matter: the newest wins whichever way the list comes", () => {
const w = [{ id: "1", name: "A", active: true }];
const runs = [{ workflowId: "1", status: "success", stoppedAt: ago(5) }, { workflowId: "1", status: "error", stoppedAt: ago(60) }];
assert.equal(adapter.summariseRuns(w, { data: runs }, NOW)[0].lastStatus, "success");
assert.equal(adapter.summariseRuns(w, { data: [...runs].reverse() }, NOW)[0].lastStatus, "success");
});
describe("the adapter against an n8n that answers", () => {
let server: Server;
let url = "";
let executionsStatus = 200;
const seen: { path: string; key: string | undefined; cursor: string | null }[] = [];
before(async () => {
server = createHttpServer((req, res) => {
const u = new URL(req.url ?? "", "http://x");
seen.push({ path: u.pathname, key: req.headers["x-n8n-api-key"] as string | undefined, cursor: u.searchParams.get("cursor") });
const send = (code: number, body: unknown) => {
res.writeHead(code, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (req.headers["x-n8n-api-key"] !== "key") return send(401, {});
if (u.pathname === "/api/v1/workflows") {
return u.searchParams.get("cursor") === "page2"
? send(200, { data: [{ id: "3", name: "Third", active: true }], nextCursor: null })
: send(200, { data: [{ id: "1", name: "First", active: true }, { id: "2", name: "Second", active: false }], nextCursor: "page2" });
}
if (u.pathname === "/api/v1/executions") {
if (executionsStatus !== 200) return send(executionsStatus, {});
return send(200, { data: [{ workflowId: "1", status: "error", stoppedAt: new Date(Date.now() - 5 * MIN).toISOString() }, { workflowId: "3", status: "success", stoppedAt: new Date(Date.now() - 9 * MIN).toISOString() }] });
}
send(404, {});
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());
test("it follows the cursor through every workflow, sends the key, and works out each one's last run", async () => {
const r = await adapter.createN8nAdapter({ url: `${url}/`, apiKey: "key" }).read();
assert.deepEqual(r.workflows.map((w) => [w.id, w.active, w.lastStatus, w.failed24h]), [["1", true, "error", 1], ["2", false, null, 0], ["3", true, "success", 0]]);
assert.equal(r.executionsKnown, true);
assert.deepEqual(seen.filter((s) => s.path === "/api/v1/workflows").map((s) => s.cursor), [null, "page2"]);
assert.ok(seen.every((s) => s.key === "key"));
});
test("when the runs cannot be read the workflows are still there, and their last runs are unknown, not fine", async () => {
executionsStatus = 500;
const r = await adapter.createN8nAdapter({ url, apiKey: "key" }).read();
assert.equal(r.executionsKnown, false);
assert.equal(r.workflows.length, 3);
assert.ok(r.workflows.every((w) => w.lastStatus === null));
executionsStatus = 200;
});
test("a refused key and a dead address are said plainly", async () => {
const bad = await adapter.createN8nAdapter({ url, apiKey: "nope" }).ping();
assert.equal(bad.ok, false);
assert.match(bad.error ?? "", /did not accept the API key/);
assert.equal((await adapter.createN8nAdapter({ url, apiKey: "key" }).ping()).ok, true);
assert.equal((await adapter.createN8nAdapter({ url: "http://127.0.0.1:1", apiKey: "key" }).ping()).ok, false);
});
});
});
describe("what is stored", () => {
let id = 0;
before(async () => {
id = (await db.insert(integrations).values({ type: "n8n", name: "Flows", baseUrl: "http://10.9.6.1:5678", config: JSON.stringify({ url: "http://10.9.6.1:5678", apiKey: "key" }) }).returning({ id: integrations.id }))[0].id;
});
const state = async () => (await db.select().from(n8nState).where(eq(n8nState.integrationId, id)))[0];
test("the state is replaced at each read", async () => {
await n8.storeRead(id, read([wf("1", "A"), wf("2", "B")]), new Date());
assert.deepEqual(n8.workflowsOf(await state()).map((w) => w.name), ["A", "B"]);
await n8.storeRead(id, read([wf("3", "C")], { executionsKnown: false }), new Date());
const s = await state();
assert.deepEqual([n8.workflowsOf(s).map((w) => w.name), s.executionsKnown], [["C"], false]);
});
test("a read that failed is said, and what was stored is left alone", async () => {
const ok = await n8.syncN8n(id, new Date(), async () => read([wf("1", "Kept")]));
assert.equal(ok.ok, true);
const failed = await n8.syncN8n(id, new Date(), async () => {
throw new Error("connection refused");
});
assert.deepEqual([failed.ok, failed.error], [false, "connection refused"]);
assert.equal(failed.at, ok.at);
assert.deepEqual(n8.workflowsOf(await state()).map((w) => w.name), ["Kept"]);
assert.equal(n8.isN8nFresh((await n8.n8nSyncStatuses())[id], Date.now()), false);
});
});
describe("what is wrong", () => {
const row = (workflows: Workflow[], executionsKnown = true) => ({ workflows: JSON.stringify(workflows), executionsKnown });
const keys = (r: ReturnType<typeof row>) => n8.n8nProblems(1, "Flows", r, NOW).map((p) => `${p.key}=${p.severity}`);
test("an active workflow whose last run failed is a warning, one alert for each; it says when, and if it crashed or failed more than once", () => {
const p = n8.n8nProblems(1, "Flows", row([wf("a", "Backup notifier", { lastStatus: "error", lastAt: ago(20), failed24h: 3 }), wf("b", "Mailer", { lastStatus: "crashed", lastAt: ago(300), failed24h: 1 })]), NOW);
assert.deepEqual(p.map((x) => [x.key, x.severity]), [["n8n-failed:1:a", "warning"], ["n8n-failed:1:b", "warning"]]);
assert.equal(p[0].message, "Flows: the workflow “Backup notifier” failed on its last run (20 min ago), and 3 runs failed in the last day");
assert.match(p[1].message, /“Mailer” failed on its last run \(5 h ago, crashed\)$/);
});
test("one that succeeded since, one that is switched off, one whose failure is over a week old and one never seen run say nothing", () => {
assert.deepEqual(
keys(row([wf("a", "A", { lastStatus: "success" }), wf("b", "B", { active: false, lastStatus: "error", lastAt: ago(10) }), wf("c", "C", { lastStatus: "error", lastAt: new Date(NOW.getTime() - 8 * DAY).toISOString() }), wf("d", "D", { lastStatus: null, lastAt: null })])),
[],
);
});
test("when the runs could not be read nothing is claimed", () => {
assert.deepEqual(keys(row([wf("a", "A", { lastStatus: "error", lastAt: ago(5) })], false)), []);
});
});
describe("the health pass, the page and /metrics", () => {
let nId = 0;
let app: Server;
let base = "";
const fresh = () => setInternalFlag("n8nSync", JSON.stringify({ [nId]: { at: new Date().toISOString(), ok: true, error: null } }));
const stale = () => setInternalFlag("n8nSync", JSON.stringify({ [nId]: { at: new Date(Date.now() - 30 * MIN).toISOString(), ok: true, error: null } }));
before(async () => {
await db.insert(users).values([{ oidcSub: "viewer", name: "Vic", role: "viewer" }]);
nId = (await db.insert(integrations).values({ type: "n8n", name: "Automations", baseUrl: "http://10.9.6.2:5678", config: JSON.stringify({ url: "http://10.9.6.2:5678", apiKey: "key" }) }).returning({ id: integrations.id }))[0].id;
const t = new Date();
await n8.storeRead(nId, read([wf("a", "Backup notifier", { lastStatus: "error", lastAt: ago(20, t), failed24h: 2 }), wf("b", "Fine", { lastAt: ago(5, t) }), wf("c", "Off", { active: false })]), t);
await fresh();
const a = express();
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "viewer" } };
next();
});
a.use("/api/n8n", n8nRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/n8n`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("the health pass judges it from what is stored, and holds it when the last read is old", async () => {
let { snapshot, held } = await collectSnapshot();
assert.deepEqual(snapshot.n8n?.map((n) => n.integrationId), [nId]);
const list = evaluateHealth(snapshot, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, Date.now(), { skipServers: true });
const c = list.find((x) => x.key === `n8n-failed:${nId}:a`)!;
assert.deepEqual([c.severity, c.source], ["warning", `n8n:${nId}`]);
assert.equal(subjectOfConditionKey(c.key), `integration:${nId}`, "maintenance on it silences its alerts");
assert.equal(list.some((x) => x.key.startsWith(`n8n-failed:${nId}:b`) || x.key.startsWith(`n8n-failed:${nId}:c`)), false);
await stale();
({ snapshot, held } = await collectSnapshot());
assert.deepEqual([snapshot.n8n ?? [], held.has(`n8n:${nId}`)], [[], true]);
await fresh();
});
test("the Alerts page shows it in the Automation kind, with where to look", async () => {
invalidateAlertsCache();
const report = await getAlerts(true);
const a = report.alerts.find((x) => x.id === `automation:n8n-failed:${nId}:a`)!;
assert.deepEqual([a.severity, a.category, a.source, a.link, a.target?.type, a.target?.id], ["warning", "automation", "n8n", "/n8n", "integration", nId]);
});
test("anyone signed in sees each n8n and its workflows; an old reading says nothing is wrong", async () => {
const body = (await (await fetch(base)).json()) as { n8ns: { id: number; fresh: boolean; state: { workflows: { name: string; lastStatus: string | null }[]; executionsKnown: boolean } | null; problems: unknown[] }[] };
const n = body.n8ns.find((x) => x.id === nId)!;
assert.deepEqual([n.fresh, n.state?.executionsKnown, n.state?.workflows.map((w) => [w.name, w.lastStatus])], [true, true, [["Backup notifier", "error"], ["Fine", "success"], ["Off", "success"]]]);
assert.equal(n.problems.length, 1);
await stale();
const old = (await (await fetch(base)).json()) as typeof body;
assert.deepEqual(old.n8ns.find((x) => x.id === nId)!.problems, []);
await fresh();
});
test("/metrics has the workflows, those switched on and those failing — only while the reading is recent", async () => {
let text = await buildMetrics(new Date());
assert.match(text, /homelab_n8n_workflows\{name="Automations"\} 3/);
assert.match(text, /homelab_n8n_workflows_active\{name="Automations"\} 2/);
assert.match(text, /homelab_n8n_workflows_failing\{name="Automations"\} 1/);
await stale();
text = await buildMetrics(new Date());
assert.doesNotMatch(text, /homelab_n8n/);
});
});
+257
View File
@@ -0,0 +1,257 @@
/**
* NetBox: reading what its API says (status, counts, the addresses in pages), which servers are not in it, what is stored, what is wrong, the health pass, the alerts, the page's
* route and /metrics.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer as createHttpServer, type Server } from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-netbox-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, integrations, netboxState } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const adapter = await import("../src/integrations/netbox/adapter.js");
const nb = await import("../src/services/netboxSync.js");
const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js");
const { subjectOfConditionKey } = await import("../src/services/maintenance.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { netboxRouter } = await import("../src/routes/netbox.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { setInternalFlag } = await import("../src/services/settingsStore.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const MIN = 60_000;
type Read = import("../src/integrations/netbox/adapter.js").NetBoxRead;
const read = (extra: Partial<Read> = {}): Read => ({ version: "4.2.5", workersRunning: 1, devices: 12, virtualMachines: 30, prefixes: 8, ipAddresses: 4, ips: ["10.0.0.5", "10.0.0.6", "10.0.0.7", "fd00::1"], ...extra });
describe("what NetBox's API says", () => {
test("the status: version and workers; the workers are null when it does not say", () => {
assert.deepEqual(adapter.parseStatus({ "netbox-version": "4.2.5", "rq-workers-running": 2 }), { version: "4.2.5", workersRunning: 2 });
assert.deepEqual(adapter.parseStatus({ "netbox-version": "4.2.5", "rq-workers-running": 0 }), { version: "4.2.5", workersRunning: 0 });
assert.deepEqual(adapter.parseStatus({}), { version: null, workersRunning: null });
});
test("counts, and addresses without their prefix length", () => {
assert.equal(adapter.parseCount({ count: 12, results: [] }), 12);
assert.equal(adapter.parseCount({}), null);
assert.deepEqual(adapter.parseAddresses({ results: [{ address: "10.0.0.5/24" }, { address: "FD00::1/64" }, { address: "" }, {}] }), ["10.0.0.5", "fd00::1"]);
});
test("a v2 token (nbt_…) goes as a Bearer token, an older one as Token", () => {
assert.equal(adapter.authorization("abc123"), "Token abc123");
assert.equal(adapter.authorization("nbt_key.secret"), "Bearer nbt_key.secret");
});
describe("the adapter against a NetBox that answers", () => {
let server: Server;
let url = "";
const seen: string[] = [];
const TOTAL = 1500;
before(async () => {
server = createHttpServer((req, res) => {
const u = new URL(req.url ?? "", "http://x");
seen.push(`${req.headers.authorization} ${u.pathname}${u.search}`);
const send = (code: number, body: unknown) => {
res.writeHead(code, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (req.headers.authorization !== "Token tok") return send(403, {});
if (u.pathname === "/api/status/") return send(200, { "netbox-version": "4.2.5", "rq-workers-running": 0 });
if (u.pathname === "/api/dcim/devices/") return send(200, { count: 12, results: [] });
if (u.pathname === "/api/virtualization/virtual-machines/") return send(500, {}); // the rest is worth having, not worth failing for
if (u.pathname === "/api/ipam/prefixes/") return send(200, { count: 8, results: [] });
if (u.pathname === "/api/ipam/ip-addresses/") {
if (u.searchParams.get("limit") === "1") return send(200, { count: TOTAL, results: [] });
const offset = Number(u.searchParams.get("offset") ?? 0);
const limit = Number(u.searchParams.get("limit") ?? 50);
const results = Array.from({ length: Math.max(0, Math.min(limit, TOTAL - offset)) }, (_, i) => ({ address: `10.1.${Math.floor((offset + i) / 250)}.${((offset + i) % 250) + 1}/24` }));
return send(200, { count: TOTAL, results });
}
send(404, {});
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());
test("it reads the counts, the workers and every address in pages; a count that cannot be read is null", async () => {
const r = await adapter.createNetBoxAdapter({ url: `${url}/`, token: "tok" }).read();
assert.deepEqual([r.version, r.workersRunning, r.devices, r.virtualMachines, r.prefixes, r.ipAddresses], ["4.2.5", 0, 12, null, 8, 1500]);
assert.equal(r.ips?.length, 1500, "both pages");
assert.ok(r.ips?.includes("10.1.0.1") && r.ips?.includes("10.1.5.250"));
const pages = seen.filter((s) => s.includes("brief=1")).map((s) => /offset=(\d+)/.exec(s)![1]);
assert.deepEqual(pages, ["0", "1000"]);
});
test("a refused token and a dead address are said plainly", async () => {
const bad = await adapter.createNetBoxAdapter({ url, token: "nope" }).ping();
assert.equal(bad.ok, false);
assert.match(bad.error ?? "", /did not accept the token/);
assert.equal((await adapter.createNetBoxAdapter({ url, token: "tok" }).ping()).ok, true);
assert.equal((await adapter.createNetBoxAdapter({ url: "http://127.0.0.1:1", token: "tok" }).ping()).ok, false);
});
});
});
describe("which servers are not in NetBox", () => {
const s = (id: number, name: string, ips: string[]) => ({ id, name, ips });
test("a server none of whose addresses are in NetBox is listed; one with any address in it is not", () => {
const out = nb.serversMissingFromNetBox([s(1, "web", ["10.0.0.5", "172.17.0.1"]), s(2, "nas", ["10.0.0.9"]), s(3, "pi", ["10.0.0.7"])], ["10.0.0.5", "10.0.0.7"]);
assert.deepEqual(out.map((x) => x.name), ["nas"]);
});
test("a server that reports no usable address is not judged: loopback, link-local and nothing are nothing to look for", () => {
const out = nb.serversMissingFromNetBox([s(1, "a", []), s(2, "b", ["127.0.0.1", "169.254.3.4", "fe80::1", "::1"]), s(3, "c", ["10.0.0.99"])], ["10.0.0.5"]);
assert.deepEqual(out.map((x) => x.name), ["c"]);
});
test("an address that is the same on every host (Docker's bridge) is taken out before it is weighed, and the order is by name", () => {
const excluded = (ip: string) => ip.startsWith("172.17.");
const out = nb.serversMissingFromNetBox([s(1, "zeta", ["172.17.0.1"]), s(2, "beta", ["172.17.0.1", "10.0.0.2"]), s(3, "alpha", ["10.0.0.1"])], ["10.0.0.2"], excluded);
assert.deepEqual(out.map((x) => [x.name, x.ips]), [["alpha", ["10.0.0.1"]]], "zeta has only the bridge address, beta is in NetBox");
});
test("case and spaces do not matter", () => {
assert.deepEqual(nb.serversMissingFromNetBox([s(1, "v6", [" FD00::1 "])], ["fd00::1"]), []);
});
});
describe("what is stored", () => {
let id = 0;
before(async () => {
id = (await db.insert(integrations).values({ type: "netbox", name: "IPAM", baseUrl: "http://10.9.7.1", config: JSON.stringify({ url: "http://10.9.7.1", token: "tok" }) }).returning({ id: integrations.id }))[0].id;
});
const state = async () => (await db.select().from(netboxState).where(eq(netboxState.integrationId, id)))[0];
test("the state is replaced at each read; addresses that could not be listed stay null, not empty", async () => {
await nb.storeRead(id, read(), new Date());
assert.deepEqual(nb.ipsOf(await state()), ["10.0.0.5", "10.0.0.6", "10.0.0.7", "fd00::1"]);
await nb.storeRead(id, read({ ips: null, workersRunning: null }), new Date());
const s = await state();
assert.deepEqual([nb.ipsOf(s), s.workersRunning], [null, null]);
});
test("a read that failed is said, and what was stored is left alone", async () => {
const ok = await nb.syncNetBox(id, new Date(), async () => read({ version: "4.2.6" }));
assert.equal(ok.ok, true);
const failed = await nb.syncNetBox(id, new Date(), async () => {
throw new Error("connection refused");
});
assert.deepEqual([failed.ok, failed.error], [false, "connection refused"]);
assert.equal(failed.at, ok.at);
assert.equal((await state()).version, "4.2.6");
assert.equal(nb.isNetBoxFresh((await nb.netboxSyncStatuses())[id], Date.now()), false);
});
});
describe("what is wrong", () => {
test("no worker running is a warning; one running, or not said, is nothing", () => {
assert.deepEqual(nb.netboxProblems(1, "IPAM", { workersRunning: 0 }).map((p) => [p.key, p.severity]), [["netbox-workers:1", "warning"]]);
assert.match(nb.netboxProblems(1, "IPAM", { workersRunning: 0 })[0].message, /no background worker is running/);
assert.deepEqual(nb.netboxProblems(1, "IPAM", { workersRunning: 1 }), []);
assert.deepEqual(nb.netboxProblems(1, "IPAM", { workersRunning: null }), []);
});
});
describe("the health pass, the page and /metrics", () => {
let nbId = 0;
let app: Server;
let base = "";
const ids: Record<string, number> = {};
const fresh = () => setInternalFlag("netboxSync", JSON.stringify({ [nbId]: { at: new Date().toISOString(), ok: true, error: null } }));
const stale = () => setInternalFlag("netboxSync", JSON.stringify({ [nbId]: { at: new Date(Date.now() - 30 * MIN).toISOString(), ok: true, error: null } }));
before(async () => {
await db.insert(users).values([{ oidcSub: "viewer", name: "Vic", role: "viewer" }]);
nbId = (await db.insert(integrations).values({ type: "netbox", name: "NetBox", baseUrl: "http://10.9.7.2", config: JSON.stringify({ url: "http://10.9.7.2", token: "tok" }) }).returning({ id: integrations.id }))[0].id;
const mk = async (name: string, ips: string[]) => (await db.insert(servers).values({ name, apiTokenHash: name, apiTokenPrefix: name, ipAddresses: JSON.stringify(ips) }).returning({ id: servers.id }))[0].id;
ids.web = await mk("web", ["10.0.0.5"]);
ids.nas = await mk("nas", ["10.0.0.99"]);
await nb.storeRead(nbId, read({ workersRunning: 0, ipAddresses: 4 }), new Date());
await fresh();
const a = express();
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "viewer" } };
next();
});
a.use("/api/netbox", netboxRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/netbox`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("the health pass judges it from what is stored, and holds it when the last read is old", async () => {
let { snapshot, held } = await collectSnapshot();
assert.deepEqual(snapshot.netbox?.map((n) => n.integrationId), [nbId]);
const list = evaluateHealth(snapshot, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, Date.now(), { skipServers: true });
const c = list.find((x) => x.key === `netbox-workers:${nbId}`)!;
assert.deepEqual([c.severity, c.source], ["warning", `netbox:${nbId}`]);
assert.equal(subjectOfConditionKey(c.key), `integration:${nbId}`, "maintenance on it silences its alerts");
await stale();
({ snapshot, held } = await collectSnapshot());
assert.deepEqual([snapshot.netbox ?? [], held.has(`netbox:${nbId}`)], [[], true]);
await fresh();
});
test("the Alerts page shows it under Services, with where to look", async () => {
invalidateAlertsCache();
const report = await getAlerts(true);
const a = report.alerts.find((x) => x.id === `services:netbox-workers:${nbId}`)!;
assert.deepEqual([a.severity, a.category, a.source, a.link, a.target?.type, a.target?.id], ["warning", "services", "NetBox", "/netbox", "integration", nbId]);
});
test("anyone signed in sees each NetBox, the counts and which servers are not in it; an old reading says nothing is wrong", async () => {
const body = (await (await fetch(base)).json()) as { netboxes: { id: number; fresh: boolean; state: { workersRunning: number; missing: { name: string }[] | null; truncated: boolean } | null; problems: unknown[] }[] };
const n = body.netboxes.find((x) => x.id === nbId)!;
assert.deepEqual([n.fresh, n.state?.workersRunning, n.state?.missing?.map((m) => m.name), n.state?.truncated], [true, 0, ["nas"], false]);
assert.equal(n.problems.length, 1);
await stale();
const old = (await (await fetch(base)).json()) as typeof body;
assert.deepEqual(old.netboxes.find((x) => x.id === nbId)!.problems, []);
await fresh();
});
test("when NetBox has more addresses than were read, the page says what is missing may be in the rest; when they could not be listed, nothing is claimed", async () => {
await nb.storeRead(nbId, read({ ipAddresses: 9000 }), new Date());
const body = (await (await fetch(base)).json()) as { netboxes: { id: number; state: { truncated: boolean; missing: unknown } }[] };
assert.equal(body.netboxes.find((x) => x.id === nbId)!.state.truncated, true);
await nb.storeRead(nbId, read({ ips: null }), new Date());
const none = (await (await fetch(base)).json()) as typeof body;
assert.equal(none.netboxes.find((x) => x.id === nbId)!.state.missing, null);
await nb.storeRead(nbId, read({ workersRunning: 0, ipAddresses: 4 }), new Date());
});
test("/metrics has the workers and the counts — only while the reading is recent", async () => {
let text = await buildMetrics(new Date());
assert.match(text, /homelab_netbox_workers\{name="NetBox"\} 0/);
assert.match(text, /homelab_netbox_devices\{name="NetBox"\} 12/);
assert.match(text, /homelab_netbox_ip_addresses\{name="NetBox"\} 4/);
await stale();
text = await buildMetrics(new Date());
assert.doesNotMatch(text, /homelab_netbox/);
});
});
+244
View File
@@ -0,0 +1,244 @@
/**
* Nextcloud: reading what its API says (status.php, and the monitoring app with an administrator's app password), what is stored, what is wrong, the health pass, the alerts, the
* page's route and /metrics.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer as createHttpServer, type Server } from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-nextcloud-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, integrations, nextcloudState } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const adapter = await import("../src/integrations/nextcloud/adapter.js");
const nc = await import("../src/services/nextcloudSync.js");
const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js");
const { subjectOfConditionKey } = await import("../src/services/maintenance.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { nextcloudRouter } = await import("../src/routes/nextcloud.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { setInternalFlag } = await import("../src/services/settingsStore.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const MIN = 60_000;
type Read = import("../src/integrations/nextcloud/adapter.js").NextcloudRead;
const read = (extra: Partial<Read> = {}): Read => ({ version: "31.0.2", maintenance: false, needsUpgrade: false, detail: true, freeBytes: 500e9, users: 4, files: 120000, active24h: 2, appUpdates: 1, ...extra });
const INFO = { ocs: { meta: { status: "ok" }, data: { nextcloud: { system: { version: "31.0.2.1", freespace: 412e9, apps: { num_installed: 40, num_updates_available: 3, app_updates: { calendar: "5.1.0" } } }, storage: { num_users: 4, num_files: 120000 } }, activeUsers: { last5minutes: 1, last24hours: 2 } } } };
describe("what Nextcloud's API says", () => {
test("status.php: the version, maintenance mode and a database that wants upgrading", () => {
assert.deepEqual(adapter.parseStatus({ installed: true, maintenance: false, needsDbUpgrade: false, version: "31.0.2.1", versionstring: "31.0.2" }), { version: "31.0.2", maintenance: false, needsUpgrade: false, installed: true });
const m = adapter.parseStatus({ installed: true, maintenance: 1, needsDbUpgrade: "1", version: "31.0.2.1" });
assert.deepEqual([m.maintenance, m.needsUpgrade, m.version], [true, true, "31.0.2.1"]);
assert.equal(adapter.parseStatus({ installed: false }).installed, false);
assert.deepEqual(adapter.parseStatus(null), { version: null, maintenance: false, needsUpgrade: false, installed: true });
});
test("the monitoring app: room left, users, files, who was active and the app updates", () => {
assert.deepEqual(adapter.parseServerInfo(INFO), { freeBytes: 412e9, users: 4, files: 120000, active24h: 2, appUpdates: 3 });
assert.equal(adapter.parseServerInfo({ ocs: { meta: {} } }), null, "no data is not an empty answer");
assert.equal(adapter.parseServerInfo({ ocs: { data: { nextcloud: { system: { apps: { app_updates: { a: "1", b: "2" } } } } } } })?.appUpdates, 2, "counted from the list when no count is given");
assert.equal(adapter.parseServerInfo({ ocs: { data: { nextcloud: { system: {} } } } })?.appUpdates, null);
});
describe("the adapter against a Nextcloud that answers", () => {
let server: Server;
let url = "";
let infoStatus = 200;
let installed = true;
const seen: { path: string; auth: string | undefined; ocs: string | undefined }[] = [];
before(async () => {
server = createHttpServer((req, res) => {
const path = (req.url ?? "").split("?")[0];
seen.push({ path, auth: req.headers.authorization, ocs: req.headers["ocs-apirequest"] as string | undefined });
const send = (code: number, body: unknown) => {
res.writeHead(code, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (path === "/status.php") return send(200, { installed, maintenance: true, needsDbUpgrade: false, versionstring: "31.0.2" });
if (path === "/ocs/v2.php/apps/serverinfo/api/v1/info") return infoStatus === 200 ? send(200, INFO) : send(infoStatus, {});
send(404, {});
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());
test("without a login only the status is read, and nothing is sent to the monitoring app", async () => {
seen.length = 0;
const r = await adapter.createNextcloudAdapter({ url: `${url}/` }).read();
assert.deepEqual([r.version, r.maintenance, r.detail, r.freeBytes], ["31.0.2", true, false, null]);
assert.deepEqual(seen.map((s) => s.path), ["/status.php"]);
});
test("with an administrator's app password the monitoring app is asked, the way Nextcloud wants it", async () => {
seen.length = 0;
const r = await adapter.createNextcloudAdapter({ url, username: "admin", appPassword: "pw" }).read();
assert.deepEqual([r.detail, r.freeBytes, r.users, r.active24h, r.appUpdates], [true, 412e9, 4, 2, 3]);
const info = seen.find((s) => s.path.includes("serverinfo"))!;
assert.equal(info.auth, `Basic ${Buffer.from("admin:pw").toString("base64")}`);
assert.equal(info.ocs, "true");
});
test("a login that is refused (or an app that is off) costs only the detail, not the status", async () => {
infoStatus = 401;
const r = await adapter.createNextcloudAdapter({ url, username: "admin", appPassword: "wrong" }).read();
assert.deepEqual([r.version, r.detail, r.freeBytes], ["31.0.2", false, null]);
const ping = await adapter.createNextcloudAdapter({ url, username: "admin", appPassword: "wrong" }).ping();
assert.equal(ping.ok, false);
assert.match(ping.error ?? "", /did not accept the login/);
infoStatus = 200;
});
test("one that says it is not installed, and a dead address, are said", async () => {
installed = false;
await assert.rejects(() => adapter.createNextcloudAdapter({ url }).read(), /not installed/);
installed = true;
assert.equal((await adapter.createNextcloudAdapter({ url: "http://127.0.0.1:1" }).ping()).ok, false);
});
});
});
describe("what is stored", () => {
let id = 0;
before(async () => {
id = (await db.insert(integrations).values({ type: "nextcloud", name: "Cloud", baseUrl: "http://10.9.8.1", config: JSON.stringify({ url: "http://10.9.8.1" }) }).returning({ id: integrations.id }))[0].id;
});
const state = async () => (await db.select().from(nextcloudState).where(eq(nextcloudState.integrationId, id)))[0];
test("the state is replaced at each read; what was not asked stays null", async () => {
await nc.storeRead(id, read(), new Date());
assert.deepEqual([(await state()).freeBytes, (await state()).appUpdates], [500e9, 1]);
await nc.storeRead(id, read({ detail: false, freeBytes: null, users: null, files: null, active24h: null, appUpdates: null }), new Date());
const s = await state();
assert.deepEqual([s.detail, s.freeBytes, s.appUpdates], [false, null, null]);
});
test("a read that failed is said, and what was stored is left alone", async () => {
const ok = await nc.syncNextcloud(id, new Date(), async () => read({ version: "31.0.3" }));
assert.equal(ok.ok, true);
const failed = await nc.syncNextcloud(id, new Date(), async () => {
throw new Error("connection refused");
});
assert.deepEqual([failed.ok, failed.error], [false, "connection refused"]);
assert.equal(failed.at, ok.at);
assert.equal((await state()).version, "31.0.3");
assert.equal(nc.isNextcloudFresh((await nc.nextcloudSyncStatuses())[id], Date.now()), false);
});
});
describe("what is wrong", () => {
const row = (extra: Partial<Parameters<typeof nc.nextcloudProblems>[2]> = {}) => ({ maintenance: false, needsUpgrade: false, freeBytes: 500e9 as number | null, ...extra });
const keys = (r: ReturnType<typeof row>) => nc.nextcloudProblems(1, "Cloud", r).map((p) => `${p.key}=${p.severity}`);
test("nothing wrong says nothing; what was not asked is not a problem", () => {
assert.deepEqual(keys(row()), []);
assert.deepEqual(keys(row({ freeBytes: null })), []);
});
test("a database that wants upgrading is critical, maintenance mode a warning, each saying what to run", () => {
assert.deepEqual(keys(row({ needsUpgrade: true, maintenance: true })), ["nextcloud-upgrade:1=critical", "nextcloud-maintenance:1=warning"]);
const p = nc.nextcloudProblems(1, "Cloud", row({ needsUpgrade: true, maintenance: true }));
assert.match(p[0].message, /occ upgrade/);
assert.match(p[1].message, /occ maintenance:mode --off/);
});
test("the room for the data: a warning under 20 GB, critical under 5 GB", () => {
assert.deepEqual(keys(row({ freeBytes: 20e9 })), []);
assert.deepEqual(keys(row({ freeBytes: 19e9 })), ["nextcloud-disk:1=warning"]);
assert.deepEqual(keys(row({ freeBytes: 4e9 })), ["nextcloud-disk:1=critical"]);
assert.match(nc.nextcloudProblems(1, "Cloud", row({ freeBytes: 4.2e9 }))[0].message, /only 3\.9 GB is left/);
});
});
describe("the health pass, the page and /metrics", () => {
let ncId = 0;
let app: Server;
let base = "";
const fresh = () => setInternalFlag("nextcloudSync", JSON.stringify({ [ncId]: { at: new Date().toISOString(), ok: true, error: null } }));
const stale = () => setInternalFlag("nextcloudSync", JSON.stringify({ [ncId]: { at: new Date(Date.now() - 30 * MIN).toISOString(), ok: true, error: null } }));
before(async () => {
await db.insert(users).values([{ oidcSub: "viewer", name: "Vic", role: "viewer" }]);
ncId = (await db.insert(integrations).values({ type: "nextcloud", name: "Family cloud", baseUrl: "http://10.9.8.2", config: JSON.stringify({ url: "http://10.9.8.2" }) }).returning({ id: integrations.id }))[0].id;
await nc.storeRead(ncId, read({ maintenance: true, needsUpgrade: true, freeBytes: 3e9 }), new Date());
await fresh();
const a = express();
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "viewer" } };
next();
});
a.use("/api/nextcloud", nextcloudRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/nextcloud`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("the health pass judges it from what is stored, and holds it when the last read is old", async () => {
let { snapshot, held } = await collectSnapshot();
assert.deepEqual(snapshot.nextcloud?.map((n) => n.integrationId), [ncId]);
const list = evaluateHealth(snapshot, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, Date.now(), { skipServers: true });
const c = list.find((x) => x.key === `nextcloud-upgrade:${ncId}`)!;
assert.deepEqual([c.severity, c.source], ["critical", `nextcloud:${ncId}`]);
for (const k of ["nextcloud-upgrade", "nextcloud-maintenance", "nextcloud-disk"]) assert.equal(subjectOfConditionKey(`${k}:${ncId}`), `integration:${ncId}`, "maintenance on it silences its alerts");
await stale();
({ snapshot, held } = await collectSnapshot());
assert.deepEqual([snapshot.nextcloud ?? [], held.has(`nextcloud:${ncId}`)], [[], true]);
await fresh();
});
test("the Alerts page shows them: the sign-in problems under Services, the disk under Disk, with where to look", async () => {
invalidateAlertsCache();
const report = await getAlerts(true);
const up = report.alerts.find((x) => x.id === `services:nextcloud-upgrade:${ncId}`)!;
assert.deepEqual([up.severity, up.category, up.source, up.link, up.target?.type, up.target?.id], ["critical", "services", "Nextcloud", "/nextcloud", "integration", ncId]);
assert.equal(report.alerts.find((x) => x.id === `disk:nextcloud-disk:${ncId}`)?.severity, "critical");
assert.ok(report.alerts.some((x) => x.id === `services:nextcloud-maintenance:${ncId}`));
});
test("anyone signed in sees each Nextcloud; an old reading says nothing is wrong", async () => {
const body = (await (await fetch(base)).json()) as { nextclouds: { id: number; fresh: boolean; state: { maintenance: boolean; freeBytes: number; appUpdates: number } | null; problems: unknown[] }[] };
const n = body.nextclouds.find((x) => x.id === ncId)!;
assert.deepEqual([n.fresh, n.state?.maintenance, n.state?.freeBytes, n.state?.appUpdates], [true, true, 3e9, 1]);
assert.equal(n.problems.length, 3);
await stale();
const old = (await (await fetch(base)).json()) as typeof body;
assert.deepEqual(old.nextclouds.find((x) => x.id === ncId)!.problems, []);
await fresh();
});
test("/metrics has maintenance mode, the room left, who was active and the app updates — only while the reading is recent", async () => {
let text = await buildMetrics(new Date());
assert.match(text, /homelab_nextcloud_maintenance_mode\{name="Family cloud"\} 1/);
assert.match(text, /homelab_nextcloud_free_bytes\{name="Family cloud"\} 3000000000|homelab_nextcloud_free_bytes\{name="Family cloud"\} 3e\+9/);
assert.match(text, /homelab_nextcloud_active_users_24h\{name="Family cloud"\} 2/);
assert.match(text, /homelab_nextcloud_app_updates\{name="Family cloud"\} 1/);
await stale();
text = await buildMetrics(new Date());
assert.doesNotMatch(text, /homelab_nextcloud/);
});
});
+6
View File
@@ -16,6 +16,9 @@ import Authentik from "./pages/Authentik";
import Prometheus from "./pages/Prometheus"; import Prometheus from "./pages/Prometheus";
import Immich from "./pages/Immich"; import Immich from "./pages/Immich";
import Paperless from "./pages/Paperless"; import Paperless from "./pages/Paperless";
import N8n from "./pages/N8n";
import NetBox from "./pages/NetBox";
import Nextcloud from "./pages/Nextcloud";
import ServiceLevels from "./pages/ServiceLevels"; import ServiceLevels from "./pages/ServiceLevels";
import Duplicati from "./pages/Duplicati"; import Duplicati from "./pages/Duplicati";
import MyWorkPage from "./pages/MyWork"; import MyWorkPage from "./pages/MyWork";
@@ -168,6 +171,9 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/prometheus" element={<Prometheus />} /> <Route path="/prometheus" element={<Prometheus />} />
<Route path="/immich" element={<Immich />} /> <Route path="/immich" element={<Immich />} />
<Route path="/paperless" element={<Paperless />} /> <Route path="/paperless" element={<Paperless />} />
<Route path="/n8n" element={<N8n />} />
<Route path="/netbox" element={<NetBox />} />
<Route path="/nextcloud" element={<Nextcloud />} />
<Route path="/service-levels" element={<ServiceLevels user={user} />} /> <Route path="/service-levels" element={<ServiceLevels user={user} />} />
<Route path="/duplicati" element={<Duplicati user={user} />} /> <Route path="/duplicati" element={<Duplicati user={user} />} />
<Route path="/ports" element={<Ports user={user} />} /> <Route path="/ports" element={<Ports user={user} />} />
+73 -1
View File
@@ -1228,6 +1228,69 @@ export interface JellyfinInfo {
problems: { key: string; message: string; severity: "warning" | "critical" }[]; problems: { key: string; message: string; severity: "warning" | "critical" }[];
} }
/** One Nextcloud, as its page shows it. */
export interface NextcloudInfo {
id: number;
name: string;
enabled: boolean;
status: { at: string; ok: boolean; error: string | null } | null;
fresh: boolean;
state: {
version: string | null;
maintenance: boolean;
needsUpgrade: boolean;
/** Whether the monitoring app answered (an administrator's app password is needed). */
detail: boolean;
freeBytes: number | null;
users: number | null;
files: number | null;
active24h: number | null;
appUpdates: number | null;
updatedAt: string;
} | null;
problems: { key: string; message: string; severity: "warning" | "critical" }[];
}
/** One NetBox, as its page shows it. */
export interface NetBoxInfo {
id: number;
name: string;
enabled: boolean;
status: { at: string; ok: boolean; error: string | null } | null;
fresh: boolean;
state: {
version: string | null;
/** Null when NetBox did not say. */
workersRunning: number | null;
devices: number | null;
virtualMachines: number | null;
prefixes: number | null;
ipAddresses: number | null;
/** The servers this app manages that have no address in NetBox; null when NetBox's addresses could not be listed. */
missing: { id: number; name: string; ips: string[] }[] | null;
/** NetBox has more addresses than were read, so what is missing may be among the rest. */
truncated: boolean;
updatedAt: string;
} | null;
problems: { key: string; message: string; severity: "warning" | "critical" }[];
}
/** One n8n, as its page shows it. */
export interface N8nInfo {
id: number;
name: string;
enabled: boolean;
status: { at: string; ok: boolean; error: string | null } | null;
fresh: boolean;
state: {
workflows: { id: string; name: string; active: boolean; lastStatus: string | null; lastAt: string | null; failed24h: number }[];
/** False when the runs could not be asked: the last outcomes are then unknown. */
executionsKnown: boolean;
updatedAt: string;
} | null;
problems: { key: string; message: string; severity: "warning" | "critical" }[];
}
/** One Paperless-ngx, as its page shows it. */ /** One Paperless-ngx, as its page shows it. */
export interface PaperlessInfo { export interface PaperlessInfo {
id: number; id: number;
@@ -2243,7 +2306,7 @@ export interface ManualTaskInput {
enabled?: boolean; enabled?: boolean;
} }
export type IntegrationType = "proxmox" | "synology" | "semaphore" | "tailscale" | "gitea" | "dockhand" | "uptimekuma" | "phpipam" | "pbs" | "osticket" | "unifi" | "npm" | "homeassistant" | "ups" | "jellyfin" | "authentik" | "prometheus" | "immich" | "paperless"; export type IntegrationType = "proxmox" | "synology" | "semaphore" | "tailscale" | "gitea" | "dockhand" | "uptimekuma" | "phpipam" | "pbs" | "osticket" | "unifi" | "npm" | "homeassistant" | "ups" | "jellyfin" | "authentik" | "prometheus" | "immich" | "paperless" | "nextcloud" | "netbox" | "n8n";
export interface IntegrationField { export interface IntegrationField {
key: string; key: string;
@@ -2790,6 +2853,15 @@ export const api = {
update: (id: number, data: ServiceLevelInput) => request<{ ok: boolean }>(`/api/service-levels/${id}`, { method: "PUT", body: JSON.stringify(data) }), update: (id: number, data: ServiceLevelInput) => request<{ ok: boolean }>(`/api/service-levels/${id}`, { method: "PUT", body: JSON.stringify(data) }),
remove: (id: number) => request<void>(`/api/service-levels/${id}`, { method: "DELETE" }), remove: (id: number) => request<void>(`/api/service-levels/${id}`, { method: "DELETE" }),
}, },
nextcloud: {
list: () => request<{ nextclouds: NextcloudInfo[] }>("/api/nextcloud"),
},
netbox: {
list: () => request<{ netboxes: NetBoxInfo[] }>("/api/netbox"),
},
n8n: {
list: () => request<{ n8ns: N8nInfo[] }>("/api/n8n"),
},
paperless: { paperless: {
list: () => request<{ paperlesses: PaperlessInfo[] }>("/api/paperless"), list: () => request<{ paperlesses: PaperlessInfo[] }>("/api/paperless"),
}, },
+3
View File
@@ -52,6 +52,9 @@ const INTEGRATION_TYPE_LABELS: Record<IntegrationType, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
const DNS_PROVIDER_LABELS: Record<DnsProviderType, string> = { const DNS_PROVIDER_LABELS: Record<DnsProviderType, string> = {
@@ -23,6 +23,9 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
export default function IntegrationEditForm({ export default function IntegrationEditForm({
+3
View File
@@ -22,6 +22,9 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
export default function IntegrationForm({ onCreated, onCancel }: { onCreated: () => void; onCancel: () => void }) { export default function IntegrationForm({ onCreated, onCancel }: { onCreated: () => void; onCancel: () => void }) {
+3
View File
@@ -24,6 +24,9 @@ const TYPE_LABELS: Record<string, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
export type WallLevel = "critical" | "warning" | "ok" | "unsure"; export type WallLevel = "critical" | "warning" | "ok" | "unsure";
+6
View File
@@ -17,6 +17,9 @@ import {
IconPlugConnected, IconPlugConnected,
IconBrandDocker, IconBrandDocker,
IconAffiliate, IconAffiliate,
IconCloud,
IconBinaryTree,
IconBolt,
IconFileText, IconFileText,
IconFlame, IconFlame,
IconLock, IconLock,
@@ -131,6 +134,9 @@ export const NAV: NavEntry[] = [
{ to: "/prometheus", label: "Prometheus", icon: <IconFlame size={20} /> }, { to: "/prometheus", label: "Prometheus", icon: <IconFlame size={20} /> },
{ to: "/immich", label: "Immich", icon: <IconPhoto size={20} /> }, { to: "/immich", label: "Immich", icon: <IconPhoto size={20} /> },
{ to: "/paperless", label: "Paperless", icon: <IconFileText size={20} /> }, { to: "/paperless", label: "Paperless", icon: <IconFileText size={20} /> },
{ to: "/n8n", label: "n8n", icon: <IconBolt size={20} /> },
{ to: "/netbox", label: "NetBox", icon: <IconBinaryTree size={20} /> },
{ to: "/nextcloud", label: "Nextcloud", icon: <IconCloud size={20} /> },
{ to: "/service-levels", label: "Service levels", icon: <IconTargetArrow size={20} /> }, { to: "/service-levels", label: "Service levels", icon: <IconTargetArrow size={20} /> },
], ],
}, },
+6
View File
@@ -34,6 +34,9 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
/** The page each kind of integration gets once it is connected. */ /** The page each kind of integration gets once it is connected. */
@@ -57,6 +60,9 @@ const TYPE_PAGES: Record<IntegrationType, string> = {
prometheus: "/prometheus", prometheus: "/prometheus",
immich: "/immich", immich: "/immich",
paperless: "/paperless", paperless: "/paperless",
n8n: "/n8n",
netbox: "/netbox",
nextcloud: "/nextcloud",
}; };
/** "pve.example.lan" out of "https://pve.example.lan:8006/", or the text itself when it isn't an address. */ /** "pve.example.lan" out of "https://pve.example.lan:8006/", or the text itself when it isn't an address. */
+126
View File
@@ -0,0 +1,126 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type N8nInfo } from "../api/client";
import EmptyState from "../components/EmptyState";
import TimeAgo from "../components/TimeAgo";
import { BlockSkeleton } from "../components/Skeleton";
import { ErrorNotice } from "../components/Notice";
import { readableError } from "../utils/errors";
function Badge({ n }: { n: N8nInfo }) {
if (!n.enabled) return <span className="badge bg-secondary-lt text-secondary">Off</span>;
if (!n.state) return <span className="badge bg-secondary-lt text-secondary">Not read yet</span>;
if (!n.fresh) return <span className="badge bg-yellow-lt text-yellow">Can&rsquo;t be read</span>;
if (n.problems.length > 0) return <span className="badge bg-yellow-lt text-yellow">Needs a look</span>;
return <span className="badge bg-green-lt text-green">In order</span>;
}
const failed = (status: string | null) => status === "error" || status === "crashed";
function LastRun({ status, at }: { status: string | null; at: string | null }) {
if (!status || !at) return <span className="text-secondary">no run seen</span>;
return (
<>
<span className={`badge ${failed(status) ? "bg-red-lt text-red" : status === "success" ? "bg-green-lt text-green" : "bg-secondary-lt text-secondary"}`}>{failed(status) ? (status === "crashed" ? "Crashed" : "Failed") : status === "success" ? "Succeeded" : status}</span>{" "}
<span className="text-secondary">
<TimeAgo value={at} />
</span>
</>
);
}
function Card({ n }: { n: N8nInfo }) {
const s = n.state;
const workflows = s ? [...s.workflows].sort((a, b) => Number(failed(b.lastStatus) && b.active) - Number(failed(a.lastStatus) && a.active) || Number(b.active) - Number(a.active) || a.name.localeCompare(b.name)) : [];
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">
{n.name} <Badge n={n} />
</h3>
<div className="card-actions text-secondary small">{n.status ? <>read <TimeAgo value={n.status.at} /></> : "not read yet"}</div>
</div>
<div className="card-body">
{n.status && !n.status.ok && <div className="text-yellow mb-2">The last attempt to read it failed: {n.status.error ?? "no answer"}.</div>}
{n.problems.length > 0 && (
<ul className="mb-3">
{n.problems.map((x) => (
<li key={x.key}>{x.message}</li>
))}
</ul>
)}
{s && (
<>
{!s.executionsKnown && <div className="text-yellow mb-2">n8n did not give its runs, so what each workflow last did is not known.</div>}
<div className="text-secondary small mb-2">
{s.workflows.length} workflow{s.workflows.length === 1 ? "" : "s"}, {s.workflows.filter((w) => w.active).length} switched on — judged from the newest 250 runs
</div>
{workflows.length === 0 ? (
<div className="text-secondary">No workflows.</div>
) : (
<div className="table-responsive">
<table className="table table-vcenter table-sm">
<thead>
<tr>
<th>Workflow</th>
<th>On</th>
<th>Last run</th>
<th className="text-end">Failed in the last day</th>
</tr>
</thead>
<tbody>
{workflows.map((w) => (
<tr key={w.id} className={w.active ? "" : "text-secondary"}>
<td>{w.name}</td>
<td>{w.active ? "yes" : "off"}</td>
<td>{s.executionsKnown ? <LastRun status={w.lastStatus} at={w.lastAt} /> : "—"}</td>
<td className={`text-end ${w.failed24h > 0 ? "text-red fw-bold" : ""}`}>{w.failed24h}</td>
</tr>
))}
</tbody>
</table>
</div>
)}
</>
)}
</div>
</div>
);
}
/** n8n: which automations have quietly stopped working — an active workflow whose last run failed. */
export default function N8n() {
const [data, setData] = useState<N8nInfo[] | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
const load = () =>
api.n8n
.list()
.then((r) => {
setData(r.n8ns);
setError(null);
})
.catch((err) => setError(readableError(err)));
void load();
const t = setInterval(load, 60_000);
return () => clearInterval(t);
}, []);
return (
<>
<h2 className="page-title">n8n</h2>
<p className="text-secondary">
Which automations have quietly stopped working. It is read every few minutes with an API key (see <Link to="/integrations">Integrations</Link>). A <strong>workflow that is switched on and failed on its last run</strong> is an alert — see <Link to="/alerts">Alerts</Link>; it clears when the workflow next succeeds.
</p>
<ErrorNotice error={error} />
{!data && !error && <BlockSkeleton lines={4} />}
{data && data.length === 0 && (
<div className="card">
<div className="card-body">
<EmptyState title="No n8n connected" hint="Add one under Integrations with its address and an API key (Settings → n8n API)." />
</div>
</div>
)}
{data?.map((n) => <Card key={n.id} n={n} />)}
</>
);
}
+123
View File
@@ -0,0 +1,123 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type NetBoxInfo } from "../api/client";
import EmptyState from "../components/EmptyState";
import TimeAgo from "../components/TimeAgo";
import { BlockSkeleton } from "../components/Skeleton";
import { ErrorNotice } from "../components/Notice";
import { readableError } from "../utils/errors";
function Badge({ n }: { n: NetBoxInfo }) {
if (!n.enabled) return <span className="badge bg-secondary-lt text-secondary">Off</span>;
if (!n.state) return <span className="badge bg-secondary-lt text-secondary">Not read yet</span>;
if (!n.fresh) return <span className="badge bg-yellow-lt text-yellow">Can&rsquo;t be read</span>;
if (n.problems.length > 0) return <span className="badge bg-yellow-lt text-yellow">Needs a look</span>;
return <span className="badge bg-green-lt text-green">In order</span>;
}
const count = (v: number | null) => (v === null ? "—" : v.toLocaleString());
function Tile({ label, value, bad }: { label: string; value: string; bad?: boolean }) {
return (
<div className="col-6 col-lg-3">
<div className="card card-sm h-100">
<div className="card-body">
<div className="text-secondary small">{label}</div>
<div className={`h2 mb-0 ${bad ? "text-red" : ""}`}>{value}</div>
</div>
</div>
</div>
);
}
function Card({ n }: { n: NetBoxInfo }) {
const s = n.state;
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">
{n.name} <Badge n={n} />
</h3>
<div className="card-actions text-secondary small">{n.status ? <>read <TimeAgo value={n.status.at} /></> : "not read yet"}</div>
</div>
<div className="card-body">
{n.status && !n.status.ok && <div className="text-yellow mb-2">The last attempt to read it failed: {n.status.error ?? "no answer"}.</div>}
{n.problems.length > 0 && (
<ul className="mb-3">
{n.problems.map((x) => (
<li key={x.key}>{x.message}</li>
))}
</ul>
)}
{s && (
<>
<div className="text-secondary small mb-3">{s.version ? `version ${s.version}` : ""}</div>
<div className="row g-3 mb-3">
<Tile label="Workers running" value={count(s.workersRunning)} bad={s.workersRunning === 0} />
<Tile label="Devices" value={count(s.devices)} />
<Tile label="Virtual machines" value={count(s.virtualMachines)} />
<Tile label="IP addresses" value={count(s.ipAddresses)} />
</div>
<h4>Servers with no address in NetBox</h4>
{s.missing === null ? (
<div className="text-secondary">NetBox&rsquo;s addresses could not be listed, so nothing can be said about what is missing.</div>
) : s.missing.length === 0 ? (
<div className="text-secondary">Every server this app manages has an address in NetBox.</div>
) : (
<>
<ul className="mb-2">
{s.missing.map((m) => (
<li key={m.id}>
<Link to={`/servers/${m.id}`}>{m.name}</Link> <span className="text-secondary">({m.ips.slice(0, 3).join(", ")}{m.ips.length > 3 ? ", …" : ""})</span>
</li>
))}
</ul>
<div className="text-secondary small">
A list, not an alert: documentation that is behind is not an outage.{s.truncated ? " NetBox has more addresses than were read, so some of these may be in the part that was not." : ""}
</div>
</>
)}
</>
)}
</div>
</div>
);
}
/** NetBox: whether it is working — above all its background workers — and which of the servers you run are not written down in it. */
export default function NetBox() {
const [data, setData] = useState<NetBoxInfo[] | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
const load = () =>
api.netbox
.list()
.then((r) => {
setData(r.netboxes);
setError(null);
})
.catch((err) => setError(readableError(err)));
void load();
const t = setInterval(load, 60_000);
return () => clearInterval(t);
}, []);
return (
<>
<h2 className="page-title">NetBox</h2>
<p className="text-secondary">
Whether NetBox is working, and whether it is complete. It is read every few minutes with an API token (see <Link to="/integrations">Integrations</Link>). <strong>No background worker running</strong> is an alert — see <Link to="/alerts">Alerts</Link>; the servers missing from NetBox are only listed.
</p>
<ErrorNotice error={error} />
{!data && !error && <BlockSkeleton lines={4} />}
{data && data.length === 0 && (
<div className="card">
<div className="card-body">
<EmptyState title="No NetBox connected" hint="Add one under Integrations with its address and an API token (your profile → API Tokens; read-only is enough)." />
</div>
</div>
)}
{data?.map((n) => <Card key={n.id} n={n} />)}
</>
);
}
+131
View File
@@ -0,0 +1,131 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type NextcloudInfo } from "../api/client";
import EmptyState from "../components/EmptyState";
import TimeAgo from "../components/TimeAgo";
import { BlockSkeleton } from "../components/Skeleton";
import { ErrorNotice } from "../components/Notice";
import { formatBytes } from "./settings/StorageSettings";
import { readableError } from "../utils/errors";
function Badge({ n }: { n: NextcloudInfo }) {
if (!n.enabled) return <span className="badge bg-secondary-lt text-secondary">Off</span>;
if (!n.state) return <span className="badge bg-secondary-lt text-secondary">Not read yet</span>;
if (!n.fresh) return <span className="badge bg-yellow-lt text-yellow">Can&rsquo;t be read</span>;
if (n.problems.some((x) => x.severity === "critical")) return <span className="badge bg-red-lt text-red">Needs a look</span>;
if (n.problems.length > 0) return <span className="badge bg-yellow-lt text-yellow">Needs a look</span>;
return <span className="badge bg-green-lt text-green">In order</span>;
}
const count = (v: number | null) => (v === null ? "—" : v.toLocaleString());
function Card({ n }: { n: NextcloudInfo }) {
const s = n.state;
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">
{n.name} <Badge n={n} />
</h3>
<div className="card-actions text-secondary small">{n.status ? <>read <TimeAgo value={n.status.at} /></> : "not read yet"}</div>
</div>
<div className="card-body">
{n.status && !n.status.ok && <div className="text-yellow mb-2">The last attempt to read it failed: {n.status.error ?? "no answer"}.</div>}
{n.problems.length > 0 && (
<ul className="mb-3">
{n.problems.map((x) => (
<li key={x.key} className={x.severity === "critical" ? "text-red" : ""}>
{x.message}
</li>
))}
</ul>
)}
{s && (
<>
<div className="text-secondary small mb-3">
{[s.version ? `version ${s.version}` : null, s.maintenance ? "in maintenance mode" : null, s.needsUpgrade ? "needs its database upgrading" : null].filter(Boolean).join(" · ")}
</div>
{s.detail ? (
<div className="row g-3">
<div className="col-6 col-lg-3">
<div className="card card-sm h-100">
<div className="card-body">
<div className="text-secondary small">Free for data</div>
<div className="h2 mb-0">{s.freeBytes === null ? "—" : formatBytes(s.freeBytes)}</div>
</div>
</div>
</div>
<div className="col-6 col-lg-3">
<div className="card card-sm h-100">
<div className="card-body">
<div className="text-secondary small">Users</div>
<div className="h2 mb-0">{count(s.users)}</div>
<div className="text-secondary small">{s.active24h === null ? "" : `${s.active24h} active in the last day`}</div>
</div>
</div>
</div>
<div className="col-6 col-lg-3">
<div className="card card-sm h-100">
<div className="card-body">
<div className="text-secondary small">Files</div>
<div className="h2 mb-0">{count(s.files)}</div>
</div>
</div>
</div>
<div className="col-6 col-lg-3">
<div className="card card-sm h-100">
<div className="card-body">
<div className="text-secondary small">App updates waiting</div>
<div className="h2 mb-0">{count(s.appUpdates)}</div>
</div>
</div>
</div>
</div>
) : (
<div className="text-secondary">
Only its status is read. Give the integration an administrator&rsquo;s name and an <strong>app password</strong> (Personal settings → Security → Devices &amp; sessions) and the room left for the data, the users and the app updates are read too.
</div>
)}
</>
)}
</div>
</div>
);
}
/** Nextcloud: whether people can sign in — maintenance mode, a database that wants upgrading — and how much room is left for the data. */
export default function Nextcloud() {
const [data, setData] = useState<NextcloudInfo[] | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
const load = () =>
api.nextcloud
.list()
.then((r) => {
setData(r.nextclouds);
setError(null);
})
.catch((err) => setError(readableError(err)));
void load();
const t = setInterval(load, 60_000);
return () => clearInterval(t);
}, []);
return (
<>
<h2 className="page-title">Nextcloud</h2>
<p className="text-secondary">
Whether people can sign in. It is read every few minutes (see <Link to="/integrations">Integrations</Link>). <strong>Maintenance mode left on</strong>, <strong>a database that wants upgrading</strong> and <strong>the data disk running low</strong> are alerts — see <Link to="/alerts">Alerts</Link>.
</p>
<ErrorNotice error={error} />
{!data && !error && <BlockSkeleton lines={4} />}
{data && data.length === 0 && (
<div className="card">
<div className="card-body">
<EmptyState title="No Nextcloud connected" hint="Add one under Integrations with its address; an administrator's name and an app password add the disk, the users and the app updates." />
</div>
</div>
)}
{data?.map((n) => <Card key={n.id} n={n} />)}
</>
);
}
+3
View File
@@ -32,6 +32,9 @@ const INTEGRATION_NAMES: Record<string, string> = {
prometheus: "Prometheus", prometheus: "Prometheus",
immich: "Immich", immich: "Immich",
paperless: "Paperless-ngx", paperless: "Paperless-ngx",
n8n: "n8n",
netbox: "NetBox",
nextcloud: "Nextcloud",
}; };
function ColorList({ function ColorList({