Endpoint checks, restore tests, and a change log

- Endpoints: HTTP, TCP and TLS checks with answer time, uptime and certificate expiry; failures and expiring certificates become alerts and notifications; linked to a server for maintenance and tags
- Restore tests: record when a backup was restored and how it went; failed tests always alert, overdue ones with a limit set; shown on the Backup coverage page
- Change log: one filterable timeline made from the audit log and the incident record

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 23:03:56 +02:00
1 parent c3bc6bbf72
commit 92726568b7
37 files changed
+6918 -21

No files matched your search

+5
View File
@@ -69,6 +69,11 @@ All gauges. Labels are escaped as Prometheus requires.
| `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. | | `homelab_inventory_power_watts` | | Typical draw of everything in the inventory. |
| `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. | | `homelab_backup_guests` | `status` | Proxmox guests by how well they are backed up: `ok`, `stale`, `failed`, `none`, `unknown`. From what the last backup check read; a scrape never goes to Proxmox or Backup Server itself, and the series is left out until something has been read. |
| `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. | | `homelab_backup_last_age_seconds` | `guest`, `vmid`, `integration` | How long ago a guest's newest backup was. |
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
| `homelab_scrape_duration_seconds` | | How long building the answer took. | | `homelab_scrape_duration_seconds` | | How long building the answer took. |
| `homelab_scrape_timestamp_seconds` | | When the answer was built. | | `homelab_scrape_timestamp_seconds` | | When the answer was built. |
+53 -4
View File
@@ -52,6 +52,7 @@ what the app actually creates.
erDiagram erDiagram
users ||--o{ audit_log : "actor (set null)" users ||--o{ audit_log : "actor (set null)"
users ||--o{ push_subscriptions : "cascade" users ||--o{ push_subscriptions : "cascade"
endpoint_checks ||--o{ endpoint_results : "cascade"
integration_credentials ||--o{ integrations : "credential (set null)" integration_credentials ||--o{ integrations : "credential (set null)"
integration_credentials ||--o{ dns_providers : "credential (set null)" integration_credentials ||--o{ dns_providers : "credential (set null)"
dns_providers ||--o{ dns_zones_cache : "cascade" dns_providers ||--o{ dns_zones_cache : "cascade"
@@ -65,9 +66,9 @@ erDiagram
integrations ||--o{ alert_tickets : "osTicket used (set null)" integrations ||--o{ alert_tickets : "osTicket used (set null)"
``` ```
Fourteen tables stand alone, with no foreign keys: `settings`, `secrets`, Sixteen tables stand alone, with no foreign keys: `settings`, `secrets`,
`ipam_entries`, `domains`, `diag_log`, `notification_queue`, `ipam_entries`, `domains`, `diag_log`, `notification_queue`,
`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`, `dependencies`. `maintenance_windows` and `maintenance_aliases` also have no `consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`, `dependencies`, `endpoint_checks`, `restore_tests`. `maintenance_windows` and `maintenance_aliases` also have no
foreign key — see [soft references](#soft-references-not-foreign-keys). foreign key — see [soft references](#soft-references-not-foreign-keys).
## Tables ## Tables
@@ -104,7 +105,7 @@ Who changed what. Written by the app on every change (see
| `id` | integer PK | | | `id` | integer PK | |
| `actor_user_id` | integer → `users.id`, **set null** on delete | Null for automatic actions, and for entries whose user was deleted. | | `actor_user_id` | integer → `users.id`, **set null** on delete | Null for automatic actions, and for entries whose user was deleted. |
| `actor_label` | text | The user's name/email as it was at the time (or `system`), so an entry still reads correctly after the account is gone. | | `actor_label` | text | The user's name/email as it was at the time (or `system`), so an entry still reads correctly after the account is gone. |
| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `dependency` (`create`, `delete` — what needs what), `push` (`subscribe`, `unsubscribe` — a browser added or removed, never its address or keys), `share_link` (a link made or removed, and `update_status_page` when what a status page shows is changed — never the secret), `runbook`, `alert` (`ack`, `unack` — with the alert's id; on the thing it is about, so they show on a server's Activity card), `incident` (`note`, `note_removed`), `db_backup` (`create`, `upload`, `download`, `delete`, `restore_staged`, `restore_cancelled`, `settings`), `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. | | `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `dependency` (`create`, `delete` — what needs what), `endpoint` (`create`, `update`, `delete`), `restore_test` (`record`, `delete`), `push` (`subscribe`, `unsubscribe` — a browser added or removed, never its address or keys), `share_link` (a link made or removed, and `update_status_page` when what a status page shows is changed — never the secret), `runbook`, `alert` (`ack`, `unack` — with the alert's id; on the thing it is about, so they show on a server's Activity card), `incident` (`note`, `note_removed`), `db_backup` (`create`, `upload`, `download`, `delete`, `restore_staged`, `restore_cancelled`, `settings`), `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. |
| `action` | text NOT NULL | `create`, `update`, `delete`, `start`, `stop`, … — free text. | | `action` | text NOT NULL | `create`, `update`, `delete`, `start`, `stop`, … — free text. |
| `target_type`, `target_id` | text | What it happened to. `target_id` is text so it can hold any kind of ID; there's no foreign key. | | `target_type`, `target_id` | text | What it happened to. `target_id` is text so it can hold any kind of ID; there's no foreign key. |
| `detail` | text | JSON, free-form context (what changed, names, counts). Secrets are never put here. | | `detail` | text | JSON, free-form context (what changed, names, counts). Secrets are never put here. |
@@ -623,6 +624,53 @@ A browser or phone that asked for push notifications, one row per device.
A subscription is removed when the push service says it is gone (HTTP 404 or 410), when its owner or an administrator removes it, and all of them are removed if the server's own key pair has to be made again. A subscription is removed when the push service says it is gone (HTTP 404 or 410), when its owner or an administrator removes it, and all of them are removed if the server's own key pair has to be made again.
#### `endpoint_checks`
A web address, TCP port or TLS certificate the app asks about every few minutes (see the README). The last answer is kept on the row.
| Column | Type | Notes |
|---|---|---|
| `id` | integer PK | |
| `name` | text NOT NULL | |
| `kind` | text NOT NULL | `http`, `tcp` or `tls`. |
| `target`, `port` | text NOT NULL, integer | http: the address. tcp and tls: the host, and the port. |
| `expect_status`, `keyword` | text | http only: the good status codes (`200-204, 401`; null = 200–399) and a word the page must contain. |
| `insecure` | bool NOT NULL, default 0 | Accept a certificate that isn't trusted (its expiry is still watched). |
| `interval_minutes`, `timeout_seconds`, `failures_before_alert`, `warn_days` | integer NOT NULL | Defaults 5, 10, 2 and 14. |
| `server_id` | integer | The server it is about. A plain reference, **not a foreign key**: maintenance on that server silences the check's alerts; one whose server is gone is just unlinked. |
| `enabled` | bool NOT NULL, default 1 | |
| `last_checked_at`, `last_ok`, `last_message`, `last_response_ms` | text, bool, text, integer | How the latest ask went. |
| `cert_expires_at` | text | ISO; when the certificate it showed runs out. |
| `consecutive_failures` | integer NOT NULL, default 0 | A problem once this reaches `failures_before_alert`. |
| `last_change_at` | text | When it last went from working to failing or back. |
| `created_at` | text NOT NULL | |
#### `endpoint_results`
One row per answer, kept for two weeks: uptime, average answer time and the run of recent results come from these.
| Column | Type | Notes |
|---|---|---|
| `id` | integer PK | |
| `check_id` | integer NOT NULL | → `endpoint_checks.id`, deleted with it (indexed with `at`). |
| `at` | text NOT NULL | ISO. |
| `ok` | bool NOT NULL | |
| `ms`, `message` | integer, text | |
#### `restore_tests`
Each time somebody tried to restore a backup, and how it went. History is kept; the newest per subject is what counts.
| Column | Type | Notes |
|---|---|---|
| `id` | integer PK | |
| `subject_key` | text NOT NULL | `guest:<integration id>:<vmid>` for a Proxmox guest, `custom:<name>` for anything else. Not a foreign key (indexed with `tested_at`). |
| `label` | text NOT NULL | What to call it. |
| `tested_at` | text NOT NULL | `YYYY-MM-DD`. |
| `result` | text NOT NULL | `ok`, `partial`, `failed` or `not_needed`. |
| `note`, `tested_by` | text | |
| `created_at` | text NOT NULL | |
#### `metrics_tokens` #### `metrics_tokens`
Read-only access to `/metrics` for Prometheus. Only a hash of each secret is stored. Read-only access to `/metrics` for Prometheus. Only a hash of each secret is stored.
@@ -730,7 +778,7 @@ that were never changed aren't stored; the app fills in defaults when reading.
| `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. | | `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. |
| `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. | | `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. |
| `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | | `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
| `healthChecks` | `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be). A server can override each of these (see `servers.thresholds`). | | `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be). A server can override each of these (see `servers.thresholds`). |
| `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). | | `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). |
| `display` | `dateFormat`, `timeFormat`, `pageSize`. | | `display` | `dateFormat`, `timeFormat`, `pageSize`. |
| `providerColors`, `integrationColors` | Badge colours, by provider / integration type. | | `providerColors`, `integrationColors` | Badge colours, by provider / integration type. |
@@ -769,6 +817,7 @@ a stale value is possible and the app treats it as "nothing there":
- `maintenance_windows.target_id` (→ a server, integration or DNS provider, by `target_type`) - `maintenance_windows.target_id` (→ a server, integration or DNS provider, by `target_type`)
- `maintenance_aliases.target_id` (the same, by `target_type`) - `maintenance_aliases.target_id` (the same, by `target_type`)
- `endpoint_checks.server_id` (→ a server) and `restore_tests.subject_key` (→ a Proxmox guest, by integration and vmid)
- `dependencies.from_id` and `to_id` (→ a server or integration, by their `_type`) - `dependencies.from_id` and `to_id` (→ a server or integration, by their `_type`)
- `audit_log.target_id` - `audit_log.target_id`
- `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider) - `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider)
+8
View File
@@ -64,6 +64,14 @@ rest, and the alert starts over if it goes away and comes back. Problems under m
informational never escalate. It is looked at every five minutes, so a step can arrive a few minutes after its time. Every escalation is written in the audit informational never escalate. It is looked at every five minutes, so a step can arrive a few minutes after its time. Every escalation is written in the audit
log (category *alert*, action *escalate*). log (category *alert*, action *escalate*).
## Endpoint checks and restore tests
An **endpoint check** that fails several times in a row (the check's own number, 2 by default) is announced like any other server problem — under the *server offline,
disk full …* switch (`healthAlerts`) — once when it starts, with a recovery message when it answers again; a certificate that is within its warning time is announced
the same way. For routing they count as *Monitoring* (a check that fails) and *Expiry* (a certificate), and carry the tags of the server the check is linked to; maintenance on that
server holds them back. A **restore test** that fails, or one that is overdue (when a limit is set), is on the Alerts page and in the incident record but is not sent as a
notification of its own — it is something to look at when you are looking at backups.
## Push to a phone or browser ## Push to a phone or browser
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
+22
View File
@@ -435,6 +435,18 @@ refused. "Nothing — it runs on its own" stops the guess for a server that isn'
guest (a physical machine that happens to share a Proxmox node's name), so its guest (a physical machine that happens to share a Proxmox node's name), so its
alerts always stand alone. alerts always stand alone.
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
kept, `/metrics` has each subject's age and result, and everything is in the change log.
**The change log** (Operations → Change log, operators and administrators) is one list of what has changed, newest first and grouped by day: agents updated,
settings changed (with what to what), maintenance started and ended, dependencies edited, hardware added, endpoint checks and restore tests, alerts raised and cleared —
each as a sentence. Narrow it by kind of change, by server, by how far back (a day to everything kept), or by a word; "Show older" goes on from where it stopped; CSV
export. A server's Activity card links to its slice of it. Sign-ins and people's own data exports aren't changes and are left out. The Audit Log remains the raw table.
**Dependencies beyond servers** (Infrastructure → Dependencies). Say that Gitea needs the server it is installed on, or that a server **Dependencies beyond servers** (Infrastructure → Dependencies). Say that Gitea needs the server it is installed on, or that a server
needs Proxmox to answer. When what is needed is down, the alert of what needs it is folded under the first one exactly like a needs Proxmox to answer. When what is needed is down, the alert of what needs it is folded under the first one exactly like a
guest under its host — shown, labelled "caused by …", left out of notifications and tickets — and the root cause says what else it guest under its host — shown, labelled "caused by …", left out of notifications and tickets — and the root cause says what else it
@@ -549,6 +561,16 @@ text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a
of commands, backticks inline code; links become clickable, and nothing typed in a of commands, backticks inline code; links become clickable, and nothing typed in a
note can run as HTML. note can run as HTML.
**Endpoint checks** (Network → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
also want a word in the page; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
its expiry is watched either way. A check that has **failed several times in a row** (2 by default, so a blip isn't) is a *Monitoring* alert and one notification, with
another when it answers again; a certificate within its warning time (14 days by default; critical in the last three) is an *Expiry* alert. A check can be linked to a
server, so maintenance on it silences the check's alerts and the server's tags can route them. Everyone can look; administrators set checks up (each one makes this
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
dashboard and in `/metrics`. Results are kept for two weeks.
**Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed **Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox (from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox
+6
View File
@@ -27,6 +27,8 @@ page's own top-level link.
| Backup coverage | `/backups` | ✅ | ✅ | ✅ | | Backup coverage | `/backups` | ✅ | ✅ | ✅ |
| Inventory | `/inventory` | ✅ | ✅ | ✅ | | Inventory | `/inventory` | ✅ | ✅ | ✅ |
| Dependencies | `/dependencies` | ✅ | ✅ | ✅ | | Dependencies | `/dependencies` | ✅ | ✅ | ✅ |
| Endpoints | `/endpoints` | ✅ | ✅ | ✅ |
| Change log | `/changes` | — | ✅ | ✅ |
| Docker | `/docker` | ✅ | ✅ | ✅ | | Docker | `/docker` | ✅ | ✅ | ✅ |
| Tailscale | `/tailscale` | ✅ | ✅ | ✅ | | Tailscale | `/tailscale` | ✅ | ✅ | ✅ |
| **Network** | | | | | | **Network** | | | | |
@@ -95,6 +97,10 @@ even further, to admin only:
- **Inventory**: everyone can see the hardware (and print its labels); adding, editing and removing needs - **Inventory**: everyone can see the hardware (and print its labels); adding, editing and removing needs
operator, and so does *Refresh from agent*; the price of power used for the running cost is admin-only. operator, and so does *Refresh from agent*; the price of power used for the running cost is admin-only.
- **Dependencies**: everyone can see what needs what; adding and removing one needs operator. - **Dependencies**: everyone can see what needs what; adding and removing one needs operator.
- **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere);
*Check now* needs operator.
- **Restore tests**: everyone sees them on the Backup coverage page; recording one needs operator; removing one is administrator-only.
- **Change log**: operators and administrators (it is made from the audit log, which has the same readers).
- **Push notifications**: an administrator switches the channel on (Settings → Notifications). Each person, whatever their role, adds - **Push notifications**: an administrator switches the channel on (Settings → Notifications). Each person, whatever their role, adds
and removes their own browsers and can send a test to them; an administrator can remove anyone's. and removes their own browsers and can send a test to them; an administrator can remove anyone's.
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs - **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
+48
View File
@@ -0,0 +1,48 @@
CREATE TABLE `endpoint_checks` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`name` text NOT NULL,
`kind` text NOT NULL,
`target` text NOT NULL,
`port` integer,
`expect_status` text,
`keyword` text,
`insecure` integer DEFAULT false NOT NULL,
`interval_minutes` integer DEFAULT 5 NOT NULL,
`timeout_seconds` integer DEFAULT 10 NOT NULL,
`failures_before_alert` integer DEFAULT 2 NOT NULL,
`warn_days` integer DEFAULT 14 NOT NULL,
`server_id` integer,
`enabled` integer DEFAULT true NOT NULL,
`last_checked_at` text,
`last_ok` integer,
`last_message` text,
`last_response_ms` integer,
`cert_expires_at` text,
`consecutive_failures` integer DEFAULT 0 NOT NULL,
`last_change_at` text,
`created_at` text DEFAULT (current_timestamp) NOT NULL
);
--> statement-breakpoint
CREATE TABLE `endpoint_results` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`check_id` integer NOT NULL,
`at` text NOT NULL,
`ok` integer NOT NULL,
`ms` integer,
`message` text,
FOREIGN KEY (`check_id`) REFERENCES `endpoint_checks`(`id`) ON UPDATE no action ON DELETE cascade
);
--> statement-breakpoint
CREATE INDEX `endpoint_results_check_idx` ON `endpoint_results` (`check_id`,`at`);--> statement-breakpoint
CREATE TABLE `restore_tests` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`subject_key` text NOT NULL,
`label` text NOT NULL,
`tested_at` text NOT NULL,
`result` text NOT NULL,
`note` text,
`tested_by` text,
`created_at` text DEFAULT (current_timestamp) NOT NULL
);
--> statement-breakpoint
CREATE INDEX `restore_tests_subject_idx` ON `restore_tests` (`subject_key`,`tested_at`);
File diff suppressed because it is too large. Load diff
+7
View File
@@ -274,6 +274,13 @@
"when": 1792354148200, "when": 1792354148200,
"tag": "0038_wise_malcolm_colcord", "tag": "0038_wise_malcolm_colcord",
"breakpoints": true "breakpoints": true
},
{
"idx": 39,
"version": "6",
"when": 1792356659440,
"tag": "0039_old_blur",
"breakpoints": true
} }
] ]
} }
+4
View File
@@ -17,6 +17,8 @@ import { incidentsRouter } from "./routes/incidents.js";
import { inventoryRouter } from "./routes/inventory.js"; import { inventoryRouter } from "./routes/inventory.js";
import { dependenciesRouter } from "./routes/dependencies.js"; import { dependenciesRouter } from "./routes/dependencies.js";
import { pushRouter } from "./routes/push.js"; import { pushRouter } from "./routes/push.js";
import { endpointsRouter } from "./routes/endpoints.js";
import { changesRouter } from "./routes/changes.js";
import { backupsRouter } from "./routes/backups.js"; import { backupsRouter } from "./routes/backups.js";
import { metricsRouter, metricsTokensRouter } from "./routes/metrics.js"; import { metricsRouter, metricsTokensRouter } from "./routes/metrics.js";
import { eventWebhooksRouter } from "./routes/eventWebhooks.js"; import { eventWebhooksRouter } from "./routes/eventWebhooks.js";
@@ -110,6 +112,8 @@ export function createApp() {
app.use("/api/inventory", inventoryRouter); app.use("/api/inventory", inventoryRouter);
app.use("/api/dependencies", dependenciesRouter); app.use("/api/dependencies", dependenciesRouter);
app.use("/api/push", pushRouter); app.use("/api/push", pushRouter);
app.use("/api/endpoints", endpointsRouter);
app.use("/api/changes", changesRouter);
app.use("/api/backups", backupsRouter); app.use("/api/backups", backupsRouter);
app.use("/api/metrics-tokens", metricsTokensRouter); app.use("/api/metrics-tokens", metricsTokensRouter);
app.use("/api/event-webhooks", eventWebhooksRouter); app.use("/api/event-webhooks", eventWebhooksRouter);
+62
View File
@@ -127,6 +127,68 @@ export const pushSubscriptions = sqliteTable("push_subscriptions", {
categories: text("categories"), categories: text("categories"),
}); });
// A web address, TCP port or TLS certificate the app asks about every few minutes — see services/endpointChecks.ts. The last answer is kept on the row;
// every answer is also one row in endpoint_results (kept two weeks), for uptime and answer times.
export const endpointChecks = sqliteTable("endpoint_checks", {
id: integer("id").primaryKey({ autoIncrement: true }),
name: text("name").notNull(),
kind: text("kind").notNull(), // http | tcp | tls
target: text("target").notNull(), // http: the web address; tcp and tls: the host
port: integer("port"), // tcp and tls
expectStatus: text("expect_status"), // http: "200-299, 301"; null = 200-399
keyword: text("keyword"), // http: text the page must contain
insecure: integer("insecure", { mode: "boolean" }).notNull().default(false), // accept a certificate that isn't trusted (self-signed, a private CA)
intervalMinutes: integer("interval_minutes").notNull().default(5),
timeoutSeconds: integer("timeout_seconds").notNull().default(10),
failuresBeforeAlert: integer("failures_before_alert").notNull().default(2),
warnDays: integer("warn_days").notNull().default(14), // warn this long before a certificate expires
serverId: integer("server_id"), // soft reference: the server this is about, so its maintenance silences the alerts
enabled: integer("enabled", { mode: "boolean" }).notNull().default(true),
lastCheckedAt: text("last_checked_at"),
lastOk: integer("last_ok", { mode: "boolean" }),
lastMessage: text("last_message"),
lastResponseMs: integer("last_response_ms"),
certExpiresAt: text("cert_expires_at"), // ISO
consecutiveFailures: integer("consecutive_failures").notNull().default(0),
lastChangeAt: text("last_change_at"), // when it last went from working to failing or back
createdAt: text("created_at")
.notNull()
.default(sql`(current_timestamp)`),
});
export const endpointResults = sqliteTable(
"endpoint_results",
{
id: integer("id").primaryKey({ autoIncrement: true }),
checkId: integer("check_id")
.notNull()
.references(() => endpointChecks.id, { onDelete: "cascade" }),
at: text("at").notNull(), // ISO
ok: integer("ok", { mode: "boolean" }).notNull(),
ms: integer("ms"),
message: text("message"),
},
(t) => [index("endpoint_results_check_idx").on(t.checkId, t.at)],
);
// One time somebody tried to restore a backup, and how it went — see services/restoreTests.ts. History is kept; the newest per subject is what counts.
export const restoreTests = sqliteTable(
"restore_tests",
{
id: integer("id").primaryKey({ autoIncrement: true }),
subjectKey: text("subject_key").notNull(), // guest:<integration id>:<vmid> for a Proxmox guest, custom:<slug> for anything else
label: text("label").notNull(),
testedAt: text("tested_at").notNull(), // YYYY-MM-DD
result: text("result").notNull(), // ok | partial | failed | not_needed
note: text("note"),
testedBy: text("tested_by"),
createdAt: text("created_at")
.notNull()
.default(sql`(current_timestamp)`),
},
(t) => [index("restore_tests_subject_idx").on(t.subjectKey, t.testedAt)],
);
// "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose // "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose
// end was deleted is ignored. When B is down, A's alert is folded under B's — see services/dependencies.ts. // end was deleted is ignored. When B is down, A's alert is folded under B's — see services/dependencies.ts.
export const dependencies = sqliteTable( export const dependencies = sqliteTable(
+2
View File
@@ -20,6 +20,7 @@ import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceSchedule
import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js"; import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js";
import { initUpdateRuns } from "./services/updateRuns.js"; import { initUpdateRuns } from "./services/updateRuns.js";
import { initEscalation } from "./services/escalation.js"; import { initEscalation } from "./services/escalation.js";
import { initEndpointMonitor } from "./services/endpointMonitor.js";
import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js"; import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js";
import { initIncidentRecorder } from "./services/incidents.js"; import { initIncidentRecorder } from "./services/incidents.js";
import { getAlerts } from "./services/alerts.js"; import { getAlerts } from "./services/alerts.js";
@@ -59,6 +60,7 @@ initUpdateRuns();
await initSelfBackupScheduler(); await initSelfBackupScheduler();
initIncidentRecorder(() => getAlerts(false)); initIncidentRecorder(() => getAlerts(false));
initEscalation(); initEscalation();
initEndpointMonitor();
createApp().listen(env.port, () => { createApp().listen(env.port, () => {
console.log(`homelab-manager listening on port ${env.port}`); console.log(`homelab-manager listening on port ${env.port}`);
+70 -2
View File
@@ -1,6 +1,14 @@
import { Router } from "express"; import { Router } from "express";
import { requireAuth } from "../auth/middleware.js"; import { desc, eq } from "drizzle-orm";
import { z } from "zod";
import { db } from "../db/client.js";
import { restoreTests } from "../db/schema.js";
import { requireAuth, requireRole } from "../auth/middleware.js";
import { recordAudit } from "../services/audit.js";
import { invalidateAlertsCache } from "../services/alerts.js";
import { DEFAULT_STALE_HOURS, getBackupCoverage } from "../services/backupCoverage.js"; import { DEFAULT_STALE_HOURS, getBackupCoverage } from "../services/backupCoverage.js";
import { customKey, guestKey, KEY_PATTERN, latestTests, restoreInfo, RESULTS } from "../services/restoreTests.js";
import { getSettings } from "../services/settingsStore.js";
import { asyncHandler } from "../utils/asyncHandler.js"; import { asyncHandler } from "../utils/asyncHandler.js";
export const backupsRouter = Router(); export const backupsRouter = Router();
@@ -9,8 +17,68 @@ export const backupsRouter = Router();
backupsRouter.use(requireAuth); backupsRouter.use(requireAuth);
// `staleDays`: how old the newest backup may be before a guest counts as stale (default 2). `refresh=1` asks Proxmox again now. // `staleDays`: how old the newest backup may be before a guest counts as stale (default 2). `refresh=1` asks Proxmox again now.
// Each guest also says how its newest restore test went (or that there was none).
backupsRouter.get("/coverage", asyncHandler(async (req, res) => { backupsRouter.get("/coverage", asyncHandler(async (req, res) => {
const days = Number(req.query.staleDays); const days = Number(req.query.staleDays);
const staleHours = Number.isFinite(days) && days >= 1 && days <= 60 ? Math.round(days * 24) : DEFAULT_STALE_HOURS; const staleHours = Number.isFinite(days) && days >= 1 && days <= 60 ? Math.round(days * 24) : DEFAULT_STALE_HOURS;
res.json(await getBackupCoverage({ staleHours, refresh: req.query.refresh === "1" })); const report = await getBackupCoverage({ staleHours, refresh: req.query.refresh === "1" });
const now = new Date();
const limit = (await getSettings()).healthChecks.restoreTestDays;
const latest = await latestTests();
res.json({
...report,
restoreTestDays: limit,
guests: report.guests.map((g) => ({ ...g, restoreKey: guestKey(g.integrationId, g.vmid), restoreTest: restoreInfo(latest.get(guestKey(g.integrationId, g.vmid)), now, limit) })),
});
}));
// The history of restore tests, and how each thing that isn't a Proxmox guest (a share, a database) stands.
backupsRouter.get("/restore-tests", asyncHandler(async (_req, res) => {
const now = new Date();
const limit = (await getSettings()).healthChecks.restoreTestDays;
const latest = await latestTests();
const recent = await db.select().from(restoreTests).orderBy(desc(restoreTests.testedAt), desc(restoreTests.id)).limit(100);
res.json({
limitDays: limit,
tests: recent,
custom: [...latest.values()].filter((r) => r.subjectKey.startsWith("custom:")).map((r) => ({ key: r.subjectKey, label: r.label, ...restoreInfo(r, now, limit) })).sort((a, b) => a.label.localeCompare(b.label)),
});
}));
const testSchema = z.object({
/** guest:<integration>:<vmid>, or left out for something that isn't a guest (then it is told apart by its label). */
key: z.string().optional(),
label: z.string().trim().min(1).max(120),
testedAt: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).optional(),
result: z.enum(RESULTS),
note: z.string().trim().max(500).optional(),
});
backupsRouter.post("/restore-tests", requireRole("operator"), asyncHandler(async (req, res) => {
const parsed = testSchema.safeParse(req.body);
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Say what was tested, how it went, and on what day (YYYY-MM-DD).", details: parsed.error.flatten() });
const d = parsed.data;
const key = d.key ?? customKey(d.label);
if (!key || !KEY_PATTERN.test(key)) return res.status(400).json({ error: "invalid_key", message: "That can't be told apart from other things; give it a name with some letters or digits." });
const today = new Date().toISOString().slice(0, 10);
const testedAt = d.testedAt ?? today;
if (Number.isNaN(Date.parse(`${testedAt}T00:00:00Z`)) || new Date(`${testedAt}T00:00:00Z`).toISOString().slice(0, 10) !== testedAt) return res.status(400).json({ error: "invalid_date", message: "That isn't a real date." });
if (testedAt > today) return res.status(400).json({ error: "future", message: "A restore test can't be in the future." });
const actor = req.currentUser!;
const [row] = await db.insert(restoreTests).values({ subjectKey: key, label: d.label, testedAt, result: d.result, note: d.note || null, testedBy: actor.name ?? actor.email ?? actor.oidcSub }).returning();
await recordAudit({ actor, category: "restore_test", action: "record", targetType: "restore_test", targetId: row.id, detail: { name: row.label, result: row.result, testedAt, note: row.note } });
invalidateAlertsCache();
res.status(201).json({ test: row });
}));
// A mistaken entry can be taken out; that is an administrator's to do, since it is what the alerts are judged on.
backupsRouter.delete("/restore-tests/:id", requireRole("admin"), asyncHandler(async (req, res) => {
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
const [before] = await db.select().from(restoreTests).where(eq(restoreTests.id, id)).limit(1);
if (!before) return res.status(404).json({ error: "not_found" });
await db.delete(restoreTests).where(eq(restoreTests.id, id));
await recordAudit({ actor: req.currentUser!, category: "restore_test", action: "delete", targetType: "restore_test", targetId: id, detail: { name: before.label, testedAt: before.testedAt } });
invalidateAlertsCache();
res.status(204).end();
})); }));
+27
View File
@@ -0,0 +1,27 @@
import { Router } from "express";
import { requireAuth, requireRole } from "../auth/middleware.js";
import { CHANGE_KINDS, KIND_LABELS, getChangeLog, type ChangeKind } from "../services/changeLog.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const changesRouter = Router();
// Like the audit log it is made from: operators and administrators.
changesRouter.use(requireAuth, requireRole("operator"));
// ?days=7 &kind=settings &serverId=3 &q=word &before=ISO (the next page) &limit=200
changesRouter.get("/", asyncHandler(async (req, res) => {
const days = Number(req.query.days);
const serverId = typeof req.query.serverId === "string" && /^\d+$/.test(req.query.serverId) ? Number(req.query.serverId) : undefined;
const kind = CHANGE_KINDS.find((k) => k === req.query.kind) as ChangeKind | undefined;
const before = typeof req.query.before === "string" && !Number.isNaN(Date.parse(req.query.before)) ? req.query.before : undefined;
const limit = Number(req.query.limit);
const result = await getChangeLog({
days: Number.isFinite(days) && days >= 1 ? days : 7,
kind,
serverId,
q: typeof req.query.q === "string" ? req.query.q.slice(0, 100) : undefined,
before,
limit: Number.isFinite(limit) && limit >= 1 ? limit : 200,
});
res.json({ ...result, kinds: CHANGE_KINDS.map((k) => ({ id: k, label: KIND_LABELS[k] })) });
}));
+2 -2
View File
@@ -108,7 +108,7 @@ dependenciesRouter.post("/", requireRole("operator"), asyncHandler(async (req, r
return res.status(409).json({ error: "loop", message: `${graph.names.get(to)} already depends on ${graph.names.get(from)}, so this would make them wait for each other.` }); return res.status(409).json({ error: "loop", message: `${graph.names.get(to)} already depends on ${graph.names.get(from)}, so this would make them wait for each other.` });
} }
const [row] = await db.insert(dependencies).values(d).returning(); const [row] = await db.insert(dependencies).values(d).returning();
await recordAudit({ actor: req.currentUser!, category: "dependency", action: "create", targetType: "dependency", targetId: row.id, detail: { from: graph.names.get(from), to: graph.names.get(to) } }); await recordAudit({ actor: req.currentUser!, category: "dependency", action: "create", targetType: "dependency", targetId: row.id, detail: { from: graph.names.get(from), to: graph.names.get(to), fromKey: from, toKey: to } });
invalidateAlertsCache(); invalidateAlertsCache();
res.status(201).json({ id: row.id }); res.status(201).json({ id: row.id });
})); }));
@@ -126,7 +126,7 @@ dependenciesRouter.delete("/:id", requireRole("operator"), asyncHandler(async (r
action: "delete", action: "delete",
targetType: "dependency", targetType: "dependency",
targetId: id, targetId: id,
detail: { from: graph.names.get(nodeKey(before.fromType as "server", before.fromId)) ?? `${before.fromType} #${before.fromId}`, to: graph.names.get(nodeKey(before.toType as "server", before.toId)) ?? `${before.toType} #${before.toId}` }, detail: { from: graph.names.get(nodeKey(before.fromType as "server", before.fromId)) ?? `${before.fromType} #${before.fromId}`, to: graph.names.get(nodeKey(before.toType as "server", before.toId)) ?? `${before.toType} #${before.toId}`, fromKey: `${before.fromType}:${before.fromId}`, toKey: `${before.toType}:${before.toId}` },
}); });
invalidateAlertsCache(); invalidateAlertsCache();
res.status(204).end(); res.status(204).end();
+134
View File
@@ -0,0 +1,134 @@
import { Router } from "express";
import { eq } from "drizzle-orm";
import { z } from "zod";
import { db } from "../db/client.js";
import { endpointChecks, servers } from "../db/schema.js";
import { requireAuth, requireRole } from "../auth/middleware.js";
import { recordAudit } from "../services/audit.js";
import { invalidateAlertsCache } from "../services/alerts.js";
import { CHECK_KINDS, isStatusSpec, loadStats, runCheck } from "../services/endpointChecks.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const endpointsRouter = Router();
endpointsRouter.use(requireAuth);
const MAX_CHECKS = 100;
const checkSchema = z
.object({
name: z.string().trim().min(1).max(100),
kind: z.enum(CHECK_KINDS),
target: z.string().trim().min(1).max(500),
port: z.number().int().min(1).max(65535).nullable().optional(),
expectStatus: z.string().trim().max(60).nullable().optional(),
keyword: z.string().max(200).nullable().optional(),
insecure: z.boolean().optional(),
intervalMinutes: z.number().int().min(1).max(1440).optional(),
timeoutSeconds: z.number().int().min(1).max(60).optional(),
failuresBeforeAlert: z.number().int().min(1).max(20).optional(),
warnDays: z.number().int().min(1).max(90).optional(),
serverId: z.number().int().positive().nullable().optional(),
enabled: z.boolean().optional(),
})
.superRefine((v, ctx) => {
if (v.kind === "http") {
let ok = false;
try {
const u = new URL(v.target);
ok = u.protocol === "http:" || u.protocol === "https:";
} catch {
ok = false;
}
if (!ok) ctx.addIssue({ code: "custom", path: ["target"], message: "A web check needs an address that starts with http:// or https://." });
if (v.expectStatus && !isStatusSpec(v.expectStatus)) ctx.addIssue({ code: "custom", path: ["expectStatus"], message: "Use status codes like 200, or ranges like 200-299, separated by commas." });
} else {
if (v.port == null) ctx.addIssue({ code: "custom", path: ["port"], message: v.kind === "tcp" ? "A port check needs a port." : "A certificate check needs a port (usually 443)." });
if (/[/\s]/.test(v.target)) ctx.addIssue({ code: "custom", path: ["target"], message: "Give just the host name or address, without http:// or a path." });
}
});
/** A blank text field is no value. */
const blank = (v: string | null | undefined) => (v === undefined || v === null || v.trim() === "" ? null : v.trim());
// Everyone signed in can see how things are doing, like the pages the same information appears on.
endpointsRouter.get("/", asyncHandler(async (_req, res) => {
const rows = await db.select().from(endpointChecks).orderBy(endpointChecks.name);
const stats = await loadStats(rows.map((r) => r.id));
const serverNames = new Map((await db.select({ id: servers.id, name: servers.name }).from(servers)).map((s) => [s.id, s.name]));
res.json({
checks: rows.map((r) => ({ ...r, server: r.serverId !== null && serverNames.has(r.serverId) ? { id: r.serverId, name: serverNames.get(r.serverId)! } : null, stats: stats.get(r.id) })),
servers: [...serverNames].map(([id, name]) => ({ id, name })).sort((a, b) => a.name.localeCompare(b.name)),
});
}));
function valuesOf(d: z.infer<typeof checkSchema>) {
return {
name: d.name,
kind: d.kind,
target: d.target,
port: d.kind === "http" ? null : (d.port ?? null),
expectStatus: d.kind === "http" ? blank(d.expectStatus) : null,
keyword: d.kind === "http" ? blank(d.keyword) : null,
insecure: d.insecure ?? false,
intervalMinutes: d.intervalMinutes ?? 5,
timeoutSeconds: d.timeoutSeconds ?? 10,
failuresBeforeAlert: d.failuresBeforeAlert ?? 2,
warnDays: d.warnDays ?? 14,
serverId: d.serverId ?? null,
enabled: d.enabled ?? true,
};
}
async function serverExists(id: number | null | undefined) {
return id == null || (await db.select({ id: servers.id }).from(servers).where(eq(servers.id, id))).length > 0;
}
// Asking things on the network is an administrator's to set up.
endpointsRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => {
const parsed = checkSchema.safeParse(req.body);
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "That isn't a check.", details: parsed.error.flatten() });
if ((await db.select({ id: endpointChecks.id }).from(endpointChecks)).length >= MAX_CHECKS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_CHECKS} checks.` });
if (!(await serverExists(parsed.data.serverId))) return res.status(400).json({ error: "unknown_server", message: "That server doesn't exist." });
const [row] = await db.insert(endpointChecks).values(valuesOf(parsed.data)).returning();
await recordAudit({ actor: req.currentUser!, category: "endpoint", action: "create", targetType: "endpoint", targetId: row.id, detail: { name: row.name, kind: row.kind, target: row.target } });
res.status(201).json({ check: row });
}));
endpointsRouter.put("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
const parsed = checkSchema.safeParse(req.body);
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.success ? "That isn't a check." : (parsed.error.issues[0]?.message ?? "That isn't a check."), details: parsed.success ? undefined : parsed.error.flatten() });
const [before] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id)).limit(1);
if (!before) return res.status(404).json({ error: "not_found" });
if (!(await serverExists(parsed.data.serverId))) return res.status(400).json({ error: "unknown_server", message: "That server doesn't exist." });
const v = valuesOf(parsed.data);
// What is asked changed, so what was last found out about it no longer stands.
const reset = v.kind !== before.kind || v.target !== before.target || v.port !== before.port ? { lastCheckedAt: null, lastOk: null, lastMessage: null, lastResponseMs: null, certExpiresAt: null, consecutiveFailures: 0 } : {};
const [row] = await db.update(endpointChecks).set({ ...v, ...reset }).where(eq(endpointChecks.id, id)).returning();
await recordAudit({ actor: req.currentUser!, category: "endpoint", action: "update", targetType: "endpoint", targetId: id, detail: { name: row.name, kind: row.kind, target: row.target } });
invalidateAlertsCache();
res.json({ check: row });
}));
endpointsRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
const [before] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id)).limit(1);
if (!before) return res.status(404).json({ error: "not_found" });
await db.delete(endpointChecks).where(eq(endpointChecks.id, id));
await recordAudit({ actor: req.currentUser!, category: "endpoint", action: "delete", targetType: "endpoint", targetId: id, detail: { name: before.name } });
invalidateAlertsCache();
res.status(204).end();
}));
// Ask now, instead of at the next time it is due.
endpointsRouter.post("/:id/check", requireRole("operator"), asyncHandler(async (req, res) => {
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
const [row] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id)).limit(1);
if (!row) return res.status(404).json({ error: "not_found" });
const result = await runCheck(row);
invalidateAlertsCache();
res.json({ result });
}));
+1 -1
View File
@@ -143,7 +143,7 @@ const updateSchema = z.object({
.partial() .partial()
.optional(), .optional(),
healthChecks: z healthChecks: z
.object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160) }) .object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730) })
.partial() .partial()
.optional(), .optional(),
quietHours: z quietHours: z
+35 -1
View File
@@ -11,7 +11,7 @@
*/ */
import { eq } from "drizzle-orm"; import { eq } from "drizzle-orm";
import { db } from "../db/client.js"; import { db } from "../db/client.js";
import { integrations, secrets, servers } from "../db/schema.js"; import { endpointChecks, integrations, secrets, servers } from "../db/schema.js";
import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js"; import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js";
import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js"; import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js";
import { getBackupCoverage, staleBackups } from "./backupCoverage.js"; import { getBackupCoverage, staleBackups } from "./backupCoverage.js";
@@ -34,6 +34,8 @@ import { getFailingSources } from "./integrationHealthMonitor.js";
import { getSettings } from "./settingsStore.js"; import { getSettings } from "./settingsStore.js";
import { sourceLabel } from "./notify.js"; import { sourceLabel } from "./notify.js";
import { alertIdOfNode, dependencyCover, integrationTypeCause, takenByType } from "./dependencies.js"; import { alertIdOfNode, dependencyCover, integrationTypeCause, takenByType } from "./dependencies.js";
import { endpointConditions } from "./endpointChecks.js";
import { latestTests, restoreAlerts } from "./restoreTests.js";
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js"; import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
export interface AlertsReport { export interface AlertsReport {
@@ -62,6 +64,7 @@ const SOURCES: Record<string, { label: string; link: string }> = {
osticket: { label: "osTicket", link: "/osticket" }, osticket: { label: "osTicket", link: "/osticket" },
secrets: { label: "Secrets", link: "/secrets" }, secrets: { label: "Secrets", link: "/secrets" },
domains: { label: "Domains", link: "/domains" }, domains: { label: "Domains", link: "/domains" },
endpoint: { label: "Endpoint", link: "/endpoints" },
}; };
const SOURCE_TIMEOUT_MS = 20_000; const SOURCE_TIMEOUT_MS = 20_000;
@@ -268,6 +271,37 @@ export async function collectAlerts(): Promise<AlertsReport> {
target: s.guest.server ? targetOf("server", s.guest.server.id) : integrationTarget(s.guest.integrationId), target: s.guest.server ? targetOf("server", s.guest.server.id) : integrationTarget(s.guest.integrationId),
}); });
} }
// A restore that failed, or hasn't been tried in too long (the limit is in Settings; a failure is always raised).
const { healthChecks } = await getSettings();
for (const r of restoreAlerts(report, await latestTests(), healthChecks.restoreTestDays, now)) {
const g = r.guest;
push({
category: "backup",
severity: r.severity,
type: g ? "proxmox" : "homelab",
message: r.message,
key: r.key,
link: "/backups",
silenced: g ? subjects.has(`integration:${g.integrationId}`) || (g.server !== null && subjects.has(`server:${g.server.id}`)) : false,
target: g ? (g.server ? targetOf("server", g.server.id) : integrationTarget(g.integrationId)) : null,
});
}
}),
// What the endpoint checks have found: an address, port or certificate that has stopped being fine.
check("Endpoint checks", async () => {
for (const c of endpointConditions(await db.select().from(endpointChecks), new Date())) {
push({
category: c.category,
severity: c.severity,
type: "endpoint",
message: c.message,
key: c.key,
link: "/endpoints",
silenced: c.serverId !== null && subjects.has(`server:${c.serverId}`),
target: c.serverId !== null ? targetOf("server", c.serverId) : null,
});
}
}), }),
// Homelab Manager's own backups: if they are switched on, there should be a recent one. This is what turns a backup that has quietly // Homelab Manager's own backups: if they are switched on, there should be a recent one. This is what turns a backup that has quietly
+307
View File
@@ -0,0 +1,307 @@
/**
* The change log: one timeline of what has changed in the homelab and in this app — agents updated, settings changed, maintenance started and
* ended, dependencies edited, hardware added, endpoints set up, alerts raised and cleared — each in a sentence, newest first, and filterable
* by what kind of thing it was, by server, by time and by a word. It is made from the audit log and the incident record, so it needs nothing
* of its own; the Audit Log page remains the raw table behind it.
*/
import { and, desc, gte, or, lte } from "drizzle-orm";
import { db } from "../db/client.js";
import { alertIncidents, auditLog, endpointChecks, integrations, servers } from "../db/schema.js";
import { auditTimeToIso, describeAudit, type ActivityTone } from "./serverActivity.js";
export const CHANGE_KINDS = ["servers", "maintenance", "alerts", "settings", "integrations", "dependencies", "inventory", "backups", "endpoints", "other"] as const;
export type ChangeKind = (typeof CHANGE_KINDS)[number];
export const KIND_LABELS: Record<ChangeKind, string> = {
servers: "Servers and agents",
maintenance: "Maintenance",
alerts: "Alerts and incidents",
settings: "Settings and access",
integrations: "Integrations, DNS and secrets",
dependencies: "Dependencies",
inventory: "Inventory",
backups: "Backups and restore tests",
endpoints: "Endpoint checks",
other: "Other",
};
const KIND_OF_CATEGORY: Record<string, ChangeKind> = {
server: "servers",
task: "servers",
update_run: "servers",
tag: "servers",
maintenance: "maintenance",
alert: "alerts",
incident: "alerts",
alert_ticket: "alerts",
settings: "settings",
user: "settings",
push: "settings",
share_link: "settings",
metrics_token: "settings",
webhook: "settings",
integration: "integrations",
dns: "integrations",
secret: "integrations",
domain: "integrations",
ipam: "integrations",
network: "integrations",
consistency: "integrations",
dependency: "dependencies",
inventory: "inventory",
runbook: "other",
db_backup: "backups",
restore_test: "backups",
endpoint: "endpoints",
};
/** Sign-ins and a person's own data export are not changes to the homelab; the diagnostic log has its own page. */
const LEFT_OUT = new Set(["session", "privacy", "diag_log"]);
export interface ChangeEntry {
/** Stable, so a row can be keyed on it. */
id: string;
/** ISO. */
at: string;
kind: ChangeKind;
tone: ActivityTone;
title: string;
detail: string | null;
/** A person's name, or "system" for the app's own checks; null when nobody did it. */
actor: string | null;
/** What it is about, and where to read more. */
target: { type: string; id: string | null; name: string | null } | null;
link: string | null;
/** The server it concerns, when it concerns one (so the log can be narrowed to a server). */
serverId: number | null;
}
const humanize = (s: string) => {
const t = s.replace(/[_-]+/g, " ");
return t.charAt(0).toUpperCase() + t.slice(1);
};
function parseDetail(detail: string | null): Record<string, unknown> {
if (!detail) return {};
try {
const v = JSON.parse(detail);
return v && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : {};
} catch {
return {};
}
}
const str = (v: unknown) => (typeof v === "string" && v ? v : null);
const MAX_SUMMARY = 220;
/** "notifications: healthAlerts true → false" for a settings update; nothing for what is only a name. */
function settingsSummary(detail: Record<string, unknown>): string | null {
const changes = detail.changes;
if (!changes || typeof changes !== "object") return null;
const parts: string[] = [];
for (const [section, fields] of Object.entries(changes as Record<string, Record<string, unknown>>)) {
for (const [field, change] of Object.entries(fields ?? {})) {
if (change && typeof change === "object" && "to" in (change as object)) {
const c = change as { from: unknown; to: unknown };
parts.push(`${section}.${field}: ${JSON.stringify(c.from)} → ${JSON.stringify(c.to)}`);
} else parts.push(`${section}.${field} changed`);
}
}
const text = parts.join(" · ");
return text.length > MAX_SUMMARY ? `${text.slice(0, MAX_SUMMARY)}…` : text || null;
}
/** What is left of the detail once the name and what is already in the sentence are taken out. */
function genericSummary(detail: Record<string, unknown>, skip: string[] = []): string | null {
const parts = Object.entries(detail)
.filter(([k, v]) => !["name", "message", ...skip].includes(k) && v !== null && v !== undefined && v !== "" && !(Array.isArray(v) && v.length === 0))
.map(([k, v]) => `${k}: ${typeof v === "object" ? JSON.stringify(v) : String(v)}`);
const text = parts.join(" · ");
return text.length > MAX_SUMMARY ? `${text.slice(0, MAX_SUMMARY)}…` : text || null;
}
/** One sentence for an audit entry. Servers, alerts and update runs have their own wording (the same as a server's Activity card); the rest is made here. */
export function describeChange(category: string, action: string, detail: Record<string, unknown>): { title: string; tone: ActivityTone; detail: string | null } {
const name = str(detail.name);
const minutes = typeof detail.minutes === "number" ? detail.minutes : null;
const forHow = minutes !== null ? (minutes >= 120 ? `${Math.round(minutes / 60)} h` : `${minutes} min`) : null;
switch (`${category}/${action}`) {
case "maintenance/start":
return { title: `Maintenance started${name ? ` on ${name}` : ""}`, tone: "info", detail: [forHow && `for ${forHow}`, str(detail.reason)].filter(Boolean).join(" · ") || null };
case "maintenance/extend":
return { title: `Maintenance extended${name ? ` on ${name}` : ""}`, tone: "info", detail: [forHow && `for ${forHow} from now`, str(detail.reason)].filter(Boolean).join(" · ") || null };
case "maintenance/end":
return { title: `Maintenance ended${name ? ` on ${name}` : ""}`, tone: "info", detail: null };
case "maintenance/start_bulk":
return { title: "Maintenance started on many at once", tone: "info", detail: genericSummary(detail) };
case "dependency/create":
return { title: `${str(detail.from) ?? "Something"} now needs ${str(detail.to) ?? "something"}`, tone: "neutral", detail: null };
case "dependency/delete":
return { title: `${str(detail.from) ?? "Something"} no longer needs ${str(detail.to) ?? "something"}`, tone: "neutral", detail: null };
case "inventory/create":
return { title: `Hardware added${name ? `: ${name}` : ""}`, tone: "neutral", detail: genericSummary(detail, ["kind"]) };
case "inventory/update":
return { title: `Hardware changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: genericSummary(detail) };
case "inventory/delete":
return { title: `Hardware removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
case "inventory/refresh_from_agent":
return { title: `Hardware details taken from the agent${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
case "inventory/update_config":
return { title: "Price of power changed", tone: "neutral", detail: null };
case "endpoint/create":
return { title: `Endpoint check added${name ? `: ${name}` : ""}`, tone: "neutral", detail: [str(detail.kind), str(detail.target)].filter(Boolean).join(" · ") || null };
case "endpoint/update":
return { title: `Endpoint check changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: [str(detail.kind), str(detail.target)].filter(Boolean).join(" · ") || null };
case "endpoint/delete":
return { title: `Endpoint check removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
case "restore_test/record": {
const result = str(detail.result);
const tone: ActivityTone = result === "failed" ? "bad" : result === "not_needed" ? "neutral" : "good";
const said = result === "ok" ? "passed" : result === "partial" ? "partly worked" : result === "failed" ? "failed" : "marked as not needed";
return { title: `Restore test of ${name ?? "something"} ${said}`, tone, detail: [str(detail.testedAt), str(detail.note)].filter(Boolean).join(" · ") || null };
}
case "restore_test/delete":
return { title: `A restore test of ${name ?? "something"} was removed`, tone: "neutral", detail: str(detail.testedAt) };
case "settings/update": {
const sections = Array.isArray(detail.sections) ? (detail.sections as string[]).join(", ") : null;
return { title: `Settings changed${sections ? `: ${sections}` : ""}`, tone: "neutral", detail: settingsSummary(detail) };
}
case "push/subscribe":
return { title: `A browser was added for push notifications${str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: "neutral", detail: null };
case "push/unsubscribe":
return { title: `A browser was removed from push notifications${str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: "neutral", detail: null };
case "alert/escalate":
return { title: "An alert nobody had acknowledged was escalated", tone: "bad", detail: [str(detail.message), typeof detail.afterMinutes === "number" ? `step after ${detail.afterMinutes} min` : null].filter(Boolean).join(" · ") || null };
}
if (category === "server" || category === "alert" || category === "update_run") {
const d = describeAudit(category, action, detail);
return { title: d.title, tone: d.tone, detail: d.detail };
}
return { title: `${humanize(category)}: ${action.replace(/_/g, " ")}${name ? ` — ${name}` : ""}`, tone: "neutral", detail: genericSummary(detail) };
}
export interface ChangeQuery {
/** How far back, in days (default 7). */
days?: number;
kind?: ChangeKind;
serverId?: number;
/** A word or two that must all appear somewhere in the entry. */
q?: string;
/** Only entries older than this (ISO), for the next page. */
before?: string;
limit?: number;
now?: Date;
}
const sqlTime = (d: Date) => d.toISOString().slice(0, 19).replace("T", " ");
function serverIdOf(targetType: string | null, targetId: string | null, detail: Record<string, unknown>): number | null {
if (targetType === "server" && targetId && /^\d+$/.test(targetId)) return Number(targetId);
// Some entries about something else name the servers they touch by key ("server:3").
for (const k of ["fromKey", "toKey", "key"]) {
const m = /^server:(\d+)$/.exec(typeof detail[k] === "string" ? (detail[k] as string) : "");
if (m) return Number(m[1]);
}
return null;
}
const touchesServer = (e: ChangeEntry, id: number, detail: Record<string, unknown>) =>
e.serverId === id || ["fromKey", "toKey"].some((k) => detail[k] === `server:${id}`);
/** The log, newest first, narrowed as asked. `more` says whether older entries are left beyond the ones returned. */
export async function getChangeLog(query: ChangeQuery = {}): Promise<{ entries: ChangeEntry[]; more: boolean }> {
const now = query.now ?? new Date();
const days = Math.min(Math.max(query.days ?? 7, 1), 400);
const limit = Math.min(Math.max(query.limit ?? 200, 1), 500);
const since = new Date(now.getTime() - days * 86_400_000);
const until = query.before ? new Date(query.before) : null;
const [serverRows, integrationRows, endpointRows] = await Promise.all([
db.select({ id: servers.id, name: servers.name }).from(servers),
db.select({ id: integrations.id, name: integrations.name }).from(integrations),
db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks),
]);
const serverName = new Map(serverRows.map((s) => [s.id, s.name]));
const integrationName = new Map(integrationRows.map((i) => [i.id, i.name]));
const endpointName = new Map(endpointRows.map((e) => [e.id, e.name]));
const entries: { entry: ChangeEntry; detail: Record<string, unknown> }[] = [];
const audits = await db
.select()
.from(auditLog)
.where(until ? and(gte(auditLog.createdAt, sqlTime(since)), lte(auditLog.createdAt, sqlTime(until))) : gte(auditLog.createdAt, sqlTime(since)))
.orderBy(desc(auditLog.createdAt), desc(auditLog.id))
.limit(5000);
for (const a of audits) {
if (LEFT_OUT.has(a.category)) continue;
const detail = parseDetail(a.detail);
const d = describeChange(a.category, a.action, detail);
const serverId = serverIdOf(a.targetType, a.targetId, detail);
const idNum = a.targetId && /^\d+$/.test(a.targetId) ? Number(a.targetId) : null;
let targetName = str(detail.name);
let link: string | null = null;
if (a.targetType === "server" && idNum !== null) {
targetName = serverName.get(idNum) ?? targetName;
link = serverName.has(idNum) ? `/servers/${idNum}` : null;
} else if (a.targetType === "integration" && idNum !== null) {
targetName = integrationName.get(idNum) ?? targetName;
link = "/integrations";
} else if (a.targetType === "endpoint" && idNum !== null) {
targetName = endpointName.get(idNum) ?? targetName;
link = "/endpoints";
} else if (a.targetType === "asset" && idNum !== null) link = `/inventory?asset=${idNum}`;
else if (a.targetType === "restore_test") link = "/backups";
else if (a.targetType === "dependency") link = "/dependencies";
else if (a.targetType === "settings") link = "/settings/notifications";
entries.push({
detail,
entry: {
id: `audit-${a.id}`,
at: auditTimeToIso(a.createdAt),
kind: KIND_OF_CATEGORY[a.category] ?? "other",
tone: d.tone,
title: d.title,
detail: d.detail,
actor: a.actorLabel,
target: a.targetType ? { type: a.targetType, id: a.targetId, name: targetName } : null,
link,
serverId,
},
});
}
// Alerts raised and cleared, from the incident record.
const incidents = await db
.select()
.from(alertIncidents)
.where(or(gte(alertIncidents.openedAt, since.toISOString()), gte(alertIncidents.resolvedAt, since.toISOString())))
.orderBy(desc(alertIncidents.openedAt))
.limit(1000);
for (const i of incidents) {
const serverId = i.targetType === "server" ? i.targetId : null;
const base = { kind: "alerts" as const, actor: null, target: i.targetType ? { type: i.targetType, id: i.targetId === null ? null : String(i.targetId), name: i.targetName } : null, link: `/incidents/${i.id}`, serverId };
if (i.openedAt >= since.toISOString() && (!until || i.openedAt <= until.toISOString())) {
entries.push({ detail: {}, entry: { ...base, id: `incident-open-${i.id}`, at: i.openedAt, tone: i.peakSeverity === "critical" ? "bad" : "info", title: `Alert raised: ${i.message}`, detail: i.causedBy ? `explained by ${i.causedBy}` : null } });
}
if (i.resolvedAt && i.resolvedAt >= since.toISOString() && (!until || i.resolvedAt <= until.toISOString())) {
entries.push({ detail: {}, entry: { ...base, id: `incident-close-${i.id}`, at: i.resolvedAt, tone: "good", title: `Alert cleared: ${i.message}`, detail: null } });
}
}
const terms = (query.q ?? "").toLowerCase().split(/\s+/).filter(Boolean);
const kept = entries
.filter(({ entry, detail }) => {
if (query.kind && entry.kind !== query.kind) return false;
if (query.serverId !== undefined && !touchesServer(entry, query.serverId, detail)) return false;
if (until && entry.at >= until.toISOString()) return false; // strictly older than the cursor
if (terms.length > 0) {
const hay = `${entry.title} ${entry.detail ?? ""} ${entry.actor ?? ""} ${entry.target?.name ?? ""} ${KIND_LABELS[entry.kind]}`.toLowerCase();
if (!terms.every((t) => hay.includes(t))) return false;
}
return true;
})
.map((e) => e.entry)
.sort((a, b) => b.at.localeCompare(a.at) || b.id.localeCompare(a.id));
return { entries: kept.slice(0, limit), more: kept.length > limit };
}
+350
View File
@@ -0,0 +1,350 @@
/**
* Endpoint checks: the app asks a web address, a TCP port or a TLS certificate whether it is fine, every few minutes, and remembers how it
* went — answer time, uptime, when the certificate runs out. A check that keeps failing becomes an alert (and a notification), and so does a
* certificate that is about to expire. This is for the simple "is it up?" that doesn't need a monitoring server of its own.
*
* What is asked is typed in by an administrator, so it can't be just anything: loopback, link-local (cloud metadata) and similar addresses
* are refused, and the connection goes to the one address that passed the check rather than to the name again (utils/networkGuard.ts).
* LAN and public hosts are fine; that is what these are for.
*/
import http from "node:http";
import https from "node:https";
import net from "node:net";
import tls from "node:tls";
import type { TLSSocket } from "node:tls";
import { and, asc, eq, gte, inArray, lt } from "drizzle-orm";
import { db } from "../db/client.js";
import { endpointChecks, endpointResults } from "../db/schema.js";
import { resolveTarget } from "../utils/networkGuard.js";
export const CHECK_KINDS = ["http", "tcp", "tls"] as const;
export type CheckKind = (typeof CHECK_KINDS)[number];
export type CheckRow = typeof endpointChecks.$inferSelect;
export interface ProbeResult {
ok: boolean;
/** How long it took to answer, when it did. */
ms: number | null;
message: string;
/** ISO — when the certificate it showed runs out (https and tls checks). */
certExpiresAt?: string | null;
}
export const KEEP_RESULT_DAYS = 14;
const MAX_BODY_BYTES = 64 * 1024;
const MAX_REDIRECTS = 5;
/** "200-299, 301" → a test for a status code. Anything unreadable falls back to the usual 200–399. */
export function statusAccepter(spec: string | null | undefined): (code: number) => boolean {
const ranges: [number, number][] = [];
for (const part of (spec ?? "").split(",")) {
const m = /^\s*(\d{3})(?:\s*-\s*(\d{3}))?\s*$/.exec(part);
if (m) ranges.push([Number(m[1]), Number(m[2] ?? m[1])]);
}
if (ranges.length === 0) ranges.push([200, 399]);
return (code) => ranges.some(([lo, hi]) => code >= lo && code <= hi);
}
/** Whether a status spec is one statusAccepter would read (for refusing a typo when it is saved). */
export const isStatusSpec = (spec: string) => spec.split(",").every((p) => /^\s*\d{3}(\s*-\s*\d{3})?\s*$/.test(p));
const certExpiry = (socket: unknown): string | null => {
try {
const cert = (socket as TLSSocket).getPeerCertificate?.();
if (!cert || !cert.valid_to) return null;
const t = new Date(cert.valid_to);
return Number.isNaN(t.getTime()) ? null : t.toISOString();
} catch {
return null;
}
};
/** What to tell a person when the certificate can't be trusted: the usual reasons, said plainly. */
function trustProblem(code: string | undefined, text: string): string {
const known: Record<string, string> = {
DEPTH_ZERO_SELF_SIGNED_CERT: "it is self-signed",
SELF_SIGNED_CERT_IN_CHAIN: "it is signed by a certificate authority this server doesn't know",
UNABLE_TO_VERIFY_LEAF_SIGNATURE: "it is signed by a certificate authority this server doesn't know",
UNABLE_TO_GET_ISSUER_CERT_LOCALLY: "it is signed by a certificate authority this server doesn't know",
CERT_HAS_EXPIRED: "it has expired",
ERR_TLS_CERT_ALTNAME_INVALID: "it isn't for this name",
};
return `The certificate can't be trusted: ${(code && known[code]) || text}. If that is expected for this service, tick "Accept a certificate that isn't trusted".`;
}
interface OnceResult {
status: number;
location: string | null;
body: string;
cert: string | null;
}
function requestOnce(url: URL, address: string, insecure: boolean, timeoutMs: number, wantBody: boolean): Promise<OnceResult> {
const family = net.isIP(address) === 6 ? 6 : 4;
return new Promise((resolve, reject) => {
const isHttps = url.protocol === "https:";
const bare = url.hostname.replace(/^\[|\]$/g, "");
const req = (isHttps ? https : http).request(
{
host: bare,
port: url.port ? Number(url.port) : isHttps ? 443 : 80,
method: "GET",
path: `${url.pathname}${url.search}`,
headers: { "user-agent": "HomelabManager-endpoint-check", accept: "*/*", host: url.host },
// Connect to the address that passed the check, not to the name again.
lookup: ((_host: string, options: { all?: boolean }, cb: (...args: unknown[]) => void) => (options?.all ? cb(null, [{ address, family }]) : cb(null, address, family))) as never,
timeout: timeoutMs,
...(isHttps ? { rejectUnauthorized: !insecure, servername: net.isIP(bare) === 0 ? bare : undefined } : {}),
},
(res) => {
const cert = isHttps ? certExpiry(res.socket) : null;
const chunks: Buffer[] = [];
let size = 0;
const finish = () => resolve({ status: res.statusCode ?? 0, location: typeof res.headers.location === "string" ? res.headers.location : null, body: Buffer.concat(chunks).toString("utf8"), cert });
if (!wantBody) {
res.resume();
res.on("end", finish);
res.on("close", finish);
return;
}
res.on("data", (c: Buffer) => {
size += c.length;
chunks.push(c);
if (size >= MAX_BODY_BYTES) {
res.destroy();
finish();
}
});
res.on("end", finish);
res.on("close", finish);
},
);
req.on("timeout", () => req.destroy(new Error(`no answer within ${Math.round(timeoutMs / 1000)} s`)));
req.on("error", reject);
req.end();
});
}
async function probeHttp(c: CheckRow): Promise<ProbeResult> {
const started = performance.now();
const timeoutMs = c.timeoutSeconds * 1000;
const accepts = statusAccepter(c.expectStatus);
let url: URL;
try {
url = new URL(c.target);
if (url.protocol !== "http:" && url.protocol !== "https:") throw new Error("not http");
} catch {
return { ok: false, ms: null, message: "The address isn't a web address (it should start with http:// or https://)." };
}
let cert: string | null = null;
for (let hops = 0; ; hops++) {
const left = timeoutMs - (performance.now() - started);
if (left <= 0) return { ok: false, ms: null, message: `no answer within ${c.timeoutSeconds} s` };
let once: OnceResult;
try {
const address = await resolveTarget(url.hostname.replace(/^\[|\]$/g, ""), { allowPrivate: true });
once = await requestOnce(url, address, c.insecure, left, !!c.keyword);
} catch (err) {
const e = err as NodeJS.ErrnoException;
const text = err instanceof Error ? err.message : String(err);
const trust = url.protocol === "https:" && /^(DEPTH_ZERO|SELF_SIGNED|UNABLE_TO|CERT_|ERR_TLS_CERT)/.test(e.code ?? "");
return { ok: false, ms: null, message: trust ? trustProblem(e.code, text) : text, certExpiresAt: cert };
}
cert = once.cert ?? cert;
// Redirects are only followed when there is a keyword to look for in the page they lead to; otherwise a redirect is an answer.
if (c.keyword && [301, 302, 303, 307, 308].includes(once.status) && once.location && hops < MAX_REDIRECTS) {
url = new URL(once.location, url);
continue;
}
const ms = Math.round(performance.now() - started);
if (!accepts(once.status)) return { ok: false, ms, message: `answered ${once.status}`, certExpiresAt: cert };
if (c.keyword && !once.body.includes(c.keyword)) return { ok: false, ms, message: `answered ${once.status}, but the page doesn't contain “${c.keyword}”`, certExpiresAt: cert };
return { ok: true, ms, message: `answered ${once.status} in ${ms} ms`, certExpiresAt: cert };
}
}
async function probeTcp(c: CheckRow): Promise<ProbeResult> {
const started = performance.now();
let address: string;
try {
address = await resolveTarget(c.target, { allowPrivate: true });
} catch (err) {
return { ok: false, ms: null, message: err instanceof Error ? err.message : String(err) };
}
return new Promise((resolve) => {
const socket = net.connect({ host: address, port: c.port ?? 0 });
const done = (r: ProbeResult) => {
socket.destroy();
resolve(r);
};
socket.setTimeout(c.timeoutSeconds * 1000, () => done({ ok: false, ms: null, message: `no answer within ${c.timeoutSeconds} s` }));
socket.on("connect", () => {
const ms = Math.round(performance.now() - started);
done({ ok: true, ms, message: `port ${c.port} answered in ${ms} ms` });
});
socket.on("error", (err: NodeJS.ErrnoException) => done({ ok: false, ms: null, message: err.code === "ECONNREFUSED" ? `port ${c.port} refused the connection` : err.message }));
});
}
async function probeTls(c: CheckRow): Promise<ProbeResult> {
const started = performance.now();
let address: string;
try {
address = await resolveTarget(c.target, { allowPrivate: true });
} catch (err) {
return { ok: false, ms: null, message: err instanceof Error ? err.message : String(err) };
}
return new Promise((resolve) => {
const socket = tls.connect({ host: address, port: c.port ?? 443, servername: net.isIP(c.target) === 0 ? c.target : undefined, rejectUnauthorized: false }, () => {
const ms = Math.round(performance.now() - started);
const expires = certExpiry(socket);
const authorized = socket.authorized;
const reason = socket.authorizationError ? String(socket.authorizationError) : "";
socket.end();
if (!expires) return resolve({ ok: false, ms, message: "The server showed no certificate." });
const days = Math.floor((Date.parse(expires) - Date.now()) / 86_400_000);
if (days < 0) return resolve({ ok: false, ms, message: `The certificate expired on ${expires.slice(0, 10)}.`, certExpiresAt: expires });
if (!authorized && !c.insecure) return resolve({ ok: false, ms, message: trustProblem(reason, reason || "it didn't verify"), certExpiresAt: expires });
resolve({ ok: true, ms, message: `The certificate is good until ${expires.slice(0, 10)} (${days} days).`, certExpiresAt: expires });
});
socket.setTimeout(c.timeoutSeconds * 1000, () => {
socket.destroy();
resolve({ ok: false, ms: null, message: `no answer within ${c.timeoutSeconds} s` });
});
socket.on("error", (err) => {
resolve({ ok: false, ms: null, message: /wrong version number|packet length too long/i.test(err.message) ? `${c.target}:${c.port ?? 443} isn't speaking TLS — is that the right port?` : err.message });
});
});
}
export function probe(c: CheckRow): Promise<ProbeResult> {
return c.kind === "tcp" ? probeTcp(c) : c.kind === "tls" ? probeTls(c) : probeHttp(c);
}
/** Asks once, and writes down how it went: the check's current state, and one more point in its history. */
export async function runCheck(c: CheckRow, now: Date = new Date(), probeFn: (c: CheckRow) => Promise<ProbeResult> = probe): Promise<ProbeResult> {
let result: ProbeResult;
try {
result = await probeFn(c);
} catch (err) {
result = { ok: false, ms: null, message: err instanceof Error ? err.message : String(err) };
}
const iso = now.toISOString();
const flipped = c.lastOk === null || c.lastOk !== result.ok;
await db
.update(endpointChecks)
.set({
lastCheckedAt: iso,
lastOk: result.ok,
lastMessage: result.message.slice(0, 300),
lastResponseMs: result.ms,
certExpiresAt: result.certExpiresAt === undefined ? c.certExpiresAt : result.certExpiresAt,
consecutiveFailures: result.ok ? 0 : c.consecutiveFailures + 1,
lastChangeAt: flipped ? iso : c.lastChangeAt,
})
.where(eq(endpointChecks.id, c.id));
await db.insert(endpointResults).values({ checkId: c.id, at: iso, ok: result.ok, ms: result.ms, message: result.message.slice(0, 300) });
return result;
}
/** Runs every enabled check that is due, a few at a time. Returns how many it asked. */
export async function runDueChecks(now: Date = new Date(), probeFn?: (c: CheckRow) => Promise<ProbeResult>): Promise<number> {
const rows = await db.select().from(endpointChecks).where(eq(endpointChecks.enabled, true)).orderBy(asc(endpointChecks.id));
const due = rows.filter((c) => !c.lastCheckedAt || now.getTime() - Date.parse(c.lastCheckedAt) >= c.intervalMinutes * 60_000 - 5_000);
let next = 0;
const worker = async () => {
while (next < due.length) {
const c = due[next++];
await runCheck(c, now, probeFn);
}
};
await Promise.all(Array.from({ length: Math.min(5, due.length) }, worker));
return due.length;
}
export async function pruneResults(now: Date = new Date()): Promise<void> {
await db.delete(endpointResults).where(lt(endpointResults.at, new Date(now.getTime() - KEEP_RESULT_DAYS * 86_400_000).toISOString()));
}
// ─── What the history says ─────────────────────────────────────────────────
export interface CheckStats {
uptime24h: number | null;
uptime7d: number | null;
avgMs24h: number | null;
/** The last results, oldest first: how the recent checks went. */
recent: { at: string; ok: boolean; ms: number | null }[];
}
/** Pure: uptime, average answer time and the recent run of results, from a check's stored results. */
export function summariseResults(results: { at: string; ok: boolean; ms: number | null }[], now: Date): CheckStats {
const sorted = [...results].sort((a, b) => a.at.localeCompare(b.at));
const since = (ms: number) => sorted.filter((r) => now.getTime() - Date.parse(r.at) <= ms);
const pct = (list: typeof sorted) => (list.length === 0 ? null : Math.round((list.filter((r) => r.ok).length / list.length) * 1000) / 10);
const day = since(86_400_000);
const timed = day.filter((r) => r.ok && r.ms !== null);
return {
uptime24h: pct(day),
uptime7d: pct(since(7 * 86_400_000)),
avgMs24h: timed.length === 0 ? null : Math.round(timed.reduce((n, r) => n + (r.ms ?? 0), 0) / timed.length),
recent: sorted.slice(-30),
};
}
export async function loadStats(ids: number[], now: Date = new Date()): Promise<Map<number, CheckStats>> {
const out = new Map<number, CheckStats>();
if (ids.length === 0) return out;
const rows = await db
.select({ checkId: endpointResults.checkId, at: endpointResults.at, ok: endpointResults.ok, ms: endpointResults.ms })
.from(endpointResults)
.where(and(inArray(endpointResults.checkId, ids), gte(endpointResults.at, new Date(now.getTime() - 7 * 86_400_000).toISOString())));
const by = new Map<number, { at: string; ok: boolean; ms: number | null }[]>();
for (const r of rows) by.set(r.checkId, [...(by.get(r.checkId) ?? []), r]);
for (const id of ids) out.set(id, summariseResults(by.get(id) ?? [], now));
return out;
}
// ─── What is wrong ─────────────────────────────────────────────────────────
export interface EndpointCondition {
key: string;
source: "endpoint";
message: string;
severity: "critical" | "warning";
/** "monitoring" for a check that fails, "expiry" for a certificate running out. */
category: "monitoring" | "expiry";
checkId: number;
serverId: number | null;
}
const daysLeft = (iso: string, now: Date) => Math.floor((Date.parse(iso) - now.getTime()) / 86_400_000);
/**
* The problems the checks stand for. A check must have failed several times in a row before it is one (so a blip isn't), and a certificate is
* one from its warning time on — critical in the last three days. The key carries the server a check is linked to, so maintenance on it silences them.
*/
export function endpointConditions(rows: CheckRow[], now: Date): EndpointCondition[] {
const out: EndpointCondition[] = [];
for (const c of rows) {
if (!c.enabled) continue;
const suffix = c.serverId !== null ? `:server:${c.serverId}` : "";
const left = c.certExpiresAt ? daysLeft(c.certExpiresAt, now) : null;
const certGone = left !== null && left < 0;
if (c.lastOk === false && c.consecutiveFailures >= c.failuresBeforeAlert && !(c.kind === "tls" && certGone)) {
out.push({ key: `endpoint:${c.id}${suffix}`, source: "endpoint", category: "monitoring", severity: "critical", checkId: c.id, serverId: c.serverId, message: `${c.name} isn't answering — ${c.lastMessage ?? "no answer"}` });
}
if (c.certExpiresAt && left !== null && left <= c.warnDays) {
out.push({
key: `endpoint-cert:${c.id}${suffix}`,
source: "endpoint",
category: "expiry",
severity: left <= 3 ? "critical" : "warning",
checkId: c.id,
serverId: c.serverId,
message: `The certificate for ${c.name} ${left < 0 ? `expired ${-left} day${-left === 1 ? "" : "s"} ago` : left === 0 ? "expires today" : `expires in ${left} day${left === 1 ? "" : "s"}`} (${c.certExpiresAt.slice(0, 10)})`,
});
}
}
return out;
}
+59
View File
@@ -0,0 +1,59 @@
/**
* The checks' own rhythm: every minute, ask the ones that are due, then see what has started or stopped being a problem and say so — a
* notification when a check has failed enough times in a row, another when it works again, the same for a certificate running out. It
* uses the same bookkeeping as the server and automation checks (each problem announced once, remembered across a restart), so
* a maintenance window on the server a check is linked to holds its problems back, and they alert after it if they are still there.
*/
import { db } from "../db/client.js";
import { endpointChecks } from "../db/schema.js";
import { diffConditions, type ActiveState, type HealthCondition } from "./healthMonitor.js";
import { activeSubjects } from "./maintenance.js";
import { endpointConditions, pruneResults, runDueChecks, type CheckRow, type ProbeResult } from "./endpointChecks.js";
import { notifyHealthIssues, notifyHealthRecovered } from "./notify.js";
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
const STATE_FLAG = "endpointActiveConditions";
async function loadState(): Promise<ActiveState> {
const raw = await getInternalFlag(STATE_FLAG);
if (!raw) return {};
try {
return JSON.parse(raw);
} catch {
return {};
}
}
/** One pass: ask what is due, then announce what changed. Returns what it did. */
export async function runEndpointPass(now: Date = new Date(), probeFn?: (c: CheckRow) => Promise<ProbeResult>): Promise<{ asked: number; announced: number; recovered: number }> {
const asked = await runDueChecks(now, probeFn);
const rows = await db.select().from(endpointChecks);
const current: HealthCondition[] = endpointConditions(rows, now).map((c) => ({ key: c.key, source: c.source, message: c.message, severity: c.severity }));
const { added, resolved, next } = diffConditions(await loadState(), current, new Set(), await activeSubjects(now));
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
if (added.length > 0) await notifyHealthIssues(added);
if (resolved.length > 0) await notifyHealthRecovered(resolved);
return { asked, announced: added.length, recovered: resolved.length };
}
let timer: ReturnType<typeof setInterval> | null = null;
/** Looks every minute (each check has its own interval, so most passes ask nobody). Once a day it clears out old results. */
export function initEndpointMonitor(): void {
if (timer) return;
let lastPrune = 0;
const tick = async () => {
try {
await runEndpointPass();
if (Date.now() - lastPrune > 24 * 3600_000) {
lastPrune = Date.now();
await pruneResults();
}
} catch (err) {
console.error("[endpoints] the pass failed:", err);
}
};
setTimeout(() => void tick(), 20_000).unref();
timer = setInterval(() => void tick(), 60_000);
timer.unref();
}
+2
View File
@@ -25,6 +25,8 @@ export async function listUpcomingWindows(now: Date = new Date()): Promise<Activ
export function subjectOfConditionKey(key: string): string | null { export function subjectOfConditionKey(key: string): string | null {
let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)(?::|$)/.exec(key); let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)(?::|$)/.exec(key);
if (m) return `server:${m[1]}`; if (m) return `server:${m[1]}`;
m = /^endpoint(?:-cert)?:\d+:server:(\d+)$/.exec(key);
if (m) return `server:${m[1]}`;
m = /^disk:proxmox:(\d+):/.exec(key); m = /^disk:proxmox:(\d+):/.exec(key);
if (m) return `integration:${m[1]}`; if (m) return `integration:${m[1]}`;
m = /^(?:synology-volume|synology-disk|disk:synology):(\d+):/.exec(key); m = /^(?:synology-volume|synology-disk|disk:synology):(\d+):/.exec(key);
+19 -1
View File
@@ -4,7 +4,8 @@
* integration on its own, so a scrape every fifteen seconds costs a few database reads (the alerts are the app's own minute-old result). * integration on its own, so a scrape every fifteen seconds costs a few database reads (the alerts are the app's own minute-old result).
*/ */
import { db } from "../db/client.js"; import { db } from "../db/client.js";
import { assets, integrations, maintenanceWindows, secrets, servers } from "../db/schema.js"; import { assets, endpointChecks, integrations, maintenanceWindows, secrets, servers } from "../db/schema.js";
import { latestTests } from "./restoreTests.js";
import { env } from "../env.js"; import { env } from "../env.js";
import { getAlerts } from "./alerts.js"; import { getAlerts } from "./alerts.js";
import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js"; import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js";
@@ -53,6 +54,22 @@ class Exposition {
} }
} }
/** Endpoint checks and restore tests, for /metrics. */
async function addEndpointMetrics(out: Exposition, now: Date) {
for (const c of await db.select().from(endpointChecks)) {
if (!c.enabled || c.lastOk === null) continue;
const labels = { name: c.name, kind: c.kind };
out.add("homelab_endpoint_up", "gauge", "1 if the endpoint check's last answer was fine, else 0.", c.lastOk ? 1 : 0, labels);
if (c.lastResponseMs !== null) out.add("homelab_endpoint_response_milliseconds", "gauge", "How long the endpoint took to answer the last time.", c.lastResponseMs, labels);
if (c.certExpiresAt) out.add("homelab_endpoint_certificate_days_left", "gauge", "Days until the certificate the endpoint shows expires (negative once it has).", Math.floor((Date.parse(c.certExpiresAt) - now.getTime()) / 86_400_000), labels);
}
for (const r of (await latestTests()).values()) {
if (r.result === "not_needed") continue;
out.add("homelab_restore_test_age_days", "gauge", "Days since a restore of this was last tested.", Math.max(0, Math.floor((now.getTime() - Date.parse(`${r.testedAt}T12:00:00Z`)) / 86_400_000)), { subject: r.label });
out.add("homelab_restore_test_passed", "gauge", "1 if the last restore test passed (fully or partly), 0 if it failed.", r.result === "failed" ? 0 : 1, { subject: r.label });
}
}
export async function buildMetrics(now: Date = new Date()): Promise<string> { export async function buildMetrics(now: Date = new Date()): Promise<string> {
const started = Date.now(); const started = Date.now();
const out = new Exposition(); const out = new Exposition();
@@ -177,5 +194,6 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
out.add("homelab_scrape_duration_seconds", "gauge", "How long building this answer took.", (Date.now() - started) / 1000); out.add("homelab_scrape_duration_seconds", "gauge", "How long building this answer took.", (Date.now() - started) / 1000);
out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec); out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec);
await addEndpointMetrics(out, now);
return out.toString(); return out.toString();
} }
+3 -1
View File
@@ -109,6 +109,8 @@ export function metaOfConditionKey(key: string, severity?: string): NotifyMeta {
else if (key.startsWith("failed-units:")) category = "services"; else if (key.startsWith("failed-units:")) category = "services";
else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk"; else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk";
else if (key.startsWith("automation:")) category = "automation"; else if (key.startsWith("automation:")) category = "automation";
const server = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)/.exec(key); else if (key.startsWith("endpoint-cert:")) category = "expiry";
else if (key.startsWith("endpoint:")) category = "monitoring";
const server = /^(?:(?:offline|disk|failed-units|smart|zfs):server:|endpoint(?:-cert)?:\d+:server:)(\d+)/.exec(key);
return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null }; return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null };
} }
+94
View File
@@ -0,0 +1,94 @@
/**
* Restore tests: a backup nobody has ever restored is a hope, not a backup. Somebody records when they tried to restore a guest (or anything else
* — a share, a database) and how it went, and the app keeps the history, shows each guest's newest test beside its backups, and speaks up
* when a test failed (always) or when none has been done in too long (if a time limit is set under Settings → Notifications).
*/
import { desc } from "drizzle-orm";
import { db } from "../db/client.js";
import { restoreTests } from "../db/schema.js";
import type { CoverageGuest, CoverageReport } from "./backupCoverage.js";
export const RESULTS = ["ok", "partial", "failed", "not_needed"] as const;
export type RestoreResult = (typeof RESULTS)[number];
export type TestRow = typeof restoreTests.$inferSelect;
/** What a subject's newest test amounts to: how it went, or that none was done (never) or that it is too old (stale). */
export type RestoreStatus = RestoreResult | "never" | "stale";
export interface RestoreInfo {
status: RestoreStatus;
/** YYYY-MM-DD of the newest test; null when there is none. */
testedAt: string | null;
ageDays: number | null;
note: string | null;
testedBy: string | null;
}
export const guestKey = (integrationId: number, vmid: number) => `guest:${integrationId}:${vmid}`;
export const KEY_PATTERN = /^(guest:\d+:\d+|custom:[a-z0-9][a-z0-9-]{0,60})$/;
/** "Photos share" → "custom:photos-share". */
export function customKey(label: string): string | null {
const slug = label
.toLowerCase()
.normalize("NFKD")
.replace(/[̀-ͯ]/g, "")
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 60);
return slug ? `custom:${slug}` : null;
}
const ageInDays = (testedAt: string, now: Date) => Math.floor((now.getTime() - Date.parse(`${testedAt}T12:00:00Z`)) / 86_400_000);
/** Pure: what the newest test (if any) says, against the time limit (0 = no limit). */
export function restoreInfo(latest: Pick<TestRow, "result" | "testedAt" | "note" | "testedBy"> | undefined, now: Date, limitDays: number): RestoreInfo {
if (!latest) return { status: "never", testedAt: null, ageDays: null, note: null, testedBy: null };
const ageDays = ageInDays(latest.testedAt, now);
let status: RestoreStatus = latest.result as RestoreResult;
if ((status === "ok" || status === "partial") && limitDays > 0 && ageDays > limitDays) status = "stale";
return { status, testedAt: latest.testedAt, ageDays, note: latest.note, testedBy: latest.testedBy };
}
/** The newest test of each subject (the one with the latest date, and the later one if two share a date). */
export async function latestTests(): Promise<Map<string, TestRow>> {
const rows = await db.select().from(restoreTests).orderBy(desc(restoreTests.testedAt), desc(restoreTests.id));
const out = new Map<string, TestRow>();
for (const r of rows) if (!out.has(r.subjectKey)) out.set(r.subjectKey, r);
return out;
}
export interface RestoreAlert {
key: string;
severity: "critical" | "warning";
message: string;
guest: CoverageGuest | null;
}
const daysWords = (d: number) => `${d} day${d === 1 ? "" : "s"}`;
/**
* What to speak up about. A test that failed is always a problem (the backups may not be usable) until a newer one passes. With a limit set,
* a guest that has backups but whose newest passing test is older than the limit — or that has never had one — is too, and so is anything
* else somebody chose to track. A guest marked "no need to test" never is.
*/
export function restoreAlerts(report: CoverageReport, latest: Map<string, TestRow>, limitDays: number, now: Date): RestoreAlert[] {
const out: RestoreAlert[] = [];
const seen = new Set<string>();
for (const g of report.guests) {
const key = guestKey(g.integrationId, g.vmid);
seen.add(key);
const info = restoreInfo(latest.get(key), now, limitDays);
const who = `${g.name} (#${g.vmid}) [${g.integrationName}]`;
if (info.status === "failed") out.push({ key: `restore-test:${key}`, severity: "critical", guest: g, message: `${who}: the last restore test (${info.testedAt}) failed${info.note ? ` — ${info.note}` : ""}` });
else if (limitDays > 0 && g.lastBackupAt !== null && info.status === "never") out.push({ key: `restore-test:${key}`, severity: "warning", guest: g, message: `${who}: a restore has never been tested` });
else if (limitDays > 0 && info.status === "stale") out.push({ key: `restore-test:${key}`, severity: "warning", guest: g, message: `${who}: its last restore test was ${daysWords(info.ageDays ?? 0)} ago (the limit is ${daysWords(limitDays)})` });
}
for (const [key, row] of latest) {
if (seen.has(key) || key.startsWith("guest:")) continue;
const info = restoreInfo(row, now, limitDays);
if (info.status === "failed") out.push({ key: `restore-test:${key}`, severity: "critical", guest: null, message: `${row.label}: the last restore test (${row.testedAt}) failed${row.note ? ` — ${row.note}` : ""}` });
else if (info.status === "stale") out.push({ key: `restore-test:${key}`, severity: "warning", guest: null, message: `${row.label}: its last restore test was ${daysWords(info.ageDays ?? 0)} ago (the limit is ${daysWords(limitDays)})` });
}
return out;
}
+3 -1
View File
@@ -111,6 +111,8 @@ export interface HealthCheckSettings {
diskForecastDays: number; diskForecastDays: number;
/** A Proxmox guest whose newest backup is older than this many hours is raised as stale. 0 turns it off. */ /** A Proxmox guest whose newest backup is older than this many hours is raised as stale. 0 turns it off. */
backupStaleHours: number; backupStaleHours: number;
/** A restore test older than this many days (or none at all) is a problem for a guest that has backups; 0 = never (a failed test always is). */
restoreTestDays: number;
} }
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */ /** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */
@@ -257,7 +259,7 @@ const DEFAULTS: AppSettings = {
agents: { autoUpdate: false }, agents: { autoUpdate: false },
selfBackup: { enabled: true, time: "03:30", keep: 14 }, selfBackup: { enabled: true, time: "03:30", keep: 14 },
signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" }, signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" },
healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48 }, healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0 },
// Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every // Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every
// Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN. // Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN.
consistency: { excludedRanges: ["172.16.0.0/12"] }, consistency: { excludedRanges: ["172.16.0.0/12"] },
+162
View File
@@ -0,0 +1,162 @@
/**
* The change log: the sentences, what is left out, the filters (kind, server, time, a word), paging, and who may read it.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-changes-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, auditLog, alertIncidents } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { describeChange, getChangeLog } = await import("../src/services/changeLog.js");
const { changesRouter } = await import("../src/routes/changes.js");
await runMigrations();
describe("the sentences", () => {
test("maintenance, dependencies, hardware, restore tests and settings read as sentences", () => {
assert.deepEqual(describeChange("maintenance", "start", { name: "nas", minutes: 60, reason: "reboot" }), { title: "Maintenance started on nas", tone: "info", detail: "for 60 min · reboot" });
assert.equal(describeChange("maintenance", "extend", { name: "nas", minutes: 240 }).detail, "for 4 h from now");
assert.equal(describeChange("dependency", "create", { from: "Gitea", to: "nas" }).title, "Gitea now needs nas");
assert.equal(describeChange("dependency", "delete", { from: "Gitea", to: "nas" }).title, "Gitea no longer needs nas");
assert.equal(describeChange("inventory", "create", { name: "NAS", kind: "storage" }).title, "Hardware added: NAS");
assert.deepEqual(describeChange("restore_test", "record", { name: "elsa", result: "failed", testedAt: "2026-10-01", note: "no boot" }), { title: "Restore test of elsa failed", tone: "bad", detail: "2026-10-01 · no boot" });
assert.equal(describeChange("restore_test", "record", { name: "elsa", result: "ok" }).tone, "good");
const s = describeChange("settings", "update", { sections: ["notifications", "webhook"], changes: { notifications: { healthAlerts: { from: true, to: false } }, webhook: { url: "(changed)" } } });
assert.equal(s.title, "Settings changed: notifications, webhook");
assert.equal(s.detail, 'notifications.healthAlerts: true → false · webhook.url changed', "a secret's value is never in it");
});
test("a server's own entries use the wording of its Activity card; anything else is made readable", () => {
assert.equal(describeChange("server", "agent_updated", { from: 4, to: 6 }).title, "Agent updated to version 6");
assert.equal(describeChange("alert", "ack", { assignee: "Olle", hours: 4 }).title, "Acknowledged, handed to Olle");
assert.deepEqual(describeChange("dns", "add_record", { name: "a.example.com", type: "A" }), { title: "Dns: add record — a.example.com", tone: "neutral", detail: "type: A" });
});
});
describe("through the routes", () => {
let app: Server;
let base = "";
let who = "operator";
let nas = 0;
let elsa = 0;
const audit = (category: string, action: string, extra: { targetType?: string; targetId?: string; detail?: unknown; actor?: string; at: string }) =>
db.insert(auditLog).values({ category, action, targetType: extra.targetType ?? null, targetId: extra.targetId ?? null, detail: extra.detail === undefined ? null : JSON.stringify(extra.detail), actorLabel: extra.actor ?? "Ada", createdAt: extra.at });
const ago = (min: number) => new Date(Date.now() - min * 60_000).toISOString().slice(0, 19).replace("T", " ");
before(async () => {
await db.insert(users).values([
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
const made = await db.insert(servers).values([{ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n" }, { name: "elsa", apiTokenHash: "e", apiTokenPrefix: "e" }]).returning({ id: servers.id });
nas = made[0].id;
elsa = made[1].id;
await audit("maintenance", "start", { targetType: "server", targetId: String(nas), detail: { name: "nas", minutes: 60, reason: "reboot" }, at: ago(10) });
await audit("server", "agent_updated", { targetType: "server", targetId: String(elsa), detail: { from: 4, to: 6 }, actor: "system", at: ago(20) });
await audit("dependency", "create", { targetType: "dependency", targetId: "1", detail: { from: "Gitea", to: "nas", fromKey: "integration:1", toKey: `server:${nas}` }, at: ago(30) });
await audit("settings", "update", { targetType: "settings", detail: { sections: ["display"], changes: { display: { pageSize: { from: 20, to: 50 } } } }, at: ago(40) });
await audit("session", "login", { at: ago(5) });
await audit("privacy", "export_own_data", { at: ago(6) });
await audit("inventory", "create", { targetType: "asset", targetId: "7", detail: { name: "Core switch", kind: "network" }, at: ago(60 * 24 * 3) });
await audit("server", "update", { targetType: "server", targetId: String(nas), detail: { name: "nas" }, at: ago(60 * 24 * 20) });
await db.insert(alertIncidents).values({ alertKey: "offline:offline:server:1", severity: "critical", peakSeverity: "critical", category: "offline", source: "Server", message: "nas hasn't reported for 3 h", targetType: "server", targetId: nas, targetName: "nas", openedAt: new Date(Date.now() - 50 * 60_000).toISOString(), resolvedAt: new Date(Date.now() - 15 * 60_000).toISOString(), lastSeenAt: new Date().toISOString() });
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/changes", changesRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/changes`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const get = async (query = "") => (await fetch(`${base}${query}`)).json();
const titles = (r: { entries: { title: string }[] }) => r.entries.map((e) => e.title);
test("everything of the last week, newest first, with the alerts raised and cleared in between; sign-ins and data exports are not changes", async () => {
const r = await get();
assert.deepEqual(titles(r), [
"Maintenance started on nas",
"Alert cleared: nas hasn't reported for 3 h",
"Agent updated to version 6",
"Gitea now needs nas",
"Settings changed: display",
"Alert raised: nas hasn't reported for 3 h",
"Hardware added: Core switch",
]);
assert.ok(!r.entries.some((e: { title: string }) => /login|export/i.test(e.title)));
const maint = r.entries[0];
assert.deepEqual([maint.kind, maint.actor, maint.link, maint.serverId], ["maintenance", "Ada", `/servers/${nas}`, nas]);
assert.equal(r.entries.find((e: { title: string }) => e.title.startsWith("Alert raised")).link.startsWith("/incidents/"), true);
assert.equal(r.kinds.length, 10);
});
test("by kind, by how far back, and by a word", async () => {
assert.deepEqual(titles(await get("?kind=maintenance")), ["Maintenance started on nas"]);
assert.deepEqual(titles(await get("?kind=alerts")).length, 2);
assert.equal((await get("?days=1")).entries.length, 6, "the hardware added three days ago is out");
assert.equal((await get("?days=30")).entries.length, 8, "and a month covers the old edit too");
assert.deepEqual(titles(await get("?q=gitea")), ["Gitea now needs nas"]);
assert.deepEqual(titles(await get("?q=version%20elsa")), ["Agent updated to version 6"], "every word must appear somewhere, the server's name included");
assert.deepEqual((await get("?kind=nonsense")).entries.length, 7, "an unknown kind is no filter");
});
test("by server: its own entries, its alerts, and what mentions it by key — not the other server's", async () => {
assert.deepEqual(titles(await get(`?serverId=${nas}`)), ["Maintenance started on nas", "Alert cleared: nas hasn't reported for 3 h", "Gitea now needs nas", "Alert raised: nas hasn't reported for 3 h"]);
assert.deepEqual(titles(await get(`?serverId=${elsa}`)), ["Agent updated to version 6"]);
assert.deepEqual((await get("?serverId=99999")).entries, []);
});
test("paged: a limit says there is more, and 'before' goes on from where it stopped", async () => {
const first = await get("?limit=3");
assert.equal(first.entries.length, 3);
assert.equal(first.more, true);
const next = await get(`?limit=3&before=${encodeURIComponent(first.entries[2].at)}`);
assert.deepEqual(titles(next), ["Gitea now needs nas", "Settings changed: display", "Alert raised: nas hasn't reported for 3 h"]);
assert.ok(next.entries.every((e: { at: string }) => e.at < first.entries[2].at));
assert.equal(next.more, true, "the hardware added three days ago is still to come");
const last = await get(`?limit=3&before=${encodeURIComponent(next.entries[2].at)}`);
assert.deepEqual(titles(last), ["Hardware added: Core switch"]);
assert.equal(last.more, false);
});
test("operators and administrators can read it; a viewer can't", async () => {
who = "viewer";
assert.equal((await fetch(base)).status, 403);
who = "operator";
assert.equal((await fetch(base)).status, 200);
});
test("the service on its own gives the same answer for any moment it is asked about", async () => {
const future = await getChangeLog({ now: new Date(Date.now() + 40 * 86_400_000), days: 1 });
assert.deepEqual(future.entries, [], "nothing in the day before a month from now");
});
});
+407
View File
@@ -0,0 +1,407 @@
/**
* Endpoint checks: what counts as an answer, the probes themselves (against servers this machine runs), what a run records, the problems they
* stand for, and what the routes, the alerts and the notifications do with them.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { createServer as createHttpServer, type Server } from "node:http";
import { createServer as createHttpsServer } from "node:https";
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { networkInterfaces, tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-endpoints-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, endpointChecks, endpointResults } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { statusAccepter, isStatusSpec, summariseResults, endpointConditions, probe, runCheck, runDueChecks } = await import("../src/services/endpointChecks.js");
const { runEndpointPass } = await import("../src/services/endpointMonitor.js");
const { endpointsRouter } = await import("../src/routes/endpoints.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { startOrExtendWindow } = await import("../src/services/maintenance.js");
const { updateSettings } = await import("../src/services/settingsStore.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
type Row = typeof endpointChecks.$inferSelect;
const NOW = new Date("2026-10-18T12:00:00Z");
const row = (extra: Partial<Row> = {}): Row => ({
id: 1, name: "Gitea", kind: "http", target: "https://git.lab/", port: null, expectStatus: null, keyword: null, insecure: false, intervalMinutes: 5, timeoutSeconds: 10, failuresBeforeAlert: 2, warnDays: 14,
serverId: null, enabled: true, lastCheckedAt: null, lastOk: null, lastMessage: null, lastResponseMs: null, certExpiresAt: null, consecutiveFailures: 0, lastChangeAt: null, createdAt: "2026-10-01 00:00:00", ...extra,
});
describe("what counts as an answer", () => {
test("status codes, ranges and the default of 200 to 399", () => {
const usual = statusAccepter(null);
assert.deepEqual([199, 200, 301, 399, 400, 500].map(usual), [false, true, true, true, false, false]);
const custom = statusAccepter("200-204, 401");
assert.deepEqual([200, 204, 205, 401, 403].map(custom), [true, true, false, true, false]);
assert.equal(statusAccepter("rubbish")(200), true, "unreadable falls back to the usual");
assert.equal(isStatusSpec("200, 301-302"), true);
assert.equal(isStatusSpec("2xx"), false);
});
test("uptime, the average answer time and the recent run come from the stored results", () => {
const at = (min: number) => new Date(NOW.getTime() - min * 60_000).toISOString();
const stats = summariseResults(
[
{ at: at(10), ok: true, ms: 100 },
{ at: at(20), ok: true, ms: 300 },
{ at: at(30), ok: false, ms: null },
{ at: at(60 * 30), ok: false, ms: null }, // yesterday and a bit
{ at: at(60 * 24 * 6), ok: true, ms: 50 },
],
NOW,
);
assert.equal(stats.uptime24h, 66.7, "2 of 3 in the last day");
assert.equal(stats.uptime7d, 60, "3 of 5 in the last week");
assert.equal(stats.avgMs24h, 200, "only answers that came count");
assert.equal(stats.recent[0].at, at(60 * 24 * 6), "oldest first");
assert.deepEqual(summariseResults([], NOW), { uptime24h: null, uptime7d: null, avgMs24h: null, recent: [] });
});
});
describe("what is a problem", () => {
test("a check has to fail several times in a row; a blip isn't", () => {
assert.deepEqual(endpointConditions([row({ lastOk: false, consecutiveFailures: 1, lastMessage: "answered 500" })], NOW), []);
const [c] = endpointConditions([row({ lastOk: false, consecutiveFailures: 2, lastMessage: "answered 500" })], NOW);
assert.equal(c.key, "endpoint:1");
assert.equal(c.category, "monitoring");
assert.equal(c.severity, "critical");
assert.match(c.message, /Gitea isn't answering — answered 500/);
assert.deepEqual(endpointConditions([row({ lastOk: false, consecutiveFailures: 9, enabled: false })], NOW), [], "a check that is switched off says nothing");
assert.deepEqual(endpointConditions([row({ lastOk: true, consecutiveFailures: 0 })], NOW), []);
});
test("a certificate is a problem from its warning time on, critical in the last three days, and the key carries the server it is linked to", () => {
const days = (n: number) => new Date(NOW.getTime() + n * 86_400_000).toISOString();
assert.deepEqual(endpointConditions([row({ certExpiresAt: days(20) })], NOW), []);
const [warn] = endpointConditions([row({ certExpiresAt: days(10), serverId: 4 })], NOW);
assert.equal(warn.key, "endpoint-cert:1:server:4");
assert.deepEqual([warn.category, warn.severity], ["expiry", "warning"]);
assert.match(warn.message, /expires in 10 days/);
assert.equal(endpointConditions([row({ certExpiresAt: days(2) })], NOW)[0].severity, "critical");
assert.match(endpointConditions([row({ certExpiresAt: days(-3) })], NOW)[0].message, /expired 3 days ago/);
});
test("a certificate check whose certificate has expired says so once, not also that it isn't answering", () => {
const expired = new Date(NOW.getTime() - 86_400_000).toISOString();
const list = endpointConditions([row({ kind: "tls", target: "git.lab", port: 443, lastOk: false, consecutiveFailures: 5, lastMessage: "The certificate expired", certExpiresAt: expired })], NOW);
assert.deepEqual(list.map((c) => c.category), ["expiry"]);
});
});
/** An address of this machine that isn't loopback (the checks refuse that), if it has one. */
function lanAddress(): string | null {
for (const list of Object.values(networkInterfaces())) {
for (const i of list ?? []) if (i.family === "IPv4" && !i.internal && !i.address.startsWith("169.254.")) return i.address;
}
return null;
}
const LAN = lanAddress();
describe("asking", { skip: LAN === null && "this machine has no address other than loopback" }, () => {
let web: Server;
let port = 0;
let sleeping: Server;
let sleepingPort = 0;
const hits: string[] = [];
before(async () => {
web = createHttpServer((req, res) => {
hits.push(req.url ?? "");
if (req.url === "/ok") return void res.end("<h1>Welcome to the lab</h1>");
if (req.url === "/broken") {
res.statusCode = 500;
return void res.end("oops");
}
if (req.url === "/moved") {
res.statusCode = 301;
res.setHeader("location", "/ok");
return void res.end();
}
res.statusCode = 404;
res.end("no");
});
await new Promise<void>((r) => web.listen(0, "0.0.0.0", r));
port = (web.address() as AddressInfo).port;
// Accepts the connection and never answers.
sleeping = createHttpServer(() => {});
await new Promise<void>((r) => sleeping.listen(0, "0.0.0.0", r));
sleepingPort = (sleeping.address() as AddressInfo).port;
});
after(() => {
web.close();
sleeping.close();
});
const http = (path: string, extra: Partial<Row> = {}) => row({ kind: "http", target: `http://${LAN}:${port}${path}`, timeoutSeconds: 3, ...extra });
test("a web address that answers is fine, with how long it took", async () => {
const r = await probe(http("/ok"));
assert.equal(r.ok, true);
assert.match(r.message, /answered 200 in \d+ ms/);
assert.ok(r.ms !== null && r.ms >= 0);
});
test("a wrong status fails, unless that is what was expected; a redirect is an answer", async () => {
assert.match((await probe(http("/broken"))).message, /answered 500/);
assert.equal((await probe(http("/broken", { expectStatus: "500" }))).ok, true);
assert.equal((await probe(http("/missing"))).ok, false);
assert.equal((await probe(http("/moved"))).ok, true, "301 is within 200 to 399");
});
test("a word the page must contain: found after following a redirect, or missing", async () => {
assert.equal((await probe(http("/ok", { keyword: "Welcome" }))).ok, true);
const missing = await probe(http("/ok", { keyword: "Goodbye" }));
assert.equal(missing.ok, false);
assert.match(missing.message, /doesn't contain “Goodbye”/);
assert.equal((await probe(http("/moved", { keyword: "Welcome" }))).ok, true, "with a word to find, the redirect is followed to the page");
});
test("no answer in time, and a refused connection, say so", async () => {
const slow = await probe(row({ kind: "http", target: `http://${LAN}:${sleepingPort}/`, timeoutSeconds: 1 }));
assert.equal(slow.ok, false);
assert.match(slow.message, /no answer within 1 s/);
const refused = await probe(row({ kind: "tcp", target: LAN!, port: 1, timeoutSeconds: 2 }));
assert.equal(refused.ok, false);
const open = await probe(row({ kind: "tcp", target: LAN!, port, timeoutSeconds: 2 }));
assert.equal(open.ok, true);
assert.match(open.message, new RegExp(`port ${port} answered in \\d+ ms`));
});
test("it will not be pointed at this machine itself or at the cloud's metadata address", async () => {
for (const target of ["http://127.0.0.1:1/", "http://169.254.169.254/latest/meta-data/"]) {
const r = await probe(row({ kind: "http", target }));
assert.equal(r.ok, false);
assert.match(r.message, /can't be used|isn't allowed/);
}
assert.equal((await probe(row({ kind: "tcp", target: "127.0.0.1", port: 22 }))).ok, false);
assert.equal((await probe(row({ kind: "http", target: "ftp://example.com/" }))).ok, false);
});
test("a certificate that can't be trusted fails unless it is accepted, and either way its expiry is read", async () => {
const work = mkdtempSync(join(tmpdir(), "hm-cert-"));
const made = spawnSync("openssl", ["req", "-x509", "-newkey", "rsa:2048", "-nodes", "-days", "30", "-subj", "/CN=lab.example", "-keyout", join(work, "k.pem"), "-out", join(work, "c.pem")], { encoding: "utf8" });
if (made.status !== 0) {
rmSync(work, { recursive: true, force: true });
return; // no openssl here: the rest is covered by what the pure parts say
}
const secure = createHttpsServer({ key: readFileSync(join(work, "k.pem")), cert: readFileSync(join(work, "c.pem")) }, (_req, res) => res.end("secure"));
await new Promise<void>((r) => secure.listen(0, "0.0.0.0", r));
const sp = (secure.address() as AddressInfo).port;
try {
const strict = await probe(row({ kind: "http", target: `https://${LAN}:${sp}/`, timeoutSeconds: 5 }));
assert.equal(strict.ok, false);
assert.match(strict.message, /certificate can't be trusted: it is self-signed/);
const lenient = await probe(row({ kind: "http", target: `https://${LAN}:${sp}/`, insecure: true, timeoutSeconds: 5 }));
assert.equal(lenient.ok, true);
assert.ok(lenient.certExpiresAt && Date.parse(lenient.certExpiresAt) > Date.now() + 25 * 86_400_000, "about 30 days left");
const tlsStrict = await probe(row({ kind: "tls", target: LAN!, port: sp, timeoutSeconds: 5 }));
assert.equal(tlsStrict.ok, false);
assert.match(tlsStrict.message, /can't be trusted/);
const tlsLenient = await probe(row({ kind: "tls", target: LAN!, port: sp, insecure: true, timeoutSeconds: 5 }));
assert.equal(tlsLenient.ok, true);
assert.match(tlsLenient.message, /good until \d{4}-\d{2}-\d{2}/);
const plain = await probe(row({ kind: "tls", target: LAN!, port, timeoutSeconds: 5 }));
assert.equal(plain.ok, false);
assert.match(plain.message, /isn't speaking TLS/);
} finally {
secure.close();
rmSync(work, { recursive: true, force: true });
}
});
});
describe("a run, a pass, the routes, the alerts and the notifications", () => {
let app: Server;
let base = "";
let who = "admin";
let serverId = 0;
const messages: string[] = [];
let hook: Server;
let answer: { ok: boolean; ms: number | null; message: string; certExpiresAt?: string | null } = { ok: true, ms: 12, message: "answered 200 in 12 ms" };
before(async () => {
await db.insert(users).values([
{ oidcSub: "admin", name: "Ada", role: "admin" },
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
serverId = (await db.insert(servers).values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", lastSeenAt: new Date().toISOString() }).returning({ id: servers.id }))[0].id;
hook = createHttpServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", (c) => chunks.push(c));
req.on("end", () => {
messages.push(String(JSON.parse(Buffer.concat(chunks).toString()).content));
res.end("{}");
});
});
await new Promise<void>((r) => hook.listen(0, "127.0.0.1", r));
await updateSettings({ webhook: { enabled: true, url: `http://127.0.0.1:${(hook.address() as AddressInfo).port}/` } });
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/endpoints", endpointsRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/endpoints`;
});
after(() => {
app.close();
hook.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const send = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
const fake = async () => answer;
let id = 0;
test("an administrator adds a check; a wrong one is refused with a reason; nobody else can", async () => {
const res = await send("POST", "/", { name: "Gitea", kind: "http", target: "https://git.lab/", serverId });
assert.equal(res.status, 201);
id = (await res.json()).check.id;
for (const bad of [
{ name: "x", kind: "http", target: "git.lab" },
{ name: "x", kind: "http", target: "https://a/", expectStatus: "2xx" },
{ name: "x", kind: "tcp", target: "host" },
{ name: "x", kind: "tcp", target: "http://host", port: 80 },
{ name: "x", kind: "tls", target: "host", port: 70000 },
{ name: "x", kind: "ping", target: "host" },
{ name: "x", kind: "http", target: "https://a/", serverId: 99999 },
]) assert.equal((await send("POST", "/", bad)).status, 400, JSON.stringify(bad));
who = "operator";
assert.equal((await send("POST", "/", { name: "y", kind: "tcp", target: "h", port: 22 })).status, 403);
who = "viewer";
assert.equal((await send("GET", "/")).status, 200, "everyone can look");
who = "admin";
});
test("a run records the answer on the check and as a point in its history", async () => {
const [c] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id));
answer = { ok: true, ms: 40, message: "answered 200 in 40 ms", certExpiresAt: new Date(NOW.getTime() + 3 * 86_400_000).toISOString() };
await runCheck(c, new Date("2026-10-18T12:00:00Z"), fake);
const [after1] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id));
assert.deepEqual([after1.lastOk, after1.lastResponseMs, after1.consecutiveFailures, after1.lastChangeAt], [true, 40, 0, "2026-10-18T12:00:00.000Z"]);
assert.ok(after1.certExpiresAt);
assert.equal((await db.select().from(endpointResults).where(eq(endpointResults.checkId, id))).length, 1);
answer = { ok: false, ms: null, message: "answered 500" };
await runCheck(after1, new Date("2026-10-18T12:05:00Z"), fake);
const [failed] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id));
assert.deepEqual([failed.lastOk, failed.consecutiveFailures, failed.lastChangeAt, failed.lastMessage], [false, 1, "2026-10-18T12:05:00.000Z", "answered 500"]);
await runCheck(failed, new Date("2026-10-18T12:10:00Z"), fake);
const [again] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id));
assert.deepEqual([again.consecutiveFailures, again.lastChangeAt], [2, "2026-10-18T12:05:00.000Z"], "the time of the change is when it began failing, not each failure");
});
test("only the checks that are due are asked", async () => {
await db.update(endpointChecks).set({ lastCheckedAt: "2026-10-18T12:10:00.000Z" }).where(eq(endpointChecks.id, id));
let asked = 0;
const counting = async () => {
asked++;
return answer;
};
assert.equal(await runDueChecks(new Date("2026-10-18T12:12:00Z"), counting), 0, "two minutes after a five-minute check");
assert.equal(await runDueChecks(new Date("2026-10-18T12:15:30Z"), counting), 1);
assert.equal(asked, 1);
await db.update(endpointChecks).set({ enabled: false }).where(eq(endpointChecks.id, id));
assert.equal(await runDueChecks(new Date("2026-10-18T13:00:00Z"), counting), 0, "a check that is switched off is never asked");
await db.update(endpointChecks).set({ enabled: true }).where(eq(endpointChecks.id, id));
});
test("a check that keeps failing is an alert, silenced by maintenance on its server, and a certificate running out is another", async () => {
await db.update(endpointChecks).set({ lastOk: false, consecutiveFailures: 3, lastMessage: "answered 500", certExpiresAt: new Date(Date.now() + 2 * 86_400_000).toISOString() }).where(eq(endpointChecks.id, id));
invalidateAlertsCache();
let alerts = (await getAlerts(true)).alerts.filter((a) => a.source === "Endpoint");
const down = alerts.find((a) => a.category === "monitoring")!;
const cert = alerts.find((a) => a.category === "expiry")!;
assert.match(down.message, /Gitea isn't answering — answered 500/);
assert.equal(down.severity, "critical");
assert.equal(down.link, "/endpoints");
assert.deepEqual(down.target && [down.target.type, down.target.id], ["server", serverId]);
assert.match(cert.message, /expires in/);
assert.equal(cert.severity, "critical", "two days left");
assert.equal(down.silenced, false);
await startOrExtendWindow("server", serverId, 60, "work", "Ada");
invalidateAlertsCache();
alerts = (await getAlerts(true)).alerts.filter((a) => a.source === "Endpoint");
assert.ok(alerts.length === 2 && alerts.every((a) => a.silenced), "both are under the server's maintenance");
});
test("the pass tells you once when a check starts failing, and once when it is fine again", async () => {
const { listActiveWindows } = await import("../src/services/maintenance.js");
const { maintenanceWindows } = await import("../src/db/schema.js");
for (const w of await listActiveWindows()) await db.delete(maintenanceWindows).where(eq(maintenanceWindows.id, w.id));
await db.update(endpointChecks).set({ lastOk: true, consecutiveFailures: 0, lastCheckedAt: null, certExpiresAt: null }).where(eq(endpointChecks.id, id));
messages.length = 0;
answer = { ok: false, ms: null, message: "answered 503" };
const t = (min: number) => new Date(Date.now() + min * 60_000);
let r = await runEndpointPass(t(0), fake);
assert.equal(r.announced, 0, "the first failure is a blip");
r = await runEndpointPass(t(6), fake);
assert.equal(r.announced, 1, "the second is a problem");
assert.match(messages.join("\n"), /Gitea isn't answering — answered 503/);
const sent = messages.length;
await runEndpointPass(t(12), fake);
assert.equal(messages.length, sent, "and it isn't said again");
answer = { ok: true, ms: 9, message: "answered 200 in 9 ms" };
r = await runEndpointPass(t(18), fake);
assert.equal(r.recovered, 1);
assert.match(messages[messages.length - 1], /Gitea isn't answering/, "the recovery names what cleared");
});
test("check now is for operators and answers at once; the list carries the figures; metrics have them", async () => {
who = "viewer";
assert.equal((await send("POST", `/${id}/check`)).status, 403);
who = "operator";
// the real probe, aimed at a name nothing answers to (and refused if it were loopback): it says what happened
await db.update(endpointChecks).set({ target: "http://127.0.0.1:1/" }).where(eq(endpointChecks.id, id));
const res = await send("POST", `/${id}/check`);
assert.equal(res.status, 200);
assert.equal((await res.json()).result.ok, false);
who = "viewer";
const list = await (await send("GET", "/")).json();
const c = list.checks.find((x: { id: number }) => x.id === id);
assert.equal(c.server.name, "nas");
assert.ok(c.stats.recent.length >= 1);
assert.equal(typeof c.stats.uptime24h, "number");
assert.match(await buildMetrics(), /homelab_endpoint_up\{name="Gitea",kind="http"\} 0/);
});
test("changing what is asked forgets what was found; removing it takes its history", async () => {
who = "admin";
const res = await send("PUT", `/${id}`, { name: "Gitea", kind: "http", target: "https://git2.lab/", serverId });
assert.equal(res.status, 200);
const [c] = await db.select().from(endpointChecks).where(eq(endpointChecks.id, id));
assert.deepEqual([c.lastOk, c.consecutiveFailures, c.lastMessage], [null, 0, null]);
assert.equal((await send("DELETE", `/${id}`)).status, 204);
assert.equal((await db.select().from(endpointResults).where(eq(endpointResults.checkId, id))).length, 0);
assert.equal((await send("DELETE", `/${id}`)).status, 404);
});
});
+210
View File
@@ -0,0 +1,210 @@
/**
* Restore tests: what the newest test of something amounts to, when to speak up, and the routes that record them.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-restore-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, restoreTests, auditLog } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { restoreInfo, restoreAlerts, customKey, guestKey } = await import("../src/services/restoreTests.js");
const { setCoverageLoaderForTests, buildCoverage } = await import("../src/services/backupCoverage.js");
const { backupsRouter } = await import("../src/routes/backups.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { updateSettings } = await import("../src/services/settingsStore.js");
const { buildMetrics } = await import("../src/services/metrics.js");
await runMigrations();
const NOW = new Date("2026-10-18T12:00:00Z");
const daysAgo = (n: number) => new Date(NOW.getTime() - n * 86_400_000).toISOString().slice(0, 10);
const test1 = (result: string, ago: number, extra: Record<string, unknown> = {}) => ({ result, testedAt: daysAgo(ago), note: null as string | null, testedBy: "Ada", ...extra });
describe("what the newest test amounts to", () => {
test("how it went; none at all is never; an old pass is stale once there is a limit; a failure or 'not needed' doesn't age", () => {
assert.equal(restoreInfo(undefined, NOW, 90).status, "never");
assert.deepEqual(restoreInfo(test1("ok", 10), NOW, 90), { status: "ok", testedAt: daysAgo(10), ageDays: 10, note: null, testedBy: "Ada" });
assert.equal(restoreInfo(test1("ok", 100), NOW, 90).status, "stale");
assert.equal(restoreInfo(test1("ok", 100), NOW, 0).status, "ok", "no limit, no staleness");
assert.equal(restoreInfo(test1("partial", 100), NOW, 90).status, "stale");
assert.equal(restoreInfo(test1("failed", 400), NOW, 90).status, "failed");
assert.equal(restoreInfo(test1("not_needed", 400), NOW, 90).status, "not_needed");
});
test("something that isn't a guest is told apart by its name", () => {
assert.equal(customKey("Photos share"), "custom:photos-share");
assert.equal(customKey(" Gitea — database! "), "custom:gitea-database");
assert.equal(customKey("Åkerblom's NAS"), "custom:akerblom-s-nas");
assert.equal(customKey("!!!"), null);
assert.equal(guestKey(2, 105), "guest:2:105");
});
});
const guest = (vmid: number, name: string) => ({ vmid, name, node: "pve1", type: "qemu" as const, status: "running", cpu: null, maxmem: null, mem: null, uptime: null });
const fixture = () => {
const loaded = {
proxmox: [{ integrationId: 1, integrationName: "Felicia", guests: [guest(100, "elsa"), guest(101, "anna"), guest(102, "zoe")], jobs: [{ id: "j", enabled: true, schedule: "02:00", storage: "pbs", node: null, allGuests: true, vmids: null, exclude: null }], tasks: [] }],
pbs: [{ integrationName: "Backup server", groups: [{ backupType: "vm", backupId: "100", backupTime: Math.floor(Date.now() / 1000) - 3600, verification: "ok" as const }, { backupType: "vm", backupId: "101", backupTime: Math.floor(Date.now() / 1000) - 7200, verification: "ok" as const }] }],
sources: [{ kind: "proxmox" as const, name: "Felicia", ok: true }],
};
return loaded;
};
describe("when to speak up", () => {
const report = () => ({ guests: buildCoverage({ ...fixture(), servers: [], now: NOW }), counts: {} as never, sources: [], staleHours: 48, generatedAt: NOW.toISOString() });
const row = (key: string, label: string, result: string, ago: number) => [key, { id: 1, subjectKey: key, label, testedAt: daysAgo(ago), result, note: null, testedBy: null, createdAt: "" }] as const;
test("with no limit only a failure speaks; with one, a guest that has backups and no recent test does too", () => {
const g = report();
assert.deepEqual(restoreAlerts(g, new Map(), 0, NOW), [], "no limit, nothing tested: quiet");
const limited = restoreAlerts(g, new Map(), 90, NOW);
assert.deepEqual(limited.map((a) => [a.guest?.name, a.severity]), [["elsa", "warning"], ["anna", "warning"]], "zoe has no backups to restore from, so nothing to say");
assert.match(limited[0].message, /elsa \(#100\) \[Felicia\]: a restore has never been tested/);
const failed = restoreAlerts(g, new Map([row("guest:1:100", "elsa", "failed", 3)]), 0, NOW);
assert.equal(failed.length, 1);
assert.equal(failed[0].severity, "critical");
assert.match(failed[0].message, /the last restore test \(.*\) failed/);
});
test("a recent pass, 'not needed' and an old pass under the limit are all fine; an old one isn't", () => {
const g = report();
const tests = new Map([row("guest:1:100", "elsa", "ok", 10), row("guest:1:101", "anna", "not_needed", 500)]);
assert.deepEqual(restoreAlerts(g, tests, 90, NOW), []);
const old = restoreAlerts(g, new Map([row("guest:1:100", "elsa", "ok", 120), row("guest:1:101", "anna", "ok", 1)]), 90, NOW);
assert.equal(old.length, 1);
assert.match(old[0].message, /last restore test was 120 days ago \(the limit is 90 days\)/);
});
test("something that isn't a guest is told of when it failed, or when it is stale", () => {
const g = report();
const tests = new Map([row("custom:photos", "Photos share", "ok", 200), row("custom:db", "Gitea database", "failed", 5), row("custom:fine", "Fine", "ok", 3), row("guest:9:9", "gone guest", "ok", 999)]);
const out = restoreAlerts(g, tests, 90, NOW);
assert.deepEqual(out.map((a) => a.message.split(":")[0]).sort(), ["Gitea database", "Photos share", "anna (#101) [Felicia]", "elsa (#100) [Felicia]"].sort());
assert.equal(restoreAlerts(g, tests, 0, NOW).filter((a) => !a.guest).length, 1, "with no limit only the failure");
});
});
describe("through the routes, the page, the alerts and the figures", () => {
let app: Server;
let base = "";
let who = "operator";
before(async () => {
await db.insert(users).values([
{ oidcSub: "admin", name: "Ada", role: "admin" },
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
setCoverageLoaderForTests(async () => fixture());
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/backups", backupsRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/backups`;
});
after(() => {
setCoverageLoaderForTests(null);
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const send = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
const today = new Date().toISOString().slice(0, 10);
test("an operator records a test; a viewer can't; a wrong one is refused", async () => {
assert.equal((await send("POST", "/restore-tests", { key: "guest:1:100", label: "elsa", result: "ok", note: "booted and the data was there" })).status, 201);
who = "viewer";
assert.equal((await send("POST", "/restore-tests", { key: "guest:1:101", label: "anna", result: "ok" })).status, 403);
who = "operator";
for (const bad of [
{ key: "guest:1:101", label: "anna", result: "excellent" },
{ key: "guest:one:two", label: "anna", result: "ok" },
{ key: "../etc", label: "anna", result: "ok" },
{ key: "guest:1:101", label: "anna", result: "ok", testedAt: "2999-01-01" },
{ key: "guest:1:101", label: "anna", result: "ok", testedAt: "2026-02-30" },
{ label: "!!!", result: "ok" },
{ key: "guest:1:101", label: "", result: "ok" },
]) assert.equal((await send("POST", "/restore-tests", bad)).status, 400, JSON.stringify(bad));
assert.equal((await db.select().from(auditLog)).filter((e) => e.category === "restore_test" && e.action === "record").length, 1);
});
test("the Backups page's data says how each guest's restore went, or that nobody has tried", async () => {
const cov = await (await send("GET", "/coverage")).json();
const by = Object.fromEntries(cov.guests.map((g: { name: string }) => [g.name, g]));
assert.equal(by.elsa.restoreKey, "guest:1:100");
assert.deepEqual([by.elsa.restoreTest.status, by.elsa.restoreTest.testedAt, by.elsa.restoreTest.note], ["ok", today, "booted and the data was there"]);
assert.equal(by.anna.restoreTest.status, "never");
assert.equal(cov.restoreTestDays, 0);
});
test("something that isn't a guest gets a key from its name, and is listed with how it stands", async () => {
const res = await send("POST", "/restore-tests", { label: "Photos share", result: "ok", testedAt: daysAgo(40) });
assert.equal(res.status, 201);
assert.equal((await res.json()).test.subjectKey, "custom:photos-share");
const list = await (await send("GET", "/restore-tests")).json();
assert.deepEqual(list.custom.map((c: { key: string; status: string }) => [c.key, c.status]), [["custom:photos-share", "ok"]]);
assert.equal(list.tests.length, 2);
});
test("a failed test is an alert straight away; with a limit, never-tested and old ones are too; marking 'not needed' ends it", async () => {
await send("POST", "/restore-tests", { key: "guest:1:101", label: "anna", result: "failed", note: "the disk wouldn't attach" });
invalidateAlertsCache();
let alerts = (await getAlerts(true)).alerts.filter((a) => a.id.startsWith("backup:restore-test:"));
assert.deepEqual(alerts.map((a) => [a.id, a.severity]), [["backup:restore-test:guest:1:101", "critical"]]);
assert.match(alerts[0].message, /anna \(#101\).*failed — the disk wouldn't attach/);
assert.equal(alerts[0].link, "/backups");
await updateSettings({ healthChecks: { restoreTestDays: 30 } });
invalidateAlertsCache();
alerts = (await getAlerts(true)).alerts.filter((a) => a.id.startsWith("backup:restore-test:"));
assert.deepEqual(alerts.map((a) => a.id).sort(), ["backup:restore-test:custom:photos-share", "backup:restore-test:guest:1:101"], "elsa is fine; the photos share was last tested 40 days ago");
await send("POST", "/restore-tests", { key: "guest:1:101", label: "anna", result: "not_needed", note: "a scratch machine" });
invalidateAlertsCache();
alerts = (await getAlerts(true)).alerts.filter((a) => a.id.startsWith("backup:restore-test:"));
assert.deepEqual(alerts.map((a) => a.id), ["backup:restore-test:custom:photos-share"]);
await updateSettings({ healthChecks: { restoreTestDays: 0 } });
});
test("metrics say how old each test is and whether it passed", async () => {
const text = await buildMetrics();
assert.match(text, /homelab_restore_test_age_days\{subject="elsa"\} 0/);
assert.match(text, /homelab_restore_test_age_days\{subject="Photos share"\} 40/);
assert.match(text, /homelab_restore_test_passed\{subject="elsa"\} 1/);
assert.doesNotMatch(text, /subject="anna"/, "'not needed' isn't measured");
});
test("only an administrator takes an entry out, and that is recorded", async () => {
const [first] = await db.select().from(restoreTests);
assert.equal((await send("DELETE", `/restore-tests/${first.id}`)).status, 403);
who = "admin";
assert.equal((await send("DELETE", `/restore-tests/${first.id}`)).status, 204);
assert.equal((await send("DELETE", `/restore-tests/${first.id}`)).status, 404);
assert.equal((await db.select().from(auditLog)).filter((e) => e.category === "restore_test" && e.action === "delete").length, 1);
});
});
+5 -1
View File
@@ -34,6 +34,8 @@ import IncidentDetail from "./pages/IncidentDetail";
import Inventory from "./pages/Inventory"; import Inventory from "./pages/Inventory";
import InventoryLabels from "./pages/InventoryLabels"; import InventoryLabels from "./pages/InventoryLabels";
import Dependencies from "./pages/Dependencies"; import Dependencies from "./pages/Dependencies";
import Endpoints from "./pages/Endpoints";
import ChangeLog from "./pages/ChangeLog";
import Backups from "./pages/Backups"; import Backups from "./pages/Backups";
import Domains from "./pages/Domains"; import Domains from "./pages/Domains";
import Consistency from "./pages/Consistency"; import Consistency from "./pages/Consistency";
@@ -151,7 +153,9 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/inventory" element={<Inventory user={user} />} /> <Route path="/inventory" element={<Inventory user={user} />} />
<Route path="/inventory/labels" element={<InventoryLabels />} /> <Route path="/inventory/labels" element={<InventoryLabels />} />
<Route path="/dependencies" element={<Dependencies user={user} />} /> <Route path="/dependencies" element={<Dependencies user={user} />} />
<Route path="/backups" element={<Backups />} /> <Route path="/endpoints" element={<Endpoints user={user} />} />
<Route path="/changes" element={user.role === "viewer" ? <Navigate to="/" replace /> : <ChangeLog />} />
<Route path="/backups" element={<Backups user={user} />} />
<Route path="/docker" element={<Docker user={user} />} /> <Route path="/docker" element={<Docker user={user} />} />
<Route path="/tailscale" element={<Tailscale user={user} />} /> <Route path="/tailscale" element={<Tailscale user={user} />} />
<Route path="/semaphore" element={<Semaphore user={user} />} /> <Route path="/semaphore" element={<Semaphore user={user} />} />
+106
View File
@@ -497,6 +497,8 @@ export interface HealthCheckSettings {
diskForecastDays: number; diskForecastDays: number;
/** A Proxmox guest whose newest backup is older than this many hours is raised as stale; 0 is off. */ /** A Proxmox guest whose newest backup is older than this many hours is raised as stale; 0 is off. */
backupStaleHours: number; backupStaleHours: number;
/** A restore test older than this many days is a problem; 0 = off (a failed test always is). */
restoreTestDays: number;
} }
export interface QuietHoursSettings { export interface QuietHoursSettings {
@@ -810,9 +812,34 @@ export interface CoverageGuest {
status: CoverageStatus; status: CoverageStatus;
failure: string | null; failure: string | null;
server: { id: number; name: string } | null; server: { id: number; name: string } | null;
/** The key a restore test of this guest is recorded under. */
restoreKey?: string;
/** How the newest restore test of it went. */
restoreTest?: RestoreInfo;
}
export type RestoreResult = "ok" | "partial" | "failed" | "not_needed";
export type RestoreStatus = RestoreResult | "never" | "stale";
export interface RestoreInfo {
status: RestoreStatus;
testedAt: string | null;
ageDays: number | null;
note: string | null;
testedBy: string | null;
}
export interface RestoreTestEntry {
id: number;
subjectKey: string;
label: string;
testedAt: string;
result: RestoreResult;
note: string | null;
testedBy: string | null;
createdAt: string;
} }
export interface BackupCoverage { export interface BackupCoverage {
restoreTestDays?: number;
guests: CoverageGuest[]; guests: CoverageGuest[];
counts: Record<CoverageStatus, number> & { total: number }; counts: Record<CoverageStatus, number> & { total: number };
sources: { kind: "proxmox" | "pbs"; name: string; ok: boolean; error?: string }[]; sources: { kind: "proxmox" | "pbs"; name: string; ok: boolean; error?: string }[];
@@ -820,6 +847,68 @@ export interface BackupCoverage {
generatedAt: string; generatedAt: string;
} }
export type EndpointKind = "http" | "tcp" | "tls";
export interface EndpointStats {
uptime24h: number | null;
uptime7d: number | null;
avgMs24h: number | null;
recent: { at: string; ok: boolean; ms: number | null }[];
}
export interface EndpointCheck {
id: number;
name: string;
kind: EndpointKind;
target: string;
port: number | null;
expectStatus: string | null;
keyword: string | null;
insecure: boolean;
intervalMinutes: number;
timeoutSeconds: number;
failuresBeforeAlert: number;
warnDays: number;
serverId: number | null;
enabled: boolean;
lastCheckedAt: string | null;
lastOk: boolean | null;
lastMessage: string | null;
lastResponseMs: number | null;
certExpiresAt: string | null;
consecutiveFailures: number;
lastChangeAt: string | null;
server: { id: number; name: string } | null;
stats?: EndpointStats;
}
export interface EndpointInput {
name: string;
kind: EndpointKind;
target: string;
port?: number | null;
expectStatus?: string | null;
keyword?: string | null;
insecure?: boolean;
intervalMinutes?: number;
timeoutSeconds?: number;
failuresBeforeAlert?: number;
warnDays?: number;
serverId?: number | null;
enabled?: boolean;
}
export type ChangeKind = "servers" | "maintenance" | "alerts" | "settings" | "integrations" | "dependencies" | "inventory" | "backups" | "endpoints" | "other";
export interface ChangeEntry {
id: string;
at: string;
kind: ChangeKind;
tone: "good" | "bad" | "neutral" | "info";
title: string;
detail: string | null;
actor: string | null;
target: { type: string; id: string | null; name: string | null } | null;
link: string | null;
serverId: number | null;
}
export type AssetKind = "server" | "storage" | "network" | "power" | "client" | "other"; export type AssetKind = "server" | "storage" | "network" | "power" | "client" | "other";
export type DependencyNodeType = "server" | "integration"; export type DependencyNodeType = "server" | "integration";
@@ -1757,6 +1846,23 @@ export const api = {
}, },
backups: { backups: {
coverage: (staleDays: number, refresh = false) => request<BackupCoverage>(`/api/backups/coverage?staleDays=${staleDays}${refresh ? "&refresh=1" : ""}`), coverage: (staleDays: number, refresh = false) => request<BackupCoverage>(`/api/backups/coverage?staleDays=${staleDays}${refresh ? "&refresh=1" : ""}`),
restoreTests: () => request<{ limitDays: number; tests: RestoreTestEntry[]; custom: (RestoreInfo & { key: string; label: string })[] }>("/api/backups/restore-tests"),
recordRestoreTest: (data: { key?: string; label: string; testedAt?: string; result: RestoreResult; note?: string }) => request<{ test: RestoreTestEntry }>("/api/backups/restore-tests", { method: "POST", body: JSON.stringify(data) }),
removeRestoreTest: (id: number) => request<void>(`/api/backups/restore-tests/${id}`, { method: "DELETE" }),
},
endpoints: {
list: () => request<{ checks: EndpointCheck[]; servers: { id: number; name: string }[] }>("/api/endpoints"),
create: (data: EndpointInput) => request<{ check: EndpointCheck }>("/api/endpoints", { method: "POST", body: JSON.stringify(data) }),
update: (id: number, data: EndpointInput) => request<{ check: EndpointCheck }>(`/api/endpoints/${id}`, { method: "PUT", body: JSON.stringify(data) }),
remove: (id: number) => request<void>(`/api/endpoints/${id}`, { method: "DELETE" }),
check: (id: number) => request<{ result: { ok: boolean; ms: number | null; message: string } }>(`/api/endpoints/${id}/check`, { method: "POST" }),
},
changes: {
list: (params: { days?: number; kind?: string; serverId?: number; q?: string; before?: string; limit?: number }) => {
const q = new URLSearchParams();
for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== "") q.set(k, String(v));
return request<{ entries: ChangeEntry[]; more: boolean; kinds: { id: ChangeKind; label: string }[] }>(`/api/changes?${q.toString()}`);
},
}, },
push: { push: {
key: () => request<{ publicKey: string }>("/api/push/key"), key: () => request<{ publicKey: string }>("/api/push/key"),
+6
View File
@@ -1,4 +1,5 @@
import { useEffect, useState } from "react"; import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { IconAlertTriangle, IconCircleCheck, IconEdit, IconTool } from "@tabler/icons-react"; import { IconAlertTriangle, IconCircleCheck, IconEdit, IconTool } from "@tabler/icons-react";
import { api, type ServerActivityEvent } from "../api/client"; import { api, type ServerActivityEvent } from "../api/client";
import { formatDateTime } from "../utils/date"; import { formatDateTime } from "../utils/date";
@@ -39,6 +40,11 @@ export default function ServerActivity({ serverId }: { serverId: number }) {
<div className="card mb-3"> <div className="card mb-3">
<div className="card-header"> <div className="card-header">
<h3 className="card-title">Activity</h3> <h3 className="card-title">Activity</h3>
<div className="card-actions">
<Link to={`/changes?serverId=${serverId}&days=90`} className="small">
Everything about this server in the change log
</Link>
</div>
</div> </div>
<div className="card-body"> <div className="card-body">
{failed ? ( {failed ? (
+4
View File
@@ -26,6 +26,8 @@ import {
IconDownload, IconDownload,
IconBox, IconBox,
IconHierarchy2, IconHierarchy2,
IconHeartbeat,
IconGitCommit,
IconShieldCheck, IconShieldCheck,
IconActivityHeartbeat, IconActivityHeartbeat,
IconCopyCheck, IconCopyCheck,
@@ -89,6 +91,7 @@ export const NAV: NavEntry[] = [
{ to: "/domains", label: "Domains", icon: <IconWorldWww size={20} /> }, { to: "/domains", label: "Domains", icon: <IconWorldWww size={20} /> },
{ to: "/ipam", label: "IP Addresses", icon: <IconNetwork size={20} /> }, { to: "/ipam", label: "IP Addresses", icon: <IconNetwork size={20} /> },
{ to: "/ports", label: "Ports", icon: <IconPlug size={20} /> }, { to: "/ports", label: "Ports", icon: <IconPlug size={20} /> },
{ to: "/endpoints", label: "Endpoints", icon: <IconHeartbeat size={20} /> },
{ to: "/consistency", label: "Consistency", icon: <IconListCheck size={20} /> }, { to: "/consistency", label: "Consistency", icon: <IconListCheck size={20} /> },
], ],
}, },
@@ -109,6 +112,7 @@ export const NAV: NavEntry[] = [
items: [ items: [
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> }, { to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
{ to: "/incidents", label: "Incidents", icon: <IconTimeline size={20} /> }, { to: "/incidents", label: "Incidents", icon: <IconTimeline size={20} /> },
{ to: "/changes", label: "Change log", icon: <IconGitCommit size={20} />, minRole: "operator" },
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> }, { to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
{ to: "/updates", label: "Updates", icon: <IconDownload size={20} /> }, { to: "/updates", label: "Updates", icon: <IconDownload size={20} /> },
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> }, { to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
+222 -6
View File
@@ -1,6 +1,8 @@
import { useEffect, useState } from "react"; import { useEffect, useState } from "react";
import { Link, useSearchParams } from "react-router-dom"; import { Link, useSearchParams } from "react-router-dom";
import { api, type BackupCoverage, type CoverageGuest, type CoverageStatus } from "../api/client"; import { api, type BackupCoverage, type CoverageGuest, type CoverageStatus, type CurrentUser, type RestoreInfo, type RestoreResult, type RestoreStatus, type RestoreTestEntry } from "../api/client";
import { confirmDialog } from "../utils/dialogs";
import { toast } from "../utils/toast";
import Pagination from "../components/Pagination"; import Pagination from "../components/Pagination";
import SortableTh from "../components/SortableTh"; import SortableTh from "../components/SortableTh";
import TableToolbar from "../components/TableToolbar"; import TableToolbar from "../components/TableToolbar";
@@ -30,11 +32,79 @@ const STATUS: Record<CoverageStatus, { label: string; badge: string; hint: strin
const STALE_CHOICES = [1, 2, 3, 7, 14]; const STALE_CHOICES = [1, 2, 3, 7, 14];
const RESTORE: Record<RestoreStatus, { label: string; badge: string }> = {
never: { label: "Never tested", badge: "bg-secondary-lt text-secondary" },
ok: { label: "Restored fine", badge: "bg-green-lt text-green" },
partial: { label: "Partly worked", badge: "bg-yellow-lt text-yellow" },
failed: { label: "Restore failed", badge: "bg-red-lt text-red" },
stale: { label: "Test is old", badge: "bg-yellow-lt text-yellow" },
not_needed: { label: "No need to test", badge: "bg-secondary-lt text-secondary" },
};
const RESULT_WORDS: Record<RestoreResult, string> = { ok: "It restored and worked", partial: "It restored, but not everything worked", failed: "It failed", not_needed: "No need to test this one" };
interface TestDraft {
/** A guest's key, or null for something that isn't a Proxmox guest. */
key: string | null;
label: string;
testedAt: string;
result: RestoreResult;
note: string;
}
const todayIso = () => new Date().toISOString().slice(0, 10);
const ageWords = (r: RestoreInfo) => (r.ageDays === null ? "" : r.ageDays === 0 ? "today" : r.ageDays === 1 ? "yesterday" : `${r.ageDays} days ago`);
/** /**
* What is and isn't backed up: every Proxmox guest, whether a backup job covers it, when it was last backed up (from Proxmox Backup Server * What is and isn't backed up: every Proxmox guest, whether a backup job covers it, when it was last backed up (from Proxmox Backup Server
* snapshots and Proxmox's own backup tasks), and whether that went well — one page instead of an alert per gap. * snapshots and Proxmox's own backup tasks), and whether that went well — one page instead of an alert per gap.
*/ */
export default function Backups() { export default function Backups({ user }: { user?: CurrentUser }) {
const canRecord = user?.role === "admin" || user?.role === "operator";
const isAdmin = user?.role === "admin";
const [tests, setTests] = useState<RestoreTestEntry[] | null>(null);
const [custom, setCustom] = useState<(RestoreInfo & { key: string; label: string })[]>([]);
const [draft, setDraft] = useState<TestDraft | null>(null);
const [saving, setSaving] = useState(false);
function loadTests() {
api.backups
.restoreTests()
.then((res) => {
setTests(res.tests);
setCustom(res.custom);
})
.catch(() => setTests([]));
}
useEffect(loadTests, []);
async function saveTest(e: React.FormEvent) {
e.preventDefault();
if (!draft) return;
setSaving(true);
try {
await api.backups.recordRestoreTest({ key: draft.key ?? undefined, label: draft.label.trim(), testedAt: draft.testedAt, result: draft.result, note: draft.note.trim() || undefined });
toast.success("Recorded.");
setDraft(null);
loadTests();
load();
} catch (err) {
setError(readableError(err));
} finally {
setSaving(false);
}
}
async function removeTest(t: RestoreTestEntry) {
if (!(await confirmDialog({ title: "Remove a restore test", message: `Remove the test of "${t.label}" from ${t.testedAt}?`, confirmLabel: "Remove", danger: true }))) return;
try {
await api.backups.removeRestoreTest(t.id);
loadTests();
load();
} catch (err) {
setError(readableError(err));
}
}
const [params, setParams] = useSearchParams(); const [params, setParams] = useSearchParams();
const staleParam = Number(params.get("stale")); const staleParam = Number(params.get("stale"));
const staleDays = STALE_CHOICES.includes(staleParam) ? staleParam : 2; const staleDays = STALE_CHOICES.includes(staleParam) ? staleParam : 2;
@@ -83,8 +153,8 @@ export default function Backups() {
if (!sorted) return; if (!sorted) return;
downloadCsv( downloadCsv(
"backup-coverage.csv", "backup-coverage.csv",
["Guest", "Type", "ID", "Node", "Cluster", "In a backup job", "Last backup", "Source", "Status", "Problem"], ["Guest", "Type", "ID", "Node", "Cluster", "In a backup job", "Last backup", "Source", "Status", "Problem", "Restore test", "Tested on"],
sorted.map((r) => [r.name, r.type === "qemu" ? "VM" : "Container", r.vmid, r.node, r.integrationName, r.inJob ? "yes" : "no", r.lastBackupAt ?? "", r.lastSource === "pbs" ? "Backup Server" : r.lastSource === "task" ? "Proxmox task" : "", STATUS[r.status].label, r.failure ?? ""]), sorted.map((r) => [r.name, r.type === "qemu" ? "VM" : "Container", r.vmid, r.node, r.integrationName, r.inJob ? "yes" : "no", r.lastBackupAt ?? "", r.lastSource === "pbs" ? "Backup Server" : r.lastSource === "task" ? "Proxmox task" : "", STATUS[r.status].label, r.failure ?? "", r.restoreTest ? RESTORE[r.restoreTest.status].label : "", r.restoreTest?.testedAt ?? ""]),
); );
} }
@@ -168,12 +238,13 @@ export default function Backups() {
<SortableTh<Row> label="In a job" sortKeyName="inJob" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="In a job" sortKeyName="inJob" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Last backup" sortKeyName="lastBackupAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="Last backup" sortKeyName="lastBackupAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Status" sortKeyName="rank" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="Status" sortKeyName="rank" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<th>Restore test</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
{report === null && !error && ( {report === null && !error && (
<tr> <tr>
<td colSpan={4}> <td colSpan={5}>
<BlockSkeleton lines={4} /> <BlockSkeleton lines={4} />
</td> </td>
</tr> </tr>
@@ -209,11 +280,28 @@ export default function Backups() {
</span> </span>
{r.failure && <div className="text-secondary small">{r.failure}</div>} {r.failure && <div className="text-secondary small">{r.failure}</div>}
</td> </td>
<td>
{r.restoreTest && (
<>
<span className={`badge ${RESTORE[r.restoreTest.status].badge}`} title={r.restoreTest.note ?? undefined}>
{RESTORE[r.restoreTest.status].label}
</span>
{r.restoreTest.testedAt && <div className="text-secondary small">{ageWords(r.restoreTest)}</div>}
</>
)}
{canRecord && r.restoreKey && (
<div>
<button type="button" className="btn btn-link btn-sm p-0" onClick={() => setDraft({ key: r.restoreKey!, label: r.name, testedAt: todayIso(), result: "ok", note: "" })}>
Record a test
</button>
</div>
)}
</td>
</tr> </tr>
))} ))}
{report !== null && sorted?.length === 0 && ( {report !== null && sorted?.length === 0 && (
<tr> <tr>
<td colSpan={4} className="text-secondary text-center"> <td colSpan={5} className="text-secondary text-center">
{rows.length === 0 ? ( {rows.length === 0 ? (
<> <>
No Proxmox guests to show. Connect a Proxmox integration under <Link to="/integrations">Integrations</Link>. No Proxmox guests to show. Connect a Proxmox integration under <Link to="/integrations">Integrations</Link>.
@@ -233,6 +321,134 @@ export default function Backups() {
name its guests, so a guest backed up to something other than a Backup Server reads “Unknown” rather than being guessed at. The result is kept for a minute; “Check now” asks again. name its guests, so a guest backed up to something other than a Backup Server reads “Unknown” rather than being guessed at. The result is kept for a minute; “Check now” asks again.
</div> </div>
</div> </div>
{draft && (
<form className="card mt-3" onSubmit={saveTest}>
<div className="card-header">
<h3 className="card-title">Record a restore test{draft.key ? ` of ${draft.label}` : ""}</h3>
</div>
<div className="card-body row g-3">
{draft.key === null && (
<div className="col-md-5">
<label className="form-label" htmlFor="rt-label">
What was restored
</label>
<input id="rt-label" className="form-control" required maxLength={120} placeholder="e.g. Photos share, Gitea database" value={draft.label} onChange={(e) => setDraft({ ...draft, label: e.target.value })} />
</div>
)}
<div className="col-md-3">
<label className="form-label" htmlFor="rt-date">
On
</label>
<input id="rt-date" type="date" className="form-control" required max={todayIso()} value={draft.testedAt} onChange={(e) => setDraft({ ...draft, testedAt: e.target.value })} />
</div>
<div className="col-md-4">
<label className="form-label" htmlFor="rt-result">
How it went
</label>
<select id="rt-result" className="form-select" value={draft.result} onChange={(e) => setDraft({ ...draft, result: e.target.value as RestoreResult })}>
{(Object.keys(RESULT_WORDS) as RestoreResult[]).map((r) => (
<option key={r} value={r}>
{RESULT_WORDS[r]}
</option>
))}
</select>
</div>
<div className="col-12">
<label className="form-label" htmlFor="rt-note">
Note (optional)
</label>
<input id="rt-note" className="form-control" maxLength={500} placeholder="e.g. restored to a scratch VM, booted, the database was there" value={draft.note} onChange={(e) => setDraft({ ...draft, note: e.target.value })} />
</div>
</div>
<div className="card-footer d-flex gap-2">
<button type="submit" className="btn btn-primary" disabled={saving || draft.label.trim() === ""}>
{saving ? "Saving…" : "Record it"}
</button>
<button type="button" className="btn" onClick={() => setDraft(null)}>
Cancel
</button>
</div>
</form>
)}
<div className="card mt-3">
<div className="card-header">
<h3 className="card-title">Restore tests</h3>
{canRecord && !draft && (
<div className="card-actions">
<button className="btn btn-sm" onClick={() => setDraft({ key: null, label: "", testedAt: todayIso(), result: "ok", note: "" })}>
Record a test of something else
</button>
</div>
)}
</div>
<div className="card-body text-secondary small pb-0">
A backup nobody has restored is a hope, not a backup. Record each time you try one — for a guest above, or for anything else (a share, a database). A failed test is raised on the
Alerts page; under Settings → Notifications you can also ask to be reminded when a guest with backups hasn&rsquo;t had one in a while{report?.restoreTestDays ? ` (now: ${report.restoreTestDays} days)` : ""}.
</div>
{custom.length > 0 && (
<div className="card-body pb-0">
<div className="text-secondary small mb-1">Other things being tracked</div>
<div className="d-flex flex-wrap gap-2">
{custom.map((c) => (
<span key={c.key} className={`badge ${RESTORE[c.status].badge}`}>
{c.label} · {RESTORE[c.status].label.toLowerCase()} {ageWords(c)}
</span>
))}
</div>
</div>
)}
<div className="table-responsive">
<table className="table table-vcenter card-table table-stack">
<thead>
<tr>
<th>Date</th>
<th>What</th>
<th>Result</th>
<th>Note</th>
{isAdmin && <th className="w-1">Actions</th>}
</tr>
</thead>
<tbody>
{tests === null && (
<tr>
<td colSpan={isAdmin ? 5 : 4}>
<BlockSkeleton lines={2} />
</td>
</tr>
)}
{tests?.map((t) => (
<tr key={t.id}>
<td className="stack-label" data-label="Date">{t.testedAt}</td>
<td>{t.label}</td>
<td>
<span className={`badge ${RESTORE[t.result].badge}`}>{RESTORE[t.result].label}</span>
</td>
<td className="text-secondary stack-wide">
{t.note ?? "—"}
{t.testedBy && <span className="small"> · {t.testedBy}</span>}
</td>
{isAdmin && (
<td className="stack-wide">
<button className="btn btn-sm btn-outline-danger" onClick={() => void removeTest(t)}>
Remove
</button>
</td>
)}
</tr>
))}
{tests !== null && tests.length === 0 && (
<tr>
<td colSpan={isAdmin ? 5 : 4} className="text-secondary text-center">
No restore has been recorded yet.{canRecord ? " After you next try one, record it here." : ""}
</td>
</tr>
)}
</tbody>
</table>
</div>
</div>
</> </>
); );
} }
+209
View File
@@ -0,0 +1,209 @@
import { useEffect, useState } from "react";
import { Link, useSearchParams } from "react-router-dom";
import { api, type ChangeEntry, type ChangeKind } from "../api/client";
import { ErrorNotice } from "../components/Notice";
import { BlockSkeleton } from "../components/Skeleton";
import { downloadCsv } from "../utils/csv";
import { formatDate, formatTime } from "../utils/date";
import { readableError } from "../utils/errors";
const TONE_DOT: Record<ChangeEntry["tone"], string> = { good: "#2fb344", bad: "#e05252", info: "#4299e1", neutral: "#8a94a6" };
const RANGES: { days: number; label: string }[] = [
{ days: 1, label: "Last 24 hours" },
{ days: 7, label: "Last 7 days" },
{ days: 30, label: "Last 30 days" },
{ days: 90, label: "Last 90 days" },
{ days: 400, label: "Everything kept" },
];
const PAGE = 100;
/**
* What has changed, in one list: agents updated, settings changed, maintenance started, dependencies edited, alerts raised and cleared — each as a
* sentence, newest first, grouped by day. Narrow it to a kind of change, a server, a stretch of time or a word. The Audit Log is the raw table behind it.
*/
export default function ChangeLog() {
const [params, setParams] = useSearchParams();
const days = RANGES.some((r) => r.days === Number(params.get("days"))) ? Number(params.get("days")) : 7;
const kind = params.get("kind") ?? "";
const serverId = /^\d+$/.test(params.get("serverId") ?? "") ? Number(params.get("serverId")) : undefined;
const q = params.get("q") ?? "";
const [text, setText] = useState(q);
const [entries, setEntries] = useState<ChangeEntry[] | null>(null);
const [more, setMore] = useState(false);
const [kinds, setKinds] = useState<{ id: ChangeKind; label: string }[]>([]);
const [servers, setServers] = useState<{ id: number; name: string }[]>([]);
const [error, setError] = useState<string | null>(null);
const [loadingMore, setLoadingMore] = useState(false);
const setParam = (key: string, value: string | null) =>
setParams(
(prev) => {
const next = new URLSearchParams(prev);
if (value) next.set(key, value);
else next.delete(key);
return next;
},
{ replace: true },
);
useEffect(() => {
api.servers
.list()
.then((res) => setServers(res.servers.map((s) => ({ id: s.id, name: s.name })).sort((a, b) => a.name.localeCompare(b.name))))
.catch(() => {});
}, []);
// What is searched for follows what is typed, a moment after it stops.
useEffect(() => {
const t = setTimeout(() => setParam("q", text.trim() || null), 300);
return () => clearTimeout(t);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [text]);
useEffect(() => {
let alive = true;
setEntries(null);
api.changes
.list({ days, kind, serverId, q, limit: PAGE })
.then((res) => {
if (!alive) return;
setEntries(res.entries);
setMore(res.more);
setKinds(res.kinds);
setError(null);
})
.catch((err) => alive && setError(readableError(err)));
return () => {
alive = false;
};
}, [days, kind, serverId, q]);
async function older() {
if (!entries || entries.length === 0) return;
setLoadingMore(true);
try {
const res = await api.changes.list({ days, kind, serverId, q, limit: PAGE, before: entries[entries.length - 1].at });
setEntries([...entries, ...res.entries]);
setMore(res.more);
} catch (err) {
setError(readableError(err));
} finally {
setLoadingMore(false);
}
}
// Grouped by the day it happened (in this browser's time).
const days_: { day: string; items: ChangeEntry[] }[] = [];
for (const e of entries ?? []) {
const day = formatDate(new Date(e.at));
const last = days_[days_.length - 1];
if (last && last.day === day) last.items.push(e);
else days_.push({ day, items: [e] });
}
function exportCsv() {
downloadCsv(
"change-log.csv",
["When", "Kind", "What", "Details", "By", "About"],
(entries ?? []).map((e) => [e.at, kinds.find((k) => k.id === e.kind)?.label ?? e.kind, e.title, e.detail ?? "", e.actor ?? "", e.target?.name ?? ""]),
);
}
return (
<>
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
<h2 className="page-title mb-0">Change log</h2>
<button className="btn btn-outline-secondary ms-auto" onClick={exportCsv} disabled={!entries || entries.length === 0}>
Export CSV
</button>
</div>
<p className="text-secondary">
What changed and when: agents updated, settings changed, maintenance started and ended, dependencies edited, alerts raised and cleared. The <Link to="/audit-log">Audit Log</Link> is the raw table behind it.
</p>
<ErrorNotice error={error} />
<div className="card mb-3">
<div className="card-body d-flex flex-wrap gap-2 align-items-center">
<input className="form-control w-auto flex-grow-1" style={{ maxWidth: "22rem" }} type="search" aria-label="Search the change log" placeholder="Search: a server, a person, a word…" value={text} onChange={(e) => setText(e.target.value)} />
<select className="form-select w-auto" aria-label="Kind of change" value={kind} onChange={(e) => setParam("kind", e.target.value || null)}>
<option value="">All kinds</option>
{kinds.map((k) => (
<option key={k.id} value={k.id}>
{k.label}
</option>
))}
</select>
<select className="form-select w-auto" aria-label="Server" value={serverId ?? ""} onChange={(e) => setParam("serverId", e.target.value || null)}>
<option value="">Every server</option>
{servers.map((s) => (
<option key={s.id} value={s.id}>
{s.name}
</option>
))}
</select>
<select className="form-select w-auto" aria-label="How far back" value={days} onChange={(e) => setParam("days", e.target.value === "7" ? null : e.target.value)}>
{RANGES.map((r) => (
<option key={r.days} value={r.days}>
{r.label}
</option>
))}
</select>
</div>
</div>
{entries === null && !error && <BlockSkeleton lines={5} />}
{entries !== null && entries.length === 0 && (
<div className="card">
<div className="card-body text-secondary text-center">
Nothing {q || kind || serverId ? "matches that" : "has changed"} in {RANGES.find((r) => r.days === days)?.label.toLowerCase()}.
{(q || kind || serverId || days < 400) && " Try a wider range or fewer filters."}
</div>
</div>
)}
{days_.map((group) => (
<div className="card mb-3" key={group.day}>
<div className="card-header">
<h3 className="card-title">{group.day}</h3>
</div>
<div className="list-group list-group-flush">
{group.items.map((e) => (
<div className="list-group-item" key={e.id}>
<div className="d-flex gap-3">
<span className="flex-shrink-0 text-secondary small text-nowrap" style={{ width: "3.2rem" }}>
{formatTime(new Date(e.at), false)}
</span>
<span aria-hidden="true" className="flex-shrink-0 mt-2" style={{ width: 8, height: 8, borderRadius: "50%", background: TONE_DOT[e.tone] }} />
<div className="flex-fill" style={{ minWidth: 0 }}>
<div className="text-break">
{e.title}
{e.link && (
<Link to={e.link} className="ms-2 small">
open
</Link>
)}
</div>
{e.detail && <div className="text-secondary small text-break">{e.detail}</div>}
<div className="text-secondary small">
{kinds.find((k) => k.id === e.kind)?.label ?? e.kind}
{e.actor ? ` · ${e.actor}` : ""}
</div>
</div>
</div>
</div>
))}
</div>
</div>
))}
{more && (
<div className="text-center mb-3">
<button className="btn" onClick={() => void older()} disabled={loadingMore}>
{loadingMore ? "Loading…" : "Show older"}
</button>
</div>
)}
</>
);
}
+53
View File
@@ -9,6 +9,7 @@ import {
api, api,
type CurrentUser, type CurrentUser,
type DomainList, type DomainList,
type EndpointCheck,
type IntegrationSummary, type IntegrationSummary,
type IntegrationType, type IntegrationType,
type OsTicketTicketsResponse, type OsTicketTicketsResponse,
@@ -362,9 +363,14 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
const [serverSummary, setServerSummary] = useState<ServerSummary | null>(null); const [serverSummary, setServerSummary] = useState<ServerSummary | null>(null);
const [serverError, setServerError] = useState<string | null>(null); const [serverError, setServerError] = useState<string | null>(null);
const [domainList, setDomainList] = useState<DomainList | null>(null); const [domainList, setDomainList] = useState<DomainList | null>(null);
const [endpointChecks, setEndpointChecks] = useState<EndpointCheck[] | null>(null);
const [domainError, setDomainError] = useState<string | null>(null); const [domainError, setDomainError] = useState<string | null>(null);
useEffect(() => { useEffect(() => {
api.endpoints
.list()
.then((res) => setEndpointChecks(res.checks))
.catch(() => setEndpointChecks([]));
api.domains api.domains
.list() .list()
.then(setDomainList) .then(setDomainList)
@@ -604,11 +610,16 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
{ id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 }, { id: "people", title: "Let someone else in (optional)", description: "Others sign in with Authentik and appear under Users; you decide their role.", to: "/users", action: "Open Users", done: (userCount ?? 0) > 1 },
]; ];
// What the endpoint checks say: the ones that keep failing, and the certificates that are running out.
const endpointsDown = (endpointChecks ?? []).filter((c) => c.enabled && c.lastOk === false && c.consecutiveFailures >= c.failuresBeforeAlert);
const endpointCerts = (endpointChecks ?? []).filter((c) => c.enabled && c.certExpiresAt && Math.floor((Date.parse(c.certExpiresAt) - Date.now()) / 86_400_000) <= c.warnDays).length;
// What has nothing set up yet. These get one list at the bottom instead of a grey "Not configured" card each. // What has nothing set up yet. These get one list at the bottom instead of a grey "Not configured" card each.
const notSetUp: { label: string; to: string; what: string }[] = []; const notSetUp: { label: string; to: string; what: string }[] = [];
if (dnsStats?.totalProviders === 0) notSetUp.push({ label: "DNS", to: "/dns", what: "connect a DNS provider to see zones and records" }); if (dnsStats?.totalProviders === 0) notSetUp.push({ label: "DNS", to: "/dns", what: "connect a DNS provider to see zones and records" });
if (secrets !== null && totalSecrets === 0) notSetUp.push({ label: "Secrets", to: "/secrets", what: "track when a token, certificate or password expires" }); if (secrets !== null && totalSecrets === 0) notSetUp.push({ label: "Secrets", to: "/secrets", what: "track when a token, certificate or password expires" });
if (serverSummary?.total === 0) notSetUp.push({ label: "Servers", to: "/servers", what: "add a server and install its agent" }); if (serverSummary?.total === 0) notSetUp.push({ label: "Servers", to: "/servers", what: "add a server and install its agent" });
if (endpointChecks !== null && endpointChecks.length === 0) notSetUp.push({ label: "Endpoints", to: "/endpoints", what: "ask a web address, a port or a certificate whether it is fine" });
if (domainList !== null && domains.length === 0) notSetUp.push({ label: "Domains", to: "/domains", what: "appear after a DNS zone is added and the daily check has run" }); if (domainList !== null && domains.length === 0) notSetUp.push({ label: "Domains", to: "/domains", what: "appear after a DNS zone is added and the daily check has run" });
if (integrations !== null) { if (integrations !== null) {
for (const { type, label } of WIDGETS) { for (const { type, label } of WIDGETS) {
@@ -931,6 +942,48 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
</> </>
)} )}
</WidgetCard> </WidgetCard>
<WidgetCard
label="Endpoints"
hide={endpointChecks !== null && endpointChecks.length === 0}
badge={
!endpointChecks ? undefined : endpointsDown.length > 0 ? (
<span className="badge bg-red-lt text-red">{endpointsDown.length} not answering</span>
) : endpointCerts > 0 ? (
<span className="badge bg-yellow-lt text-yellow">
{endpointCerts} certificate{endpointCerts === 1 ? "" : "s"} to renew
</span>
) : (
<span className="badge bg-green-lt text-green">All answering</span>
)
}
>
{!endpointChecks ? (
<WidgetSkeleton />
) : (
<>
<div className="row g-2 mb-3">
<div className="col-4">
<MiniStat label="Checks" value={endpointChecks.filter((c) => c.enabled).length} />
</div>
<div className="col-4">
<MiniStat label="Up" value={endpointChecks.filter((c) => c.enabled && c.lastOk === true).length} />
</div>
<div className="col-4">
<MiniStat label="Down" value={endpointsDown.length} accent={endpointsDown.length > 0 ? "#ef4444" : undefined} />
</div>
</div>
{endpointsDown.slice(0, 3).map((c) => (
<div key={c.id} className="text-red small">
{c.name} — {c.lastMessage}
</div>
))}
<div className="text-secondary small mt-2">
<Link to="/endpoints">All checks</Link>
</div>
</>
)}
</WidgetCard>
</div> </div>
<SectionHeader <SectionHeader
+494
View File
@@ -0,0 +1,494 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type CurrentUser, type EndpointCheck, type EndpointInput, type EndpointKind } from "../api/client";
import Pagination from "../components/Pagination";
import SortableTh from "../components/SortableTh";
import TableToolbar from "../components/TableToolbar";
import TimeAgo from "../components/TimeAgo";
import { ErrorNotice } from "../components/Notice";
import { BlockSkeleton } from "../components/Skeleton";
import { usePagination } from "../hooks/usePagination";
import { useSortable } from "../hooks/useSortable";
import { useTableSearch } from "../hooks/useTableSearch";
import { downloadCsv } from "../utils/csv";
import { confirmDialog } from "../utils/dialogs";
import { readableError } from "../utils/errors";
import { toast } from "../utils/toast";
const KIND_LABELS: Record<EndpointKind, string> = { http: "Web address", tcp: "Port", tls: "Certificate" };
const KIND_HINTS: Record<EndpointKind, string> = {
http: "Asks the address and wants a good answer (and, if you like, a word in the page).",
tcp: "Opens a connection to the port and sees whether anything answers.",
tls: "Reads the certificate the service shows and warns before it runs out.",
};
type State = "waiting" | "up" | "failing" | "down" | "off";
const STATE_BADGE: Record<State, { label: string; badge: string }> = {
waiting: { label: "Not asked yet", badge: "bg-secondary-lt text-secondary" },
up: { label: "Up", badge: "bg-green-lt text-green" },
failing: { label: "Failing", badge: "bg-yellow-lt text-yellow" },
down: { label: "Down", badge: "bg-red-lt text-red" },
off: { label: "Switched off", badge: "bg-secondary-lt text-secondary" },
};
const RANK: Record<State, number> = { down: 0, failing: 1, waiting: 2, up: 3, off: 4 };
function stateOf(c: EndpointCheck): State {
if (!c.enabled) return "off";
if (c.lastOk === null) return "waiting";
if (c.lastOk) return "up";
return c.consecutiveFailures >= c.failuresBeforeAlert ? "down" : "failing";
}
const daysLeft = (iso: string | null) => (iso ? Math.floor((Date.parse(iso) - Date.now()) / 86_400_000) : null);
interface Row extends EndpointCheck {
state: State;
rank: number;
certDays: number | null;
uptime24h: number | null;
}
/** The last results as a row of small ticks: green for an answer, red for none. */
function Ticks({ recent }: { recent: { at: string; ok: boolean }[] }) {
if (recent.length === 0) return <span className="text-secondary">—</span>;
return (
<span className="d-inline-flex gap-1 align-middle" role="img" aria-label={`The last ${recent.length} checks: ${recent.filter((r) => r.ok).length} answered`}>
{recent.map((r) => (
<span key={r.at} title={`${r.at.slice(0, 16).replace("T", " ")} UTC — ${r.ok ? "answered" : "no good answer"}`} style={{ width: 4, height: 14, borderRadius: 1, background: r.ok ? "#2fb344" : "#e05252", display: "inline-block" }} />
))}
</span>
);
}
interface Draft {
id: number | null;
name: string;
kind: EndpointKind;
target: string;
port: string;
expectStatus: string;
keyword: string;
insecure: boolean;
intervalMinutes: string;
timeoutSeconds: string;
failuresBeforeAlert: string;
warnDays: string;
serverId: string;
enabled: boolean;
}
const EMPTY: Draft = { id: null, name: "", kind: "http", target: "", port: "", expectStatus: "", keyword: "", insecure: false, intervalMinutes: "5", timeoutSeconds: "10", failuresBeforeAlert: "2", warnDays: "14", serverId: "", enabled: true };
const toDraft = (c: EndpointCheck): Draft => ({
id: c.id,
name: c.name,
kind: c.kind,
target: c.target,
port: c.port === null ? "" : String(c.port),
expectStatus: c.expectStatus ?? "",
keyword: c.keyword ?? "",
insecure: c.insecure,
intervalMinutes: String(c.intervalMinutes),
timeoutSeconds: String(c.timeoutSeconds),
failuresBeforeAlert: String(c.failuresBeforeAlert),
warnDays: String(c.warnDays),
serverId: c.serverId === null ? "" : String(c.serverId),
enabled: c.enabled,
});
const toInput = (d: Draft): EndpointInput => ({
name: d.name.trim(),
kind: d.kind,
target: d.target.trim(),
port: d.kind === "http" || d.port.trim() === "" ? null : Number(d.port),
expectStatus: d.kind === "http" && d.expectStatus.trim() ? d.expectStatus.trim() : null,
keyword: d.kind === "http" && d.keyword ? d.keyword : null,
insecure: d.insecure,
intervalMinutes: Number(d.intervalMinutes),
timeoutSeconds: Number(d.timeoutSeconds),
failuresBeforeAlert: Number(d.failuresBeforeAlert),
warnDays: Number(d.warnDays),
serverId: d.serverId === "" ? null : Number(d.serverId),
enabled: d.enabled,
});
/**
* "Is it up?" for the simple cases: a web address, a TCP port, a certificate. The app asks every few minutes, remembers how it went, and
* raises an alert (and a notification) when a check keeps failing or a certificate is about to run out. Everyone can look; an administrator
* sets them up, since each one makes this server connect somewhere.
*/
export default function Endpoints({ user }: { user: CurrentUser }) {
const isAdmin = user.role === "admin";
const canCheck = user.role === "admin" || user.role === "operator";
const [checks, setChecks] = useState<EndpointCheck[] | null>(null);
const [servers, setServers] = useState<{ id: number; name: string }[]>([]);
const [error, setError] = useState<string | null>(null);
const [draft, setDraft] = useState<Draft | null>(null);
const [busy, setBusy] = useState(false);
const [asking, setAsking] = useState<number | null>(null);
function load() {
api.endpoints
.list()
.then((res) => {
setChecks(res.checks);
setServers(res.servers);
})
.catch((err) => setError(readableError(err)));
}
useEffect(() => {
load();
// The results come in on their own every few minutes: look again now and then.
const t = setInterval(load, 60_000);
return () => clearInterval(t);
}, []);
const rows: Row[] = (checks ?? []).map((c) => ({ ...c, state: stateOf(c), rank: RANK[stateOf(c)], certDays: daysLeft(c.certExpiresAt), uptime24h: c.stats?.uptime24h ?? null }));
const tableSearch = useTableSearch(rows);
const { sorted, sortKey, sortDir, requestSort } = useSortable(tableSearch.filtered, "rank", "asc", { url: true });
const page = usePagination(sorted);
const down = rows.filter((r) => r.state === "down" || r.state === "failing").length;
const certs = rows.filter((r) => r.certDays !== null && r.certDays <= r.warnDays).length;
async function save(e: React.FormEvent) {
e.preventDefault();
if (!draft) return;
setBusy(true);
setError(null);
try {
if (draft.id === null) await api.endpoints.create(toInput(draft));
else await api.endpoints.update(draft.id, toInput(draft));
toast.success(draft.id === null ? `Added ${draft.name}.` : "Saved.");
setDraft(null);
load();
} catch (err) {
setError(readableError(err));
} finally {
setBusy(false);
}
}
async function remove(c: EndpointCheck) {
if (!(await confirmDialog({ title: "Remove a check", message: `Remove "${c.name}" and its history?`, confirmLabel: "Remove", danger: true }))) return;
try {
await api.endpoints.remove(c.id);
load();
} catch (err) {
setError(readableError(err));
}
}
async function askNow(c: EndpointCheck) {
setAsking(c.id);
try {
const { result } = await api.endpoints.check(c.id);
if (result.ok) toast.success(`${c.name}: ${result.message}`);
else toast.error(`${c.name}: ${result.message}`);
load();
} catch (err) {
setError(readableError(err));
} finally {
setAsking(null);
}
}
function exportCsv() {
downloadCsv(
"endpoint-checks.csv",
["Name", "Kind", "Target", "Port", "Status", "Last answer", "Answer time (ms)", "Uptime 24 h (%)", "Uptime 7 d (%)", "Certificate expires", "Last checked", "Server"],
(sorted ?? []).map((r) => [r.name, KIND_LABELS[r.kind], r.target, r.port ?? "", STATE_BADGE[r.state].label, r.lastMessage ?? "", r.lastResponseMs ?? "", r.stats?.uptime24h ?? "", r.stats?.uptime7d ?? "", r.certExpiresAt?.slice(0, 10) ?? "", r.lastCheckedAt ?? "", r.server?.name ?? ""]),
);
}
const set = <K extends keyof Draft>(key: K, value: Draft[K]) => setDraft((d) => (d ? { ...d, [key]: value } : d));
const cards = [
{ label: "Checks", value: checks ? String(rows.length) : "…", color: "" },
{ label: "Not answering", value: checks ? String(down) : "…", color: down > 0 ? "text-red" : "text-green" },
{ label: "Certificates to renew", value: checks ? String(certs) : "…", color: certs > 0 ? "text-yellow" : "" },
];
return (
<>
<div className="d-flex align-items-center mb-1">
<h2 className="page-title mb-0">Endpoints</h2>
{isAdmin && !draft && (
<button className="btn btn-primary ms-auto" onClick={() => setDraft({ ...EMPTY })}>
Add a check
</button>
)}
</div>
<p className="text-secondary">
The app asks a web address, a port or a certificate whether it is fine, every few minutes, and says so on the <Link to="/alerts">Alerts</Link> page and in notifications when it keeps failing or a
certificate is about to run out.
</p>
<ErrorNotice error={error} />
<div className="row row-cards mb-3">
{cards.map((c) => (
<div className="col-6 col-md-4" key={c.label}>
<div className="card card-sm">
<div className="card-body">
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
<div className="text-secondary">{c.label}</div>
</div>
</div>
</div>
))}
</div>
{draft && (
<form className="card mb-3" onSubmit={save}>
<div className="card-header">
<h3 className="card-title">{draft.id === null ? "Add a check" : `Edit ${draft.name || "the check"}`}</h3>
</div>
<div className="card-body row g-3">
<div className="col-md-4">
<label className="form-label" htmlFor="ep-name">
Name
</label>
<input id="ep-name" className="form-control" required maxLength={100} placeholder="e.g. Gitea" value={draft.name} onChange={(e) => set("name", e.target.value)} />
</div>
<div className="col-md-4">
<label className="form-label" htmlFor="ep-kind">
What to ask
</label>
<select id="ep-kind" className="form-select" value={draft.kind} onChange={(e) => set("kind", e.target.value as EndpointKind)}>
{(Object.keys(KIND_LABELS) as EndpointKind[]).map((k) => (
<option key={k} value={k}>
{KIND_LABELS[k]}
</option>
))}
</select>
<div className="form-hint text-secondary small mt-1">{KIND_HINTS[draft.kind]}</div>
</div>
<div className="col-md-4">
<label className="form-label" htmlFor="ep-server">
It is about the server
</label>
<select id="ep-server" className="form-select" value={draft.serverId} onChange={(e) => set("serverId", e.target.value)}>
<option value="">— none in particular —</option>
{servers.map((s) => (
<option key={s.id} value={s.id}>
{s.name}
</option>
))}
</select>
<div className="form-hint text-secondary small mt-1">Maintenance on that server silences this check&rsquo;s alerts, and its tags can route them.</div>
</div>
<div className={draft.kind === "http" ? "col-md-8" : "col-md-5"}>
<label className="form-label" htmlFor="ep-target">
{draft.kind === "http" ? "Address" : "Host name or address"}
</label>
<input
id="ep-target"
className="form-control"
required
maxLength={500}
placeholder={draft.kind === "http" ? "https://git.example.lan/" : "git.example.lan"}
value={draft.target}
onChange={(e) => set("target", e.target.value)}
/>
</div>
{draft.kind !== "http" && (
<div className="col-md-3">
<label className="form-label" htmlFor="ep-port">
Port
</label>
<input id="ep-port" type="number" min={1} max={65535} className="form-control" required value={draft.port} placeholder={draft.kind === "tls" ? "443" : ""} onChange={(e) => set("port", e.target.value)} />
</div>
)}
{draft.kind === "http" && (
<>
<div className="col-md-2">
<label className="form-label" htmlFor="ep-status">
Good answers
</label>
<input id="ep-status" className="form-control" placeholder="200-399" value={draft.expectStatus} onChange={(e) => set("expectStatus", e.target.value)} />
</div>
<div className="col-md-2">
<label className="form-label" htmlFor="ep-keyword">
Page contains
</label>
<input id="ep-keyword" className="form-control" maxLength={200} value={draft.keyword} onChange={(e) => set("keyword", e.target.value)} />
</div>
</>
)}
<div className="col-md-2">
<label className="form-label" htmlFor="ep-interval">
Every
</label>
<div className="input-group">
<input id="ep-interval" type="number" min={1} max={1440} className="form-control" value={draft.intervalMinutes} onChange={(e) => set("intervalMinutes", e.target.value)} />
<span className="input-group-text">min</span>
</div>
</div>
<div className="col-md-2">
<label className="form-label" htmlFor="ep-timeout">
Give up after
</label>
<div className="input-group">
<input id="ep-timeout" type="number" min={1} max={60} className="form-control" value={draft.timeoutSeconds} onChange={(e) => set("timeoutSeconds", e.target.value)} />
<span className="input-group-text">s</span>
</div>
</div>
<div className="col-md-3">
<label className="form-label" htmlFor="ep-fails">
Alert after
</label>
<div className="input-group">
<input id="ep-fails" type="number" min={1} max={20} className="form-control" value={draft.failuresBeforeAlert} onChange={(e) => set("failuresBeforeAlert", e.target.value)} />
<span className="input-group-text">failures in a row</span>
</div>
</div>
<div className="col-md-3">
<label className="form-label" htmlFor="ep-warn">
Warn about the certificate
</label>
<div className="input-group">
<input id="ep-warn" type="number" min={1} max={90} className="form-control" value={draft.warnDays} onChange={(e) => set("warnDays", e.target.value)} />
<span className="input-group-text">days before</span>
</div>
</div>
<div className="col-12 d-flex flex-wrap gap-4">
<label className="form-check mb-0">
<input type="checkbox" className="form-check-input" checked={draft.insecure} onChange={(e) => set("insecure", e.target.checked)} />
<span className="form-check-label">Accept a certificate that isn&rsquo;t trusted (self-signed, a private authority). Its expiry is still watched.</span>
</label>
<label className="form-check mb-0">
<input type="checkbox" className="form-check-input" checked={draft.enabled} onChange={(e) => set("enabled", e.target.checked)} />
<span className="form-check-label">Switched on</span>
</label>
</div>
<div className="col-12 text-secondary small">Only addresses this server may reach are accepted: not itself (loopback) and not the cloud metadata address. Your own network is fine.</div>
</div>
<div className="card-footer d-flex gap-2">
<button type="submit" className="btn btn-primary" disabled={busy || draft.name.trim() === "" || draft.target.trim() === ""}>
{busy ? "Saving…" : "Save"}
</button>
<button type="button" className="btn" onClick={() => setDraft(null)}>
Cancel
</button>
</div>
</form>
)}
<div className="card">
<TableToolbar search={tableSearch.query} onSearch={tableSearch.setQuery} placeholder="Search checks…" shown={sorted?.length} total={rows.length} onExport={exportCsv} exportDisabled={!sorted || sorted.length === 0} />
<div className="table-responsive">
<table className="table table-vcenter card-table table-stack">
<thead>
<tr>
<SortableTh<Row> label="Check" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Status" sortKeyName="rank" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Answer" sortKeyName="lastResponseMs" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Uptime" sortKeyName="uptime24h" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Certificate" sortKeyName="certDays" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Checked" sortKeyName="lastCheckedAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
{(canCheck || isAdmin) && <th className="w-1">Actions</th>}
</tr>
</thead>
<tbody>
{checks === null && !error && (
<tr>
<td colSpan={7}>
<BlockSkeleton lines={3} />
</td>
</tr>
)}
{page.pageItems?.map((c) => (
<tr key={c.id}>
<td className="stack-wide">
<strong>{c.name}</strong> <span className="badge bg-secondary-lt text-secondary">{KIND_LABELS[c.kind]}</span>
<div className="text-secondary small text-break">
{c.target}
{c.port !== null ? `:${c.port}` : ""}
</div>
{c.server && (
<div className="small">
about <Link to={`/servers/${c.server.id}`}>{c.server.name}</Link>
</div>
)}
</td>
<td>
<span className={`badge ${STATE_BADGE[c.state].badge}`}>{STATE_BADGE[c.state].label}</span>
{c.lastOk === false && c.lastMessage && <div className="text-secondary small">{c.lastMessage}</div>}
{c.lastOk === false && c.state === "failing" && (
<div className="text-secondary small">
failed {c.consecutiveFailures} of {c.failuresBeforeAlert}
</div>
)}
</td>
<td className="stack-label" data-label="Answer">
{c.lastResponseMs === null ? <span className="text-secondary">—</span> : `${c.lastResponseMs} ms`}
{c.stats?.avgMs24h != null && <div className="text-secondary small">avg {c.stats.avgMs24h} ms</div>}
</td>
<td className="stack-label" data-label="Uptime">
{c.stats?.uptime24h == null ? (
<span className="text-secondary">—</span>
) : (
<>
{c.stats.uptime24h}% <span className="text-secondary small">24 h</span>
<div className="text-secondary small">{c.stats.uptime7d}% 7 d</div>
</>
)}
<div>
<Ticks recent={c.stats?.recent ?? []} />
</div>
</td>
<td className="stack-label" data-label="Certificate">
{c.certDays === null ? (
<span className="text-secondary">—</span>
) : c.certDays < 0 ? (
<span className="badge bg-red-lt text-red" title={c.certExpiresAt?.slice(0, 10)}>
expired {-c.certDays} d ago
</span>
) : c.certDays <= c.warnDays ? (
<span className={`badge ${c.certDays <= 3 ? "bg-red-lt text-red" : "bg-yellow-lt text-yellow"}`} title={c.certExpiresAt?.slice(0, 10)}>
{c.certDays} d left
</span>
) : (
<span className="text-secondary">{c.certDays} d left</span>
)}
</td>
<td className="text-secondary stack-label" data-label="Checked">
{c.lastCheckedAt ? <TimeAgo value={c.lastCheckedAt} /> : "—"}
</td>
{(canCheck || isAdmin) && (
<td className="stack-wide">
<div className="btn-list flex-nowrap">
{canCheck && (
<button className="btn btn-sm btn-outline-primary" disabled={asking === c.id} onClick={() => void askNow(c)}>
{asking === c.id ? "Asking…" : "Check now"}
</button>
)}
{isAdmin && (
<>
<button className="btn btn-sm btn-outline-secondary" onClick={() => setDraft(toDraft(c))}>
Edit
</button>
<button className="btn btn-sm btn-outline-danger" onClick={() => void remove(c)}>
Remove
</button>
</>
)}
</div>
</td>
)}
</tr>
))}
{checks !== null && sorted?.length === 0 && (
<tr>
<td colSpan={7} className="text-secondary text-center">
{tableSearch.active ? "Nothing matches that." : `Nothing is being checked yet.${isAdmin ? " Add the first check: a web address, a port, or a certificate." : ""}`}
</td>
</tr>
)}
</tbody>
</table>
</div>
<Pagination page={page.page} pageCount={page.pageCount} totalCount={page.totalCount} onPageChange={page.setPage} />
</div>
</>
);
}
@@ -81,7 +81,7 @@ const DIGEST_DAYS: { value: number; label: string }[] = [
{ value: 6, label: "Saturday" }, { value: 6, label: "Saturday" },
{ value: 0, label: "Sunday" }, { value: 0, label: "Sunday" },
]; ];
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48 }; const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0 };
type TestResult = { ok: boolean; message: string } | null; type TestResult = { ok: boolean; message: string } | null;
type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends boolean ? K : never }[keyof NotificationEvents]; type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends boolean ? K : never }[keyof NotificationEvents];
@@ -745,6 +745,15 @@ export default function NotificationSettings() {
hint="Shown on the Alerts page for every guest whose newest backup (on Proxmox or a Backup Server) is older than this, and sent once a day with the other backup messages. A server can have its own limit. 0 turns it off." hint="Shown on the Alerts page for every guest whose newest backup (on Proxmox or a Backup Server) is older than this, and sent once a day with the other backup messages. A server can have its own limit. 0 turns it off."
onChange={(v) => setHealthChecks((h) => ({ ...h, backupStaleHours: v }))} onChange={(v) => setHealthChecks((h) => ({ ...h, backupStaleHours: v }))}
/> />
<NumberField
label="A restore should have been tested within"
value={healthChecks.restoreTestDays}
min={0}
max={730}
unit="days"
hint="On the Alerts page, for every guest that has backups but no restore test recorded on the Backups page (or none that recent). A test that failed is always raised, whatever this says. 0 turns the reminder off."
onChange={(v) => setHealthChecks((h) => ({ ...h, restoreTestDays: v }))}
/>
{toggle("pbsVerificationCheck", "Proxmox Backup Server snapshot failed verification")} {toggle("pbsVerificationCheck", "Proxmox Backup Server snapshot failed verification")}
{toggle("dockerUpdateCheck", "Docker image update available")} {toggle("dockerUpdateCheck", "Docker image update available")}
</div> </div>