diff --git a/DATABASE.md b/DATABASE.md index 4e8f673..8c5994e 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -215,6 +215,7 @@ what it last reported. | `mem_total_bytes`, `mem_used_bytes` | integer | | | `disks` | text | JSON, see below. Agent-reported. | | `listening_ports` | text | JSON, see below. What the agent sees bound on the host. | +| `port_baseline` | text | JSON array like `["tcp:22","tcp:443"]`: what the host was listening on when its agent first said so. A TCP port listening later that is not in it, and not noted on the Ports card, is a "new port" alert. Null until the first report that carries ports; never changed after. | | `last_port_scan` | text | JSON summary of the latest network scan *from this app*. | | `tags` | text | JSON `string[]` of normalised tag names. | | `proxmox_integration_id` | integer → `integrations.id` | The database has no `ON DELETE` rule here; the app clears the link itself — see [below](#the-proxmox-links-foreign-key). | @@ -546,6 +547,45 @@ Resolved rows older than 400 days are deleted, with their notes. | `body` | text NOT NULL | Up to 4000 characters. | | `created_at` | text NOT NULL | ISO. | +#### `patch_plans` + +Updates on a schedule, one server at a time (services/patchPlans.ts). A plan only says when, to whom and how; each server is updated by an update run. + +| Column | Type | Notes | +|---|---|---| +| `id`, `name`, `enabled` | | | +| `days` | text NOT NULL | JSON: weekdays, 0 = Sunday … 6. | +| `start_time`, `timezone` | text NOT NULL | `HH:MM` on the clock of an IANA zone. | +| `tags`, `server_ids` | text NOT NULL | JSON arrays: the servers with any of these tags, and these servers by id. | +| `integration_id`, `project_id`, `template_id`, `template_name` | | The Semaphore playbook (no foreign key, like update runs). | +| `limit_from` | text NOT NULL, default `name` | What ansible's `--limit` is for each server: its name here, or its host name. | +| `stop_on_failure`, `only_with_updates`, `silence_dependents` | bool NOT NULL, default true | Stop at the first failure; leave out a server with nothing waiting; silence what depends on a server while it is updated. | +| `window_minutes` | integer NOT NULL, default 90 | The longest a server is under maintenance. | +| `last_run_date` | text | The local day it last started on, so it starts once a day even across a restart. | +| `created_by`, `created_at` | | | + +#### `patch_plan_runs` + +One time a plan ran, and how it went for each server. + +| Column | Type | Notes | +|---|---|---| +| `id` | integer PK | | +| `plan_id` | integer NOT NULL | → `patch_plans.id`, deleted with it. | +| `plan_name` | text NOT NULL | As it was. | +| `trigger` | text NOT NULL | `schedule` or `manual`. | +| `status` | text NOT NULL | `running`, `done`, `failed` or `cancelled`. | +| `items` | text NOT NULL | JSON: `[{serverId, name, status (pending/running/done/failed/skipped), updateRunId, message, windowIds}]`. | +| `started_by`, `message`, `started_at`, `finished_at` | | | + +#### `endpoint_daily` + +One row per endpoint check per UTC day: `total` times asked, `ok` of them answered, `ms_sum`/`ms_count` for the average answer time. Kept about two years (`endpoint_results` keeps two weeks); started from the results that existed when it was added. The monthly report's endpoint availability is worked from it. Primary key (`check_id`, `day`); deleted with its check. + +#### `renewals` + +A certificate, domain registration or secret whose expiry moved at least 20 days later when the app re-read it: `kind` (`certificate`, `domain`, `secret`), `subject` (its name as it was), `at`, `from_date`, `to_date`. Written once per renewal; read by the monthly report. + #### `update_runs` A playbook run in Semaphore for one server, wrapped in a maintenance window and followed until the server reports again. @@ -811,8 +851,9 @@ that were never changed aren't stored; the app fills in defaults when reading. | `selfBackup` | The daily backup of this database: `enabled` (true), `time` (`HH:MM`, default `03:30`, in the notification time zone) and `keep` (14 — how many daily ones are kept). | | `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. | | `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. | +| `monthlyReport` | The monthly report: `enabled` (false), `day` (1–28, default 1) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | | `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | -| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be). A server can override each of these (see `servers.thresholds`). | +| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be), `newPortAlerts` (true — warn when a server starts listening on a TCP port that is not in its baseline and not noted). A server can override each of these (see `servers.thresholds`). | | `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). | | `display` | `dateFormat`, `timeFormat`, `pageSize`. | | `providerColors`, `integrationColors` | Badge colours, by provider / integration type. | @@ -851,7 +892,7 @@ a stale value is possible and the app treats it as "nothing there": - `maintenance_windows.target_id` (→ a server, integration or DNS provider, by `target_type`) - `maintenance_aliases.target_id` (the same, by `target_type`) -- `endpoint_checks.server_id` and `heartbeats.server_id` (→ a server) and `restore_tests.subject_key` (→ a Proxmox guest, by integration and vmid) +- `endpoint_checks.server_id` and `heartbeats.server_id` (→ a server), and `patch_plans.integration_id`, `.tags` and `.server_ids` (→ a Semaphore integration, server tags, servers) and `restore_tests.subject_key` (→ a Proxmox guest, by integration and vmid) - `dependencies.from_id` and `to_id` (→ a server or integration, by their `_type`) - `audit_log.target_id` - `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider) diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index 126f5fe..3e4f765 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -161,6 +161,21 @@ schedule. A scheduled backup of the app's own database that fails sends one message through the channels ("Homelab Manager — its own backup failed", with the reason), under the same toggle as server and disk alerts (*Server offline, disk nearly full…*). The same problem is also an alert on the Alerts page of the Backups kind — "the newest backup of Homelab Manager's own database is … old" once it is more than two days old, critical after a week, and "no backup of its own yet" with the reason when the very first one fails. No alert is raised for a new install that hasn't tried yet, or when daily backups are switched off. See [BACKUP.md](BACKUP.md). +## Monthly report + +| Notification | Fires when | +|---|---| +| "Homelab monthly report — " | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. | + +Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there +is something to say. The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page. + +## Patch plans and new ports + +A **patch plan** run sends one message when it ends — "Patch plan "Lab nights" finished" or "didn't finish cleanly" — with a line for each server, under the *update run results* switch (the same one as a single update run's message). A server +that begins **listening on a new TCP port** is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when the port is noted or stops listening; for routing it counts as *Monitoring*, +carries the tags of the server, and maintenance on the server holds it back. + ## Weekly digest | Notification | Fires when | diff --git a/README.md b/README.md index 1ddd420..c2d2f8c 100644 --- a/README.md +++ b/README.md @@ -529,6 +529,17 @@ If the newest backup gets more than two days old, or a scheduled one fails, it i **A backup does not contain `CREDENTIALS_ENCRYPTION_KEY`**, which decrypts the credentials stored in it, and it is still on the same machine until you copy it away: see [BACKUP.md](BACKUP.md). +**Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days +and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore +playbook that applies updates. When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run +(below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server +(a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers +after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for +each server. A plan starts once a day at most (the day it last started is kept), up to half an hour late if the app was down at its time — later than that, that day is skipped rather than +surprising anyone mid-morning — and never overlaps itself. A run that is going is followed through a restart of this app, and can be stopped (the servers not yet started are left alone; a playbook +already started keeps running in Semaphore). Plans appear in **Coming up** as maintenance, with how many servers they update, and the runs in the change log and the monthly report. Everyone can see +plans and how they went; administrators make, change and remove them; operators run and stop them. + **Update runs** (Updates → Update…, operators). Apply a server's waiting updates in one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server under maintenance (so nothing alerts while it works and restarts), runs the playbook @@ -675,6 +686,13 @@ alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n status board. Both are described, with the exact series and message formats, in [API.md](API.md). +**Monthly report** (Operations → Monthly report; and, if switched on, a message on a day of the month). How a calendar month went, on one page that can be read, printed and copied as text: +**availability** (servers, from when each stopped reporting and reported again; endpoints, from every check's answers, kept day by day for years; integrations), **incidents** (how many began and +ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the +month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone; +the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them, +and the backups are a snapshot, not a history. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*. + **Weekly digest** (Settings → Notifications, off until switched on) sends one message a week through your notification channels: what is wrong now, what was down and for how long, what happened, disks heading for full, things about to expire, updates waiting @@ -781,6 +799,11 @@ never colour alone; the search box and dialogs are proper dialogs (keyboard trapped and restored); and the text colours were checked against the contrast minimum in both themes. Motion is turned down for those who ask the system to. +**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that +nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up +(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on +it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there. + **Ports** — each server's detail page has a Ports card for finding free ports and remembering what each one is for. "Scan…" runs a TCP scan of a port range from the app against the server's address (private addresses only, up to diff --git a/ROLES.md b/ROLES.md index 8492612..4dd925b 100644 --- a/ROLES.md +++ b/ROLES.md @@ -106,6 +106,9 @@ even further, to admin only: and removes their own browsers and can send a test to them; an administrator can remove anyone's. - **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs are going; starting an update run (**Update…**) or stopping one needs operator. +- **Patch plans** (Updates → Patch plans) can be seen by everyone, with how they went; making, changing, switching off and removing one needs administrator (it will run on its own, at night); + **Run now** and **Stop** need operator, like an update run. The **Monthly report** page is for everyone signed in; sending it now, and its schedule (Settings), are administrators'. + **Mark as expected** on a server's Ports card (which ends a new-port alert) needs operator, like any note on a port. - **Acknowledging an alert, handing it over, and taking it back** need operator (the person it is handed to must be an operator or administrator); everyone sees who has what. **Incidents** (Alerts → History) are readable by everyone signed in; writing a note needs operator, and a note can be removed by its author or an administrator. diff --git a/server/drizzle/0043_puzzling_prism.sql b/server/drizzle/0043_puzzling_prism.sql new file mode 100644 index 0000000..d44fdab --- /dev/null +++ b/server/drizzle/0043_puzzling_prism.sql @@ -0,0 +1,66 @@ +CREATE TABLE `endpoint_daily` ( + `check_id` integer NOT NULL, + `day` text NOT NULL, + `total` integer DEFAULT 0 NOT NULL, + `ok` integer DEFAULT 0 NOT NULL, + `ms_sum` integer DEFAULT 0 NOT NULL, + `ms_count` integer DEFAULT 0 NOT NULL, + PRIMARY KEY(`check_id`, `day`), + FOREIGN KEY (`check_id`) REFERENCES `endpoint_checks`(`id`) ON UPDATE no action ON DELETE cascade +); +--> statement-breakpoint +CREATE TABLE `patch_plan_runs` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `plan_id` integer NOT NULL, + `plan_name` text NOT NULL, + `trigger` text NOT NULL, + `started_by` text, + `status` text NOT NULL, + `message` text, + `items` text DEFAULT '[]' NOT NULL, + `started_at` text NOT NULL, + `finished_at` text, + FOREIGN KEY (`plan_id`) REFERENCES `patch_plans`(`id`) ON UPDATE no action ON DELETE cascade +); +--> statement-breakpoint +CREATE INDEX `patch_plan_runs_plan_idx` ON `patch_plan_runs` (`plan_id`,`started_at`);--> statement-breakpoint +CREATE INDEX `patch_plan_runs_status_idx` ON `patch_plan_runs` (`status`);--> statement-breakpoint +CREATE TABLE `patch_plans` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `name` text NOT NULL, + `enabled` integer DEFAULT true NOT NULL, + `days` text NOT NULL, + `start_time` text NOT NULL, + `timezone` text NOT NULL, + `tags` text DEFAULT '[]' NOT NULL, + `server_ids` text DEFAULT '[]' NOT NULL, + `integration_id` integer NOT NULL, + `project_id` integer NOT NULL, + `template_id` integer NOT NULL, + `template_name` text NOT NULL, + `limit_from` text DEFAULT 'name' NOT NULL, + `stop_on_failure` integer DEFAULT true NOT NULL, + `only_with_updates` integer DEFAULT true NOT NULL, + `silence_dependents` integer DEFAULT true NOT NULL, + `window_minutes` integer DEFAULT 90 NOT NULL, + `last_run_date` text, + `created_by` text, + `created_at` text DEFAULT (current_timestamp) NOT NULL +); +--> statement-breakpoint +CREATE TABLE `renewals` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `kind` text NOT NULL, + `subject` text NOT NULL, + `at` text NOT NULL, + `from_date` text NOT NULL, + `to_date` text NOT NULL +); +--> statement-breakpoint +CREATE INDEX `renewals_at_idx` ON `renewals` (`at`);--> statement-breakpoint +ALTER TABLE `servers` ADD `port_baseline` text; +--> statement-breakpoint +-- The daily rollup starts with what the endpoint results already hold (they are kept two weeks), so the first report is not blank. +INSERT INTO `endpoint_daily` (`check_id`, `day`, `total`, `ok`, `ms_sum`, `ms_count`) +SELECT `check_id`, substr(`at`, 1, 10), count(*), sum(`ok`), sum(CASE WHEN `ok` = 1 AND `ms` IS NOT NULL THEN `ms` ELSE 0 END), sum(CASE WHEN `ok` = 1 AND `ms` IS NOT NULL THEN 1 ELSE 0 END) +FROM `endpoint_results` GROUP BY `check_id`, substr(`at`, 1, 10); diff --git a/server/drizzle/meta/0043_snapshot.json b/server/drizzle/meta/0043_snapshot.json new file mode 100644 index 0000000..055122f --- /dev/null +++ b/server/drizzle/meta/0043_snapshot.json @@ -0,0 +1,4345 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "603c307c-a4f9-455b-8a9e-8a7d63c04d98", + "prevId": "f4714db3-8bbe-47c4-b5f4-29928617b04c", + "tables": { + "alert_acks": { + "name": "alert_acks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "assignee_user_id": { + "name": "assignee_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "assignee_name": { + "name": "assignee_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "acked_by": { + "name": "acked_by", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "acked_at": { + "name": "acked_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "until": { + "name": "until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "alert_acks_key_idx": { + "name": "alert_acks_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": true + } + }, + "foreignKeys": { + "alert_acks_assignee_user_id_users_id_fk": { + "name": "alert_acks_assignee_user_id_users_id_fk", + "tableFrom": "alert_acks", + "tableTo": "users", + "columnsFrom": [ + "assignee_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_incidents": { + "name": "alert_incidents", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "peak_severity": { + "name": "peak_severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "link": { + "name": "link", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_name": { + "name": "target_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "caused_by": { + "name": "caused_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "opened_at": { + "name": "opened_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resolved_at": { + "name": "resolved_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "missing_since": { + "name": "missing_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "escalated_step": { + "name": "escalated_step", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "escalated_at": { + "name": "escalated_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_incidents_key_idx": { + "name": "alert_incidents_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + }, + "alert_incidents_opened_idx": { + "name": "alert_incidents_opened_idx", + "columns": [ + "opened_at" + ], + "isUnique": false + }, + "alert_incidents_target_idx": { + "name": "alert_incidents_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_tickets": { + "name": "alert_tickets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "seen_count": { + "name": "seen_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ticket_number": { + "name": "ticket_number", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "grouped": { + "name": "grouped", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ticketed_at": { + "name": "ticketed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cleared_at": { + "name": "cleared_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_tickets_key_idx": { + "name": "alert_tickets_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + } + }, + "foreignKeys": { + "alert_tickets_integration_id_integrations_id_fk": { + "name": "alert_tickets_integration_id_integrations_id_fk", + "tableFrom": "alert_tickets", + "tableTo": "integrations", + "columnsFrom": [ + "integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assets": { + "name": "assets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'other'" + }, + "manufacturer": { + "name": "manufacturer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "serial": { + "name": "serial", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_date": { + "name": "purchase_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_price": { + "name": "purchase_price", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warranty_until": { + "name": "warranty_until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "end_of_support": { + "name": "end_of_support", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "replacement_cost": { + "name": "replacement_cost", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rack_unit": { + "name": "rack_unit", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "power_watts": { + "name": "power_watts", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_log": { + "name": "audit_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "actor_label": { + "name": "actor_label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "audit_log_actor_user_id_users_id_fk": { + "name": "audit_log_actor_user_id_users_id_fk", + "tableFrom": "audit_log", + "tableTo": "users", + "columnsFrom": [ + "actor_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "consistency_ignores": { + "name": "consistency_ignores", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "consistency_ignores_key_unique": { + "name": "consistency_ignores_key_unique", + "columns": [ + "key" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dependencies": { + "name": "dependencies", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "from_type": { + "name": "from_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "from_id": { + "name": "from_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "to_type": { + "name": "to_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "to_id": { + "name": "to_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "dependencies_edge_idx": { + "name": "dependencies_edge_idx", + "columns": [ + "from_type", + "from_id", + "to_type", + "to_id" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "diag_log": { + "name": "diag_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "operation": { + "name": "operation", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ok": { + "name": "ok", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "latency_ms": { + "name": "latency_ms", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_providers": { + "name": "dns_providers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_type": { + "name": "provider_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "dns_providers_credential_id_integration_credentials_id_fk": { + "name": "dns_providers_credential_id_integration_credentials_id_fk", + "tableFrom": "dns_providers", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_records_cache": { + "name": "dns_records_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "record_id": { + "name": "record_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ttl": { + "name": "ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "priority": { + "name": "priority", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxied": { + "name": "proxied", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "dns_records_cache_provider_id_dns_providers_id_fk": { + "name": "dns_records_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_records_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_zones_cache": { + "name": "dns_zones_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_name": { + "name": "zone_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "synced_at": { + "name": "synced_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "dns_zones_cache_provider_zone_idx": { + "name": "dns_zones_cache_provider_zone_idx", + "columns": [ + "provider_id", + "zone_id" + ], + "isUnique": true + } + }, + "foreignKeys": { + "dns_zones_cache_provider_id_dns_providers_id_fk": { + "name": "dns_zones_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_zones_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "domains": { + "name": "domains", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'manual'" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "registrar": { + "name": "registrar", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "lookup_source": { + "name": "lookup_source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_ok_at": { + "name": "last_checked_ok_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "domains_name_unique": { + "name": "domains_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "endpoint_checks": { + "name": "endpoint_checks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target": { + "name": "target", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "port": { + "name": "port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expect_status": { + "name": "expect_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "insecure": { + "name": "insecure", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "interval_minutes": { + "name": "interval_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 5 + }, + "timeout_seconds": { + "name": "timeout_seconds", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 10 + }, + "failures_before_alert": { + "name": "failures_before_alert", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 2 + }, + "warn_days": { + "name": "warn_days", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 14 + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_ok": { + "name": "last_ok", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_message": { + "name": "last_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_response_ms": { + "name": "last_response_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cert_expires_at": { + "name": "cert_expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "consecutive_failures": { + "name": "consecutive_failures", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "last_change_at": { + "name": "last_change_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "endpoint_daily": { + "name": "endpoint_daily", + "columns": { + "check_id": { + "name": "check_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "day": { + "name": "day", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "total": { + "name": "total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "ok": { + "name": "ok", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "ms_sum": { + "name": "ms_sum", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "ms_count": { + "name": "ms_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + } + }, + "indexes": {}, + "foreignKeys": { + "endpoint_daily_check_id_endpoint_checks_id_fk": { + "name": "endpoint_daily_check_id_endpoint_checks_id_fk", + "tableFrom": "endpoint_daily", + "tableTo": "endpoint_checks", + "columnsFrom": [ + "check_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "endpoint_daily_check_id_day_pk": { + "columns": [ + "check_id", + "day" + ], + "name": "endpoint_daily_check_id_day_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "endpoint_results": { + "name": "endpoint_results", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "check_id": { + "name": "check_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "at": { + "name": "at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ok": { + "name": "ok", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ms": { + "name": "ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "endpoint_results_check_idx": { + "name": "endpoint_results_check_idx", + "columns": [ + "check_id", + "at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "endpoint_results_check_id_endpoint_checks_id_fk": { + "name": "endpoint_results_check_id_endpoint_checks_id_fk", + "tableFrom": "endpoint_results", + "tableTo": "endpoint_checks", + "columnsFrom": [ + "check_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "event_webhooks": { + "name": "event_webhooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "secret": { + "name": "secret", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "events": { + "name": "events", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_attempt_at": { + "name": "last_attempt_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_result": { + "name": "last_result", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "heartbeat_pings": { + "name": "heartbeat_pings", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "heartbeat_id": { + "name": "heartbeat_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "at": { + "name": "at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "heartbeat_pings_idx": { + "name": "heartbeat_pings_idx", + "columns": [ + "heartbeat_id", + "at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "heartbeat_pings_heartbeat_id_heartbeats_id_fk": { + "name": "heartbeat_pings_heartbeat_id_heartbeats_id_fk", + "tableFrom": "heartbeat_pings", + "tableTo": "heartbeats", + "columnsFrom": [ + "heartbeat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "heartbeats": { + "name": "heartbeats", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "period_minutes": { + "name": "period_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "grace_minutes": { + "name": "grace_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 10 + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_ping_at": { + "name": "last_ping_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_status": { + "name": "last_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_message": { + "name": "last_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_duration_ms": { + "name": "last_duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_start_at": { + "name": "last_start_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ping_count": { + "name": "ping_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "heartbeats_token_unique": { + "name": "heartbeats_token_unique", + "columns": [ + "token" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "incident_notes": { + "name": "incident_notes", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "incident_id": { + "name": "incident_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "author": { + "name": "author", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "incident_notes_incident_idx": { + "name": "incident_notes_incident_idx", + "columns": [ + "incident_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "incident_notes_incident_id_alert_incidents_id_fk": { + "name": "incident_notes_incident_id_alert_incidents_id_fk", + "tableFrom": "incident_notes", + "tableTo": "alert_incidents", + "columnsFrom": [ + "incident_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integration_credentials": { + "name": "integration_credentials", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "encrypted_secret": { + "name": "encrypted_secret", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integrations": { + "name": "integrations", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "integrations_credential_id_integration_credentials_id_fk": { + "name": "integrations_credential_id_integration_credentials_id_fk", + "tableFrom": "integrations", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "ipam_entries": { + "name": "ipam_entries", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "vendor": { + "name": "vendor", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "ipam_entries_ip_address_unique": { + "name": "ipam_entries_ip_address_unique", + "columns": [ + "ip_address" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_aliases": { + "name": "maintenance_aliases", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "address": { + "name": "address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "maintenance_aliases_unique_idx": { + "name": "maintenance_aliases_unique_idx", + "columns": [ + "address", + "target_type", + "target_id" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_schedules": { + "name": "maintenance_schedules", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "targets": { + "name": "targets", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "days": { + "name": "days", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "start_time": { + "name": "start_time", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_minutes": { + "name": "duration_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timezone": { + "name": "timezone", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_windows": { + "name": "maintenance_windows", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ends_at": { + "name": "ends_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "metrics_tokens": { + "name": "metrics_tokens", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "metrics_tokens_token_hash_unique": { + "name": "metrics_tokens_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "notification_queue": { + "name": "notification_queue", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "audience": { + "name": "audience", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "patch_plan_runs": { + "name": "patch_plan_runs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "plan_id": { + "name": "plan_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "plan_name": { + "name": "plan_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "trigger": { + "name": "trigger", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "started_by": { + "name": "started_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "items": { + "name": "items", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "finished_at": { + "name": "finished_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "patch_plan_runs_plan_idx": { + "name": "patch_plan_runs_plan_idx", + "columns": [ + "plan_id", + "started_at" + ], + "isUnique": false + }, + "patch_plan_runs_status_idx": { + "name": "patch_plan_runs_status_idx", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "patch_plan_runs_plan_id_patch_plans_id_fk": { + "name": "patch_plan_runs_plan_id_patch_plans_id_fk", + "tableFrom": "patch_plan_runs", + "tableTo": "patch_plans", + "columnsFrom": [ + "plan_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "patch_plans": { + "name": "patch_plans", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "days": { + "name": "days", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "start_time": { + "name": "start_time", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timezone": { + "name": "timezone", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "server_ids": { + "name": "server_ids", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_id": { + "name": "template_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_name": { + "name": "template_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "limit_from": { + "name": "limit_from", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'name'" + }, + "stop_on_failure": { + "name": "stop_on_failure", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "only_with_updates": { + "name": "only_with_updates", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "silence_dependents": { + "name": "silence_dependents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "window_minutes": { + "name": "window_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 90 + }, + "last_run_date": { + "name": "last_run_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "port_forwards": { + "name": "port_forwards", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "external_port": { + "name": "external_port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination": { + "name": "destination", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "internal_port": { + "name": "internal_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "port_forwards_server_id_servers_id_fk": { + "name": "port_forwards_server_id_servers_id_fk", + "tableFrom": "port_forwards", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "push_subscriptions": { + "name": "push_subscriptions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "endpoint": { + "name": "endpoint", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "p256dh": { + "name": "p256dh", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "auth": { + "name": "auth", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_success_at": { + "name": "last_success_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "min_severity": { + "name": "min_severity", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "categories": { + "name": "categories", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "push_subscriptions_endpoint_unique": { + "name": "push_subscriptions_endpoint_unique", + "columns": [ + "endpoint" + ], + "isUnique": true + } + }, + "foreignKeys": { + "push_subscriptions_user_id_users_id_fk": { + "name": "push_subscriptions_user_id_users_id_fk", + "tableFrom": "push_subscriptions", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "renewals": { + "name": "renewals", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "subject": { + "name": "subject", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "at": { + "name": "at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "from_date": { + "name": "from_date", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "to_date": { + "name": "to_date", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "renewals_at_idx": { + "name": "renewals_at_idx", + "columns": [ + "at" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "restore_tests": { + "name": "restore_tests", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "subject_key": { + "name": "subject_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "tested_at": { + "name": "tested_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "result": { + "name": "result", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tested_by": { + "name": "tested_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "restore_tests_subject_idx": { + "name": "restore_tests_subject_idx", + "columns": [ + "subject_key", + "tested_at" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "runbooks": { + "name": "runbooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "runbooks_target_idx": { + "name": "runbooks_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "scheduled_tasks": { + "name": "scheduled_tasks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "schedule_type": { + "name": "schedule_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'agent'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "command": { + "name": "command", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "schedule_expression": { + "name": "schedule_expression", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "next_run_at": { + "name": "next_run_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "raw_metadata": { + "name": "raw_metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_stale": { + "name": "is_stale", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "scheduled_tasks_server_id_servers_id_fk": { + "name": "scheduled_tasks_server_id_servers_id_fk", + "tableFrom": "scheduled_tasks", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "secrets": { + "name": "secrets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'generic'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expiry_date": { + "name": "expiry_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warn_days": { + "name": "warn_days", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 30 + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_host": { + "name": "check_host", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_port": { + "name": "check_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_links": { + "name": "server_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "server_links_server_id_servers_id_fk": { + "name": "server_links_server_id_servers_id_fk", + "tableFrom": "server_links", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_metrics": { + "name": "server_metrics", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metric": { + "name": "metric", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "real", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "recorded_at": { + "name": "recorded_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "server_metrics_lookup_idx": { + "name": "server_metrics_lookup_idx", + "columns": [ + "server_id", + "metric", + "recorded_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "server_metrics_server_id_servers_id_fk": { + "name": "server_metrics_server_id_servers_id_fk", + "tableFrom": "server_metrics", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_ports": { + "name": "server_ports", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "port": { + "name": "port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "open": { + "name": "open", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "last_seen_open_at": { + "name": "last_seen_open_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "server_ports_unique": { + "name": "server_ports_unique", + "columns": [ + "server_id", + "port", + "protocol" + ], + "isUnique": true + } + }, + "foreignKeys": { + "server_ports_server_id_servers_id_fk": { + "name": "server_ports_server_id_servers_id_fk", + "tableFrom": "server_ports", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "servers": { + "name": "servers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "hostname": { + "name": "hostname", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "os_type": { + "name": "os_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'linux'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "api_token_hash": { + "name": "api_token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "api_token_prefix": { + "name": "api_token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ip_addresses": { + "name": "ip_addresses", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_model": { + "name": "cpu_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_cores": { + "name": "cpu_cores", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_load_percent": { + "name": "cpu_load_percent", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_total_bytes": { + "name": "mem_total_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_used_bytes": { + "name": "mem_used_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disks": { + "name": "disks", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "listening_ports": { + "name": "listening_ports", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_port_scan": { + "name": "last_port_scan", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_integration_id": { + "name": "proxmox_integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_node": { + "name": "proxmox_node", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_guest_type": { + "name": "proxmox_guest_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_vmid": { + "name": "proxmox_vmid", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hide_proxmox_link": { + "name": "hide_proxmox_link", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "parent_server_id": { + "name": "parent_server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "no_parent": { + "name": "no_parent", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "pending_updates": { + "name": "pending_updates", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_pending_since": { + "name": "updates_pending_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_health": { + "name": "host_health", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_diagnostics": { + "name": "agent_diagnostics", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "port_baseline": { + "name": "port_baseline", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_manufacturer": { + "name": "hw_manufacturer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_model": { + "name": "hw_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_serial": { + "name": "hw_serial", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "thresholds": { + "name": "thresholds", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_version": { + "name": "agent_version", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "servers_proxmox_integration_id_integrations_id_fk": { + "name": "servers_proxmox_integration_id_integrations_id_fk", + "tableFrom": "servers", + "tableTo": "integrations", + "columnsFrom": [ + "proxmox_integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "settings": { + "name": "settings", + "columns": { + "key": { + "name": "key", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "share_links": { + "name": "share_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'summary'" + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'wall'" + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "share_links_token_hash_unique": { + "name": "share_links_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "status_items": { + "name": "status_items", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "link_id": { + "name": "link_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "position": { + "name": "position", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "monitor_id": { + "name": "monitor_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "status_items_link_idx": { + "name": "status_items_link_idx", + "columns": [ + "link_id", + "position" + ], + "isUnique": false + } + }, + "foreignKeys": { + "status_items_link_id_share_links_id_fk": { + "name": "status_items_link_id_share_links_id_fk", + "tableFrom": "status_items", + "tableTo": "share_links", + "columnsFrom": [ + "link_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "tag_definitions": { + "name": "tag_definitions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "tag_definitions_name_unique": { + "name": "tag_definitions_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "update_runs": { + "name": "update_runs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_id": { + "name": "template_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_name": { + "name": "template_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "host_limit": { + "name": "host_limit", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_id": { + "name": "task_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "task_status": { + "name": "task_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_id": { + "name": "window_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_created": { + "name": "window_created", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_by": { + "name": "started_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_ended_at": { + "name": "task_ended_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_reported_at": { + "name": "host_reported_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finished_at": { + "name": "finished_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_before": { + "name": "updates_before", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_after": { + "name": "updates_after", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reboot_required": { + "name": "reboot_required", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "poll_errors": { + "name": "poll_errors", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + } + }, + "indexes": { + "update_runs_server_idx": { + "name": "update_runs_server_idx", + "columns": [ + "server_id", + "started_at" + ], + "isUnique": false + }, + "update_runs_status_idx": { + "name": "update_runs_status_idx", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "update_runs_server_id_servers_id_fk": { + "name": "update_runs_server_id_servers_id_fk", + "tableFrom": "update_runs", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "users": { + "name": "users", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "oidc_sub": { + "name": "oidc_sub", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'viewer'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_login_at": { + "name": "last_login_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "favorites": { + "name": "favorites", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_looked_at": { + "name": "last_looked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "groups": { + "name": "groups", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dashboard_layout": { + "name": "dashboard_layout", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "picture_url": { + "name": "picture_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "users_oidc_sub_unique": { + "name": "users_oidc_sub_unique", + "columns": [ + "oidc_sub" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/server/drizzle/meta/_journal.json b/server/drizzle/meta/_journal.json index 06b765a..96dc6af 100644 --- a/server/drizzle/meta/_journal.json +++ b/server/drizzle/meta/_journal.json @@ -302,6 +302,13 @@ "when": 1792361094159, "tag": "0042_silly_omega_red", "breakpoints": true + }, + { + "idx": 43, + "version": "6", + "when": 1792443011097, + "tag": "0043_puzzling_prism", + "breakpoints": true } ] } \ No newline at end of file diff --git a/server/src/app.ts b/server/src/app.ts index a368313..4f3e051 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -50,6 +50,8 @@ import { alertTicketsRouter } from "./routes/alertTickets.js"; import { avatarsRouter } from "./routes/avatars.js"; import { heartbeatsRouter } from "./routes/heartbeats.js"; import { agendaRouter } from "./routes/agenda.js"; +import { patchPlansRouter } from "./routes/patchPlans.js"; +import { reportsRouter } from "./routes/reports.js"; import { publicCalendarRouter } from "./routes/publicCalendar.js"; import { heartbeatPingRouter } from "./routes/heartbeatPing.js"; import { apiErrorHandler } from "./utils/apiErrorHandler.js"; @@ -113,6 +115,8 @@ export function createApp() { app.use("/api/avatars", avatarsRouter); app.use("/api/heartbeats", heartbeatsRouter); app.use("/api/agenda", agendaRouter); + app.use("/api/patch-plans", patchPlansRouter); + app.use("/api/reports", reportsRouter); app.use("/api/uptime", uptimeRouter); app.use("/api/share-links", shareLinksRouter); app.use("/api/runbooks", runbooksRouter); diff --git a/server/src/db/schema.ts b/server/src/db/schema.ts index 77c3f69..66f13b9 100644 --- a/server/src/db/schema.ts +++ b/server/src/db/schema.ts @@ -1,5 +1,5 @@ import { sql } from "drizzle-orm"; -import { sqliteTable, text, integer, real, uniqueIndex, index } from "drizzle-orm/sqlite-core"; +import { sqliteTable, text, integer, real, uniqueIndex, index, primaryKey } from "drizzle-orm/sqlite-core"; // ─── Users & roles ────────────────────────────────────────────────────────── @@ -447,6 +447,10 @@ export const servers = sqliteTable("servers", { // What the agent says it could not collect, as JSON: {items: [{check, status, reason?, detail?}], reportedAt} — see services/agentDiagnostics.ts. Null until an agent that can say so has. agentDiagnostics: text("agent_diagnostics"), + // The ports this server was listening on when its agent first said so, as a JSON array like ["tcp:22","tcp:443"]. A port that is listening later and is + // not in this list, and not noted on the Ports page, is a "new port" (services/newPorts.ts). Null until the first report that carries ports. + portBaseline: text("port_baseline"), + // What the machine is, as its firmware says (agents report it): maker, model and serial number. Used to fill in the inventory. hwManufacturer: text("hw_manufacturer"), hwModel: text("hw_model"), @@ -846,6 +850,89 @@ export const updateRuns = sqliteTable( (t) => [index("update_runs_server_idx").on(t.serverId, t.startedAt), index("update_runs_status_idx").on(t.status)], ); +// ─── Patch plans: updates, on a schedule, one server at a time ───────────── + +// "Sundays at 02:00, the servers tagged lab, one at a time, stop at the first failure." A plan only says when, to whom and how; each server is updated by an +// ordinary update run (services/updateRuns.ts), and the plan waits for it to finish before the next — see services/patchPlans.ts. +export const patchPlans = sqliteTable("patch_plans", { + id: integer("id").primaryKey({ autoIncrement: true }), + name: text("name").notNull(), + enabled: integer("enabled", { mode: "boolean" }).notNull().default(true), + days: text("days").notNull(), // JSON: weekdays, 0 = Sunday … 6 = Saturday + startTime: text("start_time").notNull(), // "HH:MM" on the clock of `timezone` + timezone: text("timezone").notNull(), + tags: text("tags").notNull().default("[]"), // JSON string[]: servers that have any of these tags + serverIds: text("server_ids").notNull().default("[]"), // JSON number[]: and these servers by name, whatever their tags + integrationId: integer("integration_id").notNull(), // the Semaphore integration; no FK, like update runs + projectId: integer("project_id").notNull(), + templateId: integer("template_id").notNull(), + templateName: text("template_name").notNull(), + limitFrom: text("limit_from").notNull().default("name"), // what ansible's --limit is for a server: its name here, or its host name + stopOnFailure: integer("stop_on_failure", { mode: "boolean" }).notNull().default(true), + onlyWithUpdates: integer("only_with_updates", { mode: "boolean" }).notNull().default(true), // leave out a server that has nothing waiting + silenceDependents: integer("silence_dependents", { mode: "boolean" }).notNull().default(true), // what depends on a server is silenced while it is updated + windowMinutes: integer("window_minutes").notNull().default(90), // at most this long under maintenance per server + lastRunDate: text("last_run_date"), // the local day (YYYY-MM-DD) it last started on — so it starts once a day, even if the app is restarted + createdBy: text("created_by"), + createdAt: text("created_at") + .notNull() + .default(sql`(current_timestamp)`), +}); + +// One time a plan ran, and how it went for each server: items is JSON [{serverId, name, status, updateRunId, message, windowIds}]. +export const patchPlanRuns = sqliteTable( + "patch_plan_runs", + { + id: integer("id").primaryKey({ autoIncrement: true }), + planId: integer("plan_id") + .notNull() + .references(() => patchPlans.id, { onDelete: "cascade" }), + planName: text("plan_name").notNull(), // as it was, so a renamed plan's history still reads right + trigger: text("trigger").notNull(), // schedule | manual + startedBy: text("started_by"), + status: text("status").notNull(), // running | done | failed | cancelled + message: text("message"), + items: text("items").notNull().default("[]"), + startedAt: text("started_at").notNull(), // ISO + finishedAt: text("finished_at"), // ISO + }, + (t) => [index("patch_plan_runs_plan_idx").on(t.planId, t.startedAt), index("patch_plan_runs_status_idx").on(t.status)], +); + +// ─── The monthly report's long memory ─────────────────────────────────────── + +// One row per endpoint check per day: how many times it was asked, how many answered, and the answer times — kept for years, where the individual +// results are kept for two weeks. What the monthly report's availability is worked from (services/endpointChecks.ts). +export const endpointDaily = sqliteTable( + "endpoint_daily", + { + checkId: integer("check_id") + .notNull() + .references(() => endpointChecks.id, { onDelete: "cascade" }), + day: text("day").notNull(), // YYYY-MM-DD, UTC + total: integer("total").notNull().default(0), + ok: integer("ok").notNull().default(0), + msSum: integer("ms_sum").notNull().default(0), // of the answers that were good and timed + msCount: integer("ms_count").notNull().default(0), + }, + (t) => [primaryKey({ columns: [t.checkId, t.day] })], +); + +// A certificate, domain registration or secret whose expiry moved later: it was renewed. Written when the app sees it happen, so it only goes back +// as far as the version that began watching (services/renewals.ts). +export const renewals = sqliteTable( + "renewals", + { + id: integer("id").primaryKey({ autoIncrement: true }), + kind: text("kind").notNull(), // certificate | domain | secret + subject: text("subject").notNull(), // its name, as it was + at: text("at").notNull(), // ISO — when it was seen + fromDate: text("from_date").notNull(), // YYYY-MM-DD — the expiry before + toDate: text("to_date").notNull(), // YYYY-MM-DD — and after + }, + (t) => [index("renewals_at_idx").on(t.at)], +); + // ─── Inventory: the hardware behind it all ────────────────────────────────── export const assetKinds = ["server", "storage", "network", "power", "client", "other"] as const; diff --git a/server/src/index.ts b/server/src/index.ts index 0fdb651..ab48205 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -13,6 +13,7 @@ import { initProxmoxBackupScheduler } from "./services/proxmoxBackupScheduler.js import { initPbsVerificationScheduler } from "./services/pbsVerificationScheduler.js"; import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js"; import { initWeeklyDigestScheduler } from "./services/weeklyDigestScheduler.js"; +import { initMonthlyReportScheduler } from "./services/monthlyReportScheduler.js"; import { initMaintenanceEvents } from "./services/maintenanceEvents.js"; import { initHealthScheduler } from "./services/healthScheduler.js"; import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js"; @@ -21,6 +22,7 @@ import { initMaintenanceScheduleScheduler } from "./services/maintenanceSchedule import { initUpdateRuns } from "./services/updateRuns.js"; import { initEscalation } from "./services/escalation.js"; import { initEndpointMonitor } from "./services/endpointMonitor.js"; +import { initPatchPlans } from "./services/patchPlans.js"; import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js"; import { initIncidentRecorder } from "./services/incidents.js"; import { getAlerts } from "./services/alerts.js"; @@ -51,6 +53,7 @@ await initProxmoxBackupScheduler(); await initPbsVerificationScheduler(); await initQuietHoursScheduler(); await initWeeklyDigestScheduler(); +await initMonthlyReportScheduler(); initMaintenanceEvents(); await initHealthScheduler(); await initAlertTicketScheduler(); @@ -61,6 +64,7 @@ await initSelfBackupScheduler(); initIncidentRecorder(() => getAlerts(false)); initEscalation(); initEndpointMonitor(); +initPatchPlans(); createApp().listen(env.port, () => { console.log(`homelab-manager listening on port ${env.port}`); diff --git a/server/src/routes/patchPlans.ts b/server/src/routes/patchPlans.ts new file mode 100644 index 0000000..bf6d41e --- /dev/null +++ b/server/src/routes/patchPlans.ts @@ -0,0 +1,196 @@ +import { Router } from "express"; +import { eq } from "drizzle-orm"; +import { z } from "zod"; +import { db } from "../db/client.js"; +import { patchPlans, servers } from "../db/schema.js"; +import { requireAuth, requireRole } from "../auth/middleware.js"; +import { recordAudit } from "../services/audit.js"; +import { isValidTimeZone } from "../services/maintenanceSchedules.js"; +import { activeRunOf, cancelPlanRun, listPlanRuns, MAX_PLANS, parseItems, planDays, planOccurrences, planServerIds, planServers, planTags, PlanError, startPlanRun, type PlanRow, type PlanRunRow } from "../services/patchPlans.js"; +import { InvalidTagError, normalizeTags, parseTags } from "../services/serverTags.js"; +import { asyncHandler } from "../utils/asyncHandler.js"; + +export const patchPlansRouter = Router(); + +// Everyone signed in can see what is planned and how it went, like the Updates page. Making a plan is an administrator's (it will run on its own, at night); +// starting one now or stopping one that is running is an operator's, like an update run. +patchPlansRouter.use(requireAuth); + +const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN); + +const planSchema = z + .object({ + name: z.string().trim().min(1).max(100), + enabled: z.boolean().default(true), + days: z.array(z.number().int().min(0).max(6)).min(1).max(7), + startTime: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/, "Use a time like 02:00"), + timezone: z.string().trim().min(1).max(60), + tags: z.array(z.string()).max(12).default([]), + serverIds: z.array(z.number().int().positive()).max(200).default([]), + integrationId: z.number().int().positive(), + projectId: z.number().int().positive(), + templateId: z.number().int().positive(), + templateName: z.string().trim().min(1).max(200), + limitFrom: z.enum(["name", "hostname"]).default("name"), + stopOnFailure: z.boolean().default(true), + onlyWithUpdates: z.boolean().default(true), + silenceDependents: z.boolean().default(true), + windowMinutes: z.number().int().min(15).max(480).default(90), + }) + .refine((d) => isValidTimeZone(d.timezone), { message: "That isn't a time zone the system knows (use a name like Europe/Stockholm).", path: ["timezone"] }) + .refine((d) => d.tags.length > 0 || d.serverIds.length > 0, { message: "Choose which servers: give a tag, or name at least one server.", path: ["tags"] }); + +function valuesOf(d: z.infer) { + return { + name: d.name, + enabled: d.enabled, + days: JSON.stringify([...new Set(d.days)].sort()), + startTime: d.startTime, + timezone: d.timezone, + tags: JSON.stringify(normalizeTags(d.tags)), + serverIds: JSON.stringify([...new Set(d.serverIds)]), + integrationId: d.integrationId, + projectId: d.projectId, + templateId: d.templateId, + templateName: d.templateName, + limitFrom: d.limitFrom, + stopOnFailure: d.stopOnFailure, + onlyWithUpdates: d.onlyWithUpdates, + silenceDependents: d.silenceDependents, + windowMinutes: d.windowMinutes, + }; +} + +const shapeRun = (r: PlanRunRow) => ({ + id: r.id, + planId: r.planId, + planName: r.planName, + trigger: r.trigger, + startedBy: r.startedBy, + status: r.status, + message: r.message, + startedAt: r.startedAt, + finishedAt: r.finishedAt, + items: parseItems(r.items).map(({ windowIds: _w, ...i }) => i), +}); + +async function shapePlan(p: PlanRow, all: { id: number; name: string; tags: string | null }[], now: Date) { + const covered = planServers(p, all); + const next = planOccurrences(p, now, new Date(now.getTime() + 15 * 86_400_000)).filter((o) => o.start.getTime() > now.getTime()).slice(0, 3); + const active = await activeRunOf(p.id); + return { + id: p.id, + name: p.name, + enabled: p.enabled, + days: planDays(p), + startTime: p.startTime, + timezone: p.timezone, + tags: planTags(p), + serverIds: planServerIds(p), + integrationId: p.integrationId, + projectId: p.projectId, + templateId: p.templateId, + templateName: p.templateName, + limitFrom: p.limitFrom, + stopOnFailure: p.stopOnFailure, + onlyWithUpdates: p.onlyWithUpdates, + silenceDependents: p.silenceDependents, + windowMinutes: p.windowMinutes, + createdBy: p.createdBy, + servers: covered, + next: next.map((o) => o.start.toISOString()), + running: active !== null, + }; +} + +patchPlansRouter.get("/", asyncHandler(async (_req, res) => { + const now = new Date(); + const rows = await db.select().from(patchPlans).orderBy(patchPlans.name); + const all = await db.select({ id: servers.id, name: servers.name, tags: servers.tags }).from(servers); + const tags = [...new Set(all.flatMap((s) => parseTags(s.tags)))].sort((a, b) => a.localeCompare(b, "sv")); + res.json({ + plans: await Promise.all(rows.map((p) => shapePlan(p, all, now))), + runs: (await listPlanRuns({ limit: 15 })).map(shapeRun), + servers: all.map((s) => ({ id: s.id, name: s.name, tags: parseTags(s.tags) })).sort((a, b) => a.name.localeCompare(b.name)), + tags, + }); +})); + +/** The message for a rejected body, as a sentence. */ +const reason = (parsed: { error: z.ZodError }) => parsed.error.issues[0]?.message ?? "That isn't a plan."; + +patchPlansRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => { + const parsed = planSchema.safeParse(req.body); + if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: reason(parsed), details: parsed.error.flatten() }); + if ((await db.select({ id: patchPlans.id }).from(patchPlans)).length >= MAX_PLANS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_PLANS} plans.` }); + let values; + try { + values = valuesOf(parsed.data); + } catch (err) { + if (err instanceof InvalidTagError) return res.status(400).json({ error: "invalid_tag", message: err.message }); + throw err; + } + const actor = req.currentUser!; + const [row] = await db.insert(patchPlans).values({ ...values, createdBy: actor.name ?? actor.email ?? actor.oidcSub }).returning(); + await recordAudit({ actor, category: "patch_plan", action: "create", targetType: "patch_plan", targetId: row.id, detail: { name: row.name, days: planDays(row), time: row.startTime } }); + res.status(201).json({ plan: await shapePlan(row, await db.select({ id: servers.id, name: servers.name, tags: servers.tags }).from(servers), new Date()) }); +})); + +patchPlansRouter.put("/:id", requireRole("admin"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + const parsed = planSchema.safeParse(req.body); + if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.success ? "That isn't a plan." : reason(parsed) }); + const [before] = await db.select().from(patchPlans).where(eq(patchPlans.id, id)).limit(1); + if (!before) return res.status(404).json({ error: "not_found" }); + let values; + try { + values = valuesOf(parsed.data); + } catch (err) { + if (err instanceof InvalidTagError) return res.status(400).json({ error: "invalid_tag", message: err.message }); + throw err; + } + // A changed day or time is a new schedule: what it last started on no longer says whether today's has happened. + const rescheduled = values.days !== before.days || values.startTime !== before.startTime || values.timezone !== before.timezone; + const [row] = await db.update(patchPlans).set({ ...values, ...(rescheduled ? { lastRunDate: null } : {}) }).where(eq(patchPlans.id, id)).returning(); + await recordAudit({ actor: req.currentUser!, category: "patch_plan", action: "update", targetType: "patch_plan", targetId: id, detail: { name: row.name, days: planDays(row), time: row.startTime } }); + res.json({ plan: await shapePlan(row, await db.select({ id: servers.id, name: servers.name, tags: servers.tags }).from(servers), new Date()) }); +})); + +patchPlansRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" }); + const [before] = await db.select().from(patchPlans).where(eq(patchPlans.id, id)).limit(1); + if (!before) return res.status(404).json({ error: "not_found" }); + if (await activeRunOf(id)) return res.status(409).json({ error: "running", message: "It is running now. Stop the run first." }); + await db.delete(patchPlans).where(eq(patchPlans.id, id)); + await recordAudit({ actor: req.currentUser!, category: "patch_plan", action: "delete", targetType: "patch_plan", targetId: id, detail: { name: before.name } }); + res.status(204).end(); +})); + +// Run it now, instead of at its time (it still starts again at its time). +patchPlansRouter.post("/:id/run", requireRole("operator"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" }); + const actor = req.currentUser!; + try { + const run = await startPlanRun(id, { trigger: "manual", actor: actor.name ?? actor.email ?? actor.oidcSub }); + res.status(201).json({ run: shapeRun(run) }); + } catch (err) { + if (err instanceof PlanError) return res.status(err.status).json({ error: err.code, message: err.message }); + throw err; + } +})); + +patchPlansRouter.post("/runs/:id/cancel", requireRole("operator"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" }); + const actor = req.currentUser!; + try { + const run = await cancelPlanRun(id, actor.name ?? actor.email ?? actor.oidcSub); + if (!run) return res.status(404).json({ error: "not_found" }); + res.json({ run: shapeRun(run) }); + } catch (err) { + if (err instanceof PlanError) return res.status(err.status).json({ error: err.code, message: err.message }); + throw err; + } +})); diff --git a/server/src/routes/reports.ts b/server/src/routes/reports.ts new file mode 100644 index 0000000..2228143 --- /dev/null +++ b/server/src/routes/reports.ts @@ -0,0 +1,40 @@ +import { Router } from "express"; +import { requireAuth, requireRole } from "../auth/middleware.js"; +import { buildMonthlyReport, isMonthId, lastFullMonth, monthlyReportText, recentMonths } from "../services/monthlyReport.js"; +import { recordAudit } from "../services/audit.js"; +import { notifyNow } from "../services/notify.js"; +import { getSettings } from "../services/settingsStore.js"; +import { asyncHandler } from "../utils/asyncHandler.js"; + +export const reportsRouter = Router(); + +reportsRouter.use(requireAuth); + +/** The month asked for, or the last whole one; null when what was asked is not a month. */ +async function monthFrom(query: unknown): Promise<{ id: string; months: { id: string; label: string }[] } | null> { + const tz = (await getSettings()).notifications.timezone || "UTC"; + const months = recentMonths(new Date(), tz); + if (query === undefined || query === "") return { id: lastFullMonth(new Date(), tz), months }; + if (!isMonthId(query) || !months.some((m) => m.id === query)) return null; + return { id: query, months }; +} + +// How a month went, for everyone signed in: it is only what the pages it comes from already show them. ?month=2026-09, the last whole month by default; +// ?format=text is the plain-text version a notification carries. +reportsRouter.get("/monthly", asyncHandler(async (req, res) => { + const picked = await monthFrom(req.query.month); + if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months, like 2026-09." }); + const report = await buildMonthlyReport(picked.id); + if (req.query.format === "text") return res.json(monthlyReportText(report)); + res.json({ report, months: picked.months }); +})); + +// Send it now to the notification channels, as if it were the day. Administrators. +reportsRouter.post("/monthly/send", requireRole("admin"), asyncHandler(async (req, res) => { + const picked = await monthFrom((req.body as { month?: unknown } | undefined)?.month); + if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months, like 2026-09." }); + const { title, message } = monthlyReportText(await buildMonthlyReport(picked.id)); + await notifyNow(title, message); + await recordAudit({ actor: req.currentUser!, category: "report", action: "send_monthly", targetType: "report", detail: { month: picked.id } }); + res.json({ sent: true }); +})); diff --git a/server/src/routes/serverPorts.ts b/server/src/routes/serverPorts.ts index 10e6864..e702285 100644 --- a/server/src/routes/serverPorts.ts +++ b/server/src/routes/serverPorts.ts @@ -7,6 +7,8 @@ import { requireRole } from "../auth/middleware.js"; import { recordAudit } from "../services/audit.js"; import { beginScan, endScan, MAX_SCAN_SPAN, resolveScanTarget, scanPorts, toRanges } from "../services/portScan.js"; import { type AgentPort, groupAgentPorts, parseJson } from "../services/agentPorts.js"; +import { newListeningPorts, parseBaseline, portKey } from "../services/newPorts.js"; +import { getSettings } from "../services/settingsStore.js"; import { asyncHandler } from "../utils/asyncHandler.js"; // Mounted under /api/servers/:id/ports by the servers router, which has already required a signed-in user. @@ -37,6 +39,8 @@ export interface PortEntry { agent: { addresses: string[]; process: string | null; localOnly: boolean } | null; /** "open" if anything is using it; "reserved" if it only has a note. */ state: "open" | "reserved"; + /** Began listening after the first report and nobody has noted it: a "new port" alert is on until it is noted or stops listening. */ + isNew: boolean; } async function buildPortList(serverId: number) { @@ -46,6 +50,11 @@ async function buildPortList(serverId: number) { const agentRaw = parseJson(server.listeningPorts, null); const agent = groupAgentPorts(agentRaw ?? []); + // What is new: listening, reachable from outside, not in the baseline and not noted — the same rule the alert uses. + const expected = new Set(rows.filter((r) => r.label || r.comment).map((r) => portKey(r.protocol, r.port))); + const alerting = (await getSettings()).healthChecks.newPortAlerts; + const fresh = new Set(alerting ? newListeningPorts(agentRaw, parseBaseline(server.portBaseline), expected).map((p) => portKey(p.protocol, p.port)) : []); + const entries = new Map(); for (const row of rows) { const key = `${row.protocol}:${row.port}`; @@ -59,6 +68,7 @@ async function buildPortList(serverId: number) { lastSeenOpenAt: row.lastSeenOpenAt, agent: agent.get(key) ?? null, state: "reserved", + isNew: fresh.has(key), }); } for (const [key, info] of agent) { @@ -74,6 +84,7 @@ async function buildPortList(serverId: number) { lastSeenOpenAt: null, agent: info, state: "reserved", + isNew: fresh.has(key), }); } for (const entry of entries.values()) { diff --git a/server/src/routes/settings.ts b/server/src/routes/settings.ts index 86f0342..cb716b7 100644 --- a/server/src/routes/settings.ts +++ b/server/src/routes/settings.ts @@ -13,6 +13,7 @@ import { scheduleLogRetentionPurge } from "../services/logRetentionScheduler.js" import { purgeOldLogs } from "../services/logRetention.js"; import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js"; import { scheduleWeeklyDigest } from "../services/weeklyDigestScheduler.js"; +import { scheduleMonthlyReport } from "../services/monthlyReportScheduler.js"; import { buildDigest, sendDigest } from "../services/weeklyDigest.js"; import { enabledChannelCount } from "../services/notify.js"; import { CATEGORIES, CHANNEL_IDS } from "../services/notifyRouting.js"; @@ -143,7 +144,7 @@ const updateSchema = z.object({ .partial() .optional(), healthChecks: z - .object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730) }) + .object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730), newPortAlerts: z.boolean() }) .partial() .optional(), quietHours: z @@ -165,6 +166,10 @@ const updateSchema = z.object({ .object({ enabled: z.boolean(), day: z.number().int().min(0).max(6), time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/) }) .partial() .optional(), + monthlyReport: z + .object({ enabled: z.boolean(), day: z.number().int().min(1).max(28), time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/) }) + .partial() + .optional(), }); settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => { @@ -200,6 +205,9 @@ settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => { if (parsed.data.digest || parsed.data.notifications) { await scheduleWeeklyDigest(); } + if (parsed.data.monthlyReport || parsed.data.notifications) { + await scheduleMonthlyReport(); + } await recordAudit({ actor: req.currentUser!, diff --git a/server/src/services/agenda.ts b/server/src/services/agenda.ts index 6fc984b..3daed9c 100644 --- a/server/src/services/agenda.ts +++ b/server/src/services/agenda.ts @@ -9,10 +9,11 @@ import { createHash } from "node:crypto"; import { and, eq } from "drizzle-orm"; import { db } from "../db/client.js"; -import { assets, domains, endpointChecks, integrations, maintenanceSchedules, secrets, servers } from "../db/schema.js"; +import { assets, domains, endpointChecks, integrations, maintenanceSchedules, patchPlans, secrets, servers } from "../db/schema.js"; import { collectTailscaleKeyExpiries } from "./tailscaleKeyExpiryScheduler.js"; import { describeTarget, listUpcomingWindows } from "./maintenance.js"; import { occurrencesBetween, targetsOf } from "./maintenanceSchedules.js"; +import { planDays, planOccurrences, planServers } from "./patchPlans.js"; import { parsePendingUpdates } from "./pendingUpdates.js"; import { latestTests } from "./restoreTests.js"; import { getSettings } from "./settingsStore.js"; @@ -88,6 +89,8 @@ export interface AgendaData { /** Windows that have been set up but not begun, one per target. */ windows: { startedAt: string; endsAt: string; reason: string | null; source: string | null; targetName: string }[]; schedules: { id: number; name: string; reason: string | null; days: string; startTime: string; durationMinutes: number; timezone: string; targetNames: string[] }[]; + /** Patch plans that are switched on, with how many servers each covers now. */ + patchPlans: { id: number; name: string; days: string; startTime: string; timezone: string; servers: number }[]; /** The newest restore test of each subject. */ restoreTests: { subjectKey: string; label: string; testedAt: string; result: string }[]; restoreLimitDays: number; @@ -140,6 +143,10 @@ export async function loadAgendaData(now: Date = new Date()): Promise p.enabled); + const allServers = planRows.length > 0 ? await db.select({ id: servers.id, name: servers.name, tags: servers.tags }).from(servers) : []; + const plans: AgendaData["patchPlans"] = planRows.map((p) => ({ id: p.id, name: p.name, days: p.days, startTime: p.startTime, timezone: p.timezone, servers: planServers(p, allServers).length })); + const updates: AgendaData["updates"] = []; for (const s of serverRows) { const u = parsePendingUpdates(s.pendingUpdates); @@ -163,6 +170,7 @@ export async function loadAgendaData(now: Date = new Date()): Promise ({ subjectKey: t.subjectKey, label: t.label, testedAt: t.testedAt, result: t.result })), restoreLimitDays: settings.healthChecks.restoreTestDays, updates, @@ -223,7 +231,7 @@ export function agendaFromData(data: AgendaData, now: Date, tz: string, days: nu g.names.push(w.targetName); groups.set(key, g); } - const timed = (id: string, start: string, end: string, title: string, detail: string | null) => + const timed = (id: string, start: string, end: string | null, title: string, detail: string | null) => items.push({ id, kind: "maintenance", at: start, endsAt: end, allDay: false, title, detail, link: "/maintenance", serverId: null, daysLeft: daysBetween(today, localDate(new Date(start), tz)) }); const names = (list: string[]) => { const sorted = [...list].sort((a, b) => a.localeCompare(b)); @@ -236,6 +244,13 @@ export function agendaFromData(data: AgendaData, now: Date, tz: string, days: nu } } + // A patch plan is maintenance too: when it will start and on how many servers. + for (const p of data.patchPlans) { + for (const occ of planOccurrences(p, now, new Date(lastMs)).filter((o) => o.start.getTime() > now.getTime()).slice(0, 60)) { + timed(`plan:${p.id}:${occ.date}`, occ.start.toISOString(), null, `Patch plan: ${p.name}`, `updates ${plural(p.servers, "server")}, one at a time`); + } + } + // Updates that have been waiting: one line for all of them, from the longest wait. if (data.updates.length > 0) { const since = data.updates.map((u) => u.since).filter((s): s is string => !!s).map((s) => localDate(new Date(s), tz)).sort()[0] ?? today; diff --git a/server/src/services/alerts.ts b/server/src/services/alerts.ts index 601b2aa..c12c2a9 100644 --- a/server/src/services/alerts.ts +++ b/server/src/services/alerts.ts @@ -79,8 +79,8 @@ const SEVERITY_ORDER: Record = { critical: 0, warning: 1, /** Which kind of source, and which one, a health/automation condition key belongs to. */ function originOfConditionKey(key: string): { type: string; id: number | null; category: AlertCategory } { - let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)/.exec(key); - if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : key.startsWith("failed-units") ? "services" : "disk" }; + let m = /^(?:offline|disk|failed-units|smart|zfs|new-port):server:(\d+)/.exec(key); + if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : key.startsWith("failed-units") ? "services" : key.startsWith("new-port") ? "monitoring" : "disk" }; m = /^disk:proxmox:(\d+):/.exec(key); if (m) return { type: "proxmox", id: Number(m[1]), category: "disk" }; m = /^(?:synology-volume|synology-disk|disk:synology):(\d+):/.exec(key); diff --git a/server/src/services/changeLog.ts b/server/src/services/changeLog.ts index 1d061ac..bffa5b9 100644 --- a/server/src/services/changeLog.ts +++ b/server/src/services/changeLog.ts @@ -54,6 +54,8 @@ const KIND_OF_CATEGORY: Record = { restore_test: "backups", endpoint: "endpoints", heartbeat: "endpoints", + patch_plan: "servers", + report: "other", }; /** Sign-ins and a person's own data export are not changes to the homelab; the diagnostic log has its own page. */ @@ -155,6 +157,22 @@ export function describeChange(category: string, action: string, detail: Record< return { title: `Endpoint check changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: [str(detail.kind), str(detail.target)].filter(Boolean).join(" · ") || null }; case "endpoint/delete": return { title: `Endpoint check removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null }; + case "report/send_monthly": + return { title: `Monthly report sent${str(detail.month) ? ` (${str(detail.month)})` : ""}`, tone: "neutral", detail: null }; + case "patch_plan/create": + return { title: `Patch plan added${name ? `: ${name}` : ""}`, tone: "neutral", detail: null }; + case "patch_plan/update": + return { title: `Patch plan changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null }; + case "patch_plan/delete": + return { title: `Patch plan removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null }; + case "patch_plan/run": + return { title: `Patch plan started${name ? `: ${name}` : ""}`, tone: "neutral", detail: str(detail.trigger) === "schedule" ? "by its schedule" : str(detail.by) ? `by ${str(detail.by)}` : null }; + case "patch_plan/finished": + return { title: `Patch plan finished${name ? `: ${name}` : ""}`, tone: "good", detail: str(detail.message) }; + case "patch_plan/failed": + return { title: `Patch plan didn't finish cleanly${name ? `: ${name}` : ""}`, tone: "bad", detail: str(detail.message) }; + case "patch_plan/cancelled": + return { title: `Patch plan stopped${name ? `: ${name}` : ""}`, tone: "neutral", detail: str(detail.message) }; case "heartbeat/create": return { title: `Heartbeat added${name ? `: ${name}` : ""}`, tone: "neutral", detail: typeof detail.periodMinutes === "number" ? `expected every ${detail.periodMinutes} min` : null }; case "heartbeat/update": @@ -260,6 +278,7 @@ export async function getChangeLog(query: ChangeQuery = {}): Promise<{ entries: targetName = endpointName.get(idNum) ?? targetName; link = "/endpoints"; } else if (a.targetType === "heartbeat") link = "/heartbeats"; + else if (a.targetType === "patch_plan") link = "/updates?view=plans"; else if (a.targetType === "asset" && idNum !== null) link = `/inventory?asset=${idNum}`; else if (a.targetType === "restore_test") link = "/backups"; else if (a.targetType === "dependency") link = "/map?view=dependencies"; diff --git a/server/src/services/domainMonitor.ts b/server/src/services/domainMonitor.ts index 64fac42..faf3444 100644 --- a/server/src/services/domainMonitor.ts +++ b/server/src/services/domainMonitor.ts @@ -5,6 +5,7 @@ import { defaultDeps, lookupRegistration, normalizeDomain, resolveRegistration, import { computeSecretStatus } from "./secretStatus.js"; import { notifyDomainExpiry } from "./notify.js"; import { getSettings } from "./settingsStore.js"; +import { noteRenewal } from "./renewals.js"; export type DomainRow = typeof domains.$inferSelect; @@ -48,6 +49,8 @@ export async function checkDomain(id: number, deps: LookupDeps = defaultDeps): P if (!row) return null; const result = await lookupRegistration(row.name, deps); const [updated] = await db.update(domains).set(outcomeFields(result, row, new Date(deps.now()).toISOString())).where(eq(domains.id, id)).returning(); + // A registration that now runs much longer than it did was renewed. + await noteRenewal("domain", row.name, row.expiresAt, updated.expiresAt, new Date(deps.now())); return updated; } diff --git a/server/src/services/endpointChecks.ts b/server/src/services/endpointChecks.ts index a826cff..84f334f 100644 --- a/server/src/services/endpointChecks.ts +++ b/server/src/services/endpointChecks.ts @@ -12,9 +12,10 @@ import https from "node:https"; import net from "node:net"; import tls from "node:tls"; import type { TLSSocket } from "node:tls"; -import { and, asc, eq, gte, inArray, lt } from "drizzle-orm"; +import { and, asc, eq, gte, inArray, lt, sql } from "drizzle-orm"; import { db } from "../db/client.js"; -import { endpointChecks, endpointResults } from "../db/schema.js"; +import { endpointChecks, endpointDaily, endpointResults } from "../db/schema.js"; +import { noteRenewal } from "./renewals.js"; import { resolveTarget } from "../utils/networkGuard.js"; export const CHECK_KINDS = ["http", "tcp", "tls"] as const; @@ -244,9 +245,33 @@ export async function runCheck(c: CheckRow, now: Date = new Date(), probeFn: (c: }) .where(eq(endpointChecks.id, c.id)); await db.insert(endpointResults).values({ checkId: c.id, at: iso, ok: result.ok, ms: result.ms, message: result.message.slice(0, 300) }); + await addToDaily(c.id, iso, result); + // A certificate that is now good for much longer than it was has been renewed. + if (result.certExpiresAt) await noteRenewal("certificate", c.name, c.certExpiresAt, result.certExpiresAt, now); return result; } +/** How long the daily figures are kept: years, where the individual results are kept two weeks. */ +export const KEEP_DAILY_DAYS = 800; + +/** Adds one result to its day: the monthly report's availability is worked from these, long after the results themselves are gone. */ +async function addToDaily(checkId: number, atIso: string, result: ProbeResult): Promise { + const timed = result.ok && result.ms !== null; + const ok = result.ok ? 1 : 0; + const ms = timed ? (result.ms as number) : 0; + try { + await db + .insert(endpointDaily) + .values({ checkId, day: atIso.slice(0, 10), total: 1, ok, msSum: ms, msCount: timed ? 1 : 0 }) + .onConflictDoUpdate({ + target: [endpointDaily.checkId, endpointDaily.day], + set: { total: sql`${endpointDaily.total} + 1`, ok: sql`${endpointDaily.ok} + ${ok}`, msSum: sql`${endpointDaily.msSum} + ${ms}`, msCount: sql`${endpointDaily.msCount} + ${timed ? 1 : 0}` }, + }); + } catch (err) { + console.error("[endpoints] couldn't add to the daily figures:", err); + } +} + /** Runs every enabled check that is due, a few at a time. Returns how many it asked. */ export async function runDueChecks(now: Date = new Date(), probeFn?: (c: CheckRow) => Promise): Promise { const rows = await db.select().from(endpointChecks).where(eq(endpointChecks.enabled, true)).orderBy(asc(endpointChecks.id)); @@ -264,6 +289,7 @@ export async function runDueChecks(now: Date = new Date(), probeFn?: (c: CheckRo export async function pruneResults(now: Date = new Date()): Promise { await db.delete(endpointResults).where(lt(endpointResults.at, new Date(now.getTime() - KEEP_RESULT_DAYS * 86_400_000).toISOString())); + await db.delete(endpointDaily).where(lt(endpointDaily.day, new Date(now.getTime() - KEEP_DAILY_DAYS * 86_400_000).toISOString().slice(0, 10))); } // ─── What the history says ───────────────────────────────────────────────── diff --git a/server/src/services/healthMonitor.ts b/server/src/services/healthMonitor.ts index 89e23f6..aea54a7 100644 --- a/server/src/services/healthMonitor.ts +++ b/server/src/services/healthMonitor.ts @@ -1,6 +1,6 @@ import { eq } from "drizzle-orm"; import { db } from "../db/client.js"; -import { integrations, servers } from "../db/schema.js"; +import { integrations, serverPorts, servers } from "../db/schema.js"; import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createProxmoxAdapter, type ProxmoxNodeStats } from "../integrations/proxmox/adapter.js"; import { createSynologyAdapter, type SynologyStorageInfo } from "../integrations/synology/adapter.js"; @@ -11,6 +11,8 @@ import { activeSubjects, subjectOfConditionKey } from "./maintenance.js"; import { dependencyCover, HEALTH_STATE_FLAG } from "./dependencies.js"; import { getFailingSources } from "./integrationHealthMonitor.js"; import { hostHealthConditions, parseHostHealth, withoutIgnoredServices, type HostHealth } from "./hostHealth.js"; +import { newListeningPorts, newPortMessage, parseBaseline, portKey, type NewPort } from "./newPorts.js"; +import { parseJson, type AgentPort } from "./agentPorts.js"; import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js"; const STATE_FLAG = HEALTH_STATE_FLAG; @@ -39,6 +41,8 @@ export interface ServerSnapshot { health?: HostHealth | null; /** This server's own thresholds, where it has any: they replace the global ones for it. */ thresholds?: ServerThresholds | null; + /** TCP ports that began listening after the baseline and that nobody has noted (services/newPorts.ts); empty when the check is off. */ + newPorts?: NewPort[]; } export interface ProxmoxSnapshot { integrationId: number; @@ -123,6 +127,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, // A server that never reported has no agent to go quiet; and an offline one's disk figures are stale, so neither is judged on disks. if (s.lastSeenAt && !offline) { for (const c of hostHealthConditions(s.id, s.name, s.health ?? null)) out.push({ key: c.key, source: "server", message: c.message, severity: c.severity }); + for (const p of s.newPorts ?? []) out.push({ key: `new-port:server:${s.id}:${portKey(p.protocol, p.port)}`, source: "server", message: newPortMessage(s.name, p), severity: "warning" }); for (const d of s.disks) { const p = usage(d.usedBytes, d.sizeBytes); // A disk can have its own limit, or be left alone altogether; otherwise the server's, then the global one. @@ -299,6 +304,17 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel const held = new Set(); const snapshot: HealthSnapshot = { servers: [], proxmox: [], synology: [] }; + const { healthChecks: checks } = await getSettings(); + // The ports somebody has said are expected (a label or a comment on the Ports page); a row that only records an open scan says nothing. + const expectedByServer = new Map>(); + if (checks.newPortAlerts) { + for (const r of await db.select().from(serverPorts)) { + if (!r.label && !r.comment) continue; + const set = expectedByServer.get(r.serverId) ?? new Set(); + set.add(portKey(r.protocol, r.port)); + expectedByServer.set(r.serverId, set); + } + } for (const s of await db.select().from(servers)) { let disks: ServerSnapshot["disks"] = []; try { @@ -306,7 +322,8 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel } catch { // a malformed report shouldn't stop every other server being checked } - snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: withoutIgnoredServices(parseHostHealth(s.hostHealth), parseThresholds(s.thresholds)?.ignoreServices), thresholds: parseThresholds(s.thresholds) }); + const newPorts = checks.newPortAlerts ? newListeningPorts(parseJson(s.listeningPorts, null), parseBaseline(s.portBaseline), expectedByServer.get(s.id) ?? new Set()) : []; + snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: withoutIgnoredServices(parseHostHealth(s.hostHealth), parseThresholds(s.thresholds)?.ignoreServices), thresholds: parseThresholds(s.thresholds), newPorts }); } const rows = await db diff --git a/server/src/services/maintenance.ts b/server/src/services/maintenance.ts index cff5595..6d348fd 100644 --- a/server/src/services/maintenance.ts +++ b/server/src/services/maintenance.ts @@ -23,7 +23,7 @@ export async function listUpcomingWindows(now: Date = new Date()): Promise String(n).padStart(2, "0"); + +export function isMonthId(v: unknown): v is string { + return typeof v === "string" && /^\d{4}-(0[1-9]|1[0-2])$/.test(v); +} + +/** The year and month `now` is in, on the clock of `tz`. */ +function monthOf(now: Date, tz: string): { year: number; month: number } { + const parts = new Intl.DateTimeFormat("en-CA", { timeZone: tz, year: "numeric", month: "2-digit" }).formatToParts(now); + return { year: Number(parts.find((p) => p.type === "year")?.value), month: Number(parts.find((p) => p.type === "month")?.value) }; +} + +export const monthId = (year: number, month: number) => `${year}-${pad(month)}`; +const shift = (year: number, month: number, by: number) => { + const index = year * 12 + (month - 1) + by; + return { year: Math.floor(index / 12), month: (index % 12) + 1 }; +}; + +export function monthLabel(id: string): string { + const [y, m] = id.split("-").map(Number); + return new Date(Date.UTC(y, m - 1, 15)).toLocaleDateString("en-GB", { month: "long", year: "numeric", timeZone: "UTC" }); +} + +/** The month before the one `now` is in: what a report sent at the start of a month is about. */ +export function lastFullMonth(now: Date, tz: string): string { + const cur = monthOf(now, tz); + const prev = shift(cur.year, cur.month, -1); + return monthId(prev.year, prev.month); +} + +/** This month and the eleven before it, newest first. */ +export function recentMonths(now: Date, tz: string, count = 12): { id: string; label: string }[] { + const cur = monthOf(now, tz); + return Array.from({ length: count }, (_, i) => { + const m = shift(cur.year, cur.month, -i); + const id = monthId(m.year, m.month); + return { id, label: monthLabel(id) }; + }); +} + +/** The instants a month begins and ends at, on the clock of `tz`. */ +export function monthBounds(id: string, tz: string): { start: Date; end: Date } { + const [y, m] = id.split("-").map(Number); + const next = shift(y, m, 1); + return { start: zonedTime(y, m, 1, 0, 0, tz), end: zonedTime(next.year, next.month, 1, 0, 0, tz) }; +} + +// ─── Gathering ───────────────────────────────────────────────────────────── + +const round1 = (n: number) => Math.round(n * 10) / 10; + +export async function buildMonthlyReport(id: string, now: Date = new Date()): Promise { + const settings = await getSettings(); + const tz = settings.notifications.timezone || "UTC"; + const { start, end: fullEnd } = monthBounds(id, tz); + const partial = fullEnd.getTime() > now.getTime(); + const end = partial ? now : fullEnd; + const days = Math.max(1, Math.ceil((end.getTime() - start.getTime()) / DAY_MS)); + const startIso = start.toISOString(); + const endIso = end.toISOString(); + + // Availability. Servers and integrations from the audit log's record of going away and coming back; laid over the month's days ending where the month does. + const uptime = await getUptime(end, days); + const serverNames = new Map((await db.select({ id: servers.id, name: servers.name }).from(servers)).map((s) => [s.id, s.name])); + const integrationNames = new Map((await db.select({ id: integrations.id, name: integrations.name }).from(integrations)).map((i) => [i.id, i.name])); + const rowsOf = (series: Record, names: Map): AvailabilityRow[] => + Object.entries(series) + .map(([key, s]) => ({ id: Number(key), name: names.get(Number(key)) ?? `#${key}`, percent: s.percent, downMinutes: s.buckets.reduce((n, b) => n + b.d, 0) })) + .filter((r) => names.has(r.id)) + .sort((a, b) => (a.percent ?? 101) - (b.percent ?? 101) || a.name.localeCompare(b.name)); + + // Endpoints: from the daily figures, for the days the month covers. The figures are per UTC day, so a day at either end can belong to a neighbouring month + // by a few hours (more than the time zone is ahead of UTC); a month that is not over takes in today. + const [yy, mm] = id.split("-").map(Number); + const following = shift(yy, mm, 1); + const firstDay = `${id}-01`; + const lastDayExclusive = partial ? new Date(now.getTime() + DAY_MS).toISOString().slice(0, 10) : `${monthId(following.year, following.month)}-01`; + const dailyRows = await db.select().from(endpointDaily).where(and(gte(endpointDaily.day, firstDay), lt(endpointDaily.day, lastDayExclusive))); + const checkNames = new Map((await db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks)).map((c) => [c.id, c.name])); + const perCheck = new Map(); + for (const r of dailyRows) { + const t = perCheck.get(r.checkId) ?? { total: 0, ok: 0, msSum: 0, msCount: 0 }; + t.total += r.total; + t.ok += r.ok; + t.msSum += r.msSum; + t.msCount += r.msCount; + perCheck.set(r.checkId, t); + } + const endpointRows: EndpointRow[] = [...perCheck] + .filter(([cid, t]) => checkNames.has(cid) && t.total > 0) + .map(([cid, t]) => ({ id: cid, name: checkNames.get(cid)!, percent: round1((t.ok / t.total) * 100), avgMs: t.msCount > 0 ? Math.round(t.msSum / t.msCount) : null, checks: t.total })) + .sort((a, b) => (a.percent ?? 101) - (b.percent ?? 101) || a.name.localeCompare(b.name)); + + // Incidents. + const all = await listIncidents({ now: end, limit: 3000 }); + const inMonth = (iso: string) => iso >= startIso && iso < endIso; + const began = all.filter((i) => inMonth(i.openedAt)); + const resolved = all.filter((i) => i.resolvedAt !== null && inMonth(i.resolvedAt)); + const stillOpen = all.filter((i) => i.openedAt < endIso && (i.resolvedAt === null || i.resolvedAt >= endIso)).length; + const closedHere = began.filter((i) => i.resolvedAt !== null && inMonth(i.resolvedAt)); + const minutesOf = (ms: number) => Math.max(1, Math.round(ms / 60_000)); + const totalMinutes = closedHere.reduce((n, i) => n + minutesOf(i.durationMs), 0); + const byDuration = [...began].sort((a, b) => b.durationMs - a.durationMs); + const categories = new Map(); + for (const i of began) categories.set(i.category, (categories.get(i.category) ?? 0) + 1); + + // Updates, by hand and by plan. + const runRows = await db.select().from(updateRuns).where(and(gte(updateRuns.startedAt, startIso), lt(updateRuns.startedAt, endIso))); + const perServer = new Map(); + let installed = 0; + for (const r of runRows) { + const gained = r.status === "done" && r.updatesBefore !== null && r.updatesAfter !== null ? Math.max(0, r.updatesBefore - r.updatesAfter) : 0; + installed += gained; + const t = perServer.get(r.serverId) ?? { runs: 0, failed: 0, installed: 0 }; + t.runs++; + if (r.status === "failed") t.failed++; + t.installed += gained; + perServer.set(r.serverId, t); + } + const planRows = await db.select().from(patchPlanRuns).where(and(gte(patchPlanRuns.startedAt, startIso), lt(patchPlanRuns.startedAt, endIso))); + + // Restore tests, and the backups as they are now. + const tests = await db.select().from(restoreTests).where(and(gte(restoreTests.testedAt, firstDay), lt(restoreTests.testedAt, lastDayExclusive))); + const coverage = await getCachedBackupCoverage(now); + + const renewalRows = await renewalsBetween(start, end); + + return { + month: id, + label: monthLabel(id), + from: startIso, + to: endIso, + partial, + timezone: tz, + servers: rowsOf(uptime.servers, serverNames), + integrations: rowsOf(uptime.integrations, integrationNames), + endpoints: endpointRows, + incidents: { + began: began.length, + resolved: resolved.length, + stillOpen, + totalMinutes, + averageMinutes: closedHere.length > 0 ? Math.round(totalMinutes / closedHere.length) : null, + longest: byDuration[0] ? { id: byDuration[0].id, message: byDuration[0].message, minutes: minutesOf(byDuration[0].durationMs) } : null, + byCategory: [...categories].map(([category, count]) => ({ category, count })).sort((a, b) => b.count - a.count || a.category.localeCompare(b.category)), + list: byDuration.slice(0, MAX_LISTED).map((i) => ({ id: i.id, message: i.message, where: i.target?.name ?? i.source, severity: i.peakSeverity, openedAt: i.openedAt, resolvedAt: i.resolvedAt, minutes: minutesOf(i.durationMs) })), + }, + updates: { + runs: runRows.length, + done: runRows.filter((r) => r.status === "done").length, + failed: runRows.filter((r) => r.status === "failed").length, + installed, + byServer: [...perServer] + .map(([sid, t]) => ({ name: serverNames.get(sid) ?? `#${sid}`, ...t })) + .sort((a, b) => b.runs - a.runs || a.name.localeCompare(b.name)), + planRuns: { runs: planRows.length, done: planRows.filter((r) => r.status === "done").length, failed: planRows.filter((r) => r.status === "failed").length }, + }, + restoreTests: tests.map((t) => ({ label: t.label, testedAt: t.testedAt, result: t.result, note: t.note })).sort((a, b) => a.testedAt.localeCompare(b.testedAt)), + backups: coverage ? { ok: coverage.counts.ok, stale: coverage.counts.stale, failed: coverage.counts.failed, none: coverage.counts.none, unknown: coverage.counts.unknown, total: coverage.counts.total, asOf: coverage.generatedAt } : null, + renewals: renewalRows.map((r) => ({ kind: r.kind, subject: r.subject, at: r.at, fromDate: r.fromDate, toDate: r.toDate })), + }; +} + +// ─── As a message ────────────────────────────────────────────────────────── + +const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; +const RESULT_WORDS: Record = { ok: "passed", partial: "partly worked", failed: "failed", not_needed: "marked as not needed" }; + +function average(rows: AvailabilityRow[]): number | null { + const known = rows.filter((r) => r.percent !== null); + return known.length === 0 ? null : round1(known.reduce((n, r) => n + (r.percent as number), 0) / known.length); +} + +/** The report as plain text, for a notification or an email: short, a topic only where there is something to say. */ +export function monthlyReportText(r: MonthlyReport): { title: string; message: string } { + const sections: string[] = []; + const lines = (heading: string, body: string[]) => body.length > 0 && sections.push(`${heading.toUpperCase()}\n${body.join("\n")}`); + + const avail: string[] = []; + const serverAvg = average(r.servers); + if (serverAvg !== null) { + avail.push(`Servers: ${serverAvg}% on average (${plural(r.servers.length, "server")}).`); + for (const s of r.servers.filter((x) => x.downMinutes > 0).slice(0, 6)) avail.push(`- ${s.name}: ${s.percent}% (${spanWords(s.downMinutes)} down)`); + } + if (r.endpoints.length > 0) { + const avg = round1(r.endpoints.reduce((n, e) => n + (e.percent ?? 0), 0) / r.endpoints.length); + avail.push(`Endpoints: ${avg}% on average (${plural(r.endpoints.length, "check")}).`); + for (const e of r.endpoints.filter((x) => (x.percent ?? 100) < 100).slice(0, 6)) avail.push(`- ${e.name}: ${e.percent}%${e.avgMs !== null ? `, ${e.avgMs} ms on average` : ""}`); + } + const integrationAvg = average(r.integrations); + if (integrationAvg !== null) avail.push(`Integrations: ${integrationAvg}% on average.`); + lines("Availability", avail); + + const inc = r.incidents; + if (inc.began > 0 || inc.resolved > 0 || inc.stillOpen > 0) { + const body = [`${plural(inc.began, "incident")} began, ${inc.resolved} ended, ${inc.stillOpen} still open.`]; + if (inc.averageMinutes !== null) body.push(`A resolved incident took ${spanWords(inc.averageMinutes)} on average.`); + for (const i of inc.list.slice(0, 3)) body.push(`- ${i.message} — ${spanWords(i.minutes)}${i.resolvedAt ? "" : " and still open"}`); + lines("Incidents", body); + } else if (serverAvg !== null || r.endpoints.length > 0) { + lines("Incidents", ["None."]); + } + + const u = r.updates; + const upd: string[] = []; + if (u.runs > 0) upd.push(`${plural(u.runs, "update run")}: ${u.done} done${u.failed > 0 ? `, ${u.failed} failed` : ""}${u.installed > 0 ? ` — ${plural(u.installed, "update")} installed` : ""}.`); + if (u.planRuns.runs > 0) upd.push(`Patch plans ran ${plural(u.planRuns.runs, "time")}${u.planRuns.failed > 0 ? `, ${u.planRuns.failed} not cleanly` : ", all cleanly"}.`); + lines("Updates", upd); + + const bk: string[] = []; + for (const t of r.restoreTests) bk.push(`- Restore test of ${t.label} (${t.testedAt}) ${RESULT_WORDS[t.result] ?? t.result}`); + if (r.restoreTests.length === 0) bk.push("No restore test was recorded."); + if (r.backups) bk.push(`Backups now: ${r.backups.ok} of ${plural(r.backups.total, "guest")} recent${r.backups.failed + r.backups.stale + r.backups.none > 0 ? `; ${r.backups.failed} failed, ${r.backups.stale} stale, ${r.backups.none} without a backup` : ""}.`); + lines("Backups", bk); + + lines("Renewed", r.renewals.slice(0, 10).map((x) => `- ${x.kind === "domain" ? "Domain" : "Certificate"} ${x.subject}: ${x.fromDate} → ${x.toDate}`)); + + const title = `Homelab monthly report — ${r.label}${r.partial ? " (so far)" : ""}`; + return { title, message: sections.length > 0 ? sections.join("\n\n") : "Nothing was recorded for this month." }; +} diff --git a/server/src/services/monthlyReportScheduler.ts b/server/src/services/monthlyReportScheduler.ts new file mode 100644 index 0000000..10cbee7 --- /dev/null +++ b/server/src/services/monthlyReportScheduler.ts @@ -0,0 +1,43 @@ +import schedule from "node-schedule"; +import { getSettings } from "./settingsStore.js"; +import { notify } from "./notify.js"; +import { buildMonthlyReport, lastFullMonth, monthlyReportText } from "./monthlyReport.js"; + +let currentJob: schedule.Job | null = null; + +/** The cron rule for "this day of the month at HH:MM" (1–28, so every month has it). */ +export function cronForMonthlyReport(day: number, time: string): string { + const [h, m] = time.split(":").map(Number); + const d = Number.isInteger(day) && day >= 1 && day <= 28 ? day : 1; + return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} ${d} * *`; +} + +/** Builds last month's report and sends it through the channels (held with other notifications during quiet hours). */ +export async function sendMonthlyReport(now: Date = new Date()): Promise<{ title: string; message: string }> { + const tz = (await getSettings()).notifications.timezone || "UTC"; + const text = monthlyReportText(await buildMonthlyReport(lastFullMonth(now, tz), now)); + await notify(text.title, text.message); + return text; +} + +/** (Re)schedules the monthly report per the current settings. Call again after they change. */ +export async function scheduleMonthlyReport(): Promise { + if (currentJob) { + currentJob.cancel(); + currentJob = null; + } + const { monthlyReport, notifications } = await getSettings(); + if (!monthlyReport.enabled) { + console.log("[report] monthly report is off, nothing scheduled"); + return; + } + const rule = cronForMonthlyReport(monthlyReport.day, monthlyReport.time); + currentJob = schedule.scheduleJob({ rule, tz: notifications.timezone }, () => { + sendMonthlyReport().catch((err) => console.error("[report] couldn't send the monthly report:", err)); + }); + console.log(`Monthly report scheduled: ${rule} (${notifications.timezone})`); +} + +export async function initMonthlyReportScheduler(): Promise { + await scheduleMonthlyReport(); +} diff --git a/server/src/services/newPorts.ts b/server/src/services/newPorts.ts new file mode 100644 index 0000000..a2917cc --- /dev/null +++ b/server/src/services/newPorts.ts @@ -0,0 +1,60 @@ +/** + * New ports: a server that starts listening on something it did not before is worth a look — a service somebody started, or something that should not be + * there. The first report that carries ports is taken as the baseline: what the server was listening on, taken as normal. After that a TCP port that is + * listening on an address other than this machine's own, is not in the baseline, and has not been noted on the Ports page, is a warning (the Alerts page, + * and a notification) until the port is noted ("this is expected") or stops listening. + * + * Left out on purpose: UDP (clients and discovery open them all the time), ports in the dynamic range (49152 and up, where programs pick a free one), and + * anything bound only to the loopback address, which nothing else on the network can reach. + */ +import { isLoopback, type AgentPort } from "./agentPorts.js"; + +/** From here up, a port is usually one a program picked for itself, not a service. */ +export const DYNAMIC_PORTS_FROM = 49152; + +export interface NewPort { + protocol: "tcp"; + port: number; + /** What is listening, as the agent says; null when it does not. */ + process: string | null; +} + +export const portKey = (protocol: string, port: number) => `${protocol}:${port}`; + +/** What is stored, read back as a set of "tcp:443"; null when there is no baseline yet (or it can't be read). */ +export function parseBaseline(stored: string | null): Set | null { + if (!stored) return null; + try { + const v = JSON.parse(stored); + return Array.isArray(v) ? new Set(v.filter((k): k is string => typeof k === "string")) : null; + } catch { + return null; + } +} + +/** The baseline a first report makes: everything it listens on, in any protocol, so nothing about it is ever "new". */ +export function baselineFrom(listening: Pick[]): string { + return JSON.stringify([...new Set(listening.map((p) => portKey(p.protocol, p.port)))].sort()); +} + +/** + * Pure: the ports that are new — listening now, reachable from outside this machine, in neither the baseline nor the noted ports. `expected` is the set of + * "tcp:8080" somebody gave a label or a comment on the Ports page. With no baseline yet, nothing is new. + */ +export function newListeningPorts(listening: AgentPort[] | null | undefined, baseline: Set | null, expected: Set): NewPort[] { + if (!listening || baseline === null) return []; + const found = new Map(); + for (const p of listening) { + if (p.protocol !== "tcp" || !Number.isInteger(p.port) || p.port <= 0 || p.port >= DYNAMIC_PORTS_FROM) continue; + if (isLoopback(p.address ?? "")) continue; + const key = portKey("tcp", p.port); + if (baseline.has(key) || expected.has(key)) continue; + const seen = found.get(p.port); + if (seen) seen.process = seen.process ?? (p.process || null); + else found.set(p.port, { protocol: "tcp", port: p.port, process: p.process || null }); + } + return [...found.values()].sort((a, b) => a.port - b.port); +} + +export const newPortMessage = (serverName: string, p: NewPort) => + `${serverName}: a new port is listening — ${p.port}/tcp${p.process ? ` (${p.process})` : ""}. If it should be there, note it on the server's Ports card.`; diff --git a/server/src/services/notify.ts b/server/src/services/notify.ts index 6ad2e82..2526c24 100644 --- a/server/src/services/notify.ts +++ b/server/src/services/notify.ts @@ -449,6 +449,16 @@ export async function notifyUpdateRunFinished(run: { serverName: string; templat ); } +/** How a patch plan's run ended: one message, with a line for each server. Held back or sent with the other update-run results. */ +export async function notifyPatchPlanFinished(run: { planName: string; ok: boolean; message: string; lines: string[] }): Promise { + if (!(await eventEnabled("updateRunResults"))) return; + await notify( + run.ok ? `Homelab Manager — Patch plan "${run.planName}" finished` : `Homelab Manager — Patch plan "${run.planName}" didn't finish cleanly`, + `${run.message}${run.lines.length > 0 ? `\n\n${run.lines.join("\n")}` : ""}`, + { category: "updates", severity: run.ok ? "warning" : "critical" }, + ); +} + export async function notifyPbsVerificationFailed( failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[], ): Promise { diff --git a/server/src/services/notifyRouting.ts b/server/src/services/notifyRouting.ts index d38c879..c3195df 100644 --- a/server/src/services/notifyRouting.ts +++ b/server/src/services/notifyRouting.ts @@ -107,11 +107,12 @@ export function metaOfConditionKey(key: string, severity?: string): NotifyMeta { let category: AlertCategory | null = null; if (key.startsWith("offline:")) category = "offline"; else if (key.startsWith("failed-units:")) category = "services"; + else if (key.startsWith("new-port:")) category = "monitoring"; else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk"; else if (key.startsWith("automation:")) category = "automation"; else if (key.startsWith("heartbeat")) category = "automation"; else if (key.startsWith("endpoint-cert:")) category = "expiry"; else if (key.startsWith("endpoint:")) category = "monitoring"; - const server = /^(?:(?:offline|disk|failed-units|smart|zfs):server:|(?:endpoint(?:-cert)?|heartbeat(?:-fail)?):\d+:server:)(\d+)/.exec(key); + const server = /^(?:(?:offline|disk|failed-units|smart|zfs|new-port):server:|(?:endpoint(?:-cert)?|heartbeat(?:-fail)?):\d+:server:)(\d+)/.exec(key); return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null }; } diff --git a/server/src/services/patchPlans.ts b/server/src/services/patchPlans.ts new file mode 100644 index 0000000..ea9bf19 --- /dev/null +++ b/server/src/services/patchPlans.ts @@ -0,0 +1,321 @@ +/** + * Patch plans: "Sundays at 02:00, the servers tagged lab, one at a time, stop at the first failure." + * + * A plan only says when, to whom and how. When it is due (or somebody presses Run now) it makes a *run*: a list of the servers it matches, and goes down + * the list one server at a time. Each server is updated by an ordinary update run (services/updateRuns.ts) — the server is put under maintenance, the + * Semaphore playbook is run limited to it, the app waits for it to report again — and the plan waits for that to finish before it starts the next. What + * depends on the server is silenced for the same time (it will look down while the server restarts). If a server's update fails and the plan says to stop, + * the servers after it are left alone and the run says why; otherwise the plan carries on. A message at the end says how each server went. + * + * A plan starts once a day at most: the day it last started on is kept, so a restart of this app at 02:10 neither misses it nor repeats it. If the app was + * down at the time, it still starts, up to half an hour late; later than that, that day is skipped. + */ +import { and, desc, eq, inArray } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { maintenanceWindows, patchPlanRuns, patchPlans, servers, updateRuns } from "../db/schema.js"; +import { invalidateAlertsCache } from "./alerts.js"; +import { recordAudit } from "./audit.js"; +import { impactOf } from "./dependencies.js"; +import { startOrExtendWindow } from "./maintenance.js"; +import { occurrencesBetween, type Occurrence } from "./maintenanceSchedules.js"; +import { notifyPatchPlanFinished } from "./notify.js"; +import { parseTags } from "./serverTags.js"; +import { parsePendingUpdates } from "./pendingUpdates.js"; +import { ACTIVE_STATUSES, cancelUpdateRun, startUpdateRun, UpdateRunError } from "./updateRuns.js"; + +export type PlanRow = typeof patchPlans.$inferSelect; +export type PlanRunRow = typeof patchPlanRuns.$inferSelect; +export type ItemStatus = "pending" | "running" | "done" | "failed" | "skipped"; +export type RunStatus = "running" | "done" | "failed" | "cancelled"; + +export interface PlanItem { + serverId: number; + name: string; + status: ItemStatus; + updateRunId: number | null; + message: string | null; + /** Windows made for what depends on this server, which end with it. */ + windowIds: number[]; +} + +/** How late a plan may still start, if the app was not running at its time. */ +export const START_GRACE_MS = 30 * 60_000; +export const POLL_INTERVAL_MS = 20_000; +export const MAX_PLANS = 30; + +export class PlanError extends Error { + constructor( + readonly status: number, + readonly code: string, + message: string, + ) { + super(message); + } +} + +// ─── Reading what is stored ──────────────────────────────────────────────── + +function jsonArray(stored: string | null | undefined): T[] { + if (!stored) return []; + try { + const v = JSON.parse(stored); + return Array.isArray(v) ? (v as T[]) : []; + } catch { + return []; + } +} + +export const parseItems = (stored: string | null | undefined): PlanItem[] => jsonArray(stored).filter((i) => i && typeof i.serverId === "number"); +export const planDays = (plan: Pick): number[] => jsonArray(plan.days).filter((d) => Number.isInteger(d)); +export const planTags = (plan: Pick): string[] => parseTags(plan.tags); +export const planServerIds = (plan: Pick): number[] => jsonArray(plan.serverIds).filter((i) => Number.isInteger(i)); + +// ─── Who a plan is for, and when ─────────────────────────────────────────── + +/** Pure: the servers a plan covers — those with any of its tags, and those named — by name. */ +export function planServers(plan: Pick, all: { id: number; name: string; tags: string | null }[]): { id: number; name: string }[] { + const tags = new Set(planTags(plan)); + const ids = new Set(planServerIds(plan)); + return all + .filter((s) => ids.has(s.id) || parseTags(s.tags).some((t) => tags.has(t))) + .map((s) => ({ id: s.id, name: s.name })) + .sort((a, b) => a.name.localeCompare(b.name)); +} + +/** The times a plan happens, from `from` to `to`. A plan has no length; the minute it starts is what matters. */ +export function planOccurrences(plan: Pick, from: Date, to: Date): Occurrence[] { + return occurrencesBetween({ days: plan.days, startTime: plan.startTime, durationMinutes: 1, timezone: plan.timezone }, from, to); +} + +/** Pure: the occurrence a plan should start for now — its time has come, it is no more than the grace late, and it has not started on that day. */ +export function dueOccurrence(plan: Pick, now: Date, graceMs: number = START_GRACE_MS): Occurrence | null { + if (!plan.enabled) return null; + const found = planOccurrences(plan, new Date(now.getTime() - graceMs - 60_000), new Date(now.getTime() + 1000)).filter((o) => o.start.getTime() <= now.getTime() && now.getTime() - o.start.getTime() <= graceMs && o.date !== plan.lastRunDate); + return found.length > 0 ? found[found.length - 1] : null; +} + +// ─── Starting a run ──────────────────────────────────────────────────────── + +const ACTIVE_RUN: RunStatus[] = ["running"]; + +export async function activeRunOf(planId: number): Promise { + const [run] = await db.select().from(patchPlanRuns).where(and(eq(patchPlanRuns.planId, planId), inArray(patchPlanRuns.status, ACTIVE_RUN))).limit(1); + return run ?? null; +} + +export async function startPlanRun(planId: number, options: { trigger: "schedule" | "manual"; actor: string; now?: Date }): Promise { + const now = options.now ?? new Date(); + const [plan] = await db.select().from(patchPlans).where(eq(patchPlans.id, planId)).limit(1); + if (!plan) throw new PlanError(404, "not_found", "That plan doesn't exist."); + if (await activeRunOf(plan.id)) throw new PlanError(409, "already_running", `"${plan.name}" is already running.`); + const matched = planServers(plan, await db.select({ id: servers.id, name: servers.name, tags: servers.tags }).from(servers)); + if (matched.length === 0) throw new PlanError(400, "no_servers", "No server has this plan's tags or is named in it, so there is nothing to update."); + const items: PlanItem[] = matched.map((s) => ({ serverId: s.id, name: s.name, status: "pending", updateRunId: null, message: null, windowIds: [] })); + const [run] = await db + .insert(patchPlanRuns) + .values({ planId: plan.id, planName: plan.name, trigger: options.trigger, startedBy: options.actor, status: "running", message: `Updating ${matched.length} server${matched.length === 1 ? "" : "s"}, one at a time.`, items: JSON.stringify(items), startedAt: now.toISOString() }) + .returning(); + await recordAudit({ category: "patch_plan", action: "run", targetType: "patch_plan", targetId: plan.id, detail: { name: plan.name, run: run.id, trigger: options.trigger, servers: matched.length, by: options.actor } }); + return run; +} + +// ─── Following a run ─────────────────────────────────────────────────────── + +async function endWindows(ids: number[], now: Date): Promise { + if (ids.length === 0) return; + await db.update(maintenanceWindows).set({ endsAt: now.toISOString() }).where(inArray(maintenanceWindows.id, ids)); + invalidateAlertsCache(); +} + +/** Silences what depends on the server for as long as it is being updated; returns the windows it made (ones it only extended belong to someone else). */ +async function silenceDependents(plan: PlanRow, serverId: number, serverName: string, actor: string, now: Date): Promise { + if (!plan.silenceDependents) return []; + const made: number[] = []; + try { + const impact = await impactOf("server", serverId, now); + for (const d of impact?.dependents ?? []) { + const started = await startOrExtendWindow(d.type, d.id, plan.windowMinutes, `Depends on ${serverName} — patch plan "${plan.name}"`.slice(0, 200), actor, now); + if (started && !started.extended) made.push(started.window.id); + } + } catch (err) { + console.error("[patchPlans] couldn't silence what depends on a server:", err); + } + return made; +} + +const words = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; + +/** What a finished run says, in a line. */ +export function summariseItems(items: PlanItem[]): string { + const count = (s: ItemStatus) => items.filter((i) => i.status === s).length; + const parts = [`${count("done")} updated`]; + if (count("failed") > 0) parts.push(`${count("failed")} failed`); + if (count("skipped") > 0) parts.push(`${count("skipped")} left alone`); + return `${parts.join(", ")} of ${words(items.length, "server")}.`; +} + +async function saveRun(run: PlanRunRow, items: PlanItem[], extra: Partial = {}): Promise { + await db.update(patchPlanRuns).set({ items: JSON.stringify(items), ...extra }).where(eq(patchPlanRuns.id, run.id)); +} + +async function finishRun(run: PlanRunRow, plan: PlanRow | null, items: PlanItem[], status: RunStatus, now: Date, why?: string): Promise { + const message = [summariseItems(items), why].filter(Boolean).join(" "); + await saveRun(run, items, { status, message, finishedAt: now.toISOString() }); + await recordAudit({ category: "patch_plan", action: status === "done" ? "finished" : status, targetType: "patch_plan", targetId: run.planId, detail: { name: run.planName, run: run.id, message } }); + if (status !== "cancelled") { + const lines = items.map((i) => `${i.status === "done" ? "✓" : i.status === "failed" ? "✕" : "–"} ${i.name}${i.message ? ` — ${i.message}` : ""}`); + await notifyPatchPlanFinished({ planName: run.planName, ok: status === "done", message, lines }).catch((err) => console.error("[patchPlans] couldn't send the result:", err)); + } + void plan; +} + +/** When a server's turn is over and it failed: stop (leaving the rest alone) or carry on. */ +function afterFailure(plan: PlanRow, items: PlanItem[]): boolean { + if (!plan.stopOnFailure) return false; + for (const i of items) if (i.status === "pending") Object.assign(i, { status: "skipped", message: "Not started, because an earlier server failed." }); + return true; +} + +async function advanceRun(run: PlanRunRow, now: Date): Promise { + const [plan] = await db.select().from(patchPlans).where(eq(patchPlans.id, run.planId)).limit(1); + const items = parseItems(run.items); + if (!plan) return; // deleted: the run goes with it + + // The server whose turn it is: has its update run finished? + const current = items.find((i) => i.status === "running"); + if (current) { + const [ur] = current.updateRunId !== null ? await db.select().from(updateRuns).where(eq(updateRuns.id, current.updateRunId)).limit(1) : []; + if (ur && ACTIVE_STATUSES.includes(ur.status as (typeof ACTIVE_STATUSES)[number])) return; + const ok = ur?.status === "done"; + current.status = ok ? "done" : "failed"; + current.message = ur?.message ?? "The update run is gone."; + await endWindows(current.windowIds, now); + if (!ok && afterFailure(plan, items)) { + await finishRun(run, plan, items, "failed", now, `Stopped after ${current.name}.`); + return; + } + await saveRun(run, items); + } + + // The next one. + for (;;) { + const next = items.find((i) => i.status === "pending"); + if (!next) { + await finishRun(run, plan, items, items.some((i) => i.status === "failed") ? "failed" : "done", now); + return; + } + const [server] = await db.select().from(servers).where(eq(servers.id, next.serverId)).limit(1); + if (!server) { + Object.assign(next, { status: "skipped", message: "The server was removed." }); + continue; + } + if (plan.onlyWithUpdates) { + const updates = parsePendingUpdates(server.pendingUpdates); + if (!updates || updates.count === 0) { + Object.assign(next, { status: "skipped", message: "No updates were waiting." }); + continue; + } + } + const actor = `Patch plan "${plan.name}"`; + try { + const ur = await startUpdateRun( + { + serverId: server.id, + integrationId: plan.integrationId, + projectId: plan.projectId, + templateId: plan.templateId, + limit: plan.limitFrom === "hostname" && server.hostname ? server.hostname : server.name, + windowMinutes: plan.windowMinutes, + actor, + }, + now, + ); + Object.assign(next, { status: "running", updateRunId: ur.id, message: "The update run has started.", windowIds: await silenceDependents(plan, server.id, server.name, actor, now) }); + await saveRun(run, items); + return; // one at a time: this one is followed until it is over + } catch (err) { + const message = err instanceof UpdateRunError || err instanceof Error ? err.message : String(err); + Object.assign(next, { status: "failed", message }); + if (afterFailure(plan, items)) { + await finishRun(run, plan, items, "failed", now, `Stopped at ${next.name}.`); + return; + } + } + } +} + +/** One pass over every plan run that is under way. Returns how many it looked at (for the tests). */ +export async function advancePlanRuns(now: Date = new Date()): Promise { + const active = await db.select().from(patchPlanRuns).where(inArray(patchPlanRuns.status, ACTIVE_RUN)); + for (const run of active) { + try { + await advanceRun(run, now); + } catch (err) { + console.error(`[patchPlans] run ${run.id}:`, err); + } + } + return active.length; +} + +/** Starts the plans whose time has come. Returns how many it started. */ +export async function startDuePlans(now: Date = new Date()): Promise { + let started = 0; + for (const plan of await db.select().from(patchPlans)) { + const due = dueOccurrence(plan, now); + if (!due) continue; + if (await activeRunOf(plan.id)) continue; // still on the last one: today is skipped rather than overlapping + // Written down first, so that whatever happens next it is not tried again every twenty seconds for half an hour. + await db.update(patchPlans).set({ lastRunDate: due.date }).where(eq(patchPlans.id, plan.id)); + try { + await startPlanRun(plan.id, { trigger: "schedule", actor: "the schedule", now }); + started++; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + await db.insert(patchPlanRuns).values({ planId: plan.id, planName: plan.name, trigger: "schedule", startedBy: "the schedule", status: "failed", message: `Didn't start: ${message}`, items: "[]", startedAt: now.toISOString(), finishedAt: now.toISOString() }); + } + } + return started; +} + +export async function cancelPlanRun(id: number, actor: string, now: Date = new Date()): Promise { + const [run] = await db.select().from(patchPlanRuns).where(eq(patchPlanRuns.id, id)).limit(1); + if (!run) return null; + if (run.status !== "running") throw new PlanError(409, "not_active", "That run is already over."); + const items = parseItems(run.items); + for (const i of items) { + if (i.status === "pending") Object.assign(i, { status: "skipped", message: `Not started: stopped by ${actor}.` }); + else if (i.status === "running") { + if (i.updateRunId !== null) await cancelUpdateRun(i.updateRunId, actor, now).catch(() => undefined); + await endWindows(i.windowIds, now); + Object.assign(i, { status: "failed", message: `Stopped being followed by ${actor}. A playbook already started keeps running in Semaphore.` }); + } + } + const [plan] = await db.select().from(patchPlans).where(eq(patchPlans.id, run.planId)).limit(1); + await finishRun(run, plan ?? null, items, "cancelled", now, `Stopped by ${actor}.`); + const [after] = await db.select().from(patchPlanRuns).where(eq(patchPlanRuns.id, id)).limit(1); + return after; +} + +/** Recent runs, those still going first. */ +export async function listPlanRuns(options: { planId?: number; limit?: number } = {}): Promise { + const rows = await db + .select() + .from(patchPlanRuns) + .where(options.planId !== undefined ? eq(patchPlanRuns.planId, options.planId) : undefined) + .orderBy(desc(patchPlanRuns.startedAt)) + .limit(options.limit ?? 20); + return [...rows.filter((r) => r.status === "running"), ...rows.filter((r) => r.status !== "running")]; +} + +let timer: ReturnType | null = null; + +/** Every twenty seconds: start what is due, follow what is going. A run that was under way when the app last stopped is picked up where it was. */ +export function initPatchPlans(): void { + if (timer) return; + timer = setInterval(() => { + void startDuePlans() + .then(() => advancePlanRuns()) + .catch((err) => console.error("[patchPlans]", err)); + }, POLL_INTERVAL_MS); + timer.unref(); +} diff --git a/server/src/services/renewals.ts b/server/src/services/renewals.ts new file mode 100644 index 0000000..bd8595b --- /dev/null +++ b/server/src/services/renewals.ts @@ -0,0 +1,44 @@ +/** + * Renewals: a certificate, domain registration or secret whose expiry moved later was renewed. The app sees it happen when it re-reads the date — an endpoint + * check seeing a new certificate, the registry saying a domain now runs longer, a secret's live certificate being replaced — and writes it down, so the monthly + * report can say "certificates renewed: 3". It only goes back as far as the version that began watching; nothing is guessed about earlier. + */ +import { and, eq, gte, lt } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { renewals } from "../db/schema.js"; + +/** An expiry has to move at least this far for it to be a renewal (a registry rounding a date by a day is not one). */ +export const MIN_RENEWAL_DAYS = 20; +const DAY_MS = 86_400_000; + +export type RenewalKind = "certificate" | "domain" | "secret"; + +const dayOf = (value: string | null | undefined): string | null => (value && /^\d{4}-\d{2}-\d{2}/.test(value) ? value.slice(0, 10) : null); + +/** Pure: whether the expiry went from one date to a clearly later one. Either may be a full ISO time; only the day counts. */ +export function isRenewal(from: string | null | undefined, to: string | null | undefined): boolean { + const a = dayOf(from); + const b = dayOf(to); + if (!a || !b) return false; + return (Date.parse(`${b}T00:00:00Z`) - Date.parse(`${a}T00:00:00Z`)) / DAY_MS >= MIN_RENEWAL_DAYS; +} + +/** Writes down a renewal if the dates say it was one (and the same one is not already written down). Never throws: it must not hold up the check that saw it. */ +export async function noteRenewal(kind: RenewalKind, subject: string, from: string | null | undefined, to: string | null | undefined, now: Date = new Date()): Promise { + try { + if (!isRenewal(from, to)) return false; + const fromDate = dayOf(from)!; + const toDate = dayOf(to)!; + const [same] = await db.select({ id: renewals.id }).from(renewals).where(and(eq(renewals.kind, kind), eq(renewals.subject, subject), eq(renewals.toDate, toDate))).limit(1); + if (same) return false; + await db.insert(renewals).values({ kind, subject, at: now.toISOString(), fromDate, toDate }); + return true; + } catch (err) { + console.error("[renewals] couldn't record a renewal:", err); + return false; + } +} + +export async function renewalsBetween(from: Date, to: Date) { + return db.select().from(renewals).where(and(gte(renewals.at, from.toISOString()), lt(renewals.at, to.toISOString()))).orderBy(renewals.at); +} diff --git a/server/src/services/settingsStore.ts b/server/src/services/settingsStore.ts index 9dcafc2..172c393 100644 --- a/server/src/services/settingsStore.ts +++ b/server/src/services/settingsStore.ts @@ -113,6 +113,17 @@ export interface HealthCheckSettings { backupStaleHours: number; /** A restore test older than this many days (or none at all) is a problem for a guest that has backups; 0 = never (a failed test always is). */ restoreTestDays: number; + /** A server that starts listening on a TCP port it did not at first, and that is not noted on its Ports card, is a warning (services/newPorts.ts). */ + newPortAlerts: boolean; +} + +/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. Off until switched on. */ +export interface MonthlyReportSettings { + enabled: boolean; + /** Day of the month, 1–28 (every month has it), in notifications.timezone. */ + day: number; + /** "HH:MM", in notifications.timezone. */ + time: string; } /** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */ @@ -225,6 +236,7 @@ export interface AppSettings { quietHours: QuietHoursSettings; healthChecks: HealthCheckSettings; digest: DigestSettings; + monthlyReport: MonthlyReportSettings; signIn: SignInSettings; profilePictures: ProfilePictureSettings; inventory: InventorySettings; @@ -267,12 +279,13 @@ const DEFAULTS: AppSettings = { logRetention: { enabled: false, retentionDays: 90, intervalHours: 24 }, quietHours: { enabled: false, start: "22:00", end: "07:00" }, digest: { enabled: false, day: 1, time: "08:00" }, + monthlyReport: { enabled: false, day: 1, time: "08:00" }, inventory: { currency: "", pricePerKwh: null }, agents: { autoUpdate: false }, selfBackup: { enabled: true, time: "03:30", keep: 14 }, profilePictures: { gravatar: false, identityProvider: false }, signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" }, - healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0 }, + healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true }, // Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every // Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN. consistency: { excludedRanges: ["172.16.0.0/12"] }, diff --git a/server/src/services/taskSync.ts b/server/src/services/taskSync.ts index 88d0967..31726d8 100644 --- a/server/src/services/taskSync.ts +++ b/server/src/services/taskSync.ts @@ -5,6 +5,7 @@ import { recordServerMetrics } from "./serverMetrics.js"; import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js"; import { nextHostHealth, type IncomingHostHealth } from "./hostHealth.js"; import { recordAudit } from "./audit.js"; +import { baselineFrom } from "./newPorts.js"; import { fillInventoryFromServer } from "./inventoryAutofill.js"; import { nextDiagnostics, type AgentDiagnostic } from "./agentDiagnostics.js"; @@ -113,7 +114,7 @@ export async function syncServerTasks(serverId: number, report: AgentReport) { } const system = report.system; - const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince, hostHealth: servers.hostHealth, agentVersion: servers.agentVersion }).from(servers).where(eq(servers.id, serverId)); + const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince, hostHealth: servers.hostHealth, agentVersion: servers.agentVersion, portBaseline: servers.portBaseline }).from(servers).where(eq(servers.id, serverId)); await db .update(servers) .set({ @@ -144,6 +145,8 @@ export async function syncServerTasks(serverId: number, report: AgentReport) { : {}), // Only stored when the agent reported it, so an agent that predates this doesn't wipe the field. ...(system?.listening_ports ? { listeningPorts: JSON.stringify(system.listening_ports) } : {}), + // The first report that carries ports says what is normal for this server; a port that appears later is "new" (services/newPorts.ts). + ...(system?.listening_ports && before && before.portBaseline === null ? { portBaseline: baselineFrom(system.listening_ports) } : {}), }) .where(eq(servers.id, serverId)); diff --git a/server/src/services/tlsCheck.ts b/server/src/services/tlsCheck.ts index 6e08c73..eeaf444 100644 --- a/server/src/services/tlsCheck.ts +++ b/server/src/services/tlsCheck.ts @@ -4,6 +4,7 @@ import { eq, isNotNull } from "drizzle-orm"; import { db } from "../db/client.js"; import { secrets } from "../db/schema.js"; import { resolveTarget } from "../utils/networkGuard.js"; +import { noteRenewal } from "./renewals.js"; /** * Opens a real TLS connection and returns the server certificate's expiry as @@ -79,6 +80,8 @@ export async function refreshTlsSecret(id: number): Promise[0]; const NOW = new Date("2026-10-19T12:00:00Z"); -const empty = (extra: Partial = {}): Data => ({ secrets: [], domains: [], endpoints: [], tailscale: [], assets: [], windows: [], schedules: [], restoreTests: [], restoreLimitDays: 0, updates: [], notes: [], ...extra }); +const empty = (extra: Partial = {}): Data => ({ secrets: [], domains: [], endpoints: [], tailscale: [], assets: [], windows: [], schedules: [], patchPlans: [], restoreTests: [], restoreLimitDays: 0, updates: [], notes: [], ...extra }); const titles = (data: Data, days?: number, tz = "UTC") => agendaFromData(data, NOW, tz, days).items.map((i) => [i.daysLeft, i.kind, i.title]); describe("what is gathered, and how it is dated", () => { diff --git a/server/test/monthlyReport.test.ts b/server/test/monthlyReport.test.ts new file mode 100644 index 0000000..f2291ac --- /dev/null +++ b/server/test/monthlyReport.test.ts @@ -0,0 +1,289 @@ +/** + * The monthly report: which month is which, what renewals and daily figures are kept, what the report gathers for a month, how it reads as a message, and the + * routes and schedule around it. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { createServer as createHttpServer, type Server } from "node:http"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; + +const dir = mkdtempSync(join(tmpdir(), "hm-monthly-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers, auditLog, alertIncidents, updateRuns, patchPlanRuns, patchPlans, restoreTests, renewals, endpointChecks, endpointDaily, endpointResults } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const { isMonthId, lastFullMonth, recentMonths, monthBounds, monthLabel, buildMonthlyReport, monthlyReportText } = await import("../src/services/monthlyReport.js"); +const { isRenewal, noteRenewal } = await import("../src/services/renewals.js"); +const { runCheck, pruneResults } = await import("../src/services/endpointChecks.js"); +const { cronForMonthlyReport } = await import("../src/services/monthlyReportScheduler.js"); +const { reportsRouter } = await import("../src/routes/reports.js"); +const { updateSettings } = await import("../src/services/settingsStore.js"); +const { sql, eq } = await import("drizzle-orm"); + +await runMigrations(); + +describe("which month is which", () => { + test("the last whole month, and the twelve to choose from, on the clock of the time zone", () => { + assert.equal(lastFullMonth(new Date("2026-10-19T12:00:00Z"), "UTC"), "2026-09"); + assert.equal(lastFullMonth(new Date("2027-01-02T12:00:00Z"), "UTC"), "2026-12", "across the new year"); + assert.equal(lastFullMonth(new Date("2026-10-31T23:30:00Z"), "Europe/Stockholm"), "2026-10", "it is already November in Stockholm"); + const months = recentMonths(new Date("2026-10-19T12:00:00Z"), "UTC"); + assert.equal(months.length, 12); + assert.deepEqual([months[0], months[1], months[11]], [{ id: "2026-10", label: "October 2026" }, { id: "2026-09", label: "September 2026" }, { id: "2025-11", label: "November 2025" }]); + assert.equal(monthLabel("2026-02"), "February 2026"); + }); + + test("a month begins and ends where the clock of the time zone says, summer or winter", () => { + const sept = monthBounds("2026-09", "Europe/Stockholm"); + assert.deepEqual([sept.start.toISOString(), sept.end.toISOString()], ["2026-08-31T22:00:00.000Z", "2026-09-30T22:00:00.000Z"]); + const oct = monthBounds("2026-10", "Europe/Stockholm"); + assert.equal(oct.end.toISOString(), "2026-10-31T23:00:00.000Z", "the clocks went back on the last Sunday"); + assert.equal(monthBounds("2026-12", "UTC").end.toISOString(), "2027-01-01T00:00:00.000Z"); + }); + + test("only a real month is a month", () => { + for (const ok of ["2026-09", "1999-12"]) assert.equal(isMonthId(ok), true, ok); + for (const bad of ["2026-13", "2026-00", "2026-9", "26-09", "2026-09-01", "", undefined, 202609]) assert.equal(isMonthId(bad), false, String(bad)); + }); + + test("the schedule is a day of the month, 1 to 28 so every month has it", () => { + assert.equal(cronForMonthlyReport(5, "07:30"), "30 7 5 * *"); + assert.equal(cronForMonthlyReport(31, "07:30"), "30 7 1 * *"); + assert.equal(cronForMonthlyReport(0, "bad"), "0 8 1 * *"); + }); +}); + +describe("renewals and the daily figures", () => { + test("an expiry that moves a clear step later is a renewal; a day's rounding, a move earlier and a gap in the data are not", () => { + assert.equal(isRenewal("2026-10-01", "2026-12-30"), true); + assert.equal(isRenewal("2026-10-01T08:00:00.000Z", "2026-11-20"), true, "a full time or a day, only the day counts"); + assert.equal(isRenewal("2026-10-01", "2026-10-02"), false); + assert.equal(isRenewal("2026-10-01", "2026-10-20"), false, "nineteen days is not enough"); + assert.equal(isRenewal("2026-10-21", "2026-10-01"), false); + assert.equal(isRenewal(null, "2026-12-01"), false, "there was nothing to compare with"); + assert.equal(isRenewal("2026-10-01", "not a date"), false); + }); + + test("a renewal is written down once", async () => { + assert.equal(await noteRenewal("domain", "example.se", "2026-10-01", "2027-10-01"), true); + assert.equal(await noteRenewal("domain", "example.se", "2026-10-02", "2027-10-01"), false, "the same one seen again"); + assert.equal(await noteRenewal("domain", "example.se", "2026-10-01", "2026-10-03"), false); + assert.equal((await db.select().from(renewals).where(eq(renewals.subject, "example.se"))).length, 1); + await db.delete(renewals); + }); + + test("each answer adds to its day; a certificate that is replaced is a renewal; old days go", async () => { + const [check] = await db.insert(endpointChecks).values({ name: "Gitea", kind: "http", target: "https://git.lab/" }).returning(); + const answer = (ok: boolean, ms: number | null, cert?: string) => async () => ({ ok, ms, message: "m", certExpiresAt: cert }); + const t = (h: number) => new Date(`2026-09-10T0${h}:00:00Z`); + let row = (await db.select().from(endpointChecks).where(eq(endpointChecks.id, check.id)))[0]; + await runCheck(row, t(1), answer(true, 100, "2026-10-05T00:00:00.000Z")); + row = (await db.select().from(endpointChecks).where(eq(endpointChecks.id, check.id)))[0]; + await runCheck(row, t(2), answer(true, 300, "2026-10-05T00:00:00.000Z")); + row = (await db.select().from(endpointChecks).where(eq(endpointChecks.id, check.id)))[0]; + await runCheck(row, t(3), answer(false, null, "2026-10-05T00:00:00.000Z")); + const [day] = await db.select().from(endpointDaily).where(eq(endpointDaily.checkId, check.id)); + assert.deepEqual([day.day, day.total, day.ok, day.msSum, day.msCount], ["2026-09-10", 3, 2, 400, 2], "a failure counts as a check but adds no answer time"); + assert.equal((await db.select().from(renewals)).length, 0, "the same certificate three times is not a renewal"); + + row = (await db.select().from(endpointChecks).where(eq(endpointChecks.id, check.id)))[0]; + await runCheck(row, t(4), answer(true, 100, "2026-12-04T00:00:00.000Z")); + const [renewed] = await db.select().from(renewals); + assert.deepEqual([renewed.kind, renewed.subject, renewed.fromDate, renewed.toDate], ["certificate", "Gitea", "2026-10-05", "2026-12-04"]); + + await db.insert(endpointDaily).values({ checkId: check.id, day: "2020-01-01", total: 1, ok: 1 }); + await pruneResults(new Date("2026-10-19T00:00:00Z")); + assert.equal((await db.select().from(endpointDaily).where(eq(endpointDaily.day, "2020-01-01"))).length, 0, "kept for years, not for ever"); + assert.equal((await db.select().from(endpointDaily).where(eq(endpointDaily.day, "2026-09-10"))).length, 1); + await db.delete(renewals); + }); + + test("the migration started the daily figures from the results that already existed", async () => { + const [check] = await db.insert(endpointChecks).values({ name: "Backfilled", kind: "tcp", target: "h", port: 22 }).returning(); + await db.insert(endpointResults).values([ + { checkId: check.id, at: "2026-10-01T01:00:00.000Z", ok: true, ms: 10 }, + { checkId: check.id, at: "2026-10-01T02:00:00.000Z", ok: true, ms: 30 }, + { checkId: check.id, at: "2026-10-01T03:00:00.000Z", ok: false, ms: null }, + { checkId: check.id, at: "2026-10-02T01:00:00.000Z", ok: true, ms: 50 }, + ]); + const migration = readFileSync(new URL("../drizzle/0043_puzzling_prism.sql", import.meta.url), "utf8"); + const statement = migration.split("--> statement-breakpoint").pop()!; + assert.match(statement, /INSERT INTO `endpoint_daily`/); + await db.run(sql.raw(statement)); + const rows = (await db.select().from(endpointDaily).where(eq(endpointDaily.checkId, check.id))).sort((a, b) => a.day.localeCompare(b.day)); + assert.deepEqual(rows.map((r) => [r.day, r.total, r.ok, r.msSum, r.msCount]), [["2026-10-01", 3, 2, 40, 2], ["2026-10-02", 1, 1, 50, 1]]); + }); +}); + +describe("a month's report", () => { + let nasId = 0; + let checkId = 0; + const audit = (action: string, id: number, at: string) => ({ category: "server", action, targetType: "server", targetId: String(id), createdAt: at }); + const incident = (extra: Partial) => ({ alertKey: "k", severity: "warning", peakSeverity: "warning", category: "disk", source: "Server", message: "a disk", openedAt: "2026-09-02T00:00:00.000Z", lastSeenAt: "2026-09-02T00:00:00.000Z", ...extra }); + + before(async () => { + await updateSettings({ notifications: { timezone: "UTC" } }); + await db.delete(endpointChecks); // the earlier tests left checks of their own, and their daily figures go with them + nasId = (await db.insert(servers).values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", lastSeenAt: "2026-10-01T00:00:00.000Z", createdAt: "2026-01-01 00:00:00" }).returning({ id: servers.id }))[0].id; + const elsaId = (await db.insert(servers).values({ name: "elsa", apiTokenHash: "e", apiTokenPrefix: "e", lastSeenAt: "2026-10-01T00:00:00.000Z", createdAt: "2026-01-01 00:00:00" }).returning({ id: servers.id }))[0].id; + void elsaId; + // nas was offline 13 hours in September, and a night in August that does not count + await db.insert(auditLog).values([audit("went_offline", nasId, "2026-09-10 08:00:00"), audit("back_online", nasId, "2026-09-10 21:00:00"), audit("went_offline", nasId, "2026-08-02 01:00:00"), audit("back_online", nasId, "2026-08-02 05:00:00")]); + await db.insert(alertIncidents).values([ + incident({ message: "nas: / is 96% full", category: "disk", openedAt: "2026-09-03T10:00:00.000Z", resolvedAt: "2026-09-03T12:00:00.000Z", peakSeverity: "critical" }), + incident({ message: "elsa stopped reporting", category: "offline", openedAt: "2026-09-12T10:00:00.000Z", resolvedAt: "2026-09-12T10:30:00.000Z", targetType: "server", targetId: 2, targetName: "elsa" }), + incident({ message: "Backup of anna failed", category: "backup", openedAt: "2026-09-20T00:00:00.000Z" }), + incident({ message: "in August", openedAt: "2026-08-20T00:00:00.000Z", resolvedAt: "2026-08-20T01:00:00.000Z" }), + incident({ message: "began in August, ended in September", openedAt: "2026-08-31T20:00:00.000Z", resolvedAt: "2026-09-01T02:00:00.000Z" }), + ]); + await db.insert(updateRuns).values([ + { serverId: nasId, integrationId: 1, projectId: 1, templateId: 1, templateName: "t", hostLimit: "nas", status: "done", startedAt: "2026-09-05T02:00:00.000Z", updatesBefore: 12, updatesAfter: 2 }, + { serverId: nasId, integrationId: 1, projectId: 1, templateId: 1, templateName: "t", hostLimit: "nas", status: "failed", startedAt: "2026-09-12T02:00:00.000Z", updatesBefore: 3, updatesAfter: null }, + { serverId: nasId, integrationId: 1, projectId: 1, templateId: 1, templateName: "t", hostLimit: "nas", status: "done", startedAt: "2026-10-05T02:00:00.000Z", updatesBefore: 9, updatesAfter: 0 }, + ]); + const [plan] = await db.insert(patchPlans).values({ name: "Lab", days: "[0]", startTime: "02:00", timezone: "UTC", integrationId: 1, projectId: 1, templateId: 1, templateName: "t" }).returning(); + await db.insert(patchPlanRuns).values([ + { planId: plan.id, planName: "Lab", trigger: "schedule", status: "done", startedAt: "2026-09-06T02:00:00.000Z" }, + { planId: plan.id, planName: "Lab", trigger: "schedule", status: "failed", startedAt: "2026-09-13T02:00:00.000Z" }, + ]); + await db.insert(restoreTests).values([{ subjectKey: "custom:photos", label: "Photos share", testedAt: "2026-09-14", result: "ok" }, { subjectKey: "custom:old", label: "Old", testedAt: "2026-08-14", result: "ok" }]); + await db.insert(renewals).values([{ kind: "certificate", subject: "git.lab", at: "2026-09-12T03:00:00.000Z", fromDate: "2026-09-14", toDate: "2026-12-13" }, { kind: "domain", subject: "example.se", at: "2026-10-03T03:00:00.000Z", fromDate: "2026-10-04", toDate: "2027-10-04" }]); + checkId = (await db.insert(endpointChecks).values({ name: "Gitea", kind: "http", target: "https://git.lab/" }).returning({ id: endpointChecks.id }))[0].id; + await db.insert(endpointDaily).values([ + { checkId, day: "2026-09-01", total: 100, ok: 100, msSum: 10_000, msCount: 100 }, + { checkId, day: "2026-09-30", total: 100, ok: 90, msSum: 9_000, msCount: 90 }, + { checkId, day: "2026-08-31", total: 100, ok: 0 }, + { checkId, day: "2026-10-01", total: 100, ok: 0 }, + ]); + }); + + const NOW = new Date("2026-10-19T12:00:00Z"); + + test("availability: what was down and for how long, from what the audit log recorded; the neighbouring months are not in it", async () => { + const r = await buildMonthlyReport("2026-09", NOW); + assert.deepEqual([r.month, r.label, r.partial, r.timezone], ["2026-09", "September 2026", false, "UTC"]); + const nas = r.servers.find((s) => s.name === "nas")!; + assert.equal(nas.downMinutes, 13 * 60); + assert.equal(nas.percent, 98.2, "13 of 720 hours down"); + assert.equal(r.servers[0].name, "nas", "the worst first"); + assert.deepEqual(r.endpoints.map((e) => [e.name, e.percent, e.avgMs, e.checks]), [["Gitea", 95, 100, 200]], "the first and last day of the month, not the days either side"); + }); + + test("incidents: how many began and ended, the time they took, the longest, by kind", async () => { + const i = (await buildMonthlyReport("2026-09", NOW)).incidents; + assert.deepEqual([i.began, i.resolved, i.stillOpen], [3, 3, 1], "the open one is still open; the August one that ended in September ended in it"); + assert.equal(i.totalMinutes, 150, "2 h and 30 min, of those that began and ended in the month"); + assert.equal(i.averageMinutes, 75); + assert.equal(i.longest!.message, "Backup of anna failed", "still open, so it has been going the longest"); + assert.deepEqual(i.byCategory, [{ category: "backup", count: 1 }, { category: "disk", count: 1 }, { category: "offline", count: 1 }]); + assert.deepEqual(i.list.map((x) => x.message), ["Backup of anna failed", "nas: / is 96% full", "elsa stopped reporting"]); + assert.equal(i.list[2].where, "elsa"); + }); + + test("updates, patch plans, restore tests and renewals for that month only", async () => { + const r = await buildMonthlyReport("2026-09", NOW); + assert.deepEqual([r.updates.runs, r.updates.done, r.updates.failed, r.updates.installed], [2, 1, 1, 10]); + assert.deepEqual(r.updates.byServer, [{ name: "nas", runs: 2, failed: 1, installed: 10 }]); + assert.deepEqual(r.updates.planRuns, { runs: 2, done: 1, failed: 1 }); + assert.deepEqual(r.restoreTests.map((t) => t.label), ["Photos share"]); + assert.deepEqual(r.renewals.map((x) => [x.kind, x.subject]), [["certificate", "git.lab"]]); + assert.equal(r.backups, null, "no backup coverage has been read here"); + const oct = await buildMonthlyReport("2026-10", NOW); + assert.equal(oct.partial, true, "October is not over"); + assert.deepEqual(oct.renewals.map((x) => x.subject), ["example.se"]); + assert.equal(oct.updates.runs, 1); + }); + + test("as a message: a section only where there is something to say, with the numbers that matter", async () => { + const { title, message } = monthlyReportText(await buildMonthlyReport("2026-09", NOW)); + assert.equal(title, "Homelab monthly report — September 2026"); + assert.match(message, /^AVAILABILITY\nServers: 9\d\.\d% on average \(2 servers\)\.\n- nas: 9\d\.\d% \(13 h down\)/); + assert.match(message, /Endpoints: 95% on average \(1 check\)\.\n- Gitea: 95%, 100 ms on average/); + assert.match(message, /INCIDENTS\n3 incidents began, 3 ended, 1 still open\.\nA resolved incident took 1 h 15 min on average\./); + assert.match(message, /- Backup of anna failed — 11 d and still open/); + assert.match(message, /UPDATES\n2 update runs: 1 done, 1 failed — 10 updates installed\.\nPatch plans ran 2 times, 1 not cleanly\./); + assert.match(message, /BACKUPS\n- Restore test of Photos share \(2026-09-14\) passed/); + assert.match(message, /RENEWED\n- Certificate git\.lab: 2026-09-14 → 2026-12-13/); + const partial = monthlyReportText(await buildMonthlyReport("2026-10", NOW)); + assert.equal(partial.title, "Homelab monthly report — October 2026 (so far)"); + const empty = monthlyReportText(await buildMonthlyReport("2025-01", NOW)); + assert.equal(empty.message.includes("Nothing was recorded") || !/INCIDENTS\n[1-9]/.test(empty.message), true); + }); +}); + +describe("through the routes", () => { + let app: Server; + let hook: Server; + let base = ""; + let who = "viewer"; + const messages: string[] = []; + + before(async () => { + await db.insert(users).values([ + { oidcSub: "viewer", name: "Vic", role: "viewer" }, + { oidcSub: "admin", name: "Ada", role: "admin" }, + ]); + hook = createHttpServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + messages.push(String(JSON.parse(Buffer.concat(chunks).toString()).content)); + res.end("{}"); + }); + }); + await new Promise((r) => hook.listen(0, "127.0.0.1", r)); + await updateSettings({ webhook: { enabled: true, url: `http://127.0.0.1:${(hook.address() as AddressInfo).port}/` } }); + const a = express(); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: who } }; + next(); + }); + a.use("/api/reports", reportsRouter); + app = await new Promise((resolve) => { + const s = a.listen(0, "127.0.0.1", () => resolve(s)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/reports`; + }); + + after(() => { + app.close(); + hook.close(); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } + }); + + test("anyone signed in can read it: the last whole month by default, the twelve to choose from, or one of them; nothing else is a month", async () => { + const body = await (await fetch(`${base}/monthly`)).json(); + assert.equal(body.months.length, 12); + assert.equal(body.report.month, body.months[1].id, "last month, not this one"); + assert.equal((await (await fetch(`${base}/monthly?month=${body.months[0].id}`)).json()).report.partial, true); + for (const bad of ["2026-13", "2019-01", "abc", "2026-9"]) assert.equal((await fetch(`${base}/monthly?month=${bad}`)).status, 400, bad); + const text = await (await fetch(`${base}/monthly?format=text`)).json(); + assert.match(text.title, /^Homelab monthly report — /); + assert.equal(typeof text.message, "string"); + }); + + test("an administrator can send it now to the channels; nobody else can", async () => { + assert.equal((await fetch(`${base}/monthly/send`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}" })).status, 403); + who = "admin"; + const res = await fetch(`${base}/monthly/send`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}" }); + assert.equal(res.status, 200); + assert.equal(messages.length, 1); + assert.equal((await fetch(`${base}/monthly/send`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ month: "nope" }) })).status, 400); + assert.equal((await db.select().from(auditLog).where(eq(auditLog.action, "send_monthly"))).length, 1); + }); +}); diff --git a/server/test/newPorts.test.ts b/server/test/newPorts.test.ts new file mode 100644 index 0000000..ed03056 --- /dev/null +++ b/server/test/newPorts.test.ts @@ -0,0 +1,187 @@ +/** + * New ports: what counts as one, that the first report is the baseline, and what the health check, the Alerts page, maintenance, routing and + * the server's Ports card do with them. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import type { Server } from "node:http"; + +const dir = mkdtempSync(join(tmpdir(), "hm-newports-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers, serverPorts } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const { newListeningPorts, baselineFrom, parseBaseline, newPortMessage } = await import("../src/services/newPorts.js"); +const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js"); +const { syncServerTasks } = await import("../src/services/taskSync.js"); +const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js"); +const { startOrExtendWindow } = await import("../src/services/maintenance.js"); +const { metaOfConditionKey } = await import("../src/services/notifyRouting.js"); +const { subjectOfConditionKey } = await import("../src/services/maintenance.js"); +const { serverPortsRouter } = await import("../src/routes/serverPorts.js"); +const { updateSettings } = await import("../src/services/settingsStore.js"); +const { eq } = await import("drizzle-orm"); + +await runMigrations(); + +const sock = (port: number, extra: Partial<{ protocol: "tcp" | "udp"; address: string; process: string }> = {}) => ({ protocol: "tcp" as const, port, address: "0.0.0.0", ...extra }); +const none = new Set(); + +describe("what counts as a new port", () => { + const baseline = parseBaseline(baselineFrom([sock(22), sock(443), sock(53, { protocol: "udp" })])); + + test("the baseline is everything the first report listened on, in any protocol", () => { + assert.deepEqual([...baseline!].sort(), ["tcp:22", "tcp:443", "udp:53"]); + assert.equal(parseBaseline(null), null); + assert.equal(parseBaseline("not json"), null); + assert.equal(parseBaseline(JSON.stringify({ not: "a list" })), null); + }); + + test("with no baseline yet, nothing is new", () => { + assert.deepEqual(newListeningPorts([sock(8080)], null, none), []); + assert.deepEqual(newListeningPorts(null, baseline, none), []); + }); + + test("a TCP port that is listening and was not in the baseline is new, with what is listening on it", () => { + const found = newListeningPorts([sock(22), sock(8080, { process: "node" }), sock(8080, { address: "::", process: "" }), sock(3000)], baseline, none); + assert.deepEqual(found, [{ protocol: "tcp", port: 3000, process: null }, { protocol: "tcp", port: 8080, process: "node" }]); + }); + + test("not UDP, not the dynamic range, not a port bound to this machine alone", () => { + const found = newListeningPorts([sock(5353, { protocol: "udp" }), sock(49152), sock(60000), sock(9000, { address: "127.0.0.1" }), sock(9001, { address: "::1" }), sock(9002, { address: "[::1]" }), sock(9003, { address: "127.0.0.1" }), sock(9003, { address: "10.0.0.5" })], baseline, none); + assert.deepEqual(found.map((p) => p.port), [9003], "one socket on a real address is enough"); + assert.deepEqual(newListeningPorts([sock(49151)], baseline, none).map((p) => p.port), [49151], "the last port before the dynamic range is still a service"); + }); + + test("a port somebody noted as expected is not new", () => { + assert.deepEqual(newListeningPorts([sock(8080), sock(9090)], baseline, new Set(["tcp:8080"])).map((p) => p.port), [9090]); + }); + + test("the message says what it is and what to do", () => { + assert.equal(newPortMessage("nas", { protocol: "tcp", port: 8080, process: "node" }), "nas: a new port is listening — 8080/tcp (node). If it should be there, note it on the server's Ports card."); + }); +}); + +describe("the health check, the alert and the routing", () => { + const snap = (extra: object) => ({ servers: [{ id: 3, name: "nas", lastSeenAt: new Date().toISOString(), disks: [], ...extra }], proxmox: [], synology: [] }); + const limits = { serverOfflineMinutes: 60, diskUsagePercent: 90 }; + + test("each new port is a warning keyed by server and port; an offline server's are not judged", () => { + const list = evaluateHealth(snap({ newPorts: [{ protocol: "tcp", port: 8080, process: "node" }, { protocol: "tcp", port: 9090, process: null }] }), limits, Date.now()); + assert.deepEqual(list.map((c) => [c.key, c.severity, c.source]), [["new-port:server:3:tcp:8080", "warning", "server"], ["new-port:server:3:tcp:9090", "warning", "server"]]); + assert.match(list[0].message, /nas: a new port is listening — 8080\/tcp \(node\)/); + const offline = evaluateHealth(snap({ lastSeenAt: new Date(Date.now() - 5 * 3600_000).toISOString(), newPorts: [{ protocol: "tcp", port: 8080, process: null }] }), limits, Date.now()); + assert.equal(offline.some((c) => c.key.startsWith("new-port")), false); + }); + + test("maintenance on the server holds it back, and routing treats it as monitoring with that server's tags", () => { + assert.equal(subjectOfConditionKey("new-port:server:3:tcp:8080"), "server:3"); + assert.deepEqual(metaOfConditionKey("new-port:server:3:tcp:8080", "warning"), { category: "monitoring", severity: "warning", serverId: 3 }); + }); +}); + +describe("from a report to the Ports card", () => { + let app: Server; + let base = ""; + let serverId = 0; + let who = "operator"; + const report = (ports: ReturnType[]) => ({ tasks: [], system: { listening_ports: ports } }); + + before(async () => { + await db.insert(users).values([ + { oidcSub: "operator", name: "Olga", role: "operator" }, + { oidcSub: "viewer", name: "Vic", role: "viewer" }, + ]); + serverId = (await db.insert(servers).values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", lastSeenAt: new Date().toISOString() }).returning({ id: servers.id }))[0].id; + const a = express(); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown; currentUser: unknown }).session = { user: { sub: who } }; + next(); + }); + // The real servers router puts a signed-in user on the request before this; here the role is read the same way the middleware does. + const { requireAuth } = await import("../src/auth/middleware.js"); + a.use("/api/servers/:id/ports", requireAuth, serverPortsRouter); + app = await new Promise((resolve) => { + const s = a.listen(0, "127.0.0.1", () => resolve(s)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/servers/${serverId}/ports`; + }); + + after(() => { + app.close(); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } + }); + + const newKeys = async () => (await collectSnapshot()).snapshot.servers.find((s) => s.id === serverId)!.newPorts!.map((p) => p.port); + + test("the first report that carries ports is the baseline: nothing in it is new, and it is never changed by a later one", async () => { + await syncServerTasks(serverId, report([sock(22), sock(443)])); + let [row] = await db.select().from(servers).where(eq(servers.id, serverId)); + assert.deepEqual(JSON.parse(row.portBaseline!), ["tcp:22", "tcp:443"]); + assert.deepEqual(await newKeys(), []); + await syncServerTasks(serverId, report([sock(22), sock(443), sock(8080, { process: "node" })])); + [row] = await db.select().from(servers).where(eq(servers.id, serverId)); + assert.deepEqual(JSON.parse(row.portBaseline!), ["tcp:22", "tcp:443"], "8080 is not folded into what is normal"); + assert.deepEqual(await newKeys(), [8080]); + }); + + test("a report with no ports at all does not start a baseline", async () => { + const [other] = await db.insert(servers).values({ name: "bare", apiTokenHash: "b", apiTokenPrefix: "b" }).returning({ id: servers.id }); + await syncServerTasks(other.id, { tasks: [], system: {} }); + const [row] = await db.select().from(servers).where(eq(servers.id, other.id)); + assert.equal(row.portBaseline, null); + }); + + test("it is an alert on the Alerts page, in monitoring, linked to the server — and silenced by maintenance on it", async () => { + invalidateAlertsCache(); + const alert = (await getAlerts(true)).alerts.find((a) => a.message.includes("a new port is listening"))!; + assert.ok(alert, "the alert is there"); + assert.deepEqual([alert.category, alert.severity, alert.silenced], ["monitoring", "warning", false]); + assert.deepEqual(alert.target && [alert.target.type, alert.target.id], ["server", serverId]); + await startOrExtendWindow("server", serverId, 30, "work", "Ada"); + invalidateAlertsCache(); + assert.equal((await getAlerts(true)).alerts.find((a) => a.message.includes("a new port is listening"))!.silenced, true); + }); + + test("the Ports card marks it new, and noting it as expected ends the alert; a scan-only row does not", async () => { + who = "viewer"; + let list = (await (await fetch(base)).json()).ports as { port: number; isNew: boolean }[]; + assert.deepEqual(list.filter((p) => p.isNew).map((p) => p.port), [8080]); + await db.insert(serverPorts).values({ serverId, port: 8080, protocol: "tcp", open: true }); + assert.deepEqual(await newKeys(), [8080], "a row that only records an open scan says nothing"); + await db.delete(serverPorts).where(eq(serverPorts.port, 8080)); + + who = "operator"; + const res = await fetch(base, { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ port: 8080, protocol: "tcp", label: "Expected" }) }); + assert.equal(res.status, 200); + list = (await res.json()).ports; + assert.equal(list.find((p) => p.port === 8080)!.isNew, false); + assert.deepEqual(await newKeys(), []); + }); + + test("it can be switched off, and then nothing is new", async () => { + await syncServerTasks(serverId, report([sock(22), sock(443), sock(8080), sock(9999)])); + assert.deepEqual(await newKeys(), [9999]); + await updateSettings({ healthChecks: { newPortAlerts: false } }); + assert.deepEqual(await newKeys(), []); + who = "viewer"; + assert.equal(((await (await fetch(base)).json()).ports as { isNew: boolean }[]).some((p) => p.isNew), false); + await updateSettings({ healthChecks: { newPortAlerts: true } }); + }); +}); diff --git a/server/test/patchPlans.test.ts b/server/test/patchPlans.test.ts new file mode 100644 index 0000000..b7c3bec --- /dev/null +++ b/server/test/patchPlans.test.ts @@ -0,0 +1,407 @@ +/** + * Patch plans: who a plan covers, when it is due, that it goes one server at a time and waits for each, what it does when one fails, what it silences, + * the message at the end, and who may do what. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { createServer as createHttpServer, type Server } from "node:http"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; + +const dir = mkdtempSync(join(tmpdir(), "hm-patchplans-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers, patchPlans, patchPlanRuns, updateRuns, maintenanceWindows } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const { planServers, dueOccurrence, summariseItems, startPlanRun, advancePlanRuns, startDuePlans, cancelPlanRun, parseItems, PlanError } = await import("../src/services/patchPlans.js"); +const { advanceUpdateRuns, setSemaphoreForTests } = await import("../src/services/updateRuns.js"); +const { listActiveWindows } = await import("../src/services/maintenance.js"); +const { agendaFromData } = await import("../src/services/agenda.js"); +const { patchPlansRouter } = await import("../src/routes/patchPlans.js"); +const { updateSettings } = await import("../src/services/settingsStore.js"); +const { eq } = await import("drizzle-orm"); + +await runMigrations(); + +// ── a Semaphore that answers as told ── +const fake = { runs: [] as { limit?: string }[], task: { id: 41, status: "waiting", created: null as string | null, start: null as string | null, end: null as string | null }, refuseFor: new Set() }; +setSemaphoreForTests(async (integrationId) => + integrationId === 9 + ? { + ping: async () => ({ ok: true }), + listTemplatesWithStatus: async () => [], + checkTemplates: async () => ({ templates: [{ id: 7, projectId: 2, projectName: "Homelab", name: "Apply updates", playbook: "u.yml", lastTask: null }], failedProjectIds: [] }), + runTemplate: async (_p: number, _t: number, options?: { limit?: string }) => { + if (options?.limit && fake.refuseFor.has(options.limit)) throw new Error("Semaphore is having a moment"); + fake.runs.push({ limit: options?.limit }); + return { ...fake.task } as never; + }, + getTask: async () => ({ ...fake.task }) as never, + } + : null, +); + +const T0 = new Date("2026-10-18T02:05:00.000Z"); // a Sunday +const at = (m: number) => new Date(T0.getTime() + m * 60_000); + +describe("who a plan covers, and when it is due", () => { + const all = [ + { id: 1, name: "web", tags: JSON.stringify(["lab"]) }, + { id: 2, name: "api", tags: JSON.stringify(["prod"]) }, + { id: 3, name: "ångström", tags: JSON.stringify(["lab", "prod"]) }, + { id: 4, name: "db", tags: null }, + ]; + + test("servers with any of the tags, and those named, by name; each only once", () => { + assert.deepEqual(planServers({ tags: JSON.stringify(["lab"]), serverIds: "[]" }, all).map((s) => s.name), ["ångström", "web"].sort((x, y) => x.localeCompare(y))); + assert.deepEqual(planServers({ tags: JSON.stringify(["lab"]), serverIds: JSON.stringify([4, 3]) }, all).map((s) => s.name), ["db", "ångström", "web"].sort((x, y) => x.localeCompare(y)), "in the order the app sorts names"); + assert.deepEqual(planServers({ tags: "[]", serverIds: "[]" }, all), [], "a plan that names no one covers no one"); + }); + + const plan = { enabled: true, days: JSON.stringify([0]), startTime: "02:00", timezone: "UTC", lastRunDate: null as string | null }; + + test("due from its time to half an hour after, once a day, only on its days", () => { + assert.equal(dueOccurrence(plan, new Date("2026-10-18T01:59:00Z")), null, "not yet"); + assert.equal(dueOccurrence(plan, new Date("2026-10-18T02:00:00Z"))?.date, "2026-10-18"); + assert.equal(dueOccurrence(plan, new Date("2026-10-18T02:29:00Z"))?.date, "2026-10-18", "the app may have been down for a while"); + assert.equal(dueOccurrence(plan, new Date("2026-10-18T02:31:00Z")), null, "too late: that day is skipped"); + assert.equal(dueOccurrence({ ...plan, lastRunDate: "2026-10-18" }, new Date("2026-10-18T02:10:00Z")), null, "already started today"); + assert.equal(dueOccurrence(plan, new Date("2026-10-19T02:10:00Z")), null, "Monday is not a day it runs"); + assert.equal(dueOccurrence({ ...plan, enabled: false }, new Date("2026-10-18T02:10:00Z")), null); + }); + + test("its time is on the clock of its time zone, winter and summer", () => { + const stockholm = { ...plan, timezone: "Europe/Stockholm" }; + assert.equal(dueOccurrence(stockholm, new Date("2026-10-18T00:05:00Z"))?.date, "2026-10-18", "02:05 in Stockholm is 00:05 UTC while summer time lasts"); + assert.equal(dueOccurrence(stockholm, new Date("2026-11-01T01:05:00Z"))?.date, "2026-11-01", "and 01:05 UTC after the clocks go back"); + assert.equal(dueOccurrence(stockholm, new Date("2026-11-01T00:05:00Z")), null); + }); + + test("a finished run says how each went in one line", () => { + const item = (status: string) => ({ serverId: 1, name: "x", status, updateRunId: null, message: null, windowIds: [] }) as never; + assert.equal(summariseItems([item("done"), item("done"), item("failed"), item("skipped")]), "2 updated, 1 failed, 1 left alone of 4 servers."); + assert.equal(summariseItems([item("done")]), "1 updated of 1 server."); + }); + + test("a plan is in the agenda as maintenance, with how many servers it updates", () => { + const data = { secrets: [], domains: [], endpoints: [], tailscale: [], assets: [], windows: [], schedules: [], restoreTests: [], restoreLimitDays: 0, updates: [], notes: [], patchPlans: [{ id: 5, name: "Lab nights", days: JSON.stringify([0]), startTime: "02:00", timezone: "UTC", servers: 3 }] }; + const items = agendaFromData(data, new Date("2026-10-19T12:00:00Z"), "UTC", 20).items.filter((i) => i.id.startsWith("plan:")); + assert.deepEqual(items.map((i) => [i.title, i.detail, i.at.slice(0, 16), i.kind]), [["Patch plan: Lab nights", "updates 3 servers, one at a time", "2026-10-25T02:00", "maintenance"], ["Patch plan: Lab nights", "updates 3 servers, one at a time", "2026-11-01T02:00", "maintenance"], ["Patch plan: Lab nights", "updates 3 servers, one at a time", "2026-11-08T02:00", "maintenance"]]); + }); +}); + +describe("a run, from start to end", () => { + let hook: Server; + const messages: string[] = []; + const ids: Record = {}; + let planId = 0; + + const plan = (extra: Partial = {}) => ({ + name: "Lab nights", days: JSON.stringify([0]), startTime: "02:00", timezone: "UTC", tags: JSON.stringify(["lab"]), serverIds: "[]", integrationId: 9, projectId: 2, templateId: 7, templateName: "Apply updates", limitFrom: "name", stopOnFailure: true, onlyWithUpdates: true, silenceDependents: true, windowMinutes: 90, ...extra, + }); + const updates = (count: number, reportedAt = T0.toISOString()) => JSON.stringify({ manager: "apt", count, security: 0, rebootRequired: false, reportedAt }); + const runOf = async (id: number) => (await db.select().from(patchPlanRuns).where(eq(patchPlanRuns.id, id)))[0]; + const itemsOf = async (id: number) => parseItems((await runOf(id)).items); + + /** The playbook for the server being updated finishes and the server reports again — as the update run follows. */ + async function finishCurrent(server: string, minute: number) { + fake.task = { id: 41, status: "success", created: null, start: null, end: at(minute).toISOString() }; + await advanceUpdateRuns(at(minute + 1)); + await db.update(servers).set({ lastSeenAt: at(minute + 2).toISOString(), pendingUpdates: updates(0, at(minute + 2).toISOString()) }).where(eq(servers.id, ids[server])); + await advanceUpdateRuns(at(minute + 3)); + fake.task = { id: 41, status: "waiting", created: null, start: null, end: null }; + } + + async function reset(extra: Partial = {}) { + await db.delete(patchPlans); + await db.delete(updateRuns); + await db.delete(maintenanceWindows); // the tests share one clock, so a window left by the last would still be open + fake.runs = []; + fake.refuseFor.clear(); + fake.task = { id: 41, status: "waiting", created: null, start: null, end: null }; + for (const name of ["web", "api"]) await db.update(servers).set({ lastSeenAt: T0.toISOString(), pendingUpdates: updates(4) }).where(eq(servers.id, ids[name])); + await db.update(servers).set({ lastSeenAt: T0.toISOString(), pendingUpdates: updates(0) }).where(eq(servers.id, ids.idle)); + messages.length = 0; + planId = (await db.insert(patchPlans).values(plan(extra)).returning({ id: patchPlans.id }))[0].id; + } + + before(async () => { + for (const name of ["web", "api", "idle"]) { + ids[name] = (await db.insert(servers).values({ name, apiTokenHash: name, apiTokenPrefix: name, tags: JSON.stringify(["lab"]), lastSeenAt: T0.toISOString() }).returning({ id: servers.id }))[0].id; + } + ids.prod = (await db.insert(servers).values({ name: "prod", apiTokenHash: "p", apiTokenPrefix: "p", tags: JSON.stringify(["prod"]), lastSeenAt: T0.toISOString(), pendingUpdates: updates(9) }).returning({ id: servers.id }))[0].id; + // a guest that runs on web: what depends on web + ids.guest = (await db.insert(servers).values({ name: "guest", apiTokenHash: "g", apiTokenPrefix: "g", lastSeenAt: T0.toISOString(), parentServerId: ids.web }).returning({ id: servers.id }))[0].id; + hook = createHttpServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + messages.push(String(JSON.parse(Buffer.concat(chunks).toString()).content)); + res.end("{}"); + }); + }); + await new Promise((r) => hook.listen(0, "127.0.0.1", r)); + await updateSettings({ webhook: { enabled: true, url: `http://127.0.0.1:${(hook.address() as AddressInfo).port}/` } }); + }); + + after(() => { + hook.close(); + setSemaphoreForTests(null); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } + }); + + test("one server at a time, each waited for, those with nothing waiting left alone, and a message at the end", async () => { + await reset(); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + assert.deepEqual((await itemsOf(run.id)).map((i) => [i.name, i.status]), [["api", "pending"], ["idle", "pending"], ["web", "pending"]], "by name"); + + await advancePlanRuns(T0); + assert.deepEqual((await itemsOf(run.id)).map((i) => i.status), ["running", "pending", "pending"]); + assert.deepEqual(fake.runs.map((r) => r.limit), ["api"], "only the first has been started"); + assert.equal((await listActiveWindows(T0)).some((w) => w.targetType === "server" && w.targetId === ids.api), true, "and it is under maintenance"); + + await advancePlanRuns(at(1)); + assert.equal(fake.runs.length, 1, "while it is still running, nothing else starts"); + + await finishCurrent("api", 5); + await advancePlanRuns(at(9)); + assert.deepEqual((await itemsOf(run.id)).map((i) => i.status), ["done", "skipped", "running"], "idle has nothing waiting, so it is passed over, and web is next"); + assert.deepEqual(fake.runs.map((r) => r.limit), ["api", "web"]); + assert.equal((await itemsOf(run.id))[1].message, "No updates were waiting."); + assert.equal((await listActiveWindows(at(9))).some((w) => w.targetId === ids.api), false, "api's maintenance ended with its update"); + + await finishCurrent("web", 15); + await advancePlanRuns(at(19)); + const done = await runOf(run.id); + assert.equal(done.status, "done"); + assert.equal(done.message, "2 updated, 1 left alone of 3 servers."); + assert.equal(messages.length >= 3, true, "each update run says how it went, and so does the plan"); + const last = messages[messages.length - 1]; + assert.match(last, /2 updated, 1 left alone of 3 servers\./); + assert.match(last, /✓ api/); + assert.match(last, /– idle — No updates were waiting\./); + assert.match(last, /✓ web/); + }); + + test("what depends on a server is silenced while it is updated, and let go when it is done", async () => { + await reset({ tags: "[]", serverIds: JSON.stringify([ids.web]) }); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + const during = await listActiveWindows(T0); + assert.equal(during.some((w) => w.targetId === ids.guest), true, "the guest that runs on web"); + assert.match(during.find((w) => w.targetId === ids.guest)!.reason ?? "", /Depends on web — patch plan "Lab nights"/); + assert.equal((await itemsOf(run.id))[0].windowIds.length, 1); + await finishCurrent("web", 5); + await advancePlanRuns(at(9)); + assert.equal((await listActiveWindows(at(9))).some((w) => w.targetId === ids.guest), false); + assert.equal((await runOf(run.id)).status, "done"); + }); + + test("it can be told not to silence what depends on a server", async () => { + await reset({ tags: "[]", serverIds: JSON.stringify([ids.web]), silenceDependents: false }); + await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + assert.equal((await listActiveWindows(T0)).some((w) => w.targetId === ids.guest), false); + }); + + test("a failure stops the plan and leaves the rest alone, saying where it stopped", async () => { + await reset(); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + fake.task = { id: 41, status: "error", created: null, start: null, end: null }; + await advanceUpdateRuns(at(2)); + await advancePlanRuns(at(3)); + const after = await runOf(run.id); + assert.equal(after.status, "failed"); + assert.deepEqual((await itemsOf(run.id)).map((i) => [i.name, i.status]), [["api", "failed"], ["idle", "skipped"], ["web", "skipped"]]); + assert.equal((await itemsOf(run.id))[2].message, "Not started, because an earlier server failed."); + assert.match(after.message ?? "", /1 failed, 2 left alone of 3 servers\. Stopped after api\./); + assert.deepEqual(fake.runs.map((r) => r.limit), ["api"], "web was never touched"); + assert.match(messages[messages.length - 1], /✕ api/); + assert.equal(messages.some((m) => m.includes("didn't finish cleanly")), true); + }); + + test("told to carry on, it does: the failed one is reported and the rest are updated", async () => { + await reset({ stopOnFailure: false }); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + fake.task = { id: 41, status: "error", created: null, start: null, end: null }; + await advanceUpdateRuns(at(2)); + fake.task = { id: 41, status: "waiting", created: null, start: null, end: null }; + await advancePlanRuns(at(3)); + assert.deepEqual((await itemsOf(run.id)).map((i) => i.status), ["failed", "skipped", "running"], "web is next"); + await finishCurrent("web", 10); + await advancePlanRuns(at(14)); + const done = await runOf(run.id); + assert.equal(done.status, "failed", "something failed, so the run says so"); + assert.equal(done.message, "1 updated, 1 failed, 1 left alone of 3 servers."); + }); + + test("a server that can't be started counts as a failure; a server with nothing to update can be updated all the same if asked", async () => { + await reset({ onlyWithUpdates: false, tags: "[]", serverIds: JSON.stringify([ids.idle]) }); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + assert.deepEqual(fake.runs.map((r) => r.limit), ["idle"], "it had nothing waiting but the plan was told not to mind"); + await updateRunsCancelAll(); + await cancelPlanRun(run.id, "Olle", at(1)); + + await reset({ tags: "[]", serverIds: JSON.stringify([ids.web]) }); + fake.refuseFor.add("web"); + const refused = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + const items = await itemsOf(refused.id); + assert.equal(items[0].status, "failed"); + assert.match(items[0].message ?? "", /Semaphore wouldn't start it: Semaphore is having a moment/); + assert.equal((await runOf(refused.id)).status, "failed"); + }); + + async function updateRunsCancelAll() { + await db.update(updateRuns).set({ status: "cancelled" }); + } + + test("a plan does not overlap itself, and one with no servers says so rather than doing nothing silently", async () => { + await reset(); + await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await assert.rejects(startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }), (e: unknown) => e instanceof PlanError && e.code === "already_running"); + await db.update(patchPlanRuns).set({ status: "cancelled" }); + + await db.update(patchPlans).set({ tags: JSON.stringify(["nothing-has-this"]) }).where(eq(patchPlans.id, planId)); + await assert.rejects(startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }), (e: unknown) => e instanceof PlanError && e.code === "no_servers"); + }); + + test("stopping a run that is going leaves the rest alone, ends the maintenance it made and stops the update run being followed", async () => { + await reset(); + const run = await startPlanRun(planId, { trigger: "manual", actor: "Olle", now: T0 }); + await advancePlanRuns(T0); + const stopped = await cancelPlanRun(run.id, "Olga", at(2)); + assert.equal(stopped!.status, "cancelled"); + assert.deepEqual(parseItems(stopped!.items).map((i) => i.status), ["failed", "skipped", "skipped"]); + assert.equal((await listActiveWindows(at(2))).some((w) => w.targetId === ids.api), false); + assert.equal((await db.select().from(updateRuns))[0].status, "cancelled"); + await assert.rejects(cancelPlanRun(run.id, "Olga", at(3)), (e: unknown) => e instanceof PlanError && e.code === "not_active"); + assert.equal(await cancelPlanRun(99999, "Olga"), null); + }); + + test("the schedule starts a plan once on its day, and a restart of the app does not start it again", async () => { + await reset(); + assert.equal(await startDuePlans(T0), 1); + const [row] = await db.select().from(patchPlans).where(eq(patchPlans.id, planId)); + assert.equal(row.lastRunDate, "2026-10-18"); + assert.equal((await db.select().from(patchPlanRuns))[0].trigger, "schedule"); + await db.update(patchPlanRuns).set({ status: "done" }); + assert.equal(await startDuePlans(at(10)), 0, "not twice in a day"); + assert.equal(await startDuePlans(new Date("2026-10-25T02:03:00Z")), 1, "but the next Sunday it does"); + }); + + test("a plan that is due but has no servers leaves a failed run to say so", async () => { + await reset({ tags: JSON.stringify(["nothing-has-this"]) }); + await db.delete(patchPlanRuns); + assert.equal(await startDuePlans(T0), 0); + const [run] = await db.select().from(patchPlanRuns); + assert.deepEqual([run.status, run.trigger], ["failed", "schedule"]); + assert.match(run.message ?? "", /^Didn't start: No server has this plan's tags/); + }); +}); + +describe("through the routes", () => { + let app: Server; + let base = ""; + let who = "admin"; + const body = (extra: object = {}) => ({ name: "Weekend patching", days: [6, 0], startTime: "03:30", timezone: "Europe/Stockholm", tags: ["Lab"], serverIds: [], integrationId: 9, projectId: 2, templateId: 7, templateName: "Apply updates", ...extra }); + + before(async () => { + await db.insert(users).values([ + { oidcSub: "admin", name: "Ada", role: "admin" }, + { oidcSub: "operator", name: "Olga", role: "operator" }, + { oidcSub: "viewer", name: "Vic", role: "viewer" }, + ]); + const a = express(); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: who } }; + next(); + }); + a.use("/api/patch-plans", patchPlansRouter); + app = await new Promise((resolve) => { + const s = a.listen(0, "127.0.0.1", () => resolve(s)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/patch-plans`; + await db.delete(patchPlans); + await db.delete(patchPlanRuns); + }); + + after(() => app.close()); + + const send = (method: string, path: string, b?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: b === undefined ? undefined : JSON.stringify(b) }); + let id = 0; + + test("an administrator makes a plan, with its tags written the way tags are; a wrong one is refused with a reason; nobody else can", async () => { + const res = await send("POST", "/", body()); + assert.equal(res.status, 201); + const plan = (await res.json()).plan; + id = plan.id; + assert.deepEqual([plan.tags, plan.days, plan.limitFrom, plan.stopOnFailure, plan.onlyWithUpdates, plan.windowMinutes], [["lab"], [0, 6], "name", true, true, 90]); + assert.deepEqual(plan.servers.map((s: { name: string }) => s.name), ["api", "idle", "web"], "the three tagged lab"); + assert.equal(plan.next.length, 3, "its next times, so nobody has to work out when"); + for (const bad of [{ days: [] }, { startTime: "3:30" }, { timezone: "Mars/Olympus" }, { tags: [], serverIds: [] }, { tags: ["bad tag!"] }, { windowMinutes: 5 }, { name: " " }, { integrationId: 0 }]) { + const r = await send("POST", "/", body(bad)); + assert.equal(r.status, 400, JSON.stringify(bad)); + assert.ok((await r.json()).message.length > 0); + } + who = "operator"; + assert.equal((await send("POST", "/", body())).status, 403); + assert.equal((await send("PUT", `/${id}`, body())).status, 403); + assert.equal((await send("DELETE", `/${id}`)).status, 403); + who = "viewer"; + const list = await (await send("GET", "/")).json(); + assert.equal(list.plans.length, 1, "everyone can see what is planned"); + assert.ok(list.tags.includes("lab") && list.servers.length >= 4); + }); + + test("changing when it happens forgets which day it last started on; changing anything else does not", async () => { + who = "admin"; + await db.update(patchPlans).set({ lastRunDate: "2026-10-18" }).where(eq(patchPlans.id, id)); + await send("PUT", `/${id}`, body({ name: "Renamed" })); + assert.equal((await db.select().from(patchPlans).where(eq(patchPlans.id, id)))[0].lastRunDate, "2026-10-18"); + await send("PUT", `/${id}`, body({ name: "Renamed", startTime: "04:00" })); + assert.equal((await db.select().from(patchPlans).where(eq(patchPlans.id, id)))[0].lastRunDate, null); + assert.equal((await send("PUT", "/99999", body())).status, 404); + }); + + test("an operator can run it now and stop it; a viewer cannot; and it cannot be removed while it is running", async () => { + who = "viewer"; + assert.equal((await send("POST", `/${id}/run`)).status, 403); + who = "operator"; + const res = await send("POST", `/${id}/run`); + assert.equal(res.status, 201); + const run = (await res.json()).run; + assert.equal(run.status, "running"); + assert.equal(run.trigger, "manual"); + assert.ok(run.items.every((i: { windowIds?: unknown }) => i.windowIds === undefined), "the windows it made are its own business"); + assert.equal((await send("POST", `/${id}/run`)).status, 409, "not twice at once"); + who = "admin"; + assert.equal((await send("DELETE", `/${id}`)).status, 409); + who = "operator"; + const stopped = await send("POST", `/runs/${run.id}/cancel`); + assert.equal(stopped.status, 200); + assert.equal((await stopped.json()).run.status, "cancelled"); + assert.equal((await send("POST", `/runs/${run.id}/cancel`)).status, 409); + assert.equal((await send("POST", "/runs/99999/cancel")).status, 404); + who = "admin"; + assert.equal((await send("DELETE", `/${id}`)).status, 204); + assert.equal((await db.select().from(patchPlanRuns)).length, 0, "its history goes with it"); + }); +}); diff --git a/web/src/App.tsx b/web/src/App.tsx index 950750c..32581b3 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -33,6 +33,7 @@ import InventoryLifecycle from "./pages/InventoryLifecycle"; import Endpoints from "./pages/Endpoints"; import Heartbeats from "./pages/Heartbeats"; import ComingUp from "./pages/ComingUp"; +import MonthlyReport from "./pages/MonthlyReport"; import ChangeLog from "./pages/ChangeLog"; import Backups from "./pages/Backups"; import Consistency from "./pages/Consistency"; @@ -158,6 +159,7 @@ function Shelled({ user }: { user: CurrentUser }) { } /> } /> } /> + } /> : } /> } /> } /> diff --git a/web/src/api/client.ts b/web/src/api/client.ts index 6dc359d..17500ca 100644 --- a/web/src/api/client.ts +++ b/web/src/api/client.ts @@ -508,6 +508,17 @@ export interface HealthCheckSettings { backupStaleHours: number; /** A restore test older than this many days is a problem; 0 = off (a failed test always is). */ restoreTestDays: number; + /** A server that starts listening on a TCP port it did not at first, and that is not noted on its Ports card, is a warning. */ + newPortAlerts: boolean; +} + +/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. */ +export interface MonthlyReportSettings { + enabled: boolean; + /** Day of the month, 1–28, in the notification time zone. */ + day: number; + /** "HH:MM", in the notification time zone. */ + time: string; } export interface QuietHoursSettings { @@ -543,6 +554,7 @@ export interface AppSettings { digest: DigestSettings; signIn: SignInSettings; profilePictures: ProfilePictureSettings; + monthlyReport: MonthlyReportSettings; } export type AppSettingsPatch = { [K in keyof AppSettings]?: Partial }; @@ -905,6 +917,94 @@ export interface EndpointInput { enabled?: boolean; } +export type PatchPlanItemStatus = "pending" | "running" | "done" | "failed" | "skipped"; +export interface PatchPlanItem { + serverId: number; + name: string; + status: PatchPlanItemStatus; + updateRunId: number | null; + message: string | null; +} +export interface PatchPlanRun { + id: number; + planId: number; + planName: string; + trigger: "schedule" | "manual"; + startedBy: string | null; + status: "running" | "done" | "failed" | "cancelled"; + message: string | null; + startedAt: string; + finishedAt: string | null; + items: PatchPlanItem[]; +} +/** Updates on a schedule, one server at a time. */ +export interface PatchPlan { + id: number; + name: string; + enabled: boolean; + /** 0 = Sunday … 6 = Saturday. */ + days: number[]; + startTime: string; + timezone: string; + tags: string[]; + serverIds: number[]; + integrationId: number; + projectId: number; + templateId: number; + templateName: string; + limitFrom: "name" | "hostname"; + stopOnFailure: boolean; + onlyWithUpdates: boolean; + silenceDependents: boolean; + windowMinutes: number; + createdBy: string | null; + /** The servers it covers right now. */ + servers: { id: number; name: string }[]; + /** ISO — its next times. */ + next: string[]; + running: boolean; +} +export type PatchPlanInput = Omit; + +export interface AvailabilityRow { + id: number; + name: string; + percent: number | null; + downMinutes: number; +} +export interface MonthlyReport { + month: string; + label: string; + from: string; + to: string; + partial: boolean; + timezone: string; + servers: AvailabilityRow[]; + integrations: AvailabilityRow[]; + endpoints: { id: number; name: string; percent: number | null; avgMs: number | null; checks: number }[]; + incidents: { + began: number; + resolved: number; + stillOpen: number; + totalMinutes: number; + averageMinutes: number | null; + longest: { id: number; message: string; minutes: number } | null; + byCategory: { category: string; count: number }[]; + list: { id: number; message: string; where: string; severity: string; openedAt: string; resolvedAt: string | null; minutes: number }[]; + }; + updates: { + runs: number; + done: number; + failed: number; + installed: number; + byServer: { name: string; runs: number; failed: number; installed: number }[]; + planRuns: { runs: number; done: number; failed: number }; + }; + restoreTests: { label: string; testedAt: string; result: string; note: string | null }[]; + backups: { ok: number; stale: number; failed: number; none: number; unknown: number; total: number; asOf: string } | null; + renewals: { kind: string; subject: string; at: string; fromDate: string; toDate: string }[]; +} + /** A check the app could set up for you, from what it already knows. */ export interface EndpointSuggestion { key: string; @@ -1322,6 +1422,8 @@ export interface PortEntry { lastSeenOpenAt: string | null; agent: { addresses: string[]; process: string | null; localOnly: boolean } | null; state: "open" | "reserved"; + /** Began listening after the first report and nobody has noted it: a "new port" alert is on. */ + isNew: boolean; } export interface PortScanSummary { @@ -1947,6 +2049,19 @@ export const api = { start: (data: UpdateRunInput) => request<{ run: UpdateRun }>("/api/update-runs", { method: "POST", body: JSON.stringify(data) }), cancel: (id: number) => request<{ status: string }>(`/api/update-runs/${id}/cancel`, { method: "POST" }), }, + patchPlans: { + list: () => request<{ plans: PatchPlan[]; runs: PatchPlanRun[]; servers: { id: number; name: string; tags: string[] }[]; tags: string[] }>("/api/patch-plans"), + create: (data: PatchPlanInput) => request<{ plan: PatchPlan }>("/api/patch-plans", { method: "POST", body: JSON.stringify(data) }), + update: (id: number, data: PatchPlanInput) => request<{ plan: PatchPlan }>(`/api/patch-plans/${id}`, { method: "PUT", body: JSON.stringify(data) }), + remove: (id: number) => request(`/api/patch-plans/${id}`, { method: "DELETE" }), + run: (id: number) => request<{ run: PatchPlanRun }>(`/api/patch-plans/${id}/run`, { method: "POST" }), + cancel: (runId: number) => request<{ run: PatchPlanRun }>(`/api/patch-plans/runs/${runId}/cancel`, { method: "POST" }), + }, + reports: { + monthly: (month?: string) => request<{ report: MonthlyReport; months: { id: string; label: string }[] }>(`/api/reports/monthly${month ? `?month=${month}` : ""}`), + monthlyText: (month?: string) => request<{ title: string; message: string }>(`/api/reports/monthly?format=text${month ? `&month=${month}` : ""}`), + sendMonthly: (month?: string) => request<{ sent: boolean }>("/api/reports/monthly/send", { method: "POST", body: JSON.stringify({ month }) }), + }, backups: { coverage: (staleDays: number, refresh = false) => request(`/api/backups/coverage?staleDays=${staleDays}${refresh ? "&refresh=1" : ""}`), restoreTests: () => request<{ limitDays: number; tests: RestoreTestEntry[]; custom: (RestoreInfo & { key: string; label: string })[] }>("/api/backups/restore-tests"), diff --git a/web/src/components/PatchPlans.tsx b/web/src/components/PatchPlans.tsx new file mode 100644 index 0000000..904973d --- /dev/null +++ b/web/src/components/PatchPlans.tsx @@ -0,0 +1,495 @@ +import { useEffect, useMemo, useState } from "react"; +import { Link } from "react-router-dom"; +import { api, type CurrentUser, type IntegrationSummary, type PatchPlan, type PatchPlanInput, type PatchPlanItemStatus, type PatchPlanRun, type SemaphoreTemplate } from "../api/client"; +import TimeAgo from "./TimeAgo"; +import EmptyState from "./EmptyState"; +import { ErrorNotice } from "./Notice"; +import { BlockSkeleton } from "./Skeleton"; +import { confirmDialog } from "../utils/dialogs"; +import { formatDateTime } from "../utils/date"; +import { readableError } from "../utils/errors"; +import { toast } from "../utils/toast"; + +const WEEK: { day: number; label: string }[] = [ + { day: 1, label: "Mon" }, + { day: 2, label: "Tue" }, + { day: 3, label: "Wed" }, + { day: 4, label: "Thu" }, + { day: 5, label: "Fri" }, + { day: 6, label: "Sat" }, + { day: 0, label: "Sun" }, +]; +const LONG_DAY = ["Sundays", "Mondays", "Tuesdays", "Wednesdays", "Thursdays", "Fridays", "Saturdays"]; + +const ITEM_BADGE: Record = { + pending: { label: "Waiting its turn", badge: "bg-secondary-lt text-secondary" }, + running: { label: "Updating", badge: "bg-blue-lt text-blue" }, + done: { label: "Updated", badge: "bg-green-lt text-green" }, + failed: { label: "Failed", badge: "bg-red-lt text-red" }, + skipped: { label: "Left alone", badge: "bg-secondary-lt text-secondary" }, +}; +const RUN_BADGE: Record = { + running: { label: "Running", badge: "bg-blue-lt text-blue" }, + done: { label: "Finished", badge: "bg-green-lt text-green" }, + failed: { label: "Didn't finish cleanly", badge: "bg-red-lt text-red" }, + cancelled: { label: "Stopped", badge: "bg-secondary-lt text-secondary" }, +}; + +/** "Sundays at 02:00", "Mondays to Fridays at 03:30" is too clever — the days are just listed. */ +function whenWords(days: number[], time: string): string { + const sorted = [...days].sort((a, b) => ((a + 6) % 7) - ((b + 6) % 7)); + return `${sorted.map((d) => LONG_DAY[d]).join(", ")} at ${time}`; +} + +const WINDOWS = [ + { minutes: 45, label: "45 minutes" }, + { minutes: 90, label: "1½ hours" }, + { minutes: 180, label: "3 hours" }, + { minutes: 360, label: "6 hours" }, +]; + +const zones = (): string[] => { + try { + return (Intl as unknown as { supportedValuesOf?: (k: string) => string[] }).supportedValuesOf?.("timeZone") ?? ["UTC"]; + } catch { + return ["UTC"]; + } +}; +const browserZone = () => { + try { + return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC"; + } catch { + return "UTC"; + } +}; + +interface Template { + key: string; + integration: IntegrationSummary; + template: SemaphoreTemplate; + label: string; +} + +interface Draft { + id: number | null; + name: string; + enabled: boolean; + days: number[]; + startTime: string; + timezone: string; + tags: string[]; + serverIds: number[]; + template: string; + limitFrom: "name" | "hostname"; + stopOnFailure: boolean; + onlyWithUpdates: boolean; + silenceDependents: boolean; + windowMinutes: number; +} + +const EMPTY: Draft = { id: null, name: "", enabled: true, days: [0], startTime: "02:00", timezone: browserZone(), tags: [], serverIds: [], template: "", limitFrom: "name", stopOnFailure: true, onlyWithUpdates: true, silenceDependents: true, windowMinutes: 90 }; + +function Progress({ run, canStop, onStop }: { run: PatchPlanRun; canStop: boolean; onStop: () => void }) { + return ( +
+
+ {run.planName} + {RUN_BADGE[run.status].label} + + started {run.trigger === "schedule" ? "by its schedule" : `by ${run.startedBy ?? "someone"}`} + + {run.status === "running" && canStop && ( + + )} +
+ {run.message &&
{run.message}
} +
+ {run.items.map((i) => ( +
+ + {ITEM_BADGE[i.status].label} + + {i.name} + {i.message && {i.message}} +
+ ))} +
+
+ ); +} + +/** + * Patch plans: "Sundays at 02:00, the servers tagged lab, one at a time, stop at the first failure." A plan updates each server with an ordinary update run — the + * server is put under maintenance, the playbook runs for it alone, the app waits for it to report again — and goes on to the next only when that one is over. What + * depends on a server is silenced while it is updated. Everyone can see the plans and how they went; an administrator makes them; an operator can run one now or stop it. + */ +export default function PatchPlans({ user }: { user: CurrentUser }) { + const isAdmin = user.role === "admin"; + const canRun = isAdmin || user.role === "operator"; + const [data, setData] = useState<{ plans: PatchPlan[]; runs: PatchPlanRun[]; servers: { id: number; name: string; tags: string[] }[]; tags: string[] } | null>(null); + const [templates, setTemplates] = useState(null); + const [error, setError] = useState(null); + const [draft, setDraft] = useState(null); + const [busy, setBusy] = useState(false); + + function load() { + api.patchPlans + .list() + .then((res) => { + setData(res); + setError(null); + }) + .catch((err) => setError(readableError(err))); + } + const running = data?.runs.some((r) => r.status === "running") ?? false; + useEffect(() => { + load(); + }, []); + // A run that is going is looked at every few seconds; otherwise now and then. + useEffect(() => { + const t = setInterval(load, running ? 10_000 : 60_000); + return () => clearInterval(t); + }, [running]); + + // The playbooks to choose from, asked for only when somebody starts making a plan. + useEffect(() => { + if (!draft || templates !== null) return; + let alive = true; + api.integrations + .list() + .then(async (res) => { + const sems = res.integrations.filter((i) => i.type === "semaphore" && i.enabled); + const all = await Promise.all(sems.map((integration) => api.integrations.semaphore.templates(integration.id).then((r) => r.templates.map((template): Template => ({ key: `${integration.id}:${template.projectId}:${template.id}`, integration, template, label: `${template.name} — ${template.projectName}${sems.length > 1 ? ` (${integration.name})` : ""}` }))).catch((): Template[] => []))); + if (alive) setTemplates(all.flat()); + }) + .catch(() => alive && setTemplates([])); + return () => { + alive = false; + }; + }, [draft, templates]); + + const covered = useMemo(() => { + if (!draft || !data) return []; + return data.servers.filter((s) => draft.serverIds.includes(s.id) || s.tags.some((t) => draft.tags.includes(t))); + }, [draft, data]); + + function edit(p: PatchPlan) { + setDraft({ id: p.id, name: p.name, enabled: p.enabled, days: p.days, startTime: p.startTime, timezone: p.timezone, tags: p.tags, serverIds: p.serverIds, template: `${p.integrationId}:${p.projectId}:${p.templateId}`, limitFrom: p.limitFrom, stopOnFailure: p.stopOnFailure, onlyWithUpdates: p.onlyWithUpdates, silenceDependents: p.silenceDependents, windowMinutes: p.windowMinutes }); + } + + async function save(e: React.FormEvent) { + e.preventDefault(); + if (!draft) return; + // Editing a plan whose playbook was not looked up still keeps what it has. + const picked = templates?.find((t) => t.key === draft.template); + const existing = draft.id !== null ? data?.plans.find((p) => p.id === draft.id) : undefined; + const input: PatchPlanInput | null = picked + ? { name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: picked.integration.id, projectId: picked.template.projectId, templateId: picked.template.id, templateName: picked.template.name, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes } + : existing && draft.template === `${existing.integrationId}:${existing.projectId}:${existing.templateId}` + ? { name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: existing.integrationId, projectId: existing.projectId, templateId: existing.templateId, templateName: existing.templateName, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes } + : null; + if (!input) { + setError("Choose the playbook that applies updates."); + return; + } + setBusy(true); + setError(null); + try { + if (draft.id === null) await api.patchPlans.create(input); + else await api.patchPlans.update(draft.id, input); + toast.success(draft.id === null ? `Added ${draft.name}.` : "Saved."); + setDraft(null); + load(); + } catch (err) { + setError(readableError(err)); + } finally { + setBusy(false); + } + } + + async function runNow(p: PatchPlan) { + const names = p.servers.map((s) => s.name); + if (!(await confirmDialog({ title: `Run ${p.name} now?`, message: `It updates ${names.length === 1 ? names[0] : `${names.length} servers (${names.slice(0, 5).join(", ")}${names.length > 5 ? " …" : ""})`}, one at a time${p.stopOnFailure ? ", and stops at the first failure" : ""}. ${p.onlyWithUpdates ? "Servers with nothing waiting are left alone. " : ""}Each is put under maintenance while it is updated and may restart.`, confirmLabel: "Run it now" }))) return; + try { + await api.patchPlans.run(p.id); + toast.success(`Started ${p.name}.`); + load(); + } catch (err) { + setError(readableError(err)); + } + } + + async function stop(run: PatchPlanRun) { + if (!(await confirmDialog({ title: "Stop this run?", message: "The servers not yet started are left alone. A playbook that has already started keeps running in Semaphore; the app only stops following it.", confirmLabel: "Stop", danger: true }))) return; + try { + await api.patchPlans.cancel(run.id); + load(); + } catch (err) { + setError(readableError(err)); + } + } + + async function remove(p: PatchPlan) { + if (!(await confirmDialog({ title: "Remove a plan", message: `Remove "${p.name}" and its history?`, confirmLabel: "Remove", danger: true }))) return; + try { + await api.patchPlans.remove(p.id); + load(); + } catch (err) { + setError(readableError(err)); + } + } + + const set = (key: K, value: Draft[K]) => setDraft((d) => (d ? { ...d, [key]: value } : d)); + const toggle = (list: T[], v: T) => (list.includes(v) ? list.filter((x) => x !== v) : [...list, v]); + + return ( + <> +

+ Updates on a schedule: Sundays at 02:00, the servers tagged lab, one at a time, stop at the first failure. Each server is updated by an ordinary update run — put under maintenance, the playbook run for it + alone, the app waits for it to report again — and the plan goes on to the next only when that one is over. What depends on a server is silenced while it is updated. A message at the end says how each went. +

+ + + {data && data.runs.length > 0 && ( +
+
+

Recent runs

+
+
+ {data.runs.slice(0, running ? 15 : 5).map((r) => ( + void stop(r)} /> + ))} +
+
+ )} + + {draft && data && ( +
+
+

{draft.id === null ? "Make a patch plan" : `Edit ${draft.name || "the plan"}`}

+
+
+
+ + set("name", e.target.value)} /> +
+
+
Days
+
+ {WEEK.map((d) => ( + + ))} +
+
+
+ + set("startTime", e.target.value)} /> +
+
+ + set("timezone", e.target.value)} /> + + {zones().map((z) => ( + +
A name like Europe/Stockholm, so 02:00 stays 02:00 in summer and winter.
+
+
+ + +
It ends sooner when that server's update does.
+
+ +
+
Which servers
+ {data.tags.length === 0 ? ( +
No server has a tag yet — tag servers on their pages (for example “lab”), or name them below.
+ ) : ( +
+ {data.tags.map((t) => ( + + ))} +
+ )} +
+ Or name servers one by one{draft.serverIds.length > 0 ? ` (${draft.serverIds.length})` : ""} +
+ {data.servers.map((s) => ( + + ))} +
+
+
{covered.length === 0 ? "No server is covered yet." : <>It would update {covered.length} server{covered.length === 1 ? "" : "s"}, by name: {covered.map((s) => s.name).join(", ")}.}
+
+ +
+ + {templates === null ? ( + + ) : templates.length === 0 && draft.id === null ? ( +
+ There is no Semaphore connected, or it has no templates. Add one under Integrations. +
+ ) : ( + + )} +
+
+ + +
Must match the host name in the Semaphore inventory.
+
+ +
+ + + + +
+
+
+ + +
+
+ )} + +
+
+

Plans

+ {isAdmin && !draft && ( +
+ +
+ )} +
+ {data === null && !error && } + {data && data.plans.length === 0 && ( + setDraft({ ...EMPTY, timezone: browserZone() }) } : undefined} + /> + )} + {data && data.plans.length > 0 && ( +
+ + + + + + + + {(canRun || isAdmin) && } + + + + {data.plans.map((p) => ( + + + + + + {(canRun || isAdmin) && ( + + )} + + ))} + +
PlanWhenServersNextActions
+ {p.name} {!p.enabled && Switched off} {p.running && Running} +
+ {p.templateName} · {p.stopOnFailure ? "stops at the first failure" : "carries on after a failure"} +
+
+ {whenWords(p.days, p.startTime)} +
{p.timezone}
+
+ {p.servers.length === 0 ? none match : p.servers.map((s) => s.name).join(", ")} +
{[...p.tags.map((t) => `tag ${t}`), p.serverIds.length > 0 ? `${p.serverIds.length} named` : null].filter(Boolean).join(" · ")}
+
+ {p.enabled && p.next[0] ? formatDateTime(new Date(p.next[0])) : —} + +
+ {canRun && ( + + )} + {isAdmin && ( + <> + + + + )} +
+
+
+ )} +
+ + ); +} diff --git a/web/src/components/ServerPorts.tsx b/web/src/components/ServerPorts.tsx index a984357..3d86738 100644 --- a/web/src/components/ServerPorts.tsx +++ b/web/src/components/ServerPorts.tsx @@ -37,6 +37,11 @@ function StatusBadges({ entry }: { entry: PortEntry }) { if (entry.state === "reserved") return Reserved; return (
+ {entry.isNew && ( + + New + + )} {entry.scanOpen && ( Reachable @@ -187,6 +192,17 @@ export default function ServerPorts({ } } + /** "This is expected": a note on the port ends the new-port alert. */ + async function markExpected(entry: PortEntry) { + setError(null); + try { + const res = await api.servers.ports.save(serverId, { port: entry.port, protocol: entry.protocol, label: "Expected", comment: "" }); + setData((prev) => (prev ? { ...prev, ports: res.ports } : prev)); + } catch (err) { + setError(readableError(err)); + } + } + async function removeNote(entry: PortEntry) { if (entry.id === null) return; const what = entry.state === "reserved" ? `Remove the reservation for ${entry.protocol}/${entry.port}?` : `Clear the note on ${entry.protocol}/${entry.port}?`; @@ -482,6 +498,11 @@ export default function ServerPorts({ {canEdit && (
+ {p.isNew && ( + + )} diff --git a/web/src/layout/nav.tsx b/web/src/layout/nav.tsx index 6045a18..c964dcf 100644 --- a/web/src/layout/nav.tsx +++ b/web/src/layout/nav.tsx @@ -32,6 +32,7 @@ import { IconSitemap, IconCalendarEvent, IconHeartRateMonitor, + IconReportAnalytics, } from "@tabler/icons-react"; import type { CurrentUser } from "../api/client"; @@ -126,6 +127,7 @@ export const NAV: NavEntry[] = [ { to: "/maintenance", label: "Maintenance", icon: }, { to: "/updates", label: "Updates", icon: }, { to: "/coming-up", label: "Coming up", icon: }, + { to: "/reports", label: "Monthly report", icon: }, { to: "/changes", label: "Change log", icon: , minRole: "operator" }, ], }, diff --git a/web/src/pages/ComingUp.tsx b/web/src/pages/ComingUp.tsx index dcda3cd..f3d8b89 100644 --- a/web/src/pages/ComingUp.tsx +++ b/web/src/pages/ComingUp.tsx @@ -147,7 +147,8 @@ export default function ComingUp({ user }: { user: CurrentUser }) { const seen = new Set(); let hidden = 0; const kept = withinRange.filter((i) => { - const schedule = /^schedule:(\d+):/.exec(i.id)?.[1]; + const m = /^(schedule|plan):(\d+):/.exec(i.id); + const schedule = m ? `${m[1]}:${m[2]}` : null; if (!schedule) return true; if (seen.has(schedule)) { hidden++; diff --git a/web/src/pages/Dashboard.tsx b/web/src/pages/Dashboard.tsx index ad26855..a15ee6b 100644 --- a/web/src/pages/Dashboard.tsx +++ b/web/src/pages/Dashboard.tsx @@ -630,7 +630,8 @@ export default function Dashboard({ user }: { user: CurrentUser }) { const seenSchedules = new Set(); const soon = (agenda?.items ?? []).filter((i) => { if (i.kind === "updates") return false; - const schedule = /^schedule:(\d+):/.exec(i.id)?.[1]; + const m = /^(schedule|plan):(\d+):/.exec(i.id); + const schedule = m ? `${m[1]}:${m[2]}` : null; if (!schedule) return true; if (seenSchedules.has(schedule)) return false; seenSchedules.add(schedule); diff --git a/web/src/pages/MonthlyReport.tsx b/web/src/pages/MonthlyReport.tsx new file mode 100644 index 0000000..cd6a878 --- /dev/null +++ b/web/src/pages/MonthlyReport.tsx @@ -0,0 +1,413 @@ +import { useEffect, useState } from "react"; +import { Link, useSearchParams } from "react-router-dom"; +import { IconPrinter } from "@tabler/icons-react"; +import { api, type AvailabilityRow, type CurrentUser, type MonthlyReport as Report } from "../api/client"; +import { ErrorNotice } from "../components/Notice"; +import { BlockSkeleton } from "../components/Skeleton"; +import { CATEGORY_NAMES, incidentPath, SEVERITY_BADGE } from "../components/IncidentHistory"; +import { formatDate, formatDateTime } from "../utils/date"; +import { readableError } from "../utils/errors"; +import { toast } from "../utils/toast"; + +/** "2 h 40 min", "35 min", "3 d 4 h". */ +function span(minutes: number): string { + const m = Math.max(1, Math.round(minutes)); + if (m < 60) return `${m} min`; + const h = Math.floor(m / 60); + if (h < 24) return m % 60 && h < 10 ? `${h} h ${m % 60} min` : `${h} h`; + const d = Math.floor(h / 24); + return h % 24 ? `${d} d ${h % 24} h` : `${d} d`; +} + +const RESULT_WORDS: Record = { + ok: { label: "Passed", badge: "bg-green-lt text-green" }, + partial: { label: "Partly worked", badge: "bg-yellow-lt text-yellow" }, + failed: { label: "Failed", badge: "bg-red-lt text-red" }, + not_needed: { label: "Not needed", badge: "bg-secondary-lt text-secondary" }, +}; + +const average = (rows: { percent: number | null }[]) => { + const known = rows.filter((r) => r.percent !== null); + return known.length === 0 ? null : Math.round((known.reduce((n, r) => n + (r.percent as number), 0) / known.length) * 10) / 10; +}; + +const percentBadge = (p: number | null) => (p === null ? "text-secondary" : p >= 99.9 ? "text-green" : p >= 99 ? "" : p >= 95 ? "text-yellow" : "text-red"); + +function Availability({ title, rows, hint }: { title: string; rows: AvailabilityRow[]; hint?: string }) { + if (rows.length === 0) return null; + return ( +
+

{title}

+ {hint &&
{hint}
} +
+ + + + + + + + + + {rows.map((r) => ( + + + + + + ))} + +
NameUpDown for
{r.name}{r.percent === null ? "—" : `${r.percent}%`}{r.downMinutes > 0 ? span(r.downMinutes) : "—"}
+
+
+ ); +} + +/** + * How a month went, in one page that can be read, printed or sent: how available the servers and endpoints were, the incidents and how long they took, the + * updates applied, restore tests and the state of the backups, and what was renewed. The same report can be sent on a day of the month (Settings → + * Notifications → Monthly report). + */ +export default function MonthlyReport({ user }: { user: CurrentUser }) { + const [params, setParams] = useSearchParams(); + const month = params.get("month") ?? undefined; + const [data, setData] = useState<{ report: Report; months: { id: string; label: string }[] } | null>(null); + const [error, setError] = useState(null); + + useEffect(() => { + let alive = true; + setData(null); + api.reports + .monthly(month) + .then((res) => { + if (!alive) return; + setData(res); + setError(null); + }) + .catch((err) => alive && setError(readableError(err))); + return () => { + alive = false; + }; + }, [month]); + + async function copyText() { + try { + const t = await api.reports.monthlyText(data?.report.month); + await navigator.clipboard.writeText(`${t.title}\n\n${t.message}`); + toast.success("Copied the report as text."); + } catch (err) { + toast.error(readableError(err)); + } + } + + const r = data?.report; + const serverAvg = r ? average(r.servers) : null; + const endpointAvg = r ? average(r.endpoints) : null; + const cards = r + ? [ + { label: "Servers up, on average", value: serverAvg === null ? "—" : `${serverAvg}%`, color: percentBadge(serverAvg) }, + { label: "Endpoints answering", value: endpointAvg === null ? "—" : `${endpointAvg}%`, color: percentBadge(endpointAvg) }, + { label: "Incidents began", value: String(r.incidents.began), color: r.incidents.began > 0 ? "text-yellow" : "text-green" }, + { label: "Update runs", value: String(r.updates.runs), color: r.updates.failed > 0 ? "text-yellow" : "" }, + { label: "Renewed", value: String(r.renewals.length), color: "" }, + ] + : []; + + return ( +
+
+

Monthly report

+
+ {data && ( + + )} + + +
+
+

+ How the month went, in one page: availability, incidents and how long they took, updates applied, restore tests and backups, and what was renewed. + {user.role === "admin" ? ( + <> + {" "} + It can be sent on a day of the month: Settings → Notifications. + + ) : null} +

+ + {!data && !error && } + + {r && ( + <> +

{r.label}

+
+ {formatDate(new Date(r.from))} – {formatDate(new Date(r.to))} + {r.partial ? " · the month is not over, so this is the month so far" : ""} · time zone {r.timezone} +
+ +
+ {cards.map((c) => ( +
+
+
+
{c.value}
+
{c.label}
+
+
+
+ ))} +
+ +
+
+

Availability

+
+
+ {r.servers.length === 0 && r.endpoints.length === 0 && r.integrations.length === 0 ? ( +
Nothing was measured for this month. Servers are measured from when their agent first reported, endpoints from when they were first checked.
+ ) : ( + <> + + {r.endpoints.length > 0 && ( +
+

Endpoints

+
From every check's answers, day by day (the days are UTC days).
+
+ + + + + + + + + + + {r.endpoints.map((e) => ( + + + + + + + ))} + +
CheckAnsweredAverage answerTimes asked
{e.name}{e.percent === null ? "—" : `${e.percent}%`}{e.avgMs === null ? "—" : `${e.avgMs} ms`}{e.checks}
+
+
+ )} + + + )} +
+
+ +
+
+

Incidents

+
+
+ {r.incidents.began === 0 && r.incidents.resolved === 0 && r.incidents.stillOpen === 0 ? ( +
No incident began or ended this month.
+ ) : ( + <> +

+ {r.incidents.began} began, {r.incidents.resolved} ended{r.incidents.stillOpen > 0 ? <>, {r.incidents.stillOpen} {r.partial ? "open" : "still open at the end of the month"} : null}. + {r.incidents.averageMinutes !== null && <> One that began and ended in the month took {span(r.incidents.averageMinutes)} on average, {span(r.incidents.totalMinutes)} in all.} +

+ {r.incidents.byCategory.length > 0 && ( +
+ {r.incidents.byCategory.map((c) => ( + + {CATEGORY_NAMES[c.category as keyof typeof CATEGORY_NAMES] ?? c.category} · {c.count} + + ))} +
+ )} +
+ + + + + + + + + + + + {r.incidents.list.map((i) => ( + + + + + + + + ))} + +
BeganWhereWhatWorstLasted
{formatDateTime(new Date(i.openedAt))}{i.where} + {i.message} + + {i.severity} + + {span(i.minutes)} + {!i.resolvedAt && and open} +
+
+ {r.incidents.began > r.incidents.list.length &&
The {r.incidents.list.length} longest of {r.incidents.began}. All are under Alerts → History.
} + + )} +
+
+ +
+
+

Updates

+
+
+ {r.updates.runs === 0 && r.updates.planRuns.runs === 0 ? ( +
No update run was made from here this month.
+ ) : ( + <> +

+ {r.updates.runs} update run{r.updates.runs === 1 ? "" : "s"}: {r.updates.done} done{r.updates.failed > 0 ? `, ${r.updates.failed} failed` : ""} + {r.updates.installed > 0 ? ` — ${r.updates.installed} update${r.updates.installed === 1 ? "" : "s"} installed` : ""}. + {r.updates.planRuns.runs > 0 && ( + <> + {" "} + Patch plans ran {r.updates.planRuns.runs} time{r.updates.planRuns.runs === 1 ? "" : "s"} + {r.updates.planRuns.failed > 0 ? `, ${r.updates.planRuns.failed} not cleanly` : ", all cleanly"}. + + )} +

+ {r.updates.byServer.length > 0 && ( +
+ + + + + + + + + + + {r.updates.byServer.map((s) => ( + + + + + + + ))} + +
ServerRunsFailedInstalled
{s.name}{s.runs} 0 ? "text-red" : "text-secondary"}`}>{s.failed || "—"}{s.installed || "—"}
+
+ )} + + )} +
+
+ +
+
+

Backups and restore tests

+
+
+ {r.restoreTests.length === 0 ? ( +

No restore test was recorded this month.

+ ) : ( +
+ + + + + + + + + + + {r.restoreTests.map((t, i) => ( + + + + + + + ))} + +
TestedWhatResultNote
{t.testedAt}{t.label} + {RESULT_WORDS[t.result]?.label ?? t.result} + {t.note ?? ""}
+
+ )} + {r.backups ? ( +

+ Backups as they are now (read {formatDate(new Date(r.backups.asOf))}): {r.backups.ok} of {r.backups.total} guest{r.backups.total === 1 ? "" : "s"} have a recent one + {r.backups.failed + r.backups.stale + r.backups.none + r.backups.unknown > 0 && ( + <> + ; {r.backups.failed} failed, {r.backups.stale} stale, {r.backups.none} without any{r.backups.unknown > 0 ? `, ${r.backups.unknown} unknown` : ""} + + )} + . See Backup coverage. +

+ ) : ( +

The backups have not been read since this app started, so their state is not shown here. Open Backup coverage to read them.

+ )} +
+
+ +
+
+

Renewed

+
+
+ {r.renewals.length === 0 ? ( +
No certificate or domain was seen to be renewed this month.
+ ) : ( +
+ + + + + + + + + + + {r.renewals.map((x, i) => ( + + + + + + + ))} + +
SeenWhatWas good untilIs good until
{formatDate(new Date(x.at))} + {x.kind === "domain" ? "Domain" : "Certificate"} {x.subject} + {x.fromDate}{x.toDate}
+
+ )} +
Renewals are noticed when the app re-reads an expiry date, so only those since it began watching are here.
+
+
+ + )} +
+ ); +} diff --git a/web/src/pages/Updates.tsx b/web/src/pages/Updates.tsx index de1e36a..666b24e 100644 --- a/web/src/pages/Updates.tsx +++ b/web/src/pages/Updates.tsx @@ -1,4 +1,4 @@ -import { useEffect, useState } from "react"; +import { useEffect, useState, type ReactNode } from "react"; import { Link, useSearchParams } from "react-router-dom"; import { api, type AgentLatest, type CurrentUser, type DockhandContainer, type UpdatesServer } from "../api/client"; import { agentState, agentVersionWords } from "../utils/agent"; @@ -15,6 +15,8 @@ import { readableError } from "../utils/errors"; import ExportTableButton from "../components/ExportTableButton"; import UpdateRunDrawer from "../components/UpdateRunDrawer"; import UpdateRunsCard from "../components/UpdateRunsCard"; +import ViewTabs from "../components/ViewTabs"; +import PatchPlans from "../components/PatchPlans"; import { EmptyRow } from "../components/EmptyState"; interface Row { @@ -56,7 +58,34 @@ const DAY_MS = 86_400_000; * whether a restart is due, and how long they have been waiting) and the Docker containers that have a newer image. It only looks — * nothing here installs anything. */ +const updateTabs = (view: "waiting" | "plans") => ( + +); + +/** What is waiting to be installed, and — as patch plans — updating on a schedule, one server at a time (`?view=plans`). */ export default function Updates({ user }: { user: CurrentUser }) { + const [params] = useSearchParams(); + if (params.get("view") === "plans") { + return ( + <> +
+

Updates

+
+ {updateTabs("plans")} + + + ); + } + return ; +} + +function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) { const canRun = user.role === "admin" || user.role === "operator"; // The server being updated now (its drawer is open), and a counter that makes the runs card look again after one was started. const [updating, setUpdating] = useState(null); @@ -154,9 +183,10 @@ export default function Updates({ user }: { user: CurrentUser }) { return ( <> -
+

Updates

+ {tabs}

What the agents say is waiting to be installed on each server, and the Docker containers that have a newer image. {canRun ? "Update… applies a server's updates through a Semaphore playbook: it silences the server, runs the playbook, waits for the server to report again and tells you what is left." : "Operators can apply a server's updates from here."}

diff --git a/web/src/pages/settings/NotificationSettings.tsx b/web/src/pages/settings/NotificationSettings.tsx index 8c44116..f3a64a6 100644 --- a/web/src/pages/settings/NotificationSettings.tsx +++ b/web/src/pages/settings/NotificationSettings.tsx @@ -12,6 +12,7 @@ import { type EscalationSettings, type NotificationEvents, type DigestSettings, + type MonthlyReportSettings, type QuietHoursSettings, type HealthCheckSettings, } from "../../api/client"; @@ -81,7 +82,8 @@ const DIGEST_DAYS: { value: number; label: string }[] = [ { value: 6, label: "Saturday" }, { value: 0, label: "Sunday" }, ]; -const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0 }; +const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true }; +const DEFAULT_MONTHLY: MonthlyReportSettings = { enabled: false, day: 1, time: "08:00" }; type TestResult = { ok: boolean; message: string } | null; type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends boolean ? K : never }[keyof NotificationEvents]; @@ -240,6 +242,10 @@ export default function NotificationSettings() { const [digestPreview, setDigestPreview] = useState<{ title: string; message: string } | null>(null); const [digestBusy, setDigestBusy] = useState<"preview" | "send" | null>(null); const [digestResult, setDigestResult] = useState(null); + const [monthly, setMonthly] = useState(DEFAULT_MONTHLY); + const [monthlyPreview, setMonthlyPreview] = useState<{ title: string; message: string } | null>(null); + const [monthlyBusy, setMonthlyBusy] = useState<"preview" | "send" | null>(null); + const [monthlyResult, setMonthlyResult] = useState(null); const [queuedCount, setQueuedCount] = useState(null); const [flushing, setFlushing] = useState(false); const [flushResult, setFlushResult] = useState(null); @@ -253,7 +259,7 @@ export default function NotificationSettings() { const [testingWebhook, setTestingWebhook] = useState(false); const [testWebhookResult, setTestWebhookResult] = useState(null); - const snapshot = JSON.stringify({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest }); + const snapshot = JSON.stringify({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest, monthlyReport: monthly }); const dirty = baseline !== null && snapshot !== baseline; useEffect(() => { @@ -277,7 +283,9 @@ export default function NotificationSettings() { setHealthChecks(s.healthChecks); const d = s.digest ?? DEFAULT_DIGEST; setDigest(d); - setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, routing: rt, escalation: es, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d })); + const mo = s.monthlyReport ?? DEFAULT_MONTHLY; + setMonthly(mo); + setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, routing: rt, escalation: es, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d, monthlyReport: mo })); }) .catch((err) => setLoadError(err instanceof Error ? err.message : String(err))) .finally(() => setLoading(false)); @@ -324,7 +332,7 @@ export default function NotificationSettings() { setSaveError(null); setSaved(false); try { - await api.settings.update({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest }); + await api.settings.update({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest, monthlyReport: monthly }); setBaseline(snapshot); setWebPushSaved(webPush.enabled); setSaved(true); @@ -362,6 +370,31 @@ export default function NotificationSettings() { } } + async function handleMonthlyPreview() { + setMonthlyBusy("preview"); + setMonthlyResult(null); + try { + setMonthlyPreview(await api.reports.monthlyText()); + } catch (err) { + setMonthlyResult({ ok: false, message: readableError(err) }); + } finally { + setMonthlyBusy(null); + } + } + + async function handleMonthlySend() { + setMonthlyBusy("send"); + setMonthlyResult(null); + try { + await api.reports.sendMonthly(); + setMonthlyResult({ ok: true, message: "Sent through the channels that are on." }); + } catch (err) { + setMonthlyResult({ ok: false, message: readableError(err) }); + } finally { + setMonthlyBusy(null); + } + } + async function handleFlushNow() { setFlushing(true); setFlushResult(null); @@ -702,6 +735,16 @@ export default function NotificationSettings() { hint="From each agent server's own history, shown on the Alerts page and dashboard (not sent as a notification). Needs a few days of steady growth to say anything. 0 turns it off." onChange={(v) => setHealthChecks((h) => ({ ...h, diskForecastDays: v }))} /> +
@@ -893,6 +936,62 @@ export default function NotificationSettings() { +
+
+
+

Monthly report

+
+
+

+ One message a month with last month in numbers: how available the servers and endpoints were, the incidents and how long they took, updates applied (by hand and by patch plans), + restore tests, and what was renewed. The same report is on the Monthly report page, to read and print. Quiet hours hold it like any other notification. +

+ +
+
+ + +
+
+ + setMonthly((m) => ({ ...m, time: e.target.value }))} /> +
+
+
+ + +
+
The day and time are saved with the Save button; the preview and "send now" don't depend on them. The day is at most the 28th, so every month has it.
+ + {monthlyPreview && ( +
+
{monthlyPreview.title}
+
+                      {monthlyPreview.message}
+                    
+
+ )} +
+
+
+
diff --git a/web/src/styles.css b/web/src/styles.css index 2e86393..dd46ddf 100644 --- a/web/src/styles.css +++ b/web/src/styles.css @@ -1549,10 +1549,27 @@ html[data-density="roomy"] .table > :not(caption) > * > * { @media print { .no-print, - .navbar-vertical { + .navbar-vertical, + .navbar-toggler, + .breadcrumb, + .toast-stack { display: none !important; } + /* The monthly report: each section stays whole on a page, in plain black on white. */ + .report-page .card { + break-inside: avoid; + border: 1px solid #ccc; + box-shadow: none; + } + + .report-page, + .report-page .card, + .report-page .text-secondary { + color: #000 !important; + background: #fff !important; + } + .page-wrapper { margin: 0 !important; padding: 0 !important;