diff --git a/API.md b/API.md index b816e9b..452bf5b 100644 --- a/API.md +++ b/API.md @@ -56,6 +56,11 @@ All gauges. Labels are escaped as Prometheus requires. | `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. | | `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). | | `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). | +| `homelab_alerts_acknowledged` | | Alerts somebody has acknowledged (and that aren't silenced). Still counted in `homelab_alerts` above. | +| `homelab_incidents_open` | | Incidents that have begun and not yet ended. | +| `homelab_selfbackup_snapshots` | | How many backups of Homelab Manager's own database are kept. | +| `homelab_selfbackup_last_success_timestamp_seconds` | | When the newest of them was made, as a Unix time. | +| `homelab_selfbackup_last_attempt_ok` | | 1 if the last attempt to back it up worked, 0 if it failed (left out until one has been tried). | | `homelab_alerts_silenced` | | Alerts held back by a maintenance window. | | `homelab_alerts_grouped` | | Alerts folded under a server that is down. | | `homelab_integration_up` | `integration`, `type` | 1 if the integration's latest calls work, 0 if failing; left out until something has called it. | @@ -68,7 +73,7 @@ All gauges. Labels are escaped as Prometheus requires. | `homelab_scrape_timestamp_seconds` | | When the answer was built. | Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`, -`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`. +`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`, `time() - homelab_selfbackup_last_success_timestamp_seconds > 2 * 86400`. ## Event webhooks diff --git a/BACKUP.md b/BACKUP.md index 66bfa1e..b579160 100644 --- a/BACKUP.md +++ b/BACKUP.md @@ -4,6 +4,7 @@ Everything Homelab Manager knows is in **one SQLite file**, plus a handful of se `.env`. A backup is those two things — and the part people get wrong is keeping them **apart** (see [Keeping backups safe](#keeping-backups-safe)). +- [The built-in backups (Settings → Backup)](#the-built-in-backups-settings--backup) - [What to back up](#what-to-back-up) - [Taking a backup](#taking-a-backup) - [Checking a backup](#checking-a-backup) @@ -18,6 +19,34 @@ Everything Homelab Manager knows is in **one SQLite file**, plus a handful of se The commands assume the standard Docker setup from the README: you run them in the folder that holds `docker-compose.yml`, `.env` and `data/`, and the service is called `homelab-manager`. +## The built-in backups (Settings → Backup) + +You don't need the snippets below for the everyday case: Homelab Manager takes a backup of its own +database **every day** (03:30 in the notification time zone, the newest 14 kept — both changeable, and it +can be switched off), checks each one as it makes it, and keeps them in `data/backups/` with the same +names the snippets use. **Settings → Backup** shows them and does the rest: + +- **Back up now** makes one on the spot. **Download** takes a copy to your computer (the quickest way + off the machine). **Check** opens a backup and says whether SQLite finds it sound, how many users, + servers and integrations it holds, and whether it is from an older or newer version. +- **Upload a backup…** takes a file from somewhere else (another machine, the script below) and keeps it + if it checks out. A file that isn't a Homelab Manager database is refused and nothing is kept. +- **Restore…** checks the backup and *stages* it. Nothing changes until the app next starts — restart it + (`docker compose restart homelab-manager`) and, before anything opens the database, the staged file + takes the database's place. The database that was in use is **kept** in `data/backups/` as + `…-pre-restore.sqlite`, so a restore that wasn't what you wanted is undone by restoring that file. If + something stops it half way, the old database is put back. A backup from a **newer** version than the + one running is refused; an older one is fine (the start-up brings it up to date). **Cancel the restore** + until you restart if you change your mind. +- If the newest backup is more than two days old, or a scheduled one fails, there is an **alert** (and a + message through your channels), and `/metrics` has `homelab_selfbackup_last_success_timestamp_seconds` + for your own monitoring. + +Three things it does **not** do, and the rest of this page is about: it does not hold +`CREDENTIALS_ENCRYPTION_KEY` (keep that separately — see [Keeping backups safe](#keeping-backups-safe)), +it does not copy anything off the machine on its own (Download, or copy `data/backups/` with your usual +tool), and it does not replace a restore drill. Access to all of it is for administrators only. + ## What to back up | What | Where | Why | If you lose it | @@ -275,7 +304,17 @@ What you can get back, from least to most work: ## What has been tested, and what hasn't -Tested on a scratch database made with the real migrations: +**The built-in backups** were tested with the app's own tests (a backup holds what was there and not what came +after; two in the same second don't collide; garbage, someone else's database and a newer version's +database are refused; the newest N are kept; a staged restore is checked, can be cancelled and puts nothing +half-copied in place; a restore that can't complete leaves the database exactly as it was) and by a real +restore: a backup was taken, a user added, the restore staged, the actual built app (`node dist/index.js`) +restarted on the same data folder — the user added after the backup was gone, the previous database was +kept as `…-pre-restore.sqlite`, and the next daily backup was scheduled. **Not** tested: running it inside the +real Docker container, and a very large database (the copy is made on this machine's disk and takes about as +long as copying the file). + +The snippet-based way below was tested on a scratch database made with the real migrations: - The snapshot (`VACUUM INTO`) taken from a second process **while the app was writing**: it completed in tens of milliseconds, passed the integrity check, held all migrations, and an diff --git a/DATABASE.md b/DATABASE.md index 73092f2..ec91522 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -23,8 +23,10 @@ what the app actually creates. |---|---| | **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. | | **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. | -| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0031`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | -| **Tables** | 31 application tables, plus drizzle's own `__drizzle_migrations`. | +| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0032`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | +| **Tables** | 34 application tables, plus drizzle's own `__drizzle_migrations`. | + +**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`. **Not in the database:** @@ -62,9 +64,9 @@ erDiagram integrations ||--o{ alert_tickets : "osTicket used (set null)" ``` -Twelve tables stand alone, with no foreign keys: `settings`, `secrets`, +Thirteen tables stand alone, with no foreign keys: `settings`, `secrets`, `ipam_entries`, `domains`, `diag_log`, `notification_queue`, -`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`. `maintenance_windows` and `maintenance_aliases` also have no +`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`. `maintenance_windows` and `maintenance_aliases` also have no foreign key — see [soft references](#soft-references-not-foreign-keys). ## Tables @@ -101,7 +103,7 @@ Who changed what. Written by the app on every change (see | `id` | integer PK | | | `actor_user_id` | integer → `users.id`, **set null** on delete | Null for automatic actions, and for entries whose user was deleted. | | `actor_label` | text | The user's name/email as it was at the time (or `system`), so an entry still reads correctly after the account is gone. | -| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `share_link` (a link made or removed — never the secret), `runbook`, `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. | +| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `share_link` (a link made or removed — never the secret), `runbook`, `alert` (`ack`, `unack` — with the alert's id; on the thing it is about, so they show on a server's Activity card), `incident` (`note`, `note_removed`), `db_backup` (`create`, `upload`, `download`, `delete`, `restore_staged`, `restore_cancelled`, `settings`), `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. | | `action` | text NOT NULL | `create`, `update`, `delete`, `start`, `stop`, … — free text. | | `target_type`, `target_id` | text | What it happened to. `target_id` is text so it can hold any kind of ID; there's no foreign key. | | `detail` | text | JSON, free-form context (what changed, names, counts). Secrets are never put here. | @@ -490,6 +492,51 @@ checks see it. A row is one *occurrence*: when an alert clears and later returns | `ticketed_at` | text | ISO; when the ticket was opened. | | `cleared_at` | text | ISO; null while the alert is still active. Rows cleared more than 180 days ago are deleted. | +#### `alert_acks` + +"I'm on it" for one alert. **One row per alert id.** + +| Column | Type | Notes | +|---|---|---| +| `id` | integer PK | | +| `alert_key` | text NOT NULL, unique | The alert's stable id (e.g. `offline:offline:server:3`), the same key the Alerts page uses; **not a foreign key**. | +| `note` | text NOT NULL | Up to 500 characters; may be empty. | +| `assignee_user_id` | integer | → `users.id`, set to null if the user is removed. | +| `assignee_name` | text | The name as it was when handed over, so it still reads if the user is removed. | +| `acked_by` | text NOT NULL | | +| `acked_at` | text NOT NULL | ISO. | +| `until` | text | ISO; when it runs out. Null means until the alert is gone. | +| `last_seen_at` | text NOT NULL | ISO; the last time the alert was there. A row whose alert has been gone for 30 minutes is deleted. | + +#### `alert_incidents` + +One time an alert was raised and later went away. When the same alert comes back it is a new row. Information-only alerts are not recorded. + +| Column | Type | Notes | +|---|---|---| +| `id` | integer PK | | +| `alert_key` | text NOT NULL | The alert's id (indexed). Not a foreign key. | +| `severity`, `peak_severity` | text NOT NULL | As it is now (or was at the end), and the worst it got to. | +| `category`, `source`, `message`, `link` | text | As the Alerts page showed them; `message` is the latest wording. | +| `target_type`, `target_id`, `target_name` | text, integer, text | The server, integration or DNS provider it was about, when there is one (indexed). A plain reference, **not a foreign key**. | +| `caused_by` | text | The host whose outage this was folded under. | +| `opened_at` | text NOT NULL | ISO. | +| `resolved_at` | text | ISO; null while open. Set to when it first went missing, once it has been missing long enough. | +| `last_seen_at` | text NOT NULL | ISO. | +| `missing_since` | text | ISO; set while it has been missing from the last look(s), before it is called over. | + +Resolved rows older than 400 days are deleted, with their notes. + +#### `incident_notes` + +| Column | Type | Notes | +|---|---|---| +| `id` | integer PK | | +| `incident_id` | integer NOT NULL | → `alert_incidents.id`, deleted with it. | +| `author` | text NOT NULL | The person's name at the time. | +| `body` | text NOT NULL | Up to 4000 characters. | +| `created_at` | text NOT NULL | ISO. | + #### `update_runs` A playbook run in Semaphore for one server, wrapped in a maintenance window and followed until the server reports again. @@ -623,6 +670,7 @@ that were never changed aren't stored; the app fills in defaults when reading. | `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). | | `quietHours` | `enabled`, `start`, `end`. | | `signIn` | Roles from groups: `roleFromGroups` (false), `groupsClaim` (`groups`), `adminGroups`, `operatorGroups`, `otherwise` (`viewer` or `keep`). | +| `selfBackup` | The daily backup of this database: `enabled` (true), `time` (`HH:MM`, default `03:30`, in the notification time zone) and `keep` (14 — how many daily ones are kept). | | `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. | | `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. | | `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index 6112976..434577e 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -99,6 +99,10 @@ schedule. |---|---| | "Notifications from quiet hours" | Once, at quiet hours' configured end time, **only if enabled and only if at least one notification was held** during the window. Bundles every held notification's title and message into one message, then clears the queue. | +## Backups of Homelab Manager itself + +A scheduled backup of the app's own database that fails sends one message through the channels ("Homelab Manager — its own backup failed", with the reason), under the same toggle as server and disk alerts (*Server offline, disk nearly full…*). The same problem is also an alert on the Alerts page of the Backups kind — "the newest backup of Homelab Manager's own database is … old" once it is more than two days old, critical after a week, and "no backup of its own yet" with the reason when the very first one fails. No alert is raised for a new install that hasn't tried yet, or when daily backups are switched off. See [BACKUP.md](BACKUP.md). + ## Weekly digest | Notification | Fires when | @@ -108,7 +112,7 @@ schedule. It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say: what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage); **what happened** (servers that stopped reporting and came back, services that stopped answering, maintenance); -**disks heading for full**; secrets and domains that are **expiring** or have expired (and hardware whose warranty ends within 60 days); guests that are **not backed up**, have a failed or stale backup; **updates waiting** (how many +**incidents** (how many began and ended, still open, the longest); **disks heading for full**; secrets and domains that are **expiring** or have expired (and hardware whose warranty ends within 60 days); guests that are **not backed up**, have a failed or stale backup; **updates waiting** (how many servers, how many security fixes, who needs a restart, the longest wait); and **planned maintenance** in the next seven days. A week with nothing in it says "A quiet week: nothing needs attention." On the settings page, *Preview* shows what it would say right now and *Send it now* sends it at once (through the channels that are on, whatever the time). diff --git a/README.md b/README.md index de225a6..7c12ad0 100644 --- a/README.md +++ b/README.md @@ -449,6 +449,38 @@ said — an older agent, or another package manager — shows "not reported", ne to date"; run its install command again to update its agent (how, and how to tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). +**Acknowledging and handing over alerts.** On the Alerts page, **Acknowledge…** (operators) says +"somebody is on this": an optional note, who has it (an administrator or operator), and how long it should +be left alone (an hour to a week, or until the alert is gone). It changes nothing about the problem — the +alert is still raised, still counted by `/metrics` and still on the wall display — but the alert shows who +has it and why, the sidebar count, the dashboard strip and the Alerts cards count only what nobody is on, +and the list can be filtered to "nobody is on it" or "handed to me". An acknowledgement that runs out puts +the alert back among the unattended; it goes by itself once the alert has been gone for half an hour, so +the same problem returning next month is not "still being dealt with". It is in the audit log and in the +server's history. + +**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an +incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer +after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the +maintenance windows, what was changed on the server, update runs, other alerts about the same thing that +began around the same time, and the notes people wrote (operators can add them; the author or an +administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look +every five minutes, so one that comes and goes between two looks may be missed. An alert is only called +resolved after it has been missing for a couple of minutes in a row — thirty when a check couldn't run, and +never in the first minutes after the app starts, when server alerts are held back on purpose. Resolved +incidents are kept for 400 days. The weekly digest has an *Incidents* section. + +**Database backups** (Settings → Backup). A copy of Homelab Manager's own database is taken every day (03:30 +in the notification time zone, the newest 14 kept; on unless you turn it off), checked as it is made, and +kept in `data/backups/`. The page lists them with **Download**, **Check**, **Restore…** and **Delete**, +makes one now (**Back up now**), takes one from somewhere else (**Upload a backup…**) and sets the schedule. +A restore is checked first, then staged, and takes effect the next time the app starts — restart it +(`docker compose restart homelab-manager`); the database that was in use is kept as a "before a restore" +backup, so a restore can be undone. A backup made by a *newer* version than the one running is refused. +If the newest backup gets more than two days old, or a scheduled one fails, it is an alert (and a message). +**A backup does not contain `CREDENTIALS_ENCRYPTION_KEY`**, which decrypts the credentials stored in it, +and it is still on the same machine until you copy it away: see [BACKUP.md](BACKUP.md). + **Update runs** (Updates → Update…, operators). Apply a server's waiting updates in one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server under maintenance (so nothing alerts while it works and restarts), runs the playbook diff --git a/ROLES.md b/ROLES.md index 89fa9e4..29d2bf7 100644 --- a/ROLES.md +++ b/ROLES.md @@ -95,6 +95,12 @@ even further, to admin only: operator; the price of power used for the running cost is admin-only. - **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs are going; starting an update run (**Update…**) or stopping one needs operator. +- **Acknowledging an alert, handing it over, and taking it back** need operator (the person it is handed to must be an operator or + administrator); everyone sees who has what. **Incidents** are readable by everyone signed in; writing a note needs operator, and a note + can be removed by its author or an administrator. +- **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are + administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials + (encrypted). - **Alert limits** on a server's page: everyone can see the ones a server has of its own; only an administrator changes them. - **Host health** (failed services, disk health, ZFS pools) is shown to everyone on a server's page; **Update agents automatically** on the Servers page is admin-only. diff --git a/server/drizzle/0032_daffy_captain_stacy.sql b/server/drizzle/0032_daffy_captain_stacy.sql new file mode 100644 index 0000000..57cdf3b --- /dev/null +++ b/server/drizzle/0032_daffy_captain_stacy.sql @@ -0,0 +1,46 @@ +CREATE TABLE `alert_acks` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `alert_key` text NOT NULL, + `note` text DEFAULT '' NOT NULL, + `assignee_user_id` integer, + `assignee_name` text, + `acked_by` text NOT NULL, + `acked_at` text NOT NULL, + `until` text, + `last_seen_at` text NOT NULL, + FOREIGN KEY (`assignee_user_id`) REFERENCES `users`(`id`) ON UPDATE no action ON DELETE set null +); +--> statement-breakpoint +CREATE UNIQUE INDEX `alert_acks_key_idx` ON `alert_acks` (`alert_key`);--> statement-breakpoint +CREATE TABLE `alert_incidents` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `alert_key` text NOT NULL, + `severity` text NOT NULL, + `peak_severity` text NOT NULL, + `category` text NOT NULL, + `source` text NOT NULL, + `message` text NOT NULL, + `link` text, + `target_type` text, + `target_id` integer, + `target_name` text, + `caused_by` text, + `opened_at` text NOT NULL, + `resolved_at` text, + `last_seen_at` text NOT NULL, + `missing_since` text +); +--> statement-breakpoint +CREATE INDEX `alert_incidents_key_idx` ON `alert_incidents` (`alert_key`);--> statement-breakpoint +CREATE INDEX `alert_incidents_opened_idx` ON `alert_incidents` (`opened_at`);--> statement-breakpoint +CREATE INDEX `alert_incidents_target_idx` ON `alert_incidents` (`target_type`,`target_id`);--> statement-breakpoint +CREATE TABLE `incident_notes` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `incident_id` integer NOT NULL, + `author` text NOT NULL, + `body` text NOT NULL, + `created_at` text NOT NULL, + FOREIGN KEY (`incident_id`) REFERENCES `alert_incidents`(`id`) ON UPDATE no action ON DELETE cascade +); +--> statement-breakpoint +CREATE INDEX `incident_notes_incident_idx` ON `incident_notes` (`incident_id`); \ No newline at end of file diff --git a/server/drizzle/meta/0032_snapshot.json b/server/drizzle/meta/0032_snapshot.json new file mode 100644 index 0000000..aae13d9 --- /dev/null +++ b/server/drizzle/meta/0032_snapshot.json @@ -0,0 +1,3062 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "74105475-bb46-4505-8f16-876699aff9d4", + "prevId": "f8e70de6-9a27-4382-b3ff-ebf54dfc76e3", + "tables": { + "alert_acks": { + "name": "alert_acks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "assignee_user_id": { + "name": "assignee_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "assignee_name": { + "name": "assignee_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "acked_by": { + "name": "acked_by", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "acked_at": { + "name": "acked_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "until": { + "name": "until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "alert_acks_key_idx": { + "name": "alert_acks_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": true + } + }, + "foreignKeys": { + "alert_acks_assignee_user_id_users_id_fk": { + "name": "alert_acks_assignee_user_id_users_id_fk", + "tableFrom": "alert_acks", + "tableTo": "users", + "columnsFrom": [ + "assignee_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_incidents": { + "name": "alert_incidents", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "peak_severity": { + "name": "peak_severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "link": { + "name": "link", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_name": { + "name": "target_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "caused_by": { + "name": "caused_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "opened_at": { + "name": "opened_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resolved_at": { + "name": "resolved_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "missing_since": { + "name": "missing_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_incidents_key_idx": { + "name": "alert_incidents_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + }, + "alert_incidents_opened_idx": { + "name": "alert_incidents_opened_idx", + "columns": [ + "opened_at" + ], + "isUnique": false + }, + "alert_incidents_target_idx": { + "name": "alert_incidents_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_tickets": { + "name": "alert_tickets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "seen_count": { + "name": "seen_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ticket_number": { + "name": "ticket_number", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "grouped": { + "name": "grouped", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ticketed_at": { + "name": "ticketed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cleared_at": { + "name": "cleared_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_tickets_key_idx": { + "name": "alert_tickets_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + } + }, + "foreignKeys": { + "alert_tickets_integration_id_integrations_id_fk": { + "name": "alert_tickets_integration_id_integrations_id_fk", + "tableFrom": "alert_tickets", + "tableTo": "integrations", + "columnsFrom": [ + "integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assets": { + "name": "assets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'other'" + }, + "manufacturer": { + "name": "manufacturer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "serial": { + "name": "serial", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_date": { + "name": "purchase_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_price": { + "name": "purchase_price", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warranty_until": { + "name": "warranty_until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rack_unit": { + "name": "rack_unit", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "power_watts": { + "name": "power_watts", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_log": { + "name": "audit_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "actor_label": { + "name": "actor_label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "audit_log_actor_user_id_users_id_fk": { + "name": "audit_log_actor_user_id_users_id_fk", + "tableFrom": "audit_log", + "tableTo": "users", + "columnsFrom": [ + "actor_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "consistency_ignores": { + "name": "consistency_ignores", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "consistency_ignores_key_unique": { + "name": "consistency_ignores_key_unique", + "columns": [ + "key" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "diag_log": { + "name": "diag_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "operation": { + "name": "operation", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ok": { + "name": "ok", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "latency_ms": { + "name": "latency_ms", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_providers": { + "name": "dns_providers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_type": { + "name": "provider_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "dns_providers_credential_id_integration_credentials_id_fk": { + "name": "dns_providers_credential_id_integration_credentials_id_fk", + "tableFrom": "dns_providers", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_records_cache": { + "name": "dns_records_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "record_id": { + "name": "record_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ttl": { + "name": "ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "priority": { + "name": "priority", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxied": { + "name": "proxied", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "dns_records_cache_provider_id_dns_providers_id_fk": { + "name": "dns_records_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_records_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_zones_cache": { + "name": "dns_zones_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_name": { + "name": "zone_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "synced_at": { + "name": "synced_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "dns_zones_cache_provider_zone_idx": { + "name": "dns_zones_cache_provider_zone_idx", + "columns": [ + "provider_id", + "zone_id" + ], + "isUnique": true + } + }, + "foreignKeys": { + "dns_zones_cache_provider_id_dns_providers_id_fk": { + "name": "dns_zones_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_zones_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "domains": { + "name": "domains", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'manual'" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "registrar": { + "name": "registrar", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "lookup_source": { + "name": "lookup_source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_ok_at": { + "name": "last_checked_ok_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "domains_name_unique": { + "name": "domains_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "event_webhooks": { + "name": "event_webhooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "secret": { + "name": "secret", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "events": { + "name": "events", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_attempt_at": { + "name": "last_attempt_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_result": { + "name": "last_result", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "incident_notes": { + "name": "incident_notes", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "incident_id": { + "name": "incident_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "author": { + "name": "author", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "incident_notes_incident_idx": { + "name": "incident_notes_incident_idx", + "columns": [ + "incident_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "incident_notes_incident_id_alert_incidents_id_fk": { + "name": "incident_notes_incident_id_alert_incidents_id_fk", + "tableFrom": "incident_notes", + "tableTo": "alert_incidents", + "columnsFrom": [ + "incident_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integration_credentials": { + "name": "integration_credentials", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "encrypted_secret": { + "name": "encrypted_secret", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integrations": { + "name": "integrations", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "integrations_credential_id_integration_credentials_id_fk": { + "name": "integrations_credential_id_integration_credentials_id_fk", + "tableFrom": "integrations", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "ipam_entries": { + "name": "ipam_entries", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "vendor": { + "name": "vendor", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "ipam_entries_ip_address_unique": { + "name": "ipam_entries_ip_address_unique", + "columns": [ + "ip_address" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_aliases": { + "name": "maintenance_aliases", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "address": { + "name": "address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "maintenance_aliases_unique_idx": { + "name": "maintenance_aliases_unique_idx", + "columns": [ + "address", + "target_type", + "target_id" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_schedules": { + "name": "maintenance_schedules", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "targets": { + "name": "targets", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "days": { + "name": "days", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "start_time": { + "name": "start_time", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_minutes": { + "name": "duration_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timezone": { + "name": "timezone", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_windows": { + "name": "maintenance_windows", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ends_at": { + "name": "ends_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "metrics_tokens": { + "name": "metrics_tokens", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "metrics_tokens_token_hash_unique": { + "name": "metrics_tokens_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "notification_queue": { + "name": "notification_queue", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "port_forwards": { + "name": "port_forwards", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "external_port": { + "name": "external_port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination": { + "name": "destination", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "internal_port": { + "name": "internal_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "port_forwards_server_id_servers_id_fk": { + "name": "port_forwards_server_id_servers_id_fk", + "tableFrom": "port_forwards", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "runbooks": { + "name": "runbooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "runbooks_target_idx": { + "name": "runbooks_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "scheduled_tasks": { + "name": "scheduled_tasks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "schedule_type": { + "name": "schedule_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'agent'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "command": { + "name": "command", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "schedule_expression": { + "name": "schedule_expression", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "next_run_at": { + "name": "next_run_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "raw_metadata": { + "name": "raw_metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_stale": { + "name": "is_stale", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "scheduled_tasks_server_id_servers_id_fk": { + "name": "scheduled_tasks_server_id_servers_id_fk", + "tableFrom": "scheduled_tasks", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "secrets": { + "name": "secrets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'generic'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expiry_date": { + "name": "expiry_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warn_days": { + "name": "warn_days", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 30 + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_host": { + "name": "check_host", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_port": { + "name": "check_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_links": { + "name": "server_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "server_links_server_id_servers_id_fk": { + "name": "server_links_server_id_servers_id_fk", + "tableFrom": "server_links", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_metrics": { + "name": "server_metrics", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metric": { + "name": "metric", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "real", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "recorded_at": { + "name": "recorded_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "server_metrics_lookup_idx": { + "name": "server_metrics_lookup_idx", + "columns": [ + "server_id", + "metric", + "recorded_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "server_metrics_server_id_servers_id_fk": { + "name": "server_metrics_server_id_servers_id_fk", + "tableFrom": "server_metrics", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_ports": { + "name": "server_ports", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "port": { + "name": "port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "open": { + "name": "open", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "last_seen_open_at": { + "name": "last_seen_open_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "server_ports_unique": { + "name": "server_ports_unique", + "columns": [ + "server_id", + "port", + "protocol" + ], + "isUnique": true + } + }, + "foreignKeys": { + "server_ports_server_id_servers_id_fk": { + "name": "server_ports_server_id_servers_id_fk", + "tableFrom": "server_ports", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "servers": { + "name": "servers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "hostname": { + "name": "hostname", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "os_type": { + "name": "os_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'linux'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "api_token_hash": { + "name": "api_token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "api_token_prefix": { + "name": "api_token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ip_addresses": { + "name": "ip_addresses", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_model": { + "name": "cpu_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_cores": { + "name": "cpu_cores", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_load_percent": { + "name": "cpu_load_percent", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_total_bytes": { + "name": "mem_total_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_used_bytes": { + "name": "mem_used_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disks": { + "name": "disks", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "listening_ports": { + "name": "listening_ports", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_port_scan": { + "name": "last_port_scan", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_integration_id": { + "name": "proxmox_integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_node": { + "name": "proxmox_node", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_guest_type": { + "name": "proxmox_guest_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_vmid": { + "name": "proxmox_vmid", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hide_proxmox_link": { + "name": "hide_proxmox_link", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "parent_server_id": { + "name": "parent_server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "no_parent": { + "name": "no_parent", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "pending_updates": { + "name": "pending_updates", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_pending_since": { + "name": "updates_pending_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_health": { + "name": "host_health", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "thresholds": { + "name": "thresholds", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_version": { + "name": "agent_version", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "servers_proxmox_integration_id_integrations_id_fk": { + "name": "servers_proxmox_integration_id_integrations_id_fk", + "tableFrom": "servers", + "tableTo": "integrations", + "columnsFrom": [ + "proxmox_integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "settings": { + "name": "settings", + "columns": { + "key": { + "name": "key", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "share_links": { + "name": "share_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'summary'" + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "share_links_token_hash_unique": { + "name": "share_links_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "tag_definitions": { + "name": "tag_definitions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "tag_definitions_name_unique": { + "name": "tag_definitions_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "update_runs": { + "name": "update_runs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_id": { + "name": "template_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_name": { + "name": "template_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "host_limit": { + "name": "host_limit", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_id": { + "name": "task_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "task_status": { + "name": "task_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_id": { + "name": "window_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_created": { + "name": "window_created", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_by": { + "name": "started_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_ended_at": { + "name": "task_ended_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_reported_at": { + "name": "host_reported_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finished_at": { + "name": "finished_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_before": { + "name": "updates_before", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_after": { + "name": "updates_after", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reboot_required": { + "name": "reboot_required", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "poll_errors": { + "name": "poll_errors", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + } + }, + "indexes": { + "update_runs_server_idx": { + "name": "update_runs_server_idx", + "columns": [ + "server_id", + "started_at" + ], + "isUnique": false + }, + "update_runs_status_idx": { + "name": "update_runs_status_idx", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "update_runs_server_id_servers_id_fk": { + "name": "update_runs_server_id_servers_id_fk", + "tableFrom": "update_runs", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "users": { + "name": "users", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "oidc_sub": { + "name": "oidc_sub", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'viewer'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_login_at": { + "name": "last_login_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "favorites": { + "name": "favorites", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_looked_at": { + "name": "last_looked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "groups": { + "name": "groups", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dashboard_layout": { + "name": "dashboard_layout", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "users_oidc_sub_unique": { + "name": "users_oidc_sub_unique", + "columns": [ + "oidc_sub" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/server/drizzle/meta/_journal.json b/server/drizzle/meta/_journal.json index b18dde2..4c5f03e 100644 --- a/server/drizzle/meta/_journal.json +++ b/server/drizzle/meta/_journal.json @@ -225,6 +225,13 @@ "when": 1792278718505, "tag": "0031_sad_lizard", "breakpoints": true + }, + { + "idx": 32, + "version": "6", + "when": 1792281947686, + "tag": "0032_daffy_captain_stacy", + "breakpoints": true } ] } \ No newline at end of file diff --git a/server/src/app.ts b/server/src/app.ts index a9ffa1d..5b81d1e 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -12,6 +12,8 @@ import { uptimeRouter } from "./routes/uptime.js"; import { shareLinksRouter } from "./routes/shareLinks.js"; import { runbooksRouter } from "./routes/runbooks.js"; import { updateRunsRouter } from "./routes/updateRuns.js"; +import { dbBackupsRouter } from "./routes/dbBackups.js"; +import { incidentsRouter } from "./routes/incidents.js"; import { inventoryRouter } from "./routes/inventory.js"; import { backupsRouter } from "./routes/backups.js"; import { metricsRouter, metricsTokensRouter } from "./routes/metrics.js"; @@ -100,6 +102,8 @@ export function createApp() { app.use("/api/share-links", shareLinksRouter); app.use("/api/runbooks", runbooksRouter); app.use("/api/update-runs", updateRunsRouter); + app.use("/api/db-backups", dbBackupsRouter); + app.use("/api/incidents", incidentsRouter); app.use("/api/inventory", inventoryRouter); app.use("/api/backups", backupsRouter); app.use("/api/metrics-tokens", metricsTokensRouter); diff --git a/server/src/db/applyRestore.ts b/server/src/db/applyRestore.ts new file mode 100644 index 0000000..1323615 --- /dev/null +++ b/server/src/db/applyRestore.ts @@ -0,0 +1,71 @@ +/** + * Finishes a restore that was staged from Settings → Backup. It has to happen before the database is opened, which is why it is a module + * of its own that index.ts imports first: by the time anything else loads, the file is already the one that was chosen. + * + * Nothing is deleted. The database that was in use is moved into the backups folder (as "…-pre-restore.sqlite") before the chosen one is put + * in its place, so a restore that wasn't what was wanted can itself be undone by restoring that file. If anything goes wrong half way, the + * old file is put back and the staged one is left where it is. + */ +import "../loadEnv.js"; +import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { resolveDataPath } from "../paths.js"; + +import { PENDING_FILE, PENDING_NOTE, RESULT_NOTE, type RestoreResult } from "./restoreFiles.js"; + +const stamp = (d: Date) => d.toISOString().slice(0, 19).replace(/[:T]/g, "-"); + +/** Puts a staged database in place, if one is waiting. Returns what it did, or null if nothing was staged. */ +export function applyPendingRestore(dbFile: string, now: Date = new Date()): RestoreResult | null { + const dataDir = dirname(dbFile); + const pending = join(dataDir, PENDING_FILE); + if (!existsSync(pending)) return null; + // Only a file is ever put in place of the database. + if (!statSync(pending).isFile()) { + console.error(`[restore] ${PENDING_FILE} isn't a file, so it was left alone and the database is as it was.`); + return null; + } + + let note: { from?: string; by?: string | null } = {}; + try { + note = JSON.parse(readFileSync(join(dataDir, PENDING_NOTE), "utf8")); + } catch { + // the note only says where it came from; the file is what matters + } + + const backups = join(dataDir, "backups"); + let keptAs: string | null = null; + let moved = false; + try { + mkdirSync(backups, { recursive: true }); + if (existsSync(dbFile)) { + let name = `homelab-manager-${stamp(now)}-pre-restore.sqlite`; + for (let n = 2; existsSync(join(backups, name)); n++) name = `homelab-manager-${stamp(now)}-pre-restore-${n}.sqlite`; + renameSync(dbFile, join(backups, name)); + keptAs = name; + moved = true; + } + // A journal left by the old file would be taken for the new one's. + for (const suffix of ["-wal", "-shm", "-journal"]) rmSync(dbFile + suffix, { force: true }); + renameSync(pending, dbFile); + } catch (err) { + if (moved && keptAs && !existsSync(dbFile)) { + try { + renameSync(join(backups, keptAs), dbFile); + } catch { + // nothing more can be done; the old file is still in the backups folder under its new name + } + } + console.error("[restore] couldn't put the staged backup in place; the database is as it was:", err instanceof Error ? err.message : err); + return null; + } + + const result: RestoreResult = { restoredAt: now.toISOString(), from: note.from ?? "an uploaded backup", keptAs, by: note.by ?? null }; + writeFileSync(join(dataDir, RESULT_NOTE), JSON.stringify(result)); + rmSync(join(dataDir, PENDING_NOTE), { force: true }); + console.log(`[restore] restored the database from ${result.from}${keptAs ? `; the one that was in use is kept as backups/${keptAs}` : ""}.`); + return result; +} + +// Run it now, on the database this process is about to open. +applyPendingRestore(resolveDataPath(process.env.DATABASE_PATH ?? "../data/homelab-manager.sqlite")); diff --git a/server/src/db/client.ts b/server/src/db/client.ts index 47266f0..3f062c2 100644 --- a/server/src/db/client.ts +++ b/server/src/db/client.ts @@ -12,4 +12,4 @@ const client = createClient({ url: `file:${dbPath}` }); await client.execute("PRAGMA foreign_keys = ON;"); export const db = drizzle(client, { schema }); -export { client }; +export { client, dbPath }; diff --git a/server/src/db/restoreFiles.ts b/server/src/db/restoreFiles.ts new file mode 100644 index 0000000..8c12c96 --- /dev/null +++ b/server/src/db/restoreFiles.ts @@ -0,0 +1,13 @@ +// The names a staged restore uses in the data folder. Kept apart from applyRestore.ts, which does something when it is imported. +export const PENDING_FILE = "restore-pending.sqlite"; +export const PENDING_NOTE = "restore-pending.json"; +export const RESULT_NOTE = "restore-result.json"; + +export interface RestoreResult { + restoredAt: string; + /** The backup that is now the database. */ + from: string; + /** Where the database that was in use went. */ + keptAs: string | null; + by: string | null; +} diff --git a/server/src/db/schema.ts b/server/src/db/schema.ts index 6817721..f6d008b 100644 --- a/server/src/db/schema.ts +++ b/server/src/db/schema.ts @@ -572,6 +572,63 @@ export const runbooks = sqliteTable( (t) => [index("runbooks_target_idx").on(t.targetType, t.targetId)], ); +// ─── Acknowledgements and incidents ───────────────────────────────────────── + +// "I'm on it" for one alert — see services/alertAcks.ts. Keyed by the alert's stable id (not a foreign key to anything); gone once the alert has been. +export const alertAcks = sqliteTable( + "alert_acks", + { + id: integer("id").primaryKey({ autoIncrement: true }), + alertKey: text("alert_key").notNull(), + note: text("note").notNull().default(""), + assigneeUserId: integer("assignee_user_id").references(() => users.id, { onDelete: "set null" }), + assigneeName: text("assignee_name"), // as it was when handed over, so it still reads if the user is removed + ackedBy: text("acked_by").notNull(), + ackedAt: text("acked_at").notNull(), // ISO + until: text("until"), // ISO; null = until the alert is gone + lastSeenAt: text("last_seen_at").notNull(), // ISO; the last time the alert was there + }, + (t) => [uniqueIndex("alert_acks_key_idx").on(t.alertKey)], +); + +// One time an alert was raised and later went away — see services/incidents.ts. When the same alert comes back it is a new row. +export const alertIncidents = sqliteTable( + "alert_incidents", + { + id: integer("id").primaryKey({ autoIncrement: true }), + alertKey: text("alert_key").notNull(), + severity: text("severity").notNull(), // as it is now (or was when it ended) + peakSeverity: text("peak_severity").notNull(), + category: text("category").notNull(), + source: text("source").notNull(), + message: text("message").notNull(), // the latest wording + link: text("link"), + targetType: text("target_type"), // server | integration | dns_provider, when the alert is about one of those + targetId: integer("target_id"), + targetName: text("target_name"), + causedBy: text("caused_by"), // the host whose outage this was folded under + openedAt: text("opened_at").notNull(), // ISO + resolvedAt: text("resolved_at"), // ISO; null while it is open + lastSeenAt: text("last_seen_at").notNull(), // ISO + missingSince: text("missing_since"), // ISO; set while it has been absent from the last look(s), before it is called over + }, + (t) => [index("alert_incidents_key_idx").on(t.alertKey), index("alert_incidents_opened_idx").on(t.openedAt), index("alert_incidents_target_idx").on(t.targetType, t.targetId)], +); + +export const incidentNotes = sqliteTable( + "incident_notes", + { + id: integer("id").primaryKey({ autoIncrement: true }), + incidentId: integer("incident_id") + .notNull() + .references(() => alertIncidents.id, { onDelete: "cascade" }), + author: text("author").notNull(), + body: text("body").notNull(), + createdAt: text("created_at").notNull(), // ISO + }, + (t) => [index("incident_notes_incident_idx").on(t.incidentId)], +); + // ─── Update runs: apply a server's updates and see it through ────────────── // A playbook run in Semaphore for one server, wrapped in a maintenance window and followed until the server reports again — see diff --git a/server/src/index.ts b/server/src/index.ts index dd5a7d3..b313b54 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,4 +1,6 @@ import "./loadEnv.js"; +// Before anything opens the database: puts a restore that was staged from Settings → Backup in place. +import "./db/applyRestore.js"; import { env, warnIfAuthNotConfigured, checkSessionSecret } from "./env.js"; import { createApp } from "./app.js"; import { runMigrations } from "./db/migrate.js"; @@ -17,6 +19,9 @@ import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js"; import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js"; import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js"; import { initUpdateRuns } from "./services/updateRuns.js"; +import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js"; +import { initIncidentRecorder } from "./services/incidents.js"; +import { getAlerts } from "./services/alerts.js"; { // Sessions are signed with this secret. Without one, the app would sign them with a value that's in the source code. @@ -50,6 +55,8 @@ await initAlertTicketScheduler(); await initKumaMaintenanceScheduler(); await initMaintenanceScheduleScheduler(); initUpdateRuns(); +await initSelfBackupScheduler(); +initIncidentRecorder(() => getAlerts(false)); createApp().listen(env.port, () => { console.log(`homelab-manager listening on port ${env.port}`); diff --git a/server/src/routes/alerts.ts b/server/src/routes/alerts.ts index aa15789..5aa407e 100644 --- a/server/src/routes/alerts.ts +++ b/server/src/routes/alerts.ts @@ -1,6 +1,10 @@ import { Router } from "express"; -import { requireAuth } from "../auth/middleware.js"; +import { z } from "zod"; +import { requireAuth, requireRole } from "../auth/middleware.js"; import { getAlerts } from "../services/alerts.js"; +import { AckError, assignableUsers, attentionCounts, clearAck, loadAcks, MAX_HOURS, MAX_NOTE, setAck } from "../services/alertAcks.js"; +import { recordAudit } from "../services/audit.js"; +import { openIncidentIds } from "../services/incidents.js"; import { ticketsForAlerts, type AlertTicketInfo } from "../services/alertTickets.js"; import { asyncHandler } from "../utils/asyncHandler.js"; @@ -19,5 +23,68 @@ alertsRouter.get("/", asyncHandler(async (req, res) => { } catch (err) { console.error("[alerts] couldn't look up tickets:", err); } - res.json({ ...report, alerts: report.alerts.map((a) => ({ ...a, ticket: tickets.get(a.id) ?? null })) }); + // Who has said they are on it, and the incident record each alert is part of. Neither may stop the page from loading. + let acks = new Map> extends Map ? V : never>(); + let incidents = new Map(); + try { + acks = await loadAcks(); + incidents = await openIncidentIds(report.alerts.map((a) => a.id)); + } catch (err) { + console.error("[alerts] couldn't look up acknowledgements:", err); + } + const attention = attentionCounts(report.alerts, acks); + res.json({ + ...report, + // `counts` still counts everything, as /metrics and the wall do; `attention` leaves out what somebody has said they are dealing with. + counts: { ...report.counts, acknowledged: attention.acknowledged, attention: { critical: attention.critical, warning: attention.warning } }, + alerts: report.alerts.map((a) => ({ ...a, ticket: tickets.get(a.id) ?? null, ack: acks.get(a.id) ?? null, incidentId: incidents.get(a.id) ?? null })), + }); +})); + +// Who an alert can be handed to. +alertsRouter.get("/assignees", requireRole("operator"), asyncHandler(async (_req, res) => { + res.json({ users: await assignableUsers() }); +})); + +const ackSchema = z.object({ + alertId: z.string().min(1).max(600), + note: z.string().max(MAX_NOTE).default(""), + assigneeId: z.number().int().positive().nullable().default(null), + /** How long it lasts; null (or left out) is until the alert is gone. */ + hours: z.number().int().min(1).max(MAX_HOURS).nullable().default(null), +}); + +const auditTargetOf = (alert: { target: { type: string; id: number } | null }) => (alert.target ? { targetType: alert.target.type, targetId: alert.target.id } : {}); + +// "I'm on it": acknowledge an alert, say why, hand it to someone, and say how long it should be left alone. +alertsRouter.put("/ack", requireRole("operator"), asyncHandler(async (req, res) => { + const parsed = ackSchema.safeParse(req.body); + if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "Say which alert, and optionally why, who has it and for how long.", details: parsed.error.flatten() }); + const alert = (await getAlerts(false)).alerts.find((a) => a.id === parsed.data.alertId); + if (!alert) return res.status(404).json({ error: "not_found", message: "That alert isn't there any more." }); + const actor = req.currentUser!; + try { + const { ack, before } = await setAck({ ...parsed.data, actor: actor.name ?? actor.email ?? actor.oidcSub }); + await recordAudit({ + actor, + category: "alert", + action: "ack", + ...auditTargetOf(alert), + detail: { alertId: alert.id, message: alert.message, note: ack.note, assignee: ack.assignee?.name ?? null, hours: parsed.data.hours, changed: before !== null }, + }); + res.json({ ack }); + } catch (err) { + if (err instanceof AckError) return res.status(err.status).json({ error: err.code, message: err.message }); + throw err; + } +})); + +alertsRouter.delete("/ack", requireRole("operator"), asyncHandler(async (req, res) => { + const alertId = typeof req.query.alertId === "string" ? req.query.alertId : ""; + if (!alertId) return res.status(400).json({ error: "invalid_id" }); + const removed = await clearAck(alertId); + if (!removed) return res.status(404).json({ error: "not_found" }); + const alert = (await getAlerts(false)).alerts.find((a) => a.id === alertId); + await recordAudit({ actor: req.currentUser!, category: "alert", action: "unack", ...(alert ? auditTargetOf(alert) : {}), detail: { alertId, message: alert?.message ?? null } }); + res.status(204).end(); })); diff --git a/server/src/routes/dbBackups.ts b/server/src/routes/dbBackups.ts new file mode 100644 index 0000000..c77bf7b --- /dev/null +++ b/server/src/routes/dbBackups.ts @@ -0,0 +1,135 @@ +import express, { Router } from "express"; +import { z } from "zod"; +import { requireAuth, requireRole } from "../auth/middleware.js"; +import { recordAudit } from "../services/audit.js"; +import { getSettings, updateSettings } from "../services/settingsStore.js"; +import { + backupDir, + cancelStagedRestore, + checkSnapshotFile, + createSnapshot, + deleteSnapshot, + dismissLastRestore, + importUploaded, + lastAttempt, + lastRestore, + listSnapshots, + pruneSnapshots, + snapshotPath, + stageRestore, + stagedRestore, +} from "../services/selfBackup.js"; +import { nextBackupRun, scheduleSelfBackup } from "../services/selfBackupScheduler.js"; +import { asyncHandler } from "../utils/asyncHandler.js"; + +export const dbBackupsRouter = Router(); + +// A backup holds everything the app knows — the audit log, every server's address, the hashed agent tokens and all the stored credentials +// (encrypted) — so all of it, looking and doing, is for administrators. +dbBackupsRouter.use(requireAuth, requireRole("admin")); + +/** A database file is far bigger than anything else this app is sent, and is only ever read from an administrator. */ +const UPLOAD_LIMIT = "1gb"; + +const settingsSchema = z.object({ + enabled: z.boolean(), + time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/, "Use a time like 03:30"), + keep: z.number().int().min(1).max(90), +}); + +const actorName = (req: express.Request) => req.currentUser!.name ?? req.currentUser!.email ?? req.currentUser!.oidcSub; + +async function overview() { + const { selfBackup } = await getSettings(); + return { + settings: selfBackup, + snapshots: listSnapshots(), + staged: stagedRestore(), + lastRestore: lastRestore(), + lastAttempt: await lastAttempt(), + nextRun: nextBackupRun()?.toISOString() ?? null, + folder: backupDir(), + }; +} + +dbBackupsRouter.get("/", asyncHandler(async (_req, res) => { + res.json(await overview()); +})); + +dbBackupsRouter.put("/settings", asyncHandler(async (req, res) => { + const parsed = settingsSchema.safeParse(req.body); + if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "A time and how many to keep.", details: parsed.error.flatten() }); + const before = (await getSettings()).selfBackup; + await updateSettings({ selfBackup: parsed.data }); + await scheduleSelfBackup(); + pruneSnapshots(parsed.data.keep); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "settings", targetType: "settings", detail: { from: before, to: parsed.data } }); + res.json(await overview()); +})); + +// Take one now. It is checked as it is made, and pruned like the scheduled ones. +dbBackupsRouter.post("/", asyncHandler(async (req, res) => { + try { + const { info, check } = await createSnapshot("manual"); + pruneSnapshots((await getSettings()).selfBackup.keep); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "create", targetType: "db_backup", targetId: info.name, detail: { sizeBytes: info.sizeBytes } }); + res.status(201).json({ snapshot: info, check }); + } catch (err) { + res.status(500).json({ error: "backup_failed", message: err instanceof Error ? err.message : String(err) }); + } +})); + +// A backup somebody has (from another machine, from the script in BACKUP.md): kept if it checks out, and can then be restored like any other. +dbBackupsRouter.post("/upload", express.raw({ type: () => true, limit: UPLOAD_LIMIT }), asyncHandler(async (req, res) => { + if (!Buffer.isBuffer(req.body) || req.body.length === 0) return res.status(400).json({ error: "no_file", message: "Choose a backup file to upload." }); + try { + const { info, check } = await importUploaded(req.body); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "upload", targetType: "db_backup", targetId: info.name, detail: { sizeBytes: info.sizeBytes } }); + res.status(201).json({ snapshot: info, check }); + } catch (err) { + res.status(400).json({ error: "unusable", message: err instanceof Error ? err.message : String(err) }); + } +})); + +dbBackupsRouter.post("/restore/cancel", asyncHandler(async (req, res) => { + const had = cancelStagedRestore(); + if (had) await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "restore_cancelled" }); + res.json({ cancelled: had }); +})); + +dbBackupsRouter.post("/restore-result/dismiss", asyncHandler(async (_req, res) => { + dismissLastRestore(); + res.json({ ok: true }); +})); + +dbBackupsRouter.post("/:name/check", asyncHandler(async (req, res) => { + const file = snapshotPath(req.params.name); + if (!file) return res.status(404).json({ error: "not_found" }); + res.json({ check: await checkSnapshotFile(file) }); +})); + +dbBackupsRouter.get("/:name/download", asyncHandler(async (req, res) => { + const file = snapshotPath(req.params.name); + if (!file) return res.status(404).json({ error: "not_found" }); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "download", targetType: "db_backup", targetId: req.params.name }); + res.setHeader("Cache-Control", "no-store"); + res.download(file, req.params.name); +})); + +// Checks the backup and stages it to replace the database the next time the app starts. Nothing changes until then, and the database in +// use is kept (as a "pre-restore" backup) when it does. +dbBackupsRouter.post("/:name/restore", asyncHandler(async (req, res) => { + try { + const check = await stageRestore(req.params.name, actorName(req)); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "restore_staged", targetType: "db_backup", targetId: req.params.name }); + res.json({ staged: stagedRestore(), check }); + } catch (err) { + res.status(400).json({ error: "cannot_restore", message: err instanceof Error ? err.message : String(err) }); + } +})); + +dbBackupsRouter.delete("/:name", asyncHandler(async (req, res) => { + if (!deleteSnapshot(req.params.name)) return res.status(404).json({ error: "not_found" }); + await recordAudit({ actor: req.currentUser!, category: "db_backup", action: "delete", targetType: "db_backup", targetId: req.params.name }); + res.status(204).end(); +})); diff --git a/server/src/routes/incidents.ts b/server/src/routes/incidents.ts new file mode 100644 index 0000000..7651a3a --- /dev/null +++ b/server/src/routes/incidents.ts @@ -0,0 +1,51 @@ +import { Router } from "express"; +import { z } from "zod"; +import { requireAuth, requireRole } from "../auth/middleware.js"; +import { recordAudit } from "../services/audit.js"; +import { addNote, getIncident, listIncidents, MAX_NOTE, removeNote } from "../services/incidents.js"; +import { asyncHandler } from "../utils/asyncHandler.js"; + +export const incidentsRouter = Router(); + +// Everyone signed in can read what happened — it is made of what the Alerts page, the server pages and the Maintenance page already show. +// Writing a note is for operators. +incidentsRouter.use(requireAuth); + +const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN); + +incidentsRouter.get("/", asyncHandler(async (_req, res) => { + res.json({ incidents: await listIncidents() }); +})); + +incidentsRouter.get("/:id", asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" }); + const detail = await getIncident(id); + if (!detail) return res.status(404).json({ error: "not_found" }); + res.json(detail); +})); + +const noteSchema = z.object({ body: z.string().trim().min(1, "Write something first.").max(MAX_NOTE) }); + +incidentsRouter.post("/:id/notes", requireRole("operator"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + const parsed = noteSchema.safeParse(req.body); + if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.success ? undefined : parsed.error.issues[0]?.message }); + const actor = req.currentUser!; + const note = await addNote(id, actor.name ?? actor.email ?? actor.oidcSub, parsed.data.body); + if (!note) return res.status(404).json({ error: "not_found" }); + await recordAudit({ actor, category: "incident", action: "note", targetType: "incident", targetId: id }); + res.status(201).json({ id: note.id }); +})); + +incidentsRouter.delete("/:id/notes/:noteId", requireRole("operator"), asyncHandler(async (req, res) => { + const id = idOf(req.params.id); + const noteId = idOf(req.params.noteId); + if (!Number.isInteger(id) || !Number.isInteger(noteId)) return res.status(400).json({ error: "invalid_id" }); + const actor = req.currentUser!; + const outcome = await removeNote(id, noteId, { name: actor.name ?? actor.email ?? actor.oidcSub, isAdmin: actor.role === "admin" }); + if (outcome === "not_found") return res.status(404).json({ error: "not_found" }); + if (outcome === "forbidden") return res.status(403).json({ error: "forbidden", message: "A note can be removed by the person who wrote it, or an administrator." }); + await recordAudit({ actor, category: "incident", action: "note_removed", targetType: "incident", targetId: id }); + res.status(204).end(); +})); diff --git a/server/src/services/alertAcks.ts b/server/src/services/alertAcks.ts new file mode 100644 index 0000000..f32c817 --- /dev/null +++ b/server/src/services/alertAcks.ts @@ -0,0 +1,133 @@ +/** + * "I'm on it": an acknowledgement says that somebody has seen an alert, may say who is dealing with it and why, and may run out after a + * while. It changes nothing about the problem — the alert is still raised, still counted by /metrics and still on the wall display — but + * the people looking at the app can tell what nobody has looked at yet from what somebody has, and the sidebar and dashboard count only the + * former. + * + * An acknowledgement belongs to the alert's id (the same stable key the Alerts page keys rows on). It goes away on its own once the alert + * has been gone for a while, so the same problem coming back later is a new alert that nobody has looked at, not one that is "still being + * dealt with" from last month. + */ +import { and, eq, inArray, lt } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { alertAcks, users } from "../db/schema.js"; +import type { Alert } from "./alertTypes.js"; + +export interface AckInfo { + note: string; + assignee: { id: number | null; name: string } | null; + ackedBy: string; + ackedAt: string; + /** ISO — when it runs out; null means it lasts until the alert is gone. */ + until: string | null; + /** Still in force: has no end, or the end hasn't come. */ + active: boolean; +} + +/** An alert that has been absent this long loses its acknowledgement (a blip of a few minutes doesn't). */ +export const GONE_AFTER_MS = 30 * 60_000; +export const MAX_NOTE = 500; +export const MAX_HOURS = 24 * 30; + +export class AckError extends Error { + constructor( + readonly status: number, + readonly code: string, + message: string, + ) { + super(message); + } +} + +const shape = (r: typeof alertAcks.$inferSelect, now: Date): AckInfo => ({ + note: r.note, + assignee: r.assigneeName ? { id: r.assigneeUserId, name: r.assigneeName } : null, + ackedBy: r.ackedBy, + ackedAt: r.ackedAt, + until: r.until, + active: r.until === null || Date.parse(r.until) > now.getTime(), +}); + +export async function loadAcks(now: Date = new Date()): Promise> { + const rows = await db.select().from(alertAcks); + return new Map(rows.map((r) => [r.alertKey, shape(r, now)])); +} + +export async function loadAck(alertKey: string, now: Date = new Date()): Promise { + const [row] = await db.select().from(alertAcks).where(eq(alertAcks.alertKey, alertKey)).limit(1); + return row ? shape(row, now) : null; +} + +export interface SetAckInput { + alertId: string; + note: string; + /** A user to hand it to; null for nobody in particular. */ + assigneeId: number | null; + /** How long the acknowledgement lasts; null means until the alert is gone. */ + hours: number | null; + actor: string; +} + +/** Who an alert can be handed to: people who can act on it. */ +export async function assignableUsers(): Promise<{ id: number; name: string; role: string }[]> { + const rows = await db.select().from(users).where(inArray(users.role, ["admin", "operator"])); + return rows.map((u) => ({ id: u.id, name: u.name ?? u.email ?? u.oidcSub, role: u.role })).sort((a, b) => a.name.localeCompare(b.name)); +} + +/** Saves an acknowledgement, replacing an earlier one on the same alert. Returns what changed, for the audit log. */ +export async function setAck(input: SetAckInput, now: Date = new Date()): Promise<{ ack: AckInfo; before: AckInfo | null }> { + const note = input.note.trim(); + if (note.length > MAX_NOTE) throw new AckError(400, "note_too_long", `Keep the note under ${MAX_NOTE} characters.`); + if (input.hours !== null && !(Number.isInteger(input.hours) && input.hours >= 1 && input.hours <= MAX_HOURS)) throw new AckError(400, "invalid_duration", "Choose how long it lasts, up to thirty days — or until the alert is gone."); + + let assigneeName: string | null = null; + if (input.assigneeId !== null) { + const [u] = await db.select().from(users).where(eq(users.id, input.assigneeId)).limit(1); + if (!u || (u.role !== "admin" && u.role !== "operator")) throw new AckError(400, "invalid_assignee", "An alert can be handed to an administrator or an operator."); + assigneeName = u.name ?? u.email ?? u.oidcSub; + } + + const before = await loadAck(input.alertId, now); + const until = input.hours === null ? null : new Date(now.getTime() + input.hours * 3600_000).toISOString(); + const values = { note, assigneeUserId: input.assigneeId, assigneeName, ackedBy: input.actor, ackedAt: now.toISOString(), until, lastSeenAt: now.toISOString() }; + await db.insert(alertAcks).values({ alertKey: input.alertId, ...values }).onConflictDoUpdate({ target: alertAcks.alertKey, set: values }); + return { ack: (await loadAck(input.alertId, now))!, before }; +} + +export async function clearAck(alertId: string): Promise { + const before = await loadAck(alertId); + if (!before) return null; + await db.delete(alertAcks).where(eq(alertAcks.alertKey, alertId)); + return before; +} + +/** + * Called after every collection of the alerts: the acknowledgements of alerts that are there are marked as seen, and those of alerts that + * have been gone for half an hour are dropped. Returns how many were dropped. + */ +export async function reconcileAcks(presentIds: Set, now: Date = new Date()): Promise { + const rows = await db.select().from(alertAcks); + const seen = rows.filter((r) => presentIds.has(r.alertKey)).map((r) => r.id); + if (seen.length > 0) await db.update(alertAcks).set({ lastSeenAt: now.toISOString() }).where(inArray(alertAcks.id, seen)); + const cutoff = new Date(now.getTime() - GONE_AFTER_MS).toISOString(); + const gone = rows.filter((r) => !presentIds.has(r.alertKey) && r.lastSeenAt < cutoff).map((r) => r.id); + if (gone.length > 0) await db.delete(alertAcks).where(and(inArray(alertAcks.id, gone), lt(alertAcks.lastSeenAt, cutoff))); + return gone.length; +} + +/** What still needs somebody to look: the alerts that aren't silenced and that nobody has acknowledged (or whose acknowledgement has run out). */ +export function attentionCounts(alerts: Pick[], acks: Map): { critical: number; warning: number; acknowledged: number } { + let critical = 0; + let warning = 0; + let acknowledged = 0; + for (const a of alerts) { + if (a.silenced) continue; + if (acks.get(a.id)?.active) { + acknowledged++; + continue; + } + if (a.severity === "critical") critical++; + else if (a.severity === "warning") warning++; + } + return { critical, warning, acknowledged }; +} diff --git a/server/src/services/alerts.ts b/server/src/services/alerts.ts index 6b9d3d5..1146119 100644 --- a/server/src/services/alerts.ts +++ b/server/src/services/alerts.ts @@ -15,6 +15,9 @@ import { integrations, secrets, servers } from "../db/schema.js"; import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js"; import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js"; import { getBackupCoverage, staleBackups } from "./backupCoverage.js"; +import { recordFromReport } from "./incidents.js"; +import { reconcileAcks } from "./alertAcks.js"; +import { lastAttempt as lastSelfBackup, newestSnapshot, STALE_AFTER_HOURS as SELF_BACKUP_STALE_HOURS } from "./selfBackup.js"; import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createUptimeKumaAdapter } from "../integrations/uptimekuma/adapter.js"; import { createOsTicketAdapter } from "../integrations/osticket/adapter.js"; @@ -53,6 +56,7 @@ const SOURCES: Record = { gitea: { label: "Gitea", link: "/gitea" }, dockhand: { label: "Docker", link: "/docker" }, tailscale: { label: "Tailscale", link: "/tailscale" }, + homelab: { label: "Homelab Manager", link: "/settings/backup" }, pbs: { label: "Proxmox Backup", link: "/pbs" }, uptimekuma: { label: "Uptime Kuma", link: "/uptime-kuma" }, osticket: { label: "osTicket", link: "/osticket" }, @@ -262,6 +266,29 @@ export async function collectAlerts(): Promise { } }), + // Homelab Manager's own backups: if they are switched on, there should be a recent one. This is what turns a backup that has quietly + // stopped working — a full disk, a folder that went read-only — into something somebody hears about before they need it. + check("Homelab Manager's own backups", async () => { + const { selfBackup } = await getSettings(); + if (!selfBackup.enabled) return; + const newest = newestSnapshot(); + const last = await lastSelfBackup(); + const failing = last && !last.ok ? ` The last attempt failed: ${last.error}` : ""; + if (!newest) { + if (failing) push({ category: "backup", severity: "warning", type: "homelab", message: `Homelab Manager has no backup of its own yet.${failing}`, key: "self-backup" }); + return; + } + const ageHours = (Date.now() - Date.parse(newest.at)) / 3600_000; + if (ageHours <= SELF_BACKUP_STALE_HOURS) return; + push({ + category: "backup", + severity: ageHours >= 24 * 7 ? "critical" : "warning", + type: "homelab", + message: `The newest backup of Homelab Manager's own database is ${ageHours < 72 ? `${Math.round(ageHours)} h` : `${Math.floor(ageHours / 24)} days`} old.${failing}`, + key: "self-backup", + }); + }), + check("Backup verification", async () => { const { problems, sourceFailures } = await collectPbsProblems({ skipSilenced: false }); for (const p of problems) { @@ -475,6 +502,8 @@ export async function getAlerts(force: boolean): Promise { cache = { at: Date.now(), report }; + // Remembered for the incident record and the acknowledgements — in the background, and never allowed to hold up or break the answer. + void recordFromReport(report).then(() => reconcileAcks(new Set(report.alerts.map((a) => a.id)))).catch((err) => console.error("[alerts] couldn't update the incident record:", err)); return report; }) .finally(() => { diff --git a/server/src/services/incidents.ts b/server/src/services/incidents.ts new file mode 100644 index 0000000..aaec1a4 --- /dev/null +++ b/server/src/services/incidents.ts @@ -0,0 +1,326 @@ +/** + * Incidents: each time an alert is raised and later goes away is one incident, with the moment it began and the moment it ended — so + * "what happened on Tuesday night?" has an answer after the alert itself is long gone. Around each one the page puts together everything the + * app knows about that time: who acknowledged it and why, the maintenance windows, what was changed on the server, the update runs, the + * other alerts that came with it, and the notes people wrote. + * + * The record is made from the alerts as they are collected (and a check every five minutes so it doesn't depend on anybody looking), by + * comparing what is there now with what was open. An alert that isn't there is only called resolved after it has been missing for a couple + * of minutes in a row — much longer when a check couldn't run, since "not there" then may only mean "couldn't look" — and never in the first + * minutes after the app starts, when server alerts are held back on purpose. + */ +import { and, asc, desc, eq, gte, inArray, isNotNull, isNull, lte, ne, or } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { alertIncidents, auditLog, incidentNotes, maintenanceWindows, updateRuns } from "../db/schema.js"; +import type { Alert } from "./alertTypes.js"; +import { describeAudit, auditTimeToIso, type ActivityTone } from "./serverActivity.js"; +import { startupGraceRemainingMs } from "./healthMonitor.js"; +import { loadAck, type AckInfo } from "./alertAcks.js"; + +export type IncidentRow = typeof alertIncidents.$inferSelect; + +/** How long an alert must be missing before the incident is called over. */ +export const MISSING_GRACE_MS = 2 * 60_000; +/** The same, when a check couldn't run on the last look: not there may then only mean couldn't look. */ +export const MISSING_GRACE_BLIND_MS = 30 * 60_000; +/** Resolved incidents are kept this long. */ +export const KEEP_DAYS = 400; +/** Other alerts raised this close to an incident, about the same thing, are shown with it. */ +export const RELATED_WINDOW_MS = 15 * 60_000; + +const RANK: Record = { info: 0, warning: 1, critical: 2 }; + +export interface SnapshotInput { + alerts: Alert[]; + /** The checks that couldn't run on this look. */ + blind: number; + generatedAt?: string; +} + +export interface SnapshotResult { + opened: number[]; + resolved: number[]; +} + +/** + * Brings the incidents in line with the alerts as they are now. Information-only alerts aren't incidents. Safe to call as often as likes: + * the same look twice changes nothing. + */ +export async function recordAlertSnapshot(input: SnapshotInput, now: Date = new Date(), options: { inGrace?: boolean } = {}): Promise { + const iso = now.toISOString(); + const present = new Map(input.alerts.filter((a) => a.severity !== "info").map((a) => [a.id, a])); + const open = await db.select().from(alertIncidents).where(isNull(alertIncidents.resolvedAt)); + const openByKey = new Map(open.map((i) => [i.alertKey, i])); + const result: SnapshotResult = { opened: [], resolved: [] }; + + for (const [key, a] of present) { + const existing = openByKey.get(key); + if (existing) { + const peak = (RANK[a.severity] ?? 0) > (RANK[existing.peakSeverity] ?? 0) ? a.severity : existing.peakSeverity; + await db + .update(alertIncidents) + .set({ severity: a.severity, peakSeverity: peak, message: a.message, causedBy: a.causedBy?.name ?? null, lastSeenAt: iso, missingSince: null }) + .where(eq(alertIncidents.id, existing.id)); + } else { + const [row] = await db + .insert(alertIncidents) + .values({ + alertKey: key, + severity: a.severity, + peakSeverity: a.severity, + category: a.category, + source: a.source, + message: a.message, + link: a.link, + targetType: a.target?.type ?? null, + targetId: a.target?.id ?? null, + targetName: a.target?.name ?? null, + causedBy: a.causedBy?.name ?? null, + openedAt: iso, + lastSeenAt: iso, + }) + .returning({ id: alertIncidents.id }); + result.opened.push(row.id); + } + } + + // What was open and isn't there now. Nothing is called over while the app is still settling after a start, when server alerts are held back. + const grace = input.blind > 0 ? MISSING_GRACE_BLIND_MS : MISSING_GRACE_MS; + for (const inc of open) { + if (present.has(inc.alertKey)) continue; + if (options.inGrace) continue; + if (inc.missingSince === null) { + await db.update(alertIncidents).set({ missingSince: iso }).where(eq(alertIncidents.id, inc.id)); + } else if (now.getTime() - Date.parse(inc.missingSince) >= grace) { + // Over since the first look it was missing, not since this one. + await db.update(alertIncidents).set({ resolvedAt: inc.missingSince, missingSince: null }).where(eq(alertIncidents.id, inc.id)); + result.resolved.push(inc.id); + } + } + return result; +} + +/** Called by the collection of the alerts: records what it found, and never lets a problem here stop the alerts from being served. */ +export async function recordFromReport(report: { alerts: Alert[]; couldntCheck: unknown[] }, now: Date = new Date()): Promise { + try { + await recordAlertSnapshot({ alerts: report.alerts, blind: report.couldntCheck.length }, now, { inGrace: startupGraceRemainingMs(now.getTime()) > 0 }); + } catch (err) { + console.error("[incidents] couldn't record the alerts:", err); + } +} + +/** Resolved incidents older than the keeping time are removed (with their notes). */ +export async function pruneIncidents(now: Date = new Date()): Promise { + const cutoff = new Date(now.getTime() - KEEP_DAYS * 86_400_000).toISOString(); + const old = await db.select({ id: alertIncidents.id }).from(alertIncidents).where(and(isNotNull(alertIncidents.resolvedAt), lte(alertIncidents.resolvedAt, cutoff))); + if (old.length === 0) return 0; + await db.delete(alertIncidents).where(inArray(alertIncidents.id, old.map((o) => o.id))); + return old.length; +} + +// ─── Reading them ─────────────────────────────────────────────────────────── + +export interface IncidentSummary { + id: number; + alertKey: string; + severity: string; + peakSeverity: string; + category: string; + source: string; + message: string; + link: string | null; + target: { type: string; id: number; name: string } | null; + causedBy: string | null; + openedAt: string; + resolvedAt: string | null; + /** In milliseconds: until it was resolved, or until now while it is open. */ + durationMs: number; + open: boolean; + notes: number; +} + +const summarize = (r: IncidentRow, notes: number, now: Date): IncidentSummary => ({ + id: r.id, + alertKey: r.alertKey, + severity: r.severity, + peakSeverity: r.peakSeverity, + category: r.category, + source: r.source, + message: r.message, + link: r.link, + target: r.targetType && r.targetId !== null ? { type: r.targetType, id: r.targetId, name: r.targetName ?? `${r.targetType} #${r.targetId}` } : null, + causedBy: r.causedBy, + openedAt: r.openedAt, + resolvedAt: r.resolvedAt, + durationMs: Math.max(0, (r.resolvedAt ? Date.parse(r.resolvedAt) : now.getTime()) - Date.parse(r.openedAt)), + open: r.resolvedAt === null, + notes, +}); + +/** Open incidents first (newest first), then the resolved ones, newest first. */ +export async function listIncidents(options: { limit?: number; now?: Date } = {}): Promise { + const now = options.now ?? new Date(); + const limit = options.limit ?? 300; + const open = await db.select().from(alertIncidents).where(isNull(alertIncidents.resolvedAt)).orderBy(desc(alertIncidents.openedAt)); + const closed = await db.select().from(alertIncidents).where(isNotNull(alertIncidents.resolvedAt)).orderBy(desc(alertIncidents.openedAt)).limit(limit); + const rows = [...open, ...closed]; + const noteRows = rows.length > 0 ? await db.select({ incidentId: incidentNotes.incidentId }).from(incidentNotes).where(inArray(incidentNotes.incidentId, rows.map((r) => r.id))) : []; + const counts = new Map(); + for (const n of noteRows) counts.set(n.incidentId, (counts.get(n.incidentId) ?? 0) + 1); + return rows.map((r) => summarize(r, counts.get(r.id) ?? 0, now)); +} + +/** The open incident for each alert id, so the Alerts page can link to it. */ +export async function openIncidentIds(alertKeys: string[]): Promise> { + if (alertKeys.length === 0) return new Map(); + const rows = await db.select({ id: alertIncidents.id, key: alertIncidents.alertKey }).from(alertIncidents).where(and(isNull(alertIncidents.resolvedAt), inArray(alertIncidents.alertKey, alertKeys))); + return new Map(rows.map((r) => [r.key, r.id])); +} + +export type TimelineKind = "alert" | "ack" | "maintenance" | "change" | "health" | "run" | "related" | "note"; + +export interface TimelineEvent { + /** ISO. */ + at: string; + kind: TimelineKind; + tone: ActivityTone; + title: string; + detail: string | null; + actor: string | null; + /** For a related alert or a note: what it points at. */ + incidentId?: number; + noteId?: number; +} + +export interface IncidentDetail { + incident: IncidentSummary; + /** The acknowledgement in force now, if the alert is still open. */ + ack: AckInfo | null; + timeline: TimelineEvent[]; + notes: { id: number; author: string; body: string; at: string }[]; +} + +function parse(detail: string | null): Record { + if (!detail) return {}; + try { + const v = JSON.parse(detail); + return v && typeof v === "object" ? (v as Record) : {}; + } catch { + return {}; + } +} + +const words = (ms: number) => { + const m = Math.round(ms / 60_000); + if (m < 90) return `${Math.max(1, m)} min`; + const h = ms / 3600_000; + return h < 48 ? `${Math.round(h * 10) / 10} h` : `${Math.round(h / 24)} days`; +}; + +/** Everything the app knows about one incident, in the order it happened. */ +export async function getIncident(id: number, now: Date = new Date()): Promise { + const [row] = await db.select().from(alertIncidents).where(eq(alertIncidents.id, id)).limit(1); + if (!row) return null; + const notes = await db.select().from(incidentNotes).where(eq(incidentNotes.incidentId, id)).orderBy(asc(incidentNotes.createdAt)); + const incident = summarize(row, notes.length, now); + + const from = Date.parse(row.openedAt); + const to = row.resolvedAt ? Date.parse(row.resolvedAt) : now.getTime(); + const fromIso = new Date(from - 30 * 60_000).toISOString(); + const toIso = new Date(to + 15 * 60_000).toISOString(); + const events: TimelineEvent[] = []; + + events.push({ at: row.openedAt, kind: "alert", tone: row.peakSeverity === "critical" ? "bad" : "info", title: `Alert raised (${row.severity})`, detail: row.message, actor: null }); + if (row.peakSeverity !== row.severity) { + events.push({ at: row.resolvedAt ?? row.lastSeenAt, kind: "alert", tone: "neutral", title: `At its worst it was ${row.peakSeverity}`, detail: null, actor: null }); + } + if (row.causedBy) events.push({ at: row.openedAt, kind: "alert", tone: "info", title: `Held back: caused by ${row.causedBy}`, detail: "Folded under the alert for the machine it runs on.", actor: null }); + if (row.resolvedAt) events.push({ at: row.resolvedAt, kind: "alert", tone: "good", title: "Resolved", detail: `It lasted ${words(to - from)}.`, actor: null }); + + // Acknowledgements and hand-overs of this alert, from the audit log. + const audits = await db.select().from(auditLog).where(and(gte(auditLog.createdAt, fromIso.slice(0, 19).replace("T", " ")), lte(auditLog.createdAt, toIso.slice(0, 19).replace("T", " ")))).orderBy(asc(auditLog.createdAt)); + for (const a of audits) { + const d = parse(a.detail); + if (a.category === "alert") { + if (d.alertId !== row.alertKey) continue; + const described = describeAudit(a.category, a.action, d); + events.push({ at: auditTimeToIso(a.createdAt), kind: "ack", tone: described.tone, title: described.title, detail: described.detail, actor: a.actorLabel }); + continue; + } + // Things done to, or noticed about, the thing the alert is about. + if (row.targetType && row.targetId !== null && a.targetType === row.targetType && a.targetId === String(row.targetId) && a.category !== "maintenance") { + const described = describeAudit(a.category, a.action, d); + events.push({ at: auditTimeToIso(a.createdAt), kind: described.kind === "health" ? "health" : "change", tone: described.tone, title: described.title, detail: described.detail, actor: a.actorLabel }); + } + } + + if (row.targetType && row.targetId !== null) { + const windows = await db + .select() + .from(maintenanceWindows) + .where(and(eq(maintenanceWindows.targetType, row.targetType as "server"), eq(maintenanceWindows.targetId, row.targetId), lte(maintenanceWindows.startedAt, new Date(to).toISOString()), gte(maintenanceWindows.endsAt, new Date(from).toISOString()))); + for (const w of windows) { + if (w.endsAt <= w.startedAt) continue; // withdrawn before it began + events.push({ at: w.startedAt, kind: "maintenance", tone: "info", title: "Maintenance began — alerts silenced", detail: w.reason, actor: w.createdBy }); + if (w.endsAt <= toIso) events.push({ at: w.endsAt, kind: "maintenance", tone: "neutral", title: "Maintenance ended", detail: null, actor: null }); + } + if (row.targetType === "server") { + const runs = await db.select().from(updateRuns).where(and(eq(updateRuns.serverId, row.targetId), lte(updateRuns.startedAt, new Date(to).toISOString()), or(isNull(updateRuns.finishedAt), gte(updateRuns.finishedAt, new Date(from).toISOString()))!)); + for (const r of runs) { + events.push({ at: r.startedAt, kind: "run", tone: "info", title: `Update run started: ${r.templateName}`, detail: `by ${r.startedBy ?? "someone"}`, actor: r.startedBy }); + if (r.finishedAt) events.push({ at: r.finishedAt, kind: "run", tone: r.status === "done" ? "good" : "bad", title: r.status === "done" ? "Update run finished" : r.status === "cancelled" ? "Update run stopped being followed" : "Update run didn't finish", detail: r.message, actor: null }); + } + } + + // Other alerts about the same thing that began around the same time. + const related = await db + .select() + .from(alertIncidents) + .where(and(eq(alertIncidents.targetType, row.targetType), eq(alertIncidents.targetId, row.targetId), ne(alertIncidents.id, row.id), gte(alertIncidents.openedAt, new Date(from - RELATED_WINDOW_MS).toISOString()), lte(alertIncidents.openedAt, new Date(Math.max(to, from) + RELATED_WINDOW_MS).toISOString()))); + for (const r of related) events.push({ at: r.openedAt, kind: "related", tone: "info", title: `Another alert was raised: ${r.category} (${r.severity})`, detail: r.message, actor: null, incidentId: r.id }); + } + + for (const n of notes) events.push({ at: n.createdAt, kind: "note", tone: "neutral", title: `${n.author} wrote a note`, detail: n.body, actor: n.author, noteId: n.id }); + + events.sort((a, b) => a.at.localeCompare(b.at)); + return { incident, ack: row.resolvedAt ? null : await loadAck(row.alertKey, now), timeline: events, notes: notes.map((n) => ({ id: n.id, author: n.author, body: n.body, at: n.createdAt })) }; +} + +export const MAX_NOTE = 4000; + +export async function addNote(incidentId: number, author: string, body: string, now: Date = new Date()): Promise<{ id: number } | null> { + const [inc] = await db.select({ id: alertIncidents.id }).from(alertIncidents).where(eq(alertIncidents.id, incidentId)).limit(1); + if (!inc) return null; + const [row] = await db.insert(incidentNotes).values({ incidentId, author, body: body.trim(), createdAt: now.toISOString() }).returning({ id: incidentNotes.id }); + return row; +} + +export async function removeNote(incidentId: number, noteId: number, actor: { name: string; isAdmin: boolean }): Promise<"removed" | "not_found" | "forbidden"> { + const [n] = await db.select().from(incidentNotes).where(and(eq(incidentNotes.id, noteId), eq(incidentNotes.incidentId, incidentId))).limit(1); + if (!n) return "not_found"; + if (n.author !== actor.name && !actor.isAdmin) return "forbidden"; + await db.delete(incidentNotes).where(eq(incidentNotes.id, noteId)); + return "removed"; +} + +let timer: ReturnType | null = null; + +/** A look at the alerts every five minutes, so incidents are recorded whether or not anybody has the page open. Once a day it tidies up old ones. */ +export function initIncidentRecorder(lookAtAlerts: () => Promise): void { + if (timer) return; + let lastPrune = 0; + const tick = async () => { + try { + await lookAtAlerts(); + if (Date.now() - lastPrune > 24 * 3600_000) { + lastPrune = Date.now(); + await pruneIncidents(); + } + } catch (err) { + console.error("[incidents] the five-minute look failed:", err); + } + }; + setTimeout(() => void tick(), 60_000).unref(); + timer = setInterval(() => void tick(), 5 * 60_000); + timer.unref(); +} diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 531f6e0..796a447 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -13,6 +13,9 @@ import { parsePendingUpdates } from "./pendingUpdates.js"; import { parseHostHealth, poolIsFine } from "./hostHealth.js"; import { parseThresholds } from "./serverThresholds.js"; import { getCachedBackupCoverage } from "./backupCoverage.js"; +import { lastAttempt as lastSelfBackup, listSnapshots } from "./selfBackup.js"; +import { loadAcks, attentionCounts } from "./alertAcks.js"; +import { listIncidents } from "./incidents.js"; import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js"; import { computeSecretStatus } from "./secretStatus.js"; import { getSettings } from "./settingsStore.js"; @@ -115,6 +118,18 @@ export async function buildMetrics(now: Date = new Date()): Promise { console.error("[metrics] couldn't read the uptime:", err); } + // ── Homelab Manager's own backups ── + try { + const snapshots = listSnapshots(); + out.add("homelab_selfbackup_snapshots", "gauge", "How many backups of Homelab Manager's own database are kept.", snapshots.length); + const newest = snapshots.find((s) => s.kind === "scheduled" || s.kind === "manual"); + if (newest) out.add("homelab_selfbackup_last_success_timestamp_seconds", "gauge", "When the newest backup of Homelab Manager's own database was made, as a Unix time.", Math.floor(Date.parse(newest.at) / 1000)); + const last = await lastSelfBackup(); + if (last) out.add("homelab_selfbackup_last_attempt_ok", "gauge", "1 if the last attempt to back up Homelab Manager's own database worked, 0 if it failed.", last.ok ? 1 : 0); + } catch (err) { + console.error("[metrics] couldn't read the backups folder:", err); + } + // ── Alerts ── try { const report = await getAlerts(false); @@ -127,6 +142,9 @@ export async function buildMetrics(now: Date = new Date()): Promise { } } for (const severity of ["critical", "warning", "info"] as const) out.add("homelab_alerts", "gauge", "Active alerts by severity (not counting silenced ones).", report.counts[severity], { severity }); + // Somebody has said they are dealing with these (still raised, and still counted above). + out.add("homelab_alerts_acknowledged", "gauge", "Alerts somebody has acknowledged and that are not silenced by maintenance.", attentionCounts(report.alerts, await loadAcks(now)).acknowledged); + out.add("homelab_incidents_open", "gauge", "Incidents that have begun and not yet ended.", (await listIncidents({ limit: 1, now })).filter((i) => i.open).length); out.add("homelab_alerts_silenced", "gauge", "Alerts held back by a maintenance window.", report.counts.silenced); out.add("homelab_alerts_grouped", "gauge", "Alerts folded under a server that is down.", report.counts.grouped); } catch (err) { diff --git a/server/src/services/selfBackup.ts b/server/src/services/selfBackup.ts new file mode 100644 index 0000000..d1f7c7f --- /dev/null +++ b/server/src/services/selfBackup.ts @@ -0,0 +1,323 @@ +/** + * Backups of Homelab Manager itself. Everything it knows is one SQLite file, so a backup is a consistent copy of that file (`VACUUM INTO`, + * which is safe while the app is working) kept in the data folder's `backups/`: taken every day, the newest few kept, each one checked as + * it is made, and downloadable, uploadable and restorable from Settings → Backup. + * + * What a backup is not: it does not hold `CREDENTIALS_ENCRYPTION_KEY`, which decrypts the credentials stored in it, and it isn't off the + * machine until someone copies it there. Both are said where backups are shown. + * + * A restore can't swap a database that is open, so it is staged (checked, then copied beside the database) and takes effect when the app + * next starts — see db/applyRestore.ts, which does it before anything opens the file. + */ +import { createClient } from "@libsql/client"; +import { copyFileSync, chmodSync, existsSync, mkdirSync, mkdtempSync, openSync, readdirSync, readFileSync, readSync, closeSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { client, dbPath } from "../db/client.js"; +import { PENDING_FILE, PENDING_NOTE, RESULT_NOTE, type RestoreResult } from "../db/restoreFiles.js"; +import { getInternalFlag, setInternalFlag } from "./settingsStore.js"; + +export type SnapshotKind = "scheduled" | "manual" | "pre-restore" | "uploaded"; + +export interface SnapshotInfo { + name: string; + kind: SnapshotKind; + sizeBytes: number; + /** ISO — when the file was written. */ + at: string; +} + +export interface SnapshotCheck { + ok: boolean; + /** SQLite's own verdict ("ok", or what it found). */ + integrity: string | null; + /** How many schema changes it has had; goes up with each release that changes the database. */ + migrations: number | null; + /** How many this version of the app knows. */ + appMigrations: number; + /** Made by a newer version of the app than this one: it can't be restored here. */ + newerThanApp: boolean; + counts: { users: number; servers: number; integrations: number } | null; + /** Why it isn't usable, when it isn't. */ + problem: string | null; +} + +const NAME = /^homelab-manager-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}(?:-(manual|pre-restore|uploaded))?(?:-\d+)?\.sqlite$/; +export const isSnapshotName = (name: string) => NAME.test(name); + +const dataDir = () => dirname(dbPath); +export const backupDir = () => join(dataDir(), "backups"); + +const stamp = (d: Date) => d.toISOString().slice(0, 19).replace(/[:T]/g, "-"); + +function kindOf(name: string): SnapshotKind { + const m = NAME.exec(name); + return (m?.[1] as SnapshotKind | undefined) ?? "scheduled"; +} + +/** How many schema changes this version of the app has (the entries in its migration journal). */ +export function appMigrationCount(): number { + try { + const journal = JSON.parse(readFileSync(join(dirname(fileURLToPath(import.meta.url)), "..", "..", "drizzle", "meta", "_journal.json"), "utf8")) as { entries?: unknown[] }; + return journal.entries?.length ?? 0; + } catch { + return 0; + } +} + +function hasSqliteHeader(file: string): boolean { + try { + const fd = openSync(file, "r"); + try { + const buf = Buffer.alloc(16); + readSync(fd, buf, 0, 16, 0); + return buf.toString("latin1") === "SQLite format 3\0"; + } finally { + closeSync(fd); + } + } catch { + return false; + } +} + +/** A scratch folder for looking at a copy of a backup, and the way to be rid of it (best effort: some systems hold an opened database until the process ends). */ +function scratch(): { file: string; done: () => void } { + const folder = mkdtempSync(join(tmpdir(), "hlm-backup-")); + return { + file: join(folder, "db.sqlite"), + done: () => { + try { + rmSync(folder, { recursive: true, force: true }); + } catch { + // left for the system's own tidying of its temporary folder + } + }, + }; +} + +/** + * Looks at a backup: does SQLite think it is sound, is it one of ours, and is it from this version or an older one? It looks at a copy in a + * temporary folder, never at the file itself, so a backup in the backups folder is never held open — it can always be deleted, anywhere. + */ +export async function checkSnapshotFile(file: string): Promise { + if (!existsSync(file)) return checkOpened(file); + if (!hasSqliteHeader(file)) return checkOpened(file); + const tmp = scratch(); + try { + copyFileSync(file, tmp.file); + return await checkOpened(tmp.file); + } finally { + tmp.done(); + } +} + +/** The check itself, on a file nobody else cares about the handle of. */ +async function checkOpened(file: string): Promise { + const appMigrations = appMigrationCount(); + const fail = (problem: string, extra: Partial = {}): SnapshotCheck => ({ ok: false, integrity: null, migrations: null, appMigrations, newerThanApp: false, counts: null, problem, ...extra }); + if (!existsSync(file)) return fail("The file isn't there."); + if (!hasSqliteHeader(file)) return fail("That isn't a database file."); + const c = createClient({ url: `file:${file}` }); + try { + const one = async (sql: string) => Object.values((await c.execute(sql)).rows[0] ?? {})[0]; + const integrity = String(await one("PRAGMA integrity_check")); + const tables = new Set((await c.execute("select name from sqlite_master where type = 'table'")).rows.map((r) => String(r.name))); + for (const needed of ["users", "servers", "integrations", "settings", "__drizzle_migrations"]) { + if (!tables.has(needed)) return fail(`It isn't a Homelab Manager database (no "${needed}" table).`, { integrity }); + } + const migrations = Number(await one("select count(*) from __drizzle_migrations")); + const counts = { users: Number(await one("select count(*) from users")), servers: Number(await one("select count(*) from servers")), integrations: Number(await one("select count(*) from integrations")) }; + const newerThanApp = appMigrations > 0 && migrations > appMigrations; + if (integrity !== "ok") return fail(`SQLite found problems in it: ${integrity}`, { integrity, migrations, counts, newerThanApp }); + if (newerThanApp) return fail("It was made by a newer version of Homelab Manager than this one, and an older version can't use it. Update first.", { integrity, migrations, counts, newerThanApp }); + return { ok: true, integrity, migrations, appMigrations, newerThanApp: false, counts, problem: null }; + } catch (err) { + return fail(`It couldn't be read as a database: ${err instanceof Error ? err.message : String(err)}`); + } finally { + c.close(); + } +} + +/** A name for a new file that isn't taken. */ +function freshName(kind: SnapshotKind, now: Date): string { + const base = `homelab-manager-${stamp(now)}${kind === "scheduled" ? "" : `-${kind}`}`; + let name = `${base}.sqlite`; + for (let n = 2; existsSync(join(backupDir(), name)); n++) name = `${base}-${n}.sqlite`; + return name; +} + +export interface LastAttempt { + at: string; + ok: boolean; + kind: SnapshotKind; + error: string | null; + name: string | null; +} +const ATTEMPT_FLAG = "selfBackupLastAttempt"; + +export async function lastAttempt(): Promise { + try { + const v = await getInternalFlag(ATTEMPT_FLAG); + return v ? (JSON.parse(v) as LastAttempt) : null; + } catch { + return null; + } +} + +/** Writes a consistent copy of the database into the backups folder and checks it. Throws, saying why, if it can't — and records the attempt either way. */ +export async function createSnapshot(kind: SnapshotKind = "manual", now: Date = new Date()): Promise<{ info: SnapshotInfo; check: SnapshotCheck }> { + const tmp = scratch(); + try { + mkdirSync(backupDir(), { recursive: true }); + const name = freshName(kind, now); + const file = join(backupDir(), name); + // Made and checked in a temporary folder first, and only then put in the backups folder: what is there is always whole and sound. + await client.execute(`VACUUM INTO '${tmp.file.replace(/'/g, "''")}'`); + const check = await checkOpened(tmp.file); + if (!check.ok) throw new Error(`The copy didn't check out, so it was thrown away: ${check.problem}`); + copyFileSync(tmp.file, file); + try { + chmodSync(file, 0o600); + } catch { + // not every file system has modes + } + const info: SnapshotInfo = { name, kind, sizeBytes: statSync(file).size, at: now.toISOString() }; + await setInternalFlag(ATTEMPT_FLAG, JSON.stringify({ at: now.toISOString(), ok: true, kind, error: null, name } satisfies LastAttempt)); + return { info, check }; + } catch (err) { + const error = err instanceof Error ? err.message : String(err); + await setInternalFlag(ATTEMPT_FLAG, JSON.stringify({ at: now.toISOString(), ok: false, kind, error, name: null } satisfies LastAttempt)); + throw err; + } finally { + tmp.done(); + } +} + +export function listSnapshots(): SnapshotInfo[] { + if (!existsSync(backupDir())) return []; + return readdirSync(backupDir()) + .filter(isSnapshotName) + .map((name) => { + const st = statSync(join(backupDir(), name)); + return { name, kind: kindOf(name), sizeBytes: st.size, at: st.mtime.toISOString() }; + }) + .sort((a, b) => b.at.localeCompare(a.at) || b.name.localeCompare(a.name)); +} + +/** What is kept: the newest `keep` scheduled backups, and a few of each other kind (a restore's "before" copy is worth keeping longer than a day). */ +export const KEEP_OTHER: Record, number> = { manual: 10, "pre-restore": 3, uploaded: 3 }; + +export function pruneSnapshots(keep: number): string[] { + const removed: string[] = []; + const all = listSnapshots(); + const limitFor = (kind: SnapshotKind) => (kind === "scheduled" ? Math.max(1, keep) : KEEP_OTHER[kind]); + for (const kind of ["scheduled", "manual", "pre-restore", "uploaded"] as const) { + for (const s of all.filter((x) => x.kind === kind).slice(limitFor(kind))) { + rmSync(join(backupDir(), s.name), { force: true }); + removed.push(s.name); + } + } + return removed; +} + +export function snapshotPath(name: string): string | null { + if (!isSnapshotName(name)) return null; + const file = join(backupDir(), name); + return existsSync(file) ? file : null; +} + +export function deleteSnapshot(name: string): boolean { + const file = snapshotPath(name); + if (!file) return false; + rmSync(file, { force: true }); + return true; +} + +/** A file somebody uploaded, kept as a backup if it checks out (and thrown away if not). */ +export async function importUploaded(bytes: Buffer, now: Date = new Date()): Promise<{ info: SnapshotInfo; check: SnapshotCheck }> { + const tmp = scratch(); + try { + writeFileSync(tmp.file, bytes); + const check = await checkOpened(tmp.file); + if (!check.ok) throw new Error(check.problem ?? "That file isn't usable."); + mkdirSync(backupDir(), { recursive: true }); + const name = freshName("uploaded", now); + const file = join(backupDir(), name); + copyFileSync(tmp.file, file); + try { + chmodSync(file, 0o600); + } catch { + // not every file system has modes + } + return { info: { name, kind: "uploaded", sizeBytes: statSync(file).size, at: now.toISOString() }, check }; + } finally { + tmp.done(); + } +} + +// ─── Restoring ────────────────────────────────────────────────────────────── + +export interface StagedRestore { + from: string; + stagedAt: string; + by: string | null; +} + +export function stagedRestore(): StagedRestore | null { + try { + if (!existsSync(join(dataDir(), PENDING_FILE))) return null; + const note = JSON.parse(readFileSync(join(dataDir(), PENDING_NOTE), "utf8")) as { from?: string; stagedAt?: string; by?: string | null }; + return { from: note.from ?? "a backup", stagedAt: note.stagedAt ?? "", by: note.by ?? null }; + } catch { + return existsSync(join(dataDir(), PENDING_FILE)) ? { from: "a backup", stagedAt: "", by: null } : null; + } +} + +/** Checks a backup and stages it to replace the database the next time the app starts. Throws, saying why, if it can't be used. */ +export async function stageRestore(name: string, by: string | null, now: Date = new Date()): Promise { + const file = snapshotPath(name); + if (!file) throw new Error("There is no backup with that name."); + const check = await checkSnapshotFile(file); + if (!check.ok) throw new Error(check.problem ?? "That backup isn't usable."); + const pending = join(dataDir(), PENDING_FILE); + const tmp = `${pending}.copying`; + copyFileSync(file, tmp); + renameSync(tmp, pending); // whole, or not there at all: the start-up step never sees half a file + try { + chmodSync(pending, 0o600); + } catch { + // not every file system has modes + } + writeFileSync(join(dataDir(), PENDING_NOTE), JSON.stringify({ from: name, stagedAt: now.toISOString(), by })); + return check; +} + +export function cancelStagedRestore(): boolean { + const had = existsSync(join(dataDir(), PENDING_FILE)); + rmSync(join(dataDir(), PENDING_FILE), { force: true }); + rmSync(join(dataDir(), PENDING_NOTE), { force: true }); + return had; +} + +/** What the last restore did, once, for the page to say. */ +export function lastRestore(): RestoreResult | null { + try { + return JSON.parse(readFileSync(join(dataDir(), RESULT_NOTE), "utf8")) as RestoreResult; + } catch { + return null; + } +} + +export function dismissLastRestore(): void { + rmSync(join(dataDir(), RESULT_NOTE), { force: true }); +} + +// ─── How current the backups are ──────────────────────────────────────────── + +/** Backups older than this are an alert: a day's slack on top of the daily schedule. */ +export const STALE_AFTER_HOURS = 50; + +export function newestSnapshot(): SnapshotInfo | null { + return listSnapshots().find((s) => s.kind === "scheduled" || s.kind === "manual") ?? null; +} diff --git a/server/src/services/selfBackupScheduler.ts b/server/src/services/selfBackupScheduler.ts new file mode 100644 index 0000000..5e43166 --- /dev/null +++ b/server/src/services/selfBackupScheduler.ts @@ -0,0 +1,65 @@ +import schedule from "node-schedule"; +import { getSettings } from "./settingsStore.js"; +import { createSnapshot, lastAttempt, newestSnapshot, pruneSnapshots } from "./selfBackup.js"; +import { notify } from "./notify.js"; + +let currentJob: schedule.Job | null = null; + +/** The cron rule for "every day at HH:MM". */ +export function cronForBackup(time: string): string { + const [h, m] = time.split(":").map(Number); + return `${Number.isFinite(m) ? m : 30} ${Number.isFinite(h) ? h : 3} * * *`; +} + +/** One scheduled backup: take it, then drop the oldest beyond how many are kept. Says so through the channels if it fails. */ +export async function runScheduledBackup(now: Date = new Date()): Promise<{ name: string } | null> { + const { selfBackup, notifications } = await getSettings(); + try { + const { info } = await createSnapshot("scheduled", now); + pruneSnapshots(selfBackup.keep); + return { name: info.name }; + } catch (err) { + console.error("[selfBackup] the scheduled backup failed:", err instanceof Error ? err.message : err); + if (notifications.healthAlerts !== false) { + await notify("Homelab Manager — its own backup failed", `The scheduled backup of Homelab Manager's database didn't work: ${err instanceof Error ? err.message : String(err)}`).catch(() => {}); + } + return null; + } +} + +/** The next time it will run, or null when it is off. */ +export function nextBackupRun(): Date | null { + const next = currentJob?.nextInvocation(); + return next ? new Date(next.getTime()) : null; +} + +/** (Re)schedules the daily backup per the current settings. Call again after they change. */ +export async function scheduleSelfBackup(): Promise { + if (currentJob) { + currentJob.cancel(); + currentJob = null; + } + const { selfBackup, notifications } = await getSettings(); + if (!selfBackup.enabled) { + console.log("[selfBackup] daily backups are off, nothing scheduled"); + return; + } + currentJob = schedule.scheduleJob({ rule: cronForBackup(selfBackup.time), tz: notifications.timezone }, () => { + void runScheduledBackup(); + }); + console.log(`Daily backup scheduled at ${selfBackup.time} (${notifications.timezone})`); +} + +/** Schedules it, and takes one soon if the app has been off through its time (or has never taken one). */ +export async function initSelfBackupScheduler(): Promise { + await scheduleSelfBackup(); + const { selfBackup } = await getSettings(); + if (!selfBackup.enabled) return; + const newest = newestSnapshot(); + const overdue = !newest || Date.now() - Date.parse(newest.at) > 26 * 3600_000; + // Only retry a failing one at the next scheduled time, not at every restart. + const last = await lastAttempt(); + if (overdue && !(last && !last.ok && Date.now() - Date.parse(last.at) < 6 * 3600_000)) { + setTimeout(() => void runScheduledBackup(), 30_000).unref(); + } +} diff --git a/server/src/services/serverActivity.ts b/server/src/services/serverActivity.ts index 42305f3..fd473b3 100644 --- a/server/src/services/serverActivity.ts +++ b/server/src/services/serverActivity.ts @@ -72,6 +72,17 @@ export function describeAudit(category: string, action: string, detail: Record= 48 ? `${Math.round(detail.hours / 24)} days` : `${detail.hours} h`}` : "until the alert is gone"; + switch (action) { + case "ack": + return { title: assignee ? `Acknowledged, handed to ${assignee}` : "Acknowledged", tone: "info", kind: "change", detail: [note, lasts].filter(Boolean).join(" · ") }; + case "unack": + return { title: "Acknowledgement taken back", tone: "neutral", kind: "change", detail: null }; + } + } if (category === "update_run") { const template = typeof detail.template === "string" ? detail.template : null; switch (action) { diff --git a/server/src/services/settingsStore.ts b/server/src/services/settingsStore.ts index 821389b..db0b041 100644 --- a/server/src/services/settingsStore.ts +++ b/server/src/services/settingsStore.ts @@ -129,6 +129,15 @@ export interface AgentSettings { autoUpdate: boolean; } +/** The daily backup of Homelab Manager's own database (see services/selfBackup.ts). On unless an administrator turns it off. */ +export interface SelfBackupSettings { + enabled: boolean; + /** "HH:MM", in the notification time zone. */ + time: string; + /** How many of the daily backups are kept; the oldest beyond that are deleted. */ + keep: number; +} + /** What the inventory needs to work out a running cost. Without a price per kWh it shows draw but no cost. */ export interface InventorySettings { /** A short name shown after amounts ("SEK", "kr", "$"); may be blank. */ @@ -190,6 +199,7 @@ export interface AppSettings { digest: DigestSettings; signIn: SignInSettings; inventory: InventorySettings; + selfBackup: SelfBackupSettings; agents: AgentSettings; consistency: ConsistencySettings; nameGenerator: NameGeneratorSettings; @@ -227,6 +237,7 @@ const DEFAULTS: AppSettings = { digest: { enabled: false, day: 1, time: "08:00" }, inventory: { currency: "", pricePerKwh: null }, agents: { autoUpdate: false }, + selfBackup: { enabled: true, time: "03:30", keep: 14 }, signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" }, healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48 }, // Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every diff --git a/server/src/services/weeklyDigest.ts b/server/src/services/weeklyDigest.ts index 243db5f..5c4b48d 100644 --- a/server/src/services/weeklyDigest.ts +++ b/server/src/services/weeklyDigest.ts @@ -20,6 +20,8 @@ import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js"; import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js"; import { getSettings } from "./settingsStore.js"; import { getUptime } from "./uptimeHistory.js"; +import { attentionCounts, loadAcks } from "./alertAcks.js"; +import { listIncidents } from "./incidents.js"; export const DIGEST_DAYS = 7; const DAY_MS = 24 * 3600_000; @@ -90,11 +92,30 @@ export async function buildDigest(now: Date = new Date()): Promise { if (report.counts.silenced > 0) held.push(`${report.counts.silenced} silenced by maintenance`); if (report.counts.grouped > 0) held.push(`${report.counts.grouped} folded under a server that is down`); if (held.length > 0) lines.push(`(${held.join(", ")})`); + const acknowledged = attentionCounts(report.alerts, await loadAcks(now)).acknowledged; + if (acknowledged > 0) lines.push(`(${acknowledged} of them acknowledged — somebody is on it)`); sections.push({ heading: "Right now", lines, attention: active.length > 0 || report.couldntCheck.length > 0 }); } catch (err) { console.error("[digest] couldn't read the alerts:", err); } + // ── Incidents ── + try { + const since = now.getTime() - DIGEST_DAYS * DAY_MS; + const all = await listIncidents({ now }); + const begun = all.filter((i) => Date.parse(i.openedAt) >= since); + const ended = all.filter((i) => i.resolvedAt !== null && Date.parse(i.resolvedAt) >= since); + const open = all.filter((i) => i.open); + if (begun.length > 0 || ended.length > 0) { + const lines = [`${plural(begun.length, "incident")} began, ${ended.length} ended, ${open.length} still open.`]; + const longest = [...ended].sort((a, b) => b.durationMs - a.durationMs).slice(0, 3); + lines.push(...longest.map((i) => `- ${i.message} — lasted ${spanWords(i.durationMs / 60_000)}`)); + sections.push({ heading: "Incidents", lines, attention: open.length > 0 }); + } + } catch (err) { + console.error("[digest] couldn't read the incidents:", err); + } + // ── Uptime ── try { const uptime = await getUptime(now, DIGEST_DAYS); diff --git a/server/test/ackAndIncidents.test.ts b/server/test/ackAndIncidents.test.ts new file mode 100644 index 0000000..d5ccccb --- /dev/null +++ b/server/test/ackAndIncidents.test.ts @@ -0,0 +1,453 @@ +/** + * Acknowledging and handing over alerts, and the incident record: when an alert began and ended, and what happened around it. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import type { Server } from "node:http"; + +const dir = mkdtempSync(join(tmpdir(), "hm-incidents-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers, auditLog, maintenanceWindows, updateRuns, alertAcks, alertIncidents, incidentNotes } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const acks = await import("../src/services/alertAcks.js"); +const inc = await import("../src/services/incidents.js"); +const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js"); +const { getServerActivity } = await import("../src/services/serverActivity.js"); +const { buildMetrics } = await import("../src/services/metrics.js"); +const { buildDigest } = await import("../src/services/weeklyDigest.js"); +const { alertsRouter } = await import("../src/routes/alerts.js"); +const { incidentsRouter } = await import("../src/routes/incidents.js"); +const { hashToken } = await import("../src/services/tokens.js"); +const { eq } = await import("drizzle-orm"); + +await runMigrations(); + +const T0 = new Date("2026-10-17T12:00:00.000Z"); +const at = (minutes: number) => new Date(T0.getTime() + minutes * 60_000); +const alert = (id: string, extra: Record = {}) => + ({ id, severity: "warning" as const, category: "disk" as const, source: "Server", message: `${id} is a problem`, link: null, silenced: false, target: null, ...extra }) as never; + +let adminId = 0; +let operatorId = 0; +let viewerId = 0; +before(async () => { + const rows = await db + .insert(users) + .values([ + { oidcSub: "ada", name: "Ada", role: "admin" }, + { oidcSub: "olle", name: "Olle", role: "operator" }, + { oidcSub: "vera", name: "Vera", role: "viewer" }, + ]) + .returning(); + [adminId, operatorId, viewerId] = rows.map((r) => r.id); +}); +after(() => { + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } +}); + +describe("acknowledging an alert", () => { + test("a note, who has it and for how long are kept; a second one replaces the first; taking it back removes it", async () => { + const first = await acks.setAck({ alertId: "disk:server:1:/", note: " looking at it ", assigneeId: operatorId, hours: 4, actor: "Ada" }, T0); + assert.equal(first.before, null); + assert.equal(first.ack.note, "looking at it"); + assert.deepEqual(first.ack.assignee, { id: operatorId, name: "Olle" }); + assert.equal(first.ack.ackedBy, "Ada"); + assert.equal(first.ack.until, at(240).toISOString()); + assert.equal(first.ack.active, true); + + const second = await acks.setAck({ alertId: "disk:server:1:/", note: "", assigneeId: null, hours: null, actor: "Olle" }, at(10)); + assert.equal(second.before!.note, "looking at it"); + assert.equal(second.ack.assignee, null); + assert.equal(second.ack.until, null, "until the alert is gone"); + assert.equal((await db.select().from(alertAcks)).length, 1, "one per alert"); + + assert.equal((await acks.clearAck("disk:server:1:/"))!.ackedBy, "Olle"); + assert.equal(await acks.clearAck("disk:server:1:/"), null); + assert.equal((await acks.loadAcks()).size, 0); + }); + + test("only people who can act can be handed an alert, and the note and the time are limited", async () => { + const refused = async (input: Partial[0]>, code: string) => + assert.rejects(acks.setAck({ alertId: "x", note: "", assigneeId: null, hours: null, actor: "Ada", ...input }, T0), (e: unknown) => e instanceof acks.AckError && e.code === code, code); + await refused({ assigneeId: viewerId }, "invalid_assignee"); + await refused({ assigneeId: 99999 }, "invalid_assignee"); + await refused({ note: "x".repeat(acks.MAX_NOTE + 1) }, "note_too_long"); + await refused({ hours: 0 }, "invalid_duration"); + await refused({ hours: acks.MAX_HOURS + 1 }, "invalid_duration"); + await refused({ hours: 1.5 }, "invalid_duration"); + assert.deepEqual((await acks.assignableUsers()).map((u) => u.name), ["Ada", "Olle"], "no viewers in the list"); + }); + + test("it runs out when it was meant to, and only then does the alert count as unattended again", async () => { + await acks.setAck({ alertId: "a:1", note: "", assigneeId: null, hours: 2, actor: "Ada" }, T0); + await acks.setAck({ alertId: "a:2", note: "", assigneeId: null, hours: null, actor: "Ada" }, T0); + const alerts = [ + { id: "a:1", severity: "critical", silenced: false }, + { id: "a:2", severity: "warning", silenced: false }, + { id: "a:3", severity: "critical", silenced: false }, + { id: "a:4", severity: "warning", silenced: true }, + ] as { id: string; severity: "critical" | "warning" | "info"; silenced: boolean }[]; + assert.deepEqual(acks.attentionCounts(alerts, await acks.loadAcks(at(60))), { critical: 1, warning: 0, acknowledged: 2 }, "a:3 is the only one nobody has looked at; a silenced one is neither"); + const later = await acks.loadAcks(at(150)); + assert.equal(later.get("a:1")!.active, false); + assert.deepEqual(acks.attentionCounts(alerts, later), { critical: 2, warning: 0, acknowledged: 1 }, "a:1 is unattended again; a:2 has no end"); + await db.delete(alertAcks); + }); + + test("it goes when the alert has been gone for half an hour — not for a blip, and not if the alert is back", async () => { + await acks.setAck({ alertId: "gone", note: "", assigneeId: null, hours: null, actor: "Ada" }, T0); + await acks.setAck({ alertId: "stays", note: "", assigneeId: null, hours: null, actor: "Ada" }, T0); + assert.equal(await acks.reconcileAcks(new Set(["stays"]), at(10)), 0, "ten minutes missing is a blip"); + assert.equal(await acks.reconcileAcks(new Set(["stays"]), at(29)), 0); + assert.equal(await acks.reconcileAcks(new Set(["stays"]), at(31)), 1); + assert.deepEqual([...(await acks.loadAcks()).keys()], ["stays"]); + // a problem that returns later is a new one, nobody having looked at it + assert.equal((await acks.loadAcks()).has("gone"), false); + await db.delete(alertAcks); + }); +}); + +describe("the incident record", () => { + const look = (alerts: unknown[], minutes: number, options: { blind?: number; inGrace?: boolean } = {}) => inc.recordAlertSnapshot({ alerts: alerts as never, blind: options.blind ?? 0 }, at(minutes), { inGrace: options.inGrace }); + const rows = () => db.select().from(alertIncidents).orderBy(alertIncidents.id); + + test("an alert that appears begins an incident; the same look again changes nothing; information-only alerts aren't incidents", async () => { + await db.delete(alertIncidents); + const a = alert("disk:server:1:/", { target: { type: "server", id: 1, name: "elsa" } }); + const r1 = await look([a, alert("info:thing", { severity: "info" })], 0); + assert.equal(r1.opened.length, 1); + assert.deepEqual((await look([a], 1)).opened, []); + const [row] = await rows(); + assert.equal(row.alertKey, "disk:server:1:/"); + assert.equal(row.openedAt, at(0).toISOString()); + assert.equal(row.resolvedAt, null); + assert.equal(row.targetName, "elsa"); + assert.equal(row.lastSeenAt, at(1).toISOString()); + assert.equal((await rows()).length, 1); + }); + + test("it keeps the worst it got to and the latest wording", async () => { + await look([alert("disk:server:1:/", { severity: "critical", message: "now 97% full", target: { type: "server", id: 1, name: "elsa" } })], 5); + await look([alert("disk:server:1:/", { severity: "warning", message: "back to 91% full", target: { type: "server", id: 1, name: "elsa" } })], 6); + const [row] = await rows(); + assert.equal(row.severity, "warning"); + assert.equal(row.peakSeverity, "critical"); + assert.equal(row.message, "back to 91% full"); + }); + + test("it is only called over after being missing for a couple of minutes in a row, and then it ended when it was first missing", async () => { + const a = alert("disk:server:1:/", { target: { type: "server", id: 1, name: "elsa" } }); + await look([], 10); + let [row] = await rows(); + assert.equal(row.resolvedAt, null); + assert.equal(row.missingSince, at(10).toISOString()); + await look([a], 11); // it was only a blip + [row] = await rows(); + assert.equal(row.missingSince, null, "back before it was called over: the count starts again"); + assert.equal(row.resolvedAt, null); + + await look([], 20); + await look([], 21); + [row] = await rows(); + assert.equal(row.resolvedAt, null, "one minute isn't enough"); + const done = await look([], 23); + assert.equal(done.resolved.length, 1); + [row] = await rows(); + assert.equal(row.resolvedAt, at(20).toISOString(), "it ended when it first wasn't there, not when that was confirmed"); + assert.equal(row.missingSince, null); + }); + + test("when it comes back it is a new incident, and the old one stays as it was", async () => { + const a = alert("disk:server:1:/", { target: { type: "server", id: 1, name: "elsa" } }); + const r = await look([a], 60); + assert.equal(r.opened.length, 1); + const all = await rows(); + assert.equal(all.length, 2); + assert.equal(all[0].resolvedAt, at(20).toISOString()); + assert.equal(all[1].resolvedAt, null); + assert.equal(all[1].peakSeverity, "warning"); + }); + + test("after a check couldn't run it takes much longer to call anything over, and never in the first minutes after the app starts", async () => { + await db.delete(alertIncidents); + const a = alert("proxmox-backup:1:pve1", { category: "backup" }); + await look([a], 0); + await look([], 5, { blind: 1 }); + await look([], 20, { blind: 1 }); + assert.equal((await rows())[0].resolvedAt, null, "fifteen minutes missing isn't enough while a check is failing"); + assert.equal((await look([], 36, { blind: 1 })).resolved.length, 1); + + await db.delete(alertIncidents); + await look([a], 100); + await look([], 101, { inGrace: true }); + await look([], 200, { inGrace: true }); + const [row] = await rows(); + assert.equal(row.resolvedAt, null, "server alerts are held back on purpose for a while after the app starts, so missing then means nothing"); + assert.equal(row.missingSince, null); + }); + + test("it is recorded from the alerts as they are collected, without anyone asking, and the Alerts page links to it", async () => { + await db.delete(alertIncidents); + const [s] = await db.insert(servers).values({ name: "quiet", apiTokenHash: hashToken("q"), apiTokenPrefix: "q", lastSeenAt: new Date(Date.now() - 5 * 3600_000).toISOString() }).returning(); + invalidateAlertsCache(); + await getAlerts(true); + let found: typeof alertIncidents.$inferSelect | undefined; + for (let i = 0; i < 40 && !found; i++) { + found = (await rows()).find((r) => r.alertKey === `offline:offline:server:${s.id}`); + if (!found) await new Promise((r) => setTimeout(r, 50)); + } + assert.ok(found, "an incident for the offline server"); + assert.equal(found.severity, "critical"); + assert.deepEqual([found.targetType, found.targetId, found.targetName], ["server", s.id, "quiet"]); + assert.equal((await inc.openIncidentIds([found.alertKey])).get(found.alertKey), found.id); + await db.delete(servers).where(eq(servers.id, s.id)); + }); + + test("resolved ones are kept for a long while and then removed with their notes; open ones never", async () => { + await db.delete(alertIncidents); + const old = new Date(Date.now() - (inc.KEEP_DAYS + 5) * 86_400_000).toISOString(); + const [oldRow] = await db.insert(alertIncidents).values({ alertKey: "old", severity: "warning", peakSeverity: "warning", category: "disk", source: "Server", message: "old", openedAt: old, resolvedAt: old, lastSeenAt: old }).returning(); + await db.insert(incidentNotes).values({ incidentId: oldRow.id, author: "Ada", body: "n", createdAt: old }); + const [openRow] = await db.insert(alertIncidents).values({ alertKey: "ancient-open", severity: "warning", peakSeverity: "warning", category: "disk", source: "Server", message: "still open", openedAt: old, lastSeenAt: old }).returning(); + assert.equal(await inc.pruneIncidents(), 1); + assert.deepEqual((await rows()).map((r) => r.id), [openRow.id]); + assert.equal((await db.select().from(incidentNotes)).length, 0); + }); +}); + +describe("the timeline around an incident", () => { + let serverId = 0; + let incidentId = 0; + + before(async () => { + await db.delete(alertIncidents); + const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("e"), apiTokenPrefix: "e" }).returning(); + serverId = s.id; + const [row] = await db + .insert(alertIncidents) + .values({ alertKey: `offline:offline:server:${serverId}`, severity: "critical", peakSeverity: "critical", category: "offline", source: "Server", message: "elsa hasn't reported for 2 h", targetType: "server", targetId: serverId, targetName: "elsa", openedAt: at(0).toISOString(), resolvedAt: at(95).toISOString(), lastSeenAt: at(95).toISOString() }) + .returning(); + incidentId = row.id; + const stamp = (m: number) => at(m).toISOString().slice(0, 19).replace("T", " "); + const audit = (m: number, category: string, action: string, detail: object, actor = "system", target: object = { targetType: "server", targetId: String(serverId) }) => + db.insert(auditLog).values({ actorLabel: actor, category, action, detail: JSON.stringify(detail), createdAt: stamp(m), ...target }); + await audit(1, "server", "went_offline", { message: "elsa hasn't reported for 2 h" }); + await audit(5, "alert", "ack", { alertId: row.alertKey, message: "x", note: "power cut in the cellar", assignee: "Olle", hours: 4 }, "Ada"); + await audit(6, "alert", "ack", { alertId: "some:other:alert", note: "not this one" }, "Ada"); + await audit(30, "server", "update", { name: "elsa" }, "Ada"); + await audit(90, "server", "back_online", {}); + await audit(40, "server", "update", {}, "Ada", { targetType: "server", targetId: "9999" }); // another server + await db.insert(maintenanceWindows).values({ targetType: "server", targetId: serverId, reason: "UPS swap", startedAt: at(20).toISOString(), endsAt: at(60).toISOString(), createdBy: "Ada" }); + await db.insert(maintenanceWindows).values({ targetType: "server", targetId: serverId, reason: "withdrawn", startedAt: at(25).toISOString(), endsAt: at(25).toISOString() }); + await db.insert(updateRuns).values({ serverId, integrationId: 1, projectId: 1, templateId: 1, templateName: "Apply updates", hostLimit: "elsa", status: "done", message: "No updates are waiting now; no restart is needed.", startedBy: "Ada", startedAt: at(61).toISOString(), finishedAt: at(80).toISOString() }); + await db.insert(alertIncidents).values({ alertKey: `disk:server:${serverId}:/`, severity: "warning", peakSeverity: "warning", category: "disk", source: "Server", message: "/ is 93% full", targetType: "server", targetId: serverId, targetName: "elsa", openedAt: at(8).toISOString(), lastSeenAt: at(8).toISOString() }); + await db.insert(alertIncidents).values({ alertKey: `disk:server:${serverId}:/old`, severity: "warning", peakSeverity: "warning", category: "disk", source: "Server", message: "long ago", targetType: "server", targetId: serverId, targetName: "elsa", openedAt: at(-600).toISOString(), lastSeenAt: at(-600).toISOString() }); + await inc.addNote(incidentId, "Olle", "Confirmed: the building lost power.", at(15)); + }); + + test("it puts everything in the order it happened, and only what belongs to this incident", async () => { + const d = (await inc.getIncident(incidentId, at(100)))!; + assert.equal(d.incident.open, false); + assert.equal(d.incident.durationMs, 95 * 60_000); + const titles = d.timeline.map((e) => `${e.at.slice(11, 16)} ${e.kind}: ${e.title}`); + assert.deepEqual(titles, [ + "12:00 alert: Alert raised (critical)", + "12:01 health: Stopped reporting — marked offline", + "12:05 ack: Acknowledged, handed to Olle", + "12:08 related: Another alert was raised: disk (warning)", + "12:15 note: Olle wrote a note", + "12:20 maintenance: Maintenance began — alerts silenced", + "12:30 change: Details changed", + "12:55 maintenance: Maintenance ended".replace("12:55", "13:00"), + "13:01 run: Update run started: Apply updates", + "13:20 run: Update run finished", + "13:30 health: Reporting again", + "13:35 alert: Resolved", + ]); + }); + + test("the details say what was written and by whom", async () => { + const d = (await inc.getIncident(incidentId, at(100)))!; + const ack = d.timeline.find((e) => e.kind === "ack")!; + assert.equal(ack.actor, "Ada"); + assert.match(ack.detail!, /power cut in the cellar · for 4 h/); + const note = d.timeline.find((e) => e.kind === "note")!; + assert.equal(note.detail, "Confirmed: the building lost power."); + assert.ok(note.noteId); + const related = d.timeline.find((e) => e.kind === "related")!; + assert.match(related.detail!, /93% full/); + assert.ok(related.incidentId); + assert.equal(d.timeline.find((e) => e.title === "Resolved")!.detail, "It lasted 1.6 h."); + assert.equal(d.ack, null, "no acknowledgement to show on one that is over"); + assert.equal(d.notes.length, 1); + assert.equal((await inc.getIncident(999999)), null); + }); + + test("an open one runs to now and shows the acknowledgement in force", async () => { + const [row] = await db.select().from(alertIncidents).where(eq(alertIncidents.alertKey, `disk:server:${serverId}:/`)); + await acks.setAck({ alertId: row.alertKey, note: "freeing space", assigneeId: operatorId, hours: null, actor: "Ada" }, at(9)); + const d = (await inc.getIncident(row.id, at(120)))!; + assert.equal(d.incident.open, true); + assert.equal(d.incident.durationMs, 112 * 60_000); + assert.equal(d.ack!.note, "freeing space"); + assert.equal(d.ack!.assignee!.name, "Olle"); + assert.ok(!d.timeline.some((e) => e.title === "Resolved")); + await db.delete(alertAcks); + }); + + test("the list has the open ones first, then the rest newest first, with how many notes each has", async () => { + const list = await inc.listIncidents({ now: at(120) }); + assert.equal(list[0].open, true); + const firstClosed = list.findIndex((i) => !i.open); + assert.ok(list.slice(0, firstClosed).every((i) => i.open) && list.slice(firstClosed).every((i) => !i.open)); + assert.equal(list.find((i) => i.id === incidentId)!.notes, 1); + assert.deepEqual(list.filter((i) => !i.open).map((i) => i.openedAt), list.filter((i) => !i.open).map((i) => i.openedAt).sort().reverse()); + }); +}); + +describe("in the weekly digest", () => { + test("the week's incidents are counted, the longest are named, and acknowledged alerts are said to be", async () => { + const digest = await buildDigest(at(120)); + assert.match(digest.message, /INCIDENTS/); + assert.match(digest.message, /\d+ incidents? began, 1 ended, \d+ still open/); + assert.match(digest.message, /elsa hasn't reported for 2 h — lasted 1 h 35 min/); + }); +}); + +describe("through the routes", () => { + let app: Server; + let base = ""; + let who = "ada"; + let serverId = 0; + let alertId = ""; + let incidentId = 0; + const call = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) }); + + before(async () => { + await db.delete(alertIncidents); + const [s] = await db.insert(servers).values({ name: "zoe", apiTokenHash: hashToken("z"), apiTokenPrefix: "z", lastSeenAt: new Date(Date.now() - 5 * 3600_000).toISOString() }).returning(); + serverId = s.id; + alertId = `offline:offline:server:${serverId}`.replace("offline:offline", "offline:offline"); + const a = express(); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: who } }; + next(); + }); + a.use("/api/alerts", alertsRouter); + a.use("/api/incidents", incidentsRouter); + app = await new Promise((resolve) => { + const x = a.listen(0, "127.0.0.1", () => resolve(x)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`; + invalidateAlertsCache(); + const report = await (await call("GET", "/alerts?refresh=1")).json(); + alertId = report.alerts.find((x: { id: string }) => x.id.includes(`:server:${serverId}`)).id; + for (let i = 0; i < 40 && !incidentId; i++) { + incidentId = (await db.select().from(alertIncidents)).find((r) => r.alertKey === alertId)?.id ?? 0; + if (!incidentId) await new Promise((r) => setTimeout(r, 50)); + } + }); + after(() => app.close()); + + test("operators acknowledge and hand over; viewers can look but not; an alert that isn't there can't be", async () => { + who = "vera"; + assert.equal((await call("PUT", "/alerts/ack", { alertId })).status, 403); + assert.equal((await call("GET", "/alerts/assignees")).status, 403); + assert.equal((await call("DELETE", `/alerts/ack?alertId=${encodeURIComponent(alertId)}`)).status, 403); + assert.equal((await call("GET", "/alerts")).status, 200); + + who = "olle"; + const names = (await (await call("GET", "/alerts/assignees")).json()).users.map((u: { name: string }) => u.name); + assert.deepEqual(names, ["Ada", "Olle"]); + assert.equal((await call("PUT", "/alerts/ack", { alertId: "no:such:alert" })).status, 404); + assert.equal((await call("PUT", "/alerts/ack", { alertId: "" })).status, 400); + assert.equal((await call("PUT", "/alerts/ack", { alertId, hours: 0 })).status, 400); + const bad = await call("PUT", "/alerts/ack", { alertId, assigneeId: viewerId }); + assert.equal(bad.status, 400); + assert.equal((await bad.json()).error, "invalid_assignee"); + + const res = await call("PUT", "/alerts/ack", { alertId, note: "power cut", assigneeId: adminId, hours: 6 }); + assert.equal(res.status, 200); + const { ack } = await res.json(); + assert.equal(ack.assignee.name, "Ada"); + assert.equal(ack.ackedBy, "Olle"); + assert.equal(ack.active, true); + }); + + test("the Alerts page shows it on the alert, leaves it out of what needs attention, and links to the incident", async () => { + invalidateAlertsCache(); + const report = await (await call("GET", "/alerts")).json(); + const a = report.alerts.find((x: { id: string }) => x.id === alertId); + assert.equal(a.ack.note, "power cut"); + assert.equal(a.incidentId, incidentId); + assert.ok(report.counts.acknowledged >= 1); + assert.ok(report.counts.attention.critical < report.counts.critical, "the whole count is still the whole count; attention leaves out the acknowledged"); + const metrics = await buildMetrics(); + assert.match(metrics, /homelab_alerts_acknowledged [1-9]/); + assert.match(metrics, /homelab_incidents_open [1-9]/); + }); + + test("it is in the audit log and in the server's history, and taking it back is too", async () => { + const entry = (await db.select().from(auditLog)).filter((e) => e.category === "alert" && e.action === "ack").pop()!; + assert.equal(entry.targetType, "server"); + assert.equal(entry.targetId, String(serverId)); + assert.equal(JSON.parse(entry.detail!).assignee, "Ada"); + const activity = await getServerActivity(serverId); + assert.ok(activity.some((e) => e.title === "Acknowledged, handed to Ada" && /power cut/.test(e.detail ?? ""))); + + assert.equal((await call("DELETE", `/alerts/ack?alertId=${encodeURIComponent(alertId)}`)).status, 204); + assert.equal((await call("DELETE", `/alerts/ack?alertId=${encodeURIComponent(alertId)}`)).status, 404); + invalidateAlertsCache(); + assert.equal((await (await call("GET", "/alerts")).json()).alerts.find((x: { id: string }) => x.id === alertId).ack, null); + assert.ok((await getServerActivity(serverId)).some((e) => e.title === "Acknowledgement taken back")); + }); + + test("everyone signed in can read the incidents; operators write notes; a note can be removed by its author or an administrator", async () => { + who = "vera"; + const list = await (await call("GET", "/incidents")).json(); + assert.ok(list.incidents.some((i: { id: number }) => i.id === incidentId)); + assert.equal((await call("GET", `/incidents/${incidentId}`)).status, 200); + assert.equal((await call("GET", "/incidents/999999")).status, 404); + assert.equal((await call("GET", "/incidents/x")).status, 400); + assert.equal((await call("POST", `/incidents/${incidentId}/notes`, { body: "hello" })).status, 403); + + who = "olle"; + assert.equal((await call("POST", `/incidents/${incidentId}/notes`, { body: " " })).status, 400); + assert.equal((await call("POST", "/incidents/999999/notes", { body: "x" })).status, 404); + const made = await call("POST", `/incidents/${incidentId}/notes`, { body: "It was the UPS." }); + assert.equal(made.status, 201); + const noteId = (await made.json()).id; + const detail = await (await call("GET", `/incidents/${incidentId}`)).json(); + assert.ok(detail.timeline.some((e: { kind: string; detail: string }) => e.kind === "note" && e.detail === "It was the UPS.")); + + // another operator can't remove it, its author and an administrator can + await db.insert(users).values({ oidcSub: "olga", name: "Olga", role: "operator" }); + who = "olga"; + assert.equal((await call("DELETE", `/incidents/${incidentId}/notes/${noteId}`)).status, 403); + who = "olle"; + assert.equal((await call("DELETE", `/incidents/${incidentId}/notes/${noteId}`)).status, 204); + assert.equal((await call("DELETE", `/incidents/${incidentId}/notes/${noteId}`)).status, 404); + const again = await (await call("POST", `/incidents/${incidentId}/notes`, { body: "second" })).json(); + who = "ada"; + assert.equal((await call("DELETE", `/incidents/${incidentId}/notes/${again.id}`)).status, 204); + const actions = (await db.select().from(auditLog)).filter((e) => e.category === "incident").map((e) => e.action); + assert.ok(actions.includes("note") && actions.includes("note_removed")); + }); +}); diff --git a/server/test/selfBackup.test.ts b/server/test/selfBackup.test.ts new file mode 100644 index 0000000..d57ffd9 --- /dev/null +++ b/server/test/selfBackup.test.ts @@ -0,0 +1,453 @@ +/** + * Backups of Homelab Manager itself: taking them, checking them, keeping the newest, staging and applying a restore, the routes around + * them, and what the alerts and /metrics say about how current they are. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, utimesSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import type { Server } from "node:http"; + +const dir = mkdtempSync(join(tmpdir(), "hm-selfbackup-")); +process.env.DATABASE_PATH = join(dir, "data", "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { createClient } = await import("@libsql/client"); +const { db, client } = await import("../src/db/client.js"); +const { users, auditLog } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const sb = await import("../src/services/selfBackup.js"); +const { runScheduledBackup, cronForBackup } = await import("../src/services/selfBackupScheduler.js"); +const { applyPendingRestore } = await import("../src/db/applyRestore.js"); +const { PENDING_FILE, PENDING_NOTE, RESULT_NOTE } = await import("../src/db/restoreFiles.js"); +const { dbBackupsRouter } = await import("../src/routes/dbBackups.js"); +const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js"); +const { buildMetrics } = await import("../src/services/metrics.js"); +const { getSettings, updateSettings, setInternalFlag } = await import("../src/services/settingsStore.js"); +const schedule = (await import("node-schedule")).default; + +await runMigrations(); +after(async () => { + await schedule.gracefulShutdown(); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } +}); + +const at = (iso: string) => new Date(iso); +/** Opens a copy of a backup, never the backup itself: an opened database can't be deleted on some systems until the process ends. */ +const open = (file: string) => { + const copy = join(mkdtempSync(join(dir, "copy-")), "db.sqlite"); + copyFileSync(file, copy); + return createClient({ url: `file:${copy}` }); +}; +const wipeBackups = () => rmSync(sb.backupDir(), { recursive: true, force: true }); + +describe("taking and checking a backup", () => { + test("a backup is a consistent copy that opens, passes SQLite's check, and holds what was there", async () => { + await db.insert(users).values({ oidcSub: "ada", name: "Ada", role: "admin" }); + const { info, check } = await sb.createSnapshot("manual", at("2026-10-17T12:00:00Z")); + assert.equal(info.name, "homelab-manager-2026-10-17-12-00-00-manual.sqlite"); + assert.equal(info.kind, "manual"); + assert.ok(info.sizeBytes > 10_000); + assert.equal(check.ok, true); + assert.equal(check.integrity, "ok"); + assert.equal(check.migrations, sb.appMigrationCount()); + assert.equal(check.counts!.users, 1); + + // and it really holds it: open the file on its own and read the row back + const copy = open(join(sb.backupDir(), info.name)); + const rows = (await copy.execute("select name from users")).rows; + copy.close(); + assert.deepEqual(rows.map((r) => r.name), ["Ada"]); + }); + + test("changes made after it was taken are not in it", async () => { + const { info } = await sb.createSnapshot("manual", at("2026-10-17T12:01:00Z")); + await db.insert(users).values({ oidcSub: "bo", name: "Bo", role: "viewer" }); + const copy = open(join(sb.backupDir(), info.name)); + assert.equal(Number((await copy.execute("select count(*) as n from users")).rows[0].n), 1); + copy.close(); + await db.delete(users).where((await import("drizzle-orm")).eq(users.oidcSub, "bo")); + }); + + test("two taken in the same second don't collide", async () => { + const a = await sb.createSnapshot("manual", at("2026-10-17T13:00:00Z")); + const b = await sb.createSnapshot("manual", at("2026-10-17T13:00:00Z")); + assert.notEqual(a.info.name, b.info.name); + assert.match(b.info.name, /-manual-2\.sqlite$/); + }); + + test("files that aren't a usable backup are said to be, and why", async () => { + const junk = join(dir, "junk.sqlite"); + writeFileSync(junk, "this is not a database at all"); + assert.match((await sb.checkSnapshotFile(junk)).problem!, /isn't a database file/); + assert.match((await sb.checkSnapshotFile(join(dir, "missing.sqlite"))).problem!, /isn't there/); + + // a real SQLite file that isn't ours + const other = join(dir, "other.sqlite"); + const c = createClient({ url: `file:${other}` }); + await c.execute("create table cats (name text)"); + c.close(); + assert.match((await sb.checkSnapshotFile(other)).problem!, /isn't a Homelab Manager database/); + + // one of ours, made by a newer version (more schema changes than this one has) + const newer = join(dir, "newer.sqlite"); + await client.execute(`VACUUM INTO '${newer.replace(/'/g, "''")}'`); + const n = createClient({ url: `file:${newer}` }); + for (let i = 0; i < 3; i++) await n.execute({ sql: "insert into __drizzle_migrations (hash, created_at) values (?, ?)", args: [`future-${i}`, 9_999_999_999_999 + i] }); + n.close(); + const check = await sb.checkSnapshotFile(newer); + assert.equal(check.ok, false); + assert.equal(check.newerThanApp, true); + assert.match(check.problem!, /newer version/); + assert.ok(check.migrations! > check.appMigrations); + }); + + test("an older one is fine: this version brings it up to date when it starts", async () => { + const older = join(dir, "older.sqlite"); + await client.execute(`VACUUM INTO '${older.replace(/'/g, "''")}'`); + const o = createClient({ url: `file:${older}` }); + await o.execute("delete from __drizzle_migrations where rowid = (select max(rowid) from __drizzle_migrations)"); + o.close(); + const check = await sb.checkSnapshotFile(older); + assert.equal(check.ok, true); + assert.equal(check.migrations, sb.appMigrationCount() - 1); + }); + + test("only names of our own files are ever acted on, whatever is asked for", () => { + for (const bad of ["../test.sqlite", "..\\test.sqlite", "homelab-manager-2026-10-17-12-00-00.sqlite/../../x", "homelab-manager-x.sqlite", "test.sqlite", "homelab-manager-2026-10-17-12-00-00.sqlite.bak", "/etc/passwd", ""]) { + assert.equal(sb.isSnapshotName(bad), false, bad); + assert.equal(sb.snapshotPath(bad), null, bad); + assert.equal(sb.deleteSnapshot(bad), false, bad); + } + for (const good of ["homelab-manager-2026-10-17-12-00-00.sqlite", "homelab-manager-2026-10-17-12-00-00-manual.sqlite", "homelab-manager-2026-10-17-12-00-00-pre-restore-2.sqlite", "homelab-manager-2026-10-17-12-00-00-uploaded.sqlite"]) { + assert.equal(sb.isSnapshotName(good), true, good); + } + }); +}); + +describe("keeping the newest", () => { + const fake = (name: string, mtime: string) => { + mkdirSync(sb.backupDir(), { recursive: true }); + const f = join(sb.backupDir(), name); + writeFileSync(f, "x"); + utimesSync(f, at(mtime), at(mtime)); + }; + const names = () => sb.listSnapshots().map((s) => s.name); + + test("the newest N of the daily ones stay, and each other kind has its own, smaller allowance", () => { + wipeBackups(); + for (let d = 1; d <= 6; d++) fake(`homelab-manager-2026-10-0${d}-03-30-00.sqlite`, `2026-10-0${d}T03:30:00Z`); + for (let d = 1; d <= 12; d++) fake(`homelab-manager-2026-10-${String(d).padStart(2, "0")}-09-00-00-manual.sqlite`, `2026-10-${String(d).padStart(2, "0")}T09:00:00Z`); + for (let d = 1; d <= 5; d++) fake(`homelab-manager-2026-10-0${d}-10-00-00-pre-restore.sqlite`, `2026-10-0${d}T10:00:00Z`); + const removed = sb.pruneSnapshots(3); + const left = names(); + assert.equal(left.filter((n) => !/-(manual|pre-restore|uploaded)/.test(n)).length, 3); + assert.equal(left.filter((n) => n.includes("-manual")).length, 10); + assert.equal(left.filter((n) => n.includes("-pre-restore")).length, 3); + assert.equal(removed.length, 3 + 2 + 2); + assert.ok(left.includes("homelab-manager-2026-10-06-03-30-00.sqlite") && !left.includes("homelab-manager-2026-10-01-03-30-00.sqlite"), "the oldest went, not the newest"); + // other files in the folder are left alone + writeFileSync(join(sb.backupDir(), "notes.txt"), "mine"); + sb.pruneSnapshots(1); + assert.ok(existsSync(join(sb.backupDir(), "notes.txt"))); + wipeBackups(); + }); + + test("a scheduled run takes one and trims to what is kept, and records that it worked", async () => { + wipeBackups(); + await updateSettings({ selfBackup: { keep: 2 } }); + for (const d of ["2026-10-14T03:30:00Z", "2026-10-15T03:30:00Z", "2026-10-16T03:30:00Z"]) assert.ok(await runScheduledBackup(at(d))); + assert.deepEqual(names(), ["homelab-manager-2026-10-16-03-30-00.sqlite", "homelab-manager-2026-10-15-03-30-00.sqlite"]); + const last = await sb.lastAttempt(); + assert.equal(last!.ok, true); + assert.equal(last!.kind, "scheduled"); + assert.equal(cronForBackup("03:30"), "30 3 * * *"); + assert.equal(cronForBackup("23:05"), "5 23 * * *"); + await updateSettings({ selfBackup: { keep: 14 } }); + }); + + test("a backup that can't be made is recorded as having failed, says why, and leaves no half file", async () => { + wipeBackups(); + const realExecute = client.execute.bind(client); + (client as unknown as { execute: unknown }).execute = async (stmt: unknown) => { + if (typeof stmt === "string" && stmt.startsWith("VACUUM INTO")) throw new Error("SQLITE_FULL: database or disk is full"); + return realExecute(stmt as never); + }; + try { + await assert.rejects(sb.createSnapshot("scheduled", at("2026-10-17T03:30:00Z")), /SQLITE_FULL/); + const last = await sb.lastAttempt(); + assert.equal(last!.ok, false); + assert.match(last!.error!, /disk is full/); + assert.deepEqual(sb.listSnapshots(), [], "nothing half-written is left in the backups folder"); + assert.equal(await runScheduledBackup(at("2026-10-17T03:31:00Z")), null, "the scheduled run swallows it (and says so through the channels)"); + } finally { + (client as unknown as { execute: unknown }).execute = realExecute; + } + assert.ok(await runScheduledBackup(at("2026-10-17T03:32:00Z")), "and the next one works"); + assert.equal((await sb.lastAttempt())!.ok, true); + wipeBackups(); + }); +}); + +describe("restoring", () => { + test("a staged restore is checked first, kept beside the database, can be cancelled, and refuses what isn't usable", async () => { + wipeBackups(); + const { info } = await sb.createSnapshot("manual", at("2026-10-17T15:00:00Z")); + assert.equal(sb.stagedRestore(), null); + + await assert.rejects(sb.stageRestore("../../etc/passwd", "Ada"), /no backup with that name/); + await assert.rejects(sb.stageRestore("homelab-manager-2026-01-01-00-00-00.sqlite", "Ada"), /no backup with that name/); + writeFileSync(join(sb.backupDir(), "homelab-manager-2026-10-17-16-00-00-uploaded.sqlite"), "garbage"); + await assert.rejects(sb.stageRestore("homelab-manager-2026-10-17-16-00-00-uploaded.sqlite", "Ada"), /isn't a database file/); + assert.equal(sb.stagedRestore(), null, "a refused one stages nothing"); + + const check = await sb.stageRestore(info.name, "Ada", at("2026-10-17T15:05:00Z")); + assert.equal(check.ok, true); + const staged = sb.stagedRestore()!; + assert.deepEqual(staged, { from: info.name, stagedAt: "2026-10-17T15:05:00.000Z", by: "Ada" }); + const dataDir = join(dir, "data"); + assert.ok(existsSync(join(dataDir, PENDING_FILE))); + assert.equal(readdirSync(dataDir).some((f) => f.endsWith(".copying")), false, "no half-copied file is left"); + assert.equal(statSync(join(dataDir, PENDING_FILE)).size, info.sizeBytes); + + assert.equal(sb.cancelStagedRestore(), true); + assert.equal(sb.stagedRestore(), null); + assert.equal(existsSync(join(dataDir, PENDING_FILE)), false); + assert.equal(sb.cancelStagedRestore(), false); + wipeBackups(); + }); + + test("at start-up the staged file replaces the database, the old one is kept, and nothing happens twice", () => { + const folder = mkdtempSync(join(dir, "apply-")); + const dbFile = join(folder, "app.sqlite"); + writeFileSync(dbFile, "OLD DATABASE"); + writeFileSync(`${dbFile}-wal`, "stale journal"); + writeFileSync(join(folder, PENDING_FILE), "NEW DATABASE"); + writeFileSync(join(folder, PENDING_NOTE), JSON.stringify({ from: "homelab-manager-2026-10-01-03-30-00.sqlite", by: "Ada" })); + + const result = applyPendingRestore(dbFile, at("2026-10-17T16:00:00Z"))!; + assert.equal(readFileSync(dbFile, "utf8"), "NEW DATABASE"); + assert.equal(existsSync(`${dbFile}-wal`), false, "a journal from the old file would be taken for the new one's"); + assert.equal(result.keptAs, "homelab-manager-2026-10-17-16-00-00-pre-restore.sqlite"); + assert.equal(readFileSync(join(folder, "backups", result.keptAs!), "utf8"), "OLD DATABASE", "what was in use is kept, not deleted"); + assert.equal(result.from, "homelab-manager-2026-10-01-03-30-00.sqlite"); + assert.equal(result.by, "Ada"); + assert.equal(existsSync(join(folder, PENDING_FILE)), false); + assert.equal(existsSync(join(folder, PENDING_NOTE)), false); + assert.deepEqual(JSON.parse(readFileSync(join(folder, RESULT_NOTE), "utf8")), result); + + assert.equal(applyPendingRestore(dbFile, at("2026-10-17T16:01:00Z")), null, "nothing staged any more"); + assert.equal(readFileSync(dbFile, "utf8"), "NEW DATABASE"); + }); + + test("with no database there yet (a fresh machine), the staged one simply becomes it", () => { + const folder = mkdtempSync(join(dir, "fresh-")); + const dbFile = join(folder, "app.sqlite"); + writeFileSync(join(folder, PENDING_FILE), "NEW DATABASE"); + const result = applyPendingRestore(dbFile, at("2026-10-17T16:00:00Z"))!; + assert.equal(readFileSync(dbFile, "utf8"), "NEW DATABASE"); + assert.equal(result.keptAs, null); + assert.equal(result.from, "an uploaded backup", "no note, no name — said plainly"); + }); + + test("a restore that can't be completed leaves the database exactly as it was", () => { + // the folder for keeping the old one can't be made (a file is in its place) + const folder = mkdtempSync(join(dir, "fail-")); + const dbFile = join(folder, "app.sqlite"); + writeFileSync(dbFile, "OLD DATABASE"); + writeFileSync(join(folder, PENDING_FILE), "NEW DATABASE"); + writeFileSync(join(folder, "backups"), "I am a file, not a folder"); + assert.equal(applyPendingRestore(dbFile, at("2026-10-17T16:00:00Z")), null); + assert.equal(readFileSync(dbFile, "utf8"), "OLD DATABASE", "the database is as it was"); + assert.equal(readFileSync(join(folder, PENDING_FILE), "utf8"), "NEW DATABASE", "and the staged one is still there to try again"); + assert.equal(existsSync(join(folder, RESULT_NOTE)), false); + + // something staged that isn't a file is never put in the database's place + const odd = mkdtempSync(join(dir, "odd-")); + writeFileSync(join(odd, "app.sqlite"), "OLD DATABASE"); + mkdirSync(join(odd, PENDING_FILE)); + assert.equal(applyPendingRestore(join(odd, "app.sqlite"), at("2026-10-17T16:00:00Z")), null); + assert.equal(readFileSync(join(odd, "app.sqlite"), "utf8"), "OLD DATABASE"); + }); +}); + +describe("through the routes", () => { + let app: Server; + let base = ""; + let who = "ada"; + const call = (method: string, path: string, body?: unknown, raw?: Buffer) => + fetch(`${base}${path}`, raw ? { method, headers: { "content-type": "application/octet-stream" }, body: new Uint8Array(raw) } : { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) }); + + before(async () => { + await db.insert(users).values({ oidcSub: "operator", name: "Olle", role: "operator" }); + const a = express(); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: who } }; + next(); + }); + a.use("/api/db-backups", dbBackupsRouter); + app = await new Promise((resolve) => { + const x = a.listen(0, "127.0.0.1", () => resolve(x)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/db-backups`; + wipeBackups(); + }); + after(() => app.close()); + + test("everything about backups is for administrators only", async () => { + who = "operator"; + for (const [method, path] of [["GET", "/"], ["POST", "/"], ["PUT", "/settings"], ["POST", "/upload"], ["GET", "/x/download"], ["POST", "/x/restore"], ["POST", "/x/check"], ["DELETE", "/x"], ["POST", "/restore/cancel"]] as const) { + assert.equal((await call(method, path, method === "GET" || method === "DELETE" ? undefined : {})).status, 403, `${method} ${path}`); + } + who = "ada"; + assert.equal((await call("GET", "/")).status, 200); + }); + + test("settings are checked and kept, and change what is kept at once", async () => { + assert.equal((await call("PUT", "/settings", { enabled: true, time: "25:00", keep: 7 })).status, 400); + assert.equal((await call("PUT", "/settings", { enabled: true, time: "03:30", keep: 0 })).status, 400); + assert.equal((await call("PUT", "/settings", { enabled: true, time: "03:30", keep: 500 })).status, 400); + const res = await call("PUT", "/settings", { enabled: true, time: "04:15", keep: 7 }); + assert.equal(res.status, 200); + const body = await res.json(); + assert.deepEqual(body.settings, { enabled: true, time: "04:15", keep: 7 }); + assert.ok(body.nextRun, "it says when the next one is"); + assert.equal((await getSettings()).selfBackup.time, "04:15"); + const off = await (await call("PUT", "/settings", { enabled: false, time: "04:15", keep: 7 })).json(); + assert.equal(off.nextRun, null, "off means nothing is scheduled"); + await call("PUT", "/settings", { enabled: true, time: "03:30", keep: 14 }); + const entry = (await db.select().from(auditLog)).filter((e) => e.category === "db_backup" && e.action === "settings").pop()!; + assert.deepEqual(JSON.parse(entry.detail!).to, { enabled: true, time: "03:30", keep: 14 }); + }); + + test("take one, see it listed, download it byte for byte, check it, and delete it", async () => { + const made = await call("POST", "/"); + assert.equal(made.status, 201); + const { snapshot, check } = await made.json(); + assert.equal(check.ok, true); + assert.match(snapshot.name, /-manual\.sqlite$/); + + const overview = await (await call("GET", "/")).json(); + assert.equal(overview.snapshots.length, 1); + assert.equal(overview.snapshots[0].name, snapshot.name); + assert.equal(overview.lastAttempt.ok, true); + assert.equal(overview.staged, null); + + const download = await call("GET", `/${snapshot.name}/download`); + assert.equal(download.status, 200); + assert.equal(download.headers.get("cache-control"), "no-store"); + assert.match(download.headers.get("content-disposition") ?? "", new RegExp(snapshot.name.replace(/\./g, "\\."))); + const bytes = Buffer.from(await download.arrayBuffer()); + assert.deepEqual(bytes, readFileSync(join(sb.backupDir(), snapshot.name))); + + assert.equal((await (await call("POST", `/${snapshot.name}/check`)).json()).check.integrity, "ok"); + assert.equal((await call("GET", "/homelab-manager-nonsense.sqlite/download")).status, 404); + assert.equal((await call("GET", `/..%2Fapp.sqlite/download`)).status, 404); + assert.equal((await call("DELETE", `/${snapshot.name}`)).status, 204); + assert.equal((await call("DELETE", `/${snapshot.name}`)).status, 404); + + const actions = (await db.select().from(auditLog)).filter((e) => e.category === "db_backup").map((e) => e.action); + for (const wanted of ["create", "download", "delete"]) assert.ok(actions.includes(wanted), wanted); + }); + + test("a backup from elsewhere can be uploaded and then restored like any other; a bad file is refused and leaves nothing", async () => { + const { info } = await sb.createSnapshot("manual", at("2026-10-17T18:00:00Z")); + const bytes = readFileSync(join(sb.backupDir(), info.name)); + const before = sb.listSnapshots().length; + + const up = await call("POST", "/upload", undefined, bytes); + assert.equal(up.status, 201); + const uploaded = (await up.json()).snapshot; + assert.match(uploaded.name, /-uploaded\.sqlite$/); + assert.equal(sb.listSnapshots().length, before + 1); + + const bad = await call("POST", "/upload", undefined, Buffer.from("not a database")); + assert.equal(bad.status, 400); + assert.match((await bad.json()).message, /isn't a database file/); + assert.equal((await call("POST", "/upload", undefined, Buffer.alloc(0))).status, 400); + assert.equal(sb.listSnapshots().length, before + 1, "nothing was kept from the bad ones"); + + const restore = await call("POST", `/${uploaded.name}/restore`); + assert.equal(restore.status, 200); + assert.equal((await restore.json()).staged.from, uploaded.name); + assert.equal((await (await call("GET", "/")).json()).staged.from, uploaded.name); + assert.equal((await call("POST", "/homelab-manager-2026-01-01-00-00-00.sqlite/restore")).status, 400); + assert.equal((await (await call("POST", "/restore/cancel")).json()).cancelled, true); + assert.equal((await (await call("GET", "/")).json()).staged, null); + assert.ok((await db.select().from(auditLog)).some((e) => e.action === "restore_staged")); + assert.ok((await db.select().from(auditLog)).some((e) => e.action === "restore_cancelled")); + }); + + test("what the last restore did is shown until it is dismissed", async () => { + writeFileSync(join(dir, "data", RESULT_NOTE), JSON.stringify({ restoredAt: "2026-10-17T16:00:00.000Z", from: "x.sqlite", keptAs: "y.sqlite", by: "Ada" })); + assert.equal((await (await call("GET", "/")).json()).lastRestore.keptAs, "y.sqlite"); + assert.equal((await call("POST", "/restore-result/dismiss")).status, 200); + assert.equal((await (await call("GET", "/")).json()).lastRestore, null); + }); +}); + +describe("how current the backups are, in the alerts and /metrics", () => { + const selfAlert = async () => { + invalidateAlertsCache(); + return (await getAlerts(true)).alerts.find((a) => a.id === "backup:self-backup"); + }; + const oneAged = async (hoursAgo: number) => { + wipeBackups(); + const { info } = await sb.createSnapshot("scheduled", new Date()); + const when = new Date(Date.now() - hoursAgo * 3600_000); + utimesSync(join(sb.backupDir(), info.name), when, when); + }; + + test("no alert while they are recent, or when none was ever tried; a warning when stale, critical after a week", async () => { + wipeBackups(); + assert.equal(await selfAlert(), undefined, "nothing yet, and nothing failed: a new install isn't scolded"); + await oneAged(5); + assert.equal(await selfAlert(), undefined); + await oneAged(60); + const stale = await selfAlert(); + assert.equal(stale!.severity, "warning"); + assert.match(stale!.message, /60 h old/); + assert.equal(stale!.source, "Homelab Manager"); + assert.equal(stale!.link, "/settings/backup"); + await oneAged(24 * 8); + assert.equal((await selfAlert())!.severity, "critical"); + assert.match((await selfAlert())!.message, /8 days old/); + }); + + test("when the last attempt failed, the alert says why; and nothing is said at all while backups are switched off", async () => { + await oneAged(60); + await setInternalFlag("selfBackupLastAttempt", JSON.stringify({ at: new Date().toISOString(), ok: false, kind: "scheduled", error: "ENOSPC: no space left on device", name: null })); + assert.match((await selfAlert())!.message, /The last attempt failed: ENOSPC: no space left on device/); + + wipeBackups(); + const first = await selfAlert(); + assert.match(first!.message, /no backup of its own yet\. The last attempt failed|no backup of its own yet/, "a failure with nothing to show for it is said"); + + await updateSettings({ selfBackup: { enabled: false } }); + assert.equal(await selfAlert(), undefined); + await updateSettings({ selfBackup: { enabled: true } }); + }); + + test("/metrics says how many are kept, when the newest was made and whether the last try worked", async () => { + await oneAged(2); + await sb.createSnapshot("manual", new Date()); + const text = await buildMetrics(); + assert.match(text, /homelab_selfbackup_snapshots 2/); + const ts = /homelab_selfbackup_last_success_timestamp_seconds (\d+)/.exec(text); + assert.ok(ts && Math.abs(Number(ts[1]) - Date.now() / 1000) < 120, "the manual one just made is the newest"); + assert.match(text, /homelab_selfbackup_last_attempt_ok 1/); + wipeBackups(); + }); +}); diff --git a/server/test/updateRuns.test.ts b/server/test/updateRuns.test.ts index 3da812b..9da326c 100644 --- a/server/test/updateRuns.test.ts +++ b/server/test/updateRuns.test.ts @@ -282,9 +282,9 @@ describe("following a run to the end", () => { const titles = events.filter((e) => /Update run/.test(e.title)).map((e) => e.title); // (the tests before this one wrote their own lines too, so this looks for the kinds rather than the number) for (const wanted of ["Update run started", "Update run didn't finish", "Update run finished", "Update run stopped being followed"]) assert.ok(titles.includes(wanted), `${wanted} in ${JSON.stringify([...new Set(titles)])}`); - const failedLine = events.find((e) => e.title === "Update run didn't finish")!; - assert.equal(failedLine.tone, "bad"); - assert.match(failedLine.detail!, /The playbook failed in Semaphore/); + const failedLines = events.filter((e) => e.title === "Update run didn't finish"); + assert.ok(failedLines.every((e) => e.tone === "bad"), "a run that didn't finish reads as bad"); + assert.ok(failedLines.some((e) => /The playbook failed in Semaphore/.test(e.detail ?? "")), "and says why"); assert.ok((await db.select().from(auditLog)).some((a) => a.category === "update_run" && a.action === "failed")); // nothing is held in memory: a fresh pass over the table is all it takes diff --git a/web/src/App.tsx b/web/src/App.tsx index d4fe582..6f54952 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -29,6 +29,8 @@ import Alerts from "./pages/Alerts"; import Generator from "./pages/Generator"; import Maintenance from "./pages/Maintenance"; import Updates from "./pages/Updates"; +import Incidents from "./pages/Incidents"; +import IncidentDetail from "./pages/IncidentDetail"; import Inventory from "./pages/Inventory"; import Backups from "./pages/Backups"; import Domains from "./pages/Domains"; @@ -155,6 +157,8 @@ function Shelled({ user }: { user: CurrentUser }) { } /> } /> } /> + } /> + } /> request(`/api/alerts${refresh ? "?refresh=1" : ""}`), + assignees: () => request<{ users: { id: number; name: string; role: string }[] }>("/api/alerts/assignees"), + ack: (data: AckInput) => request<{ ack: AlertAck }>("/api/alerts/ack", { method: "PUT", body: JSON.stringify(data) }), + unack: (alertId: string) => request(`/api/alerts/ack?alertId=${encodeURIComponent(alertId)}`, { method: "DELETE" }), + }, + incidents: { + list: () => request<{ incidents: IncidentSummary[] }>("/api/incidents"), + get: (id: number) => request(`/api/incidents/${id}`), + addNote: (id: number, body: string) => request<{ id: number }>(`/api/incidents/${id}/notes`, { method: "POST", body: JSON.stringify({ body }) }), + removeNote: (id: number, noteId: number) => request(`/api/incidents/${id}/notes/${noteId}`, { method: "DELETE" }), + }, + dbBackups: { + overview: () => request("/api/db-backups"), + create: () => request<{ snapshot: DbSnapshot; check: DbSnapshotCheck }>("/api/db-backups", { method: "POST" }), + check: (name: string) => request<{ check: DbSnapshotCheck }>(`/api/db-backups/${encodeURIComponent(name)}/check`, { method: "POST" }), + restore: (name: string) => request<{ staged: DbStagedRestore | null; check: DbSnapshotCheck }>(`/api/db-backups/${encodeURIComponent(name)}/restore`, { method: "POST" }), + cancelRestore: () => request<{ cancelled: boolean }>("/api/db-backups/restore/cancel", { method: "POST" }), + dismissRestoreResult: () => request<{ ok: true }>("/api/db-backups/restore-result/dismiss", { method: "POST" }), + remove: (name: string) => request(`/api/db-backups/${encodeURIComponent(name)}`, { method: "DELETE" }), + saveSettings: (data: DbBackupSettings) => request("/api/db-backups/settings", { method: "PUT", body: JSON.stringify(data) }), + /** A backup file chosen on this computer, sent as it is. */ + upload: async (file: File) => { + const res = await fetch("/api/db-backups/upload", { method: "POST", headers: { "Content-Type": "application/octet-stream" }, body: file, credentials: "same-origin" }); + if (!res.ok) throw new Error(await res.text().catch(() => `Upload failed (${res.status})`)); + return (await res.json()) as { snapshot: DbSnapshot; check: DbSnapshotCheck }; + }, + downloadUrl: (name: string) => `/api/db-backups/${encodeURIComponent(name)}/download`, }, alertTickets: { get: () => request("/api/alert-tickets"), diff --git a/web/src/components/AckDrawer.tsx b/web/src/components/AckDrawer.tsx new file mode 100644 index 0000000..42b165f --- /dev/null +++ b/web/src/components/AckDrawer.tsx @@ -0,0 +1,138 @@ +import { useEffect, useState } from "react"; +import { api, type AlertItem, type CurrentUser } from "../api/client"; +import { formatDateTime } from "../utils/date"; +import { readableError } from "../utils/errors"; +import { toast } from "../utils/toast"; +import DetailDrawer from "./DetailDrawer"; +import { ErrorNotice } from "./Notice"; +import TimeAgo from "./TimeAgo"; + +const LENGTHS: { hours: number | null; label: string }[] = [ + { hours: null, label: "Until the alert is gone" }, + { hours: 1, label: "1 hour" }, + { hours: 4, label: "4 hours" }, + { hours: 24, label: "1 day" }, + { hours: 72, label: "3 days" }, + { hours: 168, label: "1 week" }, +]; + +/** What an acknowledgement says, in a few words for a table row: who, and how long it lasts. */ +export function ackWords(ack: NonNullable): string { + const who = ack.assignee ? `${ack.assignee.name} has it` : `${ack.ackedBy} is on it`; + return ack.until ? `${who} until ${formatDateTime(new Date(ack.until))}` : who; +} + +/** + * "I'm on it": say that an alert has been seen, why it is what it is, who has it, and how long it should be left alone. It changes nothing about + * the problem (the alert stays, and is still counted by /metrics) but it is no longer one that nobody has looked at. + */ +export default function AckDrawer({ alert, me, onClose, onSaved }: { alert: AlertItem; me: CurrentUser; onClose: () => void; onSaved: () => void }) { + const existing = alert.ack ?? null; + const [note, setNote] = useState(existing?.note ?? ""); + const [assignee, setAssignee] = useState(existing?.assignee?.id ? String(existing.assignee.id) : ""); + const [hours, setHours] = useState(existing?.until ? "" : ""); + const [people, setPeople] = useState<{ id: number; name: string }[]>([]); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + let alive = true; + api.alerts + .assignees() + .then((res) => alive && setPeople(res.users)) + .catch(() => {}); + return () => { + alive = false; + }; + }, []); + + async function save() { + setBusy(true); + setError(null); + try { + await api.alerts.ack({ alertId: alert.id, note: note.trim(), assigneeId: assignee === "" ? null : Number(assignee), hours: hours === "" ? null : Number(hours) }); + toast.success(assignee && Number(assignee) !== me.id ? "Acknowledged and handed over." : "Acknowledged."); + onSaved(); + onClose(); + } catch (err) { + setError(readableError(err)); + } finally { + setBusy(false); + } + } + + async function takeBack() { + setBusy(true); + try { + await api.alerts.unack(alert.id); + toast.success("Acknowledgement taken back."); + onSaved(); + onClose(); + } catch (err) { + setError(readableError(err)); + } finally { + setBusy(false); + } + } + + return ( + +
{alert.message}
+ + {existing && ( +
+
+ {existing.active ? ackWords(existing) : "Its acknowledgement has run out"} +
+ Acknowledged by {existing.ackedBy} . +
+
+
+ )} + +
+ +