Runbooks per alert kind, high-vulnerability threshold, wall display extras

A "What to do" note (text and/or link) can be written once for a whole kind of alert and
shows beside every alert of that kind; alerts now carry their kind and source type.
New optional setting: warn when a Docker host has N or more high vulnerabilities (off by
default). The wall display gains tiles (Hetzner and game servers running, critical
vulnerabilities, stuck downloads) and a summary line for each group of integrations,
also in share links. Migration 0074; docs and tests included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-27 21:15:45 +01:00
1 parent d9083adb5b
commit 3d9a2310bb
35 files changed
+9129 -30

No files matched your search

+17 -4
View File
@@ -23,8 +23,8 @@ what the app actually creates.
|---|---|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0073`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 84 application tables, plus drizzle's own `__drizzle_migrations`. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0074`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 85 application tables, plus drizzle's own `__drizzle_migrations`. |
**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`.
@@ -105,10 +105,10 @@ erDiagram
event_webhooks ||--o{ webhook_deliveries : "cascade"
```
These 21 tables are without a foreign key to another table, and — but for `integration_credentials`, which `integrations` and `dns_providers` point at — with none from another table to them; where they refer to something else it is by id or name, see
These 22 tables are without a foreign key to another table, and — but for `integration_credentials`, which `integrations` and `dns_providers` point at — with none from another table to them; where they refer to something else it is by id or name, see
[soft references](#soft-references-not-foreign-keys): `settings`, `secrets`, `ipam_entries`, `domains`, `diag_log`, `notification_queue`, `consistency_ignores`, `tag_definitions`,
`maintenance_windows`, `maintenance_aliases`, `maintenance_schedules`, `metrics_tokens`, `restore_tests`, `dependencies`, `runbooks`, `renewals`, `assets`, `notes`,
`alert_rules`, `alert_snoozes` and `integration_credentials`.
`alert_rules`, `alert_snoozes`, `alert_kind_runbooks` and `integration_credentials`.
## Tables
@@ -684,6 +684,19 @@ Notes about what to do when something is wrong, attached to a server, integratio
| `updated_by` | text | The person's name at the time. |
| `created_at`, `updated_at` | text NOT NULL | ISO. |
#### `alert_kind_runbooks`
What to do first about a **kind** of alert, whichever server or integration it is about — shown beside every alert of that kind on the Alerts page (*What to do*). The kind is the first part of an alert's key (`disk`, `hetzner-cert`, `sonarr-queue`). One row for each kind; no foreign keys.
| Column | Type | Notes |
|---|---|---|
| `kind` | text PK | |
| `title` | text NOT NULL | |
| `body` | text NOT NULL, default empty | Plain text with the little formatting runbooks use; never rendered as HTML. May be empty when there is a `url`. |
| `url` | text | A page to open (`http` or `https` only); null when there is none. |
| `updated_by` | text | The person's name at the time. |
| `updated_at` | text NOT NULL | ISO. |
#### `assets`
The hardware inventory.
+1 -1
View File
@@ -59,7 +59,7 @@ the dashboard views themselves load fine. (UniFi, Nginx Proxy Manager, Home Assi
- **Suggested checks:** each container's image and published ports are also used to suggest an endpoint check on the app's own health address ([MONITORING.md](docs/MONITORING.md#endpoint-checks-and-heartbeats)). Ports are read from whatever shape Dockhand's container list carries them in (Docker's own, `"0.0.0.0:8080->80/tcp"` strings or inspect's map) — **not checked against a live Dockhand**; if none is read, the Docker host's listening ports from its agent stand in.
- **Container health:** on the 15-minute health pass the app also reads every container's state and status text and raises alerts for what is broken — see [Docker containers that are broken](docs/MONITORING.md#docker-containers-that-are-broken). A Docker host that Dockhand
cannot read (its Hawser agent is down) is an alert too, and its containers' alerts are held, not cleared. The Docker page shows each container's problem and has a **Leave alone** button.
- **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host (a setting turns it off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*.
- **Vulnerability scans:** every fifteen minutes the app also reads what Dockhand's scans have stored — `GET /api/vulnerabilities/count?env=` for each environment, and `GET /api/vulnerabilities?env=&severity=critical&limit=100` only where that count is above zero (needs the *images: view* permission). Nothing is scanned from here. **Critical** findings are a warning per Docker host — and so are **high** ones once a host has as many as a number you set (*Also warn when a Docker host has this many high vulnerabilities*, off by default) — (a setting turns the critical one off — see [MONITORING.md](docs/MONITORING.md#docker-containers-that-are-broken)); the Docker page lists them. **Written from Dockhand's source and OpenAPI file, not tried against a live Dockhand** — its published docs show the lists at the top level of the answer, its code puts the counts under `summary`; the reader follows the code and reads nothing it does not find as *unknown*.
### Uptime Kuma
+4 -4
View File
@@ -131,8 +131,8 @@ All modules from the original plan are built:
container (from Dockhand's own cached update check, plus a button
to trigger a fresh one), a live running/total widget (with an
updates-available count), and the **vulnerabilities** Dockhand's own
scans found per host (critical ones listed; a warning is raised for
critical ones, and it can be switched off).
scans found per host (critical ones listed; a warning is raised for the critical ones — and, if you give a number, for hosts with that many high ones — and
it can be switched off).
- **Uptime Kuma** — every monitor's status (up/down/pending/maintenance),
response time, and 24h/30d uptime, with certificate days remaining where
applicable. Each monitor is matched to one of your servers when its
@@ -421,7 +421,7 @@ with a sentence underneath — "If it goes down, 4 things go with it: …" — s
**Wall display** (`/wall`, also on the dashboard and the user menu) is a
full-screen status board for a spare monitor or tablet: the overall state in
large type, servers online, alerts, maintenance windows and failing integrations,
large type, servers online, alerts, maintenance windows and failing integrations — and, for what is connected, how many Hetzner servers and game servers are running, how many critical vulnerabilities and stuck downloads there are, and one line for each group of the menu (Infrastructure, Apps …) saying whether it is well —
the problems themselves, and each integration's status, refreshing every 30
seconds with the screen kept awake where the browser allows. Esc returns to the
app.
@@ -613,7 +613,7 @@ check the UPS first, then…" — on a server's page (the Runbooks card) or an
integration's card. A note can be for anything about the thing, or for one kind of
alert (a disk filling up). On the Alerts page an alert about a server or integration
shows a Runbook link when a note fits it, and opens it in a side panel with the notes
that fit that alert first. Everyone signed in reads them; operators write them. The
that fit that alert first. A note can also be written once for a whole **kind** of alert — the *What to do* link beside an alert, "when a Hetzner certificate is about to end: renew it in the console, then…", with an optional page to open — and then shows beside every alert of that kind, whatever it is about. Everyone signed in reads them; operators write them. The
text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a block
of commands, backticks inline code; links become clickable, and nothing typed in a
note can run as HTML.
+1 -1
View File
@@ -122,7 +122,7 @@ even further, to admin only:
badge on the Alerts page, linking to the ticket in osTicket); switching the
feature on, choosing what opens a ticket, and sending a test ticket are
under Settings → Notifications → Alert tickets, which is admin-only. **Closing a ticket** along with an alert (the tick box in the Close drawer, for one alert or several picked together) is an operator's, like closing the alert; the login it writes with, and **Test closing**, are administrators'.
- **Runbooks**: everyone reads them (on a server's page, an integration's card,
- **Runbooks**: everyone reads them (on a server's page, an integration's card, the *What to do* link beside an alert for a whole kind of alert,
next to an alert); writing, editing and removing one needs operator.
- **Inventory**: everyone can see the hardware (and print its labels); adding, editing and removing needs
operator, and so does *Refresh from agent*; the price of power used for the running cost is admin-only.
+3 -2
View File
@@ -155,7 +155,8 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) |
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
@@ -421,7 +422,7 @@ Maintenance on the Docker integration holds these alerts like its other ones. If
Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`).
**Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other.
**To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
**High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host.
## Endpoint checks and heartbeats
+8
View File
@@ -0,0 +1,8 @@
CREATE TABLE `alert_kind_runbooks` (
`kind` text PRIMARY KEY NOT NULL,
`title` text NOT NULL,
`body` text DEFAULT '' NOT NULL,
`url` text,
`updated_by` text,
`updated_at` text NOT NULL
);
File diff suppressed because it is too large. Load diff
+7
View File
@@ -519,6 +519,13 @@
"when": 1793126767536,
"tag": "0073_chilly_silver_surfer",
"breakpoints": true
},
{
"idx": 74,
"version": "6",
"when": 1793128598267,
"tag": "0074_open_the_hood",
"breakpoints": true
}
]
}
+2
View File
@@ -70,6 +70,7 @@ import { hetznerRouter } from "./routes/hetzner.js";
import { speedtestRouter } from "./routes/speedtest.js";
import { fragnetRouter } from "./routes/fragnet.js";
import { arrRouter } from "./routes/arr.js";
import { wallRouter } from "./routes/wall.js";
import { costsRouter } from "./routes/costs.js";
import { nextcloudRouter } from "./routes/nextcloud.js";
import { serviceLevelsRouter } from "./routes/serviceLevels.js";
@@ -218,6 +219,7 @@ export function createApp() {
app.use("/api/speedtest", speedtestRouter);
app.use("/api/fragnet", fragnetRouter);
app.use("/api/arr", arrRouter);
app.use("/api/wall", wallRouter);
app.use("/api/costs", costsRouter);
app.use("/api/nextcloud", nextcloudRouter);
app.use("/api/service-levels", serviceLevelsRouter);
+12
View File
@@ -884,6 +884,18 @@ export const runbooks = sqliteTable(
(t) => [index("runbooks_target_idx").on(t.targetType, t.targetId)],
);
// What to do first about a KIND of alert, whichever server or integration it is about — "when a Hetzner certificate is about to end: renew it in the console, then check the
// load balancer". The kind is the first part of an alert's key ("disk", "hetzner-cert", "sonarr-queue"). Shown beside every alert of that kind on the Alerts page, as a note and/or a
// link. Read by routes/runbooks.ts.
export const alertKindRunbooks = sqliteTable("alert_kind_runbooks", {
kind: text("kind").primaryKey(),
title: text("title").notNull(),
body: text("body").notNull().default(""),
url: text("url"), // a page to open (the host's documentation, a wiki page); null when there is none
updatedBy: text("updated_by"),
updatedAt: text("updated_at").notNull(), // ISO
});
// ─── Acknowledgements and incidents ─────────────────────────────────────────
// "I'm on it" for one alert — see services/alertAcks.ts. Keyed by the alert's stable id (not a foreign key to anything); gone once the alert has been.
+1
View File
@@ -605,6 +605,7 @@ integrationsRouter.get("/:id/dockhand/vulnerabilities", asyncHandler(async (req,
const status = statuses[id] ?? null;
res.json({
alerts: settings.healthChecks.vulnerabilityAlerts,
highThreshold: settings.healthChecks.vulnerabilityHighThreshold ?? 0,
status,
fresh: isDockerVulnFresh(status ?? undefined, Date.now()),
environments: rows
+46 -1
View File
@@ -2,7 +2,7 @@ import { Router } from "express";
import { and, eq } from "drizzle-orm";
import { z } from "zod";
import { db } from "../db/client.js";
import { dnsProviders, integrations, maintenanceTargetTypes, networkDevices, runbooks, servers } from "../db/schema.js";
import { alertKindRunbooks, dnsProviders, integrations, maintenanceTargetTypes, networkDevices, runbooks, servers } from "../db/schema.js";
import { requireAuth, requireRole } from "../auth/middleware.js";
import { recordAudit } from "../services/audit.js";
import { describeTarget } from "../services/maintenance.js";
@@ -25,6 +25,22 @@ const bodySchema = z.object({
});
const createSchema = bodySchema.extend({ targetType: z.enum(maintenanceTargetTypes), targetId: z.number().int().positive() });
/** What an alert kind looks like: the first part of a key, letters, digits and dashes. */
const KIND = /^[A-Za-z0-9][A-Za-z0-9_-]{0,59}$/;
const kindSchema = z.object({
title: z.string().trim().min(1).max(120),
body: z.string().max(20_000).default(""),
/** A page to open; http or https only. */
url: z
.string()
.trim()
.max(500)
.nullable()
.optional()
.transform((v) => (v === undefined || v === null || v === "" ? null : v))
.refine((v) => v === null || /^https?:\/\/[^\s]+$/i.test(v), "A web address starting with http:// or https://"),
});
const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN);
// What there is for one thing: the full text.
@@ -55,6 +71,35 @@ runbooksRouter.get("/summary", asyncHandler(async (_req, res) => {
res.json({ runbooks: rows.filter((r) => exists[r.targetType].has(r.targetId)) });
}));
// What there is for kinds of alert (by what they are about, wherever it happens), for everyone: it is what to do first, and is read by the Alerts page.
runbooksRouter.get("/kinds", asyncHandler(async (_req, res) => {
res.json({ kinds: await db.select().from(alertKindRunbooks).orderBy(alertKindRunbooks.kind) });
}));
runbooksRouter.put("/kinds/:kind", requireRole("operator"), asyncHandler(async (req, res) => {
const kind = String(req.params.kind);
const parsed = kindSchema.safeParse(req.body);
if (!KIND.test(kind)) return res.status(400).json({ error: "invalid_kind" });
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Give it a title, and a web address starting with http:// or https:// if you add one.", details: parsed.error.flatten() });
if (parsed.data.body.trim() === "" && parsed.data.url === null) return res.status(400).json({ error: "invalid_body", message: "Write what to do, or give a page to open." });
const actor = req.currentUser!;
const [before] = await db.select({ kind: alertKindRunbooks.kind }).from(alertKindRunbooks).where(eq(alertKindRunbooks.kind, kind)).limit(1);
const values = { title: parsed.data.title, body: parsed.data.body, url: parsed.data.url, updatedBy: actor.name ?? actor.email ?? actor.oidcSub, updatedAt: new Date().toISOString() };
const [row] = await db.insert(alertKindRunbooks).values({ kind, ...values }).onConflictDoUpdate({ target: alertKindRunbooks.kind, set: values }).returning();
await recordAudit({ actor, category: "runbook", action: before ? "update" : "create", targetType: "alert_kind", targetId: kind, detail: { title: parsed.data.title } });
res.json({ runbook: row });
}));
runbooksRouter.delete("/kinds/:kind", requireRole("operator"), asyncHandler(async (req, res) => {
const kind = String(req.params.kind);
if (!KIND.test(kind)) return res.status(400).json({ error: "invalid_kind" });
const [before] = await db.select().from(alertKindRunbooks).where(eq(alertKindRunbooks.kind, kind)).limit(1);
if (!before) return res.status(404).json({ error: "not_found" });
await db.delete(alertKindRunbooks).where(eq(alertKindRunbooks.kind, kind));
await recordAudit({ actor: req.currentUser!, category: "runbook", action: "delete", targetType: "alert_kind", targetId: kind, detail: { title: before.title } });
res.status(204).end();
}));
runbooksRouter.post("/", requireRole("operator"), asyncHandler(async (req, res) => {
const parsed = createSchema.safeParse(req.body);
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Give it a title and some text, and say what it is for.", details: parsed.error.flatten() });
+1 -1
View File
@@ -144,7 +144,7 @@ const updateSchema = z.object({
.partial()
.optional(),
healthChecks: z
.object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730), newPortAlerts: z.boolean(), flapRaises: z.number().int().min(0).max(20), deviceOfflineMinutes: z.number().int().min(5).max(1440), osSupportWarnDays: z.number().int().min(0).max(365), baselineAlerts: z.boolean(), containerAlerts: z.boolean(), vulnerabilityAlerts: z.boolean().default(true), containerIgnore: z.array(z.string().trim().min(1).max(100)).max(200) })
.object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730), newPortAlerts: z.boolean(), flapRaises: z.number().int().min(0).max(20), deviceOfflineMinutes: z.number().int().min(5).max(1440), osSupportWarnDays: z.number().int().min(0).max(365), baselineAlerts: z.boolean(), containerAlerts: z.boolean(), vulnerabilityAlerts: z.boolean().default(true), vulnerabilityHighThreshold: z.number().int().min(0).max(10000).default(0), containerIgnore: z.array(z.string().trim().min(1).max(100)).max(200) })
.partial()
.optional(),
quietHours: z
+12
View File
@@ -0,0 +1,12 @@
import { Router } from "express";
import { requireAuth } from "../auth/middleware.js";
import { wallExtrasNow } from "../services/wallExtras.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const wallRouter = Router();
wallRouter.use(requireAuth);
/** What the signed-in wall display shows besides the four tiles it works out itself: the extra tiles (Hetzner, game servers, vulnerabilities, stuck downloads) and a summary for each group of integrations. */
wallRouter.get("/extras", asyncHandler(async (_req, res) => {
res.json(await wallExtrasNow());
}));
+7
View File
@@ -6,9 +6,16 @@ export type AlertSeverity = "critical" | "warning" | "info";
/** What kind of problem — drives the filter on the Alerts page. */
export type AlertCategory = "offline" | "disk" | "services" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
/** The kind of an alert: the first part of its key ("disk:server:3:/" is a "disk" alert, "hetzner-cert:2:20" a "hetzner-cert" one). What a kind-wide runbook is attached to. */
export const kindOfKey = (key: string): string => key.split(":")[0] || key;
export interface Alert {
/** Stable, so the page can key rows on it. */
id: string;
/** What kind of alert it is (see kindOfKey). */
kind: string;
/** The kind of source it is about — an integration's type ("hetzner", "sonarr"), "server", "slo" … — which the wall groups by. */
type: string;
severity: AlertSeverity;
category: AlertCategory;
/** Where it comes from, as a short name: "Server", "Proxmox", "Secrets", ... */
+4 -2
View File
@@ -43,7 +43,7 @@ import { alertIdOfNode, dependencyCover, integrationTypeCause, takenByType } fro
import { endpointConditions } from "./endpointChecks.js";
import { heartbeatConditions } from "./heartbeats.js";
import { latestTests, restoreAlerts } from "./restoreTests.js";
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
import { kindOfKey, type Alert, type AlertCategory, type AlertSeverity, type SourceFailure } from "./alertTypes.js";
export interface AlertsReport {
alerts: Alert[];
@@ -125,7 +125,7 @@ function originOfConditionKey(key: string): { type: string; id: number | null; c
if (m) return { type: "slo", id: Number(m[1]), category: "monitoring" };
m = /^container-(?:restarting|unhealthy|stopped|created|host):(\d+):/.exec(key);
if (m) return { type: "dockhand", id: Number(m[1]), category: "services" };
m = /^docker-vuln:(\d+):/.exec(key);
m = /^docker-vuln(?:-high)?:(\d+):/.exec(key);
if (m) return { type: "dockhand", id: Number(m[1]), category: "services" };
m = /^speedtest-(?:slow|failing):(\d+)/.exec(key);
if (m) return { type: "speedtest", id: Number(m[1]), category: "monitoring" };
@@ -198,6 +198,8 @@ export async function collectAlerts(): Promise<AlertsReport> {
const meta = SOURCES[a.type];
alerts.push({
id: `${a.category}:${a.key}`,
kind: kindOfKey(a.key),
type: a.type,
severity: a.severity,
category: a.category,
source: meta?.label ?? sourceLabel(a.type),
+12 -2
View File
@@ -120,10 +120,20 @@ export interface DockerVulnProblem {
severity: "warning";
}
/** Pure: a warning for each Docker host whose stored scans have critical findings. A host whose summary is unknown says nothing. */
export function dockerVulnProblems(integrationId: number, rows: Pick<StateRow, "environmentId" | "environmentName" | "critical" | "high" | "findings">[]): DockerVulnProblem[] {
/**
* Pure: a warning for each Docker host whose stored scans have critical findings — and, when `highThreshold` is above 0, one for each whose scans have at least that many *high* ones. A host
* whose summary is unknown says nothing.
*/
export function dockerVulnProblems(integrationId: number, rows: Pick<StateRow, "environmentId" | "environmentName" | "critical" | "high" | "findings">[], highThreshold = 0): DockerVulnProblem[] {
const out: DockerVulnProblem[] = [];
for (const r of [...rows].sort((a, b) => a.environmentName.localeCompare(b.environmentName))) {
if (highThreshold > 0 && r.high !== null && r.high >= highThreshold) {
out.push({
key: `docker-vuln-high:${integrationId}:${r.environmentId}`,
severity: "warning",
message: `${r.environmentName}: ${r.high} high ${r.high === 1 ? "vulnerability" : "vulnerabilities"} in the scanned images (you asked to hear of ${highThreshold} or more)`,
});
}
if (r.critical === null || r.critical <= 0) continue;
const findings = findingsOf(r);
const images = [...new Set(findings.map((f) => f.imageName))];
+1 -1
View File
@@ -761,7 +761,7 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel
held.add(`dockhand:${row.id}:vuln`);
continue;
}
(snapshot.dockerVulns ??= []).push({ integrationId: row.id, problems: dockerVulnProblems(row.id, stored) });
(snapshot.dockerVulns ??= []).push({ integrationId: row.id, problems: dockerVulnProblems(row.id, stored, checks.vulnerabilityHighThreshold ?? 0) });
}
} catch (err) {
console.error("[health] couldn't judge the vulnerability scans:", err instanceof Error ? err.message : err);
+12
View File
@@ -0,0 +1,12 @@
/**
* The kinds of integration in the groups the sidebar has, in its order. The same list as TYPE_GROUPS in web/src/utils/integrationTypes.ts — a test compares the two, so a kind added to one
* and not the other is caught. A kind in no group (an older one) is simply not in any group's summary.
*/
export const INTEGRATION_GROUPS: { id: string; label: string; types: string[] }[] = [
{ id: "infrastructure", label: "Infrastructure", types: ["proxmox", "synology", "dockhand", "hetzner", "fragnet", "tailscale"] },
{ id: "apps", label: "Apps", types: ["homeassistant", "jellyfin", "sonarr", "radarr", "prowlarr", "immich", "paperless", "nextcloud", "authentik"] },
{ id: "backups", label: "Backups", types: ["pbs"] },
{ id: "network", label: "Network", types: ["unifi", "npm", "phpipam", "netbox"] },
{ id: "monitoring", label: "Monitoring", types: ["uptimekuma", "prometheus", "ups", "speedtest"] },
{ id: "automation", label: "Automation", types: ["semaphore", "gitea", "osticket", "n8n"] },
];
+1 -1
View File
@@ -31,7 +31,7 @@ export function subjectOfConditionKey(key: string): string | null {
if (m) return `server:${m[1]}`;
m = /^container-(?:restarting|unhealthy|stopped|created|host):(\d+):/.exec(key);
if (m) return `integration:${m[1]}`;
m = /^docker-vuln:(\d+):/.exec(key);
m = /^docker-vuln(?:-high)?:(\d+):/.exec(key);
if (m) return `integration:${m[1]}`;
m = /^jellyfin-(?:restart|task|logins):(\d+)/.exec(key);
if (m) return `integration:${m[1]}`;
+8
View File
@@ -10,6 +10,7 @@ import { getAlerts } from "./alerts.js";
import { getSourceStatus } from "./integrationHealthMonitor.js";
import { describeTarget, listActiveWindows, activeSubjects } from "./maintenance.js";
import { buildServerSummary } from "./serverSummary.js";
import { buildWallExtras, type WallGroup, type WallTile } from "./wallExtras.js";
import { getSettings } from "./settingsStore.js";
import type { ShareLinkDetail } from "../db/schema.js";
@@ -23,6 +24,9 @@ export interface PublicWall {
/** Windows running now. */
maintenance: number;
integrationsFailing: number;
/** What else is worth a glance (Hetzner servers running, critical vulnerabilities …) and a summary for each group of integrations — numbers and group names only, so every detail level has them. */
tiles: WallTile[];
groups: WallGroup[];
/** Only with detail "names". */
problems?: { id: string; severity: "critical" | "warning"; message: string }[];
maintenanceWindows?: { id: number; name: string; endsAt: string }[];
@@ -54,6 +58,8 @@ export async function buildPublicWall(detail: ShareLinkDetail, now: Date = new D
const warning = report.counts.warning;
const level: PublicWall["level"] = critical > 0 ? "critical" : warning > 0 ? "warning" : report.couldntCheck.length > 0 ? "unsure" : "ok";
const extras = await buildWallExtras(enabled.map((i) => ({ id: i.id, type: i.type })), new Set(failing.map((i) => i.type)), report.alerts, now.getTime());
const out: PublicWall = {
detail,
level,
@@ -62,6 +68,8 @@ export async function buildPublicWall(detail: ShareLinkDetail, now: Date = new D
alerts: { critical, warning },
maintenance: windows.length,
integrationsFailing: failing.length,
tiles: extras.tiles,
groups: extras.groups,
generatedAt: now.toISOString(),
display: { dateFormat: settings.display.dateFormat, timeFormat: settings.display.timeFormat },
};
+3 -1
View File
@@ -144,6 +144,8 @@ export interface HealthCheckSettings {
containerAlerts: boolean;
/** Whether critical vulnerabilities found by Dockhand's scans are alerted (services/dockerVulnSync.ts). */
vulnerabilityAlerts: boolean;
/** A Docker host whose scanned images have at least this many HIGH findings is a warning too (0 = off, the default; critical ones are alerted whatever this is, while vulnerabilityAlerts is on). */
vulnerabilityHighThreshold: number;
/** Containers to leave alone: a name, or "host/name"; * stands for anything. */
containerIgnore: string[];
}
@@ -329,7 +331,7 @@ const DEFAULTS: AppSettings = {
selfBackup: { enabled: true, time: "03:30", keep: 14 },
profilePictures: { gravatar: false, identityProvider: false },
signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" },
healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true, containerAlerts: true, vulnerabilityAlerts: true, containerIgnore: [] },
healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true, containerAlerts: true, vulnerabilityAlerts: true, vulnerabilityHighThreshold: 0, containerIgnore: [] },
// Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every
// Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN.
consistency: { excludedRanges: ["172.16.0.0/12"] },
+152
View File
@@ -0,0 +1,152 @@
/**
* What the wall display shows beyond its four fixed tiles (servers online, alerts, in maintenance, integrations failing): a tile for each thing that is connected and worth a glance — Hetzner
* servers running, game servers running, critical vulnerabilities, downloads that are stuck — and a summary for each group of integrations the menu has (Infrastructure, Apps, Network …): how
* many are connected, how many are failing, how many alerts they have. Only numbers and the names of the groups, so a share link that gives "numbers only" may carry all of it.
*
* A tile is there only when its source is connected and was read lately: a number read an hour ago is not shown as how things are.
*/
import { and, eq, inArray } from "drizzle-orm";
import { db } from "../db/client.js";
import { arrState, dockhandVulnerabilities, fragnetState, hetznerState, integrations } from "../db/schema.js";
import type { Alert } from "./alertTypes.js";
import { arrSyncStatuses, isArrFresh, storedOf as arrStoredOf, stuckItems } from "./arrSync.js";
import { dockerVulnSyncStatuses, isDockerVulnFresh } from "./dockerVulnSync.js";
import { fragnetSyncStatuses, isFragnetFresh, serversOf as fragnetServersOf } from "./fragnetSync.js";
import { hetznerSyncStatuses, isHetznerFresh, serversOf as hetznerServersOf } from "./hetznerSync.js";
import { getAlerts } from "./alerts.js";
import { getSourceStatus } from "./integrationHealthMonitor.js";
import { INTEGRATION_GROUPS } from "./integrationGroups.js";
export type WallTileState = "ok" | "warn" | "bad" | "info";
export interface WallTile {
id: string;
label: string;
figure: string;
state: WallTileState;
}
export interface WallGroup {
id: string;
label: string;
/** Connected (and switched on). */
integrations: number;
failing: number;
alerts: { critical: number; warning: number };
}
export interface WallExtras {
tiles: WallTile[];
groups: WallGroup[];
}
/** Pure: the groups, each with what is connected in it, what is failing and what is alerting — a group with nothing connected is left out. */
export function buildGroups(connected: { id: number; type: string }[], failing: Set<string>, alerts: Pick<Alert, "type" | "severity" | "silenced">[]): WallGroup[] {
const out: WallGroup[] = [];
for (const g of INTEGRATION_GROUPS) {
const mine = connected.filter((i) => g.types.includes(i.type));
if (mine.length === 0) continue;
const active = alerts.filter((a) => !a.silenced && g.types.includes(a.type));
out.push({
id: g.id,
label: g.label,
integrations: mine.length,
failing: mine.filter((i) => failing.has(i.type)).length,
alerts: { critical: active.filter((a) => a.severity === "critical").length, warning: active.filter((a) => a.severity === "warning").length },
});
}
return out;
}
const plural = (n: number, one: string, many: string) => (n === 1 ? one : many);
/** Pure: how many of some servers run, as a tile — "3/3"; a stop is a warning for a cloud server that is billed, and only a note for a game server, which is often stopped on purpose. */
export function runningTile(id: string, label: string, running: number, total: number, stopIs: "warn" | "info"): WallTile {
return { id, label, figure: `${running}/${total}`, state: running === total ? "ok" : stopIs };
}
export async function buildWallExtras(
connected: { id: number; type: string }[],
failing: Set<string>,
alerts: Pick<Alert, "type" | "severity" | "silenced">[],
nowMs: number = Date.now(),
): Promise<WallExtras> {
const tiles: WallTile[] = [];
const ids = (types: string[]) => connected.filter((i) => types.includes(i.type)).map((i) => i.id);
try {
const hz = ids(["hetzner"]);
if (hz.length > 0) {
const statuses = await hetznerSyncStatuses();
let running = 0;
let total = 0;
let read = false;
for (const row of await db.select().from(hetznerState)) {
if (!hz.includes(row.integrationId) || !isHetznerFresh(statuses[row.integrationId], nowMs)) continue;
read = true;
for (const s of hetznerServersOf(row)) {
total++;
if (s.status === "running") running++;
}
}
if (read && total > 0) tiles.push(runningTile("hetzner", "Hetzner servers running", running, total, "warn"));
}
const fn = ids(["fragnet"]);
if (fn.length > 0) {
const statuses = await fragnetSyncStatuses();
let running = 0;
let total = 0;
let read = false;
for (const row of await db.select().from(fragnetState)) {
if (!fn.includes(row.integrationId) || !isFragnetFresh(statuses[row.integrationId], nowMs)) continue;
read = true;
for (const s of fragnetServersOf(row)) {
if (s.suspended) continue;
total++;
if (s.state === "running") running++;
}
}
if (read && total > 0) tiles.push(runningTile("game-servers", "game servers running", running, total, "info"));
}
const dh = ids(["dockhand"]);
if (dh.length > 0) {
const statuses = await dockerVulnSyncStatuses();
let critical = 0;
let known = false;
for (const row of await db.select().from(dockhandVulnerabilities).where(inArray(dockhandVulnerabilities.integrationId, dh))) {
if (!isDockerVulnFresh(statuses[row.integrationId], nowMs) || row.critical === null) continue;
known = true;
critical += row.critical;
}
if (known) tiles.push({ id: "vulnerabilities", label: `critical ${plural(critical, "vulnerability", "vulnerabilities")}`, figure: String(critical), state: critical > 0 ? "bad" : "ok" });
}
const media = ids(["sonarr", "radarr"]);
if (media.length > 0) {
const statuses = await arrSyncStatuses();
let stuck = 0;
let known = false;
for (const row of await db.select().from(arrState).where(inArray(arrState.integrationId, media))) {
const state = arrStoredOf(row);
if (!isArrFresh(statuses[row.integrationId], nowMs) || state.queue === null) continue;
known = true;
stuck += stuckItems(state.queue, nowMs).length;
}
if (known) tiles.push({ id: "stuck-downloads", label: `stuck ${plural(stuck, "download", "downloads")}`, figure: String(stuck), state: stuck > 0 ? "warn" : "ok" });
}
} catch (err) {
// A tile that cannot be worked out is simply not there; the board must never fail for the sake of an extra.
console.error("[wall] couldn't work out the extra tiles:", err instanceof Error ? err.message : err);
}
return { tiles, groups: buildGroups(connected, failing, alerts) };
}
/** The extras for the signed-in wall, which has none of it read yet: the connected integrations, which kinds are failing, and the alerts there are. */
export async function wallExtrasNow(nowMs: number = Date.now()): Promise<WallExtras> {
const [rows, report] = await Promise.all([db.select({ id: integrations.id, type: integrations.type }).from(integrations).where(and(eq(integrations.enabled, true))), getAlerts(false)]);
const failing = new Set(rows.filter((r) => getSourceStatus(r.type).status === "failing").map((r) => r.type));
return buildWallExtras(rows, failing, report.alerts, nowMs);
}
+252
View File
@@ -0,0 +1,252 @@
/**
* Runbooks for a KIND of alert (written once, shown beside every alert of that kind) and the wall display's extras: a tile for each connected thing worth a glance (Hetzner servers running, game
* servers running, critical vulnerabilities, stuck downloads) and a summary for each group of integrations the menu has.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import type { Server } from "node:http";
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-kinds-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, integrations, auditLog, dockhandVulnerabilities } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { runbooksRouter } = await import("../src/routes/runbooks.js");
const { wallRouter } = await import("../src/routes/wall.js");
const { kindOfKey } = await import("../src/services/alertTypes.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const hz = await import("../src/services/hetznerSync.js");
const fn = await import("../src/services/fragnetSync.js");
const arr = await import("../src/services/arrSync.js");
const wall = await import("../src/services/wallExtras.js");
const { INTEGRATION_GROUPS } = await import("../src/services/integrationGroups.js");
const { buildPublicWall } = await import("../src/services/publicWall.js");
const { setInternalFlag } = await import("../src/services/settingsStore.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const MIN = 60_000;
const HOUR = 60 * MIN;
const iso = (ms: number) => new Date(ms).toISOString();
describe("the kind of an alert", () => {
test("is the first part of its key", () => {
assert.equal(kindOfKey("disk:server:3:/"), "disk");
assert.equal(kindOfKey("hetzner-cert:2:20"), "hetzner-cert");
assert.equal(kindOfKey("sonarr-queue:7"), "sonarr-queue");
assert.equal(kindOfKey("plain"), "plain");
});
});
describe("runbooks for a kind of alert", () => {
let app: Server;
let base = "";
const call = (sub: string, method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json", "x-sub": sub }, body: body === undefined ? undefined : JSON.stringify(body) });
before(async () => {
await db.insert(users).values([{ oidcSub: "viewer", name: "Vic", role: "viewer" }, { oidcSub: "operator", name: "Olle", role: "operator" }]);
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: String(req.headers["x-sub"] ?? "viewer") } };
next();
});
a.use("/api/runbooks", runbooksRouter);
a.use("/api/wall", wallRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => app.close());
test("an operator writes one, everyone signed in reads the list; a viewer cannot write", async () => {
assert.equal((await call("viewer", "PUT", "/runbooks/kinds/hetzner-cert", { title: "x", body: "y" })).status, 403);
const put = await call("operator", "PUT", "/runbooks/kinds/hetzner-cert", { title: "If a certificate is about to end", body: "Renew it in the Hetzner console.\n\n- then check the load balancer", url: "https://wiki.example.lan/certs" });
assert.equal(put.status, 200);
const list = (await (await call("viewer", "GET", "/runbooks/kinds")).json()) as { kinds: { kind: string; title: string; body: string; url: string | null; updatedBy: string }[] };
assert.deepEqual(list.kinds.map((k) => [k.kind, k.title, k.url, k.updatedBy]), [["hetzner-cert", "If a certificate is about to end", "https://wiki.example.lan/certs", "Olle"]]);
assert.match(list.kinds[0].body, /Renew it in the Hetzner console/);
});
test("writing it again changes it (one for each kind), with a link alone or a text alone", async () => {
assert.equal((await call("operator", "PUT", "/runbooks/kinds/hetzner-cert", { title: "Certificates", body: "", url: "https://wiki.example.lan/certs" })).status, 200);
assert.equal((await call("operator", "PUT", "/runbooks/kinds/disk", { title: "A disk is filling", body: "Check what grew: du -sh /*" })).status, 200);
const list = (await (await call("viewer", "GET", "/runbooks/kinds")).json()) as { kinds: { kind: string; title: string; url: string | null }[] };
assert.deepEqual(list.kinds.map((k) => [k.kind, k.title, k.url]), [["disk", "A disk is filling", null], ["hetzner-cert", "Certificates", "https://wiki.example.lan/certs"]]);
});
test("it must say something, be titled, and a link must be a web address; a kind is letters, digits and dashes", async () => {
assert.equal((await call("operator", "PUT", "/runbooks/kinds/disk", { title: "x", body: " " })).status, 400, "nothing to say");
assert.equal((await call("operator", "PUT", "/runbooks/kinds/disk", { title: "", body: "y" })).status, 400, "no title");
assert.equal((await call("operator", "PUT", "/runbooks/kinds/disk", { title: "x", body: "y", url: "javascript:alert(1)" })).status, 400, "not a web address");
assert.equal((await call("operator", "PUT", "/runbooks/kinds/disk", { title: "x", body: "y", url: "ftp://example.lan/x" })).status, 400);
assert.equal((await call("operator", "PUT", "/runbooks/kinds/a%20b", { title: "x", body: "y" })).status, 400, "a kind with a space");
assert.equal((await call("operator", "PUT", `/runbooks/kinds/${"k".repeat(61)}`, { title: "x", body: "y" })).status, 400, "too long");
const list = (await (await call("viewer", "GET", "/runbooks/kinds")).json()) as { kinds: { kind: string; title: string }[] };
assert.equal(list.kinds.find((k) => k.kind === "disk")?.title, "A disk is filling", "what was refused changed nothing");
});
test("it can be taken away by an operator, and says so when there is none; every change is in the audit log", async () => {
assert.equal((await call("viewer", "DELETE", "/runbooks/kinds/disk")).status, 403);
assert.equal((await call("operator", "DELETE", "/runbooks/kinds/disk")).status, 204);
assert.equal((await call("operator", "DELETE", "/runbooks/kinds/disk")).status, 404);
const entries = (await db.select().from(auditLog)).filter((e) => e.targetType === "alert_kind").map((e) => `${e.action}:${e.targetId}`);
assert.deepEqual(entries, ["create:hetzner-cert", "update:hetzner-cert", "create:disk", "delete:disk"]);
});
});
describe("an alert says what kind and what source it is", () => {
let id = 0;
before(async () => {
id = (await db.insert(integrations).values({ type: "sonarr", name: "TV", baseUrl: "http://sonarr", config: JSON.stringify({}) }).returning({ id: integrations.id }))[0].id;
await arr.storeRead(id, { version: "4", health: [{ source: "DownloadClientCheck", type: "error", message: "No download client is available" }], queue: { total: 0, items: [] }, disks: [], missing: 0, indexers: null }, new Date());
await setInternalFlag("arrSync", JSON.stringify({ [id]: { at: new Date().toISOString(), ok: true, error: null } }));
});
after(async () => {
await arr.forgetArr(id);
await db.delete(integrations).where(eq(integrations.id, id));
});
test("kind from the key, type from where it came from", async () => {
invalidateAlertsCache();
const a = (await getAlerts(true)).alerts.find((x) => x.id.includes(`sonarr-health:${id}:`))!;
assert.deepEqual([a.kind, a.type], ["sonarr-health", "sonarr"]);
});
});
describe("the wall's groups", () => {
test("the groups are the ones the menu has: the server's list is the web's list", () => {
const text = readFileSync(join(import.meta.dirname, "../../web/src/utils/integrationTypes.ts"), "utf8");
const webGroups = [...text.matchAll(/\{ id: "([a-z]+)", label: "([A-Za-z]+)", types: \[([^\]]*)\] \}/g)].map((m) => ({ id: m[1], label: m[2], types: [...m[3].matchAll(/"([a-z0-9]+)"/g)].map((t) => t[1]) }));
assert.ok(webGroups.length >= 6, "the web's groups were found");
assert.equal(JSON.stringify(INTEGRATION_GROUPS), JSON.stringify(webGroups));
});
const connected = [{ id: 1, type: "proxmox" }, { id: 2, type: "hetzner" }, { id: 3, type: "jellyfin" }, { id: 4, type: "sonarr" }, { id: 5, type: "legacy-kind" }];
const alert = (type: string, severity: "critical" | "warning" | "info", silenced = false) => ({ type, severity, silenced });
test("a group with something connected is there, with how many are connected, failing and alerting; a group with nothing is not", () => {
const g = wall.buildGroups(connected, new Set(["hetzner"]), [alert("proxmox", "critical"), alert("hetzner", "warning"), alert("hetzner", "warning"), alert("sonarr", "warning"), alert("jellyfin", "info")]);
assert.deepEqual(g.map((x) => [x.id, x.integrations, x.failing, x.alerts.critical, x.alerts.warning]), [["infrastructure", 2, 1, 1, 2], ["apps", 2, 0, 0, 1]]);
});
test("a silenced alert does not count, nor does one about something in no group", () => {
const g = wall.buildGroups(connected, new Set(), [alert("proxmox", "critical", true), alert("server", "critical"), alert("legacy-kind", "warning")]);
assert.deepEqual(g.map((x) => [x.id, x.alerts.critical, x.alerts.warning]), [["infrastructure", 0, 0], ["apps", 0, 0]]);
});
test("running tiles: all running is ok; a stopped cloud server is a warning, a stopped game server only a note", () => {
assert.deepEqual({ ...wall.runningTile("a", "x", 3, 3, "warn") }, { id: "a", label: "x", figure: "3/3", state: "ok" });
assert.equal(wall.runningTile("a", "x", 2, 3, "warn").state, "warn");
assert.equal(wall.runningTile("a", "x", 2, 3, "info").state, "info");
});
});
describe("the wall's tiles, from what is stored", () => {
const ids: Record<string, number> = {};
const mk = async (type: "hetzner" | "fragnet" | "dockhand" | "sonarr", name: string) => (await db.insert(integrations).values({ type, name, baseUrl: "http://x", config: JSON.stringify({}) }).returning({ id: integrations.id }))[0].id;
const fresh = async () => {
const at = { at: new Date().toISOString(), ok: true, error: null };
await setInternalFlag("hetznerSync", JSON.stringify({ [ids.hz]: at }));
await setInternalFlag("fragnetSync", JSON.stringify({ [ids.fn]: at }));
await setInternalFlag("dockerVulnSync", JSON.stringify({ [ids.dh]: at }));
await setInternalFlag("arrSync", JSON.stringify({ [ids.sonarr]: at }));
};
const stale = async () => {
const at = { at: new Date(Date.now() - 3 * HOUR).toISOString(), ok: true, error: null };
await setInternalFlag("hetznerSync", JSON.stringify({ [ids.hz]: at }));
await setInternalFlag("fragnetSync", JSON.stringify({ [ids.fn]: at }));
await setInternalFlag("dockerVulnSync", JSON.stringify({ [ids.dh]: at }));
await setInternalFlag("arrSync", JSON.stringify({ [ids.sonarr]: at }));
};
const connected = () => Object.values(ids).map((id, i) => ({ id, type: ["hetzner", "fragnet", "dockhand", "sonarr"][i] }));
const tileOf = (t: Awaited<ReturnType<typeof wall.buildWallExtras>>, id: string) => t.tiles.find((x) => x.id === id);
before(async () => {
ids.hz = await mk("hetzner", "Prod");
ids.fn = await mk("fragnet", "Games");
ids.dh = await mk("dockhand", "Docker");
ids.sonarr = await mk("sonarr", "TV");
const now = Date.now();
await hz.storeRead(ids.hz, { servers: [{ id: 1, name: "web", status: "running", outgoingTraffic: null }, { id: 2, name: "db", status: "off", outgoingTraffic: null }], volumes: null, images: null, primaryIps: null, prices: null, loadBalancers: null, floatingIps: null, certificates: null } as never, new Date());
await fn.storeRead(ids.fn, { servers: [{ id: "aaaa1111", name: "survival", state: "running", suspended: false }, { id: "bbbb2222", name: "rust", state: "offline", suspended: false }, { id: "cccc3333", name: "old", state: "offline", suspended: true }] } as never, new Date());
await db.insert(dockhandVulnerabilities).values([
{ integrationId: ids.dh, environmentId: 1, environmentName: "Jackie", critical: 3, high: 7, medium: 1, low: 0, imagesScanned: 5, totalImages: 5, findings: "[]", updatedAt: new Date().toISOString() },
{ integrationId: ids.dh, environmentId: 2, environmentName: "Anna", critical: 2, high: 0, medium: 0, low: 0, imagesScanned: 2, totalImages: 2, findings: "[]", updatedAt: new Date().toISOString() },
{ integrationId: ids.dh, environmentId: 3, environmentName: "Unknown", critical: null, high: null, medium: null, low: null, imagesScanned: null, totalImages: null, findings: "[]", updatedAt: new Date().toISOString() },
]);
const item = (id: number, extra: Record<string, unknown> = {}) => ({ id, title: `t${id}`, status: "downloading", tracked: "ok", state: "downloading", messages: [], size: 1, sizeLeft: 1, added: iso(now - 30 * MIN), etaAt: null, protocol: null, client: null, indexer: null, ...extra });
await arr.storeRead(ids.sonarr, { version: "4", health: [], queue: { total: 3, items: [item(1), item(2, { tracked: "error", added: iso(now - 5 * HOUR) }), item(3, { state: "importBlocked", added: iso(now - 4 * HOUR) })] }, disks: [], missing: 0, indexers: null }, new Date());
await fresh();
});
after(async () => {
for (const id of Object.values(ids)) await db.delete(integrations).where(eq(integrations.id, id));
await hz.forgetHetzner(ids.hz);
await fn.forgetFragnet(ids.fn);
await arr.forgetArr(ids.sonarr);
});
test("Hetzner servers running (a stop is a warning), game servers running (a stop is a note, a suspended one is not counted), critical vulnerabilities (an unknown host adds nothing), stuck downloads", async () => {
const x = await wall.buildWallExtras(connected(), new Set(), []);
assert.deepEqual({ ...tileOf(x, "hetzner") }, { id: "hetzner", label: "Hetzner servers running", figure: "1/2", state: "warn" });
assert.deepEqual({ ...tileOf(x, "game-servers") }, { id: "game-servers", label: "game servers running", figure: "1/2", state: "info" });
assert.deepEqual({ ...tileOf(x, "vulnerabilities") }, { id: "vulnerabilities", label: "critical vulnerabilities", figure: "5", state: "bad" });
assert.deepEqual({ ...tileOf(x, "stuck-downloads") }, { id: "stuck-downloads", label: "stuck downloads", figure: "2", state: "warn" });
assert.deepEqual(x.groups.map((g) => g.id), ["infrastructure", "apps"]);
});
test("a reading that is old is not shown as how things are; nothing connected, no tile", async () => {
await stale();
const x = await wall.buildWallExtras(connected(), new Set(), []);
assert.deepEqual(x.tiles, []);
await fresh();
assert.deepEqual((await wall.buildWallExtras([], new Set(), [])).tiles, []);
assert.deepEqual((await wall.buildWallExtras([{ id: ids.hz, type: "hetzner" }], new Set(), [])).tiles.map((t) => t.id), ["hetzner"], "only what is connected");
});
test("the share link carries them at every level (numbers and group names only), and the signed-in wall gets them from its own route", async () => {
for (const detail of ["summary", "names"] as const) {
const w = await buildPublicWall(detail);
assert.deepEqual(w.tiles.map((t) => t.id).sort(), ["game-servers", "hetzner", "stuck-downloads", "vulnerabilities"], detail);
assert.deepEqual(w.groups.map((g) => g.label), ["Infrastructure", "Apps"], detail);
assert.doesNotMatch(JSON.stringify([w.tiles, w.groups]), /web|db|survival|Jackie/, "no names of anything");
}
const a = express();
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "viewer" } };
next();
});
a.use("/api/wall", wallRouter);
const server = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
try {
const body = (await (await fetch(`http://127.0.0.1:${(server.address() as AddressInfo).port}/api/wall/extras`)).json()) as { tiles: { id: string }[]; groups: { id: string }[] };
assert.deepEqual(body.tiles.map((t) => t.id).sort(), ["game-servers", "hetzner", "stuck-downloads", "vulnerabilities"]);
assert.deepEqual(body.groups.map((g) => g.id), ["infrastructure", "apps"]);
} finally {
server.close();
}
});
});
after(() => {
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
+40
View File
@@ -152,6 +152,22 @@ describe("the fake Dockhand", () => {
assert.deepEqual(dv.dockerVulnProblems(id, [{ ...jackie, critical: 0, high: 50 }]), []);
});
test("high findings are raised only when a number is set, from that many, and then beside the critical ones", async () => {
const jackie = (await rows()).find((r) => r.environmentName === "Jackie")!;
const many = { ...jackie, critical: 0, high: 12 };
assert.deepEqual(dv.dockerVulnProblems(id, [many], 0), [], "off by default");
assert.deepEqual(dv.dockerVulnProblems(id, [many]), [], "and with no number given");
assert.deepEqual(dv.dockerVulnProblems(id, [many], 13), [], "below the number");
const p = dv.dockerVulnProblems(id, [many], 10);
assert.deepEqual(p.map((x) => [x.key, x.severity]), [[`docker-vuln-high:${id}:1`, "warning"]]);
assert.equal(p[0].message, "Jackie: 12 high vulnerabilities in the scanned images (you asked to hear of 10 or more)");
assert.equal(dv.dockerVulnProblems(id, [{ ...many, high: 12 }], 12).length, 1, "at the number counts");
assert.deepEqual(dv.dockerVulnProblems(id, [{ ...many, high: 1 }], 1).map((x) => x.message), ["Jackie: 1 high vulnerability in the scanned images (you asked to hear of 1 or more)"]);
assert.deepEqual(dv.dockerVulnProblems(id, [{ ...many, high: null }], 1), [], "an unknown count says nothing");
const both = dv.dockerVulnProblems(id, [{ ...jackie, critical: 2, high: 40 }], 10);
assert.deepEqual(both.map((x) => x.key), [`docker-vuln-high:${id}:1`, `docker-vuln:${id}:1`]);
});
describe("the health pass, the Alerts page, the Docker page and /metrics", () => {
let app: Server;
let base = "";
@@ -224,6 +240,30 @@ describe("the fake Dockhand", () => {
assert.equal((await getSettings()).healthChecks.vulnerabilityAlerts, true);
});
test("high findings: off by default (an old setting without the number is read as 0); with a number the host's high count is an alert of its own, under maintenance like the other", async () => {
const checks = (await getSettings()).healthChecks;
assert.equal(checks.vulnerabilityHighThreshold, 0);
const { vulnerabilityHighThreshold: _dropped, ...older } = checks;
await updateSettings({ healthChecks: older as never });
assert.equal((await getSettings()).healthChecks.vulnerabilityHighThreshold, 0);
const highIds = async () => {
invalidateAlertsCache();
return (await getAlerts(true)).alerts.map((x) => x.id).filter((i) => i.includes("docker-vuln-high:"));
};
assert.deepEqual(await highIds(), []);
await updateSettings({ healthChecks: { ...checks, vulnerabilityHighThreshold: 5 } });
assert.ok((await highIds()).includes(`services:docker-vuln-high:${id}:1`), "Jackie has 7 high findings");
const a = (await getAlerts(true)).alerts.find((x) => x.id === `services:docker-vuln-high:${id}:1`)!;
assert.deepEqual([a.severity, a.source, a.link, a.target?.type], ["warning", "Docker", "/docker", "integration"]);
assert.match(a.message, /Jackie: 7 high vulnerabilities/);
assert.equal(subjectOfConditionKey(`docker-vuln-high:${id}:1`), `integration:${id}`);
await updateSettings({ healthChecks: { ...checks, vulnerabilityHighThreshold: 8 } });
assert.ok(!(await highIds()).includes(`services:docker-vuln-high:${id}:1`), "7 is under 8");
await updateSettings({ healthChecks: { ...checks, vulnerabilityAlerts: false, vulnerabilityHighThreshold: 5 } });
assert.deepEqual(await highIds(), [], "the switch for the vulnerability alerts turns this off as well");
await updateSettings({ healthChecks: checks });
});
test("anyone signed in sees each host's counts, the critical findings and how many images are scanned", async () => {
const body = (await (await fetch(`${base}/vulnerabilities`, { headers: { connection: "close" } })).json()) as { alerts: boolean; fresh: boolean; environments: { environmentName: string; critical: number; high: number; imagesScanned: number; totalImages: number; findings: { cve: string }[] }[] };
assert.deepEqual([body.alerts, body.fresh], [true, true]);
+46
View File
@@ -67,6 +67,23 @@ export interface ShareLink {
}
/** What a share link may show: numbers, and with "names" also the names of what is wrong. Built on the server. */
/** An extra tile on the wall display (Hetzner servers running, critical vulnerabilities …). */
export interface WallTile {
id: string;
label: string;
figure: string;
state: "ok" | "warn" | "bad" | "info";
}
/** A summary of one group of integrations (the menu's Infrastructure, Apps …) for the wall display: how many are connected, failing, and how many alerts they have. */
export interface WallGroup {
id: string;
label: string;
integrations: number;
failing: number;
alerts: { critical: number; warning: number };
}
export interface PublicWall {
detail: ShareLinkDetail;
level: "ok" | "warning" | "critical" | "unsure";
@@ -75,6 +92,8 @@ export interface PublicWall {
alerts: { critical: number; warning: number };
maintenance: number;
integrationsFailing: number;
tiles?: WallTile[];
groups?: WallGroup[];
problems?: { id: string; severity: "critical" | "warning"; message: string }[];
maintenanceWindows?: { id: number; name: string; endsAt: string }[];
integrations?: { id: number; name: string; type: string; status: "ok" | "failing" | "unknown" }[];
@@ -540,6 +559,8 @@ export interface HealthCheckSettings {
/** A container that keeps restarting, is unhealthy, stopped unexpectedly or never started, and a Docker host that cannot be reached, is an alert. */
containerAlerts: boolean;
vulnerabilityAlerts: boolean;
/** A Docker host with at least this many high findings is a warning too; 0 = off. */
vulnerabilityHighThreshold: number;
/** Containers to leave alone: a name, or "host/name"; * stands for anything. */
containerIgnore: string[];
}
@@ -2150,11 +2171,26 @@ export interface Runbook {
/** A runbook's title and what it is for, without its text. */
export type RunbookSummary = Pick<Runbook, "id" | "targetType" | "targetId" | "category" | "title">;
/** What to do first about a kind of alert ("disk", "hetzner-cert" …), whichever server or integration it is about. */
export interface AlertKindRunbook {
kind: string;
title: string;
body: string;
/** A page to open; null when there is none. */
url: string | null;
updatedBy: string | null;
updatedAt: string;
}
export type AlertSeverity = "critical" | "warning" | "info";
export type AlertCategory = "offline" | "disk" | "services" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
export interface AlertItem {
id: string;
/** What kind of alert it is: the first part of its key ("disk", "hetzner-cert"). A runbook can be written for a kind. */
kind: string;
/** The kind of source it is about: an integration's type, "server" … */
type: string;
severity: AlertSeverity;
category: AlertCategory;
source: string;
@@ -2804,6 +2840,8 @@ export interface DockhandVulnerabilityEnvironment {
export interface DockhandVulnerabilitiesResponse {
/** Whether critical findings are alerted (Settings → Notifications). */
alerts: boolean;
/** A host with at least this many high findings is alerted too; 0 = never. */
highThreshold: number;
status: { at: string; ok: boolean; error: string | null } | null;
fresh: boolean;
environments: DockhandVulnerabilityEnvironment[];
@@ -3380,6 +3418,11 @@ export const api = {
update: (id: number, data: { title: string; body: string; category: AlertCategory | null }) =>
request<{ runbook: Runbook }>(`/api/runbooks/${id}`, { method: "PUT", body: JSON.stringify(data) }),
remove: (id: number) => request<void>(`/api/runbooks/${id}`, { method: "DELETE" }),
kinds: {
list: () => request<{ kinds: AlertKindRunbook[] }>("/api/runbooks/kinds"),
save: (kind: string, data: { title: string; body: string; url: string | null }) => request<{ runbook: AlertKindRunbook }>(`/api/runbooks/kinds/${encodeURIComponent(kind)}`, { method: "PUT", body: JSON.stringify(data) }),
remove: (kind: string) => request<void>(`/api/runbooks/kinds/${encodeURIComponent(kind)}`, { method: "DELETE" }),
},
},
shareLinks: {
list: () => request<{ links: ShareLink[] }>("/api/share-links"),
@@ -3401,6 +3444,9 @@ export const api = {
},
},
// Opened by someone who isn't signed in: no cookies, and the status is kept so a link that has ended can say so.
wall: {
extras: () => request<{ tiles: WallTile[]; groups: WallGroup[] }>("/api/wall/extras"),
},
publicWall: {
get: async (token: string): Promise<PublicWall> => {
const res = await fetch(`/api/public/wall/${encodeURIComponent(token)}`, { credentials: "omit", cache: "no-store" });
+180
View File
@@ -0,0 +1,180 @@
import { useState } from "react";
import { IconBook, IconExternalLink } from "@tabler/icons-react";
import { api, type AlertKindRunbook } from "../api/client";
import { ALERT_KIND_RUNBOOKS_CHANGED, useAlertKindRunbooks } from "../hooks/useAlertKindRunbooks";
import { readableError } from "../utils/errors";
import { removeWithUndo } from "../utils/undoable";
import { toast } from "../utils/toast";
import DetailDrawer from "./DetailDrawer";
import RunbookText from "./RunbookText";
import TimeAgo from "./TimeAgo";
/** How a kind of alert reads in a sentence; any kind not listed is its own name with the dashes taken out. */
const KIND_WORDS: Record<string, string> = {
offline: "a server that stopped reporting",
disk: "a disk filling up",
"failed-units": "failed services",
smart: "a disk reporting trouble",
zfs: "a ZFS pool in trouble",
"docker-vuln": "critical vulnerabilities in Docker images",
"docker-vuln-high": "many high vulnerabilities in Docker images",
"container-restarting": "a container that keeps restarting",
"container-unhealthy": "an unhealthy container",
"container-stopped": "a container that stopped",
"hetzner-off": "a Hetzner server that is off",
"hetzner-traffic": "a Hetzner traffic quota nearly used",
"hetzner-forecast": "a Hetzner traffic quota the month will pass",
"hetzner-backup": "a Hetzner backup that is late",
"hetzner-lb": "an unhealthy Hetzner load balancer",
"hetzner-cert": "a Hetzner certificate about to end",
"hetzner-fip": "a Hetzner floating IP pointing at nothing",
"fragnet-backup": "a game server backup that is late",
"fragnet-disk": "a game server disk filling up",
"fragnet-memory": "a game server's memory running out",
"sonarr-health": "Sonarr reporting a problem",
"sonarr-queue": "stuck downloads in Sonarr",
"radarr-health": "Radarr reporting a problem",
"radarr-queue": "stuck downloads in Radarr",
"prowlarr-health": "Prowlarr reporting a problem",
"speedtest-slow": "a slow internet line",
"speedtest-failing": "failing speed tests",
};
export const kindWords = (kind: string): string => KIND_WORDS[kind] ?? `${kind.replace(/[-_]+/g, " ")} alerts`;
/**
* A small "What to do" link beside an alert: what is written down for this KIND of alert, whichever server or integration it is about (the runbooks of one thing
* are the other link). Shown where there is something to read and, for those who can write them, a quiet "Add what to do" where there is not. A link, if one was
* given, opens in a new tab from the panel.
*/
export default function AlertKindRunbookEntry({ kind, canEdit, offerAdd = true }: { kind: string; canEdit: boolean; /** Show "Add what to do" for those who can, when there is none. */ offerAdd?: boolean }) {
const [open, setOpen] = useState(false);
const all = useAlertKindRunbooks();
const here = all?.find((r) => r.kind === kind) ?? null;
if (all === null || (here === null && !(canEdit && offerAdd))) return null;
return (
<>
<button
type="button"
className={`btn btn-link btn-sm p-0 d-inline-flex align-items-center gap-1 align-baseline ${here ? "" : "text-secondary"}`}
title={here ? here.title : `Write down what to do about ${kindWords(kind)}`}
onClick={() => setOpen(true)}
>
<IconBook size={14} aria-hidden="true" />
{here ? "What to do" : "Add what to do"}
</button>
{open && (
<DetailDrawer title="What to do" subtitle={kindWords(kind)} onClose={() => setOpen(false)}>
<Panel kind={kind} existing={here} canEdit={canEdit} onDone={() => setOpen(false)} />
</DetailDrawer>
)}
</>
);
}
function Panel({ kind, existing, canEdit, onDone }: { kind: string; existing: AlertKindRunbook | null; canEdit: boolean; onDone: () => void }) {
const [editing, setEditing] = useState(existing === null && canEdit);
const [title, setTitle] = useState(existing?.title ?? `If it is ${kindWords(kind)}`);
const [body, setBody] = useState(existing?.body ?? "");
const [url, setUrl] = useState(existing?.url ?? "");
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const changed = () => window.dispatchEvent(new Event(ALERT_KIND_RUNBOOKS_CHANGED));
async function save(e: React.FormEvent) {
e.preventDefault();
setBusy(true);
setError(null);
try {
await api.runbooks.kinds.save(kind, { title: title.trim(), body, url: url.trim() === "" ? null : url.trim() });
toast.success("Saved.");
changed();
onDone();
} catch (err) {
setError(readableError(err));
} finally {
setBusy(false);
}
}
async function remove() {
if (!existing) return;
onDone();
await removeWithUndo({
remove: () => api.runbooks.kinds.remove(kind),
restore: () => api.runbooks.kinds.save(kind, { title: existing.title, body: existing.body, url: existing.url }),
message: `Removed "${existing.title}".`,
reload: changed,
});
}
if (!editing && existing) {
return (
<div>
<div className="fw-bold mb-2">{existing.title}</div>
{existing.url && (
<p>
<a className="btn btn-sm btn-outline-primary" href={existing.url} target="_blank" rel="noopener noreferrer">
<IconExternalLink size={14} className="me-1" aria-hidden="true" />
Open the page
</a>{" "}
<span className="text-secondary small text-break">{existing.url}</span>
</p>
)}
{existing.body.trim() !== "" && <RunbookText text={existing.body} />}
<div className="d-flex flex-wrap align-items-center gap-2 mt-3">
<span className="text-secondary small">
For every alert of this kind. {existing.updatedBy ? `${existing.updatedBy}, ` : ""}
<TimeAgo value={existing.updatedAt} />
</span>
{canEdit && (
<span className="ms-auto btn-list flex-nowrap">
<button type="button" className="btn btn-sm btn-outline-secondary" onClick={() => setEditing(true)}>
Edit
</button>
<button type="button" className="btn btn-sm btn-outline-danger" onClick={() => void remove()}>
Remove
</button>
</span>
)}
</div>
</div>
);
}
return (
<form onSubmit={save}>
<p className="text-secondary small">Written once, shown beside every alert of this kind — {kindWords(kind)} — on whatever it happens to. What is specific to one server or integration goes in its own runbook.</p>
<div className="mb-3">
<label className="form-label" htmlFor="akr-title">
Title
</label>
<input id="akr-title" className="form-control" required maxLength={120} value={title} onChange={(e) => setTitle(e.target.value)} />
</div>
<div className="mb-3">
<label className="form-label" htmlFor="akr-url">
A page to open (optional)
</label>
<input id="akr-url" className="form-control" type="url" maxLength={500} placeholder="https://wiki.example.lan/certificates" value={url} onChange={(e) => setUrl(e.target.value)} />
</div>
<div className="mb-2">
<label className="form-label" htmlFor="akr-body">
What to do first
</label>
<textarea id="akr-body" className="form-control" rows={9} maxLength={20000} value={body} onChange={(e) => setBody(e.target.value)} />
<div className="form-hint">
Plain text. A blank line starts a new paragraph; <code>- </code> makes a list; <code>```</code> around lines makes a block of commands; links are made clickable.
</div>
</div>
{error && <div className="text-danger small mb-2">{error}</div>}
<div className="btn-list">
<button type="submit" className="btn btn-primary" disabled={busy || title.trim() === "" || (body.trim() === "" && url.trim() === "")}>
{busy ? "Saving…" : "Save"}
</button>
<button type="button" className="btn" onClick={() => (existing ? setEditing(false) : onDone())}>
Cancel
</button>
</div>
</form>
);
}
+26
View File
@@ -1,4 +1,5 @@
import { useEffect, useState, type ReactNode } from "react";
import type { WallGroup, WallTile } from "../api/client";
import { formatDate, formatTime } from "../utils/date";
import { formatRemaining } from "../utils/duration";
@@ -48,6 +49,9 @@ export interface WallData {
problems: { id: string; severity: "critical" | "warning"; message: string }[] | null;
windows: { id: number; name: string; endsAt: string }[] | null;
integrations: { id: number; name: string; type: string; status: "ok" | "failing" | "unknown" }[] | null;
/** What else is worth a glance, and a summary of each group of integrations; absent on a board that has none. */
tiles?: WallTile[];
groups?: WallGroup[];
environmentName: string | null;
updatedAt: Date | null;
}
@@ -141,8 +145,30 @@ export default function WallView({ data, idle, footerActions }: { data: WallData
<div className="wall-tile-figure">{data.integrationsFailing === null ? "…" : data.integrationsFailing}</div>
<div className="wall-tile-label">integrations failing</div>
</div>
{(data.tiles ?? []).map((t) => (
<div key={t.id} className={`wall-tile ${t.state === "bad" ? "wall-tile-bad" : t.state === "warn" ? "wall-tile-warn" : t.state === "info" ? "wall-tile-info" : ""}`}>
<div className="wall-tile-figure">{t.figure}</div>
<div className="wall-tile-label">{t.label}</div>
</div>
))}
</section>
{(data.groups ?? []).length > 0 && (
<section className="wall-groups" aria-label="By area">
{(data.groups ?? []).map((g) => {
const alerts = g.alerts.critical + g.alerts.warning;
const level = g.failing > 0 || g.alerts.critical > 0 ? "bad" : alerts > 0 ? "warn" : "ok";
const parts = [g.failing > 0 ? `${g.failing} of ${g.integrations} failing` : null, alerts > 0 ? `${alerts} ${alerts === 1 ? "alert" : "alerts"}` : null].filter(Boolean);
return (
<div key={g.id} className={`wall-group wall-group-${level}`}>
<span aria-hidden="true">{level === "bad" ? "✕" : level === "warn" ? "▲" : "✓"}</span> <strong>{g.label}</strong>{" "}
<span className="wall-dim">{parts.length > 0 ? parts.join(" · ") : `${g.integrations} connected`}</span>
</div>
);
})}
</section>
)}
{showsDetail ? (
<section className="wall-columns">
<div>
+42
View File
@@ -0,0 +1,42 @@
import { useEffect, useSyncExternalStore } from "react";
import { api, type AlertKindRunbook } from "../api/client";
// What is written down for each kind of alert, shared by every alert on the page and fetched once — and again whenever one is added, changed or
// removed (ALERT_KIND_RUNBOOKS_CHANGED). The texts come with the list: there are few of them, and the drawer should open at once.
export const ALERT_KIND_RUNBOOKS_CHANGED = "alert-kind-runbooks-changed";
let items: AlertKindRunbook[] | null = null;
let loading = false;
const listeners = new Set<() => void>();
function load(force = false): void {
if (loading || (items !== null && !force)) return;
loading = true;
api.runbooks.kinds
.list()
.then((res) => {
items = res.kinds;
listeners.forEach((l) => l());
})
.catch(() => {
// Only a convenience on top of the page: without it the alerts simply show no runbook link.
})
.finally(() => {
loading = false;
});
}
if (typeof window !== "undefined") window.addEventListener(ALERT_KIND_RUNBOOKS_CHANGED, () => load(true));
function subscribe(listener: () => void): () => void {
listeners.add(listener);
return () => {
listeners.delete(listener);
};
}
/** What is written down for each kind of alert; null until loaded. */
export function useAlertKindRunbooks(): AlertKindRunbook[] | null {
useEffect(() => load(), []);
return useSyncExternalStore(subscribe, () => items);
}
+17 -3
View File
@@ -15,6 +15,7 @@ import TableToolbar from "../components/TableToolbar";
import SilenceButton from "../components/SilenceButton";
import RunPlaybookButton from "../components/RunPlaybookButton";
import RunbookEntry from "../components/RunbookEntry";
import AlertKindRunbookEntry from "../components/AlertKindRunbook";
import AckDrawer, { ackWords } from "../components/AckDrawer";
import CloseDrawer, { CloseBulkDrawer } from "../components/CloseDrawer";
import SnoozeDrawer, { SnoozeBulkDrawer, snoozeWords } from "../components/SnoozeDrawer";
@@ -202,6 +203,18 @@ function AlertsNow({ user, tabs }: { user: CurrentUser; tabs: React.ReactNode })
const withTicket = (report?.alerts ?? []).filter((a) => ids.has(a.id) && a.ticket?.number && !a.ticket.closedAt);
return { closable: withTicket.filter((a) => a.ticket?.canClose).map((a) => a.ticket!.number as string), other: withTicket.filter((a) => !a.ticket?.canClose).length };
}, [pickedIds, report]);
// "Add what to do" is offered once for each kind of alert on the page (on the first of them), not beside every one; what has been written shows beside all of them.
const firstOfKind = useMemo(() => {
const seen = new Set<string>();
const first = new Set<string>();
for (const a of pageItems ?? []) {
if (!seen.has(a.kind)) {
seen.add(a.kind);
first.add(a.id);
}
}
return first;
}, [pageItems]);
const shownIds = useMemo(() => (sorted ?? []).filter((a) => !a.silenced).map((a) => a.id), [sorted]);
const allShownPicked = shownIds.length > 0 && shownIds.every((id) => picked.has(id));
function togglePicked(id: string) {
@@ -536,9 +549,10 @@ function AlertsNow({ user, tabs }: { user: CurrentUser; tabs: React.ReactNode })
<Link to={incidentPath(a.incidentId)}>Timeline</Link>
</div>
)}
{a.target && (
<div>
<RunbookEntry targetType={a.target.type} targetId={a.target.id} targetName={a.target.name} canEdit={canSilence} forCategory={a.category} />
{(a.target || a.kind) && (
<div className="d-flex flex-wrap gap-3">
{a.target && <RunbookEntry targetType={a.target.type} targetId={a.target.id} targetName={a.target.name} canEdit={canSilence} forCategory={a.category} />}
{a.kind && <AlertKindRunbookEntry kind={a.kind} canEdit={canSilence} offerAdd={firstOfKind.has(a.id)} />}
</div>
)}
{a.affects && a.affects.length > 0 && (
+8 -1
View File
@@ -109,7 +109,14 @@ export default function DockerVulnerabilities({ integrationId }: { integrationId
From the scans Dockhand has already run on its images (it scans when it is told to — nothing is scanned from here). An image that was never scanned is not known to be clean.{" "}
{data.alerts ? (
<>
<strong>Critical</strong> findings are an alert — see <Link to="/alerts">Alerts</Link>; high, medium and low are only shown here.
<strong>Critical</strong> findings are an alert — see <Link to="/alerts">Alerts</Link>;{" "}
{data.highThreshold > 0 ? (
<>
so is a host with <strong>{data.highThreshold} or more high</strong> ones (<Link to="/settings/notifications">Settings → Notifications</Link>); medium and low are only shown here.
</>
) : (
<>high, medium and low are only shown here (high ones can be alerted from a number you choose under Settings → Notifications).</>
)}
</>
) : (
<>
+2
View File
@@ -88,6 +88,8 @@ export default function SharedWall() {
alerts: wall ? wall.alerts : null,
maintenance: wall?.maintenance ?? 0,
integrationsFailing: wall ? wall.integrationsFailing : null,
tiles: wall?.tiles,
groups: wall?.groups,
problems: wall?.problems ?? null,
windows: wall?.maintenanceWindows ?? null,
integrations: (wall?.integrations as WallData["integrations"]) ?? null,
+7 -2
View File
@@ -1,6 +1,6 @@
import { useEffect, useState } from "react";
import { Link, useNavigate } from "react-router-dom";
import { api, type IntegrationSummary, type MaintenanceWindow, type ServerSummary } from "../api/client";
import { api, type IntegrationSummary, type MaintenanceWindow, type ServerSummary, type WallGroup, type WallTile } from "../api/client";
import { alertBadge, useAlertsReport } from "../hooks/useAlertsReport";
import { useEnvironment } from "../utils/environment";
import WallView, { REFRESH_MS, wallHeadline, type WallData } from "../components/WallView";
@@ -16,6 +16,7 @@ export default function Wall() {
const [servers, setServers] = useState<ServerSummary | null>(null);
const [integrations, setIntegrations] = useState<IntegrationSummary[] | null>(null);
const [windows, setWindows] = useState<MaintenanceWindow[]>([]);
const [extras, setExtras] = useState<{ tiles: WallTile[]; groups: WallGroup[] } | null>(null);
const [updatedAt, setUpdatedAt] = useState<Date | null>(null);
const [idle, setIdle] = useState(false);
@@ -23,11 +24,13 @@ export default function Wall() {
let alive = true;
const load = () => {
reload();
Promise.allSettled([api.servers.summary(), api.integrations.list(), api.maintenance.list()]).then(([s, i, m]) => {
Promise.allSettled([api.servers.summary(), api.integrations.list(), api.maintenance.list(), api.wall.extras()]).then(([s, i, m, x]) => {
if (!alive) return;
if (s.status === "fulfilled") setServers(s.value);
if (i.status === "fulfilled") setIntegrations(i.value.integrations);
if (m.status === "fulfilled") setWindows(m.value.windows);
// The extras are only an addition: a board that cannot get them shows the rest.
if (x.status === "fulfilled") setExtras(x.value);
setUpdatedAt(new Date());
});
};
@@ -72,6 +75,8 @@ export default function Wall() {
alerts: report ? { critical: report.counts.critical, warning: report.counts.warning } : null,
maintenance: windows.length,
integrationsFailing: integrations ? failing.length : null,
tiles: extras?.tiles,
groups: extras?.groups,
problems: active.map((a) => ({ id: a.id, severity: a.severity as "critical" | "warning", message: a.message })),
windows: windows.map((w) => ({ id: w.id, name: w.targetName, endsAt: w.endsAt })),
integrations: (integrations ?? [])
@@ -84,7 +84,7 @@ const DIGEST_DAYS: { value: number; label: string }[] = [
{ value: 6, label: "Saturday" },
{ value: 0, label: "Sunday" },
];
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true, containerAlerts: true, vulnerabilityAlerts: true, containerIgnore: [] };
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true, containerAlerts: true, vulnerabilityAlerts: true, vulnerabilityHighThreshold: 0, containerIgnore: [] };
const DEFAULT_MONTHLY: MonthlyReportSettings = { enabled: false, quarterly: false, annual: false, day: 1, time: "08:00" };
type TestResult = { ok: boolean; message: string } | null;
@@ -904,11 +904,22 @@ export default function NotificationSettings() {
<span className="form-check-label">
Warn about critical vulnerabilities in Docker images
<span className="form-hint d-block">
From the scans Dockhand has already run (nothing is scanned from here): a warning for each Docker host whose scanned images have one or more critical findings. High, medium and low findings are shown on the Docker
page but never alerted. Dockhand can send its own vulnerability notifications too; with both switched on you hear twice.
From the scans Dockhand has already run (nothing is scanned from here): a warning for each Docker host whose scanned images have one or more critical findings. Medium and low findings are shown on the Docker
page but never alerted (high ones only if you set a number below). Dockhand can send its own vulnerability notifications too; with both switched on you hear twice.
</span>
</span>
</label>
<div className="mt-3">
<NumberField
label="Also warn when a Docker host has this many high vulnerabilities"
value={healthChecks.vulnerabilityHighThreshold}
min={0}
max={10000}
unit="or more (0 = never)"
hint="Off by default: scanners find high-severity issues in almost every image, so a low number is noise. Counted over the scanned images of each host, as Dockhand reports them; the critical ones are alerted either way while the box above is ticked."
onChange={(v) => setHealthChecks((h) => ({ ...h, vulnerabilityHighThreshold: v }))}
/>
</div>
</div>
</div>
</div>
+33
View File
@@ -837,6 +837,39 @@ main:focus {
box-shadow: inset 0 0 0 3px #4dabf7;
}
/* One line for each group of integrations the menu has: a symbol, its name and how it is. */
.wall-groups {
display: flex;
flex-wrap: wrap;
gap: clamp(0.5rem, 1.2vw, 1rem);
}
.wall-group {
padding: 0.5em 0.9em;
background: #121c2f;
border-radius: 999px;
font-size: clamp(0.9rem, 1.5vw, 1.3rem);
box-shadow: inset 0 0 0 2px transparent;
}
.wall-group-ok {
color: #8ce99a;
}
.wall-group-warn {
color: #ffc247;
box-shadow: inset 0 0 0 2px #ffc247;
}
.wall-group-bad {
color: #ff8787;
box-shadow: inset 0 0 0 2px #ff6b6b;
}
.wall-group .wall-dim {
color: #adb5bd;
}
.wall-columns {
display: grid;
grid-template-columns: 3fr 2fr;