diff --git a/DATABASE.md b/DATABASE.md index 5db5da4..871c4bf 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -1131,7 +1131,7 @@ that were never changed aren't stored; the app fills in defaults when reading. | `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. | | `monthlyReport` | The monthly report: `enabled` (false), `day` (1–28, default 1) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | | `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. | -| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be), `newPortAlerts` (true — warn when a server starts listening on a TCP port that is not in its baseline and not noted), `flapRaises` (3; 0 = off — raises of one alert within an hour that make it "flapping" so its messages are held; see NOTIFICATIONS.md), `deviceOfflineMinutes` (10 — how long a UniFi device may be offline before it is raised). A server can override each of these (see `servers.thresholds`). | +| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be), `newPortAlerts` (true — warn when a server starts listening on a TCP port that is not in its baseline and not noted), `flapRaises` (3; 0 = off — raises of one alert within an hour that make it "flapping" so its messages are held; see NOTIFICATIONS.md), `deviceOfflineMinutes` (10 — how long a UniFi device may be offline before it is raised), `osSupportWarnDays` (90; 0 = off), `baselineAlerts` (true — warn when CPU or memory over the last hour is far above what the server's own history says is normal for this hour of the week; see docs/MONITORING.md). The baselines themselves are not stored: they are worked out from `server_metrics_rollup` (hourly rows, `grain = 'h'`) when needed. A server can override each of these (see `servers.thresholds`). | | `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). | | `display` | `dateFormat`, `timeFormat`, `pageSize`. | | `providerColors`, `integrationColors` | Badge colours, by provider / integration type. | diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index d8bd578..9128aae 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -217,6 +217,12 @@ A **patch plan** run sends one message when it ends — "Patch plan "Lab nights" that begins **listening on a new TCP port** is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when the port is noted or stops listening; for routing it counts as *Monitoring*, carries the tags of the server, and maintenance on the server holds it back. +## Unusual for the time + +A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see [MONITORING.md](docs/MONITORING.md#what-is-normal-for-the-time)) is announced like the other server problems — under *Server offline, disk +nearly full …* (`healthAlerts`), once, with a recovery message when it is back toward normal. For routing it counts as *Monitoring*, carries the server's tags, and maintenance on the server holds it back. It has its own switch under +Settings → Notifications (*Servers and disks*), on by default; with no four weeks of history for the hour there is nothing to announce. + ## Weekly digest | Notification | Fires when | diff --git a/README.md b/README.md index 5d78ad6..877c678 100644 --- a/README.md +++ b/README.md @@ -809,6 +809,13 @@ nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notif (where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on it yourself. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there. +**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's +averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey +**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least +30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The +warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by +maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*). + **A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too (on the Alerts page, the weekly digest and in `/metrics`). The first start after the upgrade builds the hours and days from the samples already kept, once. diff --git a/docs/MONITORING.md b/docs/MONITORING.md index cafc4a7..cab8b70 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -9,6 +9,7 @@ This page explains what Homelab Manager watches, how often, how a problem become - [Servers and the agent](#servers-and-the-agent) - [From a reading to an alert](#from-a-reading-to-an-alert) - [The built-in checks](#the-built-in-checks) +- [What is normal for the time](#what-is-normal-for-the-time) - [Your own alert rules](#your-own-alert-rules) - [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats) - [Integrations](#integrations) @@ -116,6 +117,7 @@ Every threshold here is a default; where it can be changed, the place is named. | **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable | | **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable | | **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable | +| **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Settings → Notifications | | **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card | on | Settings → Notifications | | **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications | | **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications | @@ -132,6 +134,27 @@ Every threshold here is a default; where it can be changed, the place is named. | **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) | | **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules | +## What is normal for the time + +A fixed limit cannot say that a server is busy *for a Tuesday at 03:00*. So for each server's **CPU load** and **memory use** the app keeps a baseline for every hour of the week (168 of them, in the **notification time zone**): the **median** of +that hour's averages over the previous weeks (up to ten; the hourly figures are kept four months), and a **robust spread** (the median absolute deviation, so one odd week does not stretch it). An hour needs **four weeks** behind it before +it has a baseline at all; until then nothing is said. The hour in progress and hours with fewer than two samples are not counted. + +It is **quiet by design**. A reading is a warning only when **all** of these hold, judged on the last hour of samples (at least three, the newest under 30 minutes old): + +| | CPU | Memory | +|---|---|---| +| The last hour's average is at least | 50 % | 70 % | +| …and above the usual by at least | 30 points | 15 points | +| …and above it by at least | four spreads (the spread is taken as at least 4 points) | four spreads (at least 2 points) | +| …and the lowest sample in that hour is still above the usual by | 15 points | 7.5 points | + +The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health +condition (`baseline:cpu:server:`, `baseline:mem:server:`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for +a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Settings → Notifications → What to send → Servers and disks. + +The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure. + ## Your own alert rules Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers: diff --git a/server/src/routes/servers.ts b/server/src/routes/servers.ts index 4cea373..817ac9f 100644 --- a/server/src/routes/servers.ts +++ b/server/src/routes/servers.ts @@ -13,6 +13,7 @@ import { describeDiagnostics, parseDiagnostics } from "../services/agentDiagnost import { checkThresholds, parseThresholds } from "../services/serverThresholds.js"; import { latestAgentVersions } from "../services/agentVersions.js"; import { getServerMetrics, MAX_RANGE_DAYS } from "../services/serverMetrics.js"; +import { baselineView } from "../services/baselines.js"; import { getServerActivity } from "../services/serverActivity.js"; import { asyncHandler } from "../utils/asyncHandler.js"; import { ID_PATTERNS } from "../utils/pathSegment.js"; @@ -277,6 +278,15 @@ serversRouter.get("/:id/metrics", asyncHandler(async (req, res) => { res.json({ days, series: await getServerMetrics(id, days) }); })); +// What is normal for each hour of the week, for the "usual" band on the trend charts (services/baselines.ts). +serversRouter.get("/:id/baseline", asyncHandler(async (req, res) => { + const id = Number(req.params.id); + if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" }); + const [server] = await db.select({ id: servers.id }).from(servers).where(eq(servers.id, id)).limit(1); + if (!server) return res.status(404).json({ error: "not_found" }); + res.json(await baselineView(id, (await getSettings()).notifications.timezone || "UTC")); +})); + // What has happened to a server: changes to it, the health check seeing it stop reporting or a disk fill, and its maintenance // windows. Operators and above — the audit log it draws on is theirs to read. serversRouter.get("/:id/activity", requireRole("operator"), asyncHandler(async (req, res) => { diff --git a/server/src/routes/settings.ts b/server/src/routes/settings.ts index f090ff0..dd4b969 100644 --- a/server/src/routes/settings.ts +++ b/server/src/routes/settings.ts @@ -144,7 +144,7 @@ const updateSchema = z.object({ .partial() .optional(), healthChecks: z - .object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730), newPortAlerts: z.boolean(), flapRaises: z.number().int().min(0).max(20), deviceOfflineMinutes: z.number().int().min(5).max(1440), osSupportWarnDays: z.number().int().min(0).max(365) }) + .object({ serverOfflineMinutes: z.number().int().min(15).max(10080), diskUsagePercent: z.number().int().min(50).max(99), domainWarnDays: z.number().int().min(1).max(365), diskForecastDays: z.number().int().min(0).max(90), backupStaleHours: z.number().int().min(0).max(2160), restoreTestDays: z.number().int().min(0).max(730), newPortAlerts: z.boolean(), flapRaises: z.number().int().min(0).max(20), deviceOfflineMinutes: z.number().int().min(5).max(1440), osSupportWarnDays: z.number().int().min(0).max(365), baselineAlerts: z.boolean() }) .partial() .optional(), quietHours: z diff --git a/server/src/services/alerts.ts b/server/src/services/alerts.ts index 19c58e0..bf0e31d 100644 --- a/server/src/services/alerts.ts +++ b/server/src/services/alerts.ts @@ -93,6 +93,8 @@ function originOfConditionKey(key: string): { type: string; id: number | null; c if (device) return { type: "unifi", id: Number(device[1]), category: "offline" }; let m = /^rule:\d+:server:(\d+)/.exec(key); if (m) return { type: "server", id: Number(m[1]), category: "monitoring" }; + m = /^baseline:(?:cpu|mem):server:(\d+)$/.exec(key); + if (m) return { type: "server", id: Number(m[1]), category: "monitoring" }; m = /^(?:offline|disk|failed-units|smart|zfs|new-port):server:(\d+)/.exec(key); if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : key.startsWith("failed-units") ? "services" : key.startsWith("new-port") ? "monitoring" : "disk" }; m = /^disk:proxmox:(\d+):/.exec(key); @@ -225,12 +227,13 @@ export async function collectAlerts(): Promise { type: origin.type, message: c.message, key: c.key, - link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : undefined, + link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") || c.key.startsWith("baseline:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : undefined, silenced: subject !== null && subjects.has(subject), target: targetOfSubject(subject), }); } // Integrations the check couldn't read this time. (A whole-integration entry covers its nodes, so skip those.) if (held.has("server:rules")) cannotCheck("Alert rules", "couldn't be judged — see the server log", "Server and storage health"); + if (held.has("server:baseline")) cannotCheck("What is usual for the time", "couldn't be judged — see the server log", "Server and storage health"); for (const h of held) { const m = /^(proxmox|synology|unifi|ups):(\d+)(?::(.+))?$/.exec(h); if (!m || (m[3] && held.has(`${m[1]}:${m[2]}`))) continue; diff --git a/server/src/services/baselines.ts b/server/src/services/baselines.ts new file mode 100644 index 0000000..2a142a0 --- /dev/null +++ b/server/src/services/baselines.ts @@ -0,0 +1,206 @@ +/** + * What is normal for a server at this time: "this server is unusually busy for a Tuesday at 03:00", judged from its own history rather than a fixed limit. + * + * For each server's CPU load and memory use, every hour of the week (Tuesday 03:00 is one, in the notification time zone) has a baseline: the median of that hour's + * averages over the previous weeks, and a robust spread (the median absolute deviation, so one odd week doesn't stretch it). It needs four weeks of that hour before it says + * anything at all, and uses up to ten (the hourly figures are kept four months). + * + * It is quiet by design. It only speaks about a figure that is *high*, and only when all of these hold: the last hour averaged well past a floor (CPU 50 %, memory 70 %), + * far above the usual (CPU 30 points, memory 15) and at least four spreads above it, and it never dropped back toward normal during that hour — so a spike, a backup that + * always runs a little late, or a server that is simply always busy is not an alert. What it finds is an ordinary health condition (`baseline::server:`), announced, + * recovered, held back, silenced by maintenance and routed like the others. The same baseline draws the "usual" band on a server's trend charts. + */ +import { and, eq, gte, inArray } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { serverMetrics, serverMetricsRollup, servers } from "../db/schema.js"; + +export type BaselineMetric = "cpu" | "mem"; +export const BASELINE_METRICS: BaselineMetric[] = ["cpu", "mem"]; + +export const MIN_WEEKS = 4; +export const MAX_WEEKS = 10; +/** An hour with fewer samples than this is a poor average (the agent was away for most of it) and is left out. */ +export const MIN_SAMPLES_PER_HOUR = 2; +/** The last hour is what is judged: at least this many samples, the newest no older than the second number. */ +export const RECENT_MINUTES = 60; +export const MIN_RECENT_SAMPLES = 3; +export const MAX_RECENT_AGE_MS = 30 * 60_000; +/** How many spreads above the usual it must be. */ +export const SPREADS = 4; + +export const RULES: Record = { + cpu: { label: "CPU load", phrase: "CPU load", floor: 50, minDelta: 30, minSpread: 4 }, + mem: { label: "Memory in use", phrase: "memory use", floor: 70, minDelta: 15, minSpread: 2 }, +}; + +const HOUR_MS = 3_600_000; +const DAY_MS = 86_400_000; +const DAYS = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"]; + +export interface SlotId { + /** 0 = Sunday … 6 = Saturday, in the time zone. */ + dow: number; + hour: number; +} + +export interface Slot extends SlotId { + median: number; + /** A robust spread, in the figure's own points. */ + spread: number; + /** How many weeks of this hour it stands on. */ + weeks: number; +} + +const formatters = new Map(); +function formatter(tz: string): Intl.DateTimeFormat { + let f = formatters.get(tz); + if (!f) { + try { + f = new Intl.DateTimeFormat("en-US", { timeZone: tz, weekday: "short", hour: "2-digit", hourCycle: "h23" }); + } catch { + f = formatter("UTC"); + } + formatters.set(tz, f); + } + return f; +} + +const SHORT = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]; + +/** The weekday and hour a moment falls in, in a time zone (UTC if the zone isn't one). */ +export function slotOf(ms: number, tz: string): SlotId { + const parts = formatter(tz).formatToParts(new Date(ms)); + const weekday = parts.find((p) => p.type === "weekday")?.value ?? "Sun"; + const hour = Number(parts.find((p) => p.type === "hour")?.value ?? 0) % 24; + return { dow: Math.max(0, SHORT.indexOf(weekday)), hour }; +} + +export const slotWords = (s: SlotId) => `a ${DAYS[s.dow]} at ${String(s.hour).padStart(2, "0")}:00`; + +export function median(xs: number[]): number { + if (xs.length === 0) return NaN; + const s = [...xs].sort((a, b) => a - b); + const m = Math.floor(s.length / 2); + return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; +} + +/** The median absolute deviation, scaled to be comparable with a standard deviation. */ +export function robustSpread(xs: number[]): number { + const m = median(xs); + return 1.4826 * median(xs.map((x) => Math.abs(x - m))); +} + +export interface HourRow { + /** Start of the hour, epoch ms. */ + t: number; + /** That hour's average. */ + v: number; + /** How many samples it averages. */ + n: number; +} + +/** + * Pure: the baseline of every hour of the week that has enough weeks behind it (168 entries, null where it hasn't). The hour in progress and anything older than ten weeks + * are left out, as are hours too thinly sampled to be an average. + */ +export function slotTable(rows: HourRow[], tz: string, nowMs: number): (Slot | null)[] { + const byslot: number[][] = Array.from({ length: 168 }, () => []); + const current = Math.floor(nowMs / HOUR_MS) * HOUR_MS; + const oldest = nowMs - MAX_WEEKS * 7 * DAY_MS; + for (const r of rows) { + if (r.t >= current || r.t < oldest || r.n < MIN_SAMPLES_PER_HOUR || !Number.isFinite(r.v)) continue; + const s = slotOf(r.t, tz); + byslot[s.dow * 24 + s.hour].push(r.v); + } + return byslot.map((vals, i) => (vals.length >= MIN_WEEKS ? { dow: Math.floor(i / 24), hour: i % 24, median: median(vals), spread: robustSpread(vals), weeks: vals.length } : null)); +} + +export interface Unusual { + metric: BaselineMetric; + /** The last hour's average. */ + value: number; + usual: number; + weeks: number; + slot: SlotId; +} + +/** + * Pure: whether the last hour was clearly unusual for this hour of the week. `recent` are the samples (epoch ms, percent) newest last or in any order. Null when it isn't, when + * there is not enough to say, and when there is no baseline for the hour yet. + */ +export function judgeUnusual(metric: BaselineMetric, recent: { t: number; v: number }[], baseline: Slot | null, nowMs: number): Unusual | null { + if (!baseline) return null; + const rule = RULES[metric]; + const window = recent.filter((s) => s.t >= nowMs - RECENT_MINUTES * 60_000 && s.t <= nowMs); + if (window.length < MIN_RECENT_SAMPLES) return null; + if (nowMs - Math.max(...window.map((s) => s.t)) > MAX_RECENT_AGE_MS) return null; + const avg = window.reduce((n, s) => n + s.v, 0) / window.length; + const lowest = Math.min(...window.map((s) => s.v)); + const spread = Math.max(baseline.spread, rule.minSpread); + const delta = avg - baseline.median; + if (avg < rule.floor || delta < rule.minDelta || delta / spread < SPREADS) return null; + // It never came back toward normal in that hour: a state, not a spike. + if (lowest < baseline.median + rule.minDelta / 2) return null; + return { metric, value: avg, usual: baseline.median, weeks: baseline.weeks, slot: { dow: baseline.dow, hour: baseline.hour } }; +} + +export interface BaselineFinding extends Unusual { + serverId: number; + serverName: string; + message: string; +} + +export const baselineKey = (f: Pick) => `baseline:${f.metric}:server:${f.serverId}`; + +export function describeUnusual(serverName: string, u: Unusual): string { + return `${serverName}: ${RULES[u.metric].phrase} is unusually high for ${slotWords(u.slot)} — it averaged ${Math.round(u.value)}% over the last hour, where it is normally about ${Math.round(u.usual)}% (from ${u.weeks} weeks of this hour)`; +} + +// ─── From the stored history ──────────────────────────────────────────────── + +/** A server's hourly averages of one or both figures for the weeks the baseline uses. */ +async function hourlyRows(serverId: number, metrics: BaselineMetric[], nowMs: number): Promise> { + const out: Record = { cpu: [], mem: [] }; + const rows = await db + .select({ metric: serverMetricsRollup.metric, at: serverMetricsRollup.at, sum: serverMetricsRollup.sum, n: serverMetricsRollup.n }) + .from(serverMetricsRollup) + .where(and(eq(serverMetricsRollup.serverId, serverId), eq(serverMetricsRollup.grain, "h"), inArray(serverMetricsRollup.metric, metrics), gte(serverMetricsRollup.at, new Date(nowMs - MAX_WEEKS * 7 * DAY_MS).toISOString()))); + for (const r of rows) if (r.n > 0 && (r.metric === "cpu" || r.metric === "mem")) out[r.metric].push({ t: Date.parse(r.at), v: r.sum / r.n, n: r.n }); + return out; +} + +/** What is unusual about each server right now, from its stored history. Judges only servers that reported lately. */ +export async function computeBaselineFindings(timeZone: string, nowMs: number = Date.now()): Promise { + const out: BaselineFinding[] = []; + for (const s of await db.select({ id: servers.id, name: servers.name, lastSeenAt: servers.lastSeenAt }).from(servers)) { + if (!s.lastSeenAt || nowMs - Date.parse(s.lastSeenAt) > 45 * 60_000) continue; + const recentRows = await db + .select({ metric: serverMetrics.metric, at: serverMetrics.recordedAt, value: serverMetrics.value }) + .from(serverMetrics) + .where(and(eq(serverMetrics.serverId, s.id), inArray(serverMetrics.metric, BASELINE_METRICS), gte(serverMetrics.recordedAt, new Date(nowMs - RECENT_MINUTES * 60_000).toISOString()))); + if (recentRows.length === 0) continue; + const hours = await hourlyRows(s.id, BASELINE_METRICS, nowMs); + const slot = slotOf(nowMs, timeZone); + for (const metric of BASELINE_METRICS) { + const recent = recentRows.filter((r) => r.metric === metric).map((r) => ({ t: Date.parse(r.at), v: r.value })); + const table = slotTable(hours[metric], timeZone, nowMs); + const unusual = judgeUnusual(metric, recent, table[slot.dow * 24 + slot.hour], nowMs); + if (unusual) out.push({ ...unusual, serverId: s.id, serverName: s.name, message: describeUnusual(s.name, unusual) }); + } + } + return out; +} + +export interface BaselineView { + timezone: string; + /** Now, as a weekday and hour in that zone. */ + now: SlotId; + /** Every hour of the week (Sunday 00:00 first), for each figure; null where there are not enough weeks yet. */ + slots: Record; +} + +/** The baselines a server's page draws its "usual" band from. */ +export async function baselineView(serverId: number, timeZone: string, nowMs: number = Date.now()): Promise { + const hours = await hourlyRows(serverId, BASELINE_METRICS, nowMs); + return { timezone: timeZone, now: slotOf(nowMs, timeZone), slots: { cpu: slotTable(hours.cpu, timeZone, nowMs), mem: slotTable(hours.mem, timeZone, nowMs) } }; +} diff --git a/server/src/services/healthMonitor.ts b/server/src/services/healthMonitor.ts index 51f6443..3057222 100644 --- a/server/src/services/healthMonitor.ts +++ b/server/src/services/healthMonitor.ts @@ -16,6 +16,7 @@ import { newListeningPorts, newPortMessage, parseBaseline, portKey, type NewPort import { parseJson, type AgentPort } from "./agentPorts.js"; import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js"; import { computeRuleFindings, findingKey, listRules, type RuleFinding } from "./alertRules.js"; +import { baselineKey, computeBaselineFindings, type BaselineFinding } from "./baselines.js"; import { recordStorageSamples } from "./storageHistory.js"; import { isUpsFresh, upsProblems, upsSyncStatuses, type UpsStateRow } from "./upsSync.js"; @@ -78,6 +79,8 @@ export interface HealthSnapshot { ups?: UpsSnapshot[]; /** What the administrator's own alert rules find on the servers (services/alertRules.ts). */ rules?: RuleFinding[]; + /** What is clearly higher than usual for this hour of the week, from each server's own history (services/baselines.ts). */ + baselines?: BaselineFinding[]; } export interface Thresholds { serverOfflineMinutes: number; @@ -167,13 +170,17 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, } } - // The administrator's own rules: judged only for servers that are reporting (one that is not has its offline alert). + // The administrator's own rules, and what is unusual for the time: judged only for servers that are reporting (one that is not has its offline alert). if (!opts.skipServers) { const offlineIds = new Set(out.map((c) => /^offline:server:(\d+)$/.exec(c.key)?.[1]).filter((id): id is string => id !== undefined).map(Number)); for (const f of snapshot.rules ?? []) { if (offlineIds.has(f.serverId)) continue; out.push({ key: findingKey(f), source: "server:rules", message: f.message, severity: f.severity }); } + for (const f of snapshot.baselines ?? []) { + if (offlineIds.has(f.serverId)) continue; + out.push({ key: baselineKey(f), source: "server:baseline", message: f.message, severity: "warning" }); + } } for (const u of snapshot.unifi ?? []) { @@ -385,6 +392,16 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel held.add("server:rules"); } + // What is unusually high for this hour of the week. A failure here costs only this check. + if (checks.baselineAlerts) { + try { + snapshot.baselines = await computeBaselineFindings((await getSettings()).notifications.timezone || "UTC"); + } catch (err) { + console.error("[health] couldn't judge what is usual:", err instanceof Error ? err.message : err); + held.add("server:baseline"); + } + } + const rows = await db .select({ id: integrations.id, name: integrations.name, type: integrations.type }) .from(integrations) diff --git a/server/src/services/maintenance.ts b/server/src/services/maintenance.ts index 63a09b0..e2097f5 100644 --- a/server/src/services/maintenance.ts +++ b/server/src/services/maintenance.ts @@ -27,6 +27,8 @@ export function subjectOfConditionKey(key: string): string | null { if (m) return `server:${m[1]}`; m = /^rule:\d+:server:(\d+)(?::|$)/.exec(key); if (m) return `server:${m[1]}`; + m = /^baseline:(?:cpu|mem):server:(\d+)$/.exec(key); + if (m) return `server:${m[1]}`; m = /^(?:endpoint(?:-cert)?|heartbeat(?:-fail)?):\d+:server:(\d+)$/.exec(key); if (m) return `server:${m[1]}`; m = /^offline:device:(\d+)$/.exec(key); diff --git a/server/src/services/settingsStore.ts b/server/src/services/settingsStore.ts index 903ee2c..ae9b14d 100644 --- a/server/src/services/settingsStore.ts +++ b/server/src/services/settingsStore.ts @@ -121,6 +121,8 @@ export interface HealthCheckSettings { deviceOfflineMinutes: number; /** Days before the end of an operating system's support that a server is first flagged on the Alerts page. 0 turns it off. */ osSupportWarnDays: number; + /** A server whose CPU load or memory use over the last hour is clearly higher than is normal for this hour of the week is a warning (services/baselines.ts). */ + baselineAlerts: boolean; } /** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. Off until switched on. */ @@ -298,7 +300,7 @@ const DEFAULTS: AppSettings = { selfBackup: { enabled: true, time: "03:30", keep: 14 }, profilePictures: { gravatar: false, identityProvider: false }, signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" }, - healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90 }, + healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true }, // Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every // Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN. consistency: { excludedRanges: ["172.16.0.0/12"] }, diff --git a/server/test/baselines.test.ts b/server/test/baselines.test.ts new file mode 100644 index 0000000..10fac87 --- /dev/null +++ b/server/test/baselines.test.ts @@ -0,0 +1,248 @@ +/** + * What is normal for a server at this time: the hours of the week and their baselines, when a figure is clearly past them (and when it is not), what is found from the stored + * history, how it becomes an alert, and the "usual" the server's page draws. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { Server } from "node:http"; +import type { AddressInfo } from "node:net"; + +const dir = mkdtempSync(join(tmpdir(), "hm-baselines-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers, serverMetrics, serverMetricsRollup } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const b = await import("../src/services/baselines.js"); +const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js"); +const { subjectOfConditionKey } = await import("../src/services/maintenance.js"); +const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js"); +const { getSettings, updateSettings } = await import("../src/services/settingsStore.js"); +const { serversRouter } = await import("../src/routes/servers.js"); + +await runMigrations(); + +const HOUR = 3_600_000; +const DAY = 86_400_000; +const WEEK = 7 * DAY; +/** A Tuesday, 03:40 UTC. */ +const NOW = Date.parse("2026-10-20T03:40:00Z"); +const TUE_3 = Date.parse("2026-10-20T03:00:00Z"); + +describe("hours of the week", () => { + test("a moment is a weekday and an hour in the time zone, summer time included", () => { + assert.deepEqual(b.slotOf(NOW, "UTC"), { dow: 2, hour: 3 }); + assert.deepEqual(b.slotOf(NOW, "Europe/Stockholm"), { dow: 2, hour: 5 }, "UTC+2 on this date"); + assert.deepEqual(b.slotOf(NOW, "America/New_York"), { dow: 1, hour: 23 }, "still Monday evening there"); + assert.deepEqual(b.slotOf(Date.parse("2026-01-15T23:30:00Z"), "Europe/Stockholm"), { dow: 5, hour: 0 }, "UTC+1 in winter: already Friday"); + assert.deepEqual(b.slotOf(NOW, "Not/AZone"), { dow: 2, hour: 3 }, "an unknown zone is read as UTC"); + assert.equal(b.slotWords({ dow: 2, hour: 3 }), "a Tuesday at 03:00"); + }); + + test("the median, and a spread that one odd week does not stretch", () => { + assert.equal(b.median([3, 1, 2]), 2); + assert.equal(b.median([4, 1, 2, 3]), 2.5); + assert.ok(Number.isNaN(b.median([]))); + const calm = b.robustSpread([8, 9, 7, 8, 10, 8]); + const withOdd = b.robustSpread([8, 9, 7, 8, 10, 95]); + assert.ok(calm > 0 && Math.abs(calm - withOdd) < 1.5, `an outlier barely moves it (${calm} vs ${withOdd})`); + assert.equal(b.robustSpread([5, 5, 5, 5]), 0); + }); +}); + +describe("the baseline of each hour", () => { + const row = (weeksBack: number, v: number, n = 4, hourOfWeek = TUE_3) => ({ t: hourOfWeek - weeksBack * WEEK, v, n }); + + test("it needs four weeks of that hour, and says nothing before", () => { + const three = b.slotTable([row(1, 8), row(2, 9), row(3, 7)], "UTC", NOW); + assert.equal(three[2 * 24 + 3], null); + const four = b.slotTable([row(1, 8), row(2, 9), row(3, 7), row(4, 8)], "UTC", NOW); + assert.deepEqual([four[2 * 24 + 3]?.median, four[2 * 24 + 3]?.weeks, four[2 * 24 + 3]?.dow, four[2 * 24 + 3]?.hour], [8, 4, 2, 3]); + assert.equal(four.filter((s) => s !== null).length, 1, "only that hour has a baseline"); + assert.equal(four.length, 168); + }); + + test("the hour in progress, thin hours and anything older than ten weeks are left out", () => { + const rows = [row(1, 8), row(2, 9), row(3, 7), row(4, 8), { t: TUE_3, v: 99, n: 4 }, row(5, 50, 1), row(11, 60)]; + const slot = b.slotTable(rows, "UTC", NOW)[2 * 24 + 3]!; + assert.deepEqual([slot.weeks, slot.median], [4, 8], "the 99 of this hour, the one-sample hour and the eleven-week-old one don't count"); + }); + + test("the hours are those of the time zone", () => { + const rows = [1, 2, 3, 4].map((w) => row(w, 20)); + const table = b.slotTable(rows, "Europe/Stockholm", NOW); + assert.ok(table[2 * 24 + 5], "03:00 UTC is Tuesday 05:00 in Stockholm"); + assert.equal(table[2 * 24 + 3], null); + }); +}); + +describe("when it is clearly unusual", () => { + const slot = { dow: 2, hour: 3, median: 8, spread: 2, weeks: 6 }; + const recent = (values: number[], newestAgoMin = 3) => values.map((v, i) => ({ t: NOW - (newestAgoMin + (values.length - 1 - i) * 10) * 60_000, v })); + const judge = (metric: "cpu" | "mem", values: number[], s: typeof slot | null = slot, ago = 3) => b.judgeUnusual(metric, recent(values, ago), s, NOW); + + test("a busy hour, far above the usual and never back toward it, is said", () => { + const u = judge("cpu", [78, 82, 80, 76])!; + assert.ok(u); + assert.deepEqual([Math.round(u.value), u.usual, u.weeks, u.slot], [79, 8, 6, { dow: 2, hour: 3 }]); + assert.equal(b.describeUnusual("wiki-host", u), "wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79% over the last hour, where it is normally about 8% (from 6 weeks of this hour)"); + }); + + test("memory has its own, smaller, bar", () => { + const mem = { ...slot, median: 40, spread: 1 }; + assert.ok(judge("mem", [82, 84, 83], mem), "40 → 83"); + assert.equal(judge("mem", [66, 68, 67], mem), null, "under the 70 % floor, however far above the usual"); + assert.equal(judge("mem", [72, 74, 73], { ...mem, median: 65 }), null, "only 8 points above"); + }); + + test("it stays quiet unless everything agrees", () => { + assert.equal(judge("cpu", [78, 82, 80, 76], null), null, "no baseline for the hour yet"); + assert.equal(judge("cpu", [78, 82]), null, "two samples is not an hour"); + assert.equal(judge("cpu", [78, 82, 80], slot, 40), null, "the newest sample is too old"); + assert.equal(judge("cpu", [46, 48, 49, 47]), null, "below the 50 % floor"); + assert.equal(judge("cpu", [30, 33, 31, 32]), null, "high for this hour but not far enough above the usual"); + assert.equal(judge("cpu", [78, 82, 80, 76], { ...slot, spread: 20 }), null, "an hour that is always unpredictable has a wide spread; 71 above is under four of them"); + assert.equal(judge("cpu", [90, 92, 20, 91]), null, "it dropped back to normal in that hour: a spike"); + assert.equal(judge("cpu", [5, 6, 4, 7]), null, "low is never said"); + assert.equal(judge("cpu", [78, 82, 80, 76].map((v) => v), { ...slot, median: 70, spread: 3 }), null, "a server that is always busy at this hour"); + }); + + test("only the last hour counts", () => { + const old = [{ t: NOW - 3 * HOUR, v: 95 }, { t: NOW - 2 * HOUR, v: 95 }, { t: NOW - 90 * 60_000, v: 95 }]; + assert.equal(b.judgeUnusual("cpu", old, slot, NOW), null); + }); +}); + +describe("found from the stored history", () => { + let busy = 0; + let young = 0; + let away = 0; + + /** Hourly rows for a Tuesday 03:00 and some other hours, `weeks` weeks back. */ + const seedHours = async (serverId: number, weeks: number, cpu: number, mem: number, at = TUE_3) => { + for (let w = 1; w <= weeks; w++) { + for (const [metric, v] of [["cpu", cpu + (w % 3) - 1], ["mem", mem + (w % 2)]] as const) { + await db.insert(serverMetricsRollup).values({ serverId, metric, grain: "h", at: new Date(at - w * WEEK).toISOString(), sum: v * 4, n: 4, min: v - 1, max: v + 1 }); + } + } + }; + const seedRecent = async (serverId: number, cpu: number[], mem: number[], nowMs = NOW) => { + for (let i = 0; i < cpu.length; i++) { + const at = new Date(nowMs - (3 + (cpu.length - 1 - i) * 10) * 60_000).toISOString(); + await db.insert(serverMetrics).values([{ serverId, metric: "cpu", value: cpu[i], recordedAt: at }, { serverId, metric: "mem", value: mem[i], recordedAt: at }]); + } + }; + + before(async () => { + const mk = async (name: string, seen: number) => (await db.insert(servers).values({ name, apiTokenHash: name, apiTokenPrefix: name, lastSeenAt: new Date(seen).toISOString() }).returning({ id: servers.id }))[0].id; + busy = await mk("wiki-host", NOW - 3 * 60_000); + young = await mk("new-box", NOW - 3 * 60_000); + away = await mk("quiet-box", NOW - 3 * HOUR); + await seedHours(busy, 6, 8, 40); + await seedRecent(busy, [78, 82, 80, 76], [42, 41, 43, 42]); + await seedHours(young, 3, 8, 40); + await seedRecent(young, [78, 82, 80, 76], [42, 41, 43, 42]); + await seedHours(away, 6, 8, 40); + await seedRecent(away, [78, 82, 80, 76], [42, 41, 43, 42]); + }); + + test("a server that is busy for the hour is found; one with too little history, or not reporting, is not", async () => { + const found = await b.computeBaselineFindings("UTC", NOW); + assert.deepEqual(found.map((f) => [f.serverName, f.metric, b.baselineKey(f)]), [["wiki-host", "cpu", `baseline:cpu:server:${busy}`]], "memory is normal for the hour, so only the CPU"); + assert.match(found[0].message, /^wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79% over the last hour, where it is normally about \d+% \(from 6 weeks of this hour\)$/); + }); + + test("the same load at a time it is usual says nothing", async () => { + // Wednesday 03:40 is judged against Wednesdays, which have no history here + assert.deepEqual(await b.computeBaselineFindings("UTC", NOW + DAY), []); + }); + + test("the page's view has every hour of the week for each figure, and where it is now", async () => { + const v = await b.baselineView(busy, "UTC", NOW); + assert.deepEqual([v.timezone, v.now, v.slots.cpu.length, v.slots.mem.length], ["UTC", { dow: 2, hour: 3 }, 168, 168]); + assert.deepEqual([v.slots.cpu[2 * 24 + 3]?.weeks, Math.round(v.slots.cpu[2 * 24 + 3]!.median)], [6, 8]); + assert.equal(v.slots.cpu[0], null); + assert.equal((await b.baselineView(young, "UTC", NOW)).slots.cpu[2 * 24 + 3], null, "three weeks is not enough"); + }); + + test("a finding is an ordinary health condition, left to the server's own offline alert when it is offline, and kept off by the setting", async () => { + const found = await b.computeBaselineFindings("UTC", NOW); + const snap = { servers: [{ id: busy, name: "wiki-host", lastSeenAt: new Date(NOW - 3 * 60_000).toISOString(), disks: [] }], proxmox: [], synology: [], baselines: found }; + const list = evaluateHealth(snap, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, NOW); + const c = list.find((x) => x.key === `baseline:cpu:server:${busy}`)!; + assert.deepEqual([c.source, c.severity], ["server:baseline", "warning"]); + assert.equal(subjectOfConditionKey(c.key), `server:${busy}`, "maintenance on the server silences it"); + const gone = evaluateHealth({ ...snap, servers: [{ ...snap.servers[0], lastSeenAt: new Date(NOW - 5 * HOUR).toISOString() }] }, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, NOW); + assert.deepEqual(gone.map((x) => x.key), [`offline:server:${busy}`], "a server that stopped reporting is just offline"); + assert.equal(evaluateHealth(snap, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, NOW, { skipServers: true }).some((x) => x.key.startsWith("baseline:")), false); + + const checks = (await getSettings()).healthChecks; + assert.equal(checks.baselineAlerts, true, "on by default"); + await updateSettings({ healthChecks: { ...checks, baselineAlerts: false } }); + assert.equal((await collectSnapshot()).snapshot.baselines, undefined, "off: not even looked for"); + await updateSettings({ healthChecks: { ...checks, baselineAlerts: true } }); + }); +}); + +describe("on the Alerts page and the server's page", () => { + let id = 0; + let app: Server; + let base = ""; + + before(async () => { + await db.insert(users).values([{ oidcSub: "viewer", name: "Vic", role: "viewer" }]); + // History relative to the real now, so the hour being judged is the hour it really is. + const now = Date.now(); + const hourStart = Math.floor(now / HOUR) * HOUR; + id = (await db.insert(servers).values({ name: "builder", apiTokenHash: "builder", apiTokenPrefix: "b", lastSeenAt: new Date(now - 2 * 60_000).toISOString() }).returning({ id: servers.id }))[0].id; + for (let w = 1; w <= 6; w++) await db.insert(serverMetricsRollup).values({ serverId: id, metric: "cpu", grain: "h", at: new Date(hourStart - w * WEEK).toISOString(), sum: 24 + w, n: 4, min: 5, max: 9 }); + for (let i = 0; i < 4; i++) await db.insert(serverMetrics).values({ serverId: id, metric: "cpu", value: 85 + i, recordedAt: new Date(now - (2 + (3 - i) * 10) * 60_000).toISOString() }); + const a = express(); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: "viewer" } }; + next(); + }); + a.use("/api/servers", serversRouter); + app = await new Promise((resolve) => { + const s = a.listen(0, "127.0.0.1", () => resolve(s)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/servers`; + }); + after(() => { + app.close(); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } + }); + + test("the Alerts page says it, as a warning about the server, with a link to its history", async () => { + invalidateAlertsCache(); + const report = await getAlerts(true); + const a = report.alerts.find((x) => x.id === `monitoring:baseline:cpu:server:${id}`); + assert.ok(a, "the alert is there"); + assert.deepEqual([a.severity, a.category, a.source, a.link, a.target?.type, a.target?.id], ["warning", "monitoring", "Server", `/servers/${id}?view=history`, "server", id]); + assert.match(a.message, /^builder: CPU load is unusually high for a \w+day at \d\d:00/); + }); + + test("the server's page gets every hour of the week, for anyone signed in", async () => { + const res = await fetch(`${base}/${id}/baseline`, { headers: { connection: "close" } }); + assert.equal(res.status, 200); + const body = (await res.json()) as any; + assert.deepEqual([body.timezone, body.slots.cpu.length, body.slots.mem.length], ["UTC", 168, 168]); + const here = body.slots.cpu[body.now.dow * 24 + body.now.hour]; + assert.deepEqual([here.weeks, Math.round(here.median)], [6, 7]); + assert.equal((await fetch(`${base}/999/baseline`, { headers: { connection: "close" } })).status, 404); + assert.equal((await fetch(`${base}/abc/baseline`, { headers: { connection: "close" } })).status, 400); + }); +}); diff --git a/web/src/api/client.ts b/web/src/api/client.ts index a570d3e..a3325bc 100644 --- a/web/src/api/client.ts +++ b/web/src/api/client.ts @@ -301,6 +301,22 @@ export interface ServerActivityEvent { actor: string | null; } +/** What is normal for one hour of the week: the median of that hour's averages over past weeks, a robust spread, and how many weeks it stands on. */ +export interface BaselineSlot { + dow: number; + hour: number; + median: number; + spread: number; + weeks: number; +} + +/** What is normal for a server at each hour of the week (Sunday 00:00 first), in the notification time zone; null where there are not enough weeks yet. */ +export interface BaselineView { + timezone: string; + now: { dow: number; hour: number }; + slots: { cpu: (BaselineSlot | null)[]; mem: (BaselineSlot | null)[] }; +} + export interface ServerMetricSeries { metric: string; kind: "cpu" | "mem" | "disk"; @@ -519,6 +535,8 @@ export interface HealthCheckSettings { deviceOfflineMinutes: number; /** Days before an operating system's support ends that a server is flagged (0 = off). */ osSupportWarnDays: number; + /** A server whose CPU or memory over the last hour is clearly higher than is normal for this hour of the week is a warning. */ + baselineAlerts: boolean; } /** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. */ @@ -2908,6 +2926,7 @@ export const api = { detail: (id: number) => request(`/api/servers/${id}/detail`), activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`), metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`), + baseline: (id: number) => request(`/api/servers/${id}/baseline`), remove: (id: number) => request(`/api/servers/${id}`, { method: "DELETE" }), setTags: (id: number, tags: string[]) => request<{ tags: string[] }>(`/api/servers/${id}/tags`, { method: "PUT", body: JSON.stringify({ tags }) }), diff --git a/web/src/components/ServerTrends.tsx b/web/src/components/ServerTrends.tsx index a44d812..08d2586 100644 --- a/web/src/components/ServerTrends.tsx +++ b/web/src/components/ServerTrends.tsx @@ -1,5 +1,6 @@ import { useEffect, useMemo, useState } from "react"; -import { api, type ServerMetricSeries } from "../api/client"; +import { api, type BaselineView, type ServerMetricSeries } from "../api/client"; +import { hourOfWeek } from "../utils/hourOfWeek"; import TimeChart from "./TimeChart"; import RangeSelect from "./RangeSelect"; import { BlockSkeleton } from "./Skeleton"; @@ -33,6 +34,7 @@ export default function ServerTrends({ serverId }: { serverId: number }) { const [days, setDays] = useChartRange(7); const [series, setSeries] = useState(null); const [failed, setFailed] = useState(false); + const [baseline, setBaseline] = useState(null); // Fixed per load, so the right-hand end of a chart is when it was read and does not creep as the page re-renders. const [to, setTo] = useState(() => Date.now()); @@ -52,11 +54,40 @@ export default function ServerTrends({ serverId }: { serverId: number }) { }; }, [serverId, days]); + // What is usual for each hour of the week, loaded once; it only matters while the points are single samples or hours (the weekday means nothing for a day's average). + useEffect(() => { + let alive = true; + api.servers + .baseline(serverId) + .then((res) => alive && setBaseline(res)) + .catch(() => alive && setBaseline(null)); + return () => { + alive = false; + }; + }, [serverId]); + const from = to - days * 86_400_000; const rangeLabel = CHART_RANGES.find((r) => r.days === days)?.label ?? `${days} days`; const charts = useMemo( - () => (series ?? []).map((s) => ({ s, words: describeTrend(s), series: [{ name: s.label, color: levelColor(s), points: s.points }] })), - [series], + () => + (series ?? []).map((s) => { + const lines = [{ name: s.label, color: levelColor(s), points: s.points as { t: string; v: number; lo?: number; hi?: number }[] }]; + const slots = s.kind === "disk" || !baseline ? null : baseline.slots[s.kind]; + if (slots && days <= 30) { + // The usual for the hour each point falls in, as a line with a band around it: the median and two spreads either side. + const usual = s.points.flatMap((p) => { + const h = hourOfWeek(Date.parse(p.t), baseline!.timezone); + const slot = slots[h.dow * 24 + h.hour]; + if (!slot) return []; + const room = 2 * Math.max(slot.spread, s.kind === "cpu" ? 4 : 2); + return [{ t: p.t, v: slot.median, lo: Math.max(0, slot.median - room), hi: Math.min(100, slot.median + room) }]; + }); + if (usual.length > 0) lines.push({ name: "Usual for the time", color: "#868e96", points: usual }); + } + const here = s.kind === "disk" || !baseline ? null : baseline.slots[s.kind][baseline.now.dow * 24 + baseline.now.hour]; + return { s, words: describeTrend(s), series: lines, usualNow: here }; + }), + [series, baseline, days], ); return ( @@ -76,16 +107,22 @@ export default function ServerTrends({ serverId }: { serverId: number }) { ) : (
- {charts.map(({ s, words, series: lines }) => ( + {charts.map(({ s, words, series: lines, usualNow }) => (
{s.label} {s.latest !== null ? `${Math.round(s.latest)}%` : "—"} {words} + {usualNow && ( + + · usually about {Math.round(usualNow.median)}% now + + )}
))} + {charts.some((c) => c.series.length > 1) &&
The grey band is what is usual for that hour of the week, from this server’s own history (the median, two spreads either side).
} {days > 35 &&
Over {days > 120 ? "more than four months" : "more than five weeks"} each point is {days > 120 ? "a day" : "an hour"}’s average, and the shaded band is its lowest and highest.
}
)} diff --git a/web/src/pages/settings/NotificationSettings.tsx b/web/src/pages/settings/NotificationSettings.tsx index 59570d0..158a0a6 100644 --- a/web/src/pages/settings/NotificationSettings.tsx +++ b/web/src/pages/settings/NotificationSettings.tsx @@ -83,7 +83,7 @@ const DIGEST_DAYS: { value: number; label: string }[] = [ { value: 6, label: "Saturday" }, { value: 0, label: "Sunday" }, ]; -const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90 }; +const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true }; const DEFAULT_MONTHLY: MonthlyReportSettings = { enabled: false, day: 1, time: "08:00" }; type TestResult = { ok: boolean; message: string } | null; @@ -754,6 +754,17 @@ export default function NotificationSettings() { hint="Switches, access points and gateways read from a UniFi controller (Network devices). The controller is read every five minutes, and what a device needs or what needs it is folded under it when it is down." onChange={(v) => setHealthChecks((h) => ({ ...h, deviceOfflineMinutes: v }))} /> +