Say who was told in an alert's timeline

Notifications are written down with the alerts they were about, and the
incident timeline lists each one: which channels took it, which refused and
what they said, or why it was held or not sent. New alerts, recovery notices,
reminders and escalations are all covered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-10 21:14:58 +01:00
1 parent f546971000
commit 461219844b
13 files changed
+9443 -18

No files matched your search

+1 -1
View File
@@ -51,7 +51,7 @@ below; several are fixed and can't be changed from the UI.
| **Held** | Quiet hours or away mode held it for the digest (the digest is its own entry) | | **Held** | Quiet hours or away mode held it for the digest (the digest is its own entry) |
| **Not sent (rule)** | A routing rule says nobody should be told about that kind | | **Not sent (rule)** | A routing rule says nobody should be told about that kind |
Filter by status or search the title and message. **Send again** (**Send now** for a held one) sends the entry once more to every channel that is on, writes that down as a new entry and is in the change log; it does not carry the buttons an alert message had. Messages are kept 30 days (at most 1000 entries), cut at 4000 characters, and an error is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back. The *Send test* buttons in Settings are not listed. Filter by status or search the title and message. A notification about alerts also carries **which alerts it was about**, so each alert's **Timeline** (Alerts → History → the incident) says who was told and when, with the same ✓/✕ detail — including a message that was held or not sent. (A held message is listed against its alerts when it was held; the digest that later carries it is not.) **Send again** (**Send now** for a held one) sends the entry once more to every channel that is on, writes that down as a new entry and is in the change log; it does not carry the buttons an alert message had. Messages are kept 30 days (at most 1000 entries), cut at 4000 characters, and an error is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back. The *Send test* buttons in Settings are not listed.
## Who hears what ## Who hears what
+1 -1
View File
@@ -535,7 +535,7 @@ server's history.
**Incidents** (Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an **Incidents** (Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the after the alert is gone. Open one for the whole story in order: the alert, **who was told** (each notification that was about it — a new alert, a recovery notice, a reminder, an escalation — through which channels, which refused and what they said, or why it was held for quiet hours or not sent because a routing rule says nobody), who acknowledged it and why, the
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
began around the same time, and the notes people wrote (operators can add them; the author or an began around the same time, and the notes people wrote (operators can add them; the author or an
administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look
@@ -0,0 +1 @@
ALTER TABLE `notification_log` ADD `alert_keys` text;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -645,6 +645,13 @@
"when": 1794334785852, "when": 1794334785852,
"tag": "0091_adorable_wasp", "tag": "0091_adorable_wasp",
"breakpoints": true "breakpoints": true
},
{
"idx": 92,
"version": "6",
"when": 1794340893429,
"tag": "0092_parallel_the_initiative",
"breakpoints": true
} }
] ]
} }
+1
View File
@@ -1692,6 +1692,7 @@ export const notificationLog = sqliteTable(
status: text("status").notNull(), // sent | partly | failed | none (no channel to send it to) | held (quiet hours or away) | muted (a routing rule says nobody) status: text("status").notNull(), // sent | partly | failed | none (no channel to send it to) | held (quiet hours or away) | muted (a routing rule says nobody)
results: text("results").notNull().default("[]"), // JSON: [{ channel, ok, error }] results: text("results").notNull().default("[]"), // JSON: [{ channel, ok, error }]
note: text("note"), // why it was held or muted note: text("note"), // why it was held or muted
alertKeys: text("alert_keys"), // JSON string[]: the alerts it was about (their keys), so an alert's timeline can say who was told; null: not about any
resentOf: integer("resent_of"), // the entry it is a resend of resentOf: integer("resent_of"), // the entry it is a resend of
}, },
(t) => [index("notification_log_at_idx").on(t.at)], (t) => [index("notification_log_at_idx").on(t.at)],
+1 -1
View File
@@ -89,7 +89,7 @@ export async function runEscalation(now: Date = new Date()): Promise<number> {
const lines = items.map((d) => `${d.alert.severity === "critical" ? "✕" : "⚠"} ${d.alert.message}`); const lines = items.map((d) => `${d.alert.severity === "critical" ? "✕" : "⚠"} ${d.alert.message}`);
const message = `Nobody has acknowledged ${items.length === 1 ? "this alert" : `these ${items.length} alerts`} after ${words(def.afterMinutes)}:\n\n${lines.join("\n")}\n\nAcknowledge ${items.length === 1 ? "it" : "them"} on the Alerts page to stop the reminders.`; const message = `Nobody has acknowledged ${items.length === 1 ? "this alert" : `these ${items.length} alerts`} after ${words(def.afterMinutes)}:\n\n${lines.join("\n")}\n\nAcknowledge ${items.length === 1 ? "it" : "them"} on the Alerts page to stop the reminders.`;
try { try {
await notifyAudience("Homelab Manager — Still not acknowledged", message, { channels: def.channels, pushIds: null, ...(def.extraEmail ? { email: def.extraEmail } : {}) }, { quietHours: !def.evenInQuietHours, actions: await actionsFor(items.map((d) => keyOfAlertId(d.alert.id, categoryOf.get(d.alert.id) ?? ""))) }); await notifyAudience("Homelab Manager — Still not acknowledged", message, { channels: def.channels, pushIds: null, ...(def.extraEmail ? { email: def.extraEmail } : {}) }, { quietHours: !def.evenInQuietHours, keys: items.map((d) => d.alert.id), actions: await actionsFor(items.map((d) => keyOfAlertId(d.alert.id, categoryOf.get(d.alert.id) ?? ""))) });
} catch (err) { } catch (err) {
console.error("[escalation] couldn't send step", step, err); console.error("[escalation] couldn't send step", step, err);
continue; // not marked as sent: the next look tries again continue; // not marked as sent: the next look tries again
+32 -1
View File
@@ -16,6 +16,7 @@ import type { Alert } from "./alertTypes.js";
import { describeAudit, auditTimeToIso, type ActivityTone } from "./serverActivity.js"; import { describeAudit, auditTimeToIso, type ActivityTone } from "./serverActivity.js";
import { startupGraceRemainingMs } from "./healthMonitor.js"; import { startupGraceRemainingMs } from "./healthMonitor.js";
import { loadAck, type AckInfo } from "./alertAcks.js"; import { loadAck, type AckInfo } from "./alertAcks.js";
import { loggedForAlert } from "./notificationLog.js";
export type IncidentRow = typeof alertIncidents.$inferSelect; export type IncidentRow = typeof alertIncidents.$inferSelect;
@@ -177,7 +178,7 @@ export async function openIncidentIds(alertKeys: string[]): Promise<Map<string,
return new Map(rows.map((r) => [r.key, r.id])); return new Map(rows.map((r) => [r.key, r.id]));
} }
export type TimelineKind = "alert" | "ack" | "maintenance" | "change" | "health" | "run" | "related" | "note"; export type TimelineKind = "alert" | "ack" | "maintenance" | "change" | "health" | "run" | "related" | "note" | "notify";
export interface TimelineEvent { export interface TimelineEvent {
/** ISO. */ /** ISO. */
@@ -280,12 +281,42 @@ export async function getIncident(id: number, now: Date = new Date()): Promise<I
for (const r of related) events.push({ at: r.openedAt, kind: "related", tone: "info", title: `Another alert was raised: ${r.category} (${r.severity})`, detail: r.message, actor: null, incidentId: r.id }); for (const r of related) events.push({ at: r.openedAt, kind: "related", tone: "info", title: `Another alert was raised: ${r.category} (${r.severity})`, detail: r.message, actor: null, incidentId: r.id });
} }
// Who was told: each notification that was about this alert, and how each channel took it (or why it was held or not sent).
const told = await loggedForAlert(row.alertKey, new Date(from - 5 * 60_000).toISOString(), new Date(to + 30 * 60_000).toISOString());
for (const n of told) events.push({ at: n.at, ...describeNotification(n) });
for (const n of notes) events.push({ at: n.createdAt, kind: "note", tone: "neutral", title: `${n.author} wrote a note`, detail: n.body, actor: n.author, noteId: n.id }); for (const n of notes) events.push({ at: n.createdAt, kind: "note", tone: "neutral", title: `${n.author} wrote a note`, detail: n.body, actor: n.author, noteId: n.id });
events.sort((a, b) => a.at.localeCompare(b.at)); events.sort((a, b) => a.at.localeCompare(b.at));
return { incident, ack: row.resolvedAt ? null : await loadAck(row.alertKey, now), timeline: events, notes: notes.map((n) => ({ id: n.id, author: n.author, body: n.body, at: n.createdAt })) }; return { incident, ack: row.resolvedAt ? null : await loadAck(row.alertKey, now), timeline: events, notes: notes.map((n) => ({ id: n.id, author: n.author, body: n.body, at: n.createdAt })) };
} }
const CHANNEL_NAMES: Record<string, string> = { gotify: "Gotify", ntfy: "ntfy", telegram: "Telegram", pushover: "Pushover", smtp: "email", webhook: "the webhook", webPush: "push", email: "email (escalation)" };
const list = (names: string[]) => (names.length <= 1 ? (names[0] ?? "") : `${names.slice(0, -1).join(", ")} and ${names[names.length - 1]}`);
/** Pure: how one notification reads in an alert's timeline — who it reached, who it did not and why. The notification's own title says what kind it was (a new alert, a recovery, a reminder). */
export function describeNotification(n: { title: string; status: string; note: string | null; results: { channel: string; ok: boolean; error: string | null }[] }): Pick<TimelineEvent, "kind" | "tone" | "title" | "detail" | "actor"> {
const base = { kind: "notify" as const, actor: null };
const said = n.title.replace(/^Homelab Manager — /, "");
const ok = n.results.filter((r) => r.ok).map((r) => CHANNEL_NAMES[r.channel] ?? r.channel);
const bad = n.results.filter((r) => !r.ok);
const errors = bad.map((r) => `${CHANNEL_NAMES[r.channel] ?? r.channel}: ${r.error ?? "failed"}`).join("; ");
switch (n.status) {
case "sent":
return { ...base, tone: "good", title: `Told through ${list(ok)}`, detail: said };
case "partly":
return { ...base, tone: "bad", title: `Told through ${list(ok)} — ${list(bad.map((r) => CHANNEL_NAMES[r.channel] ?? r.channel))} did not take it`, detail: `${said} · ${errors}` };
case "failed":
return { ...base, tone: "bad", title: "Nobody could be told — every channel failed", detail: `${said} · ${errors}` };
case "held":
return { ...base, tone: "neutral", title: "The message was held back", detail: `${said} · ${n.note ?? "held"}` };
case "muted":
return { ...base, tone: "neutral", title: "Nobody was told — a routing rule says nobody should be", detail: said };
default:
return { ...base, tone: "neutral", title: "Nobody was told — no channel is switched on", detail: said };
}
}
export const MAX_NOTE = 4000; export const MAX_NOTE = 4000;
export async function addNote(incidentId: number, author: string, body: string, now: Date = new Date()): Promise<{ id: number } | null> { export async function addNote(incidentId: number, author: string, body: string, now: Date = new Date()): Promise<{ id: number } | null> {
+26 -3
View File
@@ -3,7 +3,7 @@
* for quiet hours, a routing rule said nobody should hear it, or no channel was switched on to send it to. Kept 30 days and at most 1000 entries. Nothing here can make sending fail: writing * for quiet hours, a routing rule said nobody should hear it, or no channel was switched on to send it to. Kept 30 days and at most 1000 entries. Nothing here can make sending fail: writing
* it down is best-effort, like the channels themselves. An error's text is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back. * it down is best-effort, like the channels themselves. An error's text is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back.
*/ */
import { and, desc, eq, like, lt, notInArray, or } from "drizzle-orm"; import { and, desc, eq, gte, like, lt, lte, notInArray, or } from "drizzle-orm";
import { db } from "../db/client.js"; import { db } from "../db/client.js";
import { notificationLog } from "../db/schema.js"; import { notificationLog } from "../db/schema.js";
import { getSettings } from "./settingsStore.js"; import { getSettings } from "./settingsStore.js";
@@ -62,13 +62,15 @@ async function trim(now: Date): Promise<void> {
} }
/** Writes down one notification: what it said, and how it went (or why it was held or muted). Returns its id, or null when it could not be written — never throws. */ /** Writes down one notification: what it said, and how it went (or why it was held or muted). Returns its id, or null when it could not be written — never throws. */
export async function recordNotification(entry: { title: string; message: string; results: ChannelResult[]; status?: LogStatus; note?: string | null; resentOf?: number | null }, now: Date = new Date()): Promise<number | null> { export const MAX_ALERT_KEYS = 50;
export async function recordNotification(entry: { title: string; message: string; results: ChannelResult[]; status?: LogStatus; note?: string | null; resentOf?: number | null; keys?: string[] }, now: Date = new Date()): Promise<number | null> {
try { try {
const secrets = await secretsNow(); const secrets = await secretsNow();
const results = entry.results.map((r) => ({ channel: r.channel, ok: r.ok, error: r.error === null ? null : clip(scrub(r.error.replace(/\s+/g, " ").trim(), secrets), MAX_ERROR) })); const results = entry.results.map((r) => ({ channel: r.channel, ok: r.ok, error: r.error === null ? null : clip(scrub(r.error.replace(/\s+/g, " ").trim(), secrets), MAX_ERROR) }));
const [row] = await db const [row] = await db
.insert(notificationLog) .insert(notificationLog)
.values({ at: now.toISOString(), title: clip(entry.title, 300), message: clip(entry.message, MAX_MESSAGE), status: entry.status ?? statusOf(results), results: JSON.stringify(results), note: entry.note ?? null, resentOf: entry.resentOf ?? null }) .values({ at: now.toISOString(), title: clip(entry.title, 300), message: clip(entry.message, MAX_MESSAGE), status: entry.status ?? statusOf(results), results: JSON.stringify(results), note: entry.note ?? null, alertKeys: entry.keys && entry.keys.length > 0 ? JSON.stringify([...new Set(entry.keys.filter((k) => typeof k === "string" && k !== ""))].slice(0, MAX_ALERT_KEYS)) : null, resentOf: entry.resentOf ?? null })
.returning({ id: notificationLog.id }); .returning({ id: notificationLog.id });
await trim(now); await trim(now);
return row.id; return row.id;
@@ -87,6 +89,27 @@ export function resultsOf(row: Pick<LogRow, "results">): ChannelResult[] {
} }
} }
/** Pure: the keys a row says it was about. */
export function keysOf(row: Pick<LogRow, "alertKeys">): string[] {
try {
const v = row.alertKeys ? JSON.parse(row.alertKeys) : [];
return Array.isArray(v) ? v.filter((k): k is string => typeof k === "string") : [];
} catch {
return [];
}
}
/** The notifications that were about one alert between two moments, oldest first — what its timeline says about who was told. */
export async function loggedForAlert(alertKey: string, fromIso: string, toIso: string): Promise<(Omit<LogRow, "results"> & { results: ChannelResult[] })[]> {
// The key is looked for as the JSON writes it; a character that LIKE treats as a wildcard can match too much, so what is found is checked again.
const rows = await db
.select()
.from(notificationLog)
.where(and(like(notificationLog.alertKeys, `%${JSON.stringify(alertKey)}%`), gte(notificationLog.at, fromIso), lte(notificationLog.at, toIso)))
.orderBy(notificationLog.at, notificationLog.id);
return rows.filter((r) => keysOf(r).includes(alertKey)).map((r) => ({ ...r, results: resultsOf(r) }));
}
export async function getLogged(id: number): Promise<LogRow | null> { export async function getLogged(id: number): Promise<LogRow | null> {
return (await db.select().from(notificationLog).where(eq(notificationLog.id, id)).limit(1))[0] ?? null; return (await db.select().from(notificationLog).where(eq(notificationLog.id, id)).limit(1))[0] ?? null;
} }
+10 -9
View File
@@ -286,7 +286,7 @@ export async function audienceFor(meta?: NotifyMeta): Promise<Audience> {
} }
/** Sends to the channels the audience names (and are switched on), and writes down how each went: the entry's id, or null if it could not be written. */ /** Sends to the channels the audience names (and are switched on), and writes down how each went: the entry's id, or null if it could not be written. */
async function sendToChannels(title: string, message: string, audience: Audience = EVERYONE, actions: NotifyActions | null = null, resentOf: number | null = null): Promise<number | null> { async function sendToChannels(title: string, message: string, audience: Audience = EVERYONE, actions: NotifyActions | null = null, resentOf: number | null = null, keys: string[] = []): Promise<number | null> {
const to = (c: ChannelId) => audience.channels === null || audience.channels.includes(c); const to = (c: ChannelId) => audience.channels === null || audience.channels.includes(c);
const jobs: [string, Promise<SendOutcome>][] = []; const jobs: [string, Promise<SendOutcome>][] = [];
if (to("gotify")) jobs.push(["gotify", sendGotify(title, message, actions)]); if (to("gotify")) jobs.push(["gotify", sendGotify(title, message, actions)]);
@@ -299,30 +299,30 @@ async function sendToChannels(title: string, message: string, audience: Audience
if (audience.email) jobs.push(["email", sendExtraEmail(title, actions ? message + footerText(actions) : message, audience.email)]); if (audience.email) jobs.push(["email", sendExtraEmail(title, actions ? message + footerText(actions) : message, audience.email)]);
const outcomes = await Promise.all(jobs.map(async ([channel, run]) => [channel, await run] as const)); const outcomes = await Promise.all(jobs.map(async ([channel, run]) => [channel, await run] as const));
const results: ChannelResult[] = outcomes.filter(([, o]) => o !== null).map(([channel, o]) => ({ channel, ok: o!.ok, error: o!.ok ? null : o!.error })); const results: ChannelResult[] = outcomes.filter(([, o]) => o !== null).map(([channel, o]) => ({ channel, ok: o!.ok, error: o!.ok ? null : o!.error }));
return recordNotification({ title, message, results, resentOf }); return recordNotification({ title, message, results, resentOf, keys });
} }
/** /**
* Sends to an audience that is already worked out: held for the quiet-hours digest if it is quiet (unless told not to), nowhere if a rule * Sends to an audience that is already worked out: held for the quiet-hours digest if it is quiet (unless told not to), nowhere if a rule
* said nobody. Escalations use this with their own channels. * said nobody. Escalations use this with their own channels.
*/ */
export async function notifyAudience(title: string, message: string, audience: Audience, options: { quietHours?: boolean; actions?: NotifyActions | null; critical?: boolean } = {}): Promise<void> { export async function notifyAudience(title: string, message: string, audience: Audience, options: { quietHours?: boolean; actions?: NotifyActions | null; critical?: boolean; /** The alerts it is about (their keys), so each one's timeline can say who was told. */ keys?: string[] } = {}): Promise<void> {
if (isMuted(audience)) { if (isMuted(audience)) {
await recordNotification({ title, message, results: [], status: "muted", note: "A routing rule says nobody should be told." }); await recordNotification({ title, message, results: [], status: "muted", note: "A routing rule says nobody should be told.", keys: options.keys });
return; return;
} }
// Held when quiet hours or an away stretch say so — unless it is critical and they let critical through (services/availability.ts). // Held when quiet hours or an away stretch say so — unless it is critical and they let critical through (services/availability.ts).
const held = options.quietHours !== false ? await holdReason(options.critical === true) : null; const held = options.quietHours !== false ? await holdReason(options.critical === true) : null;
if (held) { if (held) {
await enqueueNotification(title, message, audience.channels === null && audience.pushIds === null && !audience.email ? null : JSON.stringify(audience)); await enqueueNotification(title, message, audience.channels === null && audience.pushIds === null && !audience.email ? null : JSON.stringify(audience));
await recordNotification({ title, message, results: [], status: "held", note: held === "away" ? "Held while you are away; it is sent in the digest when you are back." : "Held for quiet hours; it is sent in the digest when they end." }); await recordNotification({ title, message, results: [], status: "held", note: held === "away" ? "Held while you are away; it is sent in the digest when you are back." : "Held for quiet hours; it is sent in the digest when they end.", keys: options.keys });
return; return;
} }
await sendToChannels(title, message, audience, options.actions ?? null); await sendToChannels(title, message, audience, options.actions ?? null, null, options.keys ?? []);
} }
/** Items that may each be for a different audience (a rule can send a disk warning one place and a server outage another): sent as one message per audience. */ /** Items that may each be for a different audience (a rule can send a disk warning one place and a server outage another): sent as one message per audience. */
async function notifyGrouped<T>(items: T[], metaOf: (item: T) => NotifyMeta, build: (group: T[]) => { title: string; message: string }, keysOf?: (item: T) => string | undefined): Promise<void> { async function notifyGrouped<T>(items: T[], metaOf: (item: T) => NotifyMeta, build: (group: T[]) => { title: string; message: string }, keysOf?: (item: T) => string | undefined, options: { actions?: boolean } = {}): Promise<void> {
const groups = new Map<string, { audience: Audience; items: T[] }>(); const groups = new Map<string, { audience: Audience; items: T[] }>();
for (const item of items) { for (const item of items) {
const audience = await audienceFor(metaOf(item)); const audience = await audienceFor(metaOf(item));
@@ -335,7 +335,8 @@ async function notifyGrouped<T>(items: T[], metaOf: (item: T) => NotifyMeta, bui
const { title, message } = build(group.items); const { title, message } = build(group.items);
// Buttons, when the message is about alerts that have a key and an administrator has switched them on (a message held for quiet hours is sent without them). // Buttons, when the message is about alerts that have a key and an administrator has switched them on (a message held for quiet hours is sent without them).
const critical = group.items.some((i) => metaOf(i).severity === "critical"); const critical = group.items.some((i) => metaOf(i).severity === "critical");
await notifyAudience(title, message, group.audience, { critical, ...(keysOf ? { actions: await actionsFor(group.items.map(keysOf)) } : {}) }); const keys = keysOf ? group.items.map(keysOf).filter((k): k is string => typeof k === "string") : [];
await notifyAudience(title, message, group.audience, { critical, keys, ...(keysOf && options.actions !== false ? { actions: await actionsFor(group.items.map(keysOf)) } : {}) });
} }
} }
@@ -657,7 +658,7 @@ export async function notifyHealthRecovered(resolved: { message: string; key?: s
await notifyGrouped(resolved, issueMeta, (group) => ({ await notifyGrouped(resolved, issueMeta, (group) => ({
title: "Homelab Manager — Health Recovered", title: "Homelab Manager — Health Recovered",
message: `${group.length} issue${group.length !== 1 ? "s" : ""} cleared:\n\n${group.map((i) => `✓ ${i.message}`).join("\n")}`, message: `${group.length} issue${group.length !== 1 ? "s" : ""} cleared:\n\n${group.map((i) => `✓ ${i.message}`).join("\n")}`,
})); }), (i) => i.key, { actions: false });
} }
/** Alerts that were snoozed, whose time is up and that are still there: said once, as a reminder. */ /** Alerts that were snoozed, whose time is up and that are still there: said once, as a reminder. */
+170
View File
@@ -0,0 +1,170 @@
/**
* Who was told, in an alert's timeline: a notification is written down with the alerts it was about, and the incident's timeline lists each one that was about its alert — how each channel took it, or why it
* was held or not sent — alongside the acknowledgements, maintenance and the rest.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer as createHttpServer, type Server } from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-alerttimeline-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const { db } = await import("../src/db/client.js");
const { alertIncidents } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const log = await import("../src/services/notificationLog.js");
const inc = await import("../src/services/incidents.js");
const { notifyHealthIssues, notifyHealthRecovered, notifyAudience } = await import("../src/services/notify.js");
const { updateSettings } = await import("../src/services/settingsStore.js");
await runMigrations();
const MIN = 60_000;
const NOW = Date.parse("2026-11-10T12:00:00Z");
const iso = (ms: number) => new Date(ms).toISOString();
const ok = (channel: string) => ({ channel, ok: true, error: null });
const bad = (channel: string, error: string) => ({ channel, ok: false, error });
describe("how a notification reads in the timeline", () => {
const d = inc.describeNotification;
test("sent: who it reached, in words, with what kind of message it was", () => {
const e = d({ title: "Homelab Manager — Health Alert", status: "sent", note: null, results: [ok("ntfy"), ok("telegram"), ok("smtp")] });
assert.deepEqual([e.kind, e.tone, e.title, e.detail], ["notify", "good", "Told through ntfy, Telegram and email", "Health Alert"]);
assert.equal(d({ title: "T", status: "sent", note: null, results: [ok("webPush")] }).title, "Told through push");
});
test("partly sent says who did not take it and what they said; failed says nobody could be told", () => {
const partly = d({ title: "Homelab Manager — Health Alert", status: "partly", note: null, results: [ok("ntfy"), bad("gotify", "Gotify error 401: unauthorised")] });
assert.deepEqual([partly.tone, partly.title], ["bad", "Told through ntfy — Gotify did not take it"]);
assert.match(partly.detail ?? "", /Gotify: Gotify error 401: unauthorised/);
const failed = d({ title: "Homelab Manager — Health Recovered", status: "failed", note: null, results: [bad("ntfy", "ntfy failed: fetch failed")] });
assert.deepEqual([failed.tone, failed.title], ["bad", "Nobody could be told — every channel failed"]);
assert.match(failed.detail ?? "", /Health Recovered · ntfy: ntfy failed/);
});
test("held, muted and nowhere to send each say so, in a neutral tone", () => {
assert.deepEqual([d({ title: "T", status: "held", note: "Held for quiet hours.", results: [] }).tone, d({ title: "T", status: "held", note: "Held for quiet hours.", results: [] }).title], ["neutral", "The message was held back"]);
assert.match(d({ title: "T", status: "held", note: "Held for quiet hours.", results: [] }).detail ?? "", /Held for quiet hours/);
assert.match(d({ title: "T", status: "muted", note: null, results: [] }).title, /routing rule says nobody/);
assert.match(d({ title: "T", status: "none", note: null, results: [] }).title, /no channel is switched on/);
});
});
describe("what a notification says it was about", () => {
test("the keys are written with it, once each, up to fifty, and read back as a list", async () => {
const id = (await log.recordNotification({ title: "t", message: "m", results: [], keys: ["a:1", "a:1", "b:2", ""] }))!;
const row = await log.getLogged(id);
assert.deepEqual(log.keysOf(row!), ["a:1", "b:2"]);
const many = (await log.recordNotification({ title: "t", message: "m", results: [], keys: Array.from({ length: 80 }, (_, i) => `k:${i}`) }))!;
assert.equal(log.keysOf((await log.getLogged(many))!).length, 50);
assert.deepEqual(log.keysOf({ alertKeys: null }), []);
for (const junk of ["nope", "{}", "5"]) assert.deepEqual(log.keysOf({ alertKeys: junk }), []);
});
test("an alert's notifications are found by its key and its time, and only its own", async () => {
await log.recordNotification({ title: "one", message: "m", results: [ok("ntfy")], keys: ["disk:server:3:/", "offline:server:4"] }, new Date(NOW - 10 * MIN));
await log.recordNotification({ title: "two", message: "m", results: [ok("ntfy")], keys: ["disk:server:3:/"] }, new Date(NOW - 3 * 60 * MIN));
await log.recordNotification({ title: "three", message: "m", results: [ok("ntfy")], keys: ["disk:server:33:/"] }, new Date(NOW - 5 * MIN));
await log.recordNotification({ title: "four", message: "m", results: [ok("ntfy")] }, new Date(NOW - 5 * MIN));
const found = await log.loggedForAlert("disk:server:3:/", iso(NOW - 60 * MIN), iso(NOW));
assert.deepEqual(found.map((r) => r.title), ["one"], "not the one three hours ago, not another server's, not one about nothing");
assert.deepEqual((await log.loggedForAlert("disk:server:3:/", iso(NOW - 4 * 60 * MIN), iso(NOW))).map((r) => r.title), ["two", "one"]);
});
test("a key with a character LIKE treats as a wildcard does not find another alert's", async () => {
await log.recordNotification({ title: "underscore", message: "m", results: [], keys: ["a_b:1"] }, new Date(NOW - MIN));
await log.recordNotification({ title: "other", message: "m", results: [], keys: ["axb:1"] }, new Date(NOW - MIN));
assert.deepEqual((await log.loggedForAlert("a_b:1", iso(NOW - 10 * MIN), iso(NOW))).map((r) => r.title), ["underscore"]);
});
});
describe("through the notifications that are sent", () => {
let hook: Server;
let url = "";
const got: string[] = [];
before(async () => {
hook = createHttpServer((req, res) => {
req.resume();
got.push(req.url ?? "");
res.writeHead(200).end("ok");
});
await new Promise<void>((resolve) => hook.listen(0, "127.0.0.1", resolve));
url = `http://127.0.0.1:${(hook.address() as AddressInfo).port}/hook`;
await updateSettings({ webhook: { enabled: true, url }, ntfy: { enabled: false } });
});
after(async () => {
hook.close();
await updateSettings({ webhook: { enabled: false }, routing: { rules: [] }, quietHours: { enabled: false, start: "22:00", end: "07:00", criticalThrough: false } });
});
const about = async (key: string) => log.loggedForAlert(key, "2000-01-01T00:00:00.000Z", "2100-01-01T00:00:00.000Z");
test("a new alert and its recovery are each written down with the alert they were about", async () => {
await notifyHealthIssues([{ message: "web-1 is down", key: "offline:server:9", severity: "critical" }, { message: "disk on web-1", key: "disk:server:9:/", severity: "warning" }]);
await notifyHealthRecovered([{ message: "web-1 is down", key: "offline:server:9", severity: "critical" }]);
const mine = await about("offline:server:9");
assert.deepEqual(mine.map((r) => [r.title.replace("Homelab Manager — ", ""), r.status]), [["Health Alert", "sent"], ["Health Recovered", "sent"]]);
assert.deepEqual((await about("disk:server:9:/")).map((r) => r.title.replace("Homelab Manager — ", "")), ["Health Alert"], "the disk alert was in the first message only");
assert.ok(got.length >= 2, "and they were really sent");
});
test("one that a routing rule silenced, or quiet hours held, says so against the alert too", async () => {
await updateSettings({ routing: { rules: [{ id: "q", name: "no disks", enabled: true, categories: ["disk"], minSeverity: null, tags: [], channels: [] }] } });
await notifyHealthIssues([{ message: "disk filling", key: "disk:server:10:/", severity: "warning" }]);
assert.deepEqual((await about("disk:server:10:/")).map((r) => r.status), ["muted"]);
await updateSettings({ routing: { rules: [] } });
const h = new Date().getUTCHours();
const pad = (n: number) => String(((n % 24) + 24) % 24).padStart(2, "0") + ":00";
await updateSettings({ quietHours: { enabled: true, start: pad(h - 1), end: pad(h + 1), criticalThrough: false }, notifications: { timezone: "UTC" } as never });
await notifyAudience("Homelab Manager — Health Alert", "held", { channels: null, pushIds: null }, { keys: ["updates:server:9"] });
assert.deepEqual((await about("updates:server:9")).map((r) => r.status), ["held"]);
await updateSettings({ quietHours: { enabled: false, start: "22:00", end: "07:00", criticalThrough: false } });
});
});
describe("in the incident's timeline", () => {
let incidentId = 0;
before(async () => {
incidentId = (
await db
.insert(alertIncidents)
.values({ alertKey: "offline:server:77", severity: "critical", peakSeverity: "critical", category: "offline", source: "Servers", message: "pve9 has not reported", link: null, targetType: null, targetId: null, targetName: null, causedBy: null, openedAt: iso(NOW - 60 * MIN), lastSeenAt: iso(NOW - 5 * MIN), resolvedAt: iso(NOW - 20 * MIN) })
.returning({ id: alertIncidents.id })
)[0].id;
await log.recordNotification({ title: "Homelab Manager — Health Alert", message: "m", results: [ok("ntfy"), bad("gotify", "Gotify error 401: nope")], keys: ["offline:server:77"] }, new Date(NOW - 59 * MIN));
await log.recordNotification({ title: "Homelab Manager — Still not acknowledged", message: "m", results: [ok("telegram")], keys: ["offline:server:77"] }, new Date(NOW - 40 * MIN));
await log.recordNotification({ title: "Homelab Manager — Health Recovered", message: "m", results: [], status: "held", note: "Held for quiet hours.", keys: ["offline:server:77"] }, new Date(NOW - 18 * MIN));
await log.recordNotification({ title: "Homelab Manager — Health Alert", message: "m", results: [ok("ntfy")], keys: ["offline:server:78"] }, new Date(NOW - 59 * MIN));
await log.recordNotification({ title: "Homelab Manager — Health Alert", message: "m", results: [ok("ntfy")], keys: ["offline:server:77"] }, new Date(NOW - 3 * 60 * MIN));
});
after(() => {
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("each notification about the alert is an event, in order among the others, and nobody else's is", async () => {
const detail = (await inc.getIncident(incidentId, new Date(NOW)))!;
const told = detail.timeline.filter((e) => e.kind === "notify");
assert.deepEqual(told.map((e) => e.title), ["Told through ntfy — Gotify did not take it", "Told through Telegram", "The message was held back"]);
assert.deepEqual(told.map((e) => e.tone), ["bad", "good", "neutral"]);
assert.match(told[1].detail ?? "", /Still not acknowledged/);
assert.deepEqual(detail.timeline.map((e) => e.at), [...detail.timeline.map((e) => e.at)].sort(), "the whole timeline is in time order");
assert.deepEqual(detail.timeline.filter((e) => e.kind === "alert").map((e) => e.title), ["Alert raised (critical)", "Resolved"]);
});
test("a notification from long before the alert, or about another one, is left out", async () => {
const detail = (await inc.getIncident(incidentId, new Date(NOW)))!;
assert.equal(detail.timeline.filter((e) => e.kind === "notify").length, 3);
});
});
+1 -1
View File
@@ -2793,7 +2793,7 @@ export interface AckInput {
hours: number | null; hours: number | null;
} }
export type TimelineKind = "alert" | "ack" | "maintenance" | "change" | "health" | "run" | "related" | "note"; export type TimelineKind = "alert" | "ack" | "maintenance" | "change" | "health" | "run" | "related" | "note" | "notify";
export interface TimelineEvent { export interface TimelineEvent {
at: string; at: string;
+2 -1
View File
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useState } from "react"; import { useCallback, useEffect, useState } from "react";
import { Link, useParams } from "react-router-dom"; import { Link, useParams } from "react-router-dom";
import { IconAlertTriangle, IconBell, IconCheck, IconClockPause, IconHandStop, IconLink, IconNote, IconPlayerPlay, IconSettings } from "@tabler/icons-react"; import { IconAlertTriangle, IconBell, IconCheck, IconClockPause, IconHandStop, IconLink, IconNote, IconPlayerPlay, IconSend, IconSettings } from "@tabler/icons-react";
import { api, type CurrentUser, type IncidentDetail as Detail, type TimelineEvent, type TimelineKind } from "../api/client"; import { api, type CurrentUser, type IncidentDetail as Detail, type TimelineEvent, type TimelineKind } from "../api/client";
import { ackWords } from "../components/AckDrawer"; import { ackWords } from "../components/AckDrawer";
import RunbookEntry from "../components/RunbookEntry"; import RunbookEntry from "../components/RunbookEntry";
@@ -24,6 +24,7 @@ const KIND_ICON: Record<TimelineKind, JSX.Element> = {
run: <IconPlayerPlay size={16} />, run: <IconPlayerPlay size={16} />,
related: <IconLink size={16} />, related: <IconLink size={16} />,
note: <IconNote size={16} />, note: <IconNote size={16} />,
notify: <IconSend size={16} />,
}; };
const TONE_COLOR: Record<TimelineEvent["tone"], string> = { good: "bg-green-lt text-green", bad: "bg-red-lt text-red", neutral: "bg-secondary-lt text-secondary", info: "bg-blue-lt text-blue" }; const TONE_COLOR: Record<TimelineEvent["tone"], string> = { good: "bg-green-lt text-green", bad: "bg-red-lt text-red", neutral: "bg-secondary-lt text-secondary", info: "bg-blue-lt text-blue" };