Add a notification history

Every notification is written down with how it went through each channel
(or why it was not sent: held for quiet hours, a routing rule said nobody, no
channel on). Records -> Notifications lists them with a filter, a search and
Send again. Errors are cleaned of every secret in the settings; kept 30 days.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-10 19:18:08 +01:00
1 parent 5c6ccc0a4b
commit 9126200295
18 files changed
+9886 -34

No files matched your search

+15
View File
@@ -38,6 +38,21 @@ it:
Where a trigger has a configurable threshold, that's noted in its row Where a trigger has a configurable threshold, that's noted in its row
below; several are fixed and can't be changed from the UI. below; several are fixed and can't be changed from the UI.
## Notification history
**Records → Notifications** (operators and administrators; also linked from Settings → Notifications) lists everything that was sent, newest first, and answers "why did I not get it?". Each entry says what it said and how it went through **each channel** — a ✓ or ✕ beside Gotify, ntfy, Telegram, Pushover, Email, Webhook or Push, with what the channel answered when it refused — and one of these:
| Status | Meaning |
|---|---|
| **Sent** | Every channel it went to took it |
| **Partly sent** | Some took it, some refused or could not be reached |
| **Failed** | No channel took it |
| **Nowhere to send it** | No channel was switched on for it |
| **Held** | Quiet hours or away mode held it for the digest (the digest is its own entry) |
| **Not sent (rule)** | A routing rule says nobody should be told about that kind |
Filter by status or search the title and message. **Send again** (**Send now** for a held one) sends the entry once more to every channel that is on, writes that down as a new entry and is in the change log; it does not carry the buttons an alert message had. Messages are kept 30 days (at most 1000 entries), cut at 4000 characters, and an error is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back. The *Send test* buttons in Settings are not listed.
## Who hears what ## Who hears what
Under **Settings → Notifications → Who hears what**. Under **Settings → Notifications → Who hears what**.
+2
View File
@@ -950,6 +950,8 @@ switched by `?view=`, so a section can be linked to: an alert about a failing se
**The status strip.** Under a server's name, on every section of its page, a row of figures says how it is at a glance: **Status** (online, with when it last reported — or *offline*, with when it did, and a note when Proxmox says the guest is still running, so a stopped agent is not read as a dead machine; for a server with no agent report, what Proxmox says, or that nothing has reported), **Maintenance** (only while a window on this server is on, with when it ends and why), **Alerts** (how many are open about this server and how bad — what a maintenance window, a snooze or a closing has put aside is not counted, but said — and a link to the Alerts page), **Updates** (what is waiting, how many of them security fixes, whether a reboot is needed), **Disk** (the fullest, against its own limit: green, yellow within ten points of it, red at it; a mount with its own limit is judged by that, and an ignored mount is left out), **CPU · memory** and **how long it has run** (for a guest Proxmox reads, or from an agent of version 8 on Linux, 7 on Windows, which say it in every report; left out while the server is offline, since it is then no longer how long it has been up). A figure nothing is known for is left out or says so — it is never filled in. It uses what the page already has plus the alerts and the maintenance windows, read again whenever the page is (the *Refresh* button). Everyone signed in sees it. **The status strip.** Under a server's name, on every section of its page, a row of figures says how it is at a glance: **Status** (online, with when it last reported — or *offline*, with when it did, and a note when Proxmox says the guest is still running, so a stopped agent is not read as a dead machine; for a server with no agent report, what Proxmox says, or that nothing has reported), **Maintenance** (only while a window on this server is on, with when it ends and why), **Alerts** (how many are open about this server and how bad — what a maintenance window, a snooze or a closing has put aside is not counted, but said — and a link to the Alerts page), **Updates** (what is waiting, how many of them security fixes, whether a reboot is needed), **Disk** (the fullest, against its own limit: green, yellow within ten points of it, red at it; a mount with its own limit is judged by that, and an ignored mount is left out), **CPU · memory** and **how long it has run** (for a guest Proxmox reads, or from an agent of version 8 on Linux, 7 on Windows, which say it in every report; left out while the server is offline, since it is then no longer how long it has been up). A figure nothing is known for is left out or says so — it is never filled in. It uses what the page already has plus the alerts and the maintenance windows, read again whenever the page is (the *Refresh* button). Everyone signed in sees it.
**Notification history.** Records → Notifications lists every notification that was sent, with how it went through each channel (or why it was not sent: held for quiet hours, a routing rule said nobody, no channel switched on), the error a channel gave when it refused, a filter and a search, and **Send again**. Kept 30 days. See NOTIFICATIONS.md.
**A shorter menu for a smaller homelab.** The menu has a page for every kind of integration the app can read, whether or not you run it. **Settings → Display → Menu → Hide the pages of integrations I have not added** (off until you switch it on, the same for everyone) leaves out of the menu, the command palette and the `g`-shortcuts the pages whose integrations are not in **Administration → Integrations**: Gitea, n8n, Grafana … as the list under the switch says (a disabled integration still counts as added; any one of Sonarr, Radarr, Lidarr, Readarr or Prowlarr keeps *Media managers*, and a Proxmox or a Backup Server keeps *Proxmox backups*). Pages that are useful without an integration — Servers, Alerts, DNS, Endpoints, Secrets, Inventory and the rest — always stay, a group with nothing left disappears, and a hidden page is still there by its address. Adding an integration brings its page back at once. **A shorter menu for a smaller homelab.** The menu has a page for every kind of integration the app can read, whether or not you run it. **Settings → Display → Menu → Hide the pages of integrations I have not added** (off until you switch it on, the same for everyone) leaves out of the menu, the command palette and the `g`-shortcuts the pages whose integrations are not in **Administration → Integrations**: Gitea, n8n, Grafana … as the list under the switch says (a disabled integration still counts as added; any one of Sonarr, Radarr, Lidarr, Readarr or Prowlarr keeps *Media managers*, and a Proxmox or a Backup Server keeps *Proxmox backups*). Pages that are useful without an integration — Servers, Alerts, DNS, Endpoints, Secrets, Inventory and the rest — always stay, a group with nothing left disappears, and a hidden page is still there by its address. Adding an integration brings its page back at once.
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts **Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
+12
View File
@@ -0,0 +1,12 @@
CREATE TABLE `notification_log` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`at` text NOT NULL,
`title` text NOT NULL,
`message` text NOT NULL,
`status` text NOT NULL,
`results` text DEFAULT '[]' NOT NULL,
`note` text,
`resent_of` integer
);
--> statement-breakpoint
CREATE INDEX `notification_log_at_idx` ON `notification_log` (`at`);
File diff suppressed because it is too large. Load diff
+7
View File
@@ -631,6 +631,13 @@
"when": 1794263161221, "when": 1794263161221,
"tag": "0089_military_tombstone", "tag": "0089_military_tombstone",
"breakpoints": true "breakpoints": true
},
{
"idx": 90,
"version": "6",
"when": 1794333852784,
"tag": "0090_watery_mathemanic",
"breakpoints": true
} }
] ]
} }
+2
View File
@@ -19,6 +19,7 @@ import { dependenciesRouter } from "./routes/dependencies.js";
import { pushRouter } from "./routes/push.js"; import { pushRouter } from "./routes/push.js";
import { endpointsRouter } from "./routes/endpoints.js"; import { endpointsRouter } from "./routes/endpoints.js";
import { changesRouter } from "./routes/changes.js"; import { changesRouter } from "./routes/changes.js";
import { notificationLogRouter } from "./routes/notificationLog.js";
import { backupsRouter } from "./routes/backups.js"; import { backupsRouter } from "./routes/backups.js";
import { metricsRouter, metricsTokensRouter } from "./routes/metrics.js"; import { metricsRouter, metricsTokensRouter } from "./routes/metrics.js";
import { eventWebhooksRouter } from "./routes/eventWebhooks.js"; import { eventWebhooksRouter } from "./routes/eventWebhooks.js";
@@ -175,6 +176,7 @@ export function createApp() {
app.use("/api/push", pushRouter); app.use("/api/push", pushRouter);
app.use("/api/endpoints", endpointsRouter); app.use("/api/endpoints", endpointsRouter);
app.use("/api/changes", changesRouter); app.use("/api/changes", changesRouter);
app.use("/api/notification-log", notificationLogRouter);
app.use("/api/backups", backupsRouter); app.use("/api/backups", backupsRouter);
app.use("/api/metrics-tokens", metricsTokensRouter); app.use("/api/metrics-tokens", metricsTokensRouter);
app.use("/api/event-webhooks", eventWebhooksRouter); app.use("/api/event-webhooks", eventWebhooksRouter);
+16
View File
@@ -1662,6 +1662,22 @@ export const jellyfinReadings = sqliteTable(
// ─── Duplicati ────────────────────────────────────────────────────────────── // ─── Duplicati ──────────────────────────────────────────────────────────────
// What was sent: one row for each notification, with how it went through each channel, so "why did I not get it?" has an answer. Kept 30 days and at most 1000 rows (services/notificationLog.ts).
export const notificationLog = sqliteTable(
"notification_log",
{
id: integer("id").primaryKey({ autoIncrement: true }),
at: text("at").notNull(), // ISO
title: text("title").notNull(),
message: text("message").notNull(), // cut at 4000 characters
status: text("status").notNull(), // sent | partly | failed | none (no channel to send it to) | held (quiet hours or away) | muted (a routing rule says nobody)
results: text("results").notNull().default("[]"), // JSON: [{ channel, ok, error }]
note: text("note"), // why it was held or muted
resentOf: integer("resent_of"), // the entry it is a resend of
},
(t) => [index("notification_log_at_idx").on(t.at)],
);
// A Duplicati that reports to this app: one row per machine running it, with the secret token its reports carry in their address (only the hash is kept). Duplicati pushes a // A Duplicati that reports to this app: one row per machine running it, with the secret token its reports carry in their address (only the hash is kept). Duplicati pushes a
// report after every operation (its own "send-http-url" option); services/duplicati.ts reads them. // report after every operation (its own "send-http-url" option); services/duplicati.ts reads them.
export const duplicatiSources = sqliteTable("duplicati_sources", { export const duplicatiSources = sqliteTable("duplicati_sources", {
+29
View File
@@ -0,0 +1,29 @@
import { Router } from "express";
import { requireAuth, requireRole } from "../auth/middleware.js";
import { recordAudit } from "../services/audit.js";
import { resendLogged } from "../services/notify.js";
import { listLogged, STATUSES, type LogStatus } from "../services/notificationLog.js";
import { asyncHandler } from "../utils/asyncHandler.js";
export const notificationLogRouter = Router();
// What was sent says what is going on in the lab, and why a channel is failing: operators and administrators, like the change log.
notificationLogRouter.use(requireAuth, requireRole("operator"));
// ?status=failed &q=word &limit=200
notificationLogRouter.get("/", asyncHandler(async (req, res) => {
const status = STATUSES.find((s) => s === req.query.status) as LogStatus | undefined;
const limit = Number(req.query.limit);
const entries = await listLogged({ status, q: typeof req.query.q === "string" ? req.query.q.slice(0, 100) : undefined, limit: Number.isFinite(limit) ? limit : 200 });
res.json({ entries, statuses: STATUSES });
}));
// Sends one again, to every channel that is on (not to the audience it first had, and without the buttons an alert carried), and writes down how that went.
notificationLogRouter.post("/:id/resend", asyncHandler(async (req, res) => {
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
const sent = await resendLogged(id);
if (!sent) return res.status(404).json({ error: "not_found" });
await recordAudit({ actor: req.currentUser!, category: "notification", action: "resend", targetType: "notification", targetId: id, detail: { newEntry: sent.id } });
res.json({ ok: true, entry: sent.id });
}));
+3
View File
@@ -47,6 +47,7 @@ const KIND_OF_CATEGORY: Record<string, ChangeKind> = {
todo: "other", todo: "other",
note: "servers", note: "servers",
webhook: "settings", webhook: "settings",
notification: "settings",
integration: "integrations", integration: "integrations",
dns: "integrations", dns: "integrations",
secret: "integrations", secret: "integrations",
@@ -194,6 +195,8 @@ export function describeChange(category: string, action: string, detail: Record<
return { title: `${typeof detail.changed === "number" ? detail.changed : "Several"} snoozed alert${detail.changed === 1 ? "" : "s"} woken at once`, tone: "neutral", detail: null }; return { title: `${typeof detail.changed === "number" ? detail.changed : "Several"} snoozed alert${detail.changed === 1 ? "" : "s"} woken at once`, tone: "neutral", detail: null };
case "inventory/bulk_change": case "inventory/bulk_change":
return { title: detail.change === "undo" ? "A change to several pieces of hardware was undone" : `${typeof detail.changed === "number" ? detail.changed : "Several"} piece${detail.changed === 1 ? "" : "s"} of hardware changed at once`, tone: "neutral", detail: genericSummary(detail, ["ids", "changed", "change"]) }; return { title: detail.change === "undo" ? "A change to several pieces of hardware was undone" : `${typeof detail.changed === "number" ? detail.changed : "Several"} piece${detail.changed === 1 ? "" : "s"} of hardware changed at once`, tone: "neutral", detail: genericSummary(detail, ["ids", "changed", "change"]) };
case "notification/resend":
return { title: "A notification was sent again", tone: "neutral", detail: null };
case "webhook/retry": case "webhook/retry":
return { title: `A webhook message was sent again${name ? `: ${name}` : str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: detail.ok === false ? "bad" : "neutral", detail: str(detail.event) }; return { title: `A webhook message was sent again${name ? `: ${name}` : str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: detail.ok === false ? "bad" : "neutral", detail: str(detail.event) };
case "duplicati/create_source": case "duplicati/create_source":
+104
View File
@@ -0,0 +1,104 @@
/**
* What was sent: every notification is written down with how it went through each channel, so "why did I not get it?" has an answer — it went through ntfy and Gotify said 401, it was held
* for quiet hours, a routing rule said nobody should hear it, or no channel was switched on to send it to. Kept 30 days and at most 1000 entries. Nothing here can make sending fail: writing
* it down is best-effort, like the channels themselves. An error's text is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back.
*/
import { and, desc, eq, like, lt, notInArray, or } from "drizzle-orm";
import { db } from "../db/client.js";
import { notificationLog } from "../db/schema.js";
import { getSettings } from "./settingsStore.js";
export const KEEP_DAYS = 30;
export const KEEP_ROWS = 1000;
export const MAX_MESSAGE = 4000;
export const MAX_ERROR = 300;
export const STATUSES = ["sent", "partly", "failed", "none", "held", "muted"] as const;
export type LogStatus = (typeof STATUSES)[number];
export interface ChannelResult {
/** gotify, ntfy, telegram, pushover, smtp, webhook, webPush, or email (an escalation to one address). */
channel: string;
ok: boolean;
error: string | null;
}
export type LogRow = typeof notificationLog.$inferSelect;
const clip = (text: string, max: number) => (text.length <= max ? text : `${text.slice(0, max - 1)}…`);
/** Pure: how a notification went, from how each channel did. Nothing switched on to send it to is "none". */
export function statusOf(results: ChannelResult[]): LogStatus {
if (results.length === 0) return "none";
const ok = results.filter((r) => r.ok).length;
return ok === results.length ? "sent" : ok === 0 ? "failed" : "partly";
}
/** Pure: text with each of these values (at least six characters, so a stray short word is not swallowed) replaced by dots. */
export function scrub(text: string, secrets: (string | null | undefined)[]): string {
let out = text;
for (const s of [...new Set(secrets.filter((x): x is string => typeof x === "string" && x.length >= 6))].sort((a, b) => b.length - a.length)) {
out = out.split(s).join("•••");
// A token inside an address is also written percent-encoded.
const encoded = encodeURIComponent(s);
if (encoded !== s) out = out.split(encoded).join("•••");
}
return out;
}
async function secretsNow(): Promise<string[]> {
const s = await getSettings();
return [s.gotify.token, s.ntfy.token, s.telegram.token, s.pushover.appToken, s.pushover.userKey, s.smtp.password, s.webhook.secret, s.webhook.url, s.telegram.chatId].filter((x): x is string => typeof x === "string");
}
let sinceTrim = 0;
/** Keeps the log to KEEP_DAYS and KEEP_ROWS: old entries first, and only now and then the count. */
async function trim(now: Date): Promise<void> {
await db.delete(notificationLog).where(lt(notificationLog.at, new Date(now.getTime() - KEEP_DAYS * 86_400_000).toISOString()));
if (++sinceTrim < 50) return;
sinceTrim = 0;
const keep = await db.select({ id: notificationLog.id }).from(notificationLog).orderBy(desc(notificationLog.at), desc(notificationLog.id)).limit(KEEP_ROWS);
if (keep.length === KEEP_ROWS) await db.delete(notificationLog).where(notInArray(notificationLog.id, keep.map((k) => k.id)));
}
/** Writes down one notification: what it said, and how it went (or why it was held or muted). Returns its id, or null when it could not be written — never throws. */
export async function recordNotification(entry: { title: string; message: string; results: ChannelResult[]; status?: LogStatus; note?: string | null; resentOf?: number | null }, now: Date = new Date()): Promise<number | null> {
try {
const secrets = await secretsNow();
const results = entry.results.map((r) => ({ channel: r.channel, ok: r.ok, error: r.error === null ? null : clip(scrub(r.error.replace(/\s+/g, " ").trim(), secrets), MAX_ERROR) }));
const [row] = await db
.insert(notificationLog)
.values({ at: now.toISOString(), title: clip(entry.title, 300), message: clip(entry.message, MAX_MESSAGE), status: entry.status ?? statusOf(results), results: JSON.stringify(results), note: entry.note ?? null, resentOf: entry.resentOf ?? null })
.returning({ id: notificationLog.id });
await trim(now);
return row.id;
} catch (err) {
console.error("[notificationLog] couldn't write it down:", err instanceof Error ? err.message : err);
return null;
}
}
export function resultsOf(row: Pick<LogRow, "results">): ChannelResult[] {
try {
const v = JSON.parse(row.results);
return Array.isArray(v) ? (v as ChannelResult[]) : [];
} catch {
return [];
}
}
export async function getLogged(id: number): Promise<LogRow | null> {
return (await db.select().from(notificationLog).where(eq(notificationLog.id, id)).limit(1))[0] ?? null;
}
/** The newest first, optionally of one status, or with a word in the title or message. */
export async function listLogged(options: { status?: LogStatus; q?: string; limit?: number } = {}): Promise<(Omit<LogRow, "results"> & { results: ChannelResult[] })[]> {
const q = options.q?.trim();
const rows = await db
.select()
.from(notificationLog)
.where(and(options.status ? eq(notificationLog.status, options.status) : undefined, q ? or(like(notificationLog.title, `%${q.replace(/[%_]/g, "")}%`), like(notificationLog.message, `%${q.replace(/[%_]/g, "")}%`)) : undefined))
.orderBy(desc(notificationLog.at), desc(notificationLog.id))
.limit(Math.min(500, Math.max(1, Math.round(options.limit ?? 200))));
return rows.map((r) => ({ ...r, results: resultsOf(r) }));
}
+83 -32
View File
@@ -12,12 +12,18 @@ import { dampRaised, dampRecovered } from "./flapGuard.js";
import { withoutSnoozed } from "./alertSnoozes.js"; import { withoutSnoozed } from "./alertSnoozes.js";
import { actionsFor, footerText, ntfyActionsHeader, type NotifyActions } from "./alertActionLinks.js"; import { actionsFor, footerText, ntfyActionsHeader, type NotifyActions } from "./alertActionLinks.js";
import { firstMatchingRule, metaOfConditionKey, pushAccepts, type ChannelId, type NotifyMeta } from "./notifyRouting.js"; import { firstMatchingRule, metaOfConditionKey, pushAccepts, type ChannelId, type NotifyMeta } from "./notifyRouting.js";
import { getLogged, recordNotification, type ChannelResult } from "./notificationLog.js";
// ─── Channel senders (best-effort — failures are logged, never thrown) ────── // ─── Channel senders (best-effort — failures are logged, never thrown) ──────
async function sendGotify(title: string, message: string, actions?: NotifyActions | null) { /** How one channel did: null when it is not switched on (or has nobody to send to), so there is nothing to say about it. */
type SendOutcome = { ok: true } | { ok: false; error: string } | null;
const OK: SendOutcome = { ok: true };
const failure = (error: string): SendOutcome => ({ ok: false, error });
async function sendGotify(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { gotify } = await getSettings(); const { gotify } = await getSettings();
if (!gotify.enabled || !gotify.url || !gotify.token) return; if (!gotify.enabled || !gotify.url || !gotify.token) return null;
const base = gotify.url.replace(/\/$/, ""); const base = gotify.url.replace(/\/$/, "");
try { try {
const res = await fetch(`${base}/message?token=${encodeURIComponent(gotify.token)}`, { const res = await fetch(`${base}/message?token=${encodeURIComponent(gotify.token)}`, {
@@ -26,9 +32,15 @@ async function sendGotify(title: string, message: string, actions?: NotifyAction
// Gotify has no buttons: a tap opens the page with them (and the address is in the text, for a client that does not tap). // Gotify has no buttons: a tap opens the page with them (and the address is in the text, for a client that does not tap).
body: JSON.stringify({ title, message: actions ? message + footerText(actions) : message, priority: gotify.priority ?? 5, ...(actions ? { extras: { "client::notification": { click: { url: actions.pageUrl } } } } : {}) }), body: JSON.stringify({ title, message: actions ? message + footerText(actions) : message, priority: gotify.priority ?? 5, ...(actions ? { extras: { "client::notification": { click: { url: actions.pageUrl } } } } : {}) }),
}); });
if (!res.ok) console.error(`[notify] Gotify error ${res.status}: ${await res.text().catch(() => "")}`); if (!res.ok) {
const error = `Gotify error ${res.status}: ${await res.text().catch(() => "")}`;
console.error(`[notify] ${error}`);
return failure(error);
}
return OK;
} catch (err) { } catch (err) {
console.error("[notify] Gotify failed:", err instanceof Error ? err.message : err); console.error("[notify] Gotify failed:", err instanceof Error ? err.message : err);
return failure(`Gotify failed: ${err instanceof Error ? err.message : err}`);
} }
} }
@@ -41,9 +53,9 @@ export function headerSafe(text: string): string {
return /^[ -~]*$/.test(text) ? text : `=?UTF-8?B?${Buffer.from(text, "utf8").toString("base64")}?=`; return /^[ -~]*$/.test(text) ? text : `=?UTF-8?B?${Buffer.from(text, "utf8").toString("base64")}?=`;
} }
async function sendNtfy(title: string, message: string, actions?: NotifyActions | null) { async function sendNtfy(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { ntfy } = await getSettings(); const { ntfy } = await getSettings();
if (!ntfy.enabled || !ntfy.url || !ntfy.topic) return; if (!ntfy.enabled || !ntfy.url || !ntfy.topic) return null;
const base = ntfy.url.replace(/\/$/, ""); const base = ntfy.url.replace(/\/$/, "");
const headers: Record<string, string> = { const headers: Record<string, string> = {
"Content-Type": "text/plain", "Content-Type": "text/plain",
@@ -57,9 +69,15 @@ async function sendNtfy(title: string, message: string, actions?: NotifyActions
} }
try { try {
const res = await fetch(`${base}/${encodeURIComponent(ntfy.topic)}`, { method: "POST", headers, body: message }); const res = await fetch(`${base}/${encodeURIComponent(ntfy.topic)}`, { method: "POST", headers, body: message });
if (!res.ok) console.error(`[notify] ntfy error ${res.status}: ${await res.text().catch(() => "")}`); if (!res.ok) {
const error = `ntfy error ${res.status}: ${await res.text().catch(() => "")}`;
console.error(`[notify] ${error}`);
return failure(error);
}
return OK;
} catch (err) { } catch (err) {
console.error("[notify] ntfy failed:", err instanceof Error ? err.message : err); console.error("[notify] ntfy failed:", err instanceof Error ? err.message : err);
return failure(`ntfy failed: ${err instanceof Error ? err.message : err}`);
} }
} }
@@ -122,29 +140,33 @@ export async function pushoverSend(cfg: { appToken: string; userKey: string; dev
if (!res.ok || json?.status !== 1) throw new Error(`Pushover: ${json?.errors?.length ? json.errors.join(", ") : `HTTP ${res.status}`}`); if (!res.ok || json?.status !== 1) throw new Error(`Pushover: ${json?.errors?.length ? json.errors.join(", ") : `HTTP ${res.status}`}`);
} }
async function sendTelegram(title: string, message: string, actions?: NotifyActions | null) { async function sendTelegram(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { telegram } = await getSettings(); const { telegram } = await getSettings();
if (!telegram.enabled || !telegram.token || !telegram.chatId) return; if (!telegram.enabled || !telegram.token || !telegram.chatId) return null;
try { try {
await telegramSend(telegram, `${title}\n\n${message}`, actions); await telegramSend(telegram, `${title}\n\n${message}`, actions);
return OK;
} catch (err) { } catch (err) {
console.error("[notify]", err instanceof Error ? err.message : "Telegram failed"); console.error("[notify]", err instanceof Error ? err.message : "Telegram failed");
return failure(err instanceof Error ? err.message : "Telegram failed");
} }
} }
async function sendPushover(title: string, message: string, actions?: NotifyActions | null) { async function sendPushover(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { pushover } = await getSettings(); const { pushover } = await getSettings();
if (!pushover.enabled || !pushover.appToken || !pushover.userKey) return; if (!pushover.enabled || !pushover.appToken || !pushover.userKey) return null;
try { try {
await pushoverSend(pushover, title, message, actions); await pushoverSend(pushover, title, message, actions);
return OK;
} catch (err) { } catch (err) {
console.error("[notify]", err instanceof Error ? err.message : "Pushover failed"); console.error("[notify]", err instanceof Error ? err.message : "Pushover failed");
return failure(err instanceof Error ? err.message : "Pushover failed");
} }
} }
async function sendSmtp(title: string, message: string, actions?: NotifyActions | null) { async function sendSmtp(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { smtp } = await getSettings(); const { smtp } = await getSettings();
if (!smtp.enabled || !smtp.host || !smtp.to || !smtp.from) return; if (!smtp.enabled || !smtp.host || !smtp.to || !smtp.from) return null;
try { try {
const transporter = nodemailer.createTransport({ const transporter = nodemailer.createTransport({
host: smtp.host, host: smtp.host,
@@ -153,14 +175,16 @@ async function sendSmtp(title: string, message: string, actions?: NotifyActions
auth: smtp.username ? { user: smtp.username, pass: smtp.password } : undefined, auth: smtp.username ? { user: smtp.username, pass: smtp.password } : undefined,
}); });
await transporter.sendMail({ from: smtp.from, to: smtp.to, subject: title, text: actions ? message + footerText(actions) : message }); await transporter.sendMail({ from: smtp.from, to: smtp.to, subject: title, text: actions ? message + footerText(actions) : message });
return OK;
} catch (err) { } catch (err) {
console.error("[notify] SMTP failed:", err instanceof Error ? err.message : err); console.error("[notify] SMTP failed:", err instanceof Error ? err.message : err);
return failure(`SMTP failed: ${err instanceof Error ? err.message : err}`);
} }
} }
async function sendWebhook(title: string, message: string, actions?: NotifyActions | null) { async function sendWebhook(title: string, message: string, actions?: NotifyActions | null): Promise<SendOutcome> {
const { webhook } = await getSettings(); const { webhook } = await getSettings();
if (!webhook.enabled || !webhook.url) return; if (!webhook.enabled || !webhook.url) return null;
const headers: Record<string, string> = { "Content-Type": "application/json" }; const headers: Record<string, string> = { "Content-Type": "application/json" };
if (webhook.secret) headers["X-Webhook-Secret"] = webhook.secret; if (webhook.secret) headers["X-Webhook-Secret"] = webhook.secret;
try { try {
@@ -169,32 +193,42 @@ async function sendWebhook(title: string, message: string, actions?: NotifyActio
headers, headers,
body: JSON.stringify({ content: `**${title}**\n${message}${actions ? footerText(actions) : ""}` }), body: JSON.stringify({ content: `**${title}**\n${message}${actions ? footerText(actions) : ""}` }),
}); });
if (!res.ok) console.error(`[notify] Webhook error ${res.status}: ${await res.text().catch(() => "")}`); if (!res.ok) {
const error = `Webhook error ${res.status}: ${await res.text().catch(() => "")}`;
console.error(`[notify] ${error}`);
return failure(error);
}
return OK;
} catch (err) { } catch (err) {
console.error("[notify] Webhook failed:", err instanceof Error ? err.message : err); console.error("[notify] Webhook failed:", err instanceof Error ? err.message : err);
return failure(`Webhook failed: ${err instanceof Error ? err.message : err}`);
} }
} }
async function sendWebPush(title: string, message: string, only: number[] | null, actions?: NotifyActions | null) { async function sendWebPush(title: string, message: string, only: number[] | null, actions?: NotifyActions | null): Promise<SendOutcome> {
const { webPush } = await getSettings(); const { webPush } = await getSettings();
if (!webPush.enabled) return; if (!webPush.enabled) return null;
// For one alert, buttons on the notification itself (browsers show two); for several, a tap opens the app. // For one alert, buttons on the notification itself (browsers show two); for several, a tap opens the app.
const buttons = actions?.direct.slice(0, 2).map((d) => ({ action: d.id, title: d.label, path: d.path })); const buttons = actions?.direct.slice(0, 2).map((d) => ({ action: d.id, title: d.label, path: d.path }));
const message_ = { title, body: message, ...(buttons?.length ? { actions: buttons } : {}) }; const message_ = { title, body: message, ...(buttons?.length ? { actions: buttons } : {}) };
try { try {
if (only === null) await sendPush(message_); if (only === null) await sendPush(message_);
else if (only.length > 0) await sendPush(message_, await db.select().from(pushSubscriptions).where(inArray(pushSubscriptions.id, only))); else if (only.length > 0) await sendPush(message_, await db.select().from(pushSubscriptions).where(inArray(pushSubscriptions.id, only)));
// No browser wants it: nothing was asked of the channel.
else return null;
return OK;
} catch (err) { } catch (err) {
console.error("[notify] Web push failed:", err instanceof Error ? err.message : err); console.error("[notify] Web push failed:", err instanceof Error ? err.message : err);
return failure(`Web push failed: ${err instanceof Error ? err.message : err}`);
} }
} }
/** Email to one address, through the email channel's server, whatever the channel's own recipient is (an escalation to someone else). */ /** Email to one address, through the email channel's server, whatever the channel's own recipient is (an escalation to someone else). */
async function sendExtraEmail(title: string, message: string, to: string) { async function sendExtraEmail(title: string, message: string, to: string): Promise<SendOutcome> {
const { smtp } = await getSettings(); const { smtp } = await getSettings();
if (!smtp.host || !smtp.from) { if (!smtp.host || !smtp.from) {
console.error("[notify] An escalation wants to email someone, but email isn't set up."); console.error("[notify] An escalation wants to email someone, but email isn't set up.");
return; return failure("An escalation wants to email someone, but email isn't set up.");
} }
try { try {
const transporter = nodemailer.createTransport({ const transporter = nodemailer.createTransport({
@@ -204,8 +238,10 @@ async function sendExtraEmail(title: string, message: string, to: string) {
auth: smtp.username ? { user: smtp.username, pass: smtp.password } : undefined, auth: smtp.username ? { user: smtp.username, pass: smtp.password } : undefined,
}); });
await transporter.sendMail({ from: smtp.from, to, subject: title, text: message }); await transporter.sendMail({ from: smtp.from, to, subject: title, text: message });
return OK;
} catch (err) { } catch (err) {
console.error("[notify] Extra email failed:", err instanceof Error ? err.message : err); console.error("[notify] Extra email failed:", err instanceof Error ? err.message : err);
return failure(`Email failed: ${err instanceof Error ? err.message : err}`);
} }
} }
@@ -249,18 +285,21 @@ export async function audienceFor(meta?: NotifyMeta): Promise<Audience> {
return { channels, pushIds }; return { channels, pushIds };
} }
async function sendToChannels(title: string, message: string, audience: Audience = EVERYONE, actions: NotifyActions | null = null): Promise<void> { /** Sends to the channels the audience names (and are switched on), and writes down how each went: the entry's id, or null if it could not be written. */
async function sendToChannels(title: string, message: string, audience: Audience = EVERYONE, actions: NotifyActions | null = null, resentOf: number | null = null): Promise<number | null> {
const to = (c: ChannelId) => audience.channels === null || audience.channels.includes(c); const to = (c: ChannelId) => audience.channels === null || audience.channels.includes(c);
await Promise.all([ const jobs: [string, Promise<SendOutcome>][] = [];
to("gotify") ? sendGotify(title, message, actions) : undefined, if (to("gotify")) jobs.push(["gotify", sendGotify(title, message, actions)]);
to("ntfy") ? sendNtfy(title, message, actions) : undefined, if (to("ntfy")) jobs.push(["ntfy", sendNtfy(title, message, actions)]);
to("telegram") ? sendTelegram(title, message, actions) : undefined, if (to("telegram")) jobs.push(["telegram", sendTelegram(title, message, actions)]);
to("pushover") ? sendPushover(title, message, actions) : undefined, if (to("pushover")) jobs.push(["pushover", sendPushover(title, message, actions)]);
to("smtp") ? sendSmtp(title, message, actions) : undefined, if (to("smtp")) jobs.push(["smtp", sendSmtp(title, message, actions)]);
to("webhook") ? sendWebhook(title, message, actions) : undefined, if (to("webhook")) jobs.push(["webhook", sendWebhook(title, message, actions)]);
to("webPush") ? sendWebPush(title, message, audience.pushIds, actions) : undefined, if (to("webPush")) jobs.push(["webPush", sendWebPush(title, message, audience.pushIds, actions)]);
audience.email ? sendExtraEmail(title, actions ? message + footerText(actions) : message, audience.email) : undefined, if (audience.email) jobs.push(["email", sendExtraEmail(title, actions ? message + footerText(actions) : message, audience.email)]);
]); const outcomes = await Promise.all(jobs.map(async ([channel, run]) => [channel, await run] as const));
const results: ChannelResult[] = outcomes.filter(([, o]) => o !== null).map(([channel, o]) => ({ channel, ok: o!.ok, error: o!.ok ? null : o!.error }));
return recordNotification({ title, message, results, resentOf });
} }
/** /**
@@ -268,10 +307,15 @@ async function sendToChannels(title: string, message: string, audience: Audience
* said nobody. Escalations use this with their own channels. * said nobody. Escalations use this with their own channels.
*/ */
export async function notifyAudience(title: string, message: string, audience: Audience, options: { quietHours?: boolean; actions?: NotifyActions | null; critical?: boolean } = {}): Promise<void> { export async function notifyAudience(title: string, message: string, audience: Audience, options: { quietHours?: boolean; actions?: NotifyActions | null; critical?: boolean } = {}): Promise<void> {
if (isMuted(audience)) return; if (isMuted(audience)) {
await recordNotification({ title, message, results: [], status: "muted", note: "A routing rule says nobody should be told." });
return;
}
// Held when quiet hours or an away stretch say so — unless it is critical and they let critical through (services/availability.ts). // Held when quiet hours or an away stretch say so — unless it is critical and they let critical through (services/availability.ts).
if (options.quietHours !== false && (await holdReason(options.critical === true))) { const held = options.quietHours !== false ? await holdReason(options.critical === true) : null;
if (held) {
await enqueueNotification(title, message, audience.channels === null && audience.pushIds === null && !audience.email ? null : JSON.stringify(audience)); await enqueueNotification(title, message, audience.channels === null && audience.pushIds === null && !audience.email ? null : JSON.stringify(audience));
await recordNotification({ title, message, results: [], status: "held", note: held === "away" ? "Held while you are away; it is sent in the digest when you are back." : "Held for quiet hours; it is sent in the digest when they end." });
return; return;
} }
await sendToChannels(title, message, audience, options.actions ?? null); await sendToChannels(title, message, audience, options.actions ?? null);
@@ -300,6 +344,13 @@ export async function notifyNow(title: string, message: string): Promise<void> {
await sendToChannels(title, message); await sendToChannels(title, message);
} }
/** Sends a notification that was logged once more, to every channel that is on, and writes down the new attempt: its entry, or null if there is no such notification. */
export async function resendLogged(id: number): Promise<{ id: number | null } | null> {
const original = await getLogged(id);
if (!original) return null;
return { id: await sendToChannels(original.title, original.message, EVERYONE, null, id) };
}
/** How many channels are switched on and filled in enough to send, so a "send it now" can say when there is nowhere to send to. */ /** How many channels are switched on and filled in enough to send, so a "send it now" can say when there is nowhere to send to. */
export async function enabledChannelCount(): Promise<number> { export async function enabledChannelCount(): Promise<number> {
const { gotify, ntfy, telegram, pushover, smtp, webhook, webPush } = await getSettings(); const { gotify, ntfy, telegram, pushover, smtp, webhook, webPush } = await getSettings();
+261
View File
@@ -0,0 +1,261 @@
/**
* The notification history: every notification written down with how each channel took it, or why it was not sent (held for quiet hours, a routing rule said nobody, no channel to
* send it to); nothing secret kept in an error; a notification sent again; who may read it; and how long it is kept.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer as createHttpServer, type Server } from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-notiflog-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, auditLog, notificationLog } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const log = await import("../src/services/notificationLog.js");
const { notify, notifyAudience } = await import("../src/services/notify.js");
const { updateSettings } = await import("../src/services/settingsStore.js");
const { notificationLogRouter } = await import("../src/routes/notificationLog.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const NTFY_TOKEN = "tk_secret_ntfy_value";
describe("pure parts", () => {
test("how a notification went, from how each channel did", () => {
assert.equal(log.statusOf([]), "none");
assert.equal(log.statusOf([{ channel: "ntfy", ok: true, error: null }]), "sent");
assert.equal(log.statusOf([{ channel: "ntfy", ok: true, error: null }, { channel: "gotify", ok: false, error: "x" }]), "partly");
assert.equal(log.statusOf([{ channel: "ntfy", ok: false, error: "x" }, { channel: "gotify", ok: false, error: "y" }]), "failed");
});
test("a secret is wiped from a text, also when it is percent-encoded, and a short word is left alone", () => {
assert.equal(log.scrub("bad token abcdef123 here", ["abcdef123"]), "bad token ••• here");
assert.equal(log.scrub("GET /x?token=a%2Fb%2Fc%2Fd%2Fe", ["a/b/c/d/e"]), "GET /x?token=•••");
assert.equal(log.scrub("the cat sat", ["cat"]), "the cat sat", "under six characters is not a secret worth wiping");
assert.equal(log.scrub("one two", [null, undefined, ""]), "one two");
assert.equal(log.scrub("abcdef and abcdefghi", ["abcdef", "abcdefghi"]), "••• and •••", "the longer first, so none of it is left behind");
});
});
describe("what is written down", () => {
let okHook: Server;
let badHook: Server;
let okUrl = "";
let badUrl = "";
const received: string[] = [];
const entries = () => log.listLogged({ limit: 500 });
before(async () => {
okHook = createHttpServer((req, res) => {
let body = "";
req.on("data", (c) => (body += c));
req.on("end", () => {
received.push(body);
res.writeHead(200).end("ok");
});
});
// A channel that refuses, and says the token back in what it says.
badHook = createHttpServer((req, res) => {
req.resume();
res.writeHead(401).end(`unauthorised: ${NTFY_TOKEN} is not accepted`);
});
await Promise.all([okHook, badHook].map((s) => new Promise<void>((resolve) => s.listen(0, "127.0.0.1", resolve))));
okUrl = `http://127.0.0.1:${(okHook.address() as AddressInfo).port}`;
badUrl = `http://127.0.0.1:${(badHook.address() as AddressInfo).port}`;
});
after(async () => {
okHook.close();
badHook.close();
await updateSettings({ webhook: { enabled: false }, ntfy: { enabled: false }, routing: { rules: [] }, quietHours: { enabled: false, start: "22:00", end: "07:00", criticalThrough: false } });
});
test("with nothing switched on, a notification is written down as having nowhere to go", async () => {
await updateSettings({ webhook: { enabled: false }, ntfy: { enabled: false } });
await notify("Homelab Manager — Nothing on", "no channels");
const [e] = await entries();
assert.deepEqual([e.title, e.status, e.results], ["Homelab Manager — Nothing on", "none", []]);
});
test("one that went through is sent, with the channel that took it", async () => {
await updateSettings({ webhook: { enabled: true, url: `${okUrl}/hook` } });
await notify("Homelab Manager — Disk full", "pve1 is at 98%");
const [e] = await entries();
assert.deepEqual([e.title, e.message, e.status, e.results.map((r) => [r.channel, r.ok, r.error])], ["Homelab Manager — Disk full", "pve1 is at 98%", "sent", [["webhook", true, null]]]);
assert.equal(received.length >= 1, true);
});
test("a channel that refuses is failed, with what it said — and without the token it said back", async () => {
await updateSettings({ webhook: { enabled: false }, ntfy: { enabled: true, url: badUrl, topic: "lab", token: NTFY_TOKEN } });
await notify("Homelab Manager — Server down", "web-1 has not reported");
const [e] = await entries();
assert.equal(e.status, "failed");
assert.equal(e.results[0].channel, "ntfy");
assert.match(e.results[0].error ?? "", /ntfy error 401: unauthorised: ••• is not accepted/);
assert.ok(!JSON.stringify(e).includes(NTFY_TOKEN), "the token is nowhere in the entry");
});
test("one channel working and one refusing is partly sent, and each says how it did", async () => {
await updateSettings({ webhook: { enabled: true, url: `${okUrl}/hook` }, ntfy: { enabled: true, url: badUrl, topic: "lab", token: NTFY_TOKEN } });
await notify("Homelab Manager — Half", "one of two");
const [e] = await entries();
assert.equal(e.status, "partly");
assert.deepEqual(e.results.map((r) => [r.channel, r.ok]).sort(), [["ntfy", false], ["webhook", true]]);
});
test("a routing rule that says nobody is written down as muted, with why", async () => {
await updateSettings({ routing: { rules: [{ id: "q", name: "quiet", enabled: true, categories: ["backup"], minSeverity: null, tags: [], channels: [] }] } });
const before = received.length;
await notify("Homelab Manager — Backup late", "x", { category: "backup", severity: "warning" });
const [e] = await entries();
assert.deepEqual([e.title, e.status, e.results], ["Homelab Manager — Backup late", "muted", []]);
assert.match(e.note ?? "", /routing rule says nobody/);
assert.equal(received.length, before, "and nothing was sent");
await updateSettings({ routing: { rules: [] } });
});
test("one held for quiet hours is written down as held, and the digest that follows is its own entry", async () => {
const h = new Date().getUTCHours();
const pad = (n: number) => String(((n % 24) + 24) % 24).padStart(2, "0") + ":00";
await updateSettings({ webhook: { enabled: true, url: `${okUrl}/hook` }, ntfy: { enabled: false }, quietHours: { enabled: true, start: pad(h - 1), end: pad(h + 1), criticalThrough: false }, notifications: { timezone: "UTC" } as never });
await notify("Homelab Manager — Held one", "wait for morning", { severity: "warning" });
const [e] = await entries();
assert.deepEqual([e.title, e.status], ["Homelab Manager — Held one", "held"]);
assert.match(e.note ?? "", /quiet hours/);
await updateSettings({ quietHours: { enabled: false, start: "22:00", end: "07:00", criticalThrough: false } });
});
test("a message is cut at 4000 characters, a title at 300, and an error at 300", async () => {
const id = await log.recordNotification({ title: "T".repeat(500), message: "M".repeat(9000), results: [{ channel: "ntfy", ok: false, error: "E".repeat(900) }] });
const row = (await db.select().from(notificationLog).where(eq(notificationLog.id, id!)))[0];
assert.deepEqual([row.title.length, row.message.length, JSON.parse(row.results)[0].error.length], [300, 4000, 300]);
});
test("it is a list newest first that can be narrowed to one status or a word", async () => {
const all = await entries();
assert.ok(all.length >= 6);
assert.deepEqual(all.map((e) => e.at), [...all.map((e) => e.at)].sort().reverse(), "newest first");
const failed = await log.listLogged({ status: "failed" });
assert.ok(failed.length >= 1 && failed.every((e) => e.status === "failed"));
const word = await log.listLogged({ q: "Disk full" });
assert.deepEqual(word.map((e) => e.title), ["Homelab Manager — Disk full"]);
assert.deepEqual(await log.listLogged({ q: "no such words anywhere" }), []);
assert.equal((await log.listLogged({ q: "100%" })).length, 0, "a percent sign in a search is not a wildcard");
});
test("it never keeps more than 30 days, and never makes sending fail", async () => {
const now = new Date();
await db.insert(notificationLog).values({ at: new Date(now.getTime() - 40 * 86_400_000).toISOString(), title: "old", message: "old", status: "sent" });
await log.recordNotification({ title: "fresh", message: "x", results: [] }, now);
assert.equal((await db.select().from(notificationLog).where(eq(notificationLog.title, "old"))).length, 0, "the old one is gone");
// A notification that cannot be written down is still sent: the writer swallows its own failures.
assert.equal(await log.recordNotification({ title: "x", message: "y", results: [], status: "sent" }, new Date("not a date")), null);
});
});
describe("the page's route", () => {
let app: Server;
let base = "";
let who = "operator";
let hook: Server;
let hookUrl = "";
let hookHits = 0;
before(async () => {
await db.insert(users).values([
{ oidcSub: "admin", name: "Ada", role: "admin" },
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
hook = createHttpServer((req, res) => {
req.resume();
hookHits++;
res.writeHead(200).end("ok");
});
await new Promise<void>((resolve) => hook.listen(0, "127.0.0.1", resolve));
hookUrl = `http://127.0.0.1:${(hook.address() as AddressInfo).port}/hook`;
await updateSettings({ webhook: { enabled: true, url: hookUrl }, ntfy: { enabled: false } });
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/notification-log", notificationLogRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/notification-log`;
});
after(async () => {
app.close();
hook.close();
await updateSettings({ webhook: { enabled: false } });
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("operators and administrators read it; a viewer does not", async () => {
for (const [role, status] of [["viewer", 403], ["operator", 200], ["admin", 200]] as const) {
who = role;
assert.equal((await fetch(base)).status, status, role);
}
who = "operator";
const body = (await (await fetch(base)).json()) as { entries: { title: string; results: unknown[] }[]; statuses: string[] };
assert.ok(body.entries.length > 0);
assert.ok(Array.isArray(body.entries[0].results));
assert.deepEqual(body.statuses, ["sent", "partly", "failed", "none", "held", "muted"]);
});
test("the list can be narrowed by status and by a word, and an unknown status is ignored", async () => {
who = "operator";
const failed = (await (await fetch(`${base}?status=failed`)).json()) as { entries: { status: string }[] };
assert.ok(failed.entries.length > 0 && failed.entries.every((e) => e.status === "failed"));
const word = (await (await fetch(`${base}?q=Disk%20full`)).json()) as { entries: { title: string }[] };
assert.deepEqual(word.entries.map((e) => e.title), ["Homelab Manager — Disk full"]);
const ignored = (await (await fetch(`${base}?status=bogus`)).json()) as { entries: unknown[] };
assert.ok(ignored.entries.length > 5);
});
test("a notification can be sent again: to every channel that is on, written down as a new entry, and in the audit log", async () => {
who = "operator";
const first = (await log.listLogged({ q: "Disk full" }))[0];
const hits = hookHits;
const res = await fetch(`${base}/${first.id}/resend`, { method: "POST" });
assert.equal(res.status, 200);
const { entry } = (await res.json()) as { entry: number };
assert.equal(hookHits, hits + 1, "it went to the channel that is on");
const again = (await db.select().from(notificationLog).where(eq(notificationLog.id, entry)))[0];
assert.deepEqual([again.title, again.message, again.status, again.resentOf], [first.title, first.message, "sent", first.id]);
const audit = (await db.select().from(auditLog)).filter((e) => e.category === "notification" && e.action === "resend");
assert.equal(audit.length, 1);
});
test("a viewer cannot send one again, a notification that is not there is not found, and an id that is not one is refused", async () => {
who = "viewer";
assert.equal((await fetch(`${base}/1/resend`, { method: "POST" })).status, 403);
who = "operator";
assert.equal((await fetch(`${base}/999999/resend`, { method: "POST" })).status, 404);
assert.equal((await fetch(`${base}/abc/resend`, { method: "POST" })).status, 400);
});
test("notifyAudience with an audience that names no channel is written down as muted too (an escalation to nobody)", async () => {
await notifyAudience("Homelab Manager — Escalation", "to nobody", { channels: [], pushIds: null });
const [e] = await log.listLogged();
assert.deepEqual([e.title, e.status], ["Homelab Manager — Escalation", "muted"]);
});
});
+2
View File
@@ -63,6 +63,7 @@ import Heartbeats from "./pages/Heartbeats";
import ComingUp from "./pages/ComingUp"; import ComingUp from "./pages/ComingUp";
import MonthlyReport from "./pages/MonthlyReport"; import MonthlyReport from "./pages/MonthlyReport";
import ChangeLog from "./pages/ChangeLog"; import ChangeLog from "./pages/ChangeLog";
import NotificationHistory from "./pages/NotificationHistory";
import BackupsHub from "./pages/BackupsHub"; import BackupsHub from "./pages/BackupsHub";
import Consistency from "./pages/Consistency"; import Consistency from "./pages/Consistency";
import Privacy from "./pages/Privacy"; import Privacy from "./pages/Privacy";
@@ -220,6 +221,7 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/coming-up" element={<ComingUp user={user} />} /> <Route path="/coming-up" element={<ComingUp user={user} />} />
<Route path="/reports" element={<MonthlyReport user={user} />} /> <Route path="/reports" element={<MonthlyReport user={user} />} />
<Route path="/changes" element={user.role === "viewer" ? <Navigate to="/" replace /> : <ChangeLog />} /> <Route path="/changes" element={user.role === "viewer" ? <Navigate to="/" replace /> : <ChangeLog />} />
<Route path="/notification-history" element={user.role === "viewer" ? <Navigate to="/" replace /> : <NotificationHistory />} />
<Route path="/backups" element={<BackupsHub user={user} />} /> <Route path="/backups" element={<BackupsHub user={user} />} />
<Route path="/docker" element={<Docker user={user} />} /> <Route path="/docker" element={<Docker user={user} />} />
<Route path="/tailscale" element={<Tailscale user={user} />} /> <Route path="/tailscale" element={<Tailscale user={user} />} />
+22
View File
@@ -2235,6 +2235,20 @@ export interface Agenda {
} }
export type ChangeKind = "servers" | "maintenance" | "alerts" | "settings" | "integrations" | "dependencies" | "inventory" | "backups" | "endpoints" | "other"; export type ChangeKind = "servers" | "maintenance" | "alerts" | "settings" | "integrations" | "dependencies" | "inventory" | "backups" | "endpoints" | "other";
/** One notification as the history lists it: what it said and how each channel took it. */
export interface NotificationEntry {
id: number;
at: string;
title: string;
message: string;
/** sent, partly, failed, none (no channel to send it to), held (quiet hours or away) or muted (a routing rule says nobody). */
status: string;
results: { channel: string; ok: boolean; error: string | null }[];
/** Why it was held or muted. */
note: string | null;
resentOf: number | null;
}
export interface ChangeEntry { export interface ChangeEntry {
id: string; id: string;
at: string; at: string;
@@ -3824,6 +3838,14 @@ export const api = {
return request<{ entries: ChangeEntry[]; more: boolean; kinds: { id: ChangeKind; label: string }[] }>(`/api/changes?${q.toString()}`); return request<{ entries: ChangeEntry[]; more: boolean; kinds: { id: ChangeKind; label: string }[] }>(`/api/changes?${q.toString()}`);
}, },
}, },
notificationLog: {
list: (params: { status?: string; q?: string; limit?: number }) => {
const q = new URLSearchParams();
for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== "") q.set(k, String(v));
return request<{ entries: NotificationEntry[]; statuses: string[] }>(`/api/notification-log?${q.toString()}`);
},
resend: (id: number) => request<{ ok: boolean; entry: number | null }>(`/api/notification-log/${id}/resend`, { method: "POST" }),
},
push: { push: {
key: () => request<{ publicKey: string }>("/api/push/key"), key: () => request<{ publicKey: string }>("/api/push/key"),
mine: () => request<{ enabled: boolean; subscriptions: PushDevice[] }>("/api/push/mine"), mine: () => request<{ enabled: boolean; subscriptions: PushDevice[] }>("/api/push/mine"),
+2
View File
@@ -48,6 +48,7 @@ import {
IconBox, IconBox,
IconHeartbeat, IconHeartbeat,
IconGitCommit, IconGitCommit,
IconBellRinging,
IconShieldLock, IconShieldLock,
IconWorldWww, IconWorldWww,
IconVersions, IconVersions,
@@ -196,6 +197,7 @@ export const NAV: NavEntry[] = [
{ to: "/reports", label: "Reports", icon: <IconReportAnalytics size={20} /> }, { to: "/reports", label: "Reports", icon: <IconReportAnalytics size={20} /> },
{ to: "/costs", label: "Costs", icon: <IconCoin size={20} /> }, { to: "/costs", label: "Costs", icon: <IconCoin size={20} /> },
{ to: "/changes", label: "Change log", icon: <IconGitCommit size={20} />, minRole: "operator" }, { to: "/changes", label: "Change log", icon: <IconGitCommit size={20} />, minRole: "operator" },
{ to: "/notification-history", label: "Notifications", icon: <IconBellRinging size={20} />, minRole: "operator" },
], ],
}, },
{ {
+176
View File
@@ -0,0 +1,176 @@
import { useEffect, useState } from "react";
import { Link, useSearchParams } from "react-router-dom";
import { api, type NotificationEntry } from "../api/client";
import EmptyState from "../components/EmptyState";
import { ErrorNotice } from "../components/Notice";
import { BlockSkeleton } from "../components/Skeleton";
import TimeAgo from "../components/TimeAgo";
import { formatDateTime } from "../utils/date";
import { readableError } from "../utils/errors";
import { channelName, isLogStatus, STATUS_BADGE, STATUS_HINT, STATUS_LABEL, STATUS_ORDER, type LogStatus } from "../utils/notificationLog";
import { toast } from "../utils/toast";
/**
* What was sent: every notification, newest first, with how it went through each channel — or why it did not: held for quiet hours, a routing rule said nobody, no channel to send it to.
* "Send again" sends it once more to every channel that is on. Kept 30 days.
*/
export default function NotificationHistory() {
const [params, setParams] = useSearchParams();
const status = isLogStatus(params.get("status") ?? "") ? (params.get("status") as LogStatus) : "";
const q = params.get("q") ?? "";
const [text, setText] = useState(q);
const [entries, setEntries] = useState<NotificationEntry[] | null>(null);
const [error, setError] = useState<string | null>(null);
const [resending, setResending] = useState<number | null>(null);
const [tick, setTick] = useState(0);
const setParam = (key: string, value: string | null) =>
setParams(
(prev) => {
const next = new URLSearchParams(prev);
if (value) next.set(key, value);
else next.delete(key);
return next;
},
{ replace: true },
);
// What is searched for follows what is typed, a moment after it stops.
useEffect(() => {
const t = setTimeout(() => setParam("q", text.trim() || null), 300);
return () => clearTimeout(t);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [text]);
useEffect(() => {
let alive = true;
api.notificationLog
.list({ status, q })
.then((res) => {
if (!alive) return;
setEntries(res.entries);
setError(null);
})
.catch((err) => alive && setError(readableError(err)));
return () => {
alive = false;
};
}, [status, q, tick]);
async function resend(e: NotificationEntry) {
setResending(e.id);
try {
await api.notificationLog.resend(e.id);
toast.success("Sent again to every channel that is on.");
setTick((n) => n + 1);
} catch (err) {
toast.error(readableError(err));
} finally {
setResending(null);
}
}
const failing = entries?.filter((e) => e.status === "failed" || e.status === "partly").length ?? 0;
return (
<>
<div className="d-flex align-items-center mb-1">
<h2 className="page-title mb-0">Notification history</h2>
</div>
<p className="text-secondary">
What was sent, to which channel, and whether it got through — or why it was not sent. Kept 30 days. The channels are set up under <Link to="/settings/notifications">Settings → Notifications</Link>.
</p>
<div className="row g-2 mb-3">
<div className="col-sm-4 col-md-3">
<select className="form-select" aria-label="Show" value={status} onChange={(e) => setParam("status", e.target.value || null)}>
<option value="">Everything</option>
{STATUS_ORDER.map((s) => (
<option key={s} value={s}>
{STATUS_LABEL[s]}
</option>
))}
</select>
</div>
<div className="col-sm-8 col-md-5">
<input className="form-control" type="search" aria-label="Search the history" placeholder="Search the title or the message" value={text} onChange={(e) => setText(e.target.value)} />
</div>
</div>
<ErrorNotice error={error} />
{!entries && !error && <BlockSkeleton lines={5} />}
{entries && entries.length === 0 && (
<div className="card">
<div className="card-body">
<EmptyState title={status || q ? "Nothing matches" : "Nothing has been sent yet"} hint={status || q ? "Try another filter." : "Every notification is listed here from now on, with how each channel took it."} />
</div>
</div>
)}
{entries && entries.length > 0 && (
<div className="card">
<div className="card-header">
<h3 className="card-title">
{entries.length} notification{entries.length === 1 ? "" : "s"}
{failing > 0 && <span className="text-red fw-normal ms-2">{failing} did not get through everywhere</span>}
</h3>
</div>
<div className="table-responsive">
<table className="table table-vcenter card-table">
<thead>
<tr>
<th>When</th>
<th>Notification</th>
<th>How it went</th>
<th className="w-1">Actions</th>
</tr>
</thead>
<tbody>
{entries.map((e) => {
const known = isLogStatus(e.status);
return (
<tr key={e.id}>
<td className="text-nowrap">
<span title={formatDateTime(new Date(e.at))}>
<TimeAgo value={e.at} />
</span>
</td>
<td style={{ maxWidth: 420 }}>
<div className="fw-bold text-break">{e.title}</div>
{e.resentOf !== null && <div className="text-secondary small">sent again from an earlier one</div>}
<details className="small">
<summary className="text-secondary">Show what it said</summary>
<pre className="mb-0 text-break" style={{ whiteSpace: "pre-wrap", maxHeight: 220, overflow: "auto" }}>
{e.message}
</pre>
</details>
</td>
<td>
<span className={`badge ${known ? STATUS_BADGE[e.status as LogStatus] : "bg-secondary-lt"}`} title={known ? STATUS_HINT[e.status as LogStatus] : undefined}>
{known ? STATUS_LABEL[e.status as LogStatus] : e.status}
</span>
{e.note && <div className="text-secondary small">{e.note}</div>}
{e.results.length > 0 && (
<ul className="list-unstyled small mb-0 mt-1">
{e.results.map((r, i) => (
<li key={i} className={r.ok ? "text-secondary" : "text-red"}>
<span aria-hidden="true">{r.ok ? "✓" : "✕"}</span> {channelName(r.channel)}
{!r.ok && r.error ? <span className="text-break"> — {r.error}</span> : null}
</li>
))}
</ul>
)}
</td>
<td>
<button className="btn btn-sm" disabled={resending === e.id} onClick={() => void resend(e)} title="Send it once more to every channel that is on">
{resending === e.id ? "Sending…" : e.status === "held" ? "Send now" : "Send again"}
</button>
</td>
</tr>
);
})}
</tbody>
</table>
</div>
</div>
)}
</>
);
}
@@ -1,6 +1,6 @@
import AwayCard from "./AwayCard"; import AwayCard from "./AwayCard";
import { useEffect, useRef, useState, type KeyboardEvent, type ReactNode } from "react"; import { useEffect, useRef, useState, type KeyboardEvent, type ReactNode } from "react";
import { useSearchParams } from "react-router-dom"; import { Link, useSearchParams } from "react-router-dom";
import AlertActionsCard from "./AlertActionsCard"; import AlertActionsCard from "./AlertActionsCard";
import { import {
api, api,
@@ -546,7 +546,12 @@ export default function NotificationSettings() {
<> <>
<div className="settings-savebar"> <div className="settings-savebar">
<div className="d-flex flex-wrap align-items-center justify-content-between gap-2"> <div className="d-flex flex-wrap align-items-center justify-content-between gap-2">
<h3 className="mb-0">Notifications</h3> <h3 className="mb-0">
Notifications
<Link className="ms-3 fs-5 fw-normal" to="/notification-history">
See what was sent
</Link>
</h3>
<div className="d-flex align-items-center gap-2"> <div className="d-flex align-items-center gap-2">
{saveError && <span className="text-danger small">{saveError}</span>} {saveError && <span className="text-danger small">{saveError}</span>}
{saved && <span className="text-success small">✓ Settings saved</span>} {saved && <span className="text-success small">✓ Settings saved</span>}
+50
View File
@@ -0,0 +1,50 @@
/** How a notification went, as the history page says it: one label and one colour for each status, and a name for each channel. */
export type LogStatus = "sent" | "partly" | "failed" | "none" | "held" | "muted";
export const STATUS_LABEL: Record<LogStatus, string> = {
sent: "Sent",
partly: "Partly sent",
failed: "Failed",
none: "Nowhere to send it",
held: "Held",
muted: "Not sent (rule)",
};
/** What each status is, in a sentence: for the filter and for a tooltip. */
export const STATUS_HINT: Record<LogStatus, string> = {
sent: "Every channel it went to took it.",
partly: "Some channels took it and some refused or could not be reached.",
failed: "No channel took it.",
none: "No channel was switched on to send it to.",
held: "Held for quiet hours or while you were away; it goes in the digest.",
muted: "A routing rule says nobody should be told about this kind.",
};
export const STATUS_BADGE: Record<LogStatus, string> = {
sent: "bg-green-lt text-green",
partly: "bg-yellow-lt text-yellow",
failed: "bg-red-lt text-red",
none: "bg-secondary-lt",
held: "bg-blue-lt text-blue",
muted: "bg-secondary-lt",
};
/** The order the filter lists them in: what needs a look first. */
export const STATUS_ORDER: LogStatus[] = ["failed", "partly", "none", "held", "muted", "sent"];
const CHANNEL_NAME: Record<string, string> = {
gotify: "Gotify",
ntfy: "ntfy",
telegram: "Telegram",
pushover: "Pushover",
smtp: "Email",
webhook: "Webhook",
webPush: "Push",
email: "Email (escalation)",
};
export const channelName = (channel: string): string => CHANNEL_NAME[channel] ?? channel;
/** Whether a stored status is one this page knows (an unknown one is shown as it is, never hidden). */
export const isLogStatus = (s: string): s is LogStatus => s in STATUS_LABEL;