Notification routing rules, per-browser push wishes, escalation of unacknowledged alerts
- Routing: rules pick channels by kind, severity and server tags (first match wins, none = nobody); mixed batches split per route; recoveries follow their alert; quiet-hours digest keeps each notification's audience - Push: each subscribed browser can ask for a minimum severity and some kinds - Escalation: up to three timed steps re-notify alerts nobody has acknowledged, through chosen channels and an optional extra email; an acknowledgement stops them - Fix ntfy: titles with an em dash made every ntfy send fail (header not Latin-1); now RFC 2047 encoded Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9caa414829
commit
43fc5e0272
21 files changed
+4766
-52
No files matched your search
@@ -134,6 +134,7 @@ cleared.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `id` | integer PK | |
|
| `id` | integer PK | |
|
||||||
| `title`, `message` | text NOT NULL | |
|
| `title`, `message` | text NOT NULL | |
|
||||||
|
| `audience` | text | JSON of who it was for (`channels`, `pushIds`, `email`) when a routing rule or an escalation picked them; null = everyone. The digest is sent to each audience separately. |
|
||||||
| `created_at` | text NOT NULL | SQLite UTC. |
|
| `created_at` | text NOT NULL | SQLite UTC. |
|
||||||
|
|
||||||
#### `maintenance_windows`
|
#### `maintenance_windows`
|
||||||
@@ -526,6 +527,8 @@ One time an alert was raised and later went away. When the same alert comes back
|
|||||||
| `resolved_at` | text | ISO; null while open. Set to when it first went missing, once it has been missing long enough. |
|
| `resolved_at` | text | ISO; null while open. Set to when it first went missing, once it has been missing long enough. |
|
||||||
| `last_seen_at` | text NOT NULL | ISO. |
|
| `last_seen_at` | text NOT NULL | ISO. |
|
||||||
| `missing_since` | text | ISO; set while it has been missing from the last look(s), before it is called over. |
|
| `missing_since` | text | ISO; set while it has been missing from the last look(s), before it is called over. |
|
||||||
|
| `escalated_step` | integer NOT NULL, default 0 | The highest escalation step already sent for this incident (0 = none); a new incident starts again. |
|
||||||
|
| `escalated_at` | text | ISO; when that step was sent. |
|
||||||
|
|
||||||
Resolved rows older than 400 days are deleted, with their notes.
|
Resolved rows older than 400 days are deleted, with their notes.
|
||||||
|
|
||||||
@@ -614,6 +617,8 @@ A browser or phone that asked for push notifications, one row per device.
|
|||||||
| `label` | text | What the person calls it, or the browser it is. |
|
| `label` | text | What the person calls it, or the browser it is. |
|
||||||
| `created_at` | text NOT NULL | ISO. |
|
| `created_at` | text NOT NULL | ISO. |
|
||||||
| `last_success_at` | text | When the push service last accepted a message for it. |
|
| `last_success_at` | text | When the push service last accepted a message for it. |
|
||||||
|
| `min_severity` | text | `warning` or `critical`: this browser only wants alerts at least that bad. Null = every severity. |
|
||||||
|
| `categories` | text | JSON array of the kinds of alert it wants (`offline`, `disk`, …). Null = every kind. |
|
||||||
|
|
||||||
A subscription is removed when the push service says it is gone (HTTP 404 or 410), when its owner or an administrator removes it, and all of them are removed if the server's own key pair has to be made again.
|
A subscription is removed when the push service says it is gone (HTTP 404 or 410), when its owner or an administrator removes it, and all of them are removed if the server's own key pair has to be made again.
|
||||||
|
|
||||||
@@ -714,6 +719,8 @@ that were never changed aren't stored; the app fills in defaults when reading.
|
|||||||
| Key | Holds |
|
| Key | Holds |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `gotify`, `ntfy`, `smtp`, `webhook` | The four notification channels: enabled, address, and credentials (token / password / secret). The credential fields are stored encrypted (prefix `enc:v1:`) — see [Retention and backups](#retention-and-backups). |
|
| `gotify`, `ntfy`, `smtp`, `webhook` | The four notification channels: enabled, address, and credentials (token / password / secret). The credential fields are stored encrypted (prefix `enc:v1:`) — see [Retention and backups](#retention-and-backups). |
|
||||||
|
| `routing` | Routing rules: `rules`, an ordered array of `{id, name, enabled, categories, minSeverity, tags, channels}` — the first enabled rule a notification matches decides which channels get it (empty `channels` = nobody). |
|
||||||
|
| `escalation` | `enabled` (false) and up to three `steps` of `{afterMinutes, minSeverity, channels, extraEmail, evenInQuietHours}`, in time order. |
|
||||||
| `webPush` | Push to phones and browsers: `enabled` (false) and `contact` (an email or https address the push services may reach whoever runs this on; blank is fine). Who is subscribed is in `push_subscriptions`; the server's key pair is below. |
|
| `webPush` | Push to phones and browsers: `enabled` (false) and `contact` (an email or https address the push services may reach whoever runs this on; blank is fine). Who is subscribed is in `push_subscriptions`; the server's key pair is below. |
|
||||||
| `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). |
|
| `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). |
|
||||||
| `quietHours` | `enabled`, `start`, `end`. |
|
| `quietHours` | `enabled`, `start`, `end`. |
|
||||||
|
|||||||
+35
-4
@@ -3,15 +3,15 @@
|
|||||||
Every notification in Homelab Manager is sent through the same pipe
|
Every notification in Homelab Manager is sent through the same pipe
|
||||||
(`server/src/services/notify.ts`) to whichever channels are enabled under
|
(`server/src/services/notify.ts`) to whichever channels are enabled under
|
||||||
**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
**Settings → Notifications**: Gotify, ntfy, SMTP (email), a generic
|
||||||
JSON webhook, and push to phones and browsers. All five fire for every notification below — there's no
|
JSON webhook, and push to phones and browsers. All five fire for every notification below unless a
|
||||||
per-event channel routing, only a per-event on/off toggle and, for the
|
**routing rule** (see *Who hears what* below) says otherwise; each event also has its on/off toggle and, for the
|
||||||
daily ones, one shared time/timezone.
|
daily ones, one shared time/timezone.
|
||||||
|
|
||||||
The settings page has three tabs, with one Save button for all of them (it
|
The settings page has four tabs, with one Save button for all of them (it
|
||||||
shows "Unsaved changes" until you save, and the browser asks before the page
|
shows "Unsaved changes" until you save, and the browser asks before the page
|
||||||
is closed with changes unsaved): **Channels** (the five channels, each a row
|
is closed with changes unsaved): **Channels** (the five channels, each a row
|
||||||
you open to fill in and send a test), **What to send** (the event toggles
|
you open to fill in and send a test), **What to send** (the event toggles
|
||||||
grouped by topic, each group's thresholds beside its switches) and **When**
|
grouped by topic, each group's thresholds beside its switches), **Who hears what** (routing rules and escalation) and **When**
|
||||||
(the time zone, the daily reminder time, quiet hours and the weekly digest).
|
(the time zone, the daily reminder time, quiet hours and the weekly digest).
|
||||||
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
|
The address keeps the tab, e.g. `/settings/notifications?tab=when`.
|
||||||
|
|
||||||
@@ -34,6 +34,36 @@ it:
|
|||||||
Where a trigger has a configurable threshold, that's noted in its row
|
Where a trigger has a configurable threshold, that's noted in its row
|
||||||
below; several are fixed and can't be changed from the UI.
|
below; several are fixed and can't be changed from the UI.
|
||||||
|
|
||||||
|
## Who hears what
|
||||||
|
|
||||||
|
Under **Settings → Notifications → Who hears what**.
|
||||||
|
|
||||||
|
**Routing rules.** By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of:
|
||||||
|
the **kind** of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how
|
||||||
|
**bad** it is (warning or worse, critical only) and the **tags** of the server it is about (any of them). **The first rule that matches decides** which
|
||||||
|
channels get it — put the specific rules first — and a rule with no channels tells nobody (the Alerts page still shows the problem, and webhooks for
|
||||||
|
events still fire). A notification no rule matches goes to every channel that is on, as before. Rules can be switched off and reordered.
|
||||||
|
|
||||||
|
- A message that holds problems for different routes is split: a disk warning can go to ntfy while the server outage in the same check goes to
|
||||||
|
everything, as two messages. When all of them take the same route it is one message, as ever.
|
||||||
|
- A recovery follows the route its alert took (the health check remembers how bad each problem was).
|
||||||
|
- What a rule can see: health and automation problems carry a kind, a severity and (for servers) the server; secret, domain and Tailscale expiry, backup
|
||||||
|
checks, Docker image updates, update runs and integration failures carry a kind and a severity. DNS changes, the weekly digest and similar carry
|
||||||
|
neither, so only rules that ask about nothing in particular match them.
|
||||||
|
- Notifications held for quiet hours keep who they were for: the digest at the end goes only to the channels their rule picked.
|
||||||
|
|
||||||
|
**Each browser's own wishes.** Under *Phone and browser*, every subscribed browser can say it only wants warnings or worse, or only critical alerts,
|
||||||
|
and only some kinds of alert — a phone for the critical things, a desktop for everything. It applies after routing: a rule that sends something to
|
||||||
|
*Phone and browser* still doesn't reach a browser that turned that kind down. A notification that says nothing about its kind or severity reaches all.
|
||||||
|
|
||||||
|
**Escalation.** When an alert has been raised for a while and **nobody has acknowledged it**, it is told again, in up to three steps — for example *after
|
||||||
|
30 minutes: push; after 2 hours: email the neighbour who holds the key*. Each step names its time, the least severity it applies to, the channels to use
|
||||||
|
(chosen freely — routing rules don't apply to escalations), an optional extra email address to tell as well (sent through the email channel's server) and
|
||||||
|
whether to send during quiet hours (otherwise it waits for them to end like any other). Every step is sent once per alert; an acknowledgement stops the
|
||||||
|
rest, and the alert starts over if it goes away and comes back. Problems under maintenance, held back because something they depend on is down, or only
|
||||||
|
informational never escalate. It is looked at every five minutes, so a step can arrive a few minutes after its time. Every escalation is written in the audit
|
||||||
|
log (category *alert*, action *escalate*).
|
||||||
|
|
||||||
## Push to a phone or browser
|
## Push to a phone or browser
|
||||||
|
|
||||||
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
|
The fifth channel, **Phone and browser**, sends the same messages as the others to browsers that have asked for them — a phone with
|
||||||
@@ -48,6 +78,7 @@ with a button to send a test and one to remove each.
|
|||||||
it is needed (RFC 8292, "VAPID"), then handed to the browser's own push service — Google's, Mozilla's or Apple's — which delivers
|
it is needed (RFC 8292, "VAPID"), then handed to the browser's own push service — Google's, Mozilla's or Apple's — which delivers
|
||||||
it. The push service sees that a message is being sent and its size, not what it says. The address of each browser's push
|
it. The push service sees that a message is being sent and its size, not what it says. The address of each browser's push
|
||||||
service is checked before use: https only, and never an address on a private network.
|
service is checked before use: https only, and never an address on a private network.
|
||||||
|
- **Browsers' own wishes** (a minimum severity, which kinds) are set per browser under the channel's row.
|
||||||
- **Quiet hours** hold push messages like every other channel, and each event's on/off toggle applies. A tap on a notification opens
|
- **Quiet hours** hold push messages like every other channel, and each event's on/off toggle applies. A tap on a notification opens
|
||||||
the Alerts page.
|
the Alerts page.
|
||||||
- **A browser that has gone** (the site's permission was removed, the app uninstalled) is dropped automatically when its push
|
- **A browser that has gone** (the site's permission was removed, the app uninstalled) is dropped automatically when its push
|
||||||
|
|||||||
@@ -436,6 +436,12 @@ so it is folded only when every enabled instance of that kind is explained. A se
|
|||||||
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
|
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
|
||||||
alerts of their own, so they aren't part of this.)
|
alerts of their own, so they aren't part of this.)
|
||||||
|
|
||||||
|
**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
|
||||||
|
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
|
||||||
|
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
|
||||||
|
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
|
||||||
|
rest). See NOTIFICATIONS.md.
|
||||||
|
|
||||||
**Seeing the impact before maintenance.** Choosing a server or integration under Maintenance → Start maintenance lists what depends on it and
|
**Seeing the impact before maintenance.** Choosing a server or integration under Maintenance → Start maintenance lists what depends on it and
|
||||||
will look down while it is — and offers to put those under maintenance too, for the same time, with the reason "Depends on …" (ticked by default;
|
will look down while it is — and offers to put those under maintenance too, for the same time, with the reason "Depends on …" (ticked by default;
|
||||||
each is its own window, listed and ended like any other, and audited). The recurring-schedule form shows the same note for what is picked and can add
|
each is its own window, listed and ended like any other, and audited). The recurring-schedule form shows the same note for what is picked and can add
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
ALTER TABLE `alert_incidents` ADD `escalated_step` integer DEFAULT 0 NOT NULL;--> statement-breakpoint
|
||||||
|
ALTER TABLE `alert_incidents` ADD `escalated_at` text;--> statement-breakpoint
|
||||||
|
ALTER TABLE `notification_queue` ADD `audience` text;--> statement-breakpoint
|
||||||
|
ALTER TABLE `push_subscriptions` ADD `min_severity` text;--> statement-breakpoint
|
||||||
|
ALTER TABLE `push_subscriptions` ADD `categories` text;
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -260,6 +260,13 @@
|
|||||||
"when": 1792341456680,
|
"when": 1792341456680,
|
||||||
"tag": "0036_cheerful_slipstream",
|
"tag": "0036_cheerful_slipstream",
|
||||||
"breakpoints": true
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 37,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1792344782975,
|
||||||
|
"tag": "0037_past_magik",
|
||||||
|
"breakpoints": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -64,6 +64,8 @@ export const notificationQueue = sqliteTable("notification_queue", {
|
|||||||
id: integer("id").primaryKey({ autoIncrement: true }),
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
title: text("title").notNull(),
|
title: text("title").notNull(),
|
||||||
message: text("message").notNull(),
|
message: text("message").notNull(),
|
||||||
|
/** JSON: who it is for (the channels and browsers a routing rule picked); null = everyone, as before routing existed. */
|
||||||
|
audience: text("audience"),
|
||||||
createdAt: text("created_at")
|
createdAt: text("created_at")
|
||||||
.notNull()
|
.notNull()
|
||||||
.default(sql`(current_timestamp)`),
|
.default(sql`(current_timestamp)`),
|
||||||
@@ -119,6 +121,10 @@ export const pushSubscriptions = sqliteTable("push_subscriptions", {
|
|||||||
.notNull()
|
.notNull()
|
||||||
.default(sql`(current_timestamp)`),
|
.default(sql`(current_timestamp)`),
|
||||||
lastSuccessAt: text("last_success_at"),
|
lastSuccessAt: text("last_success_at"),
|
||||||
|
/** This browser only wants alerts at least this bad ("warning" or "critical"); null = every severity. */
|
||||||
|
minSeverity: text("min_severity"),
|
||||||
|
/** JSON string[] of the kinds of alert it wants; null = every kind. */
|
||||||
|
categories: text("categories"),
|
||||||
});
|
});
|
||||||
|
|
||||||
// "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose
|
// "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose
|
||||||
@@ -678,6 +684,8 @@ export const alertIncidents = sqliteTable(
|
|||||||
resolvedAt: text("resolved_at"), // ISO; null while it is open
|
resolvedAt: text("resolved_at"), // ISO; null while it is open
|
||||||
lastSeenAt: text("last_seen_at").notNull(), // ISO
|
lastSeenAt: text("last_seen_at").notNull(), // ISO
|
||||||
missingSince: text("missing_since"), // ISO; set while it has been absent from the last look(s), before it is called over
|
missingSince: text("missing_since"), // ISO; set while it has been absent from the last look(s), before it is called over
|
||||||
|
escalatedStep: integer("escalated_step").notNull().default(0), // the highest escalation step already sent for this incident (0 = none)
|
||||||
|
escalatedAt: text("escalated_at"), // ISO; when that was
|
||||||
},
|
},
|
||||||
(t) => [index("alert_incidents_key_idx").on(t.alertKey), index("alert_incidents_opened_idx").on(t.openedAt), index("alert_incidents_target_idx").on(t.targetType, t.targetId)],
|
(t) => [index("alert_incidents_key_idx").on(t.alertKey), index("alert_incidents_opened_idx").on(t.openedAt), index("alert_incidents_target_idx").on(t.targetType, t.targetId)],
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js";
|
|||||||
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
|
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
|
||||||
import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js";
|
import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js";
|
||||||
import { initUpdateRuns } from "./services/updateRuns.js";
|
import { initUpdateRuns } from "./services/updateRuns.js";
|
||||||
|
import { initEscalation } from "./services/escalation.js";
|
||||||
import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js";
|
import { initSelfBackupScheduler } from "./services/selfBackupScheduler.js";
|
||||||
import { initIncidentRecorder } from "./services/incidents.js";
|
import { initIncidentRecorder } from "./services/incidents.js";
|
||||||
import { getAlerts } from "./services/alerts.js";
|
import { getAlerts } from "./services/alerts.js";
|
||||||
@@ -57,6 +58,7 @@ await initMaintenanceScheduleScheduler();
|
|||||||
initUpdateRuns();
|
initUpdateRuns();
|
||||||
await initSelfBackupScheduler();
|
await initSelfBackupScheduler();
|
||||||
initIncidentRecorder(() => getAlerts(false));
|
initIncidentRecorder(() => getAlerts(false));
|
||||||
|
initEscalation();
|
||||||
|
|
||||||
createApp().listen(env.port, () => {
|
createApp().listen(env.port, () => {
|
||||||
console.log(`homelab-manager listening on port ${env.port}`);
|
console.log(`homelab-manager listening on port ${env.port}`);
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import { requireAuth } from "../auth/middleware.js";
|
|||||||
import { recordAudit } from "../services/audit.js";
|
import { recordAudit } from "../services/audit.js";
|
||||||
import { getSettings } from "../services/settingsStore.js";
|
import { getSettings } from "../services/settingsStore.js";
|
||||||
import { checkEndpoint, getVapidKeys, sendPush } from "../services/webPush.js";
|
import { checkEndpoint, getVapidKeys, sendPush } from "../services/webPush.js";
|
||||||
|
import { CATEGORIES, parseCategories } from "../services/notifyRouting.js";
|
||||||
import { asyncHandler } from "../utils/asyncHandler.js";
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
export const pushRouter = Router();
|
export const pushRouter = Router();
|
||||||
@@ -45,7 +46,7 @@ pushRouter.get("/mine", asyncHandler(async (req, res) => {
|
|||||||
const { webPush } = await getSettings();
|
const { webPush } = await getSettings();
|
||||||
res.json({
|
res.json({
|
||||||
enabled: webPush.enabled,
|
enabled: webPush.enabled,
|
||||||
subscriptions: rows.map((r) => ({ id: r.id, label: r.label, host: hostOf(r.endpoint), endpoint: r.endpoint, createdAt: r.createdAt, lastSuccessAt: r.lastSuccessAt })),
|
subscriptions: rows.map((r) => ({ id: r.id, label: r.label, host: hostOf(r.endpoint), endpoint: r.endpoint, createdAt: r.createdAt, lastSuccessAt: r.lastSuccessAt, minSeverity: r.minSeverity === "critical" || r.minSeverity === "warning" ? r.minSeverity : null, categories: parseCategories(r.categories) })),
|
||||||
});
|
});
|
||||||
}));
|
}));
|
||||||
|
|
||||||
@@ -74,6 +75,25 @@ pushRouter.post("/subscribe", asyncHandler(async (req, res) => {
|
|||||||
res.status(201).json({ id: row.id });
|
res.status(201).json({ id: row.id });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
// What a browser wants to be told: only the worse alerts, only some kinds. Its owner's to say (an administrator may too).
|
||||||
|
const prefsSchema = z.object({
|
||||||
|
minSeverity: z.enum(["warning", "critical"]).nullable(),
|
||||||
|
categories: z.array(z.enum(CATEGORIES)).max(10).nullable(),
|
||||||
|
});
|
||||||
|
|
||||||
|
pushRouter.patch("/:id", asyncHandler(async (req, res) => {
|
||||||
|
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
|
||||||
|
const parsed = prefsSchema.safeParse(req.body);
|
||||||
|
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: "Choose a severity (or any) and the kinds of alert (or all).", details: parsed.success ? undefined : parsed.error.flatten() });
|
||||||
|
const me = req.currentUser!;
|
||||||
|
const [row] = await db.select().from(pushSubscriptions).where(eq(pushSubscriptions.id, id)).limit(1);
|
||||||
|
if (!row || (row.userId !== me.id && me.role !== "admin")) return res.status(404).json({ error: "not_found" });
|
||||||
|
const categories = parsed.data.categories && parsed.data.categories.length > 0 ? parsed.data.categories : null;
|
||||||
|
await db.update(pushSubscriptions).set({ minSeverity: parsed.data.minSeverity, categories: categories ? JSON.stringify(categories) : null }).where(eq(pushSubscriptions.id, id));
|
||||||
|
await recordAudit({ actor: me, category: "push", action: "preferences", targetType: "push_subscription", targetId: id, detail: { host: hostOf(row.endpoint), label: row.label, minSeverity: parsed.data.minSeverity, categories } });
|
||||||
|
res.json({ minSeverity: parsed.data.minSeverity, categories });
|
||||||
|
}));
|
||||||
|
|
||||||
// Stops a browser being written to. A person removes their own; an administrator may remove anyone's.
|
// Stops a browser being written to. A person removes their own; an administrator may remove anyone's.
|
||||||
pushRouter.delete("/:id", asyncHandler(async (req, res) => {
|
pushRouter.delete("/:id", asyncHandler(async (req, res) => {
|
||||||
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
|
const id = /^\d+$/.test(req.params.id) ? Number(req.params.id) : NaN;
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js";
|
|||||||
import { scheduleWeeklyDigest } from "../services/weeklyDigestScheduler.js";
|
import { scheduleWeeklyDigest } from "../services/weeklyDigestScheduler.js";
|
||||||
import { buildDigest, sendDigest } from "../services/weeklyDigest.js";
|
import { buildDigest, sendDigest } from "../services/weeklyDigest.js";
|
||||||
import { enabledChannelCount } from "../services/notify.js";
|
import { enabledChannelCount } from "../services/notify.js";
|
||||||
|
import { CATEGORIES, CHANNEL_IDS } from "../services/notifyRouting.js";
|
||||||
|
import { InvalidTagError, normalizeTags } from "../services/serverTags.js";
|
||||||
import { testGotify, testNtfy, testSmtp, testWebhook, flushQuietHoursQueue } from "../services/notify.js";
|
import { testGotify, testNtfy, testSmtp, testWebhook, flushQuietHoursQueue } from "../services/notify.js";
|
||||||
import { listQueuedNotifications } from "../services/notificationQueue.js";
|
import { listQueuedNotifications } from "../services/notificationQueue.js";
|
||||||
import { buildExportPayload, encryptExport, decryptExport, applyImportPayload, type EncryptedExportFile } from "../services/configBackup.js";
|
import { buildExportPayload, encryptExport, decryptExport, applyImportPayload, type EncryptedExportFile } from "../services/configBackup.js";
|
||||||
@@ -47,7 +49,51 @@ settingsRouter.get("/display", asyncHandler(async (_req, res) => {
|
|||||||
res.json({ display });
|
res.json({ display });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
const channelEnum = z.enum(CHANNEL_IDS);
|
||||||
|
|
||||||
|
// Routing rules: which channels hear which notifications. A rule with no channels means "tell nobody" (the Alerts page still shows it).
|
||||||
|
const routingSchema = z.object({
|
||||||
|
rules: z
|
||||||
|
.array(
|
||||||
|
z.object({
|
||||||
|
id: z.string().trim().min(1).max(40),
|
||||||
|
name: z.string().trim().min(1).max(60),
|
||||||
|
enabled: z.boolean(),
|
||||||
|
categories: z.array(z.enum(CATEGORIES)).max(10),
|
||||||
|
minSeverity: z.enum(["warning", "critical"]).nullable(),
|
||||||
|
tags: z.array(z.string().trim().min(1).max(30)).max(10),
|
||||||
|
channels: z.array(channelEnum).max(5),
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.max(30),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Escalation: a few steps in time order. Each must say who to tell — a channel, or an address.
|
||||||
|
const escalationSchema = z
|
||||||
|
.object({
|
||||||
|
enabled: z.boolean(),
|
||||||
|
steps: z
|
||||||
|
.array(
|
||||||
|
z.object({
|
||||||
|
afterMinutes: z.number().int().min(5).max(10080),
|
||||||
|
minSeverity: z.enum(["warning", "critical"]),
|
||||||
|
channels: z.array(channelEnum).max(5),
|
||||||
|
extraEmail: z.string().trim().max(200).refine((v) => v === "" || /^[^@\s]+@[^@\s]+\.[^@\s]+$/.test(v), "That isn't an email address"),
|
||||||
|
evenInQuietHours: z.boolean(),
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.max(3),
|
||||||
|
})
|
||||||
|
.superRefine((v, ctx) => {
|
||||||
|
v.steps.forEach((s, i) => {
|
||||||
|
if (s.channels.length === 0 && s.extraEmail === "") ctx.addIssue({ code: "custom", path: ["steps", i], message: "Choose a channel to tell, or an address to email." });
|
||||||
|
if (i > 0 && s.afterMinutes <= v.steps[i - 1].afterMinutes) ctx.addIssue({ code: "custom", path: ["steps", i, "afterMinutes"], message: "Each step must come later than the one before." });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
const updateSchema = z.object({
|
const updateSchema = z.object({
|
||||||
|
routing: routingSchema.optional(),
|
||||||
|
escalation: escalationSchema.optional(),
|
||||||
gotify: z.object({ enabled: z.boolean(), url: z.string(), token: z.string(), priority: z.number() }).partial().optional(),
|
gotify: z.object({ enabled: z.boolean(), url: z.string(), token: z.string(), priority: z.number() }).partial().optional(),
|
||||||
ntfy: z.object({ enabled: z.boolean(), url: z.string(), topic: z.string(), token: z.string(), priority: z.number() }).partial().optional(),
|
ntfy: z.object({ enabled: z.boolean(), url: z.string(), topic: z.string(), token: z.string(), priority: z.number() }).partial().optional(),
|
||||||
smtp: z
|
smtp: z
|
||||||
@@ -126,6 +172,14 @@ settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => {
|
|||||||
return res.status(400).json({ error: "invalid_body", details: parsed.error.flatten() });
|
return res.status(400).json({ error: "invalid_body", details: parsed.error.flatten() });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Tags in routing rules are written the way server tags are, so they match.
|
||||||
|
try {
|
||||||
|
if (parsed.data.routing) for (const rule of parsed.data.routing.rules) rule.tags = normalizeTags(rule.tags);
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof InvalidTagError) return res.status(400).json({ error: "invalid_tag", message: err.message });
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
|
||||||
const before = await getSettings();
|
const before = await getSettings();
|
||||||
const updated = await updateSettings(parsed.data);
|
const updated = await updateSettings(parsed.data);
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,112 @@
|
|||||||
|
/**
|
||||||
|
* Escalation: an alert that has been raised for a while and that nobody has acknowledged is told again — through louder channels, and
|
||||||
|
* perhaps to someone else. Steps are in order ("after 30 minutes: push; after 2 hours: email the neighbour who holds the key"), each is sent
|
||||||
|
* at most once per incident (one occurrence of an alert; it starts over if the alert goes away and comes back), and an acknowledgement stops
|
||||||
|
* the lot. Alerts that are silenced for maintenance, or held back because something they depend on is down, never escalate.
|
||||||
|
*/
|
||||||
|
import { and, eq, inArray, isNull } from "drizzle-orm";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { alertIncidents } from "../db/schema.js";
|
||||||
|
import type { Alert } from "./alertTypes.js";
|
||||||
|
import { getAlerts } from "./alerts.js";
|
||||||
|
import { loadAcks } from "./alertAcks.js";
|
||||||
|
import { recordAudit } from "./audit.js";
|
||||||
|
import { notifyAudience } from "./notify.js";
|
||||||
|
import { getSettings } from "./settingsStore.js";
|
||||||
|
import { severityRank, type EscalationStep } from "./notifyRouting.js";
|
||||||
|
|
||||||
|
export interface DueEscalation {
|
||||||
|
alert: Pick<Alert, "id" | "severity" | "message" | "source">;
|
||||||
|
incidentId: number;
|
||||||
|
/** 1-based, in the order the steps are kept. */
|
||||||
|
step: number;
|
||||||
|
definition: EscalationStep;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OpenIncident {
|
||||||
|
id: number;
|
||||||
|
openedAt: string;
|
||||||
|
escalatedStep: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Which escalations are due now. Pure. An alert gets at most one step per look — the first one that is due, applies to its severity and hasn't
|
||||||
|
* been sent — so a long-neglected alert is told about once per look rather than all at once.
|
||||||
|
*/
|
||||||
|
export function dueEscalations(input: {
|
||||||
|
alerts: Pick<Alert, "id" | "severity" | "silenced" | "message" | "source">[];
|
||||||
|
acknowledged: Set<string>;
|
||||||
|
incidents: Map<string, OpenIncident>;
|
||||||
|
steps: EscalationStep[];
|
||||||
|
now: Date;
|
||||||
|
}): DueEscalation[] {
|
||||||
|
const out: DueEscalation[] = [];
|
||||||
|
for (const alert of input.alerts) {
|
||||||
|
if (alert.silenced || alert.severity === "info" || input.acknowledged.has(alert.id)) continue;
|
||||||
|
const incident = input.incidents.get(alert.id);
|
||||||
|
if (!incident) continue;
|
||||||
|
const ageMinutes = (input.now.getTime() - Date.parse(incident.openedAt)) / 60_000;
|
||||||
|
for (let i = incident.escalatedStep; i < input.steps.length; i++) {
|
||||||
|
const def = input.steps[i];
|
||||||
|
if (ageMinutes < def.afterMinutes) break; // steps are in time order: none after this one is due either
|
||||||
|
if (severityRank(alert.severity) < severityRank(def.minSeverity)) continue; // not bad enough for this step; a later one may be for it
|
||||||
|
out.push({ alert, incidentId: incident.id, step: i + 1, definition: def });
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
const words = (minutes: number) => (minutes >= 120 && minutes % 60 === 0 ? `${minutes / 60} hours` : minutes === 60 ? "an hour" : `${minutes} minutes`);
|
||||||
|
|
||||||
|
/** One look: sends whatever is due. Returns how many alerts were escalated. */
|
||||||
|
export async function runEscalation(now: Date = new Date()): Promise<number> {
|
||||||
|
const { escalation } = await getSettings();
|
||||||
|
if (!escalation.enabled || escalation.steps.length === 0) return 0;
|
||||||
|
|
||||||
|
const report = await getAlerts(false);
|
||||||
|
const ids = report.alerts.map((a) => a.id);
|
||||||
|
if (ids.length === 0) return 0;
|
||||||
|
const open = await db.select().from(alertIncidents).where(and(isNull(alertIncidents.resolvedAt), inArray(alertIncidents.alertKey, ids)));
|
||||||
|
const acks = await loadAcks(now);
|
||||||
|
|
||||||
|
const due = dueEscalations({
|
||||||
|
alerts: report.alerts,
|
||||||
|
acknowledged: new Set([...acks].filter(([, a]) => a.active).map(([id]) => id)),
|
||||||
|
incidents: new Map(open.map((i) => [i.alertKey, { id: i.id, openedAt: i.openedAt, escalatedStep: i.escalatedStep }])),
|
||||||
|
steps: escalation.steps,
|
||||||
|
now,
|
||||||
|
});
|
||||||
|
|
||||||
|
// One message for each step, listing the alerts it is for.
|
||||||
|
const byStep = new Map<number, DueEscalation[]>();
|
||||||
|
for (const d of due) byStep.set(d.step, [...(byStep.get(d.step) ?? []), d]);
|
||||||
|
for (const [step, items] of byStep) {
|
||||||
|
const def = items[0].definition;
|
||||||
|
const lines = items.map((d) => `${d.alert.severity === "critical" ? "✕" : "⚠"} ${d.alert.message}`);
|
||||||
|
const message = `Nobody has acknowledged ${items.length === 1 ? "this alert" : `these ${items.length} alerts`} after ${words(def.afterMinutes)}:\n\n${lines.join("\n")}\n\nAcknowledge ${items.length === 1 ? "it" : "them"} on the Alerts page to stop the reminders.`;
|
||||||
|
try {
|
||||||
|
await notifyAudience("Homelab Manager — Still not acknowledged", message, { channels: def.channels, pushIds: null, ...(def.extraEmail ? { email: def.extraEmail } : {}) }, { quietHours: !def.evenInQuietHours });
|
||||||
|
} catch (err) {
|
||||||
|
console.error("[escalation] couldn't send step", step, err);
|
||||||
|
continue; // not marked as sent: the next look tries again
|
||||||
|
}
|
||||||
|
for (const d of items) {
|
||||||
|
await db.update(alertIncidents).set({ escalatedStep: step, escalatedAt: now.toISOString() }).where(eq(alertIncidents.id, d.incidentId));
|
||||||
|
await recordAudit({ category: "alert", action: "escalate", detail: { alertId: d.alert.id, message: d.alert.message, step, afterMinutes: def.afterMinutes, channels: def.channels, email: def.extraEmail || null } });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return due.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
let timer: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
||||||
|
/** Looks every five minutes (a little after the incident recorder, which uses the same fresh result). */
|
||||||
|
export function initEscalation(): void {
|
||||||
|
if (timer) return;
|
||||||
|
const tick = () =>
|
||||||
|
void runEscalation().catch((err) => console.error("[escalation] the look failed:", err));
|
||||||
|
setTimeout(tick, 120_000).unref();
|
||||||
|
timer = setInterval(tick, 5 * 60_000);
|
||||||
|
timer.unref();
|
||||||
|
}
|
||||||
@@ -224,7 +224,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
|
|||||||
// ─── State diff (pure) ──────────────────────────────────────────────────────
|
// ─── State diff (pure) ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
/** `covered`: the problem is explained by another that is being reported (a guest down because its host is) — recorded, but not announced on its own. */
|
/** `covered`: the problem is explained by another that is being reported (a guest down because its host is) — recorded, but not announced on its own. */
|
||||||
export type ActiveState = Record<string, { message: string; source: string; covered?: boolean }>;
|
export type ActiveState = Record<string, { message: string; source: string; covered?: boolean; severity?: "critical" | "warning" }>;
|
||||||
|
|
||||||
/** `held` is a set of hierarchical sources that couldn't be read this run; matches the source itself or anything beneath it. */
|
/** `held` is a set of hierarchical sources that couldn't be read this run; matches the source itself or anything beneath it. */
|
||||||
function isHeld(source: string, held: Set<string>): boolean {
|
function isHeld(source: string, held: Set<string>): boolean {
|
||||||
@@ -252,7 +252,7 @@ export function diffConditions(
|
|||||||
covered: Set<string> = new Set(),
|
covered: Set<string> = new Set(),
|
||||||
): {
|
): {
|
||||||
added: HealthCondition[];
|
added: HealthCondition[];
|
||||||
resolved: { key: string; message: string }[];
|
resolved: { key: string; message: string; severity?: "critical" | "warning" }[];
|
||||||
next: ActiveState;
|
next: ActiveState;
|
||||||
/** Every problem that began, announced or not. */
|
/** Every problem that began, announced or not. */
|
||||||
newlyDown: HealthCondition[];
|
newlyDown: HealthCondition[];
|
||||||
@@ -268,12 +268,12 @@ export function diffConditions(
|
|||||||
const visible = current.filter((c) => !isSilenced(c.key));
|
const visible = current.filter((c) => !isSilenced(c.key));
|
||||||
|
|
||||||
const next: ActiveState = {};
|
const next: ActiveState = {};
|
||||||
for (const c of visible) next[c.key] = covered.has(c.key) ? { message: c.message, source: c.source, covered: true } : { message: c.message, source: c.source };
|
for (const c of visible) next[c.key] = { message: c.message, source: c.source, ...(covered.has(c.key) ? { covered: true } : {}), ...(c.severity ? { severity: c.severity } : {}) };
|
||||||
|
|
||||||
const newlyDown = visible.filter((c) => !(c.key in previous));
|
const newlyDown = visible.filter((c) => !(c.key in previous));
|
||||||
// Announced: new and standing alone, or covered until now and no longer.
|
// Announced: new and standing alone, or covered until now and no longer.
|
||||||
const added = visible.filter((c) => !covered.has(c.key) && (!(c.key in previous) || previous[c.key].covered));
|
const added = visible.filter((c) => !covered.has(c.key) && (!(c.key in previous) || previous[c.key].covered));
|
||||||
const resolved: { key: string; message: string }[] = [];
|
const resolved: { key: string; message: string; severity?: "critical" | "warning" }[] = [];
|
||||||
const ended: { key: string; message: string }[] = [];
|
const ended: { key: string; message: string }[] = [];
|
||||||
for (const [key, entry] of Object.entries(previous)) {
|
for (const [key, entry] of Object.entries(previous)) {
|
||||||
if (key in next) continue;
|
if (key in next) continue;
|
||||||
@@ -282,7 +282,7 @@ export function diffConditions(
|
|||||||
next[key] = entry;
|
next[key] = entry;
|
||||||
} else {
|
} else {
|
||||||
ended.push({ key, message: entry.message });
|
ended.push({ key, message: entry.message });
|
||||||
if (!entry.covered) resolved.push({ key, message: entry.message });
|
if (!entry.covered) resolved.push({ key, message: entry.message, ...(entry.severity ? { severity: entry.severity } : {}) });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return { added, resolved, next, newlyDown, ended };
|
return { added, resolved, next, newlyDown, ended };
|
||||||
|
|||||||
@@ -5,11 +5,13 @@ export interface QueuedNotification {
|
|||||||
id: number;
|
id: number;
|
||||||
title: string;
|
title: string;
|
||||||
message: string;
|
message: string;
|
||||||
|
/** JSON of who it was for; null means everyone. */
|
||||||
|
audience: string | null;
|
||||||
createdAt: string;
|
createdAt: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function enqueueNotification(title: string, message: string): Promise<void> {
|
export async function enqueueNotification(title: string, message: string, audience: string | null = null): Promise<void> {
|
||||||
await db.insert(notificationQueue).values({ title, message });
|
await db.insert(notificationQueue).values({ title, message, audience });
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function listQueuedNotifications(): Promise<QueuedNotification[]> {
|
export async function listQueuedNotifications(): Promise<QueuedNotification[]> {
|
||||||
|
|||||||
+167
-35
@@ -3,6 +3,11 @@ import { getSettings, type NotificationEvents } from "./settingsStore.js";
|
|||||||
import { enqueueNotification, listQueuedNotifications, clearQueuedNotifications } from "./notificationQueue.js";
|
import { enqueueNotification, listQueuedNotifications, clearQueuedNotifications } from "./notificationQueue.js";
|
||||||
import { emitEvent } from "./eventWebhooks.js";
|
import { emitEvent } from "./eventWebhooks.js";
|
||||||
import { sendPush, subscriptionCount } from "./webPush.js";
|
import { sendPush, subscriptionCount } from "./webPush.js";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { inArray } from "drizzle-orm";
|
||||||
|
import { pushSubscriptions, servers } from "../db/schema.js";
|
||||||
|
import { parseTags } from "./serverTags.js";
|
||||||
|
import { firstMatchingRule, metaOfConditionKey, pushAccepts, type ChannelId, type NotifyMeta } from "./notifyRouting.js";
|
||||||
|
|
||||||
// ─── Channel senders (best-effort — failures are logged, never thrown) ──────
|
// ─── Channel senders (best-effort — failures are logged, never thrown) ──────
|
||||||
|
|
||||||
@@ -22,13 +27,22 @@ async function sendGotify(title: string, message: string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A header can only carry Latin-1, and every title here has an em dash. ntfy reads RFC 2047 encoded words in its headers ("=?UTF-8?B?...?="),
|
||||||
|
* so anything outside plain ASCII is sent that way.
|
||||||
|
*/
|
||||||
|
export function headerSafe(text: string): string {
|
||||||
|
// eslint-disable-next-line no-control-regex
|
||||||
|
return /^[ -~]*$/.test(text) ? text : `=?UTF-8?B?${Buffer.from(text, "utf8").toString("base64")}?=`;
|
||||||
|
}
|
||||||
|
|
||||||
async function sendNtfy(title: string, message: string) {
|
async function sendNtfy(title: string, message: string) {
|
||||||
const { ntfy } = await getSettings();
|
const { ntfy } = await getSettings();
|
||||||
if (!ntfy.enabled || !ntfy.url || !ntfy.topic) return;
|
if (!ntfy.enabled || !ntfy.url || !ntfy.topic) return;
|
||||||
const base = ntfy.url.replace(/\/$/, "");
|
const base = ntfy.url.replace(/\/$/, "");
|
||||||
const headers: Record<string, string> = {
|
const headers: Record<string, string> = {
|
||||||
"Content-Type": "text/plain",
|
"Content-Type": "text/plain",
|
||||||
Title: title,
|
Title: headerSafe(title),
|
||||||
Priority: String(ntfy.priority ?? 3),
|
Priority: String(ntfy.priority ?? 3),
|
||||||
};
|
};
|
||||||
if (ntfy.token) headers.Authorization = `Bearer ${ntfy.token}`;
|
if (ntfy.token) headers.Authorization = `Bearer ${ntfy.token}`;
|
||||||
@@ -73,18 +87,116 @@ async function sendWebhook(title: string, message: string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function sendWebPush(title: string, message: string) {
|
async function sendWebPush(title: string, message: string, only: number[] | null) {
|
||||||
const { webPush } = await getSettings();
|
const { webPush } = await getSettings();
|
||||||
if (!webPush.enabled) return;
|
if (!webPush.enabled) return;
|
||||||
try {
|
try {
|
||||||
await sendPush({ title, body: message });
|
if (only === null) await sendPush({ title, body: message });
|
||||||
|
else if (only.length > 0) await sendPush({ title, body: message }, await db.select().from(pushSubscriptions).where(inArray(pushSubscriptions.id, only)));
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error("[notify] Web push failed:", err instanceof Error ? err.message : err);
|
console.error("[notify] Web push failed:", err instanceof Error ? err.message : err);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function sendToChannels(title: string, message: string): Promise<void> {
|
/** Email to one address, through the email channel's server, whatever the channel's own recipient is (an escalation to someone else). */
|
||||||
await Promise.all([sendGotify(title, message), sendNtfy(title, message), sendSmtp(title, message), sendWebhook(title, message), sendWebPush(title, message)]);
|
async function sendExtraEmail(title: string, message: string, to: string) {
|
||||||
|
const { smtp } = await getSettings();
|
||||||
|
if (!smtp.host || !smtp.from) {
|
||||||
|
console.error("[notify] An escalation wants to email someone, but email isn't set up.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const transporter = nodemailer.createTransport({
|
||||||
|
host: smtp.host,
|
||||||
|
port: Number(smtp.port) || 587,
|
||||||
|
secure: !!smtp.secure,
|
||||||
|
auth: smtp.username ? { user: smtp.username, pass: smtp.password } : undefined,
|
||||||
|
});
|
||||||
|
await transporter.sendMail({ from: smtp.from, to, subject: title, text: message });
|
||||||
|
} catch (err) {
|
||||||
|
console.error("[notify] Extra email failed:", err instanceof Error ? err.message : err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Who a notification is for ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
/** The channels and browsers a notification goes to. `null` means every one that is on (the way it was before routing existed). */
|
||||||
|
export interface Audience {
|
||||||
|
channels: ChannelId[] | null;
|
||||||
|
/** The subscribed browsers that want it; null = all of them. */
|
||||||
|
pushIds: number[] | null;
|
||||||
|
/** Also email this address. */
|
||||||
|
email?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const EVERYONE: Audience = { channels: null, pushIds: null };
|
||||||
|
|
||||||
|
export const audienceKey = (a: Audience) => JSON.stringify({ c: a.channels ? [...a.channels].sort() : null, p: a.pushIds ? [...a.pushIds].sort((x, y) => x - y) : null, e: a.email ?? null });
|
||||||
|
|
||||||
|
/** Told to nobody: a rule said so (the Alerts page still shows it). */
|
||||||
|
const isMuted = (a: Audience) => a.channels !== null && a.channels.length === 0 && !a.email;
|
||||||
|
|
||||||
|
/** Works out who a notification is for: the first routing rule it matches picks the channels, and browsers may turn down what they don't want. */
|
||||||
|
export async function audienceFor(meta?: NotifyMeta): Promise<Audience> {
|
||||||
|
if (!meta) return EVERYONE;
|
||||||
|
const { routing } = await getSettings();
|
||||||
|
let channels: ChannelId[] | null = null;
|
||||||
|
if (routing.rules.some((r) => r.enabled)) {
|
||||||
|
let tags = meta.tags;
|
||||||
|
if (!tags && meta.serverId != null) {
|
||||||
|
const [row] = await db.select({ tags: servers.tags }).from(servers).where(inArray(servers.id, [meta.serverId]));
|
||||||
|
tags = parseTags(row?.tags);
|
||||||
|
}
|
||||||
|
channels = firstMatchingRule(routing.rules, { ...meta, tags: tags ?? [] })?.channels ?? null;
|
||||||
|
}
|
||||||
|
let pushIds: number[] | null = null;
|
||||||
|
if (channels === null || channels.includes("webPush")) {
|
||||||
|
const subs = await db.select({ id: pushSubscriptions.id, minSeverity: pushSubscriptions.minSeverity, categories: pushSubscriptions.categories }).from(pushSubscriptions);
|
||||||
|
// Only when somebody has said what they want; otherwise everyone gets it, as ever.
|
||||||
|
if (subs.some((s) => s.minSeverity || s.categories)) pushIds = subs.filter((s) => pushAccepts(s, meta)).map((s) => s.id);
|
||||||
|
}
|
||||||
|
return { channels, pushIds };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function sendToChannels(title: string, message: string, audience: Audience = EVERYONE): Promise<void> {
|
||||||
|
const to = (c: ChannelId) => audience.channels === null || audience.channels.includes(c);
|
||||||
|
await Promise.all([
|
||||||
|
to("gotify") ? sendGotify(title, message) : undefined,
|
||||||
|
to("ntfy") ? sendNtfy(title, message) : undefined,
|
||||||
|
to("smtp") ? sendSmtp(title, message) : undefined,
|
||||||
|
to("webhook") ? sendWebhook(title, message) : undefined,
|
||||||
|
to("webPush") ? sendWebPush(title, message, audience.pushIds) : undefined,
|
||||||
|
audience.email ? sendExtraEmail(title, message, audience.email) : undefined,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sends to an audience that is already worked out: held for the quiet-hours digest if it is quiet (unless told not to), nowhere if a rule
|
||||||
|
* said nobody. Escalations use this with their own channels.
|
||||||
|
*/
|
||||||
|
export async function notifyAudience(title: string, message: string, audience: Audience, options: { quietHours?: boolean } = {}): Promise<void> {
|
||||||
|
if (isMuted(audience)) return;
|
||||||
|
if (options.quietHours !== false && (await isQuietHoursActive())) {
|
||||||
|
await enqueueNotification(title, message, audience.channels === null && audience.pushIds === null && !audience.email ? null : JSON.stringify(audience));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
await sendToChannels(title, message, audience);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Items that may each be for a different audience (a rule can send a disk warning one place and a server outage another): sent as one message per audience. */
|
||||||
|
async function notifyGrouped<T>(items: T[], metaOf: (item: T) => NotifyMeta, build: (group: T[]) => { title: string; message: string }): Promise<void> {
|
||||||
|
const groups = new Map<string, { audience: Audience; items: T[] }>();
|
||||||
|
for (const item of items) {
|
||||||
|
const audience = await audienceFor(metaOf(item));
|
||||||
|
const key = audienceKey(audience);
|
||||||
|
const group = groups.get(key) ?? { audience, items: [] };
|
||||||
|
group.items.push(item);
|
||||||
|
groups.set(key, group);
|
||||||
|
}
|
||||||
|
for (const group of groups.values()) {
|
||||||
|
const { title, message } = build(group.items);
|
||||||
|
await notifyAudience(title, message, group.audience);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function currentTimeInZone(timezone: string): { hh: number; mm: number } {
|
function currentTimeInZone(timezone: string): { hh: number; mm: number } {
|
||||||
@@ -133,23 +245,32 @@ export async function enabledChannelCount(): Promise<number> {
|
|||||||
].filter(Boolean).length;
|
].filter(Boolean).length;
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notify(title: string, message: string): Promise<void> {
|
/** `meta` says what it is about, so routing rules and browsers' wishes can apply; without it, it goes to everyone. */
|
||||||
if (await isQuietHoursActive()) {
|
export async function notify(title: string, message: string, meta?: NotifyMeta): Promise<void> {
|
||||||
await enqueueNotification(title, message);
|
await notifyAudience(title, message, await audienceFor(meta));
|
||||||
return;
|
|
||||||
}
|
|
||||||
await sendToChannels(title, message);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Called by the daily scheduler at quiet-hours end (or manually) — sends everything queued during the window as one digest, then clears it. */
|
/** Called by the daily scheduler at quiet-hours end (or manually) — sends everything queued during the window as one digest, then clears it. */
|
||||||
export async function flushQuietHoursQueue(): Promise<number> {
|
export async function flushQuietHoursQueue(): Promise<number> {
|
||||||
const queued = await listQueuedNotifications();
|
const queued = await listQueuedNotifications();
|
||||||
if (queued.length === 0) return 0;
|
if (queued.length === 0) return 0;
|
||||||
const sections = queued.map((q) => `— ${q.title} —\n${q.message}`);
|
// One digest for each audience the held notifications were for.
|
||||||
|
const groups = new Map<string, typeof queued>();
|
||||||
|
for (const q of queued) groups.set(q.audience ?? "", [...(groups.get(q.audience ?? "") ?? []), q]);
|
||||||
|
for (const [stored, items] of groups) {
|
||||||
|
let audience: Audience = EVERYONE;
|
||||||
|
try {
|
||||||
|
if (stored) audience = JSON.parse(stored) as Audience;
|
||||||
|
} catch {
|
||||||
|
// unreadable: everyone, as before
|
||||||
|
}
|
||||||
|
const sections = items.map((q) => `— ${q.title} —\n${q.message}`);
|
||||||
await sendToChannels(
|
await sendToChannels(
|
||||||
"Homelab Manager — Notifications from quiet hours",
|
"Homelab Manager — Notifications from quiet hours",
|
||||||
`${queued.length} notification${queued.length !== 1 ? "s were" : " was"} held during quiet hours:\n\n${sections.join("\n\n")}`,
|
`${items.length} notification${items.length !== 1 ? "s were" : " was"} held during quiet hours:\n\n${sections.join("\n\n")}`,
|
||||||
|
audience,
|
||||||
);
|
);
|
||||||
|
}
|
||||||
await clearQueuedNotifications();
|
await clearQueuedNotifications();
|
||||||
return queued.length;
|
return queued.length;
|
||||||
}
|
}
|
||||||
@@ -172,7 +293,7 @@ export async function testGotify(cfg: { url: string; token: string; priority?: n
|
|||||||
|
|
||||||
export async function testNtfy(cfg: { url: string; topic: string; token?: string; priority?: number }): Promise<void> {
|
export async function testNtfy(cfg: { url: string; topic: string; token?: string; priority?: number }): Promise<void> {
|
||||||
const base = cfg.url.replace(/\/$/, "");
|
const base = cfg.url.replace(/\/$/, "");
|
||||||
const headers: Record<string, string> = { "Content-Type": "text/plain", Title: "Homelab Manager — Test", Priority: String(cfg.priority ?? 3) };
|
const headers: Record<string, string> = { "Content-Type": "text/plain", Title: headerSafe("Homelab Manager — Test"), Priority: String(cfg.priority ?? 3) };
|
||||||
if (cfg.token) headers.Authorization = `Bearer ${cfg.token}`;
|
if (cfg.token) headers.Authorization = `Bearer ${cfg.token}`;
|
||||||
const res = await fetch(`${base}/${encodeURIComponent(cfg.topic)}`, {
|
const res = await fetch(`${base}/${encodeURIComponent(cfg.topic)}`, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
@@ -263,7 +384,7 @@ export async function notifySecretExpiry(
|
|||||||
`Couldn't read the live certificate for ${checkFailures.length} monitored secret${checkFailures.length !== 1 ? "s" : ""} — the expiry shown may be stale:\n\n${lines.join("\n")}`,
|
`Couldn't read the live certificate for ${checkFailures.length} monitored secret${checkFailures.length !== 1 ? "s" : ""} — the expiry shown may be stale:\n\n${lines.join("\n")}`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
await notify("Homelab Manager — Secrets Alert", sections.join("\n\n"));
|
await notify("Homelab Manager — Secrets Alert", sections.join("\n\n"), { category: "expiry", severity: expiring.some((s) => s.status === "expired") ? "critical" : "warning" });
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyDockerUpdates(
|
export async function notifyDockerUpdates(
|
||||||
@@ -277,6 +398,7 @@ export async function notifyDockerUpdates(
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Docker Updates Available",
|
"Homelab Manager — Docker Updates Available",
|
||||||
`${updatesAvailable.length} container${updatesAvailable.length !== 1 ? "s have" : " has"} an image update available:\n\n${lines.join("\n")}`,
|
`${updatesAvailable.length} container${updatesAvailable.length !== 1 ? "s have" : " has"} an image update available:\n\n${lines.join("\n")}`,
|
||||||
|
{ category: "updates", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -291,6 +413,7 @@ export async function notifyProxmoxBackupFailure(
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Proxmox Backup Failed",
|
"Homelab Manager — Proxmox Backup Failed",
|
||||||
`${failures.length} node${failures.length !== 1 ? "s have" : " has"} a failing most-recent backup run:\n\n${lines.join("\n")}`,
|
`${failures.length} node${failures.length !== 1 ? "s have" : " has"} a failing most-recent backup run:\n\n${lines.join("\n")}`,
|
||||||
|
{ category: "backup", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -303,6 +426,7 @@ export async function notifyProxmoxUncoveredGuests(
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Proxmox Guests Without Backup Coverage",
|
"Homelab Manager — Proxmox Guests Without Backup Coverage",
|
||||||
`${uncovered.length} guest${uncovered.length !== 1 ? "s are" : " is"} not covered by any enabled backup job:\n\n${lines.join("\n")}`,
|
`${uncovered.length} guest${uncovered.length !== 1 ? "s are" : " is"} not covered by any enabled backup job:\n\n${lines.join("\n")}`,
|
||||||
|
{ category: "backup", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -312,6 +436,7 @@ export async function notifyStaleBackups(stale: { message: string }[]): Promise<
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Backups Gone Stale",
|
"Homelab Manager — Backups Gone Stale",
|
||||||
`${stale.length} guest${stale.length !== 1 ? "s have" : " has"} no recent backup:\n\n${stale.map((s) => s.message).join("\n")}`,
|
`${stale.length} guest${stale.length !== 1 ? "s have" : " has"} no recent backup:\n\n${stale.map((s) => s.message).join("\n")}`,
|
||||||
|
{ category: "backup", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -320,6 +445,7 @@ export async function notifyUpdateRunFinished(run: { serverName: string; templat
|
|||||||
await notify(
|
await notify(
|
||||||
run.ok ? `Homelab Manager — ${run.serverName} updated` : `Homelab Manager — Update of ${run.serverName} didn't finish`,
|
run.ok ? `Homelab Manager — ${run.serverName} updated` : `Homelab Manager — Update of ${run.serverName} didn't finish`,
|
||||||
`"${run.templateName}" on ${run.serverName}: ${run.message}`,
|
`"${run.templateName}" on ${run.serverName}: ${run.message}`,
|
||||||
|
{ category: "updates", severity: run.ok ? "warning" : "critical" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -336,28 +462,32 @@ export async function notifyPbsVerificationFailed(
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Proxmox Backup Server Verification Failed",
|
"Homelab Manager — Proxmox Backup Server Verification Failed",
|
||||||
`${failures.length} datastore${failures.length !== 1 ? "s have" : " has"} a problem:\n\n${lines.join("\n")}`,
|
`${failures.length} datastore${failures.length !== 1 ? "s have" : " has"} a problem:\n\n${lines.join("\n")}`,
|
||||||
|
{ category: "backup", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** What a problem is about, for routing: its key says the kind and the server; the rest follows its severity. */
|
||||||
|
const issueMeta = (i: { key?: string; severity?: string }): NotifyMeta => (i.key ? metaOfConditionKey(i.key, i.severity) : { category: null, severity: i.severity === "critical" ? "critical" : "warning" });
|
||||||
|
|
||||||
export async function notifyHealthIssues(issues: { message: string; key?: string; source?: string; severity?: string }[]): Promise<void> {
|
export async function notifyHealthIssues(issues: { message: string; key?: string; source?: string; severity?: string }[]): Promise<void> {
|
||||||
// The event webhooks hear of it whatever this notification is switched on for: they are a separate way of being told.
|
// The event webhooks hear of it whatever this notification is switched on for: they are a separate way of being told.
|
||||||
for (const i of issues) emitEvent("alert.opened", { alert: { key: i.key ?? null, source: i.source ?? null, severity: i.severity ?? "warning", message: i.message } });
|
for (const i of issues) emitEvent("alert.opened", { alert: { key: i.key ?? null, source: i.source ?? null, severity: i.severity ?? "warning", message: i.message } });
|
||||||
if (issues.length === 0) return;
|
if (issues.length === 0) return;
|
||||||
if (!(await eventEnabled("healthAlerts"))) return;
|
if (!(await eventEnabled("healthAlerts"))) return;
|
||||||
await notify(
|
await notifyGrouped(issues, issueMeta, (group) => ({
|
||||||
"Homelab Manager — Health Alert",
|
title: "Homelab Manager — Health Alert",
|
||||||
`${issues.length} new issue${issues.length !== 1 ? "s" : ""}:\n\n${issues.map((i) => `⚠ ${i.message}`).join("\n")}`,
|
message: `${group.length} new issue${group.length !== 1 ? "s" : ""}:\n\n${group.map((i) => `⚠ ${i.message}`).join("\n")}`,
|
||||||
);
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyHealthRecovered(resolved: { message: string; key?: string }[]): Promise<void> {
|
export async function notifyHealthRecovered(resolved: { message: string; key?: string; severity?: string }[]): Promise<void> {
|
||||||
for (const r of resolved) emitEvent("alert.resolved", { alert: { key: r.key ?? null, message: r.message } });
|
for (const r of resolved) emitEvent("alert.resolved", { alert: { key: r.key ?? null, message: r.message } });
|
||||||
if (resolved.length === 0) return;
|
if (resolved.length === 0) return;
|
||||||
if (!(await eventEnabled("healthAlerts"))) return;
|
if (!(await eventEnabled("healthAlerts"))) return;
|
||||||
await notify(
|
await notifyGrouped(resolved, issueMeta, (group) => ({
|
||||||
"Homelab Manager — Health Recovered",
|
title: "Homelab Manager — Health Recovered",
|
||||||
`${resolved.length} issue${resolved.length !== 1 ? "s" : ""} cleared:\n\n${resolved.map((i) => `✓ ${i.message}`).join("\n")}`,
|
message: `${group.length} issue${group.length !== 1 ? "s" : ""} cleared:\n\n${group.map((i) => `✓ ${i.message}`).join("\n")}`,
|
||||||
);
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyAutomationFailed(issues: { message: string; key?: string; source?: string; severity?: string }[]): Promise<void> {
|
export async function notifyAutomationFailed(issues: { message: string; key?: string; source?: string; severity?: string }[]): Promise<void> {
|
||||||
@@ -365,20 +495,20 @@ export async function notifyAutomationFailed(issues: { message: string; key?: st
|
|||||||
for (const i of issues) emitEvent("alert.opened", { alert: { key: i.key ?? null, source: i.source ?? null, severity: i.severity ?? "warning", message: i.message } });
|
for (const i of issues) emitEvent("alert.opened", { alert: { key: i.key ?? null, source: i.source ?? null, severity: i.severity ?? "warning", message: i.message } });
|
||||||
if (issues.length === 0) return;
|
if (issues.length === 0) return;
|
||||||
if (!(await eventEnabled("automationAlerts"))) return;
|
if (!(await eventEnabled("automationAlerts"))) return;
|
||||||
await notify(
|
await notifyGrouped(issues, issueMeta, (group) => ({
|
||||||
"Homelab Manager — Automation Failed",
|
title: "Homelab Manager — Automation Failed",
|
||||||
`${issues.length} run${issues.length !== 1 ? "s" : ""} failed:\n\n${issues.map((i) => `✕ ${i.message}`).join("\n")}`,
|
message: `${group.length} run${group.length !== 1 ? "s" : ""} failed:\n\n${group.map((i) => `✕ ${i.message}`).join("\n")}`,
|
||||||
);
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyAutomationRecovered(resolved: { message: string; key?: string }[]): Promise<void> {
|
export async function notifyAutomationRecovered(resolved: { message: string; key?: string; severity?: string }[]): Promise<void> {
|
||||||
for (const r of resolved) emitEvent("alert.resolved", { alert: { key: r.key ?? null, message: r.message } });
|
for (const r of resolved) emitEvent("alert.resolved", { alert: { key: r.key ?? null, message: r.message } });
|
||||||
if (resolved.length === 0) return;
|
if (resolved.length === 0) return;
|
||||||
if (!(await eventEnabled("automationAlerts"))) return;
|
if (!(await eventEnabled("automationAlerts"))) return;
|
||||||
await notify(
|
await notifyGrouped(resolved, issueMeta, (group) => ({
|
||||||
"Homelab Manager — Automation Recovered",
|
title: "Homelab Manager — Automation Recovered",
|
||||||
`${resolved.length} no longer failing:\n\n${resolved.map((i) => `✓ ${i.message}`).join("\n")}`,
|
message: `${group.length} no longer failing:\n\n${group.map((i) => `✓ ${i.message}`).join("\n")}`,
|
||||||
);
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyDomainExpiry(
|
export async function notifyDomainExpiry(
|
||||||
@@ -403,7 +533,7 @@ export async function notifyDomainExpiry(
|
|||||||
`Couldn't refresh the registration for ${staleChecks.length} domain${staleChecks.length !== 1 ? "s" : ""} — the expiry shown may be stale:\n\n${lines.join("\n")}`,
|
`Couldn't refresh the registration for ${staleChecks.length} domain${staleChecks.length !== 1 ? "s" : ""} — the expiry shown may be stale:\n\n${lines.join("\n")}`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
await notify("Homelab Manager — Domain Expiry", sections.join("\n\n"));
|
await notify("Homelab Manager — Domain Expiry", sections.join("\n\n"), { category: "expiry", severity: expiring.some((d) => d.status === "expired") ? "critical" : "warning" });
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyTailscaleKeyExpiry(
|
export async function notifyTailscaleKeyExpiry(
|
||||||
@@ -417,6 +547,7 @@ export async function notifyTailscaleKeyExpiry(
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Tailscale Key Expiry",
|
"Homelab Manager — Tailscale Key Expiry",
|
||||||
`${expiring.length} device key${expiring.length !== 1 ? "s" : ""} need attention:\n\n${lines.join("\n")}`,
|
`${expiring.length} device key${expiring.length !== 1 ? "s" : ""} need attention:\n\n${lines.join("\n")}`,
|
||||||
|
{ category: "expiry", severity: expiring.some((d) => d.daysLeft < 0) ? "critical" : "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -450,11 +581,12 @@ export async function notifyIntegrationDown(source: string, consecutiveFailures:
|
|||||||
await notify(
|
await notify(
|
||||||
"Homelab Manager — Integration Down",
|
"Homelab Manager — Integration Down",
|
||||||
`${sourceLabel(source)} has failed its last ${consecutiveFailures} call${consecutiveFailures !== 1 ? "s" : ""} in a row. Check the Diagnostic Log for details.`,
|
`${sourceLabel(source)} has failed its last ${consecutiveFailures} call${consecutiveFailures !== 1 ? "s" : ""} in a row. Check the Diagnostic Log for details.`,
|
||||||
|
{ category: "integration", severity: "warning" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function notifyIntegrationRecovered(source: string): Promise<void> {
|
export async function notifyIntegrationRecovered(source: string): Promise<void> {
|
||||||
emitEvent("alert.resolved", { alert: { key: `integration:${source}`, message: `${sourceLabel(source)} is answering again.` } });
|
emitEvent("alert.resolved", { alert: { key: `integration:${source}`, message: `${sourceLabel(source)} is answering again.` } });
|
||||||
if (!(await eventEnabled("integrationFailureAlerts"))) return;
|
if (!(await eventEnabled("integrationFailureAlerts"))) return;
|
||||||
await notify("Homelab Manager — Integration Recovered", `${sourceLabel(source)} succeeded again after failing.`);
|
await notify("Homelab Manager — Integration Recovered", `${sourceLabel(source)} succeeded again after failing.`, { category: "integration", severity: "warning" });
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
/**
|
||||||
|
* Who hears what. By default every notification goes to every channel that is switched on. Routing rules narrow that: the first rule a
|
||||||
|
* notification matches decides which channels get it (possibly none — it is then only on the Alerts page), and each browser that has asked
|
||||||
|
* for push notifications may further say it only wants some severities or kinds. This file is the pure part: what a notification is about,
|
||||||
|
* which rule matches and what a browser accepts. Sending is in notify.ts.
|
||||||
|
*/
|
||||||
|
import type { AlertCategory, AlertSeverity } from "./alertTypes.js";
|
||||||
|
|
||||||
|
export const CHANNEL_IDS = ["gotify", "ntfy", "smtp", "webhook", "webPush"] as const;
|
||||||
|
export type ChannelId = (typeof CHANNEL_IDS)[number];
|
||||||
|
|
||||||
|
export const CATEGORIES = ["offline", "disk", "services", "backup", "updates", "expiry", "automation", "integration", "monitoring", "tickets"] as const satisfies readonly AlertCategory[];
|
||||||
|
|
||||||
|
/** What a notification is about, as far as routing cares. Anything left out matches only rules that don't ask about it. */
|
||||||
|
export interface NotifyMeta {
|
||||||
|
category?: AlertCategory | null;
|
||||||
|
severity?: AlertSeverity;
|
||||||
|
/** The server it is about, whose tags a rule may ask for. */
|
||||||
|
serverId?: number | null;
|
||||||
|
/** Filled in from the server before matching. */
|
||||||
|
tags?: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RoutingRule {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
enabled: boolean;
|
||||||
|
/** Empty = any kind. */
|
||||||
|
categories: AlertCategory[];
|
||||||
|
/** null = any severity. */
|
||||||
|
minSeverity: "warning" | "critical" | null;
|
||||||
|
/** Empty = any server. Otherwise the notification must be about a server that has at least one of these tags. */
|
||||||
|
tags: string[];
|
||||||
|
/** The channels that get it. Empty = none (the Alerts page still shows it). */
|
||||||
|
channels: ChannelId[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RoutingSettings {
|
||||||
|
rules: RoutingRule[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EscalationStep {
|
||||||
|
/** Minutes an alert must have been raised, with nobody on it, before this step fires. */
|
||||||
|
afterMinutes: number;
|
||||||
|
minSeverity: "warning" | "critical";
|
||||||
|
channels: ChannelId[];
|
||||||
|
/** An address to email as well (someone else), through the email channel's server. Blank = none. */
|
||||||
|
extraEmail: string;
|
||||||
|
/** Send even during quiet hours — the point of an escalation can be to wake someone. */
|
||||||
|
evenInQuietHours: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EscalationSettings {
|
||||||
|
enabled: boolean;
|
||||||
|
steps: EscalationStep[];
|
||||||
|
}
|
||||||
|
|
||||||
|
const RANK: Record<string, number> = { info: 0, warning: 1, critical: 2 };
|
||||||
|
export const severityRank = (s: string | undefined | null) => RANK[s ?? "warning"] ?? 1;
|
||||||
|
|
||||||
|
/** Whether a rule applies to a notification. A rule that asks about something the notification doesn't say doesn't match. */
|
||||||
|
export function ruleMatches(rule: RoutingRule, meta: NotifyMeta): boolean {
|
||||||
|
if (!rule.enabled) return false;
|
||||||
|
if (rule.categories.length > 0 && !(meta.category && rule.categories.includes(meta.category))) return false;
|
||||||
|
if (rule.minSeverity && severityRank(meta.severity) < severityRank(rule.minSeverity)) return false;
|
||||||
|
if (rule.tags.length > 0) {
|
||||||
|
const have = new Set((meta.tags ?? []).map((t) => t.toLowerCase()));
|
||||||
|
if (!rule.tags.some((t) => have.has(t.toLowerCase()))) return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The first rule that applies, or null (every channel that is on). */
|
||||||
|
export function firstMatchingRule(rules: RoutingRule[], meta: NotifyMeta): RoutingRule | null {
|
||||||
|
return rules.find((r) => ruleMatches(r, meta)) ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A browser's own wishes, as stored. */
|
||||||
|
export interface PushPrefs {
|
||||||
|
minSeverity: string | null;
|
||||||
|
categories: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseCategories(stored: string | null): AlertCategory[] | null {
|
||||||
|
if (!stored) return null;
|
||||||
|
try {
|
||||||
|
const v = JSON.parse(stored);
|
||||||
|
if (!Array.isArray(v)) return null;
|
||||||
|
const list = v.filter((c): c is AlertCategory => (CATEGORIES as readonly string[]).includes(c));
|
||||||
|
return list.length > 0 ? list : null;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Whether a browser wants this notification. What the notification doesn't say can't exclude it. */
|
||||||
|
export function pushAccepts(prefs: PushPrefs, meta: NotifyMeta | undefined): boolean {
|
||||||
|
if (!meta) return true;
|
||||||
|
if (prefs.minSeverity && meta.severity && severityRank(meta.severity) < severityRank(prefs.minSeverity)) return false;
|
||||||
|
const wanted = parseCategories(prefs.categories);
|
||||||
|
if (wanted && meta.category && !wanted.includes(meta.category)) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What kind of problem a health or automation condition key is about, and which server (for its tags). */
|
||||||
|
export function metaOfConditionKey(key: string, severity?: string): NotifyMeta {
|
||||||
|
let category: AlertCategory | null = null;
|
||||||
|
if (key.startsWith("offline:")) category = "offline";
|
||||||
|
else if (key.startsWith("failed-units:")) category = "services";
|
||||||
|
else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk";
|
||||||
|
else if (key.startsWith("automation:")) category = "automation";
|
||||||
|
const server = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)/.exec(key);
|
||||||
|
return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null };
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ import { sql } from "drizzle-orm";
|
|||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { settings } from "../db/schema.js";
|
import { settings } from "../db/schema.js";
|
||||||
import { defaultThemes, type NameTheme } from "./nameThemes.js";
|
import { defaultThemes, type NameTheme } from "./nameThemes.js";
|
||||||
|
import type { EscalationSettings, RoutingSettings } from "./notifyRouting.js";
|
||||||
import { env } from "../env.js";
|
import { env } from "../env.js";
|
||||||
import { decryptSecret, encryptSecret } from "../crypto.js";
|
import { decryptSecret, encryptSecret } from "../crypto.js";
|
||||||
|
|
||||||
@@ -44,6 +45,8 @@ export interface WebPushSettings {
|
|||||||
contact: string;
|
contact: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type { RoutingSettings, EscalationSettings } from "./notifyRouting.js";
|
||||||
|
|
||||||
export interface NotificationEvents {
|
export interface NotificationEvents {
|
||||||
dnsAdd: boolean;
|
dnsAdd: boolean;
|
||||||
dnsUpdate: boolean;
|
dnsUpdate: boolean;
|
||||||
@@ -197,6 +200,10 @@ export interface AppSettings {
|
|||||||
smtp: SmtpSettings;
|
smtp: SmtpSettings;
|
||||||
webhook: WebhookSettings;
|
webhook: WebhookSettings;
|
||||||
webPush: WebPushSettings;
|
webPush: WebPushSettings;
|
||||||
|
/** Which channels hear which notifications. */
|
||||||
|
routing: RoutingSettings;
|
||||||
|
/** Re-notifying about alerts nobody has acknowledged. */
|
||||||
|
escalation: EscalationSettings;
|
||||||
notifications: NotificationEvents;
|
notifications: NotificationEvents;
|
||||||
providerColors: ProviderColors;
|
providerColors: ProviderColors;
|
||||||
integrationColors: IntegrationColors;
|
integrationColors: IntegrationColors;
|
||||||
@@ -220,6 +227,8 @@ const DEFAULTS: AppSettings = {
|
|||||||
smtp: { enabled: false, host: "", port: 587, secure: false, username: "", password: "", from: "", to: "" },
|
smtp: { enabled: false, host: "", port: 587, secure: false, username: "", password: "", from: "", to: "" },
|
||||||
webhook: { enabled: false, url: "", secret: "" },
|
webhook: { enabled: false, url: "", secret: "" },
|
||||||
webPush: { enabled: false, contact: "" },
|
webPush: { enabled: false, contact: "" },
|
||||||
|
routing: { rules: [] },
|
||||||
|
escalation: { enabled: false, steps: [] },
|
||||||
notifications: {
|
notifications: {
|
||||||
dnsAdd: true,
|
dnsAdd: true,
|
||||||
dnsUpdate: true,
|
dnsUpdate: true,
|
||||||
|
|||||||
@@ -0,0 +1,450 @@
|
|||||||
|
/**
|
||||||
|
* Who hears what (routing rules and browsers' wishes) and escalation of alerts nobody has acknowledged.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { createServer, type Server } from "node:http";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-routing-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, pushSubscriptions, alertIncidents, auditLog, notificationQueue } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { ruleMatches, firstMatchingRule, pushAccepts, metaOfConditionKey } = await import("../src/services/notifyRouting.js");
|
||||||
|
const { dueEscalations, runEscalation } = await import("../src/services/escalation.js");
|
||||||
|
const { audienceFor, notifyHealthIssues, notifyHealthRecovered, flushQuietHoursQueue, notify } = await import("../src/services/notify.js");
|
||||||
|
const { updateSettings } = await import("../src/services/settingsStore.js");
|
||||||
|
const { diffConditions } = await import("../src/services/healthMonitor.js");
|
||||||
|
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
|
||||||
|
const { setAck } = await import("../src/services/alertAcks.js");
|
||||||
|
const { settingsRouter } = await import("../src/routes/settings.js");
|
||||||
|
const { eq } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
type Rule = import("../src/services/notifyRouting.js").RoutingRule;
|
||||||
|
const rule = (extra: Partial<Rule>): Rule => ({ id: "r", name: "rule", enabled: true, categories: [], minSeverity: null, tags: [], channels: [], ...extra });
|
||||||
|
|
||||||
|
describe("routing rules", () => {
|
||||||
|
test("a rule matches on kind, severity and the server's tags; asking about something the notification doesn't say never matches", () => {
|
||||||
|
assert.equal(ruleMatches(rule({}), {}), true, "a rule that asks nothing matches everything");
|
||||||
|
assert.equal(ruleMatches(rule({ categories: ["disk"] }), { category: "disk" }), true);
|
||||||
|
assert.equal(ruleMatches(rule({ categories: ["disk"] }), { category: "offline" }), false);
|
||||||
|
assert.equal(ruleMatches(rule({ categories: ["disk"] }), {}), false, "no kind to compare");
|
||||||
|
assert.equal(ruleMatches(rule({ minSeverity: "critical" }), { severity: "warning" }), false);
|
||||||
|
assert.equal(ruleMatches(rule({ minSeverity: "critical" }), { severity: "critical" }), true);
|
||||||
|
assert.equal(ruleMatches(rule({ minSeverity: "warning" }), { severity: "critical" }), true);
|
||||||
|
assert.equal(ruleMatches(rule({ tags: ["prod"] }), { tags: ["Prod", "db"] }), true, "tags compare without regard to case");
|
||||||
|
assert.equal(ruleMatches(rule({ tags: ["prod"] }), { tags: ["lab"] }), false);
|
||||||
|
assert.equal(ruleMatches(rule({ tags: ["prod"] }), {}), false);
|
||||||
|
assert.equal(ruleMatches(rule({ enabled: false }), {}), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the first rule that matches decides; none matching is no decision", () => {
|
||||||
|
const rules = [rule({ id: "a", categories: ["disk"], channels: ["ntfy"] }), rule({ id: "b", minSeverity: "critical", channels: ["gotify", "webPush"] }), rule({ id: "c", channels: [] })];
|
||||||
|
assert.equal(firstMatchingRule(rules, { category: "disk", severity: "critical" })!.id, "a", "order matters");
|
||||||
|
assert.equal(firstMatchingRule(rules, { category: "offline", severity: "critical" })!.id, "b");
|
||||||
|
assert.equal(firstMatchingRule(rules, { category: "offline", severity: "warning" })!.id, "c");
|
||||||
|
assert.equal(firstMatchingRule([rules[0]], { category: "offline" }), null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a problem's key says what kind it is and which server", () => {
|
||||||
|
assert.deepEqual(metaOfConditionKey("offline:server:7", "critical"), { category: "offline", severity: "critical", serverId: 7 });
|
||||||
|
assert.deepEqual(metaOfConditionKey("disk:server:7:/data"), { category: "disk", severity: "warning", serverId: 7 });
|
||||||
|
assert.deepEqual(metaOfConditionKey("failed-units:server:2"), { category: "services", severity: "warning", serverId: 2 });
|
||||||
|
assert.deepEqual(metaOfConditionKey("disk:proxmox:3:local"), { category: "disk", severity: "warning", serverId: null });
|
||||||
|
assert.deepEqual(metaOfConditionKey("automation:semaphore:1:5"), { category: "automation", severity: "warning", serverId: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a browser's wishes: a floor on severity and a list of kinds; what a notification doesn't say can't exclude it", () => {
|
||||||
|
assert.equal(pushAccepts({ minSeverity: null, categories: null }, { category: "disk", severity: "warning" }), true);
|
||||||
|
assert.equal(pushAccepts({ minSeverity: "critical", categories: null }, { severity: "warning" }), false);
|
||||||
|
assert.equal(pushAccepts({ minSeverity: "critical", categories: null }, { severity: "critical" }), true);
|
||||||
|
assert.equal(pushAccepts({ minSeverity: null, categories: JSON.stringify(["offline", "disk"]) }, { category: "backup" }), false);
|
||||||
|
assert.equal(pushAccepts({ minSeverity: null, categories: JSON.stringify(["offline", "disk"]) }, { category: "disk" }), true);
|
||||||
|
assert.equal(pushAccepts({ minSeverity: "critical", categories: JSON.stringify(["disk"]) }, undefined), true, "no meta, no judgement");
|
||||||
|
assert.equal(pushAccepts({ minSeverity: null, categories: JSON.stringify(["disk"]) }, { severity: "warning" }), true, "no kind, no judgement");
|
||||||
|
assert.equal(pushAccepts({ minSeverity: null, categories: "garbage" }, { category: "backup" }), true, "unreadable wishes are no wishes");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what the health monitor remembers", () => {
|
||||||
|
test("a recovery carries how bad the problem was, so it is routed like the alert was", () => {
|
||||||
|
const first = diffConditions({}, [
|
||||||
|
{ key: "offline:server:1", source: "server", message: "a is down", severity: "critical" },
|
||||||
|
{ key: "disk:server:1:/", source: "server", message: "a disk" },
|
||||||
|
]);
|
||||||
|
assert.equal(first.next["offline:server:1"].severity, "critical");
|
||||||
|
assert.equal(first.next["disk:server:1:/"].severity, undefined);
|
||||||
|
const gone = diffConditions(first.next, []);
|
||||||
|
assert.deepEqual(gone.resolved.map((r) => [r.key, r.severity]).sort(), [["disk:server:1:/", undefined], ["offline:server:1", "critical"]]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("escalation: what is due", () => {
|
||||||
|
const steps = [
|
||||||
|
{ afterMinutes: 30, minSeverity: "warning" as const, channels: ["webPush" as const], extraEmail: "", evenInQuietHours: false },
|
||||||
|
{ afterMinutes: 120, minSeverity: "critical" as const, channels: ["smtp" as const], extraEmail: "neighbour@example.com", evenInQuietHours: true },
|
||||||
|
];
|
||||||
|
const now = new Date("2026-10-18T12:00:00Z");
|
||||||
|
const ago = (min: number) => new Date(now.getTime() - min * 60_000).toISOString();
|
||||||
|
const alert = (id: string, severity: "critical" | "warning" | "info", extra: Record<string, unknown> = {}) => ({ id, severity, silenced: false, message: id, source: "Server", ...extra });
|
||||||
|
const run = (alerts: ReturnType<typeof alert>[], incidents: Record<string, { openedAt: string; escalatedStep?: number }>, acknowledged: string[] = []) =>
|
||||||
|
dueEscalations({
|
||||||
|
alerts,
|
||||||
|
acknowledged: new Set(acknowledged),
|
||||||
|
incidents: new Map(Object.entries(incidents).map(([k, v], i) => [k, { id: i + 1, openedAt: v.openedAt, escalatedStep: v.escalatedStep ?? 0 }])),
|
||||||
|
steps,
|
||||||
|
now,
|
||||||
|
}).map((d) => [d.alert.id, d.step]);
|
||||||
|
|
||||||
|
test("a step is due once the alert has been up that long; nothing before", () => {
|
||||||
|
assert.deepEqual(run([alert("a", "warning")], { a: { openedAt: ago(29) } }), []);
|
||||||
|
assert.deepEqual(run([alert("a", "warning")], { a: { openedAt: ago(31) } }), [["a", 1]]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a step is sent once, and a neglected alert is told about one step at a time", () => {
|
||||||
|
assert.deepEqual(run([alert("a", "critical")], { a: { openedAt: ago(200), escalatedStep: 0 } }), [["a", 1]], "step 1 first, though step 2 is due too");
|
||||||
|
assert.deepEqual(run([alert("a", "critical")], { a: { openedAt: ago(200), escalatedStep: 1 } }), [["a", 2]]);
|
||||||
|
assert.deepEqual(run([alert("a", "critical")], { a: { openedAt: ago(200), escalatedStep: 2 } }), []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a step that asks for a worse severity is passed over, and later ones still apply", () => {
|
||||||
|
// a warning is never bad enough for step 2, so it ends after step 1
|
||||||
|
assert.deepEqual(run([alert("w", "warning")], { w: { openedAt: ago(500), escalatedStep: 1 } }), []);
|
||||||
|
const critOnlyFirst = [{ ...steps[0], minSeverity: "critical" as const }, { ...steps[1], minSeverity: "warning" as const }];
|
||||||
|
const due = dueEscalations({ alerts: [alert("w", "warning")], acknowledged: new Set(), incidents: new Map([["w", { id: 1, openedAt: ago(500), escalatedStep: 0 }]]), steps: critOnlyFirst, now });
|
||||||
|
assert.deepEqual(due.map((d) => d.step), [2], "step 1 wasn't for it, step 2 was");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an acknowledgement stops it; so do maintenance, being held back by another alert, and info", () => {
|
||||||
|
assert.deepEqual(run([alert("a", "critical")], { a: { openedAt: ago(200) } }, ["a"]), []);
|
||||||
|
assert.deepEqual(run([alert("a", "critical", { silenced: true })], { a: { openedAt: ago(200) } }), []);
|
||||||
|
assert.deepEqual(run([alert("a", "info")], { a: { openedAt: ago(200) } }), []);
|
||||||
|
assert.deepEqual(run([alert("a", "critical")], {}), [], "no incident recorded yet, nothing to measure from");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("through the channels", () => {
|
||||||
|
type Hit = { path: string; body: string };
|
||||||
|
const hits: Record<"gotify" | "ntfy" | "webhook", Hit[]> = { gotify: [], ntfy: [], webhook: [] };
|
||||||
|
const fake: Server[] = [];
|
||||||
|
const base: Record<string, string> = {};
|
||||||
|
let proddy = 0;
|
||||||
|
|
||||||
|
const text = (h: Hit) => {
|
||||||
|
try {
|
||||||
|
const j = JSON.parse(h.body);
|
||||||
|
return String(j.message ?? j.content ?? h.body);
|
||||||
|
} catch {
|
||||||
|
return h.body;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const clear = () => {
|
||||||
|
for (const k of Object.keys(hits) as (keyof typeof hits)[]) hits[k].length = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
for (const name of ["gotify", "ntfy", "webhook"] as const) {
|
||||||
|
const s = createServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (c) => chunks.push(c));
|
||||||
|
req.on("end", () => {
|
||||||
|
hits[name].push({ path: req.url ?? "", body: Buffer.concat(chunks).toString() });
|
||||||
|
res.statusCode = 200;
|
||||||
|
res.end("{}");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await new Promise<void>((r) => s.listen(0, "127.0.0.1", r));
|
||||||
|
fake.push(s);
|
||||||
|
base[name] = `http://127.0.0.1:${(s.address() as AddressInfo).port}`;
|
||||||
|
}
|
||||||
|
await updateSettings({
|
||||||
|
gotify: { enabled: true, url: base.gotify, token: "g" },
|
||||||
|
ntfy: { enabled: true, url: base.ntfy, topic: "lab" },
|
||||||
|
webhook: { enabled: true, url: base.webhook },
|
||||||
|
});
|
||||||
|
proddy = (await db.insert(servers).values({ name: "prod-db", apiTokenHash: "p", apiTokenPrefix: "p", tags: JSON.stringify(["prod"]) }).returning({ id: servers.id }))[0].id;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
for (const s of fake) s.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const offline = (id: number) => ({ key: `offline:server:${id}`, message: `server ${id} is offline`, severity: "critical" });
|
||||||
|
const disk = (id: number) => ({ key: `disk:server:${id}:/`, message: `disk on ${id} is full`, severity: "warning" });
|
||||||
|
|
||||||
|
test("with no rules everything goes everywhere, in one message, as it always did", async () => {
|
||||||
|
clear();
|
||||||
|
await notifyHealthIssues([offline(proddy), disk(proddy)]);
|
||||||
|
for (const k of ["gotify", "ntfy", "webhook"] as const) {
|
||||||
|
assert.equal(hits[k].length, 1, k);
|
||||||
|
assert.match(text(hits[k][0]), /2 new issues/);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a rule sends one kind to one channel; the rest still goes everywhere, as a message of its own", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [rule({ id: "d", name: "Disks to ntfy", categories: ["disk"], channels: ["ntfy"] })] } });
|
||||||
|
clear();
|
||||||
|
await notifyHealthIssues([offline(proddy), disk(proddy)]);
|
||||||
|
assert.equal(hits.gotify.length, 1);
|
||||||
|
assert.match(text(hits.gotify[0]), /1 new issue:[\s\S]*is offline/);
|
||||||
|
assert.doesNotMatch(text(hits.gotify[0]), /disk/);
|
||||||
|
assert.equal(hits.webhook.length, 1);
|
||||||
|
assert.equal(hits.ntfy.length, 2, "its own message for the disk, and the offline one with the rest");
|
||||||
|
assert.ok(hits.ntfy.some((h) => /disk on/.test(text(h)) && !/offline/.test(text(h))));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a rule can mute: nobody is told, and nothing is queued", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [rule({ id: "m", name: "Prod disks: nobody", categories: ["disk"], tags: ["prod"], channels: [] })] } });
|
||||||
|
clear();
|
||||||
|
await notifyHealthIssues([disk(proddy)]);
|
||||||
|
assert.deepEqual([hits.gotify.length, hits.ntfy.length, hits.webhook.length], [0, 0, 0]);
|
||||||
|
// the same disk problem on a server without the tag is not muted
|
||||||
|
const plain = (await db.insert(servers).values({ name: "plain", apiTokenHash: "q", apiTokenPrefix: "q" }).returning({ id: servers.id }))[0].id;
|
||||||
|
await notifyHealthIssues([disk(plain)]);
|
||||||
|
assert.equal(hits.gotify.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a recovery follows the route its alert took", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [rule({ id: "c", name: "Critical to webhook", minSeverity: "critical", channels: ["webhook"] })] } });
|
||||||
|
clear();
|
||||||
|
await notifyHealthRecovered([{ key: `offline:server:${proddy}`, message: "back", severity: "critical" }]);
|
||||||
|
assert.deepEqual([hits.gotify.length, hits.ntfy.length, hits.webhook.length], [0, 0, 1]);
|
||||||
|
clear();
|
||||||
|
await notifyHealthRecovered([{ key: `disk:server:${proddy}:/`, message: "ok", severity: "warning" }]);
|
||||||
|
assert.deepEqual([hits.gotify.length, hits.ntfy.length, hits.webhook.length], [1, 1, 1], "a warning isn't covered by that rule, so it goes everywhere");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a rule that is switched off does nothing", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [rule({ id: "x", enabled: false, channels: [] })] } });
|
||||||
|
clear();
|
||||||
|
await notifyHealthIssues([offline(proddy)]);
|
||||||
|
assert.equal(hits.webhook.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("held for quiet hours, a notification is still only for the channels its rule picked", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [rule({ id: "q", categories: ["backup"], channels: ["ntfy"] })] }, quietHours: { enabled: true, start: "00:00", end: "23:59" } });
|
||||||
|
clear();
|
||||||
|
await notify("Backups", "a backup is stale", { category: "backup", severity: "warning" });
|
||||||
|
assert.deepEqual([hits.gotify.length, hits.ntfy.length, hits.webhook.length], [0, 0, 0], "held");
|
||||||
|
const queued = await db.select().from(notificationQueue);
|
||||||
|
assert.equal(queued.length, 1);
|
||||||
|
assert.ok(queued[0].audience, "who it was for is kept");
|
||||||
|
await updateSettings({ quietHours: { enabled: false } });
|
||||||
|
assert.equal(await flushQuietHoursQueue(), 1);
|
||||||
|
assert.deepEqual([hits.gotify.length, hits.ntfy.length, hits.webhook.length], [0, 1, 0]);
|
||||||
|
assert.equal((await db.select().from(notificationQueue)).length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("browsers' wishes narrow who a push goes to, and only when somebody has stated any", async () => {
|
||||||
|
await updateSettings({ routing: { rules: [] } });
|
||||||
|
const [u] = await db.insert(users).values({ oidcSub: "u1", name: "U", role: "operator" }).returning({ id: users.id });
|
||||||
|
const sub = (n: string, extra: Partial<typeof pushSubscriptions.$inferInsert>) => db.insert(pushSubscriptions).values({ userId: u.id, endpoint: `https://push.example/${n}`, p256dh: "k", auth: "a", ...extra }).returning({ id: pushSubscriptions.id });
|
||||||
|
const [all] = await sub("all", {});
|
||||||
|
assert.equal((await audienceFor({ category: "disk", severity: "warning" })).pushIds, null, "nobody asked for less");
|
||||||
|
const [critical] = await sub("critical", { minSeverity: "critical" });
|
||||||
|
const [disks] = await sub("disks", { categories: JSON.stringify(["disk"]) });
|
||||||
|
const a = await audienceFor({ category: "disk", severity: "warning" });
|
||||||
|
assert.deepEqual([...a.pushIds!].sort(), [all.id, disks.id].sort(), "the one wanting only critical stays out");
|
||||||
|
const b = await audienceFor({ category: "backup", severity: "critical" });
|
||||||
|
assert.deepEqual([...b.pushIds!].sort(), [all.id, critical.id].sort(), "and the one wanting only disks stays out of backups");
|
||||||
|
assert.equal((await audienceFor()).pushIds, null, "a notification that says nothing about itself goes to all");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("escalation, end to end", () => {
|
||||||
|
const hits: { path: string; body: string }[] = [];
|
||||||
|
let fakeHook: Server;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
fakeHook = createServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (c) => chunks.push(c));
|
||||||
|
req.on("end", () => {
|
||||||
|
hits.push({ path: req.url ?? "", body: Buffer.concat(chunks).toString() });
|
||||||
|
res.end("{}");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await new Promise<void>((r) => fakeHook.listen(0, "127.0.0.1", r));
|
||||||
|
await db.delete(servers);
|
||||||
|
await db.insert(servers).values({ name: "late", apiTokenHash: "l", apiTokenPrefix: "l", lastSeenAt: new Date(Date.now() - 5 * 3600_000).toISOString() });
|
||||||
|
await updateSettings({
|
||||||
|
gotify: { enabled: false },
|
||||||
|
ntfy: { enabled: false },
|
||||||
|
webhook: { enabled: true, url: `http://127.0.0.1:${(fakeHook.address() as AddressInfo).port}/hook` },
|
||||||
|
routing: { rules: [] },
|
||||||
|
quietHours: { enabled: false },
|
||||||
|
escalation: {
|
||||||
|
enabled: true,
|
||||||
|
steps: [
|
||||||
|
{ afterMinutes: 30, minSeverity: "warning", channels: ["webhook"], extraEmail: "", evenInQuietHours: false },
|
||||||
|
{ afterMinutes: 90, minSeverity: "critical", channels: ["webhook"], extraEmail: "", evenInQuietHours: true },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => fakeHook.close());
|
||||||
|
|
||||||
|
const setAge = async (minutes: number) => {
|
||||||
|
await db.update(alertIncidents).set({ openedAt: new Date(Date.now() - minutes * 60_000).toISOString() });
|
||||||
|
};
|
||||||
|
const text = () => hits.map((h) => JSON.parse(h.body).content as string);
|
||||||
|
|
||||||
|
test("nothing happens until the alert has been up long enough; then the steps go out one by one, each once", async () => {
|
||||||
|
invalidateAlertsCache();
|
||||||
|
await getAlerts(true);
|
||||||
|
await new Promise((r) => setTimeout(r, 300)); // the incident is recorded in the background
|
||||||
|
assert.equal(await runEscalation(), 0, "just raised");
|
||||||
|
assert.equal(hits.length, 0);
|
||||||
|
|
||||||
|
await setAge(40);
|
||||||
|
assert.equal(await runEscalation(), 1);
|
||||||
|
assert.equal(hits.length, 1);
|
||||||
|
assert.match(text()[0], /Still not acknowledged/);
|
||||||
|
assert.match(text()[0], /late hasn't reported/);
|
||||||
|
assert.match(text()[0], /after 30 minutes/);
|
||||||
|
assert.equal(await runEscalation(), 0, "step 1 is not repeated");
|
||||||
|
|
||||||
|
await setAge(100);
|
||||||
|
assert.equal(await runEscalation(), 1);
|
||||||
|
assert.match(text()[1], /after 90 minutes/);
|
||||||
|
assert.equal(await runEscalation(), 0, "and that was the last step");
|
||||||
|
const [inc] = await db.select().from(alertIncidents);
|
||||||
|
assert.equal(inc.escalatedStep, 2);
|
||||||
|
assert.equal((await db.select().from(auditLog)).filter((e) => e.action === "escalate").length, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an acknowledgement stops it, and so does switching it off", async () => {
|
||||||
|
await db.update(alertIncidents).set({ escalatedStep: 0, escalatedAt: null });
|
||||||
|
await setAge(500);
|
||||||
|
const before = hits.length;
|
||||||
|
const alert = (await getAlerts(false)).alerts.find((a) => a.category === "offline")!;
|
||||||
|
await setAck({ alertId: alert.id, note: "on it", assigneeId: null, hours: null, actor: "Ada" });
|
||||||
|
assert.equal(await runEscalation(), 0);
|
||||||
|
await db.delete((await import("../src/db/schema.js")).alertAcks);
|
||||||
|
await updateSettings({ escalation: { enabled: false } });
|
||||||
|
assert.equal(await runEscalation(), 0);
|
||||||
|
assert.equal(hits.length, before);
|
||||||
|
await updateSettings({ escalation: { enabled: true } });
|
||||||
|
assert.equal(await runEscalation(), 1, "and it carries on once it is on and nobody is on it");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("saving the settings", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values({ oidcSub: "admin", name: "Ada", role: "admin" }).onConflictDoNothing();
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: "admin" } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/settings", settingsRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/settings`;
|
||||||
|
});
|
||||||
|
after(() => app.close());
|
||||||
|
|
||||||
|
const put = (body: unknown) => fetch(base, { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
|
||||||
|
const step = (afterMinutes: number, extra: Record<string, unknown> = {}) => ({ afterMinutes, minSeverity: "critical", channels: ["webPush"], extraEmail: "", evenInQuietHours: false, ...extra });
|
||||||
|
|
||||||
|
test("rules are kept, with their tags written the way server tags are", async () => {
|
||||||
|
const res = await put({ routing: { rules: [{ id: "a1", name: "Prod", enabled: true, categories: ["disk"], minSeverity: "critical", tags: [" Prod ", "Db Servers"], channels: ["ntfy"] }] } });
|
||||||
|
assert.equal(res.status, 200);
|
||||||
|
const saved = (await (await fetch(base)).json()).settings.routing.rules[0];
|
||||||
|
assert.deepEqual(saved.tags, ["db-servers", "prod"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a rule that isn't one is refused", async () => {
|
||||||
|
assert.equal((await put({ routing: { rules: [{ id: "a", name: "x", enabled: true, categories: ["toast"], minSeverity: null, tags: [], channels: [] }] } })).status, 400);
|
||||||
|
assert.equal((await put({ routing: { rules: [{ id: "a", name: "x", enabled: true, categories: [], minSeverity: null, tags: [], channels: ["carrier-pigeon"] }] } })).status, 400);
|
||||||
|
assert.equal((await put({ routing: { rules: [{ id: "a", name: "x", enabled: true, categories: [], minSeverity: null, tags: ["not a tag!"], channels: [] }] } })).status, 400);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("escalation steps must be in time order, say who to tell, and be few", async () => {
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(30), step(120, { extraEmail: "x@example.com", channels: [] })] } })).status, 200);
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(60), step(30)] } })).status, 400, "out of order");
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(30, { channels: [], extraEmail: "" })] } })).status, 400, "nobody to tell");
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(30, { extraEmail: "not an address" })] } })).status, 400);
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(10), step(20), step(30), step(40)] } })).status, 400, "at most three");
|
||||||
|
assert.equal((await put({ escalation: { enabled: true, steps: [step(2)] } })).status, 400, "not sooner than five minutes");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("ntfy titles", () => {
|
||||||
|
test("a title with an em dash can be sent as a header; plain ones are left as they are", async () => {
|
||||||
|
const { headerSafe } = await import("../src/services/notify.js");
|
||||||
|
assert.equal(headerSafe("Plain title"), "Plain title");
|
||||||
|
const encoded = headerSafe("Homelab Manager — Health Alert");
|
||||||
|
assert.match(encoded, /^=\?UTF-8\?B\?[A-Za-z0-9+/=]+\?=$/);
|
||||||
|
assert.equal(Buffer.from(encoded.slice(10, -2), "base64").toString("utf8"), "Homelab Manager — Health Alert");
|
||||||
|
assert.doesNotThrow(() => new Headers({ Title: encoded }), "and the runtime takes it");
|
||||||
|
assert.throws(() => new Headers({ Title: "Homelab Manager — Health Alert" }), "which it would not take as it was");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("a browser's preferences, over the routes", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "p1";
|
||||||
|
let mine = 0;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
const { pushRouter } = await import("../src/routes/push.js");
|
||||||
|
const made = await db.insert(users).values([{ oidcSub: "p1", name: "P1", role: "viewer" }, { oidcSub: "p2", name: "P2", role: "viewer" }]).returning({ id: users.id });
|
||||||
|
mine = (await db.insert(pushSubscriptions).values({ userId: made[0].id, endpoint: "https://push.example/mine", p256dh: "k", auth: "a" }).returning({ id: pushSubscriptions.id }))[0].id;
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/push", pushRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/push`;
|
||||||
|
});
|
||||||
|
after(() => app.close());
|
||||||
|
|
||||||
|
const patch = (id: number, body: unknown) => fetch(`${base}/${id}`, { method: "PATCH", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
|
||||||
|
|
||||||
|
test("a person sets what reaches a browser and reads it back; someone else's browser is not theirs to set", async () => {
|
||||||
|
const first = await patch(mine, { minSeverity: "critical", categories: ["offline", "disk"] });
|
||||||
|
assert.equal(first.status, 200, JSON.stringify(await first.clone().json()));
|
||||||
|
const list = (await (await fetch(`${base}/mine`)).json()).subscriptions[0];
|
||||||
|
assert.equal(list.minSeverity, "critical");
|
||||||
|
assert.deepEqual(list.categories, ["offline", "disk"]);
|
||||||
|
assert.equal((await patch(mine, { minSeverity: null, categories: [] })).status, 200, "an empty list is every kind");
|
||||||
|
const cleared = (await (await fetch(`${base}/mine`)).json()).subscriptions[0];
|
||||||
|
assert.deepEqual([cleared.minSeverity, cleared.categories], [null, null]);
|
||||||
|
who = "p2";
|
||||||
|
assert.equal((await patch(mine, { minSeverity: "critical", categories: null })).status, 404);
|
||||||
|
who = "p1";
|
||||||
|
assert.equal((await patch(mine, { minSeverity: "info", categories: null })).status, 400, "info never notifies, so it isn't a floor");
|
||||||
|
assert.equal((await patch(mine, { minSeverity: null, categories: ["toast"] })).status, 400);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -409,6 +409,32 @@ export interface WebhookSettings {
|
|||||||
secret: string;
|
secret: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type NotifyChannelId = "gotify" | "ntfy" | "smtp" | "webhook" | "webPush";
|
||||||
|
|
||||||
|
export interface RoutingRule {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
enabled: boolean;
|
||||||
|
categories: AlertCategory[];
|
||||||
|
minSeverity: "warning" | "critical" | null;
|
||||||
|
tags: string[];
|
||||||
|
channels: NotifyChannelId[];
|
||||||
|
}
|
||||||
|
export interface RoutingSettings {
|
||||||
|
rules: RoutingRule[];
|
||||||
|
}
|
||||||
|
export interface EscalationStep {
|
||||||
|
afterMinutes: number;
|
||||||
|
minSeverity: "warning" | "critical";
|
||||||
|
channels: NotifyChannelId[];
|
||||||
|
extraEmail: string;
|
||||||
|
evenInQuietHours: boolean;
|
||||||
|
}
|
||||||
|
export interface EscalationSettings {
|
||||||
|
enabled: boolean;
|
||||||
|
steps: EscalationStep[];
|
||||||
|
}
|
||||||
|
|
||||||
export interface WebPushSettings {
|
export interface WebPushSettings {
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
/** An email address or https address a push service can reach whoever runs this on. Blank is fine. */
|
/** An email address or https address a push service can reach whoever runs this on. Blank is fine. */
|
||||||
@@ -423,6 +449,10 @@ export interface PushDevice {
|
|||||||
endpoint: string;
|
endpoint: string;
|
||||||
createdAt: string;
|
createdAt: string;
|
||||||
lastSuccessAt: string | null;
|
lastSuccessAt: string | null;
|
||||||
|
/** This browser only wants alerts at least this bad; null = every severity. */
|
||||||
|
minSeverity: "warning" | "critical" | null;
|
||||||
|
/** Only these kinds of alert; null = every kind. */
|
||||||
|
categories: AlertCategory[] | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface NotificationEvents {
|
export interface NotificationEvents {
|
||||||
@@ -490,6 +520,8 @@ export interface AppSettings {
|
|||||||
smtp: SmtpSettings;
|
smtp: SmtpSettings;
|
||||||
webhook: WebhookSettings;
|
webhook: WebhookSettings;
|
||||||
webPush: WebPushSettings;
|
webPush: WebPushSettings;
|
||||||
|
routing: RoutingSettings;
|
||||||
|
escalation: EscalationSettings;
|
||||||
notifications: NotificationEvents;
|
notifications: NotificationEvents;
|
||||||
providerColors: Record<string, string>;
|
providerColors: Record<string, string>;
|
||||||
integrationColors: Record<string, string>;
|
integrationColors: Record<string, string>;
|
||||||
@@ -1718,6 +1750,8 @@ export const api = {
|
|||||||
mine: () => request<{ enabled: boolean; subscriptions: PushDevice[] }>("/api/push/mine"),
|
mine: () => request<{ enabled: boolean; subscriptions: PushDevice[] }>("/api/push/mine"),
|
||||||
subscribe: (data: { endpoint: string; keys: { p256dh: string; auth: string }; label?: string }) => request<{ id: number }>("/api/push/subscribe", { method: "POST", body: JSON.stringify(data) }),
|
subscribe: (data: { endpoint: string; keys: { p256dh: string; auth: string }; label?: string }) => request<{ id: number }>("/api/push/subscribe", { method: "POST", body: JSON.stringify(data) }),
|
||||||
remove: (id: number) => request<void>(`/api/push/${id}`, { method: "DELETE" }),
|
remove: (id: number) => request<void>(`/api/push/${id}`, { method: "DELETE" }),
|
||||||
|
setPrefs: (id: number, data: { minSeverity: "warning" | "critical" | null; categories: AlertCategory[] | null }) =>
|
||||||
|
request<{ minSeverity: string | null; categories: AlertCategory[] | null }>(`/api/push/${id}`, { method: "PATCH", body: JSON.stringify(data) }),
|
||||||
test: () => request<{ sent: number; removed: number; failed: number }>("/api/push/test", { method: "POST" }),
|
test: () => request<{ sent: number; removed: number; failed: number }>("/api/push/test", { method: "POST" }),
|
||||||
},
|
},
|
||||||
dependencies: {
|
dependencies: {
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { useCallback, useEffect, useState } from "react";
|
import { useCallback, useEffect, useState } from "react";
|
||||||
import { api, type PushDevice } from "../api/client";
|
import { api, type AlertCategory, type PushDevice } from "../api/client";
|
||||||
|
import { CATEGORY_LABELS } from "./RoutingEditor";
|
||||||
import { readableError } from "../utils/errors";
|
import { readableError } from "../utils/errors";
|
||||||
import { confirmDialog } from "../utils/dialogs";
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
import { toast } from "../utils/toast";
|
import { toast } from "../utils/toast";
|
||||||
@@ -15,6 +16,62 @@ function keyBytes(base64url: string): Uint8Array<ArrayBuffer> {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const KINDS = Object.keys(CATEGORY_LABELS) as AlertCategory[];
|
||||||
|
|
||||||
|
/** What one browser wants to be told: how bad, and which kinds. Saved when asked. */
|
||||||
|
function Wishes({ device, onSaved, onError }: { device: PushDevice; onSaved: () => void; onError: (message: string) => void }) {
|
||||||
|
const [severity, setSeverity] = useState<"" | "warning" | "critical">(device.minSeverity ?? "");
|
||||||
|
const [kinds, setKinds] = useState<AlertCategory[]>(device.categories ?? []);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const dirty = severity !== (device.minSeverity ?? "") || JSON.stringify([...kinds].sort()) !== JSON.stringify([...(device.categories ?? [])].sort());
|
||||||
|
|
||||||
|
async function save() {
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
await api.push.setPrefs(device.id, { minSeverity: severity === "" ? null : severity, categories: kinds.length > 0 && kinds.length < KINDS.length ? kinds : null });
|
||||||
|
toast.success("Saved.");
|
||||||
|
onSaved();
|
||||||
|
} catch (err) {
|
||||||
|
onError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="ms-3 mb-2 small">
|
||||||
|
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
||||||
|
<label htmlFor={`push-sev-${device.id}`} className="text-secondary">
|
||||||
|
Only alerts that are
|
||||||
|
</label>
|
||||||
|
<select id={`push-sev-${device.id}`} className="form-select form-select-sm w-auto" value={severity} onChange={(e) => setSeverity(e.target.value as "" | "warning" | "critical")}>
|
||||||
|
<option value="">of any severity</option>
|
||||||
|
<option value="warning">warnings or worse</option>
|
||||||
|
<option value="critical">critical</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="d-flex flex-wrap gap-3 mb-1">
|
||||||
|
{KINDS.map((k) => (
|
||||||
|
<label key={k} className="form-check mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={kinds.length === 0 || kinds.includes(k)} onChange={(e) => setKinds((prev) => {
|
||||||
|
const base = prev.length === 0 ? KINDS : prev;
|
||||||
|
const next = e.target.checked ? [...base.filter((x) => x !== k), k] : base.filter((x) => x !== k);
|
||||||
|
// At least one kind stays ticked: a browser that wants nothing is better removed.
|
||||||
|
return next.length === 0 ? prev : next;
|
||||||
|
})} />
|
||||||
|
<span className="form-check-label">{CATEGORY_LABELS[k]}</span>
|
||||||
|
</label>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
{dirty && (
|
||||||
|
<button type="button" className="btn btn-sm btn-primary" onClick={() => void save()} disabled={busy}>
|
||||||
|
Save what this browser gets
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/** A short name for this browser, to tell the devices apart in the list. */
|
/** A short name for this browser, to tell the devices apart in the list. */
|
||||||
function browserLabel(): string {
|
function browserLabel(): string {
|
||||||
const ua = navigator.userAgent;
|
const ua = navigator.userAgent;
|
||||||
@@ -142,13 +199,17 @@ export default function PushDevices({ serverEnabled }: { serverEnabled: boolean
|
|||||||
{devices && devices.length > 0 && (
|
{devices && devices.length > 0 && (
|
||||||
<ul className="list-unstyled mb-0">
|
<ul className="list-unstyled mb-0">
|
||||||
{devices.map((d) => (
|
{devices.map((d) => (
|
||||||
<li key={d.id} className="d-flex align-items-center gap-2 py-1">
|
<li key={d.id} className="d-flex flex-wrap align-items-center gap-2 py-2 border-bottom">
|
||||||
<span>{d.label || d.host}</span>
|
<span>{d.label || d.host}</span>
|
||||||
{d.endpoint === thisEndpoint && <span className="badge bg-blue-lt text-blue">this browser</span>}
|
{d.endpoint === thisEndpoint && <span className="badge bg-blue-lt text-blue">this browser</span>}
|
||||||
<span className="text-secondary small">via {d.host}</span>
|
<span className="text-secondary small">via {d.host}</span>
|
||||||
<button type="button" className="btn btn-sm btn-outline-danger ms-auto" onClick={() => void remove(d)}>
|
<button type="button" className="btn btn-sm btn-outline-danger ms-auto" onClick={() => void remove(d)}>
|
||||||
Remove
|
Remove
|
||||||
</button>
|
</button>
|
||||||
|
<div className="w-100 text-secondary small">
|
||||||
|
{d.minSeverity || d.categories ? `Gets ${d.minSeverity === "critical" ? "critical alerts" : d.minSeverity === "warning" ? "warnings and worse" : "alerts of any severity"}${d.categories ? (d.categories.length > KINDS.length / 2 ? ` about everything except ${KINDS.filter((k) => !d.categories!.includes(k)).map((c) => CATEGORY_LABELS[c].toLowerCase()).join(", ")}` : ` about ${d.categories.map((c) => CATEGORY_LABELS[c].toLowerCase()).join(", ")}`) : ""}.` : "Gets every alert."}
|
||||||
|
</div>
|
||||||
|
<Wishes device={d} onSaved={() => void load()} onError={setError} />
|
||||||
</li>
|
</li>
|
||||||
))}
|
))}
|
||||||
</ul>
|
</ul>
|
||||||
|
|||||||
@@ -0,0 +1,260 @@
|
|||||||
|
import { useState } from "react";
|
||||||
|
import type { AlertCategory, EscalationSettings, EscalationStep, NotifyChannelId, RoutingRule, RoutingSettings } from "../api/client";
|
||||||
|
|
||||||
|
export const CHANNEL_LABELS: Record<NotifyChannelId, string> = { gotify: "Gotify", ntfy: "ntfy", smtp: "Email", webhook: "Webhook", webPush: "Phone and browser" };
|
||||||
|
const CHANNELS = Object.keys(CHANNEL_LABELS) as NotifyChannelId[];
|
||||||
|
|
||||||
|
export const CATEGORY_LABELS: Record<AlertCategory, string> = {
|
||||||
|
offline: "Server down",
|
||||||
|
disk: "Disk & storage",
|
||||||
|
services: "Failing services",
|
||||||
|
backup: "Backups",
|
||||||
|
updates: "Updates",
|
||||||
|
expiry: "Expiry",
|
||||||
|
automation: "Automation",
|
||||||
|
integration: "Integration failing",
|
||||||
|
monitoring: "Monitoring",
|
||||||
|
tickets: "Tickets",
|
||||||
|
};
|
||||||
|
const CATEGORIES = Object.keys(CATEGORY_LABELS) as AlertCategory[];
|
||||||
|
|
||||||
|
const newId = () => Math.random().toString(36).slice(2, 10);
|
||||||
|
|
||||||
|
/** Tags typed as "prod, db servers": kept as typed while editing, split when it leaves the box. */
|
||||||
|
function TagsInput({ value, onChange, id }: { value: string[]; onChange: (tags: string[]) => void; id: string }) {
|
||||||
|
const [text, setText] = useState(value.join(", "));
|
||||||
|
return (
|
||||||
|
<input
|
||||||
|
id={id}
|
||||||
|
className="form-control"
|
||||||
|
placeholder="any server"
|
||||||
|
value={text}
|
||||||
|
onChange={(e) => {
|
||||||
|
setText(e.target.value);
|
||||||
|
onChange(e.target.value.split(",").map((t) => t.trim()).filter(Boolean));
|
||||||
|
}}
|
||||||
|
onBlur={() => setText(value.join(", "))}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ChannelChecks({ value, onChange, enabled, idPrefix }: { value: NotifyChannelId[]; onChange: (v: NotifyChannelId[]) => void; enabled: Record<NotifyChannelId, boolean>; idPrefix: string }) {
|
||||||
|
return (
|
||||||
|
<div className="d-flex flex-wrap gap-3">
|
||||||
|
{CHANNELS.map((c) => (
|
||||||
|
<label key={c} className="form-check mb-0">
|
||||||
|
<input
|
||||||
|
id={`${idPrefix}-${c}`}
|
||||||
|
type="checkbox"
|
||||||
|
className="form-check-input"
|
||||||
|
checked={value.includes(c)}
|
||||||
|
onChange={(e) => onChange(e.target.checked ? [...value, c] : value.filter((x) => x !== c))}
|
||||||
|
/>
|
||||||
|
<span className="form-check-label">
|
||||||
|
{CHANNEL_LABELS[c]}
|
||||||
|
{!enabled[c] && <span className="text-secondary small"> (not switched on)</span>}
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Who hears what. Routing rules decide which channels get a notification (the first rule that matches wins; no rule matching means every channel
|
||||||
|
* that is on), and escalation tells people again about alerts nobody has acknowledged.
|
||||||
|
*/
|
||||||
|
export default function RoutingEditor({
|
||||||
|
routing,
|
||||||
|
onRouting,
|
||||||
|
escalation,
|
||||||
|
onEscalation,
|
||||||
|
channelsOn,
|
||||||
|
}: {
|
||||||
|
routing: RoutingSettings;
|
||||||
|
onRouting: (r: RoutingSettings) => void;
|
||||||
|
escalation: EscalationSettings;
|
||||||
|
onEscalation: (e: EscalationSettings) => void;
|
||||||
|
channelsOn: Record<NotifyChannelId, boolean>;
|
||||||
|
}) {
|
||||||
|
const setRule = (i: number, patch: Partial<RoutingRule>) => onRouting({ rules: routing.rules.map((r, j) => (j === i ? { ...r, ...patch } : r)) });
|
||||||
|
const move = (i: number, by: -1 | 1) => {
|
||||||
|
const to = i + by;
|
||||||
|
if (to < 0 || to >= routing.rules.length) return;
|
||||||
|
const rules = [...routing.rules];
|
||||||
|
[rules[i], rules[to]] = [rules[to], rules[i]];
|
||||||
|
onRouting({ rules });
|
||||||
|
};
|
||||||
|
const setStep = (i: number, patch: Partial<EscalationStep>) => onEscalation({ ...escalation, steps: escalation.steps.map((s, j) => (j === i ? { ...s, ...patch } : s)) });
|
||||||
|
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Routing rules</h3>
|
||||||
|
<div className="card-actions">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="btn btn-sm btn-primary"
|
||||||
|
disabled={routing.rules.length >= 30}
|
||||||
|
onClick={() => onRouting({ rules: [...routing.rules, { id: newId(), name: `Rule ${routing.rules.length + 1}`, enabled: true, categories: [], minSeverity: null, tags: [], channels: [] }] })}
|
||||||
|
>
|
||||||
|
Add a rule
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
<p className="text-secondary">
|
||||||
|
By default every notification goes to every channel that is switched on. A rule narrows that for what it matches: <strong>the first rule that matches decides</strong> which channels get it, so put the specific rules first. A
|
||||||
|
rule with no channels tells nobody — the Alerts page still shows the problem. Only problems and recoveries about servers, services and the like carry a kind and a severity; a rule that asks about
|
||||||
|
something a notification doesn’t say doesn’t match it.
|
||||||
|
</p>
|
||||||
|
{routing.rules.length === 0 && <div className="text-secondary">No rules: everything goes everywhere.</div>}
|
||||||
|
{routing.rules.map((r, i) => (
|
||||||
|
<div key={r.id} className="border rounded p-3 mb-3">
|
||||||
|
<div className="d-flex align-items-center gap-2 mb-3">
|
||||||
|
<label className="form-check form-switch mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={r.enabled} onChange={(e) => setRule(i, { enabled: e.target.checked })} />
|
||||||
|
<span className="visually-hidden">Rule {r.name} is on</span>
|
||||||
|
</label>
|
||||||
|
<input className="form-control" style={{ maxWidth: "20rem" }} aria-label="Name of the rule" maxLength={60} value={r.name} onChange={(e) => setRule(i, { name: e.target.value })} />
|
||||||
|
<span className="text-secondary small">#{i + 1}</span>
|
||||||
|
<div className="btn-list ms-auto flex-nowrap">
|
||||||
|
<button type="button" className="btn btn-sm" aria-label={`Move ${r.name} up`} disabled={i === 0} onClick={() => move(i, -1)}>
|
||||||
|
↑
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn btn-sm" aria-label={`Move ${r.name} down`} disabled={i === routing.rules.length - 1} onClick={() => move(i, 1)}>
|
||||||
|
↓
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn btn-sm btn-outline-danger" onClick={() => onRouting({ rules: routing.rules.filter((_, j) => j !== i) })}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="row g-3">
|
||||||
|
<div className="col-12">
|
||||||
|
<div className="form-label">When it is about (none ticked: anything)</div>
|
||||||
|
<div className="d-flex flex-wrap gap-3">
|
||||||
|
{CATEGORIES.map((c) => (
|
||||||
|
<label key={c} className="form-check mb-0">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
className="form-check-input"
|
||||||
|
checked={r.categories.includes(c)}
|
||||||
|
onChange={(e) => setRule(i, { categories: e.target.checked ? [...r.categories, c] : r.categories.filter((x) => x !== c) })}
|
||||||
|
/>
|
||||||
|
<span className="form-check-label">{CATEGORY_LABELS[c]}</span>
|
||||||
|
</label>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor={`rule-sev-${r.id}`}>
|
||||||
|
At least this bad
|
||||||
|
</label>
|
||||||
|
<select id={`rule-sev-${r.id}`} className="form-select" value={r.minSeverity ?? ""} onChange={(e) => setRule(i, { minSeverity: e.target.value === "" ? null : (e.target.value as "warning" | "critical") })}>
|
||||||
|
<option value="">Any severity</option>
|
||||||
|
<option value="warning">Warning or worse</option>
|
||||||
|
<option value="critical">Critical only</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-8">
|
||||||
|
<label className="form-label" htmlFor={`rule-tags-${r.id}`}>
|
||||||
|
On a server with one of these tags
|
||||||
|
</label>
|
||||||
|
<TagsInput id={`rule-tags-${r.id}`} value={r.tags} onChange={(tags) => setRule(i, { tags })} />
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<div className="form-label">Tell</div>
|
||||||
|
<ChannelChecks value={r.channels} onChange={(channels) => setRule(i, { channels })} enabled={channelsOn} idPrefix={`rule-${r.id}`} />
|
||||||
|
{r.channels.length === 0 && <div className="text-warning small mt-1">No channel ticked: nobody is told. The Alerts page still shows it.</div>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
{routing.rules.length > 0 && <div className="text-secondary small">Anything no rule matches goes to every channel that is on. A phone or browser can also turn down what it doesn’t want, under Phone and browser.</div>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="card">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Escalation</h3>
|
||||||
|
<div className="card-actions">
|
||||||
|
<label className="form-check form-switch mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={escalation.enabled} onChange={(e) => onEscalation({ ...escalation, enabled: e.target.checked })} />
|
||||||
|
<span className="form-check-label">On</span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
<p className="text-secondary">
|
||||||
|
When an alert has been raised for a while and nobody has acknowledged it, tell again — through a louder channel, and perhaps someone else. Each step is sent once per alert, in order, and an acknowledgement stops the
|
||||||
|
rest. Problems under maintenance, or held back because something they depend on is down, never escalate.
|
||||||
|
</p>
|
||||||
|
{escalation.steps.length === 0 && <div className="text-secondary mb-2">No steps yet.</div>}
|
||||||
|
{escalation.steps.map((s, i) => (
|
||||||
|
<div key={i} className="border rounded p-3 mb-3">
|
||||||
|
<div className="d-flex align-items-center mb-3">
|
||||||
|
<strong>Step {i + 1}</strong>
|
||||||
|
<button type="button" className="btn btn-sm btn-outline-danger ms-auto" onClick={() => onEscalation({ ...escalation, steps: escalation.steps.filter((_, j) => j !== i) })}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div className="row g-3">
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor={`esc-after-${i}`}>
|
||||||
|
After it has been up for
|
||||||
|
</label>
|
||||||
|
<div className="input-group">
|
||||||
|
<input id={`esc-after-${i}`} type="number" min={5} max={10080} className="form-control" value={s.afterMinutes} onChange={(e) => setStep(i, { afterMinutes: Number(e.target.value) })} />
|
||||||
|
<span className="input-group-text">min</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor={`esc-sev-${i}`}>
|
||||||
|
For alerts that are
|
||||||
|
</label>
|
||||||
|
<select id={`esc-sev-${i}`} className="form-select" value={s.minSeverity} onChange={(e) => setStep(i, { minSeverity: e.target.value as "warning" | "critical" })}>
|
||||||
|
<option value="warning">Warning or worse</option>
|
||||||
|
<option value="critical">Critical only</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor={`esc-mail-${i}`}>
|
||||||
|
Also email (someone else)
|
||||||
|
</label>
|
||||||
|
<input id={`esc-mail-${i}`} type="email" className="form-control" placeholder="neighbour@example.com" value={s.extraEmail} onChange={(e) => setStep(i, { extraEmail: e.target.value })} />
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<div className="form-label">Tell</div>
|
||||||
|
<ChannelChecks value={s.channels} onChange={(channels) => setStep(i, { channels })} enabled={channelsOn} idPrefix={`esc-${i}`} />
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<label className="form-check mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={s.evenInQuietHours} onChange={(e) => setStep(i, { evenInQuietHours: e.target.checked })} />
|
||||||
|
<span className="form-check-label">Send even during quiet hours</span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="btn btn-outline-primary"
|
||||||
|
disabled={escalation.steps.length >= 3}
|
||||||
|
onClick={() => {
|
||||||
|
const last = escalation.steps[escalation.steps.length - 1];
|
||||||
|
onEscalation({
|
||||||
|
...escalation,
|
||||||
|
steps: [...escalation.steps, { afterMinutes: last ? last.afterMinutes + 60 : 30, minSeverity: "critical", channels: [], extraEmail: "", evenInQuietHours: false }],
|
||||||
|
});
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Add a step
|
||||||
|
</button>
|
||||||
|
<div className="text-secondary small mt-2">Looked at every five minutes, so a step can come up to five minutes after its time.</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -8,6 +8,8 @@ import {
|
|||||||
type SmtpSettings,
|
type SmtpSettings,
|
||||||
type WebhookSettings,
|
type WebhookSettings,
|
||||||
type WebPushSettings,
|
type WebPushSettings,
|
||||||
|
type RoutingSettings,
|
||||||
|
type EscalationSettings,
|
||||||
type NotificationEvents,
|
type NotificationEvents,
|
||||||
type DigestSettings,
|
type DigestSettings,
|
||||||
type QuietHoursSettings,
|
type QuietHoursSettings,
|
||||||
@@ -17,6 +19,7 @@ import { readableError } from "../../utils/errors";
|
|||||||
import { ErrorNotice } from "../../components/Notice";
|
import { ErrorNotice } from "../../components/Notice";
|
||||||
import { PageSkeleton } from "../../components/Skeleton";
|
import { PageSkeleton } from "../../components/Skeleton";
|
||||||
import PushDevices from "../../components/PushDevices";
|
import PushDevices from "../../components/PushDevices";
|
||||||
|
import RoutingEditor from "../../components/RoutingEditor";
|
||||||
|
|
||||||
const TIMEZONES = [
|
const TIMEZONES = [
|
||||||
"UTC",
|
"UTC",
|
||||||
@@ -46,6 +49,8 @@ const DEFAULT_NTFY: NtfySettings = { enabled: false, url: "https://ntfy.sh", top
|
|||||||
const DEFAULT_SMTP: SmtpSettings = { enabled: false, host: "", port: 587, secure: false, username: "", password: "", from: "", to: "" };
|
const DEFAULT_SMTP: SmtpSettings = { enabled: false, host: "", port: 587, secure: false, username: "", password: "", from: "", to: "" };
|
||||||
const DEFAULT_WEBHOOK: WebhookSettings = { enabled: false, url: "", secret: "" };
|
const DEFAULT_WEBHOOK: WebhookSettings = { enabled: false, url: "", secret: "" };
|
||||||
const DEFAULT_WEBPUSH: WebPushSettings = { enabled: false, contact: "" };
|
const DEFAULT_WEBPUSH: WebPushSettings = { enabled: false, contact: "" };
|
||||||
|
const DEFAULT_ROUTING: RoutingSettings = { rules: [] };
|
||||||
|
const DEFAULT_ESCALATION: EscalationSettings = { enabled: false, steps: [] };
|
||||||
const DEFAULT_NOTIFICATIONS: NotificationEvents = {
|
const DEFAULT_NOTIFICATIONS: NotificationEvents = {
|
||||||
dnsAdd: true,
|
dnsAdd: true,
|
||||||
dnsUpdate: true,
|
dnsUpdate: true,
|
||||||
@@ -84,6 +89,7 @@ type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends
|
|||||||
const TABS = [
|
const TABS = [
|
||||||
{ id: "channels", label: "Channels" },
|
{ id: "channels", label: "Channels" },
|
||||||
{ id: "events", label: "What to send" },
|
{ id: "events", label: "What to send" },
|
||||||
|
{ id: "routing", label: "Who hears what" },
|
||||||
{ id: "when", label: "When" },
|
{ id: "when", label: "When" },
|
||||||
] as const;
|
] as const;
|
||||||
type TabId = (typeof TABS)[number]["id"];
|
type TabId = (typeof TABS)[number]["id"];
|
||||||
@@ -225,6 +231,8 @@ export default function NotificationSettings() {
|
|||||||
const [webPush, setWebPush] = useState<WebPushSettings>(DEFAULT_WEBPUSH);
|
const [webPush, setWebPush] = useState<WebPushSettings>(DEFAULT_WEBPUSH);
|
||||||
// Whether push is on as saved: people can only add their browsers once the server knows it is on.
|
// Whether push is on as saved: people can only add their browsers once the server knows it is on.
|
||||||
const [webPushSaved, setWebPushSaved] = useState(false);
|
const [webPushSaved, setWebPushSaved] = useState(false);
|
||||||
|
const [routing, setRouting] = useState<RoutingSettings>(DEFAULT_ROUTING);
|
||||||
|
const [escalation, setEscalation] = useState<EscalationSettings>(DEFAULT_ESCALATION);
|
||||||
const [notifications, setNotifications] = useState<NotificationEvents>(DEFAULT_NOTIFICATIONS);
|
const [notifications, setNotifications] = useState<NotificationEvents>(DEFAULT_NOTIFICATIONS);
|
||||||
const [quietHours, setQuietHours] = useState<QuietHoursSettings>(DEFAULT_QUIET_HOURS);
|
const [quietHours, setQuietHours] = useState<QuietHoursSettings>(DEFAULT_QUIET_HOURS);
|
||||||
const [healthChecks, setHealthChecks] = useState<HealthCheckSettings>(DEFAULT_HEALTH_CHECKS);
|
const [healthChecks, setHealthChecks] = useState<HealthCheckSettings>(DEFAULT_HEALTH_CHECKS);
|
||||||
@@ -245,7 +253,7 @@ export default function NotificationSettings() {
|
|||||||
const [testingWebhook, setTestingWebhook] = useState(false);
|
const [testingWebhook, setTestingWebhook] = useState(false);
|
||||||
const [testWebhookResult, setTestWebhookResult] = useState<TestResult>(null);
|
const [testWebhookResult, setTestWebhookResult] = useState<TestResult>(null);
|
||||||
|
|
||||||
const snapshot = JSON.stringify({ gotify, ntfy, smtp, webhook, webPush, notifications, quietHours, healthChecks, digest });
|
const snapshot = JSON.stringify({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest });
|
||||||
const dirty = baseline !== null && snapshot !== baseline;
|
const dirty = baseline !== null && snapshot !== baseline;
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -260,12 +268,16 @@ export default function NotificationSettings() {
|
|||||||
const wp = s.webPush ?? DEFAULT_WEBPUSH;
|
const wp = s.webPush ?? DEFAULT_WEBPUSH;
|
||||||
setWebPush(wp);
|
setWebPush(wp);
|
||||||
setWebPushSaved(wp.enabled);
|
setWebPushSaved(wp.enabled);
|
||||||
|
const rt = s.routing ?? DEFAULT_ROUTING;
|
||||||
|
const es = s.escalation ?? DEFAULT_ESCALATION;
|
||||||
|
setRouting(rt);
|
||||||
|
setEscalation(es);
|
||||||
setNotifications(s.notifications);
|
setNotifications(s.notifications);
|
||||||
setQuietHours(s.quietHours);
|
setQuietHours(s.quietHours);
|
||||||
setHealthChecks(s.healthChecks);
|
setHealthChecks(s.healthChecks);
|
||||||
const d = s.digest ?? DEFAULT_DIGEST;
|
const d = s.digest ?? DEFAULT_DIGEST;
|
||||||
setDigest(d);
|
setDigest(d);
|
||||||
setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d }));
|
setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, routing: rt, escalation: es, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d }));
|
||||||
})
|
})
|
||||||
.catch((err) => setLoadError(err instanceof Error ? err.message : String(err)))
|
.catch((err) => setLoadError(err instanceof Error ? err.message : String(err)))
|
||||||
.finally(() => setLoading(false));
|
.finally(() => setLoading(false));
|
||||||
@@ -312,7 +324,7 @@ export default function NotificationSettings() {
|
|||||||
setSaveError(null);
|
setSaveError(null);
|
||||||
setSaved(false);
|
setSaved(false);
|
||||||
try {
|
try {
|
||||||
await api.settings.update({ gotify, ntfy, smtp, webhook, webPush, notifications, quietHours, healthChecks, digest });
|
await api.settings.update({ gotify, ntfy, smtp, webhook, webPush, routing, escalation, notifications, quietHours, healthChecks, digest });
|
||||||
setBaseline(snapshot);
|
setBaseline(snapshot);
|
||||||
setWebPushSaved(webPush.enabled);
|
setWebPushSaved(webPush.enabled);
|
||||||
setSaved(true);
|
setSaved(true);
|
||||||
@@ -777,6 +789,17 @@ export default function NotificationSettings() {
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* ── Who hears what ── */}
|
||||||
|
<div role="tabpanel" id="panel-routing" aria-labelledby="tab-routing" hidden={tab !== "routing"} className="pt-3">
|
||||||
|
<RoutingEditor
|
||||||
|
routing={routing}
|
||||||
|
onRouting={setRouting}
|
||||||
|
escalation={escalation}
|
||||||
|
onEscalation={setEscalation}
|
||||||
|
channelsOn={{ gotify: gotify.enabled && gotifyComplete, ntfy: ntfy.enabled && ntfyComplete, smtp: smtp.enabled && smtpComplete, webhook: webhook.enabled && webhookComplete, webPush: webPush.enabled }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
{/* ── When ── */}
|
{/* ── When ── */}
|
||||||
<div role="tabpanel" id="panel-when" aria-labelledby="tab-when" hidden={tab !== "when"} className="pt-3">
|
<div role="tabpanel" id="panel-when" aria-labelledby="tab-when" hidden={tab !== "when"} className="pt-3">
|
||||||
<p className="text-secondary">When notifications are sent: the time zone everything here runs in, the daily reminders, hours to keep quiet, and the weekly digest.</p>
|
<p className="text-secondary">When notifications are sent: the time zone everything here runs in, the daily reminders, hours to keep quiet, and the weekly digest.</p>
|
||||||
|
|||||||
Reference in new issue
Block a user