diff --git a/NOTIFICATIONS.md b/NOTIFICATIONS.md index fc1fefb..9b95336 100644 --- a/NOTIFICATIONS.md +++ b/NOTIFICATIONS.md @@ -89,6 +89,20 @@ schedule. |---|---| | "Notifications from quiet hours" | Once, at quiet hours' configured end time, **only if enabled and only if at least one notification was held** during the window. Bundles every held notification's title and message into one message, then clears the queue. | +## The Alerts page + +**Operations → Alerts** shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and +export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs +in three ways: + +- It ignores the "Notify on" toggles. Turning a notification off doesn't hide the problem from the page. +- Problems under a **maintenance window** are kept on the list, marked *silenced* and counted separately, instead of being dropped. +- It also lists things nothing notifies about: Uptime Kuma monitors that are down, osTicket tickets that are overdue, and any integration that's + failing its last few calls (before the threshold that triggers an "integration down" notification). + +It runs the checks live when opened (a recent result is reused for a minute), and shows what it couldn't read at the top, so a missing section means +"couldn't check" and not "all clear". + ## What does *not* send a notification Worth calling out explicitly, since it's easy to assume everything in diff --git a/README.md b/README.md index f6ddca4..73aa214 100644 --- a/README.md +++ b/README.md @@ -255,6 +255,21 @@ already failing, so old failures aren't announced. Toggle it under Settings → Notifications. For Gitea this follows the repo's most recent run on any workflow or branch, the same as the Gitea page shows. +**Alerts** (Operations → Alerts, visible to every role) lists everything that's +wrong right now in one place, instead of waiting for a notification or visiting +each page: servers that stopped reporting, full or nearly full disks and volumes +(critical from 95%), Synology volume/disk problems, failed or uncovered Proxmox +backups, failed Proxmox Backup Server verifications, container image updates, +secrets, domains and Tailscale keys that are expired or about to be, failed +Semaphore/Gitea runs, Uptime Kuma monitors that are down, overdue osTicket +tickets, and integrations whose calls keep failing. It runs the same checks that +send the notifications — so the two can't disagree — but ignores the on/off +toggles, since it's for looking at rather than being interrupted by. Problems +under a maintenance window stay listed, marked silenced and counted separately. +It checks live (a recent result is reused for a minute; "Check now" forces a +fresh one), and anything it couldn't read is called out at the top rather than +quietly treated as fine. + **Maintenance mode** silences alerts about one server, integration, or DNS provider while you work on it (server offline / disk, storage and Synology health, Proxmox backup alerts, Proxmox Backup Server verification alerts, and diff --git a/ROLES.md b/ROLES.md index 5cce5c5..1bd2051 100644 --- a/ROLES.md +++ b/ROLES.md @@ -37,6 +37,7 @@ page's own top-level link. | Gitea | `/gitea` | ✅ | ✅ | ✅ | | Secrets | `/secrets` | ✅ | ✅ | ✅ | | **Operations** | | | | | +| Alerts | `/alerts` | ✅ | ✅ | ✅ | | Maintenance | `/maintenance` | ✅ | ✅ | ✅ | | Uptime Kuma | `/uptime-kuma` | ✅ | ✅ | ✅ | | osTicket | `/osticket` | ✅ | ✅ | ✅ | diff --git a/server/src/index.ts b/server/src/index.ts index 83e3ded..5c6991f 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -29,6 +29,7 @@ import { consistencyRouter } from "./routes/consistency.js"; import { privacyRouter } from "./routes/privacy.js"; import { tagsRouter } from "./routes/tags.js"; import { portsRouter } from "./routes/ports.js"; +import { alertsRouter } from "./routes/alerts.js"; import { initSecretExpiryScheduler } from "./services/secretExpiryScheduler.js"; import { initTailscaleKeyExpiryScheduler } from "./services/tailscaleKeyExpiryScheduler.js"; import { initLogRetentionScheduler } from "./services/logRetentionScheduler.js"; @@ -104,6 +105,7 @@ app.use("/api/consistency", consistencyRouter); app.use("/api/privacy", privacyRouter); app.use("/api/tags", tagsRouter); app.use("/api/ports", portsRouter); +app.use("/api/alerts", alertsRouter); if (existsSync(webDist)) { app.use(express.static(webDist)); diff --git a/server/src/routes/alerts.ts b/server/src/routes/alerts.ts new file mode 100644 index 0000000..22c9103 --- /dev/null +++ b/server/src/routes/alerts.ts @@ -0,0 +1,14 @@ +import { Router } from "express"; +import { requireAuth } from "../auth/middleware.js"; +import { getAlerts } from "../services/alerts.js"; +import { asyncHandler } from "../utils/asyncHandler.js"; + +export const alertsRouter = Router(); + +alertsRouter.use(requireAuth); + +// Everyone signed in can see it — it's the same server, disk and backup status the other pages already show. +// `?refresh=1` asks for a fresh check instead of a recent one. +alertsRouter.get("/", asyncHandler(async (req, res) => { + res.json(await getAlerts(req.query.refresh === "1")); +})); diff --git a/server/src/services/alertTypes.ts b/server/src/services/alertTypes.ts new file mode 100644 index 0000000..204993b --- /dev/null +++ b/server/src/services/alertTypes.ts @@ -0,0 +1,28 @@ +// Shared by the alerts page's collector (services/alerts.ts) and the scheduled checks it reuses, which can't import +// that file themselves without going round in a circle. + +export type AlertSeverity = "critical" | "warning" | "info"; + +/** What kind of problem — drives the filter on the Alerts page. */ +export type AlertCategory = "offline" | "disk" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets"; + +export interface Alert { + /** Stable, so the page can key rows on it. */ + id: string; + severity: AlertSeverity; + category: AlertCategory; + /** Where it comes from, as a short name: "Server", "Proxmox", "Secrets", ... */ + source: string; + message: string; + /** In-app page that shows more, if there is one. */ + link: string | null; + /** Under an active maintenance window: still a real problem, but notifications for it are held back. */ + silenced: boolean; +} + +/** One integration that couldn't be read while collecting — so the page never mistakes "couldn't check" for "all clear". */ +export interface SourceFailure { + integrationId: number; + integrationName: string; + message: string; +} diff --git a/server/src/services/alerts.ts b/server/src/services/alerts.ts new file mode 100644 index 0000000..2c9b394 --- /dev/null +++ b/server/src/services/alerts.ts @@ -0,0 +1,387 @@ +/** + * Everything that's wrong right now, in one list — the Alerts page. Nothing here decides what counts as a problem: it asks + * the same checks that send the notifications (health, backups, updates, expiry, automation, ...) and turns what they find + * into a flat list, so the page and the notifications can't disagree. Unlike the notifications it ignores the per-event + * on/off toggles (the page is for looking at, not for being interrupted by) and keeps problems that are under a + * maintenance window, flagged as silenced rather than dropped. + * + * It reads live (a handful of API calls per integration), so a result is kept for a short while rather than re-run for + * every viewer, and every source has a time limit so one hung integration can't hang the page. A source that can't be + * read is reported as such — silence from it must never look like "all clear". + */ +import { eq } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { integrations, secrets } from "../db/schema.js"; +import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; +import { createUptimeKumaAdapter } from "../integrations/uptimekuma/adapter.js"; +import { createOsTicketAdapter } from "../integrations/osticket/adapter.js"; +import { activeSubjects, isSourceInMaintenance, subjectOfConditionKey } from "./maintenance.js"; +import { collectSnapshot, evaluateHealth, startupGraceRemainingMs } from "./healthMonitor.js"; +import { collectAutomation, evaluateAutomation } from "./automationMonitor.js"; +import { collectDockerUpdates } from "./dockerUpdateScheduler.js"; +import { collectProxmoxBackupProblems } from "./proxmoxBackupScheduler.js"; +import { collectPbsProblems } from "./pbsVerificationScheduler.js"; +import { collectTailscaleKeyExpiries } from "./tailscaleKeyExpiryScheduler.js"; +import { collectDomainAlerts } from "./domainMonitor.js"; +import { computeSecretStatus } from "./secretStatus.js"; +import { getFailingSources } from "./integrationHealthMonitor.js"; +import { getSettings } from "./settingsStore.js"; +import { sourceLabel } from "./notify.js"; +import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js"; + +export interface AlertsReport { + alerts: Alert[]; + /** Active problems by severity — those under a maintenance window are counted apart, not in these. */ + counts: { critical: number; warning: number; info: number; silenced: number }; + /** Things that couldn't be checked, and which checks that leaves blind. */ + couldntCheck: { name: string; error: string; affects: string[] }[]; + /** Context worth knowing about how complete the picture is. */ + notes: string[]; + generatedAt: string; +} + +/** Where each kind of source lives in the app, and what to call it. */ +const SOURCES: Record = { + server: { label: "Server", link: "/servers" }, + proxmox: { label: "Proxmox", link: "/proxmox" }, + synology: { label: "Synology", link: "/synology" }, + semaphore: { label: "Semaphore", link: "/semaphore" }, + gitea: { label: "Gitea", link: "/gitea" }, + dockhand: { label: "Docker", link: "/docker" }, + tailscale: { label: "Tailscale", link: "/tailscale" }, + pbs: { label: "Proxmox Backup", link: "/pbs" }, + uptimekuma: { label: "Uptime Kuma", link: "/uptime-kuma" }, + osticket: { label: "osTicket", link: "/osticket" }, + secrets: { label: "Secrets", link: "/secrets" }, + domains: { label: "Domains", link: "/domains" }, +}; + +const SOURCE_TIMEOUT_MS = 20_000; +/** How long a result is reused. */ +const CACHE_MS = 60_000; +/** Pressing Refresh over and over shouldn't hammer every integration — a result younger than this is reused even then. */ +const MIN_REFRESH_MS = 10_000; + +const SEVERITY_ORDER: Record = { critical: 0, warning: 1, info: 2 }; + +/** Which kind of source, and which one, a health/automation condition key belongs to. */ +function originOfConditionKey(key: string): { type: string; id: number | null; category: AlertCategory } { + let m = /^(?:offline|disk):server:(\d+)/.exec(key); + if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : "disk" }; + m = /^disk:proxmox:(\d+):/.exec(key); + if (m) return { type: "proxmox", id: Number(m[1]), category: "disk" }; + m = /^(?:synology-volume|synology-disk|disk:synology):(\d+):/.exec(key); + if (m) return { type: "synology", id: Number(m[1]), category: "disk" }; + m = /^automation:(semaphore|gitea):(\d+):/.exec(key); + if (m) return { type: m[1], id: Number(m[2]), category: "automation" }; + return { type: "server", id: null, category: "disk" }; +} + +function withTimeout(work: Promise): Promise { + let timer: ReturnType; + const limit = new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`timed out after ${SOURCE_TIMEOUT_MS / 1000}s`)), SOURCE_TIMEOUT_MS); + }); + return Promise.race([work, limit]).finally(() => clearTimeout(timer)); +} + +const errorText = (err: unknown) => (err instanceof Error ? err.message : String(err)); +const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; + +export async function collectAlerts(): Promise { + const alerts: Alert[] = []; + const notes: string[] = []; + const blind = new Map }>(); + const subjects = await activeSubjects(); + const intRows = await db.select({ id: integrations.id, name: integrations.name, type: integrations.type, enabled: integrations.enabled }).from(integrations); + const intById = new Map(intRows.map((r) => [r.id, r])); + + function push(a: { category: AlertCategory; severity: AlertSeverity; type: string; message: string; key: string; link?: string | null; silenced?: boolean }) { + const meta = SOURCES[a.type]; + alerts.push({ + id: `${a.category}:${a.key}`, + severity: a.severity, + category: a.category, + source: meta?.label ?? sourceLabel(a.type), + message: a.message, + link: a.link === undefined ? (meta?.link ?? null) : a.link, + silenced: !!a.silenced, + }); + } + + function cannotCheck(name: string, error: string, check: string) { + const entry = blind.get(name) ?? { error, affects: new Set() }; + entry.affects.add(check); + blind.set(name, entry); + } + const cannotCheckIntegration = (f: SourceFailure, check: string) => { + const row = intById.get(f.integrationId); + cannotCheck(`${row ? (SOURCES[row.type]?.label ?? row.type) : "Integration"} “${f.integrationName}”`, f.message, check); + }; + const silencedIntegration = (id: number) => subjects.has(`integration:${id}`); + + // One entry per check. A check that blows up, or hangs, only costs its own section of the page. + async function check(name: string, work: () => Promise) { + try { + await withTimeout(work()); + } catch (err) { + cannotCheck(name, errorText(err), name); + } + } + + await Promise.all([ + check("Server and storage health", async () => { + const { healthChecks } = await getSettings(); + const { snapshot, held } = await collectSnapshot(); + const now = Date.now(); + const grace = startupGraceRemainingMs(now); + if (grace > 0) { + notes.push( + `Server-offline and server-disk checks are paused for another ${Math.ceil(grace / 60_000)} min after the app restarted, so agents get a chance to report before any server is judged.`, + ); + } + for (const c of evaluateHealth(snapshot, healthChecks, now, { skipServers: grace > 0 })) { + const origin = originOfConditionKey(c.key); + const subject = subjectOfConditionKey(c.key); + push({ + category: origin.category, + severity: c.severity ?? "warning", + type: origin.type, + message: c.message, + key: c.key, + link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}` : undefined, + silenced: subject !== null && subjects.has(subject), + }); + } + // Integrations the check couldn't read this time. (A whole-integration entry covers its nodes, so skip those.) + for (const h of held) { + const m = /^(proxmox|synology):(\d+)(?::(.+))?$/.exec(h); + if (!m || (m[3] && held.has(`${m[1]}:${m[2]}`))) continue; + const row = intById.get(Number(m[2])); + cannotCheck(`${SOURCES[m[1]].label} “${row?.name ?? `#${m[2]}`}”${m[3] ? ` (node ${m[3]})` : ""}`, "couldn't be read — see the Diagnostic Log", "Server and storage health"); + } + }), + + check("Automation runs", async () => { + const { items, held } = await collectAutomation(); + for (const c of evaluateAutomation(items).conditions) { + const origin = originOfConditionKey(c.key); + const subject = subjectOfConditionKey(c.key); + push({ category: "automation", severity: "warning", type: origin.type, message: c.message, key: c.key, silenced: subject !== null && subjects.has(subject) }); + } + for (const h of held) { + const m = /^(semaphore|gitea):(\d+)$/.exec(h); + if (!m) continue; + const row = intById.get(Number(m[2])); + cannotCheck(`${SOURCES[m[1]].label} “${row?.name ?? `#${m[2]}`}”`, "couldn't be read — see the Diagnostic Log", "Automation runs"); + } + }), + + check("Proxmox backups", async () => { + const { failures, uncovered, sourceFailures } = await collectProxmoxBackupProblems({ skipSilenced: false }); + for (const f of failures) { + push({ + category: "backup", + severity: "critical", + type: "proxmox", + message: `Latest backup on ${f.node}${f.guestId ? ` (guest ${f.guestId})` : ""} didn't succeed [${f.integrationName}]: ${f.status}`, + key: `proxmox-backup:${f.integrationId}:${f.node}`, + silenced: f.silenced, + }); + } + for (const u of uncovered) { + push({ + category: "backup", + severity: "warning", + type: "proxmox", + message: `${u.guestName} (#${u.vmid}) on ${u.node} isn't covered by any backup job [${u.integrationName}]`, + key: `proxmox-uncovered:${u.integrationId}:${u.vmid}`, + silenced: u.silenced, + }); + } + sourceFailures.forEach((f) => cannotCheckIntegration(f, "Proxmox backups")); + }), + + check("Backup verification", async () => { + const { problems, sourceFailures } = await collectPbsProblems({ skipSilenced: false }); + for (const p of problems) { + push({ + category: "backup", + severity: p.error ? "warning" : "critical", + type: "pbs", + message: p.error + ? `Datastore "${p.datastore}" couldn't be read [${p.integrationName}]: ${p.error}` + : `Datastore "${p.datastore}" has ${plural(p.failedCount, "snapshot")} that failed verification [${p.integrationName}]`, + key: `pbs:${p.integrationId}:${p.datastore}`, + silenced: p.silenced, + }); + } + sourceFailures.forEach((f) => cannotCheckIntegration(f, "Backup verification")); + }), + + check("Image updates", async () => { + const { items, failures } = await collectDockerUpdates(); + for (const u of items) { + push({ + category: "updates", + severity: "info", + type: "dockhand", + message: `${u.containerName} [${u.environmentName}, ${u.integrationName}] has an image update available${u.newerVersion ? ` → ${u.newerVersion}` : ""}`, + key: `docker:${u.integrationId}:${u.environmentName}:${u.containerName}`, + silenced: silencedIntegration(u.integrationId), + }); + } + failures.forEach((f) => cannotCheckIntegration(f, "Image updates")); + }), + + check("Tailscale keys", async () => { + const { items, failures } = await collectTailscaleKeyExpiries(); + for (const k of items) { + push({ + category: "expiry", + severity: k.daysLeft < 0 ? "critical" : "warning", + type: "tailscale", + message: k.daysLeft < 0 ? `Key for ${k.deviceLabel} [${k.integrationName}] has expired` : `Key for ${k.deviceLabel} [${k.integrationName}] expires in ${plural(k.daysLeft, "day")}`, + key: `tailscale-key:${k.integrationId}:${k.deviceLabel}`, + silenced: silencedIntegration(k.integrationId), + }); + } + failures.forEach((f) => cannotCheckIntegration(f, "Tailscale keys")); + }), + + check("Uptime Kuma", async () => { + for (const row of intRows.filter((r) => r.type === "uptimekuma" && r.enabled)) { + try { + const loaded = await loadIntegrationConfig(row.id); + if (!loaded) continue; + for (const m of await createUptimeKumaAdapter(loaded.config as any).listMonitors()) { + if (m.status !== "down") continue; + push({ + category: "monitoring", + severity: "critical", + type: "uptimekuma", + message: `Monitor "${m.name}" is down${m.target ? ` (${m.target}${m.port ? `:${m.port}` : ""})` : ""} [${row.name}]`, + key: `kuma:${row.id}:${m.id}`, + silenced: silencedIntegration(row.id), + }); + } + } catch (err) { + cannotCheckIntegration({ integrationId: row.id, integrationName: row.name, message: errorText(err) }, "Uptime Kuma"); + } + } + }), + + check("osTicket", async () => { + for (const row of intRows.filter((r) => r.type === "osticket" && r.enabled)) { + try { + const loaded = await loadIntegrationConfig(row.id); + if (!loaded) continue; + const overdue = (await createOsTicketAdapter(loaded.config as any).listOpenTickets()).filter((t) => t.isOverdue).length; + if (overdue > 0) { + push({ + category: "tickets", + severity: "warning", + type: "osticket", + message: `${plural(overdue, "open ticket")} ${overdue === 1 ? "is" : "are"} overdue [${row.name}]`, + key: `osticket:${row.id}`, + silenced: silencedIntegration(row.id), + }); + } + } catch (err) { + cannotCheckIntegration({ integrationId: row.id, integrationName: row.name, message: errorText(err) }, "osTicket"); + } + } + }), + + check("Secrets", async () => { + for (const s of await db.select().from(secrets)) { + const status = computeSecretStatus(s.expiryDate, s.warnDays); + if (status.status === "expired") { + push({ category: "expiry", severity: "critical", type: "secrets", message: `Secret "${s.name}" expired ${plural(Math.abs(status.daysLeft), "day")} ago (${s.expiryDate})`, key: `secret:${s.id}` }); + } else if (status.status === "expiring") { + push({ category: "expiry", severity: "warning", type: "secrets", message: `Secret "${s.name}" expires in ${plural(status.daysLeft, "day")} (${s.expiryDate})`, key: `secret:${s.id}` }); + } + if (s.checkHost && s.lastCheckError) { + push({ + category: "expiry", + severity: "warning", + type: "secrets", + message: `Couldn't read the live certificate for "${s.name}" (${s.checkHost}:${s.checkPort ?? 443}) — the expiry shown may be stale: ${s.lastCheckError}`, + key: `secret-check:${s.id}`, + }); + } + } + }), + + check("Domains", async () => { + const { expiring, staleChecks } = await collectDomainAlerts(); + for (const d of expiring) { + push({ + category: "expiry", + severity: d.status === "expired" ? "critical" : "warning", + type: "domains", + message: d.status === "expired" ? `Domain ${d.name} expired on ${d.expiresAt}` : `Domain ${d.name} expires in ${plural(d.daysLeft, "day")} (${d.expiresAt})`, + key: `domain:${d.name}`, + }); + } + for (const s of staleChecks) { + push({ category: "expiry", severity: "info", type: "domains", message: `Couldn't refresh the registration for ${s.name} — the expiry shown may be stale: ${s.error}`, key: `domain-stale:${s.name}` }); + } + }), + + check("Integration failures", async () => { + const { notifications } = await getSettings(); + for (const f of getFailingSources()) { + push({ + category: "integration", + severity: f.alerted ? "critical" : "warning", + type: f.source, + message: `${sourceLabel(f.source)} has failed its last ${plural(f.consecutiveFailures, "call")} in a row${f.alerted ? "" : ` (a notification goes out after ${notifications.integrationFailureThreshold})`} — see the Diagnostic Log`, + key: `failing:${f.source}`, + link: null, + silenced: await isSourceInMaintenance(f.source), + }); + } + }), + ]); + + alerts.sort( + (a, b) => + Number(a.silenced) - Number(b.silenced) || + SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity] || + a.source.localeCompare(b.source) || + a.message.localeCompare(b.message), + ); + + const active = alerts.filter((a) => !a.silenced); + return { + alerts, + counts: { + critical: active.filter((a) => a.severity === "critical").length, + warning: active.filter((a) => a.severity === "warning").length, + info: active.filter((a) => a.severity === "info").length, + silenced: alerts.length - active.length, + }, + couldntCheck: [...blind.entries()].map(([name, v]) => ({ name, error: v.error, affects: [...v.affects] })), + notes, + generatedAt: new Date().toISOString(), + }; +} + +let cache: { at: number; report: AlertsReport } | null = null; +let inFlight: Promise | null = null; + +/** The current alerts, reusing a recent result unless `force` asks for a fresh one (and even then not more than once every few seconds). */ +export async function getAlerts(force: boolean): Promise { + const age = cache ? Date.now() - cache.at : Infinity; + if (cache && age < (force ? MIN_REFRESH_MS : CACHE_MS)) return { ...cache.report, cached: true }; + inFlight ??= collectAlerts() + .then((report) => { + cache = { at: Date.now(), report }; + return report; + }) + .finally(() => { + inFlight = null; + }); + return { ...(await inFlight), cached: false }; +} diff --git a/server/src/services/automationMonitor.ts b/server/src/services/automationMonitor.ts index a9ee4d7..d1e2ec1 100644 --- a/server/src/services/automationMonitor.ts +++ b/server/src/services/automationMonitor.ts @@ -62,7 +62,7 @@ export function evaluateAutomation(items: AutomationItem[]): { conditions: Healt // ─── Collection (I/O) ─────────────────────────────────────────────────────── -async function collect(): Promise<{ items: AutomationItem[]; held: Set; readable: number }> { +export async function collectAutomation(): Promise<{ items: AutomationItem[]; held: Set; readable: number }> { const items: AutomationItem[] = []; const held = new Set(); let readable = 0; @@ -138,7 +138,7 @@ async function loadState(): Promise { * the first time this feature runs) that would otherwise be a wall of alerts about failures that are months old. */ export async function runAutomationCheck(now: number = Date.now()): Promise<{ added: number; resolved: number; baseline: boolean }> { - const { items, held: readHeld, readable } = await collect(); + const { items, held: readHeld, readable } = await collectAutomation(); const stored = await loadState(); // Nothing could be read at all (or nothing is configured): don't spend the "first pass" on an empty picture. diff --git a/server/src/services/dockerUpdateScheduler.ts b/server/src/services/dockerUpdateScheduler.ts index 91b5bda..29f8a0f 100644 --- a/server/src/services/dockerUpdateScheduler.ts +++ b/server/src/services/dockerUpdateScheduler.ts @@ -5,17 +5,28 @@ import { integrations } from "../db/schema.js"; import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createDockhandAdapter } from "../integrations/dockhand/adapter.js"; import { notifyDockerUpdates } from "./notify.js"; +import type { SourceFailure } from "./alertTypes.js"; import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js"; const LAST_RUN_FLAG = "dockerUpdateCheckLastRunDate"; -async function checkDockerUpdates(): Promise { +export interface DockerUpdate { + integrationId: number; + integrationName: string; + containerName: string; + environmentName: string; + newerVersion: string | null; +} + +/** Every container with an image update waiting, across the enabled Dockhand integrations. Shared with the Alerts page. */ +export async function collectDockerUpdates(): Promise<{ items: DockerUpdate[]; failures: SourceFailure[] }> { const rows = await db .select({ id: integrations.id, name: integrations.name }) .from(integrations) .where(and(eq(integrations.type, "dockhand"), eq(integrations.enabled, true))); - const updatesAvailable: { integrationName: string; containerName: string; environmentName: string; newerVersion: string | null }[] = []; + const updatesAvailable: DockerUpdate[] = []; + const failures: SourceFailure[] = []; for (const row of rows) { try { @@ -28,6 +39,7 @@ async function checkDockerUpdates(): Promise { for (const c of containers) { if (!c.updateAvailable) continue; updatesAvailable.push({ + integrationId: row.id, integrationName: row.name, containerName: c.name, environmentName: c.environmentName, @@ -36,10 +48,15 @@ async function checkDockerUpdates(): Promise { } } catch (err) { console.error(`[dockerUpdate] check failed for integration ${row.id}:`, err); + failures.push({ integrationId: row.id, integrationName: row.name, message: err instanceof Error ? err.message : String(err) }); } } - await notifyDockerUpdates(updatesAvailable); + return { items: updatesAvailable, failures }; +} + +async function checkDockerUpdates(): Promise { + await notifyDockerUpdates((await collectDockerUpdates()).items); } async function checkDockerUpdatesOnce(): Promise { diff --git a/server/src/services/healthMonitor.ts b/server/src/services/healthMonitor.ts index a1294a0..55bec76 100644 --- a/server/src/services/healthMonitor.ts +++ b/server/src/services/healthMonitor.ts @@ -21,6 +21,8 @@ export interface HealthCondition { /** Where the data came from, hierarchically ("server", "proxmox:3", "proxmox:3:pve1"). Used to hold a condition when its source can't be reached. */ source: string; message: string; + /** How bad, for the Alerts page. Notifications don't use it. Left out means "warning". */ + severity?: "critical" | "warning"; } export interface ServerSnapshot { @@ -91,6 +93,8 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, const out: HealthCondition[] = []; const limit = thresholds.diskUsagePercent; const pct = (n: number) => `${Math.round(n)}%`; + /** Over the configured threshold is a warning; practically out of room is critical. */ + const fullness = (p: number): "critical" | "warning" => (p >= 95 ? "critical" : "warning"); if (!opts.skipServers) { for (const s of snapshot.servers) { @@ -103,6 +107,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `offline:server:${s.id}`, source: "server", message: `${s.name} hasn't reported for ${formatDuration(age)}`, + severity: "critical", }); } } @@ -115,6 +120,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `disk:server:${s.id}:${d.mount}`, source: "server", message: `${s.name}: ${d.mount} is ${pct(p)} full (${formatBytes(d.usedBytes)} of ${formatBytes(d.sizeBytes)})`, + severity: fullness(p), }); } } @@ -132,6 +138,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `disk:proxmox:${px.integrationId}:${node.node}:rootfs`, source: `proxmox:${px.integrationId}:${node.node}`, message: `${px.integrationName} / ${node.node}: root filesystem is ${pct(rootP)} full`, + severity: fullness(rootP), }); } for (const st of node.storages) { @@ -146,12 +153,14 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `disk:proxmox:${px.integrationId}:storage:${st.id}`, source: `proxmox:${px.integrationId}:shared`, message: `${px.integrationName}: shared storage "${st.id}" is ${pct(p)} full (${formatBytes(st.usedBytes ?? 0)} of ${formatBytes(st.totalBytes ?? 0)})`, + severity: fullness(p), }); } else { out.push({ key: `disk:proxmox:${px.integrationId}:${node.node}:storage:${st.id}`, source: `proxmox:${px.integrationId}:${node.node}`, message: `${px.integrationName} / ${node.node}: storage "${st.id}" is ${pct(p)} full (${formatBytes(st.usedBytes ?? 0)} of ${formatBytes(st.totalBytes ?? 0)})`, + severity: fullness(p), }); } } @@ -166,6 +175,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `synology-volume:${syn.integrationId}:${v.id}`, source: src, message: `${syn.integrationName}: volume ${v.id} status is "${v.status}"`, + severity: "critical", }); } const p = usage(v.sizeUsed, v.sizeTotal); @@ -174,6 +184,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `disk:synology:${syn.integrationId}:${v.id}`, source: src, message: `${syn.integrationName}: volume ${v.id} is ${pct(p)} full (${formatBytes(v.sizeUsed ?? 0)} of ${formatBytes(v.sizeTotal ?? 0)})`, + severity: fullness(p), }); } } @@ -188,6 +199,8 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds, key: `synology-disk:${syn.integrationId}:${d.id}`, source: src, message: `${syn.integrationName}: disk ${d.name || d.id} — ${problems.join(", ")}`, + // A disk that's no longer "normal" is failing; a SMART warning or a threshold crossing is the early notice. + severity: d.status && d.status.toLowerCase() !== "normal" ? "critical" : "warning", }); } } @@ -247,7 +260,12 @@ export function diffConditions( // ─── Collection (I/O) ─────────────────────────────────────────────────────── -async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; held: Set }> { +/** How much longer, in ms, servers are exempt from being judged after a restart (their agents get a chance to report first). 0 once it's over. */ +export function startupGraceRemainingMs(now: number = Date.now()): number { + return Math.max(0, STARTUP_GRACE_MS - (now - PROCESS_START)); +} + +export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; held: Set }> { const held = new Set(); const snapshot: HealthSnapshot = { servers: [], proxmox: [], synology: [] }; diff --git a/server/src/services/integrationHealthMonitor.ts b/server/src/services/integrationHealthMonitor.ts index 1faf851..d9466e7 100644 --- a/server/src/services/integrationHealthMonitor.ts +++ b/server/src/services/integrationHealthMonitor.ts @@ -17,6 +17,13 @@ interface SourceHealth { */ const health = new Map(); +/** Services whose most recent calls have been failing right now, for the Alerts page. `alerted` means a "down" notification has gone out for the streak. */ +export function getFailingSources(): { source: string; consecutiveFailures: number; alerted: boolean }[] { + return [...health.entries()] + .filter(([, state]) => state.consecutiveFailures > 0) + .map(([source, state]) => ({ source, consecutiveFailures: state.consecutiveFailures, alerted: state.alerted })); +} + /** Called after every diagnostic-log entry is recorded, to track consecutive failures per source and alert on threshold-cross / recovery. */ export async function trackIntegrationHealth(source: string, ok: boolean): Promise { const state = health.get(source) ?? { consecutiveFailures: 0, alerted: false }; diff --git a/server/src/services/notify.ts b/server/src/services/notify.ts index 5a20bc7..c278e0a 100644 --- a/server/src/services/notify.ts +++ b/server/src/services/notify.ts @@ -382,9 +382,13 @@ const SOURCE_LABELS: Record = { semaphore: "Semaphore", gitea: "Gitea", dockhand: "Dockhand", + uptimekuma: "Uptime Kuma", + phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", + osticket: "osTicket", }; -function sourceLabel(source: string): string { +export function sourceLabel(source: string): string { return SOURCE_LABELS[source] ?? source; } diff --git a/server/src/services/pbsVerificationScheduler.ts b/server/src/services/pbsVerificationScheduler.ts index f566b92..87ef729 100644 --- a/server/src/services/pbsVerificationScheduler.ts +++ b/server/src/services/pbsVerificationScheduler.ts @@ -6,21 +6,39 @@ import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createPbsAdapter } from "../integrations/pbs/adapter.js"; import { notifyPbsVerificationFailed } from "./notify.js"; import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js"; -import { isInMaintenance } from "./maintenance.js"; +import { activeSubjects } from "./maintenance.js"; +import type { SourceFailure } from "./alertTypes.js"; const LAST_RUN_FLAG = "pbsVerificationCheckLastRunDate"; -async function checkPbsVerification(): Promise { +export interface PbsProblem { + integrationId: number; + integrationName: string; + datastore: string; + failedCount: number; + error: string | null; + silenced: boolean; +} + +/** + * Datastores with snapshots that failed verification, or that couldn't be read at all, across the enabled PBS + * integrations. The daily check skips integrations under a maintenance window altogether; the Alerts page passes + * skipSilenced: false to see them, flagged. + */ +export async function collectPbsProblems(opts: { skipSilenced: boolean }): Promise<{ problems: PbsProblem[]; sourceFailures: SourceFailure[] }> { const rows = await db .select({ id: integrations.id, name: integrations.name }) .from(integrations) .where(and(eq(integrations.type, "pbs"), eq(integrations.enabled, true))); - const failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[] = []; + const failures: PbsProblem[] = []; + const sourceFailures: SourceFailure[] = []; + const silencedSubjects = await activeSubjects(); for (const row of rows) { // A PBS host being worked on can't be reached reliably; this check is daily, so tomorrow's pass covers it. - if (await isInMaintenance("integration", row.id)) continue; + const silenced = silencedSubjects.has(`integration:${row.id}`); + if (silenced && opts.skipSilenced) continue; try { const loaded = await loadIntegrationConfig(row.id); if (!loaded) continue; @@ -29,17 +47,22 @@ async function checkPbsVerification(): Promise { for (const d of datastores) { if (d.error) { - failures.push({ integrationName: row.name, datastore: d.name, failedCount: 0, error: d.error }); + failures.push({ integrationId: row.id, integrationName: row.name, datastore: d.name, failedCount: 0, error: d.error, silenced }); } else if (d.failedCount > 0) { - failures.push({ integrationName: row.name, datastore: d.name, failedCount: d.failedCount, error: null }); + failures.push({ integrationId: row.id, integrationName: row.name, datastore: d.name, failedCount: d.failedCount, error: null, silenced }); } } } catch (err) { console.error(`[pbsVerification] check failed for integration ${row.id}:`, err); + sourceFailures.push({ integrationId: row.id, integrationName: row.name, message: err instanceof Error ? err.message : String(err) }); } } - await notifyPbsVerificationFailed(failures); + return { problems: failures, sourceFailures }; +} + +async function checkPbsVerification(): Promise { + await notifyPbsVerificationFailed((await collectPbsProblems({ skipSilenced: true })).problems); } async function checkPbsVerificationOnce(): Promise { diff --git a/server/src/services/proxmoxBackupScheduler.ts b/server/src/services/proxmoxBackupScheduler.ts index 982f323..2ccf657 100644 --- a/server/src/services/proxmoxBackupScheduler.ts +++ b/server/src/services/proxmoxBackupScheduler.ts @@ -6,22 +6,50 @@ import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createProxmoxAdapter, guestsWithoutBackupCoverage, type ProxmoxBackupTask } from "../integrations/proxmox/adapter.js"; import { notifyProxmoxBackupFailure, notifyProxmoxUncoveredGuests } from "./notify.js"; import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js"; -import { isInMaintenance } from "./maintenance.js"; +import { activeSubjects } from "./maintenance.js"; +import type { SourceFailure } from "./alertTypes.js"; const LAST_RUN_FLAG = "proxmoxBackupCheckLastRunDate"; -async function checkProxmoxBackups(): Promise { +export interface BackupFailure { + integrationId: number; + integrationName: string; + node: string; + guestId: string | null; + status: string; + silenced: boolean; +} + +export interface UncoveredGuest { + integrationId: number; + integrationName: string; + guestName: string; + vmid: number; + node: string; + silenced: boolean; +} + +/** + * Failed latest backups and guests no backup job covers, across the enabled Proxmox integrations. The daily check skips + * integrations under a maintenance window altogether (a host being worked on can't run or report backups, and tomorrow's + * pass covers it); the Alerts page passes skipSilenced: false to see them, flagged. + */ +export async function collectProxmoxBackupProblems(opts: { + skipSilenced: boolean; +}): Promise<{ failures: BackupFailure[]; uncovered: UncoveredGuest[]; sourceFailures: SourceFailure[] }> { const rows = await db .select({ id: integrations.id, name: integrations.name }) .from(integrations) .where(and(eq(integrations.type, "proxmox"), eq(integrations.enabled, true))); - const failures: { integrationName: string; node: string; guestId: string | null; status: string }[] = []; - const uncovered: { integrationName: string; guestName: string; vmid: number; node: string }[] = []; + const failures: BackupFailure[] = []; + const uncovered: UncoveredGuest[] = []; + const sourceFailures: SourceFailure[] = []; + const silencedSubjects = await activeSubjects(); for (const row of rows) { - // A host being worked on can't run or report backups; this check is daily, so tomorrow's pass covers it. - if (await isInMaintenance("integration", row.id)) continue; + const silenced = silencedSubjects.has(`integration:${row.id}`); + if (silenced && opts.skipSilenced) continue; try { const loaded = await loadIntegrationConfig(row.id); if (!loaded) continue; @@ -44,18 +72,24 @@ async function checkProxmoxBackups(): Promise { for (const [node, task] of latestByNode) { if (!task.ok && task.status !== "running") { - failures.push({ integrationName: row.name, node, guestId: task.guestId, status: task.status }); + failures.push({ integrationId: row.id, integrationName: row.name, node, guestId: task.guestId, status: task.status, silenced }); } } for (const g of guestsWithoutBackupCoverage(guests, jobs)) { - uncovered.push({ integrationName: row.name, guestName: g.name, vmid: g.vmid, node: g.node }); + uncovered.push({ integrationId: row.id, integrationName: row.name, guestName: g.name, vmid: g.vmid, node: g.node, silenced }); } } catch (err) { console.error(`[proxmoxBackup] check failed for integration ${row.id}:`, err); + sourceFailures.push({ integrationId: row.id, integrationName: row.name, message: err instanceof Error ? err.message : String(err) }); } } + return { failures, uncovered, sourceFailures }; +} + +async function checkProxmoxBackups(): Promise { + const { failures, uncovered } = await collectProxmoxBackupProblems({ skipSilenced: true }); await notifyProxmoxBackupFailure(failures); await notifyProxmoxUncoveredGuests(uncovered); } diff --git a/server/src/services/tailscaleKeyExpiryScheduler.ts b/server/src/services/tailscaleKeyExpiryScheduler.ts index 142fd23..a11fe20 100644 --- a/server/src/services/tailscaleKeyExpiryScheduler.ts +++ b/server/src/services/tailscaleKeyExpiryScheduler.ts @@ -5,17 +5,27 @@ import { integrations } from "../db/schema.js"; import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; import { createTailscaleAdapter, isKeyExpiringSoon } from "../integrations/tailscale/adapter.js"; import { notifyTailscaleKeyExpiry } from "./notify.js"; +import type { SourceFailure } from "./alertTypes.js"; import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js"; const LAST_RUN_FLAG = "tailscaleKeyCheckLastRunDate"; -async function checkTailscaleKeyExpiry(): Promise { +export interface TailscaleKeyExpiry { + integrationId: number; + integrationName: string; + deviceLabel: string; + daysLeft: number; +} + +/** Every device key that's expired or about to, across the enabled Tailscale integrations. Shared with the Alerts page. */ +export async function collectTailscaleKeyExpiries(): Promise<{ items: TailscaleKeyExpiry[]; failures: SourceFailure[] }> { const rows = await db .select({ id: integrations.id, name: integrations.name }) .from(integrations) .where(and(eq(integrations.type, "tailscale"), eq(integrations.enabled, true))); - const expiring: { integrationName: string; deviceLabel: string; daysLeft: number }[] = []; + const expiring: TailscaleKeyExpiry[] = []; + const failures: SourceFailure[] = []; const now = Date.now(); for (const row of rows) { @@ -27,14 +37,19 @@ async function checkTailscaleKeyExpiry(): Promise { for (const d of devices) { if (!isKeyExpiringSoon(d, now)) continue; const daysLeft = Math.floor((new Date(d.keyExpiry!).getTime() - now) / 86_400_000); - expiring.push({ integrationName: row.name, deviceLabel: d.label || d.hostname, daysLeft }); + expiring.push({ integrationId: row.id, integrationName: row.name, deviceLabel: d.label || d.hostname, daysLeft }); } } catch (err) { console.error(`[tailscaleKeyExpiry] check failed for integration ${row.id}:`, err); + failures.push({ integrationId: row.id, integrationName: row.name, message: err instanceof Error ? err.message : String(err) }); } } - await notifyTailscaleKeyExpiry(expiring); + return { items: expiring, failures }; +} + +async function checkTailscaleKeyExpiry(): Promise { + await notifyTailscaleKeyExpiry((await collectTailscaleKeyExpiries()).items); } async function checkTailscaleKeyExpiryOnce(): Promise { diff --git a/web/src/App.tsx b/web/src/App.tsx index 341c744..e1787d9 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -24,6 +24,7 @@ import UptimeKuma from "./pages/UptimeKuma"; import PbsBackup from "./pages/PbsBackup"; import OsTicket from "./pages/OsTicket"; import AdminLinks from "./pages/AdminLinks"; +import Alerts from "./pages/Alerts"; import Generator from "./pages/Generator"; import Maintenance from "./pages/Maintenance"; import Domains from "./pages/Domains"; @@ -38,8 +39,8 @@ import CacheSettings from "./pages/settings/CacheSettings"; import LogSettings from "./pages/settings/LogSettings"; import BackupSettings from "./pages/settings/BackupSettings"; import AppShell from "./layout/AppShell"; -import { setDateTimeSettings } from "./utils/date"; import DialogHost from "./components/DialogHost"; +import { setDateTimeSettings } from "./utils/date"; import { setPageSize } from "./utils/pageSize"; const roleRank: Record = { viewer: 0, operator: 1, admin: 2 }; @@ -114,6 +115,7 @@ export default function App() { } /> } /> } /> + } /> } /> + ); - } diff --git a/web/src/api/client.ts b/web/src/api/client.ts index 7937f13..afe70c5 100644 --- a/web/src/api/client.ts +++ b/web/src/api/client.ts @@ -397,6 +397,30 @@ export interface ServerDetail { links: ServerLink[]; } +export type AlertSeverity = "critical" | "warning" | "info"; +export type AlertCategory = "offline" | "disk" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets"; + +export interface AlertItem { + id: string; + severity: AlertSeverity; + category: AlertCategory; + source: string; + message: string; + link: string | null; + /** Under a maintenance window: still a real problem, but its notifications are held back. */ + silenced: boolean; +} + +export interface AlertsReport { + alerts: AlertItem[]; + counts: { critical: number; warning: number; info: number; silenced: number }; + couldntCheck: { name: string; error: string; affects: string[] }[]; + notes: string[]; + generatedAt: string; + /** This is a recent result being reused, not a fresh check. */ + cached: boolean; +} + export interface AdminLink { id: number; serverId: number; @@ -1006,6 +1030,9 @@ export const api = { syncProxmox: () => request("/api/ipam/sync-proxmox", { method: "POST" }), syncPhpIpam: () => request("/api/ipam/sync-phpipam", { method: "POST" }), }, + alerts: { + list: (refresh = false) => request(`/api/alerts${refresh ? "?refresh=1" : ""}`), + }, ports: { agent: () => request("/api/ports/agent"), forwards: { diff --git a/web/src/layout/AppShell.tsx b/web/src/layout/AppShell.tsx index b98507e..b10a929 100644 --- a/web/src/layout/AppShell.tsx +++ b/web/src/layout/AppShell.tsx @@ -32,6 +32,7 @@ import { IconTicket, IconPlug, IconExternalLink, + IconAlertTriangle, } from "@tabler/icons-react"; import { api, type CurrentUser, type MaintenanceWindow } from "../api/client"; import { formatRemaining } from "../utils/duration"; @@ -101,6 +102,7 @@ const NAV: NavEntry[] = [ label: "Operations", icon: , items: [ + { to: "/alerts", label: "Alerts", icon: }, { to: "/maintenance", label: "Maintenance", icon: }, { to: "/uptime-kuma", label: "Uptime Kuma", icon: }, { to: "/osticket", label: "osTicket", icon: }, diff --git a/web/src/pages/Alerts.tsx b/web/src/pages/Alerts.tsx new file mode 100644 index 0000000..7e2b061 --- /dev/null +++ b/web/src/pages/Alerts.tsx @@ -0,0 +1,245 @@ +import { useEffect, useMemo, useState } from "react"; +import { Link } from "react-router-dom"; +import { api, type AlertCategory, type AlertItem, type AlertSeverity, type AlertsReport, type CurrentUser } from "../api/client"; +import { useSortable } from "../hooks/useSortable"; +import SortableTh from "../components/SortableTh"; +import { usePagination } from "../hooks/usePagination"; +import Pagination from "../components/Pagination"; +import { downloadCsv } from "../utils/csv"; +import { formatDateTime } from "../utils/date"; +import { formatAgo } from "../utils/duration"; +import { readableError } from "../utils/errors"; + +const SEVERITY: Record = { + critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 }, + warning: { label: "Warning", badge: "bg-yellow-lt text-yellow", rank: 1 }, + info: { label: "Info", badge: "bg-blue-lt text-blue", rank: 2 }, +}; + +const CATEGORY_LABELS: Record = { + offline: "Server down", + disk: "Disk & storage", + backup: "Backups", + updates: "Updates", + expiry: "Expiry", + automation: "Automation", + integration: "Integration failing", + monitoring: "Monitoring", + tickets: "Tickets", +}; + +type Row = AlertItem & { rank: number }; + +export default function Alerts(_props: { user: CurrentUser }) { + const [report, setReport] = useState(null); + const [error, setError] = useState(null); + const [loading, setLoading] = useState(false); + const [severity, setSeverity] = useState<"all" | AlertSeverity>("all"); + const [category, setCategory] = useState<"all" | AlertCategory>("all"); + const [search, setSearch] = useState(""); + const [showSilenced, setShowSilenced] = useState(true); + + function load(refresh = false) { + setLoading(true); + return api.alerts + .list(refresh) + .then((res) => { + setReport(res); + setError(null); + }) + .catch((err) => setError(readableError(err))) + .finally(() => setLoading(false)); + } + + useEffect(() => { + void load(); + }, []); + + const rows: Row[] = useMemo(() => { + if (!report) return []; + const q = search.trim().toLowerCase(); + return report.alerts + .filter((a) => (severity === "all" || a.severity === severity) && (category === "all" || a.category === category) && (showSilenced || !a.silenced)) + .filter((a) => !q || [a.message, a.source, CATEGORY_LABELS[a.category]].some((v) => v.toLowerCase().includes(q))) + .map((a) => ({ ...a, rank: SEVERITY[a.severity].rank })); + }, [report, severity, category, search, showSilenced]); + + // No initial sort: the server already puts active problems first, worst first. + const { sorted, sortKey, sortDir, requestSort } = useSortable(rows); + const { pageItems, page, setPage, pageCount, totalCount } = usePagination(sorted); + + const presentCategories = useMemo(() => [...new Set((report?.alerts ?? []).map((a) => a.category))], [report]); + + function exportCsv() { + downloadCsv( + "alerts.csv", + ["Severity", "Category", "Source", "Alert", "Silenced"], + (sorted ?? []).map((a) => [SEVERITY[a.severity].label, CATEGORY_LABELS[a.category], a.source, a.message, a.silenced ? "yes" : "no"]), + ); + } + + const counts = report?.counts; + const nothingFound = report !== null && report.alerts.length === 0; + const cards: { label: string; value: number | undefined; color: string }[] = [ + { label: "Critical", value: counts?.critical, color: "text-red" }, + { label: "Warnings", value: counts?.warning, color: "text-yellow" }, + { label: "Info", value: counts?.info, color: "text-blue" }, + { label: "Silenced (maintenance)", value: counts?.silenced, color: "text-secondary" }, + ]; + + return ( + <> +
+

Alerts

+
+ + +
+
+
+ What's wrong right now across your servers and integrations — full disks, servers that stopped reporting, failed backups, updates waiting, + things about to expire. It runs the same checks that send notifications, whether or not those notifications are switched on. + {report && ( + <> + {" "} + + Checked {formatAgo(report.generatedAt)} + {report.cached ? " (a recent result, reused)" : ""}. + + + )} +
+ {error &&
{error}
} + + {!report && !error && ( +
+
Checking everything — this can take a few seconds…
+
+ )} + + {report && ( + <> +
+ {cards.map((c) => ( +
+
+
+
{c.label}
+
{c.value ?? "—"}
+
+
+
+ ))} +
+ + {report.couldntCheck.length > 0 && ( +
+
+
Couldn't check everything — what's missing below isn't necessarily fine
+
    + {report.couldntCheck.map((c) => ( +
  • + {c.name} — {c.error} + {c.affects.length > 0 && (affects: {c.affects.join(", ")})} +
    • + ))} +
+
+
+ )} + {report.notes.map((n) => ( +
+ {n} +
+ ))} + + {nothingFound ? ( +
+
+ {report.couldntCheck.length === 0 ? ( + <> +
All clear
+
Nothing needs attention right now.
+ + ) : ( +
No problems found in what could be checked — see the warning above for what couldn't.
+ )} +
+
+ ) : ( +
+
+
+ setSearch(e.target.value)} /> + + + {(counts?.silenced ?? 0) > 0 && ( + + )} +
+
+
+ + + + label="Severity" sortKeyName="rank" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + label="Kind" sortKeyName="category" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + label="Source" sortKeyName="source" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + label="Alert" sortKeyName="message" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + + + + {(pageItems ?? []).map((a) => ( + + + + + + + ))} + {(sorted ?? []).length === 0 && ( + + + + )} + +
+ {SEVERITY[a.severity].label} + {CATEGORY_LABELS[a.category]}{a.link ? {a.source} : a.source} + {a.message} + {a.silenced && ( + + silenced + + )} +
+ Nothing matches these filters. +
+
+ +
+ )} + + )} + + ); +}