Files
Homelab-manager/web/src/pages/Maintenance.tsx
T
bobbanandClaude Sonnet 5 aae4f0d74f Add maintenance mode to silence alerts while working on a server or integration
Rebooting Proxmox or patching a server triggered failure/offline alerts
you then had to dismiss. A maintenance window silences alerts about one
server, integration, or DNS provider for a chosen time. New Maintenance
page (start with a duration and optional reason, end early, see what's
silenced and what isn't) and a banner in the app shell so every signed-in
user can see what is currently silenced. Starting/ending is operator-only
and audit-logged; starting one on a target that already has a window
restarts its clock instead of stacking.

Silenced for the target: server offline/disk alerts, Proxmox/Synology
storage and health alerts, Proxmox backup alerts, and "integration down"
alerts. Not silenced: expiry and update reminders, DNS change notices.

The design goal is that this cannot hide a real outage:
- Every window has a required end (5 min to 7 days); there is no
  open-ended option, so a forgotten window expires by itself.
- A silenced problem is deliberately NOT recorded as "known". If it is
  still present when the window ends it alerts then, as new. A problem
  that was already alerted before the window stays known, so it isn't
  repeated, and is reported cleared only after the window ends.
- Failure alerts keep counting failures during a window without marking
  themselves alerted, so an outage that outlasts the window alerts on the
  very next failed call.

Known limitation, stated on the page: integration-failure alerts are
tracked per service TYPE (all "proxmox"), not per configured instance, so
a window on one Proxmox integration also silences a failure on a second
Proxmox integration while it's open. Fixing that means threading the
integration id through every adapter and the diagnostic log, which is a
much larger change than this feature.

Also moved the API-error-message helper out of Secrets.tsx into a shared
util now that two pages use it. New table maintenance_windows (migration
0008).

Verified with 44 checks: the condition-key-to-subject mapping (including
server:3 vs server:33), the diff rules with silenced subjects (new problem
not recorded, alerts when the window ends; already-known one carried and
not repeated; clears only after the window), window expiry and
integration/DNS-provider source matching, the failure tracker end to end
against a webhook (silent during a window while an unrelated service still
alerts; outage that outlasts the window alerts on the next failure and
only once; fail-and-recover fully inside a window sends nothing), a full
health pass against a real window, and the real router with a stubbed
session (role rules, duration bounds including the missing-duration case,
extend-not-stack, 404s, deleted targets hidden, audit entries). Real dev
database mtime untouched.

Not done: I haven't clicked through the new page or banner in a browser
(they sit behind the Authentik login); it builds and the API behind it is
tested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-26 02:23:06 +02:00

250 lines
9.4 KiB
TypeScript

import { useEffect, useState } from "react";
import { api, type CurrentUser, type MaintenanceTargets, type MaintenanceTargetType, type MaintenanceWindow } from "../api/client";
import { formatDateTime } from "../utils/date";
import { formatRemaining } from "../utils/duration";
import { readableError } from "../utils/errors";
const DURATIONS: { minutes: number; label: string }[] = [
{ minutes: 30, label: "30 minutes" },
{ minutes: 60, label: "1 hour" },
{ minutes: 120, label: "2 hours" },
{ minutes: 240, label: "4 hours" },
{ minutes: 480, label: "8 hours" },
{ minutes: 1440, label: "24 hours" },
{ minutes: 4320, label: "3 days" },
];
// Tells the app shell's banner to refetch right away instead of waiting for its next poll.
function announceChange() {
window.dispatchEvent(new Event("maintenance-changed"));
}
export default function Maintenance({ user }: { user: CurrentUser }) {
const canEdit = user.role === "admin" || user.role === "operator";
const [windows, setWindows] = useState<MaintenanceWindow[] | null>(null);
const [targets, setTargets] = useState<MaintenanceTargets | null>(null);
const [error, setError] = useState<string | null>(null);
const [now, setNow] = useState(() => Date.now());
const [target, setTarget] = useState("");
const [minutes, setMinutes] = useState(60);
const [reason, setReason] = useState("");
const [busy, setBusy] = useState(false);
function load() {
api.maintenance
.list()
.then((res) => {
setWindows(res.windows);
setTargets(res.targets);
})
.catch((err) => setError(readableError(err)));
}
useEffect(() => {
load();
const tick = setInterval(() => setNow(Date.now()), 30_000);
return () => clearInterval(tick);
}, []);
async function start(e: React.FormEvent) {
e.preventDefault();
const [targetType, id] = target.split(":");
if (!targetType || !id) return;
setError(null);
setBusy(true);
try {
await api.maintenance.start({
targetType: targetType as MaintenanceTargetType,
targetId: Number(id),
minutes,
reason: reason.trim() || undefined,
});
setTarget("");
setReason("");
load();
announceChange();
} catch (err) {
setError(readableError(err));
} finally {
setBusy(false);
}
}
async function end(w: MaintenanceWindow) {
setError(null);
try {
await api.maintenance.end(w.id);
load();
announceChange();
} catch (err) {
setError(readableError(err));
}
}
return (
<>
<h2 className="page-title mb-3">Maintenance</h2>
<div className="text-secondary mb-3">
Silence alerts about one server or integration while you work on it. Every window ends on its own — anything
still wrong when it ends alerts then, so nothing stays hidden.
</div>
{error && <div className="alert alert-danger">{error}</div>}
{canEdit && (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">Start maintenance</h3>
</div>
<form onSubmit={start}>
<div className="card-body row g-3">
<div className="col-md-4">
<label className="form-label">What are you working on?</label>
<select className="form-select" required value={target} onChange={(e) => setTarget(e.target.value)}>
<option value="">Choose…</option>
{targets && targets.servers.length > 0 && (
<optgroup label="Servers">
{targets.servers.map((s) => (
<option key={`s${s.id}`} value={`server:${s.id}`}>
{s.name}
</option>
))}
</optgroup>
)}
{targets && targets.integrations.length > 0 && (
<optgroup label="Integrations">
{targets.integrations.map((i) => (
<option key={`i${i.id}`} value={`integration:${i.id}`}>
{i.name} ({i.type})
</option>
))}
</optgroup>
)}
{targets && targets.dnsProviders.length > 0 && (
<optgroup label="DNS providers">
{targets.dnsProviders.map((p) => (
<option key={`d${p.id}`} value={`dns_provider:${p.id}`}>
{p.name} ({p.providerType})
</option>
))}
</optgroup>
)}
</select>
</div>
<div className="col-md-3">
<label className="form-label">For</label>
<select className="form-select" value={minutes} onChange={(e) => setMinutes(Number(e.target.value))}>
{DURATIONS.map((d) => (
<option key={d.minutes} value={d.minutes}>
{d.label}
</option>
))}
</select>
</div>
<div className="col-md-5">
<label className="form-label">Reason (optional)</label>
<input
className="form-control"
maxLength={200}
placeholder="e.g. kernel update and reboot"
value={reason}
onChange={(e) => setReason(e.target.value)}
/>
</div>
</div>
<div className="card-footer">
<button type="submit" className="btn btn-primary" disabled={busy || !target}>
{busy ? "Starting…" : "Start maintenance"}
</button>
</div>
</form>
</div>
)}
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">Active</h3>
</div>
<div className="table-responsive">
<table className="table table-vcenter card-table">
<thead>
<tr>
<th>Target</th>
<th>Reason</th>
<th>Ends</th>
<th>Started by</th>
{canEdit && <th className="w-1">Actions</th>}
</tr>
</thead>
<tbody>
{windows?.map((w) => (
<tr key={w.id}>
<td>
{w.targetName} <span className="text-secondary">· {w.targetKind}</span>
</td>
<td className="text-secondary">{w.reason ?? "—"}</td>
<td>
<span className="badge bg-yellow-lt text-yellow me-2">{formatRemaining(w.endsAt, now)} left</span>
<span className="text-secondary small">{formatDateTime(new Date(w.endsAt))}</span>
</td>
<td className="text-secondary">{w.createdBy ?? "—"}</td>
{canEdit && (
<td>
<button className="btn btn-sm btn-outline-secondary" onClick={() => end(w)}>
End now
</button>
</td>
)}
</tr>
))}
{windows?.length === 0 && (
<tr>
<td colSpan={canEdit ? 5 : 4} className="text-secondary text-center">
Nothing is in maintenance — all alerts are active.
</td>
</tr>
)}
</tbody>
</table>
</div>
</div>
<div className="card">
<div className="card-header">
<h3 className="card-title">What gets silenced</h3>
</div>
<div className="card-body">
<div className="row">
<div className="col-md-6">
<div className="fw-bold mb-1">Silenced for the target</div>
<ul className="text-secondary">
<li>Server offline and disk-full alerts (for a server).</li>
<li>Storage, volume and disk-health alerts (for a Proxmox or Synology integration).</li>
<li>Backup-failure and uncovered-guest alerts (for a Proxmox integration).</li>
<li>
"Integration down" alerts for that <em>type</em> of service — these are tracked per type (e.g. all
Proxmox), not per instance, so with two Proxmox integrations a failure on the other one is silenced
too while a window is open.
</li>
</ul>
</div>
<div className="col-md-6">
<div className="fw-bold mb-1">Not silenced</div>
<ul className="text-secondary">
<li>Secret and certificate expiry, Tailscale key expiry, Docker update reminders.</li>
<li>DNS record change notifications.</li>
</ul>
<div className="fw-bold mb-1">When a window ends</div>
<div className="text-secondary">
A problem that started during it and is still there alerts on the next check (within 15 minutes). One
that was already alerted before it began and has since cleared is reported as cleared; one that
started and cleared entirely inside the window is never reported.
</div>
</div>
</div>
</div>
</div>
</>
);
}