Alert when an integration or DNS provider fails repeatedly
The Diagnostic Log already records every outbound call's success or failure, but nothing acted on it — you'd only notice an integration was down by happening to open its page. Adds a per-source consecutive- failure counter (in-memory, reset on restart, same durability tier as the diag log's own ring buffer) hooked into recordDiagEntry: crossing the configurable threshold (default 3) sends one "down" notification on every configured channel, and a "recovered" notification fires once it succeeds again — no repeat spam while it stays down. New "Integration/DNS provider failing repeatedly" toggle and threshold field under Settings -> Notifications. Verified end-to-end against an isolated scratch database with a real local HTTP server standing in for the webhook channel: 5 consecutive failures produced exactly one "Down" notification (at the 3rd failure, correctly naming "3 calls"), a subsequent success produced exactly one "Recovered" notification, and two more failures on a fresh streak triggered nothing (below threshold) — confirmed the real dev database's mtime was untouched throughout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
10d123b18a
commit
b5a4c6e2d9
7 files changed
+105
No files matched your search
@@ -44,6 +44,8 @@ const DEFAULT_NOTIFICATIONS: NotificationEvents = {
|
||||
tailscaleKeyCheck: true,
|
||||
secretCheckTime: "08:00",
|
||||
timezone: "UTC",
|
||||
integrationFailureAlerts: true,
|
||||
integrationFailureThreshold: 3,
|
||||
};
|
||||
|
||||
type TestResult = { ok: boolean; message: string } | null;
|
||||
@@ -489,6 +491,7 @@ export default function NotificationSettings() {
|
||||
{ key: "dnsDelete" as const, label: "DNS record deleted" },
|
||||
{ key: "secretCheck" as const, label: "Secret expiry reminder" },
|
||||
{ key: "tailscaleKeyCheck" as const, label: "Tailscale key expiry reminder" },
|
||||
{ key: "integrationFailureAlerts" as const, label: "Integration/DNS provider failing repeatedly" },
|
||||
].map(({ key, label }) => (
|
||||
<label key={key} className="form-check mb-2">
|
||||
<input
|
||||
@@ -500,6 +503,26 @@ export default function NotificationSettings() {
|
||||
<span className="form-check-label">{label}</span>
|
||||
</label>
|
||||
))}
|
||||
<div className="row g-2 mt-2">
|
||||
<div className="col-6">
|
||||
<label className="form-label">Alert after</label>
|
||||
<div className="input-group">
|
||||
<input
|
||||
type="number"
|
||||
className="form-control"
|
||||
min={1}
|
||||
max={20}
|
||||
value={notifications.integrationFailureThreshold}
|
||||
disabled={!notifications.integrationFailureAlerts}
|
||||
onChange={(e) => setNotifications((n) => ({ ...n, integrationFailureThreshold: Number(e.target.value) }))}
|
||||
/>
|
||||
<span className="input-group-text">consecutive failures</span>
|
||||
</div>
|
||||
<div className="form-hint">
|
||||
Based on the Diagnostic Log. A recovery notice is sent once it succeeds again.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{(() => {
|
||||
const dailyChecksEnabled = notifications.secretCheck || notifications.tailscaleKeyCheck;
|
||||
return (
|
||||
|
||||
Reference in new issue
Block a user