The data was all being collected (agent last-seen, per-disk usage,
Proxmox storage, Synology volume/disk health) but nothing acted on
it, so a dead server or a full disk was only noticed by opening the
right page.
A new health pass runs every 15 minutes (matching the agent's default
report interval) and raises one notification when a problem starts and
one when it clears: a server's agent silent past a threshold (default
60 min), a server disk / Proxmox storage or root filesystem / Synology
volume at or above a usage threshold (default 90%), and a Synology
volume or disk that isn't "normal", has bad SMART, bad sectors past the
threshold, or life remaining below it. Both thresholds and an on/off
toggle live under Settings -> Notifications.
The parts that make this trustworthy rather than noisy:
- A problem is keyed by identity, so it alerts once and not every run;
a shared Proxmox storage listed by every node is one problem, not
one per node.
- Active problems persist across restarts, so a rebuild doesn't
re-alert everything already known.
- If a source can't be read on a given run (Proxmox/Synology
unreachable, one node lacking privileges) its existing problems are
held, not reported "cleared" and then re-alerted when it comes back —
the integration-failure alert already owns "the integration is down".
- For 20 minutes after startup server-derived problems are held too:
agents couldn't report while the app was down, so judging them then
would report every server offline after any restart.
- An offline server's disk figures are stale and are not judged; a
server that never reported has no agent and raises nothing.
- Tracking continues while the toggle is off (only sending is gated),
so turning it back on doesn't dump every long-standing problem.
Timestamps without a zone (SQLite's format) are read as UTC; the test
runs on a UTC+2 machine, where reading them as local time gives a
different answer.
Verified with 32 checks: the evaluation rules and the state diff as
pure functions (exact thresholds, the proxmox:1 vs proxmox:10 prefix
trap, the flapping sequence), then a whole pass against a real
Proxmox adapter talking to a fake HTTPS cluster (one node returning
403, the whole API down, a shared storage on two nodes, a node that
recovers), a webhook receiver, the real DB, and the persisted state.
Not exercised end-to-end: the Synology collection path — its rules are
tested on data shaped exactly like the adapter's output types, but I
did not stand up a fake DSM. Real dev database mtime untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A certificate secret's expiry was only ever what someone typed in, so
a renewed cert (or a wrong date) meant the app's reminders were
silently wrong. A certificate secret can now be given a host:port; the
app opens a real TLS connection and reads the certificate's actual
expiry — on create/edit (if the host changes), daily, and via a
per-row "Check now" — and keeps expiryDate in sync. Because the daily
refresh runs before the existing expiry check, the reminder is always
computed from what's actually being served.
Verification is deliberately off for the connection: homelab services
routinely serve self-signed/internal-CA certs, and an already-expired
one is exactly the case worth reporting, which a verifying connection
would refuse before exposing the dates.
Failure handling avoids the silent-staleness this is meant to fix: a
failed check keeps the last known date, records why on the row (shown
as a "Check failed" badge), and is listed in the daily secrets
notification. Creating a monitored secret whose host can't be reached
and with no manual date is rejected with the reason rather than saved
blank. A non-TLS port (the likeliest typo) gets a plain-language
error instead of raw OpenSSL output.
Server-side connections to a user-supplied host:port need the same
operator role that already gates editing secrets (and running Semaphore
templates, which is strictly more powerful); the host is validated
against a strict character set before any connection is made.
New nullable secrets columns (check_host, check_port, last_checked_at,
last_check_error) via migration 0007; existing rows are unaffected.
Verified against real TLS servers (openssl-generated certs) and the
real secrets router with a stubbed session: a live 45-day cert read
back as the correct date via both an IP host (no SNI) and a hostname;
an already-expired cert reported its past date and shows as expired;
refused connections, a server that accepts but never answers (times
out), and a plain non-TLS server each produced a descriptive error
rather than a hang or crash. Through the router: create with a host
and no date reads the date; unreachable host with no date -> 400 with
the reason; unreachable with a manual date -> saved with the error
recorded; host on a non-certificate type and an invalid host string
-> 400; a hand-typed date on a monitored secret is ignored; changing
the host re-checks immediately; changing the type away from
certificate ends monitoring; a viewer gets 403 on Check now. 23 checks,
all passing (a first re-run showed 2 spurious failures that were leftover
rows from the previous run's scratch database, confirmed by a clean re-run).
Real dev database mtime untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A failing backup run is visible now, but a guest with no backup job
covering it in the first place was still a silent gap. Rather than
depending on Proxmox's /cluster/backup-info/not-backed-up-guests
endpoint (only exists on newer PVE versions), this derives coverage
from data already fetched: a guest counts as covered if any enabled
job either lists its vmid directly, or backs up "all guests" (scoped
to the job's node, if it has one) without excluding it.
Adds a warning banner plus a full table to the Proxmox page's Backups
card, and extends the existing daily "Proxmox backup failed"
notification (relabeled to mention this too) to also list uncovered
guests, gated by the same toggle.
Verified the coverage logic directly (it's a pure function, so no
fake server needed) across 7 cases: no jobs at all, an all-guests job
with an exclude list, a specific-vmids job, a node-scoped job that
shouldn't cover a guest on a different node, a disabled job providing
no real coverage, two jobs whose combined scope covers everything
neither would alone, and a realistic mixed scenario — all passed.
Confirmed the real dev database's mtime was untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Instant notifications (DNS changes, integration failure/recovery
alerts) had no way to avoid pinging overnight. Adds a "Quiet hours"
window under Settings -> Notifications: notifications that would fire
during the window are held in a new notification_queue table instead
of sent immediately, then delivered as one combined digest at the end
time (in the same timezone already used for the daily checks) via a
new scheduled flush job. The scheduled daily checks (secret expiry,
Tailscale key, Docker updates, Proxmox backups) already only fire
once at a chosen time, so this mainly matters for the instant ones.
Includes a live "N queued" indicator with a manual "Flush now" button
for visibility, and correctly falls back to sending immediately
whenever the feature is disabled (the default).
Gating lives at the single choke point every notification already
flows through (notify()), so no per-event-type wiring was needed.
Verified against a fake webhook receiver on an isolated scratch
database: exhaustively checked the midnight-wraparound window math
(9 cases including exact-boundary inclusive/exclusive edges) against
synthetic "now" values rather than depending on when the test
happens to run, then end-to-end through the real notify()/
flushQuietHoursQueue() functions with a window constructed around the
actual current time — confirmed a notification during the window
queues instead of sending, the flush produces one digest with the
original title/message intact and clears the queue, a notification
outside the window sends immediately, and disabling the feature
entirely sends immediately regardless of the window. Confirmed the
real dev database's mtime was untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proxmox already runs vzdump backups, but nothing in the app said
whether they were actually succeeding — a silent backup failure is
one of the more dangerous blind spots a homelab admin can have. Adds
a "Backups" card to the Proxmox page: configured backup job
schedules (storage target, which guests, enabled/disabled) from
GET /cluster/backup, and recent vzdump task history per node from
GET /nodes/{node}/tasks?typefilter=vzdump, with a banner at the top
if the most recent run didn't succeed.
New "Proxmox backup failed" notification toggle under Settings ->
Notifications, on the same daily schedule as the other checks. The
scheduler checks each node's own most-recent vzdump run independently
(not just the single most recent task overall) so one node's healthy
backup can't mask another node's failing one in a multi-node cluster.
Known limitation, documented in the adapter's own header comment:
Proxmox's task list doesn't reliably expose which specific guest
failed within an "all guests" job — only the task's own log text has
that — so this surfaces job- and task-level status rather than
guessing at per-guest outcomes.
Verified against a fake Proxmox server (real self-signed HTTPS, since
the adapter's node:https usage can't be monkey-patched under ESM)
reproducing the documented /cluster/backup and task-list response
shapes: job parsing (all-guests+exclude vs specific-vmids+disabled)
correct, task OK/failure parsing correct, and the critical multi-node
scenario confirmed — one node's failing latest run flagged, the
other's healthy latest run correctly left alone, with exactly one
notification of the right content. This reproduces Proxmox's
documented API shape rather than a live-verified one; flag if the
real cluster's response differs in some way this didn't anticipate.
Confirmed the real dev database's mtime was untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dockhand's pending-update counts were only visible if you happened to
open the Docker page. New "Docker image update available" toggle
under Settings -> Notifications, sharing the same daily
time/timezone as the secret and Tailscale key expiry reminders (same
node-schedule reschedule-on-settings-change pattern as those two).
Reads each enabled Dockhand integration's already-cached update-check
results via listContainers() rather than triggering a fresh
per-container registry lookup, so it costs nothing extra beyond what
the Docker page itself already fetches, and lists every container
with an update pending across all environments/integrations in one
notification.
Verified end-to-end against a fake local Dockhand server (one
environment, two containers, one flagged with a pending update) and a
fake webhook receiver on an isolated scratch database: the check
correctly found only the flagged container (with its newerVersion),
sent exactly one notification with the right title/content, excluded
the up-to-date container, and sent nothing at all when the setting
was toggled off despite still finding the same pending update.
Confirmed the real dev database's mtime was untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Diagnostic Log already records every outbound call's success or
failure, but nothing acted on it — you'd only notice an integration
was down by happening to open its page. Adds a per-source consecutive-
failure counter (in-memory, reset on restart, same durability tier as
the diag log's own ring buffer) hooked into recordDiagEntry: crossing
the configurable threshold (default 3) sends one "down" notification
on every configured channel, and a "recovered" notification fires once
it succeeds again — no repeat spam while it stays down. New
"Integration/DNS provider failing repeatedly" toggle and threshold
field under Settings -> Notifications.
Verified end-to-end against an isolated scratch database with a real
local HTTP server standing in for the webhook channel: 5 consecutive
failures produced exactly one "Down" notification (at the 3rd
failure, correctly naming "3 calls"), a subsequent success produced
exactly one "Recovered" notification, and two more failures on a
fresh streak triggered nothing (below threshold) — confirmed the real
dev database's mtime was untouched throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Tailscale page already showed per-device key expiry; extend the
existing daily-reminder infrastructure (currently only for Secrets)
to push it out through the configured notification channels too, the
same way expiring secrets already are.
New tailscaleKeyExpiryScheduler.ts mirrors secretExpiryScheduler.ts:
runs once at startup (skipped if already run today) and daily
thereafter, checking every enabled Tailscale integration's devices for
keys expiring within the warning window and calling notify() with the
results. Reuses the exact same daily time/timezone setting as the
secret-expiry check (one "Daily reminder time" control, two
independent on/off toggles) rather than adding a second schedule for
users to configure.
Centralized the expiring-soon threshold and check (previously only
duplicated in the /synology and /tailscale route summaries) into
adapter.ts as `KEY_EXPIRY_WARN_DAYS` / `isKeyExpiringSoon()`, and
updated the devices route to use it instead of its own inline copy.
New `tailscaleKeyCheck` notification-event toggle (default on) in
settings, alongside the existing secret-expiry one.
Verified end-to-end against a temp SQLite DB + real migrations: a
tailscale integration pointed at a mock Tailscale API (one device
expiring in 10 days, one with key-expiry disabled) with the webhook
channel enabled and pointed at a mock receiver — confirmed the
scheduler's startup check queries the DB correctly, decrypts the
integration's credential, calls the adapter, filters out the
disabled-expiry device, and delivers a webhook payload naming only
the expiring device.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Settings page was a "coming soon" placeholder. Port Sloth Manager's
settings feature set: Gotify/ntfy/SMTP/webhook notification channels
(each with its own test-send button), per-event toggles (DNS record
added/updated/deleted, a daily secret-expiry digest with configurable
time/timezone), and per-provider DNS badge color customization.
Settings persist in the existing `settings` key/value table via a new
settingsStore service; a notify service fans a message out to every
enabled channel. DNS record add/update/delete now fire notifications,
and a node-schedule job re-arms itself whenever the notification
settings change. Removed the now-superseded GOTIFY_URL/GOTIFY_TOKEN
env vars in favor of in-app configuration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bobbanandClaude Sonnet 5