Commit Graph
27 Commits
Author SHA1 Message Date
bobbanandClaude Sonnet 5.5 93aaf5476d Lidarr/Readarr, integration inspector, app versions, grouped alerts
Lidarr and Readarr join the shared *arr reader. Integrations get an Inspect button that
runs the reader once and shows what each request returned as shapes (never values) and
where the result is empty. Operations -> App versions compares each application's version
with its newest GitHub release (lookup off until an admin switches it on; optional alert).
The Alerts page can group by source, kind of problem or type of alert. The shared HTTP
helper no longer reuses connections (stale-socket resets). Migration 0078; docs and tests
included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 20:07:36 +01:00
bobbanandClaude Sonnet 5.5 551eb1597a AdGuard Home, Grafana, Docker image age, foldable server page cards
New integration kinds adguard (status, statistics, filter lists and a real DNS question
over UDP; alerts for DNS down/not running, protection left off, stale lists) and grafana
(alert rules firing or failing, data-source health). Optional warning for running Docker
images built more than N days ago (off by default). A server's overview cards can be
folded and arranged, kept per user. Migrations 0075-0077; docs, metrics and tests included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 22:13:03 +01:00
bobbanandClaude Sonnet 5.5 3d9a2310bb Runbooks per alert kind, high-vulnerability threshold, wall display extras
A "What to do" note (text and/or link) can be written once for a whole kind of alert and
shows beside every alert of that kind; alerts now carry their kind and source type.
New optional setting: warn when a Docker host has N or more high vulnerabilities (off by
default). The wall display gains tiles (Hetzner and game servers running, critical
vulnerabilities, stuck downloads) and a summary line for each group of integrations,
also in share links. Migration 0074; docs and tests included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 21:15:45 +01:00
bobbanandClaude Sonnet 5.5 d9083adb5b Sonarr/Radarr/Prowlarr, Hetzner load balancers/floating IPs/certificates, traffic forecast
New integration kinds sonarr, radarr and prowlarr (shared adapter and sync, page under
Apps -> Media managers): own health findings, stuck downloads, nearly full disks.
Hetzner now reads load balancers (target health), floating IPs and certificates, includes
them in the cost estimate, and forecasts the month's traffic against the quota.
Migrations 0072 and 0073; docs, metrics and tests included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 20:04:55 +01:00
bobbanandClaude Sonnet 5.5 9a3d5c466a Fragnet game servers: games, addresses, backups, power actions
New integration kind "fragnet" reading Fragnet's Pterodactyl-based panel through the
client API (account key). Game servers page under Infrastructure with game, IP:port,
state, memory/disk, backups, schedules and databases; alerts for late backups and
nearly full disk/memory; operators can start/stop/restart/kill (audited). Metrics,
docs and tests included; migration 0071 adds fragnet_state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 19:01:36 +01:00
bobbanandClaude Sonnet 5.5 f86661f6c9 Speedtest Tracker, Costs, "Having trouble" on status pages, grouped Integrations and Dashboard
- Speedtest Tracker integration: charts, alerts for a slow line (median of the
  last three tests under a share of what is paid for) and for failing tests.
- Costs page (Records): things paid for by hand plus Hetzner's estimate, totals
  per currency, bills rolled forward into Coming up as Renewals.
- Status pages: a "Having trouble" state for things that answer but have an open
  alert; Speedtest reads as the internet.
- Integrations page and kind picker grouped like the sidebar (one shared types
  file); Notifications "What to send" shown one group at a time; Dashboard
  integration sections follow the menu groups, with tiles for integrations that
  have no widget.
- Docs, tests, and ROLES.md menu table brought up to date.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 23:23:46 +01:00
bobbanandClaude Sonnet 5.5 24b708abbe Hetzner Cloud integration: servers, traffic quota, backups, volumes and cost
One integration per project (read-only API token, read every 5 minutes).
Alerts for a server that was running and is off, stuck statuses, traffic
at 80%/100% of the monthly quota, rescue mode left on, and late backups.
The Hetzner page lists servers, volumes, snapshots and a cost estimate; a
server the app knows by its agent gets a Hetzner card on its own page.
/metrics and docs updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 22:51:17 +01:00
bobbanandClaude Sonnet 5.5 5a2b320175 Dockhand vulnerability scans: warn about critical findings, with a setting to turn it off
Reads what Dockhand's scans have already stored (never starts one) every
15 minutes and raises a warning per Docker host with critical findings.
Unknown counts and stale readings are held, not treated as clean.
Settings -> Notifications has a switch (on by default); the Docker page
shows counts, scan coverage and the critical findings; /metrics and docs
updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 22:34:06 +01:00
bobbanandClaude Sonnet 5.5 65214f6f0f Nextcloud cron and Nextcloud's own checks; bulk close and open again
Nextcloud: its own "Cron last run" setup check (GET /settings/setupcheck,
Nextcloud 28+, found by class name) raises an alert when background jobs have
stopped; other checks that report an error are one alert, warnings are shown
on the page. The core version Nextcloud says is available is shown too. Before
28, cron and the checks are unknown, never "fine".

Alerts page: bulk Close... (one reason, written on each incident, undoable) and
Open again. Bulk Wake now leaves closed alerts alone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 22:12:10 +01:00
bobbanandClaude Sonnet 5.5 4ff0b44943 Nextcloud, NetBox and n8n integrations
Three read-only integrations, each with a monitoring page, alerts, maintenance
silencing and held-when-unreadable handling.

Nextcloud: database needing upgrade (critical), maintenance mode, and the data
disk running low; users, files and app updates with an admin app password.
NetBox: no background worker running, counts, and a list of the servers this
app manages that have no address in NetBox (a list, not an alert).
n8n: an active workflow whose last run failed, clearing when it next succeeds.

Disk messages for Immich, Paperless and Nextcloud now use the same units as
the pages that show them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 23:13:13 +02:00
bobbanandClaude Sonnet 5.5 251a4a9d73 Prometheus, Immich and Paperless-ngx integrations
Three read-only integrations, each with a monitoring page, alerts, maintenance
silencing and held-when-unreadable handling.

Prometheus: scrape targets that stop answering (5 minutes, one alert per job)
and alerts that Prometheus's own rules fire (critical when labelled so).
Immich: disk under the library filling, failed background jobs, counts.
Paperless-ngx: worker/database/Redis in error, documents that failed to
import, index and sanity check, disk. Immich and Paperless also in /metrics.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 22:33:17 +02:00
bobbanandClaude Sonnet 5.5 aebe98bf15 Authentik: know when the front door is about to shut
New Authentik integration (read-only, every 5 minutes): version, workers,
outposts and their health, certificates and API tokens that expire, failed
sign-ins and configuration errors. Monitoring -> Authentik page.

Alerts: an outpost that serves apps has gone quiet (critical), certificate or
token about to run out (critical close to it), no worker, failed sign-ins,
configuration errors. Held by maintenance, held when unreadable, in /metrics.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 22:09:38 +02:00
bobbanandClaude Sonnet 5.5 0146eb8bca Jellyfin: who is watching, and don't update a machine mid-film
New Jellyfin integration (read-only, every 2 minutes): version, restart wanted,
streams and transcoding, failed scheduled tasks, item counts, failed sign-ins.
Monitoring -> Jellyfin page with a peak-streams history, three alerts, /metrics.

When Jellyfin runs on a managed server, an update run asks first if people are
streaming (409 "watching" unless startAnyway), and a patch plan waits up to two
hours for them before updating that server.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 22:01:18 +02:00
bobbanandClaude Sonnet 5.5 2f082ca5c2 Duplicati: know when a backup failed or has gone quiet
Duplicati reports each operation to a secret address (its own send-http-url option), so
nothing calls Duplicati and no password for it is kept. Each machine is a source; each job
is learned by name and shows its last result, duration, size, changes and a history.
A failed run is a warning (critical from the second in a row); one that has not succeeded
for longer than expected (per job, 2 days by default) is a warning, critical at three
times that. Maintenance on the machine's server silences them; /metrics series added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 21:36:13 +02:00
bobbanandClaude Sonnet 5.5 0fa50c2d5a Broken Docker containers are alerts
From Dockhand, on the health pass: a container that keeps restarting (critical, with
its exit code), one whose own health check fails, one that stopped unexpectedly, one
created and never started for an hour, and a Docker host Dockhand cannot reach. A stop
is only news when it was seen happen, so stacks switched off long ago stay quiet.
A Leave alone button on the Docker page and a list in Settings; on by default.

Also fixes a date-dependent storage-forecast test that pinned its clock while the code
under test reads the real one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 21:03:56 +02:00
bobbanandClaude Sonnet 5.5 31044b2b64 What is normal for the time: baselines per hour of the week
Each server's CPU load and memory use get a baseline for every hour of the week
(median and robust spread over up to ten weeks, four needed). A clearly high last
hour that never dropped back toward normal is a warning; everything else stays
quiet. Server History -> Trends draws a usual line and band. Setting under
Notifications, on by default.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 21:57:57 +02:00
bobbanandClaude Sonnet 5.5 232a5661ab A UPS in the map: read over NUT or SNMP, alerts, shutdown order
New UPS integration (NUT or SNMP v2c, read only). Monitoring -> UPS shows charge,
runtime, load and mains voltage with history and the times mains power was lost.
Mains lost and a nearly empty battery are critical alerts; battery to be replaced
and a heavy load are warnings. The UPS is a node in the dependency map and counts
as down while on battery, so servers that go offline with it are folded under its
alert; the page lists what is on it in the order to shut it down. /metrics series
and docs added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 21:47:31 +02:00
bobbanandClaude Sonnet 5.5 24ae8db380 Alert rules of your own, storage forecasts for Proxmox and Synology, and snooze
- Monitoring -> Alert rules: CPU, memory, disk use and disk temperature limits with a time, severity and scope (all, one server, a tag); try before saving; findings are ordinary health conditions
- Proxmox storages and Synology volumes are written down on each health pass (hourly, 400 days) and forecast like server disks: badge and history chart on the pages, alert, digest line, /metrics gauge
- Snooze an alert until a time; held back like maintenance, messages held, announced once again if still there when it ends
- Docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 23:38:36 +02:00
bobbanandClaude Sonnet 5.5 396e074108 UniFi integration: network devices, Wi-Fi and networks, as part of the dependency map
A UniFi controller (API key on Network 9+, or a read-only local login on any
version) is read every five minutes. Its switches, access points and gateways
are kept with state, IP, uptime, firmware and uplink; its networks and Wi-Fi
names beside them (passphrases are never read). Nothing is written to the
controller.

Network > Network devices lists them. Devices are a new node type in the
dependency map: uplinks become dependencies by themselves, others can be
written by hand. A device offline past a limit is an offline alert, folds what
depends on it, can be silenced with maintenance and have runbooks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 00:35:48 +02:00
bobbanandClaude Sonnet 5.5 be6c5552bd Patch plans, a monthly report and new-port alerts
Patch plans (Updates -> Patch plans): updates on a schedule, one server at a time, each by an
ordinary update run, stopping at the first failure by default, silencing what depends on a
server while it is updated, with one message at the end.

Monthly report (Operations -> Monthly report, and optionally a message on a day of the month):
availability, incidents, updates, restore tests, backups and renewals for a calendar month,
printable. Backed by daily endpoint figures and a record of renewals.

New ports: the first report with ports is the baseline; a TCP port that begins listening later and
is not noted on the server's Ports card is a warning, cleared by Mark as expected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 23:16:14 +02:00
bobbanandClaude Sonnet 5.5 4e0c6927b2 Heartbeats, Coming up and suggested checks; Operations menu tidied
Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app
alerts when the call is late or the job says it failed. Coming up: one agenda of everything
dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests,
waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested
checks: web ports and certificates the app already knows about, added with a click.

Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group
(Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under
Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow
the real clock.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 21:42:34 +02:00
bobbanandClaude Sonnet 5.5 92726568b7 Endpoint checks, restore tests, and a change log
- Endpoints: HTTP, TCP and TLS checks with answer time, uptime and certificate expiry; failures and expiring certificates become alerts and notifications; linked to a server for maintenance and tags
- Restore tests: record when a backup was restored and how it went; failed tests always alert, overdue ones with a limit set; shown on the Backup coverage page
- Change log: one filterable timeline made from the audit log and the incident record

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 23:03:56 +02:00
bobbanandClaude Sonnet 5.5 5dc6cea69c Host health from the Linux agent, agent self-update, and disk forecast in /metrics
- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates;
  SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox
- Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new
  "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series
- Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified
  download, atomic replace; GET /api/agent/report/update
- homelab_server_disk_days_to_full in /metrics
- Tests (server, real PowerShell against a fake server) and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 00:54:24 +02:00
bobbanandClaude Sonnet 5.5 9e483f2163 Mirror Uptime Kuma's maintenance schedule as maintenance windows
Uptime Kuma shares its maintenance windows only over its dashboard Socket.IO
connection, which takes a login. With a login username and password on the
integration, "Sync schedule" on the Maintenance page (and a 15-minute
background pass when switched on) reads each window's running and next time
slot and the monitors it covers, and makes a maintenance window on every
server those monitors match. Windows that haven't started are listed as
Scheduled and silence nothing until they begin; ones ended or cancelled by
hand stay ended, and ones removed in Kuma are removed here.

maintenance_windows gets a source column (migration 0017) and active windows
now require their start to have passed. Adds socket.io-client, and tests
against a stand-in Uptime Kuma speaking the same protocol.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-11 23:43:47 +02:00
bobbanandClaude Sonnet 5 bf7f73f6b6 Import maintenance windows from Uptime Kuma
New "Import from Uptime Kuma" action on the Maintenance page (operator):
pick an Uptime Kuma integration and a duration, and it starts (or
extends) a maintenance window here for every server whose address
matches a monitor Uptime Kuma currently reports as being in maintenance,
reusing the existing monitor-to-server matching from the Uptime Kuma
integration itself.

Uptime Kuma's metrics endpoint only exposes a monitor's *current* status,
not its scheduled start/end time (there's no API for that), so this
deliberately doesn't try to mirror Uptime Kuma's own schedule -- it starts
a window for the duration you choose, the same bounded/required-end
window this feature has always used. Running it again while Kuma is
still in maintenance extends the same window rather than stacking a
second one; when Kuma later shows nothing in maintenance, already-active
windows are left alone rather than force-ended, since ending them isn't
something only Uptime Kuma's state should decide. Two Kuma monitors that
match the same server are deduped to one window. Monitors with no
matching server are reported back by name so nothing is silently missed,
and monitors that aren't in maintenance are ignored entirely.

The manual "Start maintenance" endpoint's start-or-extend logic (dedupe,
pruning old rows, the response shape) is now a shared
services/maintenance.ts function instead of living only in that route
handler, so the import path can't drift from how a manual window behaves.
Likewise the server-matching helper gained a small toMatchableServers()
so the existing Uptime Kuma monitors route and this new one build the
same match input the same way instead of each parsing server rows on
their own.

No schema change -- imported windows are ordinary maintenance windows;
their Uptime-Kuma origin is only in the reason text ("Imported from
Uptime Kuma (<integration>): <monitor>"), visible in the Active table and
the audit log like any other window.

Verified with 28 backend checks (the refactored manual start/extend
flow as a regression check; roles; unknown/wrong-type/disabled
integration; validation; nothing-in-maintenance; matching including
same-server dedup and unmatched monitors; re-running extends rather than
duplicating; windows left alone once Kuma exits maintenance; upstream
failure; audit entries) and by driving the real Maintenance page against
the real routers in a browser: import, re-import (extends), the
nothing-in-maintenance state, and the viewer view (no edit card, active
windows still visible). Real dev database mtime untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-29 19:11:35 +02:00
bobbanandClaude Sonnet 5 4118062405 Alert when a Semaphore template or Gitea workflow run fails
Every 15 minutes (on the existing health-check timer) the app reads the
latest run of each Semaphore template and each Gitea repo's latest
workflow run. A failed one raises one notification, and another when a
later run succeeds. It is state-based like the server health alerts, so a
job that fails every night alerts on the first failure, not every night.
Gitea alerts include the run's link. Toggle: Settings > Notifications.

What counts:
- Semaphore "error" is a failure, "success" is a pass. A run that is
  waiting, running, stopped by hand or rejected is neither, so it leaves
  the previous state alone: a run in progress must not clear a failure it
  hasn't fixed yet, and a manual stop isn't a failure.
- Gitea failure/success likewise; running, waiting, blocked, cancelled and
  skipped leave things as they were.
- A failing template that gets another failing run does not re-alert.

Not mistaking "couldn't read" for "fixed":
- Semaphore's template listing swallowed per-project errors, so a project
  that failed to load looked like a project with no templates. A new
  checkTemplates adapter method reports which projects failed, and their
  failures are held rather than cleared.
- Gitea reports a run it couldn't fetch as null, the same as "no runs";
  both leave the repo's state alone.
- An unreachable integration holds all of its failures. Nothing is cleared
  or re-announced while it is down.

The first pass only records what is already failing without announcing
it, so upgrading (or adding an integration to a fresh install) doesn't
produce a wall of alerts about months-old failures. That baseline is not
spent while nothing could be read.

Maintenance windows on a Semaphore or Gitea integration silence its
failure alerts with the same rules as the health alerts: a problem that
starts during a window alerts when it ends, and one already announced
stays known. The diff logic is reused from the health monitor rather than
copied. Maintenance page text updated.

Known limit: for Gitea this follows the repo's most recent run on any
workflow or branch, matching what the Gitea page shows; a failure in one
workflow can be masked by a later success of another.

Verified with 53 checks against fake Semaphore and Gitea servers and a
webhook receiver: classification, baseline (including not being consumed
when nothing is readable), single alert per failure, no repeat, in-progress/
stopped/cancelled runs, recovery and re-failure, unreadable project,
unreadable integration, run-fetch errors, maintenance windows (silenced,
then announced after), the toggle, disabled integrations and repos without
Actions. Real dev database mtime untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-26 03:11:59 +02:00
bobbanandClaude Sonnet 5 aae4f0d74f Add maintenance mode to silence alerts while working on a server or integration
Rebooting Proxmox or patching a server triggered failure/offline alerts
you then had to dismiss. A maintenance window silences alerts about one
server, integration, or DNS provider for a chosen time. New Maintenance
page (start with a duration and optional reason, end early, see what's
silenced and what isn't) and a banner in the app shell so every signed-in
user can see what is currently silenced. Starting/ending is operator-only
and audit-logged; starting one on a target that already has a window
restarts its clock instead of stacking.

Silenced for the target: server offline/disk alerts, Proxmox/Synology
storage and health alerts, Proxmox backup alerts, and "integration down"
alerts. Not silenced: expiry and update reminders, DNS change notices.

The design goal is that this cannot hide a real outage:
- Every window has a required end (5 min to 7 days); there is no
  open-ended option, so a forgotten window expires by itself.
- A silenced problem is deliberately NOT recorded as "known". If it is
  still present when the window ends it alerts then, as new. A problem
  that was already alerted before the window stays known, so it isn't
  repeated, and is reported cleared only after the window ends.
- Failure alerts keep counting failures during a window without marking
  themselves alerted, so an outage that outlasts the window alerts on the
  very next failed call.

Known limitation, stated on the page: integration-failure alerts are
tracked per service TYPE (all "proxmox"), not per configured instance, so
a window on one Proxmox integration also silences a failure on a second
Proxmox integration while it's open. Fixing that means threading the
integration id through every adapter and the diagnostic log, which is a
much larger change than this feature.

Also moved the API-error-message helper out of Secrets.tsx into a shared
util now that two pages use it. New table maintenance_windows (migration
0008).

Verified with 44 checks: the condition-key-to-subject mapping (including
server:3 vs server:33), the diff rules with silenced subjects (new problem
not recorded, alerts when the window ends; already-known one carried and
not repeated; clears only after the window), window expiry and
integration/DNS-provider source matching, the failure tracker end to end
against a webhook (silent during a window while an unrelated service still
alerts; outage that outlasts the window alerts on the next failure and
only once; fail-and-recover fully inside a window sends nothing), a full
health pass against a real window, and the real router with a stubbed
session (role rules, duration bounds including the missing-duration case,
extend-not-stack, 404s, deleted targets hidden, audit entries). Real dev
database mtime untouched.

Not done: I haven't clicked through the new page or banner in a browser
(they sit behind the Authentik login); it builds and the API behind it is
tested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-26 02:23:06 +02:00