Commit Graph
31 Commits
Author SHA1 Message Date
bobbanandClaude Sonnet 5.5 4a7b1494c8 Rename the Monthly report menu entry to Reports
The page also gives quarterly and annual reports. The route is unchanged; the
scheduled monthly message in Notifications keeps its name.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-08 22:26:03 +01:00
bobbanandClaude Sonnet 5.5 31f4debb9a Add Telegram and Pushover as notification channels
Both send to one fixed address each (api.telegram.org, api.pushover.net), take part in routing rules and the channel count, have a Send test that shows the service's own refusal, and keep their tokens and keys encrypted and out of errors and the audit log. A Telegram message carries one button, and a Pushover message one link, to the page where an alert can be acknowledged, silenced or snoozed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-05 00:30:29 +01:00
bobbanandClaude Sonnet 5.5 ccb863e7c3 Move Notifications back to Settings and link to it from Alert settings
Alert settings keeps Alert rules, Service levels and Alert tickets; administrators get a link to Settings -> Notifications.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-03 20:49:09 +01:00
bobbanandClaude Sonnet 5.5 4269213275 Put alert rules, service levels, notifications and alert tickets on one Alert settings page, in four tabs
Old addresses (/alert-rules, /service-levels, /settings/notifications) lead to the right tab.
Notifications and Alert tickets stay administrator-only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-03 20:38:15 +01:00
bobbanandClaude Sonnet 5.5 78822346b1 Bulk close can close the osTicket tickets; Alerts top-level, Records group
Closing several alerts together can now also close their osTicket tickets
(one by one, each written on its incident and in the audit log; one that
fails does not stop the rest; Undo does not reopen tickets). The single
and bulk close share one helper. Alerts moves to the top of the sidebar,
and Monthly report and Change log move to a new Records group. Docs updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 20:20:28 +01:00
bobbanandClaude Sonnet 5.5 5ae61a1ece Close the osTicket ticket along with an alert
The Close drawer can also close the alert's osTicket ticket. osTicket's API
cannot, so this writes to its database with a separate, optional login: the
ticket is set to Closed, a note in its thread says who closed it and why, and
the closed event is recorded, in one transaction. Tables and columns are
checked first, only an open ticket is closed, and a ticket closed by somebody
else at the same moment is left alone. Summary tickets are not closed from here.

"Test closing" (Alert tickets settings) tries every write and rolls it back.
If the ticket cannot be closed the alert is still closed and the reason is
said, noted on the incident and audited.

Not run against a real osTicket: tested against a pretend database only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 21:27:38 +01:00
bobbanandClaude Sonnet 5.5 0fa50c2d5a Broken Docker containers are alerts
From Dockhand, on the health pass: a container that keeps restarting (critical, with
its exit code), one whose own health check fails, one that stopped unexpectedly, one
created and never started for an hour, and a Docker host Dockhand cannot reach. A stop
is only news when it was seen happen, so stacks switched off long ago stay quiet.
A Leave alone button on the Docker page and a list in Settings; on by default.

Also fixes a date-dependent storage-forecast test that pinned its clock while the code
under test reads the real one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 21:03:56 +02:00
bobbanandClaude Sonnet 5.5 16b24f5436 Away mode, and critical alerts through quiet hours
For a one-person homelab, "on call" is when you can be reached. Settings ->
Notifications -> When -> Away holds notifications while you are away (a banner on
every page says so) and sends one digest when it ends, at its time or when you say
you are back. Critical alerts can still come through. Quiet hours get the same
"still send critical alerts" option. Escalation steps marked to send in quiet hours
are sent while away too. What is held for one reason is not released while the
other still holds it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 19:14:29 +02:00
bobbanandClaude Sonnet 5.5 3b31b16796 Service-level targets with error budgets, and quarterly and annual reports
Monitoring -> Service levels: a target (99.9 %) on an endpoint check, a server or an
integration, measured over calendar months. Shows the figure, downtime used, the
error budget left, and how the last six months went. 80 % of the budget used is a
warning and an overspent budget a critical alert. Endpoints are measured day by day,
servers and integrations to the minute from the audit log.

The report page gains Quarter and Year (same figures, with a Service levels section),
and two settings send the quarterly and annual report after the month that ends them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 22:20:07 +02:00
bobbanandClaude Sonnet 5.5 31044b2b64 What is normal for the time: baselines per hour of the week
Each server's CPU load and memory use get a baseline for every hour of the week
(median and robust spread over up to ten weeks, four needed). A clearly high last
hour that never dropped back toward normal is a warning; everything else stays
quiet. Server History -> Trends draws a usual line and band. Setting under
Notifications, on by default.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 21:57:57 +02:00
bobbanandClaude Sonnet 5.5 aebf3c6ce1 Buttons on notifications: acknowledge, silence and snooze from the message
- A switch (off by default) under Settings -> Notifications -> Who hears what
- ntfy gets buttons (three for one alert, one link for several); phone push gets two buttons handled by the service worker; Gotify, email and the chat webhook get the link as text; escalation messages carry them too
- Each link is a signed token for the alerts in that one message only, good for 24 hours, for three actions; opening it changes nothing, only a POST acts; everything is audited as "from a notification link"
- Docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 20:42:04 +02:00
bobbanandClaude Sonnet 5.5 c2e76262ca Documentation: guides in docs/, and the reference docs brought up to date
- New docs/ folder: SSO, API, Prometheus and Grafana, webhooks, heartbeats and calendar, how monitoring works, and an index; the old root API.md is split across them
- Read API: alerts carry snoozedUntil
- DATABASE.md: table and migration counts, indexes, relations, integration config, settings and internal flags, delete rules, retention
- ROLES.md: the sidebar table rebuilt from the menu
- NOTIFICATIONS.md, INTEGRATIONS.md, BACKUP.md, AGENT_UPDATE.md: what had drifted
- README: links to the guides; accounts cannot be deleted in the app

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 00:09:43 +02:00
bobbanandClaude Sonnet 5.5 24ae8db380 Alert rules of your own, storage forecasts for Proxmox and Synology, and snooze
- Monitoring -> Alert rules: CPU, memory, disk use and disk temperature limits with a time, severity and scope (all, one server, a tag); try before saving; findings are ordinary health conditions
- Proxmox storages and Synology volumes are written down on each health pass (hourly, 400 days) and forecast like server disks: badge and history chart on the pages, alert, digest line, /metrics gauge
- Snooze an alert until a time; held back like maintenance, messages held, announced once again if still there when it ends
- Docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 23:38:36 +02:00
bobbanandClaude Sonnet 5.5 396e074108 UniFi integration: network devices, Wi-Fi and networks, as part of the dependency map
A UniFi controller (API key on Network 9+, or a read-only local login on any
version) is read every five minutes. Its switches, access points and gateways
are kept with state, IP, uptime, firmware and uplink; its networks and Wi-Fi
names beside them (passphrases are never read). Nothing is written to the
controller.

Network > Network devices lists them. Devices are a new node type in the
dependency map: uplinks become dependencies by themselves, others can be
written by hand. A device offline past a limit is an offline alert, folds what
depends on it, can be silenced with maintenance and have runbooks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 00:35:48 +02:00
bobbanandClaude Sonnet 5.5 73a06ebe63 Flapping alerts, a noise report and bulk actions with undo
Flapping: an alert raised three times within an hour is announced once as
flapping, its further raise/clear messages are held until it has been calm
for an hour, and one message then says whether it is still active. Event
webhooks still hear everything. The state survives restarts.

Alerts > Noise lists the alerts that keep coming back (7-180 days): how often,
how long, most in an hour, and a hint on what helps.

Bulk actions on Alerts (acknowledge, remove acknowledgement, silence), Servers
(tag, silence, runs on) and Inventory (location and dates, print labels). Each
reports what it left alone and why, and has an Undo on the toast.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 23:55:09 +02:00
bobbanandClaude Sonnet 5.5 be6c5552bd Patch plans, a monthly report and new-port alerts
Patch plans (Updates -> Patch plans): updates on a schedule, one server at a time, each by an
ordinary update run, stopping at the first failure by default, silencing what depends on a
server while it is updated, with one message at the end.

Monthly report (Operations -> Monthly report, and optionally a message on a day of the month):
availability, incidents, updates, restore tests, backups and renewals for a calendar month,
printable. Backed by daily endpoint figures and a record of renewals.

New ports: the first report with ports is the baseline; a TCP port that begins listening later and
is not noted on the server's Ports card is a warning, cleared by Mark as expected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 23:16:14 +02:00
bobbanandClaude Sonnet 5.5 ddd0aa622a Settings down to eight tabs; Domains become a tab of DNS
Notifications gains Alert tickets, Display gains Badge colours, and a new Data tab holds Backup,
Logs and Cache. Domains is the 'Domain names' tab of DNS. Old addresses redirect; the docs, the
Privacy page and the links from alerts and the agenda name the new places.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 22:36:24 +02:00
bobbanandClaude Sonnet 5.5 1432b148e9 Fold four pages into the pages they belong to; add a Backups menu group
Dependencies become a view of the Map (graph with the editor under it), Sessions the
'Signed in now' tab of Users, the Audit Log the 'Raw audit log' tab of the Change log, and the
Diagnostic Log the 'Call log' tab of Integrations (admins). Backup coverage and Proxmox Backup
move to a new Backups group. Old addresses redirect; a shared ViewTabs component does the tabs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 22:21:21 +02:00
bobbanandClaude Sonnet 5.5 4e0c6927b2 Heartbeats, Coming up and suggested checks; Operations menu tidied
Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app
alerts when the call is late or the job says it failed. Coming up: one agenda of everything
dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests,
waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested
checks: web ports and certificates the app already knows about, added with a click.

Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group
(Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under
Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow
the real clock.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 21:42:34 +02:00
bobbanandClaude Sonnet 5.5 92726568b7 Endpoint checks, restore tests, and a change log
- Endpoints: HTTP, TCP and TLS checks with answer time, uptime and certificate expiry; failures and expiring certificates become alerts and notifications; linked to a server for maintenance and tags
- Restore tests: record when a backup was restored and how it went; failed tests always alert, overdue ones with a limit set; shown on the Backup coverage page
- Change log: one filterable timeline made from the audit log and the incident record

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 23:03:56 +02:00
bobbanandClaude Sonnet 5.5 43fc5e0272 Notification routing rules, per-browser push wishes, escalation of unacknowledged alerts
- Routing: rules pick channels by kind, severity and server tags (first match wins, none = nobody); mixed batches split per route; recoveries follow their alert; quiet-hours digest keeps each notification's audience
- Push: each subscribed browser can ask for a minimum severity and some kinds
- Escalation: up to three timed steps re-notify alerts nobody has acknowledged, through chosen channels and an optional extra email; an acknowledgement stops them
- Fix ntfy: titles with an em dash made every ntfy send fail (header not Latin-1); now RFC 2047 encoded

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 19:44:45 +02:00
bobbanandClaude Sonnet 5.5 ec9c2f73f0 Dependencies beyond servers, web push, inventory auto-fill with QR labels, Windows agent health
- Dependencies: servers and integrations can depend on each other; alerts are folded under the root cause, and a failure explained by something down isn't announced
- Web push: hand-written VAPID and RFC 8291 encryption, per-device subscriptions, a fifth notification channel, service worker handlers
- Inventory: maker/model/serial from the agents fill blank fields, "Refresh from agent", QR label printing and ?asset= deep links
- Windows agent (v4) reports failed services, disk health, Storage Spaces pools and hardware identity; Linux agent (v5) reports hardware identity

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 18:53:30 +02:00
bobbanandClaude Sonnet 5.5 2744a84719 Acknowledge and hand over alerts, incident timelines, and built-in database backups
- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what
  needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics
- Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents,
  incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs,
  related alerts and notes; Incidents page and a digest section
- Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download,
  check, upload, delete, and restore staged and applied at start-up before the database opens
  (pre-restore copy kept); alerts and /metrics for stale or failing backups
- Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 02:21:08 +02:00
bobbanandClaude Sonnet 5.5 7a5b623444 Stale-backup alerts, per-server alert limits, and update runs from the Updates page
- Backups: a guest whose newest backup is older than its limit is an alert (backup kind),
  in the daily backup message, the digest and /metrics (homelab_backup_guests, ..._last_age_seconds)
- Per-server thresholds (servers.thresholds, migration 0030): offline minutes, disk percent,
  forecast window, backup age, and per-disk limits or "leave alone"; applied everywhere the
  global ones were; saving clears the alerts cache
- Update runs (update_runs, migration 0031): Update... on the Updates page puts the server under
  maintenance, runs a Semaphore playbook limited to it, waits for the host to report, and
  reports what is left and whether a restart is due; poller survives restarts
- Notification toggle for update-run results; tests and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 01:25:09 +02:00
bobbanandClaude Sonnet 5.5 5dc6cea69c Host health from the Linux agent, agent self-update, and disk forecast in /metrics
- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates;
  SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox
- Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new
  "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series
- Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified
  download, atomic replace; GET /api/agent/report/update
- homelab_server_disk_days_to_full in /metrics
- Tests (server, real PowerShell against a fake server) and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 00:54:24 +02:00
bobbanandClaude Sonnet 5.5 c17e9aa140 Add runbooks, backup coverage, inventory, roles from groups, Prometheus metrics and event webhooks
- Runbooks: notes per server/integration, shown next to matching alerts
- Backup coverage page across Proxmox and PBS, with a digest section
- Hardware inventory with warranty, power and running cost
- Authentik group to role mapping with preview and safeguards
- /metrics (bearer token) and signed outgoing event webhooks
- Docs: README, DATABASE, NOTIFICATIONS, ROLES and new API.md

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 20:41:00 +02:00
bobbanandClaude Sonnet 5.5 c36e2092c5 Split the notification settings into Channels, What to send and When tabs
- Channels are compact rows (on/off, status, one-line summary) that open to edit and test.
- Event toggles are grouped by topic with their thresholds beside them.
- Time zone, daily reminder, quiet hours and the weekly digest sit together under When.
- A sticky save bar with an unsaved-changes marker; the tab is kept in the address.
- The time zone is always editable, and a channel can be filled in before it is switched on.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 20:01:54 +02:00
bobbanandClaude Sonnet 5.5 aa94d84f6c Add a weekly digest and report which version of the agent each server runs
- Weekly digest through the notification channels: alerts now, a week of uptime, what happened, disks
  heading for full, expiring secrets and domains, updates waiting, planned maintenance. Off until switched
  on in Settings -> Notifications; Preview and Send it now there. Setting key 'digest'.
- Agents say their version in every report (Linux and Windows scripts are version 2); the server compares
  it with the scripts it serves and marks agents that are behind on the Servers, server and Updates pages.
  servers.agent_version, migration 0026.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 19:49:36 +02:00
bobbanandClaude Sonnet 5.5 aec553f122 Document alert tickets
README, INTEGRATIONS (the osTicket ticket API: fields, making the key, what is sent, what it can't do,
what wasn't verified), NOTIFICATIONS (the rules, in one table), ROLES, DATABASE (the new table, the
settings key, the internal flags, what deleting an integration does), and the Privacy page (the one new
thing that leaves the app: an alert's text, to your osTicket).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07 23:41:47 +02:00
bobbanandClaude Sonnet 5.5 447f33fff6 Add an Alerts page under Operations listing everything that's wrong now
One list of the current problems across servers and integrations, instead
of waiting for a notification or visiting each page: servers that stopped
reporting, full or nearly full disks and volumes (critical from 95%),
Synology volume/disk problems, failed or uncovered Proxmox backups,
failed Proxmox Backup Server verifications, container image updates,
expired or expiring secrets/domains/Tailscale keys, failed Semaphore and
Gitea runs, Uptime Kuma monitors that are down, overdue osTicket tickets,
and integrations whose calls keep failing. Visible to every role, with
severity and kind filters, search, sorting, CSV export and "Check now".

It runs the same checks that send the notifications rather than a second
copy of them: the detection in the health, automation, Proxmox backup,
PBS, Docker update and Tailscale key checks is pulled out into shared
collectors that both the schedulers and the page call, so the two can't
disagree about what counts as a problem. Notification behaviour is
unchanged, including the scheduled backup checks skipping integrations
under a maintenance window. Unlike the notifications the page ignores the
on/off toggles, and keeps problems under a maintenance window, marked
silenced and counted apart.

It reads live, so a result is reused for a minute (and Refresh can't
re-run everything more than once every ten seconds), and every source has
a 20 s limit so one hung integration can't hang the page. Anything it
couldn't read is called out at the top instead of looking like all clear,
and server checks pause for the same 20 minutes after a restart as the
notifications do, with a note saying so.

Also gives the newer integrations (PBS, osTicket, Uptime Kuma, phpIPAM)
proper names in "integration down" notifications instead of their ids.

Verified through the real routes against a scratch database with fake
backends (offline and full-disk servers, secrets and domains, a silenced
server, a fake PBS with failed verification, a hanging integration, a
refused one, a failing-calls streak, caching, the restart grace period,
auth), and by rendering the real page against that data in a browser:
filters, search, silenced toggle, sorting, Check now, dark mode.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 23:07:57 +02:00
bobbanandClaude Sonnet 5 4e48377348 Document every notification this app sends in NOTIFICATIONS.md
Covers all 16 notification types (daily reminders, state-based health/
automation alerts, real-time DNS and integration-down alerts, and the
quiet-hours digest), what triggers each, and what does and doesn't
respect quiet hours and maintenance windows. Every trigger condition
and threshold was cross-checked against the actual scheduler/monitor
code, not just the settings labels.

Also fixes the "Daily reminder time" hint text in NotificationSettings,
which had gone stale — it listed only 4 of the 6 checks that actually
share that schedule (missing domain expiry and PBS verification).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-29 23:48:19 +02:00