Commit Graph
5 Commits
Author SHA1 Message Date
bobbanandClaude Sonnet 5.5 24ae8db380 Alert rules of your own, storage forecasts for Proxmox and Synology, and snooze
- Monitoring -> Alert rules: CPU, memory, disk use and disk temperature limits with a time, severity and scope (all, one server, a tag); try before saving; findings are ordinary health conditions
- Proxmox storages and Synology volumes are written down on each health pass (hourly, 400 days) and forecast like server disks: badge and history chart on the pages, alert, digest line, /metrics gauge
- Snooze an alert until a time; held back like maintenance, messages held, announced once again if still there when it ends
- Docs updated

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 23:38:36 +02:00
bobbanandClaude Sonnet 5.5 73a06ebe63 Flapping alerts, a noise report and bulk actions with undo
Flapping: an alert raised three times within an hour is announced once as
flapping, its further raise/clear messages are held until it has been calm
for an hour, and one message then says whether it is still active. Event
webhooks still hear everything. The state survives restarts.

Alerts > Noise lists the alerts that keep coming back (7-180 days): how often,
how long, most in an hour, and a hint on what helps.

Bulk actions on Alerts (acknowledge, remove acknowledgement, silence), Servers
(tag, silence, runs on) and Inventory (location and dates, print labels). Each
reports what it left alone and why, and has an Undo on the toast.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 23:55:09 +02:00
bobbanandClaude Sonnet 5.5 2744a84719 Acknowledge and hand over alerts, incident timelines, and built-in database backups
- Alerts can be acknowledged (note, assignee, duration); acknowledged alerts are left out of what
  needs attention (sidebar, dashboard strip, cards) but stay raised and counted by /metrics
- Incidents: each time an alert is raised and goes away is recorded (alert_acks, alert_incidents,
  incident_notes, migration 0032), with a timeline of acks, maintenance, changes, update runs,
  related alerts and notes; Incidents page and a digest section
- Backups of Homelab Manager's own database: daily checked VACUUM INTO copies, retention, download,
  check, upload, delete, and restore staged and applied at start-up before the database opens
  (pre-restore copy kept); alerts and /metrics for stale or failing backups
- Docs: README, BACKUP, DATABASE, NOTIFICATIONS, ROLES, API; tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-18 02:21:08 +02:00
bobbanandClaude Sonnet 5.5 467d1a1a06 Open an osTicket ticket for an alert that lasts
With Settings -> Alert tickets switched on (off by default), every 15 minutes the problems on the
Alerts page are looked at and one that is new and has lasted gets one ticket in osTicket, through
its API (POST api/tickets.json, key in X-API-Key). One problem opens one ticket however many checks
see it; the app remembers what it has seen in a new alert_tickets table (migration 0015).

- Which: critical and warning by default, every kind; severities and kinds are settings.
- A problem must be seen in 2 checks in a row first (1-4, a setting), so a blip that fixes itself,
  like one failed call to an integration, never opens a ticket.
- One ticket per occurrence; a problem that clears and returns within 60 minutes is the same one.
- More than 5 new problems in one check become a single summary ticket.
- Never for alerts under a maintenance window, nor about osTicket itself or its overdue tickets.
- Problems already there when it is switched on are left alone by default (the first check records
  them as a baseline); can be set to ticket them too.
- If osTicket can't be reached nothing is recorded, so the next check retries; if a source couldn't
  be read, a problem isn't treated as gone. Every ticket is audited.

The ticket API address, key (stored encrypted) and self-signed option are optional fields on the
osTicket integration, which otherwise still only reads the database. The settings page can send a
test ticket and run a check now; the Alerts page shows each alert's ticket number, linked to osTicket.

osTicket's API can't close or update a ticket, so a ticket opened here is the end of this app's part
in it. Tested against a stand-in osTicket, not a real one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07 23:41:44 +02:00
bobbanandClaude Sonnet 5.5 447f33fff6 Add an Alerts page under Operations listing everything that's wrong now
One list of the current problems across servers and integrations, instead
of waiting for a notification or visiting each page: servers that stopped
reporting, full or nearly full disks and volumes (critical from 95%),
Synology volume/disk problems, failed or uncovered Proxmox backups,
failed Proxmox Backup Server verifications, container image updates,
expired or expiring secrets/domains/Tailscale keys, failed Semaphore and
Gitea runs, Uptime Kuma monitors that are down, overdue osTicket tickets,
and integrations whose calls keep failing. Visible to every role, with
severity and kind filters, search, sorting, CSV export and "Check now".

It runs the same checks that send the notifications rather than a second
copy of them: the detection in the health, automation, Proxmox backup,
PBS, Docker update and Tailscale key checks is pulled out into shared
collectors that both the schedulers and the page call, so the two can't
disagree about what counts as a problem. Notification behaviour is
unchanged, including the scheduled backup checks skipping integrations
under a maintenance window. Unlike the notifications the page ignores the
on/off toggles, and keeps problems under a maintenance window, marked
silenced and counted apart.

It reads live, so a result is reused for a minute (and Refresh can't
re-run everything more than once every ten seconds), and every source has
a 20 s limit so one hung integration can't hang the page. Anything it
couldn't read is called out at the top instead of looking like all clear,
and server checks pause for the same 20 minutes after a restart as the
notifications do, with a note saying so.

Also gives the newer integrations (PBS, osTicket, Uptime Kuma, phpIPAM)
proper names in "integration down" notifications instead of their ids.

Verified through the real routes against a scratch database with fake
backends (offline and full-disk servers, secrets and domains, a silenced
server, a fake PBS with failed verification, a hanging integration, a
refused one, a failing-calls streak, caching, the restart grace period,
auth), and by rendering the real page against that data in a browser:
filters, search, silenced toggle, sorting, Check now, dark mode.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 23:07:57 +02:00