An update run can name a second Semaphore playbook that restarts the server once the updates are in and it says a restart is due. The run keeps the maintenance window up, follows the restart, waits for the server to report again and checks from its uptime (or its restart flag clearing) that it did start again. A server that only needs a restart has a Restart button on the Updates page, and a patch plan can carry the restart playbook too. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
39 KiB
Notifications
Every notification in Homelab Manager is sent through the same pipe
(server/src/services/notify.ts) to whichever channels are enabled under
Settings → Notifications: Gotify, ntfy, Telegram, Pushover, SMTP (email), a generic
JSON webhook, and push to phones and browsers. All seven fire for every notification below unless a
routing rule (see Who hears what below) says otherwise; each event also has its on/off toggle and, for the
daily ones, one shared time/timezone.
The settings page has four tabs, with one Save button for all of them (it
shows "Unsaved changes" until you save, and the browser asks before the page
is closed with changes unsaved): Channels (the seven channels, each a row
you open to fill in and send a test), What to send (the event toggles
grouped by topic, each group's thresholds beside its switches), Who hears what (routing rules and escalation) and When
(the time zone, the daily reminder time, quiet hours and the weekly digest).
The address keeps the tab, e.g. /settings/notifications?tab=when.
Two settings apply to every notification regardless of what triggered it:
- Quiet hours (Settings → Notifications → Quiet hours): while enabled and inside the configured window, a notification is held instead of sent immediately, then delivered as a single digest at the window's end time. This matters most for the real-time alerts below — the daily reminders already fire once at a time you pick, usually outside the window anyway. Still send critical alerts (off by default) lets a critical one — a server down, a disk full, mains power lost — through at once, so only the warnings wait for the morning.
- Away (Settings → Notifications → When, and a banner on every page while it is on): "I'm away until Sunday". See Away.
- Maintenance windows (the Maintenance page): opening one for a server, integration, or DNS provider silences the alerts that target it specifically (offline/disk-full for a server; storage, backup, and automation-failure alerts for an integration; "integration down" for that whole service type) — see the Maintenance page's own "What gets silenced" panel for the exact list.
Where a trigger has a configurable threshold, that's noted in its row below; several are fixed and can't be changed from the UI.
Notification history
Records → Notifications (operators and administrators; also linked from Settings → Notifications) lists everything that was sent, newest first, and answers "why did I not get it?". Each entry says what it said and how it went through each channel — a ✓ or ✕ beside Gotify, ntfy, Telegram, Pushover, Email, Webhook or Push, with what the channel answered when it refused — and one of these:
| Status | Meaning |
|---|---|
| Sent | Every channel it went to took it |
| Partly sent | Some took it, some refused or could not be reached |
| Failed | No channel took it |
| Nowhere to send it | No channel was switched on for it |
| Held | Quiet hours or away mode held it for the digest (the digest is its own entry) |
| Not sent (rule) | A routing rule says nobody should be told about that kind |
Filter by status or search the title and message. A notification about alerts also carries which alerts it was about, so each alert's Timeline (Alerts → History → the incident) says who was told and when, with the same ✓/✕ detail — including a message that was held or not sent. (A held message is listed against its alerts when it was held; the digest that later carries it is not.) Send again (Send now for a held one) sends the entry once more to every channel that is on, writes that down as a new entry and is in the change log; it does not carry the buttons an alert message had. Messages are kept 30 days (at most 1000 entries), cut at 4000 characters, and an error is cleaned of every token, key and password in the settings before it is kept, since a channel may say them back. The Send test buttons in Settings are not listed.
Who hears what
Under Settings → Notifications → Who hears what.
Routing rules. By default every notification goes to every channel that is switched on. A rule narrows that for what it matches, on any of: the kind of problem (server down, disk and storage, failing services, backups, updates, expiry, automation, integration failing …), how bad it is (warning or worse, critical only) and the tags of the server it is about (any of them). The first rule that matches decides which channels get it — put the specific rules first — and a rule with no channels tells nobody (the Alerts page still shows the problem, and webhooks for events still fire). A notification no rule matches goes to every channel that is on, as before. Rules can be switched off and reordered.
- A message that holds problems for different routes is split: a disk warning can go to ntfy while the server outage in the same check goes to everything, as two messages. When all of them take the same route it is one message, as ever.
- A recovery follows the route its alert took (the health check remembers how bad each problem was).
- What a rule can see: health and automation problems carry a kind, a severity and (for servers) the server; secret, domain and Tailscale expiry, backup checks, Docker image updates, update runs and integration failures carry a kind and a severity. DNS changes, the weekly digest and similar carry neither, so only rules that ask about nothing in particular match them.
- Notifications held for quiet hours keep who they were for: the digest at the end goes only to the channels their rule picked.
Each browser's own wishes. Under Phone and browser, every subscribed browser can say it only wants warnings or worse, or only critical alerts, and only some kinds of alert — a phone for the critical things, a desktop for everything. It applies after routing: a rule that sends something to Phone and browser still doesn't reach a browser that turned that kind down. A notification that says nothing about its kind or severity reaches all.
Escalation. When an alert has been raised for a while and nobody has acknowledged it, it is told again, in up to three steps — for example after 30 minutes: push; after 2 hours: email the neighbour who holds the key. Each step names its time, the least severity it applies to, the channels to use (chosen freely — routing rules don't apply to escalations), an optional extra email address to tell as well (sent through the email channel's server) and whether to send during quiet hours and while away (otherwise it waits for them to end like any other). Every step is sent once per alert; an acknowledgement stops the rest, and the alert starts over if it goes away and comes back. Problems under maintenance, held back because something they depend on is down, or only informational never escalate. It is looked at every five minutes, so a step can arrive a few minutes after its time. Every escalation is written in the audit log (category alert, action escalate).
Endpoint checks and restore tests
An endpoint check that fails several times in a row (the check's own number, 2 by default) is announced like any other server problem — under the server offline,
disk full … switch (healthAlerts) — once when it starts, with a recovery message when it answers again; a certificate that is within its warning time is announced
the same way. For routing they count as Monitoring (a check that fails) and Expiry (a certificate), and carry the tags of the server the check is linked to; maintenance on that
server holds them back. A restore test that fails, or one that is overdue (when a limit is set), is on the Alerts page and in the incident record but is not sent as a
notification of its own — it is something to look at when you are looking at backups.
Telegram and Pushover
Two more ways to a phone, for those who already live in one of them. Both are services with one address each (api.telegram.org, api.pushover.net), so nothing typed into the settings decides where a message is sent. Each has its own Send test in
Settings → Notifications → Channels, which sends with what is typed there (saved or not) and shows what the service said if it refuses.
Telegram sends through a bot of your own, to one chat. Make the bot by messaging @BotFather (/newbot) and paste its token; the chat id is a number (a group's is negative) or @yourchannel — send the bot a message (or add it to the
group or channel and post there), then open https://api.telegram.org/bot<token>/getUpdates and read "chat":{"id":…}. A bot cannot message a person who has not written to it first. The message is the title, a blank line and the text, as plain text
(nothing in it is read as formatting), cut to Telegram's 4096-character limit. Send silently sends without a sound or a buzz.
Pushover needs an application token (pushover.net → Create an Application/API Token) and your user key (or a group's), both from your Pushover account; a device name limits it to one of your devices. Priority runs from lowest (no notification) to emergency, which repeats every minute for an hour until it is acknowledged in the Pushover app. The message is cut to Pushover's 1024 characters and the title to 250.
The token and the keys are stored encrypted with the same key as the other credentials, and are never put in an error message or the audit log (a change to them is recorded as the field's name). A message that is refused is logged with what the service said; it never stops the other channels. Both take part in routing rules like any channel.
Push to a phone or browser
The fifth channel, Phone and browser, sends the same messages as the others to browsers that have asked for them — a phone with this page added to its home screen, a desktop browser — with no app, account or extra service. An administrator switches it on under Settings → Notifications → Channels (the optional contact is given to the browsers' push services, as the standard asks), and then each person opens the same row and presses Notify this browser; the browser asks permission and the row lists their browsers, with a button to send a test and one to remove each.
- It needs https. Browsers only allow push for pages served over https (or from
localhost), so a copy opened over plain http on a LAN address shows why it can't be used instead of the button. - How it travels. The message is encrypted for each browser (RFC 8291) and signed with a key pair this server makes the first time it is needed (RFC 8292, "VAPID"), then handed to the browser's own push service — Google's, Mozilla's or Apple's — which delivers it. The push service sees that a message is being sent and its size, not what it says. The address of each browser's push service is checked before use: https only, and never an address on a private network.
- Browsers' own wishes (a minimum severity, which kinds) are set per browser under the channel's row.
- Quiet hours hold push messages like every other channel, and each event's on/off toggle applies. A tap on a notification opens the Alerts page.
- A browser that has gone (the site's permission was removed, the app uninstalled) is dropped automatically when its push service says so. If the server's key pair is ever lost (the credentials key changed), everyone has to press the button again.
Daily reminders
These six checks share one schedule: Settings → Notifications → Daily reminder time (default 08:00) and Timezone (default UTC). Unlike the state-based alerts further down, these re-send every day the condition is still true — there's no "only once" de-duplication, so an expired secret you haven't renewed yet will be mentioned again at the next day's check, and the day after that.
Each one also runs once at server startup if it hasn't already run today (e.g. after an upgrade or a period offline), so you're not waiting until the next scheduled time to catch up.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Secret expiry | Secret expiry reminder | Any tracked secret (API token, SSL cert, password, generic) is expired or within its own configured warning window. SSL certificates with a host:port are re-checked live first, so this reflects the real current expiry, not a stale saved date. A certificate that couldn't be re-checked live is mentioned separately, so the expiry shown may be stale. | Per-secret warning threshold, set when adding/editing that secret |
| Domain expiry | Domain registration expiring or expired (daily reminder) | Any tracked domain registration is expired or within the warning window; every domain is re-looked-up against its registry first. A domain whose registry lookup has been failing for 3+ days is also mentioned, separately, as "may be stale." Registries that don't publish an expiry (e.g. .de, .eu) are tracked but never trigger this. |
Settings → Notifications → Health checks → Domain expiry warning (default 30 days) |
| Tailscale key expiry | Tailscale key expiry reminder | Any device's node key (across every enabled Tailscale integration) is within 30 days of expiring, or already expired. Devices with key expiry disabled are skipped. | Fixed at 30 days |
| Docker containers broken | Warn when a Docker container is broken | A container keeps restarting, is unhealthy, stopped unexpectedly or never started, or a Docker host cannot be reached, as Dockhand reports on the 15-minute pass — announced once with the server problems (Server offline, disk nearly full …, healthAlerts), with a recovery message when it runs again. See MONITORING.md. |
Its own switch; Containers to leave alone |
| Docker image updates | Docker image update available | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
| Proxmox backups — gone stale | (same toggle as the Proxmox backup check) | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → A guest's backup is too old after (default 48 h; 0 off); a server can have its own limit on its page |
| Game server turned on or off | A game server (Fragnet) was turned on or off | A read of a Fragnet account shows a game server on (running or starting) when the one before showed it off, or off (stopped or stopping) when it showed it on — however it came about: someone pressed Start or Stop, the game crashed, the host stopped it. One message for what changed in one read, a line for each server (name, game, account); starting counts as on, so a start is said once. A server met for the first time, one whose state could not be read and one being installed or moved say nothing. Goes through the routing rules as a services notification and waits for quiet hours like any other that is not critical; the accounts are read every five minutes, so it can be that late. On unless switched off. | Settings → Notifications → What to send |
| Update run finished | An update run you started finished, or didn't | An update run started from the Updates page ended: what is still waiting and whether a restart is due (or, for a run that restarts the server afterwards, that it was restarted and is back), or why it failed. A restart started on its own says "restarted" or "Restart of … didn't finish". Sent at once (not held for the daily check). | Settings → Notifications → What to send |
| Proxmox backups | Proxmox backup failed or a guest has no coverage | Two independent conditions, both under this one toggle: (1) a node's most recent vzdump backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. |
Not configurable |
| Proxmox Backup Server verification | Proxmox Backup Server snapshot failed verification | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup ran, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
State-based alerts
These two run on a fixed 15-minute interval (matching the agent's own default report interval) — not the daily reminder time above, and not user-configurable. Unlike the daily reminders, these are state-based: a problem is announced once when it first appears, and once more when it clears — not repeated on every 15-minute pass while it continues. A condition that can't currently be read (an integration that's down, a server whose agent hasn't reported) is held exactly as it was rather than cleared or re-announced, so a temporary read failure can't fake a recovery. On first-ever run (or right after upgrading to this feature), whatever's already failing is recorded silently as the starting baseline rather than announced all at once.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Health — server offline | Server offline, disk nearly full, or Synology volume/disk problem | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → Server offline after (default 60 min) |
| Health — host health | (same toggle as the others in this group) | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
| Health — disk usage | (same toggle) | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → Disk usage alert at (default 90%) |
| Health — Synology status | (same toggle) | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
| Automation — failed run | Semaphore template or Gitea workflow run failed | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
Real-time alerts
These fire immediately, as the triggering action happens — not on any schedule.
| Event | "Notify on" toggle | Fires when |
|---|---|---|
| DNS record added | DNS record added | A DNS record is created against any DNS provider through this app (manually, or via any automated action that writes one). |
| DNS record updated | DNS record updated | An existing DNS record is edited. |
| DNS record deleted | DNS record deleted | A DNS record is deleted. |
| Integration/DNS provider down | Integration/DNS provider failing repeatedly | Any integration or DNS provider adapter's calls fail a configurable number of times in a row. Tracked per adapter type (e.g. "proxmox"), not per individual integration row — with two Proxmox integrations, a streak of failures on either one counts toward the same total. Only fires once per failure streak (not on every failure past the threshold), and is skipped entirely if that type is currently in a maintenance window, or if it is explained by something it needs that is down (see Dependencies in the README): a service that starts failing while the server it runs on is offline is not announced, and if it is still failing once that server is back, the next failed call alerts. |
| Integration/DNS provider recovered | (same toggle) | The next call for that source succeeds, after a "down" alert was already sent for the current streak. |
Digest
| Notification | Fires when |
|---|---|
| "Notifications from quiet hours" | Once, at quiet hours' configured end time, only if enabled and only if at least one notification was held during the window. Bundles every held notification's title and message into one message, then clears the queue. |
Backups of Homelab Manager itself
A scheduled backup of the app's own database that fails sends one message through the channels ("Homelab Manager — its own backup failed", with the reason), under the same toggle as server and disk alerts (Server offline, disk nearly full…). The same problem is also an alert on the Alerts page of the Backups kind — "the newest backup of Homelab Manager's own database is … old" once it is more than two days old, critical after a week, and "no backup of its own yet" with the reason when the very first one fails. No alert is raised for a new install that hasn't tried yet, or when daily backups are switched off. See BACKUP.md.
Monthly report
| Notification | Fires when |
|---|---|
| "Homelab monthly report — " | On the chosen day of the month (1–28) and time, in the notification time zone, only if switched on under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there is something to say, now with a Service levels topic (how each target did against its error budget). The same report is on the Reports page. Preview last month and Send last month now are on the settings page.
With Also send a quarterly report on, "Homelab quarterly report — Q3 2026" follows the monthly one after the month that ends a quarter (March, June, September, December), and with Also send an annual report on, "Homelab annual report — 2026" follows the December one. Both are the same report over three or twelve months, sent the same way; both are off until switched on.
Service levels
A service-level target (Alert settings → Service levels) is announced like the other server problems — under Server offline, disk nearly full … (healthAlerts), once, with a recovery message when it clears — when 80 % of the month's error budget is used
(a warning: "Gitea: 90% of the error budget for its 99.9% target is used up for October 2026 — down 40 min of 45 min allowed, 5 min left") and again when it is overspent (critical: "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed").
It is Monitoring for routing, and it ends when the month turns over. Maintenance does not hold it back: planned downtime counts. Each target has its own switch.
Network devices
A switch, access point or gateway that a UniFi controller has not seen online for longer than the limit (Settings → Notifications → Servers and disks, 10 minutes) is announced like a server that stopped reporting, under
Server offline, disk nearly full … (healthAlerts): "Hallway AP (access point, U6 Pro) has been offline for 45 min", with a recovery message when it is back. For routing it counts as Server down; maintenance on the
device holds it back; and what depends on it (or it is connected through, if that is down too) is folded under it and named in its message ("— also takes down …"). The controller is read every five minutes; if it cannot be
read for twenty, its devices are held — neither announced nor cleared.
Your own alert rules, and snoozing
An alert rule (Alert settings → Alert rules) that is past its limit is announced like any other health alert (Health Alert, with a recovery message), counted as Monitoring for routing with the rule's own severity, and held back by maintenance on the server. It is checked on the health pass, every 15 minutes.
An alert that is snoozed (Alerts page → Snooze…) is not announced or recovered by the health and automation checks until the snooze ends. When it ends and the alert is still there, one "Snooze ended" message (⏰) says so, through the usual routing and quiet hours; if the alert went away, nothing is sent. Event webhooks still receive every raise and recovery.
Buttons on notifications
Switched on under Settings → Notifications → Who hears what → Buttons on notifications (off by default), a notification about alerts that have a key — the health and automation problems, a flapping alert, a snooze that ended, and the escalation reminder — can be acted on from the message:
| Channel | One alert | Several |
|---|---|---|
| ntfy | Three buttons (Acknowledge, Silence 1 h — only when the alert is about something a window can silence — and Snooze) that POST to the alert's own link, and a tap that opens the page | One button, Choose what to do, to a page that lists them |
| Phone and browser push | Two buttons on the notification itself (Acknowledge, Silence 1 h); a press sends it and shows how it went. A tap opens the Alerts page | A tap opens the Alerts page |
| Telegram | One button under the message, Acknowledge, silence or snooze, that opens the page. (If Telegram will not take the button's address — one on a private network, say — the link is added to the text instead) | The same |
| Pushover | The link is attached to the message (Acknowledge, silence or snooze), and a tap opens the page | The same |
| Gotify | The link is added to the text, and a tap opens the page | The same |
| Email, the chat webhook, an escalation's extra address | The link, as a last line of the text | The same |
Each link is a signed token for the alerts in that message only, valid for 24 hours, for three actions: acknowledge, silence the thing the alert is about for an hour, snooze until tomorrow 09:00 (in the notification time zone). Opening it shows a page that says what each button would do and changes nothing; only a button press does anything. What was done is in the audit log, from a notification link, with the address it came from. Anyone who can read the notification can use it for those 24 hours. A message held for quiet hours is sent without buttons; recovery messages never have them.
Away
For a stretch rather than every night. Settings → Notifications → When → Away takes the time you will be back (with shortcuts: tomorrow evening, Sunday evening, a week, two weeks), optionally a start later than now (to plan a holiday ahead), a note to yourself and whether to still send critical alerts. While it lasts:
- notifications are held — the same queue quiet hours use — and a banner on every page says that someone is away and how many are held;
- critical alerts still go out at once, if that box is ticked (it is by default); with it off, nothing does;
- at the time you gave, or when you press I'm back, what was held is sent as one digest — "Homelab Manager — Notifications from while you were away" — to the channels each was meant for, and the away stretch is cleared. It is looked at every minute, and also right after the app restarts, so it ends within a minute of its time;
- an escalation step that says send even during quiet hours is sent while away too — that is what it is for, for example an email to the neighbour who holds the key;
- if quiet hours are on at that moment, the digest waits for them to end; if you come back in the middle of an away stretch's quiet hours nothing is lost.
What it does not do: it does not stop the alerts themselves — they are on the Alerts page, in the event webhooks and in /metrics as ever, so a Home Assistant or Grafana setup still sees everything; it only holds the messages to you. Away mode is an
administrator's to set (it decides who is told what), everyone sees the banner, and starting, changing and ending it are in the audit log and the Change log. Times typed there are the clock of the screen they are typed on.
Flapping alerts
A health alert that is raised, clears and is raised again is announced the first times as usual. When the same alert has been raised three times within an hour (Settings → Notifications → Servers and disks; 0 = off) it is flapping:
- one message, "Flapping alert", says "↻ — raised 3 times within an hour";
- its raise and recovery messages are then held back (the alert stays on the Alerts page with a flapping badge and still counts for the menu badge and /metrics);
- once it has been calm for an hour, one "Alert settled" message says whether it is "⚠ still active" or "✓ cleared", and messages for it are sent as usual again.
This follows the routing rules and quiet hours like other health messages, and is checked every five minutes. Event webhooks still receive every raise and recovery — only the messages to people are held. The Alerts → Noise tab lists the alerts that come back most often.
Patch plans and new ports
A patch plan run sends one message when it ends — "Patch plan "Lab nights" finished" or "didn't finish cleanly" — with a line for each server, under the update run results switch (the same one as a single update run's message). A server
that begins listening on a new TCP port is announced like the other server problems — under Server offline, disk nearly full … (healthAlerts), once, with a recovery message when the port is noted or stops listening; for routing it counts as Monitoring,
carries the tags of the server, and maintenance on the server holds it back.
Unusual for the time
A server whose CPU load or memory use over the last hour is clearly higher than is normal for that hour of that weekday (see MONITORING.md) is announced like the other server problems — under Server offline, disk
nearly full … (healthAlerts), once, with a recovery message when it is back toward normal. For routing it counts as Monitoring, carries the server's tags, and maintenance on the server holds it back. It has its own switch under
Settings → Notifications (Servers and disks), on by default; with no four weeks of history for the hour there is nothing to announce.
Weekly digest
| Notification | Fires when |
|---|---|
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), only if switched on — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say: what is wrong right now; uptime over the last seven days (what was down and for how long, with the percentage); what happened (servers that stopped reporting and came back, services that stopped answering, maintenance); incidents (how many began and ended, still open, the longest); disks heading for full; secrets and domains that are expiring or have expired (and hardware whose warranty ends within 60 days); guests that are not backed up, have a failed or stale backup; updates waiting (how many servers, how many security fixes, who needs a restart, the longest wait); and planned maintenance in the next seven days. A week with nothing in it says "A quiet week: nothing needs attention." On the settings page, Preview shows what it would say right now and Send it now sends it at once (through the channels that are on, whatever the time).
The Alerts page
Alerts (the Now view; History lists every incident that has come and gone) shows what these notifications are about — the problems that exist right now — as a list you can look at, filter, and export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs in a few ways:
- It ignores the "Notify on" toggles. Turning a notification off doesn't hide the problem from the page.
- Problems under a maintenance window are kept on the list, marked silenced and counted separately, instead of being dropped. So are snoozed ones, marked with when they come back.
- It also lists what is on the Alerts page only: forecasts, operating systems out of support, a restore test that failed or is overdue — and, at the top, what could not be checked.
- It also lists things nothing notifies about: Uptime Kuma monitors that are down, osTicket tickets that are overdue, and any integration that's failing its last few calls (before the threshold that triggers an "integration down" notification).
It runs the checks live when opened (a recent result is reused for a minute), and shows what it couldn't read at the top, so a missing section means "couldn't check" and not "all clear". An alert that has an osTicket ticket (see the next section) shows its ticket number.
Tickets in osTicket
Not a notification, but it acts on the same alerts: with Alert settings → Alert tickets switched on (off by default), a problem on the Alerts page opens a ticket in osTicket. It runs every 15 minutes, starting about a minute and a half after the app starts. It never closes a ticket by itself; closing is something you choose, in the Close drawer on the Alerts page, when the integration has a separate database login for it (see INTEGRATIONS.md).
| Rule | What it means |
|---|---|
| Which problems | The severities and kinds you choose — critical and warning, every kind by default. Info (updates waiting and the like) is off. |
| Must last | A problem must be there in 2 checks in a row (15–30 minutes) before it opens a ticket, so a blip that fixes itself — one failed call to an integration, a monitor that flickers — never does. Can be set from 1 (at once) to 4. |
| One ticket per problem | However many checks see it, once. After a ticket is open, the problem isn't ticketed again while it lasts — even if someone closes the ticket in osTicket. |
| Comes back | A problem that clears and returns within 60 minutes is the same problem (no new ticket). After longer, it is a new one and gets a new ticket. |
| Many at once | More than 5 new problems in one check become a single summary ticket listing them all, so one outage doesn't fill the queue. Up to 5 get a ticket each. |
| Maintenance windows | A problem under one never opens a ticket. If it is still there after the window ends, it does. |
| Never about osTicket | Overdue tickets, and osTicket's own calls failing, never open a ticket — it would go to the thing that is down, or add to the pile. |
| Already there at switch-on | Left alone by default: the first check after switching on records what is wrong as the starting line, and only problems that start after it get a ticket. Can be set to ticket those too. Switching it off and on again starts a new line. |
| If osTicket can't be reached | Nothing is recorded as opened; the next check tries again. A refused key, a wrong address and similar are shown on the settings page and in the Diagnostic Log. |
| If something can't be read | When a check couldn't read everything (an integration was down), nothing is treated as gone, so a problem isn't called cleared just because its source couldn't be looked at. |
The ticket says what is wrong, the severity, the kind, where it comes from, when it was first seen, and links back to the problem in this app. osTicket's
API can't update or close a ticket, so it isn't touched again: close it in osTicket when the problem is dealt with. Every ticket opened (and every test
ticket) is in the Audit Log as alert_ticket, and changes to the settings as settings / update_alert_tickets. "Send a test ticket" and "Check now"
on the settings page use the saved settings.
What does not send a notification
Worth calling out explicitly, since it's easy to assume everything in the app alerts on something:
-
Forecasts — a server's disk, a Proxmox storage or a Synology volume that will be full within the forecast window — are on the Alerts page, in the weekly digest and in /metrics, but are not sent as they happen.
-
Operating-system support — a server whose OS has passed, or is about to pass, its end of support — is on the Alerts page (a warning once it has ended, a note while it is coming), not sent.
-
A snoozed alert: its raise and recovery messages from the health and automation checks are held until the snooze ends (then one "Snooze ended" message, if it is still there). The summaries other checks send — stale backups, expiring secrets, Docker updates — are not held.
-
osTicket's own tickets — the integration lists open tickets, but there is no notification for it (e.g. nothing pings you about a newly-overdue ticket); the page and the Alerts page show them. (The other direction — an alert opening a ticket — is the section above.)
-
phpIPAM sync, Uptime Kuma monitor/maintenance import, and any other manual "Sync from X" action — these run when you click the button and report their result on the page itself, not via a notification.
-
Test buttons (Settings → Notifications → each channel's "Send test") send a one-off message through that one channel only, to confirm it's wired up correctly — not a real event and not affected by quiet hours.
-
Consistency reports, the Diagnostic Log (Integrations → Call log), and the Audit Log (Change log → Raw audit log) are all read-only views you check yourself; none of them push a notification on their own (the Diagnostic Log's failures are what feed the integration-down alert above, but reading the log itself never triggers anything).
Heartbeats
A heartbeat that doesn't hear from its job in time is announced like any other problem — under the server offline, disk full … switch (healthAlerts) — once when it is late, with a
recovery message when the next call comes in. A job that calls its /fail address is announced the same way, as critical, at once. For routing both are Automation, and carry the tags of
the server the heartbeat is linked to; maintenance on that server holds them back, and they alert after it if they are still there. Quiet hours, escalation and push preferences apply as to the rest.