README and INTEGRATIONS: what it reads, the config and token (15.0 or later), the DNS test, the alerts and metrics, that its API has no CPU, memory or disk figures, and how it differs from the Technitium DNS provider. ROLES: the page. DATABASE: technitium_state, migrations to 0081, 91 tables. MONITORING: the five alerts. PROMETHEUS-GRAFANA: the homelab_technitium_* metrics. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
76 KiB
How monitoring works
This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the "why did it do that?" page. Setting things up is in the other pages (SSO, API, Prometheus and Grafana, webhooks, heartbeats); the list of every notification is NOTIFICATIONS.md.
- The big picture
- What is watched, and how often
- Servers and the agent
- From a reading to an alert
- The built-in checks
- What is normal for the time
- Service levels
- Your own alert rules
- Duplicati backups
- Nextcloud
- Hetzner Cloud
- Game servers (Fragnet)
- Media managers
- App versions
- AdGuard Home
- Technitium DNS
- Grafana
- Speedtest
- NetBox
- n8n
- Immich
- Paperless-ngx
- Prometheus
- Authentik
- Jellyfin
- Docker containers that are broken
- Endpoint checks and heartbeats
- Integrations
- Forecasts
- Dependencies: one outage, one alert
- What happens to an alert — notify, acknowledge, silence, snooze, flapping, escalate
- Incidents
- History and how long it is kept
- Where to see it
- Tuning, in questions and answers
- What monitoring does not do
The big picture
agents ─────────┐
integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · close · silence
endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push
heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines)
your own rules ─┘ ├──► incidents (the record) and history
└──► /metrics and the read API
Three ideas run through everything:
- A check turns readings into conditions. A condition is "this specific problem exists right now" and has a stable key (
offline:server:3,disk:server:3:/) — the same problem always has the same key. - Alerts are announced on a change, not on a schedule. A problem is announced once when it appears and once when it ends. It is not repeated every time it is looked at.
- Silence is never "all clear". When something cannot be read (an integration is down, a server's agent is quiet), what depended on it is held exactly as it was — neither raised nor cleared — and the Alerts page lists it under Couldn't check everything. The app never turns "I couldn't look" into "nothing is wrong".
What is watched, and how often
| What | How it gets to the app | How often |
|---|---|---|
| Servers (Linux, Windows) | The agent on each server pushes a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects to a server. | Every 15 minutes by default (set when the agent is installed) |
| Proxmox, Synology | The health pass reads them: node and storage usage, volume and disk status. | Every 15 minutes |
| UniFi | A sync reads devices, networks, Wi-Fi names and clients and keeps them. | Every 5 minutes |
| Home Assistant | A sync reads the entities you chose to watch and keeps their state and a history. | Every 5 minutes |
| UPS (NUT or SNMP) | A sync reads each UPS and keeps its state and a history. | Every minute |
| Nginx Proxy Manager | Read when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute |
| Semaphore, Gitea | The automation check reads the latest run of each template or workflow. | Every 15 minutes |
| Endpoint checks (web address, TCP port, TLS certificate) | The app asks each one itself. | Each check has its own interval (5 minutes by default); the scheduler looks every minute for the ones that are due |
| Heartbeats (cron jobs) | The job calls an address when it finishes. | When the job runs; judged every minute |
| Secrets, certificates, domains, Tailscale keys | Their expiry dates, with the certificate fetched live. | Daily, at the reminder time in Settings → Notifications |
| Docker image updates, Proxmox backup results, Proxmox Backup verification | Read and judged. | Daily, at the same time |
| Your own alert rules | Judged from the history the agents' reports build. | On the 15-minute health pass |
| Every alert, together | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to 1 minute old · incidents: every 5 minutes · snoozes: every minute · escalation and flapping: every 5 minutes |
After the app restarts, server-based checks are paused for 20 minutes so agents get a chance to report before any server is judged.
Servers and the agent
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that only pushes. Each server has its own token, which can do one thing: replace that server's own report. The app replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the README and AGENT_UPDATE.md.
When is a server "offline"? When its agent has not reported for longer than the offline limit — 60 minutes by default (Settings → Notifications → Servers and disks), or the server's own limit if it has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that never reported is "unknown": it has no agent to go quiet, so nothing alerts.
While a server is offline its other figures are stale, so it is not also judged on disks, services or your rules — its offline alert says it all.
Per-server settings. A server's Settings section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for
one disk or "never alert about this disk", and services never to count as failed (names or patterns such as gupdate*). Whatever it does not replace follows the global setting.
What the report is used for: the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support
check, pending updates, the dependency map, and /metrics.
From a reading to an alert
There are two paths, designed so they cannot disagree:
1. The state-based checks (the notifications). Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the list of conditions that exist right now. The app compares that list with the one it remembered from last time:
- new → announced (a notification, an event webhook, a row in the incident record);
- gone → announced as resolved;
- still there → nothing;
- its source could not be read this time → carried over untouched (not resolved, not repeated).
The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all.
2. The Alerts page. It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), live, and turns what they find into one flat list. It ignores the per-event on/off toggles — the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a 20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under Couldn't check everything.
Each alert has:
| Severity | critical, warning or info. Things that are gone (offline, a failing SMART disk, a disk over 95%) are critical; things that are coming (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. |
| Category | offline, disk, services, backup, updates, expiry, automation, integration, monitoring, tickets — what the filter on the page and the routing rules work on. |
| Source | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … |
| Target | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. |
| Key / id | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. |
The built-in checks
Every threshold here is a default; where it can be changed, the place is named.
| Check | Raised when | Default | Changed in |
|---|---|---|---|
| Server offline | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
| Disk nearly full | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, critical from 95% | Settings → Notifications; per server, per disk |
| Disk filling up | A disk on a steady climb will be full within the window (Forecasts) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
| Failed services | A Linux agent reports failed systemd units | any | per server (ignore list) |
| Disk health | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | not configurable | |
| ZFS / storage pool | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | not configurable | |
| Synology volume / disk | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | not configurable | |
| Unusual for the time | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday (below) | on · warning | Settings → Notifications |
| New port | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's Ports card (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Settings → Notifications |
| Network device offline | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
| Operating system support | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
| Updates waiting | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | Updates page | |
| Backup stale / failed | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
| Restore test | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| Expiry | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| Docker updates | A container image has a newer version | Settings → Notifications | |
| Nextcloud can't be used / disk low | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; below |
| Hetzner server off / traffic / rescue / backup | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; below |
| Sonarr / Radarr / Prowlarr | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; below |
| AdGuard Home | DNS has not answered for ten minutes, or its DNS server is not running (critical); protection left off for an hour, or filter lists not updated for a week | on, with the integration | AdGuard Home page; below |
| Technitium DNS | DNS has not answered for ten minutes, or a secondary zone has expired (critical); a zone cannot sync, blocking left off for an hour, or block lists overdue for an update (warning) | on, with the integration | Technitium DNS page; below |
| Grafana | An alert rule is firing (critical when labelled critical or page), a rule cannot be evaluated, or a data source fails its health check | on, with the integration | Grafana page; below |
| Game server backup / disk / memory | A Fragnet game server's newest backup is older than the limit (7 days) or none, its disk is over 90% of the plan (critical from 98%), or its memory is over 90% while it runs | on, with the integration | Game servers page; below |
| Internet slow / speed tests failing | The middle of the last three speed tests is under the share you set of what the line is sold at (only with those numbers given), or the last three tests all failed | on, with the integration | Speedtest page; below |
| NetBox has no worker | No background worker is running | on, with the integration | NetBox page; below |
| n8n workflow failed | A workflow that is switched on failed on its last run | on, with the integration | n8n page; below |
| Immich disk filling / jobs failing | The disk under the library is 90 % full (critical at 95 %), or 10 or more background jobs have failed | on, with the integration | Immich page; below |
| Paperless-ngx needs a look | The worker, database or Redis is in error (critical), documents failed to import, the index or sanity check is in error, or the disk is 90 % full | on, with the integration | Paperless page; below |
| Prometheus target down / alert firing | A scrape target has not answered for five minutes, or one of Prometheus's own alert rules is firing | on, with the integration | Prometheus page; below |
| Authentik needs a look | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; below |
| Jellyfin needs a look | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; below |
| Duplicati backup failed / overdue | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; below |
| Docker container broken | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; below |
| Docker image has critical vulnerabilities | Dockhand's own scans of a Docker host's images found one or more critical findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; below |
| Docker host has many high vulnerabilities | The same scans found at least the number of high findings you set (Also warn when a Docker host has this many high vulnerabilities; 0, the default, never) | off | Settings → Notifications; below |
| Docker host has old running images | A Docker host has running containers whose images were built more than the number of days you set ago (Warn about running Docker images that were built more than …; 0, the default, never) — scanned or not | off | Settings → Notifications; below |
| App behind its newest release | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; below |
| Automation failed | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | not configurable | |
| Integration failing | An integration's calls have failed several times in a row (Integrations) | 3 calls | Settings → Notifications |
| Endpoint check failing / certificate running out | See below | per check | |
| Heartbeat late / failed | See below | per heartbeat | |
| Home Assistant / Nginx Proxy Manager | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | Home Assistant page | |
| Service level | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; below |
| UPS | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | UPS page; INTEGRATIONS.md | |
| Your own rules | Below | Monitoring → Alert rules |
What is normal for the time
A fixed limit cannot say that a server is busy for a Tuesday at 03:00. So for each server's CPU load and memory use the app keeps a baseline for every hour of the week (168 of them, in the notification time zone): the median of that hour's averages over the previous weeks (up to ten; the hourly figures are kept four months), and a robust spread (the median absolute deviation, so one odd week does not stretch it). An hour needs four weeks behind it before it has a baseline at all; until then nothing is said. The hour in progress and hours with fewer than two samples are not counted.
It is quiet by design. A reading is a warning only when all of these hold, judged on the last hour of samples (at least three, the newest under 30 minutes old):
| CPU | Memory | |
|---|---|---|
| The last hour's average is at least | 50 % | 70 % |
| …and above the usual by at least | 30 points | 15 points |
| …and above it by at least | four spreads (the spread is taken as at least 4 points) | four spreads (at least 2 points) |
| …and the lowest sample in that hour is still above the usual by | 15 points | 7.5 points |
The last line is what separates a state from a spike: if the figure dropped back toward normal at any time in the hour, it is not said. Only a high reading is ever raised — low is never news. What it finds is an ordinary health
condition (baseline:cpu:server:<id>, baseline:mem:server:<id>, Monitoring, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for
a server that has stopped reporting (its offline alert says that). Switch it off with Warn when CPU or memory is unusually high for the time under Settings → Notifications → What to send → Servers and disks.
The same baselines draw the grey usual line and band on a server's History → Trends (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
Service levels
Monitoring → Service levels holds a number against a service: 99.9 % for Gitea. A target is set on an endpoint check, a server or an integration and is judged over calendar months (on the notification time zone's clock). Its error budget is the downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and where today's pace would end the month, once a day has passed), and how each of the last six months went.
How downtime is measured, so the figure can be trusted — and where it cannot:
| Measured on | From | Precision |
|---|---|---|
| Endpoint check | The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. | A day's failed share is spread over the whole day |
| Server | The moments the audit log recorded it stopping reporting and reporting again | To the minute; only from when the server was created |
| Integration | The moments calls to that kind of service started failing and worked again | To the minute; only from when the app began recording them |
Planned maintenance is not taken out: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing.
When 80 % of the budget is used it is a warning, and when it is overspent a critical alert (key slo:<id>, Monitoring, source Service level): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks —
and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in /metrics (homelab_service_level_*) and in the monthly, quarterly and annual reports (below).
Your own alert rules
Where the built-in checks do not fit, Monitoring → Alert rules (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
- What to watch: CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
- When: above or below a limit, and — for load, memory and disk use — for how long. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time.
- How serious: warning or critical. For which servers: all of them, one, or those with a tag.
- Try it now shows which servers it would be raised on, from current data, before you save it.
A rule's finding is an ordinary alert with the key rule:<rule id>:server:<server id>[:<disk>], in the Monitoring category: announced and resolved like the others, held back by maintenance on the
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at when the server reports (right after its report is stored — so a figure that has been past its limit long enough is raised within seconds of the report, not up to fifteen minutes later) and again
on every 15-minute health pass, which is also what notices a rule that was deleted or switched off and what judges the servers that have not reported. "For 15 minutes" is still counted in the readings the reports build (one every 10–15 minutes), so it
means the readings over 15 minutes, as before. A server that is offline or has not reported for 45 minutes is not judged.
The check after a report looks at that server's rules only and changes nothing else in what is known: the built-in checks, the other servers and the integrations wait for the pass, and what the pass would say is the same. It is not done for a server looked at in the last minute, nor during the 20-minute grace after the app starts, and it is never run at the same moment as a pass (one waits for the other). What is unusual for the time (the baselines) still waits for the pass.
Duplicati backups
Duplicati tells this app how each backup went (INTEGRATIONS.md says how to set it up), and Backups → Duplicati shows every job: when it last succeeded, how the last run ended, how long it took, how big the source is, how many files changed, and where it stands.
| State | Meaning | Alert |
|---|---|---|
| In order | The last run worked and it is not overdue | — |
| Warnings | The last run finished with warnings (a locked file, say). It still counts as a backup | none |
| Failed | The last run ended in Error or Fatal |
warning; critical from the second failure in a row |
| Overdue | It has not succeeded for longer than it is expected to — 2 days unless you set it per job (6 hours to 90 days) | warning; critical at three times that |
The alert says which job, on which machine, why (the first error Duplicati reported) and when the last good backup was. Only the Backup operation counts: a compact, a repair or a verification does not say whether the data is safe. A job can be muted (no alert; it is still listed), and one that no longer exists can be forgotten (with its history; if Duplicati reports it again it is learned again). Each machine can be tied to a server, so maintenance on that server silences its backups' alerts.
What is kept: the last result of each job, and each report for 400 days or 400 reports. /metrics has homelab_duplicati_backup_ok, homelab_duplicati_last_success_age_seconds and homelab_duplicati_source_size_bytes for each job. A job is only learned from its first report, so a job that has
never reported is not known to be missing; and because the reports are pushed, the app being down when a backup finishes loses that report (the backup shows as overdue only once it is).
Nextcloud
An integration of type Nextcloud (INTEGRATIONS.md) is read every five minutes, and Apps → Nextcloud shows its version, whether it is usable, and — with an administrator's app password — the room left for the data, the users, the files and the app updates waiting.
| Alert | When | Severity |
|---|---|---|
| Database needs upgrading | status.php says needsDbUpgrade — nobody can use Nextcloud until occ upgrade is run |
critical |
| Maintenance mode on | status.php says maintenance — nobody can sign in |
warning |
| Data disk low | Less than 20 GB free where the data is kept (needs the login) | warning; critical under 5 GB |
| Background jobs not running | Nextcloud's own Cron last run check says error — the last job ran more than an hour ago (needs the login; Nextcloud 28 or later) | warning |
| Nextcloud's own checks | One or more of its other setup checks report an error; one alert naming up to three | warning |
Maintenance on the integration holds them. A reading older than 20 minutes is not believed. Cron is read through Nextcloud's own setup checks (the Monitoring app has no field for it), so it needs the login and Nextcloud 28 or later; before that it is not known and never counted as fine. Warnings from those checks (HTTP headers, a PHP setting) are shown on the page, not alerted.
Hetzner Cloud
An integration of type Hetzner Cloud (INTEGRATIONS.md) is one project, read every five minutes, and Infrastructure → Hetzner Cloud shows its servers (status, type, place, addresses, this month's traffic against what the type includes, whether backups are on), volumes, snapshots, and roughly what the project costs. Nothing is started, stopped or changed.
| Alert | When | Severity |
|---|---|---|
| Server off | A server the app saw running is now off (Hetzner keeps billing a server that is off). One off for a week, or never seen running by the app, is left alone — as a stopped container is | warning |
| Stuck in a status | A server has been starting, stopping, migrating, rebuilding or initializing for half an hour | warning |
| Traffic | A server has sent 80% of its included traffic for the month; at 100% it is critical (the excess is billed) | warning; critical at 100% |
| Rescue mode left on | Rescue mode has been switched on for an hour without a reboot — the next reboot (a kernel update, say) would come up in the rescue system and the server would not come back | warning |
| Backups late | A server has backups enabled and the newest is older than the backup older than limit (48 hours by default — the same setting as for Proxmox guests; 0 turns it off), or none has been made and the server is older than that | warning |
| Traffic forecast | A server is on course to pass its included traffic before the month ends (by the last week's rate, or the month's average after three days) while it has used under 80% — the alert says on which day. Not raised from 80% on: the traffic alert covers that | warning |
| Load balancer unhealthy | A load balancer has a target Hetzner judges unhealthy; critical when none of its targets is healthy | warning; critical when all |
| Certificate ending | A certificate that is in use ends within 14 days (critical within 3, or past its end), or Hetzner could not issue or renew a managed one. A certificate nothing uses is not raised | warning; critical within 3 days |
| Floating IP unattached | A floating IP points at no server — it is billed all the same | warning |
Hetzner does not say when a server went off or when rescue mode was switched on, so "since when" is when the app first saw it in a read. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). Snapshots, volumes (including ones attached to no server, which are billed all the same) and the cost are shown, not alerted.
The cost is an estimate from Hetzner's own price list, including VAT: the servers, their backups (a percentage of the server's price), volumes, snapshots, primary IPs, load balancers and floating IPs. It leaves out traffic over the quota, so it is a floor, not a bill.
The traffic forecast is made from the month's outgoing traffic, written down about every six hours for two weeks: the rate of the last week (at least two days of it) or, with too little, the month's average so far (after three days). It assumes the month is the calendar month in UTC — not checked against a live project — and a counter that goes down starts the history over. The page shows under each server's traffic bar either the day the quota is passed or what the month is on course to come to.
Load balancers, floating IPs and certificates are listed on the Hetzner page — a load balancer with how many of its targets are healthy, a floating IP with the server it points at, a certificate with when it ends (and the ones nothing uses) — and their prices are in the cost estimate. /metrics has each load balancer's unhealthy targets, each certificate's end and each server's forecast month.
On a server's page, a Hetzner Cloud card appears when one of the project's servers is that server — matched by its public or private address, else by name — with its status, type, newest backup and traffic. A Hetzner server with no agent is listed as not in the app's servers. /metrics has homelab_hetzner_server_running, homelab_hetzner_traffic_outgoing_bytes, homelab_hetzner_traffic_included_bytes and homelab_hetzner_monthly_cost.
Media managers (Sonarr, Radarr, Prowlarr)
An integration of type Sonarr, Radarr, Prowlarr, Lidarr or Readarr (INTEGRATIONS.md) is read every five minutes, and Apps → Media managers shows, for each: what the application says is wrong with itself, its download queue (the stuck ones first, with why), its disks, how many wanted items have no file (Sonarr, Radarr) and, for Prowlarr, which indexers it has switched off after failures. Nothing is searched, grabbed or changed.
| Alert | When | Severity |
|---|---|---|
| It says it is unwell | One of the application's own health checks is a warning or an error (a download client that cannot be reached, a root folder gone, indexers unavailable…). The alert quotes it. A notice — an update being available — is shown on the page, not raised | warning |
| Downloads stuck | Downloads that have been in the queue for two hours or more and are judged warning or error by the application, have failed, or are blocked from importing. One alert for all of an application's, naming the first three, how long the oldest has been there, and the reason it gives | warning |
| Disk nearly full | A disk the application uses has under 10% free, critical under 5%. Folders on one disk count as one (the same size and free space), and a disk under 2 GB is not a media disk | warning; critical under 5% |
Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). A finding is raised under a key made from the check and its words with the numbers left out, so "3 indexers unavailable" becoming "4" is the same alert, and different words are a new one.
Game servers (Fragnet)
An integration of type Fragnet game servers (INTEGRATIONS.md) is one account on Fragnet's game panel (a Pterodactyl fork), read every five minutes, and Infrastructure → Game servers shows its servers: the game, every address and port, whether it runs, memory and disk against the plan, the newest backup, and (opened) the backups, schedules and SFTP address. Operators can start, stop, restart and kill a server there.
| Alert | When | Severity |
|---|---|---|
| Backup late | The server's plan allows backups and its newest successful backup is older than warn when the newest backup is older than (7 days by default; 0 turns it off), or it has none and the app has known the server for that long. A backup that failed does not count; the message says when the newest attempt failed. A server can be left out by name | warning |
| Disk nearly full | The server has used 90% of its disk (warn when disk or memory is fuller than); critical from 98% — a full disk stops the game saving | warning; critical from 98% |
| Memory nearly full | The server uses 90% of its memory limit while it runs — past the limit the host stops the game | warning |
A server that is stopped is not an alert (game servers are often stopped on purpose), and one that is suspended or being installed is not judged. "Since when" a server has been in its state is when the app first saw it in a read. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). /metrics has the running state, disk and memory against their limits and when the newest backup finished.
AdGuard Home
An integration of type AdGuard Home (INTEGRATIONS.md) is read every two minutes, and Network → AdGuard Home shows whether DNS answers (a real question over UDP, with how long it took), whether protection is on, the queries and how many were blocked, the names blocked most and the clients asking most, and its filter lists. Nothing is changed.
| Alert | When | Severity |
|---|---|---|
| DNS server not running | AdGuard Home says its DNS server is not running — nothing that uses it can look up a name | critical |
| DNS not answering | The test question has gone unanswered (or been answered with a server failure or a refusal) in every read for ten minutes. "No such name" counts as an answer | critical |
| Protection off | Protection has been switched off for an hour — someone paused it and forgot; nothing is being blocked | warning |
| Filter lists stale | Lists that are switched on have not been updated for a week (or never) while filtering is on — one alert for all of them | warning |
"Since when" is when the app first saw it in a read. Maintenance on the integration holds the alerts; a reading older than ten minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). /metrics has whether it runs, protects and answers, how long an answer took, and the queries and blocked counts.
Technitium DNS
An integration of type Technitium DNS (INTEGRATIONS.md) is read every two minutes, and Network → Technitium DNS shows whether DNS answers (a real question over UDP, with how long it took), whether blocking is on, how long the server has run, the last day's queries and how they ended, the clients and names asked about and blocked most, the cache against its limit, the block lists and when they are next updated, the zones that have a problem and the DHCP leases. Nothing is changed. It does not report the CPU, memory or disk of the machine it runs on — Technitium's API has none; a server agent does.
| Alert | When | Severity |
|---|---|---|
| DNS not answering | The test question has gone unanswered (or been answered with a server failure or a refusal) in every read for ten minutes. "No such name" counts as an answer | critical |
| Zone expired | A secondary zone has expired — it no longer answers. One alert for all the zones with a problem; it is critical if any of them has expired | critical |
| Zone cannot sync | Secondary zones that cannot sync with their primary and are going stale (none has expired yet) | warning |
| Blocking off | Blocking has been switched off for an hour while block lists are set up — someone paused it and forgot; nothing is being blocked. With no block lists, off is how it is and nothing is said | warning |
| Block lists overdue | The block lists' next update was due more than six hours ago and has not happened | warning |
The server's own internal zones and zones that are switched off are not judged. "Since when" is when the app first saw it in a read. Maintenance on the integration holds the alerts; a reading older than ten minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). /metrics has whether it answers and how long an answer took, whether blocking is on, how long it has run, the last day's counts, the cache, the zones with a problem and the DHCP leases.
Grafana
An integration of type Grafana (INTEGRATIONS.md) is read every five minutes, and Monitoring → Grafana shows which alert rules are firing, pending or cannot be evaluated, and how each data source tests. Nothing is silenced or changed.
| Alert | When | Severity |
|---|---|---|
| Rule firing | A Grafana-managed alert rule is firing (a pending one is only shown). The alert names the rule, its folder and group, since when, and its summary | critical when the rule's severity label is critical or page, else warning |
| Rule cannot be evaluated | Grafana reports the rule in an error state — an alert that cannot alert is worse than a quiet one — with the error | warning |
| Data source failing | A data source fails Grafana's own health check. One whose type has no check is not tested, never failing | warning |
Grafana can notify about its own alerts as well; with both switched on you hear twice. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed. /metrics has the rules, those firing, those in error and the failing data sources.
App versions
Operations → App versions lists each connected application whose version is known, with the newest release of it and how far behind it is: a patch (the third number), a minor update (the second), a major version (the first; for Authentik, whose versions are dates, a new year). A build number after the third number is ignored, and prefixes (v, n8n@, version/) are taken off. What is newer here than the newest release (a beta) is ahead.
The newest release is asked of GitHub (GET /repos/{owner}/{repo}/releases/latest, which leaves out pre-releases and drafts) once a day, and only for an application whose version has been read — and only while an administrator has switched the lookup on. It sends nothing about you or your systems but the name of the repository, and is the only thing this app asks of an address that is not yours. GitHub limits requests without a sign-in (60 an hour); a lookup that is refused keeps the release it had, says why on the page and is tried again at the next pass.
| Alert | When | Severity |
|---|---|---|
| App behind its newest release | A newer release has been out for the number of days set (Warn when a newer release has been out for … days; 0, the default, never) and the application has not caught up. The alert says which release, how long, and whether it is a patch, a minor update or a new major version | warning |
Maintenance on the integration holds the alert. It is under Updates on the Alerts page. /metrics has homelab_app_update_available (1 if a newer release is out) for each application, while the lookup is on.
Speedtest
An integration of type Speedtest Tracker (INTEGRATIONS.md) is read every ten minutes, and Monitoring → Speedtest shows the newest test, the middle of the last three and charts of download, upload and ping over up to 90 days. This app never starts a test: Speedtest Tracker runs them on its own schedule.
| Alert | When | Severity |
|---|---|---|
| The internet is slow | The middle of the newest three completed tests is under the share (60% unless you set another) of what you pay for — download and upload each judged only if you gave its number | warning |
| Speed tests failing | The newest three tests that completed or failed all failed — the internet is down, or Speedtest Tracker cannot reach Ookla | warning |
The middle (the median) of three is judged so that one test that ran while somebody was streaming does not raise an alert. Without the download you pay for and the upload you pay for (on the integration) nothing is judged slow — a speed alone says nothing — and only failing tests raise an alert. Maintenance on the integration holds the alerts; a reading older than 45 minutes is not believed. Results are kept 90 days.
NetBox
An integration of type NetBox (INTEGRATIONS.md) is read every five minutes, and Network → NetBox shows its workers, what it holds and — the other half — which of your servers are not in it.
| Alert | When | Severity |
|---|---|---|
| No background worker | /api/status/ says no RQ worker is running — webhooks, scripts, jobs and reports are not run, although the pages load |
warning |
The list of servers with no address in NetBox is not an alert: it is the app's servers (from what their agents report) against NetBox's addresses, leaving out Docker bridge addresses and the other ranges excluded on the Consistency page. A server counts as missing only if it has addresses and none is in NetBox.
n8n
An integration of type n8n (INTEGRATIONS.md) is read every five minutes, and Automation → n8n lists the workflows with their last run and the failures of the last day.
| Alert | When | Severity |
|---|---|---|
| Workflow failed | A workflow that is switched on failed (or crashed) on its last run, in the last week. One alert per workflow; it clears when the workflow next succeeds | warning |
A workflow that is switched off says nothing, and what each workflow last did is judged from n8n's newest 250 runs. If n8n gives its workflows but not its runs, the last outcomes are unknown and nothing is claimed. Maintenance on the integration holds the alerts.
Immich
An integration of type Immich (INTEGRATIONS.md) is read every five minutes, and Apps → Immich shows the photo and video counts, what the library takes, the disk under it and each background job queue with how many failed. Photos cannot be made again, and Immich fails quietly:
| Alert | When | Severity |
|---|---|---|
| Disk under the library filling | 90 % full | warning; critical from 95 % |
| Background jobs failed | 10 or more failed jobs, all queues together; the three worst queues are named | warning |
Failed jobs stay counted until they are run again or cleared in Immich (Administration → Jobs), so that alert stays until you do. A queue with a backlog of waiting jobs is not an alert — a big import has one for days. Maintenance on the integration holds them. A reading older than 20 minutes is not believed.
Paperless-ngx
An integration of type Paperless-ngx (INTEGRATIONS.md) is read every five minutes, and Apps → Paperless shows the documents and the inbox, the disk, Paperless's own system status and the imports that failed in the last week.
| Alert | When | Severity |
|---|---|---|
| Database, Redis or the worker in error | The system status says ERROR — a scan dropped in the folder just sits there when the worker is down |
critical |
| Index or classifier in error | The system status says ERROR |
warning |
| Sanity check found problems | The system status says WARNING for the sanity check |
warning |
| Documents failed to import | One or more unacknowledged failures in the last week; the files and the latest error are named | warning |
| Disk filling | 90 % full | warning; critical from 95 % |
A classifier that has not been trained yet reports a warning in Paperless and is not shown as a problem. The system status needs a superuser's token; with another, the page says it is unknown and only the failed imports and the counts are judged. Maintenance on the integration holds the alerts.
Prometheus
An integration of type Prometheus (INTEGRATIONS.md) is read every two minutes, and Monitoring → Prometheus shows its targets by job (how many up, how many down), the ones not answering and the alerts its rules have firing. It makes what you already measure there part of the same list, with the same notifications, silencing and incidents as the rest.
| Alert | When | Severity |
|---|---|---|
| Targets not answering | One or more targets of a job have been down for five minutes. One alert per job, naming up to three instances and the first scrape error | warning |
| Alert firing | One of Prometheus's rules is firing. One alert per alert name ("DiskFull (3 firing)"), with the first summary | critical if any has severity="critical" (or page), else warning |
Pending alerts, info / none severities and the heartbeats every pipeline carries (Watchdog, DeadMansSwitch, InfoInhibitor) are not shown as problems. Maintenance on the integration holds them. A reading older than ten minutes is not believed. Alerts that
Grafana manages itself are not in Prometheus's list; to see one here, write it as a Prometheus rule instead. Nothing from Prometheus is exported back to /metrics.
Authentik
An integration of type Authentik (INTEGRATIONS.md) is read every five minutes, and Apps → Authentik shows its version, workers, each outpost (and whether it is reporting), the certificates and API tokens that run out — soonest first — and the failed sign-ins of the last day. Authentik is the front door to everything behind it, so what is raised early is what would shut it:
| Alert | When | Severity |
|---|---|---|
| Outpost has not reported in | An outpost that serves applications had no instance report within five minutes | critical |
| Outpost runs another version | An instance runs a version other than the one Authentik expects | warning |
| Certificate runs out | Within 14 days (a certificate Authentik renews itself is left out) | warning; critical within 3 days and after |
| API token or app password runs out | Within 7 days | warning; critical within 2 days. The token this app reads Authentik with counts too |
| No worker connected | The workers endpoint answers with none | warning |
| Failed sign-ins | 20 or more in the last day | warning |
| Configuration errors | 3 or more in the last day, with the newest message | warning |
Maintenance on the integration holds them. A reading older than 20 minutes is not believed: nothing is raised or cleared from it, and the Alerts page says Authentik could not be checked. A token that does not expire, a verification token (they live for minutes) and an outpost with no applications are never alerts.
Jellyfin
An integration of type Jellyfin (INTEGRATIONS.md) is read every two minutes, and Apps → Jellyfin shows whether it is up, who is streaming right now (and whether that is transcoded, which is what loads the machine), the scheduled tasks that failed, and a chart of the most that were streaming at once. It raises three warnings: a restart is wanted, a scheduled task failed, 10 or more sign-ins failed in a day (the last is the one to look at if the server is open to the internet). Maintenance on the integration holds them.
Its other use is not cutting off a film. When the Jellyfin runs on a server the app knows, an update run on that server asks first if people are streaming (the update drawer says who; Update anyway goes ahead), and a patch plan waits for them for up to two hours before it updates that server. Only a recent reading counts, so a Jellyfin that cannot be read never holds an update up. The match is by address: the Jellyfin's address must be the server's IP address or host name as the agent reports it (a Jellyfin behind a proxy on another machine is not matched).
Docker containers that are broken
If you use Dockhand, the health pass (every 15 minutes) also reads each container's state and status text — the same text the Docker page shows — and raises an alert (category Failing services, source Docker) for:
| What | When | Severity |
|---|---|---|
| Restart loop | Docker says the container is restarting ("Restarting (127) 39 seconds ago") | critical |
| Unhealthy | The container's own health check fails ("Up 6 days (unhealthy)") | warning |
| Stopped unexpectedly | It is exited or dead, and it was running at the last look, or it has never been seen and stopped within the last day (from "Exited (0) 15 minutes ago") | warning |
| Never started | It was created and has stayed "Created" for an hour | warning |
| Docker host unreachable | Dockhand could not read one of its environments (a Hawser or Docker agent that is down) | warning |
The hard part is stopped: many containers are stopped on purpose (a stack that is switched off, a one-shot job, a watchtower that ran and exited). So a stop is only news when it was seen happen — a container that has been down all along (a stack
that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already
long stopped. The scanner containers Dockhand starts (dockhand-grype-*, dockhand-trivy-*) are always left alone.
(Settings → Notifications → What to send shows one group at a time — Servers and disks, Secrets, keys and domains, Backups and updates, Integrations and automation, DNS changes — and the settings search opens the group a setting is in.)
Leaving a container alone: the Leave alone button on the Docker page (administrators; Watch again undoes it), or the list under Settings → Notifications → What to send → Containers to leave alone — one per line, a name or host/name, * for anything.
Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with Warn when a
Docker container is broken. What the app has seen of each container is kept between looks (in settings, key _internal:containerMemory).
Critical vulnerabilities. Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads what Dockhand has already stored (never starting a scan, so it costs Dockhand nothing) and raises a warning for each Docker host whose scanned images have one or more critical findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (Vulnerabilities, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is unknown, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other.
High findings are never alerted unless you set a number under Also warn when a Docker host has this many high vulnerabilities (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. To switch it off: Settings → Notifications → What to send → Warn about critical vulnerabilities in Docker images (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. /metrics has homelab_docker_vulnerabilities (by severity) and homelab_docker_images_scanned for each host.
Endpoint checks and heartbeats
Endpoint checks (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds:
| Kind | What it asks |
|---|---|
http |
Fetches an http(s):// address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. 200-299, 301), and — if you ask — the page contains a keyword, and/or a value in a JSON answer is within its limit (see below). Follows up to 5 redirects when it has a keyword or value to look for. |
tcp |
Opens a connection to a host and port. |
tls |
Connects and reads the certificate; the answer is the days left. |
Per check: interval (1–1440 minutes, default 5), timeout (default 10 s), failures before an alert (default 2 in a row — one blip is not an alert), certificate warning days (default 14), an optional server it is linked to (maintenance on that server silences it, and the alert links to it), and accept a self-signed certificate. Each answer — time, result, message — is kept, with hourly and daily summaries for the charts and uptime figures.
- A check fails an alert (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A certificate within its warning days is its own alert (
endpoint-cert:…). - Checking a value, not just "is it up". A web check can read a number or a word out of a JSON answer and judge it. Say where it is —
power.now,state,items[2].name, or for a Prometheus querydata.result[0].value[1]([0]is the first place in a list;["a.b"]for a name with a dot in it) — and how it should compare: is below, is at most, is above, is at least, is equal to, is anything but, contains, or is there. A number given as text ("42.5", which is how Prometheus gives them) counts as a number; equality compares numbers as numbers and anything else as the same text, capital letters included. An optional unit (W,%) is shown after a number. The status is judged first, then the value: a value outside its limit is a failure that says so ("power.now is 5200 W, but should be below 5000 W"), and the alert reads Meter — power.now is 5200 W, but … rather than "isn't answering". It is also a failure if the answer isn't JSON, the path isn't in it ("it stops at power"), or the answer is more than 64 KB. The value read is remembered with each result: shown in the Answer column, charted under History… (as far back as results are kept, 14 days), in/metricsashomelab_endpoint_value(numbers only) and in the read API. Examples: Home Assistant's/api/states/sensor.power(pathstate), a Plex or Frigate status, an inverter's output, a Prometheus query (/api/v1/query?query=…). - A header, for an API that wants a token. A web check can send one header with its request (usually
Authorization: Bearer …). The value is stored encrypted whenCREDENTIALS_ENCRYPTION_KEYis set, is never sent back to the browser (the form says Saved — leave blank to keep it), and goes only to the address you gave — not to another host a redirect leads to. - What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine.
- Apps in your containers. With Dockhand connected, an image the app recognises (Immich, Paperless-ngx, Nextcloud, Vaultwarden, Authentik, Gitea, Grafana, Prometheus, Jellyfin, n8n, ntfy, Gotify, Nginx Proxy Manager, Open WebUI, Ollama, SearXNG, Wiki.js,
BookStack, Homebox, Semaphore, Meilisearch, and a few dozen with just a web page) is offered as a check on that app's own health address, with what a good answer looks like set up as a value check: Immich's
/api/server/pingmust sayres=pong, Nextcloud's/status.phpmaintenance=false, Gitea's/api/healthzstatus=pass, Grafana's/api/healthdatabase=ok, Prometheus's/-/healthymust contain "Healthy". It is asked on the Docker host's address — the server the app knows under the Docker environment's name in Dockhand, first usable IPv4 — at the port the container publishes (from Dockhand; if Dockhand does not say, the one the host's agent sees listening that the app is known to use). Only running containers, only hosts the app knows, and only what no check covers already is offered; where an app is recognised the plain "something answers on 8080" suggestion for that port is replaced by it. The paths are those the projects publish for the purpose, not tried against live instances here: a wrong one is a failing check you can edit. Uptime Kuma's monitors are not looked at, so something you already watch there may be offered — Not for me stops it. - The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under Endpoints → Suggested, and proxied hosts Nginx Proxy Manager serves.
Heartbeats (Monitoring → Heartbeats) are the other direction: a cron job or script calls an address when it finishes, and the app alerts when it does not. A heartbeat has a period and a grace; it is
late when both have passed since the last call (a warning), and a call to /fail is critical at once. Details and examples: HEARTBEATS-AND-CALENDAR.md.
Both are judged every minute, with the same remember-and-compare bookkeeping as the other checks.
Integrations
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are read by the app, using credentials an administrator enters under Integrations; what each needs is in INTEGRATIONS.md. Every call is recorded in the Diagnostic Log.
- An integration is "failing" when its last 3 calls in a row (Settings → Notifications) failed. Failures are counted per kind of service (all Proxmox integrations share a streak), in memory, so a restart starts the count again. A down notification goes out at the threshold and a recovered one on the next success. The moments are written to the audit log, and they feed the uptime record on the integration's page.
- When an integration cannot be read, what it reported is held, not cleared. A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check. The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (20 minutes; a UPS, 5).
- An integration can explain other alerts. A service that stops answering because the integration it needs is failing is folded under it (Dependencies).
Forecasts
A forecast says "this will be full in N days" from a straight line through the recent history, and is deliberately cautious:
- It needs at least six readings covering at least three days, and a steady climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that wobbles, is not forecast. A slope under 0.05 points a day counts as flat.
- It looks at the last two weeks; if those are flat it looks at the last ninety days, so a slow, steady climb is seen too.
- A disk that is already over its limit is left to the ordinary "nearly full" alert.
- It is shown as an alert when it falls within the window (Settings → Notifications → Warn when a disk will fill within, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
homelab_server_disk_days_to_full/homelab_storage_days_to_fullin/metrics. It is not sent as a notification the moment it appears. - Servers: from the agents' history. Proxmox storages and Synology volumes: the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
Dependencies: one outage, one alert
If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a graph of what needs what:
- a server runs on a host (set by hand, or worked out from its Proxmox node);
- a UniFi device is connected through another, and a server is plugged into a switch or access point (matched by IP address);
- dependencies you write (Map → Dependencies: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out.
When something is down and what depends on it is down too, the dependants' alerts are folded under the first: shown nested, marked caused by …, not announced on their own, and the host's message says "— also takes down web and db". If the host comes back and a dependant stays down, it is announced then, as new: an outage is never swallowed.
What happens to an alert
Several things can quiet an alert. They are different, and it matters which you use:
| Promise about the thing? | Hides it? | Notifications | Ends | |
|---|---|---|---|---|
| Acknowledge ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in /metrics; leaves the needs attention count |
Stops escalation | When you choose (an hour … 30 days), or when the alert has been gone half an hour |
| Snooze ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; one reminder at the end if it is still there | At the time you set (up to 90 days), or earlier with Wake now |
| Close ("dealt with, or accepted") | Yes — with who and why, and the reason is written on the incident | Yes, held back like a snooze | Raise/clear messages held; one message only if it gets worse | Never by time: when it gets more serious than it was, when it has been gone half an hour, or with Open again |
| Maintenance window | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked silenced | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as new |
| Folded under a host | — automatic | Nested under its cause | Not announced on its own | When the cause ends |
| Flapping | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm |
Who hears about it
Notifications go to the channels that are on (Gotify, ntfy, email, a webhook, push to phones and browsers), narrowed by routing rules (first match wins: by kind, by severity, by the server's tags) and each browser's own wishes. Quiet hours hold messages and send one digest at the end of the window (critical alerts can be let through); Away mode does the same for a stretch you set, "until Sunday". Every message the app can send, and what toggles it, is in NOTIFICATIONS.md. Event webhooks are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included (WEBHOOKS.md).
Flapping
An alert raised three times within an hour (Settings → Notifications; 0 = off) is called flapping: one message says so, its further raise and clear messages are held, and it is marked on the page. After an hour with no raise or clear, one settled message says whether it is still there. The alert itself is untouched: still raised, still counted. Alerts → Noise lists what keeps coming back.
Escalation
An alert raised for a while with nobody on it can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes.
Maintenance
Maintenance (a window of 5 minutes to a week, started by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under maintenance with it. A silenced problem is not remembered as known, so one still there when the window ends alerts then, as new.
Incidents
Every time an alert was raised and later went away is an incident: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its
timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts every five minutes whether or not anyone has
the page open; the same alert coming back is a new incident. Resolved incidents are kept 400 days. They are on Alerts → History, in the read API (/api/v1/incidents) and in the
monthly report.
What changed before an incident
Each incident page has a What changed before this? card (and GET /api/v1/incidents/:id/before): the things the app knows happened in the 30 minutes, 2, 6 or 24 hours before the alert began,
ranked. The sources are the audit log (settings, DNS, dependencies, integrations, a server being changed, a neighbour going offline or filling up), maintenance windows that began or ended, update runs and patch
plans, other alerts raised first, and a step in the server's own figures (a disk that jumped, memory that fell away).
Each is scored from three things:
| How it counts | |
|---|---|
| How related | The same thing the incident is about counts most; then something it needs (the host it runs on, the switch it is plugged into, a dependency you wrote — the nearest count more); then a change that touches everything (a setting, DNS); then something that needs it; and last something with no link the app knows of. |
| What kind | A maintenance window ending or a host going down is likelier than a change of settings, which is likelier than a note being written. Logins, notes, runbooks, tags and the like are not listed at all. A setting that bears on this kind of alert (thresholds, routing) counts for more. |
| How close | A change five minutes before counts for much more than one two hours before — the weight falls off over about an hour and never reaches zero inside the window. |
They are shown as Likely (55 and over), Possible (25 and over) and Background (the rest, behind a button); under 8 is left out, and at most ten are shown. The audit entry "pve1 stopped reporting" and the alert about it are one event and shown once. It is a list of suspects, not a diagnosis: it says what happened and when, not why, and a change made on the machine itself — a package installed by hand, a config file edited — is not something the app can see.
History and how long it is kept
| What | Kept | Grain |
|---|---|---|
| A server's load, memory and each disk | 35 days of samples (one every 10–15 minutes), 120 days of hourly summaries, ~800 days of daily summaries | summaries carry the lowest and highest |
| Endpoint check answers | 14 days of every answer, hourly summaries for 120 days, daily summaries for years | |
| Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day |
| Proxmox storage and Synology volume use | 400 days, one reading an hour | |
| Home Assistant watched numbers | 35 days, one reading every few minutes | |
| A UPS's charge, load and mains voltage | 35 days, one reading every five minutes (every minute on battery) | |
| Incidents | 400 days after they resolve | |
| Messages sent to event webhooks | The last 100 per webhook, none older than 14 days — see WEBHOOKS.md | |
| The audit log and the Diagnostic Log | Until automatic purging is switched on under Settings → Data (then 90 days by default, configurable) |
A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there.
Is it being trimmed? Settings → Data → Storage measures the database, its backups and sessions, and every table by the room it takes, and for each history says how far back it reaches against how long it is meant to be kept: trimmed (at its limit), filling up (not yet old enough to be trimmed), not trimmed (it holds far more than it should, so the trimming is not running or has not caught up) or no limit (the logs, until purging is switched on). The last two are listed at the top as needing a look. Nothing there changes anything.
Where to see it
| Shows | |
|---|---|
| Dashboard, My work, the menu badge | What needs attention now; what is yours |
| Alerts (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
| A server's page | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
| Endpoints, Heartbeats, Proxmox, Synology, Network devices, Home Assistant, UPS, Proxy hosts | Each source's own view |
| Service levels | Each target and what is left of this month's error budget |
| Weekly digest and monthly, quarterly and annual reports | The week, the month, the quarter and the year in one message |
/metrics → Prometheus, Grafana |
PROMETHEUS-GRAFANA.md |
Read API (/api/v1) |
API.md |
| Event webhooks | WEBHOOKS.md |
| Status page, wall display, calendar feed | Read-only views for others; made under Settings → Share links |
Tuning, in questions and answers
I am about to reboot a server. Start maintenance on it (Maintenance page, or Silence on an alert): its alerts are held and, if it is still down at the end, announced as new.
An alert is noisy because it keeps coming and going. Let it flap (it will be damped after three raises an hour), then look at Alerts → Noise — it says what usually helps: a longer grace, a different limit. For your own rules, raise the limit or lengthen the time.
One server legitimately runs at 92% disk. Give that server (or that one disk) its own limit — or never alert about this disk — in the server's Settings section.
I want to be reminded about something later, not silence it. Snooze it.
I have dealt with it, or I accept it, and I do not want to be reminded. Close it (Close… on the Alerts page, with a reason). It stays closed until it gets worse than it was, or the problem goes away for half an hour and returns.
I do not want to be woken for warnings. Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours.
A job on a server has to run every night. Make a heartbeat for it and call it from the end of the script.
I want my own limit, such as "memory above 85% for an hour". Add an alert rule.
An integration's alerts disappeared. Check Couldn't check everything on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why.
The app restarted and nothing about servers is alerting. For 20 minutes after a restart servers are not judged, so the agents can report first.
I want these figures in my own monitoring. /metrics for Prometheus, the read API for scripts, event webhooks to be told.
What monitoring does not do
- It does not fix anything. The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.)
- It does not see inside a server that has no agent — only what an endpoint check, a heartbeat or an integration can say about it.
- It is not a replacement for a second pair of eyes on the app itself. If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on
up{job="homelab-manager"}(PROMETHEUS-GRAFANA.md), or a check from another tool againstGET /health. - It is as fresh as its sources. A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.