# How monitoring works This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the "why did it do that?" page. Setting things up is in the other pages ([SSO](SSO.md), [API](API.md), [Prometheus and Grafana](PROMETHEUS-GRAFANA.md), [webhooks](WEBHOOKS.md), [heartbeats](HEARTBEATS-AND-CALENDAR.md)); the list of every notification is [NOTIFICATIONS.md](../NOTIFICATIONS.md). - [The big picture](#the-big-picture) - [What is watched, and how often](#what-is-watched-and-how-often) - [Servers and the agent](#servers-and-the-agent) - [From a reading to an alert](#from-a-reading-to-an-alert) - [The built-in checks](#the-built-in-checks) - [What is normal for the time](#what-is-normal-for-the-time) - [Service levels](#service-levels) - [Your own alert rules](#your-own-alert-rules) - [Duplicati backups](#duplicati-backups) - [Nextcloud](#nextcloud) - [Hetzner Cloud](#hetzner-cloud) - [Game servers (Fragnet)](#game-servers-fragnet) - [Media managers](#media-managers-sonarr-radarr-prowlarr) - [Download clients](#download-clients-qbittorrent-sabnzbd) - [App versions](#app-versions) - [AdGuard Home](#adguard-home) - [Technitium DNS](#technitium-dns) - [Grafana](#grafana) - [Speedtest](#speedtest) - [NetBox](#netbox) - [n8n](#n8n) - [Immich](#immich) - [Paperless-ngx](#paperless-ngx) - [Prometheus](#prometheus) - [Authentik](#authentik) - [Jellyfin](#jellyfin) - [Docker containers that are broken](#docker-containers-that-are-broken) - [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats) - [Integrations](#integrations) - [Forecasts](#forecasts) - [Dependencies: one outage, one alert](#dependencies-one-outage-one-alert) - [What happens to an alert](#what-happens-to-an-alert) — notify, acknowledge, silence, snooze, flapping, escalate - [Incidents](#incidents) - [History and how long it is kept](#history-and-how-long-it-is-kept) - [Where to see it](#where-to-see-it) - [Tuning, in questions and answers](#tuning-in-questions-and-answers) - [What monitoring does not do](#what-monitoring-does-not-do) ## The big picture ``` agents ─────────┐ integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · close · silence endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines) your own rules ─┘ ├──► incidents (the record) and history └──► /metrics and the read API ``` Three ideas run through everything: 1. **A check turns readings into *conditions*.** A condition is "this specific problem exists right now" and has a stable **key** (`offline:server:3`, `disk:server:3:/`) — the same problem always has the same key. 2. **Alerts are announced on a change, not on a schedule.** A problem is announced **once** when it appears and **once** when it ends. It is not repeated every time it is looked at. 3. **Silence is never "all clear".** When something cannot be *read* (an integration is down, a server's agent is quiet), what depended on it is **held exactly as it was** — neither raised nor cleared — and the Alerts page lists it under *Couldn't check everything*. The app never turns "I couldn't look" into "nothing is wrong". ## What is watched, and how often | What | How it gets to the app | How often | |---|---|---| | **Servers** (Linux, Windows) | The **agent** on each server **pushes** a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects *to* a server. | Every **15 minutes** by default (set when the agent is installed) | | **Proxmox**, **Synology** | The health pass **reads** them: node and storage usage, volume and disk status. | Every **15 minutes** | | **UniFi** | A sync **reads** devices, networks, Wi-Fi names and clients and keeps them. | Every **5 minutes** | | **Home Assistant** | A sync **reads** the entities you chose to watch and keeps their state and a history. | Every **5 minutes** | | **UPS** (NUT or SNMP) | A sync **reads** each UPS and keeps its state and a history. | Every **minute** | | **Nginx Proxy Manager** | **Read** when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute | | **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** | | **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due | | **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** | | **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications | | **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time | | **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass | | **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** | After the app **restarts**, server-based checks are paused for **20 minutes** so agents get a chance to report before any server is judged. ## Servers and the agent The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md). **When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts. While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all. **Per-server settings.** A server's *Settings* section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for one disk or "never alert about this disk", and services never to count as failed (names or patterns such as `gupdate*`). Whatever it does not replace follows the global setting. **What the report is used for:** the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support check, pending updates, the dependency map, and `/metrics`. ## From a reading to an alert There are two paths, designed so they cannot disagree: **1. The state-based checks (the notifications).** Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the **list of conditions that exist right now**. The app compares that list with the one it remembered from last time: - **new** → announced (a notification, an event webhook, a row in the incident record); - **gone** → announced as resolved; - **still there** → nothing; - **its source could not be read this time** → carried over untouched (not resolved, not repeated). The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all. **2. The Alerts page.** It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), **live**, and turns what they find into one flat list. It ignores the per-event on/off toggles — the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a 20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under *Couldn't check everything*. Each alert has: | | | |---|---| | **Severity** | `critical`, `warning` or `info`. Things that are *gone* (offline, a failing SMART disk, a disk over 95%) are critical; things that are *coming* (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. | | **Category** | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring`, `tickets` — what the filter on the page and the [routing rules](#who-hears-about-it) work on. | | **Source** | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … | | **Target** | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. | | **Key / id** | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. | ## The built-in checks Every threshold here is a default; where it can be changed, the place is named. | Check | Raised when | Default | Changed in | |---|---|---|---| | **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server | | **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk | | **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server | | **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) | | **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable | | **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable | | **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable | | **Unusual for the time** | CPU load or memory use over the last hour is far above what is normal for this hour of this weekday ([below](#what-is-normal-for-the-time)) | on · warning | Settings → Notifications | | **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card or marked as expected on **Network → Ports** (which lists every server's new ports in one place, with *Show only new*, *Mark as expected* and *Mark all as expected*) (the container that publishes it, or else a port preset, names it in the message: "8096/tcp jellyfin (container)", "3389/tcp RDP") | on | Settings → Notifications | | **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications | | **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications | | **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page | | **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server | | **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications | | **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications | | **Docker updates** | A container image has a newer version | | Settings → Notifications | | **Nextcloud can't be used / disk low** | The database needs upgrading (critical), maintenance mode is on, or little room is left for the data (critical under 5 GB) | on, with the integration | Nextcloud page; [below](#nextcloud) | | **Hetzner server off / traffic / rescue / backup** | A Hetzner Cloud server that was running is off, is stuck migrating or rebuilding, has used 80% of its month's traffic (critical at 100%), has rescue mode left on, or has backups enabled but none recent | on, with the integration | Hetzner page; [below](#hetzner-cloud) | | **Sonarr / Radarr / Prowlarr** | An application reports a health warning or error, has downloads stuck for two hours, or a disk under 10% free | on, with the integration | Media managers page; [below](#media-managers-sonarr-radarr-prowlarr) | | **qBittorrent / SABnzbd** | Downloads stuck (an error, or stalled for a day), qBittorrent without a network for 15 minutes, SABnzbd paused for six hours with work waiting or downloading nothing for an hour, or a download disk nearly full | on, with the integration | Download clients page; [below](#download-clients-qbittorrent-sabnzbd) | | **AdGuard Home** | DNS has not answered for ten minutes, or its DNS server is not running (critical); protection left off for an hour, or filter lists not updated for a week | on, with the integration | AdGuard Home page; [below](#adguard-home) | | **Technitium DNS** | DNS has not answered for ten minutes, or a secondary zone has expired (critical); a zone cannot sync, blocking left off for an hour, or block lists overdue for an update (warning) | on, with the integration | Technitium DNS page; [below](#technitium-dns) | | **Grafana** | An alert rule is firing (critical when labelled critical or page), a rule cannot be evaluated, or a data source fails its health check | on, with the integration | Grafana page; [below](#grafana) | | **Game server backup / disk / memory** | A Fragnet game server's newest backup is older than the limit (7 days) or none, its disk is over 90% of the plan (critical from 98%), or its memory is over 90% while it runs | on, with the integration | Game servers page; [below](#game-servers-fragnet) | | **Internet slow / speed tests failing** | The middle of the last three speed tests is under the share you set of what the line is sold at (only with those numbers given), or the last three tests all failed | on, with the integration | Speedtest page; [below](#speedtest) | | **NetBox has no worker** | No background worker is running | on, with the integration | NetBox page; [below](#netbox) | | **n8n workflow failed** | A workflow that is switched on failed on its last run | on, with the integration | n8n page; [below](#n8n) | | **Immich disk filling / jobs failing** | The disk under the library is 90 % full (critical at 95 %), or 10 or more background jobs have failed | on, with the integration | Immich page; [below](#immich) | | **Paperless-ngx needs a look** | The worker, database or Redis is in error (critical), documents failed to import, the index or sanity check is in error, or the disk is 90 % full | on, with the integration | Paperless page; [below](#paperless-ngx) | | **Prometheus target down / alert firing** | A scrape target has not answered for five minutes, or one of Prometheus's own alert rules is firing | on, with the integration | Prometheus page; [below](#prometheus) | | **Authentik needs a look** | An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing | on, with the integration | Authentik page; [below](#authentik) | | **Jellyfin needs a look** | Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day | on, with the integration | Jellyfin page; [below](#jellyfin) | | **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) | | **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) | | **Docker image has critical vulnerabilities** | Dockhand's own scans of a Docker host's images found one or more *critical* findings (high, medium and low are only shown — unless you set a number for high ones, below) | on | Settings → Notifications; [below](#docker-containers-that-are-broken) | | **Docker host has many high vulnerabilities** | The same scans found at least the number of *high* findings you set (*Also warn when a Docker host has this many high vulnerabilities*; 0, the default, never) | off | Settings → Notifications; [below](#docker-containers-that-are-broken) | | **Docker host has old running images** | A Docker host has running containers whose images were built more than the number of days you set ago (*Warn about running Docker images that were built more than …*; 0, the default, never) — scanned or not | off | Settings → Notifications; [below](#docker-containers-that-are-broken) | | **App behind its newest release** | An application has had a newer release out for the number of days set under Operations → App versions (0, the default, never) — needs the lookup of releases switched on | off | App versions page; [below](#app-versions) | | **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable | | **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications | | **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check | | **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat | | **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page | | **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) | | **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) | | **Your own rules** | [Below](#your-own-alert-rules) | | Alert settings → Alert rules | ## What is normal for the time A fixed limit cannot say that a server is busy *for a Tuesday at 03:00*. So for each server's **CPU load** and **memory use** the app keeps a baseline for every hour of the week (168 of them, in the **notification time zone**): the **median** of that hour's averages over the previous weeks (up to ten; the hourly figures are kept four months), and a **robust spread** (the median absolute deviation, so one odd week does not stretch it). An hour needs **four weeks** behind it before it has a baseline at all; until then nothing is said. The hour in progress and hours with fewer than two samples are not counted. It is **quiet by design**. A reading is a warning only when **all** of these hold, judged on the last hour of samples (at least three, the newest under 30 minutes old): | | CPU | Memory | |---|---|---| | The last hour's average is at least | 50 % | 70 % | | …and above the usual by at least | 30 points | 15 points | | …and above it by at least | four spreads (the spread is taken as at least 4 points) | four spreads (at least 2 points) | | …and the lowest sample in that hour is still above the usual by | 15 points | 7.5 points | The last line is what separates a **state** from a **spike**: if the figure dropped back toward normal at any time in the hour, it is not said. Only a *high* reading is ever raised — low is never news. What it finds is an ordinary health condition (`baseline:cpu:server:`, `baseline:mem:server:`, *Monitoring*, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for a server that has stopped reporting (its offline alert says that). Switch it off with *Warn when CPU or memory is unusually high for the time* under Settings → Notifications → What to send → Servers and disks. The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure. ## Service levels **Alert settings → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and where today's pace would end the month, once a day has passed), and how each of the last six months went. How downtime is measured, so the figure can be trusted — and where it cannot: | Measured on | From | Precision | |---|---|---| | **Endpoint check** | The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. | A day's failed share is spread over the whole day | | **Server** | The moments the audit log recorded it stopping reporting and reporting again | To the minute; only from when the server was created | | **Integration** | The moments calls to that *kind* of service started failing and worked again | To the minute; only from when the app began recording them | **Planned maintenance is not taken out**: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing. When **80 %** of the budget is used it is a **warning**, and when it is overspent a **critical** alert (key `slo:`, *Monitoring*, source *Service level*): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks — and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in `/metrics` (`homelab_service_level_*`) and in the monthly, quarterly and annual reports ([below](#history-and-how-long-it-is-kept)). ## Your own alert rules Where the built-in checks do not fit, **Alert settings → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers: - **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature. - **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time. - **How serious:** warning or critical. **For which servers:** all of them, one, or those with a tag. - **Try it now** shows which servers it would be raised on, from current data, before you save it. A rule's finding is an **ordinary alert** with the key `rule::server:[:]`, in the *Monitoring* category: announced and resolved like the others, held back by maintenance on the server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at **when the server reports** (right after its report is stored — so a figure that has been past its limit long enough is raised within seconds of the report, not up to fifteen minutes later) and again on every 15-minute health pass, which is also what notices a rule that was deleted or switched off and what judges the servers that have not reported. "For 15 minutes" is still counted in the readings the reports build (one every 10–15 minutes), so it means the readings over 15 minutes, as before. A server that is offline or has not reported for 45 minutes is not judged. The check after a report looks at **that server's rules only** and changes nothing else in what is known: the built-in checks, the other servers and the integrations wait for the pass, and what the pass would say is the same. It is not done for a server looked at in the last minute, nor during the 20-minute grace after the app starts, and it is never run at the same moment as a pass (one waits for the other). What is *unusual for the time* (the baselines) still waits for the pass. ## Duplicati backups Duplicati tells this app how each backup went ([INTEGRATIONS.md](../INTEGRATIONS.md#duplicati) says how to set it up), and **Backups → Duplicati** shows every job: when it last succeeded, how the last run ended, how long it took, how big the source is, how many files changed, and where it stands. | State | Meaning | Alert | |---|---|---| | **In order** | The last run worked and it is not overdue | — | | **Warnings** | The last run finished with warnings (a locked file, say). It still counts as a backup | none | | **Failed** | The last run ended in `Error` or `Fatal` | warning; **critical from the second failure in a row** | | **Overdue** | It has not succeeded for longer than it is **expected** to — 2 days unless you set it per job (6 hours to 90 days) | warning; **critical at three times that** | The alert says which job, on which machine, why (the first error Duplicati reported) and when the last good backup was. Only the *Backup* operation counts: a compact, a repair or a verification does not say whether the data is safe. A job can be **muted** (no alert; it is still listed), and one that no longer exists can be **forgotten** (with its history; if Duplicati reports it again it is learned again). Each machine can be tied to a server, so **maintenance on that server silences its backups' alerts**. What is kept: the last result of each job, and each report for 400 days or 400 reports. `/metrics` has `homelab_duplicati_backup_ok`, `homelab_duplicati_last_success_age_seconds` and `homelab_duplicati_source_size_bytes` for each job. A job is only learned from its first report, so a job that has *never* reported is not known to be missing; and because the reports are pushed, **the app being down when a backup finishes** loses that report (the backup shows as overdue only once it is). ## Nextcloud An integration of type *Nextcloud* ([INTEGRATIONS.md](../INTEGRATIONS.md#nextcloud)) is read every five minutes, and **Apps → Nextcloud** shows its version, whether it is usable, and — with an administrator's app password — the room left for the data, the users, the files and the app updates waiting. | Alert | When | Severity | |---|---|---| | **Database needs upgrading** | status.php says `needsDbUpgrade` — nobody can use Nextcloud until `occ upgrade` is run | critical | | **Maintenance mode on** | status.php says `maintenance` — nobody can sign in | warning | | **Data disk low** | Less than 20 GB free where the data is kept (needs the login) | warning; critical under 5 GB | | **Background jobs not running** | Nextcloud's own *Cron last run* check says error — the last job ran more than an hour ago (needs the login; Nextcloud 28 or later) | warning | | **Nextcloud's own checks** | One or more of its other setup checks report an error; one alert naming up to three | warning | Maintenance on the integration holds them. A reading older than 20 minutes is not believed. Cron is read through Nextcloud's own setup checks (the Monitoring app has no field for it), so it needs the login and Nextcloud **28 or later**; before that it is *not known* and never counted as fine. Warnings from those checks (HTTP headers, a PHP setting) are shown on the page, not alerted. ## Hetzner Cloud An integration of type *Hetzner Cloud* ([INTEGRATIONS.md](../INTEGRATIONS.md#hetzner-cloud)) is one project, read every five minutes, and **Infrastructure → Hetzner Cloud** shows its servers (status, type, place, addresses, this month's traffic against what the type includes, whether backups are on), volumes, snapshots, and roughly what the project costs. Nothing is started, stopped or changed. | Alert | When | Severity | |---|---|---| | **Server off** | A server the app saw running is now off (Hetzner keeps billing a server that is off). One off for **a week**, or never seen running by the app, is left alone — as a stopped container is | warning | | **Stuck in a status** | A server has been starting, stopping, migrating, rebuilding or initializing for **half an hour** | warning | | **Traffic** | A server has sent **80%** of its included traffic for the month; at **100%** it is critical (the excess is billed) | warning; critical at 100% | | **Rescue mode left on** | Rescue mode has been switched on for **an hour** without a reboot — the next reboot (a kernel update, say) would come up in the rescue system and the server would not come back | warning | | **Backups late** | A server has backups enabled and the newest is older than the *backup older than* limit (48 hours by default — the same setting as for Proxmox guests; 0 turns it off), or none has been made and the server is older than that | warning | | **Traffic forecast** | A server is **on course to pass its included traffic before the month ends** (by the last week's rate, or the month's average after three days) while it has used under 80% — the alert says on which day. Not raised from 80% on: the traffic alert covers that | warning | | **Load balancer unhealthy** | A load balancer has a target Hetzner judges unhealthy; critical when none of its targets is healthy | warning; critical when all | | **Certificate ending** | A certificate that is in use ends within **14 days** (critical within 3, or past its end), or Hetzner could not issue or renew a managed one. A certificate nothing uses is not raised | warning; critical within 3 days | | **Floating IP unattached** | A floating IP points at no server — it is billed all the same | warning | Hetzner does not say when a server went off or when rescue mode was switched on, so "since when" is **when the app first saw it** in a read. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). Snapshots, volumes (including ones attached to no server, which are billed all the same) and the cost are shown, not alerted. **The cost** is an estimate from Hetzner's own price list, including VAT: the servers, their backups (a percentage of the server's price), volumes, snapshots, primary IPs, load balancers and floating IPs. It leaves out traffic over the quota, so it is a floor, not a bill. **The traffic forecast** is made from the month's outgoing traffic, written down about every six hours for two weeks: the rate of the last week (at least two days of it) or, with too little, the month's average so far (after three days). It assumes the month is the calendar month in UTC — not checked against a live project — and a counter that goes down starts the history over. The page shows under each server's traffic bar either the day the quota is passed or what the month is on course to come to. **Load balancers, floating IPs and certificates** are listed on the Hetzner page — a load balancer with how many of its targets are healthy, a floating IP with the server it points at, a certificate with when it ends (and the ones nothing uses) — and their prices are in the cost estimate. `/metrics` has each load balancer's unhealthy targets, each certificate's end and each server's forecast month. **On a server's page**, a *Hetzner Cloud* card appears when one of the project's servers is that server — matched by its public or private address, else by name — with its status, type, newest backup and traffic. A Hetzner server with no agent is listed as *not in the app's servers*. `/metrics` has `homelab_hetzner_server_running`, `homelab_hetzner_traffic_outgoing_bytes`, `homelab_hetzner_traffic_included_bytes` and `homelab_hetzner_monthly_cost`. ## Media managers (Sonarr, Radarr, Prowlarr) An integration of type *Sonarr*, *Radarr*, *Prowlarr*, *Lidarr* or *Readarr* ([INTEGRATIONS.md](../INTEGRATIONS.md#sonarr-radarr-and-prowlarr)) is read every five minutes, and **Apps → Media managers** shows, for each: what the application says is wrong with itself, its download queue (the stuck ones first, with why), its disks, how many wanted items have no file (Sonarr, Radarr) and, for Prowlarr, which indexers it has switched off after failures. Nothing is searched, grabbed or changed. | Alert | When | Severity | |---|---|---| | **It says it is unwell** | One of the application's own health checks is a **warning** or an **error** (a download client that cannot be reached, a root folder gone, indexers unavailable…). The alert quotes it. A *notice* — an update being available — is shown on the page, not raised | warning | | **Downloads stuck** | Downloads that have been in the queue for **two hours or more** and are judged warning or error by the application, have failed, or are blocked from importing. One alert for all of an application's, naming the first three, how long the oldest has been there, and the reason it gives | warning | | **Disk nearly full** | A disk the application uses has under **10%** free, critical under 5%. Folders on one disk count as one (the same size and free space), and a disk under 2 GB is not a media disk | warning; critical under 5% | Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). A finding is raised under a key made from the check and its words with the numbers left out, so "3 indexers unavailable" becoming "4" is the same alert, and different words are a new one. ## Download clients (qBittorrent, SABnzbd) An integration of type *qBittorrent* or *SABnzbd* ([INTEGRATIONS.md](../INTEGRATIONS.md#qbittorrent-and-sabnzbd)) is read every five minutes, and **Apps → Download clients** shows, for each: how fast it is downloading (and, for qBittorrent, uploading), the items in its queue and their states (the ones needing a look first), whether qBittorrent can reach the network or SABnzbd is paused, the room its download folder has, what failed in the last day (SABnzbd) and SABnzbd's own warnings. Nothing is added, paused, resumed or removed. | Alert | When | Severity | |---|---|---| | **Downloads stuck** | A download is in an **error** state (qBittorrent's `error` and `missingFiles`), or is **stalled** — nobody to get it from — with nothing having moved for **a day** (its last activity, or when it was added). One alert for all of a client's, naming the first three and why for the worst | warning | | **No connection** | qBittorrent has torrents loaded and has said `disconnected` in every read for **15 minutes** — typically a VPN it is bound to has dropped. `firewalled` (no incoming connections) is shown, not raised: it still downloads | warning | | **Paused and forgotten** | SABnzbd's whole queue has been paused for **six hours** with downloads waiting | warning | | **Downloading nothing** | SABnzbd is not paused, has downloads waiting, and its speed has been zero for **an hour** — a news server that does not answer or refuses the login | warning | | **Disk nearly full** | A download folder has under **10%** free (SABnzbd, which says how big its disks are), critical under 5%; or, for qBittorrent, which says only what is free, under **20 GB**, critical under 5 GB. A disk under 2 GB is not a download disk | warning; critical under 5% / 5 GB | "Since when" is what the app saw: the time of the first read that showed it, carried from read to read, so a restart of the app does not start the clock again. A failed download in SABnzbd's history is shown for the last day, not raised: failures are normal on Usenet, and Sonarr and Radarr try another release (they raise it themselves when downloads stay stuck — see [Media managers](#media-managers-sonarr-radarr-prowlarr)). Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). ## Game servers (Fragnet) An integration of type *Fragnet game servers* ([INTEGRATIONS.md](../INTEGRATIONS.md#fragnet-game-servers)) is one account on Fragnet's game panel (a Pterodactyl fork), read every five minutes, and **Infrastructure → Game servers** shows its servers: the game, every address and port, whether it runs, memory and disk against the plan, the newest backup, and (opened) the backups, schedules and SFTP address. Operators can start, stop, restart and kill a server there. | Alert | When | Severity | |---|---|---| | **Backup late** | The server's plan allows backups and its newest **successful** backup is older than *warn when the newest backup is older than* (7 days by default; 0 turns it off), or it has none and the app has known the server for that long. A backup that failed does not count; the message says when the newest attempt failed. A server can be left out by name | warning | | **Disk nearly full** | The server has used 90% of its disk (*warn when disk or memory is fuller than*); critical from 98% — a full disk stops the game saving | warning; critical from 98% | | **Memory nearly full** | The server uses 90% of its memory limit **while it runs** — past the limit the host stops the game | warning | A server that is **stopped** is not an alert (game servers are often stopped on purpose), and one that is suspended or being installed is not judged. "Since when" a server has been in its state is **when the app first saw it** in a read. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). `/metrics` has the running state, disk and memory against their limits and when the newest backup finished. ## AdGuard Home An integration of type *AdGuard Home* ([INTEGRATIONS.md](../INTEGRATIONS.md#adguard-home)) is read every two minutes, and **Network → AdGuard Home** shows whether DNS answers (a real question over UDP, with how long it took), whether protection is on, the queries and how many were blocked, the names blocked most and the clients asking most, and its filter lists. Nothing is changed. | Alert | When | Severity | |---|---|---| | **DNS server not running** | AdGuard Home says its DNS server is not running — nothing that uses it can look up a name | critical | | **DNS not answering** | The test question has gone unanswered (or been answered with a server failure or a refusal) in every read for **ten minutes**. "No such name" counts as an answer | critical | | **Protection off** | Protection has been switched off for **an hour** — someone paused it and forgot; nothing is being blocked | warning | | **Filter lists stale** | Lists that are switched on have not been updated for **a week** (or never) while filtering is on — one alert for all of them | warning | "Since when" is **when the app first saw it** in a read. Maintenance on the integration holds the alerts; a reading older than ten minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). `/metrics` has whether it runs, protects and answers, how long an answer took, and the queries and blocked counts. ## Technitium DNS An integration of type *Technitium DNS* ([INTEGRATIONS.md](../INTEGRATIONS.md#technitium-dns)) is read every two minutes, and **Network → Technitium DNS** shows whether DNS answers (a real question over UDP, with how long it took), whether blocking is on, how long the server has run, the last day's queries and how they ended, the clients and names asked about and blocked most, the cache against its limit, the block lists and when they are next updated, the zones that have a problem and the DHCP leases. Nothing is changed. It does not report the CPU, memory or disk of the machine it runs on — Technitium's API has none; a server agent does. | Alert | When | Severity | |---|---|---| | **DNS not answering** | The test question has gone unanswered (or been answered with a server failure or a refusal) in every read for **ten minutes**. "No such name" counts as an answer | critical | | **Zone expired** | A secondary zone has expired — it no longer answers. One alert for all the zones with a problem; it is critical if any of them has expired | critical | | **Zone cannot sync** | Secondary zones that cannot sync with their primary and are going stale (none has expired yet) | warning | | **Blocking off** | Blocking has been switched off for **an hour** while block lists are set up — someone paused it and forgot; nothing is being blocked. With no block lists, off is how it is and nothing is said | warning | | **Block lists overdue** | The block lists' next update was due more than **six hours** ago and has not happened | warning | The server's own internal zones and zones that are switched off are not judged. "Since when" is **when the app first saw it** in a read. Maintenance on the integration holds the alerts; a reading older than ten minutes is not believed (they are neither raised nor cleared, and the Alerts page says it could not check). `/metrics` has whether it answers and how long an answer took, whether blocking is on, how long it has run, the last day's counts, the cache, the zones with a problem and the DHCP leases. ## Grafana An integration of type *Grafana* ([INTEGRATIONS.md](../INTEGRATIONS.md#grafana)) is read every five minutes, and **Monitoring → Grafana** shows which alert rules are firing, pending or cannot be evaluated, and how each data source tests. Nothing is silenced or changed. | Alert | When | Severity | |---|---|---| | **Rule firing** | A Grafana-managed alert rule is firing (a pending one is only shown). The alert names the rule, its folder and group, since when, and its summary | critical when the rule's `severity` label is `critical` or `page`, else warning | | **Rule cannot be evaluated** | Grafana reports the rule in an error state — an alert that cannot alert is worse than a quiet one — with the error | warning | | **Data source failing** | A data source fails Grafana's own health check. One whose type has no check is *not tested*, never failing | warning | Grafana can notify about its own alerts as well; with both switched on you hear twice. Maintenance on the integration holds the alerts; a reading older than 20 minutes is not believed. `/metrics` has the rules, those firing, those in error and the failing data sources. ## App versions **Operations → App versions** lists each connected application whose version is known, with the newest release of it and how far behind it is: *a patch* (the third number), *a minor update* (the second), *a major version* (the first; for Authentik, whose versions are dates, a new year). A build number after the third number is ignored, and prefixes (`v`, `n8n@`, `version/`) are taken off. What is newer here than the newest release (a beta) is *ahead*. The newest release is asked of **GitHub** (`GET /repos/{owner}/{repo}/releases/latest`, which leaves out pre-releases and drafts) **once a day, and only for an application whose version has been read** — and **only while an administrator has switched the lookup on**. It sends nothing about you or your systems but the name of the repository, and is the only thing this app asks of an address that is not yours. GitHub limits requests without a sign-in (60 an hour); a lookup that is refused keeps the release it had, says why on the page and is tried again at the next pass. | Alert | When | Severity | |---|---|---| | **App behind its newest release** | A newer release has been out for the number of days set (*Warn when a newer release has been out for … days*; 0, the default, never) and the application has not caught up. The alert says which release, how long, and whether it is a patch, a minor update or a new major version | warning | Maintenance on the integration holds the alert. It is under *Updates* on the Alerts page. `/metrics` has `homelab_app_update_available` (1 if a newer release is out) for each application, while the lookup is on. ## Speedtest An integration of type *Speedtest Tracker* ([INTEGRATIONS.md](../INTEGRATIONS.md#speedtest-tracker)) is read every ten minutes, and **Monitoring → Speedtest** shows the newest test, the middle of the last three and charts of download, upload and ping over up to 90 days. This app never starts a test: Speedtest Tracker runs them on its own schedule. | Alert | When | Severity | |---|---|---| | **The internet is slow** | The middle of the newest three completed tests is under the share (60% unless you set another) of what you pay for — download and upload each judged only if you gave its number | warning | | **Speed tests failing** | The newest three tests that completed or failed all failed — the internet is down, or Speedtest Tracker cannot reach Ookla | warning | The *middle* (the median) of three is judged so that one test that ran while somebody was streaming does not raise an alert. Without *the download you pay for* and *the upload you pay for* (on the integration) nothing is judged slow — a speed alone says nothing — and only failing tests raise an alert. Maintenance on the integration holds the alerts; a reading older than 45 minutes is not believed. Results are kept 90 days. ## NetBox An integration of type *NetBox* ([INTEGRATIONS.md](../INTEGRATIONS.md#netbox)) is read every five minutes, and **Network → NetBox** shows its workers, what it holds and — the other half — **which of your servers are not in it**. | Alert | When | Severity | |---|---|---| | **No background worker** | `/api/status/` says no RQ worker is running — webhooks, scripts, jobs and reports are not run, although the pages load | warning | The list of servers with no address in NetBox is not an alert: it is the app's servers (from what their agents report) against NetBox's addresses, leaving out Docker bridge addresses and the other ranges excluded on the Consistency page. A server counts as missing only if it has addresses and none is in NetBox. ## n8n An integration of type *n8n* ([INTEGRATIONS.md](../INTEGRATIONS.md#n8n)) is read every five minutes, and **Automation → n8n** lists the workflows with their last run and the failures of the last day. | Alert | When | Severity | |---|---|---| | **Workflow failed** | A workflow that is switched on failed (or crashed) on its last run, in the last week. One alert per workflow; it clears when the workflow next succeeds | warning | A workflow that is switched off says nothing, and what each workflow last did is judged from n8n's **newest 250 runs**. If n8n gives its workflows but not its runs, the last outcomes are unknown and nothing is claimed. Maintenance on the integration holds the alerts. ## Immich An integration of type *Immich* ([INTEGRATIONS.md](../INTEGRATIONS.md#immich)) is read every five minutes, and **Apps → Immich** shows the photo and video counts, what the library takes, the disk under it and each background job queue with how many failed. Photos cannot be made again, and Immich fails quietly: | Alert | When | Severity | |---|---|---| | **Disk under the library filling** | 90 % full | warning; critical from 95 % | | **Background jobs failed** | 10 or more failed jobs, all queues together; the three worst queues are named | warning | Failed jobs stay counted until they are run again or cleared in Immich (Administration → Jobs), so that alert stays until you do. A queue with a backlog of *waiting* jobs is not an alert — a big import has one for days. Maintenance on the integration holds them. A reading older than 20 minutes is not believed. ## Paperless-ngx An integration of type *Paperless-ngx* ([INTEGRATIONS.md](../INTEGRATIONS.md#paperless-ngx)) is read every five minutes, and **Apps → Paperless** shows the documents and the inbox, the disk, Paperless's own system status and the imports that failed in the last week. | Alert | When | Severity | |---|---|---| | **Database, Redis or the worker in error** | The system status says `ERROR` — a scan dropped in the folder just sits there when the worker is down | critical | | **Index or classifier in error** | The system status says `ERROR` | warning | | **Sanity check found problems** | The system status says `WARNING` for the sanity check | warning | | **Documents failed to import** | One or more unacknowledged failures in the last week; the files and the latest error are named | warning | | **Disk filling** | 90 % full | warning; critical from 95 % | A classifier that has not been trained yet reports a *warning* in Paperless and is not shown as a problem. The system status needs a superuser's token; with another, the page says it is unknown and only the failed imports and the counts are judged. Maintenance on the integration holds the alerts. ## Prometheus An integration of type *Prometheus* ([INTEGRATIONS.md](../INTEGRATIONS.md#prometheus)) is read every two minutes, and **Monitoring → Prometheus** shows its targets by job (how many up, how many down), the ones not answering and the alerts its rules have firing. It makes what you already measure there part of the same list, with the same notifications, silencing and incidents as the rest. | Alert | When | Severity | |---|---|---| | **Targets not answering** | One or more targets of a job have been down for five minutes. One alert per job, naming up to three instances and the first scrape error | warning | | **Alert firing** | One of Prometheus's rules is firing. One alert per alert name ("DiskFull (3 firing)"), with the first summary | critical if any has `severity="critical"` (or `page`), else warning | Pending alerts, `info` / `none` severities and the heartbeats every pipeline carries (`Watchdog`, `DeadMansSwitch`, `InfoInhibitor`) are not shown as problems. Maintenance on the integration holds them. A reading older than ten minutes is not believed. Alerts that Grafana manages itself are not in Prometheus's list; to see one here, write it as a Prometheus rule instead. Nothing from Prometheus is exported back to `/metrics`. ## Authentik An integration of type *Authentik* ([INTEGRATIONS.md](../INTEGRATIONS.md#authentik)) is read every five minutes, and **Apps → Authentik** shows its version, workers, each outpost (and whether it is reporting), the certificates and API tokens that run out — soonest first — and the failed sign-ins of the last day. Authentik is the front door to everything behind it, so what is raised early is what would shut it: | Alert | When | Severity | |---|---|---| | **Outpost has not reported in** | An outpost that serves applications had no instance report within five minutes | critical | | **Outpost runs another version** | An instance runs a version other than the one Authentik expects | warning | | **Certificate runs out** | Within 14 days (a certificate Authentik renews itself is left out) | warning; critical within 3 days and after | | **API token or app password runs out** | Within 7 days | warning; critical within 2 days. The token this app reads Authentik with counts too | | **No worker connected** | The workers endpoint answers with none | warning | | **Failed sign-ins** | 20 or more in the last day | warning | | **Configuration errors** | 3 or more in the last day, with the newest message | warning | Maintenance on the integration holds them. A reading older than 20 minutes is not believed: nothing is raised or cleared from it, and the Alerts page says Authentik could not be checked. A token that does not expire, a verification token (they live for minutes) and an outpost with no applications are never alerts. ## Jellyfin An integration of type *Jellyfin* ([INTEGRATIONS.md](../INTEGRATIONS.md#jellyfin)) is read every two minutes, and **Apps → Jellyfin** shows whether it is up, who is **streaming** right now (and whether that is transcoded, which is what loads the machine), the scheduled tasks that failed, and a chart of the most that were streaming at once. It raises three warnings: **a restart is wanted**, **a scheduled task failed**, **10 or more sign-ins failed in a day** (the last is the one to look at if the server is open to the internet). Maintenance on the integration holds them. Its other use is **not cutting off a film**. When the Jellyfin runs on a server the app knows, an update run on that server asks first if people are streaming (the update drawer says who; *Update anyway* goes ahead), and a patch plan waits for them for up to two hours before it updates that server. Only a recent reading counts, so a Jellyfin that cannot be read never holds an update up. The match is by address: the Jellyfin's address must be the server's IP address or host name as the agent reports it (a Jellyfin behind a proxy on another machine is not matched). ## Docker containers that are broken If you use Dockhand, the health pass (every 15 minutes) also reads each container's **state** and **status text** — the same text the Docker page shows — and raises an alert (category *Failing services*, source *Docker*) for: | What | When | Severity | |---|---|---| | **Restart loop** | Docker says the container is restarting ("Restarting (127) 39 seconds ago") | critical | | **Unhealthy** | The container's own health check fails ("Up 6 days (unhealthy)") | warning | | **Stopped unexpectedly** | It is exited or dead, and it was **running at the last look**, or it has never been seen and stopped **within the last day** (from "Exited (0) 15 minutes ago") | warning | | **Never started** | It was created and has stayed "Created" for **an hour** | warning | | **Docker host unreachable** | Dockhand could not read one of its environments (a Hawser or Docker agent that is down) | warning | The hard part is *stopped*: many containers are stopped on purpose (a stack that is switched off, a one-shot job, a watchtower that ran and exited). So a stop is only news when it was **seen happen** — a container that has been down all along (a stack that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone. (Settings → Notifications → *What to send* shows one group at a time — *Servers and disks*, *Secrets, keys and domains*, *Backups and updates*, *Integrations and automation*, *DNS changes* — and the settings search opens the group a setting is in.) **Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything. Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`). **Critical vulnerabilities.** Dockhand can scan images (Grype or Trivy) — by hand, after a pull, or for a whole host. Every fifteen minutes the app reads **what Dockhand has already stored** (never starting a scan, so it costs Dockhand nothing) and raises a *warning* for each Docker host whose scanned images have one or more **critical** findings, naming the images and how many of the findings have a fix. High, medium and low findings are shown on the Docker page (*Vulnerabilities*, with the critical ones listed: CVE, package, installed and fixed version, image, and the containers that use it) but are never alerted. An image that was never scanned is not known to be clean — the page says how many of a host's images have a scan. A host whose count cannot be read is *unknown*, not clean, and says nothing. A reading older than 45 minutes is not believed: the alert is neither raised nor cleared and the Alerts page says it could not check. Maintenance on the Docker integration holds it, and an alert can be closed or snoozed like any other. **High findings** are never alerted unless you set a number under *Also warn when a Docker host has this many high vulnerabilities* (0, the default, is never): a host whose scanned images have at least that many high findings then gets a warning of its own ("Jackie: 12 high vulnerabilities in the scanned images"), beside the critical one. Scanners find high-severity issues in almost every image, so a low number is mostly noise. **To switch it off:** Settings → Notifications → What to send → *Warn about critical vulnerabilities in Docker images* (on by default; it turns the high ones off too). Dockhand can also send its own vulnerability notifications; with both on you hear twice. `/metrics` has `homelab_docker_vulnerabilities` (by `severity`) and `homelab_docker_images_scanned` for each host. ## Endpoint checks and heartbeats **Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds: | Kind | What it asks | |---|---| | `http` | Fetches an `http(s)://` address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. `200-299, 301`), and — if you ask — the page contains a keyword, and/or **a value in a JSON answer is within its limit** (see below). Follows up to 5 redirects when it has a keyword or value to look for. | | `tcp` | Opens a connection to a host and port. | | `tls` | Connects and reads the certificate; the answer is the days left. | Per check: **interval** (1–1440 minutes, default 5), **timeout** (default 10 s), **failures before an alert** (default 2 in a row — one blip is not an alert), **certificate warning days** (default 14), an optional **server it is linked to** (maintenance on that server silences it, and the alert links to it), and *accept a self-signed certificate*. Each answer — time, result, message — is kept, with hourly and daily summaries for the charts and uptime figures. - A check **fails an alert** (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A **certificate** within its warning days is its own alert (`endpoint-cert:…`). - **Checking a value, not just "is it up".** A web check can read a number or a word out of a JSON answer and judge it. Say **where** it is — `power.now`, `state`, `items[2].name`, or for a Prometheus query `data.result[0].value[1]` (`[0]` is the first place in a list; `["a.b"]` for a name with a dot in it) — and **how it should compare**: *is below*, *is at most*, *is above*, *is at least*, *is equal to*, *is anything but*, *contains*, or *is there*. A number given as text (`"42.5"`, which is how Prometheus gives them) counts as a number; equality compares numbers as numbers and anything else as the same text, capital letters included. An optional **unit** (`W`, `%`) is shown after a number. The status is judged first, then the value: a value outside its limit is a failure that says so ("power.now is 5200 W, but should be below 5000 W"), and the alert reads *Meter — power.now is 5200 W, but …* rather than "isn't answering". It is also a failure if the answer isn't JSON, the path isn't in it ("it stops at power"), or the answer is more than 64 KB. The value read is remembered with each result: shown in the *Answer* column, charted under **History…** (as far back as results are kept, 14 days), in `/metrics` as `homelab_endpoint_value` (numbers only) and in the read API. Examples: Home Assistant's `/api/states/sensor.power` (path `state`), a Plex or Frigate status, an inverter's output, a Prometheus query (`/api/v1/query?query=…`). - **A header, for an API that wants a token.** A web check can send one header with its request (usually `Authorization: Bearer …`). The value is stored **encrypted** when `CREDENTIALS_ENCRYPTION_KEY` is set, is never sent back to the browser (the form says *Saved — leave blank to keep it*), and goes only to the address you gave — **not to another host a redirect leads to**. - What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine. - **Apps in your containers.** With Dockhand connected, an image the app recognises (Immich, Paperless-ngx, Nextcloud, Vaultwarden, Authentik, Gitea, Grafana, Prometheus, Jellyfin, n8n, ntfy, Gotify, Nginx Proxy Manager, Open WebUI, Ollama, SearXNG, Wiki.js, BookStack, Homebox, Semaphore, Meilisearch, and a few dozen with just a web page) is offered as a check on **that app's own health address**, with what a good answer looks like set up as a [value check](#endpoint-checks-and-heartbeats): Immich's `/api/server/ping` must say `res` = `pong`, Nextcloud's `/status.php` `maintenance` = `false`, Gitea's `/api/healthz` `status` = `pass`, Grafana's `/api/health` `database` = `ok`, Prometheus's `/-/healthy` must contain "Healthy". It is asked on the Docker host's address — the server the app knows under the Docker environment's name in Dockhand, first usable IPv4 — at the **port the container publishes** (from Dockhand; if Dockhand does not say, the one the host's agent sees listening that the app is known to use). Only running containers, only hosts the app knows, and only what no check covers already is offered; where an app is recognised the plain "something answers on 8080" suggestion for that port is replaced by it. The paths are those the projects publish for the purpose, **not tried against live instances here**: a wrong one is a failing check you can edit. Uptime Kuma's monitors are not looked at, so something you already watch there may be offered — *Not for me* stops it. - The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under *Endpoints → Suggested*, and proxied hosts Nginx Proxy Manager serves. **Heartbeats** (Monitoring → Heartbeats) are the other direction: a cron job or script **calls an address when it finishes**, and the app alerts when it does *not*. A heartbeat has a **period** and a **grace**; it is **late** when both have passed since the last call (a *warning*), and a call to `/fail` is **critical** at once. Details and examples: [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md). Both are judged **every minute**, with the same remember-and-compare bookkeeping as the other checks. ## Integrations Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app, using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**. - **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the integration's page. - **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check. The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (**20 minutes**; a UPS, **5**). - **An integration can explain other alerts.** A service that stops answering because the integration it needs is failing is folded under it ([Dependencies](#dependencies-one-outage-one-alert)). ## Forecasts A forecast says "this will be full in N days" from a **straight line through the recent history**, and is deliberately cautious: - It needs **at least six readings covering at least three days**, and a **steady** climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat. - It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too. - A disk that is **already over its limit** is left to the ordinary "nearly full" alert. - It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as `homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears. - **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history. ## Dependencies: one outage, one alert If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a **graph of what needs what**: - a server **runs on** a host (set by hand, or worked out from its Proxmox node); - a UniFi device is **connected through** another, and a **server is plugged into** a switch or access point (matched by IP address); - **dependencies you write** (*Map → Dependencies*: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out. When something is down and what depends on it is down too, the dependants' alerts are **folded under** the first: shown nested, marked *caused by …*, not announced on their own, and the host's message says "— also takes down web and db". If the host comes back and a dependant **stays** down, it is announced then, as new: **an outage is never swallowed.** ## What happens to an alert Several things can quiet an alert. They are different, and it matters which you use: | | Promise about the thing? | Hides it? | Notifications | Ends | |---|---|---|---|---| | **Acknowledge** ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in `/metrics`; leaves the *needs attention* count | Stops [escalation](#escalation) | When you choose (an hour … 30 days), or when the alert has been gone half an hour | | **Snooze** ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; **one reminder** at the end if it is still there | At the time you set (up to 90 days), or earlier with *Wake now* | | **Close** ("dealt with, or accepted") | Yes — with who and why, and the reason is written on the incident | Yes, held back like a snooze | Raise/clear messages held; **one message** only if it **gets worse** | Never by time: when it gets more serious than it was, when it has been gone half an hour, or with *Open again* | | **Maintenance window** | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked *silenced* | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as **new** | | **Folded under a host** | — automatic | Nested under its cause | Not announced on its own | When the cause ends | | **Flapping** | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm | ### Who hears about it Notifications go to the channels that are on (**Gotify, ntfy, email, a webhook, push to phones and browsers**), narrowed by **routing rules** (first match wins: by kind, by severity, by the server's tags) and each browser's own wishes. **Quiet hours** hold messages and send one digest at the end of the window (critical alerts can be let through); **Away** mode does the same for a stretch you set, "until Sunday". Every message the app can send, and what toggles it, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md). **Event webhooks** are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included ([WEBHOOKS.md](WEBHOOKS.md)). ### Flapping An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back. ### Escalation An alert raised for a while with **nobody on it** can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes. ### Maintenance *Maintenance* (a window of 5 minutes to a week, started by hand, in bulk, from a **recurring schedule**, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under maintenance with it. A silenced problem is **not remembered as known**, so one still there when the window ends **alerts then, as new**. ## Incidents Every time an alert was raised and later went away is an **incident**: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts **every five minutes** whether or not anyone has the page open; the same alert coming back is a **new** incident. Resolved incidents are kept **400 days**. They are on **Alerts → History**, in the read API (`/api/v1/incidents`) and in the monthly report. ### What changed before an incident Each incident page has a **What changed before this?** card (and `GET /api/v1/incidents/:id/before`): the things the app knows happened in the 30 minutes, 2, 6 or 24 hours before the alert began, ranked. The sources are the audit log (settings, DNS, dependencies, integrations, a server being changed, a neighbour going offline or filling up), maintenance windows that began or ended, update runs and patch plans, other alerts raised first, and a step in the server's own figures (a disk that jumped, memory that fell away). Each is scored from three things: | | How it counts | |---|---| | **How related** | The **same thing** the incident is about counts most; then **something it needs** (the host it runs on, the switch it is plugged into, a dependency you wrote — the nearest count more); then a **change that touches everything** (a setting, DNS); then **something that needs it**; and last something with **no link** the app knows of. | | **What kind** | A maintenance window ending or a host going down is likelier than a change of settings, which is likelier than a note being written. Logins, notes, runbooks, tags and the like are not listed at all. A setting that bears on this kind of alert (thresholds, routing) counts for more. | | **How close** | A change five minutes before counts for much more than one two hours before — the weight falls off over about an hour and never reaches zero inside the window. | They are shown as *Likely* (55 and over), *Possible* (25 and over) and *Background* (the rest, behind a button); under 8 is left out, and at most ten are shown. The audit entry "pve1 stopped reporting" and the alert about it are one event and shown once. It is a **list of suspects, not a diagnosis**: it says what happened and when, not why, and a change made on the machine itself — a package installed by hand, a config file edited — is not something the app can see. ## History and how long it is kept | What | Kept | Grain | |---|---|---| | A server's load, memory and each disk | **35 days** of samples (one every 10–15 minutes), **120 days** of hourly summaries, **~800 days** of daily summaries | summaries carry the lowest and highest | | Endpoint check answers | **14 days** of every answer, hourly summaries for **120 days**, daily summaries for years | | | Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day | | Proxmox storage and Synology volume use | **400 days**, one reading an hour | | | Home Assistant watched numbers | **35 days**, one reading every few minutes | | | A UPS's charge, load and mains voltage | **35 days**, one reading every five minutes (every minute on battery) | | | Incidents | **400 days** after they resolve | | | Messages sent to event webhooks | The last **100 per webhook**, none older than **14 days** — see [WEBHOOKS.md](WEBHOOKS.md#the-delivery-log-and-sending-again) | | | The audit log and the Diagnostic Log | Until automatic purging is switched on under *Settings → Data* (then 90 days by default, configurable) | | A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there. **Is it being trimmed?** *Settings → Data → Storage* measures the database, its backups and sessions, and every table by the room it takes, and for each history says how far back it reaches against how long it is meant to be kept: *trimmed* (at its limit), *filling up* (not yet old enough to be trimmed), *not trimmed* (it holds far more than it should, so the trimming is not running or has not caught up) or *no limit* (the logs, until purging is switched on). The last two are listed at the top as needing a look. Nothing there changes anything. ## Where to see it | | Shows | |---|---| | **Dashboard**, **My work**, the menu badge | What needs attention now; what is yours | | **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back | | A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it | | **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **UPS**, **Proxy hosts** | Each source's own view | | **Service levels** | Each target and what is left of this month's error budget | | **Weekly digest** and **monthly, quarterly and annual reports** | The week, the month, the quarter and the year in one message | | **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) | | **Read API** (`/api/v1`) | [API.md](API.md) | | **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) | | **Status page**, **wall display**, **calendar feed** | Read-only views for others; made under *Settings → Share links* | ## Tuning, in questions and answers **I am about to reboot a server.** Start maintenance on it (Maintenance page, or *Silence* on an alert): its alerts are held and, if it is still down at the end, announced as new. **An alert is noisy because it keeps coming and going.** Let it flap (it will be damped after three raises an hour), then look at *Alerts → Noise* — it says what usually helps: a longer grace, a different limit. For your own rules, raise the limit or lengthen the time. **One server legitimately runs at 92% disk.** Give that server (or that one disk) its own limit — or *never alert about this disk* — in the server's *Settings* section. **I want to be reminded about something later, not silence it.** Snooze it. **I have dealt with it, or I accept it, and I do not want to be reminded.** Close it (*Close…* on the Alerts page, with a reason). It stays closed until it gets worse than it was, or the problem goes away for half an hour and returns. **I do not want to be woken for warnings.** Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours. **A job on a server has to run every night.** Make a heartbeat for it and call it from the end of the script. **I want my own limit, such as "memory above 85% for an hour".** Add an alert rule. **An integration's alerts disappeared.** Check *Couldn't check everything* on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why. **The app restarted and nothing about servers is alerting.** For 20 minutes after a restart servers are not judged, so the agents can report first. **I want these figures in my own monitoring.** `/metrics` for Prometheus, the read API for scripts, event webhooks to be told. ## What monitoring does not do - **It does not fix anything.** The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.) - **It does not see inside a server that has no agent** — only what an endpoint check, a heartbeat or an integration can say about it. - **It is not a replacement for a second pair of eyes on the app itself.** If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on `up{job="homelab-manager"}` ([PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md#alerting-rules-to-start-from)), or a check from another tool against `GET /health`. - **It is as fresh as its sources.** A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.