Files
Homelab-manager/docs/MONITORING.md
T
bobbanandClaude Sonnet 5.5 aebe98bf15 Authentik: know when the front door is about to shut
New Authentik integration (read-only, every 5 minutes): version, workers,
outposts and their health, certificates and API tokens that expire, failed
sign-ins and configuration errors. Monitoring -> Authentik page.

Alerts: an outpost that serves apps has gone quiet (critical), certificate or
token about to run out (critical close to it), no worker, failed sign-ins,
configuration errors. Held by maintenance, held when unreadable, in /metrics.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 22:09:38 +02:00

46 KiB
Raw Blame History

How monitoring works

This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the "why did it do that?" page. Setting things up is in the other pages (SSO, API, Prometheus and Grafana, webhooks, heartbeats); the list of every notification is NOTIFICATIONS.md.

The big picture

 agents ─────────┐
 integrations ───┤                         ┌──► Alerts page ──► acknowledge · snooze · silence
 endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push
 heartbeats ─────┤   (rules)   "a problem" ├──► event webhooks (machines)
 your own rules ─┘                         ├──► incidents (the record) and history
                                           └──► /metrics and the read API

Three ideas run through everything:

  1. A check turns readings into conditions. A condition is "this specific problem exists right now" and has a stable key (offline:server:3, disk:server:3:/) — the same problem always has the same key.
  2. Alerts are announced on a change, not on a schedule. A problem is announced once when it appears and once when it ends. It is not repeated every time it is looked at.
  3. Silence is never "all clear". When something cannot be read (an integration is down, a server's agent is quiet), what depended on it is held exactly as it was — neither raised nor cleared — and the Alerts page lists it under Couldn't check everything. The app never turns "I couldn't look" into "nothing is wrong".

What is watched, and how often

What How it gets to the app How often
Servers (Linux, Windows) The agent on each server pushes a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects to a server. Every 15 minutes by default (set when the agent is installed)
Proxmox, Synology The health pass reads them: node and storage usage, volume and disk status. Every 15 minutes
UniFi A sync reads devices, networks, Wi-Fi names and clients and keeps them. Every 5 minutes
Home Assistant A sync reads the entities you chose to watch and keeps their state and a history. Every 5 minutes
UPS (NUT or SNMP) A sync reads each UPS and keeps its state and a history. Every minute
Nginx Proxy Manager Read when its page is opened and by the alert check, at most once a minute. On demand, cached 1 minute
Semaphore, Gitea The automation check reads the latest run of each template or workflow. Every 15 minutes
Endpoint checks (web address, TCP port, TLS certificate) The app asks each one itself. Each check has its own interval (5 minutes by default); the scheduler looks every minute for the ones that are due
Heartbeats (cron jobs) The job calls an address when it finishes. When the job runs; judged every minute
Secrets, certificates, domains, Tailscale keys Their expiry dates, with the certificate fetched live. Daily, at the reminder time in Settings → Notifications
Docker image updates, Proxmox backup results, Proxmox Backup verification Read and judged. Daily, at the same time
Your own alert rules Judged from the history the agents' reports build. On the 15-minute health pass
Every alert, together The Alerts page, the incident recorder and the snooze timer look at the whole picture. Alerts page: on open, reusing a result up to 1 minute old · incidents: every 5 minutes · snoozes: every minute · escalation and flapping: every 5 minutes

After the app restarts, server-based checks are paused for 20 minutes so agents get a chance to report before any server is judged.

Servers and the agent

The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that only pushes. Each server has its own token, which can do one thing: replace that server's own report. The app replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the README and AGENT_UPDATE.md.

When is a server "offline"? When its agent has not reported for longer than the offline limit — 60 minutes by default (Settings → Notifications → Servers and disks), or the server's own limit if it has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that never reported is "unknown": it has no agent to go quiet, so nothing alerts.

While a server is offline its other figures are stale, so it is not also judged on disks, services or your rules — its offline alert says it all.

Per-server settings. A server's Settings section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for one disk or "never alert about this disk", and services never to count as failed (names or patterns such as gupdate*). Whatever it does not replace follows the global setting.

What the report is used for: the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support check, pending updates, the dependency map, and /metrics.

From a reading to an alert

There are two paths, designed so they cannot disagree:

1. The state-based checks (the notifications). Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the list of conditions that exist right now. The app compares that list with the one it remembered from last time:

  • new → announced (a notification, an event webhook, a row in the incident record);
  • gone → announced as resolved;
  • still there → nothing;
  • its source could not be read this time → carried over untouched (not resolved, not repeated).

The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all.

2. The Alerts page. It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), live, and turns what they find into one flat list. It ignores the per-event on/off toggles — the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a 20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under Couldn't check everything.

Each alert has:

Severity critical, warning or info. Things that are gone (offline, a failing SMART disk, a disk over 95%) are critical; things that are coming (a forecast, an expiring certificate, an OS going out of support) are warnings or notes.
Category offline, disk, services, backup, updates, expiry, automation, integration, monitoring, tickets — what the filter on the page and the routing rules work on.
Source Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat …
Target What a maintenance window would silence it with (a server, an integration, a device), where it makes sense.
Key / id Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it.

The built-in checks

Every threshold here is a default; where it can be changed, the place is named.

Check Raised when Default Changed in
Server offline The agent has not reported for longer than the limit 60 min · critical Settings → Notifications; per server
Disk nearly full A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit 90% · warning, critical from 95% Settings → Notifications; per server, per disk
Disk filling up A disk on a steady climb will be full within the window (Forecasts) 14 days · warning, critical within 3 days Settings → Notifications; per server
Failed services A Linux agent reports failed systemd units any per server (ignore list)
Disk health A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) not configurable
ZFS / storage pool A ZFS pool (or Windows Storage Spaces pool) is not ONLINE not configurable
Synology volume / disk A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit not configurable
Unusual for the time CPU load or memory use over the last hour is far above what is normal for this hour of this weekday (below) on · warning Settings → Notifications
New port A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's Ports card on Settings → Notifications
Network device offline A UniFi device has been offline too long 10 min · critical Settings → Notifications
Operating system support A server's OS has passed, or is about to pass, its end of support within 90 days · note; warning once ended Settings → Notifications
Updates waiting The agent reports updates waiting (and, where it can say, security ones and a restart needed) Updates page
Backup stale / failed A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed 48 h Settings → Notifications; per server
Restore test A restore test failed, or none has been done within the window failure always; age off by default Settings → Notifications
Expiry A secret, certificate, domain or Tailscale key is within its warning window or has expired per item (domains 30 days) the item; Settings → Notifications
Docker updates A container image has a newer version Settings → Notifications
Authentik needs a look An outpost that serves apps has gone quiet (critical), a certificate or API token is about to run out (critical close to it), no worker is connected, or sign-ins or flows are failing on, with the integration Authentik page; below
Jellyfin needs a look Jellyfin wants a restart, a scheduled task (a library scan) failed, or 10 or more sign-ins failed in a day on, with the integration Jellyfin page; below
Duplicati backup failed / overdue A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should per job (2 days by default) Duplicati page; below
Docker container broken A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host on Settings → Notifications; below
Automation failed A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) not configurable
Integration failing An integration's calls have failed several times in a row (Integrations) 3 calls Settings → Notifications
Endpoint check failing / certificate running out See below per check
Heartbeat late / failed See below per heartbeat
Home Assistant / Nginx Proxy Manager An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up Home Assistant page
Service level 80 % of a target's error budget used (warning), or overspent (critical), for the month per target Service levels page; below
UPS Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) UPS page; INTEGRATIONS.md
Your own rules Below Monitoring → Alert rules

What is normal for the time

A fixed limit cannot say that a server is busy for a Tuesday at 03:00. So for each server's CPU load and memory use the app keeps a baseline for every hour of the week (168 of them, in the notification time zone): the median of that hour's averages over the previous weeks (up to ten; the hourly figures are kept four months), and a robust spread (the median absolute deviation, so one odd week does not stretch it). An hour needs four weeks behind it before it has a baseline at all; until then nothing is said. The hour in progress and hours with fewer than two samples are not counted.

It is quiet by design. A reading is a warning only when all of these hold, judged on the last hour of samples (at least three, the newest under 30 minutes old):

CPU Memory
The last hour's average is at least 50 % 70 %
…and above the usual by at least 30 points 15 points
…and above it by at least four spreads (the spread is taken as at least 4 points) four spreads (at least 2 points)
…and the lowest sample in that hour is still above the usual by 15 points 7.5 points

The last line is what separates a state from a spike: if the figure dropped back toward normal at any time in the hour, it is not said. Only a high reading is ever raised — low is never news. What it finds is an ordinary health condition (baseline:cpu:server:<id>, baseline:mem:server:<id>, Monitoring, warning): announced and recovered like the others, held back by maintenance on the server, snoozed, flapping-damped, routed, an event for webhooks — and left out for a server that has stopped reporting (its offline alert says that). Switch it off with Warn when CPU or memory is unusually high for the time under Settings → Notifications → What to send → Servers and disks.

The same baselines draw the grey usual line and band on a server's History → Trends (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.

Service levels

Monitoring → Service levels holds a number against a service: 99.9 % for Gitea. A target is set on an endpoint check, a server or an integration and is judged over calendar months (on the notification time zone's clock). Its error budget is the downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and where today's pace would end the month, once a day has passed), and how each of the last six months went.

How downtime is measured, so the figure can be trusted — and where it cannot:

Measured on From Precision
Endpoint check The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. A day's failed share is spread over the whole day
Server The moments the audit log recorded it stopping reporting and reporting again To the minute; only from when the server was created
Integration The moments calls to that kind of service started failing and worked again To the minute; only from when the app began recording them

Planned maintenance is not taken out: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing.

When 80 % of the budget is used it is a warning, and when it is overspent a critical alert (key slo:<id>, Monitoring, source Service level): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks — and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in /metrics (homelab_service_level_*) and in the monthly, quarterly and annual reports (below).

Your own alert rules

Where the built-in checks do not fit, Monitoring → Alert rules (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:

  • What to watch: CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
  • When: above or below a limit, and — for load, memory and disk use — for how long. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time.
  • How serious: warning or critical. For which servers: all of them, one, or those with a tag.
  • Try it now shows which servers it would be raised on, from current data, before you save it.

A rule's finding is an ordinary alert with the key rule:<rule id>:server:<server id>[:<disk>], in the Monitoring category: announced and resolved like the others, held back by maintenance on the server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is offline or has not reported for 45 minutes is not judged.

Duplicati backups

Duplicati tells this app how each backup went (INTEGRATIONS.md says how to set it up), and Backups → Duplicati shows every job: when it last succeeded, how the last run ended, how long it took, how big the source is, how many files changed, and where it stands.

State Meaning Alert
In order The last run worked and it is not overdue —
Warnings The last run finished with warnings (a locked file, say). It still counts as a backup none
Failed The last run ended in Error or Fatal warning; critical from the second failure in a row
Overdue It has not succeeded for longer than it is expected to — 2 days unless you set it per job (6 hours to 90 days) warning; critical at three times that

The alert says which job, on which machine, why (the first error Duplicati reported) and when the last good backup was. Only the Backup operation counts: a compact, a repair or a verification does not say whether the data is safe. A job can be muted (no alert; it is still listed), and one that no longer exists can be forgotten (with its history; if Duplicati reports it again it is learned again). Each machine can be tied to a server, so maintenance on that server silences its backups' alerts.

What is kept: the last result of each job, and each report for 400 days or 400 reports. /metrics has homelab_duplicati_backup_ok, homelab_duplicati_last_success_age_seconds and homelab_duplicati_source_size_bytes for each job. A job is only learned from its first report, so a job that has never reported is not known to be missing; and because the reports are pushed, the app being down when a backup finishes loses that report (the backup shows as overdue only once it is).

Authentik

An integration of type Authentik (INTEGRATIONS.md) is read every five minutes, and Monitoring → Authentik shows its version, workers, each outpost (and whether it is reporting), the certificates and API tokens that run out — soonest first — and the failed sign-ins of the last day. Authentik is the front door to everything behind it, so what is raised early is what would shut it:

Alert When Severity
Outpost has not reported in An outpost that serves applications had no instance report within five minutes critical
Outpost runs another version An instance runs a version other than the one Authentik expects warning
Certificate runs out Within 14 days (a certificate Authentik renews itself is left out) warning; critical within 3 days and after
API token or app password runs out Within 7 days warning; critical within 2 days. The token this app reads Authentik with counts too
No worker connected The workers endpoint answers with none warning
Failed sign-ins 20 or more in the last day warning
Configuration errors 3 or more in the last day, with the newest message warning

Maintenance on the integration holds them. A reading older than 20 minutes is not believed: nothing is raised or cleared from it, and the Alerts page says Authentik could not be checked. A token that does not expire, a verification token (they live for minutes) and an outpost with no applications are never alerts.

Jellyfin

An integration of type Jellyfin (INTEGRATIONS.md) is read every two minutes, and Monitoring → Jellyfin shows whether it is up, who is streaming right now (and whether that is transcoded, which is what loads the machine), the scheduled tasks that failed, and a chart of the most that were streaming at once. It raises three warnings: a restart is wanted, a scheduled task failed, 10 or more sign-ins failed in a day (the last is the one to look at if the server is open to the internet). Maintenance on the integration holds them.

Its other use is not cutting off a film. When the Jellyfin runs on a server the app knows, an update run on that server asks first if people are streaming (the update drawer says who; Update anyway goes ahead), and a patch plan waits for them for up to two hours before it updates that server. Only a recent reading counts, so a Jellyfin that cannot be read never holds an update up. The match is by address: the Jellyfin's address must be the server's IP address or host name as the agent reports it (a Jellyfin behind a proxy on another machine is not matched).

Docker containers that are broken

If you use Dockhand, the health pass (every 15 minutes) also reads each container's state and status text — the same text the Docker page shows — and raises an alert (category Failing services, source Docker) for:

What When Severity
Restart loop Docker says the container is restarting ("Restarting (127) 39 seconds ago") critical
Unhealthy The container's own health check fails ("Up 6 days (unhealthy)") warning
Stopped unexpectedly It is exited or dead, and it was running at the last look, or it has never been seen and stopped within the last day (from "Exited (0) 15 minutes ago") warning
Never started It was created and has stayed "Created" for an hour warning
Docker host unreachable Dockhand could not read one of its environments (a Hawser or Docker agent that is down) warning

The hard part is stopped: many containers are stopped on purpose (a stack that is switched off, a one-shot job, a watchtower that ran and exited). So a stop is only news when it was seen happen — a container that has been down all along (a stack that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already long stopped. The scanner containers Dockhand starts (dockhand-grype-*, dockhand-trivy-*) are always left alone.

Leaving a container alone: the Leave alone button on the Docker page (administrators; Watch again undoes it), or the list under Settings → Notifications → What to send → Containers to leave alone — one per line, a name or host/name, * for anything. Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with Warn when a Docker container is broken. What the app has seen of each container is kept between looks (in settings, key _internal:containerMemory).

Endpoint checks and heartbeats

Endpoint checks (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds:

Kind What it asks
http Fetches an http(s):// address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. 200-299, 301), and — if you ask — the page contains a keyword, and/or a value in a JSON answer is within its limit (see below). Follows up to 5 redirects when it has a keyword or value to look for.
tcp Opens a connection to a host and port.
tls Connects and reads the certificate; the answer is the days left.

Per check: interval (1–1440 minutes, default 5), timeout (default 10 s), failures before an alert (default 2 in a row — one blip is not an alert), certificate warning days (default 14), an optional server it is linked to (maintenance on that server silences it, and the alert links to it), and accept a self-signed certificate. Each answer — time, result, message — is kept, with hourly and daily summaries for the charts and uptime figures.

  • A check fails an alert (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A certificate within its warning days is its own alert (endpoint-cert:…).
  • Checking a value, not just "is it up". A web check can read a number or a word out of a JSON answer and judge it. Say where it is — power.now, state, items[2].name, or for a Prometheus query data.result[0].value[1] ([0] is the first place in a list; ["a.b"] for a name with a dot in it) — and how it should compare: is below, is at most, is above, is at least, is equal to, is anything but, contains, or is there. A number given as text ("42.5", which is how Prometheus gives them) counts as a number; equality compares numbers as numbers and anything else as the same text, capital letters included. An optional unit (W, %) is shown after a number. The status is judged first, then the value: a value outside its limit is a failure that says so ("power.now is 5200 W, but should be below 5000 W"), and the alert reads Meter — power.now is 5200 W, but … rather than "isn't answering". It is also a failure if the answer isn't JSON, the path isn't in it ("it stops at power"), or the answer is more than 64 KB. The value read is remembered with each result: shown in the Answer column, charted under History… (as far back as results are kept, 14 days), in /metrics as homelab_endpoint_value (numbers only) and in the read API. Examples: Home Assistant's /api/states/sensor.power (path state), a Plex or Frigate status, an inverter's output, a Prometheus query (/api/v1/query?query=…).
  • A header, for an API that wants a token. A web check can send one header with its request (usually Authorization: Bearer …). The value is stored encrypted when CREDENTIALS_ENCRYPTION_KEY is set, is never sent back to the browser (the form says Saved — leave blank to keep it), and goes only to the address you gave — not to another host a redirect leads to.
  • What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine.
  • Apps in your containers. With Dockhand connected, an image the app recognises (Immich, Paperless-ngx, Nextcloud, Vaultwarden, Authentik, Gitea, Grafana, Prometheus, Jellyfin, n8n, ntfy, Gotify, Nginx Proxy Manager, Open WebUI, Ollama, SearXNG, Wiki.js, BookStack, Homebox, Semaphore, Meilisearch, and a few dozen with just a web page) is offered as a check on that app's own health address, with what a good answer looks like set up as a value check: Immich's /api/server/ping must say res = pong, Nextcloud's /status.php maintenance = false, Gitea's /api/healthz status = pass, Grafana's /api/health database = ok, Prometheus's /-/healthy must contain "Healthy". It is asked on the Docker host's address — the server the app knows under the Docker environment's name in Dockhand, first usable IPv4 — at the port the container publishes (from Dockhand; if Dockhand does not say, the one the host's agent sees listening that the app is known to use). Only running containers, only hosts the app knows, and only what no check covers already is offered; where an app is recognised the plain "something answers on 8080" suggestion for that port is replaced by it. The paths are those the projects publish for the purpose, not tried against live instances here: a wrong one is a failing check you can edit. Uptime Kuma's monitors are not looked at, so something you already watch there may be offered — Not for me stops it.
  • The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under Endpoints → Suggested, and proxied hosts Nginx Proxy Manager serves.

Heartbeats (Monitoring → Heartbeats) are the other direction: a cron job or script calls an address when it finishes, and the app alerts when it does not. A heartbeat has a period and a grace; it is late when both have passed since the last call (a warning), and a call to /fail is critical at once. Details and examples: HEARTBEATS-AND-CALENDAR.md.

Both are judged every minute, with the same remember-and-compare bookkeeping as the other checks.

Integrations

Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are read by the app, using credentials an administrator enters under Integrations; what each needs is in INTEGRATIONS.md. Every call is recorded in the Diagnostic Log.

  • An integration is "failing" when its last 3 calls in a row (Settings → Notifications) failed. Failures are counted per kind of service (all Proxmox integrations share a streak), in memory, so a restart starts the count again. A down notification goes out at the threshold and a recovered one on the next success. The moments are written to the audit log, and they feed the uptime record on the integration's page.
  • When an integration cannot be read, what it reported is held, not cleared. A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check. The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (20 minutes; a UPS, 5).
  • An integration can explain other alerts. A service that stops answering because the integration it needs is failing is folded under it (Dependencies).

Forecasts

A forecast says "this will be full in N days" from a straight line through the recent history, and is deliberately cautious:

  • It needs at least six readings covering at least three days, and a steady climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that wobbles, is not forecast. A slope under 0.05 points a day counts as flat.
  • It looks at the last two weeks; if those are flat it looks at the last ninety days, so a slow, steady climb is seen too.
  • A disk that is already over its limit is left to the ordinary "nearly full" alert.
  • It is shown as an alert when it falls within the window (Settings → Notifications → Warn when a disk will fill within, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as homelab_server_disk_days_to_full / homelab_storage_days_to_full in /metrics. It is not sent as a notification the moment it appears.
  • Servers: from the agents' history. Proxmox storages and Synology volumes: the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.

Dependencies: one outage, one alert

If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a graph of what needs what:

  • a server runs on a host (set by hand, or worked out from its Proxmox node);
  • a UniFi device is connected through another, and a server is plugged into a switch or access point (matched by IP address);
  • dependencies you write (Map → Dependencies: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out.

When something is down and what depends on it is down too, the dependants' alerts are folded under the first: shown nested, marked caused by …, not announced on their own, and the host's message says "— also takes down web and db". If the host comes back and a dependant stays down, it is announced then, as new: an outage is never swallowed.

What happens to an alert

Several things can quiet an alert. They are different, and it matters which you use:

Promise about the thing? Hides it? Notifications Ends
Acknowledge ("I'm on it") No — it says someone has seen it, why, who has it No: still listed, still counted in /metrics; leaves the needs attention count Stops escalation When you choose (an hour … 30 days), or when the alert has been gone half an hour
Snooze ("remind me tomorrow 09:00") No Yes, held back like maintenance Raise/clear messages held; one reminder at the end if it is still there At the time you set (up to 90 days), or earlier with Wake now
Maintenance window Yes — "I'm working on this" (5 minutes to a week) Yes, marked silenced Held for the server, integration, DNS provider or device At its end; what is still there is then announced as new
Folded under a host — automatic Nested under its cause Not announced on its own When the cause ends
Flapping — automatic No Raise and clear messages held; one "flapping" message, one "settled" message After an hour of calm

Who hears about it

Notifications go to the channels that are on (Gotify, ntfy, email, a webhook, push to phones and browsers), narrowed by routing rules (first match wins: by kind, by severity, by the server's tags) and each browser's own wishes. Quiet hours hold messages and send one digest at the end of the window (critical alerts can be let through); Away mode does the same for a stretch you set, "until Sunday". Every message the app can send, and what toggles it, is in NOTIFICATIONS.md. Event webhooks are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included (WEBHOOKS.md).

Flapping

An alert raised three times within an hour (Settings → Notifications; 0 = off) is called flapping: one message says so, its further raise and clear messages are held, and it is marked on the page. After an hour with no raise or clear, one settled message says whether it is still there. The alert itself is untouched: still raised, still counted. Alerts → Noise lists what keeps coming back.

Escalation

An alert raised for a while with nobody on it can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes.

Maintenance

Maintenance (a window of 5 minutes to a week, started by hand, in bulk, from a recurring schedule, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under maintenance with it. A silenced problem is not remembered as known, so one still there when the window ends alerts then, as new.

Incidents

Every time an alert was raised and later went away is an incident: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts every five minutes whether or not anyone has the page open; the same alert coming back is a new incident. Resolved incidents are kept 400 days. They are on Operations → Alerts → History, in the read API (/api/v1/incidents) and in the monthly report.

What changed before an incident

Each incident page has a What changed before this? card (and GET /api/v1/incidents/:id/before): the things the app knows happened in the 30 minutes, 2, 6 or 24 hours before the alert began, ranked. The sources are the audit log (settings, DNS, dependencies, integrations, a server being changed, a neighbour going offline or filling up), maintenance windows that began or ended, update runs and patch plans, other alerts raised first, and a step in the server's own figures (a disk that jumped, memory that fell away).

Each is scored from three things:

How it counts
How related The same thing the incident is about counts most; then something it needs (the host it runs on, the switch it is plugged into, a dependency you wrote — the nearest count more); then a change that touches everything (a setting, DNS); then something that needs it; and last something with no link the app knows of.
What kind A maintenance window ending or a host going down is likelier than a change of settings, which is likelier than a note being written. Logins, notes, runbooks, tags and the like are not listed at all. A setting that bears on this kind of alert (thresholds, routing) counts for more.
How close A change five minutes before counts for much more than one two hours before — the weight falls off over about an hour and never reaches zero inside the window.

They are shown as Likely (55 and over), Possible (25 and over) and Background (the rest, behind a button); under 8 is left out, and at most ten are shown. The audit entry "pve1 stopped reporting" and the alert about it are one event and shown once. It is a list of suspects, not a diagnosis: it says what happened and when, not why, and a change made on the machine itself — a package installed by hand, a config file edited — is not something the app can see.

History and how long it is kept

What Kept Grain
A server's load, memory and each disk 35 days of samples (one every 10–15 minutes), 120 days of hourly summaries, ~800 days of daily summaries summaries carry the lowest and highest
Endpoint check answers 14 days of every answer, hourly summaries for 120 days, daily summaries for years
Uptime of servers and integrations The last weeks, worked out from the moments each stopped and resumed (in the audit log) one figure a day
Proxmox storage and Synology volume use 400 days, one reading an hour
Home Assistant watched numbers 35 days, one reading every few minutes
A UPS's charge, load and mains voltage 35 days, one reading every five minutes (every minute on battery)
Incidents 400 days after they resolve
Messages sent to event webhooks The last 100 per webhook, none older than 14 days — see WEBHOOKS.md
The audit log and the Diagnostic Log Until automatic purging is switched on under Settings → Data (then 90 days by default, configurable)

A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there.

Is it being trimmed? Settings → Data → Storage measures the database, its backups and sessions, and every table by the room it takes, and for each history says how far back it reaches against how long it is meant to be kept: trimmed (at its limit), filling up (not yet old enough to be trimmed), not trimmed (it holds far more than it should, so the trimming is not running or has not caught up) or no limit (the logs, until purging is switched on). The last two are listed at the top as needing a look. Nothing there changes anything.

Where to see it

Shows
Dashboard, My work, the menu badge What needs attention now; what is yours
Alerts (Now · History · Noise) Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back
A server's page Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it
Endpoints, Heartbeats, Proxmox, Synology, Network devices, Home Assistant, UPS, Proxy hosts Each source's own view
Service levels Each target and what is left of this month's error budget
Weekly digest and monthly, quarterly and annual reports The week, the month, the quarter and the year in one message
/metrics → Prometheus, Grafana PROMETHEUS-GRAFANA.md
Read API (/api/v1) API.md
Event webhooks WEBHOOKS.md
Status page, wall display, calendar feed Read-only views for others; made under Settings → Share links

Tuning, in questions and answers

I am about to reboot a server. Start maintenance on it (Maintenance page, or Silence on an alert): its alerts are held and, if it is still down at the end, announced as new.

An alert is noisy because it keeps coming and going. Let it flap (it will be damped after three raises an hour), then look at Alerts → Noise — it says what usually helps: a longer grace, a different limit. For your own rules, raise the limit or lengthen the time.

One server legitimately runs at 92% disk. Give that server (or that one disk) its own limit — or never alert about this disk — in the server's Settings section.

I want to be reminded about something later, not silence it. Snooze it.

I do not want to be woken for warnings. Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours.

A job on a server has to run every night. Make a heartbeat for it and call it from the end of the script.

I want my own limit, such as "memory above 85% for an hour". Add an alert rule.

An integration's alerts disappeared. Check Couldn't check everything on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why.

The app restarted and nothing about servers is alerting. For 20 minutes after a restart servers are not judged, so the agents can report first.

I want these figures in my own monitoring. /metrics for Prometheus, the read API for scripts, event webhooks to be told.

What monitoring does not do

  • It does not fix anything. The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.)
  • It does not see inside a server that has no agent — only what an endpoint check, a heartbeat or an integration can say about it.
  • It is not a replacement for a second pair of eyes on the app itself. If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on up{job="homelab-manager"} (PROMETHEUS-GRAFANA.md), or a check from another tool against GET /health.
  • It is as fresh as its sources. A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.