Files
Homelab-manager/docs/MONITORING.md
T
bobbanandClaude Sonnet 5.5 e114567d64 How big the app's data is, and a webhook delivery log with Send again
Settings -> Data -> Storage measures the database, backups and sessions and every
table (size, rows, growth, how long it is kept, how far back it reaches), and flags
histories that are not being trimmed or have no limit.

Event webhooks now keep a delivery log (last 100 per webhook, 14 days): what was
sent, how it went, the message itself, and Send again, which resends it exactly as it
was (same id and signature). Resends are audited and in the Change log.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-21 20:58:57 +02:00

328 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# How monitoring works
This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the
"why did it do that?" page. Setting things up is in the other pages ([SSO](SSO.md), [API](API.md), [Prometheus and Grafana](PROMETHEUS-GRAFANA.md), [webhooks](WEBHOOKS.md),
[heartbeats](HEARTBEATS-AND-CALENDAR.md)); the list of every notification is [NOTIFICATIONS.md](../NOTIFICATIONS.md).
- [The big picture](#the-big-picture)
- [What is watched, and how often](#what-is-watched-and-how-often)
- [Servers and the agent](#servers-and-the-agent)
- [From a reading to an alert](#from-a-reading-to-an-alert)
- [The built-in checks](#the-built-in-checks)
- [Your own alert rules](#your-own-alert-rules)
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
- [Integrations](#integrations)
- [Forecasts](#forecasts)
- [Dependencies: one outage, one alert](#dependencies-one-outage-one-alert)
- [What happens to an alert](#what-happens-to-an-alert) — notify, acknowledge, silence, snooze, flapping, escalate
- [Incidents](#incidents)
- [History and how long it is kept](#history-and-how-long-it-is-kept)
- [Where to see it](#where-to-see-it)
- [Tuning, in questions and answers](#tuning-in-questions-and-answers)
- [What monitoring does not do](#what-monitoring-does-not-do)
## The big picture
```
agents ─────────┐
integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · silence
endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push
heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines)
your own rules ─┘ ├──► incidents (the record) and history
└──► /metrics and the read API
```
Three ideas run through everything:
1. **A check turns readings into *conditions*.** A condition is "this specific problem exists right now" and has a stable **key** (`offline:server:3`, `disk:server:3:/`) — the same problem always has the same key.
2. **Alerts are announced on a change, not on a schedule.** A problem is announced **once** when it appears and **once** when it ends. It is not repeated every time it is looked at.
3. **Silence is never "all clear".** When something cannot be *read* (an integration is down, a server's agent is quiet), what depended on it is **held exactly as it was** — neither raised nor cleared —
and the Alerts page lists it under *Couldn't check everything*. The app never turns "I couldn't look" into "nothing is wrong".
## What is watched, and how often
| What | How it gets to the app | How often |
|---|---|---|
| **Servers** (Linux, Windows) | The **agent** on each server **pushes** a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects *to* a server. | Every **15 minutes** by default (set when the agent is installed) |
| **Proxmox**, **Synology** | The health pass **reads** them: node and storage usage, volume and disk status. | Every **15 minutes** |
| **UniFi** | A sync **reads** devices, networks, Wi-Fi names and clients and keeps them. | Every **5 minutes** |
| **Home Assistant** | A sync **reads** the entities you chose to watch and keeps their state and a history. | Every **5 minutes** |
| **Nginx Proxy Manager** | **Read** when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute |
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications |
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
After the app **restarts**, server-based checks are paused for **20 minutes** so agents get a chance to report before any server is judged.
## Servers and the agent
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
**Per-server settings.** A server's *Settings* section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for
one disk or "never alert about this disk", and services never to count as failed (names or patterns such as `gupdate*`). Whatever it does not replace follows the global setting.
**What the report is used for:** the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support
check, pending updates, the dependency map, and `/metrics`.
## From a reading to an alert
There are two paths, designed so they cannot disagree:
**1. The state-based checks (the notifications).** Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the **list of conditions that exist right now**. The app compares that list with
the one it remembered from last time:
- **new** → announced (a notification, an event webhook, a row in the incident record);
- **gone** → announced as resolved;
- **still there** → nothing;
- **its source could not be read this time** → carried over untouched (not resolved, not repeated).
The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all.
**2. The Alerts page.** It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), **live**, and turns what they find into one flat list. It ignores the per-event on/off toggles —
the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a
20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under *Couldn't check everything*.
Each alert has:
| | |
|---|---|
| **Severity** | `critical`, `warning` or `info`. Things that are *gone* (offline, a failing SMART disk, a disk over 95%) are critical; things that are *coming* (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. |
| **Category** | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring`, `tickets` — what the filter on the page and the [routing rules](#who-hears-about-it) work on. |
| **Source** | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … |
| **Target** | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. |
| **Key / id** | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. |
## The built-in checks
Every threshold here is a default; where it can be changed, the place is named.
| Check | Raised when | Default | Changed in |
|---|---|---|---|
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk |
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card | on | Settings → Notifications |
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
## Your own alert rules
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
- **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
- **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the
history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time.
- **How serious:** warning or critical. **For which servers:** all of them, one, or those with a tag.
- **Try it now** shows which servers it would be raised on, from current data, before you save it.
A rule's finding is an **ordinary alert** with the key `rule:<rule id>:server:<server id>[:<disk>]`, in the *Monitoring* category: announced and resolved like the others, held back by maintenance on the
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is
offline or has not reported for 45 minutes is not judged.
## Endpoint checks and heartbeats
**Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds:
| Kind | What it asks |
|---|---|
| `http` | Fetches an `http(s)://` address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. `200-299, 301`), and — if you ask — the page contains a keyword. Follows up to 5 redirects. |
| `tcp` | Opens a connection to a host and port. |
| `tls` | Connects and reads the certificate; the answer is the days left. |
Per check: **interval** (1–1440 minutes, default 5), **timeout** (default 10 s), **failures before an alert** (default 2 in a row — one blip is not an alert), **certificate warning days** (default 14), an
optional **server it is linked to** (maintenance on that server silences it, and the alert links to it), and *accept a self-signed certificate*. Each answer — time, result, message — is kept, with hourly and
daily summaries for the charts and uptime figures.
- A check **fails an alert** (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A **certificate** within its warning days is its own alert (`endpoint-cert:…`).
- What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine.
- The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under *Endpoints → Suggested*, and proxied hosts Nginx Proxy Manager serves.
**Heartbeats** (Monitoring → Heartbeats) are the other direction: a cron job or script **calls an address when it finishes**, and the app alerts when it does *not*. A heartbeat has a **period** and a **grace**; it is
**late** when both have passed since the last call (a *warning*), and a call to `/fail` is **critical** at once. Details and examples: [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md).
Both are judged **every minute**, with the same remember-and-compare bookkeeping as the other checks.
## Integrations
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
integration's page.
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (**20 minutes**).
- **An integration can explain other alerts.** A service that stops answering because the integration it needs is failing is folded under it ([Dependencies](#dependencies-one-outage-one-alert)).
## Forecasts
A forecast says "this will be full in N days" from a **straight line through the recent history**, and is deliberately cautious:
- It needs **at least six readings covering at least three days**, and a **steady** climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
## Dependencies: one outage, one alert
If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a **graph of what needs what**:
- a server **runs on** a host (set by hand, or worked out from its Proxmox node);
- a UniFi device is **connected through** another, and a **server is plugged into** a switch or access point (matched by IP address);
- **dependencies you write** (*Map → Dependencies*: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out.
When something is down and what depends on it is down too, the dependants' alerts are **folded under** the first: shown nested, marked *caused by …*, not announced on their own, and the host's message says
"— also takes down web and db". If the host comes back and a dependant **stays** down, it is announced then, as new: **an outage is never swallowed.**
## What happens to an alert
Several things can quiet an alert. They are different, and it matters which you use:
| | Promise about the thing? | Hides it? | Notifications | Ends |
|---|---|---|---|---|
| **Acknowledge** ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in `/metrics`; leaves the *needs attention* count | Stops [escalation](#escalation) | When you choose (an hour … 30 days), or when the alert has been gone half an hour |
| **Snooze** ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; **one reminder** at the end if it is still there | At the time you set (up to 90 days), or earlier with *Wake now* |
| **Maintenance window** | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked *silenced* | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as **new** |
| **Folded under a host** | — automatic | Nested under its cause | Not announced on its own | When the cause ends |
| **Flapping** | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm |
### Who hears about it
Notifications go to the channels that are on (**Gotify, ntfy, email, a webhook, push to phones and browsers**), narrowed by **routing rules** (first match wins: by kind, by severity, by the server's tags) and each
browser's own wishes. **Quiet hours** hold messages and send one digest at the end of the window. Every message the app can send, and what toggles it, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
**Event webhooks** are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included ([WEBHOOKS.md](WEBHOOKS.md)).
### Flapping
An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
### Escalation
An alert raised for a while with **nobody on it** can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra
email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes.
### Maintenance
*Maintenance* (a window of 5 minutes to a week, started by hand, in bulk, from a **recurring schedule**, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and
its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under
maintenance with it. A silenced problem is **not remembered as known**, so one still there when the window ends **alerts then, as new**.
## Incidents
Every time an alert was raised and later went away is an **incident**: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its
timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts **every five minutes** whether or not anyone has
the page open; the same alert coming back is a **new** incident. Resolved incidents are kept **400 days**. They are on **Operations → Alerts → History**, in the read API (`/api/v1/incidents`) and in the
monthly report.
### What changed before an incident
Each incident page has a **What changed before this?** card (and `GET /api/v1/incidents/:id/before`): the things the app knows happened in the 30 minutes, 2, 6 or 24 hours before the alert began,
ranked. The sources are the audit log (settings, DNS, dependencies, integrations, a server being changed, a neighbour going offline or filling up), maintenance windows that began or ended, update runs and patch
plans, other alerts raised first, and a step in the server's own figures (a disk that jumped, memory that fell away).
Each is scored from three things:
| | How it counts |
|---|---|
| **How related** | The **same thing** the incident is about counts most; then **something it needs** (the host it runs on, the switch it is plugged into, a dependency you wrote — the nearest count more); then a **change that touches everything** (a setting, DNS); then **something that needs it**; and last something with **no link** the app knows of. |
| **What kind** | A maintenance window ending or a host going down is likelier than a change of settings, which is likelier than a note being written. Logins, notes, runbooks, tags and the like are not listed at all. A setting that bears on this kind of alert (thresholds, routing) counts for more. |
| **How close** | A change five minutes before counts for much more than one two hours before — the weight falls off over about an hour and never reaches zero inside the window. |
They are shown as *Likely* (55 and over), *Possible* (25 and over) and *Background* (the rest, behind a button); under 8 is left out, and at most ten are shown. The audit entry "pve1 stopped reporting"
and the alert about it are one event and shown once. It is a **list of suspects, not a diagnosis**: it says what happened and when, not why, and a change made on the machine itself — a package installed by
hand, a config file edited — is not something the app can see.
## History and how long it is kept
| What | Kept | Grain |
|---|---|---|
| A server's load, memory and each disk | **35 days** of samples (one every 10–15 minutes), **120 days** of hourly summaries, **~800 days** of daily summaries | summaries carry the lowest and highest |
| Endpoint check answers | **14 days** of every answer, hourly summaries for **120 days**, daily summaries for years | |
| Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day |
| Proxmox storage and Synology volume use | **400 days**, one reading an hour | |
| Home Assistant watched numbers | **35 days**, one reading every few minutes | |
| Incidents | **400 days** after they resolve | |
| Messages sent to event webhooks | The last **100 per webhook**, none older than **14 days** — see [WEBHOOKS.md](WEBHOOKS.md#the-delivery-log-and-sending-again) | |
| The audit log and the Diagnostic Log | Until automatic purging is switched on under *Settings → Data* (then 90 days by default, configurable) | |
A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there.
**Is it being trimmed?** *Settings → Data → Storage* measures the database, its backups and sessions, and every table by the room it takes, and for each history says how far back it reaches against how long it is meant to be kept:
*trimmed* (at its limit), *filling up* (not yet old enough to be trimmed), *not trimmed* (it holds far more than it should, so the trimming is not running or has not caught up) or *no limit* (the logs, until purging is
switched on). The last two are listed at the top as needing a look. Nothing there changes anything.
## Where to see it
| | Shows |
|---|---|
| **Dashboard**, **My work**, the menu badge | What needs attention now; what is yours |
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **Proxy hosts** | Each source's own view |
| **Weekly digest** and **monthly report** | The week and the month in one message |
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
| **Read API** (`/api/v1`) | [API.md](API.md) |
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
| **Status page**, **wall display**, **calendar feed** | Read-only views for others; made under *Settings → Share links* |
## Tuning, in questions and answers
**I am about to reboot a server.** Start maintenance on it (Maintenance page, or *Silence* on an alert): its alerts are held and, if it is still down at the end, announced as new.
**An alert is noisy because it keeps coming and going.** Let it flap (it will be damped after three raises an hour), then look at *Alerts → Noise* — it says what usually helps: a longer grace, a different limit.
For your own rules, raise the limit or lengthen the time.
**One server legitimately runs at 92% disk.** Give that server (or that one disk) its own limit — or *never alert about this disk* — in the server's *Settings* section.
**I want to be reminded about something later, not silence it.** Snooze it.
**I do not want to be woken for warnings.** Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours.
**A job on a server has to run every night.** Make a heartbeat for it and call it from the end of the script.
**I want my own limit, such as "memory above 85% for an hour".** Add an alert rule.
**An integration's alerts disappeared.** Check *Couldn't check everything* on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why.
**The app restarted and nothing about servers is alerting.** For 20 minutes after a restart servers are not judged, so the agents can report first.
**I want these figures in my own monitoring.** `/metrics` for Prometheus, the read API for scripts, event webhooks to be told.
## What monitoring does not do
- **It does not fix anything.** The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.)
- **It does not see inside a server that has no agent** — only what an endpoint check, a heartbeat or an integration can say about it.
- **It is not a replacement for a second pair of eyes on the app itself.** If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on `up{job="homelab-manager"}`
([PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md#alerting-rules-to-start-from)), or a check from another tool against `GET /health`.
- **It is as fresh as its sources.** A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.