- New docs/ folder: SSO, API, Prometheus and Grafana, webhooks, heartbeats and calendar, how monitoring works, and an index; the old root API.md is split across them - Read API: alerts carry snoozedUntil - DATABASE.md: table and migration counts, indexes, relations, integration config, settings and internal flags, delete rules, retention - ROLES.md: the sidebar table rebuilt from the menu - NOTIFICATIONS.md, INTEGRATIONS.md, BACKUP.md, AGENT_UPDATE.md: what had drifted - README: links to the guides; accounts cannot be deleted in the app Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
305 lines
27 KiB
Markdown
305 lines
27 KiB
Markdown
# How monitoring works
|
||
|
||
This page explains what Homelab Manager watches, how often, how a problem becomes an alert, and what happens next — notifications, acknowledgements, maintenance, incidents, history. It is the
|
||
"why did it do that?" page. Setting things up is in the other pages ([SSO](SSO.md), [API](API.md), [Prometheus and Grafana](PROMETHEUS-GRAFANA.md), [webhooks](WEBHOOKS.md),
|
||
[heartbeats](HEARTBEATS-AND-CALENDAR.md)); the list of every notification is [NOTIFICATIONS.md](../NOTIFICATIONS.md).
|
||
|
||
- [The big picture](#the-big-picture)
|
||
- [What is watched, and how often](#what-is-watched-and-how-often)
|
||
- [Servers and the agent](#servers-and-the-agent)
|
||
- [From a reading to an alert](#from-a-reading-to-an-alert)
|
||
- [The built-in checks](#the-built-in-checks)
|
||
- [Your own alert rules](#your-own-alert-rules)
|
||
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
|
||
- [Integrations](#integrations)
|
||
- [Forecasts](#forecasts)
|
||
- [Dependencies: one outage, one alert](#dependencies-one-outage-one-alert)
|
||
- [What happens to an alert](#what-happens-to-an-alert) — notify, acknowledge, silence, snooze, flapping, escalate
|
||
- [Incidents](#incidents)
|
||
- [History and how long it is kept](#history-and-how-long-it-is-kept)
|
||
- [Where to see it](#where-to-see-it)
|
||
- [Tuning, in questions and answers](#tuning-in-questions-and-answers)
|
||
- [What monitoring does not do](#what-monitoring-does-not-do)
|
||
|
||
## The big picture
|
||
|
||
```
|
||
agents ─────────┐
|
||
integrations ───┤ ┌──► Alerts page ──► acknowledge · snooze · silence
|
||
endpoint checks ┼─► checks ─► conditions ─┼──► notifications ─► Gotify · ntfy · email · webhook · push
|
||
heartbeats ─────┤ (rules) "a problem" ├──► event webhooks (machines)
|
||
your own rules ─┘ ├──► incidents (the record) and history
|
||
└──► /metrics and the read API
|
||
```
|
||
|
||
Three ideas run through everything:
|
||
|
||
1. **A check turns readings into *conditions*.** A condition is "this specific problem exists right now" and has a stable **key** (`offline:server:3`, `disk:server:3:/`) — the same problem always has the same key.
|
||
2. **Alerts are announced on a change, not on a schedule.** A problem is announced **once** when it appears and **once** when it ends. It is not repeated every time it is looked at.
|
||
3. **Silence is never "all clear".** When something cannot be *read* (an integration is down, a server's agent is quiet), what depended on it is **held exactly as it was** — neither raised nor cleared —
|
||
and the Alerts page lists it under *Couldn't check everything*. The app never turns "I couldn't look" into "nothing is wrong".
|
||
|
||
## What is watched, and how often
|
||
|
||
| What | How it gets to the app | How often |
|
||
|---|---|---|
|
||
| **Servers** (Linux, Windows) | The **agent** on each server **pushes** a report: load, memory, disks, listening ports, waiting updates, failed services, SMART and ZFS health, the OS it runs, scheduled tasks. The app never connects *to* a server. | Every **15 minutes** by default (set when the agent is installed) |
|
||
| **Proxmox**, **Synology** | The health pass **reads** them: node and storage usage, volume and disk status. | Every **15 minutes** |
|
||
| **UniFi** | A sync **reads** devices, networks, Wi-Fi names and clients and keeps them. | Every **5 minutes** |
|
||
| **Home Assistant** | A sync **reads** the entities you chose to watch and keeps their state and a history. | Every **5 minutes** |
|
||
| **Nginx Proxy Manager** | **Read** when its page is opened and by the alert check, at most once a minute. | On demand, cached 1 minute |
|
||
| **Semaphore**, **Gitea** | The automation check **reads** the latest run of each template or workflow. | Every **15 minutes** |
|
||
| **Endpoint checks** (web address, TCP port, TLS certificate) | The app **asks** each one itself. | Each check has its own interval (**5 minutes** by default); the scheduler looks **every minute** for the ones that are due |
|
||
| **Heartbeats** (cron jobs) | The job **calls** an address when it finishes. | When the job runs; judged every **minute** |
|
||
| **Secrets**, **certificates**, **domains**, **Tailscale keys** | Their expiry dates, with the certificate fetched live. | **Daily**, at the reminder time in Settings → Notifications |
|
||
| **Docker image updates**, **Proxmox backup results**, **Proxmox Backup verification** | Read and judged. | **Daily**, at the same time |
|
||
| **Your own alert rules** | Judged from the history the agents' reports build. | On the **15-minute** health pass |
|
||
| **Every alert, together** | The Alerts page, the incident recorder and the snooze timer look at the whole picture. | Alerts page: on open, reusing a result up to **1 minute** old · incidents: every **5 minutes** · snoozes: every **minute** · escalation and flapping: every **5 minutes** |
|
||
|
||
After the app **restarts**, server-based checks are paused for **20 minutes** so agents get a chance to report before any server is judged.
|
||
|
||
## Servers and the agent
|
||
|
||
The agent is a small script (a systemd timer on Linux, a scheduled task on Windows) that **only pushes**. Each server has its own token, which can do one thing: replace that server's own report. The app
|
||
replies "received" and nothing more, so it has no way to make a server run anything. Installation and updating are in the [README](../README.md) and [AGENT_UPDATE.md](../AGENT_UPDATE.md).
|
||
|
||
**When is a server "offline"?** When its agent has not reported for longer than the offline limit — **60 minutes** by default (Settings → Notifications → *Servers and disks*), or the server's own limit if it
|
||
has one (minimum 15). Four reports at the default interval, so one missed report is nothing. A server that **never** reported is "unknown": it has no agent to go quiet, so nothing alerts.
|
||
|
||
While a server is offline its other figures are stale, so it is **not also judged** on disks, services or your rules — its offline alert says it all.
|
||
|
||
**Per-server settings.** A server's *Settings* section can replace, for that server only: the offline limit, the disk-full percentage, the forecast window, how old its Proxmox guest's backup may be, a limit for
|
||
one disk or "never alert about this disk", and services never to count as failed (names or patterns such as `gupdate*`). Whatever it does not replace follows the global setting.
|
||
|
||
**What the report is used for:** the server's page and history (a sample every 10–15 minutes), the disk forecast, the failed-services and SMART/ZFS checks, new-port detection, the operating-system support
|
||
check, pending updates, the dependency map, and `/metrics`.
|
||
|
||
## From a reading to an alert
|
||
|
||
There are two paths, designed so they cannot disagree:
|
||
|
||
**1. The state-based checks (the notifications).** Every 15 minutes (endpoint checks and heartbeats: every minute) a check produces the **list of conditions that exist right now**. The app compares that list with
|
||
the one it remembered from last time:
|
||
|
||
- **new** → announced (a notification, an event webhook, a row in the incident record);
|
||
- **gone** → announced as resolved;
|
||
- **still there** → nothing;
|
||
- **its source could not be read this time** → carried over untouched (not resolved, not repeated).
|
||
|
||
The remembered list is in the database, so a restart does not forget what was already announced. The first ever run records what is already failing silently, as the baseline, instead of announcing it all.
|
||
|
||
**2. The Alerts page.** It asks the same checks, plus the daily ones (expiry, backups, updates, forecasts …), **live**, and turns what they find into one flat list. It ignores the per-event on/off toggles —
|
||
the page is for looking at, not for being interrupted by — and keeps problems that are under maintenance, marked as silenced instead of dropping them. A result is reused for up to a minute, and every source has a
|
||
20-second time limit so one hung integration cannot hang the page. A check that fails or times out costs only its own section, and appears under *Couldn't check everything*.
|
||
|
||
Each alert has:
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Severity** | `critical`, `warning` or `info`. Things that are *gone* (offline, a failing SMART disk, a disk over 95%) are critical; things that are *coming* (a forecast, an expiring certificate, an OS going out of support) are warnings or notes. |
|
||
| **Category** | `offline`, `disk`, `services`, `backup`, `updates`, `expiry`, `automation`, `integration`, `monitoring`, `tickets` — what the filter on the page and the [routing rules](#who-hears-about-it) work on. |
|
||
| **Source** | Where it comes from: Server, Proxmox, Synology, UniFi, Endpoint, Heartbeat … |
|
||
| **Target** | What a maintenance window would silence it with (a server, an integration, a device), where it makes sense. |
|
||
| **Key / id** | Stable identity; the same problem keeps it. Acknowledgements, snoozes, incidents and webhooks all follow it. |
|
||
|
||
## The built-in checks
|
||
|
||
Every threshold here is a default; where it can be changed, the place is named.
|
||
|
||
| Check | Raised when | Default | Changed in |
|
||
|---|---|---|---|
|
||
| **Server offline** | The agent has not reported for longer than the limit | 60 min · critical | Settings → Notifications; per server |
|
||
| **Disk nearly full** | A server disk, a Proxmox node's root filesystem or storage, or a Synology volume is at or over the limit | 90% · warning, **critical from 95%** | Settings → Notifications; per server, per disk |
|
||
| **Disk filling up** | A disk on a steady climb will be full within the window ([Forecasts](#forecasts)) | 14 days · warning, critical within 3 days | Settings → Notifications; per server |
|
||
| **Failed services** | A Linux agent reports failed systemd units | any | per server (ignore list) |
|
||
| **Disk health** | A disk's SMART self-assessment fails (critical) or an SSD has used ≥ 90% of its rated life (warning) | | not configurable |
|
||
| **ZFS / storage pool** | A ZFS pool (or Windows Storage Spaces pool) is not ONLINE | | not configurable |
|
||
| **Synology volume / disk** | A volume's status is not "normal"; a disk's status or SMART result is not normal, or it is over the bad-sector or under the remaining-life limit | | not configurable |
|
||
| **New port** | A TCP port starts listening that was not there when the agent first reported and that nobody noted on the server's *Ports* card | on | Settings → Notifications |
|
||
| **Network device offline** | A UniFi device has been offline too long | 10 min · critical | Settings → Notifications |
|
||
| **Operating system support** | A server's OS has passed, or is about to pass, its end of support | within 90 days · note; warning once ended | Settings → Notifications |
|
||
| **Updates waiting** | The agent reports updates waiting (and, where it can say, security ones and a restart needed) | | Updates page |
|
||
| **Backup stale / failed** | A Proxmox guest's newest backup is older than its limit, the latest run failed, or no job covers it; Proxmox Backup verification failed | 48 h | Settings → Notifications; per server |
|
||
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
|
||
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
|
||
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
|
||
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
|
||
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
|
||
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
||
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
||
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
||
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
|
||
|
||
## Your own alert rules
|
||
|
||
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
|
||
|
||
- **What to watch:** CPU load, memory in use, disk use (every disk, or one by name), or disk temperature.
|
||
- **When:** above or below a limit, and — for load, memory and disk use — **for how long**. "CPU above 90% for 30 minutes" means every reading in the last 30 minutes is above 90%; the readings are the
|
||
history the agent reports build (one every 10–15 minutes), so a single dip starts the count again. A disk temperature is only known as of the last report, so it is judged right now and has no time.
|
||
- **How serious:** warning or critical. **For which servers:** all of them, one, or those with a tag.
|
||
- **Try it now** shows which servers it would be raised on, from current data, before you save it.
|
||
|
||
A rule's finding is an **ordinary alert** with the key `rule:<rule id>:server:<server id>[:<disk>]`, in the *Monitoring* category: announced and resolved like the others, held back by maintenance on the
|
||
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is
|
||
offline or has not reported for 45 minutes is not judged.
|
||
|
||
## Endpoint checks and heartbeats
|
||
|
||
**Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds:
|
||
|
||
| Kind | What it asks |
|
||
|---|---|
|
||
| `http` | Fetches an `http(s)://` address. Counts as up if the status is in the accepted set (200–399 unless you say otherwise, e.g. `200-299, 301`), and — if you ask — the page contains a keyword. Follows up to 5 redirects. |
|
||
| `tcp` | Opens a connection to a host and port. |
|
||
| `tls` | Connects and reads the certificate; the answer is the days left. |
|
||
|
||
Per check: **interval** (1–1440 minutes, default 5), **timeout** (default 10 s), **failures before an alert** (default 2 in a row — one blip is not an alert), **certificate warning days** (default 14), an
|
||
optional **server it is linked to** (maintenance on that server silences it, and the alert links to it), and *accept a self-signed certificate*. Each answer — time, result, message — is kept, with hourly and
|
||
daily summaries for the charts and uptime figures.
|
||
|
||
- A check **fails an alert** (critical) after the configured number of failures in a row, and is announced as resolved when it works again. A **certificate** within its warning days is its own alert (`endpoint-cert:…`).
|
||
- What an administrator types in cannot be just anything: loopback, link-local and cloud-metadata addresses are refused, and the connection goes to the one address that passed the check. LAN and public hosts are fine.
|
||
- The app suggests checks it can already work out (a web port a server's agent sees open, a secret with a host) under *Endpoints → Suggested*, and proxied hosts Nginx Proxy Manager serves.
|
||
|
||
**Heartbeats** (Monitoring → Heartbeats) are the other direction: a cron job or script **calls an address when it finishes**, and the app alerts when it does *not*. A heartbeat has a **period** and a **grace**; it is
|
||
**late** when both have passed since the last call (a *warning*), and a call to `/fail` is **critical** at once. Details and examples: [HEARTBEATS-AND-CALENDAR.md](HEARTBEATS-AND-CALENDAR.md).
|
||
|
||
Both are judged **every minute**, with the same remember-and-compare bookkeeping as the other checks.
|
||
|
||
## Integrations
|
||
|
||
Integrations (Proxmox, Synology, UniFi, Semaphore, Gitea, Docker, Tailscale, Uptime Kuma, osTicket, phpIPAM, Proxmox Backup, Nginx Proxy Manager, Home Assistant, and the DNS providers) are **read** by the app,
|
||
using credentials an administrator enters under *Integrations*; what each needs is in [INTEGRATIONS.md](../INTEGRATIONS.md). Every call is recorded in the **Diagnostic Log**.
|
||
|
||
- **An integration is "failing"** when its last **3 calls in a row** (Settings → Notifications) failed. Failures are counted per *kind* of service (all Proxmox integrations share a streak), in memory, so a
|
||
restart starts the count again. A *down* notification goes out at the threshold and a *recovered* one on the next success. The moments are written to the audit log, and they feed the uptime record on the
|
||
integration's page.
|
||
- **When an integration cannot be read, what it reported is held, not cleared.** A Proxmox node that cannot be read does not make its storage alerts disappear (or appear); the Alerts page says it could not check.
|
||
The synced ones (UniFi, Home Assistant) are only judged while their last good read is recent (**20 minutes**).
|
||
- **An integration can explain other alerts.** A service that stops answering because the integration it needs is failing is folded under it ([Dependencies](#dependencies-one-outage-one-alert)).
|
||
|
||
## Forecasts
|
||
|
||
A forecast says "this will be full in N days" from a **straight line through the recent history**, and is deliberately cautious:
|
||
|
||
- It needs **at least six readings covering at least three days**, and a **steady** climb — the line has to explain at least 90% of the movement (R² ≥ 0.9). A big copy that then gets deleted, or a disk that
|
||
wobbles, is **not forecast**. A slope under 0.05 points a day counts as flat.
|
||
- It looks at the **last two weeks**; if those are flat it looks at the **last ninety days**, so a slow, steady climb is seen too.
|
||
- A disk that is **already over its limit** is left to the ordinary "nearly full" alert.
|
||
- It is shown as an alert when it falls **within the window** (Settings → Notifications → *Warn when a disk will fill within*, 14 days by default; per server; 0 turns it off), as a line in the weekly digest, and as
|
||
`homelab_server_disk_days_to_full` / `homelab_storage_days_to_full` in [`/metrics`](PROMETHEUS-GRAFANA.md). It is **not** sent as a notification the moment it appears.
|
||
- **Servers:** from the agents' history. **Proxmox storages and Synology volumes:** the app writes down what each health pass reads, one reading an hour, kept 400 days, and forecasts those the same way; it is
|
||
shown beside each storage on the Proxmox page and each volume on the Synology page. Until there are a few days of readings the page says it is collecting history.
|
||
|
||
## Dependencies: one outage, one alert
|
||
|
||
If a Proxmox host goes down, its fifteen guests go quiet too. Fifteen alerts about the symptom would bury the one about the cause. The app keeps a **graph of what needs what**:
|
||
|
||
- a server **runs on** a host (set by hand, or worked out from its Proxmox node);
|
||
- a UniFi device is **connected through** another, and a **server is plugged into** a switch or access point (matched by IP address);
|
||
- **dependencies you write** (*Map → Dependencies*: "Gitea needs the server it is installed on", "the NAS needs the rack switch"), and the app offers suggestions it can work out.
|
||
|
||
When something is down and what depends on it is down too, the dependants' alerts are **folded under** the first: shown nested, marked *caused by …*, not announced on their own, and the host's message says
|
||
"— also takes down web and db". If the host comes back and a dependant **stays** down, it is announced then, as new: **an outage is never swallowed.**
|
||
|
||
## What happens to an alert
|
||
|
||
Several things can quiet an alert. They are different, and it matters which you use:
|
||
|
||
| | Promise about the thing? | Hides it? | Notifications | Ends |
|
||
|---|---|---|---|---|
|
||
| **Acknowledge** ("I'm on it") | No — it says someone has seen it, why, who has it | No: still listed, still counted in `/metrics`; leaves the *needs attention* count | Stops [escalation](#escalation) | When you choose (an hour … 30 days), or when the alert has been gone half an hour |
|
||
| **Snooze** ("remind me tomorrow 09:00") | No | Yes, held back like maintenance | Raise/clear messages held; **one reminder** at the end if it is still there | At the time you set (up to 90 days), or earlier with *Wake now* |
|
||
| **Maintenance window** | Yes — "I'm working on this" (5 minutes to a week) | Yes, marked *silenced* | Held for the server, integration, DNS provider or device | At its end; what is still there is then announced as **new** |
|
||
| **Folded under a host** | — automatic | Nested under its cause | Not announced on its own | When the cause ends |
|
||
| **Flapping** | — automatic | No | Raise and clear messages held; one "flapping" message, one "settled" message | After an hour of calm |
|
||
|
||
### Who hears about it
|
||
|
||
Notifications go to the channels that are on (**Gotify, ntfy, email, a webhook, push to phones and browsers**), narrowed by **routing rules** (first match wins: by kind, by severity, by the server's tags) and each
|
||
browser's own wishes. **Quiet hours** hold messages and send one digest at the end of the window. Every message the app can send, and what toggles it, is in [NOTIFICATIONS.md](../NOTIFICATIONS.md).
|
||
**Event webhooks** are a separate way of being told, for machines: they hear every raise and clear, flapping and snoozed included ([WEBHOOKS.md](WEBHOOKS.md)).
|
||
|
||
### Flapping
|
||
|
||
An alert **raised three times within an hour** (Settings → Notifications; 0 = off) is called *flapping*: one message says so, its further raise and clear messages are held, and it is marked on the page. After
|
||
an hour with no raise or clear, one *settled* message says whether it is still there. The alert itself is untouched: still raised, still counted. *Alerts → Noise* lists what keeps coming back.
|
||
|
||
### Escalation
|
||
|
||
An alert raised for a while with **nobody on it** can be told again in up to three steps ("after 30 minutes: push; after 2 hours: email the neighbour who has the key"), with its own channels and an optional extra
|
||
email address. An acknowledgement stops the rest; problems under maintenance, folded under a host or only informational never escalate. Looked at every five minutes.
|
||
|
||
### Maintenance
|
||
|
||
*Maintenance* (a window of 5 minutes to a week, started by hand, in bulk, from a **recurring schedule**, or mirrored from Uptime Kuma) silences what is about the target: a server's offline/disk/service alerts and
|
||
its endpoint checks and heartbeats; an integration's storage, backup and automation alerts; "integration down" for that service type; a UniFi device's offline alert. What depends on it can be put under
|
||
maintenance with it. A silenced problem is **not remembered as known**, so one still there when the window ends **alerts then, as new**.
|
||
|
||
## Incidents
|
||
|
||
Every time an alert was raised and later went away is an **incident**: when it opened and closed, how bad it got, what it was about, what explained it, who acknowledged or was handed it, and — on its
|
||
timeline — what was done to the thing in the hour around it (maintenance, a restart, a settings change). People can add notes. The recorder looks at the alerts **every five minutes** whether or not anyone has
|
||
the page open; the same alert coming back is a **new** incident. Resolved incidents are kept **400 days**. They are on **Operations → Alerts → History**, in the read API (`/api/v1/incidents`) and in the
|
||
monthly report.
|
||
|
||
## History and how long it is kept
|
||
|
||
| What | Kept | Grain |
|
||
|---|---|---|
|
||
| A server's load, memory and each disk | **35 days** of samples (one every 10–15 minutes), **120 days** of hourly summaries, **~800 days** of daily summaries | summaries carry the lowest and highest |
|
||
| Endpoint check answers | **14 days** of every answer, hourly summaries for **120 days**, daily summaries for years | |
|
||
| Uptime of servers and integrations | The last weeks, worked out from the moments each stopped and resumed (in the audit log) | one figure a day |
|
||
| Proxmox storage and Synology volume use | **400 days**, one reading an hour | |
|
||
| Home Assistant watched numbers | **35 days**, one reading every few minutes | |
|
||
| Incidents | **400 days** after they resolve | |
|
||
| The audit log and the Diagnostic Log | Until automatic purging is switched on under *Settings → Data* (then 90 days by default, configurable) | |
|
||
|
||
A chart asks for 24 hours, 7, 30, 90 days or a year and is drawn from whichever of these is fine enough and still there.
|
||
|
||
## Where to see it
|
||
|
||
| | Shows |
|
||
|---|---|
|
||
| **Dashboard**, **My work**, the menu badge | What needs attention now; what is yours |
|
||
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
|
||
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
|
||
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **Proxy hosts** | Each source's own view |
|
||
| **Weekly digest** and **monthly report** | The week and the month in one message |
|
||
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
|
||
| **Read API** (`/api/v1`) | [API.md](API.md) |
|
||
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
|
||
| **Status page**, **wall display**, **calendar feed** | Read-only views for others; made under *Settings → Share links* |
|
||
|
||
## Tuning, in questions and answers
|
||
|
||
**I am about to reboot a server.** Start maintenance on it (Maintenance page, or *Silence* on an alert): its alerts are held and, if it is still down at the end, announced as new.
|
||
|
||
**An alert is noisy because it keeps coming and going.** Let it flap (it will be damped after three raises an hour), then look at *Alerts → Noise* — it says what usually helps: a longer grace, a different limit.
|
||
For your own rules, raise the limit or lengthen the time.
|
||
|
||
**One server legitimately runs at 92% disk.** Give that server (or that one disk) its own limit — or *never alert about this disk* — in the server's *Settings* section.
|
||
|
||
**I want to be reminded about something later, not silence it.** Snooze it.
|
||
|
||
**I do not want to be woken for warnings.** Add a routing rule: kind or severity → the channels you want (a rule with no channels tells nobody; the Alerts page still shows it), and set quiet hours.
|
||
|
||
**A job on a server has to run every night.** Make a heartbeat for it and call it from the end of the script.
|
||
|
||
**I want my own limit, such as "memory above 85% for an hour".** Add an alert rule.
|
||
|
||
**An integration's alerts disappeared.** Check *Couldn't check everything* on the Alerts page: the integration could not be read, so its alerts are held. The Diagnostic Log says why.
|
||
|
||
**The app restarted and nothing about servers is alerting.** For 20 minutes after a restart servers are not judged, so the agents can report first.
|
||
|
||
**I want these figures in my own monitoring.** `/metrics` for Prometheus, the read API for scripts, event webhooks to be told.
|
||
|
||
## What monitoring does not do
|
||
|
||
- **It does not fix anything.** The agents only push, and nothing an alert says is done automatically. (Updates and playbooks can be run from the app by an operator, on purpose.)
|
||
- **It does not see inside a server that has no agent** — only what an endpoint check, a heartbeat or an integration can say about it.
|
||
- **It is not a replacement for a second pair of eyes on the app itself.** If Homelab Manager is down, nothing here speaks. Watch it from outside: a Prometheus rule on `up{job="homelab-manager"}`
|
||
([PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md#alerting-rules-to-start-from)), or a check from another tool against `GET /health`.
|
||
- **It is as fresh as its sources.** A server's figures are as old as its last report (up to 15 minutes), a forecast needs days of history, and an integration's state is the last good read.
|