Service-level targets with error budgets, and quarterly and annual reports

Monitoring -> Service levels: a target (99.9 %) on an endpoint check, a server or an
integration, measured over calendar months. Shows the figure, downtime used, the
error budget left, and how the last six months went. 80 % of the budget used is a
warning and an overspent budget a critical alert. Endpoints are measured day by day,
servers and integrations to the minute from the audit log.

The report page gains Quarter and Year (same figures, with a Service levels section),
and two settings send the quarterly and annual report after the month that ends them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-21 22:20:07 +02:00
1 parent 31044b2b64
commit 3b31b16796
31 files changed
+7706 -61

No files matched your search

+23 -1
View File
@@ -10,6 +10,7 @@ This page explains what Homelab Manager watches, how often, how a problem become
- [From a reading to an alert](#from-a-reading-to-an-alert)
- [The built-in checks](#the-built-in-checks)
- [What is normal for the time](#what-is-normal-for-the-time)
- [Service levels](#service-levels)
- [Your own alert rules](#your-own-alert-rules)
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
- [Integrations](#integrations)
@@ -131,6 +132,7 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
| **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) |
| **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) |
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
@@ -155,6 +157,25 @@ a server that has stopped reporting (its offline alert says that). Switch it off
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
## Service levels
**Monitoring → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the
downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and
where today's pace would end the month, once a day has passed), and how each of the last six months went.
How downtime is measured, so the figure can be trusted — and where it cannot:
| Measured on | From | Precision |
|---|---|---|
| **Endpoint check** | The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. | A day's failed share is spread over the whole day |
| **Server** | The moments the audit log recorded it stopping reporting and reporting again | To the minute; only from when the server was created |
| **Integration** | The moments calls to that *kind* of service started failing and worked again | To the minute; only from when the app began recording them |
**Planned maintenance is not taken out**: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing.
When **80 %** of the budget is used it is a **warning**, and when it is overspent a **critical** alert (key `slo:<id>`, *Monitoring*, source *Service level*): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks —
and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in `/metrics` (`homelab_service_level_*`) and in the monthly, quarterly and annual reports ([below](#history-and-how-long-it-is-kept)).
## Your own alert rules
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
@@ -324,7 +345,8 @@ switched on). The last two are listed at the top as needing a look. Nothing ther
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **UPS**, **Proxy hosts** | Each source's own view |
| **Weekly digest** and **monthly report** | The week and the month in one message |
| **Service levels** | Each target and what is left of this month's error budget |
| **Weekly digest** and **monthly, quarterly and annual reports** | The week, the month, the quarter and the year in one message |
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
| **Read API** (`/api/v1`) | [API.md](API.md) |
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |