Service-level targets with error budgets, and quarterly and annual reports
Monitoring -> Service levels: a target (99.9 %) on an endpoint check, a server or an integration, measured over calendar months. Shows the figure, downtime used, the error budget left, and how the last six months went. 80 % of the budget used is a warning and an overspent budget a critical alert. Endpoints are measured day by day, servers and integrations to the minute from the audit log. The report page gains Quarter and Year (same figures, with a Service levels section), and two settings send the quarterly and annual report after the month that ends them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
31044b2b64
commit
3b31b16796
31 files changed
+7706
-61
No files matched your search
+23
-1
@@ -10,6 +10,7 @@ This page explains what Homelab Manager watches, how often, how a problem become
|
||||
- [From a reading to an alert](#from-a-reading-to-an-alert)
|
||||
- [The built-in checks](#the-built-in-checks)
|
||||
- [What is normal for the time](#what-is-normal-for-the-time)
|
||||
- [Service levels](#service-levels)
|
||||
- [Your own alert rules](#your-own-alert-rules)
|
||||
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
|
||||
- [Integrations](#integrations)
|
||||
@@ -131,6 +132,7 @@ Every threshold here is a default; where it can be changed, the place is named.
|
||||
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
||||
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
||||
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
||||
| **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) |
|
||||
| **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) |
|
||||
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
|
||||
|
||||
@@ -155,6 +157,25 @@ a server that has stopped reporting (its offline alert says that). Switch it off
|
||||
|
||||
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
|
||||
|
||||
## Service levels
|
||||
|
||||
**Monitoring → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the
|
||||
downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and
|
||||
where today's pace would end the month, once a day has passed), and how each of the last six months went.
|
||||
|
||||
How downtime is measured, so the figure can be trusted — and where it cannot:
|
||||
|
||||
| Measured on | From | Precision |
|
||||
|---|---|---|
|
||||
| **Endpoint check** | The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. | A day's failed share is spread over the whole day |
|
||||
| **Server** | The moments the audit log recorded it stopping reporting and reporting again | To the minute; only from when the server was created |
|
||||
| **Integration** | The moments calls to that *kind* of service started failing and worked again | To the minute; only from when the app began recording them |
|
||||
|
||||
**Planned maintenance is not taken out**: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing.
|
||||
|
||||
When **80 %** of the budget is used it is a **warning**, and when it is overspent a **critical** alert (key `slo:<id>`, *Monitoring*, source *Service level*): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks —
|
||||
and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in `/metrics` (`homelab_service_level_*`) and in the monthly, quarterly and annual reports ([below](#history-and-how-long-it-is-kept)).
|
||||
|
||||
## Your own alert rules
|
||||
|
||||
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
|
||||
@@ -324,7 +345,8 @@ switched on). The last two are listed at the top as needing a look. Nothing ther
|
||||
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
|
||||
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
|
||||
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **UPS**, **Proxy hosts** | Each source's own view |
|
||||
| **Weekly digest** and **monthly report** | The week and the month in one message |
|
||||
| **Service levels** | Each target and what is left of this month's error budget |
|
||||
| **Weekly digest** and **monthly, quarterly and annual reports** | The week, the month, the quarter and the year in one message |
|
||||
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
|
||||
| **Read API** (`/api/v1`) | [API.md](API.md) |
|
||||
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
|
||||
|
||||
@@ -292,6 +292,8 @@ All gauges. The same table is checked by a test against what `/metrics` really s
|
||||
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
||||
| `homelab_endpoint_value` | `name`, `kind` | The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
|
||||
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
||||
| `homelab_service_level_target_percent` | `name`, `subject`, `type` | The availability a service-level target asks for. |
|
||||
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
|
||||
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
|
||||
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
|
||||
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
||||
|
||||
Reference in new issue
Block a user