Service-level targets with error budgets, and quarterly and annual reports
Monitoring -> Service levels: a target (99.9 %) on an endpoint check, a server or an integration, measured over calendar months. Shows the figure, downtime used, the error budget left, and how the last six months went. 80 % of the budget used is a warning and an overspent budget a critical alert. Endpoints are measured day by day, servers and integrations to the minute from the audit log. The report page gains Quarter and Year (same figures, with a Service levels section), and two settings send the quarterly and annual report after the month that ends them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
31044b2b64
commit
3b31b16796
31 files changed
+7706
-61
No files matched your search
+20
-3
@@ -23,8 +23,8 @@ what the app actually creates.
|
|||||||
|---|---|
|
|---|---|
|
||||||
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
|
||||||
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
|
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
|
||||||
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0054`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0055`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
||||||
| **Tables** | 64 application tables, plus drizzle's own `__drizzle_migrations`. |
|
| **Tables** | 65 application tables, plus drizzle's own `__drizzle_migrations`. |
|
||||||
|
|
||||||
**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`.
|
**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`.
|
||||||
|
|
||||||
@@ -845,6 +845,23 @@ A UPS at a moment, for the charts: one every five minutes, and every minute whil
|
|||||||
| `runtime_seconds` | integer | |
|
| `runtime_seconds` | integer | |
|
||||||
| `on_battery` | bool NOT NULL, default 0 | |
|
| `on_battery` | bool NOT NULL, default 0 | |
|
||||||
|
|
||||||
|
#### `service_levels`
|
||||||
|
|
||||||
|
A service-level target (Monitoring → Service levels): what a thing is held to, measured over calendar months. No foreign keys: the thing measured is a plain reference, and a target whose subject is removed is shown as having nothing to measure.
|
||||||
|
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | integer PK | |
|
||||||
|
| `name` | text NOT NULL | |
|
||||||
|
| `subject_type` | text NOT NULL | `endpoint` (an endpoint check), `server` or `integration`. |
|
||||||
|
| `subject_id` | integer NOT NULL | The id of that thing. |
|
||||||
|
| `target` | real NOT NULL | Percent, 90 to 99.999. |
|
||||||
|
| `alert` | bool NOT NULL, default 1 | Warn at 80 % of the month's error budget, and when it is overspent. |
|
||||||
|
| `created_by` | text | |
|
||||||
|
| `created_at` | text NOT NULL | |
|
||||||
|
|
||||||
|
Nothing about how a target is doing is stored: it is worked out when asked, from `endpoint_daily` and the audit log.
|
||||||
|
|
||||||
#### `alert_rules`
|
#### `alert_rules`
|
||||||
|
|
||||||
Limits of the administrator's own (Monitoring → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule:<id>:server:<server>[:<disk>]`.
|
Limits of the administrator's own (Monitoring → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule:<id>:server:<server>[:<disk>]`.
|
||||||
@@ -1129,7 +1146,7 @@ that were never changed aren't stored; the app fills in defaults when reading.
|
|||||||
| `selfBackup` | The daily backup of this database: `enabled` (true), `time` (`HH:MM`, default `03:30`, in the notification time zone) and `keep` (14 — how many daily ones are kept). |
|
| `selfBackup` | The daily backup of this database: `enabled` (true), `time` (`HH:MM`, default `03:30`, in the notification time zone) and `keep` (14 — how many daily ones are kept). |
|
||||||
| `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. |
|
| `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. |
|
||||||
| `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. |
|
| `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. |
|
||||||
| `monthlyReport` | The monthly report: `enabled` (false), `day` (1–28, default 1) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
|
| `monthlyReport` | The monthly report: `enabled` (false), `day` (1–28, default 1) and `time` (`HH:MM`, default `08:00`), in the notification time zone; `quarterly` and `annual` (both false) add the quarter's report after March, June, September and December, and the year's after December. |
|
||||||
| `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
|
| `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
|
||||||
| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be), `newPortAlerts` (true — warn when a server starts listening on a TCP port that is not in its baseline and not noted), `flapRaises` (3; 0 = off — raises of one alert within an hour that make it "flapping" so its messages are held; see NOTIFICATIONS.md), `deviceOfflineMinutes` (10 — how long a UniFi device may be offline before it is raised), `osSupportWarnDays` (90; 0 = off), `baselineAlerts` (true — warn when CPU or memory over the last hour is far above what the server's own history says is normal for this hour of the week; see docs/MONITORING.md). The baselines themselves are not stored: they are worked out from `server_metrics_rollup` (hourly rows, `grain = 'h'`) when needed. A server can override each of these (see `servers.thresholds`). |
|
| `healthChecks` | `restoreTestDays` (0 = off — how long a guest's last passing restore test counts; a failed one is always raised), `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page), `backupStaleHours` (48; 0 = off — how old a Proxmox guest's newest backup may be), `newPortAlerts` (true — warn when a server starts listening on a TCP port that is not in its baseline and not noted), `flapRaises` (3; 0 = off — raises of one alert within an hour that make it "flapping" so its messages are held; see NOTIFICATIONS.md), `deviceOfflineMinutes` (10 — how long a UniFi device may be offline before it is raised), `osSupportWarnDays` (90; 0 = off), `baselineAlerts` (true — warn when CPU or memory over the last hour is far above what the server's own history says is normal for this hour of the week; see docs/MONITORING.md). The baselines themselves are not stored: they are worked out from `server_metrics_rollup` (hourly rows, `grain = 'h'`) when needed. A server can override each of these (see `servers.thresholds`). |
|
||||||
| `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). |
|
| `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). |
|
||||||
|
|||||||
+10
-1
@@ -168,7 +168,16 @@ A scheduled backup of the app's own database that fails sends one message throug
|
|||||||
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
| "Homelab monthly report — <month>" | On the chosen day of the month (1–28) and time, in the notification time zone, **only if switched on** under Settings → Notifications. One plain-text message through every channel that is on, about the month just over; quiet hours hold it like any other notification. |
|
||||||
|
|
||||||
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
|
Availability of servers and endpoints, incidents (how many, how long they took, the longest), updates and patch plan runs, restore tests and the state of the backups, and what was renewed — a topic only where there
|
||||||
is something to say. The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
|
is something to say, now with a **Service levels** topic (how each target did against its error budget). The same report is on the Monthly report page. *Preview last month* and *Send last month now* are on the settings page.
|
||||||
|
|
||||||
|
With **Also send a quarterly report** on, "Homelab quarterly report — Q3 2026" follows the monthly one after the month that ends a quarter (March, June, September, December), and with **Also send an annual report** on, "Homelab annual report — 2026" follows
|
||||||
|
the December one. Both are the same report over three or twelve months, sent the same way; both are off until switched on.
|
||||||
|
|
||||||
|
## Service levels
|
||||||
|
|
||||||
|
A service-level target (Monitoring → Service levels) is announced like the other server problems — under *Server offline, disk nearly full …* (`healthAlerts`), once, with a recovery message when it clears — when **80 %** of the month's error budget is used
|
||||||
|
(a warning: "Gitea: 90% of the error budget for its 99.9% target is used up for October 2026 — down 40 min of 45 min allowed, 5 min left") and again when it is **overspent** (critical: "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed").
|
||||||
|
It is *Monitoring* for routing, and it ends when the month turns over. Maintenance does not hold it back: planned downtime counts. Each target has its own switch.
|
||||||
|
|
||||||
## Network devices
|
## Network devices
|
||||||
|
|
||||||
|
|||||||
@@ -691,12 +691,18 @@ alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n
|
|||||||
status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in
|
status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in
|
||||||
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md).
|
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md).
|
||||||
|
|
||||||
**Monthly report** (Operations → Monthly report; and, if switched on, a message on a day of the month). How a calendar month went, on one page that can be read, printed and copied as text:
|
**Service levels** (Monitoring → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
|
||||||
|
month allows about 45 minutes; 99 % about 7 hours). A target is set on an **endpoint check** (measured from its answers, day by day), a **server** or an **integration** (measured from when the audit log recorded it going away and
|
||||||
|
coming back, to the minute), and is measured over calendar months in the notification time zone. Each card shows the figure so far, the downtime used, what is allowed and what is left, how much of the budget is used (and where the
|
||||||
|
pace so far would end the month), and how each of the last six months went. When **80 %** of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over — switchable per target. Planned maintenance is
|
||||||
|
**not** taken out: down is down. A target with nothing recorded for the period says so rather than guessing. `/metrics` has the target, the figure and the minutes left; administrators set targets, everyone sees them.
|
||||||
|
|
||||||
|
**Monthly, quarterly and annual reports** (Operations → Monthly report; and, if switched on, a message on a day of the month). How a calendar month went, on one page that can be read, printed and copied as text:
|
||||||
**availability** (servers, from when each stopped reporting and reported again; endpoints, from every check's answers, kept day by day for years; integrations), **incidents** (how many began and
|
**availability** (servers, from when each stopped reporting and reported again; endpoints, from every check's answers, kept day by day for years; integrations), **incidents** (how many began and
|
||||||
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
|
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
|
||||||
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
|
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
|
||||||
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
|
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
|
||||||
and the backups are a snapshot, not a history. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*.
|
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
|
||||||
|
|
||||||
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
|
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
|
||||||
week through your notification channels: what is wrong now, what was down and for how
|
week through your notification channels: what is wrong now, what was down and for how
|
||||||
|
|||||||
@@ -57,7 +57,8 @@ page's own top-level link.
|
|||||||
| Updates (and Patch plans) | `/updates` | ✅ | ✅ | ✅ |
|
| Updates (and Patch plans) | `/updates` | ✅ | ✅ | ✅ |
|
||||||
| Coming up | `/coming-up` | ✅ | ✅ | ✅ |
|
| Coming up | `/coming-up` | ✅ | ✅ | ✅ |
|
||||||
| To do | `/todo` | ✅ | ✅ | ✅ |
|
| To do | `/todo` | ✅ | ✅ | ✅ |
|
||||||
| Monthly report | `/reports` | ✅ | ✅ | ✅ |
|
| Monthly report (and quarterly, annual) | `/reports` | ✅ | ✅ | ✅ |
|
||||||
|
| Service levels | `/service-levels` | ✅ | ✅ | ✅ |
|
||||||
| Change log (and the raw audit log, `?view=audit`) | `/changes` | ❌ | ✅ | ✅ |
|
| Change log (and the raw audit log, `?view=audit`) | `/changes` | ❌ | ✅ | ✅ |
|
||||||
| **Administration** | | | | |
|
| **Administration** | | | | |
|
||||||
| Integrations | `/integrations` | ✅ | ✅ | ✅ |
|
| Integrations | `/integrations` | ✅ | ✅ | ✅ |
|
||||||
@@ -125,6 +126,7 @@ even further, to admin only:
|
|||||||
- **Proxy hosts** (Network → Proxy hosts): everyone signed in sees what Nginx Proxy Manager serves; **Read now** needs operator; adding the integration (its login) is an administrator's job.
|
- **Proxy hosts** (Network → Proxy hosts): everyone signed in sees what Nginx Proxy Manager serves; **Read now** needs operator; adding the integration (its login) is an administrator's job.
|
||||||
- **Home Assistant** (Monitoring → Home Assistant): everyone signed in sees what is watched, its state and history; choosing what to watch, its limits, stopping and **Read now** need operator (the list of all
|
- **Home Assistant** (Monitoring → Home Assistant): everyone signed in sees what is watched, its state and history; choosing what to watch, its limits, stopping and **Read now** need operator (the list of all
|
||||||
entities is read live, so only operators see it); adding the integration (its token) is an administrator's job.
|
entities is read live, so only operators see it); adding the integration (its token) is an administrator's job.
|
||||||
|
- **Service levels** (Monitoring → Service levels): everyone signed in sees the targets and what is left of the budget; setting, changing and removing one is administrator-only, and each is in the audit log.
|
||||||
- **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map.
|
- **UPS** (Monitoring → UPS): everyone signed in sees each UPS, its history and what is on it; adding one (its address and SNMP community) is an administrator's job, like any integration, and what it powers is written in the dependency map.
|
||||||
- **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone;
|
- **Endpoints**: everyone sees the checks and how they are doing; adding, changing and removing one is administrator-only (each makes the server connect somewhere); that includes the value a check judges and the header (a token) it sends, whose value is never shown again to anyone;
|
||||||
*Check now* needs operator.
|
*Check now* needs operator.
|
||||||
|
|||||||
+23
-1
@@ -10,6 +10,7 @@ This page explains what Homelab Manager watches, how often, how a problem become
|
|||||||
- [From a reading to an alert](#from-a-reading-to-an-alert)
|
- [From a reading to an alert](#from-a-reading-to-an-alert)
|
||||||
- [The built-in checks](#the-built-in-checks)
|
- [The built-in checks](#the-built-in-checks)
|
||||||
- [What is normal for the time](#what-is-normal-for-the-time)
|
- [What is normal for the time](#what-is-normal-for-the-time)
|
||||||
|
- [Service levels](#service-levels)
|
||||||
- [Your own alert rules](#your-own-alert-rules)
|
- [Your own alert rules](#your-own-alert-rules)
|
||||||
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
|
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
|
||||||
- [Integrations](#integrations)
|
- [Integrations](#integrations)
|
||||||
@@ -131,6 +132,7 @@ Every threshold here is a default; where it can be changed, the place is named.
|
|||||||
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
|
||||||
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
| **Heartbeat late / failed** | See [below](#endpoint-checks-and-heartbeats) | | per heartbeat |
|
||||||
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
| **Home Assistant / Nginx Proxy Manager** | An entity unavailable 15 minutes or past your limit; a certificate in use ending within 14 days, or a host NPM cannot set up | | Home Assistant page |
|
||||||
|
| **Service level** | 80 % of a target's error budget used (warning), or overspent (critical), for the month | per target | Service levels page; [below](#service-levels) |
|
||||||
| **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) |
|
| **UPS** | Mains power lost (critical); battery nearly empty (critical); battery to be replaced; load of 90 % or more (warning) | | UPS page; [INTEGRATIONS.md](../INTEGRATIONS.md#ups-nut-or-snmp) |
|
||||||
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
|
| **Your own rules** | [Below](#your-own-alert-rules) | | Monitoring → Alert rules |
|
||||||
|
|
||||||
@@ -155,6 +157,25 @@ a server that has stopped reporting (its offline alert says that). Switch it off
|
|||||||
|
|
||||||
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
|
The same baselines draw the grey **usual** line and band on a server's *History → Trends* (up to 30 days; the median, two spreads either side) and the "usually about 8 % now" beside each figure.
|
||||||
|
|
||||||
|
## Service levels
|
||||||
|
|
||||||
|
**Monitoring → Service levels** holds a number against a service: **99.9 % for Gitea**. A target is set on an endpoint check, a server or an integration and is judged over **calendar months** (on the notification time zone's clock). Its **error budget** is the
|
||||||
|
downtime it allows — minutes in the month × (1 − target) — so 99.9 % allows 44.6 minutes of a 31-day month and 99 % about 7½ hours. The page shows, for this month: the figure so far, the downtime used, what is allowed, what is left, the share of the budget used (and
|
||||||
|
where today's pace would end the month, once a day has passed), and how each of the last six months went.
|
||||||
|
|
||||||
|
How downtime is measured, so the figure can be trusted — and where it cannot:
|
||||||
|
|
||||||
|
| Measured on | From | Precision |
|
||||||
|
|---|---|---|
|
||||||
|
| **Endpoint check** | The check's daily figures: each day's share of failed checks, over the part of that day that is in the period (days are UTC days). Days with no checks are not known, and count neither way. | A day's failed share is spread over the whole day |
|
||||||
|
| **Server** | The moments the audit log recorded it stopping reporting and reporting again | To the minute; only from when the server was created |
|
||||||
|
| **Integration** | The moments calls to that *kind* of service started failing and worked again | To the minute; only from when the app began recording them |
|
||||||
|
|
||||||
|
**Planned maintenance is not taken out**: down is down. A thing with no record for the period has no figure, and the page says so rather than guessing.
|
||||||
|
|
||||||
|
When **80 %** of the budget is used it is a **warning**, and when it is overspent a **critical** alert (key `slo:<id>`, *Monitoring*, source *Service level*): announced and recovered like the others, snoozed, flapping-damped, routed, an event for webhooks —
|
||||||
|
and gone when the month turns over and the budget starts again. Each target can have its alert switched off. The figures are in `/metrics` (`homelab_service_level_*`) and in the monthly, quarterly and annual reports ([below](#history-and-how-long-it-is-kept)).
|
||||||
|
|
||||||
## Your own alert rules
|
## Your own alert rules
|
||||||
|
|
||||||
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
|
Where the built-in checks do not fit, **Monitoring → Alert rules** (administrators write them, everyone sees them and where each is raised) lets you set a limit of your own on one figure of the servers:
|
||||||
@@ -324,7 +345,8 @@ switched on). The last two are listed at the top as needing a look. Nothing ther
|
|||||||
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
|
| **Alerts** (Now · History · Noise) | Every alert, with acknowledge, snooze, silence; the incident record; what keeps coming back |
|
||||||
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
|
| A **server's page** | Its figures, history, health, ports, tasks, notes, jobs, what it runs on and what runs on it |
|
||||||
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **UPS**, **Proxy hosts** | Each source's own view |
|
| **Endpoints**, **Heartbeats**, **Proxmox**, **Synology**, **Network devices**, **Home Assistant**, **UPS**, **Proxy hosts** | Each source's own view |
|
||||||
| **Weekly digest** and **monthly report** | The week and the month in one message |
|
| **Service levels** | Each target and what is left of this month's error budget |
|
||||||
|
| **Weekly digest** and **monthly, quarterly and annual reports** | The week, the month, the quarter and the year in one message |
|
||||||
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
|
| **`/metrics`** → Prometheus, Grafana | [PROMETHEUS-GRAFANA.md](PROMETHEUS-GRAFANA.md) |
|
||||||
| **Read API** (`/api/v1`) | [API.md](API.md) |
|
| **Read API** (`/api/v1`) | [API.md](API.md) |
|
||||||
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
|
| **Event webhooks** | [WEBHOOKS.md](WEBHOOKS.md) |
|
||||||
|
|||||||
@@ -292,6 +292,8 @@ All gauges. The same table is checked by a test against what `/metrics` really s
|
|||||||
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
||||||
| `homelab_endpoint_value` | `name`, `kind` | The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
|
| `homelab_endpoint_value` | `name`, `kind` | The number a value check last read from the answer (checks that judge a value only, and only when it is a number). |
|
||||||
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
||||||
|
| `homelab_service_level_target_percent` | `name`, `subject`, `type` | The availability a service-level target asks for. |
|
||||||
|
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
|
||||||
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
|
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
|
||||||
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
|
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
|
||||||
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
CREATE TABLE `service_levels` (
|
||||||
|
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
|
||||||
|
`name` text NOT NULL,
|
||||||
|
`subject_type` text NOT NULL,
|
||||||
|
`subject_id` integer NOT NULL,
|
||||||
|
`target` real NOT NULL,
|
||||||
|
`alert` integer DEFAULT true NOT NULL,
|
||||||
|
`created_by` text,
|
||||||
|
`created_at` text DEFAULT (current_timestamp) NOT NULL
|
||||||
|
);
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -386,6 +386,13 @@
|
|||||||
"when": 1792610229172,
|
"when": 1792610229172,
|
||||||
"tag": "0054_green_beast",
|
"tag": "0054_green_beast",
|
||||||
"breakpoints": true
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 55,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1792612881951,
|
||||||
|
"tag": "0055_gorgeous_the_phantom",
|
||||||
|
"breakpoints": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -56,6 +56,7 @@ import { dataUsageRouter } from "./routes/dataUsage.js";
|
|||||||
import { alertActionsRouter } from "./routes/alertActions.js";
|
import { alertActionsRouter } from "./routes/alertActions.js";
|
||||||
import { homeAssistantRouter } from "./routes/homeAssistant.js";
|
import { homeAssistantRouter } from "./routes/homeAssistant.js";
|
||||||
import { upsRouter } from "./routes/ups.js";
|
import { upsRouter } from "./routes/ups.js";
|
||||||
|
import { serviceLevelsRouter } from "./routes/serviceLevels.js";
|
||||||
import { publicApiRouter } from "./routes/publicApi.js";
|
import { publicApiRouter } from "./routes/publicApi.js";
|
||||||
import { generatorRouter } from "./routes/generator.js";
|
import { generatorRouter } from "./routes/generator.js";
|
||||||
import { alertTicketsRouter } from "./routes/alertTickets.js";
|
import { alertTicketsRouter } from "./routes/alertTickets.js";
|
||||||
@@ -186,6 +187,7 @@ export function createApp() {
|
|||||||
app.use("/api/data-usage", dataUsageRouter);
|
app.use("/api/data-usage", dataUsageRouter);
|
||||||
app.use("/api/home-assistant", homeAssistantRouter);
|
app.use("/api/home-assistant", homeAssistantRouter);
|
||||||
app.use("/api/ups", upsRouter);
|
app.use("/api/ups", upsRouter);
|
||||||
|
app.use("/api/service-levels", serviceLevelsRouter);
|
||||||
app.use("/api/todos", todosRouter);
|
app.use("/api/todos", todosRouter);
|
||||||
app.use("/api/notes", notesRouter);
|
app.use("/api/notes", notesRouter);
|
||||||
app.use("/api/generator", generatorRouter);
|
app.use("/api/generator", generatorRouter);
|
||||||
|
|||||||
@@ -1159,6 +1159,23 @@ export const haReadings = sqliteTable(
|
|||||||
(t) => [index("ha_readings_lookup_idx").on(t.integrationId, t.entityId, t.at)],
|
(t) => [index("ha_readings_lookup_idx").on(t.integrationId, t.entityId, t.at)],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// ─── Service-level targets ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// "99.9 % for Gitea": a target on an endpoint check, a server or an integration, measured over calendar months (services/serviceLevels.ts). The thing measured is a
|
||||||
|
// plain reference (type and id), not a foreign key: one whose subject is removed is shown as having nothing to measure.
|
||||||
|
export const serviceLevels = sqliteTable("service_levels", {
|
||||||
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
|
name: text("name").notNull(),
|
||||||
|
subjectType: text("subject_type").notNull(), // endpoint | server | integration
|
||||||
|
subjectId: integer("subject_id").notNull(),
|
||||||
|
target: real("target").notNull(), // percent, 90–99.999
|
||||||
|
alert: integer("alert", { mode: "boolean" }).notNull().default(true), // warn when most of the month's error budget is used, and when it is overspent
|
||||||
|
createdBy: text("created_by"),
|
||||||
|
createdAt: text("created_at")
|
||||||
|
.notNull()
|
||||||
|
.default(sql`(current_timestamp)`),
|
||||||
|
});
|
||||||
|
|
||||||
// ─── UPS ────────────────────────────────────────────────────────────────────
|
// ─── UPS ────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
// How a UPS (an integration of type "ups", read over NUT or SNMP by services/upsSync.ts) is right now: one row each, replaced at every read. `updatedAt` is the last
|
// How a UPS (an integration of type "ups", read over NUT or SNMP by services/upsSync.ts) is right now: one row each, replaced at every read. `updatedAt` is the last
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import { Router } from "express";
|
import { Router } from "express";
|
||||||
import { requireAuth, requireRole } from "../auth/middleware.js";
|
import { requireAuth, requireRole } from "../auth/middleware.js";
|
||||||
import { buildMonthlyReport, isMonthId, lastFullMonth, monthlyReportText, recentMonths } from "../services/monthlyReport.js";
|
import { buildMonthlyReport, isPeriodId, lastFullMonth, monthlyReportText, recentPeriods } from "../services/monthlyReport.js";
|
||||||
import { recordAudit } from "../services/audit.js";
|
import { recordAudit } from "../services/audit.js";
|
||||||
import { notifyNow } from "../services/notify.js";
|
import { notifyNow } from "../services/notify.js";
|
||||||
import { getSettings } from "../services/settingsStore.js";
|
import { getSettings } from "../services/settingsStore.js";
|
||||||
@@ -10,29 +10,30 @@ export const reportsRouter = Router();
|
|||||||
|
|
||||||
reportsRouter.use(requireAuth);
|
reportsRouter.use(requireAuth);
|
||||||
|
|
||||||
/** The month asked for, or the last whole one; null when what was asked is not a month. */
|
/** The month, quarter or year asked for, or the last whole month; null when what was asked is not one that can be had. */
|
||||||
async function monthFrom(query: unknown): Promise<{ id: string; months: { id: string; label: string }[] } | null> {
|
async function monthFrom(query: unknown): Promise<({ id: string } & ReturnType<typeof recentPeriods>) | null> {
|
||||||
const tz = (await getSettings()).notifications.timezone || "UTC";
|
const tz = (await getSettings()).notifications.timezone || "UTC";
|
||||||
const months = recentMonths(new Date(), tz);
|
const options = recentPeriods(new Date(), tz);
|
||||||
if (query === undefined || query === "") return { id: lastFullMonth(new Date(), tz), months };
|
if (query === undefined || query === "") return { id: lastFullMonth(new Date(), tz), ...options };
|
||||||
if (!isMonthId(query) || !months.some((m) => m.id === query)) return null;
|
const all = [...options.months, ...options.quarters, ...options.years];
|
||||||
return { id: query, months };
|
if (!isPeriodId(query) || !all.some((m) => m.id === query)) return null;
|
||||||
|
return { id: query, ...options };
|
||||||
}
|
}
|
||||||
|
|
||||||
// How a month went, for everyone signed in: it is only what the pages it comes from already show them. ?month=2026-09, the last whole month by default;
|
// How a month, a quarter or a year went, for everyone signed in: it is only what the pages it comes from already show them. ?month=2026-09 (or ?month=2026-Q3,
|
||||||
// ?format=text is the plain-text version a notification carries.
|
// ?month=2026), the last whole month by default; ?format=text is the plain-text version a notification carries.
|
||||||
reportsRouter.get("/monthly", asyncHandler(async (req, res) => {
|
reportsRouter.get("/monthly", asyncHandler(async (req, res) => {
|
||||||
const picked = await monthFrom(req.query.month);
|
const picked = await monthFrom(req.query.month ?? req.query.period);
|
||||||
if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months, like 2026-09." });
|
if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months (2026-09), the last eight quarters (2026-Q3) or the last three years (2026)." });
|
||||||
const report = await buildMonthlyReport(picked.id);
|
const report = await buildMonthlyReport(picked.id);
|
||||||
if (req.query.format === "text") return res.json(monthlyReportText(report));
|
if (req.query.format === "text") return res.json(monthlyReportText(report));
|
||||||
res.json({ report, months: picked.months });
|
res.json({ report, months: picked.months, quarters: picked.quarters, years: picked.years });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// Send it now to the notification channels, as if it were the day. Administrators.
|
// Send it now to the notification channels, as if it were the day. Administrators.
|
||||||
reportsRouter.post("/monthly/send", requireRole("admin"), asyncHandler(async (req, res) => {
|
reportsRouter.post("/monthly/send", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
const picked = await monthFrom((req.body as { month?: unknown } | undefined)?.month);
|
const picked = await monthFrom((req.body as { month?: unknown } | undefined)?.month);
|
||||||
if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months, like 2026-09." });
|
if (!picked) return res.status(400).json({ error: "invalid_month", message: "Ask for one of the last twelve months (2026-09), the last eight quarters (2026-Q3) or the last three years (2026)." });
|
||||||
const { title, message } = monthlyReportText(await buildMonthlyReport(picked.id));
|
const { title, message } = monthlyReportText(await buildMonthlyReport(picked.id));
|
||||||
await notifyNow(title, message);
|
await notifyNow(title, message);
|
||||||
await recordAudit({ actor: req.currentUser!, category: "report", action: "send_monthly", targetType: "report", detail: { month: picked.id } });
|
await recordAudit({ actor: req.currentUser!, category: "report", action: "send_monthly", targetType: "report", detail: { month: picked.id } });
|
||||||
|
|||||||
@@ -0,0 +1,84 @@
|
|||||||
|
import { Router } from "express";
|
||||||
|
import { eq } from "drizzle-orm";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { endpointChecks, integrations, servers, serviceLevels } from "../db/schema.js";
|
||||||
|
import { requireAuth, requireRole } from "../auth/middleware.js";
|
||||||
|
import { recordAudit } from "../services/audit.js";
|
||||||
|
import { invalidateAlertsCache } from "../services/alerts.js";
|
||||||
|
import { levelStatuses, listLevels, MAX_LEVELS, MAX_TARGET, MIN_TARGET, SUBJECT_TYPES, type SubjectType } from "../services/serviceLevels.js";
|
||||||
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
|
export const serviceLevelsRouter = Router();
|
||||||
|
serviceLevelsRouter.use(requireAuth);
|
||||||
|
|
||||||
|
const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN);
|
||||||
|
|
||||||
|
const levelSchema = z.object({
|
||||||
|
name: z.string().trim().min(1).max(100),
|
||||||
|
subjectType: z.enum(SUBJECT_TYPES),
|
||||||
|
subjectId: z.number().int().positive(),
|
||||||
|
target: z.number().min(MIN_TARGET, `A target is at least ${MIN_TARGET}%.`).max(MAX_TARGET, `A target is at most ${MAX_TARGET}% — 100% leaves no room for any downtime.`),
|
||||||
|
alert: z.boolean().optional(),
|
||||||
|
});
|
||||||
|
|
||||||
|
async function subjectExists(type: SubjectType, id: number): Promise<boolean> {
|
||||||
|
const table = type === "endpoint" ? endpointChecks : type === "server" ? servers : integrations;
|
||||||
|
return (await db.select({ id: table.id }).from(table).where(eq(table.id, id)).limit(1)).length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const round3 = (n: number) => Math.round(n * 1000) / 1000;
|
||||||
|
|
||||||
|
/** Everything that can have a target, for the form. */
|
||||||
|
async function subjects() {
|
||||||
|
const [e, s, i] = await Promise.all([
|
||||||
|
db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks).orderBy(endpointChecks.name),
|
||||||
|
db.select({ id: servers.id, name: servers.name }).from(servers).orderBy(servers.name),
|
||||||
|
db.select({ id: integrations.id, name: integrations.name }).from(integrations).orderBy(integrations.name),
|
||||||
|
]);
|
||||||
|
return { endpoint: e, server: s, integration: i };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every target, where this month's error budget stands and how the last months went: for everyone signed in, like the pages the figures come from.
|
||||||
|
serviceLevelsRouter.get("/", asyncHandler(async (_req, res) => {
|
||||||
|
res.json({ levels: await levelStatuses(), subjects: await subjects() });
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Setting targets is an administrator's.
|
||||||
|
serviceLevelsRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const parsed = levelSchema.safeParse(req.body);
|
||||||
|
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "That isn't a target.", details: parsed.error.flatten() });
|
||||||
|
if ((await listLevels()).length >= MAX_LEVELS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_LEVELS} targets.` });
|
||||||
|
if (!(await subjectExists(parsed.data.subjectType, parsed.data.subjectId))) return res.status(400).json({ error: "unknown_subject", message: "That doesn't exist." });
|
||||||
|
const [row] = await db
|
||||||
|
.insert(serviceLevels)
|
||||||
|
.values({ name: parsed.data.name, subjectType: parsed.data.subjectType, subjectId: parsed.data.subjectId, target: round3(parsed.data.target), alert: parsed.data.alert ?? true, createdBy: req.currentUser!.name ?? req.currentUser!.email ?? null })
|
||||||
|
.returning();
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "service_level", action: "create", targetType: "service_level", targetId: row.id, detail: { name: row.name, subjectType: row.subjectType, target: row.target } });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
res.status(201).json({ id: row.id });
|
||||||
|
}));
|
||||||
|
|
||||||
|
serviceLevelsRouter.put("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const id = idOf(req.params.id);
|
||||||
|
const parsed = levelSchema.safeParse(req.body);
|
||||||
|
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.success ? "That isn't a target." : (parsed.error.issues[0]?.message ?? "That isn't a target.") });
|
||||||
|
const [before] = await db.select().from(serviceLevels).where(eq(serviceLevels.id, id)).limit(1);
|
||||||
|
if (!before) return res.status(404).json({ error: "not_found" });
|
||||||
|
if (!(await subjectExists(parsed.data.subjectType, parsed.data.subjectId))) return res.status(400).json({ error: "unknown_subject", message: "That doesn't exist." });
|
||||||
|
await db.update(serviceLevels).set({ name: parsed.data.name, subjectType: parsed.data.subjectType, subjectId: parsed.data.subjectId, target: round3(parsed.data.target), alert: parsed.data.alert ?? before.alert }).where(eq(serviceLevels.id, id));
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "service_level", action: "update", targetType: "service_level", targetId: id, detail: { name: parsed.data.name, target: round3(parsed.data.target), ...(before.target !== parsed.data.target ? { targetFrom: before.target } : {}) } });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
res.json({ ok: true });
|
||||||
|
}));
|
||||||
|
|
||||||
|
serviceLevelsRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const id = idOf(req.params.id);
|
||||||
|
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
const [before] = await db.select().from(serviceLevels).where(eq(serviceLevels.id, id)).limit(1);
|
||||||
|
if (!before) return res.status(404).json({ error: "not_found" });
|
||||||
|
await db.delete(serviceLevels).where(eq(serviceLevels.id, id));
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "service_level", action: "delete", targetType: "service_level", targetId: id, detail: { name: before.name } });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
res.status(204).end();
|
||||||
|
}));
|
||||||
@@ -168,7 +168,7 @@ const updateSchema = z.object({
|
|||||||
.partial()
|
.partial()
|
||||||
.optional(),
|
.optional(),
|
||||||
monthlyReport: z
|
monthlyReport: z
|
||||||
.object({ enabled: z.boolean(), day: z.number().int().min(1).max(28), time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/) })
|
.object({ enabled: z.boolean(), quarterly: z.boolean(), annual: z.boolean(), day: z.number().int().min(1).max(28), time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/) })
|
||||||
.partial()
|
.partial()
|
||||||
.optional(),
|
.optional(),
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -73,6 +73,7 @@ const SOURCES: Record<string, { label: string; link: string }> = {
|
|||||||
npm: { label: "Nginx Proxy Manager", link: "/proxy-hosts" },
|
npm: { label: "Nginx Proxy Manager", link: "/proxy-hosts" },
|
||||||
homeassistant: { label: "Home Assistant", link: "/home-assistant" },
|
homeassistant: { label: "Home Assistant", link: "/home-assistant" },
|
||||||
ups: { label: "UPS", link: "/ups" },
|
ups: { label: "UPS", link: "/ups" },
|
||||||
|
slo: { label: "Service level", link: "/service-levels" },
|
||||||
secrets: { label: "Secrets", link: "/secrets" },
|
secrets: { label: "Secrets", link: "/secrets" },
|
||||||
domains: { label: "Domains", link: "/dns?view=domains" },
|
domains: { label: "Domains", link: "/dns?view=domains" },
|
||||||
endpoint: { label: "Endpoint", link: "/endpoints" },
|
endpoint: { label: "Endpoint", link: "/endpoints" },
|
||||||
@@ -105,6 +106,8 @@ function originOfConditionKey(key: string): { type: string; id: number | null; c
|
|||||||
if (m) return { type: m[1], id: Number(m[2]), category: "automation" };
|
if (m) return { type: m[1], id: Number(m[2]), category: "automation" };
|
||||||
m = /^ups-(?:battery|low|replace|load):(\d+)$/.exec(key);
|
m = /^ups-(?:battery|low|replace|load):(\d+)$/.exec(key);
|
||||||
if (m) return { type: "ups", id: Number(m[1]), category: "monitoring" };
|
if (m) return { type: "ups", id: Number(m[1]), category: "monitoring" };
|
||||||
|
m = /^slo:(\d+)$/.exec(key);
|
||||||
|
if (m) return { type: "slo", id: Number(m[1]), category: "monitoring" };
|
||||||
return { type: "server", id: null, category: "disk" };
|
return { type: "server", id: null, category: "disk" };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -227,12 +230,13 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
type: origin.type,
|
type: origin.type,
|
||||||
message: c.message,
|
message: c.message,
|
||||||
key: c.key,
|
key: c.key,
|
||||||
link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") || c.key.startsWith("baseline:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : undefined,
|
link: origin.type === "server" && origin.id !== null ? `/servers/${origin.id}${/^(failed-units|smart|zfs):/.test(c.key) ? "?view=health" : c.key.startsWith("new-port:") ? "?view=network" : c.key.startsWith("rule:") || c.key.startsWith("baseline:") ? "?view=history" : ""}` : origin.type === "unifi" ? "/network/devices" : origin.type === "ups" ? "/ups" : origin.type === "slo" ? "/service-levels" : undefined,
|
||||||
silenced: subject !== null && subjects.has(subject), target: targetOfSubject(subject),
|
silenced: subject !== null && subjects.has(subject), target: targetOfSubject(subject),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
// Integrations the check couldn't read this time. (A whole-integration entry covers its nodes, so skip those.)
|
// Integrations the check couldn't read this time. (A whole-integration entry covers its nodes, so skip those.)
|
||||||
if (held.has("server:rules")) cannotCheck("Alert rules", "couldn't be judged — see the server log", "Server and storage health");
|
if (held.has("server:rules")) cannotCheck("Alert rules", "couldn't be judged — see the server log", "Server and storage health");
|
||||||
|
if (held.has("slo")) cannotCheck("Service-level targets", "couldn't be judged — see the server log", "Server and storage health");
|
||||||
if (held.has("server:baseline")) cannotCheck("What is usual for the time", "couldn't be judged — see the server log", "Server and storage health");
|
if (held.has("server:baseline")) cannotCheck("What is usual for the time", "couldn't be judged — see the server log", "Server and storage health");
|
||||||
for (const h of held) {
|
for (const h of held) {
|
||||||
const m = /^(proxmox|synology|unifi|ups):(\d+)(?::(.+))?$/.exec(h);
|
const m = /^(proxmox|synology|unifi|ups):(\d+)(?::(.+))?$/.exec(h);
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ const KIND_OF_CATEGORY: Record<string, ChangeKind> = {
|
|||||||
metrics_token: "settings",
|
metrics_token: "settings",
|
||||||
api_token: "settings",
|
api_token: "settings",
|
||||||
alert_rule: "alerts",
|
alert_rule: "alerts",
|
||||||
|
service_level: "endpoints",
|
||||||
todo: "other",
|
todo: "other",
|
||||||
note: "servers",
|
note: "servers",
|
||||||
webhook: "settings",
|
webhook: "settings",
|
||||||
@@ -181,6 +182,12 @@ export function describeChange(category: string, action: string, detail: Record<
|
|||||||
return { title: detail.change === "undo" ? "A change to several pieces of hardware was undone" : `${typeof detail.changed === "number" ? detail.changed : "Several"} piece${detail.changed === 1 ? "" : "s"} of hardware changed at once`, tone: "neutral", detail: genericSummary(detail, ["ids", "changed", "change"]) };
|
return { title: detail.change === "undo" ? "A change to several pieces of hardware was undone" : `${typeof detail.changed === "number" ? detail.changed : "Several"} piece${detail.changed === 1 ? "" : "s"} of hardware changed at once`, tone: "neutral", detail: genericSummary(detail, ["ids", "changed", "change"]) };
|
||||||
case "webhook/retry":
|
case "webhook/retry":
|
||||||
return { title: `A webhook message was sent again${name ? `: ${name}` : str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: detail.ok === false ? "bad" : "neutral", detail: str(detail.event) };
|
return { title: `A webhook message was sent again${name ? `: ${name}` : str(detail.label) ? `: ${str(detail.label)}` : ""}`, tone: detail.ok === false ? "bad" : "neutral", detail: str(detail.event) };
|
||||||
|
case "service_level/create":
|
||||||
|
return { title: `Service-level target set${name ? `: ${name}` : ""}`, tone: "neutral", detail: typeof detail.target === "number" ? `${detail.target}%` : null };
|
||||||
|
case "service_level/update":
|
||||||
|
return { title: `Service-level target changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: typeof detail.target === "number" ? `${typeof detail.targetFrom === "number" ? `${detail.targetFrom}% → ` : ""}${detail.target}%` : null };
|
||||||
|
case "service_level/delete":
|
||||||
|
return { title: `Service-level target removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
|
||||||
case "alert_rule/create":
|
case "alert_rule/create":
|
||||||
return { title: `Alert rule added${name ? `: ${name}` : ""}`, tone: "neutral", detail: genericSummary(detail) };
|
return { title: `Alert rule added${name ? `: ${name}` : ""}`, tone: "neutral", detail: genericSummary(detail) };
|
||||||
case "alert_rule/update":
|
case "alert_rule/update":
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ import { parseJson, type AgentPort } from "./agentPorts.js";
|
|||||||
import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js";
|
import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js";
|
||||||
import { computeRuleFindings, findingKey, listRules, type RuleFinding } from "./alertRules.js";
|
import { computeRuleFindings, findingKey, listRules, type RuleFinding } from "./alertRules.js";
|
||||||
import { baselineKey, computeBaselineFindings, type BaselineFinding } from "./baselines.js";
|
import { baselineKey, computeBaselineFindings, type BaselineFinding } from "./baselines.js";
|
||||||
|
import { levelFindings, levelKey, type LevelFinding } from "./serviceLevels.js";
|
||||||
import { recordStorageSamples } from "./storageHistory.js";
|
import { recordStorageSamples } from "./storageHistory.js";
|
||||||
import { isUpsFresh, upsProblems, upsSyncStatuses, type UpsStateRow } from "./upsSync.js";
|
import { isUpsFresh, upsProblems, upsSyncStatuses, type UpsStateRow } from "./upsSync.js";
|
||||||
|
|
||||||
@@ -81,6 +82,8 @@ export interface HealthSnapshot {
|
|||||||
rules?: RuleFinding[];
|
rules?: RuleFinding[];
|
||||||
/** What is clearly higher than usual for this hour of the week, from each server's own history (services/baselines.ts). */
|
/** What is clearly higher than usual for this hour of the week, from each server's own history (services/baselines.ts). */
|
||||||
baselines?: BaselineFinding[];
|
baselines?: BaselineFinding[];
|
||||||
|
/** Service-level targets whose error budget for the month is mostly or wholly used (services/serviceLevels.ts). */
|
||||||
|
levels?: LevelFinding[];
|
||||||
}
|
}
|
||||||
export interface Thresholds {
|
export interface Thresholds {
|
||||||
serverOfflineMinutes: number;
|
serverOfflineMinutes: number;
|
||||||
@@ -183,6 +186,9 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A service-level target that has used most of its month's error budget, or all of it.
|
||||||
|
for (const f of snapshot.levels ?? []) out.push({ key: levelKey(f), source: "slo", message: f.message, severity: f.severity });
|
||||||
|
|
||||||
for (const u of snapshot.unifi ?? []) {
|
for (const u of snapshot.unifi ?? []) {
|
||||||
const limitMs = (thresholds.deviceOfflineMinutes ?? 10) * 60_000;
|
const limitMs = (thresholds.deviceOfflineMinutes ?? 10) * 60_000;
|
||||||
for (const d of u.devices) {
|
for (const d of u.devices) {
|
||||||
@@ -402,6 +408,14 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Service-level targets: how much of the month's error budget is used. A failure here costs only this check.
|
||||||
|
try {
|
||||||
|
snapshot.levels = await levelFindings();
|
||||||
|
} catch (err) {
|
||||||
|
console.error("[health] couldn't judge the service-level targets:", err instanceof Error ? err.message : err);
|
||||||
|
held.add("slo");
|
||||||
|
}
|
||||||
|
|
||||||
const rows = await db
|
const rows = await db
|
||||||
.select({ id: integrations.id, name: integrations.name, type: integrations.type })
|
.select({ id: integrations.id, name: integrations.name, type: integrations.type })
|
||||||
.from(integrations)
|
.from(integrations)
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import { eq } from "drizzle-orm";
|
|||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { assets, endpointChecks, heartbeats, integrations, maintenanceWindows, secrets, servers, upsState } from "../db/schema.js";
|
import { assets, endpointChecks, heartbeats, integrations, maintenanceWindows, secrets, servers, upsState } from "../db/schema.js";
|
||||||
import { isUpsFresh, upsSyncStatuses } from "./upsSync.js";
|
import { isUpsFresh, upsSyncStatuses } from "./upsSync.js";
|
||||||
|
import { levelStatuses } from "./serviceLevels.js";
|
||||||
import { latestTests } from "./restoreTests.js";
|
import { latestTests } from "./restoreTests.js";
|
||||||
import { heartbeatState, parseStamp } from "./heartbeats.js";
|
import { heartbeatState, parseStamp } from "./heartbeats.js";
|
||||||
import { env } from "../env.js";
|
import { env } from "../env.js";
|
||||||
@@ -58,6 +59,17 @@ class Exposition {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Each service-level target, for /metrics: its target, how it is doing this month and how much error budget is left. */
|
||||||
|
async function addServiceLevelMetrics(out: Exposition, now: Date) {
|
||||||
|
for (const s of await levelStatuses(now, { history: false })) {
|
||||||
|
const labels = { name: s.name, subject: s.subjectName ?? "", type: s.subjectType };
|
||||||
|
out.add("homelab_service_level_target_percent", "gauge", "The availability a service-level target asks for.", s.target, labels);
|
||||||
|
if (s.measure.percent === null) continue;
|
||||||
|
out.add("homelab_service_level_availability_percent", "gauge", "How available the thing has been so far this calendar month.", s.measure.percent, labels);
|
||||||
|
out.add("homelab_service_level_error_budget_remaining_minutes", "gauge", "Downtime still allowed this month before the target is missed (negative once it is).", Math.round(s.budget.remainingMinutes * 10) / 10, labels);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** Each UPS, for /metrics: whether it is on battery, how full the battery is, how long it would last and how loaded it is. */
|
/** Each UPS, for /metrics: whether it is on battery, how full the battery is, how long it would last and how loaded it is. */
|
||||||
async function addUpsMetrics(out: Exposition, now: Date) {
|
async function addUpsMetrics(out: Exposition, now: Date) {
|
||||||
const fresh = new Set<number>();
|
const fresh = new Set<number>();
|
||||||
@@ -239,6 +251,7 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
|
|||||||
out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec);
|
out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec);
|
||||||
await addEndpointMetrics(out, now);
|
await addEndpointMetrics(out, now);
|
||||||
await addUpsMetrics(out, now);
|
await addUpsMetrics(out, now);
|
||||||
|
await addServiceLevelMetrics(out, now);
|
||||||
await addHeartbeatMetrics(out, now);
|
await addHeartbeatMetrics(out, now);
|
||||||
return out.toString();
|
return out.toString();
|
||||||
}
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/**
|
/**
|
||||||
* The monthly report: how a month went — availability of servers, integrations and endpoints, incidents and how long they took, updates applied (by hand and by
|
* The monthly report (and, by the same figures over longer stretches, the quarterly and annual ones): how a month went — availability of servers, integrations and endpoints, incidents and how long they took, updates applied (by hand and by
|
||||||
* patch plans), restore tests and the state of the backups, and what was renewed. Everything in it the app already knows; this gathers it for one calendar month
|
* patch plans), restore tests and the state of the backups, and what was renewed. Everything in it the app already knows; this gathers it for one calendar month
|
||||||
* (on the clock of the notification time zone) into one structure that the page shows, prints and exports, and into a plain-text version that can be sent like the
|
* (on the clock of the notification time zone) into one structure that the page shows, prints and exports, and into a plain-text version that can be sent like the
|
||||||
* weekly digest.
|
* weekly digest.
|
||||||
@@ -18,6 +18,7 @@ import { renewalsBetween } from "./renewals.js";
|
|||||||
import { getSettings } from "./settingsStore.js";
|
import { getSettings } from "./settingsStore.js";
|
||||||
import { getUptime } from "./uptimeHistory.js";
|
import { getUptime } from "./uptimeHistory.js";
|
||||||
import { spanWords } from "./weeklyDigest.js";
|
import { spanWords } from "./weeklyDigest.js";
|
||||||
|
import { levelResultsFor, type PeriodResult } from "./serviceLevels.js";
|
||||||
|
|
||||||
const DAY_MS = 86_400_000;
|
const DAY_MS = 86_400_000;
|
||||||
const MAX_LISTED = 12;
|
const MAX_LISTED = 12;
|
||||||
@@ -38,10 +39,14 @@ export interface EndpointRow {
|
|||||||
checks: number;
|
checks: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type PeriodKind = "month" | "quarter" | "year";
|
||||||
|
|
||||||
export interface MonthlyReport {
|
export interface MonthlyReport {
|
||||||
/** "2026-09". */
|
/** What stretch it is about: a month (id "2026-09"), a quarter ("2026-Q3") or a year ("2026"). */
|
||||||
|
period: PeriodKind;
|
||||||
|
/** The id of that stretch. (Named for the month, which is what it was made for.) */
|
||||||
month: string;
|
month: string;
|
||||||
/** "September 2026". */
|
/** "September 2026", "Q3 2026", "2026". */
|
||||||
label: string;
|
label: string;
|
||||||
/** ISO — the start of the month and its end (or now, for a month that is not over). */
|
/** ISO — the start of the month and its end (or now, for a month that is not over). */
|
||||||
from: string;
|
from: string;
|
||||||
@@ -52,6 +57,8 @@ export interface MonthlyReport {
|
|||||||
servers: AvailabilityRow[];
|
servers: AvailabilityRow[];
|
||||||
integrations: AvailabilityRow[];
|
integrations: AvailabilityRow[];
|
||||||
endpoints: EndpointRow[];
|
endpoints: EndpointRow[];
|
||||||
|
/** How each service-level target did over the stretch (those with a figure for it). */
|
||||||
|
serviceLevels: PeriodResult[];
|
||||||
incidents: {
|
incidents: {
|
||||||
began: number;
|
began: number;
|
||||||
resolved: number;
|
resolved: number;
|
||||||
@@ -94,6 +101,17 @@ function monthOf(now: Date, tz: string): { year: number; month: number } {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export const monthId = (year: number, month: number) => `${year}-${pad(month)}`;
|
export const monthId = (year: number, month: number) => `${year}-${pad(month)}`;
|
||||||
|
|
||||||
|
export function isQuarterId(v: unknown): boolean {
|
||||||
|
return typeof v === "string" && /^\d{4}-Q[1-4]$/.test(v);
|
||||||
|
}
|
||||||
|
export function isYearId(v: unknown): boolean {
|
||||||
|
return typeof v === "string" && /^\d{4}$/.test(v);
|
||||||
|
}
|
||||||
|
export const isPeriodId = (v: unknown): v is string => isMonthId(v) || isQuarterId(v) || isYearId(v);
|
||||||
|
export function periodKind(id: string): PeriodKind {
|
||||||
|
return isQuarterId(id) ? "quarter" : isYearId(id) ? "year" : "month";
|
||||||
|
}
|
||||||
const shift = (year: number, month: number, by: number) => {
|
const shift = (year: number, month: number, by: number) => {
|
||||||
const index = year * 12 + (month - 1) + by;
|
const index = year * 12 + (month - 1) + by;
|
||||||
return { year: Math.floor(index / 12), month: (index % 12) + 1 };
|
return { year: Math.floor(index / 12), month: (index % 12) + 1 };
|
||||||
@@ -121,6 +139,61 @@ export function recentMonths(now: Date, tz: string, count = 12): { id: string; l
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function periodLabel(id: string): string {
|
||||||
|
if (isQuarterId(id)) return `Q${id.slice(6)} ${id.slice(0, 4)}`;
|
||||||
|
return isYearId(id) ? id : monthLabel(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The year and quarter (1–4) `now` is in. */
|
||||||
|
function quarterOf(now: Date, tz: string): { year: number; quarter: number } {
|
||||||
|
const { year, month } = monthOf(now, tz);
|
||||||
|
return { year, quarter: Math.ceil(month / 3) };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The last whole quarter ("2026-Q2" in July), and the last whole year. */
|
||||||
|
export function lastFullQuarter(now: Date, tz: string): string {
|
||||||
|
const { year, quarter } = quarterOf(now, tz);
|
||||||
|
return quarter === 1 ? `${year - 1}-Q4` : `${year}-Q${quarter - 1}`;
|
||||||
|
}
|
||||||
|
export const lastFullYear = (now: Date, tz: string) => String(monthOf(now, tz).year - 1);
|
||||||
|
|
||||||
|
/** What can be asked for: this month and eleven before it, this quarter and seven before it, this year and two before it — newest first. */
|
||||||
|
export function recentPeriods(now: Date, tz: string): { months: { id: string; label: string }[]; quarters: { id: string; label: string }[]; years: { id: string; label: string }[] } {
|
||||||
|
const { year, quarter } = quarterOf(now, tz);
|
||||||
|
const quarters = Array.from({ length: 8 }, (_, i) => {
|
||||||
|
const index = year * 4 + (quarter - 1) - i;
|
||||||
|
const id = `${Math.floor(index / 4)}-Q${(index % 4) + 1}`;
|
||||||
|
return { id, label: periodLabel(id) };
|
||||||
|
});
|
||||||
|
const years = Array.from({ length: 3 }, (_, i) => ({ id: String(year - i), label: String(year - i) }));
|
||||||
|
return { months: recentMonths(now, tz), quarters, years };
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PeriodBounds {
|
||||||
|
start: Date;
|
||||||
|
end: Date;
|
||||||
|
/** The first UTC day of it and of what follows, as "YYYY-MM-DD", for the figures kept per UTC day. */
|
||||||
|
firstDay: string;
|
||||||
|
nextFirstDay: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The instants a month, quarter or year begins and ends at, on the clock of `tz`. */
|
||||||
|
export function periodBounds(id: string, tz: string): PeriodBounds {
|
||||||
|
if (isYearId(id)) {
|
||||||
|
const y = Number(id);
|
||||||
|
return { start: zonedTime(y, 1, 1, 0, 0, tz), end: zonedTime(y + 1, 1, 1, 0, 0, tz), firstDay: `${y}-01-01`, nextFirstDay: `${y + 1}-01-01` };
|
||||||
|
}
|
||||||
|
if (isQuarterId(id)) {
|
||||||
|
const y = Number(id.slice(0, 4));
|
||||||
|
const first = (Number(id.slice(6)) - 1) * 3 + 1;
|
||||||
|
const next = shift(y, first, 3);
|
||||||
|
return { start: zonedTime(y, first, 1, 0, 0, tz), end: zonedTime(next.year, next.month, 1, 0, 0, tz), firstDay: `${y}-${pad(first)}-01`, nextFirstDay: `${monthId(next.year, next.month)}-01` };
|
||||||
|
}
|
||||||
|
const [y, m] = id.split("-").map(Number);
|
||||||
|
const next = shift(y, m, 1);
|
||||||
|
return { ...monthBounds(id, tz), firstDay: `${id}-01`, nextFirstDay: `${monthId(next.year, next.month)}-01` };
|
||||||
|
}
|
||||||
|
|
||||||
/** The instants a month begins and ends at, on the clock of `tz`. */
|
/** The instants a month begins and ends at, on the clock of `tz`. */
|
||||||
export function monthBounds(id: string, tz: string): { start: Date; end: Date } {
|
export function monthBounds(id: string, tz: string): { start: Date; end: Date } {
|
||||||
const [y, m] = id.split("-").map(Number);
|
const [y, m] = id.split("-").map(Number);
|
||||||
@@ -135,7 +208,8 @@ const round1 = (n: number) => Math.round(n * 10) / 10;
|
|||||||
export async function buildMonthlyReport(id: string, now: Date = new Date()): Promise<MonthlyReport> {
|
export async function buildMonthlyReport(id: string, now: Date = new Date()): Promise<MonthlyReport> {
|
||||||
const settings = await getSettings();
|
const settings = await getSettings();
|
||||||
const tz = settings.notifications.timezone || "UTC";
|
const tz = settings.notifications.timezone || "UTC";
|
||||||
const { start, end: fullEnd } = monthBounds(id, tz);
|
const bounds = periodBounds(id, tz);
|
||||||
|
const { start, end: fullEnd } = bounds;
|
||||||
const partial = fullEnd.getTime() > now.getTime();
|
const partial = fullEnd.getTime() > now.getTime();
|
||||||
const end = partial ? now : fullEnd;
|
const end = partial ? now : fullEnd;
|
||||||
const days = Math.max(1, Math.ceil((end.getTime() - start.getTime()) / DAY_MS));
|
const days = Math.max(1, Math.ceil((end.getTime() - start.getTime()) / DAY_MS));
|
||||||
@@ -154,10 +228,8 @@ export async function buildMonthlyReport(id: string, now: Date = new Date()): Pr
|
|||||||
|
|
||||||
// Endpoints: from the daily figures, for the days the month covers. The figures are per UTC day, so a day at either end can belong to a neighbouring month
|
// Endpoints: from the daily figures, for the days the month covers. The figures are per UTC day, so a day at either end can belong to a neighbouring month
|
||||||
// by a few hours (more than the time zone is ahead of UTC); a month that is not over takes in today.
|
// by a few hours (more than the time zone is ahead of UTC); a month that is not over takes in today.
|
||||||
const [yy, mm] = id.split("-").map(Number);
|
const firstDay = bounds.firstDay;
|
||||||
const following = shift(yy, mm, 1);
|
const lastDayExclusive = partial ? new Date(now.getTime() + DAY_MS).toISOString().slice(0, 10) : bounds.nextFirstDay;
|
||||||
const firstDay = `${id}-01`;
|
|
||||||
const lastDayExclusive = partial ? new Date(now.getTime() + DAY_MS).toISOString().slice(0, 10) : `${monthId(following.year, following.month)}-01`;
|
|
||||||
const dailyRows = await db.select().from(endpointDaily).where(and(gte(endpointDaily.day, firstDay), lt(endpointDaily.day, lastDayExclusive)));
|
const dailyRows = await db.select().from(endpointDaily).where(and(gte(endpointDaily.day, firstDay), lt(endpointDaily.day, lastDayExclusive)));
|
||||||
const checkNames = new Map((await db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks)).map((c) => [c.id, c.name]));
|
const checkNames = new Map((await db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks)).map((c) => [c.id, c.name]));
|
||||||
const perCheck = new Map<number, { total: number; ok: number; msSum: number; msCount: number }>();
|
const perCheck = new Map<number, { total: number; ok: number; msSum: number; msCount: number }>();
|
||||||
@@ -175,7 +247,7 @@ export async function buildMonthlyReport(id: string, now: Date = new Date()): Pr
|
|||||||
.sort((a, b) => (a.percent ?? 101) - (b.percent ?? 101) || a.name.localeCompare(b.name));
|
.sort((a, b) => (a.percent ?? 101) - (b.percent ?? 101) || a.name.localeCompare(b.name));
|
||||||
|
|
||||||
// Incidents.
|
// Incidents.
|
||||||
const all = await listIncidents({ now: end, limit: 3000 });
|
const all = await listIncidents({ now: end, limit: 20000 });
|
||||||
const inMonth = (iso: string) => iso >= startIso && iso < endIso;
|
const inMonth = (iso: string) => iso >= startIso && iso < endIso;
|
||||||
const began = all.filter((i) => inMonth(i.openedAt));
|
const began = all.filter((i) => inMonth(i.openedAt));
|
||||||
const resolved = all.filter((i) => i.resolvedAt !== null && inMonth(i.resolvedAt));
|
const resolved = all.filter((i) => i.resolvedAt !== null && inMonth(i.resolvedAt));
|
||||||
@@ -209,8 +281,9 @@ export async function buildMonthlyReport(id: string, now: Date = new Date()): Pr
|
|||||||
const renewalRows = await renewalsBetween(start, end);
|
const renewalRows = await renewalsBetween(start, end);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
|
period: periodKind(id),
|
||||||
month: id,
|
month: id,
|
||||||
label: monthLabel(id),
|
label: periodLabel(id),
|
||||||
from: startIso,
|
from: startIso,
|
||||||
to: endIso,
|
to: endIso,
|
||||||
partial,
|
partial,
|
||||||
@@ -218,6 +291,7 @@ export async function buildMonthlyReport(id: string, now: Date = new Date()): Pr
|
|||||||
servers: rowsOf(uptime.servers, serverNames),
|
servers: rowsOf(uptime.servers, serverNames),
|
||||||
integrations: rowsOf(uptime.integrations, integrationNames),
|
integrations: rowsOf(uptime.integrations, integrationNames),
|
||||||
endpoints: endpointRows,
|
endpoints: endpointRows,
|
||||||
|
serviceLevels: await levelResultsFor(start, fullEnd, now),
|
||||||
incidents: {
|
incidents: {
|
||||||
began: began.length,
|
began: began.length,
|
||||||
resolved: resolved.length,
|
resolved: resolved.length,
|
||||||
@@ -274,6 +348,13 @@ export function monthlyReportText(r: MonthlyReport): { title: string; message: s
|
|||||||
if (integrationAvg !== null) avail.push(`Integrations: ${integrationAvg}% on average.`);
|
if (integrationAvg !== null) avail.push(`Integrations: ${integrationAvg}% on average.`);
|
||||||
lines("Availability", avail);
|
lines("Availability", avail);
|
||||||
|
|
||||||
|
if (r.serviceLevels.length > 0) {
|
||||||
|
const missed = r.serviceLevels.filter((l) => !l.met);
|
||||||
|
const body = [`${r.serviceLevels.length - missed.length} of ${plural(r.serviceLevels.length, "target")} met${r.partial ? " so far" : ""}.`];
|
||||||
|
for (const l of r.serviceLevels.slice(0, 8)) body.push(`- ${l.name}: ${l.percent}% against ${l.target}% — ${l.met ? "met" : "MISSED"} (down ${l.downMinutes > 0 ? spanWords(l.downMinutes) : "not at all"}, ${l.allowedMinutes >= 1 ? spanWords(l.allowedMinutes) : "under a minute"} allowed)`);
|
||||||
|
lines("Service levels", body);
|
||||||
|
}
|
||||||
|
|
||||||
const inc = r.incidents;
|
const inc = r.incidents;
|
||||||
if (inc.began > 0 || inc.resolved > 0 || inc.stillOpen > 0) {
|
if (inc.began > 0 || inc.resolved > 0 || inc.stillOpen > 0) {
|
||||||
const body = [`${plural(inc.began, "incident")} began, ${inc.resolved} ended, ${inc.stillOpen} still open.`];
|
const body = [`${plural(inc.began, "incident")} began, ${inc.resolved} ended, ${inc.stillOpen} still open.`];
|
||||||
@@ -298,6 +379,7 @@ export function monthlyReportText(r: MonthlyReport): { title: string; message: s
|
|||||||
|
|
||||||
lines("Renewed", r.renewals.slice(0, 10).map((x) => `- ${x.kind === "domain" ? "Domain" : "Certificate"} ${x.subject}: ${x.fromDate} → ${x.toDate}`));
|
lines("Renewed", r.renewals.slice(0, 10).map((x) => `- ${x.kind === "domain" ? "Domain" : "Certificate"} ${x.subject}: ${x.fromDate} → ${x.toDate}`));
|
||||||
|
|
||||||
const title = `Homelab monthly report — ${r.label}${r.partial ? " (so far)" : ""}`;
|
const noun = r.period === "year" ? "annual" : r.period === "quarter" ? "quarterly" : "monthly";
|
||||||
return { title, message: sections.length > 0 ? sections.join("\n\n") : "Nothing was recorded for this month." };
|
const title = `Homelab ${noun} report — ${r.label}${r.partial ? " (so far)" : ""}`;
|
||||||
|
return { title, message: sections.length > 0 ? sections.join("\n\n") : `Nothing was recorded for this ${r.period}.` };
|
||||||
}
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
import schedule from "node-schedule";
|
import schedule from "node-schedule";
|
||||||
import { getSettings } from "./settingsStore.js";
|
import { getSettings } from "./settingsStore.js";
|
||||||
import { notify } from "./notify.js";
|
import { notify } from "./notify.js";
|
||||||
import { buildMonthlyReport, lastFullMonth, monthlyReportText } from "./monthlyReport.js";
|
import { buildMonthlyReport, lastFullMonth, lastFullQuarter, lastFullYear, monthlyReportText } from "./monthlyReport.js";
|
||||||
|
|
||||||
let currentJob: schedule.Job | null = null;
|
let currentJob: schedule.Job | null = null;
|
||||||
|
|
||||||
@@ -12,11 +12,25 @@ export function cronForMonthlyReport(day: number, time: string): string {
|
|||||||
return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} ${d} * *`;
|
return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} ${d} * *`;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Builds last month's report and sends it through the channels (held with other notifications during quiet hours). */
|
/**
|
||||||
|
* Builds last month's report and sends it through the channels (held with other notifications during quiet hours). When that month closed a quarter and the quarterly
|
||||||
|
* report is on, the quarter's report follows it; when it closed the year and the annual one is on, the year's.
|
||||||
|
*/
|
||||||
export async function sendMonthlyReport(now: Date = new Date()): Promise<{ title: string; message: string }> {
|
export async function sendMonthlyReport(now: Date = new Date()): Promise<{ title: string; message: string }> {
|
||||||
const tz = (await getSettings()).notifications.timezone || "UTC";
|
const settings = await getSettings();
|
||||||
const text = monthlyReportText(await buildMonthlyReport(lastFullMonth(now, tz), now));
|
const tz = settings.notifications.timezone || "UTC";
|
||||||
|
const month = lastFullMonth(now, tz);
|
||||||
|
const text = monthlyReportText(await buildMonthlyReport(month, now));
|
||||||
await notify(text.title, text.message);
|
await notify(text.title, text.message);
|
||||||
|
const closing = Number(month.slice(5));
|
||||||
|
if (settings.monthlyReport.quarterly && closing % 3 === 0) {
|
||||||
|
const q = monthlyReportText(await buildMonthlyReport(lastFullQuarter(now, tz), now));
|
||||||
|
await notify(q.title, q.message);
|
||||||
|
}
|
||||||
|
if (settings.monthlyReport.annual && closing === 12) {
|
||||||
|
const y = monthlyReportText(await buildMonthlyReport(lastFullYear(now, tz), now));
|
||||||
|
await notify(y.title, y.message);
|
||||||
|
}
|
||||||
return text;
|
return text;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,244 @@
|
|||||||
|
/**
|
||||||
|
* Service-level targets: "99.9 % for Gitea", and what is left of the error budget this month. A target is set on something the app already keeps the record of — an
|
||||||
|
* endpoint check, a server, an integration — and is measured over calendar months (on the clock of the notification time zone):
|
||||||
|
*
|
||||||
|
* allowed downtime = minutes in the month × (1 − target) — that is the error budget
|
||||||
|
* used = minutes the thing was down so far
|
||||||
|
* left = allowed − used
|
||||||
|
*
|
||||||
|
* An endpoint's downtime is worked out from its daily check figures (each day's share of failed checks, over the part of the day that is within the period); a server's and an integration's
|
||||||
|
* from the moments the audit log recorded them going away and coming back, to the minute. Planned maintenance is *not* taken out: down is down. Nothing is guessed — a
|
||||||
|
* thing with no record for the period has no figure, and says so.
|
||||||
|
*
|
||||||
|
* A target whose budget is mostly used (80 %) is a warning, and one that is overspent a critical alert, until the month turns over and the budget starts again.
|
||||||
|
*/
|
||||||
|
import { and, gte, lt } from "drizzle-orm";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { endpointChecks, endpointDaily, integrations, servers, serviceLevels } from "../db/schema.js";
|
||||||
|
import { monthBounds, recentMonths } from "./monthlyReport.js";
|
||||||
|
import { getSettings } from "./settingsStore.js";
|
||||||
|
import { getUptimeRange, type UptimeRange } from "./uptimeHistory.js";
|
||||||
|
import { spanWords } from "./weeklyDigest.js";
|
||||||
|
|
||||||
|
export const SUBJECT_TYPES = ["endpoint", "server", "integration"] as const;
|
||||||
|
export type SubjectType = (typeof SUBJECT_TYPES)[number];
|
||||||
|
export const MAX_LEVELS = 50;
|
||||||
|
export const MIN_TARGET = 90;
|
||||||
|
export const MAX_TARGET = 99.999;
|
||||||
|
/** The share of the budget used at which it is a warning. */
|
||||||
|
export const WARN_AT_PERCENT = 80;
|
||||||
|
export const HISTORY_MONTHS = 6;
|
||||||
|
|
||||||
|
const DAY_MS = 86_400_000;
|
||||||
|
|
||||||
|
export type LevelRow = typeof serviceLevels.$inferSelect;
|
||||||
|
export type BudgetState = "ok" | "at_risk" | "breached" | "no_data";
|
||||||
|
|
||||||
|
export interface Measure {
|
||||||
|
/** Share of the known time spent up, in percent (three decimals); null when nothing is known. */
|
||||||
|
percent: number | null;
|
||||||
|
downMinutes: number;
|
||||||
|
knownMinutes: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Budget {
|
||||||
|
allowedMinutes: number;
|
||||||
|
/** Of the budget, in percent (can pass 100). Null without a figure. */
|
||||||
|
usedPercent: number | null;
|
||||||
|
/** Negative when overspent. */
|
||||||
|
remainingMinutes: number;
|
||||||
|
state: BudgetState;
|
||||||
|
/** Of the budget, at the pace so far, by the end of the period; null before a day has passed or without a figure. */
|
||||||
|
projectedPercent: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const round3 = (n: number) => Math.round(n * 1000) / 1000;
|
||||||
|
|
||||||
|
/** Pure: the error budget of a target over a period of `totalMinutes`, and where it stands after `elapsedMinutes` with `downMinutes` used. */
|
||||||
|
export function judgeBudget(target: number, totalMinutes: number, elapsedMinutes: number, measure: Measure): Budget {
|
||||||
|
const allowed = totalMinutes * (1 - target / 100);
|
||||||
|
if (measure.percent === null) return { allowedMinutes: allowed, usedPercent: null, remainingMinutes: allowed, state: "no_data", projectedPercent: null };
|
||||||
|
const used = measure.downMinutes;
|
||||||
|
const usedPercent = allowed > 0 ? (used / allowed) * 100 : used > 0 ? Infinity : 0;
|
||||||
|
const state: BudgetState = used > allowed ? "breached" : usedPercent >= WARN_AT_PERCENT ? "at_risk" : "ok";
|
||||||
|
const projected = elapsedMinutes >= 1440 && allowed > 0 ? ((used / elapsedMinutes) * totalMinutes / allowed) * 100 : null;
|
||||||
|
return { allowedMinutes: allowed, usedPercent: Number.isFinite(usedPercent) ? Math.round(usedPercent * 10) / 10 : 999, remainingMinutes: allowed - used, state, projectedPercent: projected === null ? null : Math.round(projected) };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A budget in words: "43 min", "4 h". Below a minute it is "under a minute". */
|
||||||
|
export const budgetWords = (minutes: number) => (minutes < 1 ? "under a minute" : spanWords(minutes));
|
||||||
|
|
||||||
|
/** Pure: what an alert about a budget says. */
|
||||||
|
export function budgetMessage(name: string, target: number, periodLabel: string, b: Budget, m: Measure): string {
|
||||||
|
if (b.state === "breached") return `${name} has missed its ${target}% target for ${periodLabel}: down ${budgetWords(m.downMinutes)}, ${budgetWords(b.allowedMinutes)} allowed`;
|
||||||
|
return `${name}: ${Math.round(b.usedPercent ?? 0)}% of the error budget for its ${target}% target is used up for ${periodLabel} — down ${budgetWords(m.downMinutes)} of ${budgetWords(b.allowedMinutes)} allowed, ${budgetWords(b.remainingMinutes)} left`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Measuring ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/** What is shared between the measurements of one request, so each stretch of the audit log and of the daily figures is read once. */
|
||||||
|
export interface MeasureContext {
|
||||||
|
now: Date;
|
||||||
|
uptime: Map<string, Promise<UptimeRange>>;
|
||||||
|
daily: Map<string, Promise<{ checkId: number; day: string; total: number; ok: number }[]>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const newContext = (now: Date = new Date()): MeasureContext => ({ now, uptime: new Map(), daily: new Map() });
|
||||||
|
|
||||||
|
const dayOf = (ms: number) => new Date(ms).toISOString().slice(0, 10);
|
||||||
|
|
||||||
|
/** How a thing did between two moments (the end is clipped to now). */
|
||||||
|
export async function measure(type: SubjectType, subjectId: number, start: Date, end: Date, ctx: MeasureContext): Promise<Measure> {
|
||||||
|
const to = Math.min(end.getTime(), ctx.now.getTime());
|
||||||
|
const from = start.getTime();
|
||||||
|
if (to <= from) return { percent: null, downMinutes: 0, knownMinutes: 0 };
|
||||||
|
if (type === "endpoint") {
|
||||||
|
const key = `${dayOf(from)}|${dayOf(to + DAY_MS - 1)}`;
|
||||||
|
let rows = ctx.daily.get(key);
|
||||||
|
if (!rows) {
|
||||||
|
rows = (async () => db.select({ checkId: endpointDaily.checkId, day: endpointDaily.day, total: endpointDaily.total, ok: endpointDaily.ok }).from(endpointDaily).where(and(gte(endpointDaily.day, dayOf(from)), lt(endpointDaily.day, dayOf(to + DAY_MS - 1)))))();
|
||||||
|
ctx.daily.set(key, rows);
|
||||||
|
}
|
||||||
|
// Day by day: the share of that day's checks that failed, over the part of the day that is within the period. Days with no checks are not known, so they count neither way.
|
||||||
|
let known = 0;
|
||||||
|
let down = 0;
|
||||||
|
for (const r of await rows) {
|
||||||
|
if (r.checkId !== subjectId || r.total <= 0) continue;
|
||||||
|
const dayStart = Date.parse(`${r.day}T00:00:00Z`);
|
||||||
|
const overlap = Math.min(to, dayStart + DAY_MS) - Math.max(from, dayStart);
|
||||||
|
if (overlap <= 0) continue;
|
||||||
|
known += overlap / 60_000;
|
||||||
|
down += (1 - r.ok / r.total) * (overlap / 60_000);
|
||||||
|
}
|
||||||
|
if (known <= 0) return { percent: null, downMinutes: 0, knownMinutes: 0 };
|
||||||
|
return { percent: round3((1 - down / known) * 100), downMinutes: Math.round(down), knownMinutes: Math.round(known) };
|
||||||
|
}
|
||||||
|
const key = `${from}|${to}`;
|
||||||
|
let range = ctx.uptime.get(key);
|
||||||
|
if (!range) {
|
||||||
|
range = getUptimeRange(start, new Date(to), ctx.now);
|
||||||
|
ctx.uptime.set(key, range);
|
||||||
|
}
|
||||||
|
const t = (await range)[type === "server" ? "servers" : "integrations"][subjectId];
|
||||||
|
if (!t || t.known <= 0) return { percent: null, downMinutes: 0, knownMinutes: 0 };
|
||||||
|
return { percent: round3((1 - t.down / t.known) * 100), downMinutes: t.down, knownMinutes: t.known };
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface LevelStatus {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
subjectType: SubjectType;
|
||||||
|
subjectId: number;
|
||||||
|
/** The name of the thing measured; null when it has been removed. */
|
||||||
|
subjectName: string | null;
|
||||||
|
target: number;
|
||||||
|
alert: boolean;
|
||||||
|
createdBy: string | null;
|
||||||
|
period: { id: string; label: string; from: string; to: string; totalMinutes: number; elapsedMinutes: number };
|
||||||
|
measure: Measure;
|
||||||
|
budget: Budget;
|
||||||
|
/** The months before this one, newest first. */
|
||||||
|
history: { id: string; label: string; percent: number | null; downMinutes: number; allowedMinutes: number; met: boolean | null }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
async function subjectNames(): Promise<Record<SubjectType, Map<number, string>>> {
|
||||||
|
const [e, s, i] = await Promise.all([db.select({ id: endpointChecks.id, name: endpointChecks.name }).from(endpointChecks), db.select({ id: servers.id, name: servers.name }).from(servers), db.select({ id: integrations.id, name: integrations.name }).from(integrations)]);
|
||||||
|
return { endpoint: new Map(e.map((r) => [r.id, r.name])), server: new Map(s.map((r) => [r.id, r.name])), integration: new Map(i.map((r) => [r.id, r.name])) };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listLevels(): Promise<LevelRow[]> {
|
||||||
|
return (await db.select().from(serviceLevels)).sort((a, b) => a.name.localeCompare(b.name) || a.id - b.id);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every target, where this month's budget stands, and how each of the last months went. */
|
||||||
|
export async function levelStatuses(now: Date = new Date(), options: { history?: boolean } = {}): Promise<LevelStatus[]> {
|
||||||
|
const rows = await listLevels();
|
||||||
|
if (rows.length === 0) return [];
|
||||||
|
const tz = (await getSettings()).notifications.timezone || "UTC";
|
||||||
|
const months = recentMonths(now, tz, HISTORY_MONTHS + 1);
|
||||||
|
const current = months[0].id;
|
||||||
|
const { start, end } = monthBounds(current, tz);
|
||||||
|
const totalMinutes = (end.getTime() - start.getTime()) / 60_000;
|
||||||
|
const elapsedMinutes = Math.max(0, (Math.min(now.getTime(), end.getTime()) - start.getTime()) / 60_000);
|
||||||
|
const names = await subjectNames();
|
||||||
|
const ctx = newContext(now);
|
||||||
|
const out: LevelStatus[] = [];
|
||||||
|
for (const r of rows) {
|
||||||
|
const type = r.subjectType as SubjectType;
|
||||||
|
const m = await measure(type, r.subjectId, start, end, ctx);
|
||||||
|
const history: LevelStatus["history"] = [];
|
||||||
|
if (options.history !== false) {
|
||||||
|
for (const past of months.slice(1)) {
|
||||||
|
const b = monthBounds(past.id, tz);
|
||||||
|
const total = (b.end.getTime() - b.start.getTime()) / 60_000;
|
||||||
|
const pm = await measure(type, r.subjectId, b.start, b.end, ctx);
|
||||||
|
const allowed = total * (1 - r.target / 100);
|
||||||
|
history.push({ id: past.id, label: past.label, percent: pm.percent, downMinutes: pm.downMinutes, allowedMinutes: allowed, met: pm.percent === null ? null : pm.downMinutes <= allowed });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.push({
|
||||||
|
id: r.id,
|
||||||
|
name: r.name,
|
||||||
|
subjectType: type,
|
||||||
|
subjectId: r.subjectId,
|
||||||
|
subjectName: names[type].get(r.subjectId) ?? null,
|
||||||
|
target: r.target,
|
||||||
|
alert: r.alert,
|
||||||
|
createdBy: r.createdBy,
|
||||||
|
period: { id: current, label: months[0].label, from: start.toISOString(), to: end.toISOString(), totalMinutes: Math.round(totalMinutes), elapsedMinutes: Math.round(elapsedMinutes) },
|
||||||
|
measure: m,
|
||||||
|
budget: judgeBudget(r.target, totalMinutes, elapsedMinutes, m),
|
||||||
|
history,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface LevelFinding {
|
||||||
|
levelId: number;
|
||||||
|
name: string;
|
||||||
|
state: "at_risk" | "breached";
|
||||||
|
severity: "warning" | "critical";
|
||||||
|
message: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const levelKey = (f: Pick<LevelFinding, "levelId">) => `slo:${f.levelId}`;
|
||||||
|
|
||||||
|
/** The targets whose budget is mostly or wholly used this month, for the ones that are to alert. */
|
||||||
|
export async function levelFindings(now: Date = new Date()): Promise<LevelFinding[]> {
|
||||||
|
const out: LevelFinding[] = [];
|
||||||
|
for (const s of await levelStatuses(now, { history: false })) {
|
||||||
|
if (!s.alert || (s.budget.state !== "at_risk" && s.budget.state !== "breached")) continue;
|
||||||
|
out.push({ levelId: s.id, name: s.name, state: s.budget.state, severity: s.budget.state === "breached" ? "critical" : "warning", message: budgetMessage(s.name, s.target, s.period.label, s.budget, s.measure) });
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PeriodResult {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
subjectType: SubjectType;
|
||||||
|
target: number;
|
||||||
|
percent: number | null;
|
||||||
|
downMinutes: number;
|
||||||
|
/** Over the whole period (even one that is not over yet). */
|
||||||
|
allowedMinutes: number;
|
||||||
|
usedPercent: number | null;
|
||||||
|
/** Whether it stayed within its budget; null without a figure. For a period not over, "so far". */
|
||||||
|
met: boolean | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** How each target did over a period (a month, a quarter, a year): for the reports. */
|
||||||
|
export async function levelResultsFor(start: Date, end: Date, now: Date = new Date()): Promise<PeriodResult[]> {
|
||||||
|
const rows = await listLevels();
|
||||||
|
const ctx = newContext(now);
|
||||||
|
const total = (end.getTime() - start.getTime()) / 60_000;
|
||||||
|
const out: PeriodResult[] = [];
|
||||||
|
for (const r of rows) {
|
||||||
|
const m = await measure(r.subjectType as SubjectType, r.subjectId, start, end, ctx);
|
||||||
|
if (m.percent === null) continue;
|
||||||
|
const b = judgeBudget(r.target, total, (Math.min(now.getTime(), end.getTime()) - start.getTime()) / 60_000, m);
|
||||||
|
out.push({ id: r.id, name: r.name, subjectType: r.subjectType as SubjectType, target: r.target, percent: m.percent, downMinutes: m.downMinutes, allowedMinutes: Math.round(b.allowedMinutes * 10) / 10, usedPercent: b.usedPercent, met: m.downMinutes <= b.allowedMinutes });
|
||||||
|
}
|
||||||
|
return out.sort((a, b) => Number(a.met) - Number(b.met) || (b.usedPercent ?? 0) - (a.usedPercent ?? 0) || a.name.localeCompare(b.name));
|
||||||
|
}
|
||||||
@@ -128,6 +128,10 @@ export interface HealthCheckSettings {
|
|||||||
/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. Off until switched on. */
|
/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. Off until switched on. */
|
||||||
export interface MonthlyReportSettings {
|
export interface MonthlyReportSettings {
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
|
/** Also send the quarter's report after the month that ends it (March, June, September, December). */
|
||||||
|
quarterly: boolean;
|
||||||
|
/** Also send the year's report after December. */
|
||||||
|
annual: boolean;
|
||||||
/** Day of the month, 1–28 (every month has it), in notifications.timezone. */
|
/** Day of the month, 1–28 (every month has it), in notifications.timezone. */
|
||||||
day: number;
|
day: number;
|
||||||
/** "HH:MM", in notifications.timezone. */
|
/** "HH:MM", in notifications.timezone. */
|
||||||
@@ -293,7 +297,7 @@ const DEFAULTS: AppSettings = {
|
|||||||
logRetention: { enabled: false, retentionDays: 90, intervalHours: 24 },
|
logRetention: { enabled: false, retentionDays: 90, intervalHours: 24 },
|
||||||
quietHours: { enabled: false, start: "22:00", end: "07:00" },
|
quietHours: { enabled: false, start: "22:00", end: "07:00" },
|
||||||
digest: { enabled: false, day: 1, time: "08:00" },
|
digest: { enabled: false, day: 1, time: "08:00" },
|
||||||
monthlyReport: { enabled: false, day: 1, time: "08:00" },
|
monthlyReport: { enabled: false, quarterly: false, annual: false, day: 1, time: "08:00" },
|
||||||
inventory: { currency: "", pricePerKwh: null },
|
inventory: { currency: "", pricePerKwh: null },
|
||||||
agents: { autoUpdate: false },
|
agents: { autoUpdate: false },
|
||||||
alertActions: { enabled: false },
|
alertActions: { enabled: false },
|
||||||
|
|||||||
@@ -34,13 +34,17 @@ export interface UptimeSeries {
|
|||||||
percent: number;
|
percent: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface Segment {
|
||||||
|
from: number;
|
||||||
|
to: number;
|
||||||
|
down: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Splits [knownFrom, now] into up and down stretches from the recorded transitions and lays them over `days` consecutive 24-hour
|
* Splits [knownFrom, now] into up and down stretches from the recorded transitions. Before the first transition the thing is taken to have been in the opposite state
|
||||||
* buckets ending at `now`. Before the first transition the thing is taken to have been in the opposite state (a "went offline" means it
|
* (a "went offline" means it was up until then); with no transitions at all it has been up the whole time.
|
||||||
* was up until then); with no transitions at all it has been up the whole time.
|
|
||||||
*/
|
*/
|
||||||
export function computeBuckets(transitions: Transition[], knownFrom: number, now: number, days: number): UptimeBucket[] {
|
export function segmentsOf(transitions: Transition[], knownFrom: number, now: number): Segment[] {
|
||||||
const start = now - days * DAY_MS;
|
|
||||||
const sorted = [...transitions].sort((a, b) => a.at - b.at);
|
const sorted = [...transitions].sort((a, b) => a.at - b.at);
|
||||||
|
|
||||||
// State at knownFrom: after any transitions that happened before it, otherwise the opposite of the first one that follows.
|
// State at knownFrom: after any transitions that happened before it, otherwise the opposite of the first one that follows.
|
||||||
@@ -49,7 +53,7 @@ export function computeBuckets(transitions: Transition[], knownFrom: number, now
|
|||||||
while (i < sorted.length && sorted[i].at <= knownFrom) state = sorted[i++].down;
|
while (i < sorted.length && sorted[i].at <= knownFrom) state = sorted[i++].down;
|
||||||
if (i === 0 && sorted.length > 0) state = !sorted[0].down;
|
if (i === 0 && sorted.length > 0) state = !sorted[0].down;
|
||||||
|
|
||||||
const segments: { from: number; to: number; down: boolean }[] = [];
|
const segments: Segment[] = [];
|
||||||
let cursor = Math.min(knownFrom, now);
|
let cursor = Math.min(knownFrom, now);
|
||||||
for (; i < sorted.length; i++) {
|
for (; i < sorted.length; i++) {
|
||||||
const t = Math.min(sorted[i].at, now);
|
const t = Math.min(sorted[i].at, now);
|
||||||
@@ -58,6 +62,26 @@ export function computeBuckets(transitions: Transition[], knownFrom: number, now
|
|||||||
state = sorted[i].down;
|
state = sorted[i].down;
|
||||||
}
|
}
|
||||||
if (now > cursor) segments.push({ from: cursor, to: now, down: state });
|
if (now > cursor) segments.push({ from: cursor, to: now, down: state });
|
||||||
|
return segments;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Minutes known, and of those spent down, within [from, to]. */
|
||||||
|
export function minutesWithin(segments: Segment[], from: number, to: number): { known: number; down: number } {
|
||||||
|
let known = 0;
|
||||||
|
let down = 0;
|
||||||
|
for (const seg of segments) {
|
||||||
|
const overlap = Math.min(seg.to, to) - Math.max(seg.from, from);
|
||||||
|
if (overlap <= 0) continue;
|
||||||
|
known += overlap;
|
||||||
|
if (seg.down) down += overlap;
|
||||||
|
}
|
||||||
|
return { known: Math.round(known / 60_000), down: Math.round(down / 60_000) };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The segments laid over `days` consecutive 24-hour buckets ending at `now`. */
|
||||||
|
export function computeBuckets(transitions: Transition[], knownFrom: number, now: number, days: number): UptimeBucket[] {
|
||||||
|
const start = now - days * DAY_MS;
|
||||||
|
const segments = segmentsOf(transitions, knownFrom, now);
|
||||||
|
|
||||||
const known = new Array<number>(days).fill(0);
|
const known = new Array<number>(days).fill(0);
|
||||||
const down = new Array<number>(days).fill(0);
|
const down = new Array<number>(days).fill(0);
|
||||||
@@ -121,10 +145,49 @@ export interface UptimeReport {
|
|||||||
integrations: Record<number, UptimeSeries>;
|
integrations: Record<number, UptimeSeries>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function getUptime(now: Date = new Date(), days: number = DEFAULT_DAYS): Promise<UptimeReport> {
|
/** How long a server or integration was known, and down, between two moments. */
|
||||||
const nowMs = now.getTime();
|
export interface UptimeTotals {
|
||||||
const rows = await db.select().from(auditLog).where(inArray(auditLog.action, ["went_offline", "back_online", "went_failing", "recovered"]));
|
known: number;
|
||||||
|
down: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface UptimeRange {
|
||||||
|
servers: Record<number, UptimeTotals>;
|
||||||
|
integrations: Record<number, UptimeTotals>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Exactly how long each server and integration was down between `from` and `to` (the end is clipped to now), to the minute — for a target that must be measured over a
|
||||||
|
* calendar period rather than in whole days back from now. Only the ones known for some of it are listed.
|
||||||
|
*/
|
||||||
|
export async function getUptimeRange(from: Date, to: Date, now: Date = new Date()): Promise<UptimeRange> {
|
||||||
|
const nowMs = now.getTime();
|
||||||
|
const end = Math.min(to.getTime(), nowMs);
|
||||||
|
const start = from.getTime();
|
||||||
|
const out: UptimeRange = { servers: {}, integrations: {} };
|
||||||
|
if (end <= start) return out;
|
||||||
|
const { serverEvents, sourceEvents } = await loadTransitions(nowMs);
|
||||||
|
|
||||||
|
for (const s of await db.select({ id: servers.id, createdAt: servers.createdAt, lastSeenAt: servers.lastSeenAt }).from(servers)) {
|
||||||
|
const events = serverEvents.get(s.id) ?? [];
|
||||||
|
if (!s.lastSeenAt && events.length === 0) continue;
|
||||||
|
const created = Date.parse(auditTimeToIso(s.createdAt));
|
||||||
|
const totals = minutesWithin(segmentsOf(events, Number.isFinite(created) ? created : nowMs, nowMs), start, end);
|
||||||
|
if (totals.known > 0) out.servers[s.id] = totals;
|
||||||
|
}
|
||||||
|
const since = await ensureHistoryStart(nowMs);
|
||||||
|
for (const i of await db.select({ id: integrations.id, type: integrations.type, createdAt: integrations.createdAt }).from(integrations)) {
|
||||||
|
const created = Date.parse(auditTimeToIso(i.createdAt));
|
||||||
|
const knownFrom = Math.max(Number.isFinite(created) ? created : 0, since);
|
||||||
|
const totals = minutesWithin(segmentsOf(sourceEvents.get(i.type) ?? [], knownFrom, nowMs), start, end);
|
||||||
|
if (totals.known > 0) out.integrations[i.id] = totals;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The recorded moments servers went away and came back, and integrations started failing and worked again. */
|
||||||
|
async function loadTransitions(nowMs: number): Promise<{ serverEvents: Map<number, Transition[]>; sourceEvents: Map<string, Transition[]> }> {
|
||||||
|
const rows = await db.select().from(auditLog).where(inArray(auditLog.action, ["went_offline", "back_online", "went_failing", "recovered"]));
|
||||||
const serverEvents = new Map<number, Transition[]>();
|
const serverEvents = new Map<number, Transition[]>();
|
||||||
const sourceEvents = new Map<string, Transition[]>();
|
const sourceEvents = new Map<string, Transition[]>();
|
||||||
for (const r of rows) {
|
for (const r of rows) {
|
||||||
@@ -141,6 +204,12 @@ export async function getUptime(now: Date = new Date(), days: number = DEFAULT_D
|
|||||||
sourceEvents.set(r.targetId, list);
|
sourceEvents.set(r.targetId, list);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
return { serverEvents, sourceEvents };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function getUptime(now: Date = new Date(), days: number = DEFAULT_DAYS): Promise<UptimeReport> {
|
||||||
|
const nowMs = now.getTime();
|
||||||
|
const { serverEvents, sourceEvents } = await loadTransitions(nowMs);
|
||||||
|
|
||||||
const out: UptimeReport = { days, endsAt: now.toISOString(), servers: {}, integrations: {} };
|
const out: UptimeReport = { days, endsAt: now.toISOString(), servers: {}, integrations: {} };
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,369 @@
|
|||||||
|
/**
|
||||||
|
* Service-level targets: the error budget of a target over a month, how a thing is measured (an endpoint from its daily figures, a server and an integration from the audit
|
||||||
|
* log, to the minute), how the months before it went, when it is an alert, and the quarterly and annual reports that carry the results.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { createServer as createHttpServer, type Server } from "node:http";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-slo-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, integrations, endpointChecks, endpointDaily, auditLog, serviceLevels } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const slo = await import("../src/services/serviceLevels.js");
|
||||||
|
const uptime = await import("../src/services/uptimeHistory.js");
|
||||||
|
const report = await import("../src/services/monthlyReport.js");
|
||||||
|
const { sendMonthlyReport } = await import("../src/services/monthlyReportScheduler.js");
|
||||||
|
const { evaluateHealth, collectSnapshot } = await import("../src/services/healthMonitor.js");
|
||||||
|
const { buildMetrics } = await import("../src/services/metrics.js");
|
||||||
|
const { serviceLevelsRouter } = await import("../src/routes/serviceLevels.js");
|
||||||
|
const { reportsRouter } = await import("../src/routes/reports.js");
|
||||||
|
const { setInternalFlag, updateSettings } = await import("../src/services/settingsStore.js");
|
||||||
|
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
|
||||||
|
const { eq } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
const MIN = 60_000;
|
||||||
|
const DAY = 86_400_000;
|
||||||
|
/** The 20th of October, noon: 19.5 days into a 31-day month. */
|
||||||
|
const NOW = new Date("2026-10-20T12:00:00Z");
|
||||||
|
const OCT_MINUTES = 31 * 1440;
|
||||||
|
|
||||||
|
describe("the error budget", () => {
|
||||||
|
const m = (down: number, percent: number | null = 99): import("../src/services/serviceLevels.js").Measure => ({ percent, downMinutes: down, knownMinutes: 1000 });
|
||||||
|
|
||||||
|
test("a target allows the minutes it leaves over, and says how much of that is used", () => {
|
||||||
|
const ok = slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m(10));
|
||||||
|
assert.equal(Math.round(ok.allowedMinutes * 100) / 100, 44.64);
|
||||||
|
assert.deepEqual([ok.state, ok.usedPercent, Math.round(ok.remainingMinutes * 10) / 10], ["ok", 22.4, 34.6]);
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m(36)).state, "at_risk", "80 % or more used");
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m(35)).state, "ok", "78 %");
|
||||||
|
const over = slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m(50));
|
||||||
|
assert.deepEqual([over.state, over.usedPercent, Math.round(over.remainingMinutes)], ["breached", 112, -5]);
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m(44)).state, "at_risk", "within the budget, though barely");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("no figure is no data, and a budget of nothing is overspent by any downtime", () => {
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 100, m(0, null)).state, "no_data");
|
||||||
|
assert.equal(slo.judgeBudget(100, OCT_MINUTES, 100, m(1)).state, "breached");
|
||||||
|
assert.equal(slo.judgeBudget(100, OCT_MINUTES, 100, m(0)).state, "ok");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the pace so far is projected to the end of the month, but not from the first hours", () => {
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 14_400, m(10)).projectedPercent, 69, "10 minutes in 10 days → 31 in the month → 69 % of 44.6");
|
||||||
|
assert.equal(slo.judgeBudget(99.9, OCT_MINUTES, 600, m(10)).projectedPercent, null, "ten hours into the month says little");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the words an alert uses", () => {
|
||||||
|
const m1 = m(36);
|
||||||
|
const b1 = slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m1);
|
||||||
|
assert.equal(slo.budgetMessage("Gitea", 99.9, "October 2026", b1, m1), "Gitea: 81% of the error budget for its 99.9% target is used up for October 2026 — down 36 min of 45 min allowed, 9 min left");
|
||||||
|
const m2 = m(50);
|
||||||
|
assert.equal(slo.budgetMessage("Gitea", 99.9, "October 2026", slo.judgeBudget(99.9, OCT_MINUTES, 28_080, m2), m2), "Gitea has missed its 99.9% target for October 2026: down 50 min, 45 min allowed");
|
||||||
|
assert.equal(slo.budgetWords(0.4), "under a minute");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("measuring", () => {
|
||||||
|
let checkId = 0;
|
||||||
|
let emptyCheckId = 0;
|
||||||
|
let serverId = 0;
|
||||||
|
let integrationId = 0;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
checkId = (await db.insert(endpointChecks).values({ name: "Gitea", kind: "http", target: "https://git.lab/" }).returning({ id: endpointChecks.id }))[0].id;
|
||||||
|
emptyCheckId = (await db.insert(endpointChecks).values({ name: "Unasked", kind: "http", target: "https://x.lab/" }).returning({ id: endpointChecks.id }))[0].id;
|
||||||
|
// October: 288 checks a day, ten of them failed on the 5th; the 20th is half done. September: a whole day failed.
|
||||||
|
const day = (d: string, total: number, ok: number) => db.insert(endpointDaily).values({ checkId, day: d, total, ok, msSum: 0, msCount: 0 });
|
||||||
|
for (let d = 1; d <= 19; d++) await day(`2026-10-${String(d).padStart(2, "0")}`, 288, d === 5 ? 278 : 288);
|
||||||
|
await day("2026-10-20", 144, 144);
|
||||||
|
for (let d = 1; d <= 30; d++) await day(`2026-09-${String(d).padStart(2, "0")}`, 288, d === 10 ? 0 : 288);
|
||||||
|
|
||||||
|
serverId = (await db.insert(servers).values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", createdAt: "2026-01-01 00:00:00", lastSeenAt: NOW.toISOString() }).returning({ id: servers.id }))[0].id;
|
||||||
|
integrationId = (await db.insert(integrations).values({ type: "gitea", name: "Gitea", baseUrl: "", createdAt: "2026-01-01 00:00:00" }).returning({ id: integrations.id }))[0].id;
|
||||||
|
const audit = (action: string, at: string, extra: Record<string, string>) => db.insert(auditLog).values({ category: "server", action, targetType: "server", createdAt: at, ...extra });
|
||||||
|
await audit("went_offline", "2026-10-05 10:00:00", { targetId: String(serverId) });
|
||||||
|
await audit("back_online", "2026-10-05 10:30:00", { targetId: String(serverId) });
|
||||||
|
// an outage that straddles the start of the month: only the part in October counts
|
||||||
|
await audit("went_offline", "2026-09-30 23:00:00", { targetId: String(serverId) });
|
||||||
|
await audit("back_online", "2026-10-01 01:00:00", { targetId: String(serverId) });
|
||||||
|
await db.insert(auditLog).values([
|
||||||
|
{ category: "integration", action: "went_failing", targetType: "integration_type", targetId: "gitea", createdAt: "2026-10-12 08:00:00" },
|
||||||
|
{ category: "integration", action: "recovered", targetType: "integration_type", targetId: "gitea", createdAt: "2026-10-12 08:20:00" },
|
||||||
|
]);
|
||||||
|
await setInternalFlag("integration_history_since", "2026-01-01T00:00:00.000Z");
|
||||||
|
uptime.resetHistoryStartForTests();
|
||||||
|
});
|
||||||
|
|
||||||
|
const oct = { start: new Date("2026-10-01T00:00:00Z"), end: new Date("2026-11-01T00:00:00Z") };
|
||||||
|
|
||||||
|
test("an endpoint: the share of checks that failed, over the time that has passed", async () => {
|
||||||
|
const r = await slo.measure("endpoint", checkId, oct.start, oct.end, slo.newContext(NOW));
|
||||||
|
assert.deepEqual([r.percent, r.downMinutes, r.knownMinutes], [99.822, 50, 28_080]);
|
||||||
|
assert.deepEqual(await slo.measure("endpoint", emptyCheckId, oct.start, oct.end, slo.newContext(NOW)), { percent: null, downMinutes: 0, knownMinutes: 0 }, "no checks, no figure");
|
||||||
|
assert.equal((await slo.measure("endpoint", checkId, new Date("2026-11-01T00:00:00Z"), new Date("2026-12-01T00:00:00Z"), slo.newContext(NOW))).percent, null, "a month that has not begun");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a server: the minutes the audit log says it was away, only those within the period", async () => {
|
||||||
|
const r = await slo.measure("server", serverId, oct.start, oct.end, slo.newContext(NOW));
|
||||||
|
assert.deepEqual([r.downMinutes, r.knownMinutes], [90, 28_080], "30 on the 5th and the 60 of the night that began in September");
|
||||||
|
assert.equal(r.percent, 99.679);
|
||||||
|
const sept = await slo.measure("server", serverId, new Date("2026-09-01T00:00:00Z"), oct.start, slo.newContext(NOW));
|
||||||
|
assert.equal(sept.downMinutes, 60, "and in September, the hour before midnight");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an integration is measured by the kind of service it is", async () => {
|
||||||
|
const r = await slo.measure("integration", integrationId, oct.start, oct.end, slo.newContext(NOW));
|
||||||
|
assert.deepEqual([r.downMinutes, r.percent], [20, 99.929]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("each stretch of the audit log is read once for everything measured over the same period", async () => {
|
||||||
|
const ctx = slo.newContext(NOW);
|
||||||
|
await Promise.all([slo.measure("server", serverId, oct.start, oct.end, ctx), slo.measure("integration", integrationId, oct.start, oct.end, ctx)]);
|
||||||
|
assert.equal(ctx.uptime.size, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the exact range is what the whole-day figures are made of", async () => {
|
||||||
|
const range = await uptime.getUptimeRange(new Date(NOW.getTime() - 3 * DAY), NOW, NOW);
|
||||||
|
assert.deepEqual(Object.keys(range.servers).map(Number), [serverId]);
|
||||||
|
assert.deepEqual(await uptime.getUptimeRange(NOW, new Date(NOW.getTime() - MIN), NOW), { servers: {}, integrations: {} }, "an empty range");
|
||||||
|
const seg = uptime.segmentsOf([{ at: 100 * MIN, down: true }, { at: 130 * MIN, down: false }], 0, 200 * MIN);
|
||||||
|
assert.deepEqual(uptime.minutesWithin(seg, 90 * MIN, 120 * MIN), { known: 30, down: 20 });
|
||||||
|
assert.deepEqual(uptime.minutesWithin(seg, 300 * MIN, 400 * MIN), { known: 0, down: 0 });
|
||||||
|
});
|
||||||
|
|
||||||
|
test("this month's budget, with how each of the months before it went", async () => {
|
||||||
|
await db.insert(serviceLevels).values([
|
||||||
|
{ name: "Gitea answers", subjectType: "endpoint", subjectId: checkId, target: 99.9 },
|
||||||
|
{ name: "NAS up", subjectType: "server", subjectId: serverId, target: 99 },
|
||||||
|
{ name: "Gitea service", subjectType: "integration", subjectId: integrationId, target: 99.9, alert: false },
|
||||||
|
{ name: "Nobody asks", subjectType: "endpoint", subjectId: emptyCheckId, target: 99.9 },
|
||||||
|
]);
|
||||||
|
const all = await slo.levelStatuses(NOW);
|
||||||
|
const by = Object.fromEntries(all.map((s) => [s.name, s]));
|
||||||
|
assert.deepEqual([by["Gitea answers"].budget.state, by["Gitea answers"].measure.percent, by["Gitea answers"].subjectName], ["breached", 99.822, "Gitea"]);
|
||||||
|
assert.deepEqual([by["NAS up"].budget.state, Math.round(by["NAS up"].budget.allowedMinutes)], ["ok", 446], "99 % of 31 days leaves 446 minutes");
|
||||||
|
assert.equal(by["Nobody asks"].budget.state, "no_data");
|
||||||
|
assert.deepEqual([by["Gitea answers"].period.id, by["Gitea answers"].period.label, by["Gitea answers"].period.totalMinutes, by["Gitea answers"].period.elapsedMinutes], ["2026-10", "October 2026", OCT_MINUTES, 28_080]);
|
||||||
|
// the months before: September had a whole failed day; August has no figures
|
||||||
|
const hist = by["Gitea answers"].history;
|
||||||
|
assert.deepEqual(hist.map((h) => h.id), ["2026-09", "2026-08", "2026-07", "2026-06", "2026-05", "2026-04"]);
|
||||||
|
assert.deepEqual([hist[0].percent, hist[0].met, Math.round(hist[0].downMinutes)], [96.667, false, 1440]);
|
||||||
|
assert.deepEqual([hist[1].percent, hist[1].met], [null, null]);
|
||||||
|
assert.equal((await slo.levelStatuses(NOW, { history: false }))[0].history.length, 0, "left out when it isn't wanted");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a target whose thing has been removed has nothing to measure, and says so", async () => {
|
||||||
|
await db.insert(serviceLevels).values({ name: "Gone", subjectType: "server", subjectId: 99999, target: 99.9 });
|
||||||
|
const gone = (await slo.levelStatuses(NOW)).find((s) => s.name === "Gone")!;
|
||||||
|
assert.deepEqual([gone.subjectName, gone.budget.state], [null, "no_data"]);
|
||||||
|
await db.delete(serviceLevels).where(eq(serviceLevels.name, "Gone"));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("only targets that are to alert, and whose budget is mostly or wholly used, are findings", async () => {
|
||||||
|
const found = await slo.levelFindings(NOW);
|
||||||
|
assert.deepEqual(found.map((f) => [f.name, f.state, f.severity]), [["Gitea answers", "breached", "critical"]]);
|
||||||
|
assert.match(found[0].message, /^Gitea answers has missed its 99.9% target for October 2026: down 50 min, 45 min allowed$/);
|
||||||
|
assert.equal(slo.levelKey(found[0]), `slo:${found[0].levelId}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("how each did over a period, for a report: the missed first", async () => {
|
||||||
|
const results = await slo.levelResultsFor(oct.start, oct.end, NOW);
|
||||||
|
assert.deepEqual(results.map((r) => [r.name, r.met]), [["Gitea answers", false], ["Gitea service", true], ["NAS up", true]], "the missed first, then the most used");
|
||||||
|
assert.ok(results.every((r) => r.name !== "Nobody asks"), "one with no figure is left out");
|
||||||
|
assert.equal(results[0].allowedMinutes, 44.6);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("quarters and years", () => {
|
||||||
|
test("which are which, and what can be asked for", () => {
|
||||||
|
assert.equal(report.periodKind("2026-Q3"), "quarter");
|
||||||
|
assert.equal(report.periodKind("2026"), "year");
|
||||||
|
assert.equal(report.periodKind("2026-09"), "month");
|
||||||
|
for (const ok of ["2026-09", "2026-Q1", "2026"]) assert.equal(report.isPeriodId(ok), true, ok);
|
||||||
|
for (const bad of ["2026-Q5", "2026-Q0", "26", "2026-13", "Q3", "", undefined, 2026]) assert.equal(report.isPeriodId(bad), false, String(bad));
|
||||||
|
assert.deepEqual([report.periodLabel("2026-Q3"), report.periodLabel("2026"), report.periodLabel("2026-09")], ["Q3 2026", "2026", "September 2026"]);
|
||||||
|
assert.equal(report.lastFullQuarter(new Date("2026-10-02T09:00:00Z"), "UTC"), "2026-Q3");
|
||||||
|
assert.equal(report.lastFullQuarter(new Date("2027-01-02T09:00:00Z"), "UTC"), "2026-Q4", "across the new year");
|
||||||
|
assert.equal(report.lastFullYear(new Date("2027-01-02T09:00:00Z"), "UTC"), "2026");
|
||||||
|
const options = report.recentPeriods(new Date("2026-10-19T12:00:00Z"), "UTC");
|
||||||
|
assert.deepEqual([options.months.length, options.quarters.length, options.years.length], [12, 8, 3]);
|
||||||
|
assert.deepEqual([options.quarters[0], options.quarters[1], options.quarters[7]], [{ id: "2026-Q4", label: "Q4 2026" }, { id: "2026-Q3", label: "Q3 2026" }, { id: "2025-Q1", label: "Q1 2025" }]);
|
||||||
|
assert.deepEqual(options.years.map((y) => y.id), ["2026", "2025", "2024"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a quarter and a year begin and end where the clock of the time zone says", () => {
|
||||||
|
const q3 = report.periodBounds("2026-Q3", "UTC");
|
||||||
|
assert.deepEqual([q3.start.toISOString(), q3.end.toISOString(), q3.firstDay, q3.nextFirstDay], ["2026-07-01T00:00:00.000Z", "2026-10-01T00:00:00.000Z", "2026-07-01", "2026-10-01"]);
|
||||||
|
const q4 = report.periodBounds("2026-Q4", "Europe/Stockholm");
|
||||||
|
assert.deepEqual([q4.start.toISOString(), q4.end.toISOString(), q4.nextFirstDay], ["2026-09-30T22:00:00.000Z", "2026-12-31T23:00:00.000Z", "2027-01-01"], "summer time at the start, winter at the end");
|
||||||
|
const y = report.periodBounds("2026", "UTC");
|
||||||
|
assert.deepEqual([y.start.toISOString(), y.end.toISOString(), y.nextFirstDay], ["2026-01-01T00:00:00.000Z", "2027-01-01T00:00:00.000Z", "2027-01-01"]);
|
||||||
|
assert.equal(report.periodBounds("2026-09", "UTC").nextFirstDay, "2026-10-01", "a month is as it was");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a quarter's report is the same report over longer, with the targets in it", async () => {
|
||||||
|
const q = await report.buildMonthlyReport("2026-Q3", NOW);
|
||||||
|
assert.deepEqual([q.period, q.month, q.label, q.partial, q.from, q.to], ["quarter", "2026-Q3", "Q3 2026", false, "2026-07-01T00:00:00.000Z", "2026-10-01T00:00:00.000Z"]);
|
||||||
|
assert.deepEqual(q.serviceLevels.map((l) => [l.name, l.met]), [["Gitea answers", false], ["NAS up", true], ["Gitea service", true]], "only September has endpoint figures: a whole failed day");
|
||||||
|
const text = report.monthlyReportText(q);
|
||||||
|
assert.equal(text.title, "Homelab quarterly report — Q3 2026");
|
||||||
|
assert.match(text.message, /SERVICE LEVELS\n2 of 3 targets met\.\n- Gitea answers: 96\.667% against 99\.9% — MISSED \(down 1 d, 2 h 13 min allowed\)/);
|
||||||
|
const year = await report.buildMonthlyReport("2026", NOW);
|
||||||
|
assert.deepEqual([year.period, year.label, year.partial], ["year", "2026", true]);
|
||||||
|
assert.equal(report.monthlyReportText(year).title, "Homelab annual report — 2026 (so far)");
|
||||||
|
const month = await report.buildMonthlyReport("2026-10", NOW);
|
||||||
|
assert.equal(month.period, "month");
|
||||||
|
assert.deepEqual(month.serviceLevels.map((l) => [l.name, l.met]), [["Gitea answers", false], ["Gitea service", true], ["NAS up", true]]);
|
||||||
|
assert.equal(report.monthlyReportText(month).title, "Homelab monthly report — October 2026 (so far)");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("as alerts, in /metrics, in the routes and as scheduled messages", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "admin";
|
||||||
|
let hook: Server;
|
||||||
|
const messages: string[] = [];
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "admin", name: "Ada", role: "admin" },
|
||||||
|
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
|
||||||
|
]);
|
||||||
|
hook = createHttpServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (c) => chunks.push(c));
|
||||||
|
req.on("end", () => {
|
||||||
|
messages.push(String(JSON.parse(Buffer.concat(chunks).toString()).content));
|
||||||
|
res.end("{}");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await new Promise<void>((r) => hook.listen(0, "127.0.0.1", r));
|
||||||
|
await updateSettings({ webhook: { enabled: true, url: `http://127.0.0.1:${(hook.address() as AddressInfo).port}/` } });
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/service-levels", serviceLevelsRouter);
|
||||||
|
a.use("/api/reports", reportsRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
|
||||||
|
});
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
hook.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const send = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json", connection: "close" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
|
||||||
|
test("a finding is an ordinary health condition", async () => {
|
||||||
|
const found = await slo.levelFindings(NOW);
|
||||||
|
const list = evaluateHealth({ servers: [], proxmox: [], synology: [], levels: found }, { serverOfflineMinutes: 60, diskUsagePercent: 90 }, NOW.getTime());
|
||||||
|
const c = list.find((x) => x.key === `slo:${found[0].levelId}`)!;
|
||||||
|
assert.deepEqual([c.source, c.severity], ["slo", "critical"]);
|
||||||
|
const { snapshot } = await collectSnapshot();
|
||||||
|
assert.ok(Array.isArray(snapshot.levels), "the health pass looks");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the Alerts page says it, with a link to the page", async () => {
|
||||||
|
// measured against the real clock: put a target on something that is down now
|
||||||
|
await db.insert(servers).values({ name: "down-now", apiTokenHash: "d", apiTokenPrefix: "d", createdAt: "2020-01-01 00:00:00", lastSeenAt: new Date().toISOString() });
|
||||||
|
const [s] = await db.select().from(servers).where(eq(servers.name, "down-now"));
|
||||||
|
const startOfMonth = new Date(Date.UTC(new Date().getUTCFullYear(), new Date().getUTCMonth(), 1)).toISOString().slice(0, 19).replace("T", " ");
|
||||||
|
await db.insert(auditLog).values({ category: "server", action: "went_offline", targetType: "server", targetId: String(s.id), createdAt: startOfMonth });
|
||||||
|
await db.insert(serviceLevels).values({ name: "Always up", subjectType: "server", subjectId: s.id, target: 99.9 });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
const found = (await getAlerts(true)).alerts.find((a) => a.message.startsWith("Always up has missed"));
|
||||||
|
assert.ok(found, "the alert is there");
|
||||||
|
assert.deepEqual([found.severity, found.category, found.source, found.link], ["critical", "monitoring", "Service level", "/service-levels"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("/metrics has each target, how it is doing and what is left", async () => {
|
||||||
|
const text = await buildMetrics();
|
||||||
|
assert.match(text, /homelab_service_level_target_percent\{name="Gitea answers"[^}]*\} 99\.9/);
|
||||||
|
assert.match(text, /homelab_service_level_error_budget_remaining_minutes\{name="Always up"[^}]*\} -\d/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("anyone signed in can look; only an administrator sets, changes and removes targets, with a reason when it is wrong", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
const list = (await (await send("GET", "/service-levels")).json()) as any;
|
||||||
|
assert.ok(list.levels.length >= 4 && list.subjects.endpoint.length >= 1 && list.subjects.server.length >= 1);
|
||||||
|
assert.equal((await send("POST", "/service-levels", { name: "x", subjectType: "server", subjectId: 1, target: 99.9 })).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
const [e] = list.subjects.endpoint;
|
||||||
|
const good = { name: "Wiki", subjectType: "endpoint", subjectId: e.id, target: 99.95 };
|
||||||
|
const res = await send("POST", "/service-levels", good);
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
const id = ((await res.json()) as { id: number }).id;
|
||||||
|
for (const bad of [{ ...good, target: 89.9 }, { ...good, target: 100 }, { ...good, name: " " }, { ...good, subjectType: "dns" }, { ...good, subjectId: 99999 }]) {
|
||||||
|
const r = await send("POST", "/service-levels", bad);
|
||||||
|
assert.equal(r.status, 400, JSON.stringify(bad));
|
||||||
|
}
|
||||||
|
assert.match(((await (await send("POST", "/service-levels", { ...good, target: 100 })).json()) as { message: string }).message, /100% leaves no room/);
|
||||||
|
assert.equal((await send("PUT", `/service-levels/${id}`, { ...good, target: 99.5, name: "Wiki up" })).status, 200);
|
||||||
|
const [row] = await db.select().from(serviceLevels).where(eq(serviceLevels.id, id));
|
||||||
|
assert.deepEqual([row.name, row.target], ["Wiki up", 99.5]);
|
||||||
|
assert.equal((await send("PUT", "/service-levels/99999", good)).status, 404);
|
||||||
|
const audited = await db.select().from(auditLog).where(eq(auditLog.category, "service_level"));
|
||||||
|
assert.deepEqual(audited.map((a) => a.action).sort(), ["create", "update"]);
|
||||||
|
who = "viewer";
|
||||||
|
assert.equal((await send("DELETE", `/service-levels/${id}`)).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
assert.equal((await send("DELETE", `/service-levels/${id}`)).status, 204);
|
||||||
|
assert.equal((await send("DELETE", `/service-levels/${id}`)).status, 404);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the report route takes a month, a quarter or a year, and offers all three", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
const body = (await (await send("GET", "/reports/monthly?month=2026-Q3")).json()) as any;
|
||||||
|
assert.deepEqual([body.report.period, body.report.label], ["quarter", "Q3 2026"]);
|
||||||
|
assert.deepEqual([body.months.length, body.quarters.length, body.years.length], [12, 8, 3]);
|
||||||
|
assert.equal((await send("GET", `/reports/monthly?month=${new Date().getUTCFullYear()}`)).status, 200);
|
||||||
|
for (const bad of ["2026-Q9", "1999", "1999-Q1", "abc"]) assert.equal((await send("GET", `/reports/monthly?month=${bad}`)).status, 400, bad);
|
||||||
|
const text = (await (await send("GET", "/reports/monthly?month=2026-Q3&format=text")).json()) as { title: string };
|
||||||
|
assert.equal(text.title, "Homelab quarterly report — Q3 2026");
|
||||||
|
who = "admin";
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a quarter's and a year's report follow the monthly one when they are switched on, and only after the month that closes them", async () => {
|
||||||
|
const titles = () => messages.map((m) => /Homelab (monthly|quarterly|annual) report — [^*]+/.exec(m)?.[0] ?? "").filter(Boolean);
|
||||||
|
const base = { enabled: true, day: 1, time: "08:00" };
|
||||||
|
messages.length = 0;
|
||||||
|
await updateSettings({ monthlyReport: { ...base, quarterly: false, annual: false } });
|
||||||
|
await sendMonthlyReport(new Date("2026-10-02T09:00:00Z"));
|
||||||
|
assert.deepEqual(titles(), ["Homelab monthly report — September 2026"]);
|
||||||
|
messages.length = 0;
|
||||||
|
await updateSettings({ monthlyReport: { ...base, quarterly: true, annual: true } });
|
||||||
|
await sendMonthlyReport(new Date("2026-10-02T09:00:00Z"));
|
||||||
|
assert.deepEqual(titles(), ["Homelab monthly report — September 2026", "Homelab quarterly report — Q3 2026"], "September closes a quarter, not the year");
|
||||||
|
messages.length = 0;
|
||||||
|
await sendMonthlyReport(new Date("2026-11-02T09:00:00Z"));
|
||||||
|
assert.deepEqual(titles(), ["Homelab monthly report — October 2026"], "October closes nothing");
|
||||||
|
messages.length = 0;
|
||||||
|
await sendMonthlyReport(new Date("2027-01-02T09:00:00Z"));
|
||||||
|
assert.deepEqual(titles(), ["Homelab monthly report — December 2026", "Homelab quarterly report — Q4 2026", "Homelab annual report — 2026"]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -11,6 +11,7 @@ import ProxyHosts from "./pages/ProxyHosts";
|
|||||||
import AlertRules from "./pages/AlertRules";
|
import AlertRules from "./pages/AlertRules";
|
||||||
import HomeAssistant from "./pages/HomeAssistant";
|
import HomeAssistant from "./pages/HomeAssistant";
|
||||||
import Ups from "./pages/Ups";
|
import Ups from "./pages/Ups";
|
||||||
|
import ServiceLevels from "./pages/ServiceLevels";
|
||||||
import MyWorkPage from "./pages/MyWork";
|
import MyWorkPage from "./pages/MyWork";
|
||||||
import ApiTokens from "./pages/ApiTokens";
|
import ApiTokens from "./pages/ApiTokens";
|
||||||
import Todo from "./pages/Todo";
|
import Todo from "./pages/Todo";
|
||||||
@@ -156,6 +157,7 @@ function Shelled({ user }: { user: CurrentUser }) {
|
|||||||
<Route path="/alert-rules" element={<AlertRules user={user} />} />
|
<Route path="/alert-rules" element={<AlertRules user={user} />} />
|
||||||
<Route path="/home-assistant" element={<HomeAssistant user={user} />} />
|
<Route path="/home-assistant" element={<HomeAssistant user={user} />} />
|
||||||
<Route path="/ups" element={<Ups user={user} />} />
|
<Route path="/ups" element={<Ups user={user} />} />
|
||||||
|
<Route path="/service-levels" element={<ServiceLevels user={user} />} />
|
||||||
<Route path="/ports" element={<Ports user={user} />} />
|
<Route path="/ports" element={<Ports user={user} />} />
|
||||||
<Route path="/secrets" element={<Secrets user={user} />} />
|
<Route path="/secrets" element={<Secrets user={user} />} />
|
||||||
<Route path="/integrations" element={<Integrations user={user} />} />
|
<Route path="/integrations" element={<Integrations user={user} />} />
|
||||||
|
|||||||
+55
-1
@@ -542,6 +542,10 @@ export interface HealthCheckSettings {
|
|||||||
/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. */
|
/** The monthly report: last month's availability, incidents, updates and renewals, in one message on a day of the month. */
|
||||||
export interface MonthlyReportSettings {
|
export interface MonthlyReportSettings {
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
|
/** Also send the quarter's report after the month that ends it. */
|
||||||
|
quarterly: boolean;
|
||||||
|
/** Also send the year's report after December. */
|
||||||
|
annual: boolean;
|
||||||
/** Day of the month, 1–28, in the notification time zone. */
|
/** Day of the month, 1–28, in the notification time zone. */
|
||||||
day: number;
|
day: number;
|
||||||
/** "HH:MM", in the notification time zone. */
|
/** "HH:MM", in the notification time zone. */
|
||||||
@@ -1101,9 +1105,52 @@ export interface AvailabilityRow {
|
|||||||
percent: number | null;
|
percent: number | null;
|
||||||
downMinutes: number;
|
downMinutes: number;
|
||||||
}
|
}
|
||||||
|
export type ServiceLevelSubject = "endpoint" | "server" | "integration";
|
||||||
|
|
||||||
|
/** How one service-level target did over a stretch (a month, quarter or year) in a report. */
|
||||||
|
export interface ServiceLevelResult {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
subjectType: ServiceLevelSubject;
|
||||||
|
target: number;
|
||||||
|
percent: number | null;
|
||||||
|
downMinutes: number;
|
||||||
|
allowedMinutes: number;
|
||||||
|
usedPercent: number | null;
|
||||||
|
met: boolean | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A service-level target, where this month's error budget stands, and how the months before it went. */
|
||||||
|
export interface ServiceLevel {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
subjectType: ServiceLevelSubject;
|
||||||
|
subjectId: number;
|
||||||
|
/** The thing measured; null when it has been removed. */
|
||||||
|
subjectName: string | null;
|
||||||
|
target: number;
|
||||||
|
alert: boolean;
|
||||||
|
createdBy: string | null;
|
||||||
|
period: { id: string; label: string; from: string; to: string; totalMinutes: number; elapsedMinutes: number };
|
||||||
|
measure: { percent: number | null; downMinutes: number; knownMinutes: number };
|
||||||
|
budget: { allowedMinutes: number; usedPercent: number | null; remainingMinutes: number; state: "ok" | "at_risk" | "breached" | "no_data"; projectedPercent: number | null };
|
||||||
|
history: { id: string; label: string; percent: number | null; downMinutes: number; allowedMinutes: number; met: boolean | null }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ServiceLevelInput {
|
||||||
|
name: string;
|
||||||
|
subjectType: ServiceLevelSubject;
|
||||||
|
subjectId: number;
|
||||||
|
target: number;
|
||||||
|
alert: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
export interface MonthlyReport {
|
export interface MonthlyReport {
|
||||||
|
/** A month ("2026-09"), a quarter ("2026-Q3") or a year ("2026"). */
|
||||||
|
period: "month" | "quarter" | "year";
|
||||||
month: string;
|
month: string;
|
||||||
label: string;
|
label: string;
|
||||||
|
serviceLevels: ServiceLevelResult[];
|
||||||
from: string;
|
from: string;
|
||||||
to: string;
|
to: string;
|
||||||
partial: boolean;
|
partial: boolean;
|
||||||
@@ -2510,6 +2557,12 @@ export const api = {
|
|||||||
create: (label: string) => request<{ token: MetricsToken; secret: string }>("/api/metrics-tokens", { method: "POST", body: JSON.stringify({ label }) }),
|
create: (label: string) => request<{ token: MetricsToken; secret: string }>("/api/metrics-tokens", { method: "POST", body: JSON.stringify({ label }) }),
|
||||||
remove: (id: number) => request<void>(`/api/metrics-tokens/${id}`, { method: "DELETE" }),
|
remove: (id: number) => request<void>(`/api/metrics-tokens/${id}`, { method: "DELETE" }),
|
||||||
},
|
},
|
||||||
|
serviceLevels: {
|
||||||
|
list: () => request<{ levels: ServiceLevel[]; subjects: Record<ServiceLevelSubject, { id: number; name: string }[]> }>("/api/service-levels"),
|
||||||
|
create: (data: ServiceLevelInput) => request<{ id: number }>("/api/service-levels", { method: "POST", body: JSON.stringify(data) }),
|
||||||
|
update: (id: number, data: ServiceLevelInput) => request<{ ok: boolean }>(`/api/service-levels/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||||
|
remove: (id: number) => request<void>(`/api/service-levels/${id}`, { method: "DELETE" }),
|
||||||
|
},
|
||||||
ups: {
|
ups: {
|
||||||
list: () => request<{ ups: UpsInfo[] }>("/api/ups"),
|
list: () => request<{ ups: UpsInfo[] }>("/api/ups"),
|
||||||
history: (id: number, days: number) => request<UpsHistory>(`/api/ups/${id}/history?days=${days}`),
|
history: (id: number, days: number) => request<UpsHistory>(`/api/ups/${id}/history?days=${days}`),
|
||||||
@@ -2545,7 +2598,8 @@ export const api = {
|
|||||||
cancel: (runId: number) => request<{ run: PatchPlanRun }>(`/api/patch-plans/runs/${runId}/cancel`, { method: "POST" }),
|
cancel: (runId: number) => request<{ run: PatchPlanRun }>(`/api/patch-plans/runs/${runId}/cancel`, { method: "POST" }),
|
||||||
},
|
},
|
||||||
reports: {
|
reports: {
|
||||||
monthly: (month?: string) => request<{ report: MonthlyReport; months: { id: string; label: string }[] }>(`/api/reports/monthly${month ? `?month=${month}` : ""}`),
|
monthly: (month?: string) =>
|
||||||
|
request<{ report: MonthlyReport; months: { id: string; label: string }[]; quarters: { id: string; label: string }[]; years: { id: string; label: string }[] }>(`/api/reports/monthly${month ? `?month=${month}` : ""}`),
|
||||||
monthlyText: (month?: string) => request<{ title: string; message: string }>(`/api/reports/monthly?format=text${month ? `&month=${month}` : ""}`),
|
monthlyText: (month?: string) => request<{ title: string; message: string }>(`/api/reports/monthly?format=text${month ? `&month=${month}` : ""}`),
|
||||||
sendMonthly: (month?: string) => request<{ sent: boolean }>("/api/reports/monthly/send", { method: "POST", body: JSON.stringify({ month }) }),
|
sendMonthly: (month?: string) => request<{ sent: boolean }>("/api/reports/monthly/send", { method: "POST", body: JSON.stringify({ month }) }),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import {
|
|||||||
IconArrowsSplit2,
|
IconArrowsSplit2,
|
||||||
IconHome2,
|
IconHome2,
|
||||||
IconBatteryCharging,
|
IconBatteryCharging,
|
||||||
|
IconTargetArrow,
|
||||||
IconAdjustmentsAlt,
|
IconAdjustmentsAlt,
|
||||||
IconChecklist,
|
IconChecklist,
|
||||||
IconListDetails,
|
IconListDetails,
|
||||||
@@ -118,6 +119,7 @@ export const NAV: NavEntry[] = [
|
|||||||
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
||||||
{ to: "/home-assistant", label: "Home Assistant", icon: <IconHome2 size={20} /> },
|
{ to: "/home-assistant", label: "Home Assistant", icon: <IconHome2 size={20} /> },
|
||||||
{ to: "/ups", label: "UPS", icon: <IconBatteryCharging size={20} /> },
|
{ to: "/ups", label: "UPS", icon: <IconBatteryCharging size={20} /> },
|
||||||
|
{ to: "/service-levels", label: "Service levels", icon: <IconTargetArrow size={20} /> },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { useEffect, useState } from "react";
|
import { useEffect, useState } from "react";
|
||||||
import { Link, useSearchParams } from "react-router-dom";
|
import { Link, useSearchParams } from "react-router-dom";
|
||||||
import { IconPrinter } from "@tabler/icons-react";
|
import { IconPrinter } from "@tabler/icons-react";
|
||||||
import { api, type AvailabilityRow, type CurrentUser, type MonthlyReport as Report } from "../api/client";
|
import { api, type AvailabilityRow, type CurrentUser, type MonthlyReport as Report, type ServiceLevelResult } from "../api/client";
|
||||||
import { ErrorNotice } from "../components/Notice";
|
import { ErrorNotice } from "../components/Notice";
|
||||||
import { BlockSkeleton } from "../components/Skeleton";
|
import { BlockSkeleton } from "../components/Skeleton";
|
||||||
import { CATEGORY_NAMES, incidentPath, SEVERITY_BADGE } from "../components/IncidentHistory";
|
import { CATEGORY_NAMES, incidentPath, SEVERITY_BADGE } from "../components/IncidentHistory";
|
||||||
@@ -33,6 +33,51 @@ const average = (rows: { percent: number | null }[]) => {
|
|||||||
|
|
||||||
const percentBadge = (p: number | null) => (p === null ? "text-secondary" : p >= 99.9 ? "text-green" : p >= 99 ? "" : p >= 95 ? "text-yellow" : "text-red");
|
const percentBadge = (p: number | null) => (p === null ? "text-secondary" : p >= 99.9 ? "text-green" : p >= 99 ? "" : p >= 95 ? "text-yellow" : "text-red");
|
||||||
|
|
||||||
|
/** How each service-level target did over the stretch: its figure against the target, and the downtime against what it allowed. */
|
||||||
|
function ServiceLevelsSection({ rows, partial }: { rows: ServiceLevelResult[]; partial: boolean }) {
|
||||||
|
if (rows.length === 0) return null;
|
||||||
|
const metCount = rows.filter((l) => l.met).length;
|
||||||
|
return (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Service levels</h3>
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
<p>
|
||||||
|
<strong>{metCount}</strong> of {rows.length} {rows.length === 1 ? "target was" : "targets were"} met{partial ? " so far" : ""}. Each is judged on the downtime it allowed over the whole stretch (the error budget).{" "}
|
||||||
|
<Link to="/service-levels">Set targets</Link>
|
||||||
|
</p>
|
||||||
|
<div className="table-responsive">
|
||||||
|
<table className="table table-vcenter table-sm">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Target</th>
|
||||||
|
<th className="text-end">Target</th>
|
||||||
|
<th className="text-end">Achieved</th>
|
||||||
|
<th className="text-end">Down for</th>
|
||||||
|
<th className="text-end">Allowed</th>
|
||||||
|
<th />
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{rows.map((l) => (
|
||||||
|
<tr key={l.id}>
|
||||||
|
<td>{l.name}</td>
|
||||||
|
<td className="text-end">{l.target}%</td>
|
||||||
|
<td className={`text-end ${l.met ? "text-green" : "text-red"}`}>{l.percent}%</td>
|
||||||
|
<td className="text-end text-secondary">{l.downMinutes > 0 ? span(l.downMinutes) : "—"}</td>
|
||||||
|
<td className="text-end text-secondary">{l.allowedMinutes >= 1 ? span(l.allowedMinutes) : "under a minute"}</td>
|
||||||
|
<td>{l.met ? <span className="badge bg-green-lt text-green">met</span> : <span className="badge bg-red-lt text-red">missed</span>}</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
function Availability({ title, rows, hint }: { title: string; rows: AvailabilityRow[]; hint?: string }) {
|
function Availability({ title, rows, hint }: { title: string; rows: AvailabilityRow[]; hint?: string }) {
|
||||||
if (rows.length === 0) return null;
|
if (rows.length === 0) return null;
|
||||||
return (
|
return (
|
||||||
@@ -71,7 +116,7 @@ function Availability({ title, rows, hint }: { title: string; rows: Availability
|
|||||||
export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
||||||
const [params, setParams] = useSearchParams();
|
const [params, setParams] = useSearchParams();
|
||||||
const month = params.get("month") ?? undefined;
|
const month = params.get("month") ?? undefined;
|
||||||
const [data, setData] = useState<{ report: Report; months: { id: string; label: string }[] } | null>(null);
|
const [data, setData] = useState<{ report: Report; months: { id: string; label: string }[]; quarters: { id: string; label: string }[]; years: { id: string; label: string }[] } | null>(null);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -101,6 +146,12 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const r = data?.report;
|
const r = data?.report;
|
||||||
|
const kind = r?.period ?? (month ? (/Q/.test(month) ? "quarter" : /^\d{4}$/.test(month) ? "year" : "month") : "month");
|
||||||
|
const noun = kind === "year" ? "Annual" : kind === "quarter" ? "Quarterly" : "Monthly";
|
||||||
|
const stretch = kind === "year" ? "year" : kind === "quarter" ? "quarter" : "month";
|
||||||
|
const options = data ? (kind === "year" ? data.years : kind === "quarter" ? data.quarters : data.months) : [];
|
||||||
|
/** The newest finished one of a kind, which is what is shown when nothing is asked for. */
|
||||||
|
const lastWhole = (list: { id: string }[]) => list[1]?.id ?? list[0]?.id;
|
||||||
const serverAvg = r ? average(r.servers) : null;
|
const serverAvg = r ? average(r.servers) : null;
|
||||||
const endpointAvg = r ? average(r.endpoints) : null;
|
const endpointAvg = r ? average(r.endpoints) : null;
|
||||||
const cards = r
|
const cards = r
|
||||||
@@ -116,14 +167,23 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
return (
|
return (
|
||||||
<div className="report-page">
|
<div className="report-page">
|
||||||
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
||||||
<h2 className="page-title mb-0">Monthly report</h2>
|
<h2 className="page-title mb-0">{noun} report</h2>
|
||||||
<div className="ms-auto d-flex flex-wrap gap-2 no-print">
|
<div className="ms-auto d-flex flex-wrap gap-2 no-print">
|
||||||
{data && (
|
{data && (
|
||||||
<select className="form-select w-auto" aria-label="Month" value={r?.month ?? ""} onChange={(e) => setParams(e.target.value === data.months[1]?.id ? {} : { month: e.target.value }, { replace: true })}>
|
<div className="btn-group" role="group" aria-label="What the report covers">
|
||||||
{data.months.map((m) => (
|
{([["month", "Month", data.months], ["quarter", "Quarter", data.quarters], ["year", "Year", data.years]] as const).map(([k, label, list]) => (
|
||||||
|
<button key={k} type="button" className={`btn ${kind === k ? "btn-primary" : "btn-outline-secondary"}`} aria-pressed={kind === k} onClick={() => setParams(k === "month" ? {} : { month: lastWhole(list) ?? "" }, { replace: true })}>
|
||||||
|
{label}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{data && (
|
||||||
|
<select className="form-select w-auto" aria-label={noun === "Monthly" ? "Month" : noun === "Quarterly" ? "Quarter" : "Year"} value={r?.month ?? ""} onChange={(e) => setParams(e.target.value === lastWhole(options) && kind === "month" ? {} : { month: e.target.value }, { replace: true })}>
|
||||||
|
{options.map((m) => (
|
||||||
<option key={m.id} value={m.id}>
|
<option key={m.id} value={m.id}>
|
||||||
{m.label}
|
{m.label}
|
||||||
{m.id === data.months[0].id ? " (so far)" : ""}
|
{m.id === options[0].id ? " (so far)" : ""}
|
||||||
</option>
|
</option>
|
||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
@@ -137,11 +197,11 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<p className="text-secondary no-print">
|
<p className="text-secondary no-print">
|
||||||
How the month went, in one page: availability, incidents and how long they took, updates applied, restore tests and backups, and what was renewed.
|
How the {stretch} went, in one page: availability, how each service-level target did, incidents and how long they took, updates applied, restore tests and backups, and what was renewed.
|
||||||
{user.role === "admin" ? (
|
{user.role === "admin" ? (
|
||||||
<>
|
<>
|
||||||
{" "}
|
{" "}
|
||||||
It can be sent on a day of the month: <Link to="/settings/notifications">Settings → Notifications</Link>.
|
It can be sent on a day of the month, and a quarter's and year's report after the month that ends it: <Link to="/settings/notifications">Settings → Notifications</Link>.
|
||||||
</>
|
</>
|
||||||
) : null}
|
) : null}
|
||||||
</p>
|
</p>
|
||||||
@@ -153,7 +213,7 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
<h3 className="mb-1">{r.label}</h3>
|
<h3 className="mb-1">{r.label}</h3>
|
||||||
<div className="text-secondary mb-3">
|
<div className="text-secondary mb-3">
|
||||||
{formatDate(new Date(r.from))} – {formatDate(new Date(r.to))}
|
{formatDate(new Date(r.from))} – {formatDate(new Date(r.to))}
|
||||||
{r.partial ? " · the month is not over, so this is the month so far" : ""} · time zone {r.timezone}
|
{r.partial ? ` · the ${stretch} is not over, so this is the ${stretch} so far` : ""} · time zone {r.timezone}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="row row-cards mb-3">
|
<div className="row row-cards mb-3">
|
||||||
@@ -213,6 +273,8 @@ export default function MonthlyReport({ user }: { user: CurrentUser }) {
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<ServiceLevelsSection rows={r.serviceLevels} partial={r.partial} />
|
||||||
|
|
||||||
<div className="card mb-3">
|
<div className="card mb-3">
|
||||||
<div className="card-header">
|
<div className="card-header">
|
||||||
<h3 className="card-title">Incidents</h3>
|
<h3 className="card-title">Incidents</h3>
|
||||||
|
|||||||
@@ -0,0 +1,292 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type CurrentUser, type ServiceLevel, type ServiceLevelInput, type ServiceLevelSubject } from "../api/client";
|
||||||
|
import EmptyState from "../components/EmptyState";
|
||||||
|
import { ErrorNotice } from "../components/Notice";
|
||||||
|
import { BlockSkeleton } from "../components/Skeleton";
|
||||||
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
|
||||||
|
const SUBJECT_WORDS: Record<ServiceLevelSubject, string> = { endpoint: "Endpoint check", server: "Server", integration: "Integration" };
|
||||||
|
const SUBJECT_HINTS: Record<ServiceLevelSubject, string> = {
|
||||||
|
endpoint: "Measured from the check's answers, day by day: the share that failed.",
|
||||||
|
server: "Measured from when it stopped reporting and reported again, to the minute.",
|
||||||
|
integration: "Measured from when calls to that kind of service started failing and worked again.",
|
||||||
|
};
|
||||||
|
|
||||||
|
/** "43 min", "4 h", "2 d 3 h"; under a minute is said so. */
|
||||||
|
function span(minutes: number): string {
|
||||||
|
if (minutes < 1) return "under a minute";
|
||||||
|
const m = Math.round(minutes);
|
||||||
|
if (m < 60) return `${m} min`;
|
||||||
|
const h = Math.floor(m / 60);
|
||||||
|
if (h < 24) return m % 60 && h < 10 ? `${h} h ${m % 60} min` : `${h} h`;
|
||||||
|
const d = Math.floor(h / 24);
|
||||||
|
return h % 24 ? `${d} d ${h % 24} h` : `${d} d`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const COLOR = { ok: "#2fb344", at_risk: "#f59f00", breached: "#e05252", no_data: "#9aa0a6" } as const;
|
||||||
|
const BADGE = {
|
||||||
|
ok: { label: "Within budget", cls: "bg-green-lt text-green" },
|
||||||
|
at_risk: { label: "Budget nearly used", cls: "bg-yellow-lt text-yellow" },
|
||||||
|
breached: { label: "Target missed", cls: "bg-red-lt text-red" },
|
||||||
|
no_data: { label: "Nothing to measure yet", cls: "bg-secondary-lt text-secondary" },
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
interface Draft {
|
||||||
|
id: number | null;
|
||||||
|
name: string;
|
||||||
|
subjectType: ServiceLevelSubject;
|
||||||
|
subjectId: string;
|
||||||
|
target: string;
|
||||||
|
alert: boolean;
|
||||||
|
}
|
||||||
|
const EMPTY: Draft = { id: null, name: "", subjectType: "endpoint", subjectId: "", target: "99.9", alert: true };
|
||||||
|
|
||||||
|
function Card({ level: l, isAdmin, onEdit, onRemove }: { level: ServiceLevel; isAdmin: boolean; onEdit: () => void; onRemove: () => void }) {
|
||||||
|
const b = l.budget;
|
||||||
|
const used = b.usedPercent ?? 0;
|
||||||
|
return (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<div>
|
||||||
|
<h3 className="card-title mb-0">
|
||||||
|
{l.name} <span className={`badge ${BADGE[b.state].cls}`}>{BADGE[b.state].label}</span>
|
||||||
|
</h3>
|
||||||
|
<div className="text-secondary small">
|
||||||
|
{SUBJECT_WORDS[l.subjectType]}: {l.subjectName ?? <em>removed</em>} · target <strong>{l.target}%</strong> a month
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{isAdmin && (
|
||||||
|
<div className="card-actions btn-list flex-nowrap">
|
||||||
|
<button className="btn btn-sm btn-outline-secondary" onClick={onEdit}>
|
||||||
|
Edit
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-sm btn-outline-danger" onClick={onRemove}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
{b.state === "no_data" ? (
|
||||||
|
<div className="text-secondary">{l.subjectName === null ? "What this measured has been removed." : `Nothing has been recorded for ${l.period.label} yet. ${SUBJECT_HINTS[l.subjectType]}`}</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<div className="row g-3 mb-2">
|
||||||
|
<div className="col-6 col-md-3">
|
||||||
|
<div className="text-secondary small">So far in {l.period.label}</div>
|
||||||
|
<div className="h2 mb-0" style={{ color: COLOR[b.state] }}>
|
||||||
|
{l.measure.percent}%
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-6 col-md-3">
|
||||||
|
<div className="text-secondary small">Down</div>
|
||||||
|
<div className="h2 mb-0">{l.measure.downMinutes > 0 ? span(l.measure.downMinutes) : "—"}</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-6 col-md-3">
|
||||||
|
<div className="text-secondary small">Allowed this month</div>
|
||||||
|
<div className="h2 mb-0">{span(b.allowedMinutes)}</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-6 col-md-3">
|
||||||
|
<div className="text-secondary small">{b.remainingMinutes >= 0 ? "Left in the budget" : "Over the budget by"}</div>
|
||||||
|
<div className="h2 mb-0" style={{ color: b.remainingMinutes < 0 ? COLOR.breached : undefined }}>
|
||||||
|
{b.remainingMinutes >= 0 ? span(b.remainingMinutes) : span(-b.remainingMinutes)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="progress progress-sm mb-1" role="progressbar" aria-label="Error budget used" aria-valuenow={Math.min(100, Math.round(used))} aria-valuemin={0} aria-valuemax={100}>
|
||||||
|
<div className="progress-bar" style={{ width: `${Math.min(100, used)}%`, background: COLOR[b.state] }} />
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small">
|
||||||
|
{used >= 100 ? "The whole error budget is used" : `${Math.round(used)}% of the error budget is used`}
|
||||||
|
{b.projectedPercent !== null && b.state !== "breached" ? ` · at this pace, about ${b.projectedPercent}% by the end of the month` : ""}
|
||||||
|
{l.alert ? "" : " · no alert for this one"}
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{l.history.some((h) => h.percent !== null) && (
|
||||||
|
<div className="mt-3">
|
||||||
|
<div className="text-secondary small mb-1">The months before</div>
|
||||||
|
<div className="d-flex flex-wrap gap-2">
|
||||||
|
{l.history.map((h) => (
|
||||||
|
<span
|
||||||
|
key={h.id}
|
||||||
|
className={`badge ${h.met === null ? "bg-secondary-lt text-secondary" : h.met ? "bg-green-lt text-green" : "bg-red-lt text-red"}`}
|
||||||
|
title={h.percent === null ? `${h.label}: nothing recorded` : `${h.label}: ${h.percent}%, down ${span(h.downMinutes)} of ${span(h.allowedMinutes)} allowed`}
|
||||||
|
>
|
||||||
|
{h.label.split(" ")[0].slice(0, 3)} {h.percent === null ? "—" : `${h.percent}%`}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What each service is held to — "99.9% for Gitea" — and what is left of this month's error budget; the months before show whether it was kept. */
|
||||||
|
export default function ServiceLevels({ user }: { user: CurrentUser }) {
|
||||||
|
const isAdmin = user.role === "admin";
|
||||||
|
const [data, setData] = useState<Awaited<ReturnType<typeof api.serviceLevels.list>> | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [draft, setDraft] = useState<Draft | null>(null);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
|
||||||
|
function load() {
|
||||||
|
api.serviceLevels
|
||||||
|
.list()
|
||||||
|
.then((res) => {
|
||||||
|
setData(res);
|
||||||
|
setError(null);
|
||||||
|
})
|
||||||
|
.catch((err) => setError(readableError(err)));
|
||||||
|
}
|
||||||
|
useEffect(() => {
|
||||||
|
load();
|
||||||
|
const t = setInterval(load, 120_000);
|
||||||
|
return () => clearInterval(t);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const set = <K extends keyof Draft>(key: K, value: Draft[K]) => setDraft((d) => (d ? { ...d, [key]: value } : d));
|
||||||
|
const subjects = data && draft ? data.subjects[draft.subjectType] : [];
|
||||||
|
|
||||||
|
async function save(e: React.FormEvent) {
|
||||||
|
e.preventDefault();
|
||||||
|
if (!draft) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError(null);
|
||||||
|
const input: ServiceLevelInput = { name: draft.name.trim(), subjectType: draft.subjectType, subjectId: Number(draft.subjectId), target: Number(draft.target), alert: draft.alert };
|
||||||
|
try {
|
||||||
|
if (draft.id === null) await api.serviceLevels.create(input);
|
||||||
|
else await api.serviceLevels.update(draft.id, input);
|
||||||
|
toast.success(draft.id === null ? `Set a target for ${input.name}.` : "Saved.");
|
||||||
|
setDraft(null);
|
||||||
|
load();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function remove(l: ServiceLevel) {
|
||||||
|
if (!(await confirmDialog({ title: "Remove a target", message: `Remove the target for "${l.name}"? What was measured is kept; only the target goes.`, confirmLabel: "Remove", danger: true }))) return;
|
||||||
|
try {
|
||||||
|
await api.serviceLevels.remove(l.id);
|
||||||
|
load();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="d-flex align-items-center mb-1">
|
||||||
|
<h2 className="page-title mb-0">Service levels</h2>
|
||||||
|
{isAdmin && !draft && (
|
||||||
|
<button className="btn btn-primary ms-auto" onClick={() => setDraft({ ...EMPTY })}>
|
||||||
|
Set a target
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<p className="text-secondary">
|
||||||
|
What a service is held to — for example <strong>99.9%</strong> for Gitea — and what is left of this month’s <strong>error budget</strong>: the downtime the target allows. A month with 99.9% allows about 45 minutes. The months before show whether it
|
||||||
|
was kept. When most of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over. Planned maintenance is not taken out: down is down. The same figures are in the{" "}
|
||||||
|
<Link to="/reports">monthly, quarterly and annual reports</Link>.
|
||||||
|
</p>
|
||||||
|
<ErrorNotice error={error} />
|
||||||
|
|
||||||
|
{draft && data && (
|
||||||
|
<form className="card mb-3" onSubmit={save}>
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">{draft.id === null ? "Set a target" : `Edit ${draft.name || "the target"}`}</h3>
|
||||||
|
</div>
|
||||||
|
<div className="card-body row g-3">
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="sl-name">
|
||||||
|
Name
|
||||||
|
</label>
|
||||||
|
<input id="sl-name" className="form-control" required maxLength={100} placeholder="e.g. Gitea answers" value={draft.name} onChange={(e) => set("name", e.target.value)} />
|
||||||
|
</div>
|
||||||
|
<div className="col-md-3">
|
||||||
|
<label className="form-label" htmlFor="sl-type">
|
||||||
|
Measured on
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
id="sl-type"
|
||||||
|
className="form-select"
|
||||||
|
value={draft.subjectType}
|
||||||
|
onChange={(e) => setDraft((d) => (d ? { ...d, subjectType: e.target.value as ServiceLevelSubject, subjectId: "" } : d))}
|
||||||
|
>
|
||||||
|
{(Object.keys(SUBJECT_WORDS) as ServiceLevelSubject[]).map((k) => (
|
||||||
|
<option key={k} value={k}>
|
||||||
|
{SUBJECT_WORDS[k]}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-5">
|
||||||
|
<label className="form-label" htmlFor="sl-subject">
|
||||||
|
Which one
|
||||||
|
</label>
|
||||||
|
<select id="sl-subject" className="form-select" required value={draft.subjectId} onChange={(e) => set("subjectId", e.target.value)}>
|
||||||
|
<option value="">— choose —</option>
|
||||||
|
{subjects.map((s) => (
|
||||||
|
<option key={s.id} value={s.id}>
|
||||||
|
{s.name}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<div className="form-hint text-secondary small mt-1">{SUBJECT_HINTS[draft.subjectType]}</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="sl-target">
|
||||||
|
Target, per calendar month
|
||||||
|
</label>
|
||||||
|
<div className="input-group">
|
||||||
|
<input id="sl-target" type="number" className="form-control" required min={90} max={99.999} step="any" value={draft.target} onChange={(e) => set("target", e.target.value)} />
|
||||||
|
<span className="input-group-text">%</span>
|
||||||
|
</div>
|
||||||
|
<div className="form-hint text-secondary small mt-1">
|
||||||
|
99% allows about 7 h a month, 99.9% about 45 min, 99.99% about 4 min. Between 90 and 99.999.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-8 d-flex align-items-end">
|
||||||
|
<label className="form-check mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={draft.alert} onChange={(e) => set("alert", e.target.checked)} />
|
||||||
|
<span className="form-check-label">Warn when 80% of the month’s error budget is used, and when it is overspent</span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-footer d-flex gap-2">
|
||||||
|
<button type="submit" className="btn btn-primary" disabled={busy || draft.name.trim() === "" || draft.subjectId === ""}>
|
||||||
|
{busy ? "Saving…" : "Save"}
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn" onClick={() => setDraft(null)}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{!data && !error && <BlockSkeleton lines={4} />}
|
||||||
|
{data?.levels.length === 0 && !draft && (
|
||||||
|
<div className="card">
|
||||||
|
<div className="card-body">
|
||||||
|
<EmptyState
|
||||||
|
title="No targets are set"
|
||||||
|
hint="Say how available something should be — an endpoint check, a server, an integration — and see what is left of the month's error budget."
|
||||||
|
action={isAdmin ? { label: "Set a target", onClick: () => setDraft({ ...EMPTY }) } : undefined}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{data?.levels.map((l) => (
|
||||||
|
<Card key={l.id} level={l} isAdmin={isAdmin} onEdit={() => setDraft({ id: l.id, name: l.name, subjectType: l.subjectType, subjectId: String(l.subjectId), target: String(l.target), alert: l.alert })} onRemove={() => void remove(l)} />
|
||||||
|
))}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -84,7 +84,7 @@ const DIGEST_DAYS: { value: number; label: string }[] = [
|
|||||||
{ value: 0, label: "Sunday" },
|
{ value: 0, label: "Sunday" },
|
||||||
];
|
];
|
||||||
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true };
|
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14, backupStaleHours: 48, restoreTestDays: 0, newPortAlerts: true, flapRaises: 3, deviceOfflineMinutes: 10, osSupportWarnDays: 90, baselineAlerts: true };
|
||||||
const DEFAULT_MONTHLY: MonthlyReportSettings = { enabled: false, day: 1, time: "08:00" };
|
const DEFAULT_MONTHLY: MonthlyReportSettings = { enabled: false, quarterly: false, annual: false, day: 1, time: "08:00" };
|
||||||
|
|
||||||
type TestResult = { ok: boolean; message: string } | null;
|
type TestResult = { ok: boolean; message: string } | null;
|
||||||
type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends boolean ? K : never }[keyof NotificationEvents];
|
type EventKey = { [K in keyof NotificationEvents]: NotificationEvents[K] extends boolean ? K : never }[keyof NotificationEvents];
|
||||||
@@ -284,7 +284,7 @@ export default function NotificationSettings() {
|
|||||||
setHealthChecks(s.healthChecks);
|
setHealthChecks(s.healthChecks);
|
||||||
const d = s.digest ?? DEFAULT_DIGEST;
|
const d = s.digest ?? DEFAULT_DIGEST;
|
||||||
setDigest(d);
|
setDigest(d);
|
||||||
const mo = s.monthlyReport ?? DEFAULT_MONTHLY;
|
const mo = { ...DEFAULT_MONTHLY, ...s.monthlyReport };
|
||||||
setMonthly(mo);
|
setMonthly(mo);
|
||||||
setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, routing: rt, escalation: es, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d, monthlyReport: mo }));
|
setBaseline(JSON.stringify({ gotify: s.gotify, ntfy: s.ntfy, smtp: s.smtp, webhook: s.webhook, webPush: wp, routing: rt, escalation: es, notifications: s.notifications, quietHours: s.quietHours, healthChecks: s.healthChecks, digest: d, monthlyReport: mo }));
|
||||||
})
|
})
|
||||||
@@ -1012,6 +1012,14 @@ export default function NotificationSettings() {
|
|||||||
<input id="monthly-time" type="time" className="form-control" value={monthly.time} disabled={!monthly.enabled} onChange={(e) => setMonthly((m) => ({ ...m, time: e.target.value }))} />
|
<input id="monthly-time" type="time" className="form-control" value={monthly.time} disabled={!monthly.enabled} onChange={(e) => setMonthly((m) => ({ ...m, time: e.target.value }))} />
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
<label className="form-check mb-1">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={monthly.quarterly} disabled={!monthly.enabled} onChange={(e) => setMonthly((m) => ({ ...m, quarterly: e.target.checked }))} />
|
||||||
|
<span className="form-check-label">Also send a quarterly report, after the month that ends a quarter (March, June, September, December)</span>
|
||||||
|
</label>
|
||||||
|
<label className="form-check mb-3">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={monthly.annual} disabled={!monthly.enabled} onChange={(e) => setMonthly((m) => ({ ...m, annual: e.target.checked }))} />
|
||||||
|
<span className="form-check-label">Also send an annual report, after December</span>
|
||||||
|
</label>
|
||||||
<div className="btn-list">
|
<div className="btn-list">
|
||||||
<button type="button" className="btn btn-outline-secondary" onClick={handleMonthlyPreview} disabled={monthlyBusy !== null}>
|
<button type="button" className="btn btn-outline-secondary" onClick={handleMonthlyPreview} disabled={monthlyBusy !== null}>
|
||||||
{monthlyBusy === "preview" ? "Gathering…" : "Preview last month"}
|
{monthlyBusy === "preview" ? "Gathering…" : "Preview last month"}
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ export const SETTINGS_INDEX: SettingsEntry[] = [
|
|||||||
{ title: "Server offline after (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
|
{ title: "Server offline after (minutes)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "offline stopped reporting silent agent threshold limit minutes" },
|
||||||
{ title: "Disk usage alert at (percent)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk full storage percent threshold space" },
|
{ title: "Disk usage alert at (percent)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk full storage percent threshold space" },
|
||||||
{ title: "Warn when a disk will fill within (days)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk forecast fill growth trend days" },
|
{ title: "Warn when a disk will fill within (days)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "disk forecast fill growth trend days" },
|
||||||
|
{ title: "Quarterly and annual reports", where: "Notifications › When", to: `${N}?tab=when`, words: "report quarterly annual yearly quarter year send message monthly" },
|
||||||
{ title: "Warn when CPU or memory is unusually high for the time", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
|
{ title: "Warn when CPU or memory is unusually high for the time", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "baseline usual normal unusual unusually busy high cpu memory load weekday hour history anomaly" },
|
||||||
{ title: "Warn when a server starts listening on a new port", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "new port baseline listening tcp monitoring" },
|
{ title: "Warn when a server starts listening on a new port", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "new port baseline listening tcp monitoring" },
|
||||||
{ title: "Call an alert flapping after (raises in an hour)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
|
{ title: "Call an alert flapping after (raises in an hour)", where: "Notifications › What to send › Servers and disks", to: `${N}?tab=events`, words: "flapping flap noise repeat raised cleared again hold back" },
|
||||||
|
|||||||
Reference in new issue
Block a user