Heartbeats, Coming up and suggested checks; Operations menu tidied
Heartbeats: a cron job or script calls /api/heartbeat/<token> (and /start, /fail); the app alerts when the call is late or the job says it failed. Coming up: one agenda of everything dated (expiries, warranties, end of support, planned and recurring maintenance, restore tests, waiting updates), with a dashboard card and an iCalendar feed behind a share link. Suggested checks: web ports and certificates the app already knows about, added with a click. Menu: Incidents become Alerts -> History (old /incidents links redirect); new Monitoring group (Endpoints, Heartbeats, Uptime Kuma); osTicket under Automation, Admin Links under Infrastructure, Generator under Administration. Also makes the restore-test metrics test follow the real clock. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c05c847d17
commit
4e0c6927b2
44 files changed
+7090
-73
No files matched your search
@@ -1,8 +1,10 @@
|
|||||||
# Metrics and webhooks
|
# Metrics, webhooks, heartbeats and a calendar feed
|
||||||
|
|
||||||
Two ways for other systems to work with Homelab Manager, both set up by an administrator under
|
Ways for other systems to work with Homelab Manager:
|
||||||
**Settings → API and webhooks**: a read-only `/metrics` address for Prometheus, and webhooks that are
|
|
||||||
sent a signed message when something happens.
|
- a read-only **`/metrics`** address for Prometheus, and **event webhooks** that are sent a signed message when something happens — both set up by an administrator under **Settings → API and webhooks** (the first two sections below);
|
||||||
|
- **heartbeats**: a cron job or script calls an address when it finishes, and the app speaks up when it doesn't (further down);
|
||||||
|
- a **calendar feed** of what is coming up, for a phone's calendar to subscribe to (at the end).
|
||||||
|
|
||||||
## `/metrics` — for Prometheus
|
## `/metrics` — for Prometheus
|
||||||
|
|
||||||
@@ -72,6 +74,9 @@ All gauges. Labels are escaped as Prometheus requires.
|
|||||||
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
|
| `homelab_endpoint_up` | `name`, `kind` | 1 if the endpoint check's last answer was fine, else 0; left out until it has been asked. |
|
||||||
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
| `homelab_endpoint_response_milliseconds` | `name`, `kind` | How long it took to answer the last time. |
|
||||||
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
|
||||||
|
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |
|
||||||
|
| `homelab_heartbeat_last_ping_timestamp_seconds` | `name` | When the job last reported that it finished (Unix time). |
|
||||||
|
| `homelab_heartbeat_last_duration_seconds` | `name` | How long the job said its last run took (only when it sent `/start`). |
|
||||||
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
|
| `homelab_restore_test_age_days` | `subject` | Days since a restore of this was last tested; things marked "no need to test" are left out. |
|
||||||
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
|
| `homelab_restore_test_passed` | `subject` | 1 if the last restore test passed (fully or partly), 0 if it failed. |
|
||||||
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
|
| `homelab_scrape_duration_seconds` | | How long building the answer took. |
|
||||||
@@ -178,3 +183,47 @@ def is_from_homelab_manager(secret: str, raw_body: bytes, signature_header: str)
|
|||||||
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
|
The secret is stored encrypted when an encryption key is set, and is never shown again after it is saved.
|
||||||
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
|
Making, enabling, disabling and removing a webhook are in the audit log — with its name and events, never
|
||||||
its address or secret.
|
its address or secret.
|
||||||
|
|
||||||
|
## Heartbeats — for cron jobs and scripts
|
||||||
|
|
||||||
|
A heartbeat is made by an administrator under **Monitoring → Heartbeats**, and its address is shown there to administrators. The job calls it when it finishes:
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/heartbeat/<token> it worked
|
||||||
|
GET /api/heartbeat/<token>/start it began (optional: lets the next call say how long it took)
|
||||||
|
GET /api/heartbeat/<token>/fail it ran and went wrong: an alert at once
|
||||||
|
```
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# in cron, after the job
|
||||||
|
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxxxxxxxxxxxxxxxxxxxxxx
|
||||||
|
|
||||||
|
# in a script, with a start, a failure and a note
|
||||||
|
curl -fsS -m 10 -o /dev/null https://hm.example.lan/api/heartbeat/hb_xxxx/start
|
||||||
|
./backup.sh && curl -fsS -m 10 -o /dev/null --data-raw "3.2 GB copied" https://hm.example.lan/api/heartbeat/hb_xxxx \
|
||||||
|
|| curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" https://hm.example.lan/api/heartbeat/hb_xxxx/fail
|
||||||
|
```
|
||||||
|
|
||||||
|
- The token in the address is the whole of the access, and all it allows is to say "I'm alive": no session, no sign-in, and nothing can be read with it. GET, HEAD, POST and PUT all work.
|
||||||
|
Keep the address between the job and you; if it gets out, give the heartbeat a new address (the old one stops working at once).
|
||||||
|
- Anything sent with a call — the body of a POST, or `?msg=…` — is kept as a note, cut to one line of at most 200 characters.
|
||||||
|
- The answer is `200` and `OK`. A token that doesn't exist, a path other than the three, and a token of the wrong shape all get `404 Not found` in the same words. Calls are limited to 120 a minute from one address.
|
||||||
|
- A job is **late** when its period and the grace have passed since its last call (or since the heartbeat was made, if it has never called). That is a warning, announced once and again as cleared when the next call comes;
|
||||||
|
a call to `/fail` is critical and stays until a call that works. A heartbeat that is switched off still answers `200`.
|
||||||
|
- Both are alerts of the *Automation* category, so notification routing, tags, quiet hours and escalation apply, and maintenance on the server a heartbeat is linked to silences them.
|
||||||
|
|
||||||
|
## Calendar feed
|
||||||
|
|
||||||
|
An administrator makes a **calendar feed** under **Settings → Share links**: an address ending in `.ics` for a calendar app to subscribe to (Google Calendar, Apple Calendar, Outlook and Thunderbird all take "add a calendar from a URL").
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/public/calendar/<token>.ics
|
||||||
|
```
|
||||||
|
|
||||||
|
- Like any share link, the token is the whole of the access; only its hash is kept, and the address is shown once, when the link is made. It can have an end date and can be removed at any time.
|
||||||
|
A link that doesn't work — never made, ended, removed, or one that opens a wall or a status page instead — gets `404` in the same words. Limited to 60 requests a minute from one address.
|
||||||
|
- It carries what **Coming up** lists, for the next year: all-day events for secrets, domains, certificates, Tailscale keys, warranties, end of support and restore tests, and events with a start and an end for planned and
|
||||||
|
recurring maintenance (every occurrence, which the page folds into one line). Updates that are waiting have no day and are not in it. Something that expired in the last week is kept; older is left out.
|
||||||
|
- With **kinds only**, an event says only what kind of thing it is ("A secret expires") and has no description, so it can go on a shared calendar. With **names** it carries the name and a link back to the app.
|
||||||
|
An event's `UID` is a hash of what it is and its date, the same on every fetch.
|
||||||
|
- The feed asks to be refreshed every six hours (`REFRESH-INTERVAL`); a calendar app decides for itself how often it does.
|
||||||
+36
-5
@@ -53,6 +53,7 @@ erDiagram
|
|||||||
users ||--o{ audit_log : "actor (set null)"
|
users ||--o{ audit_log : "actor (set null)"
|
||||||
users ||--o{ push_subscriptions : "cascade"
|
users ||--o{ push_subscriptions : "cascade"
|
||||||
endpoint_checks ||--o{ endpoint_results : "cascade"
|
endpoint_checks ||--o{ endpoint_results : "cascade"
|
||||||
|
heartbeats ||--o{ heartbeat_pings : "cascade"
|
||||||
integration_credentials ||--o{ integrations : "credential (set null)"
|
integration_credentials ||--o{ integrations : "credential (set null)"
|
||||||
integration_credentials ||--o{ dns_providers : "credential (set null)"
|
integration_credentials ||--o{ dns_providers : "credential (set null)"
|
||||||
dns_providers ||--o{ dns_zones_cache : "cascade"
|
dns_providers ||--o{ dns_zones_cache : "cascade"
|
||||||
@@ -66,9 +67,9 @@ erDiagram
|
|||||||
integrations ||--o{ alert_tickets : "osTicket used (set null)"
|
integrations ||--o{ alert_tickets : "osTicket used (set null)"
|
||||||
```
|
```
|
||||||
|
|
||||||
Sixteen tables stand alone, with no foreign keys: `settings`, `secrets`,
|
Seventeen tables stand alone, with no foreign keys: `settings`, `secrets`,
|
||||||
`ipam_entries`, `domains`, `diag_log`, `notification_queue`,
|
`ipam_entries`, `domains`, `diag_log`, `notification_queue`,
|
||||||
`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`, `dependencies`, `endpoint_checks`, `restore_tests`. `maintenance_windows` and `maintenance_aliases` also have no
|
`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`, `metrics_tokens`, `event_webhooks`, `alert_incidents`, `dependencies`, `endpoint_checks`, `restore_tests`, `heartbeats`. `maintenance_windows` and `maintenance_aliases` also have no
|
||||||
foreign key — see [soft references](#soft-references-not-foreign-keys).
|
foreign key — see [soft references](#soft-references-not-foreign-keys).
|
||||||
|
|
||||||
## Tables
|
## Tables
|
||||||
@@ -253,7 +254,7 @@ tasks added by hand.
|
|||||||
#### `server_links`
|
#### `server_links`
|
||||||
|
|
||||||
Admin-page bookmarks for a server (Dockge, Webmin, Cockpit, …). Shown on the
|
Admin-page bookmarks for a server (Dockge, Webmin, Cockpit, …). Shown on the
|
||||||
server's page and summarised under Operations → Admin Links.
|
server's page and summarised under Infrastructure → Admin Links.
|
||||||
|
|
||||||
| Column | Type | Notes |
|
| Column | Type | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@@ -660,6 +661,36 @@ One row per answer, kept for two weeks: uptime, average answer time and the run
|
|||||||
| `ok` | bool NOT NULL | |
|
| `ok` | bool NOT NULL | |
|
||||||
| `ms`, `message` | integer, text | |
|
| `ms`, `message` | integer, text | |
|
||||||
|
|
||||||
|
#### `heartbeats`
|
||||||
|
|
||||||
|
A job that reports in: it calls `/api/heartbeat/<token>` when it finishes, and the app speaks up when the call doesn't come in time (services/heartbeats.ts).
|
||||||
|
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | integer PK | |
|
||||||
|
| `name` | text NOT NULL | |
|
||||||
|
| `token` | text NOT NULL, unique | The secret part of the address (`hb_…`). Kept as it is, not hashed, because the address has to be shown again; all it allows is "I'm alive". |
|
||||||
|
| `period_minutes`, `grace_minutes` | integer NOT NULL | How often it should report, and how much later is still fine (default 10). |
|
||||||
|
| `server_id` | integer | Soft reference to the server it runs on, so its maintenance silences the alerts. |
|
||||||
|
| `enabled` | bool NOT NULL, default true | A switched-off heartbeat still takes calls. |
|
||||||
|
| `notes` | text | |
|
||||||
|
| `last_ping_at`, `last_status` | text | When the last call that ended a run came in, and `ok` or `fail`. |
|
||||||
|
| `last_message`, `last_duration_ms` | text, integer | The note sent with it, and how long the run took when it said it had started. |
|
||||||
|
| `last_start_at` | text | ISO; the last `/start` call. |
|
||||||
|
| `ping_count`, `created_at` | integer, text | |
|
||||||
|
|
||||||
|
#### `heartbeat_pings`
|
||||||
|
|
||||||
|
The recent calls to each heartbeat, for its history; kept two weeks.
|
||||||
|
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | integer PK | |
|
||||||
|
| `heartbeat_id` | integer NOT NULL | → `heartbeats.id`, deleted with it (indexed with `at`). |
|
||||||
|
| `at` | text NOT NULL | ISO. |
|
||||||
|
| `kind` | text NOT NULL | `ok`, `fail` or `start`. |
|
||||||
|
| `duration_ms`, `message` | integer, text | |
|
||||||
|
|
||||||
#### `restore_tests`
|
#### `restore_tests`
|
||||||
|
|
||||||
Each time somebody tried to restore a backup, and how it went. History is kept; the newest per subject is what counts.
|
Each time somebody tried to restore a backup, and how it went. History is kept; the newest per subject is what counts.
|
||||||
@@ -716,7 +747,7 @@ links): the wall display, or a status page. Only a hash of the secret is stored.
|
|||||||
| `token_hash` | text NOT NULL, unique | SHA-256 of the secret in the address; the secret itself is shown once and never stored. |
|
| `token_hash` | text NOT NULL, unique | SHA-256 of the secret in the address; the secret itself is shown once and never stored. |
|
||||||
| `token_prefix` | text NOT NULL | The first characters of the secret, to tell links apart in the list. |
|
| `token_prefix` | text NOT NULL | The first characters of the secret, to tell links apart in the list. |
|
||||||
| `detail` | text NOT NULL, default `summary` | For a wall: `summary` (numbers only) or `names` (also the names of what is wrong, in maintenance, and the integrations). Not used by a status page. |
|
| `detail` | text NOT NULL, default `summary` | For a wall: `summary` (numbers only) or `names` (also the names of what is wrong, in maintenance, and the integrations). Not used by a status page. |
|
||||||
| `kind` | text NOT NULL, default `wall` | `wall` or `status`. A status page's address doesn't open the wall, and the other way round. |
|
| `kind` | text NOT NULL, default `wall` | `wall`, `status` or `calendar` (an iCalendar feed of what is coming up; `detail` says whether it carries names). Each kind's address opens only that kind. |
|
||||||
| `title`, `message` | text | A status page's heading and its optional message; null for a wall. |
|
| `title`, `message` | text | A status page's heading and its optional message; null for a wall. |
|
||||||
| `created_by` | text | The administrator's name at the time. |
|
| `created_by` | text | The administrator's name at the time. |
|
||||||
| `created_at` | text NOT NULL | |
|
| `created_at` | text NOT NULL | |
|
||||||
@@ -820,7 +851,7 @@ a stale value is possible and the app treats it as "nothing there":
|
|||||||
|
|
||||||
- `maintenance_windows.target_id` (→ a server, integration or DNS provider, by `target_type`)
|
- `maintenance_windows.target_id` (→ a server, integration or DNS provider, by `target_type`)
|
||||||
- `maintenance_aliases.target_id` (the same, by `target_type`)
|
- `maintenance_aliases.target_id` (the same, by `target_type`)
|
||||||
- `endpoint_checks.server_id` (→ a server) and `restore_tests.subject_key` (→ a Proxmox guest, by integration and vmid)
|
- `endpoint_checks.server_id` and `heartbeats.server_id` (→ a server) and `restore_tests.subject_key` (→ a Proxmox guest, by integration and vmid)
|
||||||
- `dependencies.from_id` and `to_id` (→ a server or integration, by their `_type`)
|
- `dependencies.from_id` and `to_id` (→ a server or integration, by their `_type`)
|
||||||
- `audit_log.target_id`
|
- `audit_log.target_id`
|
||||||
- `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider)
|
- `dns_records_cache.zone_id` (→ `dns_zones_cache.zone_id`, same provider)
|
||||||
|
|||||||
+7
-1
@@ -177,7 +177,7 @@ what it would say right now and *Send it now* sends it at once (through the chan
|
|||||||
|
|
||||||
## The Alerts page
|
## The Alerts page
|
||||||
|
|
||||||
**Operations → Alerts** shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and
|
**Operations → Alerts** (the *Now* view; *History* lists every incident that has come and gone) shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and
|
||||||
export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs
|
export. It uses the same checks as the notifications above, so a problem appears there for exactly the reason it would be notified, but it differs
|
||||||
in three ways:
|
in three ways:
|
||||||
|
|
||||||
@@ -234,3 +234,9 @@ the app alerts on something:
|
|||||||
on their own (the Diagnostic Log's failures are what feed the
|
on their own (the Diagnostic Log's failures are what feed the
|
||||||
integration-down alert above, but reading the log itself never
|
integration-down alert above, but reading the log itself never
|
||||||
triggers anything).
|
triggers anything).
|
||||||
|
|
||||||
|
## Heartbeats
|
||||||
|
|
||||||
|
A **heartbeat** that doesn't hear from its job in time is announced like any other problem — under the *server offline, disk full …* switch (`healthAlerts`) — once when it is late, with a
|
||||||
|
recovery message when the next call comes in. A job that calls its `/fail` address is announced the same way, as critical, at once. For routing both are *Automation*, and carry the tags of
|
||||||
|
the server the heartbeat is linked to; maintenance on that server holds them back, and they alert after it if they are still there. Quiet hours, escalation and push preferences apply as to the rest.
|
||||||
@@ -16,7 +16,7 @@ Authentik (OIDC), with local admin/operator/viewer roles.
|
|||||||
|
|
||||||
All modules from the original plan are built:
|
All modules from the original plan are built:
|
||||||
|
|
||||||
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
|
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Network, Monitoring, Automation, Operations, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
|
||||||
- Authentik OIDC login, roles (first user to sign in becomes admin), audit log
|
- Authentik OIDC login, roles (first user to sign in becomes admin), audit log
|
||||||
(changes made in the app, sign-ins and sign-outs with the IP they came from, new accounts, and the log's own
|
(changes made in the app, sign-ins and sign-outs with the IP they came from, new accounts, and the log's own
|
||||||
automatic trimming — attributed to "system"; settings changes show what changed, but never credentials)
|
automatic trimming — attributed to "system"; settings changes show what changed, but never credentials)
|
||||||
@@ -95,7 +95,7 @@ All modules from the original plan are built:
|
|||||||
start/stop/restart buttons right on the detail page. The detail page also has an **Admin Links** section
|
start/stop/restart buttons right on the detail page. The detail page also has an **Admin Links** section
|
||||||
(operator/admin to add/edit/remove) for bookmarking that server's own
|
(operator/admin to add/edit/remove) for bookmarking that server's own
|
||||||
admin UIs — Dockge, Webmin, Cockpit, Portainer, or anything else reachable
|
admin UIs — Dockge, Webmin, Cockpit, Portainer, or anything else reachable
|
||||||
by URL. **Operations → Admin Links** summarizes every server's admin links
|
by URL. **Infrastructure → Admin Links** summarizes every server's admin links
|
||||||
in one sortable, searchable table (server, label, URL, an "Open" link, and
|
in one sortable, searchable table (server, label, URL, an "Open" link, and
|
||||||
the same add/edit/delete as the per-server section — adding one here just
|
the same add/edit/delete as the per-server section — adding one here just
|
||||||
asks which server it belongs to), so finding or managing one doesn't mean
|
asks which server it belongs to), so finding or managing one doesn't mean
|
||||||
@@ -220,7 +220,7 @@ offered as one-click suggestions when tagging), give any tag a colour of your ch
|
|||||||
that already exists merges the two, and both rename and delete rewrite every server
|
that already exists merges the two, and both rename and delete rewrite every server
|
||||||
that carries the tag.
|
that carries the tag.
|
||||||
|
|
||||||
**Name generator** — Operations → Generator suggests server names (and usernames and
|
**Name generator** — Administration → Generator suggests server names (and usernames and
|
||||||
passwords, which are made in your browser and never stored). Server names are picked from
|
passwords, which are made in your browser and never stored). Server names are picked from
|
||||||
name lists: Swedish girl and boy names, Disney and Pixar characters, Norse mythology and
|
name lists: Swedish girl and boy names, Disney and Pixar characters, Norse mythology and
|
||||||
Astrid Lindgren to start with, or "Mixed" for all of them together. A name already used
|
Astrid Lindgren to start with, or "Mixed" for all of them together. A name already used
|
||||||
@@ -507,7 +507,7 @@ the alert back among the unattended; it goes by itself once the alert has been g
|
|||||||
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
|
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
|
||||||
server's history.
|
server's history.
|
||||||
|
|
||||||
**Incidents** (Operations → Incidents). Each time an alert is raised and later goes away is recorded as an
|
**Incidents** (Operations → Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an
|
||||||
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
|
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
|
||||||
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
|
after the alert is gone. Open one for the whole story in order: the alert, who acknowledged it and why, the
|
||||||
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
|
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
|
||||||
@@ -561,7 +561,7 @@ text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a
|
|||||||
of commands, backticks inline code; links become clickable, and nothing typed in a
|
of commands, backticks inline code; links become clickable, and nothing typed in a
|
||||||
note can run as HTML.
|
note can run as HTML.
|
||||||
|
|
||||||
**Endpoint checks** (Network → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
|
**Endpoint checks** (Monitoring → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
|
||||||
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
|
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
|
||||||
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
|
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
|
||||||
also want a word in the page; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
|
also want a word in the page; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
|
||||||
@@ -571,6 +571,32 @@ server, so maintenance on it silences the check's alerts and the server's tags c
|
|||||||
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
|
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
|
||||||
dashboard and in `/metrics`. Results are kept for two weeks.
|
dashboard and in `/metrics`. Results are kept for two weeks.
|
||||||
|
|
||||||
|
**Suggested checks** (on the Endpoints page, administrators). From what the app already knows it offers checks you could add with a click: a **web port a server's agent
|
||||||
|
sees open** (80, 443, 8080, 8443 — when a server has both the plain and the secure one, the secure one, since it also reads the certificate), under the name DNS gives that
|
||||||
|
server if there is one (an A or AAAA record with one of its addresses, or an alias to it), otherwise its own hostname or address; and a **secret that already reads a
|
||||||
|
certificate**, since a check also hears when the service stops answering. Each is linked to its server. Something a check already covers is never offered, **Not for me**
|
||||||
|
stops one being offered (and they can be brought back), and nothing is asked until you add it. Uptime Kuma's monitors are not offered: they are already being watched, and
|
||||||
|
the adapter only keeps a host and port for them, not the address.
|
||||||
|
|
||||||
|
**Heartbeats** (Monitoring → Heartbeats). "Did my 3 a.m. job actually run?" An administrator makes a heartbeat for a job — how often it should report, and how late is
|
||||||
|
still fine — and gets an address with a long random token in it. The job calls it when it finishes (`curl -fsS -m 10 -o /dev/null <address>` after the backup in cron);
|
||||||
|
the app writes that down and shows when the next call is due, the run times and the last runs as a row of ticks. A call that does **not come in time** is a *warning*
|
||||||
|
alert and one notification (the same bookkeeping as everything else: said once, again as cleared when the call comes); a job that calls `<address>/fail` — it ran and went
|
||||||
|
wrong — is a *critical* alert at once, with what it said. `/start` before the job lets the finishing call say how long it took; anything sent with a call
|
||||||
|
(`--data-raw "3.2 GB copied"`, or `?msg=`) is kept as a note, up to 200 characters. A heartbeat can be linked to the server the job runs on, so maintenance there silences
|
||||||
|
it and the server's tags route it. Everyone can look; only administrators see the addresses, make, change and remove heartbeats, or give one a **new address** if it got out.
|
||||||
|
The address needs no sign-in (it is a token, like an agent's), and all it can do is say "I'm alive": it can't read anything. Calls are limited per address, and an
|
||||||
|
unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [API.md](API.md).
|
||||||
|
|
||||||
|
**Coming up** (Operations → Coming up, and a card on the dashboard). Everything with a date ahead in one list, nearest first: secrets, domains and certificates that
|
||||||
|
expire (including the ones endpoint checks see), Tailscale keys that are expiring, hardware warranties and end of support, planned and **recurring** maintenance, restore
|
||||||
|
tests that are due (when a time limit is set), and — as one line — updates that have been waiting. What has already passed in the last 30 days is shown as past, under
|
||||||
|
*Needs attention now*. Narrow it by kind, server and how far ahead (up to a year); the filters are in the address, so a view can be linked. A schedule that repeats weekly is one line (its next time) unless
|
||||||
|
you ask for every one. An administrator can make a **calendar feed** (Settings → Share links → *A calendar feed*): an address for a phone's calendar to subscribe to, refreshed
|
||||||
|
every few hours, with each day as an all-day event and maintenance with its start and end. A feed made with *kinds only* says only "A secret expires", with no names, so it is fine on a shared
|
||||||
|
calendar; *with names* also gives the name and a link back. Like any share link the address is shown once and can be removed. Tailscale keys are asked for at most every
|
||||||
|
ten minutes, and only those the integration reports as expiring soon are listed.
|
||||||
|
|
||||||
**Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed
|
**Backup coverage** (Infrastructure → Backup coverage) answers "what isn't backed
|
||||||
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
|
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
|
||||||
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox
|
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox
|
||||||
|
|||||||
@@ -106,7 +106,7 @@ even further, to admin only:
|
|||||||
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
|
- **Backup coverage** is a read-only page for everyone. On **Updates**, everyone sees what is waiting and how update runs
|
||||||
are going; starting an update run (**Update…**) or stopping one needs operator.
|
are going; starting an update run (**Update…**) or stopping one needs operator.
|
||||||
- **Acknowledging an alert, handing it over, and taking it back** need operator (the person it is handed to must be an operator or
|
- **Acknowledging an alert, handing it over, and taking it back** need operator (the person it is handed to must be an operator or
|
||||||
administrator); everyone sees who has what. **Incidents** are readable by everyone signed in; writing a note needs operator, and a note
|
administrator); everyone sees who has what. **Incidents** (Alerts → History) are readable by everyone signed in; writing a note needs operator, and a note
|
||||||
can be removed by its author or an administrator.
|
can be removed by its author or an administrator.
|
||||||
- **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are
|
- **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are
|
||||||
administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials
|
administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials
|
||||||
@@ -117,6 +117,8 @@ even further, to admin only:
|
|||||||
on the Servers page is admin-only.
|
on the Servers page is admin-only.
|
||||||
- **Profile pictures** are shown to everyone signed in (`/api/avatars/<id>`, the same people whose names everyone already sees); which sources are allowed is under
|
- **Profile pictures** are shown to everyone signed in (`/api/avatars/<id>`, the same people whose names everyone already sees); which sources are allowed is under
|
||||||
Settings → Sign-in, so admin-only.
|
Settings → Sign-in, so admin-only.
|
||||||
|
- **Heartbeats** and the **Coming up** list are visible to everyone signed in; only administrators see a heartbeat's address, and make, change, remove or give a new address to one, and make
|
||||||
|
the calendar feed of Coming up (a share link). **Suggested checks** are administrators' like the checks themselves. A job's call to its address needs no role: the token is the access.
|
||||||
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
|
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
|
||||||
Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
|
Grafana dashboard download** are all under Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
|
||||||
- Everything under **Settings** (notification channels, badge colors,
|
- Everything under **Settings** (notification channels, badge colors,
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
CREATE TABLE `heartbeat_pings` (
|
||||||
|
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
|
||||||
|
`heartbeat_id` integer NOT NULL,
|
||||||
|
`at` text NOT NULL,
|
||||||
|
`kind` text NOT NULL,
|
||||||
|
`duration_ms` integer,
|
||||||
|
`message` text,
|
||||||
|
FOREIGN KEY (`heartbeat_id`) REFERENCES `heartbeats`(`id`) ON UPDATE no action ON DELETE cascade
|
||||||
|
);
|
||||||
|
--> statement-breakpoint
|
||||||
|
CREATE INDEX `heartbeat_pings_idx` ON `heartbeat_pings` (`heartbeat_id`,`at`);--> statement-breakpoint
|
||||||
|
CREATE TABLE `heartbeats` (
|
||||||
|
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
|
||||||
|
`name` text NOT NULL,
|
||||||
|
`token` text NOT NULL,
|
||||||
|
`period_minutes` integer NOT NULL,
|
||||||
|
`grace_minutes` integer DEFAULT 10 NOT NULL,
|
||||||
|
`server_id` integer,
|
||||||
|
`enabled` integer DEFAULT true NOT NULL,
|
||||||
|
`notes` text,
|
||||||
|
`last_ping_at` text,
|
||||||
|
`last_status` text,
|
||||||
|
`last_message` text,
|
||||||
|
`last_duration_ms` integer,
|
||||||
|
`last_start_at` text,
|
||||||
|
`ping_count` integer DEFAULT 0 NOT NULL,
|
||||||
|
`created_at` text DEFAULT (current_timestamp) NOT NULL
|
||||||
|
);
|
||||||
|
--> statement-breakpoint
|
||||||
|
CREATE UNIQUE INDEX `heartbeats_token_unique` ON `heartbeats` (`token`);
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -295,6 +295,13 @@
|
|||||||
"when": 1792359507903,
|
"when": 1792359507903,
|
||||||
"tag": "0041_open_silvermane",
|
"tag": "0041_open_silvermane",
|
||||||
"breakpoints": true
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 42,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1792361094159,
|
||||||
|
"tag": "0042_silly_omega_red",
|
||||||
|
"breakpoints": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -48,6 +48,10 @@ import { mapRouter } from "./routes/map.js";
|
|||||||
import { generatorRouter } from "./routes/generator.js";
|
import { generatorRouter } from "./routes/generator.js";
|
||||||
import { alertTicketsRouter } from "./routes/alertTickets.js";
|
import { alertTicketsRouter } from "./routes/alertTickets.js";
|
||||||
import { avatarsRouter } from "./routes/avatars.js";
|
import { avatarsRouter } from "./routes/avatars.js";
|
||||||
|
import { heartbeatsRouter } from "./routes/heartbeats.js";
|
||||||
|
import { agendaRouter } from "./routes/agenda.js";
|
||||||
|
import { publicCalendarRouter } from "./routes/publicCalendar.js";
|
||||||
|
import { heartbeatPingRouter } from "./routes/heartbeatPing.js";
|
||||||
import { apiErrorHandler } from "./utils/apiErrorHandler.js";
|
import { apiErrorHandler } from "./utils/apiErrorHandler.js";
|
||||||
import { securityHeaders } from "./utils/securityHeaders.js";
|
import { securityHeaders } from "./utils/securityHeaders.js";
|
||||||
import { sameOriginGuard } from "./utils/sameOriginGuard.js";
|
import { sameOriginGuard } from "./utils/sameOriginGuard.js";
|
||||||
@@ -72,6 +76,8 @@ export function createApp() {
|
|||||||
// The agent endpoint comes first: it reads its own (larger) body, only after checking the token, and an agent has no use for a
|
// The agent endpoint comes first: it reads its own (larger) body, only after checking the token, and an agent has no use for a
|
||||||
// session. It is also the one place a request isn't from a browser, so it isn't subject to the same-origin check below.
|
// session. It is also the one place a request isn't from a browser, so it isn't subject to the same-origin check below.
|
||||||
app.use("/api/agent/report", agentReportRouter);
|
app.use("/api/agent/report", agentReportRouter);
|
||||||
|
// A job reporting that it ran: a token in the address, no session, and no browser to be checked for where it came from.
|
||||||
|
app.use("/api/heartbeat", heartbeatPingRouter);
|
||||||
// Everything after this is reached with a session cookie, so a request that came from another origin is refused here.
|
// Everything after this is reached with a session cookie, so a request that came from another origin is refused here.
|
||||||
app.use(sameOriginGuard);
|
app.use(sameOriginGuard);
|
||||||
app.use(express.json());
|
app.use(express.json());
|
||||||
@@ -105,6 +111,8 @@ export function createApp() {
|
|||||||
app.use("/auth", authRouter);
|
app.use("/auth", authRouter);
|
||||||
app.use("/api/me", meRouter);
|
app.use("/api/me", meRouter);
|
||||||
app.use("/api/avatars", avatarsRouter);
|
app.use("/api/avatars", avatarsRouter);
|
||||||
|
app.use("/api/heartbeats", heartbeatsRouter);
|
||||||
|
app.use("/api/agenda", agendaRouter);
|
||||||
app.use("/api/uptime", uptimeRouter);
|
app.use("/api/uptime", uptimeRouter);
|
||||||
app.use("/api/share-links", shareLinksRouter);
|
app.use("/api/share-links", shareLinksRouter);
|
||||||
app.use("/api/runbooks", runbooksRouter);
|
app.use("/api/runbooks", runbooksRouter);
|
||||||
@@ -123,6 +131,7 @@ export function createApp() {
|
|||||||
app.use("/metrics", metricsRouter);
|
app.use("/metrics", metricsRouter);
|
||||||
app.use("/api/public/wall", publicWallRouter);
|
app.use("/api/public/wall", publicWallRouter);
|
||||||
app.use("/api/public/status", publicStatusRouter);
|
app.use("/api/public/status", publicStatusRouter);
|
||||||
|
app.use("/api/public/calendar", publicCalendarRouter);
|
||||||
app.use("/api/users", usersRouter);
|
app.use("/api/users", usersRouter);
|
||||||
app.use("/api/audit-log", auditLogRouter);
|
app.use("/api/audit-log", auditLogRouter);
|
||||||
app.use("/api/diag-log", diagLogRouter);
|
app.use("/api/diag-log", diagLogRouter);
|
||||||
|
|||||||
+41
-2
@@ -192,6 +192,44 @@ export const restoreTests = sqliteTable(
|
|||||||
(t) => [index("restore_tests_subject_idx").on(t.subjectKey, t.testedAt)],
|
(t) => [index("restore_tests_subject_idx").on(t.subjectKey, t.testedAt)],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// A job that reports in: a cron job or backup script calls its address when it finishes, and the app speaks up when the call doesn't come in time —
|
||||||
|
// see services/heartbeats.ts. The token is part of the address (/api/heartbeat/<token>) and is all the access a ping needs.
|
||||||
|
export const heartbeats = sqliteTable("heartbeats", {
|
||||||
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
|
name: text("name").notNull(),
|
||||||
|
token: text("token").notNull().unique(),
|
||||||
|
periodMinutes: integer("period_minutes").notNull(), // how often it should report
|
||||||
|
graceMinutes: integer("grace_minutes").notNull().default(10), // how much later than that is still fine
|
||||||
|
serverId: integer("server_id"), // soft reference: the server it runs on, so its maintenance silences the alerts
|
||||||
|
enabled: integer("enabled", { mode: "boolean" }).notNull().default(true),
|
||||||
|
notes: text("notes"),
|
||||||
|
lastPingAt: text("last_ping_at"), // ISO; the last report of an end (success or failure)
|
||||||
|
lastStatus: text("last_status"), // ok | fail
|
||||||
|
lastMessage: text("last_message"),
|
||||||
|
lastDurationMs: integer("last_duration_ms"), // when it said it had started first
|
||||||
|
lastStartAt: text("last_start_at"), // ISO; the last "started" call
|
||||||
|
pingCount: integer("ping_count").notNull().default(0),
|
||||||
|
createdAt: text("created_at")
|
||||||
|
.notNull()
|
||||||
|
.default(sql`(current_timestamp)`),
|
||||||
|
});
|
||||||
|
|
||||||
|
// The recent calls to each heartbeat, for its history; kept two weeks.
|
||||||
|
export const heartbeatPings = sqliteTable(
|
||||||
|
"heartbeat_pings",
|
||||||
|
{
|
||||||
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
|
heartbeatId: integer("heartbeat_id")
|
||||||
|
.notNull()
|
||||||
|
.references(() => heartbeats.id, { onDelete: "cascade" }),
|
||||||
|
at: text("at").notNull(), // ISO
|
||||||
|
kind: text("kind").notNull(), // ok | fail | start
|
||||||
|
durationMs: integer("duration_ms"),
|
||||||
|
message: text("message"),
|
||||||
|
},
|
||||||
|
(t) => [index("heartbeat_pings_idx").on(t.heartbeatId, t.at)],
|
||||||
|
);
|
||||||
|
|
||||||
// "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose
|
// "A needs B", for things that can be down: a server or an integration each. Polymorphic like maintenance_windows (no foreign key); an edge whose
|
||||||
// end was deleted is ignored. When B is down, A's alert is folded under B's — see services/dependencies.ts.
|
// end was deleted is ignored. When B is down, A's alert is folded under B's — see services/dependencies.ts.
|
||||||
export const dependencies = sqliteTable(
|
export const dependencies = sqliteTable(
|
||||||
@@ -628,8 +666,9 @@ export const alertTickets = sqliteTable(
|
|||||||
export const shareLinkDetails = ["summary", "names"] as const;
|
export const shareLinkDetails = ["summary", "names"] as const;
|
||||||
export type ShareLinkDetail = (typeof shareLinkDetails)[number];
|
export type ShareLinkDetail = (typeof shareLinkDetails)[number];
|
||||||
|
|
||||||
// What a link opens: the wall display (how things are, for a TV), or a status page (is each of the things I chose working, for people at home).
|
// What a link opens: the wall display (how things are, for a TV), a status page (is each of the things I chose working, for people at home), or
|
||||||
export const shareLinkKinds = ["wall", "status"] as const;
|
// a calendar feed (what is coming up, for a phone's calendar to subscribe to).
|
||||||
|
export const shareLinkKinds = ["wall", "status", "calendar"] as const;
|
||||||
export type ShareLinkKind = (typeof shareLinkKinds)[number];
|
export type ShareLinkKind = (typeof shareLinkKinds)[number];
|
||||||
|
|
||||||
// A link to the wall display that needs no sign-in, for a TV in the hall or someone who only wants to know if things are up. Only a hash
|
// A link to the wall display that needs no sign-in, for a TV in the hall or someone who only wants to know if things are up. Only a hash
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
import { Router } from "express";
|
||||||
|
import { requireAuth } from "../auth/middleware.js";
|
||||||
|
import { buildAgenda, DEFAULT_DAYS, MAX_DAYS } from "../services/agenda.js";
|
||||||
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
|
export const agendaRouter = Router();
|
||||||
|
|
||||||
|
agendaRouter.use(requireAuth);
|
||||||
|
|
||||||
|
// Everything with a date ahead, for everyone signed in: it is only what the pages it comes from already show them. `days` is how far ahead.
|
||||||
|
agendaRouter.get("/", asyncHandler(async (req, res) => {
|
||||||
|
const asked = Number(req.query.days);
|
||||||
|
const days = Number.isInteger(asked) && asked >= 1 && asked <= MAX_DAYS ? asked : DEFAULT_DAYS;
|
||||||
|
res.json(await buildAgenda(new Date(), days));
|
||||||
|
}));
|
||||||
@@ -7,6 +7,7 @@ import { requireAuth, requireRole } from "../auth/middleware.js";
|
|||||||
import { recordAudit } from "../services/audit.js";
|
import { recordAudit } from "../services/audit.js";
|
||||||
import { invalidateAlertsCache } from "../services/alerts.js";
|
import { invalidateAlertsCache } from "../services/alerts.js";
|
||||||
import { CHECK_KINDS, isStatusSpec, loadStats, runCheck } from "../services/endpointChecks.js";
|
import { CHECK_KINDS, isStatusSpec, loadStats, runCheck } from "../services/endpointChecks.js";
|
||||||
|
import { loadDismissed, loadSuggestions, saveDismissed } from "../services/endpointSuggestions.js";
|
||||||
import { asyncHandler } from "../utils/asyncHandler.js";
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
export const endpointsRouter = Router();
|
export const endpointsRouter = Router();
|
||||||
@@ -84,8 +85,54 @@ async function serverExists(id: number | null | undefined) {
|
|||||||
return id == null || (await db.select({ id: servers.id }).from(servers).where(eq(servers.id, id))).length > 0;
|
return id == null || (await db.select({ id: servers.id }).from(servers).where(eq(servers.id, id))).length > 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Checks worth setting up, from what the app already knows: a web port a server is listening on, a secret that reads a certificate. Reads what is
|
||||||
|
// stored, asks nobody. Administrators only, like the checks themselves.
|
||||||
|
endpointsRouter.get("/suggestions", requireRole("admin"), asyncHandler(async (_req, res) => {
|
||||||
|
res.json(await loadSuggestions());
|
||||||
|
}));
|
||||||
|
|
||||||
|
const keysSchema = z.object({ keys: z.array(z.string().min(1).max(600)).min(1).max(MAX_CHECKS) });
|
||||||
|
|
||||||
|
// Makes the checks for the suggestions named. What each one is comes from the server's own list, not from the request, so a request can only
|
||||||
|
// ever add what was offered.
|
||||||
|
endpointsRouter.post("/suggestions/add", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const parsed = keysSchema.safeParse(req.body);
|
||||||
|
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Name the suggestions to add." });
|
||||||
|
const offered = new Map((await loadSuggestions()).suggestions.map((s) => [s.key, s]));
|
||||||
|
let have = (await db.select({ id: endpointChecks.id }).from(endpointChecks)).length;
|
||||||
|
const added: { id: number; name: string }[] = [];
|
||||||
|
for (const key of new Set(parsed.data.keys)) {
|
||||||
|
const s = offered.get(key);
|
||||||
|
if (!s) continue;
|
||||||
|
if (have >= MAX_CHECKS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_CHECKS} checks.`, added });
|
||||||
|
const [row] = await db
|
||||||
|
.insert(endpointChecks)
|
||||||
|
.values(valuesOf({ name: s.name, kind: s.kind, target: s.target, port: s.port, serverId: s.serverId }))
|
||||||
|
.returning();
|
||||||
|
have++;
|
||||||
|
added.push({ id: row.id, name: row.name });
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "endpoint", action: "create", targetType: "endpoint", targetId: row.id, detail: { name: row.name, kind: row.kind, target: row.target, suggested: true } });
|
||||||
|
}
|
||||||
|
res.status(201).json({ added });
|
||||||
|
}));
|
||||||
|
|
||||||
|
// "Not for me": what is dismissed stops being offered, and can be brought back.
|
||||||
|
endpointsRouter.post("/suggestions/dismiss", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const parsed = keysSchema.safeParse(req.body);
|
||||||
|
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Name the suggestions to dismiss." });
|
||||||
|
const keys = await loadDismissed();
|
||||||
|
for (const k of parsed.data.keys) keys.add(k);
|
||||||
|
await saveDismissed(keys);
|
||||||
|
res.json(await loadSuggestions());
|
||||||
|
}));
|
||||||
|
|
||||||
|
endpointsRouter.post("/suggestions/restore", requireRole("admin"), asyncHandler(async (_req, res) => {
|
||||||
|
await saveDismissed(new Set());
|
||||||
|
res.json(await loadSuggestions());
|
||||||
|
}));
|
||||||
|
|
||||||
// Asking things on the network is an administrator's to set up.
|
// Asking things on the network is an administrator's to set up.
|
||||||
endpointsRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => {
|
endpointsRouter.post("/",requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
const parsed = checkSchema.safeParse(req.body);
|
const parsed = checkSchema.safeParse(req.body);
|
||||||
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "That isn't a check.", details: parsed.error.flatten() });
|
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: parsed.error.issues[0]?.message ?? "That isn't a check.", details: parsed.error.flatten() });
|
||||||
if ((await db.select({ id: endpointChecks.id }).from(endpointChecks)).length >= MAX_CHECKS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_CHECKS} checks.` });
|
if ((await db.select({ id: endpointChecks.id }).from(endpointChecks)).length >= MAX_CHECKS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_CHECKS} checks.` });
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
import express, { Router } from "express";
|
||||||
|
import { cleanMessage, recordPing, type PingKind } from "../services/heartbeats.js";
|
||||||
|
import { rateLimit } from "../utils/rateLimit.js";
|
||||||
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The address a job calls: /api/heartbeat/<token>, /api/heartbeat/<token>/start, /api/heartbeat/<token>/fail. No session and no sign-in — the
|
||||||
|
* token is the whole of the access, and all it allows is to say "I'm alive". Mounted ahead of the browser-only middleware (a script isn't a
|
||||||
|
* browser and has no cookie), so it answers GET, HEAD, POST and PUT alike: whatever `curl` or `wget` happens to send.
|
||||||
|
*
|
||||||
|
* It answers a token that doesn't exist, and a path that isn't one of the three, in the same two words, and is limited per address.
|
||||||
|
*/
|
||||||
|
export const heartbeatPingRouter = Router();
|
||||||
|
|
||||||
|
const limiter = rateLimit({ windowMs: 60_000, max: 120, message: "Too many requests. Wait a minute." });
|
||||||
|
// Whatever a script sends as the body ("3.2 GB copied") is a note; it is cut down to a line.
|
||||||
|
const body = express.text({ type: () => true, limit: "100kb" });
|
||||||
|
|
||||||
|
const KINDS: Record<string, PingKind> = { "": "ok", ok: "ok", start: "start", fail: "fail" };
|
||||||
|
|
||||||
|
heartbeatPingRouter.all("/:token/:kind?", limiter, body, asyncHandler(async (req, res) => {
|
||||||
|
res.setHeader("Cache-Control", "no-store");
|
||||||
|
res.setHeader("X-Robots-Tag", "noindex, nofollow");
|
||||||
|
if (!["GET", "HEAD", "POST", "PUT"].includes(req.method)) return res.status(405).type("text/plain").send("Method not allowed");
|
||||||
|
const kind = KINDS[(req.params.kind ?? "").toLowerCase()];
|
||||||
|
if (!kind) return res.status(404).type("text/plain").send("Not found");
|
||||||
|
const fromQuery = typeof req.query.msg === "string" ? req.query.msg : null;
|
||||||
|
const message = cleanMessage(fromQuery ?? (typeof req.body === "string" ? req.body : null));
|
||||||
|
const heartbeat = await recordPing(req.params.token, kind, message);
|
||||||
|
if (!heartbeat) return res.status(404).type("text/plain").send("Not found");
|
||||||
|
res.type("text/plain").send("OK");
|
||||||
|
}));
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
import { Router } from "express";
|
||||||
|
import { eq } from "drizzle-orm";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { heartbeats, servers } from "../db/schema.js";
|
||||||
|
import { requireAuth, requireRole } from "../auth/middleware.js";
|
||||||
|
import { recordAudit } from "../services/audit.js";
|
||||||
|
import { invalidateAlertsCache } from "../services/alerts.js";
|
||||||
|
import { dueBy, heartbeatState, listHeartbeats, loadRecent, MAX_GRACE_MINUTES, MAX_PERIOD_MINUTES, newToken } from "../services/heartbeats.js";
|
||||||
|
import { env } from "../env.js";
|
||||||
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
|
export const heartbeatsRouter = Router();
|
||||||
|
|
||||||
|
heartbeatsRouter.use(requireAuth);
|
||||||
|
|
||||||
|
const MAX_HEARTBEATS = 100;
|
||||||
|
|
||||||
|
const schema = z.object({
|
||||||
|
name: z.string().trim().min(1).max(100),
|
||||||
|
periodMinutes: z.number().int().min(1).max(MAX_PERIOD_MINUTES),
|
||||||
|
graceMinutes: z.number().int().min(0).max(MAX_GRACE_MINUTES).optional(),
|
||||||
|
serverId: z.number().int().positive().nullable().optional(),
|
||||||
|
notes: z.string().max(500).nullable().optional(),
|
||||||
|
enabled: z.boolean().optional(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const blank = (v: string | null | undefined) => (v === undefined || v === null || v.trim() === "" ? null : v.trim());
|
||||||
|
const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN);
|
||||||
|
const pathOf = (token: string) => `/api/heartbeat/${token}`;
|
||||||
|
|
||||||
|
async function serverExists(id: number | null | undefined) {
|
||||||
|
return id == null || (await db.select({ id: servers.id }).from(servers).where(eq(servers.id, id))).length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Everyone signed in can see how the jobs are doing, like the Alerts page. The address with its token is shown only to administrators: it
|
||||||
|
// is what lets anyone say "I'm alive" for the job.
|
||||||
|
heartbeatsRouter.get("/", asyncHandler(async (req, res) => {
|
||||||
|
const now = new Date();
|
||||||
|
const isAdmin = req.currentUser!.role === "admin";
|
||||||
|
const rows = await listHeartbeats();
|
||||||
|
const recent = await loadRecent(rows.map((r) => r.id));
|
||||||
|
const serverNames = new Map((await db.select({ id: servers.id, name: servers.name }).from(servers)).map((s) => [s.id, s.name]));
|
||||||
|
res.json({
|
||||||
|
baseUrl: isAdmin ? env.appBaseUrl.replace(/\/+$/, "") : null,
|
||||||
|
heartbeats: rows.map(({ token, ...r }) => ({
|
||||||
|
...r,
|
||||||
|
path: isAdmin ? pathOf(token) : null,
|
||||||
|
state: heartbeatState(r, now),
|
||||||
|
dueBy: dueBy(r).toISOString(),
|
||||||
|
server: r.serverId !== null && serverNames.has(r.serverId) ? { id: r.serverId, name: serverNames.get(r.serverId)! } : null,
|
||||||
|
recent: recent.get(r.id) ?? [],
|
||||||
|
})),
|
||||||
|
servers: [...serverNames].map(([id, name]) => ({ id, name })).sort((a, b) => a.name.localeCompare(b.name)),
|
||||||
|
});
|
||||||
|
}));
|
||||||
|
|
||||||
|
heartbeatsRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const parsed = schema.safeParse(req.body);
|
||||||
|
if (!parsed.success) return res.status(400).json({ error: "invalid_body", message: "Give it a name and say how often it should report (1 minute to 30 days).", details: parsed.error.flatten() });
|
||||||
|
if ((await db.select({ id: heartbeats.id }).from(heartbeats)).length >= MAX_HEARTBEATS) return res.status(409).json({ error: "too_many", message: `At most ${MAX_HEARTBEATS} heartbeats.` });
|
||||||
|
if (!(await serverExists(parsed.data.serverId))) return res.status(400).json({ error: "unknown_server", message: "That server doesn't exist." });
|
||||||
|
const d = parsed.data;
|
||||||
|
const [row] = await db
|
||||||
|
.insert(heartbeats)
|
||||||
|
.values({ name: d.name, token: newToken(), periodMinutes: d.periodMinutes, graceMinutes: d.graceMinutes ?? 10, serverId: d.serverId ?? null, notes: blank(d.notes), enabled: d.enabled ?? true })
|
||||||
|
.returning();
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "heartbeat", action: "create", targetType: "heartbeat", targetId: row.id, detail: { name: row.name, periodMinutes: row.periodMinutes } });
|
||||||
|
const { token, ...rest } = row;
|
||||||
|
res.status(201).json({ heartbeat: { ...rest, path: pathOf(token) } });
|
||||||
|
}));
|
||||||
|
|
||||||
|
heartbeatsRouter.put("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const id = idOf(req.params.id);
|
||||||
|
const parsed = schema.safeParse(req.body);
|
||||||
|
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body", message: "Give it a name and say how often it should report (1 minute to 30 days).", details: parsed.success ? undefined : parsed.error.flatten() });
|
||||||
|
const [before] = await db.select().from(heartbeats).where(eq(heartbeats.id, id)).limit(1);
|
||||||
|
if (!before) return res.status(404).json({ error: "not_found" });
|
||||||
|
if (!(await serverExists(parsed.data.serverId))) return res.status(400).json({ error: "unknown_server", message: "That server doesn't exist." });
|
||||||
|
const d = parsed.data;
|
||||||
|
const [row] = await db
|
||||||
|
.update(heartbeats)
|
||||||
|
.set({ name: d.name, periodMinutes: d.periodMinutes, graceMinutes: d.graceMinutes ?? before.graceMinutes, serverId: d.serverId ?? null, notes: blank(d.notes), enabled: d.enabled ?? before.enabled })
|
||||||
|
.where(eq(heartbeats.id, id))
|
||||||
|
.returning();
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "heartbeat", action: "update", targetType: "heartbeat", targetId: id, detail: { name: row.name, periodMinutes: row.periodMinutes } });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
const { token, ...rest } = row;
|
||||||
|
res.json({ heartbeat: { ...rest, path: pathOf(token) } });
|
||||||
|
}));
|
||||||
|
|
||||||
|
// A new address, when the old one has got out. The old one stops working at once.
|
||||||
|
heartbeatsRouter.post("/:id/new-address", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const id = idOf(req.params.id);
|
||||||
|
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
const [row] = await db.update(heartbeats).set({ token: newToken() }).where(eq(heartbeats.id, id)).returning();
|
||||||
|
if (!row) return res.status(404).json({ error: "not_found" });
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "heartbeat", action: "new_address", targetType: "heartbeat", targetId: id, detail: { name: row.name } });
|
||||||
|
res.json({ path: pathOf(row.token) });
|
||||||
|
}));
|
||||||
|
|
||||||
|
heartbeatsRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||||
|
const id = idOf(req.params.id);
|
||||||
|
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
const [before] = await db.select().from(heartbeats).where(eq(heartbeats.id, id)).limit(1);
|
||||||
|
if (!before) return res.status(404).json({ error: "not_found" });
|
||||||
|
await db.delete(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "heartbeat", action: "delete", targetType: "heartbeat", targetId: id, detail: { name: before.name } });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
res.status(204).end();
|
||||||
|
}));
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
import { Router } from "express";
|
||||||
|
import { NOT_FOUND, resolveShareLink } from "../services/shareLinks.js";
|
||||||
|
import { buildAgenda, buildIcs, calendarName, MAX_DAYS } from "../services/agenda.js";
|
||||||
|
import { env } from "../env.js";
|
||||||
|
import { rateLimit } from "../utils/rateLimit.js";
|
||||||
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
|
|
||||||
|
export const publicCalendarRouter = Router();
|
||||||
|
|
||||||
|
// No sign-in: the secret in the address is the whole of the access, so this reads nothing but what is coming up, and answers a link that doesn't
|
||||||
|
// work — never made, ended, taken back, or one that opens something other than a calendar — in the same words. The address ends in .ics because
|
||||||
|
// calendar apps want that.
|
||||||
|
const limiter = rateLimit({ windowMs: 60_000, max: 60, message: "Too many requests. Wait a minute." });
|
||||||
|
|
||||||
|
publicCalendarRouter.get("/:file", limiter, asyncHandler(async (req, res) => {
|
||||||
|
const token = req.params.file.endsWith(".ics") ? req.params.file.slice(0, -4) : req.params.file;
|
||||||
|
const link = await resolveShareLink(token);
|
||||||
|
if (!link || link.kind !== "calendar") return res.status(404).json(NOT_FOUND);
|
||||||
|
const feed = buildIcs(await buildAgenda(new Date(), MAX_DAYS), { name: calendarName(), detail: link.detail, baseUrl: link.detail === "names" ? env.appBaseUrl : undefined });
|
||||||
|
res.setHeader("Cache-Control", "private, max-age=300");
|
||||||
|
res.setHeader("X-Robots-Tag", "noindex, nofollow");
|
||||||
|
res.setHeader("Content-Disposition", 'inline; filename="homelab-coming-up.ics"');
|
||||||
|
res.type("text/calendar; charset=utf-8").send(feed);
|
||||||
|
}));
|
||||||
@@ -20,7 +20,7 @@ const MAX_LINKS = 20;
|
|||||||
const createSchema = z.object({
|
const createSchema = z.object({
|
||||||
label: z.string().trim().min(1).max(60),
|
label: z.string().trim().min(1).max(60),
|
||||||
kind: z.enum(shareLinkKinds).default("wall"),
|
kind: z.enum(shareLinkKinds).default("wall"),
|
||||||
// For a wall: how much it shows. A status page always shows only names and states, so this is ignored for it.
|
// For a wall: how much it shows. A status page always shows only names and states, so this is ignored for it. For a calendar: summary leaves the names out of the events.
|
||||||
detail: z.enum(shareLinkDetails).default("summary"),
|
detail: z.enum(shareLinkDetails).default("summary"),
|
||||||
title: z.string().trim().max(MAX_TITLE).optional(),
|
title: z.string().trim().max(MAX_TITLE).optional(),
|
||||||
message: z.string().trim().max(MAX_MESSAGE).optional(),
|
message: z.string().trim().max(MAX_MESSAGE).optional(),
|
||||||
@@ -59,7 +59,7 @@ shareLinksRouter.post("/", asyncHandler(async (req, res) => {
|
|||||||
targetId: link.id,
|
targetId: link.id,
|
||||||
detail: { label: link.label, kind: link.kind, detail: link.detail, expiresAt: link.expiresAt },
|
detail: { label: link.label, kind: link.kind, detail: link.detail, expiresAt: link.expiresAt },
|
||||||
});
|
});
|
||||||
res.status(201).json({ link: publicShareLink(link), token, path: `/share/${link.kind === "status" ? "status" : "wall"}/${token}` });
|
res.status(201).json({ link: publicShareLink(link), token, path: link.kind === "calendar" ? `/api/public/calendar/${token}.ics` : `/share/${link.kind === "status" ? "status" : "wall"}/${token}` });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN);
|
const idOf = (v: string) => (/^\d+$/.test(v) ? Number(v) : NaN);
|
||||||
|
|||||||
@@ -0,0 +1,343 @@
|
|||||||
|
/**
|
||||||
|
* "Coming up": everything with a date ahead, in one list — a secret, domain, certificate or Tailscale key that expires, a warranty or end of support,
|
||||||
|
* planned and recurring maintenance, a restore test that is due, and updates that have been waiting. Each already lives on a page of its own and in the
|
||||||
|
* weekly digest; this puts them side by side, oldest first, so what is about to need attention is one look, not a dozen.
|
||||||
|
*
|
||||||
|
* Built in two parts so it can be tested exactly: `loadAgendaData` reads what is stored (and, for Tailscale keys, asks Tailscale, remembered for ten
|
||||||
|
* minutes), and `agendaFromData` turns that into items. `buildIcs` writes the same items as an iCalendar feed for a phone's calendar to subscribe to.
|
||||||
|
*/
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { and, eq } from "drizzle-orm";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { assets, domains, endpointChecks, integrations, maintenanceSchedules, secrets, servers } from "../db/schema.js";
|
||||||
|
import { collectTailscaleKeyExpiries } from "./tailscaleKeyExpiryScheduler.js";
|
||||||
|
import { describeTarget, listUpcomingWindows } from "./maintenance.js";
|
||||||
|
import { occurrencesBetween, targetsOf } from "./maintenanceSchedules.js";
|
||||||
|
import { parsePendingUpdates } from "./pendingUpdates.js";
|
||||||
|
import { latestTests } from "./restoreTests.js";
|
||||||
|
import { getSettings } from "./settingsStore.js";
|
||||||
|
import { env } from "../env.js";
|
||||||
|
|
||||||
|
export const AGENDA_KINDS = ["secret", "certificate", "domain", "tailscale", "warranty", "support", "maintenance", "restore", "updates"] as const;
|
||||||
|
export type AgendaKind = (typeof AGENDA_KINDS)[number];
|
||||||
|
|
||||||
|
export const KIND_LABELS: Record<AgendaKind, string> = {
|
||||||
|
secret: "Secrets",
|
||||||
|
certificate: "Certificates",
|
||||||
|
domain: "Domains",
|
||||||
|
tailscale: "Tailscale keys",
|
||||||
|
warranty: "Warranties",
|
||||||
|
support: "End of support",
|
||||||
|
maintenance: "Maintenance",
|
||||||
|
restore: "Restore tests",
|
||||||
|
updates: "Updates waiting",
|
||||||
|
};
|
||||||
|
|
||||||
|
export interface AgendaItem {
|
||||||
|
/** Stable, with the date in it: a secret that is renewed is a different entry, and the old one just goes. */
|
||||||
|
id: string;
|
||||||
|
kind: AgendaKind;
|
||||||
|
/** YYYY-MM-DD for something that happens on a day; an ISO time for maintenance, which has a start and an end. */
|
||||||
|
at: string;
|
||||||
|
endsAt: string | null;
|
||||||
|
allDay: boolean;
|
||||||
|
title: string;
|
||||||
|
detail: string | null;
|
||||||
|
/** The page of the app that has more. */
|
||||||
|
link: string;
|
||||||
|
serverId: number | null;
|
||||||
|
/** Whole days from today to the day it falls on, in the time zone set for notifications; negative once it has passed. */
|
||||||
|
daysLeft: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Agenda {
|
||||||
|
generatedAt: string;
|
||||||
|
today: string;
|
||||||
|
items: AgendaItem[];
|
||||||
|
/** Sources that could not be read, so the agenda is missing something. */
|
||||||
|
notes: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_DAYS = 90;
|
||||||
|
export const MAX_DAYS = 400;
|
||||||
|
/** What has already happened is shown for this long (a secret that expired last week is still worth seeing). */
|
||||||
|
export const LOOK_BACK_DAYS = 30;
|
||||||
|
const DAY_MS = 86_400_000;
|
||||||
|
|
||||||
|
// ─── Dates ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const isYmd = (v: unknown): v is string => typeof v === "string" && /^\d{4}-\d{2}-\d{2}$/.test(v) && !Number.isNaN(Date.parse(`${v}T00:00:00Z`)) && new Date(`${v}T00:00:00Z`).toISOString().startsWith(v);
|
||||||
|
|
||||||
|
/** The calendar day an instant falls on, as the clock in `tz` sees it. */
|
||||||
|
export function localDate(when: Date, tz: string): string {
|
||||||
|
return new Intl.DateTimeFormat("en-CA", { timeZone: tz, year: "numeric", month: "2-digit", day: "2-digit" }).format(when);
|
||||||
|
}
|
||||||
|
|
||||||
|
const daysBetween = (fromYmd: string, toYmd: string) => Math.round((Date.parse(`${toYmd}T00:00:00Z`) - Date.parse(`${fromYmd}T00:00:00Z`)) / DAY_MS);
|
||||||
|
|
||||||
|
export const addDays = (ymd: string, n: number) => new Date(Date.parse(`${ymd}T00:00:00Z`) + n * DAY_MS).toISOString().slice(0, 10);
|
||||||
|
|
||||||
|
// ─── What is stored ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface AgendaData {
|
||||||
|
secrets: { id: number; name: string; type: string; expiryDate: string | null }[];
|
||||||
|
domains: { id: number; name: string; expiresAt: string | null }[];
|
||||||
|
endpoints: { id: number; name: string; enabled: boolean; certExpiresAt: string | null; serverId: number | null }[];
|
||||||
|
tailscale: { integrationId: number; integrationName: string; deviceLabel: string; daysLeft: number }[];
|
||||||
|
assets: { id: number; name: string; warrantyUntil: string | null; endOfSupport: string | null; serverId: number | null }[];
|
||||||
|
/** Windows that have been set up but not begun, one per target. */
|
||||||
|
windows: { startedAt: string; endsAt: string; reason: string | null; source: string | null; targetName: string }[];
|
||||||
|
schedules: { id: number; name: string; reason: string | null; days: string; startTime: string; durationMinutes: number; timezone: string; targetNames: string[] }[];
|
||||||
|
/** The newest restore test of each subject. */
|
||||||
|
restoreTests: { subjectKey: string; label: string; testedAt: string; result: string }[];
|
||||||
|
restoreLimitDays: number;
|
||||||
|
/** Servers with updates waiting. */
|
||||||
|
updates: { name: string; count: number; security: number | null; since: string | null }[];
|
||||||
|
notes: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
let tailscaleCache: { at: number; items: AgendaData["tailscale"]; failures: string[] } | null = null;
|
||||||
|
const TAILSCALE_CACHE_MS = 10 * 60_000;
|
||||||
|
export const clearAgendaCacheForTests = () => {
|
||||||
|
tailscaleCache = null;
|
||||||
|
};
|
||||||
|
|
||||||
|
async function tailscaleKeys(): Promise<{ items: AgendaData["tailscale"]; failures: string[] }> {
|
||||||
|
if (tailscaleCache && Date.now() - tailscaleCache.at < TAILSCALE_CACHE_MS) return tailscaleCache;
|
||||||
|
const rows = await db.select({ id: integrations.id }).from(integrations).where(and(eq(integrations.type, "tailscale"), eq(integrations.enabled, true)));
|
||||||
|
if (rows.length === 0) return { items: [], failures: [] };
|
||||||
|
const { items, failures } = await collectTailscaleKeyExpiries();
|
||||||
|
tailscaleCache = { at: Date.now(), items, failures: failures.map((f) => `Tailscale (${f.integrationName}): ${f.message}`) };
|
||||||
|
return tailscaleCache;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function loadAgendaData(now: Date = new Date()): Promise<AgendaData> {
|
||||||
|
const notes: string[] = [];
|
||||||
|
const settings = await getSettings();
|
||||||
|
const [secretRows, domainRows, endpointRows, assetRows, scheduleRows, serverRows, tests] = await Promise.all([
|
||||||
|
db.select({ id: secrets.id, name: secrets.name, type: secrets.type, expiryDate: secrets.expiryDate }).from(secrets),
|
||||||
|
db.select({ id: domains.id, name: domains.name, expiresAt: domains.expiresAt }).from(domains),
|
||||||
|
db.select({ id: endpointChecks.id, name: endpointChecks.name, enabled: endpointChecks.enabled, certExpiresAt: endpointChecks.certExpiresAt, serverId: endpointChecks.serverId }).from(endpointChecks),
|
||||||
|
db.select({ id: assets.id, name: assets.name, warrantyUntil: assets.warrantyUntil, endOfSupport: assets.endOfSupport, serverId: assets.serverId }).from(assets),
|
||||||
|
db.select().from(maintenanceSchedules),
|
||||||
|
db.select({ name: servers.name, pendingUpdates: servers.pendingUpdates, since: servers.updatesPendingSince }).from(servers),
|
||||||
|
latestTests(),
|
||||||
|
]);
|
||||||
|
|
||||||
|
const windows: AgendaData["windows"] = [];
|
||||||
|
for (const w of await listUpcomingWindows(now)) {
|
||||||
|
const target = await describeTarget(w.targetType, w.targetId);
|
||||||
|
if (target) windows.push({ startedAt: w.startedAt, endsAt: w.endsAt, reason: w.reason, source: w.source, targetName: target.name });
|
||||||
|
}
|
||||||
|
|
||||||
|
const schedules: AgendaData["schedules"] = [];
|
||||||
|
for (const s of scheduleRows.filter((r) => r.enabled)) {
|
||||||
|
const names: string[] = [];
|
||||||
|
for (const t of targetsOf(s)) {
|
||||||
|
const d = await describeTarget(t.type, t.id);
|
||||||
|
if (d) names.push(d.name);
|
||||||
|
}
|
||||||
|
schedules.push({ id: s.id, name: s.name, reason: s.reason, days: s.days, startTime: s.startTime, durationMinutes: s.durationMinutes, timezone: s.timezone, targetNames: names });
|
||||||
|
}
|
||||||
|
|
||||||
|
const updates: AgendaData["updates"] = [];
|
||||||
|
for (const s of serverRows) {
|
||||||
|
const u = parsePendingUpdates(s.pendingUpdates);
|
||||||
|
if (u && u.count > 0) updates.push({ name: s.name, count: u.count, security: u.security, since: s.since });
|
||||||
|
}
|
||||||
|
|
||||||
|
let tailscale: AgendaData["tailscale"] = [];
|
||||||
|
try {
|
||||||
|
const t = await tailscaleKeys();
|
||||||
|
tailscale = t.items;
|
||||||
|
notes.push(...t.failures);
|
||||||
|
} catch (err) {
|
||||||
|
notes.push(`Tailscale keys could not be read: ${err instanceof Error ? err.message : String(err)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
secrets: secretRows,
|
||||||
|
domains: domainRows,
|
||||||
|
endpoints: endpointRows,
|
||||||
|
tailscale,
|
||||||
|
assets: assetRows,
|
||||||
|
windows,
|
||||||
|
schedules,
|
||||||
|
restoreTests: [...tests.values()].map((t) => ({ subjectKey: t.subjectKey, label: t.label, testedAt: t.testedAt, result: t.result })),
|
||||||
|
restoreLimitDays: settings.healthChecks.restoreTestDays,
|
||||||
|
updates,
|
||||||
|
notes,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Making the list ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`;
|
||||||
|
|
||||||
|
/** Pure: the items, from what was read. `days` is how far ahead; what has just passed is kept for LOOK_BACK_DAYS. */
|
||||||
|
export function agendaFromData(data: AgendaData, now: Date, tz: string, days: number = DEFAULT_DAYS): Agenda {
|
||||||
|
const today = localDate(now, tz);
|
||||||
|
const horizon = Math.min(Math.max(Math.round(days), 1), MAX_DAYS);
|
||||||
|
const items: AgendaItem[] = [];
|
||||||
|
|
||||||
|
const day = (id: string, kind: AgendaKind, ymd: string | null, fn: (past: boolean) => { title: string; detail?: string | null; link: string; serverId?: number | null }) => {
|
||||||
|
if (!isYmd(ymd)) return;
|
||||||
|
const daysLeft = daysBetween(today, ymd);
|
||||||
|
if (daysLeft < -LOOK_BACK_DAYS || daysLeft > horizon) return;
|
||||||
|
const { title, detail = null, link, serverId = null } = fn(daysLeft < 0);
|
||||||
|
items.push({ id: `${id}:${ymd}`, kind, at: ymd, endsAt: null, allDay: true, title, detail, link, serverId, daysLeft });
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const s of data.secrets) {
|
||||||
|
const certificate = s.type === "ssl_certificate";
|
||||||
|
day(`secret:${s.id}`, certificate ? "certificate" : "secret", s.expiryDate, (past) => ({ title: `${certificate ? "Certificate" : "Secret"} “${s.name}” ${past ? "expired" : "expires"}`, detail: certificate ? "kept as a secret" : null, link: "/secrets" }));
|
||||||
|
}
|
||||||
|
for (const d of data.domains) day(`domain:${d.id}`, "domain", d.expiresAt, (past) => ({ title: `Domain ${d.name} ${past ? "expired" : "expires"}`, link: "/domains" }));
|
||||||
|
for (const e of data.endpoints) {
|
||||||
|
if (!e.enabled || !e.certExpiresAt) continue;
|
||||||
|
day(`endpoint-cert:${e.id}`, "certificate", e.certExpiresAt.slice(0, 10), (past) => ({ title: `The certificate for ${e.name} ${past ? "expired" : "expires"}`, detail: "seen by an endpoint check", link: "/endpoints", serverId: e.serverId }));
|
||||||
|
}
|
||||||
|
for (const t of data.tailscale) {
|
||||||
|
day(`tailscale:${t.integrationId}:${t.deviceLabel}`, "tailscale", addDays(today, t.daysLeft), (past) => ({ title: `Tailscale key for ${t.deviceLabel} ${past ? "expired" : "expires"}`, detail: t.integrationName, link: "/tailscale" }));
|
||||||
|
}
|
||||||
|
for (const a of data.assets) {
|
||||||
|
day(`warranty:${a.id}`, "warranty", a.warrantyUntil, (past) => ({ title: `Warranty on ${a.name} ${past ? "ended" : "ends"}`, link: `/inventory?asset=${a.id}`, serverId: a.serverId }));
|
||||||
|
day(`support:${a.id}`, "support", a.endOfSupport, (past) => ({ title: `Support for ${a.name} ${past ? "ended" : "ends"}`, detail: "time to plan its replacement", link: `/inventory?asset=${a.id}`, serverId: a.serverId }));
|
||||||
|
}
|
||||||
|
|
||||||
|
// A restore test is due a set time after the last good one — only if a time limit is set (Settings → Notifications).
|
||||||
|
if (data.restoreLimitDays > 0) {
|
||||||
|
for (const t of data.restoreTests) {
|
||||||
|
if ((t.result !== "ok" && t.result !== "partial") || !isYmd(t.testedAt)) continue;
|
||||||
|
day(`restore:${t.subjectKey}`, "restore", addDays(t.testedAt, data.restoreLimitDays), (past) => ({ title: `Restore test ${past ? "was due" : "due"}: ${t.label}`, detail: `last tested ${t.testedAt}`, link: "/backups" }));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Maintenance has a start and an end. Windows set up by a recurring schedule are left to the schedule (they are only made a week ahead).
|
||||||
|
const lastMs = now.getTime() + horizon * DAY_MS;
|
||||||
|
const groups = new Map<string, { start: string; end: string; reason: string | null; names: string[] }>();
|
||||||
|
for (const w of data.windows) {
|
||||||
|
if (w.source?.startsWith("schedule:") || Date.parse(w.startedAt) > lastMs) continue;
|
||||||
|
const key = `${w.startedAt}|${w.endsAt}|${w.reason ?? ""}`;
|
||||||
|
const g = groups.get(key) ?? { start: w.startedAt, end: w.endsAt, reason: w.reason, names: [] };
|
||||||
|
g.names.push(w.targetName);
|
||||||
|
groups.set(key, g);
|
||||||
|
}
|
||||||
|
const timed = (id: string, start: string, end: string, title: string, detail: string | null) =>
|
||||||
|
items.push({ id, kind: "maintenance", at: start, endsAt: end, allDay: false, title, detail, link: "/maintenance", serverId: null, daysLeft: daysBetween(today, localDate(new Date(start), tz)) });
|
||||||
|
const names = (list: string[]) => {
|
||||||
|
const sorted = [...list].sort((a, b) => a.localeCompare(b));
|
||||||
|
return sorted.length <= 3 ? sorted.join(", ") : `${sorted.slice(0, 3).join(", ")} and ${sorted.length - 3} more`;
|
||||||
|
};
|
||||||
|
for (const g of groups.values()) timed(`maintenance:${g.start}:${g.names.length}`, g.start, g.end, `Maintenance: ${names(g.names)}`, g.reason);
|
||||||
|
for (const s of data.schedules) {
|
||||||
|
for (const occ of occurrencesBetween(s, now, new Date(lastMs)).slice(0, 60)) {
|
||||||
|
timed(`schedule:${s.id}:${occ.date}`, occ.start.toISOString(), occ.end.toISOString(), `Recurring maintenance: ${s.name}`, [s.targetNames.length > 0 ? names(s.targetNames) : null, s.reason].filter(Boolean).join(" — ") || null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Updates that have been waiting: one line for all of them, from the longest wait.
|
||||||
|
if (data.updates.length > 0) {
|
||||||
|
const since = data.updates.map((u) => u.since).filter((s): s is string => !!s).map((s) => localDate(new Date(s), tz)).sort()[0] ?? today;
|
||||||
|
const security = data.updates.filter((u) => (u.security ?? 0) > 0).length;
|
||||||
|
const waited = Math.max(0, daysBetween(since, today));
|
||||||
|
items.push({
|
||||||
|
id: `updates:${since}`,
|
||||||
|
kind: "updates",
|
||||||
|
at: since,
|
||||||
|
endsAt: null,
|
||||||
|
allDay: true,
|
||||||
|
title: `Updates waiting on ${plural(data.updates.length, "server")}${security > 0 ? ` (${security} with security fixes)` : ""}`,
|
||||||
|
detail: waited > 0 ? `the longest wait is ${plural(waited, "day")}` : "reported today",
|
||||||
|
link: "/updates",
|
||||||
|
serverId: null,
|
||||||
|
daysLeft: -waited,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
items.sort((a, b) => a.at.localeCompare(b.at) || a.title.localeCompare(b.title));
|
||||||
|
return { generatedAt: now.toISOString(), today, items, notes: data.notes };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function buildAgenda(now: Date = new Date(), days: number = DEFAULT_DAYS): Promise<Agenda> {
|
||||||
|
const settings = await getSettings();
|
||||||
|
return agendaFromData(await loadAgendaData(now), now, settings.notifications.timezone || "UTC", days);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── The calendar feed ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/** Escapes a text value for iCalendar: backslash, semicolon, comma and line breaks. */
|
||||||
|
const escapeText = (s: string) => s.replace(/\\/g, "\\\\").replace(/;/g, "\\;").replace(/,/g, "\\,").replace(/\r?\n/g, "\\n");
|
||||||
|
|
||||||
|
/** Lines are folded at 75 bytes, a space starting each continuation, without splitting a character. */
|
||||||
|
export function foldLine(line: string): string {
|
||||||
|
const out: string[] = [];
|
||||||
|
let current = "";
|
||||||
|
let bytes = 0;
|
||||||
|
for (const ch of line) {
|
||||||
|
const n = Buffer.byteLength(ch, "utf8");
|
||||||
|
if (bytes + n > (out.length === 0 ? 75 : 74)) {
|
||||||
|
out.push(current);
|
||||||
|
current = "";
|
||||||
|
bytes = 0;
|
||||||
|
}
|
||||||
|
current += ch;
|
||||||
|
bytes += n;
|
||||||
|
}
|
||||||
|
out.push(current);
|
||||||
|
return out.join("\r\n ");
|
||||||
|
}
|
||||||
|
|
||||||
|
const icsDate = (ymd: string) => ymd.replace(/-/g, "");
|
||||||
|
const icsTime = (iso: string) => new Date(iso).toISOString().replace(/[-:]/g, "").replace(/\.\d{3}/, "");
|
||||||
|
|
||||||
|
/** What an event is called when the link was made to show no names: only what kind of thing it is. */
|
||||||
|
const GENERIC: Record<AgendaKind, string> = {
|
||||||
|
secret: "A secret expires",
|
||||||
|
certificate: "A certificate expires",
|
||||||
|
domain: "A domain name expires",
|
||||||
|
tailscale: "A Tailscale key expires",
|
||||||
|
warranty: "A warranty ends",
|
||||||
|
support: "Support ends for a device",
|
||||||
|
maintenance: "Planned maintenance",
|
||||||
|
restore: "A restore test is due",
|
||||||
|
updates: "Updates are waiting",
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The agenda as an iCalendar feed. A day is an all-day event; maintenance is an event from its start to its end. With `detail` "summary" an event only
|
||||||
|
* says what kind of thing it is (no names), so a feed that is shared can be shown without telling anyone what is in the homelab. Updates that
|
||||||
|
* are waiting are not events — they have no day — and are left out.
|
||||||
|
*/
|
||||||
|
export function buildIcs(agenda: Agenda, options: { name: string; detail: "summary" | "names"; baseUrl?: string }): string {
|
||||||
|
const lines = [
|
||||||
|
"BEGIN:VCALENDAR",
|
||||||
|
"VERSION:2.0",
|
||||||
|
"PRODID:-//Homelab Manager//Coming up//EN",
|
||||||
|
"CALSCALE:GREGORIAN",
|
||||||
|
"METHOD:PUBLISH",
|
||||||
|
`X-WR-CALNAME:${escapeText(options.name)}`,
|
||||||
|
"REFRESH-INTERVAL;VALUE=DURATION:PT6H",
|
||||||
|
"X-PUBLISHED-TTL:PT6H",
|
||||||
|
];
|
||||||
|
const stamp = icsTime(agenda.generatedAt);
|
||||||
|
for (const item of agenda.items) {
|
||||||
|
if (item.kind === "updates") continue;
|
||||||
|
if (item.daysLeft < -7) continue; // an old event in a subscribed calendar is just clutter
|
||||||
|
const names = options.detail === "names";
|
||||||
|
lines.push("BEGIN:VEVENT", `UID:${createHash("sha256").update(item.id).digest("hex").slice(0, 24)}@homelab-manager`, `DTSTAMP:${stamp}`);
|
||||||
|
if (item.allDay) lines.push(`DTSTART;VALUE=DATE:${icsDate(item.at)}`, `DTEND;VALUE=DATE:${icsDate(addDays(item.at, 1))}`);
|
||||||
|
else lines.push(`DTSTART:${icsTime(item.at)}`, `DTEND:${icsTime(item.endsAt ?? item.at)}`);
|
||||||
|
lines.push(`SUMMARY:${escapeText(names ? item.title : GENERIC[item.kind])}`);
|
||||||
|
if (names) {
|
||||||
|
const description = [item.detail, options.baseUrl ? `${options.baseUrl.replace(/\/+$/, "")}${item.link}` : null].filter(Boolean).join("\n");
|
||||||
|
if (description) lines.push(`DESCRIPTION:${escapeText(description)}`);
|
||||||
|
}
|
||||||
|
lines.push(`CATEGORIES:${escapeText(KIND_LABELS[item.kind])}`, "TRANSP:TRANSPARENT", "STATUS:CONFIRMED", "END:VEVENT");
|
||||||
|
}
|
||||||
|
lines.push("END:VCALENDAR");
|
||||||
|
return `${lines.map(foldLine).join("\r\n")}\r\n`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What the feed calls itself. */
|
||||||
|
export const calendarName = () => `Homelab Manager${env.environment?.name ? ` (${env.environment.name})` : ""} — coming up`;
|
||||||
@@ -11,7 +11,7 @@
|
|||||||
*/
|
*/
|
||||||
import { eq } from "drizzle-orm";
|
import { eq } from "drizzle-orm";
|
||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { endpointChecks, integrations, secrets, servers } from "../db/schema.js";
|
import { endpointChecks, heartbeats, integrations, secrets, servers } from "../db/schema.js";
|
||||||
import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js";
|
import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js";
|
||||||
import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js";
|
import { diskLimit, effectiveThresholds, parseThresholds } from "./serverThresholds.js";
|
||||||
import { getBackupCoverage, staleBackups } from "./backupCoverage.js";
|
import { getBackupCoverage, staleBackups } from "./backupCoverage.js";
|
||||||
@@ -35,6 +35,7 @@ import { getSettings } from "./settingsStore.js";
|
|||||||
import { sourceLabel } from "./notify.js";
|
import { sourceLabel } from "./notify.js";
|
||||||
import { alertIdOfNode, dependencyCover, integrationTypeCause, takenByType } from "./dependencies.js";
|
import { alertIdOfNode, dependencyCover, integrationTypeCause, takenByType } from "./dependencies.js";
|
||||||
import { endpointConditions } from "./endpointChecks.js";
|
import { endpointConditions } from "./endpointChecks.js";
|
||||||
|
import { heartbeatConditions } from "./heartbeats.js";
|
||||||
import { latestTests, restoreAlerts } from "./restoreTests.js";
|
import { latestTests, restoreAlerts } from "./restoreTests.js";
|
||||||
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
|
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
|
||||||
|
|
||||||
@@ -65,6 +66,7 @@ const SOURCES: Record<string, { label: string; link: string }> = {
|
|||||||
secrets: { label: "Secrets", link: "/secrets" },
|
secrets: { label: "Secrets", link: "/secrets" },
|
||||||
domains: { label: "Domains", link: "/domains" },
|
domains: { label: "Domains", link: "/domains" },
|
||||||
endpoint: { label: "Endpoint", link: "/endpoints" },
|
endpoint: { label: "Endpoint", link: "/endpoints" },
|
||||||
|
heartbeat: { label: "Heartbeat", link: "/heartbeats" },
|
||||||
};
|
};
|
||||||
|
|
||||||
const SOURCE_TIMEOUT_MS = 20_000;
|
const SOURCE_TIMEOUT_MS = 20_000;
|
||||||
@@ -304,6 +306,22 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
|
|
||||||
|
// Jobs that report in: one that has not done so in time, or said it failed.
|
||||||
|
check("Heartbeats", async () => {
|
||||||
|
for (const c of heartbeatConditions(await db.select().from(heartbeats), new Date())) {
|
||||||
|
push({
|
||||||
|
category: c.category,
|
||||||
|
severity: c.severity,
|
||||||
|
type: "heartbeat",
|
||||||
|
message: c.message,
|
||||||
|
key: c.key,
|
||||||
|
link: "/heartbeats",
|
||||||
|
silenced: c.serverId !== null && subjects.has(`server:${c.serverId}`),
|
||||||
|
target: c.serverId !== null ? targetOf("server", c.serverId) : null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
|
||||||
// Homelab Manager's own backups: if they are switched on, there should be a recent one. This is what turns a backup that has quietly
|
// Homelab Manager's own backups: if they are switched on, there should be a recent one. This is what turns a backup that has quietly
|
||||||
// stopped working — a full disk, a folder that went read-only — into something somebody hears about before they need it.
|
// stopped working — a full disk, a folder that went read-only — into something somebody hears about before they need it.
|
||||||
check("Homelab Manager's own backups", async () => {
|
check("Homelab Manager's own backups", async () => {
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ export const KIND_LABELS: Record<ChangeKind, string> = {
|
|||||||
dependencies: "Dependencies",
|
dependencies: "Dependencies",
|
||||||
inventory: "Inventory",
|
inventory: "Inventory",
|
||||||
backups: "Backups and restore tests",
|
backups: "Backups and restore tests",
|
||||||
endpoints: "Endpoint checks",
|
endpoints: "Endpoint checks and heartbeats",
|
||||||
other: "Other",
|
other: "Other",
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -53,6 +53,7 @@ const KIND_OF_CATEGORY: Record<string, ChangeKind> = {
|
|||||||
db_backup: "backups",
|
db_backup: "backups",
|
||||||
restore_test: "backups",
|
restore_test: "backups",
|
||||||
endpoint: "endpoints",
|
endpoint: "endpoints",
|
||||||
|
heartbeat: "endpoints",
|
||||||
};
|
};
|
||||||
|
|
||||||
/** Sign-ins and a person's own data export are not changes to the homelab; the diagnostic log has its own page. */
|
/** Sign-ins and a person's own data export are not changes to the homelab; the diagnostic log has its own page. */
|
||||||
@@ -154,6 +155,14 @@ export function describeChange(category: string, action: string, detail: Record<
|
|||||||
return { title: `Endpoint check changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: [str(detail.kind), str(detail.target)].filter(Boolean).join(" · ") || null };
|
return { title: `Endpoint check changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: [str(detail.kind), str(detail.target)].filter(Boolean).join(" · ") || null };
|
||||||
case "endpoint/delete":
|
case "endpoint/delete":
|
||||||
return { title: `Endpoint check removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
|
return { title: `Endpoint check removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
|
||||||
|
case "heartbeat/create":
|
||||||
|
return { title: `Heartbeat added${name ? `: ${name}` : ""}`, tone: "neutral", detail: typeof detail.periodMinutes === "number" ? `expected every ${detail.periodMinutes} min` : null };
|
||||||
|
case "heartbeat/update":
|
||||||
|
return { title: `Heartbeat changed${name ? `: ${name}` : ""}`, tone: "neutral", detail: typeof detail.periodMinutes === "number" ? `expected every ${detail.periodMinutes} min` : null };
|
||||||
|
case "heartbeat/new_address":
|
||||||
|
return { title: `Heartbeat given a new address${name ? `: ${name}` : ""}`, tone: "neutral", detail: "the old address no longer works" };
|
||||||
|
case "heartbeat/delete":
|
||||||
|
return { title: `Heartbeat removed${name ? `: ${name}` : ""}`, tone: "neutral", detail: null };
|
||||||
case "restore_test/record": {
|
case "restore_test/record": {
|
||||||
const result = str(detail.result);
|
const result = str(detail.result);
|
||||||
const tone: ActivityTone = result === "failed" ? "bad" : result === "not_needed" ? "neutral" : "good";
|
const tone: ActivityTone = result === "failed" ? "bad" : result === "not_needed" ? "neutral" : "good";
|
||||||
@@ -250,7 +259,8 @@ export async function getChangeLog(query: ChangeQuery = {}): Promise<{ entries:
|
|||||||
} else if (a.targetType === "endpoint" && idNum !== null) {
|
} else if (a.targetType === "endpoint" && idNum !== null) {
|
||||||
targetName = endpointName.get(idNum) ?? targetName;
|
targetName = endpointName.get(idNum) ?? targetName;
|
||||||
link = "/endpoints";
|
link = "/endpoints";
|
||||||
} else if (a.targetType === "asset" && idNum !== null) link = `/inventory?asset=${idNum}`;
|
} else if (a.targetType === "heartbeat") link = "/heartbeats";
|
||||||
|
else if (a.targetType === "asset" && idNum !== null) link = `/inventory?asset=${idNum}`;
|
||||||
else if (a.targetType === "restore_test") link = "/backups";
|
else if (a.targetType === "restore_test") link = "/backups";
|
||||||
else if (a.targetType === "dependency") link = "/dependencies";
|
else if (a.targetType === "dependency") link = "/dependencies";
|
||||||
else if (a.targetType === "settings") link = "/settings/notifications";
|
else if (a.targetType === "settings") link = "/settings/notifications";
|
||||||
@@ -280,7 +290,7 @@ export async function getChangeLog(query: ChangeQuery = {}): Promise<{ entries:
|
|||||||
.limit(1000);
|
.limit(1000);
|
||||||
for (const i of incidents) {
|
for (const i of incidents) {
|
||||||
const serverId = i.targetType === "server" ? i.targetId : null;
|
const serverId = i.targetType === "server" ? i.targetId : null;
|
||||||
const base = { kind: "alerts" as const, actor: null, target: i.targetType ? { type: i.targetType, id: i.targetId === null ? null : String(i.targetId), name: i.targetName } : null, link: `/incidents/${i.id}`, serverId };
|
const base = { kind: "alerts" as const, actor: null, target: i.targetType ? { type: i.targetType, id: i.targetId === null ? null : String(i.targetId), name: i.targetName } : null, link: `/alerts/incidents/${i.id}`, serverId };
|
||||||
if (i.openedAt >= since.toISOString() && (!until || i.openedAt <= until.toISOString())) {
|
if (i.openedAt >= since.toISOString() && (!until || i.openedAt <= until.toISOString())) {
|
||||||
entries.push({ detail: {}, entry: { ...base, id: `incident-open-${i.id}`, at: i.openedAt, tone: i.peakSeverity === "critical" ? "bad" : "info", title: `Alert raised: ${i.message}`, detail: i.causedBy ? `explained by ${i.causedBy}` : null } });
|
entries.push({ detail: {}, entry: { ...base, id: `incident-open-${i.id}`, at: i.openedAt, tone: i.peakSeverity === "critical" ? "bad" : "info", title: `Alert raised: ${i.message}`, detail: i.causedBy ? `explained by ${i.causedBy}` : null } });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,9 +5,10 @@
|
|||||||
* a maintenance window on the server a check is linked to holds its problems back, and they alert after it if they are still there.
|
* a maintenance window on the server a check is linked to holds its problems back, and they alert after it if they are still there.
|
||||||
*/
|
*/
|
||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { endpointChecks } from "../db/schema.js";
|
import { endpointChecks, heartbeats } from "../db/schema.js";
|
||||||
import { diffConditions, type ActiveState, type HealthCondition } from "./healthMonitor.js";
|
import { diffConditions, type ActiveState, type HealthCondition } from "./healthMonitor.js";
|
||||||
import { activeSubjects } from "./maintenance.js";
|
import { activeSubjects } from "./maintenance.js";
|
||||||
|
import { heartbeatConditions, prunePings } from "./heartbeats.js";
|
||||||
import { endpointConditions, pruneResults, runDueChecks, type CheckRow, type ProbeResult } from "./endpointChecks.js";
|
import { endpointConditions, pruneResults, runDueChecks, type CheckRow, type ProbeResult } from "./endpointChecks.js";
|
||||||
import { notifyHealthIssues, notifyHealthRecovered } from "./notify.js";
|
import { notifyHealthIssues, notifyHealthRecovered } from "./notify.js";
|
||||||
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
|
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
|
||||||
@@ -28,7 +29,9 @@ async function loadState(): Promise<ActiveState> {
|
|||||||
export async function runEndpointPass(now: Date = new Date(), probeFn?: (c: CheckRow) => Promise<ProbeResult>): Promise<{ asked: number; announced: number; recovered: number }> {
|
export async function runEndpointPass(now: Date = new Date(), probeFn?: (c: CheckRow) => Promise<ProbeResult>): Promise<{ asked: number; announced: number; recovered: number }> {
|
||||||
const asked = await runDueChecks(now, probeFn);
|
const asked = await runDueChecks(now, probeFn);
|
||||||
const rows = await db.select().from(endpointChecks);
|
const rows = await db.select().from(endpointChecks);
|
||||||
const current: HealthCondition[] = endpointConditions(rows, now).map((c) => ({ key: c.key, source: c.source, message: c.message, severity: c.severity }));
|
// The jobs that report in are judged in the same pass, by the same bookkeeping: a call that is overdue is announced once, and again as cleared when it comes.
|
||||||
|
const beats = await db.select().from(heartbeats);
|
||||||
|
const current: HealthCondition[] = [...endpointConditions(rows, now), ...heartbeatConditions(beats, now)].map((c) => ({ key: c.key, source: c.source, message: c.message, severity: c.severity }));
|
||||||
const { added, resolved, next } = diffConditions(await loadState(), current, new Set(), await activeSubjects(now));
|
const { added, resolved, next } = diffConditions(await loadState(), current, new Set(), await activeSubjects(now));
|
||||||
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
|
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
|
||||||
if (added.length > 0) await notifyHealthIssues(added);
|
if (added.length > 0) await notifyHealthIssues(added);
|
||||||
@@ -48,6 +51,7 @@ export function initEndpointMonitor(): void {
|
|||||||
if (Date.now() - lastPrune > 24 * 3600_000) {
|
if (Date.now() - lastPrune > 24 * 3600_000) {
|
||||||
lastPrune = Date.now();
|
lastPrune = Date.now();
|
||||||
await pruneResults();
|
await pruneResults();
|
||||||
|
await prunePings();
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error("[endpoints] the pass failed:", err);
|
console.error("[endpoints] the pass failed:", err);
|
||||||
|
|||||||
@@ -0,0 +1,186 @@
|
|||||||
|
/**
|
||||||
|
* Suggested endpoint checks: things the app already knows enough about to offer a check for, so setting up monitoring is a click instead of typing.
|
||||||
|
*
|
||||||
|
* - A web port a server's agent sees open (80, 443, 8080, 8443), with a name that points at it in DNS if there is one — "git.example.lan answers on
|
||||||
|
* 443 on nas" — as a web check, linked to that server so its maintenance silences the alerts.
|
||||||
|
* - A secret that already has a host to read the certificate from, as a certificate check: the secret only warns about the expiry, this also hears
|
||||||
|
* when the service stops answering.
|
||||||
|
*
|
||||||
|
* It only reads what is stored (no network), leaves out what a check already covers, and remembers what somebody said no to. Uptime Kuma's monitors are
|
||||||
|
* not offered: they are already being watched, and the adapter only keeps a host and port for them, not the address.
|
||||||
|
*/
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { dnsRecordsCache, endpointChecks, secrets, servers } from "../db/schema.js";
|
||||||
|
import { isLoopback } from "./agentPorts.js";
|
||||||
|
import { getInternalFlag, setInternalFlag } from "./settingsStore.js";
|
||||||
|
|
||||||
|
export interface Suggestion {
|
||||||
|
/** Stable identity, so a dismissed suggestion stays dismissed and "add" can name it. */
|
||||||
|
key: string;
|
||||||
|
kind: "http" | "tls";
|
||||||
|
name: string;
|
||||||
|
target: string;
|
||||||
|
port: number | null;
|
||||||
|
serverId: number | null;
|
||||||
|
serverName: string | null;
|
||||||
|
/** Why it is offered, in a sentence. */
|
||||||
|
reason: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SuggestionInput {
|
||||||
|
servers: { id: number; name: string; hostname: string | null; ips: string[]; ports: { protocol?: string; port: number; address?: string }[] }[];
|
||||||
|
dns: { name: string; type: string; content: string }[];
|
||||||
|
secrets: { name: string; checkHost: string | null; checkPort: number | null }[];
|
||||||
|
existing: { kind: string; target: string; port: number | null }[];
|
||||||
|
dismissed: Set<string>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The web ports worth offering, and what they speak. */
|
||||||
|
const WEB_PORTS: Record<number, "http" | "https"> = { 80: "http", 443: "https", 8080: "http", 8443: "https" };
|
||||||
|
/** The https twin of a plain port, so a server answering on both gets the one that reads the certificate. */
|
||||||
|
const SECURE_TWIN: Record<number, number> = { 80: 443, 8080: 8443 };
|
||||||
|
const MAX_NAMES_PER_PORT = 3;
|
||||||
|
const MAX_SUGGESTIONS = 200;
|
||||||
|
|
||||||
|
const cleanHost = (h: string) => h.trim().toLowerCase().replace(/\.$/, "");
|
||||||
|
const isIPv4 = (s: string) => /^\d{1,3}(\.\d{1,3}){3}$/.test(s);
|
||||||
|
const usableIPv4 = (s: string) => isIPv4(s) && !s.startsWith("127.") && !s.startsWith("169.254.");
|
||||||
|
|
||||||
|
/** Where a check's address is, in the same words for an existing check and a suggestion: "web:host:port" and "tls:host:port" (a web check on https reads the certificate too). */
|
||||||
|
function coverage(existing: SuggestionInput["existing"]): Set<string> {
|
||||||
|
const out = new Set<string>();
|
||||||
|
for (const c of existing) {
|
||||||
|
if (c.kind === "http") {
|
||||||
|
try {
|
||||||
|
const u = new URL(c.target);
|
||||||
|
const port = u.port ? Number(u.port) : u.protocol === "https:" ? 443 : 80;
|
||||||
|
const host = cleanHost(u.hostname);
|
||||||
|
out.add(`web:${host}:${port}`);
|
||||||
|
if (u.protocol === "https:") out.add(`tls:${host}:${port}`);
|
||||||
|
} catch {
|
||||||
|
// an address that isn't one covers nothing
|
||||||
|
}
|
||||||
|
} else if (c.kind === "tls" && c.port !== null) out.add(`tls:${cleanHost(c.target)}:${c.port}`);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Pure: the checks worth offering, from what is known. */
|
||||||
|
export function suggestChecks(input: SuggestionInput): { suggestions: Suggestion[]; dismissed: Suggestion[] } {
|
||||||
|
const covered = coverage(input.existing);
|
||||||
|
const all = new Map<string, Suggestion>();
|
||||||
|
|
||||||
|
for (const s of input.servers) {
|
||||||
|
const ips = new Set(s.ips.map((i) => i.trim().toLowerCase()));
|
||||||
|
const hostname = s.hostname ? cleanHost(s.hostname) : null;
|
||||||
|
// Names that point at this server in DNS: an address record with one of its addresses, and an alias to one of those names or to its hostname.
|
||||||
|
const direct = new Set<string>();
|
||||||
|
for (const r of input.dns) {
|
||||||
|
if ((r.type === "A" || r.type === "AAAA") && ips.has(r.content.trim().toLowerCase())) direct.add(cleanHost(r.name));
|
||||||
|
}
|
||||||
|
const names = new Set(direct);
|
||||||
|
for (const r of input.dns) {
|
||||||
|
if (r.type !== "CNAME") continue;
|
||||||
|
const to = cleanHost(r.content);
|
||||||
|
if (direct.has(to) || (hostname !== null && to === hostname)) names.add(cleanHost(r.name));
|
||||||
|
}
|
||||||
|
const dnsNames = [...names].filter((n) => n.includes(".") && !n.startsWith("*") && !n.endsWith(".arpa") && !/\s/.test(n)).sort((a, b) => a.length - b.length || a.localeCompare(b));
|
||||||
|
|
||||||
|
const listening = new Set(s.ports.filter((p) => (p.protocol ?? "tcp") === "tcp" && !(p.address && isLoopback(p.address))).map((p) => p.port));
|
||||||
|
for (const port of Object.keys(WEB_PORTS).map(Number).filter((p) => listening.has(p))) {
|
||||||
|
// Plain and secure on the same server: the secure one is the real service (the plain one is most likely its redirect), and reads the certificate.
|
||||||
|
if (SECURE_TWIN[port] !== undefined && listening.has(SECURE_TWIN[port])) continue;
|
||||||
|
const scheme = WEB_PORTS[port];
|
||||||
|
const fallback = hostname && hostname.includes(".") ? [hostname] : [s.ips.map((i) => i.trim()).find(usableIPv4)].filter((x): x is string => !!x);
|
||||||
|
const hosts = dnsNames.length > 0 ? dnsNames.slice(0, MAX_NAMES_PER_PORT) : fallback;
|
||||||
|
for (const host of hosts) {
|
||||||
|
const defaultPort = scheme === "https" ? 443 : 80;
|
||||||
|
const target = `${scheme}://${host}${port === defaultPort ? "" : `:${port}`}/`;
|
||||||
|
if (covered.has(`web:${host}:${port}`)) continue;
|
||||||
|
const viaDns = dnsNames.includes(host);
|
||||||
|
all.set(`http|${target}`, {
|
||||||
|
key: `http|${target}`,
|
||||||
|
kind: "http",
|
||||||
|
name: `${host}${port === defaultPort ? "" : `:${port}`}`,
|
||||||
|
target,
|
||||||
|
port: null,
|
||||||
|
serverId: s.id,
|
||||||
|
serverName: s.name,
|
||||||
|
reason: `${s.name} is listening on port ${port}${viaDns ? `, and ${host} points at it in DNS` : ""}.`,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const sec of input.secrets) {
|
||||||
|
if (!sec.checkHost || sec.checkPort === null) continue;
|
||||||
|
const host = cleanHost(sec.checkHost);
|
||||||
|
if (covered.has(`tls:${host}:${sec.checkPort}`)) continue;
|
||||||
|
all.set(`tls|${host}|${sec.checkPort}`, {
|
||||||
|
key: `tls|${host}|${sec.checkPort}`,
|
||||||
|
kind: "tls",
|
||||||
|
name: `Certificate: ${sec.name}`,
|
||||||
|
target: host,
|
||||||
|
port: sec.checkPort,
|
||||||
|
serverId: null,
|
||||||
|
serverName: null,
|
||||||
|
reason: `The secret “${sec.name}” already reads this certificate; a check also tells you when ${host} stops answering.`,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// By server, those about no server last, then by address.
|
||||||
|
const sorted = [...all.values()].sort((a, b) => Number(a.serverName === null) - Number(b.serverName === null) || (a.serverName ?? "").localeCompare(b.serverName ?? "") || a.target.localeCompare(b.target));
|
||||||
|
return {
|
||||||
|
suggestions: sorted.filter((x) => !input.dismissed.has(x.key)).slice(0, MAX_SUGGESTIONS),
|
||||||
|
dismissed: sorted.filter((x) => input.dismissed.has(x.key)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── What was said no to ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const DISMISSED_FLAG = "endpointSuggestionsDismissed";
|
||||||
|
const MAX_DISMISSED = 500;
|
||||||
|
|
||||||
|
export async function loadDismissed(): Promise<Set<string>> {
|
||||||
|
const raw = await getInternalFlag(DISMISSED_FLAG);
|
||||||
|
if (!raw) return new Set();
|
||||||
|
try {
|
||||||
|
const v = JSON.parse(raw);
|
||||||
|
return new Set(Array.isArray(v) ? v.filter((k): k is string => typeof k === "string") : []);
|
||||||
|
} catch {
|
||||||
|
return new Set();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function saveDismissed(keys: Set<string>): Promise<void> {
|
||||||
|
await setInternalFlag(DISMISSED_FLAG, JSON.stringify([...keys].slice(-MAX_DISMISSED)));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Reading what is stored ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function jsonArray<T>(stored: string | null): T[] {
|
||||||
|
if (!stored) return [];
|
||||||
|
try {
|
||||||
|
const v = JSON.parse(stored);
|
||||||
|
return Array.isArray(v) ? (v as T[]) : [];
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function loadSuggestions(): Promise<{ suggestions: Suggestion[]; dismissed: Suggestion[] }> {
|
||||||
|
const [serverRows, dnsRows, secretRows, existing, dismissed] = await Promise.all([
|
||||||
|
db.select({ id: servers.id, name: servers.name, hostname: servers.hostname, ipAddresses: servers.ipAddresses, listeningPorts: servers.listeningPorts }).from(servers),
|
||||||
|
db.select({ name: dnsRecordsCache.name, type: dnsRecordsCache.type, content: dnsRecordsCache.content }).from(dnsRecordsCache),
|
||||||
|
db.select({ name: secrets.name, checkHost: secrets.checkHost, checkPort: secrets.checkPort }).from(secrets),
|
||||||
|
db.select({ kind: endpointChecks.kind, target: endpointChecks.target, port: endpointChecks.port }).from(endpointChecks),
|
||||||
|
loadDismissed(),
|
||||||
|
]);
|
||||||
|
return suggestChecks({
|
||||||
|
servers: serverRows.map((s) => ({ id: s.id, name: s.name, hostname: s.hostname, ips: jsonArray<string>(s.ipAddresses).filter((i) => typeof i === "string"), ports: jsonArray<{ protocol?: string; port: number; address?: string }>(s.listeningPorts).filter((p) => p && Number.isInteger(p.port)) })),
|
||||||
|
dns: dnsRows,
|
||||||
|
secrets: secretRows,
|
||||||
|
existing,
|
||||||
|
dismissed,
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
/**
|
||||||
|
* Heartbeats: "did my 3 a.m. job actually run?". A cron job or backup script calls an address when it finishes; the app writes down that it did, and
|
||||||
|
* speaks up — an alert and a notification, like any other problem — when the call doesn't come in time, or when the job says it failed.
|
||||||
|
*
|
||||||
|
* Each heartbeat has an address with a long random token in it (`/api/heartbeat/<token>`) and a period: how often the job should report, plus a grace
|
||||||
|
* period for jobs that run a little late. Three calls: `…/<token>` (it worked), `…/<token>/start` (it began, so the next call can say how long it took)
|
||||||
|
* and `…/<token>/fail` (it ran and went wrong — this is an alert at once, without waiting for the time to pass). Anything after the call, in the
|
||||||
|
* body or in `?msg=`, is kept as a short note ("3.2 GB, 41 s").
|
||||||
|
*
|
||||||
|
* The token is the whole of the access a call needs, and all it allows is to say "I'm alive": it can't read anything. It is kept as it is, not hashed
|
||||||
|
* (unlike a share link), because the address has to be shown again when somebody sets the job up on another machine.
|
||||||
|
*/
|
||||||
|
import { randomBytes } from "node:crypto";
|
||||||
|
import { and, asc, desc, eq, inArray, lt } from "drizzle-orm";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { heartbeatPings, heartbeats } from "../db/schema.js";
|
||||||
|
import { invalidateAlertsCache } from "./alerts.js";
|
||||||
|
|
||||||
|
export type HeartbeatRow = typeof heartbeats.$inferSelect;
|
||||||
|
export type PingKind = "ok" | "fail" | "start";
|
||||||
|
export const PING_KINDS: readonly PingKind[] = ["ok", "fail", "start"];
|
||||||
|
|
||||||
|
export const KEEP_PING_DAYS = 14;
|
||||||
|
export const MAX_MESSAGE = 200;
|
||||||
|
/** "hb_" and then random letters and digits: what an address is allowed to carry. */
|
||||||
|
export const TOKEN_PATTERN = /^hb_[A-Za-z0-9_-]{20,64}$/;
|
||||||
|
const MIN_PERIOD = 1;
|
||||||
|
export const MAX_PERIOD_MINUTES = 60 * 24 * 30;
|
||||||
|
export const MAX_GRACE_MINUTES = 60 * 24;
|
||||||
|
/** A "started" call that is older than this is not what a later "finished" call is timing. */
|
||||||
|
const MAX_RUN_MS = 7 * 86_400_000;
|
||||||
|
|
||||||
|
export const newToken = () => `hb_${randomBytes(18).toString("base64url")}`;
|
||||||
|
|
||||||
|
/** SQLite's own timestamps ("2026-10-19 03:10:00") are UTC without a "T"; ours are ISO. Both read as the instant they are. */
|
||||||
|
export function parseStamp(value: string): number {
|
||||||
|
return Date.parse(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/.test(value) ? `${value.replace(" ", "T")}Z` : value);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A note from a job, made fit to store and show: one line, no control characters, not long. */
|
||||||
|
export function cleanMessage(raw: unknown): string | null {
|
||||||
|
if (typeof raw !== "string") return null;
|
||||||
|
// eslint-disable-next-line no-control-regex
|
||||||
|
const text = raw.replace(/[\u0000-\u001f\u007f]+/g, " ").replace(/\s+/g, " ").trim();
|
||||||
|
return text === "" ? null : text.slice(0, MAX_MESSAGE);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── What a heartbeat's state is ───────────────────────────────────────────
|
||||||
|
|
||||||
|
export type HeartbeatState = "off" | "waiting" | "up" | "running" | "late" | "failed";
|
||||||
|
|
||||||
|
/** When the next call is due by, counting the grace: from the last call, or from the day it was made if it has never called. */
|
||||||
|
export function dueBy(h: Pick<HeartbeatRow, "periodMinutes" | "graceMinutes" | "lastPingAt" | "createdAt">): Date {
|
||||||
|
const base = h.lastPingAt ? parseStamp(h.lastPingAt) : parseStamp(h.createdAt);
|
||||||
|
return new Date(base + (h.periodMinutes + h.graceMinutes) * 60_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pure: how a heartbeat is doing. A job that said it failed is failed until it next succeeds; otherwise it is late once the time (period and
|
||||||
|
* grace) has passed without a call. A new one that has not called yet is waiting, and is late if its first call is overdue too.
|
||||||
|
*/
|
||||||
|
export function heartbeatState(h: Pick<HeartbeatRow, "enabled" | "periodMinutes" | "graceMinutes" | "lastPingAt" | "lastStatus" | "lastStartAt" | "createdAt">, now: Date): HeartbeatState {
|
||||||
|
if (!h.enabled) return "off";
|
||||||
|
if (h.lastStatus === "fail") return "failed";
|
||||||
|
if (now.getTime() > dueBy(h).getTime()) return "late";
|
||||||
|
if (!h.lastPingAt) return "waiting";
|
||||||
|
if (h.lastStartAt && parseStamp(h.lastStartAt) > parseStamp(h.lastPingAt)) return "running";
|
||||||
|
return "up";
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── What is wrong ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface HeartbeatCondition {
|
||||||
|
key: string;
|
||||||
|
source: "heartbeat";
|
||||||
|
message: string;
|
||||||
|
severity: "critical" | "warning";
|
||||||
|
category: "automation";
|
||||||
|
heartbeatId: number;
|
||||||
|
serverId: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const stamp = (d: Date) => `${d.toISOString().slice(0, 16).replace("T", " ")} UTC`;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The problems the heartbeats stand for. A call that didn't come in time is a warning, and says when it was due (a time, not "N minutes ago", so
|
||||||
|
* the message doesn't change on every pass); a job that reported a failure is critical. The key carries the server a heartbeat is linked to, so
|
||||||
|
* maintenance on it silences them.
|
||||||
|
*/
|
||||||
|
export function heartbeatConditions(rows: HeartbeatRow[], now: Date): HeartbeatCondition[] {
|
||||||
|
const out: HeartbeatCondition[] = [];
|
||||||
|
for (const h of rows) {
|
||||||
|
const state = heartbeatState(h, now);
|
||||||
|
const suffix = h.serverId !== null ? `:server:${h.serverId}` : "";
|
||||||
|
if (state === "late") {
|
||||||
|
const heard = h.lastPingAt ? `last heard ${stamp(new Date(parseStamp(h.lastPingAt)))}` : "it has never reported";
|
||||||
|
out.push({ key: `heartbeat:${h.id}${suffix}`, source: "heartbeat", category: "automation", severity: "warning", heartbeatId: h.id, serverId: h.serverId, message: `${h.name} hasn't reported — it was due by ${stamp(dueBy(h))} (${heard})` });
|
||||||
|
} else if (state === "failed") {
|
||||||
|
out.push({ key: `heartbeat-fail:${h.id}${suffix}`, source: "heartbeat", category: "automation", severity: "critical", heartbeatId: h.id, serverId: h.serverId, message: `${h.name} reported a failure${h.lastMessage ? ` — ${h.lastMessage}` : ""}` });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── A call coming in ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Writes down a call. Returns the heartbeat as it is now, or null when there is no heartbeat with that token. A heartbeat that is switched off
|
||||||
|
* still takes calls (so a script isn't broken by pausing it) and notes them, so that switching it back on starts from a fresh clock.
|
||||||
|
*/
|
||||||
|
export async function recordPing(token: string, kind: PingKind, message: string | null, now: Date = new Date()): Promise<HeartbeatRow | null> {
|
||||||
|
if (!TOKEN_PATTERN.test(token)) return null;
|
||||||
|
const [h] = await db.select().from(heartbeats).where(eq(heartbeats.token, token)).limit(1);
|
||||||
|
if (!h) return null;
|
||||||
|
const iso = now.toISOString();
|
||||||
|
let durationMs: number | null = null;
|
||||||
|
let updated: HeartbeatRow;
|
||||||
|
if (kind === "start") {
|
||||||
|
[updated] = await db.update(heartbeats).set({ lastStartAt: iso }).where(eq(heartbeats.id, h.id)).returning();
|
||||||
|
} else {
|
||||||
|
if (h.lastStartAt) {
|
||||||
|
const ran = now.getTime() - parseStamp(h.lastStartAt);
|
||||||
|
const startedAfterLastEnd = !h.lastPingAt || parseStamp(h.lastStartAt) > parseStamp(h.lastPingAt);
|
||||||
|
if (startedAfterLastEnd && ran >= 0 && ran <= MAX_RUN_MS) durationMs = ran;
|
||||||
|
}
|
||||||
|
[updated] = await db
|
||||||
|
.update(heartbeats)
|
||||||
|
.set({ lastPingAt: iso, lastStatus: kind, lastMessage: message, lastDurationMs: durationMs, pingCount: h.pingCount + 1 })
|
||||||
|
.where(eq(heartbeats.id, h.id))
|
||||||
|
.returning();
|
||||||
|
}
|
||||||
|
await db.insert(heartbeatPings).values({ heartbeatId: h.id, at: iso, kind, durationMs, message });
|
||||||
|
invalidateAlertsCache();
|
||||||
|
return updated;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── History ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface PingSummary {
|
||||||
|
at: string;
|
||||||
|
kind: PingKind;
|
||||||
|
durationMs: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The last calls of each heartbeat that ended a run (success or failure), oldest first. */
|
||||||
|
export async function loadRecent(ids: number[], perHeartbeat = 30): Promise<Map<number, PingSummary[]>> {
|
||||||
|
const out = new Map<number, PingSummary[]>();
|
||||||
|
if (ids.length === 0) return out;
|
||||||
|
const rows = await db
|
||||||
|
.select({ heartbeatId: heartbeatPings.heartbeatId, at: heartbeatPings.at, kind: heartbeatPings.kind, durationMs: heartbeatPings.durationMs })
|
||||||
|
.from(heartbeatPings)
|
||||||
|
.where(and(inArray(heartbeatPings.heartbeatId, ids), inArray(heartbeatPings.kind, ["ok", "fail"])))
|
||||||
|
.orderBy(desc(heartbeatPings.at));
|
||||||
|
for (const r of rows) {
|
||||||
|
const list = out.get(r.heartbeatId) ?? [];
|
||||||
|
if (list.length < perHeartbeat) list.push({ at: r.at, kind: r.kind as PingKind, durationMs: r.durationMs });
|
||||||
|
out.set(r.heartbeatId, list);
|
||||||
|
}
|
||||||
|
for (const list of out.values()) list.reverse();
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function prunePings(now: Date = new Date()): Promise<void> {
|
||||||
|
await db.delete(heartbeatPings).where(lt(heartbeatPings.at, new Date(now.getTime() - KEEP_PING_DAYS * 86_400_000).toISOString()));
|
||||||
|
}
|
||||||
|
|
||||||
|
export const periodLimits = { min: MIN_PERIOD, max: MAX_PERIOD_MINUTES };
|
||||||
|
|
||||||
|
/** The table order every list uses. */
|
||||||
|
export const listHeartbeats = () => db.select().from(heartbeats).orderBy(asc(heartbeats.name));
|
||||||
@@ -25,7 +25,7 @@ export async function listUpcomingWindows(now: Date = new Date()): Promise<Activ
|
|||||||
export function subjectOfConditionKey(key: string): string | null {
|
export function subjectOfConditionKey(key: string): string | null {
|
||||||
let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)(?::|$)/.exec(key);
|
let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)(?::|$)/.exec(key);
|
||||||
if (m) return `server:${m[1]}`;
|
if (m) return `server:${m[1]}`;
|
||||||
m = /^endpoint(?:-cert)?:\d+:server:(\d+)$/.exec(key);
|
m = /^(?:endpoint(?:-cert)?|heartbeat(?:-fail)?):\d+:server:(\d+)$/.exec(key);
|
||||||
if (m) return `server:${m[1]}`;
|
if (m) return `server:${m[1]}`;
|
||||||
m = /^disk:proxmox:(\d+):/.exec(key);
|
m = /^disk:proxmox:(\d+):/.exec(key);
|
||||||
if (m) return `integration:${m[1]}`;
|
if (m) return `integration:${m[1]}`;
|
||||||
|
|||||||
@@ -4,8 +4,9 @@
|
|||||||
* integration on its own, so a scrape every fifteen seconds costs a few database reads (the alerts are the app's own minute-old result).
|
* integration on its own, so a scrape every fifteen seconds costs a few database reads (the alerts are the app's own minute-old result).
|
||||||
*/
|
*/
|
||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { assets, endpointChecks, integrations, maintenanceWindows, secrets, servers } from "../db/schema.js";
|
import { assets, endpointChecks, heartbeats, integrations, maintenanceWindows, secrets, servers } from "../db/schema.js";
|
||||||
import { latestTests } from "./restoreTests.js";
|
import { latestTests } from "./restoreTests.js";
|
||||||
|
import { heartbeatState, parseStamp } from "./heartbeats.js";
|
||||||
import { env } from "../env.js";
|
import { env } from "../env.js";
|
||||||
import { getAlerts } from "./alerts.js";
|
import { getAlerts } from "./alerts.js";
|
||||||
import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js";
|
import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js";
|
||||||
@@ -70,6 +71,18 @@ async function addEndpointMetrics(out: Exposition, now: Date) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Jobs that report in, for /metrics. */
|
||||||
|
async function addHeartbeatMetrics(out: Exposition, now: Date) {
|
||||||
|
for (const h of await db.select().from(heartbeats)) {
|
||||||
|
if (!h.enabled) continue;
|
||||||
|
const labels = { name: h.name };
|
||||||
|
const state = heartbeatState(h, now);
|
||||||
|
out.add("homelab_heartbeat_up", "gauge", "1 if the job has reported in time and not said it failed, else 0.", state === "late" || state === "failed" ? 0 : 1, labels);
|
||||||
|
if (h.lastPingAt) out.add("homelab_heartbeat_last_ping_timestamp_seconds", "gauge", "When the job last reported that it finished (Unix time).", Math.floor(parseStamp(h.lastPingAt) / 1000), labels);
|
||||||
|
if (h.lastDurationMs !== null) out.add("homelab_heartbeat_last_duration_seconds", "gauge", "How long the job said its last run took.", Math.round(h.lastDurationMs / 100) / 10, labels);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export async function buildMetrics(now: Date = new Date()): Promise<string> {
|
export async function buildMetrics(now: Date = new Date()): Promise<string> {
|
||||||
const started = Date.now();
|
const started = Date.now();
|
||||||
const out = new Exposition();
|
const out = new Exposition();
|
||||||
@@ -195,5 +208,6 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
|
|||||||
out.add("homelab_scrape_duration_seconds", "gauge", "How long building this answer took.", (Date.now() - started) / 1000);
|
out.add("homelab_scrape_duration_seconds", "gauge", "How long building this answer took.", (Date.now() - started) / 1000);
|
||||||
out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec);
|
out.add("homelab_scrape_timestamp_seconds", "gauge", "When this answer was built, as a Unix time.", nowSec);
|
||||||
await addEndpointMetrics(out, now);
|
await addEndpointMetrics(out, now);
|
||||||
|
await addHeartbeatMetrics(out, now);
|
||||||
return out.toString();
|
return out.toString();
|
||||||
}
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/**
|
/**
|
||||||
* The name lists behind Operations → Generator's server-name picker.
|
* The name lists behind Administration → Generator's server-name picker.
|
||||||
*
|
*
|
||||||
* The built-in lists below are hand-picked, not taken from any official source — they're a starting point. Admins can
|
* The built-in lists below are hand-picked, not taken from any official source — they're a starting point. Admins can
|
||||||
* edit every list under Settings → Names, and can replace a Swedish list with real statistics from Skatteverket.
|
* edit every list under Settings → Names, and can replace a Swedish list with real statistics from Skatteverket.
|
||||||
|
|||||||
@@ -109,8 +109,9 @@ export function metaOfConditionKey(key: string, severity?: string): NotifyMeta {
|
|||||||
else if (key.startsWith("failed-units:")) category = "services";
|
else if (key.startsWith("failed-units:")) category = "services";
|
||||||
else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk";
|
else if (/^(disk|smart|zfs|synology-volume|synology-disk):/.test(key)) category = "disk";
|
||||||
else if (key.startsWith("automation:")) category = "automation";
|
else if (key.startsWith("automation:")) category = "automation";
|
||||||
|
else if (key.startsWith("heartbeat")) category = "automation";
|
||||||
else if (key.startsWith("endpoint-cert:")) category = "expiry";
|
else if (key.startsWith("endpoint-cert:")) category = "expiry";
|
||||||
else if (key.startsWith("endpoint:")) category = "monitoring";
|
else if (key.startsWith("endpoint:")) category = "monitoring";
|
||||||
const server = /^(?:(?:offline|disk|failed-units|smart|zfs):server:|endpoint(?:-cert)?:\d+:server:)(\d+)/.exec(key);
|
const server = /^(?:(?:offline|disk|failed-units|smart|zfs):server:|(?:endpoint(?:-cert)?|heartbeat(?:-fail)?):\d+:server:)(\d+)/.exec(key);
|
||||||
return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null };
|
return { category, severity: severity === "critical" ? "critical" : "warning", serverId: server ? Number(server[1]) : null };
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,300 @@
|
|||||||
|
/**
|
||||||
|
* "Coming up": what the agenda gathers and how it dates it, the calendar feed it can be written as, and the routes — the list for people signed in
|
||||||
|
* and the feed behind a share link.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import type { Server } from "node:http";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-agenda-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, secrets, domains, assets, endpointChecks, maintenanceWindows, maintenanceSchedules, restoreTests } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { agendaFromData, buildIcs, foldLine, localDate, addDays, buildAgenda, clearAgendaCacheForTests } = await import("../src/services/agenda.js");
|
||||||
|
const { agendaRouter } = await import("../src/routes/agenda.js");
|
||||||
|
const { publicCalendarRouter } = await import("../src/routes/publicCalendar.js");
|
||||||
|
const { shareLinksRouter } = await import("../src/routes/shareLinks.js");
|
||||||
|
const { createShareLink } = await import("../src/services/shareLinks.js");
|
||||||
|
const { updateSettings } = await import("../src/services/settingsStore.js");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
type Data = Parameters<typeof agendaFromData>[0];
|
||||||
|
const NOW = new Date("2026-10-19T12:00:00Z");
|
||||||
|
const empty = (extra: Partial<Data> = {}): Data => ({ secrets: [], domains: [], endpoints: [], tailscale: [], assets: [], windows: [], schedules: [], restoreTests: [], restoreLimitDays: 0, updates: [], notes: [], ...extra });
|
||||||
|
const titles = (data: Data, days?: number, tz = "UTC") => agendaFromData(data, NOW, tz, days).items.map((i) => [i.daysLeft, i.kind, i.title]);
|
||||||
|
|
||||||
|
describe("what is gathered, and how it is dated", () => {
|
||||||
|
test("a day that something expires on, with how many days away it is; a secret that is a certificate is a certificate", () => {
|
||||||
|
const data = empty({
|
||||||
|
secrets: [
|
||||||
|
{ id: 1, name: "API key", type: "api_key", expiryDate: "2026-10-30" },
|
||||||
|
{ id: 2, name: "Wildcard", type: "ssl_certificate", expiryDate: "2026-11-02" },
|
||||||
|
{ id: 3, name: "Never", type: "generic", expiryDate: null },
|
||||||
|
{ id: 4, name: "Garbage", type: "generic", expiryDate: "2026-02-30" },
|
||||||
|
],
|
||||||
|
domains: [{ id: 1, name: "example.se", expiresAt: "2027-01-05" }],
|
||||||
|
});
|
||||||
|
assert.deepEqual(titles(data), [
|
||||||
|
[11, "secret", "Secret “API key” expires"],
|
||||||
|
[14, "certificate", "Certificate “Wildcard” expires"],
|
||||||
|
[78, "domain", "Domain example.se expires"],
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("what has just passed is shown as passed, for thirty days; what is further off than asked for is not shown", () => {
|
||||||
|
const data = empty({ secrets: [
|
||||||
|
{ id: 1, name: "Yesterday", type: "generic", expiryDate: "2026-10-18" },
|
||||||
|
{ id: 2, name: "Today", type: "generic", expiryDate: "2026-10-19" },
|
||||||
|
{ id: 3, name: "Last month", type: "generic", expiryDate: "2026-09-19" },
|
||||||
|
{ id: 4, name: "Long ago", type: "generic", expiryDate: "2026-09-18" },
|
||||||
|
{ id: 5, name: "In a year", type: "generic", expiryDate: "2027-10-19" },
|
||||||
|
] });
|
||||||
|
assert.deepEqual(titles(data, 90), [[-30, "secret", "Secret “Last month” expired"], [-1, "secret", "Secret “Yesterday” expired"], [0, "secret", "Secret “Today” expires"]]);
|
||||||
|
assert.equal(titles(data, 400).length, 4, "a year ahead, when asked for");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("certificates that endpoint checks have seen, hardware warranties and end of support — each linking to where they are kept", () => {
|
||||||
|
const agenda = agendaFromData(
|
||||||
|
empty({
|
||||||
|
endpoints: [
|
||||||
|
{ id: 1, name: "Gitea", enabled: true, certExpiresAt: "2026-11-01T08:00:00.000Z", serverId: 5 },
|
||||||
|
{ id: 2, name: "Paused", enabled: false, certExpiresAt: "2026-11-01T08:00:00.000Z", serverId: null },
|
||||||
|
{ id: 3, name: "No cert", enabled: true, certExpiresAt: null, serverId: null },
|
||||||
|
],
|
||||||
|
assets: [{ id: 9, name: "Synology DS920+", warrantyUntil: "2026-12-01", endOfSupport: "2026-12-20", serverId: 2 }],
|
||||||
|
}),
|
||||||
|
NOW,
|
||||||
|
"UTC",
|
||||||
|
);
|
||||||
|
assert.deepEqual(agenda.items.map((i) => [i.kind, i.title, i.link, i.serverId]), [
|
||||||
|
["certificate", "The certificate for Gitea expires", "/endpoints", 5],
|
||||||
|
["warranty", "Warranty on Synology DS920+ ends", "/inventory?asset=9", 2],
|
||||||
|
["support", "Support for Synology DS920+ ends", "/inventory?asset=9", 2],
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("Tailscale keys: only those the integration says are expiring, dated by days left", () => {
|
||||||
|
assert.deepEqual(titles(empty({ tailscale: [{ integrationId: 1, integrationName: "Home", deviceLabel: "laptop", daysLeft: 6 }] })), [[6, "tailscale", "Tailscale key for laptop expires"]]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a restore test is due a set time after the last good one — only with a limit set, and never for one that failed", () => {
|
||||||
|
const tests = [
|
||||||
|
{ subjectKey: "guest:1:100", label: "elsa", testedAt: "2026-08-01", result: "ok" },
|
||||||
|
{ subjectKey: "guest:1:101", label: "anna", testedAt: "2026-08-01", result: "failed" },
|
||||||
|
{ subjectKey: "custom:photos", label: "Photos share", testedAt: "2026-07-03", result: "partial" },
|
||||||
|
];
|
||||||
|
assert.deepEqual(titles(empty({ restoreTests: tests, restoreLimitDays: 0 })), []);
|
||||||
|
assert.deepEqual(titles(empty({ restoreTests: tests, restoreLimitDays: 90 })), [[-18, "restore", "Restore test was due: Photos share"], [11, "restore", "Restore test due: elsa"]]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("updates that are waiting are one line, from the longest wait", () => {
|
||||||
|
const agenda = agendaFromData(empty({ updates: [{ name: "a", count: 5, security: 2, since: "2026-10-10T08:00:00.000Z" }, { name: "b", count: 1, security: 0, since: "2026-10-17T08:00:00.000Z" }, { name: "c", count: 3, security: null, since: null }] }), NOW, "UTC");
|
||||||
|
assert.equal(agenda.items.length, 1);
|
||||||
|
assert.deepEqual([agenda.items[0].title, agenda.items[0].detail, agenda.items[0].daysLeft, agenda.items[0].link], ["Updates waiting on 3 servers (1 with security fixes)", "the longest wait is 9 days", -9, "/updates"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("maintenance that is set up is grouped by when it is, the names kept short; windows a schedule made are left to the schedule, which is read forward", () => {
|
||||||
|
const agenda = agendaFromData(
|
||||||
|
empty({
|
||||||
|
windows: [
|
||||||
|
{ startedAt: "2026-10-24T01:00:00.000Z", endsAt: "2026-10-24T03:00:00.000Z", reason: "Firmware", source: null, targetName: "nas" },
|
||||||
|
{ startedAt: "2026-10-24T01:00:00.000Z", endsAt: "2026-10-24T03:00:00.000Z", reason: "Firmware", source: null, targetName: "switch" },
|
||||||
|
{ startedAt: "2026-10-21T02:00:00.000Z", endsAt: "2026-10-21T04:00:00.000Z", reason: "Patch night", source: "schedule:1:2026-10-21", targetName: "nas" },
|
||||||
|
],
|
||||||
|
schedules: [{ id: 1, name: "Patch night", reason: null, days: JSON.stringify([3]), startTime: "02:00", durationMinutes: 120, timezone: "UTC", targetNames: ["nas", "elsa"] }],
|
||||||
|
}),
|
||||||
|
NOW,
|
||||||
|
"UTC",
|
||||||
|
30,
|
||||||
|
);
|
||||||
|
const m = agenda.items.filter((i) => i.kind === "maintenance");
|
||||||
|
assert.deepEqual(m.map((i) => [i.title, i.at.slice(0, 16), i.allDay, i.endsAt?.slice(11, 16)]), [
|
||||||
|
["Recurring maintenance: Patch night", "2026-10-21T02:00", false, "04:00"],
|
||||||
|
["Maintenance: nas, switch", "2026-10-24T01:00", false, "03:00"],
|
||||||
|
["Recurring maintenance: Patch night", "2026-10-28T02:00", false, "04:00"],
|
||||||
|
["Recurring maintenance: Patch night", "2026-11-04T02:00", false, "04:00"],
|
||||||
|
["Recurring maintenance: Patch night", "2026-11-11T02:00", false, "04:00"],
|
||||||
|
["Recurring maintenance: Patch night", "2026-11-18T02:00", false, "04:00"],
|
||||||
|
]);
|
||||||
|
assert.equal(m.filter((i) => i.at.startsWith("2026-10-21")).length, 1, "the window the schedule made and the schedule itself are one entry, not two");
|
||||||
|
assert.equal(m[1].detail, "Firmware");
|
||||||
|
assert.equal(m[0].detail, "elsa, nas");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a day is the day on the clock set for notifications: a window at 23:30 UTC is already tomorrow in Stockholm", () => {
|
||||||
|
const windows = [{ startedAt: "2026-10-19T23:30:00.000Z", endsAt: "2026-10-20T01:00:00.000Z", reason: null, source: null, targetName: "nas" }];
|
||||||
|
assert.equal(agendaFromData(empty({ windows }), NOW, "UTC").items[0].daysLeft, 0);
|
||||||
|
assert.equal(agendaFromData(empty({ windows }), NOW, "Europe/Stockholm").items[0].daysLeft, 1);
|
||||||
|
assert.equal(localDate(new Date("2026-10-19T23:30:00Z"), "Europe/Stockholm"), "2026-10-20");
|
||||||
|
assert.equal(addDays("2026-10-31", 1), "2026-11-01");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the calendar feed", () => {
|
||||||
|
const data = empty({
|
||||||
|
secrets: [{ id: 1, name: "API key, with; odd\\chars", type: "api_key", expiryDate: "2026-10-30" }],
|
||||||
|
tailscale: [{ integrationId: 1, integrationName: "Home", deviceLabel: "bobs-laptop", daysLeft: 6 }],
|
||||||
|
windows: [{ startedAt: "2026-10-24T01:00:00.000Z", endsAt: "2026-10-24T03:00:00.000Z", reason: "Firmware", source: null, targetName: "nas" }],
|
||||||
|
secretsOld: undefined,
|
||||||
|
updates: [{ name: "a", count: 1, security: 0, since: null }],
|
||||||
|
} as unknown as Partial<Data>);
|
||||||
|
const agenda = agendaFromData(data, NOW, "UTC");
|
||||||
|
const ics = buildIcs(agenda, { name: "Homelab; manager", detail: "names", baseUrl: "https://hm.lan/" });
|
||||||
|
|
||||||
|
test("it is a calendar with CRLF lines: all-day events for days, a start and an end for maintenance, nothing for waiting updates", () => {
|
||||||
|
assert.ok(ics.startsWith("BEGIN:VCALENDAR\r\nVERSION:2.0\r\n") && ics.endsWith("END:VCALENDAR\r\n"));
|
||||||
|
assert.equal(ics.includes("\n") && ics.split("\n").every((l, i, a) => i === a.length - 1 || l.endsWith("\r")), true);
|
||||||
|
assert.equal((ics.match(/BEGIN:VEVENT/g) ?? []).length, 3, "the secret, the key and the window — not the updates");
|
||||||
|
assert.match(ics, /DTSTART;VALUE=DATE:20261030\r\nDTEND;VALUE=DATE:20261031/);
|
||||||
|
assert.match(ics, /DTSTART:20261024T010000Z\r\nDTEND:20261024T030000Z/);
|
||||||
|
assert.match(ics, /X-WR-CALNAME:Homelab\\; manager/);
|
||||||
|
assert.match(ics, /TRANSP:TRANSPARENT/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("text is escaped, and a line is never more than 75 bytes, without splitting a character", () => {
|
||||||
|
assert.match(ics.replace(/\r\n /g, ""), /SUMMARY:Secret “API key\\, with\\; odd\\\\chars” expires/);
|
||||||
|
const long = foldLine(`SUMMARY:${"å".repeat(100)}`);
|
||||||
|
for (const l of long.split("\r\n")) assert.ok(Buffer.byteLength(l, "utf8") <= 75, `${Buffer.byteLength(l, "utf8")} bytes`);
|
||||||
|
assert.equal(long.replace(/\r\n /g, ""), `SUMMARY:${"å".repeat(100)}`, "and it unfolds to what it was");
|
||||||
|
for (const l of ics.split("\r\n")) assert.ok(Buffer.byteLength(l, "utf8") <= 75);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("with names, an event carries the name and a link back; each has a steady id that does not give the name away", () => {
|
||||||
|
const unfolded = ics.replace(/\r\n /g, "");
|
||||||
|
assert.match(unfolded, /DESCRIPTION:Firmware\\nhttps:\/\/hm\.lan\/maintenance/);
|
||||||
|
const uids = [...ics.matchAll(/UID:(\S+)\r\n/g)].map((m) => m[1]);
|
||||||
|
assert.equal(new Set(uids).size, 3);
|
||||||
|
assert.ok(uids.every((u) => /^[0-9a-f]{24}@homelab-manager$/.test(u)));
|
||||||
|
assert.deepEqual(uids, [...buildIcs(agenda, { name: "x", detail: "names" }).matchAll(/UID:(\S+)\r\n/g)].map((m) => m[1]), "the same on every fetch, or a subscribed calendar would duplicate everything");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("with no names, an event says only what kind of thing it is — and nothing in it gives away a name", () => {
|
||||||
|
const plain = buildIcs(agenda, { name: "Homelab", detail: "summary", baseUrl: "https://hm.lan" });
|
||||||
|
const unfolded = plain.replace(/\r\n /g, "");
|
||||||
|
assert.match(unfolded, /SUMMARY:A secret expires/);
|
||||||
|
assert.match(unfolded, /SUMMARY:A Tailscale key expires/);
|
||||||
|
assert.match(unfolded, /SUMMARY:Planned maintenance/);
|
||||||
|
assert.equal(/API key|bobs-laptop|nas|Firmware|hm\.lan|DESCRIPTION/.test(unfolded), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an event more than a week old is dropped, one from this week is kept", () => {
|
||||||
|
const old = agendaFromData(empty({ secrets: [{ id: 1, name: "Old", type: "generic", expiryDate: "2026-10-10" }, { id: 2, name: "Recent", type: "generic", expiryDate: "2026-10-15" }] }), NOW, "UTC");
|
||||||
|
assert.equal((buildIcs(old, { name: "x", detail: "names" }).match(/BEGIN:VEVENT/g) ?? []).length, 1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("from what is stored, through the routes", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "admin";
|
||||||
|
const day = (n: number) => new Date(Date.now() + n * 86_400_000).toISOString().slice(0, 10);
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "admin", name: "Ada", role: "admin" },
|
||||||
|
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
|
||||||
|
]);
|
||||||
|
const [s] = await db
|
||||||
|
.insert(servers)
|
||||||
|
.values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", pendingUpdates: JSON.stringify({ manager: "apt", count: 4, security: 1, rebootRequired: false, reportedAt: new Date().toISOString() }), updatesPendingSince: new Date(Date.now() - 3 * 86_400_000).toISOString() })
|
||||||
|
.returning({ id: servers.id });
|
||||||
|
await db.insert(secrets).values({ name: "Deploy key", type: "api_key", expiryDate: day(10) } as never);
|
||||||
|
await db.insert(domains).values({ name: "example.se", expiresAt: day(40) });
|
||||||
|
await db.insert(assets).values({ name: "Switch", kind: "network", warrantyUntil: day(20), createdAt: new Date().toISOString(), updatedAt: new Date().toISOString() } as never);
|
||||||
|
await db.insert(endpointChecks).values({ name: "Gitea", kind: "http", target: "https://git.lab/", certExpiresAt: new Date(Date.now() + 5 * 86_400_000).toISOString(), serverId: s.id });
|
||||||
|
await db.insert(maintenanceWindows).values({ targetType: "server", targetId: s.id, reason: "Disk swap", startedAt: new Date(Date.now() + 2 * 86_400_000).toISOString(), endsAt: new Date(Date.now() + 2 * 86_400_000 + 3600_000).toISOString() });
|
||||||
|
await db.insert(maintenanceSchedules).values({ name: "Patch night", targets: JSON.stringify([{ type: "server", id: s.id }]), days: JSON.stringify([0, 1, 2, 3, 4, 5, 6]), startTime: "03:00", durationMinutes: 60, timezone: "UTC" });
|
||||||
|
await db.insert(restoreTests).values({ subjectKey: "custom:photos", label: "Photos share", testedAt: day(-80), result: "ok" });
|
||||||
|
await updateSettings({ healthChecks: { restoreTestDays: 90 } });
|
||||||
|
clearAgendaCacheForTests();
|
||||||
|
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use("/api/public/calendar", publicCalendarRouter);
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/agenda", agendaRouter);
|
||||||
|
a.use("/api/share-links", shareLinksRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const x = a.listen(0, "127.0.0.1", () => resolve(x));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("everything with a date, from every source, for anyone signed in; the days asked for narrow it", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
const res = await fetch(`${base}/api/agenda`);
|
||||||
|
assert.equal(res.status, 200);
|
||||||
|
const body = await res.json();
|
||||||
|
const kinds = new Set(body.items.map((i: { kind: string }) => i.kind));
|
||||||
|
for (const k of ["secret", "domain", "warranty", "certificate", "maintenance", "restore", "updates"]) assert.ok(kinds.has(k), k);
|
||||||
|
assert.ok(body.items.some((i: { title: string }) => i.title === "Restore test due: Photos share"), "tested 80 days ago with a 90-day limit");
|
||||||
|
assert.ok(body.items.filter((i: { title: string }) => i.title.startsWith("Recurring maintenance")).length >= 60 / 1, "a schedule is read forward, not only the week that has windows");
|
||||||
|
const near = await (await fetch(`${base}/api/agenda?days=15`)).json();
|
||||||
|
assert.equal(near.items.some((i: { kind: string }) => i.kind === "domain"), false, "40 days out is beyond 15");
|
||||||
|
assert.equal(near.items.some((i: { kind: string }) => i.kind === "secret"), true);
|
||||||
|
const bad = await (await fetch(`${base}/api/agenda?days=banana`)).json();
|
||||||
|
assert.equal(bad.items.length > 0, true, "an unreadable range is the default one");
|
||||||
|
const direct = await buildAgenda();
|
||||||
|
assert.equal(direct.items.length, body.items.length, "the route and the service agree");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a calendar link is made like any share link, by an administrator, and its address is the feed", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
assert.equal((await fetch(`${base}/api/share-links`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ label: "Phone", kind: "calendar", detail: "names", expiresInDays: null }) })).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
const res = await fetch(`${base}/api/share-links`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ label: "Phone", kind: "calendar", detail: "names", expiresInDays: null }) });
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
const made = await res.json();
|
||||||
|
assert.match(made.path, /^\/api\/public\/calendar\/[A-Za-z0-9_-]{20,}\.ics$/);
|
||||||
|
const feed = await fetch(`${base}${made.path}`);
|
||||||
|
assert.equal(feed.status, 200);
|
||||||
|
assert.match(feed.headers.get("content-type") ?? "", /^text\/calendar; charset=utf-8/);
|
||||||
|
const text = (await feed.text()).replace(/\r\n /g, "");
|
||||||
|
assert.ok(text.startsWith("BEGIN:VCALENDAR"));
|
||||||
|
assert.match(text, /SUMMARY:Secret “Deploy key” expires/);
|
||||||
|
assert.match(text, /SUMMARY:Domain example\.se expires/);
|
||||||
|
const bare = made.path.replace(/\.ics$/, "");
|
||||||
|
assert.equal((await fetch(`${base}${bare}`)).status, 200, "the extension is for calendar apps; the address works without it");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a link for no names gives the feed no names; one for something else, a wrong one and an ended one all say the same thing", async () => {
|
||||||
|
const { token } = await createShareLink({ label: "Private", kind: "calendar", detail: "summary", expiresInDays: null, createdBy: "t" });
|
||||||
|
const text = (await (await fetch(`${base}/api/public/calendar/${token}.ics`)).text()).replace(/\r\n /g, "");
|
||||||
|
assert.match(text, /SUMMARY:A secret expires/);
|
||||||
|
assert.equal(/Deploy key|example\.se|Gitea|nas/.test(text), false);
|
||||||
|
|
||||||
|
const wall = await createShareLink({ label: "Wall", kind: "wall", detail: "summary", expiresInDays: null, createdBy: "t" });
|
||||||
|
const ended = await createShareLink({ label: "Old", kind: "calendar", detail: "names", expiresInDays: 1, createdBy: "t", now: new Date(Date.now() - 3 * 86_400_000) });
|
||||||
|
const answers = await Promise.all([`${wall.token}.ics`, "nonsense.ics", `${"a".repeat(30)}.ics`, `${ended.token}.ics`].map(async (p) => {
|
||||||
|
const r = await fetch(`${base}/api/public/calendar/${p}`);
|
||||||
|
return [r.status, await r.text()];
|
||||||
|
}));
|
||||||
|
assert.deepEqual(new Set(answers.map((a) => JSON.stringify(a))).size, 1, "never made, ended, and a wall's: indistinguishable");
|
||||||
|
assert.equal(answers[0][0], 404);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -115,7 +115,7 @@ describe("through the routes", () => {
|
|||||||
assert.ok(!r.entries.some((e: { title: string }) => /login|export/i.test(e.title)));
|
assert.ok(!r.entries.some((e: { title: string }) => /login|export/i.test(e.title)));
|
||||||
const maint = r.entries[0];
|
const maint = r.entries[0];
|
||||||
assert.deepEqual([maint.kind, maint.actor, maint.link, maint.serverId], ["maintenance", "Ada", `/servers/${nas}`, nas]);
|
assert.deepEqual([maint.kind, maint.actor, maint.link, maint.serverId], ["maintenance", "Ada", `/servers/${nas}`, nas]);
|
||||||
assert.equal(r.entries.find((e: { title: string }) => e.title.startsWith("Alert raised")).link.startsWith("/incidents/"), true);
|
assert.equal(r.entries.find((e: { title: string }) => e.title.startsWith("Alert raised")).link.startsWith("/alerts/incidents/"), true);
|
||||||
assert.equal(r.kinds.length, 10);
|
assert.equal(r.kinds.length, 10);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,176 @@
|
|||||||
|
/**
|
||||||
|
* Suggested endpoint checks: what is offered from servers' open web ports, DNS names and secrets, what is left out, and the routes that add or dismiss them.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import type { Server } from "node:http";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-suggest-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, secrets, dnsProviders, dnsRecordsCache, endpointChecks } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { suggestChecks } = await import("../src/services/endpointSuggestions.js");
|
||||||
|
const { endpointsRouter } = await import("../src/routes/endpoints.js");
|
||||||
|
const { eq } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
const nas = { id: 1, name: "nas", hostname: "nas.lan", ips: ["10.0.0.5"], ports: [{ protocol: "tcp", port: 443, address: "0.0.0.0" }, { protocol: "tcp", port: 22, address: "0.0.0.0" }] };
|
||||||
|
const input = (extra: Partial<Parameters<typeof suggestChecks>[0]> = {}) => ({ servers: [nas], dns: [], secrets: [], existing: [], dismissed: new Set<string>(), ...extra });
|
||||||
|
|
||||||
|
describe("what is offered", () => {
|
||||||
|
test("a web port a server listens on, by the name DNS gives it, linked to the server", () => {
|
||||||
|
const { suggestions } = suggestChecks(input({ dns: [{ name: "git.example.lan", type: "A", content: "10.0.0.5" }, { name: "other.example.lan", type: "A", content: "10.0.0.9" }] }));
|
||||||
|
assert.deepEqual(suggestions.map((s) => [s.kind, s.target, s.name, s.serverId, s.serverName]), [["http", "https://git.example.lan/", "git.example.lan", 1, "nas"]]);
|
||||||
|
assert.match(suggestions[0].reason, /nas is listening on port 443, and git\.example\.lan points at it in DNS/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an alias to one of those names, or to the server's own hostname, is offered too; wildcards, reverse names and short names are not", () => {
|
||||||
|
const { suggestions } = suggestChecks(
|
||||||
|
input({
|
||||||
|
dns: [
|
||||||
|
{ name: "git.example.lan", type: "A", content: "10.0.0.5" },
|
||||||
|
{ name: "code.example.lan", type: "CNAME", content: "git.example.lan." },
|
||||||
|
{ name: "files.example.lan", type: "CNAME", content: "nas.lan" },
|
||||||
|
{ name: "*.example.lan", type: "A", content: "10.0.0.5" },
|
||||||
|
{ name: "5.0.0.10.in-addr.arpa", type: "A", content: "10.0.0.5" },
|
||||||
|
{ name: "nas", type: "A", content: "10.0.0.5" },
|
||||||
|
{ name: "unrelated.example.lan", type: "CNAME", content: "elsewhere.lan" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.deepEqual(suggestions.map((s) => s.target), ["https://code.example.lan/", "https://files.example.lan/", "https://git.example.lan/"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("with no DNS name the server's own hostname or its address is used; with neither, nothing", () => {
|
||||||
|
assert.deepEqual(suggestChecks(input()).suggestions.map((s) => s.target), ["https://nas.lan/"]);
|
||||||
|
const bare = { ...nas, hostname: "nas", ips: ["169.254.1.1", "10.0.0.5"] };
|
||||||
|
assert.deepEqual(suggestChecks(input({ servers: [bare] })).suggestions.map((s) => s.target), ["https://10.0.0.5/"]);
|
||||||
|
assert.deepEqual(suggestChecks(input({ servers: [{ ...bare, ips: [] }] })).suggestions, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("only web ports count, only for TCP, and a port that is open on this machine alone does not", () => {
|
||||||
|
const odd = { ...nas, ports: [{ protocol: "udp", port: 443, address: "0.0.0.0" }, { protocol: "tcp", port: 22, address: "0.0.0.0" }, { protocol: "tcp", port: 8443, address: "127.0.0.1" }, { protocol: "tcp", port: 3000, address: "0.0.0.0" }] };
|
||||||
|
assert.deepEqual(suggestChecks(input({ servers: [odd] })).suggestions, []);
|
||||||
|
const mixed = { ...nas, ports: [{ protocol: "tcp", port: 8080, address: "127.0.0.1" }, { protocol: "tcp", port: 8080, address: "10.0.0.5" }] };
|
||||||
|
assert.deepEqual(suggestChecks(input({ servers: [mixed] })).suggestions.map((s) => s.target), ["http://nas.lan:8080/"], "one socket on a real address is enough, and a port that is not the default is in the address");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("plain and secure on the same server: the secure one is offered, since it reads the certificate and the plain one is most likely its redirect", () => {
|
||||||
|
const both = { ...nas, ports: [{ protocol: "tcp", port: 80, address: "::" }, { protocol: "tcp", port: 443, address: "::" }, { protocol: "tcp", port: 8080, address: "::" }] };
|
||||||
|
assert.deepEqual(suggestChecks(input({ servers: [both] })).suggestions.map((s) => s.target), ["http://nas.lan:8080/", "https://nas.lan/"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a secret that already reads a certificate is offered as a certificate check", () => {
|
||||||
|
const { suggestions } = suggestChecks(input({ servers: [], secrets: [{ name: "Wildcard cert", checkHost: "Git.Example.lan.", checkPort: 443 }, { name: "Typed in", checkHost: null, checkPort: null }] }));
|
||||||
|
assert.deepEqual(suggestions.map((s) => [s.kind, s.target, s.port, s.name, s.serverId]), [["tls", "git.example.lan", 443, "Certificate: Wildcard cert", null]]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what is left out", () => {
|
||||||
|
const dns = [{ name: "git.example.lan", type: "A", content: "10.0.0.5" }];
|
||||||
|
|
||||||
|
test("what a check already covers, however its address is written", () => {
|
||||||
|
assert.equal(suggestChecks(input({ dns, existing: [{ kind: "http", target: "https://Git.example.lan:443/login", port: null }] })).suggestions.length, 0);
|
||||||
|
assert.equal(suggestChecks(input({ dns, existing: [{ kind: "http", target: "https://git.example.lan", port: null }] })).suggestions.length, 0);
|
||||||
|
assert.equal(suggestChecks(input({ dns, existing: [{ kind: "http", target: "http://git.example.lan/", port: null }] })).suggestions.length, 1, "a plain-http check does not cover the secure port");
|
||||||
|
assert.equal(suggestChecks(input({ dns, existing: [{ kind: "tcp", target: "git.example.lan", port: 443 }] })).suggestions.length, 1, "nor does a port check, which doesn't speak to the page");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a secret's certificate is covered by a certificate check, or by a web check on https that reads it anyway", () => {
|
||||||
|
const sec = [{ name: "Cert", checkHost: "git.example.lan", checkPort: 443 }];
|
||||||
|
assert.equal(suggestChecks(input({ servers: [], secrets: sec, existing: [{ kind: "tls", target: "git.example.lan", port: 443 }] })).suggestions.length, 0);
|
||||||
|
assert.equal(suggestChecks(input({ servers: [], secrets: sec, existing: [{ kind: "http", target: "https://git.example.lan/", port: null }] })).suggestions.length, 0);
|
||||||
|
assert.equal(suggestChecks(input({ servers: [], secrets: sec, existing: [{ kind: "tls", target: "git.example.lan", port: 8443 }] })).suggestions.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("what was dismissed is listed apart, not offered", () => {
|
||||||
|
const r = suggestChecks(input({ dns, dismissed: new Set(["http|https://git.example.lan/"]) }));
|
||||||
|
assert.deepEqual([r.suggestions.length, r.dismissed.map((d) => d.target)], [0, ["https://git.example.lan/"]]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the routes", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "admin";
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "admin", name: "Ada", role: "admin" },
|
||||||
|
{ oidcSub: "operator", name: "Olga", role: "operator" },
|
||||||
|
]);
|
||||||
|
const [s] = await db
|
||||||
|
.insert(servers)
|
||||||
|
.values({ name: "nas", hostname: "nas.lan", apiTokenHash: "n", apiTokenPrefix: "n", ipAddresses: JSON.stringify(["10.0.0.5"]), listeningPorts: JSON.stringify([{ protocol: "tcp", port: 443, address: "0.0.0.0", process: "nginx" }, { protocol: "tcp", port: 8080, address: "0.0.0.0" }]) })
|
||||||
|
.returning({ id: servers.id });
|
||||||
|
const [p] = await db.insert(dnsProviders).values({ providerType: "pihole", name: "Pi-hole" }).returning({ id: dnsProviders.id });
|
||||||
|
await db.insert(dnsRecordsCache).values({ providerId: p.id, zoneId: "z", recordId: "r1", type: "A", name: "git.example.lan", content: "10.0.0.5" });
|
||||||
|
await db.insert(secrets).values({ name: "Mail cert", type: "ssl_certificate", checkHost: "mail.example.lan", checkPort: 443 } as never);
|
||||||
|
void s;
|
||||||
|
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/endpoints", endpointsRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const x = a.listen(0, "127.0.0.1", () => resolve(x));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/endpoints`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const send = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
|
||||||
|
test("administrators see what is offered, and nobody else", async () => {
|
||||||
|
who = "operator";
|
||||||
|
assert.equal((await send("GET", "/suggestions")).status, 403);
|
||||||
|
assert.equal((await send("POST", "/suggestions/add", { keys: ["x"] })).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
const body = await (await send("GET", "/suggestions")).json();
|
||||||
|
assert.deepEqual(body.suggestions.map((s: { target: string }) => s.target), ["http://git.example.lan:8080/", "https://git.example.lan/", "mail.example.lan"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("adding makes the checks the server itself offered — a key that was not offered adds nothing — linked to their server, and they stop being offered", async () => {
|
||||||
|
const res = await send("POST", "/suggestions/add", { keys: ["https://git.example.lan/", "http|https://git.example.lan/", "http|http://evil.example/", "http|https://git.example.lan/"] });
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
assert.equal((await res.json()).added.length, 1);
|
||||||
|
const [made] = await db.select().from(endpointChecks).where(eq(endpointChecks.target, "https://git.example.lan/"));
|
||||||
|
assert.deepEqual([made.kind, made.name, made.serverId !== null, made.intervalMinutes, made.enabled], ["http", "git.example.lan", true, 5, true]);
|
||||||
|
const body = await (await send("GET", "/suggestions")).json();
|
||||||
|
assert.equal(body.suggestions.some((s: { target: string }) => s.target === "https://git.example.lan/"), false);
|
||||||
|
assert.equal((await send("POST", "/suggestions/add", { keys: [] })).status, 400);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("dismissing takes one out of the offer, and restoring brings every dismissed one back", async () => {
|
||||||
|
let body = await (await send("POST", "/suggestions/dismiss", { keys: ["tls|mail.example.lan|443"] })).json();
|
||||||
|
assert.equal(body.suggestions.some((s: { kind: string }) => s.kind === "tls"), false);
|
||||||
|
assert.equal(body.dismissed.length, 1);
|
||||||
|
body = await (await send("POST", "/suggestions/restore")).json();
|
||||||
|
assert.equal(body.suggestions.some((s: { kind: string }) => s.kind === "tls"), true);
|
||||||
|
assert.equal(body.dismissed.length, 0);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,292 @@
|
|||||||
|
/**
|
||||||
|
* Heartbeats: what a job reporting in (or not) amounts to, the address it calls, what the pass announces, and who may set them up.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { createServer as createHttpServer, type Server } from "node:http";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-heartbeats-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, heartbeats, heartbeatPings } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { dueBy, heartbeatState, heartbeatConditions, cleanMessage, parseStamp, recordPing, newToken, TOKEN_PATTERN } = await import("../src/services/heartbeats.js");
|
||||||
|
const { heartbeatsRouter } = await import("../src/routes/heartbeats.js");
|
||||||
|
const { heartbeatPingRouter } = await import("../src/routes/heartbeatPing.js");
|
||||||
|
const { runEndpointPass } = await import("../src/services/endpointMonitor.js");
|
||||||
|
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
|
||||||
|
const { startOrExtendWindow } = await import("../src/services/maintenance.js");
|
||||||
|
const { metaOfConditionKey } = await import("../src/services/notifyRouting.js");
|
||||||
|
const { subjectOfConditionKey } = await import("../src/services/maintenance.js");
|
||||||
|
const { updateSettings } = await import("../src/services/settingsStore.js");
|
||||||
|
const { buildMetrics } = await import("../src/services/metrics.js");
|
||||||
|
const { eq } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
type Row = typeof heartbeats.$inferSelect;
|
||||||
|
const NOW = new Date("2026-10-19T03:00:00Z");
|
||||||
|
const row = (extra: Partial<Row> = {}): Row => ({
|
||||||
|
id: 1, name: "Nightly backup", token: "hb_x", periodMinutes: 1440, graceMinutes: 60, serverId: null, enabled: true, notes: null, lastPingAt: null, lastStatus: null, lastMessage: null,
|
||||||
|
lastDurationMs: null, lastStartAt: null, pingCount: 0, createdAt: "2026-10-18 03:00:00", ...extra,
|
||||||
|
});
|
||||||
|
const minutes = (n: number) => new Date(NOW.getTime() + n * 60_000);
|
||||||
|
|
||||||
|
describe("when a job is due, and how it is doing", () => {
|
||||||
|
test("it is due a period and the grace after its last call — or after it was made, if it never called", () => {
|
||||||
|
assert.equal(dueBy(row({ lastPingAt: "2026-10-19T02:00:00.000Z" })).toISOString(), "2026-10-20T03:00:00.000Z");
|
||||||
|
assert.equal(dueBy(row()).toISOString(), "2026-10-19T04:00:00.000Z", "SQLite's own timestamp format is read as UTC");
|
||||||
|
assert.equal(parseStamp("2026-10-18 03:00:00"), Date.parse("2026-10-18T03:00:00Z"));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("new, up, running, late, failed and switched off", () => {
|
||||||
|
const called = { lastPingAt: NOW.toISOString(), lastStatus: "ok" };
|
||||||
|
assert.equal(heartbeatState(row({ createdAt: "2026-10-19 02:00:00" }), NOW), "waiting", "made recently, has not called, not yet due");
|
||||||
|
assert.equal(heartbeatState(row(), NOW), "waiting", "due by 04:00 today");
|
||||||
|
assert.equal(heartbeatState(row(), minutes(61)), "late", "never called, and now it is overdue");
|
||||||
|
assert.equal(heartbeatState(row(called), minutes(60 * 24)), "up");
|
||||||
|
assert.equal(heartbeatState(row(called), minutes(60 * 25)), "up", "inside the grace");
|
||||||
|
assert.equal(heartbeatState(row(called), minutes(60 * 25 + 1)), "late");
|
||||||
|
assert.equal(heartbeatState(row({ ...called, lastStartAt: minutes(5).toISOString() }), minutes(10)), "running", "it began after it last finished");
|
||||||
|
assert.equal(heartbeatState(row({ ...called, lastStartAt: minutes(-5).toISOString() }), minutes(10)), "up", "a start older than the last finish is not a run in progress");
|
||||||
|
assert.equal(heartbeatState(row({ ...called, lastStatus: "fail", lastMessage: "disk full" }), minutes(10)), "failed", "said so, and does not wait for the time to pass");
|
||||||
|
assert.equal(heartbeatState(row({ enabled: false }), minutes(9999)), "off");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the problems: late is a warning with the time it was due, a reported failure is critical; each keyed by the server it runs on", () => {
|
||||||
|
const list = heartbeatConditions(
|
||||||
|
[
|
||||||
|
row({ id: 1, serverId: 7 }),
|
||||||
|
row({ id: 2, name: "Photos sync", lastPingAt: NOW.toISOString(), lastStatus: "fail", lastMessage: "exit 3" }),
|
||||||
|
row({ id: 3, name: "Fine", lastPingAt: NOW.toISOString(), lastStatus: "ok" }),
|
||||||
|
row({ id: 4, name: "Paused", enabled: false }),
|
||||||
|
],
|
||||||
|
minutes(120),
|
||||||
|
);
|
||||||
|
assert.deepEqual(list.map((c) => [c.key, c.severity, c.category, c.serverId]), [["heartbeat:1:server:7", "warning", "automation", 7], ["heartbeat-fail:2", "critical", "automation", null]]);
|
||||||
|
assert.match(list[0].message, /Nightly backup hasn't reported — it was due by 2026-10-19 04:00 UTC \(it has never reported\)/);
|
||||||
|
assert.match(list[1].message, /Photos sync reported a failure — exit 3/);
|
||||||
|
assert.deepEqual(metaOfConditionKey("heartbeat:1:server:7", "warning"), { category: "automation", severity: "warning", serverId: 7 }, "tags and categories route it");
|
||||||
|
assert.equal(subjectOfConditionKey("heartbeat-fail:2:server:9"), "server:9", "and maintenance on the server holds it back");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a note from a job is one short line", () => {
|
||||||
|
assert.equal(cleanMessage("3.2 GB\n\tcopied in 41 s\u0000"), "3.2 GB copied in 41 s");
|
||||||
|
assert.equal(cleanMessage(" "), null);
|
||||||
|
assert.equal(cleanMessage(42), null);
|
||||||
|
assert.equal(cleanMessage("x".repeat(500))?.length, 200);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an address carries a long random token, and only that shape is looked up", () => {
|
||||||
|
const t = newToken();
|
||||||
|
assert.match(t, TOKEN_PATTERN);
|
||||||
|
assert.notEqual(t, newToken());
|
||||||
|
for (const bad of ["", "hb_", "hb_short", "nope_" + "a".repeat(24), "hb_" + "a/b".repeat(10), "hb_" + "a".repeat(100)]) assert.equal(TOKEN_PATTERN.test(bad), false, bad);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the address, the routes and the pass", () => {
|
||||||
|
let app: Server;
|
||||||
|
let hook: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "admin";
|
||||||
|
let serverId = 0;
|
||||||
|
let id = 0;
|
||||||
|
let path = "";
|
||||||
|
const messages: string[] = [];
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "admin", name: "Ada", role: "admin" },
|
||||||
|
{ oidcSub: "operator", name: "Olga", role: "operator" },
|
||||||
|
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
|
||||||
|
]);
|
||||||
|
serverId = (await db.insert(servers).values({ name: "nas", apiTokenHash: "n", apiTokenPrefix: "n", lastSeenAt: new Date().toISOString() }).returning({ id: servers.id }))[0].id;
|
||||||
|
hook = createHttpServer((req, res) => {
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
req.on("data", (c) => chunks.push(c));
|
||||||
|
req.on("end", () => {
|
||||||
|
messages.push(String(JSON.parse(Buffer.concat(chunks).toString()).content));
|
||||||
|
res.end("{}");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await new Promise<void>((r) => hook.listen(0, "127.0.0.1", r));
|
||||||
|
await updateSettings({ webhook: { enabled: true, url: `http://127.0.0.1:${(hook.address() as AddressInfo).port}/` } });
|
||||||
|
|
||||||
|
const a = express();
|
||||||
|
// The ping address comes first, as in the real app: no session, no JSON parser, no sign-in.
|
||||||
|
a.use("/api/heartbeat", heartbeatPingRouter);
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/heartbeats", heartbeatsRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
hook.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const admin = (method: string, p: string, body?: unknown) => fetch(`${base}/api/heartbeats${p}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
const ping = (p: string, init?: RequestInit) => fetch(`${base}${p}`, init);
|
||||||
|
|
||||||
|
test("an administrator makes one; a wrong one is refused with a reason; nobody else can; everyone can look", async () => {
|
||||||
|
const res = await admin("POST", "/", { name: "Nightly backup", periodMinutes: 1440, graceMinutes: 60, serverId });
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
const made = (await res.json()).heartbeat;
|
||||||
|
id = made.id;
|
||||||
|
path = made.path;
|
||||||
|
assert.match(path, /^\/api\/heartbeat\/hb_[A-Za-z0-9_-]{20,}$/);
|
||||||
|
for (const bad of [{ periodMinutes: 5 }, { name: "x" }, { name: "x", periodMinutes: 0 }, { name: "x", periodMinutes: 60 * 24 * 31 }, { name: "x", periodMinutes: 5, graceMinutes: 99999 }, { name: "x", periodMinutes: 5, serverId: 99999 }, { name: " ", periodMinutes: 5 }]) {
|
||||||
|
assert.equal((await admin("POST", "/", bad)).status, 400, JSON.stringify(bad));
|
||||||
|
}
|
||||||
|
who = "operator";
|
||||||
|
assert.equal((await admin("POST", "/", { name: "y", periodMinutes: 5 })).status, 403);
|
||||||
|
assert.equal((await admin("DELETE", `/${id}`)).status, 403);
|
||||||
|
const list = await (await admin("GET", "/")).json();
|
||||||
|
assert.equal(list.heartbeats[0].path, null, "the address is for administrators: it is what lets anyone say \"I'm alive\"");
|
||||||
|
assert.equal(list.baseUrl, null);
|
||||||
|
assert.equal(list.heartbeats[0].state, "waiting");
|
||||||
|
assert.equal(list.heartbeats[0].server.name, "nas");
|
||||||
|
who = "admin";
|
||||||
|
const adminList = await (await admin("GET", "/")).json();
|
||||||
|
assert.equal(adminList.heartbeats[0].path, path);
|
||||||
|
assert.ok(adminList.baseUrl);
|
||||||
|
assert.equal("token" in adminList.heartbeats[0], false, "the token is not a field of its own");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a call is noted: success, a note in the body or the query, and how long a run took when it said it started", async () => {
|
||||||
|
assert.equal((await ping(path)).status, 200);
|
||||||
|
assert.equal(await (await ping(path, { method: "HEAD" })).text(), "");
|
||||||
|
let [h] = await db.select().from(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
assert.deepEqual([h.lastStatus, h.pingCount], ["ok", 2]);
|
||||||
|
assert.equal(heartbeatState(h, new Date()), "up");
|
||||||
|
|
||||||
|
assert.equal((await ping(`${path}/start`, { method: "POST" })).status, 200);
|
||||||
|
await new Promise((r) => setTimeout(r, 30));
|
||||||
|
const done = await ping(path, { method: "POST", body: "3.2 GB\ncopied", headers: { "content-type": "text/plain" } });
|
||||||
|
assert.equal(await done.text(), "OK");
|
||||||
|
[h] = await db.select().from(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
assert.equal(h.lastMessage, "3.2 GB copied");
|
||||||
|
assert.ok(h.lastDurationMs !== null && h.lastDurationMs >= 20, "timed from the start call");
|
||||||
|
|
||||||
|
await ping(`${path}?msg=from%20the%20query`);
|
||||||
|
[h] = await db.select().from(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
assert.equal(h.lastMessage, "from the query");
|
||||||
|
assert.equal(h.lastDurationMs, null, "no start before this one, so no duration");
|
||||||
|
const rows = await db.select().from(heartbeatPings).where(eq(heartbeatPings.heartbeatId, id));
|
||||||
|
assert.deepEqual(rows.map((r) => r.kind), ["ok", "ok", "start", "ok", "ok"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a token that does not exist, a path that is not one of the three and a method that is not a call all get no information", async () => {
|
||||||
|
const a = await ping("/api/heartbeat/hb_" + "z".repeat(24));
|
||||||
|
const b = await ping(`${path}/bogus`);
|
||||||
|
const c = await ping("/api/heartbeat/nonsense");
|
||||||
|
assert.deepEqual([a.status, b.status, c.status], [404, 404, 404]);
|
||||||
|
assert.equal(await a.text(), await b.text(), "never made, and a wrong path, read the same");
|
||||||
|
assert.equal((await ping(path, { method: "DELETE" })).status, 405);
|
||||||
|
assert.equal((await ping(path)).headers.get("cache-control"), "no-store");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a job that says it failed is an alert at once, until it next succeeds; the pass says so once, and once when it is over", async () => {
|
||||||
|
messages.length = 0;
|
||||||
|
await db.update(heartbeats).set({ serverId: null }).where(eq(heartbeats.id, id));
|
||||||
|
assert.equal((await ping(`${path}/fail`, { method: "POST", body: "exit 3: disk full" })).status, 200);
|
||||||
|
invalidateAlertsCache();
|
||||||
|
const alert = (await getAlerts(true)).alerts.find((x) => x.source === "Heartbeat")!;
|
||||||
|
assert.match(alert.message, /Nightly backup reported a failure — exit 3: disk full/);
|
||||||
|
assert.deepEqual([alert.severity, alert.category, alert.link], ["critical", "automation", "/heartbeats"]);
|
||||||
|
|
||||||
|
let r = await runEndpointPass(new Date());
|
||||||
|
assert.equal(r.announced, 1);
|
||||||
|
assert.match(messages.join("\n"), /Nightly backup reported a failure/);
|
||||||
|
const sent = messages.length;
|
||||||
|
r = await runEndpointPass(new Date());
|
||||||
|
assert.equal(r.announced + r.recovered, 0);
|
||||||
|
assert.equal(messages.length, sent, "not said again");
|
||||||
|
|
||||||
|
await ping(path);
|
||||||
|
r = await runEndpointPass(new Date());
|
||||||
|
assert.equal(r.recovered, 1);
|
||||||
|
assert.match(messages[messages.length - 1], /Nightly backup reported a failure/, "the recovery names what cleared");
|
||||||
|
invalidateAlertsCache();
|
||||||
|
assert.equal((await getAlerts(true)).alerts.filter((x) => x.source === "Heartbeat").length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a call that does not come in time is a warning once it is overdue, silenced by maintenance on the server, and cleared by the call", async () => {
|
||||||
|
messages.length = 0;
|
||||||
|
await db.update(heartbeats).set({ serverId, periodMinutes: 60, graceMinutes: 10 }).where(eq(heartbeats.id, id));
|
||||||
|
const t = (min: number) => new Date(Date.now() + min * 60_000);
|
||||||
|
let r = await runEndpointPass(t(30));
|
||||||
|
assert.equal(r.announced, 0, "not due yet");
|
||||||
|
r = await runEndpointPass(t(75));
|
||||||
|
assert.equal(r.announced, 1, "an hour and ten minutes with no call");
|
||||||
|
assert.match(messages.join("\n"), /Nightly backup hasn't reported — it was due by/);
|
||||||
|
|
||||||
|
// The Alerts page reads the real clock, so the last call is moved back instead of the clock forward.
|
||||||
|
await db.update(heartbeats).set({ lastPingAt: new Date(Date.now() - 120 * 60_000).toISOString() }).where(eq(heartbeats.id, id));
|
||||||
|
invalidateAlertsCache();
|
||||||
|
const late = (await getAlerts(true)).alerts.find((x) => x.source === "Heartbeat")!;
|
||||||
|
assert.equal(late.severity, "warning");
|
||||||
|
assert.deepEqual(late.target && [late.target.type, late.target.id], ["server", serverId]);
|
||||||
|
assert.equal(late.silenced, false);
|
||||||
|
await startOrExtendWindow("server", serverId, 60, "work", "Ada");
|
||||||
|
invalidateAlertsCache();
|
||||||
|
assert.equal((await getAlerts(true)).alerts.find((x) => x.source === "Heartbeat")!.silenced, true, "maintenance on the server it runs on silences it");
|
||||||
|
|
||||||
|
assert.match(await buildMetrics(t(75)), /homelab_heartbeat_up\{name="Nightly backup"\} 0/);
|
||||||
|
await recordPing((await db.select().from(heartbeats).where(eq(heartbeats.id, id)))[0].token, "ok", null, t(76));
|
||||||
|
r = await runEndpointPass(t(77));
|
||||||
|
assert.equal(r.recovered, 1);
|
||||||
|
assert.match(await buildMetrics(t(77)), /homelab_heartbeat_up\{name="Nightly backup"\} 1/);
|
||||||
|
assert.match(await buildMetrics(t(77)), /homelab_heartbeat_last_ping_timestamp_seconds\{name="Nightly backup"\} \d+/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a switched-off heartbeat still takes calls, says nothing, and starts a fresh clock when it is switched on again", async () => {
|
||||||
|
await admin("PUT", `/${id}`, { name: "Nightly backup", periodMinutes: 60, graceMinutes: 10, serverId: null, enabled: false });
|
||||||
|
const [h] = await db.select().from(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
assert.equal((await ping(`/api/heartbeat/${h.token}`)).status, 200, "pausing a job's heartbeat does not break its script");
|
||||||
|
assert.equal(heartbeatConditions([{ ...h, lastPingAt: "2020-01-01T00:00:00.000Z", lastStatus: "ok" }], new Date()).length, 0);
|
||||||
|
await admin("PUT", `/${id}`, { name: "Nightly backup", periodMinutes: 60, graceMinutes: 10, serverId: null, enabled: true });
|
||||||
|
const [after_] = await db.select().from(heartbeats).where(eq(heartbeats.id, id));
|
||||||
|
assert.equal(heartbeatState(after_, new Date()), "up");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a new address replaces the old one at once; removing one takes its history", async () => {
|
||||||
|
const res = await admin("POST", `/${id}/new-address`);
|
||||||
|
assert.equal(res.status, 200);
|
||||||
|
const fresh = (await res.json()).path as string;
|
||||||
|
assert.notEqual(fresh, path);
|
||||||
|
assert.equal((await ping(path)).status, 404, "the old one is dead");
|
||||||
|
assert.equal((await ping(fresh)).status, 200);
|
||||||
|
who = "viewer";
|
||||||
|
assert.equal((await admin("POST", `/${id}/new-address`)).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
assert.equal((await admin("DELETE", `/${id}`)).status, 204);
|
||||||
|
assert.equal((await db.select().from(heartbeatPings).where(eq(heartbeatPings.heartbeatId, id))).length, 0);
|
||||||
|
assert.equal((await ping(fresh)).status, 404);
|
||||||
|
assert.equal((await admin("DELETE", `/${id}`)).status, 404);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -194,7 +194,9 @@ describe("through the routes, the page, the alerts and the figures", () => {
|
|||||||
test("metrics say how old each test is and whether it passed", async () => {
|
test("metrics say how old each test is and whether it passed", async () => {
|
||||||
const text = await buildMetrics();
|
const text = await buildMetrics();
|
||||||
assert.match(text, /homelab_restore_test_age_days\{subject="elsa"\} 0/);
|
assert.match(text, /homelab_restore_test_age_days\{subject="elsa"\} 0/);
|
||||||
assert.match(text, /homelab_restore_test_age_days\{subject="Photos share"\} 40/);
|
// The test's dates come from a fixed "now" and the metrics from the real clock, so what is expected is worked out the way the metric does.
|
||||||
|
const realAge = Math.floor((Date.now() - Date.parse(`${daysAgo(40)}T12:00:00Z`)) / 86_400_000);
|
||||||
|
assert.match(text, new RegExp(`homelab_restore_test_age_days\\{subject="Photos share"\\} ${realAge}`));
|
||||||
assert.match(text, /homelab_restore_test_passed\{subject="elsa"\} 1/);
|
assert.match(text, /homelab_restore_test_passed\{subject="elsa"\} 1/);
|
||||||
assert.doesNotMatch(text, /subject="anna"/, "'not needed' isn't measured");
|
assert.doesNotMatch(text, /subject="anna"/, "'not needed' isn't measured");
|
||||||
});
|
});
|
||||||
|
|||||||
+15
-4
@@ -1,5 +1,5 @@
|
|||||||
import { useEffect, useState, type ReactNode } from "react";
|
import { useEffect, useState, type ReactNode } from "react";
|
||||||
import { Routes, Route, Navigate, useLocation } from "react-router-dom";
|
import { Routes, Route, Navigate, useLocation, useParams } from "react-router-dom";
|
||||||
import { api, type CurrentUser, type UserRole, UnauthorizedError } from "./api/client";
|
import { api, type CurrentUser, type UserRole, UnauthorizedError } from "./api/client";
|
||||||
import Login from "./pages/Login";
|
import Login from "./pages/Login";
|
||||||
import Dashboard from "./pages/Dashboard";
|
import Dashboard from "./pages/Dashboard";
|
||||||
@@ -29,13 +29,14 @@ import Alerts from "./pages/Alerts";
|
|||||||
import Generator from "./pages/Generator";
|
import Generator from "./pages/Generator";
|
||||||
import Maintenance from "./pages/Maintenance";
|
import Maintenance from "./pages/Maintenance";
|
||||||
import Updates from "./pages/Updates";
|
import Updates from "./pages/Updates";
|
||||||
import Incidents from "./pages/Incidents";
|
|
||||||
import IncidentDetail from "./pages/IncidentDetail";
|
import IncidentDetail from "./pages/IncidentDetail";
|
||||||
import Inventory from "./pages/Inventory";
|
import Inventory from "./pages/Inventory";
|
||||||
import InventoryLabels from "./pages/InventoryLabels";
|
import InventoryLabels from "./pages/InventoryLabels";
|
||||||
import InventoryLifecycle from "./pages/InventoryLifecycle";
|
import InventoryLifecycle from "./pages/InventoryLifecycle";
|
||||||
import Dependencies from "./pages/Dependencies";
|
import Dependencies from "./pages/Dependencies";
|
||||||
import Endpoints from "./pages/Endpoints";
|
import Endpoints from "./pages/Endpoints";
|
||||||
|
import Heartbeats from "./pages/Heartbeats";
|
||||||
|
import ComingUp from "./pages/ComingUp";
|
||||||
import ChangeLog from "./pages/ChangeLog";
|
import ChangeLog from "./pages/ChangeLog";
|
||||||
import Backups from "./pages/Backups";
|
import Backups from "./pages/Backups";
|
||||||
import Domains from "./pages/Domains";
|
import Domains from "./pages/Domains";
|
||||||
@@ -83,6 +84,12 @@ function RequireRole({
|
|||||||
return <>{children}</>;
|
return <>{children}</>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** An old /incidents/<id> link, sent to the same incident under Alerts. */
|
||||||
|
function OldIncidentLink() {
|
||||||
|
const { id } = useParams();
|
||||||
|
return <Navigate to={/^\d+$/.test(id ?? "") ? `/alerts/incidents/${id}` : "/alerts?view=history"} replace />;
|
||||||
|
}
|
||||||
|
|
||||||
export default function App() {
|
export default function App() {
|
||||||
const [user, setUser] = useState<CurrentUser | null | undefined>(undefined);
|
const [user, setUser] = useState<CurrentUser | null | undefined>(undefined);
|
||||||
// A share link's page is for someone who isn't signed in: it asks nothing of the app's own session.
|
// A share link's page is for someone who isn't signed in: it asks nothing of the app's own session.
|
||||||
@@ -156,6 +163,8 @@ function Shelled({ user }: { user: CurrentUser }) {
|
|||||||
<Route path="/inventory/lifecycle" element={<InventoryLifecycle />} />
|
<Route path="/inventory/lifecycle" element={<InventoryLifecycle />} />
|
||||||
<Route path="/dependencies" element={<Dependencies user={user} />} />
|
<Route path="/dependencies" element={<Dependencies user={user} />} />
|
||||||
<Route path="/endpoints" element={<Endpoints user={user} />} />
|
<Route path="/endpoints" element={<Endpoints user={user} />} />
|
||||||
|
<Route path="/heartbeats" element={<Heartbeats user={user} />} />
|
||||||
|
<Route path="/coming-up" element={<ComingUp user={user} />} />
|
||||||
<Route path="/changes" element={user.role === "viewer" ? <Navigate to="/" replace /> : <ChangeLog />} />
|
<Route path="/changes" element={user.role === "viewer" ? <Navigate to="/" replace /> : <ChangeLog />} />
|
||||||
<Route path="/backups" element={<Backups user={user} />} />
|
<Route path="/backups" element={<Backups user={user} />} />
|
||||||
<Route path="/docker" element={<Docker user={user} />} />
|
<Route path="/docker" element={<Docker user={user} />} />
|
||||||
@@ -169,8 +178,10 @@ function Shelled({ user }: { user: CurrentUser }) {
|
|||||||
<Route path="/osticket" element={<OsTicket user={user} />} />
|
<Route path="/osticket" element={<OsTicket user={user} />} />
|
||||||
<Route path="/admin-links" element={<AdminLinks user={user} />} />
|
<Route path="/admin-links" element={<AdminLinks user={user} />} />
|
||||||
<Route path="/alerts" element={<Alerts user={user} />} />
|
<Route path="/alerts" element={<Alerts user={user} />} />
|
||||||
<Route path="/incidents" element={<Incidents />} />
|
<Route path="/alerts/incidents/:id" element={<IncidentDetail user={user} />} />
|
||||||
<Route path="/incidents/:id" element={<IncidentDetail user={user} />} />
|
{/* Incidents used to be a page of their own: old links, bookmarks and notifications still arrive. */}
|
||||||
|
<Route path="/incidents" element={<Navigate to="/alerts?view=history" replace />} />
|
||||||
|
<Route path="/incidents/:id" element={<OldIncidentLink />} />
|
||||||
<Route
|
<Route
|
||||||
path="/users"
|
path="/users"
|
||||||
element={
|
element={
|
||||||
|
|||||||
+83
-1
@@ -16,7 +16,7 @@ export interface CurrentUser {
|
|||||||
|
|
||||||
export type ShareLinkDetail = "summary" | "names";
|
export type ShareLinkDetail = "summary" | "names";
|
||||||
|
|
||||||
export type ShareLinkKind = "wall" | "status";
|
export type ShareLinkKind = "wall" | "status" | "calendar";
|
||||||
export type StatusItemKind = "server" | "integration" | "monitor";
|
export type StatusItemKind = "server" | "integration" | "monitor";
|
||||||
|
|
||||||
/** One thing on a status page, as the editor sees it. */
|
/** One thing on a status page, as the editor sees it. */
|
||||||
@@ -905,6 +905,74 @@ export interface EndpointInput {
|
|||||||
enabled?: boolean;
|
enabled?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** A check the app could set up for you, from what it already knows. */
|
||||||
|
export interface EndpointSuggestion {
|
||||||
|
key: string;
|
||||||
|
kind: "http" | "tls";
|
||||||
|
name: string;
|
||||||
|
target: string;
|
||||||
|
port: number | null;
|
||||||
|
serverId: number | null;
|
||||||
|
serverName: string | null;
|
||||||
|
reason: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type HeartbeatState = "off" | "waiting" | "up" | "running" | "late" | "failed";
|
||||||
|
/** A job that reports in: a cron job or backup script calls its address when it finishes. */
|
||||||
|
export interface Heartbeat {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
periodMinutes: number;
|
||||||
|
graceMinutes: number;
|
||||||
|
serverId: number | null;
|
||||||
|
enabled: boolean;
|
||||||
|
notes: string | null;
|
||||||
|
lastPingAt: string | null;
|
||||||
|
lastStatus: "ok" | "fail" | null;
|
||||||
|
lastMessage: string | null;
|
||||||
|
lastDurationMs: number | null;
|
||||||
|
lastStartAt: string | null;
|
||||||
|
pingCount: number;
|
||||||
|
createdAt: string;
|
||||||
|
/** The address to call, without the host; only administrators are given it. */
|
||||||
|
path: string | null;
|
||||||
|
state: HeartbeatState;
|
||||||
|
/** ISO — when the next call is due by, counting the grace. */
|
||||||
|
dueBy: string;
|
||||||
|
server: { id: number; name: string } | null;
|
||||||
|
recent: { at: string; kind: "ok" | "fail"; durationMs: number | null }[];
|
||||||
|
}
|
||||||
|
export interface HeartbeatInput {
|
||||||
|
name: string;
|
||||||
|
periodMinutes: number;
|
||||||
|
graceMinutes?: number;
|
||||||
|
serverId?: number | null;
|
||||||
|
notes?: string | null;
|
||||||
|
enabled?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type AgendaKind = "secret" | "certificate" | "domain" | "tailscale" | "warranty" | "support" | "maintenance" | "restore" | "updates";
|
||||||
|
export interface AgendaItem {
|
||||||
|
id: string;
|
||||||
|
kind: AgendaKind;
|
||||||
|
/** YYYY-MM-DD for a day; an ISO time for maintenance. */
|
||||||
|
at: string;
|
||||||
|
endsAt: string | null;
|
||||||
|
allDay: boolean;
|
||||||
|
title: string;
|
||||||
|
detail: string | null;
|
||||||
|
link: string;
|
||||||
|
serverId: number | null;
|
||||||
|
/** Whole days from today; negative once it has passed. */
|
||||||
|
daysLeft: number;
|
||||||
|
}
|
||||||
|
export interface Agenda {
|
||||||
|
generatedAt: string;
|
||||||
|
today: string;
|
||||||
|
items: AgendaItem[];
|
||||||
|
notes: string[];
|
||||||
|
}
|
||||||
|
|
||||||
export type ChangeKind = "servers" | "maintenance" | "alerts" | "settings" | "integrations" | "dependencies" | "inventory" | "backups" | "endpoints" | "other";
|
export type ChangeKind = "servers" | "maintenance" | "alerts" | "settings" | "integrations" | "dependencies" | "inventory" | "backups" | "endpoints" | "other";
|
||||||
export interface ChangeEntry {
|
export interface ChangeEntry {
|
||||||
id: string;
|
id: string;
|
||||||
@@ -1891,6 +1959,20 @@ export const api = {
|
|||||||
update: (id: number, data: EndpointInput) => request<{ check: EndpointCheck }>(`/api/endpoints/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
update: (id: number, data: EndpointInput) => request<{ check: EndpointCheck }>(`/api/endpoints/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||||
remove: (id: number) => request<void>(`/api/endpoints/${id}`, { method: "DELETE" }),
|
remove: (id: number) => request<void>(`/api/endpoints/${id}`, { method: "DELETE" }),
|
||||||
check: (id: number) => request<{ result: { ok: boolean; ms: number | null; message: string } }>(`/api/endpoints/${id}/check`, { method: "POST" }),
|
check: (id: number) => request<{ result: { ok: boolean; ms: number | null; message: string } }>(`/api/endpoints/${id}/check`, { method: "POST" }),
|
||||||
|
suggestions: () => request<{ suggestions: EndpointSuggestion[]; dismissed: EndpointSuggestion[] }>("/api/endpoints/suggestions"),
|
||||||
|
addSuggestions: (keys: string[]) => request<{ added: { id: number; name: string }[] }>("/api/endpoints/suggestions/add", { method: "POST", body: JSON.stringify({ keys }) }),
|
||||||
|
dismissSuggestions: (keys: string[]) => request<{ suggestions: EndpointSuggestion[]; dismissed: EndpointSuggestion[] }>("/api/endpoints/suggestions/dismiss", { method: "POST", body: JSON.stringify({ keys }) }),
|
||||||
|
restoreSuggestions: () => request<{ suggestions: EndpointSuggestion[]; dismissed: EndpointSuggestion[] }>("/api/endpoints/suggestions/restore", { method: "POST" }),
|
||||||
|
},
|
||||||
|
heartbeats: {
|
||||||
|
list: () => request<{ heartbeats: Heartbeat[]; servers: { id: number; name: string }[]; baseUrl: string | null }>("/api/heartbeats"),
|
||||||
|
create: (data: HeartbeatInput) => request<{ heartbeat: Heartbeat }>("/api/heartbeats", { method: "POST", body: JSON.stringify(data) }),
|
||||||
|
update: (id: number, data: HeartbeatInput) => request<{ heartbeat: Heartbeat }>(`/api/heartbeats/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||||
|
remove: (id: number) => request<void>(`/api/heartbeats/${id}`, { method: "DELETE" }),
|
||||||
|
newAddress: (id: number) => request<{ path: string }>(`/api/heartbeats/${id}/new-address`, { method: "POST" }),
|
||||||
|
},
|
||||||
|
agenda: {
|
||||||
|
get: (days = 400) => request<Agenda>(`/api/agenda?days=${days}`),
|
||||||
},
|
},
|
||||||
changes: {
|
changes: {
|
||||||
list: (params: { days?: number; kind?: string; serverId?: number; q?: string; before?: string; limit?: number }) => {
|
list: (params: { days?: number; kind?: string; serverId?: number; q?: string; before?: string; limit?: number }) => {
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type EndpointSuggestion } from "../api/client";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Checks the app could set up for you, from what it already knows: a web port a server is listening on (with the name DNS gives it) and a certificate a secret already
|
||||||
|
* reads. One click adds one, a button adds them all; "Not for me" stops one being offered, and they can be brought back. Administrators only, like the checks.
|
||||||
|
*/
|
||||||
|
export default function EndpointSuggestions({ onAdded }: { onAdded: () => void }) {
|
||||||
|
const [offered, setOffered] = useState<EndpointSuggestion[] | null>(null);
|
||||||
|
const [dismissed, setDismissed] = useState<EndpointSuggestion[]>([]);
|
||||||
|
const [open, setOpen] = useState(true);
|
||||||
|
const [busy, setBusy] = useState<string | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const take = (res: { suggestions: EndpointSuggestion[]; dismissed: EndpointSuggestion[] }) => {
|
||||||
|
setOffered(res.suggestions);
|
||||||
|
setDismissed(res.dismissed);
|
||||||
|
};
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
api.endpoints
|
||||||
|
.suggestions()
|
||||||
|
.then(take)
|
||||||
|
.catch((err) => setError(readableError(err)));
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
async function add(keys: string[], label: string) {
|
||||||
|
setBusy(label);
|
||||||
|
setError(null);
|
||||||
|
try {
|
||||||
|
const { added } = await api.endpoints.addSuggestions(keys);
|
||||||
|
toast.success(added.length === 1 ? `Added ${added[0].name}. It is asked within a minute.` : `Added ${added.length} checks. They are asked within a minute.`);
|
||||||
|
take(await api.endpoints.suggestions());
|
||||||
|
onAdded();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function dismiss(keys: string[], label: string) {
|
||||||
|
setBusy(label);
|
||||||
|
try {
|
||||||
|
take(await api.endpoints.dismissSuggestions(keys));
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function bringBack() {
|
||||||
|
setBusy("restore");
|
||||||
|
try {
|
||||||
|
take(await api.endpoints.restoreSuggestions());
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (offered === null) return error ? <div className="alert alert-danger">{error}</div> : null;
|
||||||
|
if (offered.length === 0 && dismissed.length === 0) return null;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">
|
||||||
|
{offered.length > 0 ? `Suggested checks (${offered.length})` : "No suggestions right now"}
|
||||||
|
</h3>
|
||||||
|
<div className="card-actions d-flex gap-2">
|
||||||
|
{offered.length > 1 && open && (
|
||||||
|
<button className="btn btn-sm btn-primary" disabled={busy !== null} onClick={() => void add(offered.map((s) => s.key), "all")}>
|
||||||
|
{busy === "all" ? "Adding…" : `Add all ${offered.length}`}
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
{offered.length > 0 && (
|
||||||
|
<button className="btn btn-sm btn-outline-secondary" onClick={() => setOpen((o) => !o)} aria-expanded={open}>
|
||||||
|
{open ? "Hide" : "Show"}
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{error && <div className="alert alert-danger m-3 mb-0">{error}</div>}
|
||||||
|
{open && offered.length > 0 && (
|
||||||
|
<>
|
||||||
|
<div className="card-body pb-0 text-secondary">
|
||||||
|
From what the app already knows: web ports your servers are listening on (with the names DNS gives them) and certificates your secrets read. Nothing is asked until you add it.
|
||||||
|
</div>
|
||||||
|
<div className="list-group list-group-flush">
|
||||||
|
{offered.map((s) => (
|
||||||
|
<div className="list-group-item" key={s.key}>
|
||||||
|
<div className="row align-items-center g-2">
|
||||||
|
<div className="col" style={{ minWidth: 0 }}>
|
||||||
|
<div className="text-break">
|
||||||
|
<strong>{s.name}</strong> <span className="badge bg-secondary-lt text-secondary">{s.kind === "http" ? "Web address" : "Certificate"}</span>
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small text-break">
|
||||||
|
{s.target}
|
||||||
|
{s.port !== null ? `:${s.port}` : ""}
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small">
|
||||||
|
{s.reason}
|
||||||
|
{s.serverId !== null && s.serverName && (
|
||||||
|
<>
|
||||||
|
{" "}
|
||||||
|
<Link to={`/servers/${s.serverId}`}>{s.serverName}</Link>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-auto">
|
||||||
|
<div className="btn-list flex-nowrap">
|
||||||
|
<button className="btn btn-sm btn-primary" disabled={busy !== null} onClick={() => void add([s.key], s.key)}>
|
||||||
|
{busy === s.key ? "Adding…" : "Add"}
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-sm btn-outline-secondary" disabled={busy !== null} onClick={() => void dismiss([s.key], s.key)}>
|
||||||
|
Not for me
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{dismissed.length > 0 && (
|
||||||
|
<div className="card-footer text-secondary small d-flex align-items-center gap-2">
|
||||||
|
{dismissed.length} dismissed.
|
||||||
|
<button className="btn btn-sm btn-link p-0" disabled={busy !== null} onClick={() => void bringBack()}>
|
||||||
|
Offer {dismissed.length === 1 ? "it" : "them"} again
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -4,15 +4,15 @@ import { api, type AlertCategory, type IncidentSummary } from "../api/client";
|
|||||||
import { useSortable } from "../hooks/useSortable";
|
import { useSortable } from "../hooks/useSortable";
|
||||||
import { useTableSearch } from "../hooks/useTableSearch";
|
import { useTableSearch } from "../hooks/useTableSearch";
|
||||||
import { usePagination } from "../hooks/usePagination";
|
import { usePagination } from "../hooks/usePagination";
|
||||||
import SortableTh from "../components/SortableTh";
|
import SortableTh from "./SortableTh";
|
||||||
import Pagination from "../components/Pagination";
|
import Pagination from "./Pagination";
|
||||||
import TableToolbar from "../components/TableToolbar";
|
import TableToolbar from "./TableToolbar";
|
||||||
import TimeAgo from "../components/TimeAgo";
|
import TimeAgo from "./TimeAgo";
|
||||||
import { BlockSkeleton } from "../components/Skeleton";
|
import { BlockSkeleton } from "./Skeleton";
|
||||||
import { ErrorNotice } from "../components/Notice";
|
import { ErrorNotice } from "./Notice";
|
||||||
import { downloadCsv } from "../utils/csv";
|
import { downloadCsv } from "../utils/csv";
|
||||||
import { readableError } from "../utils/errors";
|
import { readableError } from "../utils/errors";
|
||||||
import { EmptyRow } from "../components/EmptyState";
|
import { EmptyRow } from "./EmptyState";
|
||||||
|
|
||||||
export const CATEGORY_NAMES: Record<AlertCategory, string> = {
|
export const CATEGORY_NAMES: Record<AlertCategory, string> = {
|
||||||
offline: "Server down",
|
offline: "Server down",
|
||||||
@@ -39,13 +39,16 @@ export function durationWords(ms: number): string {
|
|||||||
|
|
||||||
export const SEVERITY_BADGE: Record<string, string> = { critical: "bg-red-lt text-red", warning: "bg-yellow-lt text-yellow", info: "bg-blue-lt text-blue" };
|
export const SEVERITY_BADGE: Record<string, string> = { critical: "bg-red-lt text-red", warning: "bg-yellow-lt text-yellow", info: "bg-blue-lt text-blue" };
|
||||||
|
|
||||||
|
/** Where one incident's page is: under Alerts, whose History it is part of. */
|
||||||
|
export const incidentPath = (id: number) => `/alerts/incidents/${id}`;
|
||||||
|
|
||||||
type Row = IncidentSummary & { where: string; openFirst: number };
|
type Row = IncidentSummary & { where: string; openFirst: number };
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Every time something was raised as an alert and later went away: when it began, how long it lasted, how bad it got. Open ones are on top.
|
* The History side of the Alerts page: every time something was raised as an alert and later went away — when it began, how long it lasted, how bad it
|
||||||
* Open one for everything that happened around it.
|
* got. Open ones are on top. Open one for everything that happened around it.
|
||||||
*/
|
*/
|
||||||
export default function Incidents() {
|
export default function IncidentHistory() {
|
||||||
const [incidents, setIncidents] = useState<IncidentSummary[] | null>(null);
|
const [incidents, setIncidents] = useState<IncidentSummary[] | null>(null);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
const [status, setStatus] = useState<"all" | "open" | "resolved">("all");
|
const [status, setStatus] = useState<"all" | "open" | "resolved">("all");
|
||||||
@@ -93,12 +96,9 @@ export default function Incidents() {
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<div className="d-flex align-items-center mb-1">
|
|
||||||
<h2 className="page-title mb-0">Incidents</h2>
|
|
||||||
</div>
|
|
||||||
<p className="text-secondary">
|
<p className="text-secondary">
|
||||||
Each time an alert was raised and later went away is one incident, kept with when it began and ended. Open one to see everything the app knows about that time — who acknowledged it, the maintenance windows, what was changed, the update runs and the notes people wrote.
|
Each time an alert was raised and later went away is one incident, kept with when it began and ended. Open one to see everything the app knows about that time — who acknowledged it, the maintenance windows, what was changed, the update runs and the notes people wrote.
|
||||||
They are recorded from the <Link to="/alerts">Alerts</Link> every few minutes, so one that comes and goes between two looks may not be here.
|
They are recorded from the current alerts every few minutes, so one that comes and goes between two looks may not be here.
|
||||||
</p>
|
</p>
|
||||||
<ErrorNotice error={error} />
|
<ErrorNotice error={error} />
|
||||||
|
|
||||||
@@ -167,7 +167,7 @@ export default function Incidents() {
|
|||||||
<td>{CATEGORY_NAMES[i.category] ?? i.category}</td>
|
<td>{CATEGORY_NAMES[i.category] ?? i.category}</td>
|
||||||
<td>{i.target && i.target.type === "server" ? <Link to={`/servers/${i.target.id}`}>{i.where}</Link> : i.where}</td>
|
<td>{i.target && i.target.type === "server" ? <Link to={`/servers/${i.target.id}`}>{i.where}</Link> : i.where}</td>
|
||||||
<td>
|
<td>
|
||||||
<Link to={`/incidents/${i.id}`}>{i.message}</Link>
|
<Link to={incidentPath(i.id)}>{i.message}</Link>
|
||||||
{i.causedBy && <span className="badge bg-secondary-lt text-secondary ms-2">caused by {i.causedBy}</span>}
|
{i.causedBy && <span className="badge bg-secondary-lt text-secondary ms-2">caused by {i.causedBy}</span>}
|
||||||
{i.notes > 0 && (
|
{i.notes > 0 && (
|
||||||
<span className="text-secondary small ms-2">
|
<span className="text-secondary small ms-2">
|
||||||
+17
-8
@@ -15,7 +15,6 @@ import {
|
|||||||
IconUsers,
|
IconUsers,
|
||||||
IconDevices,
|
IconDevices,
|
||||||
IconHistory,
|
IconHistory,
|
||||||
IconTimeline,
|
|
||||||
IconStethoscope,
|
IconStethoscope,
|
||||||
IconSettings,
|
IconSettings,
|
||||||
IconWand,
|
IconWand,
|
||||||
@@ -36,6 +35,8 @@ import {
|
|||||||
IconExternalLink,
|
IconExternalLink,
|
||||||
IconAlertTriangle,
|
IconAlertTriangle,
|
||||||
IconSitemap,
|
IconSitemap,
|
||||||
|
IconCalendarEvent,
|
||||||
|
IconHeartRateMonitor,
|
||||||
} from "@tabler/icons-react";
|
} from "@tabler/icons-react";
|
||||||
import type { CurrentUser } from "../api/client";
|
import type { CurrentUser } from "../api/client";
|
||||||
|
|
||||||
@@ -80,6 +81,7 @@ export const NAV: NavEntry[] = [
|
|||||||
{ to: "/map", label: "Map", icon: <IconSitemap size={20} /> },
|
{ to: "/map", label: "Map", icon: <IconSitemap size={20} /> },
|
||||||
{ to: "/inventory", label: "Inventory", icon: <IconBox size={20} /> },
|
{ to: "/inventory", label: "Inventory", icon: <IconBox size={20} /> },
|
||||||
{ to: "/dependencies", label: "Dependencies", icon: <IconHierarchy2 size={20} /> },
|
{ to: "/dependencies", label: "Dependencies", icon: <IconHierarchy2 size={20} /> },
|
||||||
|
{ to: "/admin-links", label: "Admin Links", icon: <IconExternalLink size={20} /> },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -91,10 +93,19 @@ export const NAV: NavEntry[] = [
|
|||||||
{ to: "/domains", label: "Domains", icon: <IconWorldWww size={20} /> },
|
{ to: "/domains", label: "Domains", icon: <IconWorldWww size={20} /> },
|
||||||
{ to: "/ipam", label: "IP Addresses", icon: <IconNetwork size={20} /> },
|
{ to: "/ipam", label: "IP Addresses", icon: <IconNetwork size={20} /> },
|
||||||
{ to: "/ports", label: "Ports", icon: <IconPlug size={20} /> },
|
{ to: "/ports", label: "Ports", icon: <IconPlug size={20} /> },
|
||||||
{ to: "/endpoints", label: "Endpoints", icon: <IconHeartbeat size={20} /> },
|
|
||||||
{ to: "/consistency", label: "Consistency", icon: <IconListCheck size={20} /> },
|
{ to: "/consistency", label: "Consistency", icon: <IconListCheck size={20} /> },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
id: "monitoring",
|
||||||
|
label: "Monitoring",
|
||||||
|
icon: <IconHeartbeat size={20} />,
|
||||||
|
items: [
|
||||||
|
{ to: "/endpoints", label: "Endpoints", icon: <IconHeartbeat size={20} /> },
|
||||||
|
{ to: "/heartbeats", label: "Heartbeats", icon: <IconHeartRateMonitor size={20} /> },
|
||||||
|
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
||||||
|
],
|
||||||
|
},
|
||||||
{
|
{
|
||||||
id: "automation",
|
id: "automation",
|
||||||
label: "Automation",
|
label: "Automation",
|
||||||
@@ -102,6 +113,7 @@ export const NAV: NavEntry[] = [
|
|||||||
items: [
|
items: [
|
||||||
{ to: "/semaphore", label: "Semaphore", icon: <IconPlayerPlay size={20} /> },
|
{ to: "/semaphore", label: "Semaphore", icon: <IconPlayerPlay size={20} /> },
|
||||||
{ to: "/gitea", label: "Gitea", icon: <IconBrandGit size={20} /> },
|
{ to: "/gitea", label: "Gitea", icon: <IconBrandGit size={20} /> },
|
||||||
|
{ to: "/osticket", label: "osTicket", icon: <IconTicket size={20} /> },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{ to: "/secrets", label: "Secrets", icon: <IconKey size={20} /> },
|
{ to: "/secrets", label: "Secrets", icon: <IconKey size={20} /> },
|
||||||
@@ -111,14 +123,10 @@ export const NAV: NavEntry[] = [
|
|||||||
icon: <IconTool size={20} />,
|
icon: <IconTool size={20} />,
|
||||||
items: [
|
items: [
|
||||||
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
|
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
|
||||||
{ to: "/incidents", label: "Incidents", icon: <IconTimeline size={20} /> },
|
|
||||||
{ to: "/changes", label: "Change log", icon: <IconGitCommit size={20} />, minRole: "operator" },
|
|
||||||
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
|
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
|
||||||
{ to: "/updates", label: "Updates", icon: <IconDownload size={20} /> },
|
{ to: "/updates", label: "Updates", icon: <IconDownload size={20} /> },
|
||||||
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
{ to: "/coming-up", label: "Coming up", icon: <IconCalendarEvent size={20} /> },
|
||||||
{ to: "/osticket", label: "osTicket", icon: <IconTicket size={20} /> },
|
{ to: "/changes", label: "Change log", icon: <IconGitCommit size={20} />, minRole: "operator" },
|
||||||
{ to: "/admin-links", label: "Admin Links", icon: <IconExternalLink size={20} /> },
|
|
||||||
{ to: "/generator", label: "Generator", icon: <IconWand size={20} /> },
|
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -127,6 +135,7 @@ export const NAV: NavEntry[] = [
|
|||||||
icon: <IconSettings size={20} />,
|
icon: <IconSettings size={20} />,
|
||||||
items: [
|
items: [
|
||||||
{ to: "/integrations", label: "Integrations", icon: <IconPlugConnected size={20} /> },
|
{ to: "/integrations", label: "Integrations", icon: <IconPlugConnected size={20} /> },
|
||||||
|
{ to: "/generator", label: "Generator", icon: <IconWand size={20} /> },
|
||||||
{ to: "/users", label: "Users", icon: <IconUsers size={20} />, minRole: "admin" },
|
{ to: "/users", label: "Users", icon: <IconUsers size={20} />, minRole: "admin" },
|
||||||
{ to: "/sessions", label: "Sessions", icon: <IconDevices size={20} />, minRole: "admin" },
|
{ to: "/sessions", label: "Sessions", icon: <IconDevices size={20} />, minRole: "admin" },
|
||||||
{ to: "/audit-log", label: "Audit Log", icon: <IconHistory size={20} />, minRole: "operator" },
|
{ to: "/audit-log", label: "Audit Log", icon: <IconHistory size={20} />, minRole: "operator" },
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ import AckDrawer, { ackWords } from "../components/AckDrawer";
|
|||||||
import { useAlertsReport } from "../hooks/useAlertsReport";
|
import { useAlertsReport } from "../hooks/useAlertsReport";
|
||||||
import { summariseAlerts } from "../utils/alertSummary";
|
import { summariseAlerts } from "../utils/alertSummary";
|
||||||
import { EmptyRow } from "../components/EmptyState";
|
import { EmptyRow } from "../components/EmptyState";
|
||||||
|
import IncidentHistory, { incidentPath } from "../components/IncidentHistory";
|
||||||
|
|
||||||
const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: number }> = {
|
const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: number }> = {
|
||||||
critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 },
|
critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 },
|
||||||
@@ -40,7 +41,41 @@ const CATEGORY_LABELS: Record<AlertCategory, string> = {
|
|||||||
|
|
||||||
type Row = AlertItem & { rank: number };
|
type Row = AlertItem & { rank: number };
|
||||||
|
|
||||||
|
/** Now and History: what is wrong right now, and every time something was raised and went away. The same page, switched by `?view=history`. */
|
||||||
|
function ViewTabs({ view }: { view: "now" | "history" }) {
|
||||||
|
return (
|
||||||
|
<ul className="nav nav-tabs mb-3" aria-label="Alerts view">
|
||||||
|
<li className="nav-item">
|
||||||
|
<Link to="/alerts" className={`nav-link ${view === "now" ? "active" : ""}`} aria-current={view === "now" ? "page" : undefined}>
|
||||||
|
Now
|
||||||
|
</Link>
|
||||||
|
</li>
|
||||||
|
<li className="nav-item">
|
||||||
|
<Link to="/alerts?view=history" className={`nav-link ${view === "history" ? "active" : ""}`} aria-current={view === "history" ? "page" : undefined}>
|
||||||
|
History
|
||||||
|
</Link>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
export default function Alerts({ user }: { user: CurrentUser }) {
|
export default function Alerts({ user }: { user: CurrentUser }) {
|
||||||
|
const [params] = useSearchParams();
|
||||||
|
if (params.get("view") === "history") {
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="d-flex align-items-center mb-2">
|
||||||
|
<h2 className="page-title mb-0">Alerts</h2>
|
||||||
|
</div>
|
||||||
|
<ViewTabs view="history" />
|
||||||
|
<IncidentHistory />
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return <AlertsNow user={user} tabs={<ViewTabs view="now" />} />;
|
||||||
|
}
|
||||||
|
|
||||||
|
function AlertsNow({ user, tabs }: { user: CurrentUser; tabs: React.ReactNode }) {
|
||||||
const canSilence = user.role === "admin" || user.role === "operator";
|
const canSilence = user.role === "admin" || user.role === "operator";
|
||||||
// The count beside Alerts in the menu is a shared one on its own timer; after an acknowledgement it should change at once.
|
// The count beside Alerts in the menu is a shared one on its own timer; after an acknowledgement it should change at once.
|
||||||
const { reload: reloadBadge } = useAlertsReport();
|
const { reload: reloadBadge } = useAlertsReport();
|
||||||
@@ -162,6 +197,7 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
{tabs}
|
||||||
<div className="text-secondary mb-3">
|
<div className="text-secondary mb-3">
|
||||||
What's wrong right now across your servers and integrations — full disks, servers that stopped reporting, failed backups, updates waiting,
|
What's wrong right now across your servers and integrations — full disks, servers that stopped reporting, failed backups, updates waiting,
|
||||||
things about to expire. It runs the same checks that send notifications, whether or not those notifications are switched on.
|
things about to expire. It runs the same checks that send notifications, whether or not those notifications are switched on.
|
||||||
@@ -356,7 +392,7 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
)}
|
)}
|
||||||
{a.incidentId != null && (
|
{a.incidentId != null && (
|
||||||
<div className="small">
|
<div className="small">
|
||||||
<Link to={`/incidents/${a.incidentId}`}>Timeline</Link>
|
<Link to={incidentPath(a.incidentId)}>Timeline</Link>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
{a.target && (
|
{a.target && (
|
||||||
|
|||||||
@@ -0,0 +1,318 @@
|
|||||||
|
import { useEffect, useMemo, useState } from "react";
|
||||||
|
import { Link, useSearchParams } from "react-router-dom";
|
||||||
|
import { api, type Agenda, type AgendaItem, type AgendaKind, type CurrentUser } from "../api/client";
|
||||||
|
import { ErrorNotice } from "../components/Notice";
|
||||||
|
import { BlockSkeleton } from "../components/Skeleton";
|
||||||
|
import EmptyState from "../components/EmptyState";
|
||||||
|
import { downloadCsv } from "../utils/csv";
|
||||||
|
import { formatDate, formatTime } from "../utils/date";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
|
||||||
|
const KINDS: { id: AgendaKind; label: string }[] = [
|
||||||
|
{ id: "secret", label: "Secrets" },
|
||||||
|
{ id: "certificate", label: "Certificates" },
|
||||||
|
{ id: "domain", label: "Domains" },
|
||||||
|
{ id: "tailscale", label: "Tailscale keys" },
|
||||||
|
{ id: "warranty", label: "Warranties" },
|
||||||
|
{ id: "support", label: "End of support" },
|
||||||
|
{ id: "maintenance", label: "Maintenance" },
|
||||||
|
{ id: "restore", label: "Restore tests" },
|
||||||
|
{ id: "updates", label: "Updates waiting" },
|
||||||
|
];
|
||||||
|
const KIND_LABEL = Object.fromEntries(KINDS.map((k) => [k.id, k.label])) as Record<AgendaKind, string>;
|
||||||
|
|
||||||
|
const RANGES: { days: number; label: string }[] = [
|
||||||
|
{ days: 30, label: "Next 30 days" },
|
||||||
|
{ days: 90, label: "Next 3 months" },
|
||||||
|
{ days: 180, label: "Next 6 months" },
|
||||||
|
{ days: 400, label: "Next year" },
|
||||||
|
];
|
||||||
|
|
||||||
|
/** "today", "tomorrow", "in 5 days", "3 days ago". */
|
||||||
|
function relative(daysLeft: number): string {
|
||||||
|
if (daysLeft === 0) return "today";
|
||||||
|
if (daysLeft === 1) return "tomorrow";
|
||||||
|
if (daysLeft === -1) return "yesterday";
|
||||||
|
return daysLeft > 0 ? `in ${daysLeft} days` : `${-daysLeft} days ago`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A day as the app's date format writes it (the string is a plain day, so it is read as one, not shifted by the viewer's zone). */
|
||||||
|
const dayOf = (ymd: string) => {
|
||||||
|
const [y, m, d] = ymd.split("-").map(Number);
|
||||||
|
return new Date(y, m - 1, d);
|
||||||
|
};
|
||||||
|
const weekday = (ymd: string) => dayOf(ymd).toLocaleDateString(undefined, { weekday: "short" });
|
||||||
|
|
||||||
|
interface Section {
|
||||||
|
id: string;
|
||||||
|
title: string;
|
||||||
|
hint?: string;
|
||||||
|
items: AgendaItem[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function sectionsOf(items: AgendaItem[]): Section[] {
|
||||||
|
const now = items.filter((i) => i.daysLeft < 0 || i.kind === "updates");
|
||||||
|
const rest = items.filter((i) => !(i.daysLeft < 0 || i.kind === "updates"));
|
||||||
|
return [
|
||||||
|
{ id: "now", title: "Needs attention now", hint: "Already past its date, or waiting", items: now },
|
||||||
|
{ id: "today", title: "Today", items: rest.filter((i) => i.daysLeft === 0) },
|
||||||
|
{ id: "week", title: "This week", items: rest.filter((i) => i.daysLeft >= 1 && i.daysLeft <= 7) },
|
||||||
|
{ id: "month", title: "The rest of the month", hint: "8 to 30 days away", items: rest.filter((i) => i.daysLeft >= 8 && i.daysLeft <= 30) },
|
||||||
|
{ id: "later", title: "Further ahead", items: rest.filter((i) => i.daysLeft > 30) },
|
||||||
|
].filter((s) => s.items.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function When({ item }: { item: AgendaItem }) {
|
||||||
|
if (item.allDay) {
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div>
|
||||||
|
{weekday(item.at)} {formatDate(dayOf(item.at))}
|
||||||
|
</div>
|
||||||
|
{item.kind !== "updates" && <div className="text-secondary small">{relative(item.daysLeft)}</div>}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const start = new Date(item.at);
|
||||||
|
const end = item.endsAt ? new Date(item.endsAt) : null;
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div>
|
||||||
|
{start.toLocaleDateString(undefined, { weekday: "short" })} {formatDate(start)}
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small">
|
||||||
|
{formatTime(start, false)}
|
||||||
|
{end ? `–${formatTime(end, false)}` : ""} · {relative(item.daysLeft)}
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Everything with a date ahead in one list: secrets, domains and certificates that expire, Tailscale keys, warranties, end of support, planned and recurring
|
||||||
|
* maintenance, restore tests that are due, and updates that have been waiting — nearest first. Narrow it by kind, by how far ahead, or by server. The same list
|
||||||
|
* can be put in a phone's calendar (Settings → Share links → a calendar feed).
|
||||||
|
*/
|
||||||
|
export default function ComingUp({ user }: { user: CurrentUser }) {
|
||||||
|
const [params, setParams] = useSearchParams();
|
||||||
|
const days = RANGES.some((r) => r.days === Number(params.get("days"))) ? Number(params.get("days")) : 90;
|
||||||
|
const chosen = useMemo(() => new Set((params.get("kinds") ?? "").split(",").filter((k): k is AgendaKind => KINDS.some((x) => x.id === k))), [params]);
|
||||||
|
const serverId = /^\d+$/.test(params.get("server") ?? "") ? Number(params.get("server")) : null;
|
||||||
|
|
||||||
|
const [everyOccurrence, setEveryOccurrence] = useState(false);
|
||||||
|
const [agenda, setAgenda] = useState<Agenda | null>(null);
|
||||||
|
const [servers, setServers] = useState<Map<number, string>>(new Map());
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let alive = true;
|
||||||
|
const load = () =>
|
||||||
|
api.agenda
|
||||||
|
.get(400)
|
||||||
|
.then((res) => alive && (setAgenda(res), setError(null)))
|
||||||
|
.catch((err) => alive && setError(readableError(err)));
|
||||||
|
load();
|
||||||
|
api.servers
|
||||||
|
.list()
|
||||||
|
.then((res) => alive && setServers(new Map(res.servers.map((s) => [s.id, s.name]))))
|
||||||
|
.catch(() => {});
|
||||||
|
const t = setInterval(load, 5 * 60_000);
|
||||||
|
return () => {
|
||||||
|
alive = false;
|
||||||
|
clearInterval(t);
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const setParam = (key: string, value: string | null) =>
|
||||||
|
setParams(
|
||||||
|
(prev) => {
|
||||||
|
const next = new URLSearchParams(prev);
|
||||||
|
if (value) next.set(key, value);
|
||||||
|
else next.delete(key);
|
||||||
|
return next;
|
||||||
|
},
|
||||||
|
{ replace: true },
|
||||||
|
);
|
||||||
|
const toggleKind = (id: AgendaKind) => {
|
||||||
|
const next = new Set(chosen);
|
||||||
|
if (next.has(id)) next.delete(id);
|
||||||
|
else next.add(id);
|
||||||
|
setParam("kinds", [...next].join(",") || null);
|
||||||
|
};
|
||||||
|
|
||||||
|
const withinRange = useMemo(() => (agenda?.items ?? []).filter((i) => i.kind === "updates" || i.daysLeft <= days), [agenda, days]);
|
||||||
|
// A schedule that happens every week would take a line a week: by default only its next time is listed, and the rest can be asked for.
|
||||||
|
const { inRange, hiddenRecurring } = useMemo(() => {
|
||||||
|
if (everyOccurrence) return { inRange: withinRange, hiddenRecurring: 0 };
|
||||||
|
const seen = new Set<string>();
|
||||||
|
let hidden = 0;
|
||||||
|
const kept = withinRange.filter((i) => {
|
||||||
|
const schedule = /^schedule:(\d+):/.exec(i.id)?.[1];
|
||||||
|
if (!schedule) return true;
|
||||||
|
if (seen.has(schedule)) {
|
||||||
|
hidden++;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
seen.add(schedule);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
return { inRange: kept, hiddenRecurring: hidden };
|
||||||
|
}, [withinRange, everyOccurrence]);
|
||||||
|
const counts = useMemo(() => {
|
||||||
|
const out = new Map<AgendaKind, number>();
|
||||||
|
for (const i of inRange) out.set(i.kind, (out.get(i.kind) ?? 0) + 1);
|
||||||
|
return out;
|
||||||
|
}, [inRange]);
|
||||||
|
const serversWithItems = useMemo(() => [...new Set(inRange.map((i) => i.serverId).filter((s): s is number => s !== null))].sort((a, b) => (servers.get(a) ?? "").localeCompare(servers.get(b) ?? "")), [inRange, servers]);
|
||||||
|
const shown = inRange.filter((i) => (chosen.size === 0 || chosen.has(i.kind)) && (serverId === null || i.serverId === serverId));
|
||||||
|
const sections = sectionsOf(shown);
|
||||||
|
const filtered = chosen.size > 0 || serverId !== null;
|
||||||
|
|
||||||
|
const needNow = inRange.filter((i) => i.daysLeft < 0 && i.kind !== "updates").length;
|
||||||
|
const thisWeek = inRange.filter((i) => i.daysLeft >= 0 && i.daysLeft <= 7 && i.kind !== "updates").length;
|
||||||
|
const cards = [
|
||||||
|
{ label: "Past their date", value: agenda ? String(needNow) : "…", color: needNow > 0 ? "text-red" : "text-green" },
|
||||||
|
{ label: "In the next 7 days", value: agenda ? String(thisWeek) : "…", color: thisWeek > 0 ? "text-yellow" : "" },
|
||||||
|
{ label: "In the list", value: agenda ? String(shown.length) : "…", color: "" },
|
||||||
|
];
|
||||||
|
|
||||||
|
function exportCsv() {
|
||||||
|
downloadCsv(
|
||||||
|
"coming-up.csv",
|
||||||
|
["Date", "Ends", "What", "Kind", "Details", "Days from today", "Page"],
|
||||||
|
shown.map((i) => [i.at, i.endsAt ?? "", i.title, KIND_LABEL[i.kind], i.detail ?? "", i.daysLeft, i.link]),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
||||||
|
<h2 className="page-title mb-0">Coming up</h2>
|
||||||
|
<button className="btn btn-outline-secondary ms-auto" onClick={exportCsv} disabled={shown.length === 0}>
|
||||||
|
Export CSV
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<p className="text-secondary">
|
||||||
|
What is about to need attention, nearest first: things that expire, hardware that ages out, planned maintenance, restore tests that are due. Each also raises its own alert when it gets close.
|
||||||
|
{user.role === "admin" ? (
|
||||||
|
<>
|
||||||
|
{" "}
|
||||||
|
To have it in your phone’s calendar, make a <Link to="/settings/share">calendar feed</Link>.
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
" An administrator can make a calendar feed of it for your phone."
|
||||||
|
)}
|
||||||
|
</p>
|
||||||
|
<ErrorNotice error={error} />
|
||||||
|
{agenda && agenda.notes.length > 0 && (
|
||||||
|
<div className="alert alert-warning" role="status">
|
||||||
|
Something couldn’t be read, so this list may be missing entries: {agenda.notes.join(" · ")}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<div className="row row-cards mb-3">
|
||||||
|
{cards.map((c) => (
|
||||||
|
<div className="col-6 col-md-4" key={c.label}>
|
||||||
|
<div className="card card-sm">
|
||||||
|
<div className="card-body">
|
||||||
|
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
|
||||||
|
<div className="text-secondary">{c.label}</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-body">
|
||||||
|
<div className="d-flex flex-wrap gap-2 align-items-center mb-2">
|
||||||
|
<select className="form-select w-auto" aria-label="How far ahead" value={days} onChange={(e) => setParam("days", e.target.value === "90" ? null : e.target.value)}>
|
||||||
|
{RANGES.map((r) => (
|
||||||
|
<option key={r.days} value={r.days}>
|
||||||
|
{r.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
{serversWithItems.length > 0 && (
|
||||||
|
<select className="form-select w-auto" aria-label="Server" value={serverId ?? ""} onChange={(e) => setParam("server", e.target.value || null)}>
|
||||||
|
<option value="">Every server</option>
|
||||||
|
{serversWithItems.map((id) => (
|
||||||
|
<option key={id} value={id}>
|
||||||
|
{servers.get(id) ?? `Server ${id}`}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
)}
|
||||||
|
{(hiddenRecurring > 0 || everyOccurrence) && (
|
||||||
|
<label className="form-check mb-0 ms-2">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={everyOccurrence} onChange={(e) => setEveryOccurrence(e.target.checked)} />
|
||||||
|
<span className="form-check-label">Every time a recurring maintenance happens{hiddenRecurring > 0 ? ` (${hiddenRecurring} more)` : ""}</span>
|
||||||
|
</label>
|
||||||
|
)}
|
||||||
|
{filtered && (
|
||||||
|
<button className="btn btn-sm btn-link" onClick={() => setParams(new URLSearchParams(days === 90 ? {} : { days: String(days) }), { replace: true })}>
|
||||||
|
Clear filters
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<div className="d-flex flex-wrap gap-2" role="group" aria-label="Kinds of things">
|
||||||
|
{KINDS.map((k) => {
|
||||||
|
const n = counts.get(k.id) ?? 0;
|
||||||
|
const on = chosen.has(k.id);
|
||||||
|
return (
|
||||||
|
<button key={k.id} type="button" className={`btn btn-sm ${on ? "btn-primary" : "btn-outline-secondary"}`} aria-pressed={on} disabled={n === 0 && !on} onClick={() => toggleKind(k.id)}>
|
||||||
|
{k.label} <span className={`ms-1 ${on ? "" : "text-secondary"}`}>{n}</span>
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{agenda === null && !error && <BlockSkeleton lines={5} />}
|
||||||
|
{agenda !== null && sections.length === 0 && (
|
||||||
|
<div className="card">
|
||||||
|
{filtered ? (
|
||||||
|
<EmptyState filtered title="Nothing matches that" hint="Nothing with a date fits the kinds and server chosen. Try a longer range, or fewer filters." action={{ label: "Clear filters", onClick: () => setParams(new URLSearchParams(days === 90 ? {} : { days: String(days) }), { replace: true }) }} />
|
||||||
|
) : (
|
||||||
|
<EmptyState title="Nothing is coming up" hint={`Nothing expires, ends or is planned in the next ${days} days. Dates come from Secrets, Domains, Inventory (warranty and end of support), Maintenance and the endpoint checks' certificates.`} />
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{sections.map((s) => (
|
||||||
|
<div className="card mb-3" key={s.id}>
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">{s.title}</h3>
|
||||||
|
<div className="card-subtitle text-secondary ms-2">{s.hint ?? ""}</div>
|
||||||
|
</div>
|
||||||
|
<div className="list-group list-group-flush">
|
||||||
|
{s.items.map((i) => (
|
||||||
|
<div className="list-group-item" key={i.id}>
|
||||||
|
<div className="row align-items-center g-2">
|
||||||
|
<div className="col-5 col-md-3 col-lg-2">
|
||||||
|
<When item={i} />
|
||||||
|
</div>
|
||||||
|
<div className="col" style={{ minWidth: 0 }}>
|
||||||
|
<div className="text-break">
|
||||||
|
<Link to={i.link}>{i.title}</Link>
|
||||||
|
{i.serverId !== null && servers.has(i.serverId) && (
|
||||||
|
<span className="text-secondary small">
|
||||||
|
{" "}
|
||||||
|
· <Link to={`/servers/${i.serverId}`}>{servers.get(i.serverId)}</Link>
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
{i.detail && <div className="text-secondary small text-break">{i.detail}</div>}
|
||||||
|
</div>
|
||||||
|
<div className="col-auto">
|
||||||
|
<span className={`badge ${i.daysLeft < 0 && i.kind !== "updates" ? "bg-red-lt text-red" : i.daysLeft <= 7 && i.kind !== "maintenance" && i.kind !== "updates" ? "bg-yellow-lt text-yellow" : "bg-secondary-lt text-secondary"}`}>{KIND_LABEL[i.kind]}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -23,6 +23,7 @@ import StatusStrip from "../components/StatusStrip";
|
|||||||
import SetupGuide, { type SetupStep } from "../components/SetupGuide";
|
import SetupGuide, { type SetupStep } from "../components/SetupGuide";
|
||||||
import { ErrorNotice } from "../components/Notice";
|
import { ErrorNotice } from "../components/Notice";
|
||||||
import { WidgetSkeleton } from "../components/Skeleton";
|
import { WidgetSkeleton } from "../components/Skeleton";
|
||||||
|
import type { Agenda, Heartbeat } from "../api/client";
|
||||||
|
|
||||||
const DNS_TYPE_COLORS: Record<string, string> = {
|
const DNS_TYPE_COLORS: Record<string, string> = {
|
||||||
A: "#60a5fa",
|
A: "#60a5fa",
|
||||||
@@ -364,6 +365,8 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
|||||||
const [serverError, setServerError] = useState<string | null>(null);
|
const [serverError, setServerError] = useState<string | null>(null);
|
||||||
const [domainList, setDomainList] = useState<DomainList | null>(null);
|
const [domainList, setDomainList] = useState<DomainList | null>(null);
|
||||||
const [endpointChecks, setEndpointChecks] = useState<EndpointCheck[] | null>(null);
|
const [endpointChecks, setEndpointChecks] = useState<EndpointCheck[] | null>(null);
|
||||||
|
const [agenda, setAgenda] = useState<Agenda | null>(null);
|
||||||
|
const [heartbeatList, setHeartbeatList] = useState<Heartbeat[] | null>(null);
|
||||||
const [domainError, setDomainError] = useState<string | null>(null);
|
const [domainError, setDomainError] = useState<string | null>(null);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -371,6 +374,14 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
|||||||
.list()
|
.list()
|
||||||
.then((res) => setEndpointChecks(res.checks))
|
.then((res) => setEndpointChecks(res.checks))
|
||||||
.catch(() => setEndpointChecks([]));
|
.catch(() => setEndpointChecks([]));
|
||||||
|
api.agenda
|
||||||
|
.get(30)
|
||||||
|
.then(setAgenda)
|
||||||
|
.catch(() => setAgenda({ generatedAt: "", today: "", items: [], notes: [] }));
|
||||||
|
api.heartbeats
|
||||||
|
.list()
|
||||||
|
.then((res) => setHeartbeatList(res.heartbeats))
|
||||||
|
.catch(() => setHeartbeatList([]));
|
||||||
api.domains
|
api.domains
|
||||||
.list()
|
.list()
|
||||||
.then(setDomainList)
|
.then(setDomainList)
|
||||||
@@ -614,11 +625,28 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
|||||||
const endpointsDown = (endpointChecks ?? []).filter((c) => c.enabled && c.lastOk === false && c.consecutiveFailures >= c.failuresBeforeAlert);
|
const endpointsDown = (endpointChecks ?? []).filter((c) => c.enabled && c.lastOk === false && c.consecutiveFailures >= c.failuresBeforeAlert);
|
||||||
const endpointCerts = (endpointChecks ?? []).filter((c) => c.enabled && c.certExpiresAt && Math.floor((Date.parse(c.certExpiresAt) - Date.now()) / 86_400_000) <= c.warnDays).length;
|
const endpointCerts = (endpointChecks ?? []).filter((c) => c.enabled && c.certExpiresAt && Math.floor((Date.parse(c.certExpiresAt) - Date.now()) / 86_400_000) <= c.warnDays).length;
|
||||||
|
|
||||||
|
// What is coming up: the nearest dated things (not "updates waiting", which has its own card), and whether any has already passed.
|
||||||
|
// A schedule that repeats weekly is one line (its next time), not one a week.
|
||||||
|
const seenSchedules = new Set<string>();
|
||||||
|
const soon = (agenda?.items ?? []).filter((i) => {
|
||||||
|
if (i.kind === "updates") return false;
|
||||||
|
const schedule = /^schedule:(\d+):/.exec(i.id)?.[1];
|
||||||
|
if (!schedule) return true;
|
||||||
|
if (seenSchedules.has(schedule)) return false;
|
||||||
|
seenSchedules.add(schedule);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
const soonPast = soon.filter((i) => i.daysLeft < 0).length;
|
||||||
|
const soonWeek = soon.filter((i) => i.daysLeft >= 0 && i.daysLeft <= 7).length;
|
||||||
|
// The jobs that report in: late, or said they failed.
|
||||||
|
const beatsWrong = (heartbeatList ?? []).filter((h) => h.state === "late" || h.state === "failed");
|
||||||
|
|
||||||
// What has nothing set up yet. These get one list at the bottom instead of a grey "Not configured" card each.
|
// What has nothing set up yet. These get one list at the bottom instead of a grey "Not configured" card each.
|
||||||
const notSetUp: { label: string; to: string; what: string }[] = [];
|
const notSetUp: { label: string; to: string; what: string }[] = [];
|
||||||
if (dnsStats?.totalProviders === 0) notSetUp.push({ label: "DNS", to: "/dns", what: "connect a DNS provider to see zones and records" });
|
if (dnsStats?.totalProviders === 0) notSetUp.push({ label: "DNS", to: "/dns", what: "connect a DNS provider to see zones and records" });
|
||||||
if (secrets !== null && totalSecrets === 0) notSetUp.push({ label: "Secrets", to: "/secrets", what: "track when a token, certificate or password expires" });
|
if (secrets !== null && totalSecrets === 0) notSetUp.push({ label: "Secrets", to: "/secrets", what: "track when a token, certificate or password expires" });
|
||||||
if (serverSummary?.total === 0) notSetUp.push({ label: "Servers", to: "/servers", what: "add a server and install its agent" });
|
if (serverSummary?.total === 0) notSetUp.push({ label: "Servers", to: "/servers", what: "add a server and install its agent" });
|
||||||
|
if (heartbeatList !== null && heartbeatList.length === 0) notSetUp.push({ label: "Heartbeats", to: "/heartbeats", what: "have a backup or cron job report in, so you hear when it doesn't run" });
|
||||||
if (endpointChecks !== null && endpointChecks.length === 0) notSetUp.push({ label: "Endpoints", to: "/endpoints", what: "ask a web address, a port or a certificate whether it is fine" });
|
if (endpointChecks !== null && endpointChecks.length === 0) notSetUp.push({ label: "Endpoints", to: "/endpoints", what: "ask a web address, a port or a certificate whether it is fine" });
|
||||||
if (domainList !== null && domains.length === 0) notSetUp.push({ label: "Domains", to: "/domains", what: "appear after a DNS zone is added and the daily check has run" });
|
if (domainList !== null && domains.length === 0) notSetUp.push({ label: "Domains", to: "/domains", what: "appear after a DNS zone is added and the daily check has run" });
|
||||||
if (integrations !== null) {
|
if (integrations !== null) {
|
||||||
@@ -984,6 +1012,80 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
|||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
</WidgetCard>
|
</WidgetCard>
|
||||||
|
|
||||||
|
<WidgetCard
|
||||||
|
label="Coming up"
|
||||||
|
badge={
|
||||||
|
!agenda ? undefined : soonPast > 0 ? (
|
||||||
|
<span className="badge bg-red-lt text-red">{soonPast} past their date</span>
|
||||||
|
) : soonWeek > 0 ? (
|
||||||
|
<span className="badge bg-yellow-lt text-yellow">{soonWeek} this week</span>
|
||||||
|
) : (
|
||||||
|
<span className="badge bg-green-lt text-green">Nothing soon</span>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{!agenda ? (
|
||||||
|
<WidgetSkeleton />
|
||||||
|
) : soon.length === 0 ? (
|
||||||
|
<div className="text-secondary small">Nothing expires, ends or is planned in the next 30 days.</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
{soon.slice(0, 5).map((i) => (
|
||||||
|
<div key={i.id} className="d-flex gap-2 small mb-1">
|
||||||
|
<span className={`flex-shrink-0 ${i.daysLeft < 0 ? "text-red" : i.daysLeft <= 7 ? "text-yellow" : "text-secondary"}`} style={{ minWidth: "5.5rem" }}>
|
||||||
|
{i.daysLeft === 0 ? "today" : i.daysLeft === 1 ? "tomorrow" : i.daysLeft === -1 ? "yesterday" : i.daysLeft > 0 ? `in ${i.daysLeft} days` : `${-i.daysLeft} days ago`}
|
||||||
|
</span>
|
||||||
|
<Link to={i.link} className="text-break">
|
||||||
|
{i.title}
|
||||||
|
</Link>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
{soon.length > 5 && <div className="text-secondary small">+{soon.length - 5} more</div>}
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
<div className="text-secondary small mt-2">
|
||||||
|
<Link to="/coming-up">Everything coming up</Link>
|
||||||
|
</div>
|
||||||
|
</WidgetCard>
|
||||||
|
|
||||||
|
<WidgetCard
|
||||||
|
label="Heartbeats"
|
||||||
|
hide={heartbeatList !== null && heartbeatList.length === 0}
|
||||||
|
badge={
|
||||||
|
!heartbeatList ? undefined : beatsWrong.length > 0 ? (
|
||||||
|
<span className="badge bg-red-lt text-red">{beatsWrong.length} not reporting</span>
|
||||||
|
) : (
|
||||||
|
<span className="badge bg-green-lt text-green">All reporting</span>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{!heartbeatList ? (
|
||||||
|
<WidgetSkeleton />
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<div className="row g-2 mb-3">
|
||||||
|
<div className="col-4">
|
||||||
|
<MiniStat label="Jobs" value={heartbeatList.filter((h) => h.enabled).length} />
|
||||||
|
</div>
|
||||||
|
<div className="col-4">
|
||||||
|
<MiniStat label="On time" value={heartbeatList.filter((h) => h.state === "up" || h.state === "running").length} />
|
||||||
|
</div>
|
||||||
|
<div className="col-4">
|
||||||
|
<MiniStat label="Late or failed" value={beatsWrong.length} accent={beatsWrong.length > 0 ? "#ef4444" : undefined} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{beatsWrong.slice(0, 3).map((h) => (
|
||||||
|
<div key={h.id} className="text-red small">
|
||||||
|
{h.name} — {h.state === "failed" ? `said it failed${h.lastMessage ? `: ${h.lastMessage}` : ""}` : "hasn't reported in time"}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
<div className="text-secondary small mt-2">
|
||||||
|
<Link to="/heartbeats">All heartbeats</Link>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</WidgetCard>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<SectionHeader
|
<SectionHeader
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import { confirmDialog } from "../utils/dialogs";
|
|||||||
import { readableError } from "../utils/errors";
|
import { readableError } from "../utils/errors";
|
||||||
import { toast } from "../utils/toast";
|
import { toast } from "../utils/toast";
|
||||||
import { EmptyRow } from "../components/EmptyState";
|
import { EmptyRow } from "../components/EmptyState";
|
||||||
|
import EndpointSuggestions from "../components/EndpointSuggestions";
|
||||||
|
|
||||||
const KIND_LABELS: Record<EndpointKind, string> = { http: "Web address", tcp: "Port", tls: "Certificate" };
|
const KIND_LABELS: Record<EndpointKind, string> = { http: "Web address", tcp: "Port", tls: "Certificate" };
|
||||||
const KIND_HINTS: Record<EndpointKind, string> = {
|
const KIND_HINTS: Record<EndpointKind, string> = {
|
||||||
@@ -237,6 +238,8 @@ export default function Endpoints({ user }: { user: CurrentUser }) {
|
|||||||
))}
|
))}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{isAdmin && !draft && <EndpointSuggestions onAdded={load} />}
|
||||||
|
|
||||||
{draft && (
|
{draft && (
|
||||||
<form className="card mb-3" onSubmit={save}>
|
<form className="card mb-3" onSubmit={save}>
|
||||||
<div className="card-header">
|
<div className="card-header">
|
||||||
|
|||||||
@@ -0,0 +1,424 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type CurrentUser, type Heartbeat, type HeartbeatInput, type HeartbeatState } from "../api/client";
|
||||||
|
import CopyButton from "../components/CopyButton";
|
||||||
|
import Pagination from "../components/Pagination";
|
||||||
|
import SortableTh from "../components/SortableTh";
|
||||||
|
import TableToolbar from "../components/TableToolbar";
|
||||||
|
import TimeAgo from "../components/TimeAgo";
|
||||||
|
import { ErrorNotice } from "../components/Notice";
|
||||||
|
import { BlockSkeleton } from "../components/Skeleton";
|
||||||
|
import { EmptyRow } from "../components/EmptyState";
|
||||||
|
import { usePagination } from "../hooks/usePagination";
|
||||||
|
import { useSortable } from "../hooks/useSortable";
|
||||||
|
import { useTableSearch } from "../hooks/useTableSearch";
|
||||||
|
import { downloadCsv } from "../utils/csv";
|
||||||
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
|
import { formatDateTime } from "../utils/date";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
|
||||||
|
const STATE_BADGE: Record<HeartbeatState, { label: string; badge: string }> = {
|
||||||
|
waiting: { label: "Waiting for the first call", badge: "bg-secondary-lt text-secondary" },
|
||||||
|
up: { label: "On time", badge: "bg-green-lt text-green" },
|
||||||
|
running: { label: "Running", badge: "bg-blue-lt text-blue" },
|
||||||
|
late: { label: "Late", badge: "bg-yellow-lt text-yellow" },
|
||||||
|
failed: { label: "Said it failed", badge: "bg-red-lt text-red" },
|
||||||
|
off: { label: "Switched off", badge: "bg-secondary-lt text-secondary" },
|
||||||
|
};
|
||||||
|
const RANK: Record<HeartbeatState, number> = { failed: 0, late: 1, waiting: 2, running: 3, up: 4, off: 5 };
|
||||||
|
|
||||||
|
type Unit = "minutes" | "hours" | "days";
|
||||||
|
const UNIT_MINUTES: Record<Unit, number> = { minutes: 1, hours: 60, days: 1440 };
|
||||||
|
|
||||||
|
/** "every 24 hours", "every day", "every 90 minutes". */
|
||||||
|
function every(minutes: number): string {
|
||||||
|
if (minutes % 1440 === 0) return minutes === 1440 ? "every day" : `every ${minutes / 1440} days`;
|
||||||
|
if (minutes % 60 === 0) return minutes === 60 ? "every hour" : `every ${minutes / 60} hours`;
|
||||||
|
return minutes === 1 ? "every minute" : `every ${minutes} minutes`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The biggest unit that divides a period evenly, so 1440 is edited as "1 day". */
|
||||||
|
function splitPeriod(minutes: number): { value: string; unit: Unit } {
|
||||||
|
if (minutes % 1440 === 0) return { value: String(minutes / 1440), unit: "days" };
|
||||||
|
if (minutes % 60 === 0) return { value: String(minutes / 60), unit: "hours" };
|
||||||
|
return { value: String(minutes), unit: "minutes" };
|
||||||
|
}
|
||||||
|
|
||||||
|
function duration(ms: number | null): string | null {
|
||||||
|
if (ms === null) return null;
|
||||||
|
if (ms < 1000) return `${ms} ms`;
|
||||||
|
const s = Math.round(ms / 1000);
|
||||||
|
return s < 120 ? `${s} s` : s < 7200 ? `${Math.round(s / 60)} min` : `${(s / 3600).toFixed(1)} h`;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Row extends Heartbeat {
|
||||||
|
rank: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The last runs as a row of small ticks: green for a success, red for a failure. */
|
||||||
|
function Ticks({ recent }: { recent: Heartbeat["recent"] }) {
|
||||||
|
if (recent.length === 0) return <span className="text-secondary">—</span>;
|
||||||
|
return (
|
||||||
|
<span className="d-inline-flex gap-1 align-middle" role="img" aria-label={`The last ${recent.length} runs: ${recent.filter((r) => r.kind === "ok").length} reported success`}>
|
||||||
|
{recent.map((r) => (
|
||||||
|
<span key={r.at} title={`${r.at.slice(0, 16).replace("T", " ")} UTC — ${r.kind === "ok" ? "reported success" : "reported a failure"}${r.durationMs !== null ? `, took ${duration(r.durationMs)}` : ""}`} style={{ width: 4, height: 14, borderRadius: 1, background: r.kind === "ok" ? "#2fb344" : "#e05252", display: "inline-block" }} />
|
||||||
|
))}
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Draft {
|
||||||
|
id: number | null;
|
||||||
|
name: string;
|
||||||
|
every: string;
|
||||||
|
unit: Unit;
|
||||||
|
grace: string;
|
||||||
|
serverId: string;
|
||||||
|
notes: string;
|
||||||
|
enabled: boolean;
|
||||||
|
}
|
||||||
|
const EMPTY: Draft = { id: null, name: "", every: "1", unit: "days", grace: "30", serverId: "", notes: "", enabled: true };
|
||||||
|
|
||||||
|
const toDraft = (h: Heartbeat): Draft => ({ id: h.id, ...(({ value, unit }) => ({ every: value, unit }))(splitPeriod(h.periodMinutes)), name: h.name, grace: String(h.graceMinutes), serverId: h.serverId === null ? "" : String(h.serverId), notes: h.notes ?? "", enabled: h.enabled });
|
||||||
|
const toInput = (d: Draft): HeartbeatInput => ({ name: d.name.trim(), periodMinutes: Math.round(Number(d.every) * UNIT_MINUTES[d.unit]), graceMinutes: Number(d.grace), serverId: d.serverId === "" ? null : Number(d.serverId), notes: d.notes.trim() || null, enabled: d.enabled });
|
||||||
|
|
||||||
|
/** What to put in the job: the address, and the three ways to call it. */
|
||||||
|
function HowToCall({ url, onClose, name }: { url: string; onClose: () => void; name: string }) {
|
||||||
|
const cron = `0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null ${url}`;
|
||||||
|
const script = `curl -fsS -m 10 -o /dev/null ${url}/start # it began (optional: gives the run time)
|
||||||
|
./backup.sh && curl -fsS -m 10 -o /dev/null ${url} \\
|
||||||
|
|| curl -fsS -m 10 -o /dev/null --data-raw "exit code $?" ${url}/fail`;
|
||||||
|
return (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Address for {name}</h3>
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
<p className="text-secondary">
|
||||||
|
Have the job call this address when it finishes. Anyone with it can say “I’m alive” for the job — and nothing else — so keep it between the job and you. If it gets out, give the heartbeat a new address.
|
||||||
|
</p>
|
||||||
|
<div className="input-group mb-3">
|
||||||
|
<input className="form-control" readOnly value={url} aria-label="The address to call" onFocus={(e) => e.currentTarget.select()} />
|
||||||
|
<CopyButton text={url} />
|
||||||
|
</div>
|
||||||
|
<div className="mb-1 fw-bold">In cron, after the job:</div>
|
||||||
|
<div className="input-group mb-3">
|
||||||
|
<pre className="form-control mb-0 small text-break" style={{ whiteSpace: "pre-wrap" }}>{cron}</pre>
|
||||||
|
<CopyButton text={cron} />
|
||||||
|
</div>
|
||||||
|
<div className="mb-1 fw-bold">In a script, with a start and a failure:</div>
|
||||||
|
<div className="input-group mb-2">
|
||||||
|
<pre className="form-control mb-0 small text-break" style={{ whiteSpace: "pre-wrap" }}>{script}</pre>
|
||||||
|
<CopyButton text={script} />
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small">
|
||||||
|
Anything sent with the call (<code>--data-raw "3.2 GB copied"</code>, or <code>?msg=…</code>) is kept as a note, up to 200 characters. <code>/fail</code> raises an alert at once;
|
||||||
|
<code> /start</code> lets the next call say how long the run took.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-footer">
|
||||||
|
<button className="btn" onClick={onClose}>
|
||||||
|
Done
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* "Did my 3 a.m. job actually run?" A cron job or backup script calls an address when it finishes; the app writes down that it did and speaks up — an alert and a
|
||||||
|
* notification — when the call doesn't come in time, or when the job says it failed. Everyone can see how the jobs are doing; an administrator sets them up and
|
||||||
|
* holds the addresses.
|
||||||
|
*/
|
||||||
|
export default function Heartbeats({ user }: { user: CurrentUser }) {
|
||||||
|
const isAdmin = user.role === "admin";
|
||||||
|
const [items, setItems] = useState<Heartbeat[] | null>(null);
|
||||||
|
const [servers, setServers] = useState<{ id: number; name: string }[]>([]);
|
||||||
|
const [baseUrl, setBaseUrl] = useState<string | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [draft, setDraft] = useState<Draft | null>(null);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [showing, setShowing] = useState<{ name: string; path: string } | null>(null);
|
||||||
|
|
||||||
|
function load() {
|
||||||
|
api.heartbeats
|
||||||
|
.list()
|
||||||
|
.then((res) => {
|
||||||
|
setItems(res.heartbeats);
|
||||||
|
setServers(res.servers);
|
||||||
|
setBaseUrl(res.baseUrl);
|
||||||
|
})
|
||||||
|
.catch((err) => setError(readableError(err)));
|
||||||
|
}
|
||||||
|
useEffect(() => {
|
||||||
|
load();
|
||||||
|
// Calls come in on their own: look again now and then.
|
||||||
|
const t = setInterval(load, 60_000);
|
||||||
|
return () => clearInterval(t);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const urlOf = (path: string) => `${baseUrl ?? window.location.origin}${path}`;
|
||||||
|
const rows: Row[] = (items ?? []).map((h) => ({ ...h, rank: RANK[h.state] }));
|
||||||
|
const tableSearch = useTableSearch(rows);
|
||||||
|
const { sorted, sortKey, sortDir, requestSort } = useSortable(tableSearch.filtered, "rank", "asc", { url: true });
|
||||||
|
const page = usePagination(sorted);
|
||||||
|
const wrong = rows.filter((r) => r.state === "late" || r.state === "failed").length;
|
||||||
|
const waiting = rows.filter((r) => r.state === "waiting").length;
|
||||||
|
|
||||||
|
async function save(e: React.FormEvent) {
|
||||||
|
e.preventDefault();
|
||||||
|
if (!draft) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError(null);
|
||||||
|
try {
|
||||||
|
const input = toInput(draft);
|
||||||
|
if (draft.id === null) {
|
||||||
|
const { heartbeat } = await api.heartbeats.create(input);
|
||||||
|
toast.success(`Added ${heartbeat.name}.`);
|
||||||
|
// A new one is useless until the job calls it: go straight to showing how.
|
||||||
|
if (heartbeat.path) setShowing({ name: heartbeat.name, path: heartbeat.path });
|
||||||
|
} else {
|
||||||
|
await api.heartbeats.update(draft.id, input);
|
||||||
|
toast.success("Saved.");
|
||||||
|
}
|
||||||
|
setDraft(null);
|
||||||
|
load();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function remove(h: Heartbeat) {
|
||||||
|
if (!(await confirmDialog({ title: "Remove a heartbeat", message: `Remove "${h.name}" and its history? The job's calls to its address will get "not found".`, confirmLabel: "Remove", danger: true }))) return;
|
||||||
|
try {
|
||||||
|
await api.heartbeats.remove(h.id);
|
||||||
|
load();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function newAddress(h: Heartbeat) {
|
||||||
|
if (!(await confirmDialog({ title: "Give it a new address", message: `The old address for "${h.name}" stops working at once, and the job has to be changed to call the new one. Do this if the old one got out.`, confirmLabel: "Make a new address", danger: true }))) return;
|
||||||
|
try {
|
||||||
|
const { path } = await api.heartbeats.newAddress(h.id);
|
||||||
|
setShowing({ name: h.name, path });
|
||||||
|
load();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function exportCsv() {
|
||||||
|
downloadCsv(
|
||||||
|
"heartbeats.csv",
|
||||||
|
["Name", "Expected", "Grace (min)", "Status", "Last call", "Last result", "Note", "Run time (ms)", "Due by", "Calls", "Server"],
|
||||||
|
(sorted ?? []).map((r) => [r.name, every(r.periodMinutes), r.graceMinutes, STATE_BADGE[r.state].label, r.lastPingAt ?? "", r.lastStatus ?? "", r.lastMessage ?? "", r.lastDurationMs ?? "", r.dueBy, r.pingCount, r.server?.name ?? ""]),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const set = <K extends keyof Draft>(key: K, value: Draft[K]) => setDraft((d) => (d ? { ...d, [key]: value } : d));
|
||||||
|
const cards = [
|
||||||
|
{ label: "Heartbeats", value: items ? String(rows.length) : "…", color: "" },
|
||||||
|
{ label: "Late or failed", value: items ? String(wrong) : "…", color: wrong > 0 ? "text-red" : "text-green" },
|
||||||
|
{ label: "Waiting for a first call", value: items ? String(waiting) : "…", color: waiting > 0 ? "text-yellow" : "" },
|
||||||
|
];
|
||||||
|
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="d-flex align-items-center mb-1">
|
||||||
|
<h2 className="page-title mb-0">Heartbeats</h2>
|
||||||
|
{isAdmin && !draft && (
|
||||||
|
<button className="btn btn-primary ms-auto" onClick={() => setDraft({ ...EMPTY })}>
|
||||||
|
Add a heartbeat
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<p className="text-secondary">
|
||||||
|
For jobs that should run on their own — a backup, a sync, a cleanup. The job calls an address when it finishes; if the call doesn’t come in time, or the job says it failed, it appears on the{" "}
|
||||||
|
<Link to="/alerts">Alerts</Link> page and in notifications.
|
||||||
|
</p>
|
||||||
|
<ErrorNotice error={error} />
|
||||||
|
|
||||||
|
{showing && <HowToCall name={showing.name} url={urlOf(showing.path)} onClose={() => setShowing(null)} />}
|
||||||
|
|
||||||
|
<div className="row row-cards mb-3">
|
||||||
|
{cards.map((c) => (
|
||||||
|
<div className="col-6 col-md-4" key={c.label}>
|
||||||
|
<div className="card card-sm">
|
||||||
|
<div className="card-body">
|
||||||
|
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
|
||||||
|
<div className="text-secondary">{c.label}</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{draft && (
|
||||||
|
<form className="card mb-3" onSubmit={save}>
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">{draft.id === null ? "Add a heartbeat" : `Edit ${draft.name || "the heartbeat"}`}</h3>
|
||||||
|
</div>
|
||||||
|
<div className="card-body row g-3">
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="hb-name">
|
||||||
|
Name
|
||||||
|
</label>
|
||||||
|
<input id="hb-name" className="form-control" required maxLength={100} placeholder="e.g. Nightly backup of the NAS" value={draft.name} onChange={(e) => set("name", e.target.value)} />
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="hb-every">
|
||||||
|
It should report every
|
||||||
|
</label>
|
||||||
|
<div className="input-group">
|
||||||
|
<input id="hb-every" type="number" min={1} step="any" className="form-control" required value={draft.every} onChange={(e) => set("every", e.target.value)} />
|
||||||
|
<select className="form-select" aria-label="Unit" value={draft.unit} onChange={(e) => set("unit", e.target.value as Unit)} style={{ maxWidth: "7rem" }}>
|
||||||
|
<option value="minutes">minutes</option>
|
||||||
|
<option value="hours">hours</option>
|
||||||
|
<option value="days">days</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="hb-grace">
|
||||||
|
and may be late by
|
||||||
|
</label>
|
||||||
|
<div className="input-group">
|
||||||
|
<input id="hb-grace" type="number" min={0} max={1440} className="form-control" value={draft.grace} onChange={(e) => set("grace", e.target.value)} />
|
||||||
|
<span className="input-group-text">min</span>
|
||||||
|
</div>
|
||||||
|
<div className="form-hint text-secondary small mt-1">An alert comes once this much time has passed after it was due.</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="hb-server">
|
||||||
|
It runs on the server
|
||||||
|
</label>
|
||||||
|
<select id="hb-server" className="form-select" value={draft.serverId} onChange={(e) => set("serverId", e.target.value)}>
|
||||||
|
<option value="">— none in particular —</option>
|
||||||
|
{servers.map((s) => (
|
||||||
|
<option key={s.id} value={s.id}>
|
||||||
|
{s.name}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<div className="form-hint text-secondary small mt-1">Maintenance on that server silences this heartbeat’s alerts, and its tags can route them.</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-md-5">
|
||||||
|
<label className="form-label" htmlFor="hb-notes">
|
||||||
|
Notes <span className="text-secondary">(optional)</span>
|
||||||
|
</label>
|
||||||
|
<input id="hb-notes" className="form-control" maxLength={500} placeholder="Where the job lives, who to ask" value={draft.notes} onChange={(e) => set("notes", e.target.value)} />
|
||||||
|
</div>
|
||||||
|
<div className="col-md-3 d-flex align-items-end">
|
||||||
|
<label className="form-check mb-2">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={draft.enabled} onChange={(e) => set("enabled", e.target.checked)} />
|
||||||
|
<span className="form-check-label">Switched on</span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-footer d-flex gap-2">
|
||||||
|
<button type="submit" className="btn btn-primary" disabled={busy || draft.name.trim() === "" || !(Number(draft.every) > 0)}>
|
||||||
|
{busy ? "Saving…" : "Save"}
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn" onClick={() => setDraft(null)}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<div className="card">
|
||||||
|
<TableToolbar search={tableSearch.query} onSearch={tableSearch.setQuery} placeholder="Search heartbeats…" shown={sorted?.length} total={rows.length} onExport={exportCsv} exportDisabled={!sorted || sorted.length === 0} />
|
||||||
|
<div className="table-responsive">
|
||||||
|
<table className="table table-vcenter card-table table-stack">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<SortableTh<Row> label="Job" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Status" sortKeyName="rank" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Last call" sortKeyName="lastPingAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Due by" sortKeyName="dueBy" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<th>Recent runs</th>
|
||||||
|
{isAdmin && <th className="w-1">Actions</th>}
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{items === null && !error && (
|
||||||
|
<tr>
|
||||||
|
<td colSpan={6}>
|
||||||
|
<BlockSkeleton lines={3} />
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)}
|
||||||
|
{page.pageItems?.map((h) => (
|
||||||
|
<tr key={h.id}>
|
||||||
|
<td className="stack-wide">
|
||||||
|
<strong>{h.name}</strong> <span className="badge bg-secondary-lt text-secondary">{every(h.periodMinutes)}</span>
|
||||||
|
{h.notes && <div className="text-secondary small text-break">{h.notes}</div>}
|
||||||
|
{h.server && (
|
||||||
|
<div className="small">
|
||||||
|
runs on <Link to={`/servers/${h.server.id}`}>{h.server.name}</Link>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</td>
|
||||||
|
<td>
|
||||||
|
<span className={`badge ${STATE_BADGE[h.state].badge}`}>{STATE_BADGE[h.state].label}</span>
|
||||||
|
{h.state === "failed" && h.lastMessage && <div className="text-secondary small text-break">{h.lastMessage}</div>}
|
||||||
|
</td>
|
||||||
|
<td className="stack-label" data-label="Last call">
|
||||||
|
{h.lastPingAt ? (
|
||||||
|
<>
|
||||||
|
<TimeAgo value={h.lastPingAt} />
|
||||||
|
{duration(h.lastDurationMs) && <div className="text-secondary small">took {duration(h.lastDurationMs)}</div>}
|
||||||
|
{h.state !== "failed" && h.lastMessage && <div className="text-secondary small text-break">{h.lastMessage}</div>}
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<span className="text-secondary">never</span>
|
||||||
|
)}
|
||||||
|
</td>
|
||||||
|
<td className="stack-label" data-label="Due by">
|
||||||
|
{h.enabled ? (
|
||||||
|
<span className={h.state === "late" ? "text-yellow" : "text-secondary"}>{formatDateTime(new Date(h.dueBy))}</span>
|
||||||
|
) : (
|
||||||
|
<span className="text-secondary">—</span>
|
||||||
|
)}
|
||||||
|
</td>
|
||||||
|
<td className="stack-label" data-label="Recent runs">
|
||||||
|
<Ticks recent={h.recent} />
|
||||||
|
</td>
|
||||||
|
{isAdmin && (
|
||||||
|
<td className="stack-wide">
|
||||||
|
<div className="btn-list flex-nowrap">
|
||||||
|
{h.path && (
|
||||||
|
<button className="btn btn-sm btn-outline-primary" onClick={() => setShowing({ name: h.name, path: h.path! })}>
|
||||||
|
Address
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
<button className="btn btn-sm btn-outline-secondary" onClick={() => setDraft(toDraft(h))}>
|
||||||
|
Edit
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-sm btn-outline-secondary" onClick={() => void newAddress(h)}>
|
||||||
|
New address
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-sm btn-outline-danger" onClick={() => void remove(h)}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
)}
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{items !== null && sorted?.length === 0 && (
|
||||||
|
<EmptyRow colSpan={6} {...(tableSearch.active ? { filtered: () => tableSearch.setQuery("") } : { title: "No job is reporting in yet", hint: "Add a heartbeat for a backup, a sync or any job that should run on its own; it calls an address when it finishes, and you hear about it when it doesn't.", action: isAdmin ? { label: "Add a heartbeat", onClick: () => setDraft({ ...EMPTY }) } : undefined })} />
|
||||||
|
)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
<Pagination page={page.page} pageCount={page.pageCount} totalCount={page.totalCount} onPageChange={page.setPage} />
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -11,7 +11,8 @@ import { confirmDialog } from "../utils/dialogs";
|
|||||||
import { formatDateTime } from "../utils/date";
|
import { formatDateTime } from "../utils/date";
|
||||||
import { readableError } from "../utils/errors";
|
import { readableError } from "../utils/errors";
|
||||||
import { toast } from "../utils/toast";
|
import { toast } from "../utils/toast";
|
||||||
import { CATEGORY_NAMES, durationWords, SEVERITY_BADGE } from "./Incidents";
|
import { CATEGORY_NAMES, durationWords, incidentPath, SEVERITY_BADGE } from "../components/IncidentHistory";
|
||||||
|
import { useBreadcrumbTail } from "../utils/breadcrumb";
|
||||||
|
|
||||||
const KIND_ICON: Record<TimelineKind, JSX.Element> = {
|
const KIND_ICON: Record<TimelineKind, JSX.Element> = {
|
||||||
alert: <IconAlertTriangle size={16} />,
|
alert: <IconAlertTriangle size={16} />,
|
||||||
@@ -35,6 +36,8 @@ export default function IncidentDetail({ user }: { user: CurrentUser }) {
|
|||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
const [missing, setMissing] = useState(false);
|
const [missing, setMissing] = useState(false);
|
||||||
const [note, setNote] = useState("");
|
const [note, setNote] = useState("");
|
||||||
|
// Under Alerts in the breadcrumb: "Operations › Alerts › Incident".
|
||||||
|
useBreadcrumbTail("Incident");
|
||||||
const [busy, setBusy] = useState(false);
|
const [busy, setBusy] = useState(false);
|
||||||
|
|
||||||
const load = useCallback(() => {
|
const load = useCallback(() => {
|
||||||
@@ -93,7 +96,7 @@ export default function IncidentDetail({ user }: { user: CurrentUser }) {
|
|||||||
return (
|
return (
|
||||||
<div className="card">
|
<div className="card">
|
||||||
<div className="card-body text-secondary">
|
<div className="card-body text-secondary">
|
||||||
There is no incident like that (it may be old enough to have been removed). <Link to="/incidents">All incidents</Link>.
|
There is no incident like that (it may be old enough to have been removed). <Link to="/alerts?view=history">All incidents</Link>.
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
@@ -113,8 +116,8 @@ export default function IncidentDetail({ user }: { user: CurrentUser }) {
|
|||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<div className="mb-2">
|
<div className="mb-2">
|
||||||
<Link to="/incidents" className="text-secondary">
|
<Link to="/alerts?view=history" className="text-secondary">
|
||||||
← Incidents
|
← Alerts · History
|
||||||
</Link>
|
</Link>
|
||||||
</div>
|
</div>
|
||||||
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
<div className="d-flex flex-wrap align-items-center gap-2 mb-1">
|
||||||
@@ -203,7 +206,7 @@ export default function IncidentDetail({ user }: { user: CurrentUser }) {
|
|||||||
</span>
|
</span>
|
||||||
<div className="flex-fill">
|
<div className="flex-fill">
|
||||||
<div className="d-flex flex-wrap gap-2 align-items-baseline">
|
<div className="d-flex flex-wrap gap-2 align-items-baseline">
|
||||||
<strong>{e.incidentId ? <Link to={`/incidents/${e.incidentId}`}>{e.title}</Link> : e.title}</strong>
|
<strong>{e.incidentId ? <Link to={incidentPath(e.incidentId)}>{e.title}</Link> : e.title}</strong>
|
||||||
<span className="text-secondary small" title={formatDateTime(new Date(e.at))}>
|
<span className="text-secondary small" title={formatDateTime(new Date(e.at))}>
|
||||||
{formatDateTime(new Date(e.at))}
|
{formatDateTime(new Date(e.at))}
|
||||||
</span>
|
</span>
|
||||||
|
|||||||
@@ -242,7 +242,7 @@ export default function NameSettings() {
|
|||||||
</div>
|
</div>
|
||||||
<div className="card-body">
|
<div className="card-body">
|
||||||
<div className="text-secondary small mb-3">
|
<div className="text-secondary small mb-3">
|
||||||
Operations → Generator picks server names from these lists. A name is letters, digits and hyphens (å, ä and ö become a, a and o), so
|
Administration → Generator picks server names from these lists. A name is letters, digits and hyphens (å, ä and ö become a, a and o), so
|
||||||
it works as a hostname. Changes apply when you save.
|
it works as a hostname. Changes apply when you save.
|
||||||
</div>
|
</div>
|
||||||
{drafts.length === 0 ? (
|
{drafts.length === 0 ? (
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ export default function ShareSettings() {
|
|||||||
<>
|
<>
|
||||||
<h3 className="mb-3">Share links</h3>
|
<h3 className="mb-3">Share links</h3>
|
||||||
<p className="text-secondary">
|
<p className="text-secondary">
|
||||||
A share link opens a page <strong>without signing in</strong>: the <strong>wall display</strong> — how things are, on one screen, for a TV in the hall — or a <strong>status page</strong> — whether each of the things you choose is working, with names people at home will recognise, for family or housemates who only want to know "is the movie server up?". It can only look; it can't change anything.
|
A share link opens a page <strong>without signing in</strong>: the <strong>wall display</strong> — how things are, on one screen, for a TV in the hall — or a <strong>status page</strong> — whether each of the things you choose is working, with names people at home will recognise, for family or housemates who only want to know "is the movie server up?" — or a <strong>calendar feed</strong> — what is coming up (expiries, planned maintenance …), for a phone's calendar to subscribe to. It can only look; it can't change anything.
|
||||||
</p>
|
</p>
|
||||||
<div className="alert alert-warning">
|
<div className="alert alert-warning">
|
||||||
Anyone who has the address can open it, so treat it like a password: give it only to people and screens you trust, and remove it when it isn't needed. The address is shown once, when you make it.
|
Anyone who has the address can open it, so treat it like a password: give it only to people and screens you trust, and remove it when it isn't needed. The address is shown once, when you make it.
|
||||||
@@ -128,16 +128,26 @@ export default function ShareSettings() {
|
|||||||
<select id="share-kind" className="form-select" value={kind} onChange={(e) => setKind(e.target.value as ShareLinkKind)}>
|
<select id="share-kind" className="form-select" value={kind} onChange={(e) => setKind(e.target.value as ShareLinkKind)}>
|
||||||
<option value="wall">The wall display — how things are</option>
|
<option value="wall">The wall display — how things are</option>
|
||||||
<option value="status">A status page — is each thing I choose working</option>
|
<option value="status">A status page — is each thing I choose working</option>
|
||||||
|
<option value="calendar">A calendar feed — what is coming up, for a phone's calendar</option>
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
{kind === "wall" && (
|
{(kind === "wall" || kind === "calendar") && (
|
||||||
<div className="col-md-4">
|
<div className="col-md-4">
|
||||||
<label className="form-label" htmlFor="share-detail">
|
<label className="form-label" htmlFor="share-detail">
|
||||||
How much it shows
|
How much it shows
|
||||||
</label>
|
</label>
|
||||||
<select id="share-detail" className="form-select" value={detail} onChange={(e) => setDetail(e.target.value as ShareLinkDetail)}>
|
<select id="share-detail" className="form-select" value={detail} onChange={(e) => setDetail(e.target.value as ShareLinkDetail)}>
|
||||||
|
{kind === "wall" ? (
|
||||||
|
<>
|
||||||
<option value="summary">Numbers only — servers online, alerts, maintenance</option>
|
<option value="summary">Numbers only — servers online, alerts, maintenance</option>
|
||||||
<option value="names">With names — also what is wrong, and the integrations</option>
|
<option value="names">With names — also what is wrong, and the integrations</option>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<option value="summary">Kinds only — "A secret expires", no names</option>
|
||||||
|
<option value="names">With names — which secret, domain, server …</option>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
@@ -170,7 +180,9 @@ export default function ShareSettings() {
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
<div className="col-12 text-secondary small">
|
<div className="col-12 text-secondary small">
|
||||||
{kind === "wall"
|
{kind === "calendar"
|
||||||
|
? "Each secret, domain, certificate, Tailscale key, warranty, end of support, planned or recurring maintenance and restore test that is due becomes an event, refreshed by the calendar app every few hours. “Kinds only” leaves the names out, so the feed can go on a shared calendar without saying what is in the homelab; “with names” also gives the name and a link back to this app. Paste the address into your calendar app under “add a calendar from a URL” (Google Calendar, Apple Calendar, Outlook and Thunderbird all do)."
|
||||||
|
: kind === "wall"
|
||||||
? "“Numbers only” never shows a name. “With names” shows the text of the alerts (which can mention server names, disks and secrets), what is in maintenance, and the names of your integrations — never addresses, tokens or settings."
|
? "“Numbers only” never shows a name. “With names” shows the text of the alerts (which can mention server names, disks and secrets), what is in maintenance, and the names of your integrations — never addresses, tokens or settings."
|
||||||
: "A status page shows only the name you give each thing and whether it is working, being worked on, or can't be told — and, where known, how it has done over 30 days. Never an address, an alert, or what the thing really is. You choose what is on it right after making it."}
|
: "A status page shows only the name you give each thing and whether it is working, being worked on, or can't be told — and, where known, how it has done over 30 days. Never an address, an alert, or what the thing really is. You choose what is on it right after making it."}
|
||||||
</div>
|
</div>
|
||||||
@@ -208,7 +220,7 @@ export default function ShareSettings() {
|
|||||||
<td>
|
<td>
|
||||||
{l.label} <span className="text-secondary small">{l.tokenPrefix}…</span>
|
{l.label} <span className="text-secondary small">{l.tokenPrefix}…</span>
|
||||||
</td>
|
</td>
|
||||||
<td>{l.kind === "status" ? "Status page · " + l.itemCount + (l.itemCount === 1 ? " thing" : " things") : "Wall · " + DETAIL_WORDS[l.detail].toLowerCase()}</td>
|
<td>{l.kind === "status" ? "Status page · " + l.itemCount + (l.itemCount === 1 ? " thing" : " things") : l.kind === "calendar" ? "Calendar · " + (l.detail === "names" ? "with names" : "kinds only") : "Wall · " + DETAIL_WORDS[l.detail].toLowerCase()}</td>
|
||||||
<td className="text-secondary">
|
<td className="text-secondary">
|
||||||
{l.createdBy ?? "—"}, <TimeAgo value={l.createdAt} fromDb />
|
{l.createdBy ?? "—"}, <TimeAgo value={l.createdAt} fromDb />
|
||||||
</td>
|
</td>
|
||||||
@@ -229,7 +241,7 @@ export default function ShareSettings() {
|
|||||||
</tr>
|
</tr>
|
||||||
))}
|
))}
|
||||||
{links?.length === 0 && (
|
{links?.length === 0 && (
|
||||||
<EmptyRow colSpan={6} title="No share links yet" hint="A link lets a wall display or a status page be seen without signing in. Make one with the form above." />
|
<EmptyRow colSpan={6} title="No share links yet" hint="A link lets a wall display, a status page or a calendar feed be used without signing in. Make one with the form above." />
|
||||||
)}
|
)}
|
||||||
{links === null && !error && (
|
{links === null && !error && (
|
||||||
<tr>
|
<tr>
|
||||||
|
|||||||
Reference in new issue
Block a user