Host health from the Linux agent, agent self-update, and disk forecast in /metrics

- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates;
  SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox
- Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new
  "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series
- Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified
  download, atomic replace; GET /api/agent/report/update
- homelab_server_disk_days_to_full in /metrics
- Tests (server, real PowerShell against a fake server) and docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 00:54:24 +02:00
1 parent 540ab704b7
commit 5dc6cea69c
38 files changed
+3884 -73

No files matched your search

+49 -2
View File
@@ -22,7 +22,9 @@ number when it is worth installing the agent again for.
|---|---|
| 1 | Everything before the number was reported — so an agent that says no version is version 1. |
| 2 | Reports pending updates, and its own version. |
| 3 (Linux only) | Asks the package manager outside the agent's systemd sandbox, so `dnf` and `zypper` hosts can answer (below). The Windows agent stays at 2. Install again to get it: the unit file changes too. |
| 3 (Linux) | Asks the package manager outside the agent's systemd sandbox, so `dnf` and `zypper` hosts can answer (below). Install again to get it: the unit file changes too. |
| 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). |
| 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. |
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of
@@ -157,6 +159,50 @@ Get-ScheduledTaskInfo -TaskName 'Homelab Manager Agent' # La
powershell -NoProfile -ExecutionPolicy Bypass -File 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -DryRun # prints what it would send
```
## Letting agents update themselves
**Servers → "Update agents automatically"** (administrators; off by default). With it on, an agent that is behind the script
this Homelab Manager serves downloads the newer one and replaces itself at its next run — no install command to run again.
How it stays safe:
- The agent asks first (`GET /api/agent/report/update?version=N`, with its own token). The answer is yes only when the switch
is on and the agent is behind, and it carries the **SHA-256 of the script**.
- The agent downloads the script and installs it only if the checksum matches, it is newer than the running one, it says
the version that was promised, and it parses (`bash -n` on Linux, PowerShell's own parser on Windows). Anything else is thrown
away and the agent carries on as it was; a note goes to the journal (Linux) or the task output (Windows).
- The new file is written beside the old one and moved over it in one step, so there is never half a script.
- On Linux this happens in the same step that is outside the sandbox (`ExecStartPre=-+…`, below), because the sandbox keeps
`/usr` read-only for the report itself. That step runs first, so the report that follows is already the new version.
On Windows the new script runs from the next report.
- Each update is written down as a line in the server's history ("Agent updated to version 4"), and switching the setting
is in the audit log.
What it can't do:
- **Update an agent that doesn't have the update code yet.** An agent has to contain the self-update itself: Linux version 4 or
later, Windows version 3 or later. Getting there is one install command run on each host (the last one you need to run by hand);
after that, new versions arrive by themselves. A Linux host whose unit was written by version 3 keeps working: the version 4
script still understands the unit's older step name.
- **Replace the systemd unit.** Self-update replaces the script only. A release that also needs a changed unit says so in the
version table above ("install again").
- **Update an agent that isn't reporting.**
- **Choose per server.** It is one switch for all agents.
## What the Linux agent reports about the machine (version 4)
| What | Where it comes from | Notes |
|---|---|---|
| **Failed services** | `systemctl list-units --state=failed` | Read on every report (it works inside the sandbox). Shown on the server's page as *Host health* and raised as an alert of kind *Failing services*. |
| **Disk health** | `smartctl -j -H -A` for each disk it lists | Needs `smartmontools` installed (the section is simply left out without it) and the devices, so it is read in the step outside the sandbox. Taken **about once an hour** and with `-n standby`, so a disk that is asleep isn't woken: what was last known of it is kept (in `/var/lib/homelab-manager-agent`). A disk that says its SMART check **fails** is a critical alert; one that has used 90% or more of its rated life is a warning. Virtual disks have no SMART and are skipped. |
| **ZFS pools** | `zpool list` | Needs the `zpool` command. A pool in any state but ONLINE is a critical alert. |
A report that leaves one of these out leaves what was last said alone (an agent that skips a part never turns an old problem into
"all well"); an empty list is an answer ("none failed") and clears the old one. Windows doesn't report these.
To see exactly what a host finds, as root: `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` — it prints the
update figures, the failed services, the SMART readings and the ZFS pools, says whether it would update itself, and changes nothing.
## How the update check works — and why a server may stay "not reported"
The agent only *asks*; it never installs, downloads or refreshes anything.
@@ -166,12 +212,13 @@ The agent only *asks*; it never installs, downloads or refreshes anything.
| Debian, Ubuntu, Raspberry Pi OS (`apt`) | a simulated upgrade (`apt-get -s upgrade`) | Uses the package lists already on the machine. If nothing refreshes them (`apt update`, or the distribution's own daily timer), the count is only as fresh as they are. Security updates are those from a `-security` archive. |
| Fedora, RHEL and relatives (`dnf`) | `dnf check-update --cacheonly` | Reads dnf's cache only; if the cache is empty the server stays *not reported* (`sudo dnf makecache` fills it, and dnf's own timer keeps it fresh). A restart is flagged by `needs-restarting -r` when that tool is installed. |
| openSUSE, SLES (`zypper`) | `zypper list-updates` without refreshing | Security figure not available (shown as unknown). |
| Arch and relatives (`pacman`) | `pacman -Qu` | What the local copy of the sync databases says could be upgraded (what `checkupdates` lists); packages held back are left out; no security figure. A restart is flagged when the running kernel is no longer the installed one. |
| Windows | Windows Update, through its own API | Searching can take a minute or two, so it is done at most **every six hours** and kept in `%ProgramData%\HomelabManager\updates.json`. It runs in a separate process with a 150-second limit (the scheduled task itself is stopped after five minutes). If the search fails or times out it isn't tried again for an hour, and that report leaves updates out. |
**Not reported** means the agent didn't give a figure. The usual reasons:
1. **It is an older agent.** Update it (above). This is by far the most common.
2. **Another package manager** — Alpine (`apk`), Arch (`pacman`) and others aren't read yet.
2. **Another package manager** — Alpine (`apk`) and others aren't read yet. (The Linux installer needs systemd, which Alpine doesn't use.)
3. **The check couldn't run.** On Linux the agent writes a line to the journal when its package manager
couldn't be read — `journalctl -u homelab-manager-agent -n 30`. The agent itself runs inside a tight systemd
sandbox (read-only system, no hardware, minimal privileges, see `install.sh`), and `dnf` and `zypper` write to
+7 -1
View File
@@ -49,6 +49,12 @@ All gauges. Labels are escaped as Prometheus requires.
| `homelab_server_pending_security_updates` | `server` | Of those, the security fixes, when the package manager can say. |
| `homelab_server_reboot_required` | `server` | 1 if a restart is due. |
| `homelab_server_agent_outdated` | `server` | 1 if the agent is older than the one this installation would install now. |
| `homelab_server_failed_units` | `server` | Services (systemd units) that have failed, when a Linux agent reports them. |
| `homelab_server_disk_healthy` | `server`, `device` | 1 if the disk's own SMART self-assessment passes, 0 if it fails. |
| `homelab_server_disk_wear_percent` | `server`, `device` | How much of an SSD's rated life is used, where the disk says. |
| `homelab_server_disk_temperature_celsius` | `server`, `device` | A disk's temperature. |
| `homelab_server_zfs_pool_online` | `server`, `pool` | 1 if a ZFS pool is ONLINE, 0 in any other state. |
| `homelab_server_disk_days_to_full` | `server`, `mount` | Days until the disk is full at its present rate of filling; only for disks with a steady climb over the last days (left out for the rest). |
| `homelab_alerts` | `severity` | Active alerts: `critical`, `warning`, `info` (not counting silenced ones). |
| `homelab_alerts_silenced` | | Alerts held back by a maintenance window. |
| `homelab_alerts_grouped` | | Alerts folded under a server that is down. |
@@ -60,7 +66,7 @@ All gauges. Labels are escaped as Prometheus requires.
| `homelab_scrape_timestamp_seconds` | | When the answer was built. |
Some alerting ideas: `homelab_server_up == 0`, `homelab_server_disk_used_percent > 90`,
`homelab_secret_expiry_days < 14`, `homelab_alerts{severity="critical"} > 0`.
`homelab_secret_expiry_days < 14`, `homelab_server_disk_days_to_full < 7`, `homelab_alerts{severity="critical"} > 0`.
## Event webhooks
+4 -2
View File
@@ -23,7 +23,7 @@ what the app actually creates.
|---|---|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0028`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0029`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 30 application tables, plus drizzle's own `__drizzle_migrations`. |
**Not in the database:**
@@ -101,7 +101,7 @@ Who changed what. Written by the app on every change (see
| `id` | integer PK | |
| `actor_user_id` | integer → `users.id`, **set null** on delete | Null for automatic actions, and for entries whose user was deleted. |
| `actor_label` | text | The user's name/email as it was at the time (or `system`), so an entry still reads correctly after the account is gone. |
| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings`, `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `share_link` (a link made or removed — never the secret), `runbook`, `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. |
| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `share_link` (a link made or removed — never the secret), `runbook`, `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. |
| `action` | text NOT NULL | `create`, `update`, `delete`, `start`, `stop`, … — free text. |
| `target_type`, `target_id` | text | What it happened to. `target_id` is text so it can hold any kind of ID; there's no foreign key. |
| `detail` | text | JSON, free-form context (what changed, names, counts). Secrets are never put here. |
@@ -219,6 +219,7 @@ what it last reported.
| `no_parent` | bool NOT NULL, default 0 | An admin said this server runs on nothing: no link, and no guess from its Proxmox node. Choosing a server by hand, or going back to automatic, clears it. |
| `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. |
| `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. |
| `host_health` | text | JSON `{"failedUnits":[{unit,description}],"smart":[{device,model,healthy,temperatureC,wearPercent}],"zfs":[{name,health,capacityPercent}],"reportedAt":"…"}`. Each part is present only if the agent reported it, and a report that leaves one out leaves it as it was. Null until an agent says anything. Judged by the health check and shown on the server's page. |
| `agent_version` | integer | Which version of the agent script runs on it, as the agent says in each report. Null until it says; an agent that predates the number counts as version 1. Compared with the scripts the app serves to say which agents are behind. |
#### `scheduled_tasks`
@@ -601,6 +602,7 @@ that were never changed aren't stored; the app fills in defaults when reading.
| `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). |
| `quietHours` | `enabled`, `start`, `end`. |
| `signIn` | Roles from groups: `roleFromGroups` (false), `groupsClaim` (`groups`), `adminGroups`, `operatorGroups`, `otherwise` (`viewer` or `keep`). |
| `agents` | `autoUpdate` (false): whether agents that are behind replace themselves. |
| `inventory` | `currency` (a short name, may be blank) and `pricePerKwh` (null) — what the inventory's running cost is worked out with. |
| `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
| `healthChecks` | `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page). |
+1
View File
@@ -73,6 +73,7 @@ starting baseline rather than announced all at once.
| Check | "Notify on" toggle | Fires when | Configured at |
|---|---|---|---|
| Health — server offline | *Server offline, disk nearly full, or Synology volume/disk problem* | A server's agent hasn't reported for longer than the offline threshold. Held for the first 20 minutes after this app restarts, since agents haven't had a chance to report yet. | Settings → Notifications → Health checks → **Server offline after** (default 60 min) |
| Health — host health | *(same toggle as the others in this group)* | A Linux agent says a service has failed, a disk's SMART check fails (or it has used 90% or more of its rated life), or a ZFS pool is not ONLINE. A server that is currently offline isn't also judged on these (its figures are stale). Silenced by a maintenance window on the server like the others. | The same health-check settings; there is no separate threshold. |
| Health — disk usage | *(same toggle)* | A server disk, a Proxmox node's root filesystem or any of its storages, or a Synology volume, is at or above the usage threshold. A server that's currently offline isn't also judged on disk usage (its figures are stale). | Settings → Notifications → Health checks → **Disk usage alert at** (default 90%) |
| Health — Synology status | *(same toggle)* | A Synology volume's status isn't "normal", or a disk's status/SMART result isn't normal, or it's over the bad-sector or under the remaining-life threshold. | Not configurable |
| Automation — failed run | *Semaphore template or Gitea workflow run failed* | A Semaphore template's, or a Gitea repo's, most recent run status is a clear failure. A run that's still going, was cancelled, or was stopped by hand doesn't count as failed or as a recovery — it's left exactly as it was. For Gitea this follows the repo's most recent run on any workflow or branch. | Not configurable |
+9
View File
@@ -450,6 +450,15 @@ to date"; run its install command again to update its agent (how, and how to
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply
updates, run a playbook.
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
(*Host health*) and become alerts — a failed service is a warning under its own kind, a failing disk or a pool that isn't
ONLINE is critical — with the same maintenance silencing, notifications and runbook links as other alerts. Disk space
heading for full was already forecast from each server's history (alert, weekly digest, trend note); `/metrics`
now carries it too as `homelab_server_disk_days_to_full`. **Servers → Update agents automatically** (off by default) lets
an agent that is behind download the newer script from here, check it against a SHA-256 the server gives, and replace itself.
How it works and what it can't do: [AGENT_UPDATE.md](AGENT_UPDATE.md).
**Runbooks.** Write down what to do when something is wrong — "if elsa is down,
check the UPS first, then…" — on a server's page (the Runbooks card) or an
integration's card. A note can be for anything about the thing, or for one kind of
+2
View File
@@ -94,6 +94,8 @@ even further, to admin only:
- **Inventory**: everyone can see the hardware; adding, editing and removing needs
operator; the price of power used for the running cost is admin-only.
- **Backup coverage** and **Updates** are read-only pages for everyone.
- **Host health** (failed services, disk health, ZFS pools) is shown to everyone on a server's page; **Update agents automatically**
on the Servers page is admin-only.
- **Sign-in groups, metrics tokens, event webhooks and share links** are all under
Settings, so admin-only; `/metrics` itself needs a metrics token, not a login.
- Everything under **Settings** (notification channels, badge colors,
+7 -5
View File
@@ -88,13 +88,15 @@ Description=Report scheduled tasks to Homelab Manager
Type=oneshot
EnvironmentFile=$CONFIG_DIR/homelab-manager-agent.env
ExecStart=$INSTALL_DIR/homelab-manager-agent.sh
# Asking dnf or zypper what is waiting still writes to /var and /run, which the sandbox below makes read-only, so this one step runs
# first and outside it (the + does that; the - lets the report go ahead whatever happens). It only reads what the package manager
# knows and leaves the answer in the unit's private directory, which goes away when the run ends.
ExecStartPre=-+$INSTALL_DIR/homelab-manager-agent.sh --prepare-updates
# A few things can't be done inside the sandbox below, so one step runs first and outside it (the + does that; the - lets the report go
# ahead whatever happens): asking dnf or zypper what is waiting (they write to /var and /run even when only asked), reading disk health and
# ZFS pool state (the sandbox closes the devices), and, only if an administrator has switched it on in Homelab Manager, replacing the agent
# script with a newer one (the sandbox keeps /usr read-only). It leaves its answers in the unit's private directory, which goes away when
# the run ends; the one thing kept between runs is the last SMART reading, in /var/lib/homelab-manager-agent.
ExecStartPre=-+$INSTALL_DIR/homelab-manager-agent.sh --prepare
RuntimeDirectory=homelab-manager-agent
RuntimeDirectoryMode=0700
TimeoutStartSec=420
TimeoutStartSec=600
# The agent only reads (crontabs, timers, /proc, listening sockets) and makes one HTTPS request, but it runs as root. These
# keep a bug in it, or in anything it parses, from being able to change the system. It still sees every process, which is
# how it names the program behind each listening port. Directives an older systemd doesn't know are ignored.
+219 -42
View File
@@ -13,8 +13,9 @@ set -euo pipefail
# Which version of this script this is, sent with every report so Homelab Manager can tell which agents are behind. Raise it whenever a
# change here is worth installing the agent again for (and add it to the list at the top of server/src/services/agentVersions.ts).
# 1 = before the number was reported 2 = reports pending updates, and its own version
# 3 = the package manager is asked outside the sandbox (see --prepare-updates), so dnf and zypper can answer there too
AGENT_VERSION=3
# 3 = the package manager is asked outside the sandbox (see --prepare), so dnf and zypper can answer there too
# 4 = also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can update itself when told to
AGENT_VERSION=4
ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}"
if [[ -f "$ENV_FILE" ]]; then
@@ -29,16 +30,21 @@ DRY_RUN=0
[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=1
# Two extra ways to run it, neither of which reports anything:
# --prepare-updates asks the package manager what is waiting and leaves the answer in $UPDATES_FILE for the report to pick up. The
# systemd unit runs this step outside its sandbox (ExecStartPre=+), because dnf and zypper write to /var and /run
# even when only asked a question, and the sandbox makes those read-only. A failure here only means no figure.
# --check-updates for a person at a terminal: runs the same check, shows what the package manager said and the result.
# --prepare the things that can't be done inside the systemd sandbox: asks the package manager what is waiting (dnf and zypper
# write to /var and /run even when only asked), reads disk health with smartctl and pool state with zpool (both need the
# devices, which the sandbox closes), and — if an administrator has switched it on — replaces this script with a newer
# one. The answers are left in the unit's private directory for the report to pick up. The unit runs this step outside
# its sandbox (ExecStartPre=-+); a failure here only means those figures are missing. --prepare-updates is the same.
# --check-updates for a person at a terminal: runs the same checks and shows what each said, and whether it would update itself.
MODE=report
case "${1:-}" in
--prepare-updates) MODE=prepare ;;
--check-updates) MODE=check ;;
--prepare|--prepare-updates) MODE=prepare ;;
--check-updates|--check) MODE=check ;;
esac
UPDATES_FILE="${UPDATES_FILE:-/run/homelab-manager-agent/updates.json}"
HEALTH_FILE="${HEALTH_FILE:-$(dirname "$UPDATES_FILE")/health.json}"
# Kept between runs (the unit's private directory is not): a SMART reading is only taken about once an hour, so sleeping disks are left alone.
STATE_DIR="${STATE_DIR:-/var/lib/homelab-manager-agent}"
# Where the package managers' own error output goes: nowhere, unless a person asked to see it.
PM_ERR=/dev/null
[[ "$MODE" == "check" ]] && PM_ERR=/dev/stderr
@@ -228,6 +234,16 @@ collect_updates() {
if command -v needs-restarting >/dev/null 2>&1; then
needs-restarting -r >/dev/null 2>&1 || reboot="true"
fi
elif command -v pacman >/dev/null 2>&1; then
manager="pacman"
# -Qu is what the local copy of the sync databases says could be upgraded (what `checkupdates` lists, without changing anything). It
# exits 1 when there is nothing to upgrade, so an empty answer with status 1 is "none", not an error. Packages held back are left out.
out=$(LC_ALL=C timeout 60 pacman -Qu 2>"$PM_ERR") || status=$?
if [[ "$status" -eq 0 || ( "$status" -eq 1 && -z "$out" ) ]]; then
count=$(grep -v '\[ignored\]' <<<"$out" | grep -c . || true)
fi
# No security figure (pacman doesn't say). A restart is due when the kernel running is no longer the one installed.
[[ -d "/usr/lib/modules/$(uname -r)" ]] || reboot="true"
elif command -v zypper >/dev/null 2>&1; then
manager="zypper"
if out=$(LC_ALL=C timeout 120 zypper -q --no-refresh list-updates 2>"$PM_ERR"); then
@@ -248,7 +264,164 @@ collect_updates() {
fi
}
# The answer --prepare-updates left for this report, if it is there, recent and looks like what it should. Read back through jq rather than
# >>> post_report
# Sends $PAYLOAD to the server and prints the HTTP status code.
#
# Nothing sensitive goes on curl's command line: any user on this machine can read other processes' command lines, and the
# token would be there for the length of every report. So the token is handed to curl on its standard input, as a line of
# curl's own config format, and the payload and the server's reply go through files in a private directory (mktemp -d is
# readable by root only). A fixed name in /tmp is no good either — another user could plant a symlink there and have this
# script, running as root, overwrite whatever it points at.
curl_quote() {
local s=$1
s=${s//\\/\\\\}
s=${s//\"/\\\"}
printf '%s' "$s"
}
post_report() {
case "$API_TOKEN" in
*$'\n'*|*$'\r'*) echo "API_TOKEN must not contain a line break." >&2; return 1 ;;
esac
printf '%s' "$PAYLOAD" > "$HLM_TMPDIR/payload.json"
printf 'header = "Authorization: Bearer %s"\n' "$(curl_quote "$API_TOKEN")" | curl -sS "${CURL_INSECURE_FLAG[@]}" -K - \
-o "$HLM_TMPDIR/response.json" -w "%{http_code}" \
-X POST "$API_URL/api/agent/report" \
-H "Content-Type: application/json" \
--data-binary @"$HLM_TMPDIR/payload.json"
}
# <<< post_report
# A GET with the token handed to curl on its standard input (as the upload does), the reply into the file named; prints the HTTP status.
api_get() {
printf 'header = "Authorization: Bearer %s"\n' "$(curl_quote "$API_TOKEN")" | curl -sS "${CURL_INSECURE_FLAG[@]}" -K - -m 30 -o "$2" -w "%{http_code}" "$API_URL$1"
}
# Services (systemd units) that have failed. systemctl is asked the way the timers are, so this works inside the sandbox.
collect_failed_units() {
FAILED_UNITS_JSON="null"
command -v systemctl >/dev/null 2>&1 || return 0
local out
out=$(timeout 20 systemctl list-units --state=failed --plain --no-legend --no-pager --full 2>"$PM_ERR") || return 0
FAILED_UNITS_JSON=$(awk 'NF > 0 { unit = $1; $1 = $2 = $3 = $4 = ""; sub(/^ +/, ""); print unit "\t" $0 }' <<<"$out" \
| jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({unit: .[0], description: (.[1] // "")}) | .[0:100]') || FAILED_UNITS_JSON="null"
}
# Each disk's own SMART verdict, temperature and (where it says) how much of its rated life is used. Needs the devices, so it runs from
# --prepare. Asked at most about once an hour, and with -n standby so a sleeping disk is not woken: what was last known of it is kept.
collect_smart() {
SMART_JSON="null"
command -v smartctl >/dev/null 2>&1 || return 0
local cache="$STATE_DIR/smart.json" now_s previous="[]" items="[]" dev out item scan checked
now_s=$(date +%s)
if [[ -r "$cache" ]]; then
checked=$(jq -r '.checked_at // 0' "$cache" 2>/dev/null || echo 0)
previous=$(jq -c '.devices // []' "$cache" 2>/dev/null || echo "[]")
if [[ "$checked" =~ ^[0-9]+$ ]] && (( now_s - checked < 3300 )) && [[ "$MODE" != "check" ]]; then
SMART_JSON=$previous
return 0
fi
fi
scan=$(timeout 30 smartctl --scan -j 2>"$PM_ERR") || true
while IFS= read -r dev; do
[[ -n "$dev" ]] || continue
# smartctl's exit status is a bit mask that says "something was noted" more often than "it failed", so it is read from the JSON instead.
out=$(timeout 30 smartctl -j -H -A -n standby "$dev" 2>"$PM_ERR") || true
item=$(jq -c --arg dev "$dev" 'select(.smart_status != null) | {
device: $dev,
model: (.model_name // .scsi_model_name // ""),
healthy: .smart_status.passed,
temperature_c: (.temperature.current // null),
wear_percent: (.nvme_smart_health_information_log.percentage_used // ([.ata_smart_attributes.table[]? | select(.id == 177 or .id == 231 or .id == 233) | (100 - .value)] | first) // null)
}' <<<"$out" 2>/dev/null) || item=""
if [[ -z "$item" ]]; then
# Asleep, or no SMART at all (a virtual disk): keep what was known of it, if anything.
item=$(jq -c --arg dev "$dev" '.[] | select(.device == $dev)' <<<"$previous" 2>/dev/null | head -n 1) || item=""
fi
if [[ -n "$item" ]]; then items=$(jq -c --argjson i "$item" '. + [$i]' <<<"$items"); fi
done < <(jq -r '.devices[]?.name' <<<"$scan" 2>/dev/null)
SMART_JSON=$items
if mkdir -p "$STATE_DIR" 2>/dev/null; then
chmod 700 "$STATE_DIR" 2>/dev/null || true
jq -c -n --argjson at "$now_s" --argjson devices "$items" '{checked_at: $at, devices: $devices}' > "$cache.tmp" 2>/dev/null && mv -f "$cache.tmp" "$cache" 2>/dev/null || true
fi
}
# Each ZFS pool and whether it is ONLINE. Needs /dev/zfs, so it runs from --prepare too.
collect_zfs() {
ZFS_JSON="null"
command -v zpool >/dev/null 2>&1 || return 0
local out
out=$(timeout 20 zpool list -H -o name,health,capacity 2>"$PM_ERR") || return 0
ZFS_JSON=$(jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({name: .[0], health: (.[1] // "UNKNOWN"), capacity_percent: ((.[2] // "" | rtrimstr("%") | tonumber?) // null)})' <<<"$out") || ZFS_JSON="null"
}
# The disk and pool readings --prepare left for this report, if they are there and recent. Re-shaped by jq, never trusted as they stand.
use_prepared_health() {
[[ -r "$HEALTH_FILE" ]] || return 1
[[ -n "$(find "$HEALTH_FILE" -mmin -10 -print 2>/dev/null)" ]] || return 1
SMART_JSON=$(jq -c 'if (.smart | type) == "array" then [.smart[] | {device: (.device | tostring), model: ((.model // "") | tostring),
healthy: (if (.healthy | type) == "boolean" then .healthy else null end),
temperature_c: (if (.temperature_c | type) == "number" then .temperature_c else null end),
wear_percent: (if (.wear_percent | type) == "number" then .wear_percent else null end)}] else null end' "$HEALTH_FILE" 2>/dev/null) || SMART_JSON="null"
ZFS_JSON=$(jq -c 'if (.zfs | type) == "array" then [.zfs[] | {name: (.name | tostring), health: ((.health // "UNKNOWN") | tostring),
capacity_percent: (if (.capacity_percent | type) == "number" then .capacity_percent else null end)}] else null end' "$HEALTH_FILE" 2>/dev/null) || ZFS_JSON="null"
return 0
}
# Replaces this script with the one this Homelab Manager serves, when an administrator has switched automatic updates on and this agent is
# behind. Nothing is trusted but the checksum: the (authenticated) answer gives the SHA-256 of the script, and a download that doesn't
# match it, doesn't look like this script, doesn't pass bash's syntax check or isn't newer is thrown away. The new file is put beside the
# old and moved over it in one step, and the new version then carries on from here. Runs only from --prepare (outside the sandbox, which
# keeps /usr read-only for the report itself); with --check-updates it says what it would do.
self_update() {
[[ -n "$API_URL" && -n "$API_TOKEN" ]] || return 0
[[ -z "${HLM_JUST_UPDATED:-}" ]] || return 0
local self work code reply_update version sha got
self=$(readlink -f "${BASH_SOURCE[0]}" 2>/dev/null) || return 0
[[ -f "$self" ]] || return 0
work=$(mktemp -d) || return 0
code=$(api_get "/api/agent/report/update?version=$AGENT_VERSION" "$work/reply.json" 2>/dev/null) || code=""
reply_update=$(jq -r '.update // false' "$work/reply.json" 2>/dev/null || echo false)
if [[ "$code" != "200" || "$reply_update" != "true" ]]; then
[[ "$MODE" == "check" ]] && echo "Automatic update: nothing to do (it is off, or this agent is current)."
rm -rf "$work"
return 0
fi
version=$(jq -r '.version // empty' "$work/reply.json" 2>/dev/null || true)
sha=$(jq -r '.sha256 // empty' "$work/reply.json" 2>/dev/null || true)
if [[ ! "$version" =~ ^[0-9]+$ || ! "$sha" =~ ^[0-9a-f]{64}$ ]] || (( version <= AGENT_VERSION )); then
rm -rf "$work"
return 0
fi
if ! curl -fsS "${CURL_INSECURE_FLAG[@]}" -m 60 -o "$work/new.sh" "$API_URL/agent/linux/report-tasks.sh" 2>"$PM_ERR"; then
echo "Note: couldn't download the newer agent (version $version), so it was not installed." >&2
rm -rf "$work"
return 0
fi
got=$(sha256sum "$work/new.sh" 2>/dev/null | awk '{print $1}')
if [[ "$got" != "$sha" ]] || [[ "$(head -n 1 "$work/new.sh")" != "#!/usr/bin/env bash" ]] || ! bash -n "$work/new.sh" 2>/dev/null || ! grep -q "^AGENT_VERSION=$version\$" "$work/new.sh"; then
echo "Note: the newer agent (version $version) didn't check out against what the server said, so it was not installed." >&2
rm -rf "$work"
return 0
fi
if [[ "$MODE" == "check" ]]; then
echo "Automatic update: version $version is available and would be installed on the next run."
rm -rf "$work"
return 0
fi
if install -m 755 "$work/new.sh" "$self.new" && mv -f "$self.new" "$self"; then
echo "Updated the agent from version $AGENT_VERSION to version $version." >&2
rm -rf "$work"
HLM_JUST_UPDATED=1 exec "$self" --prepare
fi
rm -f "$self.new"
echo "Note: couldn't replace the agent with version $version." >&2
rm -rf "$work"
return 0
}
# The answer --prepare left for this report, if it is there, recent and looks like what it should. Read back through jq rather than
# trusted: the report only ever carries these four fields, whatever the file holds.
use_prepared_updates() {
[[ -r "$UPDATES_FILE" ]] || return 1
@@ -325,8 +498,23 @@ collect_system_info() {
UPDATES_JSON="null"
use_prepared_updates || collect_updates || UPDATES_JSON="null"
# Failed services are asked here (systemctl works inside the sandbox); disk and pool health come from --prepare. A person running --dry-run
# isn't in the sandbox, so there they are read directly.
FAILED_UNITS_JSON="null"
collect_failed_units || FAILED_UNITS_JSON="null"
SMART_JSON="null"
ZFS_JSON="null"
if ! use_prepared_health && [[ "$DRY_RUN" == "1" ]]; then
collect_smart || SMART_JSON="null"
collect_zfs || ZFS_JSON="null"
fi
HEALTH_JSON=$(jq -n -c --argjson failed "$FAILED_UNITS_JSON" --argjson smart "$SMART_JSON" --argjson zfs "$ZFS_JSON" \
'{} | (if $failed != null then . + {failed_units: $failed} else . end) | (if $smart != null then . + {smart: $smart} else . end) | (if $zfs != null then . + {zfs: $zfs} else . end)') || HEALTH_JSON="{}"
[[ "$HEALTH_JSON" == "{}" ]] && HEALTH_JSON="null"
SYSTEM_JSON=$(jq -n \
--argjson updates "$UPDATES_JSON" \
--argjson health "$HEALTH_JSON" \
--argjson ip_addresses "$ip_json" \
--argjson listening_ports "$ports_json" \
--arg cpu_model "$cpu_model" \
@@ -341,23 +529,40 @@ collect_system_info() {
memory: { total_bytes: $mem_total_bytes, used_bytes: $mem_used_bytes },
disks: $disks,
listening_ports: $listening_ports,
updates: $updates
updates: $updates,
health: $health
}')
}
if [[ "$MODE" != "report" ]]; then
collect_updates
# prepare: first see whether this script is to be replaced (if so it carries on as the new one, from the top).
[[ "$MODE" == "prepare" ]] && self_update
collect_updates || UPDATES_JSON="null"
collect_smart || SMART_JSON="null"
collect_zfs || ZFS_JSON="null"
if [[ "$MODE" == "check" ]]; then
collect_failed_units || FAILED_UNITS_JSON="null"
echo "Pending updates as they would be reported:"
jq . <<<"$UPDATES_JSON"
echo "Failed services:"
jq . <<<"$FAILED_UNITS_JSON"
echo "Disk health (SMART; null means smartctl isn't installed or this isn't root):"
jq . <<<"$SMART_JSON"
echo "ZFS pools (null means there is no zpool here):"
jq . <<<"$ZFS_JSON"
self_update
exit 0
fi
# prepare: leave the answer (or no file at all) for the report that follows. Written whole and moved into place, so a report never reads half of it.
rm -f "$UPDATES_FILE"
if [[ "$UPDATES_JSON" != "null" ]]; then
# Leave the answers (or no file at all) for the report that follows. Each is written whole and moved into place, so a report never reads half of one.
rm -f "$UPDATES_FILE" "$HEALTH_FILE"
mkdir -p "$(dirname "$UPDATES_FILE")" 2>/dev/null || exit 0
if [[ "$UPDATES_JSON" != "null" ]]; then
printf '%s\n' "$UPDATES_JSON" > "$UPDATES_FILE.tmp" 2>/dev/null && mv -f "$UPDATES_FILE.tmp" "$UPDATES_FILE" 2>/dev/null || true
fi
HEALTH_FILE_JSON=$(jq -n -c --argjson smart "${SMART_JSON:-null}" --argjson zfs "${ZFS_JSON:-null}" '{} | (if $smart != null then . + {smart: $smart} else . end) | (if $zfs != null then . + {zfs: $zfs} else . end)' 2>/dev/null || echo "{}")
if [[ "$HEALTH_FILE_JSON" != "{}" ]]; then
printf '%s\n' "$HEALTH_FILE_JSON" > "$HEALTH_FILE.tmp" 2>/dev/null && mv -f "$HEALTH_FILE.tmp" "$HEALTH_FILE" 2>/dev/null || true
fi
exit 0
fi
@@ -392,34 +597,6 @@ if [[ "$DRY_RUN" == "1" ]]; then
exit 0
fi
# >>> post_report
# Sends $PAYLOAD to the server and prints the HTTP status code.
#
# Nothing sensitive goes on curl's command line: any user on this machine can read other processes' command lines, and the
# token would be there for the length of every report. So the token is handed to curl on its standard input, as a line of
# curl's own config format, and the payload and the server's reply go through files in a private directory (mktemp -d is
# readable by root only). A fixed name in /tmp is no good either — another user could plant a symlink there and have this
# script, running as root, overwrite whatever it points at.
curl_quote() {
local s=$1
s=${s//\\/\\\\}
s=${s//\"/\\\"}
printf '%s' "$s"
}
post_report() {
case "$API_TOKEN" in
*$'\n'*|*$'\r'*) echo "API_TOKEN must not contain a line break." >&2; return 1 ;;
esac
printf '%s' "$PAYLOAD" > "$HLM_TMPDIR/payload.json"
printf 'header = "Authorization: Bearer %s"\n' "$(curl_quote "$API_TOKEN")" | curl -sS "${CURL_INSECURE_FLAG[@]}" -K - \
-o "$HLM_TMPDIR/response.json" -w "%{http_code}" \
-X POST "$API_URL/api/agent/report" \
-H "Content-Type: application/json" \
--data-binary @"$HLM_TMPDIR/payload.json"
}
# <<< post_report
HLM_TMPDIR=$(mktemp -d)
trap 'rm -rf "$HLM_TMPDIR"' EXIT
+3
View File
@@ -33,6 +33,9 @@ rm -f \
"$INSTALL_DIR/homelab-manager-agent.sh" \
/tmp/hlm-agent-response.json
# What it remembered between runs (the last disk-health reading).
rm -rf /var/lib/homelab-manager-agent
if command -v systemctl >/dev/null 2>&1; then
systemctl daemon-reload
fi
+69 -1
View File
@@ -33,7 +33,8 @@ $ErrorActionPreference = 'Stop'
# Which version of this script this is, sent with every report so Homelab Manager can tell which agents are behind. Raise it whenever a
# change here is worth installing the agent again for. 1 = before the number was reported 2 = reports pending updates, and its own version
$script:AgentVersion = 2
# 3 = can replace itself with the newer script when an administrator has switched that on in Homelab Manager
$script:AgentVersion = 3
$script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json'
@@ -344,6 +345,70 @@ function Send-Report($Config, [string]$Json) {
return Invoke-RestMethod @params
}
# ---- updating itself -------------------------------------------------------------
# A GET to the server with the token, set up the same way as the report (modern TLS on 5.1, and the self-signed switch).
function New-GetParams($Config, [string]$Path) {
$params = @{
Uri = "$($Config.ApiUrl)$Path"
Method = 'Get'
Headers = @{ Authorization = "Bearer $($Config.ApiToken)" }
TimeoutSec = 60
}
if ($PSVersionTable.PSVersion.Major -ge 6) {
if ($Config.Insecure) { $params['SkipCertificateCheck'] = $true }
} else {
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
if ($Config.Insecure) { [Net.ServicePointManager]::ServerCertificateValidationCallback = { $true } }
$params['UseBasicParsing'] = $true
}
return $params
}
# Whether a downloaded copy may replace the running script: it is newer, it is exactly the file the server vouched for (SHA-256), it says the
# version that was promised, and it parses. Anything else is thrown away.
function Test-AgentDownload([string]$Path, [string]$ExpectedSha256, [int]$NewVersion, [int]$CurrentVersion) {
if ($NewVersion -le $CurrentVersion) { return $false }
if ($ExpectedSha256 -notmatch '^[0-9a-f]{64}$') { return $false }
$got = (Get-FileHash -LiteralPath $Path -Algorithm SHA256).Hash.ToLowerInvariant()
if ($got -ne $ExpectedSha256) { return $false }
$text = [System.IO.File]::ReadAllText($Path)
if ($text -notmatch ('(?m)^\$script:AgentVersion\s*=\s*' + $NewVersion + '\s*$')) { return $false }
$tokens = $null
$errors = $null
[void][System.Management.Automation.Language.Parser]::ParseInput($text, [ref]$tokens, [ref]$errors)
return ($errors.Count -eq 0)
}
# Replaces this script with the one this Homelab Manager serves, when an administrator has switched automatic updates on and this agent is
# behind. The new one runs from the next report. Nothing goes wrong if it can't: the report carries on as it was. Only a script that is
# installed under its own name is ever replaced, so running a copy from somewhere else (a test, a checkout) never changes it.
function Update-AgentSelf($Config, [string]$Self = $PSCommandPath) {
$self = $Self
if (-not $self -or (Split-Path -Leaf $self) -ne 'homelab-manager-agent.ps1') { return }
$download = "$self.download"
try {
$ask = New-GetParams $Config "/api/agent/report/update?version=$($script:AgentVersion)"
$reply = Invoke-RestMethod @ask
if (-not $reply.update) { return }
$version = [int]$reply.version
$fetch = New-GetParams $Config '/agent/windows/report-tasks.ps1'
$fetch.Remove('Headers')
$fetch['OutFile'] = $download
$null = Invoke-WebRequest @fetch
if (-not (Test-AgentDownload $download ([string]$reply.sha256) $version $script:AgentVersion)) {
[Console]::Error.WriteLine("Note: the newer agent (version $version) didn't check out against what the server said, so it was not installed.")
return
}
Move-Item -LiteralPath $download -Destination $self -Force
[Console]::Error.WriteLine("Updated the agent from version $($script:AgentVersion) to version $version; the new one runs at the next report.")
} catch {
[Console]::Error.WriteLine("Note: couldn't check for, or install, a newer agent: $($_.Exception.Message)")
} finally {
if (Test-Path -LiteralPath $download) { Remove-Item -LiteralPath $download -Force -ErrorAction SilentlyContinue }
}
}
# ---- main -----------------------------------------------------------------------
# A plain one-line message on stderr: Write-Error would bury it in a stack trace on every manual run.
@@ -358,6 +423,9 @@ function Invoke-Agent {
Stop-Agent "API_URL and API_TOKEN must be set (environment variables, or $script:DefaultConfigPath)."
}
# A dry run reports nothing and changes nothing, so it never updates the agent either.
if (-not $DryRun) { Update-AgentSelf $config }
$tasks = @()
try {
$tasks = @(Get-ReportedTasks $config.IncludeMicrosoftTasks)
@@ -0,0 +1 @@
ALTER TABLE `servers` ADD `host_health` text;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -204,6 +204,13 @@
"when": 1792267930026,
"tag": "0028_thankful_ultron",
"breakpoints": true
},
{
"idx": 29,
"version": "6",
"when": 1792273577264,
"tag": "0029_past_monster_badoon",
"breakpoints": true
}
]
}
+3
View File
@@ -297,6 +297,9 @@ export const servers = sqliteTable("servers", {
// When updates were first seen waiting (and none were before) — how long the oldest has been waiting. Null when none are.
updatesPendingSince: text("updates_pending_since"),
// What the agent says about the machine's own health, as JSON: {failedUnits, smart, zfs, reportedAt} — see services/hostHealth.ts. Null until it says.
hostHealth: text("host_health"),
// Which version of the agent script is running on it, as the agent says in each report (see agent/*/report-tasks.*). Null until it says:
// an agent that predates the number being reported is version 1.
agentVersion: integer("agent_version"),
+45
View File
@@ -4,6 +4,8 @@ import { db } from "../db/client.js";
import { hashToken } from "../services/tokens.js";
import { syncServerTasks } from "../services/taskSync.js";
import { asyncHandler } from "../utils/asyncHandler.js";
import { getSettings } from "../services/settingsStore.js";
import { latestAgentSha256, latestAgentVersions } from "../services/agentVersions.js";
export const agentReportRouter = Router();
@@ -31,7 +33,33 @@ const updatesSchema = z.object({
reboot_required: z.boolean().optional(),
});
// Failed services, disk health, ZFS pools. Each part is checked on its own and a bad one dropped, like the rest: it must never cost the report.
const healthSchema = z.object({
failed_units: z
.array(z.object({ unit: text(200), description: text(300).optional() }))
.transform(cap(100))
.optional()
.catch(undefined),
smart: z
.array(z.object({ device: text(100), model: text(200).optional(), healthy: z.boolean().nullable().optional(), temperature_c: z.number().nullable().optional(), wear_percent: z.number().nullable().optional() }))
.transform(cap(100))
.optional()
.catch(undefined),
zfs: z
.array(z.object({ name: text(200), health: text(40), capacity_percent: z.number().nullable().optional() }))
.transform(cap(100))
.optional()
.catch(undefined),
});
const systemSchema = z.object({
health: z
.unknown()
.optional()
.transform((h) => {
const parsed = healthSchema.safeParse(h);
return parsed.success && Object.values(parsed.data).some((v) => v !== undefined) ? parsed.data : undefined;
}),
updates: z
.unknown()
.optional()
@@ -105,6 +133,23 @@ async function authenticateAgent(req: Request, res: Response, next: NextFunction
next();
}
/**
* "Should I replace myself?" — asked by an agent that has been told to keep itself up to date, before it reports. Says yes only when an
* administrator has switched automatic updates on and this agent is behind the script this app serves, and gives the script's SHA-256 so
* the agent can check what it downloads. The agent says which version it is; failing that, the last one it reported counts.
*/
agentReportRouter.get("/update", asyncHandler(authenticateAgent), asyncHandler(async (req, res) => {
const server = res.locals.agentServer as { osType: string; agentVersion: number | null };
const os = server.osType === "windows" ? "windows" : "linux";
const claimed = Number(req.query.version);
const running = Number.isInteger(claimed) && claimed >= 1 && claimed <= 100_000 ? claimed : (server.agentVersion ?? 1);
const latest = latestAgentVersions()[os];
const sha256 = latestAgentSha256(os);
const { agents } = await getSettings();
const update = agents.autoUpdate && latest !== null && sha256 !== null && running < latest;
res.json({ update, version: update ? latest : null, sha256: update ? sha256 : null });
}));
agentReportRouter.post("/", asyncHandler(authenticateAgent), express.json({ limit: AGENT_BODY_LIMIT }), asyncHandler(async (req, res) => {
const server = res.locals.agentServer as { id: number };
+1
View File
@@ -20,6 +20,7 @@ alertTicketsRouter.use(requireAuth, requireRole("admin"));
const CATEGORIES = [
{ value: "offline", label: "Server down" },
{ value: "disk", label: "Disk & storage" },
{ value: "services", label: "Failing services" },
{ value: "backup", label: "Backups" },
{ value: "updates", label: "Updates" },
{ value: "expiry", label: "Expiry" },
+1 -1
View File
@@ -15,7 +15,7 @@ runbooksRouter.use(requireAuth);
/** More than anyone needs for one thing; each is another note to keep true. */
const MAX_PER_TARGET = 20;
export const ALERT_CATEGORIES = ["offline", "disk", "backup", "updates", "expiry", "automation", "integration", "monitoring", "tickets"] as const;
export const ALERT_CATEGORIES = ["offline", "disk", "services", "backup", "updates", "expiry", "automation", "integration", "monitoring", "tickets"] as const;
const bodySchema = z.object({
title: z.string().trim().min(1).max(120),
+19 -5
View File
@@ -8,6 +8,7 @@ import { generateApiToken } from "../services/tokens.js";
import { recordAudit } from "../services/audit.js";
import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js";
import { parsePendingUpdates } from "../services/pendingUpdates.js";
import { parseHostHealth } from "../services/hostHealth.js";
import { latestAgentVersions } from "../services/agentVersions.js";
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
import { getServerActivity } from "../services/serverActivity.js";
@@ -19,14 +20,14 @@ import { serverPortsRouter } from "./serverPorts.js";
import { InvalidTagError, normalizeTags, parseTags } from "../services/serverTags.js";
import { buildServerSummary } from "../services/serverSummary.js";
import { activeSubjects } from "../services/maintenance.js";
import { getSettings } from "../services/settingsStore.js";
import { getSettings, updateSettings } from "../services/settingsStore.js";
export const serversRouter = Router();
/** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */
function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null }>(row: T) {
const { apiTokenHash: _hash, tags, pendingUpdates, ...rest } = row;
return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates) };
function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null; hostHealth: string | null }>(row: T) {
const { apiTokenHash: _hash, tags, pendingUpdates, hostHealth, ...rest } = row;
return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates), hostHealth: parseHostHealth(hostHealth) };
}
serversRouter.use(requireAuth);
@@ -43,11 +44,23 @@ serversRouter.get("/", asyncHandler(async (_req, res) => {
const rows = await db.query.servers.findMany({ orderBy: (s, { asc }) => [asc(s.name)] });
res.json({
servers: rows.map(publicServer),
// Whether agents that are behind replace themselves, so the page can show the switch.
agentAutoUpdate: (await getSettings()).agents.autoUpdate,
// What a new install would be, so the page can say which agents are behind.
agentLatest: latestAgentVersions(),
});
}));
// Whether agents that are behind download the newer script from here and replace themselves. An administrator's to switch.
serversRouter.put("/agent-update", requireRole("admin"), asyncHandler(async (req, res) => {
const parsed = z.object({ autoUpdate: z.boolean() }).safeParse(req.body);
if (!parsed.success) return res.status(400).json({ error: "invalid_body" });
const before = (await getSettings()).agents.autoUpdate;
await updateSettings({ agents: { autoUpdate: parsed.data.autoUpdate } });
await recordAudit({ actor: req.currentUser!, category: "settings", action: "agent_auto_update", targetType: "settings", detail: { from: before, to: parsed.data.autoUpdate } });
res.json({ autoUpdate: parsed.data.autoUpdate });
}));
serversRouter.post("/", requireRole("admin"), asyncHandler(async (req, res) => {
const parsed = createServerSchema.safeParse(req.body);
if (!parsed.success) {
@@ -353,6 +366,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
listeningPorts: _listeningPorts,
lastPortScan: _lastPortScan,
pendingUpdates: rawUpdates,
hostHealth: rawHealth,
...serverOut
} = server;
@@ -366,7 +380,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)),
};
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates) }, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() });
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates), hostHealth: parseHostHealth(rawHealth) }, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() });
}));
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
+16 -1
View File
@@ -6,8 +6,11 @@
*
* 1 — before the number was reported (so an agent that says nothing is version 1)
* 2 — reports pending updates, and its own version
* 3 — Linux: the package manager is asked outside the systemd sandbox, so dnf and zypper can answer too (Windows stays at 2)
* 3 — Linux: the package manager is asked outside the systemd sandbox, so dnf and zypper can answer too.
* Windows: can replace itself when an administrator has switched automatic updates on.
* 4 — Linux: also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can replace itself too
*/
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -51,6 +54,18 @@ export function latestAgentVersions(dir: string = AGENT_DIR): LatestAgentVersion
return result;
}
/**
* The SHA-256 of the script this app serves for an OS, as hex — what an agent that updates itself checks the download against, so a
* truncated or tampered-with transfer is never installed. Null if the script can't be read.
*/
export function latestAgentSha256(os: AgentOs, dir: string = AGENT_DIR): string | null {
try {
return createHash("sha256").update(readFileSync(join(dir, FILES[os]))).digest("hex");
} catch {
return null;
}
}
/**
* Whether a server's agent is behind. Only a server that reports through an agent can be: one that never has (a Proxmox guest tracked
* through its host, say) has none to update. An agent that reports no version is version 1.
+1
View File
@@ -126,6 +126,7 @@ export function planTicketRun(input: {
const CATEGORY_LABELS: Record<AlertCategory, string> = {
offline: "Server down",
disk: "Disk & storage",
services: "Failing services",
backup: "Backups",
updates: "Updates",
expiry: "Expiry",
+1 -1
View File
@@ -4,7 +4,7 @@
export type AlertSeverity = "critical" | "warning" | "info";
/** What kind of problem — drives the filter on the Alerts page. */
export type AlertCategory = "offline" | "disk" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
export type AlertCategory = "offline" | "disk" | "services" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
export interface Alert {
/** Stable, so the page can key rows on it. */
+2 -2
View File
@@ -68,8 +68,8 @@ const SEVERITY_ORDER: Record<AlertSeverity, number> = { critical: 0, warning: 1,
/** Which kind of source, and which one, a health/automation condition key belongs to. */
function originOfConditionKey(key: string): { type: string; id: number | null; category: AlertCategory } {
let m = /^(?:offline|disk):server:(\d+)/.exec(key);
if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : "disk" };
let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)/.exec(key);
if (m) return { type: "server", id: Number(m[1]), category: key.startsWith("offline") ? "offline" : key.startsWith("failed-units") ? "services" : "disk" };
m = /^disk:proxmox:(\d+):/.exec(key);
if (m) return { type: "proxmox", id: Number(m[1]), category: "disk" };
m = /^(?:synology-volume|synology-disk|disk:synology):(\d+):/.exec(key);
+5 -1
View File
@@ -9,6 +9,7 @@ import { recordAudit } from "./audit.js";
import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.js";
import { activeSubjects, subjectOfConditionKey } from "./maintenance.js";
import { coveredByParent, loadParentRows, resolveParents } from "./serverParents.js";
import { hostHealthConditions, parseHostHealth, type HostHealth } from "./hostHealth.js";
const STATE_FLAG = "healthActiveConditions";
/** After a restart the agents haven't had a chance to report yet (they run every 15 min), so server-derived conditions are held rather than judged. */
@@ -32,6 +33,8 @@ export interface ServerSnapshot {
name: string;
lastSeenAt: string | null;
disks: { mount: string; sizeBytes: number; usedBytes: number }[];
/** What the agent says about failed services, disk health and ZFS pools, if it says anything. */
health?: HostHealth | null;
}
export interface ProxmoxSnapshot {
integrationId: number;
@@ -115,6 +118,7 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
}
// A server that never reported has no agent to go quiet; and an offline one's disk figures are stale, so neither is judged on disks.
if (s.lastSeenAt && !offline) {
for (const c of hostHealthConditions(s.id, s.name, s.health ?? null)) out.push({ key: c.key, source: "server", message: c.message, severity: c.severity });
for (const d of s.disks) {
const p = usage(d.usedBytes, d.sizeBytes);
if (p !== null && p >= limit) {
@@ -296,7 +300,7 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel
} catch {
// a malformed report shouldn't stop every other server being checked
}
snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks });
snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: parseHostHealth(s.hostHealth) });
}
const rows = await db
+124
View File
@@ -0,0 +1,124 @@
/**
* What a Linux agent can say about the machine's own health, beyond its figures: services that have failed (systemd), whether each disk's
* SMART self-assessment passes and how worn it is, and whether each ZFS pool is online. Each part is optional — an agent reports only what
* it could find out (no smartctl, no ZFS, no systemd) — and a part that is absent from a report keeps what was last said, so an agent that
* skips one never turns an old problem into "all well".
*/
export interface FailedUnit {
unit: string;
description: string;
}
export interface SmartDisk {
device: string;
model: string;
/** The drive's own overall verdict: true passes, false fails, null if it didn't give one. */
healthy: boolean | null;
temperatureC: number | null;
/** NVMe and some SSDs say how much of their rated life is used, 0–100 (and past it). */
wearPercent: number | null;
}
export interface ZfsPool {
name: string;
/** ONLINE, DEGRADED, FAULTED, UNAVAIL, SUSPENDED … as zpool says it. */
health: string;
capacityPercent: number | null;
}
export interface HostHealth {
failedUnits?: FailedUnit[];
smart?: SmartDisk[];
zfs?: ZfsPool[];
/** ISO — when the agent last told us anything here. */
reportedAt: string;
}
/** As it arrives from an agent. */
export interface IncomingHostHealth {
failed_units?: { unit: string; description?: string }[];
smart?: { device: string; model?: string; healthy?: boolean | null; temperature_c?: number | null; wear_percent?: number | null }[];
zfs?: { name: string; health: string; capacity_percent?: number | null }[];
}
const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
export function parseHostHealth(stored: string | null): HostHealth | null {
if (!stored) return null;
try {
const v = JSON.parse(stored);
if (!v || typeof v !== "object") return null;
const out: HostHealth = { reportedAt: typeof v.reportedAt === "string" ? v.reportedAt : "" };
if (Array.isArray(v.failedUnits)) out.failedUnits = v.failedUnits.filter((u: any) => u && typeof u.unit === "string").map((u: any) => ({ unit: u.unit, description: typeof u.description === "string" ? u.description : "" }));
if (Array.isArray(v.smart)) {
out.smart = v.smart
.filter((d: any) => d && typeof d.device === "string")
.map((d: any) => ({ device: d.device, model: typeof d.model === "string" ? d.model : "", healthy: typeof d.healthy === "boolean" ? d.healthy : null, temperatureC: num(d.temperatureC), wearPercent: num(d.wearPercent) }));
}
if (Array.isArray(v.zfs)) out.zfs = v.zfs.filter((p: any) => p && typeof p.name === "string" && typeof p.health === "string").map((p: any) => ({ name: p.name, health: p.health, capacityPercent: num(p.capacityPercent) }));
return out;
} catch {
return null;
}
}
/** The text to store after a report: the parts the agent sent replace what was there, the parts it didn't send stay. */
export function nextHostHealth(previous: string | null, incoming: IncomingHostHealth, now: string): string | null {
const keep = parseHostHealth(previous);
const next: HostHealth = { ...(keep ?? {}), reportedAt: now };
let any = false;
if (incoming.failed_units) {
any = true;
next.failedUnits = incoming.failed_units.map((u) => ({ unit: u.unit, description: u.description ?? "" }));
}
if (incoming.smart) {
any = true;
next.smart = incoming.smart.map((d) => ({ device: d.device, model: d.model ?? "", healthy: typeof d.healthy === "boolean" ? d.healthy : null, temperatureC: num(d.temperature_c), wearPercent: num(d.wear_percent) }));
}
if (incoming.zfs) {
any = true;
next.zfs = incoming.zfs.map((p) => ({ name: p.name, health: p.health, capacityPercent: num(p.capacity_percent) }));
}
return any ? JSON.stringify(next) : previous;
}
/** A pool in any state but ONLINE has lost something (DEGRADED has lost redundancy, the rest are worse). */
export const poolIsFine = (p: ZfsPool) => p.health.toUpperCase() === "ONLINE";
/** Wear at which a drive is worth replacing soon, in percent of rated life used. */
export const WEAR_WARNING_PERCENT = 90;
export interface HostHealthCondition {
key: string;
message: string;
severity: "critical" | "warning";
}
const listOf = (names: string[], max = 4) => (names.length <= max ? names.join(", ") : `${names.slice(0, max).join(", ")} and ${names.length - max} more`);
/** The problems a server's health report amounts to. Pure, so the rules can be tested exactly. */
export function hostHealthConditions(serverId: number, serverName: string, health: HostHealth | null): HostHealthCondition[] {
if (!health) return [];
const out: HostHealthCondition[] = [];
const failed = health.failedUnits ?? [];
if (failed.length > 0) {
out.push({
key: `failed-units:server:${serverId}`,
severity: "warning",
message: `${serverName}: ${failed.length === 1 ? "1 service has" : `${failed.length} services have`} failed — ${listOf(failed.map((u) => u.unit))}`,
});
}
for (const d of health.smart ?? []) {
const label = `${d.device}${d.model ? ` (${d.model})` : ""}`;
if (d.healthy === false) {
out.push({ key: `smart:server:${serverId}:${d.device}`, severity: "critical", message: `${serverName}: the disk ${label} says it is failing (SMART self-assessment)` });
} else if (d.wearPercent !== null && d.wearPercent >= WEAR_WARNING_PERCENT) {
out.push({ key: `smart:server:${serverId}:${d.device}`, severity: "warning", message: `${serverName}: the disk ${label} has used ${Math.round(d.wearPercent)}% of its rated life` });
}
}
for (const p of health.zfs ?? []) {
if (!poolIsFine(p)) {
out.push({ key: `zfs:server:${serverId}:${p.name}`, severity: "critical", message: `${serverName}: ZFS pool ${p.name} is ${p.health}` });
}
}
return out;
}
+1 -1
View File
@@ -23,7 +23,7 @@ export async function listUpcomingWindows(now: Date = new Date()): Promise<Activ
* so conditions persisted by an earlier version still map correctly.
*/
export function subjectOfConditionKey(key: string): string | null {
let m = /^(?:offline|disk):server:(\d+)(?::|$)/.exec(key);
let m = /^(?:offline|disk|failed-units|smart|zfs):server:(\d+)(?::|$)/.exec(key);
if (m) return `server:${m[1]}`;
m = /^disk:proxmox:(\d+):/.exec(key);
if (m) return `integration:${m[1]}`;
+20
View File
@@ -10,6 +10,8 @@ import { getAlerts } from "./alerts.js";
import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js";
import { getSourceStatus } from "./integrationHealthMonitor.js";
import { parsePendingUpdates } from "./pendingUpdates.js";
import { parseHostHealth, poolIsFine } from "./hostHealth.js";
import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js";
import { computeSecretStatus } from "./secretStatus.js";
import { getSettings } from "./settingsStore.js";
import { getUptime } from "./uptimeHistory.js";
@@ -80,8 +82,26 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
if (updates.security !== null) out.add("homelab_server_pending_security_updates", "gauge", "Of those, the ones that are security fixes.", updates.security, { server });
out.add("homelab_server_reboot_required", "gauge", "1 if a restart is due.", updates.rebootRequired ? 1 : 0, { server });
}
const health = parseHostHealth(s.hostHealth);
if (health?.failedUnits) out.add("homelab_server_failed_units", "gauge", "Services (systemd units) that have failed on the server.", health.failedUnits.length, { server });
for (const d of health?.smart ?? []) {
if (d.healthy !== null) out.add("homelab_server_disk_healthy", "gauge", "1 if the disk's own SMART self-assessment passes, 0 if it fails.", d.healthy ? 1 : 0, { server, device: d.device });
if (d.wearPercent !== null) out.add("homelab_server_disk_wear_percent", "gauge", "How much of an SSD's rated life is used.", d.wearPercent, { server, device: d.device });
if (d.temperatureC !== null) out.add("homelab_server_disk_temperature_celsius", "gauge", "A disk's temperature.", d.temperatureC, { server, device: d.device });
}
for (const p of health?.zfs ?? []) out.add("homelab_server_zfs_pool_online", "gauge", "1 if a ZFS pool is ONLINE, 0 in any other state.", poolIsFine(p) ? 1 : 0, { server, pool: p.name });
out.add("homelab_server_agent_outdated", "gauge", "1 if the server's agent is older than the one this installation would install now.", isAgentOutdated(s, latest) ? 1 : 0, { server });
}
// Disks on a steady climb, from each server's own history: days until full at the present rate (left out where there is no steady trend).
try {
for (const s of serverRows) {
for (const f of disksFillingSoon(await getServerMetrics(s.id, 14, now), 3650, 101)) {
out.add("homelab_server_disk_days_to_full", "gauge", "Days until a disk is full at its present rate of filling; only for disks with a steady climb.", f.daysToFull, { server: s.name, mount: f.mount });
}
}
} catch (err) {
console.error("[metrics] couldn't work out the disk forecasts:", err);
}
try {
const uptime = await getUptime(now, 30);
for (const s of serverRows) {
+2
View File
@@ -54,6 +54,8 @@ export function describeAudit(category: string, action: string, detail: Record<s
return { title: "A disk got nearly full", tone: "bad", kind: "health", detail: message };
case "disk_ok":
return { title: "A disk is no longer nearly full", tone: "good", kind: "health", detail: null };
case "agent_updated":
return { title: `Agent updated to version ${detail.to ?? "?"}`, tone: "good", kind: "change", detail: detail.from !== undefined ? `from version ${detail.from}` : null };
case "create":
return { title: "Server added", tone: "neutral", kind: "change", detail: null };
case "update":
+8
View File
@@ -119,6 +119,12 @@ export interface SignInSettings {
otherwise: "viewer" | "keep";
}
/** How the agents out on the servers are kept up to date. Off until an administrator turns it on. */
export interface AgentSettings {
/** An agent that is behind downloads the newer script from this app, checks it and replaces itself. */
autoUpdate: boolean;
}
/** What the inventory needs to work out a running cost. Without a price per kWh it shows draw but no cost. */
export interface InventorySettings {
/** A short name shown after amounts ("SEK", "kr", "$"); may be blank. */
@@ -180,6 +186,7 @@ export interface AppSettings {
digest: DigestSettings;
signIn: SignInSettings;
inventory: InventorySettings;
agents: AgentSettings;
consistency: ConsistencySettings;
nameGenerator: NameGeneratorSettings;
alertTickets: AlertTicketSettings;
@@ -214,6 +221,7 @@ const DEFAULTS: AppSettings = {
quietHours: { enabled: false, start: "22:00", end: "07:00" },
digest: { enabled: false, day: 1, time: "08:00" },
inventory: { currency: "", pricePerKwh: null },
agents: { autoUpdate: false },
signIn: { roleFromGroups: false, groupsClaim: "groups", adminGroups: [], operatorGroups: [], otherwise: "viewer" },
healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14 },
// Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every
+15 -1
View File
@@ -3,6 +3,8 @@ import { db } from "../db/client.js";
import { scheduledTasks, servers } from "../db/schema.js";
import { recordServerMetrics } from "./serverMetrics.js";
import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js";
import { nextHostHealth, type IncomingHostHealth } from "./hostHealth.js";
import { recordAudit } from "./audit.js";
export interface IncomingTask {
scheduleType: "cron" | "systemd_timer" | "windows_task";
@@ -23,6 +25,8 @@ export interface IncomingSystemInfo {
listening_ports?: { protocol: "tcp" | "udp"; port: number; address: string; process?: string }[];
/** Updates waiting to be installed. Absent from an agent that doesn't report them (or couldn't find out this time). */
updates?: IncomingUpdates;
/** Failed services, disk health, ZFS pools — each part present only if the agent could find it out. */
health?: IncomingHostHealth;
}
export interface AgentReport {
@@ -103,13 +107,14 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
}
const system = report.system;
const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince }).from(servers).where(eq(servers.id, serverId));
const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince, hostHealth: servers.hostHealth, agentVersion: servers.agentVersion }).from(servers).where(eq(servers.id, serverId));
await db
.update(servers)
.set({
lastSeenAt: now,
// Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before.
...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}),
...(system?.health && before ? { hostHealth: nextHostHealth(before.hostHealth, system.health, now) } : {}),
...(report.hostname ? { hostname: report.hostname } : {}),
...(report.agentVersion !== undefined ? { agentVersion: report.agentVersion } : {}),
// An agent knows what it runs on better than whoever registered the server did.
@@ -132,6 +137,15 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
})
.where(eq(servers.id, serverId));
// An agent that now says a higher version than it did has been updated (by hand, or by itself): worth a line in the server's history.
if (before && report.agentVersion !== undefined && before.agentVersion !== null && report.agentVersion > before.agentVersion) {
try {
await recordAudit({ category: "server", action: "agent_updated", targetType: "server", targetId: serverId, detail: { from: before.agentVersion, to: report.agentVersion } });
} catch (err) {
console.error("[agent] couldn't record the agent update:", err);
}
}
// Keep a little history of the figures too. A failure here must not lose the report itself, which is already saved.
try {
await recordServerMetrics(serverId, system, new Date(now));
+26 -3
View File
@@ -361,7 +361,7 @@ describe("Linux agent", () => {
// The one step that runs outside the sandbox (+) is the package-manager question, which may fail without stopping the report (-),
// and it hands its answer over through the unit's own private directory, which is where the script looks for it.
assert.equal(settings.get("ExecStartPre"), "-+/usr/local/bin/homelab-manager-agent.sh --prepare-updates");
assert.equal(settings.get("ExecStartPre"), "-+/usr/local/bin/homelab-manager-agent.sh --prepare");
assert.equal(rendered.split("\n").filter((l) => /^ExecStartPre=/.test(l.trim())).length, 1, "only that one step is outside the sandbox");
assert.equal(settings.get("RuntimeDirectory"), "homelab-manager-agent");
assert.equal(settings.get("RuntimeDirectoryMode"), "0700");
@@ -399,16 +399,39 @@ describe("Linux agent", () => {
test("the update check can be run on its own, and what it leaves behind is read back through jq, not trusted", () => {
const text = readFileSync(agentScript, "utf8");
assert.ok(text.includes("--prepare-updates) MODE=prepare"));
assert.ok(text.includes("--check-updates) MODE=check"));
assert.ok(text.includes("--prepare|--prepare-updates) MODE=prepare"), "the old name still works, for units written by an earlier installer");
assert.ok(text.includes("--check-updates|--check) MODE=check"));
assert.ok(text.includes("use_prepared_updates || collect_updates"), "the report uses the prepared answer first and asks itself otherwise");
assert.ok(text.includes('find "$UPDATES_FILE" -mmin -10'), "an old answer isn't used");
assert.ok(text.includes("jq -ce 'select(type == \"object\""), "the file is re-shaped by jq, so the report only carries the four known fields");
assert.ok(text.includes('mv -f "$UPDATES_FILE.tmp" "$UPDATES_FILE"'), "written whole, then moved into place");
// disk and pool health need the devices, so they are read from the prepare step and handed over, re-shaped by jq like the updates
assert.ok(text.includes("use_prepared_health"));
assert.ok(text.includes("smartctl -j -H -A -n standby"), "a sleeping disk is not woken to be asked");
assert.ok(text.includes("zpool list -H -o name,health,capacity"));
assert.ok(text.includes("systemctl list-units --state=failed"));
assert.ok(text.includes("pacman -Qu"));
// the two modes must not need a server address or a token
assert.ok(text.includes('[[ "$MODE" == "report" && ( -z "$API_URL" || -z "$API_TOKEN" ) ]]'));
});
test("self-update takes nothing on trust but a checksum, never runs inside the sandbox, and can not loop", () => {
const text = readFileSync(agentScript, "utf8");
const body = text.slice(text.indexOf("self_update() {"), text.indexOf("# The answer --prepare left for this report"));
assert.ok(body.length > 500, "self_update wasn't found");
assert.ok(body.includes("api_get \"/api/agent/report/update?version=$AGENT_VERSION\""), "it asks the server, with its token, whether it should");
assert.ok(body.includes("sha256sum") && body.includes("\"$got\" != \"$sha\""), "the download is checked against the checksum the server gave");
assert.ok(body.includes("bash -n \"$work/new.sh\"") && body.includes("#!/usr/bin/env bash") && body.includes("^AGENT_VERSION=$version"), "and must look like this script, parse, and say the version promised");
assert.ok(body.includes("(( version <= AGENT_VERSION ))"), "only ever to a newer version");
assert.ok(body.includes("mv -f \"$self.new\" \"$self\""), "put beside the old one and moved over it in one step");
assert.ok(body.includes("HLM_JUST_UPDATED"), "the new version doesn't update again straight away");
// it is only called from the prepare step (outside the sandbox) and from the manual check, never from the report
const calls = text.split("\n").filter((l) => /(^|\s|&& )self_update(\s|$)/.test(l) && !l.includes("self_update()") && !l.trim().startsWith("#"));
assert.equal(calls.length, 2, calls.join(" | "));
const reportBegins = text.indexOf("collect_cron\ncollect_systemd_timers");
assert.ok(reportBegins > 0 && text.indexOf("self_update", reportBegins) === -1, "not after the point where a report begins");
});
test("unit files are made readable by everyone (the umask above them is for the token file)", () => {
const text = readFileSync(installScript, "utf8").replace(/\r\n/g, "\n");
assert.match(text, /chmod 644 "\$SYSTEMD_DIR\/homelab-manager-agent\.service" "\$SYSTEMD_DIR\/homelab-manager-agent\.timer"/);
@@ -0,0 +1,316 @@
/**
* What an agent can say about a machine's health (failed services, disk SMART, ZFS pools), the alerts and figures that come from it, and
* the switch that lets agents that are behind replace themselves.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-hosthealth-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, maintenanceWindows } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { hostHealthConditions, nextHostHealth, parseHostHealth } = await import("../src/services/hostHealth.js");
const { evaluateHealth } = await import("../src/services/healthMonitor.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { latestAgentSha256, latestAgentVersions } = await import("../src/services/agentVersions.js");
const { serversRouter } = await import("../src/routes/servers.js");
const { agentReportRouter } = await import("../src/routes/agentReport.js");
const { hashToken } = await import("../src/services/tokens.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const NOW = "2026-10-17T12:00:00.000Z";
const health = (extra: object = {}) => ({ reportedAt: NOW, ...extra });
describe("what a health report amounts to", () => {
test("failed services, a failing disk, a worn one and a pool that isn't online are each a problem; the healthy are not", () => {
const conditions = hostHealthConditions(7, "elsa", {
reportedAt: NOW,
failedUnits: [{ unit: "nginx.service", description: "" }, { unit: "backup.service", description: "" }],
smart: [
{ device: "/dev/sda", model: "WD Red", healthy: false, temperatureC: 40, wearPercent: null },
{ device: "/dev/nvme0", model: "Samsung", healthy: true, temperatureC: 38, wearPercent: 93 },
{ device: "/dev/sdb", model: "Fine", healthy: true, temperatureC: 31, wearPercent: 12 },
{ device: "/dev/sdc", model: "No verdict", healthy: null, temperatureC: null, wearPercent: null },
],
zfs: [{ name: "tank", health: "DEGRADED", capacityPercent: 40 }, { name: "rpool", health: "ONLINE", capacityPercent: 10 }],
});
const byKey = Object.fromEntries(conditions.map((c) => [c.key, c]));
assert.deepEqual(Object.keys(byKey).sort(), ["failed-units:server:7", "smart:server:7:/dev/nvme0", "smart:server:7:/dev/sda", "zfs:server:7:tank"]);
assert.equal(byKey["failed-units:server:7"].severity, "warning");
assert.match(byKey["failed-units:server:7"].message, /2 services have failed — nginx\.service, backup\.service/);
assert.equal(byKey["smart:server:7:/dev/sda"].severity, "critical");
assert.match(byKey["smart:server:7:/dev/sda"].message, /failing/);
assert.equal(byKey["smart:server:7:/dev/nvme0"].severity, "warning");
assert.match(byKey["smart:server:7:/dev/nvme0"].message, /93% of its rated life/);
assert.equal(byKey["zfs:server:7:tank"].severity, "critical");
assert.match(byKey["zfs:server:7:tank"].message, /DEGRADED/);
});
test("nothing said, or everything fine, is no problem; a long list of services is shortened", () => {
assert.deepEqual(hostHealthConditions(1, "x", null), []);
assert.deepEqual(hostHealthConditions(1, "x", health({ failedUnits: [], smart: [], zfs: [] })), []);
const many = Array.from({ length: 9 }, (_, i) => ({ unit: `u${i}.service`, description: "" }));
assert.match(hostHealthConditions(1, "x", health({ failedUnits: many }))[0].message, /and 5 more/);
assert.match(hostHealthConditions(1, "x", health({ failedUnits: [many[0]] }))[0].message, /1 service has failed/);
});
test("a report replaces the parts it carries and keeps the parts it doesn't", () => {
const first = nextHostHealth(null, { failed_units: [{ unit: "a.service" }], zfs: [{ name: "tank", health: "ONLINE", capacity_percent: 20 }] }, "t1");
assert.equal(parseHostHealth(first)!.failedUnits![0].unit, "a.service");
// the next report knows about disks only
const second = nextHostHealth(first, { smart: [{ device: "/dev/sda", healthy: true }] }, "t2");
const merged = parseHostHealth(second)!;
assert.equal(merged.failedUnits![0].unit, "a.service", "services kept");
assert.equal(merged.zfs![0].name, "tank", "pools kept");
assert.equal(merged.smart![0].device, "/dev/sda");
assert.equal(merged.reportedAt, "t2");
// an empty list is an answer ("none failed"), and clears the old one
assert.deepEqual(parseHostHealth(nextHostHealth(second, { failed_units: [] }, "t3"))!.failedUnits, []);
// nothing in the report: what was stored is untouched
assert.equal(nextHostHealth(second, {}, "t4"), second);
assert.equal(parseHostHealth("not json"), null);
});
test("an offline server isn't judged on its health, and an online one is", () => {
const snap = (lastSeenAt: string) => ({ servers: [{ id: 3, name: "elsa", lastSeenAt, disks: [], health: health({ failedUnits: [{ unit: "a.service", description: "" }] }) }], proxmox: [], synology: [] });
const now = Date.parse(NOW);
const limits = { serverOfflineMinutes: 60, diskUsagePercent: 90 };
const online = evaluateHealth(snap(new Date(now - 60_000).toISOString()), limits, now).map((c) => c.key);
assert.deepEqual(online, ["failed-units:server:3"]);
const offline = evaluateHealth(snap(new Date(now - 5 * 3600_000).toISOString()), limits, now).map((c) => c.key);
assert.deepEqual(offline, ["offline:server:3"]);
});
});
describe("through the report, the alerts, the figures and the page", () => {
let app: Server;
let base = "";
let id = 0;
let who = "admin";
before(async () => {
await db.insert(users).values([
{ oidcSub: "admin", name: "Ada", role: "admin" },
{ oidcSub: "operator", name: "Olle", role: "operator" },
]);
const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("tok"), apiTokenPrefix: "t" }).returning();
id = s.id;
const a = express();
a.use("/api/agent/report", agentReportRouter);
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/servers", serversRouter);
app = await new Promise<Server>((resolve) => {
const x = a.listen(0, "127.0.0.1", () => resolve(x));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const report = (system: unknown, extra: Record<string, unknown> = {}) =>
fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer tok" }, body: JSON.stringify({ hostname: "h", os_type: "linux", tasks: [], system, ...extra }) });
const detail = async () => (await (await fetch(`${base}/servers/${id}/detail`)).json()).server;
test("what an agent says is stored and shown; a garbled part costs the report nothing", async () => {
const res = await report({
health: {
failed_units: [{ unit: "nginx.service", description: "A high performance web server" }],
smart: [{ device: "/dev/sda", model: "WD Red", healthy: false, temperature_c: 41, wear_percent: null }],
zfs: "this isn't a list",
},
});
assert.equal(res.status, 202);
let s = await detail();
assert.equal(s.hostHealth.failedUnits[0].unit, "nginx.service");
assert.equal(s.hostHealth.smart[0].healthy, false);
assert.equal(s.hostHealth.zfs, undefined, "the part that made no sense was dropped");
// the list endpoint shows it too, and never as raw stored text
const row = (await (await fetch(`${base}/servers`)).json()).servers.find((x: { id: number }) => x.id === id);
assert.equal(typeof row.hostHealth, "object");
// a report with no health at all leaves it as it was; garbage in place of the whole thing too
assert.equal((await report({ cpu: { cores: 2 } })).status, 202);
assert.equal((await report({ health: 12 })).status, 202);
s = await detail();
assert.equal(s.hostHealth.failedUnits[0].unit, "nginx.service");
});
test("they appear on the Alerts page as their own kind, under the server, and a maintenance window silences them", async () => {
invalidateAlertsCache();
let alerts = (await getAlerts(true)).alerts.filter((a) => a.id.includes(`:server:${id}`));
const services = alerts.find((a) => a.id.startsWith("services:failed-units"));
assert.ok(services, JSON.stringify(alerts.map((a) => a.id)));
assert.equal(services.category, "services");
assert.equal(services.link, `/servers/${id}`);
assert.equal(services.silenced, false);
const disk = alerts.find((a) => a.id.startsWith("disk:smart"));
assert.ok(disk && disk.severity === "critical", "a failing disk is critical, in the disk category");
await db.insert(maintenanceWindows).values({ targetType: "server", targetId: id, reason: "x", startedAt: new Date(Date.now() - 60_000).toISOString(), endsAt: new Date(Date.now() + 3600_000).toISOString() });
invalidateAlertsCache();
alerts = (await getAlerts(true)).alerts.filter((a) => a.id.includes(`:server:${id}`));
assert.ok(alerts.length >= 2 && alerts.every((a) => a.silenced), "all held back under maintenance");
await db.delete(maintenanceWindows).where(eq(maintenanceWindows.targetId, id));
// the services come right once the agent says none have failed
await report({ health: { failed_units: [] } });
invalidateAlertsCache();
assert.equal((await getAlerts(true)).alerts.some((a) => a.id.startsWith("services:")), false);
});
test("/metrics carries them", async () => {
await report({ health: { failed_units: [{ unit: "a.service" }, { unit: "b.service" }], zfs: [{ name: "tank", health: "DEGRADED", capacity_percent: 12 }], smart: [{ device: "/dev/sda", healthy: true, temperature_c: 33, wear_percent: 5 }] } });
const text = await buildMetrics();
assert.match(text, /homelab_server_failed_units\{server="elsa"\} 2/);
assert.match(text, /homelab_server_zfs_pool_online\{server="elsa",pool="tank"\} 0/);
assert.match(text, /homelab_server_disk_healthy\{server="elsa",device="\/dev\/sda"\} 1/);
assert.match(text, /homelab_server_disk_wear_percent\{server="elsa",device="\/dev\/sda"\} 5/);
assert.match(text, /homelab_server_disk_temperature_celsius\{server="elsa",device="\/dev\/sda"\} 33/);
});
test("a disk on a steady climb has its days to full in /metrics, and a flat one does not", async () => {
const { serverMetrics } = await import("../src/db/schema.js");
const day = 86_400_000;
const rows = [];
for (let i = 0; i < 10; i++) {
const at = new Date(Date.now() - (10 - i) * day).toISOString();
rows.push({ serverId: id, metric: "disk:/data", value: 50 + i * 2, recordedAt: at });
rows.push({ serverId: id, metric: "disk:/", value: 30, recordedAt: at });
}
await db.insert(serverMetrics).values(rows);
const text = await buildMetrics();
const m = /homelab_server_disk_days_to_full\{server="elsa",mount="\/data"\} (\d+)/.exec(text);
assert.ok(m, text.split("\n").filter((l) => l.includes("days_to_full")).join("\n"));
assert.ok(Number(m[1]) >= 15 && Number(m[1]) <= 17, `about sixteen days (2 points a day, 32 left from 68%), got ${m[1]}`);
assert.doesNotMatch(text, /days_to_full\{server="elsa",mount="\/"\}/);
});
test("an agent that is now a newer version leaves a line in the server's history", async () => {
await report({}, { agent_version: 3 });
await report({}, { agent_version: 4 });
const activity = await (await fetch(`${base}/servers/${id}/activity`)).json();
const line = activity.events.find((e: { title: string }) => /Agent updated to version 4/.test(e.title));
assert.ok(line, JSON.stringify(activity.events.map((e: { title: string }) => e.title)));
assert.equal(line.detail, "from version 3");
assert.equal(activity.events.some((e: { title: string }) => /version 3$/.test(e.title)), false, "the first time a version is seen is not an update");
});
});
describe("agents updating themselves", () => {
let app: Server;
let base = "";
let who = "admin";
let id = 0;
before(async () => {
const [s] = await db.insert(servers).values({ name: "anna", apiTokenHash: hashToken("anna-tok"), apiTokenPrefix: "a", osType: "linux" }).returning();
id = s.id;
const [w] = await db.insert(servers).values({ name: "win", apiTokenHash: hashToken("win-tok"), apiTokenPrefix: "w", osType: "windows", agentVersion: 1 }).returning();
void w;
const a = express();
a.use("/api/agent/report", agentReportRouter);
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/servers", serversRouter);
app = await new Promise<Server>((resolve) => {
const x = a.listen(0, "127.0.0.1", () => resolve(x));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => app.close());
const ask = (token: string | null, query = "") => fetch(`${base}/agent/report/update${query}`, { headers: token ? { authorization: `Bearer ${token}` } : {} });
const put = (autoUpdate: unknown) => fetch(`${base}/servers/agent-update`, { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ autoUpdate }) });
test("only an agent that proves who it is may ask", async () => {
assert.equal((await ask(null)).status, 401);
assert.equal((await ask("nobody")).status, 401);
});
test("while the switch is off nobody is told to update, however far behind", async () => {
const list = await (await fetch(`${base}/servers`)).json();
assert.equal(list.agentAutoUpdate, false, "off until an administrator turns it on");
const reply = await (await ask("anna-tok", "?version=1")).json();
assert.deepEqual(reply, { update: false, version: null, sha256: null });
});
test("only an administrator can switch it, it is written in the audit log, and it shows on the Servers list", async () => {
who = "operator";
assert.equal((await put(true)).status, 403);
who = "admin";
assert.equal((await put("yes")).status, 400);
assert.equal((await put(true)).status, 200);
assert.equal((await (await fetch(`${base}/servers`)).json()).agentAutoUpdate, true);
const { auditLog } = await import("../src/db/schema.js");
const entry = (await db.select().from(auditLog)).find((e) => e.action === "agent_auto_update");
assert.ok(entry);
assert.deepEqual(JSON.parse(entry.detail!), { from: false, to: true });
});
test("with it on, an agent that is behind is told to, with the checksum of what it will fetch; one that is current is not", async () => {
const latest = latestAgentVersions().linux!;
const behind = await (await ask("anna-tok", `?version=${latest - 1}`)).json();
assert.equal(behind.update, true);
assert.equal(behind.version, latest);
assert.equal(behind.sha256, latestAgentSha256("linux"));
assert.match(behind.sha256, /^[0-9a-f]{64}$/);
const current = await (await ask("anna-tok", `?version=${latest}`)).json();
assert.deepEqual(current, { update: false, version: null, sha256: null });
const ahead = await (await ask("anna-tok", `?version=${latest + 5}`)).json();
assert.equal(ahead.update, false);
// no version given: what it last reported counts, and an agent that has said nothing is version 1
assert.equal((await (await ask("anna-tok")).json()).update, true);
await db.update(servers).set({ agentVersion: latest }).where(eq(servers.id, id));
assert.equal((await (await ask("anna-tok")).json()).update, false);
// nonsense is ignored in favour of that
assert.equal((await (await ask("anna-tok", "?version=banana")).json()).update, false);
});
test("each system is given its own script's checksum", async () => {
const reply = await (await ask("win-tok", "?version=1")).json();
assert.equal(reply.update, true);
assert.equal(reply.sha256, latestAgentSha256("windows"));
assert.notEqual(reply.sha256, latestAgentSha256("linux"));
});
test("the checksum is of the file that is served, byte for byte", async () => {
const { createHash } = await import("node:crypto");
const { readFileSync } = await import("node:fs");
const { AGENT_DIR } = await import("../src/services/agentVersions.js");
const bytes = readFileSync(join(AGENT_DIR, "linux", "report-tasks.sh"));
assert.equal(latestAgentSha256("linux"), createHash("sha256").update(bytes).digest("hex"));
assert.equal(latestAgentSha256("linux", join(dir, "empty")), null, "an unreadable script has no checksum, so nobody is told to fetch it");
});
});
+180
View File
@@ -0,0 +1,180 @@
/**
* The Windows agent replacing itself: only with a script the server vouched for (SHA-256), that is newer, says the version promised and
* parses; and only when it is installed under its own name. Run against a fake server, with the real script's own function.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { spawn, spawnSync } from "node:child_process";
import http from "node:http";
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
const realScript = join(repoRoot, "agent", "windows", "report-tasks.ps1");
const probe = process.platform === "win32" ? spawnSync("pwsh", ["-NoProfile", "-Command", "$PSVersionTable.PSVersion.Major"], { encoding: "utf8" }) : null;
const havePwsh = probe?.status === 0;
const sha = (b: Buffer | string) => createHash("sha256").update(b).digest("hex");
/** Runs PowerShell without blocking this process: the fake server these tests talk to lives in it, so a blocking spawn would deadlock. */
function pwshFile(file: string): Promise<{ stdout: string; stderr: string }> {
return new Promise((resolve, reject) => {
const child = spawn("pwsh", ["-NoProfile", "-NonInteractive", "-File", file]);
let stdout = "";
let stderr = "";
const timer = setTimeout(() => child.kill(), 60_000);
child.stdout.on("data", (c) => (stdout += c));
child.stderr.on("data", (c) => (stderr += c));
child.on("error", reject);
child.on("close", () => {
clearTimeout(timer);
resolve({ stdout, stderr });
});
});
}
describe("the Windows agent updating itself", { skip: !havePwsh }, () => {
let work = "";
let fake: http.Server;
let port = 0;
// What the fake server answers; each test sets it.
let reply: { update: boolean; version: number; sha256: string } = { update: false, version: 3, sha256: "" };
let served: Buffer = Buffer.alloc(0);
let asked: string[] = [];
before(async () => {
work = mkdtempSync(join(tmpdir(), "hm-winupdate-"));
fake = http.createServer((req, res) => {
asked.push(`${req.method} ${req.url} auth=${req.headers.authorization ?? "-"}`);
if (req.url?.startsWith("/api/agent/report/update")) {
res.writeHead(200, { "content-type": "application/json" });
res.end(JSON.stringify(reply));
} else if (req.url === "/agent/windows/report-tasks.ps1") {
res.writeHead(200, { "content-type": "text/plain" });
res.end(served);
} else {
res.writeHead(404).end();
}
});
await new Promise<void>((r) => fake.listen(0, "127.0.0.1", r));
port = (fake.address() as { port: number }).port;
});
after(() => {
fake?.close();
try {
rmSync(work, { recursive: true, force: true });
} catch {
// best effort
}
});
const real = () => readFileSync(realScript);
const OLD = "# the installed agent\n$script:AgentVersion = 1\nWrite-Output 'old'\n";
/** Installs an old stub under `name`, runs the real Update-AgentSelf against the fake server, returns what is left. */
async function attempt(name = "homelab-manager-agent.ps1") {
const dir = mkdtempSync(join(work, "run-"));
mkdirSync(dir, { recursive: true });
const installed = join(dir, name);
writeFileSync(installed, OLD);
asked = [];
const script = `
$ErrorActionPreference = 'Stop'
. '${realScript.replace(/'/g, "''")}'
$script:AgentVersion = 1
$config = @{ ApiUrl = 'http://127.0.0.1:${port}'; ApiToken = 'secret-token'; Insecure = $false }
Update-AgentSelf $config -Self '${installed.replace(/'/g, "''")}'
'DONE'
`;
const scriptFile = join(dir, "run.ps1");
writeFileSync(scriptFile, script);
const r = await pwshFile(scriptFile);
assert.ok(r.stdout.includes("DONE"), `the script did not finish:\n${r.stdout}\n${r.stderr}`);
return { content: readFileSync(installed), leftovers: readdirSync(dir).filter((f) => f.endsWith(".download")), stderr: r.stderr, asked: [...asked] };
}
test("a newer script that matches the checksum replaces the old one, and the token is sent only to the question", async () => {
served = real();
reply = { update: true, version: 3, sha256: sha(served) };
const out = await attempt();
assert.deepEqual(out.content, served, "the installed file is now the served script, byte for byte");
assert.deepEqual(out.leftovers, []);
assert.match(out.stderr, /Updated the agent from version 1 to version 3/);
const question = out.asked.find((l) => l.includes("/api/agent/report/update"))!;
assert.match(question, /version=1/);
assert.match(question, /auth=Bearer secret-token/);
const download = out.asked.find((l) => l.includes("/agent/windows/report-tasks.ps1"))!;
assert.match(download, /auth=-$/, "the public script is fetched without the token");
});
test("nothing happens when the server says no", async () => {
served = real();
reply = { update: false, version: 3, sha256: sha(served) };
const out = await attempt();
assert.equal(out.content.toString(), OLD);
assert.equal(out.asked.some((l) => l.includes("/agent/windows/")), false, "and it does not even download");
});
test("a download that doesn't match the checksum is thrown away", async () => {
served = real();
reply = { update: true, version: 3, sha256: sha("something else") };
const out = await attempt();
assert.equal(out.content.toString(), OLD);
assert.deepEqual(out.leftovers, []);
assert.match(out.stderr, /didn't check out/);
});
test("a script that is not newer, doesn't say the promised version, or doesn't parse is refused even if the checksum fits", async () => {
// not newer than the running version
served = real();
reply = { update: true, version: 1, sha256: sha(served) };
assert.equal((await attempt()).content.toString(), OLD, "version 1 is not newer than 1");
// says a different version than was promised
served = Buffer.from("$script:AgentVersion = 2\nWrite-Output 'x'\n");
reply = { update: true, version: 3, sha256: sha(served) };
assert.equal((await attempt()).content.toString(), OLD);
// says the right version but is not a script
served = Buffer.from("$script:AgentVersion = 3\nfunction Broken { if ( \n");
reply = { update: true, version: 3, sha256: sha(served) };
assert.equal((await attempt()).content.toString(), OLD);
});
test("a copy that isn't installed under the agent's own name is never replaced", async () => {
served = real();
reply = { update: true, version: 3, sha256: sha(served) };
const out = await attempt("report-tasks.ps1");
assert.equal(out.content.toString(), OLD);
assert.equal(out.asked.length, 0, "it does not even ask");
});
test("a server that can't be reached costs nothing", async () => {
const dir = mkdtempSync(join(work, "run-"));
const installed = join(dir, "homelab-manager-agent.ps1");
writeFileSync(installed, OLD);
const script = `
$ErrorActionPreference = 'Stop'
. '${realScript.replace(/'/g, "''")}'
$script:AgentVersion = 1
$config = @{ ApiUrl = 'http://127.0.0.1:1'; ApiToken = 'x'; Insecure = $false }
Update-AgentSelf $config -Self '${installed.replace(/'/g, "''")}'
'DONE'
`;
const scriptFile = join(dir, "run.ps1");
writeFileSync(scriptFile, script);
const r = await pwshFile(scriptFile);
assert.ok(r.stdout.includes("DONE"));
assert.match(r.stderr, /couldn't check for, or install, a newer agent/);
assert.equal(readFileSync(installed, "utf8"), OLD);
});
test("the script stays plain ASCII, as its own header asks", () => {
assert.ok(!/[^\x00-\x7f]/.test(readFileSync(realScript, "utf8")));
});
});
+13 -2
View File
@@ -529,6 +529,8 @@ export interface ServerRecord {
updatesPendingSince: string | null;
/** Which version of the agent script it runs, as the agent says; null from one that predates the number (version 1). */
agentVersion: number | null;
/** Failed services, disk health and ZFS pools, as the agent says; null until it says anything. */
hostHealth: HostHealth | null;
}
/** The agent version a new install would be, per system; null when this app can't tell. */
@@ -537,6 +539,14 @@ export interface AgentLatest {
windows: number | null;
}
/** What a Linux agent says about the machine itself. Each part is there only if the agent could find it out. */
export interface HostHealth {
failedUnits?: { unit: string; description: string }[];
smart?: { device: string; model: string; healthy: boolean | null; temperatureC: number | null; wearPercent: number | null }[];
zfs?: { name: string; health: string; capacityPercent: number | null }[];
reportedAt: string;
}
export interface PendingUpdates {
manager: string;
count: number;
@@ -727,7 +737,7 @@ export interface Runbook {
export type RunbookSummary = Pick<Runbook, "id" | "targetType" | "targetId" | "category" | "title">;
export type AlertSeverity = "critical" | "warning" | "info";
export type AlertCategory = "offline" | "disk" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
export type AlertCategory = "offline" | "disk" | "services" | "backup" | "updates" | "expiry" | "automation" | "integration" | "monitoring" | "tickets";
export interface AlertItem {
id: string;
@@ -1641,7 +1651,8 @@ export const api = {
clearCache: () => request<{ ok: true }>("/api/dns/cache/clear", { method: "POST" }),
},
servers: {
list: () => request<{ servers: ServerRecord[]; agentLatest: AgentLatest }>("/api/servers"),
list: () => request<{ servers: ServerRecord[]; agentLatest: AgentLatest; agentAutoUpdate: boolean }>("/api/servers"),
setAgentAutoUpdate: (autoUpdate: boolean) => request<{ autoUpdate: boolean }>("/api/servers/agent-update", { method: "PUT", body: JSON.stringify({ autoUpdate }) }),
create: (data: { name: string; hostname?: string; description?: string; osType?: "linux" | "windows" }) =>
request<{ server: ServerRecord; token: string }>("/api/servers", {
method: "POST",
+121
View File
@@ -0,0 +1,121 @@
import type { HostHealth } from "../api/client";
import TimeAgo from "./TimeAgo";
/** The degraded states of a ZFS pool are all "not fine", but only ONLINE is quite fine. */
const poolFine = (health: string) => health.toUpperCase() === "ONLINE";
/**
* What the agent says about the machine itself: services that have failed, whether each disk's SMART check passes and how worn it is, and
* whether each ZFS pool is online. Each part appears only if the agent reported it (no smartctl or no ZFS on the host means no part),
* and the card is left out when it has nothing to say.
*/
export default function HostHealthCard({ health }: { health: HostHealth | null }) {
if (!health) return null;
const { failedUnits, smart, zfs } = health;
const hasSmart = !!smart && smart.length > 0;
const hasZfs = !!zfs && zfs.length > 0;
if (failedUnits === undefined && !hasSmart && !hasZfs) return null;
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">Host health</h3>
{health.reportedAt && (
<div className="card-actions text-secondary small">
reported <TimeAgo value={health.reportedAt} />
</div>
)}
</div>
<div className="card-body">
{failedUnits !== undefined && (
<div className={hasSmart || hasZfs ? "mb-3" : ""}>
<div className="text-secondary small mb-1">Services</div>
{failedUnits.length === 0 ? (
<span className="badge bg-green-lt text-green">none have failed</span>
) : (
<ul className="list-unstyled mb-0">
{failedUnits.map((u) => (
<li key={u.unit}>
<span className="badge bg-red-lt text-red me-2">failed</span>
<code>{u.unit}</code>
{u.description && <span className="text-secondary small ms-2">{u.description}</span>}
</li>
))}
</ul>
)}
</div>
)}
{hasSmart && (
<div className={hasZfs ? "mb-3" : ""}>
<div className="text-secondary small mb-1">Disks (SMART)</div>
<div className="table-responsive">
<table className="table table-sm table-vcenter mb-0">
<thead>
<tr>
<th>Disk</th>
<th>Model</th>
<th>Verdict</th>
<th>Temperature</th>
<th>Life used</th>
</tr>
</thead>
<tbody>
{smart!.map((d) => (
<tr key={d.device}>
<td>
<code>{d.device}</code>
</td>
<td className="text-secondary">{d.model || "—"}</td>
<td>
{d.healthy === null ? (
<span className="text-secondary">—</span>
) : d.healthy ? (
<span className="badge bg-green-lt text-green">passes</span>
) : (
<span className="badge bg-red-lt text-red">failing</span>
)}
</td>
<td>{d.temperatureC === null ? "—" : `${Math.round(d.temperatureC)} °C`}</td>
<td>{d.wearPercent === null ? "—" : `${Math.round(d.wearPercent)}%`}</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
)}
{hasZfs && (
<div>
<div className="text-secondary small mb-1">ZFS pools</div>
<div className="table-responsive">
<table className="table table-sm table-vcenter mb-0">
<thead>
<tr>
<th>Pool</th>
<th>State</th>
<th>Used</th>
</tr>
</thead>
<tbody>
{zfs!.map((p) => (
<tr key={p.name}>
<td>
<code>{p.name}</code>
</td>
<td>
<span className={`badge ${poolFine(p.health) ? "bg-green-lt text-green" : "bg-red-lt text-red"}`}>{p.health}</span>
</td>
<td>{p.capacityPercent === null ? "—" : `${Math.round(p.capacityPercent)}%`}</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
)}
</div>
</div>
);
}
+2 -1
View File
@@ -10,7 +10,8 @@ import TimeAgo from "./TimeAgo";
export const CATEGORY_WORDS: Record<AlertCategory, string> = {
offline: "a server that stopped reporting",
disk: "a disk filling up",
disk: "a disk filling up or failing",
services: "a failed service",
backup: "a backup problem",
updates: "updates waiting",
expiry: "something expiring",
+1
View File
@@ -24,6 +24,7 @@ const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: numb
const CATEGORY_LABELS: Record<AlertCategory, string> = {
offline: "Server down",
disk: "Disk & storage",
services: "Failing services",
backup: "Backups",
updates: "Updates",
expiry: "Expiry",
+3 -1
View File
@@ -18,6 +18,7 @@ import ServerActivity from "../components/ServerActivity";
import TimeAgo from "../components/TimeAgo";
import UptimeRibbon from "../components/UptimeRibbon";
import RunsOn from "../components/RunsOn";
import HostHealthCard from "../components/HostHealthCard";
import RunPlaybook from "../components/RunPlaybook";
import RunbookPanel from "../components/RunbookPanel";
import UpdatesBadge from "../components/UpdatesBadge";
@@ -783,7 +784,7 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
<dd className="col-7">
{agentVersionWords(server.agentVersion)}{" "}
{agentState(server, detail.agentLatest) === "behind" ? (
<span className="badge bg-yellow-lt text-yellow" title="Run its install command again to update it (Servers → Manage servers).">
<span className="badge bg-yellow-lt text-yellow" title="Run its install command again to update it (Servers → Manage servers), or switch on automatic agent updates on the Servers page.">
update available
</span>
) : (
@@ -807,6 +808,7 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
</div>
</div>
<HostHealthCard health={server.hostHealth} />
{hardware.source === "agent" && <ServerTrends serverId={serverId} />}
{(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />}
<div className="card mb-3">
+30
View File
@@ -16,6 +16,7 @@ import { tagBadge, useTagColors } from "../utils/tags";
import { confirmDialog } from "../utils/dialogs";
import { ErrorNotice } from "../components/Notice";
import { toast } from "../utils/toast";
import { readableError } from "../utils/errors";
import { SearchBox } from "../components/TableToolbar";
import { useTableSearch } from "../hooks/useTableSearch";
import SaveViewButton from "../components/SaveViewButton";
@@ -36,6 +37,8 @@ export default function Servers({ user }: { user: CurrentUser }) {
// Until the first answer arrives there is nothing to say about "no servers", so show the page's shape instead of the empty message.
const [loaded, setLoaded] = useState(false);
const [agentLatest, setAgentLatest] = useState<AgentLatest | null>(null);
// Whether agents that are behind replace themselves (an administrator's switch).
const [autoUpdate, setAutoUpdate] = useState<boolean | null>(null);
const [error, setError] = useState<string | null>(null);
const [managingServers, setManagingServers] = useState(false);
const [integrationColors, setIntegrationColors] = useState<Record<string, string>>({});
@@ -76,6 +79,7 @@ export default function Servers({ user }: { user: CurrentUser }) {
.then((res) => {
setServers(res.servers);
setAgentLatest(res.agentLatest);
setAutoUpdate(res.agentAutoUpdate);
})
.catch((err) => setError(String(err)))
.finally(() => setLoaded(true));
@@ -85,6 +89,17 @@ export default function Servers({ user }: { user: CurrentUser }) {
loadServers();
}, [loadServers]);
async function changeAutoUpdate(value: boolean) {
setAutoUpdate(value);
try {
await api.servers.setAgentAutoUpdate(value);
toast.success(value ? "Agents that are behind will update themselves at their next run." : "Agents will no longer update themselves.");
} catch (err) {
setAutoUpdate(!value);
toast.error(readableError(err));
}
}
async function createServer(e: React.FormEvent) {
e.preventDefault();
setError(null);
@@ -245,6 +260,21 @@ export default function Servers({ user }: { user: CurrentUser }) {
</div>
<ErrorNotice error={error} />
{isAdmin && autoUpdate !== null && servers.length > 0 && (
<div className="card mb-3">
<div className="card-body py-2">
<label className="form-check form-switch mb-0">
<input className="form-check-input" type="checkbox" checked={autoUpdate} onChange={(e) => void changeAutoUpdate(e.target.checked)} />
<span className="form-check-label">Update agents automatically</span>
</label>
<div className="text-secondary small mt-1">
An agent that is behind this app downloads the newer script from here, checks it against a checksum the server gives, and replaces itself at its next run. Off by default. An agent needs the
update code to use it (Linux version 4 or later, Windows 3 or later); an older one needs its install command run once (Manage servers).
</div>
</div>
</div>
)}
{managingServers ? (
<>
<div className="card mb-3">