Agent diagnostics ("what the agent couldn't collect") and services to ignore per server

- Agents (Linux 6, Windows 5) note each check they couldn't make and why; the server page turns that into tips and fixes, including when the agent is behind
- Linux --prepare leaves its notes for the report; a missing prepare step is itself noted
- Services to ignore are set per server from the Host health card (names or * patterns), applied on the server so alerts and /metrics drop them at once; saving alert limits keeps the list

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-18 22:16:31 +02:00
1 parent b5384c1e03
commit c3bc6bbf72
26 files changed
+4118 -45

No files matched your search

+21 -1
View File
@@ -26,6 +26,7 @@ number when it is worth installing the agent again for.
| 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). | | 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). |
| 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. | | 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. |
| 4 (Windows) | Also reports failed services, disk health and Storage Spaces pools (below), and what the machine is (maker, model, serial number). | | 4 (Windows) | Also reports failed services, disk health and Storage Spaces pools (below), and what the machine is (maker, model, serial number). |
| 6 (Linux), 5 (Windows) | Also says **what it couldn't collect and why** (no `smartctl`, no systemd, a package manager that didn't answer, Windows Update that timed out …), shown on the server's page with what to do about it (below). |
| 5 (Linux) | Also says what the machine is — maker, model and serial number, read from the firmware — so the inventory can fill itself in (below). | | 5 (Linux) | Also says what the machine is — maker, model and serial number, read from the firmware — so the inventory can fill itself in (below). |
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
@@ -208,10 +209,29 @@ The same three parts, from what Windows itself knows (the agent runs as SYSTEM,
| What | Where it comes from | Notes | | What | Where it comes from | Notes |
|---|---|---| |---|---|---|
| **Failed services** | `Win32_Service` | A service that starts by itself, isn't running, and stopped with an error. Services that finished normally (exit code 0), that start on a trigger (1077) or that are starting up are not failures. Some services always do this (updaters, for one): list them under `ignoreServices` in `agent.json` (or `IGNORE_SERVICES` in the environment), as names or patterns like `gupdate*`. | | **Failed services** | `Win32_Service` | A service that starts by itself, isn't running, and stopped with an error. Services that finished normally (exit code 0), that start on a trigger (1077) or that are starting up are not failures. Some services always do this (updaters, for one): list them under `ignoreServices` in `agent.json` (or `IGNORE_SERVICES` in the environment), as names or patterns like `gupdate*` — or, easier and for either system, on the server's page (below). |
| **Disk health** | `Get-PhysicalDisk` and its reliability counters | Windows' own verdict: *Healthy* passes, *Warning* and *Unhealthy* fail, anything else says nothing. Temperature and wear come from the counters where the drive gives sane ones. Virtual disks are left out. | | **Disk health** | `Get-PhysicalDisk` and its reliability counters | Windows' own verdict: *Healthy* passes, *Warning* and *Unhealthy* fail, anything else says nothing. Temperature and wear come from the counters where the drive gives sane ones. Virtual disks are left out. |
| **Storage pools** | `Get-StoragePool` (Storage Spaces) | *Healthy*, *Warning* and *Unhealthy* are reported as ONLINE, DEGRADED and FAULTED, with how full the pool is, and show beside ZFS pools. A machine with no pools reports an empty list. | | **Storage pools** | `Get-StoragePool` (Storage Spaces) | *Healthy*, *Warning* and *Unhealthy* are reported as ONLINE, DEGRADED and FAULTED, with how full the pool is, and show beside ZFS pools. A machine with no pools reports an empty list. |
## Services to ignore, from the server's page
A service that always shows as failed on one machine (an updater, a unit that is meant to be stopped) can be ignored there: on the server's **Host health**
card an administrator presses **Ignore** beside it, or types a name or a pattern with `*` (`gupdate*`, `snap.*`; case doesn't matter and `.service` is
optional). It stops counting as failed — no alert, not in `/metrics` — the moment it is saved, with no agent to update, and it still shows on the page,
marked *ignored*, with a button to stop ignoring it. The list belongs to that server (it is kept with its own alert limits), and works the same for Linux and
Windows; the Windows agent's own `ignoreServices` still works as well.
## What the agent couldn't collect (Linux version 6, Windows version 5)
Each agent notes, in every report, the checks it couldn't make: nothing there to read it with (*unavailable*) or it tried and couldn't (*failed*), why in a
word, and for a failure at most a line of what the tool said. The server's page turns that into a card, **What the agent couldn't collect**, with words and a
fix for each — "Disk health isn't reported: smartctl isn't installed → install smartmontools" — so the question *why is there no disk health here?* has an
answer on the page. A tip is simply how the machine is (a virtual machine with no serial number); *look at* is something to fix. A machine with no ZFS isn't
told it has no ZFS. The card also says when the agent is behind. The wording lives in the server, so it improves without installing anything again.
On Linux, the step outside the sandbox (`--prepare`) leaves its notes with the other answers, and the report carries them; if that step didn't run at all
(an older systemd unit, or a copy run by hand) that is itself noted, because disk health and ZFS are only read there. `--check-updates` prints the same notes.
## What the machine is (Linux version 5, Windows version 4) ## What the machine is (Linux version 5, Windows version 4)
The agents say who made the machine, its model and its serial number (Linux reads `/sys/class/dmi/id`, where the serial is only The agents say who made the machine, its model and its serial number (Linux reads `/sys/class/dmi/id`, where the serial is only
+2 -1
View File
@@ -223,9 +223,10 @@ what it last reported.
| `no_parent` | bool NOT NULL, default 0 | An admin said this server runs on nothing: no link, and no guess from its Proxmox node. Choosing a server by hand, or going back to automatic, clears it. | | `no_parent` | bool NOT NULL, default 0 | An admin said this server runs on nothing: no link, and no guess from its Proxmox node. Choosing a server by hand, or going back to automatic, clears it. |
| `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. | | `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. |
| `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. | | `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. |
| `thresholds` | text | This server's own alert limits as JSON: `{"serverOfflineMinutes":…,"diskUsagePercent":…,"diskForecastDays":…,"backupStaleHours":…,"mounts":{"/data":{"percent":95,"ignore":true}}}`. Anything left out follows Settings; null when it follows Settings entirely. Ranges are checked on the way in (offline 15–10080 min, disk 50–99 %, forecast 0–90 days, backup 0–2160 h; 0 means off). | | `thresholds` | text | This server's own alert limits as JSON: `{"serverOfflineMinutes":…,"diskUsagePercent":…,"diskForecastDays":…,"backupStaleHours":…,"mounts":{"/data":{"percent":95,"ignore":true}},"ignoreServices":["gupdate*"]}`. `ignoreServices` are services never counted as failed here (names or patterns with `*`, up to 30). Anything left out follows Settings; null when it follows Settings entirely. Ranges are checked on the way in (offline 15–10080 min, disk 50–99 %, forecast 0–90 days, backup 0–2160 h; 0 means off). |
| `host_health` | text | JSON `{"failedUnits":[{unit,description}],"smart":[{device,model,healthy,temperatureC,wearPercent}],"zfs":[{name,health,capacityPercent}],"reportedAt":"…"}`. Each part is present only if the agent reported it, and a report that leaves one out leaves it as it was. Null until an agent says anything. Judged by the health check and shown on the server's page. | | `host_health` | text | JSON `{"failedUnits":[{unit,description}],"smart":[{device,model,healthy,temperatureC,wearPercent}],"zfs":[{name,health,capacityPercent}],"reportedAt":"…"}`. Each part is present only if the agent reported it, and a report that leaves one out leaves it as it was. Null until an agent says anything. Judged by the health check and shown on the server's page. |
| `hw_manufacturer`, `hw_model`, `hw_serial` | text | What the machine is, as its firmware says (the agent reports it; placeholders like "To Be Filled By O.E.M." are left out). Each is replaced only when a report has a value, so an agent that can't tell doesn't wipe what it said. Used to fill in the inventory. | | `hw_manufacturer`, `hw_model`, `hw_serial` | text | What the machine is, as its firmware says (the agent reports it; placeholders like "To Be Filled By O.E.M." are left out). Each is replaced only when a report has a value, so an agent that can't tell doesn't wipe what it said. Used to fill in the inventory. |
| `agent_diagnostics` | text | JSON `{"items":[{check,status,reason?,detail?}],"reportedAt":"…"}`: what the agent says it could not collect (`status` is `unavailable` or `failed`). Replaced by each report that carries the list (an empty list means nothing was missed) and kept when a report doesn't. Turned into words and fixes by the server; never sent to the browser as it is. |
| `agent_version` | integer | Which version of the agent script runs on it, as the agent says in each report. Null until it says; an agent that predates the number counts as version 1. Compared with the scripts the app serves to say which agents are behind. | | `agent_version` | integer | Which version of the agent script runs on it, as the agent says in each report. Null until it says; an agent that predates the number counts as version 1. Compared with the scripts the app serves to say which agents are behind. |
#### `scheduled_tasks` #### `scheduled_tasks`
+3 -1
View File
@@ -572,7 +572,9 @@ Proxmox guest) how old its backup may be — and per disk, either a limit or "no
for this disk". Whatever is left empty follows Settings, so changing the global limit for this disk". Whatever is left empty follows Settings, so changing the global limit
still reaches every server that hasn't said otherwise. They apply everywhere the global still reaches every server that hasn't said otherwise. They apply everywhere the global
ones did: the alerts, the notifications, the dashboard, the map, the forecast, the ones did: the alerts, the notifications, the dashboard, the map, the forecast, the
digest and `/metrics`. digest and `/metrics`. The same place keeps the **services to ignore** on that server — an updater that always shows as failed, say — which can also be set from its
Host health card; and a card, **What the agent couldn't collect**, tells you why a figure is missing (no `smartctl`, an update check that didn't answer …) and what
to do. See AGENT_UPDATE.md.
**Inventory** (Infrastructure → Inventory) keeps the hardware behind it all: name and **Inventory** (Infrastructure → Inventory) keeps the hardware behind it all: name and
kind, maker, model, serial, where it sits (and rack unit), when it was bought and for kind, maker, model, serial, where it sits (and rack unit), when it was bought and for
+2 -1
View File
@@ -105,7 +105,8 @@ even further, to admin only:
- **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are - **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are
administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials
(encrypted). (encrypted).
- **Alert limits** on a server's page: everyone can see the ones a server has of its own; only an administrator changes them. - **Alert limits** on a server's page: everyone can see the ones a server has of its own; only an administrator changes them. The same goes for the **services to
ignore** on its Host health card. Everyone sees *What the agent couldn't collect*.
- **Host health** (failed services, disk health, ZFS pools) is shown to everyone on a server's page; **Update agents automatically** - **Host health** (failed services, disk health, ZFS pools) is shown to everyone on a server's page; **Update agents automatically**
on the Servers page is admin-only. on the Servers page is admin-only.
- **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the - **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the
+79 -10
View File
@@ -16,7 +16,8 @@ set -euo pipefail
# 3 = the package manager is asked outside the sandbox (see --prepare), so dnf and zypper can answer there too # 3 = the package manager is asked outside the sandbox (see --prepare), so dnf and zypper can answer there too
# 4 = also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can update itself when told to # 4 = also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can update itself when told to
# 5 = also says what the machine is (maker, model, serial number), from the firmware, for the inventory # 5 = also says what the machine is (maker, model, serial number), from the firmware, for the inventory
AGENT_VERSION=5 # 6 = also says what it could not collect and why (no smartctl, no systemd, a package manager that failed ...), so the server can say what to fix
AGENT_VERSION=6
ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}" ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}"
if [[ -f "$ENV_FILE" ]]; then if [[ -f "$ENV_FILE" ]]; then
@@ -44,6 +45,8 @@ case "${1:-}" in
esac esac
UPDATES_FILE="${UPDATES_FILE:-/run/homelab-manager-agent/updates.json}" UPDATES_FILE="${UPDATES_FILE:-/run/homelab-manager-agent/updates.json}"
HEALTH_FILE="${HEALTH_FILE:-$(dirname "$UPDATES_FILE")/health.json}" HEALTH_FILE="${HEALTH_FILE:-$(dirname "$UPDATES_FILE")/health.json}"
# What --prepare could not collect and why, left for the report to carry along.
DIAG_FILE="${DIAG_FILE:-$(dirname "$UPDATES_FILE")/diagnostics.json}"
# Kept between runs (the unit's private directory is not): a SMART reading is only taken about once an hour, so sleeping disks are left alone. # Kept between runs (the unit's private directory is not): a SMART reading is only taken about once an hour, so sleeping disks are left alone.
STATE_DIR="${STATE_DIR:-/var/lib/homelab-manager-agent}" STATE_DIR="${STATE_DIR:-/var/lib/homelab-manager-agent}"
# Where the package managers' own error output goes: nowhere, unless a person asked to see it. # Where the package managers' own error output goes: nowhere, unless a person asked to see it.
@@ -55,6 +58,23 @@ if [[ "$MODE" == "report" && ( -z "$API_URL" || -z "$API_TOKEN" ) ]]; then
exit 1 exit 1
fi fi
# Things that could not be collected, each a small JSON object: which check, whether it is "unavailable" (nothing to read it with, or nothing
# there to read) or "failed" (it tried and could not), why in a word, and at most a line of what the tool said. Only problems are noted: a
# check that is not mentioned went fine. The server turns these into what to do about them.
DIAG_ITEMS=()
diag() {
local check="$1" status="$2" reason="${3:-}" detail="${4:-}"
detail=$(printf '%s' "$detail" | tr -d '\000-\037' | head -c 160)
DIAG_ITEMS+=("$(jq -n -c --arg c "$check" --arg s "$status" --arg r "$reason" --arg d "$detail" \
'{check: $c, status: $s} + (if $r != "" then {reason: $r} else {} end) + (if $d != "" then {detail: $d} else {} end)')")
}
# The notes made so far as one JSON array ([] when there are none).
diag_json() {
if (( ${#DIAG_ITEMS[@]} == 0 )); then echo "[]"; return 0; fi
printf '%s\n' "${DIAG_ITEMS[@]}" | jq -s -c '.'
}
CURL_INSECURE_FLAG=() CURL_INSECURE_FLAG=()
case "${API_INSECURE,,}" in case "${API_INSECURE,,}" in
1|true|yes) CURL_INSECURE_FLAG=(-k) ;; 1|true|yes) CURL_INSECURE_FLAG=(-k) ;;
@@ -253,9 +273,12 @@ collect_updates() {
[[ -f /run/reboot-needed ]] && reboot="true" [[ -f /run/reboot-needed ]] && reboot="true"
fi fi
if [[ -n "$manager" && ! "$count" =~ ^[0-9]+$ ]]; then if [[ -z "$manager" ]]; then
diag updates unavailable no_package_manager
elif [[ ! "$count" =~ ^[0-9]+$ ]]; then
# In the journal (systemctl status / journalctl -u homelab-manager-agent) so "not reported" has a reason to look for. # In the journal (systemctl status / journalctl -u homelab-manager-agent) so "not reported" has a reason to look for.
echo "Note: couldn't read the pending updates from $manager - the report goes without them." >&2 echo "Note: couldn't read the pending updates from $manager - the report goes without them." >&2
diag updates failed error "$manager did not answer"
fi fi
if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then
@@ -325,7 +348,10 @@ collect_hardware() {
maker=$(dmi_value sys_vendor board_vendor) maker=$(dmi_value sys_vendor board_vendor)
model=$(dmi_value product_name board_name) model=$(dmi_value product_name board_name)
serial=$(dmi_value product_serial board_serial) serial=$(dmi_value product_serial board_serial)
[[ -n "$maker$model$serial" ]] || return 0 if [[ -z "$maker$model$serial" ]]; then
diag hardware unavailable not_found
return 0
fi
HARDWARE_JSON=$(jq -n -c --arg maker "$maker" --arg model "$model" --arg serial "$serial" \ HARDWARE_JSON=$(jq -n -c --arg maker "$maker" --arg model "$model" --arg serial "$serial" \
'{} | (if $maker != "" then . + {manufacturer: $maker} else . end) | (if $model != "" then . + {model: $model} else . end) | (if $serial != "" then . + {serial: $serial} else . end)') || HARDWARE_JSON="null" '{} | (if $maker != "" then . + {manufacturer: $maker} else . end) | (if $model != "" then . + {model: $model} else . end) | (if $serial != "" then . + {serial: $serial} else . end)') || HARDWARE_JSON="null"
} }
@@ -333,9 +359,15 @@ collect_hardware() {
# Services (systemd units) that have failed. systemctl is asked the way the timers are, so this works inside the sandbox. # Services (systemd units) that have failed. systemctl is asked the way the timers are, so this works inside the sandbox.
collect_failed_units() { collect_failed_units() {
FAILED_UNITS_JSON="null" FAILED_UNITS_JSON="null"
command -v systemctl >/dev/null 2>&1 || return 0 if ! command -v systemctl >/dev/null 2>&1; then
diag failed_units unavailable no_systemd
return 0
fi
local out local out
out=$(timeout 20 systemctl list-units --state=failed --plain --no-legend --no-pager --full 2>"$PM_ERR") || return 0 if ! out=$(timeout 20 systemctl list-units --state=failed --plain --no-legend --no-pager --full 2>"$PM_ERR"); then
diag failed_units failed error "systemctl list-units did not answer"
return 0
fi
FAILED_UNITS_JSON=$(awk 'NF > 0 { unit = $1; $1 = $2 = $3 = $4 = ""; sub(/^ +/, ""); print unit "\t" $0 }' <<<"$out" \ FAILED_UNITS_JSON=$(awk 'NF > 0 { unit = $1; $1 = $2 = $3 = $4 = ""; sub(/^ +/, ""); print unit "\t" $0 }' <<<"$out" \
| jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({unit: .[0], description: (.[1] // "")}) | .[0:100]') || FAILED_UNITS_JSON="null" | jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({unit: .[0], description: (.[1] // "")}) | .[0:100]') || FAILED_UNITS_JSON="null"
} }
@@ -344,7 +376,14 @@ collect_failed_units() {
# --prepare. Asked at most about once an hour, and with -n standby so a sleeping disk is not woken: what was last known of it is kept. # --prepare. Asked at most about once an hour, and with -n standby so a sleeping disk is not woken: what was last known of it is kept.
collect_smart() { collect_smart() {
SMART_JSON="null" SMART_JSON="null"
command -v smartctl >/dev/null 2>&1 || return 0 if ! command -v smartctl >/dev/null 2>&1; then
diag smart unavailable tool_missing smartctl
return 0
fi
if [[ "$(id -u)" != "0" ]]; then
diag smart unavailable not_root
return 0
fi
local cache="$STATE_DIR/smart.json" now_s previous="[]" items="[]" dev out item scan checked local cache="$STATE_DIR/smart.json" now_s previous="[]" items="[]" dev out item scan checked
now_s=$(date +%s) now_s=$(date +%s)
if [[ -r "$cache" ]]; then if [[ -r "$cache" ]]; then
@@ -383,9 +422,15 @@ collect_smart() {
# Each ZFS pool and whether it is ONLINE. Needs /dev/zfs, so it runs from --prepare too. # Each ZFS pool and whether it is ONLINE. Needs /dev/zfs, so it runs from --prepare too.
collect_zfs() { collect_zfs() {
ZFS_JSON="null" ZFS_JSON="null"
command -v zpool >/dev/null 2>&1 || return 0 if ! command -v zpool >/dev/null 2>&1; then
diag zfs unavailable tool_missing zpool
return 0
fi
local out local out
out=$(timeout 20 zpool list -H -o name,health,capacity 2>"$PM_ERR") || return 0 if ! out=$(timeout 20 zpool list -H -o name,health,capacity 2>"$PM_ERR"); then
diag zfs failed error "zpool list did not answer"
return 0
fi
ZFS_JSON=$(jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({name: .[0], health: (.[1] // "UNKNOWN"), capacity_percent: ((.[2] // "" | rtrimstr("%") | tonumber?) // null)})' <<<"$out") || ZFS_JSON="null" ZFS_JSON=$(jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({name: .[0], health: (.[1] // "UNKNOWN"), capacity_percent: ((.[2] // "" | rtrimstr("%") | tonumber?) // null)})' <<<"$out") || ZFS_JSON="null"
} }
@@ -466,6 +511,22 @@ use_prepared_updates() {
UPDATES_JSON=$json UPDATES_JSON=$json
} }
# What this report could not collect: the notes made here, then those --prepare left (it does the outside-the-sandbox part), one for each check.
# With nothing left by --prepare — on a real run, not a dry one — that step did not run, which is worth saying: disk health and ZFS are only read there.
collect_diagnostics() {
local from_prepare="[]" shaped
if [[ -r "$DIAG_FILE" && -n "$(find "$DIAG_FILE" -mmin -10 -print 2>/dev/null)" ]]; then
from_prepare=$(jq -c 'if type == "array" then [.[] | select(type == "object" and (.check | type) == "string" and (.status | type) == "string")
| {check: (.check | tostring), status: (.status | tostring)}
+ (if ((.reason // "") | tostring) != "" then {reason: (.reason | tostring)} else {} end)
+ (if ((.detail // "") | tostring) != "" then {detail: (.detail | tostring | .[0:160])} else {} end)] | .[0:20] else [] end' "$DIAG_FILE" 2>/dev/null) || from_prepare="[]"
elif [[ "$DRY_RUN" != "1" ]]; then
diag prepare unavailable step_missing
fi
shaped=$(jq -c -n --argjson here "$(diag_json)" --argjson there "$from_prepare" '($here + $there) | unique_by(.check) | .[0:20]') || shaped="[]"
DIAGNOSTICS_JSON=$shaped
}
collect_system_info() { collect_system_info() {
local ip_json="[]" local ip_json="[]"
if command -v ip >/dev/null 2>&1; then if command -v ip >/dev/null 2>&1; then
@@ -543,6 +604,7 @@ collect_system_info() {
fi fi
HARDWARE_JSON="null" HARDWARE_JSON="null"
collect_hardware || HARDWARE_JSON="null" collect_hardware || HARDWARE_JSON="null"
collect_diagnostics
HEALTH_JSON=$(jq -n -c --argjson failed "$FAILED_UNITS_JSON" --argjson smart "$SMART_JSON" --argjson zfs "$ZFS_JSON" \ HEALTH_JSON=$(jq -n -c --argjson failed "$FAILED_UNITS_JSON" --argjson smart "$SMART_JSON" --argjson zfs "$ZFS_JSON" \
'{} | (if $failed != null then . + {failed_units: $failed} else . end) | (if $smart != null then . + {smart: $smart} else . end) | (if $zfs != null then . + {zfs: $zfs} else . end)') || HEALTH_JSON="{}" '{} | (if $failed != null then . + {failed_units: $failed} else . end) | (if $smart != null then . + {smart: $smart} else . end) | (if $zfs != null then . + {zfs: $zfs} else . end)') || HEALTH_JSON="{}"
[[ "$HEALTH_JSON" == "{}" ]] && HEALTH_JSON="null" [[ "$HEALTH_JSON" == "{}" ]] && HEALTH_JSON="null"
@@ -551,6 +613,7 @@ collect_system_info() {
--argjson updates "$UPDATES_JSON" \ --argjson updates "$UPDATES_JSON" \
--argjson health "$HEALTH_JSON" \ --argjson health "$HEALTH_JSON" \
--argjson hardware "$HARDWARE_JSON" \ --argjson hardware "$HARDWARE_JSON" \
--argjson diagnostics "$DIAGNOSTICS_JSON" \
--argjson ip_addresses "$ip_json" \ --argjson ip_addresses "$ip_json" \
--argjson listening_ports "$ports_json" \ --argjson listening_ports "$ports_json" \
--arg cpu_model "$cpu_model" \ --arg cpu_model "$cpu_model" \
@@ -567,7 +630,8 @@ collect_system_info() {
listening_ports: $listening_ports, listening_ports: $listening_ports,
updates: $updates, updates: $updates,
health: $health, health: $health,
hardware: $hardware hardware: $hardware,
diagnostics: $diagnostics
}') }')
} }
@@ -587,12 +651,17 @@ if [[ "$MODE" != "report" ]]; then
jq . <<<"$SMART_JSON" jq . <<<"$SMART_JSON"
echo "ZFS pools (null means there is no zpool here):" echo "ZFS pools (null means there is no zpool here):"
jq . <<<"$ZFS_JSON" jq . <<<"$ZFS_JSON"
collect_hardware || true
echo "What could not be collected (empty means nothing was missed):"
diag_json | jq .
self_update self_update
exit 0 exit 0
fi fi
# Leave the answers (or no file at all) for the report that follows. Each is written whole and moved into place, so a report never reads half of one. # Leave the answers (or no file at all) for the report that follows. Each is written whole and moved into place, so a report never reads half of one.
rm -f "$UPDATES_FILE" "$HEALTH_FILE" rm -f "$UPDATES_FILE" "$HEALTH_FILE" "$DIAG_FILE"
mkdir -p "$(dirname "$UPDATES_FILE")" 2>/dev/null || exit 0 mkdir -p "$(dirname "$UPDATES_FILE")" 2>/dev/null || exit 0
# The notes always go, even when there are none: a missing file is how the report knows this step didn't run.
diag_json > "$DIAG_FILE.tmp" 2>/dev/null && mv -f "$DIAG_FILE.tmp" "$DIAG_FILE" 2>/dev/null || true
if [[ "$UPDATES_JSON" != "null" ]]; then if [[ "$UPDATES_JSON" != "null" ]]; then
printf '%s\n' "$UPDATES_JSON" > "$UPDATES_FILE.tmp" 2>/dev/null && mv -f "$UPDATES_FILE.tmp" "$UPDATES_FILE" 2>/dev/null || true printf '%s\n' "$UPDATES_JSON" > "$UPDATES_FILE.tmp" 2>/dev/null && mv -f "$UPDATES_FILE.tmp" "$UPDATES_FILE" 2>/dev/null || true
fi fi
+29 -8
View File
@@ -35,10 +35,23 @@ $ErrorActionPreference = 'Stop'
# change here is worth installing the agent again for. 1 = before the number was reported 2 = reports pending updates, and its own version # change here is worth installing the agent again for. 1 = before the number was reported 2 = reports pending updates, and its own version
# 3 = can replace itself with the newer script when an administrator has switched that on in Homelab Manager # 3 = can replace itself with the newer script when an administrator has switched that on in Homelab Manager
# 4 = also reports services that have failed, disk health, Storage Spaces pools and what the machine is (maker, model, serial) # 4 = also reports services that have failed, disk health, Storage Spaces pools and what the machine is (maker, model, serial)
$script:AgentVersion = 4 # 5 = also says what it could not collect and why, so Homelab Manager can say what to fix
$script:AgentVersion = 5
$script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json' $script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json'
# Things that could not be collected: which check, whether it is 'unavailable' (nothing there to read) or 'failed' (it tried and could not), why in a
# word, and at most a line of what Windows said. Only problems are noted; a check that is not mentioned went fine.
$script:Diag = New-Object System.Collections.ArrayList
function Add-Diag([string]$Check, [string]$Status, [string]$Reason = '', [string]$Detail = '') {
$item = [ordered]@{ check = $Check; status = $Status }
if ($Reason) { $item['reason'] = $Reason }
$clean = ($Detail -replace '[\x00-\x1f]', ' ').Trim()
if ($clean.Length -gt 160) { $clean = $clean.Substring(0, 160) }
if ($clean) { $item['detail'] = $clean }
[void]$script:Diag.Add([pscustomobject]$item)
}
# ---- configuration ------------------------------------------------------------ # ---- configuration ------------------------------------------------------------
function Get-AgentConfig { function Get-AgentConfig {
@@ -287,7 +300,10 @@ function Get-PendingUpdates {
} }
# No answer is no answer: the report then leaves updates out, and the server keeps what it was last told. # No answer is no answer: the report then leaves updates out, and the server keeps what it was last told.
if ($cached.failed) { return $null } if ($cached.failed) {
Add-Diag 'updates' 'failed' 'error' 'Windows Update did not answer; it is asked again in an hour'
return $null
}
return [ordered]@{ manager = 'windows-update'; count = [int]$cached.count; security = [int]$cached.security; reboot_required = (Test-RebootRequired) } return [ordered]@{ manager = 'windows-update'; count = [int]$cached.count; security = [int]$cached.security; reboot_required = (Test-RebootRequired) }
} }
@@ -355,7 +371,7 @@ function Get-HostHealth($Config = $null) {
$health = [ordered]@{} $health = [ordered]@{}
$ignore = @() $ignore = @()
if ($Config -and $Config.IgnoreServices) { $ignore = @($Config.IgnoreServices) } if ($Config -and $Config.IgnoreServices) { $ignore = @($Config.IgnoreServices) }
try { $health['failed_units'] = @(ConvertTo-FailedUnits @(Get-CimInstance -ClassName Win32_Service -ErrorAction Stop) $ignore) } catch { Write-Warning "Reading services failed. ($($_.Exception.Message))" } try { $health['failed_units'] = @(ConvertTo-FailedUnits @(Get-CimInstance -ClassName Win32_Service -ErrorAction Stop) $ignore) } catch { Write-Warning "Reading services failed. ($($_.Exception.Message))"; Add-Diag 'failed_units' 'failed' 'error' $_.Exception.Message }
try { try {
$disks = @(Get-PhysicalDisk -ErrorAction Stop) $disks = @(Get-PhysicalDisk -ErrorAction Stop)
$counters = @{} $counters = @{}
@@ -363,12 +379,12 @@ function Get-HostHealth($Config = $null) {
try { $counters[[string]$d.DeviceId] = Get-StorageReliabilityCounter -PhysicalDisk $d -ErrorAction Stop } catch { } try { $counters[[string]$d.DeviceId] = Get-StorageReliabilityCounter -PhysicalDisk $d -ErrorAction Stop } catch { }
} }
$health['smart'] = @(ConvertTo-DiskHealth $disks $counters) $health['smart'] = @(ConvertTo-DiskHealth $disks $counters)
} catch { Write-Warning "Reading disk health failed. ($($_.Exception.Message))" } } catch { Write-Warning "Reading disk health failed. ($($_.Exception.Message))"; Add-Diag 'smart' 'failed' 'error' $_.Exception.Message }
try { try {
$health['zfs'] = @(ConvertTo-PoolHealth @(Get-StoragePool -ErrorAction Stop | Where-Object { -not $_.IsPrimordial })) $health['zfs'] = @(ConvertTo-PoolHealth @(Get-StoragePool -ErrorAction Stop | Where-Object { -not $_.IsPrimordial }))
} catch { } catch {
# A machine with no Storage Spaces pools gets an error, not an empty list: that is an answer ("no pools"), not a failure to read. # A machine with no Storage Spaces pools gets an error, not an empty list: that is an answer ("no pools"), not a failure to read.
if ($_.FullyQualifiedErrorId -like '*NotFound*') { $health['zfs'] = @() } else { Write-Warning "Reading storage pools failed. ($($_.Exception.Message))" } if ($_.FullyQualifiedErrorId -like '*NotFound*') { $health['zfs'] = @() } else { Write-Warning "Reading storage pools failed. ($($_.Exception.Message))"; Add-Diag 'zfs' 'failed' 'error' $_.Exception.Message }
} }
if ($health.Count -eq 0) { return $null } if ($health.Count -eq 0) { return $null }
return $health return $health
@@ -402,6 +418,7 @@ function Get-HardwareInfo {
} }
function Get-SystemInfo($Config = $null) { function Get-SystemInfo($Config = $null) {
$script:Diag.Clear()
$ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue | $ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue |
Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } | Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } |
ForEach-Object { $_.IPAddress } | Select-Object -Unique) ForEach-Object { $_.IPAddress } | Select-Object -Unique)
@@ -423,18 +440,22 @@ function Get-SystemInfo($Config = $null) {
# Best-effort: Windows Update being unreachable must not cost the rest of the report. # Best-effort: Windows Update being unreachable must not cost the rest of the report.
$updates = $null $updates = $null
try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))" } try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))"; Add-Diag 'updates' 'failed' 'error' $_.Exception.Message }
# The same for what is wrong with the machine, and what it is: neither may cost the rest of the report. # The same for what is wrong with the machine, and what it is: neither may cost the rest of the report.
$health = $null $health = $null
try { $health = Get-HostHealth $Config } catch { Write-Warning "Reading host health failed. ($($_.Exception.Message))" } try { $health = Get-HostHealth $Config } catch { Write-Warning "Reading host health failed. ($($_.Exception.Message))"; Add-Diag 'host_health' 'failed' 'error' $_.Exception.Message }
$hardware = $null $hardware = $null
try { $hardware = Get-HardwareInfo } catch { Write-Warning "Reading the hardware identity failed. ($($_.Exception.Message))" } try {
$hardware = Get-HardwareInfo
if (-not $hardware) { Add-Diag 'hardware' 'unavailable' 'not_found' }
} catch { Write-Warning "Reading the hardware identity failed. ($($_.Exception.Message))"; Add-Diag 'hardware' 'failed' 'error' $_.Exception.Message }
return [ordered]@{ return [ordered]@{
updates = $updates updates = $updates
health = $health health = $health
hardware = $hardware hardware = $hardware
diagnostics = @($script:Diag)
ip_addresses = @($ips) ip_addresses = @($ips)
cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) } cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) }
memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) } memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) }
@@ -0,0 +1 @@
ALTER TABLE `servers` ADD `agent_diagnostics` text;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -267,6 +267,13 @@
"when": 1792344782975, "when": 1792344782975,
"tag": "0037_past_magik", "tag": "0037_past_magik",
"breakpoints": true "breakpoints": true
},
{
"idx": 38,
"version": "6",
"when": 1792354148200,
"tag": "0038_wise_malcolm_colcord",
"breakpoints": true
} }
] ]
} }
+3
View File
@@ -341,6 +341,9 @@ export const servers = sqliteTable("servers", {
// What the agent says about the machine's own health, as JSON: {failedUnits, smart, zfs, reportedAt} — see services/hostHealth.ts. Null until it says. // What the agent says about the machine's own health, as JSON: {failedUnits, smart, zfs, reportedAt} — see services/hostHealth.ts. Null until it says.
hostHealth: text("host_health"), hostHealth: text("host_health"),
// What the agent says it could not collect, as JSON: {items: [{check, status, reason?, detail?}], reportedAt} — see services/agentDiagnostics.ts. Null until an agent that can say so has.
agentDiagnostics: text("agent_diagnostics"),
// What the machine is, as its firmware says (agents report it): maker, model and serial number. Used to fill in the inventory. // What the machine is, as its firmware says (agents report it): maker, model and serial number. Used to fill in the inventory.
hwManufacturer: text("hw_manufacturer"), hwManufacturer: text("hw_manufacturer"),
hwModel: text("hw_model"), hwModel: text("hw_model"),
+16
View File
@@ -59,7 +59,23 @@ const hardwareSchema = z.object({
serial: text(100).optional().catch(undefined), serial: text(100).optional().catch(undefined),
}); });
// What the agent could not collect: leniently read, each entry on its own, so one odd one never costs the report.
const diagnosticsSchema = z.unknown().transform((v) => {
if (!Array.isArray(v)) return undefined;
const out: { check: string; status: "unavailable" | "failed"; reason?: string; detail?: string }[] = [];
for (const i of v.slice(0, 20)) {
if (!i || typeof i !== "object") continue;
const e = i as Record<string, unknown>;
const clean = (x: unknown, max: number) => (typeof x === "string" ? x.replace(/[\u0000-\u001f]/g, " ").trim().slice(0, max) : "");
const check = clean(e.check, 40);
if (!check || (e.status !== "unavailable" && e.status !== "failed")) continue;
out.push({ check, status: e.status, ...(clean(e.reason, 40) ? { reason: clean(e.reason, 40) } : {}), ...(clean(e.detail, 160) ? { detail: clean(e.detail, 160) } : {}) });
}
return out;
});
const systemSchema = z.object({ const systemSchema = z.object({
diagnostics: diagnosticsSchema.optional(),
hardware: z hardware: z
.unknown() .unknown()
.optional() .optional()
+6 -4
View File
@@ -8,7 +8,8 @@ import { generateApiToken } from "../services/tokens.js";
import { recordAudit } from "../services/audit.js"; import { recordAudit } from "../services/audit.js";
import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js"; import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js";
import { parsePendingUpdates } from "../services/pendingUpdates.js"; import { parsePendingUpdates } from "../services/pendingUpdates.js";
import { parseHostHealth } from "../services/hostHealth.js"; import { markIgnoredServices, parseHostHealth } from "../services/hostHealth.js";
import { describeDiagnostics, parseDiagnostics } from "../services/agentDiagnostics.js";
import { checkThresholds, parseThresholds } from "../services/serverThresholds.js"; import { checkThresholds, parseThresholds } from "../services/serverThresholds.js";
import { latestAgentVersions } from "../services/agentVersions.js"; import { latestAgentVersions } from "../services/agentVersions.js";
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js"; import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
@@ -27,8 +28,8 @@ import { getSettings, updateSettings } from "../services/settingsStore.js";
export const serversRouter = Router(); export const serversRouter = Router();
/** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */ /** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */
function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null; hostHealth: string | null; thresholds: string | null }>(row: T) { function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null; hostHealth: string | null; thresholds: string | null; agentDiagnostics?: string | null }>(row: T) {
const { apiTokenHash: _hash, tags, pendingUpdates, hostHealth, thresholds, ...rest } = row; const { apiTokenHash: _hash, tags, pendingUpdates, hostHealth, thresholds, agentDiagnostics: _diagnostics, ...rest } = row;
return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates), hostHealth: parseHostHealth(hostHealth), thresholds: parseThresholds(thresholds) }; return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates), hostHealth: parseHostHealth(hostHealth), thresholds: parseThresholds(thresholds) };
} }
@@ -382,6 +383,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
pendingUpdates: rawUpdates, pendingUpdates: rawUpdates,
hostHealth: rawHealth, hostHealth: rawHealth,
thresholds: rawThresholds, thresholds: rawThresholds,
agentDiagnostics: rawDiagnostics,
...serverOut ...serverOut
} = server; } = server;
@@ -395,7 +397,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)), children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)),
}; };
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates), hostHealth: parseHostHealth(rawHealth), thresholds: parseThresholds(rawThresholds) }, thresholdDefaults: (await getSettings()).healthChecks, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() }); res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates), hostHealth: markIgnoredServices(parseHostHealth(rawHealth), parseThresholds(rawThresholds)?.ignoreServices), thresholds: parseThresholds(rawThresholds) }, diagnostics: describeDiagnostics(parseDiagnostics(rawDiagnostics), { osType: server.osType, lastSeenAt: server.lastSeenAt, agentVersion: server.agentVersion, latest: latestAgentVersions() }), thresholdDefaults: (await getSettings()).healthChecks, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() });
})); }));
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) }); const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
+130
View File
@@ -0,0 +1,130 @@
/**
* What an agent could not collect, and what to do about it. Agents (Linux version 6, Windows version 5) note each check they couldn't make —
* with whether nothing was there to read it with ("unavailable") or it tried and couldn't ("failed"), why in a word and at most a line of
* what the tool said. They say only problems; a check not mentioned went fine. The words and the fixes live here, not in the agents, so they
* can be improved without installing anything again.
*/
import { isAgentOutdated, type LatestAgentVersions } from "./agentVersions.js";
export interface AgentDiagnostic {
check: string;
status: "unavailable" | "failed";
reason?: string;
detail?: string;
}
export interface StoredDiagnostics {
items: AgentDiagnostic[];
/** ISO — the report these came from. */
reportedAt: string;
}
export interface DiagnosticNote {
check: string;
/** "warning": something to look at. "info": a tip, or something that is simply how this machine is. */
level: "warning" | "info";
title: string;
fix: string | null;
/** What the agent said about it, when it said anything. */
detail: string | null;
}
const clean = (v: unknown, max: number) => (typeof v === "string" ? v.replace(/[\u0000-\u001f]/g, " ").trim().slice(0, max) : "");
/** Reads what is stored, leniently. Null when the agent has never said (an agent that predates this, or one that has nothing to report on). */
export function parseDiagnostics(stored: string | null | undefined): StoredDiagnostics | null {
if (!stored) return null;
try {
const v = JSON.parse(stored);
if (!v || typeof v !== "object" || !Array.isArray(v.items)) return null;
const items: AgentDiagnostic[] = [];
for (const i of v.items.slice(0, 20)) {
if (!i || typeof i !== "object") continue;
const check = clean(i.check, 40);
if (!check || (i.status !== "unavailable" && i.status !== "failed")) continue;
const item: AgentDiagnostic = { check, status: i.status };
const reason = clean(i.reason, 40);
const detail = clean(i.detail, 160);
if (reason) item.reason = reason;
if (detail) item.detail = detail;
items.push(item);
}
return { items, reportedAt: typeof v.reportedAt === "string" ? v.reportedAt : "" };
} catch {
return null;
}
}
/** What to store after a report that carried notes (an empty list is an answer: nothing was missed). */
export function nextDiagnostics(incoming: AgentDiagnostic[], now: string): string {
return JSON.stringify({ items: incoming, reportedAt: now } satisfies StoredDiagnostics);
}
type Os = "linux" | "windows";
/** One note for what the agent said, or null when it is not worth a person's attention (no ZFS on a machine that has none). */
function describe(d: AgentDiagnostic, os: Os): Omit<DiagnosticNote, "detail"> | null {
const linux = os === "linux";
const key = `${d.check}/${d.status}${d.reason ? `/${d.reason}` : ""}`;
switch (key) {
case "prepare/unavailable/step_missing":
return {
check: d.check,
level: "warning",
title: "The agent's step outside its sandbox didn't run",
fix: "Disk health and ZFS pools are only read in that step. Install the agent again (Servers → Manage servers has the command) so its systemd unit has it; if it is installed, look at `journalctl -u homelab-manager-agent` on the host.",
};
case "updates/unavailable/no_package_manager":
return { check: d.check, level: "info", title: "Pending updates can't be read here", fix: "The agent understands apt, dnf, zypper and pacman, and Windows Update. This machine has none of them." };
case "failed_units/unavailable/no_systemd":
return { check: d.check, level: "info", title: "Failed services can't be read: this machine doesn't run systemd", fix: null };
case "smart/unavailable/tool_missing":
return { check: d.check, level: "info", title: "Disk health isn't reported: smartctl isn't installed", fix: "Install smartmontools on the host (apt install smartmontools, or dnf install smartmontools); the next report includes it." };
case "smart/unavailable/not_root":
return { check: d.check, level: "warning", title: "Disk health needs the agent to run as root", fix: "The install script's systemd unit runs it as root. A copy started by hand as another user can't read the disks." };
case "zfs/unavailable/tool_missing":
return null; // most machines have no ZFS, and that is not a problem
case "hardware/unavailable/not_found":
return { check: d.check, level: "info", title: "No maker, model or serial number found", fix: "Many virtual machines and single-board computers don't have them. Type them into the Inventory instead." };
}
if (d.status === "failed") {
const what: Record<string, string> = { updates: "pending updates", failed_units: "services", smart: "disk health", zfs: "storage pools", host_health: "the machine's health", hardware: "what the machine is" };
const fixes: Record<string, string> = linux
? {
updates: "Run `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` on the host to see what the package manager said (dnf and zypper answer from their caches: `sudo dnf makecache`).",
failed_units: "`systemctl list-units --state=failed` didn't answer in time on that host.",
zfs: "`zpool list` didn't answer; check that the ZFS kernel module is loaded.",
}
: {
updates: "Windows Update didn't answer (the search is tried again after an hour). Check the Windows Update service and that the machine can reach Microsoft.",
failed_units: "Reading services through WMI failed. Check that the Windows Management Instrumentation service is running.",
smart: "Get-PhysicalDisk failed. In an elevated PowerShell, run `Get-PhysicalDisk` to see why.",
zfs: "Get-StoragePool failed. In an elevated PowerShell, run `Get-StoragePool` to see why.",
host_health: "Reading the machine's health failed; the detail says why.",
hardware: "Reading the system's own details through WMI failed; the detail says why.",
};
return { check: d.check, level: "warning", title: `Reading ${what[d.check] ?? d.check} failed`, fix: fixes[d.check] ?? null };
}
return { check: d.check, level: "info", title: `${d.check.replace(/_/g, " ")} isn't available${d.reason ? ` (${d.reason.replace(/_/g, " ")})` : ""}`, fix: null };
}
/** What to tell a person about one server's agent: what it couldn't collect (with what to do) and, when it is behind, that. Warnings first. */
export function describeDiagnostics(stored: StoredDiagnostics | null, context: { osType: string; lastSeenAt: string | null; agentVersion: number | null; latest: LatestAgentVersions }): DiagnosticNote[] {
const os: Os = context.osType === "windows" ? "windows" : "linux";
const notes: DiagnosticNote[] = [];
for (const d of stored?.items ?? []) {
const note = describe(d, os);
if (note) notes.push({ ...note, detail: d.status === "failed" ? (d.detail ?? null) : null }); // what the agent said is only worth showing for a failure
}
if (isAgentOutdated({ osType: context.osType, lastSeenAt: context.lastSeenAt, agentVersion: context.agentVersion }, context.latest)) {
const current = os === "windows" ? context.latest.windows : context.latest.linux;
notes.push({
check: "agent_behind",
level: "info",
title: `This agent is version ${context.agentVersion ?? 1}; ${current} is current`,
fix: "A newer agent reports more and says more about what it couldn't read. Update it from the Updates page, or switch on automatic updates for agents in Settings.",
detail: null,
});
}
return notes.sort((a, b) => Number(b.level === "warning") - Number(a.level === "warning"));
}
+1
View File
@@ -11,6 +11,7 @@
* 4 — Linux: also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can replace itself too. * 4 — Linux: also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can replace itself too.
* Windows: also reports failed services, disk health and Storage Spaces pools, and what the machine is (maker, model, serial) * Windows: also reports failed services, disk health and Storage Spaces pools, and what the machine is (maker, model, serial)
* 5 — Linux: also says what the machine is (maker, model, serial), for the inventory * 5 — Linux: also says what the machine is (maker, model, serial), for the inventory
* 6 — Linux (and Windows 5): also says what it could not collect and why, so the server can say what to fix
*/ */
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
+2 -2
View File
@@ -10,7 +10,7 @@ import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.j
import { activeSubjects, subjectOfConditionKey } from "./maintenance.js"; import { activeSubjects, subjectOfConditionKey } from "./maintenance.js";
import { dependencyCover, HEALTH_STATE_FLAG } from "./dependencies.js"; import { dependencyCover, HEALTH_STATE_FLAG } from "./dependencies.js";
import { getFailingSources } from "./integrationHealthMonitor.js"; import { getFailingSources } from "./integrationHealthMonitor.js";
import { hostHealthConditions, parseHostHealth, type HostHealth } from "./hostHealth.js"; import { hostHealthConditions, parseHostHealth, withoutIgnoredServices, type HostHealth } from "./hostHealth.js";
import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js"; import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js";
const STATE_FLAG = HEALTH_STATE_FLAG; const STATE_FLAG = HEALTH_STATE_FLAG;
@@ -306,7 +306,7 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel
} catch { } catch {
// a malformed report shouldn't stop every other server being checked // a malformed report shouldn't stop every other server being checked
} }
snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: parseHostHealth(s.hostHealth), thresholds: parseThresholds(s.thresholds) }); snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: withoutIgnoredServices(parseHostHealth(s.hostHealth), parseThresholds(s.thresholds)?.ignoreServices), thresholds: parseThresholds(s.thresholds) });
} }
const rows = await db const rows = await db
+27
View File
@@ -8,6 +8,8 @@
export interface FailedUnit { export interface FailedUnit {
unit: string; unit: string;
description: string; description: string;
/** Set where it is shown, for a unit this server has been told to ignore. It never counts as failed. */
ignored?: boolean;
} }
export interface SmartDisk { export interface SmartDisk {
device: string; device: string;
@@ -82,6 +84,31 @@ export function nextHostHealth(previous: string | null, incoming: IncomingHostHe
return any ? JSON.stringify(next) : previous; return any ? JSON.stringify(next) : previous;
} }
const globToRegex = (pattern: string) => new RegExp(`^${pattern.trim().toLowerCase().replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*")}$`);
/** Whether a unit is one this server is told to ignore: a name, or a pattern with * ("gupdate*"); case doesn't matter and ".service" may be left off. */
export function isServiceIgnored(unit: string, patterns: string[] | undefined): boolean {
if (!patterns || patterns.length === 0) return false;
const u = unit.toLowerCase();
const bare = u.replace(/\.service$/, "");
return patterns.some((p) => {
const re = globToRegex(p);
return re.test(u) || re.test(bare);
});
}
/** The health with the ignored services taken out: what alerts and figures are worked from. */
export function withoutIgnoredServices(health: HostHealth | null, patterns: string[] | undefined): HostHealth | null {
if (!health?.failedUnits || !patterns || patterns.length === 0) return health;
return { ...health, failedUnits: health.failedUnits.filter((u) => !isServiceIgnored(u.unit, patterns)) };
}
/** The health with the ignored services marked, not removed: what the server's page shows, so they can be seen and taken off the list. */
export function markIgnoredServices(health: HostHealth | null, patterns: string[] | undefined): HostHealth | null {
if (!health?.failedUnits || !patterns || patterns.length === 0) return health;
return { ...health, failedUnits: health.failedUnits.map((u) => (isServiceIgnored(u.unit, patterns) ? { ...u, ignored: true } : u)) };
}
/** A pool in any state but ONLINE has lost something (DEGRADED has lost redundancy, the rest are worse). */ /** A pool in any state but ONLINE has lost something (DEGRADED has lost redundancy, the rest are worse). */
export const poolIsFine = (p: ZfsPool) => p.health.toUpperCase() === "ONLINE"; export const poolIsFine = (p: ZfsPool) => p.health.toUpperCase() === "ONLINE";
+3 -2
View File
@@ -10,7 +10,8 @@ import { getAlerts } from "./alerts.js";
import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js"; import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js";
import { getSourceStatus } from "./integrationHealthMonitor.js"; import { getSourceStatus } from "./integrationHealthMonitor.js";
import { parsePendingUpdates } from "./pendingUpdates.js"; import { parsePendingUpdates } from "./pendingUpdates.js";
import { parseHostHealth, poolIsFine } from "./hostHealth.js"; import { parseHostHealth, poolIsFine, withoutIgnoredServices } from "./hostHealth.js";
import { parseThresholds as parseServerThresholds } from "./serverThresholds.js";
import { parseThresholds } from "./serverThresholds.js"; import { parseThresholds } from "./serverThresholds.js";
import { getCachedBackupCoverage } from "./backupCoverage.js"; import { getCachedBackupCoverage } from "./backupCoverage.js";
import { lastAttempt as lastSelfBackup, listSnapshots } from "./selfBackup.js"; import { lastAttempt as lastSelfBackup, listSnapshots } from "./selfBackup.js";
@@ -88,7 +89,7 @@ export async function buildMetrics(now: Date = new Date()): Promise<string> {
if (updates.security !== null) out.add("homelab_server_pending_security_updates", "gauge", "Of those, the ones that are security fixes.", updates.security, { server }); if (updates.security !== null) out.add("homelab_server_pending_security_updates", "gauge", "Of those, the ones that are security fixes.", updates.security, { server });
out.add("homelab_server_reboot_required", "gauge", "1 if a restart is due.", updates.rebootRequired ? 1 : 0, { server }); out.add("homelab_server_reboot_required", "gauge", "1 if a restart is due.", updates.rebootRequired ? 1 : 0, { server });
} }
const health = parseHostHealth(s.hostHealth); const health = withoutIgnoredServices(parseHostHealth(s.hostHealth), parseServerThresholds(s.thresholds)?.ignoreServices);
if (health?.failedUnits) out.add("homelab_server_failed_units", "gauge", "Services (systemd units) that have failed on the server.", health.failedUnits.length, { server }); if (health?.failedUnits) out.add("homelab_server_failed_units", "gauge", "Services (systemd units) that have failed on the server.", health.failedUnits.length, { server });
for (const d of health?.smart ?? []) { for (const d of health?.smart ?? []) {
if (d.healthy !== null) out.add("homelab_server_disk_healthy", "gauge", "1 if the disk's own SMART self-assessment passes, 0 if it fails.", d.healthy ? 1 : 0, { server, device: d.device }); if (d.healthy !== null) out.add("homelab_server_disk_healthy", "gauge", "1 if the disk's own SMART self-assessment passes, 0 if it fails.", d.healthy ? 1 : 0, { server, device: d.device });
+23
View File
@@ -21,6 +21,8 @@ export interface ServerThresholds {
/** 0 turns the stale-backup alert off for this server's guest. */ /** 0 turns the stale-backup alert off for this server's guest. */
backupStaleHours?: number; backupStaleHours?: number;
mounts?: Record<string, MountThreshold>; mounts?: Record<string, MountThreshold>;
/** Services (units) never to count as failed on this server: names or patterns like "gupdate*" (case does not matter, ".service" is optional). */
ignoreServices?: string[];
} }
/** The global values these override. */ /** The global values these override. */
@@ -38,6 +40,8 @@ export const LIMITS = {
backupStaleHours: { min: 0, max: 2160 }, backupStaleHours: { min: 0, max: 2160 },
} as const; } as const;
export const MAX_MOUNT_RULES = 50; export const MAX_MOUNT_RULES = 50;
export const MAX_IGNORED_SERVICES = 30;
const MAX_SERVICE_PATTERN = 100;
const inRange = (v: unknown, key: keyof typeof LIMITS): v is number => typeof v === "number" && Number.isInteger(v) && v >= LIMITS[key].min && v <= LIMITS[key].max; const inRange = (v: unknown, key: keyof typeof LIMITS): v is number => typeof v === "number" && Number.isInteger(v) && v >= LIMITS[key].min && v <= LIMITS[key].max;
@@ -61,6 +65,11 @@ export function parseThresholds(stored: string | null | undefined): ServerThresh
} }
if (Object.keys(mounts).length > 0) out.mounts = mounts; if (Object.keys(mounts).length > 0) out.mounts = mounts;
} }
if (Array.isArray(v.ignoreServices)) {
const strings: string[] = (v.ignoreServices as unknown[]).filter((p): p is string => typeof p === "string");
const list = [...new Set(strings.map((p) => p.trim()).filter((p) => p.length > 0 && p.length <= MAX_SERVICE_PATTERN))].slice(0, MAX_IGNORED_SERVICES);
if (list.length > 0) out.ignoreServices = list;
}
return Object.keys(out).length > 0 ? out : null; return Object.keys(out).length > 0 ? out : null;
} catch { } catch {
return null; return null;
@@ -103,6 +112,20 @@ export function checkThresholds(input: unknown): { ok: true; value: ServerThresh
} }
if (Object.keys(mounts).length > 0) out.mounts = mounts; if (Object.keys(mounts).length > 0) out.mounts = mounts;
} }
if (v.ignoreServices !== undefined && v.ignoreServices !== null) {
if (!Array.isArray(v.ignoreServices)) return { ok: false, message: "The services to ignore should be a list." };
const list: string[] = [];
for (const raw of v.ignoreServices) {
if (typeof raw !== "string") return { ok: false, message: "Each service to ignore should be a name or a pattern." };
const p = raw.trim();
if (p.length === 0) continue;
// eslint-disable-next-line no-control-regex
if (p.length > MAX_SERVICE_PATTERN || /[\u0000-\u001f]/.test(p)) return { ok: false, message: `"${p.slice(0, 30)}…" isn't a service name or pattern (up to ${MAX_SERVICE_PATTERN} characters, no line breaks).` };
if (!list.includes(p)) list.push(p);
}
if (list.length > MAX_IGNORED_SERVICES) return { ok: false, message: `At most ${MAX_IGNORED_SERVICES} services can be ignored on one server.` };
if (list.length > 0) out.ignoreServices = list;
}
return { ok: true, value: Object.keys(out).length > 0 ? out : null }; return { ok: true, value: Object.keys(out).length > 0 ? out : null };
} }
+4
View File
@@ -6,6 +6,7 @@ import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js";
import { nextHostHealth, type IncomingHostHealth } from "./hostHealth.js"; import { nextHostHealth, type IncomingHostHealth } from "./hostHealth.js";
import { recordAudit } from "./audit.js"; import { recordAudit } from "./audit.js";
import { fillInventoryFromServer } from "./inventoryAutofill.js"; import { fillInventoryFromServer } from "./inventoryAutofill.js";
import { nextDiagnostics, type AgentDiagnostic } from "./agentDiagnostics.js";
export interface IncomingTask { export interface IncomingTask {
scheduleType: "cron" | "systemd_timer" | "windows_task"; scheduleType: "cron" | "systemd_timer" | "windows_task";
@@ -30,6 +31,8 @@ export interface IncomingSystemInfo {
health?: IncomingHostHealth; health?: IncomingHostHealth;
/** What the machine is, from its firmware: each part present only if it says anything. */ /** What the machine is, from its firmware: each part present only if it says anything. */
hardware?: { manufacturer?: string; model?: string; serial?: string }; hardware?: { manufacturer?: string; model?: string; serial?: string };
/** What the agent could not collect; an empty list says nothing was missed. */
diagnostics?: AgentDiagnostic[];
} }
export interface AgentReport { export interface AgentReport {
@@ -118,6 +121,7 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
// Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before. // Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before.
...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}), ...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}),
...(system?.health && before ? { hostHealth: nextHostHealth(before.hostHealth, system.health, now) } : {}), ...(system?.health && before ? { hostHealth: nextHostHealth(before.hostHealth, system.health, now) } : {}),
...(system?.diagnostics ? { agentDiagnostics: nextDiagnostics(system.diagnostics, now) } : {}),
...(system?.hardware?.manufacturer ? { hwManufacturer: system.hardware.manufacturer } : {}), ...(system?.hardware?.manufacturer ? { hwManufacturer: system.hardware.manufacturer } : {}),
...(system?.hardware?.model ? { hwModel: system.hardware.model } : {}), ...(system?.hardware?.model ? { hwModel: system.hardware.model } : {}),
...(system?.hardware?.serial ? { hwSerial: system.hardware.serial } : {}), ...(system?.hardware?.serial ? { hwSerial: system.hardware.serial } : {}),
+207
View File
@@ -0,0 +1,207 @@
/**
* What an agent could not collect and what to do about it, and the services a server is told to ignore.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-diag-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { describeDiagnostics, parseDiagnostics } = await import("../src/services/agentDiagnostics.js");
const { isServiceIgnored, withoutIgnoredServices, markIgnoredServices } = await import("../src/services/hostHealth.js");
const { checkThresholds, parseThresholds } = await import("../src/services/serverThresholds.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { buildMetrics } = await import("../src/services/metrics.js");
const { serversRouter } = await import("../src/routes/servers.js");
const { agentReportRouter } = await import("../src/routes/agentReport.js");
const { hashToken } = await import("../src/services/tokens.js");
await runMigrations();
const latest = { linux: 6, windows: 5 };
const ctx = (extra: Partial<{ osType: string; lastSeenAt: string | null; agentVersion: number | null }> = {}) => ({ osType: "linux", lastSeenAt: "2026-10-18T10:00:00Z", agentVersion: 6, latest, ...extra });
const stored = (items: unknown[]) => JSON.stringify({ items, reportedAt: "2026-10-18T10:00:00Z" });
describe("saying what to do about what couldn't be collected", () => {
test("a missing tool is a tip with the fix; ZFS that isn't there is not worth saying", () => {
const notes = describeDiagnostics(
parseDiagnostics(stored([
{ check: "smart", status: "unavailable", reason: "tool_missing", detail: "smartctl" },
{ check: "zfs", status: "unavailable", reason: "tool_missing", detail: "zpool" },
{ check: "hardware", status: "unavailable", reason: "not_found" },
])),
ctx(),
);
assert.deepEqual(notes.map((n) => n.check), ["smart", "hardware"], "no word about ZFS");
assert.equal(notes[0].level, "info");
assert.match(notes[0].fix!, /smartmontools/);
});
test("a failure is a warning that says what the agent said, with a fix for the system it is on", () => {
const failed = stored([{ check: "updates", status: "failed", reason: "error", detail: "apt did not answer" }]);
const linux = describeDiagnostics(parseDiagnostics(failed), ctx())[0];
assert.equal(linux.level, "warning");
assert.equal(linux.detail, "apt did not answer");
assert.match(linux.fix!, /--check-updates/);
const windows = describeDiagnostics(parseDiagnostics(failed), ctx({ osType: "windows", agentVersion: 5 }))[0];
assert.match(windows.fix!, /Windows Update/);
});
test("a step that didn't run is a warning, and warnings come first", () => {
const notes = describeDiagnostics(parseDiagnostics(stored([{ check: "hardware", status: "unavailable", reason: "not_found" }, { check: "prepare", status: "unavailable", reason: "step_missing" }])), ctx());
assert.deepEqual(notes.map((n) => [n.check, n.level]), [["prepare", "warning"], ["hardware", "info"]]);
});
test("something it has no words for is still said, rather than dropped", () => {
const notes = describeDiagnostics(parseDiagnostics(stored([{ check: "gpu_temps", status: "failed", reason: "error" }, { check: "fans", status: "unavailable", reason: "no_sensor" }])), ctx());
assert.equal(notes[0].level, "warning");
assert.match(notes[0].title, /gpu_temps/);
assert.match(notes[1].title, /fans isn't available \(no sensor\)/);
});
test("an agent that is behind is told so, whether or not it said anything", () => {
const notes = describeDiagnostics(null, ctx({ agentVersion: 3 }));
assert.equal(notes.length, 1);
assert.match(notes[0].title, /version 3; 6 is current/);
assert.deepEqual(describeDiagnostics(null, ctx()), [], "an agent that is current and silent has nothing to say");
assert.deepEqual(describeDiagnostics(null, ctx({ lastSeenAt: null, agentVersion: 3 })), [], "a server that has never reported has no agent to update");
});
test("what is stored is read leniently", () => {
assert.equal(parseDiagnostics(null), null);
assert.equal(parseDiagnostics("not json"), null);
const read = parseDiagnostics(stored([{ check: "ok", status: "fine" }, "x", { check: "smart", status: "failed", detail: "a\nb" }]))!;
assert.deepEqual(read.items, [{ check: "smart", status: "failed", detail: "a b" }]);
});
});
describe("services a server is told to ignore", () => {
test("names and patterns, in any case, with or without .service", () => {
assert.equal(isServiceIgnored("gupdate.service", ["gupdate"]), true);
assert.equal(isServiceIgnored("GUpdate", ["gupdate*"]), true);
assert.equal(isServiceIgnored("snap.lxd.activate.service", ["snap.*"]), true);
assert.equal(isServiceIgnored("nginx.service", ["gupdate*", "snap.*"]), false);
assert.equal(isServiceIgnored("nginx.service", ["nginx.service"]), true);
assert.equal(isServiceIgnored("mynginx.service", ["nginx"]), false, "a name is the whole name");
assert.equal(isServiceIgnored("a+b.service", ["a+b"]), true, "characters that mean something to a regex are only characters here");
assert.equal(isServiceIgnored("anything", []), false);
assert.equal(isServiceIgnored("anything", undefined), false);
});
test("left out for what is judged, marked for what is shown", () => {
const health = { failedUnits: [{ unit: "nginx.service", description: "" }, { unit: "gupdate", description: "Google Update" }], reportedAt: "t" };
assert.deepEqual(withoutIgnoredServices(health, ["gupdate*"])!.failedUnits!.map((u) => u.unit), ["nginx.service"]);
assert.deepEqual(markIgnoredServices(health, ["gupdate*"])!.failedUnits!.map((u) => [u.unit, u.ignored ?? false]), [["nginx.service", false], ["gupdate", true]]);
assert.equal(withoutIgnoredServices(null, ["x"]), null);
assert.deepEqual(withoutIgnoredServices(health, []), health);
});
test("the list is part of a server's own thresholds: kept tidy, and a wrong one is refused", () => {
const ok = checkThresholds({ ignoreServices: [" gupdate* ", "gupdate*", "", "snap.*"] });
assert.deepEqual(ok.ok && ok.value, { ignoreServices: ["gupdate*", "snap.*"] });
assert.equal(checkThresholds({ ignoreServices: "gupdate" }).ok, false);
assert.equal(checkThresholds({ ignoreServices: [3] }).ok, false);
assert.equal(checkThresholds({ ignoreServices: ["x".repeat(101)] }).ok, false);
assert.equal(checkThresholds({ ignoreServices: Array.from({ length: 31 }, (_, i) => `s${i}`) }).ok, false);
assert.equal(checkThresholds({ ignoreServices: [] }).ok && (checkThresholds({ ignoreServices: [] }) as { value: unknown }).value, null, "an empty list is no thresholds at all");
assert.deepEqual(parseThresholds(JSON.stringify({ ignoreServices: ["a", 3, "", "a"] })), { ignoreServices: ["a"] });
});
});
describe("through a report, the page, the alerts and the figures", () => {
let app: Server;
let base = "";
let id = 0;
before(async () => {
await db.insert(users).values([{ oidcSub: "admin", name: "Ada", role: "admin" }]);
const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("tok"), apiTokenPrefix: "t", lastSeenAt: new Date().toISOString(), agentVersion: 6 }).returning();
id = s.id;
const a = express();
a.use("/api/agent/report", agentReportRouter);
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "admin" } };
next();
});
a.use("/api/servers", serversRouter);
app = await new Promise<Server>((resolve) => {
const x = a.listen(0, "127.0.0.1", () => resolve(x));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const report = (system: unknown) =>
fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer tok" }, body: JSON.stringify({ hostname: "h", os_type: "linux", agent_version: 6, tasks: [], system }) });
const detail = async () => await (await fetch(`${base}/servers/${id}/detail`)).json();
const patch = (body: unknown) => fetch(`${base}/servers/${id}`, { method: "PATCH", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
test("what the agent says it couldn't collect is shown with what to do; garbage entries are dropped; silence keeps what was said", async () => {
assert.equal(
(await report({ diagnostics: [{ check: "smart", status: "unavailable", reason: "tool_missing", detail: "smartctl" }, { check: "x", status: "bogus" }, "junk", { status: "failed" }] })).status,
202,
);
let d = (await detail()).diagnostics;
assert.deepEqual(d.map((n: { check: string }) => n.check), ["smart"]);
assert.match(d[0].fix, /smartmontools/);
assert.equal((await report({ cpu: { cores: 2 } })).status, 202);
assert.equal((await detail()).diagnostics.length, 1, "a report that doesn't say keeps what was said");
assert.equal((await report({ diagnostics: "garbage" })).status, 202);
assert.equal((await detail()).diagnostics.length, 1, "and so does one that isn't a list");
assert.equal((await report({ diagnostics: [] })).status, 202);
assert.deepEqual((await detail()).diagnostics, [], "an empty list says nothing was missed");
const row = (await (await fetch(`${base}/servers`)).json()).servers.find((x: { id: number }) => x.id === id);
assert.equal(row.agentDiagnostics, undefined, "never as raw stored text");
});
test("a service the server is told to ignore stops being an alert and a figure, still shows on the page marked, and can be un-ignored", async () => {
assert.equal((await report({ health: { failed_units: [{ unit: "nginx.service", description: "web" }, { unit: "gupdate.service", description: "Google Update" }] } })).status, 202);
invalidateAlertsCache();
const failing = async () => (await getAlerts(true)).alerts.find((a) => a.id.startsWith("services:failed-units") && a.id.includes(`:server:${id}`));
assert.match((await failing())!.message, /2 services have failed/);
assert.equal((await patch({ thresholds: { ignoreServices: ["gupdate*"] } })).status, 200);
assert.match((await failing())!.message, /1 service has failed — nginx\.service/, "the alert cache was dropped when the list changed");
assert.match(await buildMetrics(), new RegExp(`homelab_server_failed_units\\{server="elsa"[^}]*\\} 1`));
const units = (await detail()).server.hostHealth.failedUnits;
assert.deepEqual(units.map((u: { unit: string; ignored?: boolean }) => [u.unit, u.ignored ?? false]), [["nginx.service", false], ["gupdate.service", true]]);
assert.equal((await patch({ thresholds: { ignoreServices: ["gupdate*", "nginx"] } })).status, 200);
assert.equal(await failing(), undefined, "with every failed service ignored there is nothing to alert about");
assert.equal((await patch({ thresholds: null })).status, 200);
assert.match((await failing())!.message, /2 services have failed/);
});
test("a list that isn't one is refused with a message", async () => {
const res = await patch({ thresholds: { ignoreServices: "gupdate" } });
assert.equal(res.status, 400);
assert.match((await res.json()).message, /should be a list/);
});
});
+18 -2
View File
@@ -32,9 +32,9 @@ function run(snippet: string): unknown {
} }
describe("the Windows agent's health parts", { skip: !havePwsh }, () => { describe("the Windows agent's health parts", { skip: !havePwsh }, () => {
test("it says it is version 4, and stays plain ASCII", () => { test("it says it is version 5, and stays plain ASCII", () => {
const text = readFileSync(script, "utf8"); const text = readFileSync(script, "utf8");
assert.match(text, /^\$script:AgentVersion = 4\s*$/m); assert.match(text, /^\$script:AgentVersion = 5\s*$/m);
assert.ok(!/[^\x00-\x7f]/.test(text)); assert.ok(!/[^\x00-\x7f]/.test(text));
}); });
@@ -103,6 +103,22 @@ describe("the Windows agent's health parts", { skip: !havePwsh }, () => {
assert.deepEqual(out.map((p) => [p.name, p.health, p.capacity_percent]), [["Media", "ONLINE", 25], ["Backups", "DEGRADED", 90], ["Broken", "FAULTED", null]]); assert.deepEqual(out.map((p) => [p.name, p.health, p.capacity_percent]), [["Media", "ONLINE", 25], ["Backups", "DEGRADED", 90], ["Broken", "FAULTED", null]]);
}); });
test("what could not be read is noted, with a word for why and at most a line of what Windows said", () => {
const out = run(`
function Get-CimInstance { throw "WMI is unhappy\`r\`nsecond line" }
$WarningPreference = 'SilentlyContinue'
$null = Get-HostHealth @{ IgnoreServices = @() }
Add-Diag 'hardware' 'unavailable' 'not_found'
Add-Diag 'updates' 'failed' 'error' ('x' * 400)
ConvertTo-Json -InputObject @{ items = @($script:Diag) } -Depth 4 -Compress
`) as { items: { check: string; status: string; reason?: string; detail?: string }[] };
const byCheck = Object.fromEntries(out.items.map((i) => [i.check, i]));
assert.deepEqual([byCheck.failed_units.status, byCheck.failed_units.reason], ["failed", "error"]);
assert.equal(byCheck.failed_units.detail, "WMI is unhappy second line", "line breaks are flattened");
assert.deepEqual(byCheck.hardware, { check: "hardware", status: "unavailable", reason: "not_found" }, "nothing to say about it, no detail");
assert.equal(byCheck.updates.detail!.length, 160, "cut to a line");
});
test("a maker's placeholders aren't a model or a serial number", () => { test("a maker's placeholders aren't a model or a serial number", () => {
const out = run(` const out = run(`
$t = @('To Be Filled By O.E.M.', 'Default string', 'System Serial Number', 'System Product Name', '0000000', ' ', 'None', 'Unknown', 'ASUS', 'ROG STRIX B560-F', 'PF2ABCDE', '0A1B') $t = @('To Be Filled By O.E.M.', 'Default string', 'System Serial Number', 'System Product Name', '0000000', ' ', 'None', 'Unknown', 'ASUS', 'ROG STRIX B560-F', 'PF2ABCDE', '0A1B')
+14 -1
View File
@@ -632,7 +632,7 @@ export interface AgentLatest {
/** What a Linux agent says about the machine itself. Each part is there only if the agent could find it out. */ /** What a Linux agent says about the machine itself. Each part is there only if the agent could find it out. */
export interface HostHealth { export interface HostHealth {
failedUnits?: { unit: string; description: string }[]; failedUnits?: { unit: string; description: string; /** This server is told to ignore it: it doesn't count as failed. */ ignored?: boolean }[];
smart?: { device: string; model: string; healthy: boolean | null; temperatureC: number | null; wearPercent: number | null }[]; smart?: { device: string; model: string; healthy: boolean | null; temperatureC: number | null; wearPercent: number | null }[];
zfs?: { name: string; health: string; capacityPercent: number | null }[]; zfs?: { name: string; health: string; capacityPercent: number | null }[];
reportedAt: string; reportedAt: string;
@@ -655,6 +655,17 @@ export interface ServerThresholds {
diskForecastDays?: number; diskForecastDays?: number;
backupStaleHours?: number; backupStaleHours?: number;
mounts?: Record<string, { percent?: number; ignore?: boolean }>; mounts?: Record<string, { percent?: number; ignore?: boolean }>;
/** Services never to count as failed here: names or patterns like "gupdate*". */
ignoreServices?: string[];
}
/** One thing worth telling a person about a server's agent: what it couldn't collect (and what to do), or that it is behind. */
export interface AgentDiagnosticNote {
check: string;
level: "warning" | "info";
title: string;
fix: string | null;
detail: string | null;
} }
export interface UpdateRun { export interface UpdateRun {
@@ -746,6 +757,8 @@ export interface ServerDetail {
links: ServerLink[]; links: ServerLink[];
relations: { parent: ServerRelation | null; children: ServerRelation[] }; relations: { parent: ServerRelation | null; children: ServerRelation[] };
agentLatest: AgentLatest; agentLatest: AgentLatest;
/** What its agent couldn't collect, with what to do about it. */
diagnostics: AgentDiagnosticNote[];
/** The limits in Settings, which this server follows wherever it has none of its own. */ /** The limits in Settings, which this server follows wherever it has none of its own. */
thresholdDefaults: HealthCheckSettings; thresholdDefaults: HealthCheckSettings;
} }
+39
View File
@@ -0,0 +1,39 @@
import type { AgentDiagnosticNote } from "../api/client";
/** The fixes put commands between backticks: those are shown as code. */
const withCode = (text: string) => text.split("`").map((part, i) => (i % 2 === 1 ? <code key={i}>{part}</code> : part));
/**
* What the server's agent couldn't collect, and what to do about it — the answer to "why is there no disk health here?". A tip is how this
* machine is (a virtual machine with no serial number); a warning is something to look at. Nothing to say means no card.
*/
export default function AgentDiagnostics({ notes }: { notes: AgentDiagnosticNote[] }) {
if (notes.length === 0) return null;
const warnings = notes.filter((n) => n.level === "warning").length;
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">What the agent couldn&rsquo;t collect</h3>
<div className="card-actions">{warnings > 0 ? <span className="badge bg-yellow-lt text-yellow">{warnings} to look at</span> : <span className="badge bg-blue-lt text-blue">tips</span>}</div>
</div>
<div className="list-group list-group-flush">
{notes.map((n) => (
<div key={n.check + n.title} className="list-group-item">
<div className="d-flex align-items-start gap-2">
<span className={`badge ${n.level === "warning" ? "bg-yellow-lt text-yellow" : "bg-blue-lt text-blue"} flex-shrink-0 mt-1`}>{n.level === "warning" ? "look at" : "tip"}</span>
<div>
<div>{n.title}</div>
{n.detail && (
<div className="text-secondary small">
It said: <code>{n.detail}</code>
</div>
)}
{n.fix && <div className="text-secondary small">{withCode(n.fix)}</div>}
</div>
</div>
</div>
))}
</div>
</div>
);
}
+71 -9
View File
@@ -1,3 +1,4 @@
import { useState } from "react";
import type { HostHealth } from "../api/client"; import type { HostHealth } from "../api/client";
import TimeAgo from "./TimeAgo"; import TimeAgo from "./TimeAgo";
@@ -9,18 +10,41 @@ const poolFine = (health: string) => health.toUpperCase() === "ONLINE";
* whether each storage pool (ZFS, or Storage Spaces on Windows) is online. Each part appears only if the agent reported it (no smartctl or no ZFS on the host means no part), * whether each storage pool (ZFS, or Storage Spaces on Windows) is online. Each part appears only if the agent reported it (no smartctl or no ZFS on the host means no part),
* and the card is left out when it has nothing to say. * and the card is left out when it has nothing to say.
*/ */
export default function HostHealthCard({ health }: { health: HostHealth | null }) { export default function HostHealthCard({
if (!health) return null; health,
const { failedUnits, smart, zfs } = health; ignored = [],
onIgnoredChange,
}: {
health: HostHealth | null;
/** The services this server is told to ignore (names or patterns). */
ignored?: string[];
/** Given when the person can change that list; it is saved by whoever holds it. */
onIgnoredChange?: (list: string[]) => Promise<void> | void;
}) {
const [pattern, setPattern] = useState("");
const [busy, setBusy] = useState(false);
if (!health && ignored.length === 0) return null;
const { failedUnits, smart, zfs } = health ?? { failedUnits: undefined, smart: undefined, zfs: undefined, reportedAt: "" };
async function change(list: string[]) {
if (!onIgnoredChange) return;
setBusy(true);
try {
await onIgnoredChange(list);
setPattern("");
} finally {
setBusy(false);
}
}
const hasSmart = !!smart && smart.length > 0; const hasSmart = !!smart && smart.length > 0;
const hasZfs = !!zfs && zfs.length > 0; const hasZfs = !!zfs && zfs.length > 0;
if (failedUnits === undefined && !hasSmart && !hasZfs) return null; if (failedUnits === undefined && !hasSmart && !hasZfs && ignored.length === 0) return null;
return ( return (
<div className="card mb-3"> <div className="card mb-3">
<div className="card-header"> <div className="card-header">
<h3 className="card-title">Host health</h3> <h3 className="card-title">Host health</h3>
{health.reportedAt && ( {health?.reportedAt && (
<div className="card-actions text-secondary small"> <div className="card-actions text-secondary small">
reported <TimeAgo value={health.reportedAt} /> reported <TimeAgo value={health.reportedAt} />
</div> </div>
@@ -30,15 +54,21 @@ export default function HostHealthCard({ health }: { health: HostHealth | null }
{failedUnits !== undefined && ( {failedUnits !== undefined && (
<div className={hasSmart || hasZfs ? "mb-3" : ""}> <div className={hasSmart || hasZfs ? "mb-3" : ""}>
<div className="text-secondary small mb-1">Services</div> <div className="text-secondary small mb-1">Services</div>
{failedUnits.length === 0 ? ( {failedUnits.filter((u) => !u.ignored).length === 0 ? (
<span className="badge bg-green-lt text-green">none have failed</span> <span className="badge bg-green-lt text-green">none have failed</span>
) : ( ) : null}
{failedUnits.length > 0 && (
<ul className="list-unstyled mb-0"> <ul className="list-unstyled mb-0">
{failedUnits.map((u) => ( {failedUnits.map((u) => (
<li key={u.unit}> <li key={u.unit} className={u.ignored ? "text-secondary" : undefined}>
<span className="badge bg-red-lt text-red me-2">failed</span> {u.ignored ? <span className="badge bg-secondary-lt text-secondary me-2">ignored</span> : <span className="badge bg-red-lt text-red me-2">failed</span>}
<code>{u.unit}</code> <code>{u.unit}</code>
{u.description && <span className="text-secondary small ms-2">{u.description}</span>} {u.description && <span className="text-secondary small ms-2">{u.description}</span>}
{onIgnoredChange && !u.ignored && (
<button type="button" className="btn btn-link btn-sm p-0 ms-2 align-baseline" disabled={busy} title="Stop counting this service as failed on this server" onClick={() => void change([...ignored, u.unit])}>
Ignore
</button>
)}
</li> </li>
))} ))}
</ul> </ul>
@@ -46,6 +76,38 @@ export default function HostHealthCard({ health }: { health: HostHealth | null }
</div> </div>
)} )}
{(ignored.length > 0 || onIgnoredChange) && (
<div className={hasSmart || hasZfs ? "mb-3" : ""}>
<div className="text-secondary small mb-1">Services this server ignores</div>
{ignored.length === 0 && <div className="text-secondary small mb-1">None. A service that always fails here (an updater, say) can be ignored, so it stops raising an alert.</div>}
<div className="d-flex flex-wrap gap-2 mb-1">
{ignored.map((p) => (
<span key={p} className="badge bg-secondary-lt text-secondary d-inline-flex align-items-center gap-1">
<code>{p}</code>
{onIgnoredChange && (
<button type="button" className="btn-close btn-close-sm" style={{ fontSize: "0.5rem" }} aria-label={`Stop ignoring ${p}`} disabled={busy} onClick={() => void change(ignored.filter((x) => x !== p))} />
)}
</span>
))}
</div>
{onIgnoredChange && (
<form
className="d-flex gap-2"
onSubmit={(e) => {
e.preventDefault();
const p = pattern.trim();
if (p && !ignored.includes(p)) void change([...ignored, p]);
}}
>
<input className="form-control form-control-sm" style={{ maxWidth: "18rem" }} aria-label="A service name or pattern to ignore" placeholder="a name, or a pattern like gupdate*" maxLength={100} value={pattern} onChange={(e) => setPattern(e.target.value)} />
<button type="submit" className="btn btn-sm" disabled={busy || pattern.trim() === ""}>
Ignore
</button>
</form>
)}
</div>
)}
{hasSmart && ( {hasSmart && (
<div className={hasZfs ? "mb-3" : ""}> <div className={hasZfs ? "mb-3" : ""}>
<div className="text-secondary small mb-1">Disks (SMART)</div> <div className="text-secondary small mb-1">Disks (SMART)</div>
+4 -2
View File
@@ -13,7 +13,7 @@ const FIELDS: { key: Field; label: string; unit: string; min: number; max: numbe
{ key: "backupStaleHours", label: "Its backup is too old after", unit: "hours", min: 0, max: 2160, hint: "For the Proxmox guest this server is. 0 means it is never called stale.", onlyForGuests: true }, { key: "backupStaleHours", label: "Its backup is too old after", unit: "hours", min: 0, max: 2160, hint: "For the Proxmox guest this server is. 0 means it is never called stale.", onlyForGuests: true },
]; ];
const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts") || Object.keys(t.mounts ?? {}).length > 0); const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0);
/** /**
* This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every * This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every
@@ -53,6 +53,8 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC
if (rule.percent !== undefined || rule.ignore) rules[m] = rule; if (rule.percent !== undefined || rule.ignore) rules[m] = rule;
} }
if (Object.keys(rules).length > 0) out.mounts = rules; if (Object.keys(rules).length > 0) out.mounts = rules;
// The services to ignore are edited on the health card; saving limits here must not lose them.
if (own?.ignoreServices?.length) out.ignoreServices = own.ignoreServices;
return Object.keys(out).length > 0 ? out : null; return Object.keys(out).length > 0 ? out : null;
} }
@@ -195,7 +197,7 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC
{busy ? "Saving…" : "Save"} {busy ? "Saving…" : "Save"}
</button> </button>
{has(own) && ( {has(own) && (
<button type="button" className="btn" disabled={busy} onClick={() => void save(null)}> <button type="button" className="btn" disabled={busy} onClick={() => void save(own?.ignoreServices?.length ? { ignoreServices: own.ignoreServices } : null)}>
Follow Settings again Follow Settings again
</button> </button>
)} )}
+22 -1
View File
@@ -19,6 +19,9 @@ import TimeAgo from "../components/TimeAgo";
import UptimeRibbon from "../components/UptimeRibbon"; import UptimeRibbon from "../components/UptimeRibbon";
import RunsOn from "../components/RunsOn"; import RunsOn from "../components/RunsOn";
import HostHealthCard from "../components/HostHealthCard"; import HostHealthCard from "../components/HostHealthCard";
import AgentDiagnostics from "../components/AgentDiagnostics";
import { toast } from "../utils/toast";
import { readableError } from "../utils/errors";
import ServerThresholds from "../components/ServerThresholds"; import ServerThresholds from "../components/ServerThresholds";
import RunPlaybook from "../components/RunPlaybook"; import RunPlaybook from "../components/RunPlaybook";
import RunbookPanel from "../components/RunbookPanel"; import RunbookPanel from "../components/RunbookPanel";
@@ -809,7 +812,25 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
</div> </div>
</div> </div>
<HostHealthCard health={server.hostHealth} /> <HostHealthCard
health={server.hostHealth}
ignored={server.thresholds?.ignoreServices ?? []}
onIgnoredChange={
isAdmin
? async (list) => {
try {
// The list is part of the server's own limits; the rest of them are carried over as they are.
await api.servers.update(server.id, { thresholds: { ...(server.thresholds ?? {}), ignoreServices: list.length > 0 ? list : undefined } });
toast.success(list.length > (server.thresholds?.ignoreServices?.length ?? 0) ? "Ignored. It no longer counts as failed on this server." : "Saved.");
void loadDetail();
} catch (err) {
toast.error(readableError(err));
}
}
: undefined
}
/>
<AgentDiagnostics notes={detail.diagnostics ?? []} />
{hardware.source === "agent" && <ServerTrends serverId={serverId} />} {hardware.source === "agent" && <ServerTrends serverId={serverId} />}
<ServerThresholds key={JSON.stringify(server.thresholds)} server={server} defaults={detail.thresholdDefaults} disks={(hardware.disks ?? []).map((d) => d.mount)} canEdit={isAdmin} onChanged={() => void loadDetail()} /> <ServerThresholds key={JSON.stringify(server.thresholds)} server={server} defaults={detail.thresholdDefaults} disks={(hardware.disks ?? []).map((d) => d.mount)} canEdit={isAdmin} onChanged={() => void loadDetail()} />
{(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />} {(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />}