Host health from the Linux agent, agent self-update, and disk forecast in /metrics
- Linux agent v4: failed systemd services, SMART disk health, ZFS pool state, pacman updates; SMART/ZFS and self-update run in the ExecStartPre=-+ step outside the sandbox - Server: hosts' health stored (servers.host_health, migration 0029), alerts under a new "Failing services" kind plus disk/ZFS alerts, Host health card, Prometheus series - Agent self-update (Linux v4, Windows v3): admin switch on the Servers page, checksum-verified download, atomic replace; GET /api/agent/report/update - homelab_server_disk_days_to_full in /metrics - Tests (server, real PowerShell against a fake server) and docs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
540ab704b7
commit
5dc6cea69c
38 files changed
+3884
-73
No files matched your search
+49
-2
@@ -22,7 +22,9 @@ number when it is worth installing the agent again for.
|
||||
|---|---|
|
||||
| 1 | Everything before the number was reported — so an agent that says no version is version 1. |
|
||||
| 2 | Reports pending updates, and its own version. |
|
||||
| 3 (Linux only) | Asks the package manager outside the agent's systemd sandbox, so `dnf` and `zypper` hosts can answer (below). The Windows agent stays at 2. Install again to get it: the unit file changes too. |
|
||||
| 3 (Linux) | Asks the package manager outside the agent's systemd sandbox, so `dnf` and `zypper` hosts can answer (below). Install again to get it: the unit file changes too. |
|
||||
| 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). |
|
||||
| 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. |
|
||||
|
||||
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
|
||||
an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of
|
||||
@@ -157,6 +159,50 @@ Get-ScheduledTaskInfo -TaskName 'Homelab Manager Agent' # La
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -DryRun # prints what it would send
|
||||
```
|
||||
|
||||
## Letting agents update themselves
|
||||
|
||||
**Servers → "Update agents automatically"** (administrators; off by default). With it on, an agent that is behind the script
|
||||
this Homelab Manager serves downloads the newer one and replaces itself at its next run — no install command to run again.
|
||||
|
||||
How it stays safe:
|
||||
|
||||
- The agent asks first (`GET /api/agent/report/update?version=N`, with its own token). The answer is yes only when the switch
|
||||
is on and the agent is behind, and it carries the **SHA-256 of the script**.
|
||||
- The agent downloads the script and installs it only if the checksum matches, it is newer than the running one, it says
|
||||
the version that was promised, and it parses (`bash -n` on Linux, PowerShell's own parser on Windows). Anything else is thrown
|
||||
away and the agent carries on as it was; a note goes to the journal (Linux) or the task output (Windows).
|
||||
- The new file is written beside the old one and moved over it in one step, so there is never half a script.
|
||||
- On Linux this happens in the same step that is outside the sandbox (`ExecStartPre=-+…`, below), because the sandbox keeps
|
||||
`/usr` read-only for the report itself. That step runs first, so the report that follows is already the new version.
|
||||
On Windows the new script runs from the next report.
|
||||
- Each update is written down as a line in the server's history ("Agent updated to version 4"), and switching the setting
|
||||
is in the audit log.
|
||||
|
||||
What it can't do:
|
||||
|
||||
- **Update an agent that doesn't have the update code yet.** An agent has to contain the self-update itself: Linux version 4 or
|
||||
later, Windows version 3 or later. Getting there is one install command run on each host (the last one you need to run by hand);
|
||||
after that, new versions arrive by themselves. A Linux host whose unit was written by version 3 keeps working: the version 4
|
||||
script still understands the unit's older step name.
|
||||
- **Replace the systemd unit.** Self-update replaces the script only. A release that also needs a changed unit says so in the
|
||||
version table above ("install again").
|
||||
- **Update an agent that isn't reporting.**
|
||||
- **Choose per server.** It is one switch for all agents.
|
||||
|
||||
## What the Linux agent reports about the machine (version 4)
|
||||
|
||||
| What | Where it comes from | Notes |
|
||||
|---|---|---|
|
||||
| **Failed services** | `systemctl list-units --state=failed` | Read on every report (it works inside the sandbox). Shown on the server's page as *Host health* and raised as an alert of kind *Failing services*. |
|
||||
| **Disk health** | `smartctl -j -H -A` for each disk it lists | Needs `smartmontools` installed (the section is simply left out without it) and the devices, so it is read in the step outside the sandbox. Taken **about once an hour** and with `-n standby`, so a disk that is asleep isn't woken: what was last known of it is kept (in `/var/lib/homelab-manager-agent`). A disk that says its SMART check **fails** is a critical alert; one that has used 90% or more of its rated life is a warning. Virtual disks have no SMART and are skipped. |
|
||||
| **ZFS pools** | `zpool list` | Needs the `zpool` command. A pool in any state but ONLINE is a critical alert. |
|
||||
|
||||
A report that leaves one of these out leaves what was last said alone (an agent that skips a part never turns an old problem into
|
||||
"all well"); an empty list is an answer ("none failed") and clears the old one. Windows doesn't report these.
|
||||
|
||||
To see exactly what a host finds, as root: `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` — it prints the
|
||||
update figures, the failed services, the SMART readings and the ZFS pools, says whether it would update itself, and changes nothing.
|
||||
|
||||
## How the update check works — and why a server may stay "not reported"
|
||||
|
||||
The agent only *asks*; it never installs, downloads or refreshes anything.
|
||||
@@ -166,12 +212,13 @@ The agent only *asks*; it never installs, downloads or refreshes anything.
|
||||
| Debian, Ubuntu, Raspberry Pi OS (`apt`) | a simulated upgrade (`apt-get -s upgrade`) | Uses the package lists already on the machine. If nothing refreshes them (`apt update`, or the distribution's own daily timer), the count is only as fresh as they are. Security updates are those from a `-security` archive. |
|
||||
| Fedora, RHEL and relatives (`dnf`) | `dnf check-update --cacheonly` | Reads dnf's cache only; if the cache is empty the server stays *not reported* (`sudo dnf makecache` fills it, and dnf's own timer keeps it fresh). A restart is flagged by `needs-restarting -r` when that tool is installed. |
|
||||
| openSUSE, SLES (`zypper`) | `zypper list-updates` without refreshing | Security figure not available (shown as unknown). |
|
||||
| Arch and relatives (`pacman`) | `pacman -Qu` | What the local copy of the sync databases says could be upgraded (what `checkupdates` lists); packages held back are left out; no security figure. A restart is flagged when the running kernel is no longer the installed one. |
|
||||
| Windows | Windows Update, through its own API | Searching can take a minute or two, so it is done at most **every six hours** and kept in `%ProgramData%\HomelabManager\updates.json`. It runs in a separate process with a 150-second limit (the scheduled task itself is stopped after five minutes). If the search fails or times out it isn't tried again for an hour, and that report leaves updates out. |
|
||||
|
||||
**Not reported** means the agent didn't give a figure. The usual reasons:
|
||||
|
||||
1. **It is an older agent.** Update it (above). This is by far the most common.
|
||||
2. **Another package manager** — Alpine (`apk`), Arch (`pacman`) and others aren't read yet.
|
||||
2. **Another package manager** — Alpine (`apk`) and others aren't read yet. (The Linux installer needs systemd, which Alpine doesn't use.)
|
||||
3. **The check couldn't run.** On Linux the agent writes a line to the journal when its package manager
|
||||
couldn't be read — `journalctl -u homelab-manager-agent -n 30`. The agent itself runs inside a tight systemd
|
||||
sandbox (read-only system, no hardware, minimal privileges, see `install.sh`), and `dnf` and `zypper` write to
|
||||
|
||||
Reference in new issue
Block a user