- New docs/ folder: SSO, API, Prometheus and Grafana, webhooks, heartbeats and calendar, how monitoring works, and an index; the old root API.md is split across them - Read API: alerts carry snoozedUntil - DATABASE.md: table and migration counts, indexes, relations, integration config, settings and internal flags, delete rules, retention - ROLES.md: the sidebar table rebuilt from the menu - NOTIFICATIONS.md, INTEGRATIONS.md, BACKUP.md, AGENT_UPDATE.md: what had drifted - README: links to the guides; accounts cannot be deleted in the app Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
311 lines
22 KiB
Markdown
311 lines
22 KiB
Markdown
# Updating the agents
|
|
|
|
The agent on each server — [`agent/linux`](agent/linux) or [`agent/windows`](agent/windows) — is a copy of a
|
|
script that Homelab Manager serves, taken at the time it was installed. It does **not** update itself, and
|
|
updating Homelab Manager does not touch the agents already out on your servers. When a new version of
|
|
Homelab Manager teaches the agent something new, each server needs its agent installed again to learn it.
|
|
|
|
This page is about how to do that, safely, and how to tell it worked.
|
|
|
|
## Do I need to?
|
|
|
|
| You updated Homelab Manager to… | Agents need updating? |
|
|
|---|---|
|
|
| a version that only changed the app | No. Older agents keep working: a report is read the same way as before. |
|
|
| the version that added **pending updates** (the Updates page) | Yes, to see updates. Until then the server's row says *not reported*. Everything else about it carries on as before. |
|
|
|
|
Each agent script carries a version number and sends it in every report; Homelab Manager compares it with the
|
|
scripts it serves itself, so it can tell you which agents are behind. A change to an agent is only given a new
|
|
number when it is worth installing the agent again for.
|
|
|
|
| Agent version | What it added |
|
|
|---|---|
|
|
| 1 | Everything before the number was reported — so an agent that says no version is version 1. |
|
|
| 2 | Reports pending updates, and its own version. |
|
|
| 3 (Linux) | Asks the package manager outside the agent's systemd sandbox, so `dnf` and `zypper` hosts can answer (below). Install again to get it: the unit file changes too. |
|
|
| 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). |
|
|
| 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. |
|
|
| 4 (Windows) | Also reports failed services, disk health and Storage Spaces pools (below), and what the machine is (maker, model, serial number). |
|
|
| 6 (Linux), 5 (Windows) | Also says **what it couldn't collect and why** (no `smartctl`, no systemd, a package manager that didn't answer, Windows Update that timed out …), shown on the server's page with what to do about it (below). |
|
|
| 7 (Linux), 6 (Windows) | Also says **which operating system it runs** (distribution and release, or Windows product and build), so the server's page can say when that release stops being supported (below). |
|
|
| 5 (Linux) | Also says what the machine is — maker, model and serial number, read from the firmware — so the inventory can fill itself in (below). |
|
|
|
|
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
|
|
an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of
|
|
old and new agents is fine.
|
|
|
|
**How to tell which agents are old.** Homelab Manager says so: a server whose agent is behind shows an
|
|
*agent behind* badge on the **Servers** page, *update available* beside **Agent** on its own page, and *behind* in the
|
|
**Agent** column of the **Updates** page, which also counts them under *Agents to update*. (An agent that was
|
|
updated before version numbers existed reads as "v1 or older" until it has been updated once more.) On the
|
|
machine itself:
|
|
|
|
```bash
|
|
# Linux — prints 1 or more if the script knows about updates, 0 if it doesn't
|
|
grep -c collect_updates /usr/local/bin/homelab-manager-agent.sh
|
|
```
|
|
|
|
```powershell
|
|
# Windows (elevated) — True if it does
|
|
Select-String -Path 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -Pattern 'Get-PendingUpdates' -Quiet
|
|
```
|
|
|
|
## What updating does
|
|
|
|
"Updating" is simply running the install command again. The installer is built to be run over an existing
|
|
install:
|
|
|
|
- it downloads the current agent script from your Homelab Manager and replaces the old one;
|
|
- it writes the URL and token again (the **same token keeps the same server**: its tasks, history, tags,
|
|
uptime and everything else stay exactly where they are — nothing is registered twice);
|
|
- on Linux it rewrites the systemd service and timer and reloads systemd; on Windows it replaces the
|
|
*Homelab Manager Agent* scheduled task;
|
|
- Windows sends a first report straight away so you see at once whether it worked.
|
|
|
|
It does not uninstall anything, does not touch the machine's own cron jobs, timers or tasks, and does not
|
|
install any updates itself.
|
|
|
|
Two settings go back to their defaults, so give them again if you had changed them:
|
|
|
|
- **Reporting interval** — back to 15 minutes unless you set `INTERVAL_MINUTES` on the command.
|
|
- **Windows `includeMicrosoftTasks`** — the installer rewrites `agent.json` with just the URL, token and
|
|
certificate setting. Put `INCLUDE_MICROSOFT_TASKS=true` in the environment afterwards, or add
|
|
`"includeMicrosoftTasks": true` to `agent.json` again.
|
|
|
|
## Getting the command
|
|
|
|
You need the server's URL and its token. There are two ways to have both.
|
|
|
|
### A. Reuse the token already on the machine (nothing to look up)
|
|
|
|
The token is already stored on each server, readable only by root / Administrators, so the machine can
|
|
update itself with the credentials it has. Run as root:
|
|
|
|
```bash
|
|
# Linux
|
|
sudo sh -c '. /etc/homelab-manager-agent.env; curl -fsSL "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash'
|
|
```
|
|
|
|
If Homelab Manager uses a **self-signed certificate**, the first download needs `-k` too (the agent's own
|
|
`API_INSECURE` is carried over from the file):
|
|
|
|
```bash
|
|
sudo sh -c '. /etc/homelab-manager-agent.env; curl -fsSLk "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash'
|
|
```
|
|
|
|
In an **elevated** Windows PowerShell:
|
|
|
|
```powershell
|
|
$c = Get-Content 'C:\ProgramData\HomelabManager\agent.json' | ConvertFrom-Json
|
|
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
|
$env:API_URL = $c.apiUrl; $env:API_TOKEN = $c.apiToken
|
|
if ($c.insecure) { $env:API_INSECURE = 'true'; [Net.ServicePointManager]::ServerCertificateValidationCallback = { $true } }
|
|
iex ((New-Object Net.WebClient).DownloadString("$env:API_URL/agent/windows/install.ps1"))
|
|
```
|
|
|
|
(That form is for Windows PowerShell 5.1, the one built into Windows. See the
|
|
[Windows agent README](agent/windows/README.md) for PowerShell 7.)
|
|
|
|
### B. Rotate the token and use the command the page shows
|
|
|
|
In Homelab Manager: **Servers → Manage servers → Rotate token** on that server. The page shows a fresh
|
|
install command for it (with the self-signed-certificate box if you need it). Run it on the machine.
|
|
|
|
Use this when the token is lost or you want it replaced anyway. Know that rotating takes effect **at once**:
|
|
the old token stops working, so that server's agent can't report until the new command has been run on it —
|
|
and the server will look as if it has stopped reporting (and alert) in the meantime. Do one server at a
|
|
time, and put it under [maintenance](README.md) first if the gap could be more than a few minutes.
|
|
|
|
## Updating many servers
|
|
|
|
For more than a handful, use the tool you already have. The commands in A are plain shell and PowerShell, so
|
|
Ansible or Semaphore can run them. For example, a playbook for the Linux hosts (an example — adjust the
|
|
group name, and add `-k` to `curl` for a self-signed certificate):
|
|
|
|
```yaml
|
|
- hosts: linux_servers
|
|
become: true
|
|
tasks:
|
|
- name: Update the Homelab Manager agent (same token as before)
|
|
ansible.builtin.shell: |
|
|
set -e
|
|
. /etc/homelab-manager-agent.env
|
|
curl -fsSL "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash
|
|
args:
|
|
executable: /bin/bash
|
|
changed_when: true
|
|
```
|
|
|
|
Homelab Manager can start such a template for you, limited to one host, from the server's own page (*Run a
|
|
playbook*) — a good way to try it on one machine first, then run it on the rest from Semaphore.
|
|
|
|
## Checking it worked
|
|
|
|
1. **The Updates page.** The server's row changes from *not reported* to its figures, and its **Agent** to the
|
|
current version, within one reporting interval (15 minutes by default; on Windows the installer reports straight away, and on Linux you can
|
|
start one now — below).
|
|
2. **Last seen** on the server's page moves to *just now*.
|
|
|
|
On a Linux server:
|
|
|
|
```bash
|
|
systemctl list-timers homelab-manager-agent.timer # when it next runs
|
|
sudo systemctl start homelab-manager-agent.service # report now
|
|
sudo journalctl -u homelab-manager-agent -n 30 # what it said (look for "Note:" lines)
|
|
sudo /usr/local/bin/homelab-manager-agent.sh --dry-run | jq .system.updates # what it would send; sends nothing
|
|
```
|
|
|
|
On a Windows server (elevated):
|
|
|
|
```powershell
|
|
Start-ScheduledTask -TaskName 'Homelab Manager Agent' # report now
|
|
Get-ScheduledTaskInfo -TaskName 'Homelab Manager Agent' # LastRunTime, LastTaskResult (0 = fine)
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -DryRun # prints what it would send
|
|
```
|
|
|
|
## Letting agents update themselves
|
|
|
|
**Servers → "Update agents automatically"** (administrators; off by default). With it on, an agent that is behind the script
|
|
this Homelab Manager serves downloads the newer one and replaces itself at its next run — no install command to run again.
|
|
|
|
How it stays safe:
|
|
|
|
- The agent asks first (`GET /api/agent/report/update?version=N`, with its own token). The answer is yes only when the switch
|
|
is on and the agent is behind, and it carries the **SHA-256 of the script**.
|
|
- The agent downloads the script and installs it only if the checksum matches, it is newer than the running one, it says
|
|
the version that was promised, and it parses (`bash -n` on Linux, PowerShell's own parser on Windows). Anything else is thrown
|
|
away and the agent carries on as it was; a note goes to the journal (Linux) or the task output (Windows).
|
|
- The new file is written beside the old one and moved over it in one step, so there is never half a script.
|
|
- On Linux this happens in the same step that is outside the sandbox (`ExecStartPre=-+…`, below), because the sandbox keeps
|
|
`/usr` read-only for the report itself. That step runs first, so the report that follows is already the new version.
|
|
On Windows the new script runs from the next report.
|
|
- Each update is written down as a line in the server's history ("Agent updated to version 4"), and switching the setting
|
|
is in the audit log.
|
|
|
|
What it can't do:
|
|
|
|
- **Update an agent that doesn't have the update code yet.** An agent has to contain the self-update itself: Linux version 4 or
|
|
later, Windows version 3 or later. Getting there is one install command run on each host (the last one you need to run by hand);
|
|
after that, new versions arrive by themselves. A Linux host whose unit was written by version 3 keeps working: the version 4
|
|
script still understands the unit's older step name.
|
|
- **Replace the systemd unit.** Self-update replaces the script only. A release that also needs a changed unit says so in the
|
|
version table above ("install again").
|
|
- **Update an agent that isn't reporting.**
|
|
- **Choose per server.** It is one switch for all agents.
|
|
|
|
## What the Linux agent reports about the machine (version 4)
|
|
|
|
| What | Where it comes from | Notes |
|
|
|---|---|---|
|
|
| **Failed services** | `systemctl list-units --state=failed` | Read on every report (it works inside the sandbox). Shown on the server's page as *Host health* and raised as an alert of kind *Failing services*. |
|
|
| **Disk health** | `smartctl -j -H -A` for each disk it lists | Needs `smartmontools` installed (the section is simply left out without it) and the devices, so it is read in the step outside the sandbox. Taken **about once an hour** and with `-n standby`, so a disk that is asleep isn't woken: what was last known of it is kept (in `/var/lib/homelab-manager-agent`). A disk that says its SMART check **fails** is a critical alert; one that has used 90% or more of its rated life is a warning. Virtual disks have no SMART and are skipped. |
|
|
| **ZFS pools** | `zpool list` | Needs the `zpool` command. A pool in any state but ONLINE is a critical alert. |
|
|
|
|
A report that leaves one of these out leaves what was last said alone (an agent that skips a part never turns an old problem into
|
|
"all well"); an empty list is an answer ("none failed") and clears the old one.
|
|
|
|
## What the Windows agent reports about the machine (version 4)
|
|
|
|
The same three parts, from what Windows itself knows (the agent runs as SYSTEM, so nothing needs installing):
|
|
|
|
| What | Where it comes from | Notes |
|
|
|---|---|---|
|
|
| **Failed services** | `Win32_Service` | A service that starts by itself, isn't running, and stopped with an error. Services that finished normally (exit code 0), that start on a trigger (1077) or that are starting up are not failures. Some services always do this (updaters, for one): list them under `ignoreServices` in `agent.json` (or `IGNORE_SERVICES` in the environment), as names or patterns like `gupdate*` — or, easier and for either system, on the server's page (below). |
|
|
| **Disk health** | `Get-PhysicalDisk` and its reliability counters | Windows' own verdict: *Healthy* passes, *Warning* and *Unhealthy* fail, anything else says nothing. Temperature and wear come from the counters where the drive gives sane ones. Virtual disks are left out. |
|
|
| **Storage pools** | `Get-StoragePool` (Storage Spaces) | *Healthy*, *Warning* and *Unhealthy* are reported as ONLINE, DEGRADED and FAULTED, with how full the pool is, and show beside ZFS pools. A machine with no pools reports an empty list. |
|
|
|
|
## Services to ignore, from the server's page
|
|
|
|
A service that always shows as failed on one machine (an updater, a unit that is meant to be stopped) can be ignored there: on the server's **Host health**
|
|
card an administrator presses **Ignore** beside it, or types a name or a pattern with `*` (`gupdate*`, `snap.*`; case doesn't matter and `.service` is
|
|
optional). It stops counting as failed — no alert, not in `/metrics` — the moment it is saved, with no agent to update, and it still shows on the page,
|
|
marked *ignored*, with a button to stop ignoring it. The list belongs to that server (it is kept with its own alert limits), and works the same for Linux and
|
|
Windows; the Windows agent's own `ignoreServices` still works as well.
|
|
|
|
## What the agent couldn't collect (Linux version 6, Windows version 5)
|
|
|
|
Each agent notes, in every report, the checks it couldn't make: nothing there to read it with (*unavailable*) or it tried and couldn't (*failed*), why in a
|
|
word, and for a failure at most a line of what the tool said. The server's page turns that into a card, **What the agent couldn't collect**, with words and a
|
|
fix for each — "Disk health isn't reported: smartctl isn't installed → install smartmontools" — so the question *why is there no disk health here?* has an
|
|
answer on the page. A tip is simply how the machine is (a virtual machine with no serial number); *look at* is something to fix. A machine with no ZFS isn't
|
|
told it has no ZFS. The card also says when the agent is behind. The wording lives in the server, so it improves without installing anything again.
|
|
|
|
On Linux, the step outside the sandbox (`--prepare`) leaves its notes with the other answers, and the report carries them; if that step didn't run at all
|
|
(an older systemd unit, or a copy run by hand) that is itself noted, because disk health and ZFS are only read there. `--check-updates` prints the same notes.
|
|
|
|
## What the machine is (Linux version 5, Windows version 4)
|
|
|
|
The agents say who made the machine, its model and its serial number (Linux reads `/sys/class/dmi/id`, where the serial is only
|
|
readable by root; Windows asks the computer system, the BIOS, and the motherboard when those say nothing — a home-built PC). What
|
|
a maker writes when it has nothing to say ("To Be Filled By O.E.M.", "Default string", all zeros) is left out, and many virtual
|
|
machines and single-board computers say nothing at all. The figures are kept on the server and used by the **Inventory**: a
|
|
linked entry gets its blank maker, model and serial number filled in, and *Refresh from agent* on the Inventory page replaces
|
|
what is written there with what the agent says, when you choose to.
|
|
|
|
To see exactly what a host finds, as root: `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` — it prints the
|
|
update figures, the failed services, the SMART readings and the ZFS pools, says whether it would update itself, and changes nothing.
|
|
|
|
## What the machine runs (Linux version 7, Windows version 6)
|
|
|
|
Each agent says which operating system it runs, in the report's `system.os`: `{ id, name, version }`.
|
|
|
|
- **Linux** reads `/etc/os-release` — `ID` (lower case, e.g. `ubuntu`, `debian`, `rocky`), `VERSION_ID` and `PRETTY_NAME`. On a **Proxmox VE** host (where `pveversion` exists) it says `proxmox-ve` and the
|
|
release instead, since the Debian underneath is not what needs watching.
|
|
- **Windows** says `windows`, the product name without "Microsoft " (e.g. "Windows Server 2022 Standard") and the build number — which is how client and server releases are told apart.
|
|
|
|
The server keeps it with the machine and compares the release with a built-in table of end-of-support dates (Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE, Windows
|
|
client and server). A server's page shows it in its *System* row with *supported*, *ends in N days* or *support ended*; one past its end — or inside the warning window under Settings → Notifications (*Flag an
|
|
operating system whose support ends within*, 90 days) — is on the Alerts page, and `operatingSystem` is in the read API. A release the table does not know is shown without a badge, never guessed at. An agent
|
|
older than this version reports no OS and the server shows nothing for it.
|
|
|
|
`sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` prints what it found.
|
|
|
|
## How the update check works — and why a server may stay "not reported"
|
|
|
|
The agent only *asks*; it never installs, downloads or refreshes anything.
|
|
|
|
| System | What it asks | Notes |
|
|
|---|---|---|
|
|
| Debian, Ubuntu, Raspberry Pi OS (`apt`) | a simulated upgrade (`apt-get -s upgrade`) | Uses the package lists already on the machine. If nothing refreshes them (`apt update`, or the distribution's own daily timer), the count is only as fresh as they are. Security updates are those from a `-security` archive. |
|
|
| Fedora, RHEL and relatives (`dnf`) | `dnf check-update --cacheonly` | Reads dnf's cache only; if the cache is empty the server stays *not reported* (`sudo dnf makecache` fills it, and dnf's own timer keeps it fresh). A restart is flagged by `needs-restarting -r` when that tool is installed. |
|
|
| openSUSE, SLES (`zypper`) | `zypper list-updates` without refreshing | Security figure not available (shown as unknown). |
|
|
| Arch and relatives (`pacman`) | `pacman -Qu` | What the local copy of the sync databases says could be upgraded (what `checkupdates` lists); packages held back are left out; no security figure. A restart is flagged when the running kernel is no longer the installed one. |
|
|
| Windows | Windows Update, through its own API | Searching can take a minute or two, so it is done at most **every six hours** and kept in `%ProgramData%\HomelabManager\updates.json`. It runs in a separate process with a 150-second limit (the scheduled task itself is stopped after five minutes). If the search fails or times out it isn't tried again for an hour, and that report leaves updates out. |
|
|
|
|
**Not reported** means the agent didn't give a figure. The usual reasons:
|
|
|
|
1. **It is an older agent.** Update it (above). This is by far the most common.
|
|
2. **Another package manager** — Alpine (`apk`) and others aren't read yet. (The Linux installer needs systemd, which Alpine doesn't use.)
|
|
3. **The check couldn't run.** On Linux the agent writes a line to the journal when its package manager
|
|
couldn't be read — `journalctl -u homelab-manager-agent -n 30`. The agent itself runs inside a tight systemd
|
|
sandbox (read-only system, no hardware, minimal privileges, see `install.sh`), and `dnf` and `zypper` write to
|
|
`/var` and `/run` even when only asked a question. So from agent version 3 the unit asks the package manager in
|
|
one separate step *before* the report — `ExecStartPre=-+…homelab-manager-agent.sh --prepare-updates`, where
|
|
`+` means outside the sandbox and `-` means a failure doesn't stop the report — and hands the answer to the
|
|
report through a private directory (`/run/homelab-manager-agent/updates.json`, removed when the run ends).
|
|
That step only runs the package manager's read-only questions. The report itself keeps the full sandbox.
|
|
To see what it finds on a host, as root:
|
|
|
|
```bash
|
|
sudo /usr/local/bin/homelab-manager-agent.sh --check-updates
|
|
```
|
|
|
|
It prints what the package manager said (errors included) and the figures that would be reported. To try the
|
|
whole path the way the timer does it: `sudo systemctl start homelab-manager-agent` and then
|
|
`journalctl -u homelab-manager-agent -n 30`.
|
|
4. **The package manager was busy** (another update running). It is tried again at the next report.
|
|
|
|
A server that says nothing keeps what it last said — an agent that skips a report never turns an old figure
|
|
into "up to date".
|
|
|
|
## If something goes wrong
|
|
|
|
- **The installer fails on download** — the machine can't reach Homelab Manager's `/agent/...` address, or the
|
|
certificate isn't trusted (use the self-signed forms above). Nothing has changed on the machine yet.
|
|
- **The installer ran but the server doesn't report** — check the token (a rotated token makes the old one
|
|
stop at once) and `API_URL`: `journalctl -u homelab-manager-agent` on Linux, the scheduled task's last
|
|
result on Windows.
|
|
- **To go back** — there is no older version to fetch; the agent a Homelab Manager serves is the one that
|
|
came with it. To remove an agent, use the uninstall command shown on the Servers page; the server's entry and
|
|
history stay until you delete it there.
|