Document updating the agents, and harden the Windows update check
- AGENT_UPDATE.md: when an agent needs updating, how (reuse the stored token or rotate it), how to do many servers, how to check it worked, and why a server can stay 'not reported'. - Windows agent: the Windows Update search runs in its own process with a 150 s limit (the scheduled task is stopped after five minutes), a failed search is not retried for an hour, and the cache is read the same in PowerShell 7 and Windows PowerShell 5.1. - Linux agent: says in the journal when the package manager could not be read. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
171511c6db
commit
3da5c639b1
5 files changed
+225
-18
No files matched your search
+181
@@ -0,0 +1,181 @@
|
|||||||
|
# Updating the agents
|
||||||
|
|
||||||
|
The agent on each server — [`agent/linux`](agent/linux) or [`agent/windows`](agent/windows) — is a copy of a
|
||||||
|
script that Homelab Manager serves, taken at the time it was installed. It does **not** update itself, and
|
||||||
|
updating Homelab Manager does not touch the agents already out on your servers. When a new version of
|
||||||
|
Homelab Manager teaches the agent something new, each server needs its agent installed again to learn it.
|
||||||
|
|
||||||
|
This page is about how to do that, safely, and how to tell it worked.
|
||||||
|
|
||||||
|
## Do I need to?
|
||||||
|
|
||||||
|
| You updated Homelab Manager to… | Agents need updating? |
|
||||||
|
|---|---|
|
||||||
|
| a version that only changed the app | No. Older agents keep working: a report is read the same way as before. |
|
||||||
|
| the version that added **pending updates** (the Updates page) | Yes, to see updates. Until then the server's row says *not reported*. Everything else about it carries on as before. |
|
||||||
|
|
||||||
|
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
|
||||||
|
an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of
|
||||||
|
old and new agents is fine.
|
||||||
|
|
||||||
|
**How to tell which agents are old.** The agent doesn't report a version number. The visible sign is the
|
||||||
|
**Updates** page showing *not reported* for a server that has an agent. On the machine itself:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux — prints 1 or more if the script knows about updates, 0 if it doesn't
|
||||||
|
grep -c collect_updates /usr/local/bin/homelab-manager-agent.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# Windows (elevated) — True if it does
|
||||||
|
Select-String -Path 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -Pattern 'Get-PendingUpdates' -Quiet
|
||||||
|
```
|
||||||
|
|
||||||
|
## What updating does
|
||||||
|
|
||||||
|
"Updating" is simply running the install command again. The installer is built to be run over an existing
|
||||||
|
install:
|
||||||
|
|
||||||
|
- it downloads the current agent script from your Homelab Manager and replaces the old one;
|
||||||
|
- it writes the URL and token again (the **same token keeps the same server**: its tasks, history, tags,
|
||||||
|
uptime and everything else stay exactly where they are — nothing is registered twice);
|
||||||
|
- on Linux it rewrites the systemd service and timer and reloads systemd; on Windows it replaces the
|
||||||
|
*Homelab Manager Agent* scheduled task;
|
||||||
|
- Windows sends a first report straight away so you see at once whether it worked.
|
||||||
|
|
||||||
|
It does not uninstall anything, does not touch the machine's own cron jobs, timers or tasks, and does not
|
||||||
|
install any updates itself.
|
||||||
|
|
||||||
|
Two settings go back to their defaults, so give them again if you had changed them:
|
||||||
|
|
||||||
|
- **Reporting interval** — back to 15 minutes unless you set `INTERVAL_MINUTES` on the command.
|
||||||
|
- **Windows `includeMicrosoftTasks`** — the installer rewrites `agent.json` with just the URL, token and
|
||||||
|
certificate setting. Put `INCLUDE_MICROSOFT_TASKS=true` in the environment afterwards, or add
|
||||||
|
`"includeMicrosoftTasks": true` to `agent.json` again.
|
||||||
|
|
||||||
|
## Getting the command
|
||||||
|
|
||||||
|
You need the server's URL and its token. There are two ways to have both.
|
||||||
|
|
||||||
|
### A. Reuse the token already on the machine (nothing to look up)
|
||||||
|
|
||||||
|
The token is already stored on each server, readable only by root / Administrators, so the machine can
|
||||||
|
update itself with the credentials it has. Run as root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux
|
||||||
|
sudo sh -c '. /etc/homelab-manager-agent.env; curl -fsSL "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash'
|
||||||
|
```
|
||||||
|
|
||||||
|
If Homelab Manager uses a **self-signed certificate**, the first download needs `-k` too (the agent's own
|
||||||
|
`API_INSECURE` is carried over from the file):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo sh -c '. /etc/homelab-manager-agent.env; curl -fsSLk "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash'
|
||||||
|
```
|
||||||
|
|
||||||
|
In an **elevated** Windows PowerShell:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$c = Get-Content 'C:\ProgramData\HomelabManager\agent.json' | ConvertFrom-Json
|
||||||
|
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
||||||
|
$env:API_URL = $c.apiUrl; $env:API_TOKEN = $c.apiToken
|
||||||
|
if ($c.insecure) { $env:API_INSECURE = 'true'; [Net.ServicePointManager]::ServerCertificateValidationCallback = { $true } }
|
||||||
|
iex ((New-Object Net.WebClient).DownloadString("$env:API_URL/agent/windows/install.ps1"))
|
||||||
|
```
|
||||||
|
|
||||||
|
(That form is for Windows PowerShell 5.1, the one built into Windows. See the
|
||||||
|
[Windows agent README](agent/windows/README.md) for PowerShell 7.)
|
||||||
|
|
||||||
|
### B. Rotate the token and use the command the page shows
|
||||||
|
|
||||||
|
In Homelab Manager: **Servers → Manage servers → Rotate token** on that server. The page shows a fresh
|
||||||
|
install command for it (with the self-signed-certificate box if you need it). Run it on the machine.
|
||||||
|
|
||||||
|
Use this when the token is lost or you want it replaced anyway. Know that rotating takes effect **at once**:
|
||||||
|
the old token stops working, so that server's agent can't report until the new command has been run on it —
|
||||||
|
and the server will look as if it has stopped reporting (and alert) in the meantime. Do one server at a
|
||||||
|
time, and put it under [maintenance](README.md) first if the gap could be more than a few minutes.
|
||||||
|
|
||||||
|
## Updating many servers
|
||||||
|
|
||||||
|
For more than a handful, use the tool you already have. The commands in A are plain shell and PowerShell, so
|
||||||
|
Ansible or Semaphore can run them. For example, a playbook for the Linux hosts (an example — adjust the
|
||||||
|
group name, and add `-k` to `curl` for a self-signed certificate):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- hosts: linux_servers
|
||||||
|
become: true
|
||||||
|
tasks:
|
||||||
|
- name: Update the Homelab Manager agent (same token as before)
|
||||||
|
ansible.builtin.shell: |
|
||||||
|
set -e
|
||||||
|
. /etc/homelab-manager-agent.env
|
||||||
|
curl -fsSL "$API_URL/agent/linux/install.sh" | API_URL="$API_URL" API_TOKEN="$API_TOKEN" API_INSECURE="$API_INSECURE" bash
|
||||||
|
args:
|
||||||
|
executable: /bin/bash
|
||||||
|
changed_when: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Homelab Manager can start such a template for you, limited to one host, from the server's own page (*Run a
|
||||||
|
playbook*) — a good way to try it on one machine first, then run it on the rest from Semaphore.
|
||||||
|
|
||||||
|
## Checking it worked
|
||||||
|
|
||||||
|
1. **The Updates page.** The server's row changes from *not reported* to its figures within one reporting
|
||||||
|
interval (15 minutes by default; on Windows the installer reports straight away, and on Linux you can
|
||||||
|
start one now — below).
|
||||||
|
2. **Last seen** on the server's page moves to *just now*.
|
||||||
|
|
||||||
|
On a Linux server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl list-timers homelab-manager-agent.timer # when it next runs
|
||||||
|
sudo systemctl start homelab-manager-agent.service # report now
|
||||||
|
sudo journalctl -u homelab-manager-agent -n 30 # what it said (look for "Note:" lines)
|
||||||
|
sudo /usr/local/bin/homelab-manager-agent.sh --dry-run | jq .system.updates # what it would send; sends nothing
|
||||||
|
```
|
||||||
|
|
||||||
|
On a Windows server (elevated):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
Start-ScheduledTask -TaskName 'Homelab Manager Agent' # report now
|
||||||
|
Get-ScheduledTaskInfo -TaskName 'Homelab Manager Agent' # LastRunTime, LastTaskResult (0 = fine)
|
||||||
|
powershell -NoProfile -ExecutionPolicy Bypass -File 'C:\ProgramData\HomelabManager\homelab-manager-agent.ps1' -DryRun # prints what it would send
|
||||||
|
```
|
||||||
|
|
||||||
|
## How the update check works — and why a server may stay "not reported"
|
||||||
|
|
||||||
|
The agent only *asks*; it never installs, downloads or refreshes anything.
|
||||||
|
|
||||||
|
| System | What it asks | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Debian, Ubuntu, Raspberry Pi OS (`apt`) | a simulated upgrade (`apt-get -s upgrade`) | Uses the package lists already on the machine. If nothing refreshes them (`apt update`, or the distribution's own daily timer), the count is only as fresh as they are. Security updates are those from a `-security` archive. |
|
||||||
|
| Fedora, RHEL and relatives (`dnf`) | `dnf check-update --cacheonly` | Reads dnf's cache only; if the cache is empty or unreadable the server stays *not reported*. A restart is flagged by `needs-restarting -r` when that tool is installed. |
|
||||||
|
| openSUSE, SLES (`zypper`) | `zypper list-updates` without refreshing | Security figure not available (shown as unknown). |
|
||||||
|
| Windows | Windows Update, through its own API | Searching can take a minute or two, so it is done at most **every six hours** and kept in `%ProgramData%\HomelabManager\updates.json`. It runs in a separate process with a 150-second limit (the scheduled task itself is stopped after five minutes). If the search fails or times out it isn't tried again for an hour, and that report leaves updates out. |
|
||||||
|
|
||||||
|
**Not reported** means the agent didn't give a figure. The usual reasons:
|
||||||
|
|
||||||
|
1. **It is an older agent.** Update it (above). This is by far the most common.
|
||||||
|
2. **Another package manager** — Alpine (`apk`), Arch (`pacman`) and others aren't read yet.
|
||||||
|
3. **The check couldn't run.** On Linux the agent writes a line to the journal when its package manager
|
||||||
|
couldn't be read — `journalctl -u homelab-manager-agent -n 30`. The agent runs inside a tight systemd
|
||||||
|
sandbox (read-only system, no hardware, minimal privileges, see `install.sh`), which suits `apt`'s
|
||||||
|
simulation but may stop `dnf` or `zypper` from reading their caches. If you hit that, the sandbox is what
|
||||||
|
to look at; loosen it only as far as that one need, and only on the hosts that need it.
|
||||||
|
4. **The package manager was busy** (another update running). It is tried again at the next report.
|
||||||
|
|
||||||
|
A server that says nothing keeps what it last said — an agent that skips a report never turns an old figure
|
||||||
|
into "up to date".
|
||||||
|
|
||||||
|
## If something goes wrong
|
||||||
|
|
||||||
|
- **The installer fails on download** — the machine can't reach Homelab Manager's `/agent/...` address, or the
|
||||||
|
certificate isn't trusted (use the self-signed forms above). Nothing has changed on the machine yet.
|
||||||
|
- **The installer ran but the server doesn't report** — check the token (a rotated token makes the old one
|
||||||
|
stop at once) and `API_URL`: `journalctl -u homelab-manager-agent` on Linux, the scheduled task's last
|
||||||
|
result on Windows.
|
||||||
|
- **To go back** — there is no older version to fetch; the agent a Homelab Manager serves is the one that
|
||||||
|
came with it. To remove an agent, use the uninstall command shown on the Servers page; the server's entry and
|
||||||
|
history stay until you delete it there.
|
||||||
@@ -436,7 +436,8 @@ The Linux agent asks apt, dnf or zypper (only what they already know; nothing is
|
|||||||
installed or refreshed) and the Windows agent asks Windows Update (at most every six
|
installed or refreshed) and the Windows agent asks Windows Update (at most every six
|
||||||
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
|
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
|
||||||
said — an older agent, or another package manager — shows "not reported", never "up
|
said — an older agent, or another package manager — shows "not reported", never "up
|
||||||
to date"; run its install command again to update its agent. It only looks: to apply
|
to date"; run its install command again to update its agent (how, and how to
|
||||||
|
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply
|
||||||
updates, run a playbook.
|
updates, run a playbook.
|
||||||
|
|
||||||
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
|
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
|
||||||
|
|||||||
@@ -215,6 +215,11 @@ collect_updates() {
|
|||||||
[[ -f /run/reboot-needed ]] && reboot="true"
|
[[ -f /run/reboot-needed ]] && reboot="true"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$manager" && ! "$count" =~ ^[0-9]+$ ]]; then
|
||||||
|
# In the journal (systemctl status / journalctl -u homelab-manager-agent) so "not reported" has a reason to look for.
|
||||||
|
echo "Note: couldn't read the pending updates from $manager - the report goes without them." >&2
|
||||||
|
fi
|
||||||
|
|
||||||
if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then
|
if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then
|
||||||
[[ "$security" =~ ^[0-9]+$ ]] || security="null"
|
[[ "$security" =~ ^[0-9]+$ ]] || security="null"
|
||||||
UPDATES_JSON=$(jq -n --arg manager "$manager" --argjson count "$count" --argjson security "$security" --argjson reboot "$reboot" \
|
UPDATES_JSON=$(jq -n --arg manager "$manager" --argjson count "$count" --argjson security "$security" --argjson reboot "$reboot" \
|
||||||
|
|||||||
@@ -6,9 +6,10 @@ runs as a scheduled task (as SYSTEM, every 15 minutes) and pushes to `POST /api/
|
|||||||
the machine never needs to be reachable from Homelab Manager.
|
the machine never needs to be reachable from Homelab Manager.
|
||||||
|
|
||||||
|
|
||||||
Each report also says how many updates Windows Update has waiting (and how many are security or critical, and whether a restart is due). Searching Windows Update goes over the network and can take a minute or two, so it is done at most every six hours; the answer is kept in `%ProgramData%\HomelabManager\updates.json` and reused between. Nothing is downloaded or installed.
|
Each report also says how many updates Windows Update has waiting (and how many are security or critical, and whether a restart is due). Searching Windows Update goes over the network and can take a minute or two, so it runs in a separate process with a 150-second limit and is done at most every six hours; the answer is kept in `%ProgramData%\HomelabManager\updates.json` and reused between (a search that fails is not tried again for an hour, and that report simply leaves updates out). Nothing is downloaded or installed. To update an agent that was installed before this existed, see [AGENT_UPDATE.md](../../AGENT_UPDATE.md).
|
||||||
|
|
||||||
The installer limits `%ProgramData%\HomelabManager` to SYSTEM and Administrators. The agent runs as SYSTEM, so anyone who could change its script could run code as SYSTEM; the folder is not left to inherit ProgramData's default permissions. Re-running the install one-liner applies this to an existing install.
|
The installer limits `%ProgramData%\HomelabManager` to SYSTEM and Administrators. The agent runs as SYSTEM, so anyone who could change its script could run code as SYSTEM; the folder is not left to inherit ProgramData's default permissions. Re-running the install one-liner applies this to an existing install.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
1. In Homelab Manager, **Servers → Manage servers → Add a server**, choose **Windows** as the
|
1. In Homelab Manager, **Servers → Manage servers → Add a server**, choose **Windows** as the
|
||||||
|
|||||||
@@ -232,36 +232,55 @@ function Test-RebootRequired {
|
|||||||
return $false
|
return $false
|
||||||
}
|
}
|
||||||
|
|
||||||
# Asks Windows Update what is waiting. That goes over the network and can take a minute, so it is done at most every six hours and the
|
# Asks Windows Update what is waiting. That goes over the network and can take a minute or two, and the scheduled task is stopped
|
||||||
# last answer is reused in between; whether a restart is due is cheap and always read fresh. Nothing is installed or downloaded.
|
# after five minutes, so the search runs in a separate process with a time limit, and is repeated at most every six hours; the last
|
||||||
|
# answer is reused in between. A search that fails or runs out of time is not tried again for an hour, so a broken Windows Update
|
||||||
|
# can't slow down every report. Whether a restart is due is cheap and always read fresh. Nothing is installed or downloaded.
|
||||||
function Get-PendingUpdates {
|
function Get-PendingUpdates {
|
||||||
$result = $null
|
$cached = $null
|
||||||
try {
|
try {
|
||||||
if (Test-Path $script:UpdatesCachePath) {
|
if (Test-Path $script:UpdatesCachePath) {
|
||||||
$cached = Get-Content -Raw -Path $script:UpdatesCachePath | ConvertFrom-Json
|
$cached = Get-Content -Raw -Path $script:UpdatesCachePath | ConvertFrom-Json
|
||||||
$checked = [datetime]::Parse([string]$cached.checked_at, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::RoundtripKind)
|
# PowerShell 7 hands a stored date back as a date, Windows PowerShell 5.1 as the text it was written as.
|
||||||
if (([datetime]::UtcNow - $checked.ToUniversalTime()).TotalHours -lt 6) { $result = $cached }
|
$checked = $cached.checked_at
|
||||||
|
if ($checked -isnot [datetime]) { $checked = [datetime]::Parse([string]$checked, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::RoundtripKind) }
|
||||||
|
$limitHours = 6
|
||||||
|
if ($cached.failed) { $limitHours = 1 }
|
||||||
|
if (([datetime]::UtcNow - $checked.ToUniversalTime()).TotalHours -ge $limitHours) { $cached = $null }
|
||||||
}
|
}
|
||||||
} catch { $result = $null }
|
} catch { $cached = $null }
|
||||||
|
|
||||||
if (-not $result) {
|
if (-not $cached) {
|
||||||
$session = New-Object -ComObject Microsoft.Update.Session
|
$job = Start-Job -ScriptBlock {
|
||||||
$found = $session.CreateUpdateSearcher().Search("IsInstalled=0 and IsHidden=0 and Type='Software'")
|
$session = New-Object -ComObject Microsoft.Update.Session
|
||||||
$security = 0
|
$found = $session.CreateUpdateSearcher().Search("IsInstalled=0 and IsHidden=0 and Type='Software'")
|
||||||
foreach ($u in $found.Updates) {
|
$security = 0
|
||||||
foreach ($c in $u.Categories) {
|
foreach ($u in $found.Updates) {
|
||||||
if ($c.Name -eq 'Security Updates' -or $c.Name -eq 'Critical Updates') { $security++; break }
|
foreach ($c in $u.Categories) {
|
||||||
|
if ($c.Name -eq 'Security Updates' -or $c.Name -eq 'Critical Updates') { $security++; break }
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
[pscustomobject]@{ count = [int]$found.Updates.Count; security = [int]$security }
|
||||||
}
|
}
|
||||||
$result = [pscustomobject][ordered]@{ checked_at = [datetime]::UtcNow.ToString('o'); count = [int]$found.Updates.Count; security = [int]$security }
|
$answer = $null
|
||||||
|
try {
|
||||||
|
if (Wait-Job -Job $job -Timeout 150) { $answer = Receive-Job -Job $job -ErrorAction Stop } else { Stop-Job -Job $job }
|
||||||
|
} catch { $answer = $null } finally { Remove-Job -Job $job -Force -ErrorAction SilentlyContinue }
|
||||||
|
|
||||||
|
$count = 0
|
||||||
|
$security = 0
|
||||||
|
if ($null -ne $answer) { $count = [int]$answer.count; $security = [int]$answer.security }
|
||||||
|
$cached = [pscustomobject][ordered]@{ checked_at = [datetime]::UtcNow.ToString('o'); failed = ($null -eq $answer); count = $count; security = $security }
|
||||||
try {
|
try {
|
||||||
$dir = Split-Path $script:UpdatesCachePath
|
$dir = Split-Path $script:UpdatesCachePath
|
||||||
if (-not (Test-Path $dir)) { $null = New-Item -ItemType Directory -Path $dir -Force }
|
if (-not (Test-Path $dir)) { $null = New-Item -ItemType Directory -Path $dir -Force }
|
||||||
ConvertTo-Json -InputObject $result | Set-Content -Path $script:UpdatesCachePath -Encoding UTF8
|
ConvertTo-Json -InputObject $cached | Set-Content -Path $script:UpdatesCachePath -Encoding UTF8
|
||||||
} catch { }
|
} catch { }
|
||||||
}
|
}
|
||||||
|
|
||||||
return [ordered]@{ manager = 'windows-update'; count = [int]$result.count; security = [int]$result.security; reboot_required = (Test-RebootRequired) }
|
# No answer is no answer: the report then leaves updates out, and the server keeps what it was last told.
|
||||||
|
if ($cached.failed) { return $null }
|
||||||
|
return [ordered]@{ manager = 'windows-update'; count = [int]$cached.count; security = [int]$cached.security; reboot_required = (Test-RebootRequired) }
|
||||||
}
|
}
|
||||||
|
|
||||||
function Get-SystemInfo {
|
function Get-SystemInfo {
|
||||||
|
|||||||
Reference in new issue
Block a user