Restart a server after its updates, and check that it came back

An update run can name a second Semaphore playbook that restarts the server
once the updates are in and it says a restart is due. The run keeps the
maintenance window up, follows the restart, waits for the server to report
again and checks from its uptime (or its restart flag clearing) that it did
start again. A server that only needs a restart has a Restart button on the
Updates page, and a patch plan can carry the restart playbook too.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-10 23:45:52 +01:00
1 parent 97df28c6dc
commit e4f3a54189
19 files changed
+10275 -87

No files matched your search

+13 -2
View File
@@ -616,7 +616,7 @@ and it is still on the same machine until you copy it away: see [BACKUP.md](BACK
**Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days
and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore
playbook that applies updates. When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
playbook that applies updates (and, if you like, a second one that restarts a server which says a restart is due once its updates are in — see below). When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
(below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server
(a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers
after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for
@@ -636,7 +636,18 @@ the alerts can speak) if the playbook fails, Semaphore stops answering, the serv
doesn't report again within 30 minutes, or it takes more than three hours. "Stop
following" ends the window and stops watching; it can't stop a playbook already running
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
send*). It never restarts the server itself — the playbook does, if it is written to.
send*).
**Restarting after updates.** The update drawer has a second choice, *Restart it afterwards, if it says a restart is due, with* — a Semaphore playbook that only restarts the
server (in the same Semaphore as the update's). When the updates are in and the server reports again saying a restart is due, that playbook is run for the same one server, the maintenance
window is kept up for it (at least 45 more minutes), and the server is **waited for again and checked**: it must have started again, which is read from the uptime its agent reports (its start is
the report's time minus its uptime; a start a couple of minutes before the restart began still counts) — or, where an agent reports no uptime, from its *restart needed* flag clearing. A playbook that
finished while the server did not restart is a *failed* run that says so ("check that the playbook really restarts it"); a restart playbook that fails, or can't be started, ends the run saying the
updates are in and the server still needs a restart. Without a second playbook nothing changes: the run says a restart is still needed, and an update playbook may restart the machine itself, as before.
A server that only needs a restart (nothing waiting, *restart needed* showing) has a **Restart…** button on the Updates page that does the restart on its own, with the same checks. A **patch plan** can
carry the restart playbook too: each server is updated and, if it then says so, restarted before the plan goes on to the next; a server with nothing waiting that still needs a restart is restarted instead
of left alone; the plan's maintenance time counts for the update and the restart together. Operators start them; the restart is in the server's history and the audit log, and the message at the end
says "the server was restarted and is back".
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page