Restart a server after its updates, and check that it came back
An update run can name a second Semaphore playbook that restarts the server once the updates are in and it says a restart is due. The run keeps the maintenance window up, follows the restart, waits for the server to report again and checks from its uptime (or its restart flag clearing) that it did start again. A server that only needs a restart has a Restart button on the Updates page, and a patch plan can carry the restart playbook too. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
97df28c6dc
commit
e4f3a54189
19 files changed
+10275
-87
No files matched your search
@@ -616,7 +616,7 @@ and it is still on the same machine until you copy it away: see [BACKUP.md](BACK
|
||||
|
||||
**Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days
|
||||
and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore
|
||||
playbook that applies updates. When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
|
||||
playbook that applies updates (and, if you like, a second one that restarts a server which says a restart is due once its updates are in — see below). When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
|
||||
(below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server
|
||||
(a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers
|
||||
after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for
|
||||
@@ -636,7 +636,18 @@ the alerts can speak) if the playbook fails, Semaphore stops answering, the serv
|
||||
doesn't report again within 30 minutes, or it takes more than three hours. "Stop
|
||||
following" ends the window and stops watching; it can't stop a playbook already running
|
||||
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
|
||||
send*). It never restarts the server itself — the playbook does, if it is written to.
|
||||
send*).
|
||||
|
||||
**Restarting after updates.** The update drawer has a second choice, *Restart it afterwards, if it says a restart is due, with* — a Semaphore playbook that only restarts the
|
||||
server (in the same Semaphore as the update's). When the updates are in and the server reports again saying a restart is due, that playbook is run for the same one server, the maintenance
|
||||
window is kept up for it (at least 45 more minutes), and the server is **waited for again and checked**: it must have started again, which is read from the uptime its agent reports (its start is
|
||||
the report's time minus its uptime; a start a couple of minutes before the restart began still counts) — or, where an agent reports no uptime, from its *restart needed* flag clearing. A playbook that
|
||||
finished while the server did not restart is a *failed* run that says so ("check that the playbook really restarts it"); a restart playbook that fails, or can't be started, ends the run saying the
|
||||
updates are in and the server still needs a restart. Without a second playbook nothing changes: the run says a restart is still needed, and an update playbook may restart the machine itself, as before.
|
||||
A server that only needs a restart (nothing waiting, *restart needed* showing) has a **Restart…** button on the Updates page that does the restart on its own, with the same checks. A **patch plan** can
|
||||
carry the restart playbook too: each server is updated and, if it then says so, restarted before the plan goes on to the next; a server with nothing waiting that still needs a restart is restarted instead
|
||||
of left alone; the plan's maintenance time counts for the update and the restart together. Operators start them; the restart is in the server's history and the audit log, and the message at the end
|
||||
says "the server was restarted and is back".
|
||||
|
||||
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
|
||||
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
|
||||
|
||||
Reference in new issue
Block a user