Restart a server after its updates, and check that it came back

An update run can name a second Semaphore playbook that restarts the server
once the updates are in and it says a restart is due. The run keeps the
maintenance window up, follows the restart, waits for the server to report
again and checks from its uptime (or its restart flag clearing) that it did
start again. A server that only needs a restart has a Restart button on the
Updates page, and a patch plan can carry the restart playbook too.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-10 23:45:52 +01:00
1 parent 97df28c6dc
commit e4f3a54189
19 files changed
+10267 -79

No files matched your search

+7 -1
View File
@@ -655,6 +655,7 @@ Updates on a schedule, one server at a time (services/patchPlans.ts). A plan onl
| `limit_from` | text NOT NULL, default `name` | What ansible's `--limit` is for each server: its name here, or its host name. | | `limit_from` | text NOT NULL, default `name` | What ansible's `--limit` is for each server: its name here, or its host name. |
| `stop_on_failure`, `only_with_updates`, `silence_dependents` | bool NOT NULL, default true | Stop at the first failure; leave out a server with nothing waiting; silence what depends on a server while it is updated. | | `stop_on_failure`, `only_with_updates`, `silence_dependents` | bool NOT NULL, default true | Stop at the first failure; leave out a server with nothing waiting; silence what depends on a server while it is updated. |
| `window_minutes` | integer NOT NULL, default 90 | The longest a server is under maintenance. | | `window_minutes` | integer NOT NULL, default 90 | The longest a server is under maintenance. |
| `restart_project_id`, `restart_template_id`, `restart_template_name` | integer, integer, text | A second Semaphore playbook (same integration) that restarts a server that says a restart is due after its updates — and one that only needs a restart; all null = none. |
| `last_run_date` | text | The local day it last started on, so it starts once a day even across a restart. | | `last_run_date` | text | The local day it last started on, so it starts once a day even across a restart. |
| `created_by`, `created_at` | | | | `created_by`, `created_at` | | |
@@ -700,7 +701,12 @@ A playbook run in Semaphore for one server, wrapped in a maintenance window and
| `template_name`, `host_limit` | text NOT NULL | The template's name and ansible's `--limit` it was run with (never empty: an update run is for one server). | | `template_name`, `host_limit` | text NOT NULL | The template's name and ansible's `--limit` it was run with (never empty: an update run is for one server). |
| `task_id`, `task_status` | integer, text | Semaphore's run number and its own word for how it is going. | | `task_id`, `task_status` | integer, text | Semaphore's run number and its own word for how it is going. |
| `window_id`, `window_created` | integer, bool | The maintenance window put round it, and whether this run made it (a window that was already there is extended and left to run out). | | `window_id`, `window_created` | integer, bool | The maintenance window put round it, and whether this run made it (a window that was already there is extended and left to run out). |
| `status` | text NOT NULL | `running`, `waiting_for_host`, `done`, `failed` or `cancelled`. | | `status` | text NOT NULL | `running`, `waiting_for_host`, `restarting`, `waiting_for_restart`, `done`, `failed` or `cancelled` (the first four are "under way"). |
| `kind` | text NOT NULL, default `update` | `update`, or `restart` (the template is a restart playbook and nothing else is done). |
| `restart_project_id`, `restart_template_id`, `restart_template_name` | integer, integer, text | The playbook that restarts the server once the updates are in, if one was chosen. |
| `restart_task_id`, `restart_task_status` | integer, text | Semaphore's run number for the restart, and its word for how it is going. |
| `restart_started_at`, `restart_ended_at` | text | ISO — when the restart playbook was started and when it finished. |
| `restarted` | bool NOT NULL, default 0 | The server was seen to have started again (from its reported uptime, or its restart flag clearing). |
| `message` | text | What it says now, in words. | | `message` | text | What it says now, in words. |
| `started_by` | text | | | `started_by` | text | |
| `started_at`, `task_ended_at`, `host_reported_at`, `finished_at` | text | ISO. | | `started_at`, `task_ended_at`, `host_reported_at`, `finished_at` | text | ISO. |
+1 -1
View File
@@ -148,7 +148,7 @@ until the next scheduled time to catch up.
| Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) | | Docker image updates | *Docker image update available* | Any container (across every enabled Dockhand integration) has an image update available, per Dockhand's own cached update-check results — this doesn't trigger a fresh registry lookup, just reads what the Docker page itself would show. | Not configurable (reflects Dockhand's own check interval) |
| Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page | | Proxmox backups — gone stale | *(same toggle as the Proxmox backup check)* | Once a day with the other backup messages: a guest whose newest backup (on Proxmox or a Backup Server) is older than the limit. Guests under maintenance (on the server or the Proxmox integration) are left out. | Settings → Notifications → **A guest's backup is too old after** (default 48 h; 0 off); a server can have its own limit on its page |
| Game server turned on or off | *A game server (Fragnet) was turned on or off* | A read of a Fragnet account shows a game server **on** (running or starting) when the one before showed it off, or **off** (stopped or stopping) when it showed it on — however it came about: someone pressed Start or Stop, the game crashed, the host stopped it. One message for what changed in one read, a line for each server (name, game, account); *starting* counts as on, so a start is said once. A server met for the first time, one whose state could not be read and one being installed or moved say nothing. Goes through the routing rules as a *services* notification and waits for quiet hours like any other that is not critical; the accounts are read every five minutes, so it can be that late. On unless switched off. | Settings → Notifications → What to send | | Game server turned on or off | *A game server (Fragnet) was turned on or off* | A read of a Fragnet account shows a game server **on** (running or starting) when the one before showed it off, or **off** (stopped or stopping) when it showed it on — however it came about: someone pressed Start or Stop, the game crashed, the host stopped it. One message for what changed in one read, a line for each server (name, game, account); *starting* counts as on, so a start is said once. A server met for the first time, one whose state could not be read and one being installed or moved say nothing. Goes through the routing rules as a *services* notification and waits for quiet hours like any other that is not critical; the accounts are read every five minutes, so it can be that late. On unless switched off. | Settings → Notifications → What to send |
| Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due, or why it failed. Sent at once (not held for the daily check). | Settings → Notifications → What to send | | Update run finished | *An update run you started finished, or didn't* | An update run started from the Updates page ended: what is still waiting and whether a restart is due (or, for a run that restarts the server afterwards, that it was restarted and is back), or why it failed. A restart started on its own says "restarted" or "Restart of … didn't finish". Sent at once (not held for the daily check). | Settings → Notifications → What to send |
| Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable | | Proxmox backups | *Proxmox backup failed or a guest has no coverage* | Two independent conditions, both under this one toggle: (1) a node's most recent `vzdump` backup task (across every enabled Proxmox integration) didn't succeed; (2) a VM/LXC isn't covered by any enabled backup job at all. | Not configurable |
| Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable | | Proxmox Backup Server verification | *Proxmox Backup Server snapshot failed verification* | Across every enabled PBS integration: any datastore has at least one stored snapshot whose verification state is "failed", or a datastore couldn't be read at all (e.g. a permissions problem). This is distinct from the Proxmox check above — PVE only knows a backup *ran*, PBS is the only place that knows whether the stored data still verifies. | Not configurable |
+13 -2
View File
@@ -616,7 +616,7 @@ and it is still on the same machine until you copy it away: see [BACKUP.md](BACK
**Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days **Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days
and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore
playbook that applies updates. When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run playbook that applies updates (and, if you like, a second one that restarts a server which says a restart is due once its updates are in — see below). When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
(below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server (below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server
(a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers (a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers
after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for
@@ -636,7 +636,18 @@ the alerts can speak) if the playbook fails, Semaphore stops answering, the serv
doesn't report again within 30 minutes, or it takes more than three hours. "Stop doesn't report again within 30 minutes, or it takes more than three hours. "Stop
following" ends the window and stops watching; it can't stop a playbook already running following" ends the window and stops watching; it can't stop a playbook already running
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
send*). It never restarts the server itself — the playbook does, if it is written to. send*).
**Restarting after updates.** The update drawer has a second choice, *Restart it afterwards, if it says a restart is due, with* — a Semaphore playbook that only restarts the
server (in the same Semaphore as the update's). When the updates are in and the server reports again saying a restart is due, that playbook is run for the same one server, the maintenance
window is kept up for it (at least 45 more minutes), and the server is **waited for again and checked**: it must have started again, which is read from the uptime its agent reports (its start is
the report's time minus its uptime; a start a couple of minutes before the restart began still counts) — or, where an agent reports no uptime, from its *restart needed* flag clearing. A playbook that
finished while the server did not restart is a *failed* run that says so ("check that the playbook really restarts it"); a restart playbook that fails, or can't be started, ends the run saying the
updates are in and the server still needs a restart. Without a second playbook nothing changes: the run says a restart is still needed, and an update playbook may restart the machine itself, as before.
A server that only needs a restart (nothing waiting, *restart needed* showing) has a **Restart…** button on the Updates page that does the restart on its own, with the same checks. A **patch plan** can
carry the restart playbook too: each server is updated and, if it then says so, restarted before the plan goes on to the next; a server with nothing waiting that still needs a restart is restarted instead
of left alone; the plan's maintenance time counts for the update and the restart together. Operators start them; the restart is in the server's history and the audit log, and the message at the end
says "the server was restarted and is back".
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have **Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
@@ -0,0 +1,12 @@
ALTER TABLE `patch_plans` ADD `restart_project_id` integer;--> statement-breakpoint
ALTER TABLE `patch_plans` ADD `restart_template_id` integer;--> statement-breakpoint
ALTER TABLE `patch_plans` ADD `restart_template_name` text;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `kind` text DEFAULT 'update' NOT NULL;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_project_id` integer;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_template_id` integer;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_template_name` text;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_task_id` integer;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_task_status` text;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_started_at` text;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restart_ended_at` text;--> statement-breakpoint
ALTER TABLE `update_runs` ADD `restarted` integer DEFAULT false NOT NULL;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -652,6 +652,13 @@
"when": 1794340893429, "when": 1794340893429,
"tag": "0092_parallel_the_initiative", "tag": "0092_parallel_the_initiative",
"breakpoints": true "breakpoints": true
},
{
"idx": 93,
"version": "6",
"when": 1794349689431,
"tag": "0093_reboot_after_updates",
"breakpoints": true
} }
] ]
} }
+16 -1
View File
@@ -1003,7 +1003,8 @@ export const incidentNotes = sqliteTable(
// ─── Update runs: apply a server's updates and see it through ────────────── // ─── Update runs: apply a server's updates and see it through ──────────────
// A playbook run in Semaphore for one server, wrapped in a maintenance window and followed until the server reports again — see // A playbook run in Semaphore for one server, wrapped in a maintenance window and followed until the server reports again — see
// services/updateRuns.ts. status: running | waiting_for_host | done | failed | cancelled. // services/updateRuns.ts. status: running | waiting_for_host | restarting | waiting_for_restart | done | failed | cancelled. kind: "update" (apply updates, and restart
// afterwards if a restart template is set and the server says one is due) or "restart" (the template is a restart playbook and nothing else is done).
export const updateRuns = sqliteTable( export const updateRuns = sqliteTable(
"update_runs", "update_runs",
{ {
@@ -1031,6 +1032,16 @@ export const updateRuns = sqliteTable(
updatesAfter: integer("updates_after"), updatesAfter: integer("updates_after"),
rebootRequired: integer("reboot_required", { mode: "boolean" }).notNull().default(false), rebootRequired: integer("reboot_required", { mode: "boolean" }).notNull().default(false),
pollErrors: integer("poll_errors").notNull().default(0), pollErrors: integer("poll_errors").notNull().default(0),
kind: text("kind").notNull().default("update"),
// The playbook that restarts the server afterwards (same Semaphore integration), and how that went.
restartProjectId: integer("restart_project_id"),
restartTemplateId: integer("restart_template_id"),
restartTemplateName: text("restart_template_name"),
restartTaskId: integer("restart_task_id"),
restartTaskStatus: text("restart_task_status"),
restartStartedAt: text("restart_started_at"), // ISO
restartEndedAt: text("restart_ended_at"), // ISO — when the restart playbook finished
restarted: integer("restarted", { mode: "boolean" }).notNull().default(false), // the server was seen to have started again (from its uptime, or its restart flag clearing)
}, },
(t) => [index("update_runs_server_idx").on(t.serverId, t.startedAt), index("update_runs_status_idx").on(t.status)], (t) => [index("update_runs_server_idx").on(t.serverId, t.startedAt), index("update_runs_status_idx").on(t.status)],
); );
@@ -1052,6 +1063,10 @@ export const patchPlans = sqliteTable("patch_plans", {
projectId: integer("project_id").notNull(), projectId: integer("project_id").notNull(),
templateId: integer("template_id").notNull(), templateId: integer("template_id").notNull(),
templateName: text("template_name").notNull(), templateName: text("template_name").notNull(),
// A playbook that restarts a server that says a restart is due once its updates are in (and one that only needs a restart is restarted instead of left alone); null = none.
restartProjectId: integer("restart_project_id"),
restartTemplateId: integer("restart_template_id"),
restartTemplateName: text("restart_template_name"),
limitFrom: text("limit_from").notNull().default("name"), // what ansible's --limit is for a server: its name here, or its host name limitFrom: text("limit_from").notNull().default("name"), // what ansible's --limit is for a server: its name here, or its host name
stopOnFailure: integer("stop_on_failure", { mode: "boolean" }).notNull().default(true), stopOnFailure: integer("stop_on_failure", { mode: "boolean" }).notNull().default(true),
onlyWithUpdates: integer("only_with_updates", { mode: "boolean" }).notNull().default(true), // leave out a server that has nothing waiting onlyWithUpdates: integer("only_with_updates", { mode: "boolean" }).notNull().default(true), // leave out a server that has nothing waiting
+11
View File
@@ -36,7 +36,12 @@ const planSchema = z
onlyWithUpdates: z.boolean().default(true), onlyWithUpdates: z.boolean().default(true),
silenceDependents: z.boolean().default(true), silenceDependents: z.boolean().default(true),
windowMinutes: z.number().int().min(15).max(480).default(90), windowMinutes: z.number().int().min(15).max(480).default(90),
// A playbook that restarts a server that says a restart is due after its updates (all three or none).
restartProjectId: z.number().int().positive().nullable().default(null),
restartTemplateId: z.number().int().positive().nullable().default(null),
restartTemplateName: z.string().trim().min(1).max(200).nullable().default(null),
}) })
.refine((d) => (d.restartProjectId === null) === (d.restartTemplateId === null) && (d.restartTemplateId === null) === (d.restartTemplateName === null), { message: "The restart playbook needs its project, its template and its name — or none of them.", path: ["restartTemplateId"] })
.refine((d) => isValidTimeZone(d.timezone), { message: "That isn't a time zone the system knows (use a name like Europe/Stockholm).", path: ["timezone"] }) .refine((d) => isValidTimeZone(d.timezone), { message: "That isn't a time zone the system knows (use a name like Europe/Stockholm).", path: ["timezone"] })
.refine((d) => d.tags.length > 0 || d.serverIds.length > 0, { message: "Choose which servers: give a tag, or name at least one server.", path: ["tags"] }); .refine((d) => d.tags.length > 0 || d.serverIds.length > 0, { message: "Choose which servers: give a tag, or name at least one server.", path: ["tags"] });
@@ -58,6 +63,9 @@ function valuesOf(d: z.infer<typeof planSchema>) {
onlyWithUpdates: d.onlyWithUpdates, onlyWithUpdates: d.onlyWithUpdates,
silenceDependents: d.silenceDependents, silenceDependents: d.silenceDependents,
windowMinutes: d.windowMinutes, windowMinutes: d.windowMinutes,
restartProjectId: d.restartProjectId,
restartTemplateId: d.restartTemplateId,
restartTemplateName: d.restartTemplateName,
}; };
} }
@@ -96,6 +104,9 @@ async function shapePlan(p: PlanRow, all: { id: number; name: string; tags: stri
onlyWithUpdates: p.onlyWithUpdates, onlyWithUpdates: p.onlyWithUpdates,
silenceDependents: p.silenceDependents, silenceDependents: p.silenceDependents,
windowMinutes: p.windowMinutes, windowMinutes: p.windowMinutes,
restartProjectId: p.restartProjectId,
restartTemplateId: p.restartTemplateId,
restartTemplateName: p.restartTemplateName,
createdBy: p.createdBy, createdBy: p.createdBy,
servers: covered, servers: covered,
next: next.map((o) => o.start.toISOString()), next: next.map((o) => o.start.toISOString()),
+10
View File
@@ -20,6 +20,10 @@ const startSchema = z.object({
windowMinutes: z.number().int().min(15).max(480).default(90), windowMinutes: z.number().int().min(15).max(480).default(90),
// Somebody is streaming from the machine and the person has seen the warning. // Somebody is streaming from the machine and the person has seen the warning.
startAnyway: z.boolean().optional(), startAnyway: z.boolean().optional(),
// "restart": the template is a playbook that only restarts the server.
kind: z.enum(["update", "restart"]).default("update"),
// An update run's playbook that restarts the server afterwards if it says one is due (in the same Semaphore).
restart: z.object({ projectId: z.number().int().positive(), templateId: z.number().int().positive() }).optional(),
}); });
const shape = (r: Awaited<ReturnType<typeof listUpdateRuns>>[number]) => ({ const shape = (r: Awaited<ReturnType<typeof listUpdateRuns>>[number]) => ({
@@ -41,6 +45,12 @@ const shape = (r: Awaited<ReturnType<typeof listUpdateRuns>>[number]) => ({
updatesBefore: r.updatesBefore, updatesBefore: r.updatesBefore,
updatesAfter: r.updatesAfter, updatesAfter: r.updatesAfter,
rebootRequired: r.rebootRequired, rebootRequired: r.rebootRequired,
kind: r.kind,
restartTemplate: r.restartTemplateName,
restartTaskId: r.restartTaskId,
restartTaskStatus: r.restartTaskStatus,
restartedAt: r.restartStartedAt,
restarted: r.restarted,
}); });
updateRunsRouter.get("/", asyncHandler(async (req, res) => { updateRunsRouter.get("/", asyncHandler(async (req, res) => {
+3 -2
View File
@@ -578,10 +578,11 @@ export async function notifyStaleBackups(stale: { message: string }[]): Promise<
); );
} }
export async function notifyUpdateRunFinished(run: { serverName: string; templateName: string; ok: boolean; message: string }): Promise<void> { export async function notifyUpdateRunFinished(run: { serverName: string; templateName: string; ok: boolean; message: string; kind?: "update" | "restart" }): Promise<void> {
if (!(await eventEnabled("updateRunResults"))) return; if (!(await eventEnabled("updateRunResults"))) return;
const restart = run.kind === "restart";
await notify( await notify(
run.ok ? `Homelab Manager — ${run.serverName} updated` : `Homelab Manager — Update of ${run.serverName} didn't finish`, run.ok ? `Homelab Manager — ${run.serverName} ${restart ? "restarted" : "updated"}` : `Homelab Manager — ${restart ? "Restart" : "Update"} of ${run.serverName} didn't finish`,
`"${run.templateName}" on ${run.serverName}: ${run.message}`, `"${run.templateName}" on ${run.serverName}: ${run.message}`,
{ category: "updates", severity: run.ok ? "warning" : "critical" }, { category: "updates", severity: run.ok ? "warning" : "critical" },
); );
+9 -4
View File
@@ -215,13 +215,17 @@ async function advanceRun(run: PlanRunRow, now: Date): Promise<void> {
Object.assign(next, { status: "skipped", message: "The server was removed." }); Object.assign(next, { status: "skipped", message: "The server was removed." });
continue; continue;
} }
// A server with nothing waiting is left alone — unless it only needs a restart and the plan has a playbook for that.
let restartOnly = false;
if (plan.onlyWithUpdates) { if (plan.onlyWithUpdates) {
const updates = parsePendingUpdates(server.pendingUpdates); const updates = parsePendingUpdates(server.pendingUpdates);
if (!updates || updates.count === 0) { if (updates && updates.count === 0 && updates.rebootRequired && plan.restartTemplateId !== null && plan.restartProjectId !== null) restartOnly = true;
else if (!updates || updates.count === 0) {
Object.assign(next, { status: "skipped", message: "No updates were waiting." }); Object.assign(next, { status: "skipped", message: "No updates were waiting." });
continue; continue;
} }
} }
const restartWith = plan.restartTemplateId !== null && plan.restartProjectId !== null ? { projectId: plan.restartProjectId, templateId: plan.restartTemplateId } : undefined;
const actor = `Patch plan "${plan.name}"`; const actor = `Patch plan "${plan.name}"`;
// People streaming from it (Jellyfin on this machine): its turn waits, up to two hours, rather than cutting a film off. // People streaming from it (Jellyfin on this machine): its turn waits, up to two hours, rather than cutting a film off.
const watching = await watchingOnServer(server.id, now); const watching = await watchingOnServer(server.id, now);
@@ -241,16 +245,17 @@ async function advanceRun(run: PlanRunRow, now: Date): Promise<void> {
{ {
serverId: server.id, serverId: server.id,
integrationId: plan.integrationId, integrationId: plan.integrationId,
projectId: plan.projectId, projectId: restartOnly ? restartWith!.projectId : plan.projectId,
templateId: plan.templateId, templateId: restartOnly ? restartWith!.templateId : plan.templateId,
limit: plan.limitFrom === "hostname" && server.hostname ? server.hostname : server.name, limit: plan.limitFrom === "hostname" && server.hostname ? server.hostname : server.name,
windowMinutes: plan.windowMinutes, windowMinutes: plan.windowMinutes,
actor, actor,
startAnyway: true, startAnyway: true,
...(restartOnly ? { kind: "restart" as const } : { restart: restartWith }),
}, },
now, now,
); );
Object.assign(next, { status: "running", updateRunId: ur.id, message: `${waited}The update run has started.`, windowIds: await silenceDependents(plan, server.id, server.name, actor, now) }); Object.assign(next, { status: "running", updateRunId: ur.id, message: `${waited}${restartOnly ? "Nothing to update, but it needs a restart: the restart has started." : "The update run has started."}`, windowIds: await silenceDependents(plan, server.id, server.name, actor, now) });
await saveRun(run, items); await saveRun(run, items);
return; // one at a time: this one is followed until it is over return; // one at a time: this one is followed until it is over
} catch (err) { } catch (err) {
+3 -1
View File
@@ -87,7 +87,9 @@ export function describeAudit(category: string, action: string, detail: Record<s
const template = typeof detail.template === "string" ? detail.template : null; const template = typeof detail.template === "string" ? detail.template : null;
switch (action) { switch (action) {
case "start": case "start":
return { title: "Update run started", tone: "info", kind: "change", detail: template ? `${template}, limited to ${detail.limit ?? "this server"}` : null }; return { title: detail.kind === "restart" ? "Restart started" : "Update run started", tone: "info", kind: "change", detail: template ? `${template}, limited to ${detail.limit ?? "this server"}${typeof detail.restartTemplate === "string" ? `, then a restart with ${detail.restartTemplate} if one is due` : ""}` : null };
case "restart":
return { title: "Update run is restarting the server", tone: "info", kind: "change", detail: template };
case "finished": case "finished":
return { title: "Update run finished", tone: "good", kind: "change", detail: message }; return { title: "Update run finished", tone: "good", kind: "change", detail: message };
case "failed": case "failed":
+153 -24
View File
@@ -4,8 +4,12 @@
* for the server's agent to report again, then says how many updates are still waiting and whether a restart is due, and ends the * for the server's agent to report again, then says how many updates are still waiting and whether a restart is due, and ends the
* maintenance window it made. Every step is written down on the run, which survives a restart of this app — a poller picks up where it was. * maintenance window it made. Every step is written down on the run, which survives a restart of this app — a poller picks up where it was.
* *
* What it does not do: it does not restart the server (the playbook does, if it is written to), and it cannot stop a run in Semaphore — * A restart afterwards is a second playbook, chosen when the run is started: when the server says a restart is due once its updates are in, that playbook is
* cancelling here only stops watching and ends the window. * run (the quiet is kept up for it), followed, and the server is waited for again — and it is checked that it did start again, from the uptime its agent
* reports (or, where that is not reported, from its "restart needed" flag clearing). A server that only needs a restart can be restarted on its own (kind "restart").
* Without a restart playbook the run does not restart the server (an update playbook may, if it is written to).
*
* What it cannot do: stop a run in Semaphore — cancelling here only stops watching and ends the window.
*/ */
import { and, desc, eq, inArray } from "drizzle-orm"; import { and, desc, eq, inArray } from "drizzle-orm";
import { db } from "../db/client.js"; import { db } from "../db/client.js";
@@ -20,8 +24,13 @@ import { notifyUpdateRunFinished } from "./notify.js";
import { parsePendingUpdates } from "./pendingUpdates.js"; import { parsePendingUpdates } from "./pendingUpdates.js";
export type UpdateRunRow = typeof updateRuns.$inferSelect; export type UpdateRunRow = typeof updateRuns.$inferSelect;
export type UpdateRunStatus = "running" | "waiting_for_host" | "done" | "failed" | "cancelled"; export type UpdateRunStatus = "running" | "waiting_for_host" | "restarting" | "waiting_for_restart" | "done" | "failed" | "cancelled";
export const ACTIVE_STATUSES: UpdateRunStatus[] = ["running", "waiting_for_host"]; export const ACTIVE_STATUSES: UpdateRunStatus[] = ["running", "waiting_for_host", "restarting", "waiting_for_restart"];
export type UpdateRunKind = "update" | "restart";
/** How long the quiet is kept up for a restart, counted from when its playbook is started. */
export const RESTART_WINDOW_MINUTES = 45;
/** The uptime a report gives is worked back to a start time; a start this much before the restart began still counts (clocks and report timing are not exact). */
export const BOOT_SLACK_MS = 2 * 60_000;
/** How long a run may take from start to finish before it is given up on. */ /** How long a run may take from start to finish before it is given up on. */
export const RUN_LIMIT_MS = 3 * 3600_000; export const RUN_LIMIT_MS = 3 * 3600_000;
@@ -71,9 +80,14 @@ export interface StartInput {
actor: string; actor: string;
/** Go ahead although somebody is streaming from the machine (a person who has seen the warning, or a plan that has waited long enough). */ /** Go ahead although somebody is streaming from the machine (a person who has seen the warning, or a plan that has waited long enough). */
startAnyway?: boolean; startAnyway?: boolean;
/** "restart": the template is a playbook that only restarts the server (nothing else is done). Default "update". */
kind?: UpdateRunKind;
/** A playbook (in the same Semaphore) that restarts the server once the updates are in, if it says a restart is due. Only for an update run. */
restart?: { projectId: number; templateId: number };
} }
export async function startUpdateRun(input: StartInput, now: Date = new Date()): Promise<UpdateRunRow> { export async function startUpdateRun(input: StartInput, now: Date = new Date()): Promise<UpdateRunRow> {
const kind: UpdateRunKind = input.kind ?? "update";
const [server] = await db.select().from(servers).where(eq(servers.id, input.serverId)).limit(1); const [server] = await db.select().from(servers).where(eq(servers.id, input.serverId)).limit(1);
if (!server) throw new UpdateRunError(404, "server_not_found", "That server doesn't exist."); if (!server) throw new UpdateRunError(404, "server_not_found", "That server doesn't exist.");
if (!server.lastSeenAt) throw new UpdateRunError(400, "never_reported", "This server has never reported, so there is nothing to wait for after the run."); if (!server.lastSeenAt) throw new UpdateRunError(400, "never_reported", "This server has never reported, so there is nothing to wait for after the run.");
@@ -95,21 +109,28 @@ export async function startUpdateRun(input: StartInput, now: Date = new Date()):
if (!adapter) throw new UpdateRunError(400, "no_semaphore", "That Semaphore integration doesn't exist or isn't switched on."); if (!adapter) throw new UpdateRunError(400, "no_semaphore", "That Semaphore integration doesn't exist or isn't switched on.");
let templateName: string; let templateName: string;
let restartTemplateName: string | null = null;
const restart = kind === "update" ? input.restart : undefined;
try { try {
const { templates } = await adapter.checkTemplates(); const { templates } = await adapter.checkTemplates();
const template = templates.find((t) => t.projectId === input.projectId && t.id === input.templateId); const template = templates.find((t) => t.projectId === input.projectId && t.id === input.templateId);
if (!template) throw new UpdateRunError(404, "template_not_found", "That template isn't in Semaphore (any more)."); if (!template) throw new UpdateRunError(404, "template_not_found", "That template isn't in Semaphore (any more).");
templateName = template.name; templateName = template.name;
if (restart) {
const again = templates.find((t) => t.projectId === restart.projectId && t.id === restart.templateId);
if (!again) throw new UpdateRunError(404, "restart_template_not_found", "The playbook that restarts the server isn't in Semaphore (any more).");
restartTemplateName = again.name;
}
} catch (err) { } catch (err) {
if (err instanceof UpdateRunError) throw err; if (err instanceof UpdateRunError) throw err;
throw new UpdateRunError(502, "semaphore_unreachable", `Couldn't reach Semaphore: ${err instanceof Error ? err.message : String(err)}`); throw new UpdateRunError(502, "semaphore_unreachable", `Couldn't reach Semaphore: ${err instanceof Error ? err.message : String(err)}`);
} }
// Quiet first, so nothing alerts about the server being worked on — and if the playbook can't be started the quiet is taken back. // Quiet first, so nothing alerts about the server being worked on — and if the playbook can't be started the quiet is taken back.
const started = await startOrExtendWindow("server", server.id, input.windowMinutes, `Update run: ${templateName}`, input.actor, now); const started = await startOrExtendWindow("server", server.id, input.windowMinutes, `${kind === "restart" ? "Restart" : "Update run"}: ${templateName}`, input.actor, now);
let task: SemaphoreTask; let task: SemaphoreTask;
try { try {
task = await adapter.runTemplate(input.projectId, input.templateId, { limit: input.limit, message: `Update run for ${server.name}, started in Homelab Manager by ${input.actor} (limited to ${input.limit})` }); task = await adapter.runTemplate(input.projectId, input.templateId, { limit: input.limit, message: `${kind === "restart" ? "Restart" : "Update run"} for ${server.name}, started in Homelab Manager by ${input.actor} (limited to ${input.limit})` });
} catch (err) { } catch (err) {
if (started && !started.extended) await db.update(maintenanceWindows).set({ endsAt: now.toISOString() }).where(eq(maintenanceWindows.id, started.window.id)); if (started && !started.extended) await db.update(maintenanceWindows).set({ endsAt: now.toISOString() }).where(eq(maintenanceWindows.id, started.window.id));
invalidateAlertsCache(); invalidateAlertsCache();
@@ -136,9 +157,13 @@ export async function startUpdateRun(input: StartInput, now: Date = new Date()):
startedBy: input.actor, startedBy: input.actor,
startedAt: now.toISOString(), startedAt: now.toISOString(),
updatesBefore: updates?.count ?? null, updatesBefore: updates?.count ?? null,
kind,
restartProjectId: restart?.projectId ?? null,
restartTemplateId: restart?.templateId ?? null,
restartTemplateName,
}) })
.returning(); .returning();
await recordAudit({ category: "update_run", action: "start", targetType: "server", targetId: server.id, detail: { run: run.id, template: templateName, limit: input.limit, task: task.id, by: input.actor } }); await recordAudit({ category: "update_run", action: "start", targetType: "server", targetId: server.id, detail: { run: run.id, kind, template: templateName, limit: input.limit, task: task.id, by: input.actor, ...(restartTemplateName ? { restartTemplate: restartTemplateName } : {}) } });
return run; return run;
} }
@@ -160,23 +185,114 @@ async function finish(run: UpdateRunRow, status: "done" | "failed" | "cancelled"
const [server] = await db.select({ name: servers.name }).from(servers).where(eq(servers.id, run.serverId)).limit(1); const [server] = await db.select({ name: servers.name }).from(servers).where(eq(servers.id, run.serverId)).limit(1);
await recordAudit({ category: "update_run", action: status === "done" ? "finished" : status, targetType: "server", targetId: run.serverId, detail: { run: run.id, template: run.templateName, message } }); await recordAudit({ category: "update_run", action: status === "done" ? "finished" : status, targetType: "server", targetId: run.serverId, detail: { run: run.id, template: run.templateName, message } });
if (status !== "cancelled") { if (status !== "cancelled") {
await notifyUpdateRunFinished({ serverName: server?.name ?? `#${run.serverId}`, templateName: run.templateName, ok: status === "done", message }).catch((err) => console.error("[updateRuns] couldn't send the result:", err)); await notifyUpdateRunFinished({ serverName: server?.name ?? `#${run.serverId}`, templateName: run.templateName, ok: status === "done", message, kind: run.kind === "restart" ? "restart" : "update" }).catch((err) => console.error("[updateRuns] couldn't send the result:", err));
} }
} }
const updatesWords = (n: number) => (n === 1 ? "1 update" : `${n} updates`); const updatesWords = (n: number) => (n === 1 ? "1 update" : `${n} updates`);
/** What is left of the updates, in a clause ("2 updates still waiting (10 updates installed)"). */
function updatesClause(before: number | null, after: number | null): string {
if (after === null) return "The server reported again, but doesn't say what is waiting";
let words = after === 0 ? "No updates are waiting now" : `${updatesWords(after)} still waiting`;
if (before !== null && before > after) words += ` (${updatesWords(before - after)} installed)`;
return words;
}
/** What a finished run says about the server, from the report that came after it. */ /** What a finished run says about the server, from the report that came after it. */
export function summarizeOutcome(before: number | null, after: number | null, reboot: boolean): string { export function summarizeOutcome(before: number | null, after: number | null, reboot: boolean): string {
const parts: string[] = []; return `${[updatesClause(before, after), reboot ? "a restart is still needed" : "no restart is needed"].join("; ")}.`;
if (after !== null) {
parts.push(after === 0 ? "No updates are waiting now" : `${updatesWords(after)} still waiting`);
if (before !== null && before > after) parts[0] += ` (${updatesWords(before - after)} installed)`;
} else {
parts.push("The server reported again, but doesn't say what is waiting");
} }
parts.push(reboot ? "a restart is still needed" : "no restart is needed");
return `${parts.join("; ")}.`; /**
* Whether the server has started since `sinceMs`, from the uptime in its last report (its start is the report's time minus its uptime): true or false, or null when it doesn't say.
* A start a little before `sinceMs` still counts (see BOOT_SLACK_MS).
*/
export function startedSince(server: { lastSeenAt: string | null; uptimeSeconds: number | null }, sinceMs: number): boolean | null {
if (!server.lastSeenAt || server.uptimeSeconds === null || server.uptimeSeconds === undefined) return null;
const seen = Date.parse(server.lastSeenAt);
if (Number.isNaN(seen)) return null;
return seen - server.uptimeSeconds * 1000 >= sinceMs - BOOT_SLACK_MS;
}
export interface RestartOutcome {
/** The server is taken to have started again. */
restarted: boolean;
/** Something said so (its uptime, or its "restart needed" flag clearing); false means it only reported again and nothing says whether it restarted. */
confirmed: boolean;
}
/** Did the restart happen? Its uptime says so where it is reported; failing that, the "restart needed" flag clearing; failing that, nothing can tell. */
export function restartOutcome(server: { lastSeenAt: string | null; uptimeSeconds: number | null }, updates: { rebootRequired: boolean } | null, sinceMs: number): RestartOutcome {
const booted = startedSince(server, sinceMs);
if (booted !== null) return { restarted: booted, confirmed: true };
if (updates) return { restarted: !updates.rebootRequired, confirmed: true };
return { restarted: true, confirmed: false };
}
/** What a finished restart says: for an update run, what is left of the updates and then the restart; for a restart on its own, only the restart. */
export function summarizeRestart(kind: UpdateRunKind, before: number | null, after: number | null, outcome: RestartOutcome, stillDue: boolean): string {
const lead = kind === "update" ? `${updatesClause(before, after)}; ` : "";
const subject = kind === "update" ? "the server" : "The server";
if (!outcome.restarted) {
return `${lead}${kind === "update" ? "the restart playbook finished" : "The playbook finished"}, but the server has not started again — check that the playbook really restarts it.`;
}
const how = outcome.confirmed ? "" : " (its uptime isn't reported, so that couldn't be confirmed)";
return `${lead}${subject} was restarted and is back${stillDue ? ", but says another restart is due" : ""}${how}.`;
}
/** Keeps the quiet up for at least `minutes` more; a window that has ended is made again, since the server is about to restart. Returns what changed on the run, if anything. */
async function keepQuiet(run: UpdateRunRow, minutes: number, now: Date): Promise<Partial<UpdateRunRow>> {
const until = new Date(now.getTime() + minutes * 60_000).toISOString();
if (run.windowId !== null) {
const [w] = await db.select().from(maintenanceWindows).where(eq(maintenanceWindows.id, run.windowId)).limit(1);
if (w && w.endsAt > now.toISOString()) {
if (w.endsAt < until) {
await db.update(maintenanceWindows).set({ endsAt: until }).where(eq(maintenanceWindows.id, w.id));
invalidateAlertsCache();
}
return {};
}
}
const started = await startOrExtendWindow("server", run.serverId, minutes, `Update run: restart`, run.startedBy, now);
invalidateAlertsCache();
return started ? { windowId: started.window.id, windowCreated: !started.extended } : {};
}
/** The updates are in and the server says a restart is due: run the restart playbook. If it can't be started, the run ends and says the updates are in. */
async function beginRestart(run: UpdateRunRow, server: typeof servers.$inferSelect, updates: { count: number; rebootRequired: boolean } | null, now: Date): Promise<void> {
const seen = { updatesAfter: updates?.count ?? null, rebootRequired: true, hostReportedAt: server.lastSeenAt };
const adapter = await semaphoreFor(run.integrationId);
const clause = updatesClause(run.updatesBefore, updates?.count ?? null);
if (!adapter || run.restartProjectId === null || run.restartTemplateId === null) {
await finish(run, "failed", `${clause}, but the restart could not be started: Semaphore isn't available. The server still needs a restart.`, now, seen);
return;
}
const quiet = await keepQuiet(run, RESTART_WINDOW_MINUTES, now);
let task: SemaphoreTask;
try {
task = await adapter.runTemplate(run.restartProjectId, run.restartTemplateId, { limit: run.hostLimit, message: `Restart of ${server.name} after update run #${run.id}, started in Homelab Manager (limited to ${run.hostLimit})` });
} catch (err) {
await finish(run, "failed", `${clause}, but the restart could not be started: ${err instanceof Error ? err.message : String(err)}. The server still needs a restart.`, now, seen);
return;
}
await db
.update(updateRuns)
.set({ ...seen, ...quiet, status: "restarting", restartTaskId: task.id, restartTaskStatus: task.status, restartStartedAt: now.toISOString(), pollErrors: 0, message: `${clause}. The server says a restart is due — restarting it with "${run.restartTemplateName ?? "the restart playbook"}".` })
.where(eq(updateRuns.id, run.id));
await recordAudit({ category: "update_run", action: "restart", targetType: "server", targetId: run.serverId, detail: { run: run.id, template: run.restartTemplateName, task: task.id } });
}
/** A restart is over and the server has reported again: did it start again? */
async function finishRestart(run: UpdateRunRow, server: typeof servers.$inferSelect, updates: ReturnType<typeof parsePendingUpdates>, sinceMs: number, now: Date): Promise<void> {
const outcome = restartOutcome(server, updates, sinceMs);
const stillDue = updates?.rebootRequired ?? false;
await finish(run, outcome.restarted ? "done" : "failed", summarizeRestart(run.kind === "restart" ? "restart" : "update", run.updatesBefore, updates?.count ?? null, outcome, outcome.restarted && stillDue), now, {
hostReportedAt: server.lastSeenAt,
updatesAfter: updates?.count ?? null,
rebootRequired: stillDue,
restarted: outcome.restarted,
});
} }
async function advance(run: UpdateRunRow, now: Date): Promise<void> { async function advance(run: UpdateRunRow, now: Date): Promise<void> {
@@ -185,11 +301,15 @@ async function advance(run: UpdateRunRow, now: Date): Promise<void> {
return; return;
} }
if (run.status === "running") { // The playbook is being followed: the update's, or (the second phase) the one that restarts the server.
if (run.status === "running" || run.status === "restarting") {
const restarting = run.status === "restarting";
const projectId = restarting ? run.restartProjectId : run.projectId;
const taskId = restarting ? run.restartTaskId : run.taskId;
const adapter = await semaphoreFor(run.integrationId); const adapter = await semaphoreFor(run.integrationId);
let task: SemaphoreTask | null = null; let task: SemaphoreTask | null = null;
try { try {
task = adapter && run.taskId !== null ? await adapter.getTask(run.projectId, run.taskId) : null; task = adapter && projectId !== null && taskId !== null ? await adapter.getTask(projectId, taskId) : null;
} catch { } catch {
task = null; task = null;
} }
@@ -199,22 +319,27 @@ async function advance(run: UpdateRunRow, now: Date): Promise<void> {
else await db.update(updateRuns).set({ pollErrors: errors }).where(eq(updateRuns.id, run.id)); else await db.update(updateRuns).set({ pollErrors: errors }).where(eq(updateRuns.id, run.id));
return; return;
} }
const lastStatus = restarting ? run.restartTaskStatus : run.taskStatus;
const statusField = restarting ? { restartTaskStatus: task.status } : { taskStatus: task.status };
if (!FINISHED_TASK.has(task.status)) { if (!FINISHED_TASK.has(task.status)) {
if (task.status !== run.taskStatus || run.pollErrors > 0) await db.update(updateRuns).set({ taskStatus: task.status, pollErrors: 0 }).where(eq(updateRuns.id, run.id)); if (task.status !== lastStatus || run.pollErrors > 0) await db.update(updateRuns).set({ ...statusField, pollErrors: 0 }).where(eq(updateRuns.id, run.id));
return; return;
} }
if (task.status === "success") { if (task.status === "success") {
const ended = task.end ? new Date(task.end).toISOString() : now.toISOString(); const ended = task.end ? new Date(task.end).toISOString() : now.toISOString();
await db.update(updateRuns).set({ status: "waiting_for_host", taskStatus: task.status, taskEndedAt: ended, pollErrors: 0, message: "The playbook finished. Waiting for the server to report again." }).where(eq(updateRuns.id, run.id)); if (restarting) await db.update(updateRuns).set({ status: "waiting_for_restart", ...statusField, restartEndedAt: ended, pollErrors: 0, message: "The restart playbook finished. Waiting for the server to come back and report." }).where(eq(updateRuns.id, run.id));
else await db.update(updateRuns).set({ status: "waiting_for_host", ...statusField, taskEndedAt: ended, pollErrors: 0, message: run.kind === "restart" ? "The restart playbook finished. Waiting for the server to come back and report." : "The playbook finished. Waiting for the server to report again." }).where(eq(updateRuns.id, run.id));
return; return;
} }
const why = task.status === "error" ? "The playbook failed in Semaphore" : task.status === "rejected" ? "The run was rejected in Semaphore" : "The run was stopped in Semaphore"; const why = task.status === "error" ? "The playbook failed in Semaphore" : task.status === "rejected" ? "The run was rejected in Semaphore" : "The run was stopped in Semaphore";
await finish(run, "failed", `${why} (run #${run.taskId}). The server is no longer under maintenance.`, now, { taskStatus: task.status }); if (restarting) await finish(run, "failed", `The updates are in, but the restart didn't happen. ${why} (run #${taskId}). The server still needs a restart, and is no longer under maintenance.`, now, statusField);
else await finish(run, "failed", `${why} (run #${taskId}). The server is no longer under maintenance.`, now, statusField);
return; return;
} }
// waiting_for_host: the agent has reported since the playbook ended (and, where it reports updates, those are newer than the run too). // Waiting for the server: the agent has reported since the playbook ended (and, where it reports updates, those are newer than the run too).
const ended = Date.parse(run.taskEndedAt ?? run.startedAt); const afterRestart = run.status === "waiting_for_restart";
const ended = Date.parse((afterRestart ? run.restartEndedAt : run.taskEndedAt) ?? (afterRestart ? run.restartStartedAt : null) ?? run.startedAt);
const [server] = await db.select().from(servers).where(eq(servers.id, run.serverId)).limit(1); const [server] = await db.select().from(servers).where(eq(servers.id, run.serverId)).limit(1);
if (!server) { if (!server) {
await finish(run, "failed", "The server was removed.", now); await finish(run, "failed", "The server was removed.", now);
@@ -224,6 +349,10 @@ async function advance(run: UpdateRunRow, now: Date): Promise<void> {
const updates = parsePendingUpdates(server.pendingUpdates); const updates = parsePendingUpdates(server.pendingUpdates);
const updatesFresh = updates ? Date.parse(updates.reportedAt) > ended : true; const updatesFresh = updates ? Date.parse(updates.reportedAt) > ended : true;
if (reported && updatesFresh) { if (reported && updatesFresh) {
if (afterRestart) await finishRestart(run, server, updates, Date.parse(run.restartStartedAt ?? run.startedAt), now);
else if (run.kind === "restart") await finishRestart(run, server, updates, Date.parse(run.startedAt), now);
else if (run.restartTemplateId !== null && updates?.rebootRequired) await beginRestart(run, server, updates, now);
else
await finish(run, "done", summarizeOutcome(run.updatesBefore, updates?.count ?? null, updates?.rebootRequired ?? false), now, { await finish(run, "done", summarizeOutcome(run.updatesBefore, updates?.count ?? null, updates?.rebootRequired ?? false), now, {
hostReportedAt: server.lastSeenAt, hostReportedAt: server.lastSeenAt,
updatesAfter: updates?.count ?? null, updatesAfter: updates?.count ?? null,
@@ -232,7 +361,7 @@ async function advance(run: UpdateRunRow, now: Date): Promise<void> {
return; return;
} }
if (now.getTime() - ended > HOST_WAIT_MS) { if (now.getTime() - ended > HOST_WAIT_MS) {
await finish(run, "failed", `The playbook finished, but the server hasn't reported again in ${Math.round(HOST_WAIT_MS / 60_000)} minutes. Check that it came back.`, now); await finish(run, "failed", `The ${afterRestart || run.kind === "restart" ? "restart playbook" : "playbook"} finished, but the server hasn't reported again in ${Math.round(HOST_WAIT_MS / 60_000)} minutes. Check that it came back.`, now);
} }
} }
+569
View File
@@ -0,0 +1,569 @@
/**
* Restarting a server after its updates: a second playbook, run when the server says a restart is due; the quiet kept up for it; the server waited for and checked
* to have started again (from its uptime, or its "restart needed" flag clearing); a restart on its own; and a patch plan that restarts what needs it.
* Semaphore is a stand-in that answers as told.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-restart-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, updateRuns, auditLog, maintenanceWindows, patchPlans, patchPlanRuns } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { startUpdateRun, advanceUpdateRuns, setSemaphoreForTests, startedSince, restartOutcome, summarizeRestart, UpdateRunError, HOST_WAIT_MS, RESTART_WINDOW_MINUTES, ACTIVE_STATUSES } = await import("../src/services/updateRuns.js");
const { startPlanRun, advancePlanRuns, parseItems } = await import("../src/services/patchPlans.js");
const { listActiveWindows } = await import("../src/services/maintenance.js");
const { getServerActivity } = await import("../src/services/serverActivity.js");
const { updateRunsRouter } = await import("../src/routes/updateRuns.js");
const { patchPlansRouter } = await import("../src/routes/patchPlans.js");
const { hashToken } = await import("../src/services/tokens.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
// ── a Semaphore that answers as told: two templates (updates, restart), and a task for every run ──
interface Task {
id: number;
status: string;
created: string | null;
start: string | null;
end: string | null;
}
const fake = { runs: [] as { projectId: number; templateId: number; limit?: string; message?: string }[], tasks: new Map<number, Task>(), next: 100, refuseTemplate: null as number | null };
const resetFake = () => {
fake.runs = [];
fake.tasks = new Map();
fake.next = 100;
fake.refuseTemplate = null;
};
const setTask = (id: number, patch: Partial<Task>) => fake.tasks.set(id, { ...fake.tasks.get(id)!, ...patch });
setSemaphoreForTests(async (integrationId) =>
integrationId === 9
? {
ping: async () => ({ ok: true }),
listTemplatesWithStatus: async () => [],
checkTemplates: async () => ({
templates: [
{ id: 7, projectId: 2, projectName: "Homelab", name: "Apply updates", playbook: "updates.yml", lastTask: null },
{ id: 8, projectId: 2, projectName: "Homelab", name: "Restart server", playbook: "restart.yml", lastTask: null },
],
failedProjectIds: [],
}),
runTemplate: async (projectId, templateId, options) => {
if (fake.refuseTemplate === templateId) throw new Error("Semaphore is having a moment");
fake.runs.push({ projectId, templateId, limit: options?.limit, message: options?.message });
const task = { id: fake.next++, status: "waiting", created: null, start: null, end: null };
fake.tasks.set(task.id, task);
return { ...task } as never;
},
getTask: async (_p, id) => ({ ...fake.tasks.get(id)! }) as never,
}
: null,
);
const T0 = new Date("2026-10-17T12:00:00.000Z");
const at = (minutes: number) => new Date(T0.getTime() + minutes * 60_000);
const iso = (minutes: number) => at(minutes).toISOString();
const RESTART = { projectId: 2, templateId: 8 };
const input = (serverId: number, extra: object = {}) => ({ serverId, integrationId: 9, projectId: 2, templateId: 7, limit: "elsa", windowMinutes: 90, actor: "Olle", ...extra });
const runRow = async (id: number) => (await db.select().from(updateRuns).where(eq(updateRuns.id, id)))[0];
const updatesJson = (count: number, reboot: boolean, minutes: number) => JSON.stringify({ manager: "apt", count, security: 0, rebootRequired: reboot, reportedAt: iso(minutes) });
describe("restarting after the updates", () => {
let serverId = 0;
before(async () => {
await db.insert(users).values([{ oidcSub: "admin", name: "Ada", role: "admin" }]);
const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("tok"), apiTokenPrefix: "t", lastSeenAt: iso(0), uptimeSeconds: 86_400 * 30, pendingUpdates: updatesJson(12, false, 0) }).returning();
serverId = s.id;
});
beforeEach(async () => {
resetFake();
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
await db.update(servers).set({ lastSeenAt: iso(0), uptimeSeconds: 86_400 * 30, pendingUpdates: updatesJson(12, false, 0) }).where(eq(servers.id, serverId));
});
after(() => {
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
/** Reports as the agent would: when, what is waiting, whether a restart is due, and how long it has been up. */
const report = (minutes: number, count: number, reboot: boolean, uptimeSeconds: number | null) =>
db.update(servers).set({ lastSeenAt: iso(minutes), uptimeSeconds, pendingUpdates: updatesJson(count, reboot, minutes) }).where(eq(servers.id, serverId));
/** Runs an update with a restart playbook to the point where the server says a restart is due. */
async function toRestart(start = 0) {
const run = await startUpdateRun(input(serverId, { restart: RESTART }), at(start));
setTask(run.taskId!, { status: "success", end: iso(start + 8) });
await advanceUpdateRuns(at(start + 9));
await report(start + 14, 2, true, 86_400 * 30 + 840);
await advanceUpdateRuns(at(start + 15));
return run;
}
test("an update run remembers the restart playbook, says so in its history, and refuses one that isn't in Semaphore", async () => {
const run = await startUpdateRun(input(serverId, { restart: RESTART }), T0);
assert.equal(run.kind, "update");
assert.deepEqual([run.restartProjectId, run.restartTemplateId, run.restartTemplateName], [2, 8, "Restart server"]);
assert.equal(fake.runs.length, 1, "only the update playbook is started now");
const events = await getServerActivity(serverId, at(1));
assert.ok(events.some((e) => e.title === "Update run started" && /then a restart with Restart server if one is due/.test(e.detail ?? "")));
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
fake.runs = [];
await assert.rejects(startUpdateRun(input(serverId, { restart: { projectId: 2, templateId: 999 } }), T0), (e: unknown) => e instanceof UpdateRunError && e.status === 404 && e.code === "restart_template_not_found");
assert.equal(fake.runs.length, 0, "nothing was started");
assert.equal((await listActiveWindows(T0)).length, 0, "and no quiet was left behind");
});
test("when the server says a restart is due, the restart playbook runs, is followed, and the server is checked to have started again", async () => {
const run = await toRestart();
let r = await runRow(run.id);
assert.equal(r.status, "restarting");
assert.equal(r.updatesAfter, 2);
assert.equal(r.restartStartedAt, iso(15));
assert.deepEqual(fake.runs.map((x) => [x.templateId, x.limit]), [[7, "elsa"], [8, "elsa"]], "the same server, and only that one");
assert.match(fake.runs[1].message!, /Restart of elsa after update run #\d+/);
assert.match(r.message!, /2 updates still waiting \(10 updates installed\)\. The server says a restart is due — restarting it with "Restart server"/);
assert.ok(ACTIVE_STATUSES.includes(r.status as never), "it counts as under way");
assert.ok((await db.select().from(auditLog)).some((a) => a.category === "update_run" && a.action === "restart"));
// the quiet is kept up for the restart (it was the update's 90 minutes; the restart's own is shorter, so it stays at 90)
const windows = await listActiveWindows(at(16));
assert.equal(windows.length, 1);
assert.equal(windows[0].id, r.windowId);
setTask(r.restartTaskId!, { status: "running" });
await advanceUpdateRuns(at(17));
r = await runRow(run.id);
assert.equal(r.status, "restarting");
assert.equal(r.restartTaskStatus, "running");
assert.equal(r.taskStatus, "success", "the update's own status is left as it was");
setTask(r.restartTaskId!, { status: "success", end: iso(19) });
await advanceUpdateRuns(at(20));
r = await runRow(run.id);
assert.equal(r.status, "waiting_for_restart");
assert.equal(r.restartEndedAt, iso(19));
assert.match(r.message!, /Waiting for the server to come back/);
await advanceUpdateRuns(at(22));
assert.equal((await runRow(run.id)).status, "waiting_for_restart", "the report from before the restart ended doesn't count");
await report(23, 2, false, 150); // up for two and a half minutes: it started at 20:30, after the restart began
await advanceUpdateRuns(at(24));
r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.restarted, true);
assert.equal(r.rebootRequired, false);
assert.equal(r.message, "2 updates still waiting (10 updates installed); the server was restarted and is back.");
assert.equal((await listActiveWindows(at(25))).length, 0, "the window it made is over");
});
test("a server that says no restart is due is not restarted", async () => {
const run = await startUpdateRun(input(serverId, { restart: RESTART }), T0);
setTask(run.taskId!, { status: "success", end: iso(8) });
await advanceUpdateRuns(at(9));
await report(14, 0, false, 600);
await advanceUpdateRuns(at(15));
const r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.restarted, false);
assert.equal(r.message, "No updates are waiting now (12 updates installed); no restart is needed.");
assert.equal(fake.runs.length, 1, "the restart playbook was never run");
});
test("without a restart playbook a restart that is due is only said to be, as before", async () => {
const run = await startUpdateRun(input(serverId), T0);
setTask(run.taskId!, { status: "success", end: iso(8) });
await advanceUpdateRuns(at(9));
await report(14, 2, true, 840);
await advanceUpdateRuns(at(15));
const r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.message, "2 updates still waiting (10 updates installed); a restart is still needed.");
assert.equal(fake.runs.length, 1);
});
test("if the restart playbook fails, the run says the updates are in and the server still needs a restart; the quiet ends", async () => {
const run = await toRestart();
const { restartTaskId } = await runRow(run.id);
setTask(restartTaskId!, { status: "error" });
await advanceUpdateRuns(at(18));
const r = await runRow(run.id);
assert.equal(r.status, "failed");
assert.match(r.message!, /The updates are in, but the restart didn't happen\. The playbook failed in Semaphore \(run #\d+\)\. The server still needs a restart/);
assert.equal((await listActiveWindows(at(19))).length, 0);
});
test("if the restart playbook can't be started, the run ends saying so, with the updates counted", async () => {
fake.refuseTemplate = 8;
const run = await startUpdateRun(input(serverId, { restart: RESTART }), T0);
setTask(run.taskId!, { status: "success", end: iso(8) });
await advanceUpdateRuns(at(9));
await report(14, 2, true, 840);
await advanceUpdateRuns(at(15));
const r = await runRow(run.id);
assert.equal(r.status, "failed");
assert.equal(r.message, "2 updates still waiting (10 updates installed), but the restart could not be started: Semaphore is having a moment. The server still needs a restart.");
assert.equal(r.rebootRequired, true);
assert.equal((await listActiveWindows(at(16))).length, 0);
});
test("a playbook that finished without the server restarting is a failure, from its uptime — and a restart flag that never clears is too", async () => {
// reported again, but it has been up since long before the restart began
let run = await toRestart();
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(19) });
await advanceUpdateRuns(at(20));
await report(23, 2, false, 86_400 * 30 + 1380);
await advanceUpdateRuns(at(24));
let r = await runRow(run.id);
assert.equal(r.status, "failed");
assert.equal(r.restarted, false);
assert.equal(r.message, "2 updates still waiting (10 updates installed); the restart playbook finished, but the server has not started again — check that the playbook really restarts it.");
// no uptime in its reports: the restart flag is what there is
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
run = await toRestart(100);
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(119) });
await advanceUpdateRuns(at(120));
await report(123, 2, true, null);
await advanceUpdateRuns(at(124));
r = await runRow(run.id);
assert.equal(r.status, "failed", "still says a restart is due");
assert.match(r.message!, /has not started again/);
});
test("with no uptime to go by, the flag clearing is enough; with neither, it is taken to have restarted and says it couldn't be confirmed", async () => {
let run = await toRestart();
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(19) });
await advanceUpdateRuns(at(20));
await report(23, 2, false, null);
await advanceUpdateRuns(at(24));
let r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.message, "2 updates still waiting (10 updates installed); the server was restarted and is back.");
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
run = await toRestart(100);
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(119) });
await advanceUpdateRuns(at(120));
await db.update(servers).set({ lastSeenAt: iso(123), uptimeSeconds: null, pendingUpdates: null }).where(eq(servers.id, serverId));
await advanceUpdateRuns(at(124));
r = await runRow(run.id);
assert.equal(r.status, "done");
assert.match(r.message!, /was restarted and is back \(its uptime isn't reported, so that couldn't be confirmed\)\.$/);
});
test("a restarted server that says another restart is due is done, with that said", async () => {
const run = await toRestart();
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(19) });
await advanceUpdateRuns(at(20));
await report(23, 0, true, 150);
await advanceUpdateRuns(at(24));
const r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.rebootRequired, true);
assert.match(r.message!, /the server was restarted and is back, but says another restart is due\.$/);
});
test("a server that doesn't come back from its restart is given up on, and the quiet ends", async () => {
const run = await toRestart();
setTask((await runRow(run.id)).restartTaskId!, { status: "success", end: iso(19) });
await advanceUpdateRuns(at(20));
await advanceUpdateRuns(at(19 + HOST_WAIT_MS / 60_000 + 1));
const r = await runRow(run.id);
assert.equal(r.status, "failed");
assert.match(r.message!, /The restart playbook finished, but the server hasn't reported again in 30 minutes/);
assert.equal((await listActiveWindows(at(60))).length, 0);
});
test("the quiet is made again if it ended before the restart began, and is not shortened if it is longer than the restart needs", async () => {
// a window longer than the restart needs (somebody made it longer while the update ran) is not cut short
const run = await startUpdateRun(input(serverId, { restart: RESTART }), T0);
await db.update(maintenanceWindows).set({ endsAt: iso(600) }).where(eq(maintenanceWindows.id, run.windowId!));
setTask(run.taskId!, { status: "success", end: iso(8) });
await advanceUpdateRuns(at(9));
await report(14, 2, true, 840);
await advanceUpdateRuns(at(15));
assert.equal((await runRow(run.id)).status, "restarting");
assert.equal((await listActiveWindows(at(16)))[0].endsAt, iso(600), "left as long as it was");
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
// one that has already run out is made again, for the restart
const second = await startUpdateRun(input(serverId, { restart: RESTART, windowMinutes: 15 }), at(200));
setTask(second.taskId!, { status: "success", end: iso(208) });
await advanceUpdateRuns(at(209));
await report(214, 2, true, 600);
await advanceUpdateRuns(at(216)); // the 15 minutes ran out at 215
const r = await runRow(second.id);
assert.equal(r.status, "restarting");
const windows = await listActiveWindows(at(217));
assert.equal(windows.length, 1, "quiet again for the restart");
assert.equal(Date.parse(windows[0].endsAt) - at(216).getTime(), RESTART_WINDOW_MINUTES * 60_000);
assert.equal(r.windowId, windows[0].id);
});
});
describe("restarting a server on its own", () => {
let serverId = 0;
before(async () => {
const [s] = await db.insert(servers).values({ name: "anna", apiTokenHash: hashToken("tok2"), apiTokenPrefix: "t2", lastSeenAt: iso(0), uptimeSeconds: 86_400 * 10, pendingUpdates: updatesJson(0, true, 0) }).returning();
serverId = s.id;
});
beforeEach(async () => {
resetFake();
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
await db.update(servers).set({ lastSeenAt: iso(0), uptimeSeconds: 86_400 * 10, pendingUpdates: updatesJson(0, true, 0) }).where(eq(servers.id, serverId));
});
test("a restart run runs the playbook it was given, follows it, and checks the server started again", async () => {
const run = await startUpdateRun(input(serverId, { kind: "restart", templateId: 8, limit: "anna", restart: RESTART }), T0);
assert.equal(run.kind, "restart");
assert.equal(run.templateName, "Restart server");
assert.equal(run.restartTemplateId, null, "a restart doesn't restart again");
assert.deepEqual(fake.runs.map((x) => [x.templateId, x.limit]), [[8, "anna"]]);
assert.match(fake.runs[0].message!, /^Restart for anna, started in Homelab Manager by Olle/);
const windows = await listActiveWindows(T0);
assert.equal(windows[0].reason, "Restart: Restart server");
setTask(run.taskId!, { status: "success", end: iso(4) });
await advanceUpdateRuns(at(5));
let r = await runRow(run.id);
assert.equal(r.status, "waiting_for_host");
assert.match(r.message!, /The restart playbook finished\. Waiting for the server to come back/);
await db.update(servers).set({ lastSeenAt: iso(9), uptimeSeconds: 200, pendingUpdates: updatesJson(0, false, 9) }).where(eq(servers.id, serverId));
await advanceUpdateRuns(at(10));
r = await runRow(run.id);
assert.equal(r.status, "done");
assert.equal(r.restarted, true);
assert.equal(r.message, "The server was restarted and is back.");
assert.equal((await listActiveWindows(at(11))).length, 0);
});
test("a restart that did not happen fails, in its own words", async () => {
const run = await startUpdateRun(input(serverId, { kind: "restart", templateId: 8, limit: "anna" }), T0);
setTask(run.taskId!, { status: "success", end: iso(4) });
await advanceUpdateRuns(at(5));
await db.update(servers).set({ lastSeenAt: iso(9), uptimeSeconds: 86_400 * 10 + 540, pendingUpdates: updatesJson(0, true, 9) }).where(eq(servers.id, serverId));
await advanceUpdateRuns(at(10));
const r = await runRow(run.id);
assert.equal(r.status, "failed");
assert.equal(r.message, "The playbook finished, but the server has not started again — check that the playbook really restarts it.");
});
test("it is in the server's history as a restart", async () => {
await startUpdateRun(input(serverId, { kind: "restart", templateId: 8, limit: "anna" }), T0);
const events = await getServerActivity(serverId, at(1));
assert.ok(events.some((e) => e.title === "Restart started"));
});
});
describe("working out whether a server has started again", () => {
const since = T0.getTime();
test("from its uptime: the report's time minus the uptime is when it started", () => {
assert.equal(startedSince({ lastSeenAt: iso(10), uptimeSeconds: 300 }, since), true, "started at 12:05");
assert.equal(startedSince({ lastSeenAt: iso(10), uptimeSeconds: 86_400 }, since), false);
assert.equal(startedSince({ lastSeenAt: iso(10), uptimeSeconds: 10 * 60 + 60 }, since), true, "a minute before the restart began still counts");
assert.equal(startedSince({ lastSeenAt: iso(10), uptimeSeconds: 10 * 60 + 600 }, since), false, "ten minutes before does not");
assert.equal(startedSince({ lastSeenAt: iso(10), uptimeSeconds: null }, since), null);
assert.equal(startedSince({ lastSeenAt: null, uptimeSeconds: 100 }, since), null);
});
test("the uptime is believed over the flag; the flag over nothing", () => {
assert.deepEqual(restartOutcome({ lastSeenAt: iso(10), uptimeSeconds: 300 }, { rebootRequired: true }, since), { restarted: true, confirmed: true });
assert.deepEqual(restartOutcome({ lastSeenAt: iso(10), uptimeSeconds: 86_400 }, { rebootRequired: false }, since), { restarted: false, confirmed: true });
assert.deepEqual(restartOutcome({ lastSeenAt: iso(10), uptimeSeconds: null }, { rebootRequired: false }, since), { restarted: true, confirmed: true });
assert.deepEqual(restartOutcome({ lastSeenAt: iso(10), uptimeSeconds: null }, { rebootRequired: true }, since), { restarted: false, confirmed: true });
assert.deepEqual(restartOutcome({ lastSeenAt: iso(10), uptimeSeconds: null }, null, since), { restarted: true, confirmed: false });
});
test("the words for each case", () => {
const yes = { restarted: true, confirmed: true };
assert.equal(summarizeRestart("update", 12, 0, yes, false), "No updates are waiting now (12 updates installed); the server was restarted and is back.");
assert.equal(summarizeRestart("restart", 12, 0, yes, false), "The server was restarted and is back.");
assert.equal(summarizeRestart("restart", null, null, yes, true), "The server was restarted and is back, but says another restart is due.");
assert.equal(summarizeRestart("restart", null, null, { restarted: false, confirmed: true }, true), "The playbook finished, but the server has not started again — check that the playbook really restarts it.");
});
});
describe("a patch plan that restarts what needs it", () => {
let aId = 0;
let bId = 0;
const plan = (extra: object = {}) => ({
name: "Sunday night",
enabled: true,
days: JSON.stringify([0]),
startTime: "02:00",
timezone: "UTC",
tags: "[]",
serverIds: JSON.stringify([aId, bId]),
integrationId: 9,
projectId: 2,
templateId: 7,
templateName: "Apply updates",
silenceDependents: false,
windowMinutes: 90,
...extra,
});
before(async () => {
const [a] = await db.insert(servers).values({ name: "plan-a", apiTokenHash: hashToken("pa"), apiTokenPrefix: "pa", lastSeenAt: iso(0), uptimeSeconds: 86_400, pendingUpdates: updatesJson(3, false, 0) }).returning();
const [b] = await db.insert(servers).values({ name: "plan-b", apiTokenHash: hashToken("pb"), apiTokenPrefix: "pb", lastSeenAt: iso(0), uptimeSeconds: 86_400, pendingUpdates: updatesJson(0, true, 0) }).returning();
aId = a.id;
bId = b.id;
});
beforeEach(async () => {
resetFake();
await db.update(servers).set({ lastSeenAt: iso(0), uptimeSeconds: 86_400, pendingUpdates: updatesJson(3, false, 0) }).where(eq(servers.id, aId));
await db.update(servers).set({ lastSeenAt: iso(0), uptimeSeconds: 86_400, pendingUpdates: updatesJson(0, true, 0) }).where(eq(servers.id, bId));
await db.delete(patchPlanRuns);
await db.delete(patchPlans);
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
});
test("each server is updated with the restart playbook behind it, and a server that only needs a restart is restarted instead of left alone", async () => {
const [p] = await db.insert(patchPlans).values(plan({ restartProjectId: 2, restartTemplateId: 8, restartTemplateName: "Restart server" })).returning();
const run = await startPlanRun(p.id, { trigger: "manual", actor: "Olle", now: T0 });
await advancePlanRuns(at(1));
const first = (await db.select().from(updateRuns).orderBy(updateRuns.id))[0];
assert.equal(first.kind, "update");
assert.equal(first.restartTemplateId, 8, "the update run has the restart playbook behind it");
assert.equal(first.serverId, aId);
// plan-a finishes (no restart due)
setTask(first.taskId!, { status: "success", end: iso(5) });
await advanceUpdateRuns(at(6));
await db.update(servers).set({ lastSeenAt: iso(8), uptimeSeconds: 86_400 + 480, pendingUpdates: updatesJson(0, false, 8) }).where(eq(servers.id, aId));
await advanceUpdateRuns(at(9));
await advancePlanRuns(at(10));
// plan-b has nothing waiting but needs a restart: restarted, not skipped
const runs = await db.select().from(updateRuns).orderBy(updateRuns.id);
assert.equal(runs.length, 2);
assert.equal(runs[1].serverId, bId);
assert.equal(runs[1].kind, "restart");
assert.equal(runs[1].templateId, 8);
const items = parseItems((await db.select().from(patchPlanRuns).where(eq(patchPlanRuns.id, run.id)))[0].items);
assert.equal(items[1].status, "running");
assert.match(items[1].message!, /Nothing to update, but it needs a restart: the restart has started\./);
});
test("without a restart playbook, a server with nothing waiting is left alone, as before", async () => {
const [p] = await db.insert(patchPlans).values(plan()).returning();
const run = await startPlanRun(p.id, { trigger: "manual", actor: "Olle", now: T0 });
await advancePlanRuns(at(1));
const first = (await db.select().from(updateRuns))[0];
assert.equal(first.restartTemplateId, null);
setTask(first.taskId!, { status: "success", end: iso(5) });
await advanceUpdateRuns(at(6));
await db.update(servers).set({ lastSeenAt: iso(8), pendingUpdates: updatesJson(0, false, 8) }).where(eq(servers.id, aId));
await advanceUpdateRuns(at(9));
await advancePlanRuns(at(10));
const items = parseItems((await db.select().from(patchPlanRuns).where(eq(patchPlanRuns.id, run.id)))[0].items);
assert.equal(items[1].status, "skipped");
assert.equal(items[1].message, "No updates were waiting.");
assert.equal((await db.select().from(updateRuns)).length, 1);
});
});
describe("the routes", () => {
let app: Server;
let base = "";
let who = "operator";
let serverId = 0;
const call = (method: string, path: string, body?: unknown) => fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
before(async () => {
await db.insert(users).values([
{ oidcSub: "operator", name: "Olle", role: "operator" },
{ oidcSub: "root", name: "Rut", role: "admin" },
]);
const [s] = await db.insert(servers).values({ name: "zoe", apiTokenHash: hashToken("tok3"), apiTokenPrefix: "t3", lastSeenAt: iso(0), uptimeSeconds: 100 }).returning();
serverId = s.id;
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/update-runs", updateRunsRouter);
a.use("/api/patch-plans", patchPlansRouter);
app = await new Promise<Server>((resolve) => {
const x = a.listen(0, "127.0.0.1", () => resolve(x));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
beforeEach(async () => {
resetFake();
await db.delete(updateRuns);
await db.delete(maintenanceWindows);
await db.delete(patchPlans);
});
after(() => app.close());
const body = (extra: object = {}) => ({ serverId, integrationId: 9, projectId: 2, templateId: 7, limit: "zoe", windowMinutes: 60, ...extra });
test("an operator can start an update with a restart behind it, or a restart on its own, and the run says which", async () => {
const res = await call("POST", "/update-runs", body({ restart: RESTART }));
assert.equal(res.status, 201);
const { run } = await res.json();
assert.equal(run.kind, "update");
assert.equal(run.restartTemplate, "Restart server");
assert.equal(run.restarted, false);
await db.delete(updateRuns);
const alone = await (await call("POST", "/update-runs", body({ kind: "restart", templateId: 8 }))).json();
assert.equal(alone.run.kind, "restart");
assert.equal(alone.run.template, "Restart server");
assert.equal(alone.run.restartTemplate, null);
await db.delete(updateRuns);
assert.equal((await call("POST", "/update-runs", body({ kind: "reboot" }))).status, 400);
assert.equal((await call("POST", "/update-runs", body({ restart: { projectId: 2 } }))).status, 400);
const missing = await call("POST", "/update-runs", body({ restart: { projectId: 2, templateId: 999 } }));
assert.equal(missing.status, 404);
assert.equal((await missing.json()).error, "restart_template_not_found");
});
test("a patch plan keeps its restart playbook, all three parts or none, and an administrator's save is what changes it", async () => {
const plan = (extra: object = {}) => ({ name: "Nights", days: [0], startTime: "02:00", timezone: "UTC", tags: ["lab"], integrationId: 9, projectId: 2, templateId: 7, templateName: "Apply updates", ...extra });
who = "operator";
assert.equal((await call("POST", "/patch-plans", plan())).status, 403);
who = "root";
const made = await call("POST", "/patch-plans", plan({ restartProjectId: 2, restartTemplateId: 8, restartTemplateName: "Restart server" }));
assert.equal(made.status, 201);
const { plan: saved } = await made.json();
assert.deepEqual([saved.restartProjectId, saved.restartTemplateId, saved.restartTemplateName], [2, 8, "Restart server"]);
const half = await call("POST", "/patch-plans", plan({ name: "Half", restartTemplateId: 8 }));
assert.equal(half.status, 400);
assert.match((await half.json()).message, /needs its project, its template and its name — or none of them/);
const plain = await (await call("PUT", `/patch-plans/${saved.id}`, plan())).json();
assert.deepEqual([plain.plan.restartProjectId, plain.plan.restartTemplateId, plain.plan.restartTemplateName], [null, null, null], "saving without one takes it away");
who = "operator";
});
});
+19 -1
View File
@@ -858,7 +858,7 @@ export interface UpdateRun {
taskId: number | null; taskId: number | null;
/** Semaphore's own word for the run. */ /** Semaphore's own word for the run. */
taskStatus: string | null; taskStatus: string | null;
status: "running" | "waiting_for_host" | "done" | "failed" | "cancelled"; status: "running" | "waiting_for_host" | "restarting" | "waiting_for_restart" | "done" | "failed" | "cancelled";
active: boolean; active: boolean;
message: string | null; message: string | null;
startedBy: string | null; startedBy: string | null;
@@ -868,6 +868,16 @@ export interface UpdateRun {
updatesBefore: number | null; updatesBefore: number | null;
updatesAfter: number | null; updatesAfter: number | null;
rebootRequired: boolean; rebootRequired: boolean;
/** "restart": a restart on its own (the template is the restart playbook). */
kind: "update" | "restart";
/** The playbook that restarts the server once its updates are in, if one was chosen. */
restartTemplate: string | null;
restartTaskId: number | null;
restartTaskStatus: string | null;
/** ISO — when the restart playbook was started (after the updates). */
restartedAt: string | null;
/** The server was seen to have started again. */
restarted: boolean;
} }
export interface UpdateRunInput { export interface UpdateRunInput {
@@ -879,6 +889,10 @@ export interface UpdateRunInput {
windowMinutes: number; windowMinutes: number;
/** Somebody is streaming from the machine and the person has seen the warning. */ /** Somebody is streaming from the machine and the person has seen the warning. */
startAnyway?: boolean; startAnyway?: boolean;
/** "restart": the template is a playbook that only restarts the server. */
kind?: "update" | "restart";
/** Restart the server with this playbook (same Semaphore) once the updates are in, if it says a restart is due. */
restart?: { projectId: number; templateId: number };
} }
export interface UpdatesServer { export interface UpdatesServer {
@@ -1216,6 +1230,10 @@ export interface PatchPlan {
onlyWithUpdates: boolean; onlyWithUpdates: boolean;
silenceDependents: boolean; silenceDependents: boolean;
windowMinutes: number; windowMinutes: number;
/** The playbook that restarts a server that says a restart is due after its updates (in the same Semaphore); null = none. */
restartProjectId: number | null;
restartTemplateId: number | null;
restartTemplateName: string | null;
createdBy: string | null; createdBy: string | null;
/** The servers it covers right now. */ /** The servers it covers right now. */
servers: { id: number; name: string }[]; servers: { id: number; name: string }[];
+48 -6
View File
@@ -85,9 +85,14 @@ interface Draft {
onlyWithUpdates: boolean; onlyWithUpdates: boolean;
silenceDependents: boolean; silenceDependents: boolean;
windowMinutes: number; windowMinutes: number;
/** The playbook that restarts a server that says a restart is due after its updates ("" = none); a template key like the update one. */
restart: string;
} }
const EMPTY: Draft = { id: null, name: "", enabled: true, days: [0], startTime: "02:00", timezone: browserZone(), tags: [], serverIds: [], template: "", limitFrom: "name", stopOnFailure: true, onlyWithUpdates: true, silenceDependents: true, windowMinutes: 90 }; const EMPTY: Draft = { id: null, name: "", enabled: true, days: [0], startTime: "02:00", timezone: browserZone(), tags: [], serverIds: [], template: "", limitFrom: "name", stopOnFailure: true, onlyWithUpdates: true, silenceDependents: true, windowMinutes: 90, restart: "" };
/** A plan's restart playbook as a template key ("" = none). */
const restartKeyOf = (p: PatchPlan): string => (p.restartTemplateId !== null && p.restartProjectId !== null ? `${p.integrationId}:${p.restartProjectId}:${p.restartTemplateId}` : "");
function Progress({ run, canStop, onStop }: { run: PatchPlanRun; canStop: boolean; onStop: () => void }) { function Progress({ run, canStop, onStop }: { run: PatchPlanRun; canStop: boolean; onStop: () => void }) {
return ( return (
@@ -176,7 +181,7 @@ export default function PatchPlans({ user }: { user: CurrentUser }) {
}, [draft, data]); }, [draft, data]);
function edit(p: PatchPlan) { function edit(p: PatchPlan) {
setDraft({ id: p.id, name: p.name, enabled: p.enabled, days: p.days, startTime: p.startTime, timezone: p.timezone, tags: p.tags, serverIds: p.serverIds, template: `${p.integrationId}:${p.projectId}:${p.templateId}`, limitFrom: p.limitFrom, stopOnFailure: p.stopOnFailure, onlyWithUpdates: p.onlyWithUpdates, silenceDependents: p.silenceDependents, windowMinutes: p.windowMinutes }); setDraft({ id: p.id, name: p.name, enabled: p.enabled, days: p.days, startTime: p.startTime, timezone: p.timezone, tags: p.tags, serverIds: p.serverIds, template: `${p.integrationId}:${p.projectId}:${p.templateId}`, limitFrom: p.limitFrom, stopOnFailure: p.stopOnFailure, onlyWithUpdates: p.onlyWithUpdates, silenceDependents: p.silenceDependents, windowMinutes: p.windowMinutes, restart: restartKeyOf(p) });
} }
async function save(e: React.FormEvent) { async function save(e: React.FormEvent) {
@@ -185,10 +190,26 @@ export default function PatchPlans({ user }: { user: CurrentUser }) {
// Editing a plan whose playbook was not looked up still keeps what it has. // Editing a plan whose playbook was not looked up still keeps what it has.
const picked = templates?.find((t) => t.key === draft.template); const picked = templates?.find((t) => t.key === draft.template);
const existing = draft.id !== null ? data?.plans.find((p) => p.id === draft.id) : undefined; const existing = draft.id !== null ? data?.plans.find((p) => p.id === draft.id) : undefined;
// The restart playbook: none, one picked from the list, or the one the plan already has (when the list could not be read).
const restartPicked = draft.restart ? templates?.find((t) => t.key === draft.restart) : undefined;
const restartKept = existing && draft.restart !== "" && draft.restart === restartKeyOf(existing) ? existing : undefined;
if (draft.restart !== "" && !restartPicked && !restartKept) {
setError("Choose the restart playbook again, or none.");
return;
}
if (restartPicked && restartPicked.integration.id !== (picked?.integration.id ?? existing?.integrationId)) {
setError("The restart playbook has to be in the same Semaphore as the one that applies updates.");
return;
}
const restartFields = restartPicked
? { restartProjectId: restartPicked.template.projectId, restartTemplateId: restartPicked.template.id, restartTemplateName: restartPicked.template.name }
: restartKept
? { restartProjectId: restartKept.restartProjectId, restartTemplateId: restartKept.restartTemplateId, restartTemplateName: restartKept.restartTemplateName }
: { restartProjectId: null, restartTemplateId: null, restartTemplateName: null };
const input: PatchPlanInput | null = picked const input: PatchPlanInput | null = picked
? { name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: picked.integration.id, projectId: picked.template.projectId, templateId: picked.template.id, templateName: picked.template.name, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes } ? { ...restartFields, name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: picked.integration.id, projectId: picked.template.projectId, templateId: picked.template.id, templateName: picked.template.name, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes }
: existing && draft.template === `${existing.integrationId}:${existing.projectId}:${existing.templateId}` : existing && draft.template === `${existing.integrationId}:${existing.projectId}:${existing.templateId}`
? { name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: existing.integrationId, projectId: existing.projectId, templateId: existing.templateId, templateName: existing.templateName, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes } ? { ...restartFields, name: draft.name.trim(), enabled: draft.enabled, days: draft.days, startTime: draft.startTime, timezone: draft.timezone, tags: draft.tags, serverIds: draft.serverIds, integrationId: existing.integrationId, projectId: existing.projectId, templateId: existing.templateId, templateName: existing.templateName, limitFrom: draft.limitFrom, stopOnFailure: draft.stopOnFailure, onlyWithUpdates: draft.onlyWithUpdates, silenceDependents: draft.silenceDependents, windowMinutes: draft.windowMinutes }
: null; : null;
if (!input) { if (!input) {
setError("Choose the playbook that applies updates."); setError("Choose the playbook that applies updates.");
@@ -211,7 +232,7 @@ export default function PatchPlans({ user }: { user: CurrentUser }) {
async function runNow(p: PatchPlan) { async function runNow(p: PatchPlan) {
const names = p.servers.map((s) => s.name); const names = p.servers.map((s) => s.name);
if (!(await confirmDialog({ title: `Run ${p.name} now?`, message: `It updates ${names.length === 1 ? names[0] : `${names.length} servers (${names.slice(0, 5).join(", ")}${names.length > 5 ? " …" : ""})`}, one at a time${p.stopOnFailure ? ", and stops at the first failure" : ""}. ${p.onlyWithUpdates ? "Servers with nothing waiting are left alone. " : ""}Each is put under maintenance while it is updated and may restart.`, confirmLabel: "Run it now" }))) return; if (!(await confirmDialog({ title: `Run ${p.name} now?`, message: `It updates ${names.length === 1 ? names[0] : `${names.length} servers (${names.slice(0, 5).join(", ")}${names.length > 5 ? " …" : ""})`}, one at a time${p.stopOnFailure ? ", and stops at the first failure" : ""}. ${p.onlyWithUpdates ? "Servers with nothing waiting are left alone. " : ""}Each is put under maintenance while it is updated${p.restartTemplateName ? `, and restarted (with "${p.restartTemplateName}") if it says a restart is due` : " and may restart"}.`, confirmLabel: "Run it now" }))) return;
try { try {
await api.patchPlans.run(p.id); await api.patchPlans.run(p.id);
toast.success(`Started ${p.name}.`); toast.success(`Started ${p.name}.`);
@@ -381,6 +402,26 @@ export default function PatchPlans({ user }: { user: CurrentUser }) {
<div className="form-hint text-secondary small mt-1">Must match the host name in the Semaphore inventory.</div> <div className="form-hint text-secondary small mt-1">Must match the host name in the Semaphore inventory.</div>
</div> </div>
{templates !== null && (
<div className="col-md-8">
<label className="form-label" htmlFor="pp-restart">
Restart a server afterwards, if it says a restart is due, with
</label>
<select id="pp-restart" className="form-select" value={draft.restart} onChange={(e) => set("restart", e.target.value)}>
<option value="">No second playbook (the update playbook may restart it itself)</option>
{draft.id !== null && draft.restart !== "" && !templates.some((t) => t.key === draft.restart) && <option value={draft.restart}>{data.plans.find((p) => p.id === draft.id)?.restartTemplateName ?? "the current one"} (as it is now)</option>}
{templates
.filter((t) => t.key !== draft.template && t.integration.id === Number(draft.template.split(":")[0]))
.map((t) => (
<option key={t.key} value={t.key}>
{t.label}
</option>
))}
</select>
<div className="form-hint text-secondary small mt-1">A playbook that only restarts the server, in the same Semaphore. It runs once a server's updates are in; a server with nothing waiting that still says a restart is due is restarted too. Each restart is checked: the server must report that it started again. The window below counts for the update and the restart together.</div>
</div>
)}
<div className="col-12 d-flex flex-column gap-2"> <div className="col-12 d-flex flex-column gap-2">
<label className="form-check mb-0"> <label className="form-check mb-0">
<input type="checkbox" className="form-check-input" checked={draft.stopOnFailure} onChange={(e) => set("stopOnFailure", e.target.checked)} /> <input type="checkbox" className="form-check-input" checked={draft.stopOnFailure} onChange={(e) => set("stopOnFailure", e.target.checked)} />
@@ -448,7 +489,8 @@ export default function PatchPlans({ user }: { user: CurrentUser }) {
<td className="stack-wide"> <td className="stack-wide">
<strong>{p.name}</strong> {!p.enabled && <span className="badge bg-secondary-lt text-secondary">Switched off</span>} {p.running && <span className="badge bg-blue-lt text-blue">Running</span>} <strong>{p.name}</strong> {!p.enabled && <span className="badge bg-secondary-lt text-secondary">Switched off</span>} {p.running && <span className="badge bg-blue-lt text-blue">Running</span>}
<div className="text-secondary small"> <div className="text-secondary small">
{p.templateName} · {p.stopOnFailure ? "stops at the first failure" : "carries on after a failure"} {p.templateName}
{p.restartTemplateName ? ` · restarts with ${p.restartTemplateName} if one is due` : ""} · {p.stopOnFailure ? "stops at the first failure" : "carries on after a failure"}
</div> </div>
</td> </td>
<td className="stack-label" data-label="When"> <td className="stack-label" data-label="When">
+85 -28
View File
@@ -22,25 +22,37 @@ const WINDOWS = [
]; ];
const LAST_KEY = "hlm.updateRun.template"; const LAST_KEY = "hlm.updateRun.template";
const remembered = () => { const RESTART_KEY = "hlm.updateRun.restartTemplate";
const remembered = (key: string) => {
try { try {
return localStorage.getItem(LAST_KEY) ?? ""; return localStorage.getItem(key) ?? "";
} catch { } catch {
return ""; return "";
} }
}; };
const remember = (key: string, value: string) => {
try {
localStorage.setItem(key, value);
} catch {
// a remembered choice is only a convenience
}
};
/** /**
* "Update this server now": pick the playbook that applies updates, and Homelab Manager puts the server under maintenance, runs it for this * "Update this server now" (mode "update"): pick the playbook that applies updates, and Homelab Manager puts the server under maintenance, runs it for this
* server only, follows it, waits for the server to report again and says what is left and whether a restart is due. It asks before it * server only, follows it, waits for the server to report again and says what is left and whether a restart is due. A second playbook can be chosen to
* starts, and says what it will do. * restart the server afterwards, if it says a restart is due; it is then checked that the server started again. Mode "restart" does only the restart, for a
* server that already says one is due. It asks before it starts, and says what it will do.
*/ */
export default function UpdateRunDrawer({ server, onClose, onStarted }: { server: UpdatesServer; onClose: () => void; onStarted: () => void }) { export default function UpdateRunDrawer({ server, mode = "update", onClose, onStarted }: { server: UpdatesServer; mode?: "update" | "restart"; onClose: () => void; onStarted: () => void }) {
const restartOnly = mode === "restart";
const [loaded, setLoaded] = useState<Loaded[] | null>(null); const [loaded, setLoaded] = useState<Loaded[] | null>(null);
const [failed, setFailed] = useState(false); const [failed, setFailed] = useState(false);
const [choice, setChoice] = useState(remembered); const [choice, setChoice] = useState(() => remembered(restartOnly ? RESTART_KEY : LAST_KEY));
// The playbook that restarts afterwards (an update only): "" is none.
const [restartChoice, setRestartChoice] = useState(() => remembered(RESTART_KEY));
const [limit, setLimit] = useState(server.name); const [limit, setLimit] = useState(server.name);
const [minutes, setMinutes] = useState(90); const [minutes, setMinutes] = useState(restartOnly ? 45 : 90);
const [busy, setBusy] = useState(false); const [busy, setBusy] = useState(false);
// People streaming from a Jellyfin on this machine: the update may restart it, so they are named first and the person asked. // People streaming from a Jellyfin on this machine: the update may restart it, so they are named first and the person asked.
const [watching, setWatching] = useState(""); const [watching, setWatching] = useState("");
@@ -75,25 +87,47 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
l.templates.map((t) => ({ key: `${l.integration.id}:${t.projectId}:${t.id}`, integration: l.integration, template: t, label: `${t.name} — ${t.projectName}${(loaded ?? []).length > 1 ? ` (${l.integration.name})` : ""}` })), l.templates.map((t) => ({ key: `${l.integration.id}:${t.projectId}:${t.id}`, integration: l.integration, template: t, label: `${t.name} — ${t.projectName}${(loaded ?? []).length > 1 ? ` (${l.integration.name})` : ""}` })),
); );
const picked = options.find((o) => o.key === choice); const picked = options.find((o) => o.key === choice);
// The restart playbook has to be in the same Semaphore as the update's.
const restartOptions = picked ? options.filter((o) => o.integration.id === picked.integration.id && o.key !== picked.key) : [];
const restartPicked = !restartOnly ? restartOptions.find((o) => o.key === restartChoice) : undefined;
const count = server.updates?.count ?? null; const count = server.updates?.count ?? null;
const windowWords = WINDOWS.find((w) => w.minutes === minutes)?.label ?? `${minutes} minutes`;
async function start() { async function start() {
if (!picked) return; if (!picked) return;
const ok = await confirmDialog({ const lead = watching ? `${watching[0].toUpperCase()}${watching.slice(1)}, and ${restartOnly ? "the restart" : "the update may restart the machine"}. ` : "";
const ok = await confirmDialog(
restartOnly
? {
title: `Restart ${server.name} now?`,
message: `${lead}This silences alerts for ${server.name} for up to ${windowWords}, runs "${picked.template.name}" in ${picked.integration.name} limited to ${limit.trim()} — which restarts it — and waits for the server to report again, then checks that it did start again. What runs on it, and what depends on it, will be down for a while.`,
confirmLabel: watching ? "Restart anyway" : "Restart it",
danger: true,
}
: {
title: `Update ${server.name} now?`, title: `Update ${server.name} now?`,
message: `${watching ? `${watching[0].toUpperCase()}${watching.slice(1)}, and the update may restart the machine. ` : ""}This silences alerts for ${server.name} for up to ${WINDOWS.find((w) => w.minutes === minutes)?.label ?? `${minutes} minutes`}, runs "${picked.template.name}" in ${picked.integration.name} limited to ${limit.trim()}, and waits for the server to report again. The playbook may restart it.`, message: `${lead}This silences alerts for ${server.name} for up to ${windowWords}, runs "${picked.template.name}" in ${picked.integration.name} limited to ${limit.trim()}, and waits for the server to report again. ${
confirmLabel: watching ? "Update anyway" : "Start the update", restartPicked ? `If it then says a restart is due, "${restartPicked.template.name}" restarts it — the machine will be down for a while — and Homelab Manager checks that it came back.` : "The playbook may restart it."
}); }`,
confirmLabel: watching ? "Update anyway" : restartPicked ? "Start the update and restart" : "Start the update",
},
);
if (!ok) return; if (!ok) return;
setBusy(true); setBusy(true);
try { try {
await api.updateRuns.start({ serverId: server.id, integrationId: picked.integration.id, projectId: picked.template.projectId, templateId: picked.template.id, limit: limit.trim(), windowMinutes: minutes, startAnyway: watching !== "" }); await api.updateRuns.start({
try { serverId: server.id,
localStorage.setItem(LAST_KEY, picked.key); integrationId: picked.integration.id,
} catch { projectId: picked.template.projectId,
// a remembered choice is only a convenience templateId: picked.template.id,
} limit: limit.trim(),
toast.success(`Started updating ${server.name}.`); windowMinutes: minutes,
startAnyway: watching !== "",
...(restartOnly ? { kind: "restart" as const } : restartPicked ? { restart: { projectId: restartPicked.template.projectId, templateId: restartPicked.template.id } } : {}),
});
remember(restartOnly ? RESTART_KEY : LAST_KEY, picked.key);
if (!restartOnly) remember(RESTART_KEY, restartPicked ? restartPicked.key : "");
toast.success(restartOnly ? `Started restarting ${server.name}.` : `Started updating ${server.name}.`);
onStarted(); onStarted();
onClose(); onClose();
} catch (err) { } catch (err) {
@@ -104,7 +138,7 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
} }
return ( return (
<DetailDrawer title={`Update ${server.name}`} subtitle={server.osType === "windows" ? "Windows" : "Linux"} onClose={onClose}> <DetailDrawer title={`${restartOnly ? "Restart" : "Update"} ${server.name}`} subtitle={server.osType === "windows" ? "Windows" : "Linux"} onClose={onClose}>
<div className="mb-3"> <div className="mb-3">
<UpdatesBadge updates={server.updates} /> <UpdatesBadge updates={server.updates} />
{count !== null && count > 0 && server.updates && server.updates.security !== null && <span className="text-secondary small ms-2">{server.updates.security} of them security</span>} {count !== null && count > 0 && server.updates && server.updates.security !== null && <span className="text-secondary small ms-2">{server.updates.security} of them security</span>}
@@ -116,7 +150,7 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
<BlockSkeleton lines={3} /> <BlockSkeleton lines={3} />
) : loaded.length === 0 ? ( ) : loaded.length === 0 ? (
<div className="text-secondary"> <div className="text-secondary">
There is no Semaphore connected. Add one under <Link to="/integrations">Integrations</Link>, with a playbook that applies updates, and this server can be updated from here. There is no Semaphore connected. Add one under <Link to="/integrations">Integrations</Link>, with a playbook that {restartOnly ? "restarts a server" : "applies updates"}, and this server can be {restartOnly ? "restarted" : "updated"} from here.
</div> </div>
) : options.length === 0 ? ( ) : options.length === 0 ? (
<div className="text-secondary">No templates found in Semaphore.</div> <div className="text-secondary">No templates found in Semaphore.</div>
@@ -128,18 +162,25 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
{watching && ( {watching && (
<div className="alert alert-warning" role="status"> <div className="alert alert-warning" role="status">
<strong>Somebody is watching.</strong> {watching[0].toUpperCase()} <strong>Somebody is watching.</strong> {watching[0].toUpperCase()}
{watching.slice(1)}. The update may restart this machine; you will be asked once more. {watching.slice(1)}. {restartOnly ? "The restart will cut them off" : "The update may restart this machine"}; you will be asked once more.
</div> </div>
)} )}
<ol className="text-secondary small ps-3 mb-3"> <ol className="text-secondary small ps-3 mb-3">
<li>Alerts for this server are silenced (a maintenance window).</li> <li>Alerts for this server are silenced (a maintenance window).</li>
<li>The playbook runs in Semaphore, for this server only.</li> <li>The playbook runs in Semaphore, for this server only.</li>
<li>Homelab Manager waits for the server to report again — it may be restarting.</li> <li>Homelab Manager waits for the server to report again — it may be restarting.</li>
<li>It tells you what is still waiting and whether a restart is due, and the window ends.</li> {restartOnly ? (
<li>It checks that the server did start again (from its uptime), and the window ends.</li>
) : (
<>
{restartPicked && <li>If the server then says a restart is due, the restart playbook runs, and the server is checked to have started again.</li>}
<li>It tells you what is still waiting{restartPicked ? "" : " and whether a restart is due"}, and the window ends.</li>
</>
)}
</ol> </ol>
<div className="mb-3"> <div className="mb-3">
<label className="form-label" htmlFor="ur-template"> <label className="form-label" htmlFor="ur-template">
Playbook that applies updates {restartOnly ? "Playbook that restarts the server" : "Playbook that applies updates"}
</label> </label>
<select id="ur-template" className="form-select" value={choice} onChange={(e) => setChoice(e.target.value)}> <select id="ur-template" className="form-select" value={choice} onChange={(e) => setChoice(e.target.value)}>
<option value="">Choose a template…</option> <option value="">Choose a template…</option>
@@ -150,12 +191,28 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
))} ))}
</select> </select>
</div> </div>
{!restartOnly && (
<div className="mb-3">
<label className="form-label" htmlFor="ur-restart">
Restart it afterwards, if it says a restart is due, with
</label>
<select id="ur-restart" className="form-select" value={restartPicked ? restartChoice : ""} onChange={(e) => setRestartChoice(e.target.value)} disabled={!picked}>
<option value="">No second playbook (the update playbook may restart it itself)</option>
{restartOptions.map((o) => (
<option key={o.key} value={o.key}>
{o.label}
</option>
))}
</select>
<div className="form-hint text-secondary small mt-1">A playbook that only restarts the server. It runs once the updates are in and the server has reported again.</div>
</div>
)}
<div className="mb-3"> <div className="mb-3">
<label className="form-label" htmlFor="ur-limit"> <label className="form-label" htmlFor="ur-limit">
Run it on — the host name as it is in the Semaphore inventory Run it on — the host name as it is in the Semaphore inventory
</label> </label>
<input id="ur-limit" className="form-control" value={limit} maxLength={200} onChange={(e) => setLimit(e.target.value)} placeholder="e.g. web01" /> <input id="ur-limit" className="form-control" value={limit} maxLength={200} onChange={(e) => setLimit(e.target.value)} placeholder="e.g. web01" />
<div className="form-hint text-secondary small mt-1">An update run is always for this one server, never the whole inventory.</div> <div className="form-hint text-secondary small mt-1">{restartOnly ? "A restart" : "An update run"} is always for this one server, never the whole inventory.</div>
</div> </div>
<div className="mb-3"> <div className="mb-3">
<label className="form-label" htmlFor="ur-window"> <label className="form-label" htmlFor="ur-window">
@@ -168,10 +225,10 @@ export default function UpdateRunDrawer({ server, onClose, onStarted }: { server
</option> </option>
))} ))}
</select> </select>
<div className="form-hint text-secondary small mt-1">The window ends sooner, as soon as the run does.</div> <div className="form-hint text-secondary small mt-1">The window ends sooner, as soon as the run does.{restartPicked ? " If a restart follows, the quiet is kept up for it as well." : ""}</div>
</div> </div>
<button type="button" className="btn btn-primary" disabled={!picked || busy || limit.trim() === ""} onClick={() => void start()}> <button type="button" className={`btn ${restartOnly ? "btn-warning" : "btn-primary"}`} disabled={!picked || busy || limit.trim() === ""} onClick={() => void start()}>
{busy ? "Starting…" : "Update now"} {busy ? "Starting…" : restartOnly ? "Restart now" : restartPicked ? "Update and restart" : "Update now"}
</button> </button>
</> </>
)} )}
+13 -3
View File
@@ -21,7 +21,11 @@ export function describeRun(run: UpdateRun): { label: string; badge: string } {
case "running": case "running":
return { label: TASK_WORDS[run.taskStatus ?? ""] ?? "playbook running", badge: "bg-blue-lt text-blue" }; return { label: TASK_WORDS[run.taskStatus ?? ""] ?? "playbook running", badge: "bg-blue-lt text-blue" };
case "waiting_for_host": case "waiting_for_host":
return { label: "waiting for the server to report again", badge: "bg-blue-lt text-blue" }; return { label: run.kind === "restart" ? "waiting for the server to come back" : "waiting for the server to report again", badge: "bg-blue-lt text-blue" };
case "restarting":
return { label: TASK_WORDS[run.restartTaskStatus ?? ""] ? `restart: ${TASK_WORDS[run.restartTaskStatus ?? ""]}` : "restart playbook running", badge: "bg-blue-lt text-blue" };
case "waiting_for_restart":
return { label: "waiting for the server to come back from its restart", badge: "bg-blue-lt text-blue" };
case "done": case "done":
return { label: "finished", badge: run.rebootRequired ? "bg-yellow-lt text-yellow" : "bg-green-lt text-green" }; return { label: "finished", badge: run.rebootRequired ? "bg-yellow-lt text-yellow" : "bg-green-lt text-green" };
case "failed": case "failed":
@@ -99,8 +103,9 @@ export default function UpdateRunsCard({ canEdit, reloadKey, onFinished }: { can
<Link to={`/servers/${r.serverId}`}> <Link to={`/servers/${r.serverId}`}>
<strong>{r.serverName ?? `server #${r.serverId}`}</strong> <strong>{r.serverName ?? `server #${r.serverId}`}</strong>
</Link> </Link>
<span className="text-secondary">{r.template}</span> <span className="text-secondary">{r.kind === "restart" ? "Restart · " : ""}{r.template}{r.restartTemplate && r.kind === "update" ? ` (then ${r.restartTemplate} if a restart is due)` : ""}</span>
<span className={`badge ${w.badge}`}>{w.label}</span> <span className={`badge ${w.badge}`}>{w.label}</span>
{r.status === "done" && r.restarted && <span className="badge bg-green-lt text-green">restarted</span>}
{r.status === "done" && r.rebootRequired && <span className="badge bg-yellow-lt text-yellow">restart needed</span>} {r.status === "done" && r.rebootRequired && <span className="badge bg-yellow-lt text-yellow">restart needed</span>}
<span className="text-secondary small ms-auto"> <span className="text-secondary small ms-auto">
started <TimeAgo value={r.startedAt} /> by {r.startedBy ?? "someone"} started <TimeAgo value={r.startedAt} /> by {r.startedBy ?? "someone"}
@@ -117,7 +122,12 @@ export default function UpdateRunsCard({ canEdit, reloadKey, onFinished }: { can
<div className="progress-bar progress-bar-indeterminate" /> <div className="progress-bar progress-bar-indeterminate" />
</div> </div>
)} )}
{r.taskId !== null && <div className="text-secondary small">Run #{r.taskId} in Semaphore — the playbook's output is there.</div>} {r.taskId !== null && (
<div className="text-secondary small">
Run #{r.taskId} in Semaphore — the playbook's output is there.
{r.restartTaskId !== null && <> The restart is run #{r.restartTaskId}.</>}
</div>
)}
</div> </div>
); );
})} })}
+12 -5
View File
@@ -88,7 +88,7 @@ export default function Updates({ user }: { user: CurrentUser }) {
function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) { function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) {
const canRun = user.role === "admin" || user.role === "operator"; const canRun = user.role === "admin" || user.role === "operator";
// The server being updated now (its drawer is open), and a counter that makes the runs card look again after one was started. // The server being updated now (its drawer is open), and a counter that makes the runs card look again after one was started.
const [updating, setUpdating] = useState<UpdatesServer | null>(null); const [updating, setUpdating] = useState<{ server: UpdatesServer; mode: "update" | "restart" } | null>(null);
const [runsKey, setRunsKey] = useState(0); const [runsKey, setRunsKey] = useState(0);
const [servers, setServers] = useState<UpdatesServer[] | null>(null); const [servers, setServers] = useState<UpdatesServer[] | null>(null);
const [agentLatest, setAgentLatest] = useState<AgentLatest | null>(null); const [agentLatest, setAgentLatest] = useState<AgentLatest | null>(null);
@@ -101,7 +101,7 @@ function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) {
useEffect(() => { useEffect(() => {
if (!wantedUpdate || !servers || !canRun) return; if (!wantedUpdate || !servers || !canRun) return;
const server = servers.find((s) => String(s.id) === wantedUpdate); const server = servers.find((s) => String(s.id) === wantedUpdate);
if (server) setUpdating(server); if (server) setUpdating({ server, mode: "update" });
setParams( setParams(
(prev) => { (prev) => {
const next = new URLSearchParams(prev); const next = new URLSearchParams(prev);
@@ -188,7 +188,7 @@ function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) {
</div> </div>
{tabs} {tabs}
<p className="text-secondary"> <p className="text-secondary">
What the agents say is waiting to be installed on each server, and the Docker containers that have a newer image. {canRun ? "Update… applies a server's updates through a Semaphore playbook: it silences the server, runs the playbook, waits for the server to report again and tells you what is left." : "Operators can apply a server's updates from here."} What the agents say is waiting to be installed on each server, and the Docker containers that have a newer image. {canRun ? "Update… applies a server's updates through a Semaphore playbook: it silences the server, runs the playbook, waits for the server to report again and tells you what is left — and can restart it afterwards, with a second playbook, if it says a restart is due. Restart… does only the restart." : "Operators can apply a server's updates, and restart a server that needs it, from here."}
</p> </p>
<ErrorNotice error={error} /> <ErrorNotice error={error} />
@@ -266,9 +266,16 @@ function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) {
</td> </td>
{canRun && ( {canRun && (
<td className="stack-wide"> <td className="stack-wide">
<button type="button" className="btn btn-sm btn-outline-primary" disabled={r.count === null || r.count === 0 || r.server.lastSeenAt === null} title={r.count ? "Apply this server's updates through a Semaphore playbook" : "Nothing is waiting, or the agent hasn't said"} onClick={() => setUpdating(r.server)}> <div className="btn-list flex-nowrap">
<button type="button" className="btn btn-sm btn-outline-primary" disabled={r.count === null || r.count === 0 || r.server.lastSeenAt === null} title={r.count ? "Apply this server's updates through a Semaphore playbook" : "Nothing is waiting, or the agent hasn't said"} onClick={() => setUpdating({ server: r.server, mode: "update" })}>
Update… Update…
</button> </button>
{r.restart && (
<button type="button" className="btn btn-sm btn-outline-warning" disabled={r.server.lastSeenAt === null} title="This server says a restart is due: restart it through a Semaphore playbook" onClick={() => setUpdating({ server: r.server, mode: "restart" })}>
Restart…
</button>
)}
</div>
</td> </td>
)} )}
</tr> </tr>
@@ -332,7 +339,7 @@ function WaitingNow({ user, tabs }: { user: CurrentUser; tabs: ReactNode }) {
</div> </div>
)} )}
{updating && <UpdateRunDrawer server={updating} onClose={() => setUpdating(null)} onStarted={() => setRunsKey((k) => k + 1)} />} {updating && <UpdateRunDrawer server={updating.server} mode={updating.mode} onClose={() => setUpdating(null)} onStarted={() => setRunsKey((k) => k + 1)} />}
</> </>
); );
} }