Fold alerts under their host, recurring maintenance, run playbooks from alerts, and an Updates page

- Servers can say what they run on (or it is worked out from the Proxmox node); when the host is
  down the guests' offline alerts fold under it: held back, not counted, not notified, still recorded.
- Recurring maintenance schedules made into ordinary windows a week ahead (maintenance_schedules).
- Run a Semaphore template limited to a server from its page or an alert, and watch the run.
- Agents report pending updates (apt, dnf, zypper, Windows Update); an Updates page lists them.
- Migrations 0024 and 0025.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-17 01:35:15 +02:00
1 parent 44e393f172
commit 171511c6db
45 files changed
+6769 -41

No files matched your search

+28 -5
View File
@@ -23,8 +23,8 @@ what the app actually creates.
|---|---|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0023`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 25 application tables, plus drizzle's own `__drizzle_migrations`. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0025`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 26 application tables, plus drizzle's own `__drizzle_migrations`. |
**Not in the database:**
@@ -62,9 +62,9 @@ erDiagram
integrations ||--o{ alert_tickets : "osTicket used (set null)"
```
Nine tables stand alone, with no foreign keys: `settings`, `secrets`,
Ten tables stand alone, with no foreign keys: `settings`, `secrets`,
`ipam_entries`, `domains`, `diag_log`, `notification_queue`,
`consistency_ignores`, `tag_definitions`, `share_links`. `maintenance_windows` and `maintenance_aliases` also have no
`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`. `maintenance_windows` and `maintenance_aliases` also have no
foreign key — see [soft references](#soft-references-not-foreign-keys).
## Tables
@@ -145,7 +145,27 @@ required, so a forgotten window can't silence real problems forever.
| `reason` | text | |
| `started_at`, `ends_at` | text NOT NULL | ISO. |
| `created_by` | text | Name of the person who opened it. `Uptime Kuma sync` for mirrored ones. |
| `source` | text | `kuma:<integration id>:<maintenance id>` on windows mirrored from Uptime Kuma's schedule, so a later sync can find them again; NULL for ones started by hand. A window whose `started_at` is still in the future is *scheduled*: it silences nothing until it begins. A scheduled one that is cancelled has `ends_at` set equal to `started_at`. |
| `source` | text | `kuma:<integration id>:<maintenance id>` on windows mirrored from Uptime Kuma's schedule, so a later sync can find them again; `schedule:<schedule id>:<date>` on windows made from a recurring schedule (so one is made once, even if it was cancelled); NULL for ones started by hand. A window whose `started_at` is still in the future is *scheduled*: it silences nothing until it begins. A scheduled one that is cancelled has `ends_at` set equal to `started_at`. |
#### `maintenance_schedules`
A maintenance that repeats ("Sundays 02:00–04:00 on these servers"). It only says when:
the windows that silence alerts are ordinary `maintenance_windows` rows made a week
ahead and marked `schedule:<id>:<date>`.
| Column | Type | Notes |
|---|---|---|
| `id` | integer PK | |
| `name` | text NOT NULL | e.g. "Patch night". |
| `reason` | text | |
| `targets` | text NOT NULL | JSON `[{"type":"server","id":3}, …]`. Polymorphic like `maintenance_windows`, so no foreign key; a deleted target is skipped. |
| `days` | text NOT NULL | JSON array of weekdays, 0 = Sunday … 6 = Saturday. |
| `start_time` | text NOT NULL | `HH:MM` on the clock of `timezone`. |
| `duration_minutes` | integer NOT NULL | |
| `timezone` | text NOT NULL | IANA name, so 02:00 stays 02:00 across summer and winter time. |
| `enabled` | bool NOT NULL, default 1 | Off makes no windows and takes back those that haven't begun. |
| `created_by` | text | |
| `created_at` | text NOT NULL | |
#### `maintenance_aliases`
@@ -194,6 +214,9 @@ what it last reported.
| `proxmox_guest_type` | text | `qemu` \| `lxc`. |
| `proxmox_vmid` | integer | The four `proxmox_*` columns are set together or cleared together (enforced by the API), by an admin — never by the agent. |
| `hide_proxmox_link` | bool NOT NULL, default 0 | Hides the "Proxmox link" card for servers that aren't Proxmox guests. Ignored while the server is actually linked. |
| `parent_server_id` | integer | The server this one runs on, set by an admin. **Not a foreign key** (a removed host's servers are cleared by the app). Null = work it out: a server linked to a Proxmox guest runs on the one server named after that guest's node. When the parent is down, this server's offline alert is folded under it. |
| `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. |
| `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. |
#### `scheduled_tasks`
+1 -1
View File
@@ -40,7 +40,7 @@ the dashboard views themselves load fine.
- **Config fields:** Semaphore URL, API token
- **Auth:** Bearer token
- **Actions used:** list projects, list templates, run a template (creates a task)
- **Actions used:** list projects, list templates, run a template (creates a task — optionally with a `limit` of hosts and a note saying who started it from Homelab Manager), read a task's status
- **Required access:** A user API token with access to every project you want visible, and **permission to run templates** in those projects — a viewer/guest-level project role can list templates but will fail on "Run", so the account needs at least the operator-equivalent role Semaphore's own project permissions define.
### Gitea
+39
View File
@@ -400,6 +400,45 @@ the problems themselves, and each integration's status, refreshing every 30
seconds with the screen kept awake where the browser allows. Esc returns to the
app.
**Alerts folded under their host.** A server can say which server it runs on (a
virtual machine on its Proxmox host): on its page, "Runs on" — and left alone it
is worked out when the server is linked to a Proxmox guest and exactly one
server is named after that guest's node. When the host stops reporting, the
guests' own "server offline" alerts are folded under the host's ("also takes down
anna, elsa") instead of being one alert each: they are held back like a silenced
alert, not counted, left out of notifications and osTicket tickets, and labelled
"caused by pve1" on the Alerts page. They are still recorded as down (for the
uptime ribbons and "Since you last looked"), and if the host comes back while a
guest is still down, that guest alerts then as new — an outage is never swallowed.
Links that loop back on themselves cover nothing, and setting one that would is
refused.
**Recurring maintenance** (Maintenance → Recurring) repeats a window on the days
and clock time you choose — "Sundays 02:00–04:00 on these servers" — in the time
zone of the browser it was made in, so 02:00 stays 02:00 across summer and winter
time. Each time becomes an ordinary maintenance window a week ahead (shown on the
timeline with vertical stripes, and in Scheduled), so any single one can be
cancelled and stays cancelled. Turning a schedule off or removing it takes back
the windows that haven't begun.
**Running a playbook** from an alert or a server: a server's page has a "Run a
playbook" card, and alerts about a server have a "Run…" button, which pick a
Semaphore template and run it limited to that server (ansible's `--limit`, which
you can edit to match the host name in your inventory — or untick it to run on
everything, which asks more sternly). Semaphore is told who started it, the run
shows waiting → running → success or failed, and the audit log records what ran
and where. Operators and admins only.
**Updates** (Operations → Updates) shows what each agent says is waiting to be
installed — how many, how many are security fixes, whether a restart is due, and
how long updates have been waiting — plus the Docker containers with a newer image.
The Linux agent asks apt, dnf or zypper (only what they already know; nothing is
installed or refreshed) and the Windows agent asks Windows Update (at most every six
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
said — an older agent, or another package manager — shows "not reported", never "up
to date"; run its install command again to update its agent. It only looks: to apply
updates, run a playbook.
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
card earlier or later among its neighbours and hide the ones you don't use —
hidden cards stay listed, dimmed, while you arrange so they can be brought back,
+47 -1
View File
@@ -182,6 +182,46 @@ collect_systemd_timers() {
done < <(jq -c '.[]' <<<"$timers_json")
}
# Updates waiting to be installed, as JSON in UPDATES_JSON ("null" when this host's package manager isn't one we know, or couldn't answer).
# Nothing is installed or refreshed: this reads what the package manager already knows about, so it is quick and changes nothing.
collect_updates() {
UPDATES_JSON="null"
local manager="" count="" security="null" reboot="false" out="" status=0
if command -v apt-get >/dev/null 2>&1; then
manager="apt"
# -s simulates: "Inst" lines are what an upgrade would install. Security ones come from a "-security" or "Security" archive.
if out=$(LC_ALL=C timeout 60 apt-get -s -o Debug::NoLocking=true upgrade 2>/dev/null); then
count=$(grep -c '^Inst ' <<<"$out" || true)
security=$(grep '^Inst ' <<<"$out" | grep -ci 'security' || true)
fi
[[ -f /var/run/reboot-required ]] && reboot="true"
elif command -v dnf >/dev/null 2>&1; then
manager="dnf"
# check-update exits 100 when there are updates, 0 when none, 1 on an error. --cacheonly: never goes to the network from here.
out=$(LC_ALL=C timeout 120 dnf -q check-update --cacheonly 2>/dev/null) || status=$?
if [[ "$status" -eq 0 || "$status" -eq 100 ]]; then
count=$(awk 'NF >= 3 && $1 ~ /^[A-Za-z0-9_.+-]+\.[A-Za-z0-9_]+$/ && /Obsoleting/ == 0 { n++ } /^Obsoleting/ { exit } END { print n + 0 }' <<<"$out")
security=$(LC_ALL=C timeout 120 dnf -q updateinfo list --security --cacheonly 2>/dev/null | grep -c '/Sec\.' || true)
fi
if command -v needs-restarting >/dev/null 2>&1; then
needs-restarting -r >/dev/null 2>&1 || reboot="true"
fi
elif command -v zypper >/dev/null 2>&1; then
manager="zypper"
if out=$(LC_ALL=C timeout 120 zypper -q --no-refresh list-updates 2>/dev/null); then
count=$(grep -c '^v |' <<<"$out" || true)
fi
[[ -f /run/reboot-needed ]] && reboot="true"
fi
if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then
[[ "$security" =~ ^[0-9]+$ ]] || security="null"
UPDATES_JSON=$(jq -n --arg manager "$manager" --argjson count "$count" --argjson security "$security" --argjson reboot "$reboot" \
'{manager: $manager, count: $count, security: $security, reboot_required: $reboot}')
fi
}
collect_system_info() {
local ip_json="[]"
if command -v ip >/dev/null 2>&1; then
@@ -243,7 +283,12 @@ collect_system_info() {
')
fi
# Best-effort like the rest: a package manager that misbehaves must not cost the report.
UPDATES_JSON="null"
collect_updates || UPDATES_JSON="null"
SYSTEM_JSON=$(jq -n \
--argjson updates "$UPDATES_JSON" \
--argjson ip_addresses "$ip_json" \
--argjson listening_ports "$ports_json" \
--arg cpu_model "$cpu_model" \
@@ -257,7 +302,8 @@ collect_system_info() {
cpu: { model: $cpu_model, cores: $cpu_cores, load_percent: $cpu_load_percent },
memory: { total_bytes: $mem_total_bytes, used_bytes: $mem_used_bytes },
disks: $disks,
listening_ports: $listening_ports
listening_ports: $listening_ports,
updates: $updates
}')
}
+2
View File
@@ -6,6 +6,8 @@ runs as a scheduled task (as SYSTEM, every 15 minutes) and pushes to `POST /api/
the machine never needs to be reachable from Homelab Manager.
Each report also says how many updates Windows Update has waiting (and how many are security or critical, and whether a restart is due). Searching Windows Update goes over the network and can take a minute or two, so it is done at most every six hours; the answer is kept in `%ProgramData%\HomelabManager\updates.json` and reused between. Nothing is downloaded or installed.
The installer limits `%ProgramData%\HomelabManager` to SYSTEM and Administrators. The agent runs as SYSTEM, so anyone who could change its script could run code as SYSTEM; the folder is not left to inherit ProgramData's default permissions. Re-running the install one-liner applies this to an existing install.
## Install
+49
View File
@@ -220,6 +220,50 @@ function Get-ListeningPorts {
return @($rows | Where-Object { $_.port -ge 1 -and $_.port -le 65535 } | Select-Object -First 2000)
}
# ---- updates waiting to be installed ------------------------------------------------------------
$script:UpdatesCachePath = Join-Path $env:ProgramData 'HomelabManager\updates.json'
function Test-RebootRequired {
foreach ($key in @(
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired',
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending')) {
if (Test-Path $key) { return $true }
}
return $false
}
# Asks Windows Update what is waiting. That goes over the network and can take a minute, so it is done at most every six hours and the
# last answer is reused in between; whether a restart is due is cheap and always read fresh. Nothing is installed or downloaded.
function Get-PendingUpdates {
$result = $null
try {
if (Test-Path $script:UpdatesCachePath) {
$cached = Get-Content -Raw -Path $script:UpdatesCachePath | ConvertFrom-Json
$checked = [datetime]::Parse([string]$cached.checked_at, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::RoundtripKind)
if (([datetime]::UtcNow - $checked.ToUniversalTime()).TotalHours -lt 6) { $result = $cached }
}
} catch { $result = $null }
if (-not $result) {
$session = New-Object -ComObject Microsoft.Update.Session
$found = $session.CreateUpdateSearcher().Search("IsInstalled=0 and IsHidden=0 and Type='Software'")
$security = 0
foreach ($u in $found.Updates) {
foreach ($c in $u.Categories) {
if ($c.Name -eq 'Security Updates' -or $c.Name -eq 'Critical Updates') { $security++; break }
}
}
$result = [pscustomobject][ordered]@{ checked_at = [datetime]::UtcNow.ToString('o'); count = [int]$found.Updates.Count; security = [int]$security }
try {
$dir = Split-Path $script:UpdatesCachePath
if (-not (Test-Path $dir)) { $null = New-Item -ItemType Directory -Path $dir -Force }
ConvertTo-Json -InputObject $result | Set-Content -Path $script:UpdatesCachePath -Encoding UTF8
} catch { }
}
return [ordered]@{ manager = 'windows-update'; count = [int]$result.count; security = [int]$result.security; reboot_required = (Test-RebootRequired) }
}
function Get-SystemInfo {
$ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue |
Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } |
@@ -240,7 +284,12 @@ function Get-SystemInfo {
$disks += [pscustomobject][ordered]@{ mount = [string]$d.DeviceID; size_bytes = [int64]$d.Size; used_bytes = [int64]($d.Size - $d.FreeSpace) }
}
# Best-effort: Windows Update being unreachable must not cost the rest of the report.
$updates = $null
try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))" }
return [ordered]@{
updates = $updates
ip_addresses = @($ips)
cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) }
memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) }
+3
View File
@@ -0,0 +1,3 @@
ALTER TABLE `servers` ADD `parent_server_id` integer;--> statement-breakpoint
ALTER TABLE `servers` ADD `pending_updates` text;--> statement-breakpoint
ALTER TABLE `servers` ADD `updates_pending_since` text;
+13
View File
@@ -0,0 +1,13 @@
CREATE TABLE `maintenance_schedules` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`name` text NOT NULL,
`reason` text,
`targets` text NOT NULL,
`days` text NOT NULL,
`start_time` text NOT NULL,
`duration_minutes` integer NOT NULL,
`timezone` text NOT NULL,
`enabled` integer DEFAULT true NOT NULL,
`created_by` text,
`created_at` text DEFAULT (current_timestamp) NOT NULL
);
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+14
View File
@@ -169,6 +169,20 @@
"when": 1792181083723,
"tag": "0023_premium_cyclops",
"breakpoints": true
},
{
"idx": 24,
"version": "6",
"when": 1792191100783,
"tag": "0024_chubby_jackpot",
"breakpoints": true
},
{
"idx": 25,
"version": "6",
"when": 1792191757544,
"tag": "0025_clean_the_hand",
"breakpoints": true
}
]
}
+30
View File
@@ -282,6 +282,15 @@ export const servers = sqliteTable("servers", {
// card always shows) once a server IS actually linked, so unlinking stays
// reachable.
hideProxmoxLink: integer("hide_proxmox_link", { mode: "boolean" }).notNull().default(false),
// The server this one runs on (a VM on its host), set by an admin. Null means "work it out": a guest linked to a Proxmox node runs on
// the server named after that node. When the parent stops reporting, this server's own offline alert is folded under the parent's.
parentServerId: integer("parent_server_id"),
// What the agent says is waiting to be installed, as JSON: {manager, count, security, rebootRequired, reportedAt}. Null until it reports.
pendingUpdates: text("pending_updates"),
// When updates were first seen waiting (and none were before) — how long the oldest has been waiting. Null when none are.
updatesPendingSince: text("updates_pending_since"),
});
export const scheduledTasks = sqliteTable("scheduled_tasks", {
@@ -505,3 +514,24 @@ export const shareLinks = sqliteTable("share_links", {
expiresAt: text("expires_at"), // ISO; null = never
lastUsedAt: text("last_used_at"), // ISO
});
// ─── Recurring maintenance ──────────────────────────────────────────────────
// "Every Sunday 02:00–04:00 on these servers": a schedule kept here, turned into ordinary maintenance windows a week ahead (each marked
// `schedule:<id>:<date>` in maintenance_windows.source). The windows are what silence alerts; the schedule only says when they come.
export const maintenanceSchedules = sqliteTable("maintenance_schedules", {
id: integer("id").primaryKey({ autoIncrement: true }),
name: text("name").notNull(), // e.g. "Patch night"
reason: text("reason"),
// JSON: {type: "server" | "integration" | "dns_provider", id: number}[] — polymorphic, so no foreign key; a deleted target is skipped.
targets: text("targets").notNull(),
days: text("days").notNull(), // JSON: weekdays it happens on, 0 = Sunday … 6 = Saturday
startTime: text("start_time").notNull(), // "HH:MM" on the clock of `timezone`
durationMinutes: integer("duration_minutes").notNull(),
timezone: text("timezone").notNull(), // IANA name, so "02:00" stays 02:00 across summer and winter time
enabled: integer("enabled", { mode: "boolean" }).notNull().default(true),
createdBy: text("created_by"),
createdAt: text("created_at")
.notNull()
.default(sql`(current_timestamp)`),
});
+2
View File
@@ -13,6 +13,7 @@ import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js";
import { initHealthScheduler } from "./services/healthScheduler.js";
import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js";
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js";
{
// Sessions are signed with this secret. Without one, the app would sign them with a value that's in the source code.
@@ -42,6 +43,7 @@ await initQuietHoursScheduler();
await initHealthScheduler();
await initAlertTicketScheduler();
await initKumaMaintenanceScheduler();
await initMaintenanceScheduleScheduler();
createApp().listen(env.port, () => {
console.log(`homelab-manager listening on port ${env.port}`);
+20 -4
View File
@@ -45,12 +45,20 @@ export interface SemaphoreTemplate {
lastTask: SemaphoreTask | null;
}
/** What a run may be told: which hosts of the inventory to run on (ansible's --limit), and a note shown beside the run in Semaphore. */
export interface SemaphoreRunOptions {
limit?: string;
message?: string;
}
export interface SemaphoreAdapter {
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
listTemplatesWithStatus(): Promise<SemaphoreTemplate[]>;
/** Like listTemplatesWithStatus, but says which projects couldn't be read — for callers that must not mistake "couldn't read" for "no templates". */
checkTemplates(): Promise<{ templates: SemaphoreTemplate[]; failedProjectIds: number[] }>;
runTemplate(projectId: number, templateId: number): Promise<SemaphoreTask>;
runTemplate(projectId: number, templateId: number, options?: SemaphoreRunOptions): Promise<SemaphoreTask>;
/** How a run is getting on. */
getTask(projectId: number, taskId: number): Promise<SemaphoreTask>;
}
export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapter {
@@ -134,11 +142,19 @@ export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapte
return (await checkTemplates()).templates;
}
async function runTemplate(projectId: number, templateId: number): Promise<SemaphoreTask> {
const task = await api("POST", `/project/${projectId}/tasks`, { template_id: templateId });
async function runTemplate(projectId: number, templateId: number, options: SemaphoreRunOptions = {}): Promise<SemaphoreTask> {
const task = await api("POST", `/project/${projectId}/tasks`, {
template_id: templateId,
...(options.limit ? { limit: options.limit } : {}),
...(options.message ? { message: options.message } : {}),
});
return mapTask(task)!;
}
async function getTask(projectId: number, taskId: number): Promise<SemaphoreTask> {
return mapTask(await api("GET", `/project/${projectId}/tasks/${taskId}`))!;
}
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
const start = Date.now();
try {
@@ -149,5 +165,5 @@ export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapte
}
}
return withDiagLogging("semaphore", { ping, listTemplatesWithStatus, checkTemplates, runTemplate });
return withDiagLogging("semaphore", { ping, listTemplatesWithStatus, checkTemplates, runTemplate, getTask });
}
+15
View File
@@ -23,7 +23,22 @@ const listeningPortSchema = z.object({
process: text(100).optional(),
});
// What is waiting to be installed. As lenient as the ports: a malformed answer is dropped, never allowed to cost the agent its report.
const updatesSchema = z.object({
manager: text(30).optional(),
count: z.number().int().min(0).max(100_000),
security: z.number().int().min(0).max(100_000).nullable().optional(),
reboot_required: z.boolean().optional(),
});
const systemSchema = z.object({
updates: z
.unknown()
.optional()
.transform((u) => {
const parsed = updatesSchema.safeParse(u);
return parsed.success ? parsed.data : undefined;
}),
ip_addresses: z.array(text(100)).transform(cap(200)).optional(),
cpu: z.object({ model: text(200).optional(), cores: z.number().optional(), load_percent: z.number().nullable().optional() }).optional(),
memory: z.object({ total_bytes: z.number().optional(), used_bytes: z.number().optional() }).optional(),
+34 -3
View File
@@ -633,6 +633,14 @@ integrationsRouter.get("/:id/semaphore/templates", asyncHandler(async (req, res)
}
}));
// Which hosts a run may be limited to: names, addresses and ansible patterns (commas, colons, wildcards) — and nothing that could be
// taken for an option. Semaphore hands the value to ansible as an argument, but it is still only ever shown the characters a host has.
const runBodySchema = z
.object({
limit: z.union([z.literal(""), z.string().trim().regex(/^[A-Za-z0-9][A-Za-z0-9._,:*\-]{0,199}$/, "Use host names or patterns like web01,db*")]).optional(),
})
.default({});
integrationsRouter.post(
"/:id/semaphore/projects/:projectId/templates/:templateId/run",
requireRole("operator"),
@@ -645,16 +653,25 @@ integrationsRouter.post(
if (!Number.isInteger(projectId) || !Number.isInteger(templateId)) {
return res.status(400).json({ error: "invalid_id" });
}
const body = runBodySchema.safeParse(req.body ?? {});
if (!body.success) {
return res.status(400).json({ error: "invalid_body", message: "The hosts to limit it to can only be host names or patterns like web01,db*.", details: body.error.flatten() });
}
try {
const task = await found.adapter.runTemplate(projectId, templateId);
const actor = req.currentUser!;
const limit = body.data.limit || undefined;
const task = await found.adapter.runTemplate(projectId, templateId, {
limit,
message: `Started in Homelab Manager by ${actor.name ?? actor.email ?? actor.oidcSub}${limit ? ` (limited to ${limit})` : ""}`,
});
await recordAudit({
actor: req.currentUser!,
actor,
category: "integration",
action: "run_template",
targetType: "semaphore_template",
targetId: templateId,
detail: { integrationId: found.integration.id, projectId, taskId: task.id },
detail: { integrationId: found.integration.id, projectId, taskId: task.id, ...(limit ? { limit } : {}) },
});
res.status(201).json({ task });
} catch (err) {
@@ -663,6 +680,20 @@ integrationsRouter.post(
}),
);
// How a run is getting on: waiting, running, success or error.
integrationsRouter.get("/:id/semaphore/projects/:projectId/tasks/:taskId", asyncHandler(async (req, res) => {
const found = await requireSemaphoreAdapter(req, res);
if (!found) return;
const projectId = Number(req.params.projectId);
const taskId = Number(req.params.taskId);
if (!Number.isInteger(projectId) || !Number.isInteger(taskId)) return res.status(400).json({ error: "invalid_id" });
try {
res.json({ task: await found.adapter.getTask(projectId, taskId) });
} catch (err) {
res.status(502).json({ error: err instanceof Error ? err.message : String(err) });
}
}));
// ─── Proxmox ─────────────────────────────────────────────────────────────────
async function requireProxmoxAdapter(req: Request, res: Response) {
+101 -1
View File
@@ -2,7 +2,8 @@ import { Router } from "express";
import { and, eq, inArray } from "drizzle-orm";
import { z } from "zod";
import { db } from "../db/client.js";
import { dnsProviders, integrations, maintenanceAliases, maintenanceTargetTypes, maintenanceWindows, servers } from "../db/schema.js";
import { dnsProviders, integrations, maintenanceAliases, maintenanceSchedules, maintenanceTargetTypes, maintenanceWindows, servers } from "../db/schema.js";
import { deleteSchedule, isValidTimeZone, materializeSchedules, occurrencesBetween, removeFutureWindows, targetsOf } from "../services/maintenanceSchedules.js";
import { hostOfAddress } from "../services/kumaTargets.js";
import { invalidateAlertsCache } from "../services/alerts.js";
import { requireAuth, requireRole } from "../auth/middleware.js";
@@ -268,6 +269,105 @@ maintenanceRouter.post("/end-all", requireRole("operator"), asyncHandler(async (
res.json({ ended: active.length });
}));
// ─── Recurring schedules ────────────────────────────────────────────────────
const scheduleSchema = z.object({
name: z.string().trim().min(1).max(80),
reason: z.string().trim().max(200).optional(),
targets: z.array(z.object({ type: z.enum(maintenanceTargetTypes), id: z.number().int() })).min(1).max(100),
days: z.array(z.number().int().min(0).max(6)).min(1).max(7),
startTime: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/, "Use a time like 02:00"),
durationMinutes: z.number().int().min(15).max(3 * 24 * 60),
timezone: z.string().min(1).max(64),
});
maintenanceRouter.get("/schedules", asyncHandler(async (_req, res) => {
const rows = await db.select().from(maintenanceSchedules).orderBy(maintenanceSchedules.name);
const now = new Date();
const out = [];
for (const s of rows) {
const targets = [];
for (const t of targetsOf(s)) {
const d = await describeTarget(t.type, t.id);
if (d) targets.push({ type: t.type, id: t.id, name: d.name, kind: d.kind });
}
const next = s.enabled ? occurrencesBetween(s, now, new Date(now.getTime() + 15 * 24 * 3600_000))[0] : undefined;
out.push({
id: s.id,
name: s.name,
reason: s.reason,
days: JSON.parse(s.days) as number[],
startTime: s.startTime,
durationMinutes: s.durationMinutes,
timezone: s.timezone,
enabled: s.enabled,
createdBy: s.createdBy,
targets,
nextStart: next ? next.start.toISOString() : null,
});
}
res.json({ schedules: out });
}));
maintenanceRouter.post("/schedules", requireRole("operator"), asyncHandler(async (req, res) => {
const parsed = scheduleSchema.safeParse(req.body);
if (!parsed.success) {
return res.status(400).json({ error: "invalid_body", message: "Name it, choose what it covers, the days, a start time and how long.", details: parsed.error.flatten() });
}
const data = parsed.data;
if (!isValidTimeZone(data.timezone)) return res.status(400).json({ error: "invalid_timezone", message: "That isn't a time zone this server knows." });
// Only things that exist, once each.
const seen = new Set<string>();
const targets = [];
for (const t of data.targets) {
const key = `${t.type}:${t.id}`;
if (seen.has(key)) continue;
seen.add(key);
if (!(await describeTarget(t.type, t.id))) return res.status(400).json({ error: "unknown_target", message: "One of the things chosen doesn't exist any more." });
targets.push(t);
}
const actor = req.currentUser!;
const [created] = await db
.insert(maintenanceSchedules)
.values({
name: data.name,
reason: data.reason || null,
targets: JSON.stringify(targets),
days: JSON.stringify([...new Set(data.days)].sort()),
startTime: data.startTime,
durationMinutes: data.durationMinutes,
timezone: data.timezone,
createdBy: actor.name ?? actor.email ?? actor.oidcSub,
})
.returning();
const made = await materializeSchedules(new Date(), undefined, created.id);
await recordAudit({ actor, category: "maintenance", action: "create_schedule", targetType: "maintenance_schedule", targetId: created.id, detail: { name: created.name, targets: targets.length, days: data.days, startTime: data.startTime, durationMinutes: data.durationMinutes } });
res.status(201).json({ id: created.id, windowsMade: made });
}));
maintenanceRouter.patch("/schedules/:id", requireRole("operator"), asyncHandler(async (req, res) => {
const id = Number(req.params.id);
const parsed = z.object({ enabled: z.boolean() }).safeParse(req.body);
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body" });
const [row] = await db.select().from(maintenanceSchedules).where(eq(maintenanceSchedules.id, id)).limit(1);
if (!row) return res.status(404).json({ error: "not_found" });
await db.update(maintenanceSchedules).set({ enabled: parsed.data.enabled }).where(eq(maintenanceSchedules.id, id));
if (parsed.data.enabled) await materializeSchedules(new Date(), undefined, id);
else await removeFutureWindows(id);
await recordAudit({ actor: req.currentUser!, category: "maintenance", action: parsed.data.enabled ? "enable_schedule" : "disable_schedule", targetType: "maintenance_schedule", targetId: id, detail: { name: row.name } });
res.json({ enabled: parsed.data.enabled });
}));
maintenanceRouter.delete("/schedules/:id", requireRole("operator"), asyncHandler(async (req, res) => {
const id = Number(req.params.id);
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
const [row] = await db.select().from(maintenanceSchedules).where(eq(maintenanceSchedules.id, id)).limit(1);
if (!row) return res.status(404).json({ error: "not_found" });
await deleteSchedule(id);
await recordAudit({ actor: req.currentUser!, category: "maintenance", action: "delete_schedule", targetType: "maintenance_schedule", targetId: id, detail: { name: row.name } });
res.status(204).end();
}));
const syncUptimeKumaSchema = z.object({ integrationId: z.number().int().positive() });
/** Reads Uptime Kuma's maintenance schedule now and brings the mirrored windows in line with it (the same pass that runs every 15 minutes when switched on). */
+55 -5
View File
@@ -6,6 +6,8 @@ import { servers, serverLinks, integrations, dnsRecordsCache, dnsZonesCache, dns
import { requireAuth, requireRole } from "../auth/middleware.js";
import { generateApiToken } from "../services/tokens.js";
import { recordAudit } from "../services/audit.js";
import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js";
import { parsePendingUpdates } from "../services/pendingUpdates.js";
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
import { getServerActivity } from "../services/serverActivity.js";
import { asyncHandler } from "../utils/asyncHandler.js";
@@ -21,9 +23,9 @@ import { getSettings } from "../services/settingsStore.js";
export const serversRouter = Router();
/** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */
function publicServer<T extends { apiTokenHash: string; tags: string | null }>(row: T) {
const { apiTokenHash: _hash, tags, ...rest } = row;
return { ...rest, tags: parseTags(tags) };
function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null }>(row: T) {
const { apiTokenHash: _hash, tags, pendingUpdates, ...rest } = row;
return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates) };
}
serversRouter.use(requireAuth);
@@ -113,6 +115,8 @@ const updateServerSchema = z
proxmoxGuestType: z.enum(["qemu", "lxc"]).nullable().optional(),
proxmoxVmid: z.number().int().nullable().optional(),
hideProxmoxLink: z.boolean().optional(),
// The server this one runs on. Null goes back to working it out from the Proxmox node.
parentServerId: z.number().int().nullable().optional(),
})
.refine(
(data) => {
@@ -148,6 +152,17 @@ serversRouter.patch("/:id", requireRole("admin"), asyncHandler(async (req, res)
}
}
if (parsed.data.parentServerId !== undefined && parsed.data.parentServerId !== null) {
const parentId = parsed.data.parentServerId;
const rows = await loadParentRows();
if (parentId === id || !rows.some((r) => r.id === parentId)) {
return res.status(400).json({ error: "invalid_parent", message: "Choose another server that exists." });
}
if (wouldLoop(rows, id, parentId)) {
return res.status(400).json({ error: "parent_loop", message: "That server already runs on this one, so each would be waiting for the other." });
}
}
const [updated] = await db.update(servers).set(parsed.data).where(eq(servers.id, id)).returning();
await recordAudit({
@@ -156,7 +171,7 @@ serversRouter.patch("/:id", requireRole("admin"), asyncHandler(async (req, res)
action: "update",
targetType: "server",
targetId: id,
detail: { name: updated.name },
detail: { name: updated.name, ...(parsed.data.parentServerId !== undefined ? { parentServerId: parsed.data.parentServerId } : {}) },
});
const serverOut = publicServer(updated);
@@ -173,6 +188,27 @@ serversRouter.get("/summary", asyncHandler(async (_req, res) => {
res.json(buildServerSummary(rows, healthChecks, Date.now(), await activeSubjects()));
}));
// What every server that reports through an agent says is waiting to be installed, for the Updates page. Servers whose agent hasn't
// said (it predates this, or couldn't find out) are listed with no figures rather than left out.
serversRouter.get("/updates", asyncHandler(async (_req, res) => {
const rows = await db
.select({
id: servers.id,
name: servers.name,
osType: servers.osType,
lastSeenAt: servers.lastSeenAt,
pendingUpdates: servers.pendingUpdates,
updatesPendingSince: servers.updatesPendingSince,
})
.from(servers)
.orderBy(servers.name);
res.json({
servers: rows
.filter((r) => r.lastSeenAt !== null)
.map((r) => ({ id: r.id, name: r.name, osType: r.osType, lastSeenAt: r.lastSeenAt, updates: parsePendingUpdates(r.pendingUpdates), pendingSince: r.updatesPendingSince })),
});
}));
// For the Operations > Admin Links page — every server's admin bookmarks in one place, instead of visiting
// each server's own detail page to find them.
serversRouter.get("/links", asyncHandler(async (_req, res) => {
@@ -306,10 +342,21 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
memUsedBytes: _memUsedBytes,
listeningPorts: _listeningPorts,
lastPortScan: _lastPortScan,
pendingUpdates: rawUpdates,
...serverOut
} = server;
res.json({ server: { ...serverOut, tags: parseTags(rawTags) }, ipAddresses, hardware, dnsMatches, links });
// What this server runs on, and what runs on it (set by hand, or worked out from its Proxmox node).
const parentRows = await loadParentRows();
const links2 = resolveParents(parentRows);
const nameOf = new Map(parentRows.map((r) => [r.id, r.name]));
const parentLink = links2.get(id);
const relations = {
parent: parentLink ? { id: parentLink.parentId, name: nameOf.get(parentLink.parentId) ?? "", auto: parentLink.auto } : null,
children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)),
};
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates) }, ipAddresses, hardware, dnsMatches, links, relations });
}));
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
@@ -440,6 +487,9 @@ serversRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res)
const id = Number(req.params.id);
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
// Servers that ran on this one go back to working out where they run (the link is a plain column, not a foreign key).
await db.update(servers).set({ parentServerId: null }).where(eq(servers.parentServerId, id));
const deleted = await db.delete(servers).where(eq(servers.id, id)).returning();
if (deleted.length === 0) return res.status(404).json({ error: "not_found" });
+4
View File
@@ -18,6 +18,10 @@ export interface Alert {
link: string | null;
/** Under an active maintenance window: still a real problem, but notifications for it are held back. */
silenced: boolean;
/** Set when this problem is explained by another one (a guest down because its host is): it is held back like a silenced one, but the reason is this, not maintenance. */
causedBy?: { name: string };
/** On a host's alert: the servers on it whose own alerts are folded under this one. */
affects?: string[];
/** What a maintenance window would have to be on to silence it, when there is such a thing — which is what "Silence" acts on. */
target: { type: "server" | "integration" | "dns_provider"; id: number; name: string } | null;
}
+24 -5
View File
@@ -28,12 +28,13 @@ import { computeSecretStatus } from "./secretStatus.js";
import { getFailingSources } from "./integrationHealthMonitor.js";
import { getSettings } from "./settingsStore.js";
import { sourceLabel } from "./notify.js";
import { coveredByParent, loadParentRows, resolveParents } from "./serverParents.js";
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
export interface AlertsReport {
alerts: Alert[];
/** Active problems by severity — those under a maintenance window are counted apart, not in these. */
counts: { critical: number; warning: number; info: number; silenced: number };
counts: { critical: number; warning: number; info: number; silenced: number; /** Held back because another problem explains them (a guest down because its host is). */ grouped: number };
/** Things that couldn't be checked, and which checks that leaves blind. */
couldntCheck: { name: string; error: string; affects: string[] }[];
/** Context worth knowing about how complete the picture is. */
@@ -115,7 +116,7 @@ export async function collectAlerts(): Promise<AlertsReport> {
return row ? integrationTarget(row.id) : null;
};
function push(a: { category: AlertCategory; severity: AlertSeverity; type: string; message: string; key: string; link?: string | null; silenced?: boolean; target?: Alert["target"] }) {
function push(a: { category: AlertCategory; severity: AlertSeverity; type: string; message: string; key: string; link?: string | null; silenced?: boolean; target?: Alert["target"]; causedBy?: Alert["causedBy"]; affects?: string[] }) {
const meta = SOURCES[a.type];
alerts.push({
id: `${a.category}:${a.key}`,
@@ -124,8 +125,10 @@ export async function collectAlerts(): Promise<AlertsReport> {
source: meta?.label ?? sourceLabel(a.type),
message: a.message,
link: a.link === undefined ? (meta?.link ?? null) : a.link,
silenced: !!a.silenced,
silenced: !!a.silenced || !!a.causedBy,
target: a.target ?? null,
...(a.causedBy ? { causedBy: a.causedBy } : {}),
...(a.affects?.length ? { affects: a.affects } : {}),
});
}
@@ -160,10 +163,24 @@ export async function collectAlerts(): Promise<AlertsReport> {
`Server-offline and server-disk checks are paused for another ${Math.ceil(grace / 60_000)} min after the app restarted, so agents get a chance to report before any server is judged.`,
);
}
for (const c of evaluateHealth(snapshot, healthChecks, now, { skipServers: grace > 0 })) {
const conditions = evaluateHealth(snapshot, healthChecks, now, { skipServers: grace > 0 });
// A server that is down because the one it runs on is down is folded under that one.
const offlineIds = new Set<number>();
for (const c of conditions) {
const m = /^offline:server:(\d+)$/.exec(c.key);
if (m) offlineIds.add(Number(m[1]));
}
const cause = coveredByParent(offlineIds, resolveParents(await loadParentRows()));
const taken = new Map<number, string[]>();
for (const [child, parent] of cause) taken.set(parent, [...(taken.get(parent) ?? []), serverNames.get(child) ?? `#${child}`]);
for (const c of conditions) {
const origin = originOfConditionKey(c.key);
const subject = subjectOfConditionKey(c.key);
const offlineId = /^offline:server:(\d+)$/.exec(c.key)?.[1];
const parentOf = offlineId ? cause.get(Number(offlineId)) : undefined;
push({
causedBy: parentOf !== undefined ? { name: serverNames.get(parentOf) ?? `#${parentOf}` } : undefined,
affects: offlineId ? taken.get(Number(offlineId))?.sort((a, b) => a.localeCompare(b)) : undefined,
category: origin.category,
severity: c.severity ?? "warning",
type: origin.type,
@@ -403,13 +420,15 @@ export async function collectAlerts(): Promise<AlertsReport> {
);
const active = alerts.filter((a) => !a.silenced);
const grouped = alerts.filter((a) => a.causedBy).length;
return {
alerts,
counts: {
critical: active.filter((a) => a.severity === "critical").length,
warning: active.filter((a) => a.severity === "warning").length,
info: active.filter((a) => a.severity === "info").length,
silenced: alerts.length - active.length,
silenced: alerts.length - active.length - grouped,
grouped,
},
couldntCheck: [...blind.entries()].map(([name, v]) => ({ name, error: v.error, affects: [...v.affects] })),
notes,
+54 -9
View File
@@ -8,10 +8,11 @@ import { notifyHealthIssues, notifyHealthRecovered } from "./notify.js";
import { recordAudit } from "./audit.js";
import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.js";
import { activeSubjects, subjectOfConditionKey } from "./maintenance.js";
import { coveredByParent, loadParentRows, resolveParents } from "./serverParents.js";
const STATE_FLAG = "healthActiveConditions";
/** After a restart the agents haven't had a chance to report yet (they run every 15 min), so server-derived conditions are held rather than judged. */
const STARTUP_GRACE_MS = 20 * 60 * 1000;
const STARTUP_GRACE_MS = Number(process.env.HEALTH_STARTUP_GRACE_MS ?? 20 * 60 * 1000); // the environment variable is for tests
const PROCESS_START = Date.now();
// ─── Types ──────────────────────────────────────────────────────────────────
@@ -212,7 +213,8 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
// ─── State diff (pure) ──────────────────────────────────────────────────────
export type ActiveState = Record<string, { message: string; source: string }>;
/** `covered`: the problem is explained by another that is being reported (a guest down because its host is) — recorded, but not announced on its own. */
export type ActiveState = Record<string, { message: string; source: string; covered?: boolean }>;
/** `held` is a set of hierarchical sources that couldn't be read this run; matches the source itself or anything beneath it. */
function isHeld(source: string, held: Set<string>): boolean {
@@ -226,6 +228,10 @@ function isHeld(source: string, held: Set<string>): boolean {
* now is resolved — except when its source couldn't be read this run, in
* which case it's carried over untouched. Without that, one failed poll would
* report a recovery and then re-alert the same problem on the next poll.
*
* `covered` are keys of problems explained by another (a guest that is down because its host is). They are kept in the state and
* counted in `newlyDown` / `ended` (what happened, for the record), but are neither announced as new nor as resolved — until the
* explanation goes away while they are still there, when they are announced as new: an outage is never swallowed.
*/
export function diffConditions(
previous: ActiveState,
@@ -233,7 +239,16 @@ export function diffConditions(
held: Set<string> = new Set(),
/** Subjects ("server:3", "integration:1") under an active maintenance window. */
silenced: Set<string> = new Set(),
): { added: HealthCondition[]; resolved: { key: string; message: string }[]; next: ActiveState } {
covered: Set<string> = new Set(),
): {
added: HealthCondition[];
resolved: { key: string; message: string }[];
next: ActiveState;
/** Every problem that began, announced or not. */
newlyDown: HealthCondition[];
/** Every problem that ended, announced or not. */
ended: { key: string; message: string }[];
} {
const isSilenced = (key: string) => {
const subject = subjectOfConditionKey(key);
return subject !== null && silenced.has(subject);
@@ -243,20 +258,24 @@ export function diffConditions(
const visible = current.filter((c) => !isSilenced(c.key));
const next: ActiveState = {};
for (const c of visible) next[c.key] = { message: c.message, source: c.source };
for (const c of visible) next[c.key] = covered.has(c.key) ? { message: c.message, source: c.source, covered: true } : { message: c.message, source: c.source };
const added = visible.filter((c) => !(c.key in previous));
const newlyDown = visible.filter((c) => !(c.key in previous));
// Announced: new and standing alone, or covered until now and no longer.
const added = visible.filter((c) => !covered.has(c.key) && (!(c.key in previous) || previous[c.key].covered));
const resolved: { key: string; message: string }[] = [];
const ended: { key: string; message: string }[] = [];
for (const [key, entry] of Object.entries(previous)) {
if (key in next) continue;
// Held (source unreadable) or silenced: leave it exactly as it was — neither cleared nor re-alerted.
if (isHeld(entry.source, held) || isSilenced(key)) {
next[key] = entry;
} else {
resolved.push({ key, message: entry.message });
ended.push({ key, message: entry.message });
if (!entry.covered) resolved.push({ key, message: entry.message });
}
}
return { added, resolved, next };
return { added, resolved, next, newlyDown, ended };
}
// ─── Collection (I/O) ───────────────────────────────────────────────────────
@@ -346,6 +365,14 @@ export async function recordServerTransitions(added: { key: string; message: str
await write(resolved, { offline: "back_online", disk: "disk_ok" }, false);
}
/** "a, b and c", "a, b, c, d, e and 3 more". */
export function alsoList(names: string[]): string {
const shown = names.slice(0, 5);
const rest = names.length - shown.length;
const head = shown.length > 1 ? `${shown.slice(0, -1).join(", ")} and ${shown[shown.length - 1]}` : shown[0];
return rest > 0 ? `${shown.join(", ")} and ${rest} more` : head;
}
export async function runHealthCheck(now: number = Date.now()): Promise<{ added: number; resolved: number }> {
const { healthChecks } = await getSettings();
const { snapshot, held } = await collectSnapshot();
@@ -354,12 +381,30 @@ export async function runHealthCheck(now: number = Date.now()): Promise<{ added:
if (inGrace) held.add("server");
const current = evaluateHealth(snapshot, healthChecks, now, { skipServers: inGrace });
const { added, resolved, next } = diffConditions(await loadState(), current, held, await activeSubjects(new Date(now)));
// A server that is down because the one it runs on is down is folded under that one: recorded, but not announced on its own.
const offlineIds = new Set<number>();
for (const c of current) {
const m = /^offline:server:(\d+)$/.exec(c.key);
if (m) offlineIds.add(Number(m[1]));
}
const cause = coveredByParent(offlineIds, resolveParents(await loadParentRows()));
const coveredKeys = new Set([...cause.keys()].map((id) => `offline:server:${id}`));
const { added: announced, resolved, next, newlyDown, ended } = diffConditions(await loadState(), current, held, await activeSubjects(new Date(now)), coveredKeys);
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
// A host's message says what else it takes with it.
const names = new Map(snapshot.servers.map((s) => [s.id, s.name]));
const taken = new Map<number, string[]>();
for (const [child, parent] of cause) taken.set(parent, [...(taken.get(parent) ?? []), names.get(child) ?? `#${child}`]);
const added = announced.map((c) => {
const m = /^offline:server:(\d+)$/.exec(c.key);
const children = m ? taken.get(Number(m[1])) : undefined;
return children?.length ? { ...c, message: `${c.message} — also takes down ${alsoList(children)}` } : c;
});
// State is tracked even when the alert toggle is off (the notify functions check it),
// so turning alerts back on doesn't dump every long-standing condition at once.
await recordServerTransitions(added, resolved);
await recordServerTransitions(newlyDown, ended);
if (added.length > 0) await notifyHealthIssues(added);
if (resolved.length > 0) await notifyHealthRecovered(resolved);
return { added: added.length, resolved: resolved.length };
@@ -0,0 +1,24 @@
import { materializeSchedules } from "./maintenanceSchedules.js";
// Windows are made a week ahead, so a pass is only needed now and then: to make the next day's, and to pick up a server added to a
// schedule's list. Quarter-hourly keeps a restart from leaving a gap.
const INTERVAL_MS = 15 * 60 * 1000;
const FIRST_PASS_DELAY_MS = 30 * 1000;
let timer: ReturnType<typeof setInterval> | null = null;
async function pass(): Promise<void> {
try {
const made = await materializeSchedules();
if (made > 0) console.log(`[maintenanceSchedules] made ${made} maintenance window${made === 1 ? "" : "s"} from recurring schedules`);
} catch (err) {
console.error("[maintenanceSchedules]", err instanceof Error ? err.message : err);
}
}
export async function initMaintenanceScheduleScheduler(): Promise<void> {
if (timer) return;
timer = setInterval(() => void pass(), INTERVAL_MS);
setTimeout(() => void pass(), FIRST_PASS_DELAY_MS);
console.log(`[maintenanceSchedules] making windows from recurring schedules a week ahead, checked every ${INTERVAL_MS / 60000} minutes`);
}
+145
View File
@@ -0,0 +1,145 @@
/**
* Recurring maintenance: "every Sunday 02:00–04:00 on these servers". A schedule only says when; the windows that silence alerts are
* ordinary maintenance windows made a week ahead (so they show on the timeline and can be cancelled one at a time, like any other).
* Each is marked `schedule:<id>:<date>`, which is how a later pass knows it has already made it — even if someone cancelled it.
*/
import { and, eq, gt, like } from "drizzle-orm";
import { db } from "../db/client.js";
import { maintenanceSchedules, maintenanceWindows, type MaintenanceTargetType } from "../db/schema.js";
import { invalidateAlertsCache } from "./alerts.js";
import { describeTarget } from "./maintenance.js";
export const HORIZON_DAYS = 7;
const DAY_MS = 24 * 3600_000;
export type ScheduleRow = typeof maintenanceSchedules.$inferSelect;
export interface ScheduleTarget {
type: MaintenanceTargetType;
id: number;
}
export interface Occurrence {
start: Date;
end: Date;
/** The local calendar date it starts on, "2026-10-18" — unique per schedule. */
date: string;
}
/** Whether the system knows this time zone name. */
export function isValidTimeZone(tz: string): boolean {
try {
new Intl.DateTimeFormat("en-US", { timeZone: tz });
return true;
} catch {
return false;
}
}
const parts = (utcMs: number, tz: string) => {
const f = new Intl.DateTimeFormat("en-US", { timeZone: tz, hourCycle: "h23", year: "numeric", month: "2-digit", day: "2-digit", hour: "2-digit", minute: "2-digit", second: "2-digit" });
const out: Record<string, number> = {};
for (const p of f.formatToParts(new Date(utcMs))) if (p.type !== "literal") out[p.type] = Number(p.value);
return out;
};
/** How far the clock in `tz` is ahead of UTC at that moment, in ms. */
function offsetAt(utcMs: number, tz: string): number {
const p = parts(utcMs, tz);
return Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second) - Math.floor(utcMs / 1000) * 1000;
}
/** The instant a wall-clock time in `tz` happens. Winter/summer time is settled by asking the zone for its offset at the answer. */
export function zonedTime(year: number, month: number, day: number, hour: number, minute: number, tz: string): Date {
const naive = Date.UTC(year, month - 1, day, hour, minute);
let guess = naive - offsetAt(naive, tz);
guess = naive - offsetAt(guess, tz);
return new Date(guess);
}
const pad = (n: number) => String(n).padStart(2, "0");
/**
* The times a schedule happens that touch [from, to]: those that have not ended by `from` and begin before `to`. A schedule that runs
* past midnight belongs to the day it starts on.
*/
export function occurrencesBetween(schedule: Pick<ScheduleRow, "days" | "startTime" | "durationMinutes" | "timezone">, from: Date, to: Date): Occurrence[] {
const days = new Set<number>(JSON.parse(schedule.days) as number[]);
const [hh, mm] = schedule.startTime.split(":").map(Number);
const out: Occurrence[] = [];
// Walk the calendar dates around the range (a day either side covers a schedule that started yesterday, or zone differences).
const first = parts(from.getTime() - DAY_MS, schedule.timezone);
let cursor = Date.UTC(first.year, first.month - 1, first.day);
const last = to.getTime() + DAY_MS;
for (; cursor <= last; cursor += DAY_MS) {
const d = new Date(cursor);
if (!days.has(d.getUTCDay())) continue;
const y = d.getUTCFullYear();
const m = d.getUTCMonth() + 1;
const day = d.getUTCDate();
const start = zonedTime(y, m, day, hh, mm, schedule.timezone);
const end = new Date(start.getTime() + schedule.durationMinutes * 60_000);
if (end > from && start < to) out.push({ start, end, date: `${y}-${pad(m)}-${pad(day)}` });
}
return out;
}
export function targetsOf(schedule: Pick<ScheduleRow, "targets">): ScheduleTarget[] {
try {
const v = JSON.parse(schedule.targets);
return Array.isArray(v) ? v.filter((t): t is ScheduleTarget => !!t && typeof t.id === "number" && typeof t.type === "string") : [];
} catch {
return [];
}
}
export const sourceOf = (scheduleId: number, date: string) => `schedule:${scheduleId}:${date}`;
/**
* Makes the windows for the coming week that don't exist yet. Safe to run as often as you like: a window already made (including one
* that was cancelled) is left as it is. Returns how many it made.
*/
export async function materializeSchedules(now: Date = new Date(), horizonDays: number = HORIZON_DAYS, onlyScheduleId?: number): Promise<number> {
const schedules = (await db.select().from(maintenanceSchedules)).filter((s) => s.enabled && (onlyScheduleId === undefined || s.id === onlyScheduleId));
const horizon = new Date(now.getTime() + horizonDays * DAY_MS);
let made = 0;
for (const schedule of schedules) {
const occurrences = occurrencesBetween(schedule, now, horizon);
if (occurrences.length === 0) continue;
const existing = await db.select({ source: maintenanceWindows.source, targetType: maintenanceWindows.targetType, targetId: maintenanceWindows.targetId }).from(maintenanceWindows).where(like(maintenanceWindows.source, `schedule:${schedule.id}:%`));
const have = new Set(existing.map((w) => `${w.source}|${w.targetType}:${w.targetId}`));
for (const target of targetsOf(schedule)) {
if (!(await describeTarget(target.type, target.id))) continue; // since removed: nothing left to silence
for (const occ of occurrences) {
const source = sourceOf(schedule.id, occ.date);
if (have.has(`${source}|${target.type}:${target.id}`)) continue;
await db.insert(maintenanceWindows).values({
targetType: target.type,
targetId: target.id,
reason: schedule.reason ? `${schedule.name}: ${schedule.reason}` : schedule.name,
startedAt: occ.start.toISOString(),
endsAt: occ.end.toISOString(),
createdBy: `Recurring: ${schedule.name}`,
source,
});
made++;
}
}
}
if (made > 0) invalidateAlertsCache();
return made;
}
/** Takes back the windows a schedule made that haven't begun — when it is switched off or removed. Ones under way end on their own. */
export async function removeFutureWindows(scheduleId: number, now: Date = new Date()): Promise<number> {
const gone = await db
.delete(maintenanceWindows)
.where(and(like(maintenanceWindows.source, `schedule:${scheduleId}:%`), gt(maintenanceWindows.startedAt, now.toISOString())))
.returning({ id: maintenanceWindows.id });
if (gone.length > 0) invalidateAlertsCache();
return gone.length;
}
export async function deleteSchedule(scheduleId: number): Promise<void> {
await removeFutureWindows(scheduleId);
await db.delete(maintenanceSchedules).where(eq(maintenanceSchedules.id, scheduleId));
}
+62
View File
@@ -0,0 +1,62 @@
/**
* What each server says is waiting to be installed: how many updates, how many of them security fixes, whether a restart is due — and
* how long updates have been waiting, which the agent can't know and so is worked out here from one report to the next.
*/
/** What an agent reports about updates, as it arrives. */
export interface IncomingUpdates {
/** apt, dnf, zypper, windows-update … — what it asked. */
manager?: string;
count: number;
/** Null when this package manager can't say which are security fixes. */
security?: number | null;
reboot_required?: boolean;
}
export interface PendingUpdates {
manager: string;
count: number;
security: number | null;
rebootRequired: boolean;
/** ISO — when the agent last told us. */
reportedAt: string;
}
export function parsePendingUpdates(stored: string | null): PendingUpdates | null {
if (!stored) return null;
try {
const v = JSON.parse(stored);
if (!v || typeof v !== "object" || typeof v.count !== "number") return null;
return {
manager: typeof v.manager === "string" ? v.manager : "",
count: v.count,
security: typeof v.security === "number" ? v.security : null,
rebootRequired: v.rebootRequired === true,
reportedAt: typeof v.reportedAt === "string" ? v.reportedAt : "",
};
} catch {
return null;
}
}
/**
* The columns to store after a report. Updates waiting since "since" keep that moment for as long as some are still waiting; when
* none are, it is cleared, and the next time some appear it starts again from then.
*/
export function nextUpdateState(
previous: { pendingUpdates: string | null; updatesPendingSince: string | null },
incoming: IncomingUpdates,
now: string,
): { pendingUpdates: string; updatesPendingSince: string | null } {
const before = parsePendingUpdates(previous.pendingUpdates);
const waiting = incoming.count > 0;
const since = waiting ? (before && before.count > 0 && previous.updatesPendingSince ? previous.updatesPendingSince : now) : null;
const record: PendingUpdates = {
manager: incoming.manager ?? "",
count: incoming.count,
security: typeof incoming.security === "number" ? incoming.security : null,
rebootRequired: incoming.reboot_required === true,
reportedAt: now,
};
return { pendingUpdates: JSON.stringify(record), updatesPendingSince: since };
}
+91
View File
@@ -0,0 +1,91 @@
/**
* Which server runs on which: a virtual machine on its Proxmox host, a container on a Docker box. When the host stops reporting, the
* things on it stop too, and seven alerts about seven guests only hide the one that matters — so those alerts are folded under the
* host's. The link is either set by hand ("runs on") or worked out: a server linked to a Proxmox guest runs on the server named after
* that guest's Proxmox node, when there is exactly one such server.
*/
import { db } from "../db/client.js";
import { servers } from "../db/schema.js";
export interface ParentRow {
id: number;
name: string;
hostname: string | null;
parentServerId: number | null;
proxmoxNode: string | null;
}
export interface ParentLink {
parentId: number;
/** Worked out from the Proxmox node rather than set by hand. */
auto: boolean;
}
const lc = (s: string | null | undefined) => (s ?? "").trim().toLowerCase();
/** Whether a server is the machine a Proxmox node name refers to: by its name, its hostname, or the first part of a fully qualified hostname. */
function isNode(row: ParentRow, node: string): boolean {
const n = lc(node);
if (!n) return false;
const host = lc(row.hostname);
return lc(row.name) === n || host === n || host.split(".")[0] === n;
}
/** Each server's parent, where it has one. A link set by hand wins; otherwise the unique server that is its Proxmox node. */
export function resolveParents(rows: ParentRow[]): Map<number, ParentLink> {
const byId = new Map(rows.map((r) => [r.id, r]));
const out = new Map<number, ParentLink>();
for (const row of rows) {
if (row.parentServerId !== null && row.parentServerId !== row.id && byId.has(row.parentServerId)) {
out.set(row.id, { parentId: row.parentServerId, auto: false });
continue;
}
if (row.proxmoxNode) {
const candidates = rows.filter((r) => r.id !== row.id && isNode(r, row.proxmoxNode!));
if (candidates.length === 1) out.set(row.id, { parentId: candidates[0].id, auto: true });
}
}
return out;
}
/**
* Of the servers that are down, those whose cause is another server that is down too: each maps to the topmost ancestor that is down —
* the root cause. A chain (guest → VM host → the box it hangs off) is followed all the way up; a loop in the links covers nothing.
*/
export function coveredByParent(offline: Set<number>, parents: Map<number, ParentLink>): Map<number, number> {
const covered = new Map<number, number>();
for (const id of offline) {
const seen = new Set<number>([id]);
let root: number | null = null;
let looped = false;
let link = parents.get(id);
while (link && !seen.has(link.parentId)) {
if (offline.has(link.parentId)) root = link.parentId;
seen.add(link.parentId);
link = parents.get(link.parentId);
}
if (link && seen.has(link.parentId)) looped = true;
// A loop explains nothing (each would be waiting for the other), so its members stand alone and are announced.
if (root !== null && !looped) covered.set(id, root);
}
return covered;
}
/** Would making `parentId` the parent of `id` close a loop? True if `id` is `parentId` or already sits above it. */
export function wouldLoop(rows: ParentRow[], id: number, parentId: number): boolean {
const parents = resolveParents(rows);
const seen = new Set<number>();
let cur: number | undefined = parentId;
while (cur !== undefined && !seen.has(cur)) {
if (cur === id) return true;
seen.add(cur);
cur = parents.get(cur)?.parentId;
}
return false;
}
export async function loadParentRows(): Promise<ParentRow[]> {
return db
.select({ id: servers.id, name: servers.name, hostname: servers.hostname, parentServerId: servers.parentServerId, proxmoxNode: servers.proxmoxNode })
.from(servers);
}
+6
View File
@@ -2,6 +2,7 @@ import { and, eq, notInArray } from "drizzle-orm";
import { db } from "../db/client.js";
import { scheduledTasks, servers } from "../db/schema.js";
import { recordServerMetrics } from "./serverMetrics.js";
import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js";
export interface IncomingTask {
scheduleType: "cron" | "systemd_timer" | "windows_task";
@@ -20,6 +21,8 @@ export interface IncomingSystemInfo {
memory?: { total_bytes?: number; used_bytes?: number };
disks?: { mount: string; size_bytes: number; used_bytes: number }[];
listening_ports?: { protocol: "tcp" | "udp"; port: number; address: string; process?: string }[];
/** Updates waiting to be installed. Absent from an agent that doesn't report them (or couldn't find out this time). */
updates?: IncomingUpdates;
}
export interface AgentReport {
@@ -98,10 +101,13 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
}
const system = report.system;
const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince }).from(servers).where(eq(servers.id, serverId));
await db
.update(servers)
.set({
lastSeenAt: now,
// Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before.
...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}),
...(report.hostname ? { hostname: report.hostname } : {}),
// An agent knows what it runs on better than whoever registered the server did.
...(report.osType === "linux" || report.osType === "windows" ? { osType: report.osType } : {}),
+202
View File
@@ -0,0 +1,202 @@
/**
* Recurring maintenance: when a schedule happens (across summer and winter time), and the windows made from it.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-sched-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, maintenanceWindows, maintenanceSchedules, auditLog } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { maintenanceRouter } = await import("../src/routes/maintenance.js");
const { occurrencesBetween, zonedTime, materializeSchedules, isValidTimeZone } = await import("../src/services/maintenanceSchedules.js");
const { listActiveWindows } = await import("../src/services/maintenance.js");
const { eq, like } = await import("drizzle-orm");
await runMigrations();
const sched = (days: number[], startTime: string, durationMinutes: number, timezone: string) => ({ days: JSON.stringify(days), startTime, durationMinutes, timezone });
const iso = (d: Date) => d.toISOString();
describe("when a schedule happens", () => {
test("a wall-clock time follows summer and winter time", () => {
// Sundays in Stockholm: 18 October is still summer time (UTC+2), 1 November is winter time (UTC+1).
const s = sched([0], "02:00", 120, "Europe/Stockholm");
const occ = occurrencesBetween(s, new Date("2026-10-15T00:00:00Z"), new Date("2026-11-03T00:00:00Z"));
const byDate = Object.fromEntries(occ.map((o) => [o.date, [iso(o.start), iso(o.end)]]));
assert.deepEqual(byDate["2026-10-18"], ["2026-10-18T00:00:00.000Z", "2026-10-18T02:00:00.000Z"]);
assert.deepEqual(byDate["2026-11-01"], ["2026-11-01T01:00:00.000Z", "2026-11-01T03:00:00.000Z"]);
assert.deepEqual(Object.keys(byDate).sort(), ["2026-10-18", "2026-10-25", "2026-11-01"]);
});
test("it is the same wall-clock time in another zone, and a time that doesn't exist that night still happens", () => {
assert.equal(iso(zonedTime(2026, 7, 1, 2, 0, "America/New_York")), "2026-07-01T06:00:00.000Z");
assert.equal(iso(zonedTime(2026, 1, 1, 2, 0, "America/New_York")), "2026-01-01T07:00:00.000Z");
// 29 March 2026: clocks in Stockholm jump from 02:00 to 03:00, so 02:30 isn't a time that night.
const gap = zonedTime(2026, 3, 29, 2, 30, "Europe/Stockholm");
assert.ok(Math.abs(gap.getTime() - Date.parse("2026-03-29T01:00:00Z")) <= 3600_000);
assert.equal(zonedTime(2026, 10, 18, 0, 0, "UTC").toISOString(), "2026-10-18T00:00:00.000Z");
});
test("one that runs past midnight belongs to the day it starts, and one under way counts", () => {
const s = sched([6], "23:00", 180, "UTC"); // Saturday 23:00 for three hours
const [occ] = occurrencesBetween(s, new Date("2026-10-18T00:30:00Z"), new Date("2026-10-18T06:00:00Z"));
assert.equal(occ.date, "2026-10-17");
assert.equal(iso(occ.start), "2026-10-17T23:00:00.000Z");
assert.equal(iso(occ.end), "2026-10-18T02:00:00.000Z");
assert.equal(occurrencesBetween(s, new Date("2026-10-18T02:00:00Z"), new Date("2026-10-18T06:00:00Z")).length, 0, "already over");
});
test("every day, and only the days chosen", () => {
const week = occurrencesBetween(sched([0, 1, 2, 3, 4, 5, 6], "04:00", 60, "UTC"), new Date("2026-10-12T00:00:00Z"), new Date("2026-10-19T00:00:00Z"));
assert.equal(week.length, 7);
const weekdays = occurrencesBetween(sched([1, 3], "04:00", 60, "UTC"), new Date("2026-10-12T00:00:00Z"), new Date("2026-10-19T00:00:00Z"));
assert.deepEqual(weekdays.map((o) => o.date), ["2026-10-12", "2026-10-14"]);
});
test("time zone names are checked", () => {
assert.equal(isValidTimeZone("Europe/Stockholm"), true);
assert.equal(isValidTimeZone("Mars/Olympus"), false);
});
});
describe("the schedules and their windows", () => {
let app: Server;
let base = "";
let who = "operator";
let a = 0;
let b = 0;
before(async () => {
await db.insert(users).values([
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
const made = await db
.insert(servers)
.values([
{ name: "elsa", apiTokenHash: "a", apiTokenPrefix: "a" },
{ name: "anna", apiTokenHash: "b", apiTokenPrefix: "b" },
])
.returning({ id: servers.id });
a = made[0].id;
b = made[1].id;
const app0 = express();
app0.use(express.json());
app0.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
app0.use("/api/maintenance", maintenanceRouter);
app = await new Promise<Server>((resolve) => {
const s = app0.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/maintenance`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const call = (method: string, path: string, body?: unknown) =>
fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
const good = { name: "Patch night", reason: "OS updates", targets: [{ type: "server", id: 0 }], days: [0, 1, 2, 3, 4, 5, 6], startTime: "03:00", durationMinutes: 60, timezone: "UTC" };
const windowsOf = async (id: number) => db.select().from(maintenanceWindows).where(like(maintenanceWindows.source, `schedule:${id}:%`));
let scheduleId = 0;
test("making a schedule makes the coming week's windows, once each, for every server on it", async () => {
const res = await call("POST", "/schedules", { ...good, targets: [{ type: "server", id: a }, { type: "server", id: b }, { type: "server", id: a }] });
assert.equal(res.status, 201);
const body = await res.json();
scheduleId = body.id;
const windows = await windowsOf(scheduleId);
assert.ok(body.windowsMade === windows.length && (windows.length === 14 || windows.length === 16), `made ${windows.length}`);
assert.equal(new Set(windows.map((w) => `${w.targetId}|${w.source}`)).size, windows.length, "no duplicates");
assert.ok(windows.every((w) => w.reason === "Patch night: OS updates" && w.createdBy === "Recurring: Patch night"));
assert.ok(windows.every((w) => w.endsAt > w.startedAt));
// running again makes nothing more
assert.equal(await materializeSchedules(), 0);
assert.equal((await windowsOf(scheduleId)).length, windows.length);
});
test("a window cancelled by hand isn't made again, and the list says what is next", async () => {
const before = await windowsOf(scheduleId);
const one = before.find((w) => w.startedAt > new Date().toISOString())!;
await db.update(maintenanceWindows).set({ endsAt: one.startedAt }).where(eq(maintenanceWindows.id, one.id));
assert.equal(await materializeSchedules(), 0);
assert.equal((await windowsOf(scheduleId)).length, before.length);
const list = await (await call("GET", "/schedules")).json();
assert.equal(list.schedules.length, 1);
assert.equal(list.schedules[0].targets.length, 2);
assert.deepEqual(list.schedules[0].days, [0, 1, 2, 3, 4, 5, 6]);
assert.ok(Date.parse(list.schedules[0].nextStart) > 0);
});
test("a window made from a schedule silences while it runs", async () => {
const now = Date.now();
await db.insert(maintenanceWindows).values({ targetType: "server", targetId: a, reason: "x", startedAt: new Date(now - 60_000).toISOString(), endsAt: new Date(now + 600_000).toISOString(), source: `schedule:${scheduleId}:2000-01-01` });
assert.ok((await listActiveWindows()).some((w) => w.source === `schedule:${scheduleId}:2000-01-01`));
await db.delete(maintenanceWindows).where(eq(maintenanceWindows.source, `schedule:${scheduleId}:2000-01-01`));
});
test("switching it off takes back what hasn't begun, and on makes it again", async () => {
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: false })).status, 200);
assert.equal((await windowsOf(scheduleId)).filter((w) => w.startedAt > new Date().toISOString()).length, 0);
assert.equal(await materializeSchedules(), 0, "off means nothing is made");
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: true })).status, 200);
assert.ok((await windowsOf(scheduleId)).length >= 12);
});
test("only operators change schedules; a bad one is refused", async () => {
who = "viewer";
assert.equal((await call("POST", "/schedules", { ...good, targets: [{ type: "server", id: a }] })).status, 403);
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: false })).status, 403);
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 403);
assert.equal((await call("GET", "/schedules")).status, 200, "anyone signed in can see them");
who = "operator";
const target = [{ type: "server", id: a }];
for (const bad of [
{ ...good, targets: target, name: "" },
{ ...good, targets: [] },
{ ...good, targets: target, days: [] },
{ ...good, targets: target, days: [7] },
{ ...good, targets: target, startTime: "25:00" },
{ ...good, targets: target, startTime: "3am" },
{ ...good, targets: target, durationMinutes: 5 },
{ ...good, targets: target, timezone: "Mars/Olympus" },
{ ...good, targets: [{ type: "server", id: 99999 }] },
]) {
assert.equal((await call("POST", "/schedules", bad)).status, 400, JSON.stringify(bad).slice(0, 80));
}
});
test("removing a schedule takes back what hasn't begun and is in the audit log", async () => {
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 204);
assert.equal((await db.select().from(maintenanceSchedules)).length, 0);
assert.equal((await windowsOf(scheduleId)).filter((w) => w.startedAt > new Date().toISOString()).length, 0);
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 404);
const actions = (await db.select().from(auditLog)).filter((e) => e.category === "maintenance").map((e) => e.action);
for (const wanted of ["create_schedule", "disable_schedule", "enable_schedule", "delete_schedule"]) assert.ok(actions.includes(wanted), wanted);
});
});
+306
View File
@@ -0,0 +1,306 @@
/**
* Folding a guest's "offline" alert under its host's, and what servers say is waiting to be installed.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-parents-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
// No waiting after a restart before servers are judged, or nothing here could be "offline".
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
const { diffConditions, alsoList } = await import("../src/services/healthMonitor.js");
const { resolveParents, coveredByParent, wouldLoop } = await import("../src/services/serverParents.js");
const { nextUpdateState, parsePendingUpdates } = await import("../src/services/pendingUpdates.js");
const { serversRouter } = await import("../src/routes/servers.js");
const { agentReportRouter } = await import("../src/routes/agentReport.js");
const { hashToken } = await import("../src/services/tokens.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
const row = (id: number, name: string, extra: Partial<{ hostname: string | null; parentServerId: number | null; proxmoxNode: string | null }> = {}) => ({
id,
name,
hostname: null,
parentServerId: null,
proxmoxNode: null,
...extra,
});
describe("resolveParents", () => {
test("a link set by hand wins, and a Proxmox node that is exactly one server is found", () => {
const rows = [row(1, "pve1", { hostname: "pve1.lab.example" }), row(2, "web", { proxmoxNode: "PVE1" }), row(3, "db", { parentServerId: 2, proxmoxNode: "pve1" }), row(4, "lone")];
const parents = resolveParents(rows);
assert.deepEqual(parents.get(2), { parentId: 1, auto: true });
assert.deepEqual(parents.get(3), { parentId: 2, auto: false }, "by hand beats the node");
assert.equal(parents.get(4), undefined);
assert.equal(parents.get(1), undefined);
});
test("two servers that could be the node, or none, give no parent; a dead link is ignored", () => {
const rows = [row(1, "pve1"), row(2, "pve1", { hostname: "other" }), row(3, "web", { proxmoxNode: "pve1" }), row(4, "x", { proxmoxNode: "nothing" }), row(5, "y", { parentServerId: 99 }), row(6, "z", { parentServerId: 6 })];
const parents = resolveParents(rows);
assert.equal(parents.get(3), undefined, "ambiguous");
assert.equal(parents.get(4), undefined);
assert.equal(parents.get(5), undefined);
assert.equal(parents.get(6), undefined, "not its own parent");
});
test("a server is not its own node", () => {
assert.equal(resolveParents([row(1, "pve1", { proxmoxNode: "pve1" })]).size, 0);
});
});
describe("coveredByParent", () => {
const parents = resolveParents([row(1, "host"), row(2, "vm", { parentServerId: 1 }), row(3, "ct", { parentServerId: 2 }), row(4, "other", { parentServerId: 1 })]);
test("a guest is covered only while something above it is down, and by the topmost one", () => {
assert.deepEqual([...coveredByParent(new Set([1, 2, 3]), parents)].sort(), [[2, 1], [3, 1]]);
assert.deepEqual([...coveredByParent(new Set([2, 3]), parents)], [[3, 2]], "the host is up: the VM is the cause");
assert.deepEqual([...coveredByParent(new Set([3]), parents)], [], "nothing above it is down");
assert.deepEqual([...coveredByParent(new Set([1]), parents)], []);
assert.deepEqual([...coveredByParent(new Set([1, 3]), parents)], [[3, 1]], "an up VM in between doesn't hide the cause");
});
test("a loop covers nothing, so nothing is swallowed", () => {
const loop = resolveParents([row(1, "a", { parentServerId: 2 }), row(2, "b", { parentServerId: 1 }), row(3, "c", { parentServerId: 1 })]);
assert.equal(coveredByParent(new Set([1, 2, 3]), loop).size, 0);
});
test("wouldLoop spots a link that would close a loop", () => {
const rows = [row(1, "host"), row(2, "vm", { parentServerId: 1 })];
assert.equal(wouldLoop(rows, 1, 2), true);
assert.equal(wouldLoop(rows, 1, 1), true);
assert.equal(wouldLoop(rows, 2, 1), false);
});
});
describe("diffConditions with covered problems", () => {
const c = (key: string) => ({ key, source: "server", message: key });
const names = (list: { key: string }[]) => list.map((x) => x.key).sort();
test("a problem explained by another is recorded but not announced, and neither is its end", () => {
const first = diffConditions({}, [c("offline:server:1"), c("offline:server:2")], new Set(), new Set(), new Set(["offline:server:2"]));
assert.deepEqual(names(first.added), ["offline:server:1"]);
assert.deepEqual(names(first.newlyDown), ["offline:server:1", "offline:server:2"], "both are down, for the record");
assert.equal(first.next["offline:server:2"].covered, true);
const later = diffConditions(first.next, [c("offline:server:1")], new Set(), new Set(), new Set());
assert.deepEqual(names(later.resolved), [], "the guest's end isn't announced either");
assert.deepEqual(names(later.ended), ["offline:server:2"], "but it did end");
});
test("when the explanation goes away and the problem is still there, it is announced as new", () => {
const first = diffConditions({}, [c("offline:server:1"), c("offline:server:2")], new Set(), new Set(), new Set(["offline:server:2"]));
const hostBack = diffConditions(first.next, [c("offline:server:2")], new Set(), new Set(), new Set());
assert.deepEqual(names(hostBack.added), ["offline:server:2"]);
assert.deepEqual(names(hostBack.resolved), ["offline:server:1"]);
assert.deepEqual(names(hostBack.newlyDown), [], "it was already down; nothing began");
// and only once
const again = diffConditions(hostBack.next, [c("offline:server:2")], new Set(), new Set(), new Set());
assert.deepEqual(again.added, []);
});
test("with nothing covered it works as before", () => {
const run = diffConditions({}, [c("a"), c("b")]);
assert.deepEqual(names(run.added), ["a", "b"]);
assert.deepEqual(names(run.newlyDown), ["a", "b"]);
const gone = diffConditions(run.next, [c("a")]);
assert.deepEqual(names(gone.resolved), ["b"]);
assert.deepEqual(names(gone.ended), ["b"]);
});
test("a list of names reads as words", () => {
assert.equal(alsoList(["a"]), "a");
assert.equal(alsoList(["a", "b", "c"]), "a, b and c");
assert.equal(alsoList(["a", "b", "c", "d", "e", "f", "g"]), "a, b, c, d, e and 2 more");
});
});
describe("the alerts, folded", () => {
const hoursAgo = (n: number) => new Date(Date.now() - n * 3600_000).toISOString();
let hostId = 0;
before(async () => {
const made = await db
.insert(servers)
.values([
{ name: "pve1", apiTokenHash: "h", apiTokenPrefix: "h", lastSeenAt: hoursAgo(2) },
{ name: "web", apiTokenHash: "w", apiTokenPrefix: "w", lastSeenAt: hoursAgo(2), proxmoxNode: "pve1" },
{ name: "db", apiTokenHash: "d", apiTokenPrefix: "d", lastSeenAt: hoursAgo(2), proxmoxNode: "pve1" },
{ name: "elsewhere", apiTokenHash: "e", apiTokenPrefix: "e", lastSeenAt: hoursAgo(2) },
])
.returning({ id: servers.id });
hostId = made[0].id;
});
test("the host's alert says what it takes with it, and the guests' are held back, not counted", async () => {
const report = await getAlerts(true);
const offline = report.alerts.filter((a) => a.category === "offline");
assert.equal(offline.length, 4);
const host = offline.find((a) => a.message.startsWith("pve1"))!;
assert.deepEqual(host.affects, ["db", "web"]);
assert.equal(host.silenced, false);
const web = offline.find((a) => a.message.startsWith("web"))!;
assert.deepEqual(web.causedBy, { name: "pve1" });
assert.equal(web.silenced, true, "held back like a silenced alert");
assert.equal(offline.find((a) => a.message.startsWith("elsewhere"))!.causedBy, undefined);
assert.equal(report.counts.critical, 2, "pve1 and the unrelated one");
assert.equal(report.counts.grouped, 2);
assert.equal(report.counts.silenced, 0, "none of this is maintenance");
});
test("when the host reports again the guests that are still down stand alone", async () => {
await db.update(servers).set({ lastSeenAt: new Date().toISOString() }).where(eq(servers.id, hostId));
invalidateAlertsCache(); // a look under ten seconds old is reused otherwise
const report = await getAlerts(true);
assert.equal(report.counts.critical, 3);
assert.equal(report.counts.grouped, 0);
assert.ok(report.alerts.every((a) => !a.causedBy));
});
});
describe("what servers say is waiting to be installed", () => {
test("how long updates have been waiting carries from report to report", () => {
const t0 = "2026-10-01T00:00:00.000Z";
const t1 = "2026-10-05T00:00:00.000Z";
const t2 = "2026-10-09T00:00:00.000Z";
const t3 = "2026-10-12T00:00:00.000Z";
const first = nextUpdateState({ pendingUpdates: null, updatesPendingSince: null }, { manager: "apt", count: 4, security: 1 }, t0);
assert.equal(first.updatesPendingSince, t0);
const second = nextUpdateState(first, { manager: "apt", count: 6, security: 2, reboot_required: true }, t1);
assert.equal(second.updatesPendingSince, t0, "still waiting: the clock keeps running");
assert.deepEqual(parsePendingUpdates(second.pendingUpdates), { manager: "apt", count: 6, security: 2, rebootRequired: true, reportedAt: t1 });
const cleared = nextUpdateState(second, { manager: "apt", count: 0, security: 0 }, t2);
assert.equal(cleared.updatesPendingSince, null);
assert.equal(nextUpdateState(cleared, { manager: "apt", count: 2 }, t3).updatesPendingSince, t3, "and starts again when some appear");
assert.equal(parsePendingUpdates(nextUpdateState(cleared, { count: 2 }, t3).pendingUpdates)?.security, null, "unknown security is unknown, not zero");
});
test("garbage is no updates, not an error", () => {
assert.equal(parsePendingUpdates(null), null);
assert.equal(parsePendingUpdates("{"), null);
assert.equal(parsePendingUpdates(JSON.stringify({ count: "many" })), null);
});
});
describe("the routes", () => {
let app: Server;
let base = "";
let who = "admin";
let hostId = 0;
let guestId = 0;
before(async () => {
await db.insert(users).values([
{ oidcSub: "admin", name: "Ada", role: "admin" },
{ oidcSub: "operator", name: "Olga", role: "operator" },
]);
const [host] = await db.insert(servers).values({ name: "rack", apiTokenHash: hashToken("host-token"), apiTokenPrefix: "ht", lastSeenAt: new Date().toISOString() }).returning();
const [guest] = await db.insert(servers).values({ name: "vm1", apiTokenHash: hashToken("guest-token"), apiTokenPrefix: "gt", lastSeenAt: new Date().toISOString() }).returning();
hostId = host.id;
guestId = guest.id;
const a = express();
a.use("/api/agent/report", agentReportRouter);
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/servers", serversRouter);
app = await new Promise<Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const call = (method: string, path: string, body?: unknown) =>
fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
const report = (token: string, system: unknown) =>
fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${token}` }, body: JSON.stringify({ hostname: "h", os_type: "linux", system, tasks: [] }) });
test("an agent's update counts are stored, a report without them keeps what was said, and a garbled answer costs nothing", async () => {
assert.equal((await report("guest-token", { updates: { manager: "apt", count: 7, security: 2, reboot_required: true } })).status, 202);
let [s] = await db.select().from(servers).where(eq(servers.id, guestId));
assert.deepEqual(parsePendingUpdates(s.pendingUpdates), { manager: "apt", count: 7, security: 2, rebootRequired: true, reportedAt: parsePendingUpdates(s.pendingUpdates)!.reportedAt });
const since = s.updatesPendingSince;
assert.ok(since);
assert.equal((await report("guest-token", { cpu: { cores: 2 } })).status, 202);
[s] = await db.select().from(servers).where(eq(servers.id, guestId));
assert.equal(parsePendingUpdates(s.pendingUpdates)?.count, 7, "an agent that said nothing doesn't wipe it");
assert.equal((await report("guest-token", { updates: { count: "lots" }, cpu: { cores: 4 } })).status, 202, "the report itself still goes through");
[s] = await db.select().from(servers).where(eq(servers.id, guestId));
assert.equal(s.cpuCores, 4);
assert.equal(parsePendingUpdates(s.pendingUpdates)?.count, 7);
assert.equal(s.updatesPendingSince, since);
assert.equal((await report("host-token", { updates: { manager: "dnf", count: 0 } })).status, 202);
});
test("the updates list shows every server that reports, with or without figures", async () => {
await db.insert(servers).values({ name: "silent", apiTokenHash: "s", apiTokenPrefix: "s" }); // never reported
const body = await (await call("GET", "/servers/updates")).json();
const byName = new Map(body.servers.map((x: { name: string }) => [x.name, x]));
assert.ok(!byName.has("silent"));
const vm = byName.get("vm1") as { updates: { count: number; security: number; rebootRequired: boolean }; pendingSince: string };
assert.equal(vm.updates.count, 7);
assert.equal(vm.updates.rebootRequired, true);
assert.ok(vm.pendingSince);
assert.equal((byName.get("rack") as { pendingSince: string | null }).pendingSince, null, "nothing waiting");
});
test("an admin says what a server runs on; a loop or a stranger is refused; it shows on both pages", async () => {
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: hostId })).status, 200);
const detail = await (await call("GET", `/servers/${guestId}/detail`)).json();
assert.deepEqual(detail.relations.parent, { id: hostId, name: "rack", auto: false });
const hostDetail = await (await call("GET", `/servers/${hostId}/detail`)).json();
assert.deepEqual(hostDetail.relations.children, [{ id: guestId, name: "vm1", auto: false }]);
assert.equal(detail.server.pendingUpdates.count, 7, "the detail shows what is waiting, as a figure rather than stored text");
const loop = await call("PATCH", `/servers/${hostId}`, { parentServerId: guestId });
assert.equal(loop.status, 400);
assert.equal((await loop.json()).error, "parent_loop");
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: guestId })).status, 400);
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: 99999 })).status, 400);
who = "operator";
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: null })).status, 403);
who = "admin";
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: null })).status, 200);
assert.equal((await (await call("GET", `/servers/${guestId}/detail`)).json()).relations.parent, null, "back to working it out");
});
test("removing the host sets the servers on it free", async () => {
await call("PATCH", `/servers/${guestId}`, { parentServerId: hostId });
assert.equal((await call("DELETE", `/servers/${hostId}`)).status, 204);
const [s] = await db.select().from(servers).where(eq(servers.id, guestId));
assert.equal(s.parentServerId, null);
});
});
+143
View File
@@ -0,0 +1,143 @@
/**
* Running a Semaphore template from the app, limited to one host, and watching how the run is getting on.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-sem-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, integrations, auditLog } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { integrationsRouter } = await import("../src/routes/integrations.js");
await runMigrations();
describe("running a Semaphore template", () => {
let semaphore: http.Server;
let app: http.Server;
let base = "";
let who = "operator";
let integrationId = 0;
const received: { method: string; url: string; body: Record<string, unknown> | null; auth: string | undefined }[] = [];
before(async () => {
semaphore = await new Promise<http.Server>((resolve) => {
const s = http.createServer((req, res) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
received.push({ method: req.method ?? "", url: req.url ?? "", body: raw ? JSON.parse(raw) : null, auth: req.headers.authorization });
res.setHeader("content-type", "application/json");
if (req.method === "POST" && req.url === "/api/project/3/tasks") return res.end(JSON.stringify({ id: 77, status: "waiting", created: "2026-10-17T10:00:00Z" }));
if (req.method === "GET" && req.url === "/api/project/3/tasks/77") return res.end(JSON.stringify({ id: 77, status: "running", start: "2026-10-17T10:00:05Z" }));
res.statusCode = 404;
res.end(JSON.stringify({ error: "not here" }));
});
});
s.listen(0, "127.0.0.1", () => resolve(s));
});
const url = `http://127.0.0.1:${(semaphore.address() as AddressInfo).port}`;
await db.insert(users).values([
{ oidcSub: "operator", name: "Olga", role: "operator" },
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
]);
const [row] = await db.insert(integrations).values({ type: "semaphore", name: "Semaphore", baseUrl: url, config: JSON.stringify({ url, token: "secret-token" }) }).returning();
integrationId = row.id;
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: who } };
next();
});
a.use("/api/integrations", integrationsRouter);
app = await new Promise<http.Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/integrations/${integrationId}/semaphore`;
});
after(() => {
semaphore.close();
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const run = (body?: unknown) =>
fetch(`${base}/projects/3/templates/9/run`, { method: "POST", headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
test("a run limited to a host tells Semaphore which, and who started it", async () => {
const res = await run({ limit: "elsa" });
assert.equal(res.status, 201);
assert.equal((await res.json()).task.id, 77);
const sent = received.find((r) => r.method === "POST")!;
assert.equal(sent.url, "/api/project/3/tasks");
assert.equal(sent.auth, "Bearer secret-token");
assert.equal(sent.body?.template_id, 9);
assert.equal(sent.body?.limit, "elsa");
assert.match(String(sent.body?.message), /Olga/);
assert.match(String(sent.body?.message), /limited to elsa/);
});
test("a run with no limit sends none, and the audit log records what ran and where", async () => {
received.length = 0;
assert.equal((await run()).status, 201);
assert.equal((await run({})).status, 201);
for (const r of received) {
assert.equal(r.body?.limit, undefined);
assert.doesNotMatch(String(r.body?.message), /limited/);
}
const entries = (await db.select().from(auditLog)).filter((e) => e.action === "run_template");
assert.equal(entries.length, 3);
assert.match(entries[0].detail ?? "", /"limit":"elsa"/);
assert.doesNotMatch(entries[1].detail ?? "", /limit/);
assert.ok(entries.every((e) => !(e.detail ?? "").includes("secret-token")));
});
test("anything that isn't a host or a pattern is refused before it reaches Semaphore", async () => {
received.length = 0;
for (const limit of ["--extra-vars x=1", "-e foo", "elsa; reboot", "$(id)", "a b", "", "x".repeat(201), "../etc"]) {
const res = await run({ limit });
assert.equal(res.status, limit === "" ? 201 : 400, JSON.stringify(limit).slice(0, 40));
}
// the empty one is "no limit"; nothing else was sent on
assert.equal(received.filter((r) => r.method === "POST").length, 1);
for (const ok of ["web01,db*", "elsa.lab.example", "10.0.0.5", "group:&web"]) {
if (ok.includes("&")) continue; // not a pattern this accepts
assert.equal((await run({ limit: ok })).status, 201, ok);
}
});
test("a viewer can't run anything", async () => {
who = "viewer";
received.length = 0;
assert.equal((await run({ limit: "elsa" })).status, 403);
assert.equal(received.length, 0);
who = "operator";
});
test("how a run is getting on can be asked for, and a missing run is reported", async () => {
const res = await fetch(`${base}/projects/3/tasks/77`);
assert.equal(res.status, 200);
assert.deepEqual((await res.json()).task, { id: 77, status: "running", created: null, start: "2026-10-17T10:00:05Z", end: null });
assert.equal((await fetch(`${base}/projects/3/tasks/999`)).status, 502);
assert.equal((await fetch(`${base}/projects/x/tasks/77`)).status, 400);
});
});
+2
View File
@@ -28,6 +28,7 @@ import AdminLinks from "./pages/AdminLinks";
import Alerts from "./pages/Alerts";
import Generator from "./pages/Generator";
import Maintenance from "./pages/Maintenance";
import Updates from "./pages/Updates";
import Domains from "./pages/Domains";
import Consistency from "./pages/Consistency";
import Privacy from "./pages/Privacy";
@@ -136,6 +137,7 @@ function Shelled({ user }: { user: CurrentUser }) {
<Route path="/consistency" element={<Consistency user={user} />} />
<Route path="/domains" element={<Domains user={user} />} />
<Route path="/maintenance" element={<Maintenance user={user} />} />
<Route path="/updates" element={<Updates />} />
<Route path="/docker" element={<Docker user={user} />} />
<Route path="/tailscale" element={<Tailscale user={user} />} />
<Route path="/semaphore" element={<Semaphore user={user} />} />
+78 -3
View File
@@ -490,6 +490,38 @@ export interface ServerRecord {
proxmoxVmid: number | null;
hideProxmoxLink: boolean;
tags: string[];
/** The server this one runs on, when an admin said so; null means it is worked out from the Proxmox node. */
parentServerId: number | null;
/** What the agent says is waiting to be installed; null until it has said. */
pendingUpdates: PendingUpdates | null;
/** When updates were first seen waiting (and none were before); null when none are. */
updatesPendingSince: string | null;
}
export interface PendingUpdates {
manager: string;
count: number;
/** Null when the package manager can't say which are security fixes. */
security: number | null;
rebootRequired: boolean;
reportedAt: string;
}
/** One server on the Updates page. */
export interface UpdatesServer {
id: number;
name: string;
osType: string;
lastSeenAt: string | null;
updates: PendingUpdates | null;
pendingSince: string | null;
}
/** What a server runs on, and what runs on it. `auto`: worked out from the Proxmox node rather than set by hand. */
export interface ServerRelation {
id: number;
name: string;
auto: boolean;
}
export interface ServerHardware {
@@ -531,6 +563,7 @@ export interface ServerDetail {
hardware: ServerHardware;
dnsMatches: DnsMatch[];
links: ServerLink[];
relations: { parent: ServerRelation | null; children: ServerRelation[] };
}
export type AlertSeverity = "critical" | "warning" | "info";
@@ -547,6 +580,10 @@ export interface AlertItem {
silenced: boolean;
/** What a maintenance window would go on to silence it — what the Silence button acts on. Null when nothing can silence it. */
target?: { type: MaintenanceTargetType; id: number; name: string } | null;
/** Held back because another problem explains it (a guest down because its host is): the host's alert says so too. */
causedBy?: { name: string };
/** On a host's alert: the servers on it whose own alerts are folded under this one. */
affects?: string[];
/** The osTicket ticket opened for it, if one was. */
ticket?: { number: string | null; url: string | null; grouped: boolean; openedAt: string | null } | null;
}
@@ -590,7 +627,7 @@ export interface AlertTicketOverview {
export interface AlertsReport {
alerts: AlertItem[];
counts: { critical: number; warning: number; info: number; silenced: number };
counts: { critical: number; warning: number; info: number; silenced: number; grouped: number };
couldntCheck: { name: string; error: string; affects: string[] }[];
notes: string[];
generatedAt: string;
@@ -654,6 +691,34 @@ export interface ServerUpdateInput {
proxmoxGuestType?: "qemu" | "lxc" | null;
proxmoxVmid?: number | null;
hideProxmoxLink?: boolean;
/** The server this one runs on; null goes back to working it out from the Proxmox node. */
parentServerId?: number | null;
}
/** A maintenance that repeats: "every Sunday 02:00–04:00 on these servers". */
export interface MaintenanceSchedule {
id: number;
name: string;
reason: string | null;
/** 0 = Sunday … 6 = Saturday. */
days: number[];
startTime: string;
durationMinutes: number;
timezone: string;
enabled: boolean;
createdBy: string | null;
targets: { type: MaintenanceTargetType; id: number; name: string; kind: string }[];
nextStart: string | null;
}
export interface MaintenanceScheduleInput {
name: string;
reason?: string;
targets: { type: MaintenanceTargetType; id: number }[];
days: number[];
startTime: string;
durationMinutes: number;
timezone: string;
}
export type ScheduleType = "cron" | "systemd_timer" | "windows_task" | "docker" | "backup" | "update" | "n8n_workflow" | "manual";
@@ -1230,6 +1295,12 @@ export const api = {
startBulk: (data: { scope: MaintenanceBulkScope; minutes: number; reason?: string }) =>
request<{ started: number; extended: number; total: number }>("/api/maintenance/bulk", { method: "POST", body: JSON.stringify(data) }),
endAll: () => request<{ ended: number }>("/api/maintenance/end-all", { method: "POST" }),
schedules: {
list: () => request<{ schedules: MaintenanceSchedule[] }>("/api/maintenance/schedules"),
create: (data: MaintenanceScheduleInput) => request<{ id: number; windowsMade: number }>("/api/maintenance/schedules", { method: "POST", body: JSON.stringify(data) }),
setEnabled: (id: number, enabled: boolean) => request<{ enabled: boolean }>(`/api/maintenance/schedules/${id}`, { method: "PATCH", body: JSON.stringify({ enabled }) }),
remove: (id: number) => request<void>(`/api/maintenance/schedules/${id}`, { method: "DELETE" }),
},
syncUptimeKuma: (data: { integrationId: number }) =>
request<UptimeKumaSyncResult>("/api/maintenance/sync/uptimekuma", { method: "POST", body: JSON.stringify(data) }),
importUptimeKuma: (data: { integrationId: number; minutes: number }) =>
@@ -1387,6 +1458,7 @@ export const api = {
update: (id: number, data: ServerUpdateInput) =>
request<{ server: ServerRecord }>(`/api/servers/${id}`, { method: "PATCH", body: JSON.stringify(data) }),
summary: () => request<ServerSummary>("/api/servers/summary"),
updates: () => request<{ servers: UpdatesServer[] }>("/api/servers/updates"),
detail: (id: number) => request<ServerDetail>(`/api/servers/${id}/detail`),
activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`),
metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`),
@@ -1552,11 +1624,14 @@ export const api = {
semaphore: {
templates: (integrationId: number) =>
request<SemaphoreTemplatesResponse>(`/api/integrations/${integrationId}/semaphore/templates`),
run: (integrationId: number, projectId: number, templateId: number) =>
/** `limit` runs it on those hosts only (ansible's --limit). */
run: (integrationId: number, projectId: number, templateId: number, options?: { limit?: string }) =>
request<{ task: SemaphoreTask }>(
`/api/integrations/${integrationId}/semaphore/projects/${projectId}/templates/${templateId}/run`,
{ method: "POST" },
{ method: "POST", body: JSON.stringify(options ?? {}) },
),
task: (integrationId: number, projectId: number, taskId: number) =>
request<{ task: SemaphoreTask }>(`/api/integrations/${integrationId}/semaphore/projects/${projectId}/tasks/${taskId}`),
},
proxmox: {
guests: (integrationId: number) => request<ProxmoxGuestsResponse>(`/api/integrations/${integrationId}/proxmox/guests`),
+4 -1
View File
@@ -129,7 +129,7 @@ export default function MaintenanceTimeline({ windows, now }: { windows: Mainten
return (
<span
key={w.id}
className={`timeline-bar ${running ? "timeline-bar-running" : "timeline-bar-scheduled"} ${w.source ? "timeline-bar-mirrored" : ""}`}
className={`timeline-bar ${running ? "timeline-bar-running" : "timeline-bar-scheduled"} ${w.source?.startsWith("kuma:") ? "timeline-bar-mirrored" : ""} ${w.source?.startsWith("schedule:") ? "timeline-bar-recurring" : ""}`}
style={bar(from, to)}
title={`${row.name}: ${formatDateTime(new Date(from))} – ${formatDateTime(new Date(to))}${w.reason ? ` · ${w.reason}` : ""}`}
/>
@@ -150,6 +150,9 @@ export default function MaintenanceTimeline({ windows, now }: { windows: Mainten
<span>
<span className="timeline-key timeline-bar-mirrored" /> From Uptime Kuma
</span>
<span>
<span className="timeline-key timeline-bar-recurring" /> Recurring
</span>
<span>The same windows are listed in the tables below.</span>
{further > 0 && <span>{further} further ahead than this range.</span>}
</div>
+309
View File
@@ -0,0 +1,309 @@
import { useEffect, useState } from "react";
import { api, type MaintenanceSchedule, type MaintenanceTargetType, type MaintenanceTargets } from "../api/client";
import { confirmDialog } from "../utils/dialogs";
import { readableError } from "../utils/errors";
import { toast } from "../utils/toast";
import TimeAgo from "./TimeAgo";
import { ErrorNotice } from "./Notice";
// Monday first, as a week reads; the numbers are what the server keeps (0 = Sunday).
const DAYS: { value: number; short: string }[] = [
{ value: 1, short: "Mon" },
{ value: 2, short: "Tue" },
{ value: 3, short: "Wed" },
{ value: 4, short: "Thu" },
{ value: 5, short: "Fri" },
{ value: 6, short: "Sat" },
{ value: 0, short: "Sun" },
];
const LENGTHS: { minutes: number; label: string }[] = [
{ minutes: 30, label: "30 minutes" },
{ minutes: 60, label: "1 hour" },
{ minutes: 120, label: "2 hours" },
{ minutes: 180, label: "3 hours" },
{ minutes: 240, label: "4 hours" },
{ minutes: 360, label: "6 hours" },
{ minutes: 480, label: "8 hours" },
{ minutes: 720, label: "12 hours" },
{ minutes: 1440, label: "24 hours" },
];
function daysWords(days: number[]): string {
if (days.length === 7) return "Every day";
const set = new Set(days);
if ([1, 2, 3, 4, 5].every((d) => set.has(d)) && days.length === 5) return "Weekdays";
if (days.length === 2 && set.has(6) && set.has(0)) return "Weekends";
return DAYS.filter((d) => set.has(d.value)).map((d) => d.short).join(", ");
}
/** "02:00–04:00", with "(+1 d)" when it runs into the next day. */
function timeWords(start: string, minutes: number): string {
const [h, m] = start.split(":").map(Number);
const end = h * 60 + m + minutes;
const clock = (t: number) => `${String(Math.floor((t % 1440) / 60)).padStart(2, "0")}:${String(t % 60).padStart(2, "0")}`;
return `${start}–${clock(end)}${end >= 1440 ? ` (+${Math.floor(end / 1440)} d)` : ""}`;
}
function namesWords(names: string[]): string {
return names.length <= 3 ? names.join(", ") : `${names.slice(0, 3).join(", ")} and ${names.length - 3} more`;
}
/**
* Maintenance that comes back by itself — "every Sunday 02:00–04:00 on these servers". Each time is made into an ordinary maintenance
* window a week ahead, so it shows on the timeline and in Scheduled, silences alerts while it runs, and any single one can be cancelled.
*/
export default function RecurringMaintenance({ canEdit, targets, onChanged }: { canEdit: boolean; targets: MaintenanceTargets | null; onChanged: () => void }) {
const [schedules, setSchedules] = useState<MaintenanceSchedule[] | null>(null);
const [error, setError] = useState<string | null>(null);
const [adding, setAdding] = useState(false);
const [busy, setBusy] = useState(false);
const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
const [name, setName] = useState("");
const [reason, setReason] = useState("");
const [days, setDays] = useState<number[]>([0]);
const [startTime, setStartTime] = useState("02:00");
const [minutes, setMinutes] = useState(120);
const [picked, setPicked] = useState<Set<string>>(new Set());
function load() {
api.maintenance.schedules
.list()
.then((res) => setSchedules(res.schedules))
.catch((err) => setError(readableError(err)));
}
useEffect(load, []);
const key = (type: MaintenanceTargetType, id: number) => `${type}:${id}`;
const groups: { label: string; type: MaintenanceTargetType; items: { id: number; name: string }[] }[] = [
{ label: "Servers", type: "server", items: targets?.servers ?? [] },
{ label: "Integrations", type: "integration", items: targets?.integrations ?? [] },
{ label: "DNS providers", type: "dns_provider", items: targets?.dnsProviders ?? [] },
];
function toggleGroup(type: MaintenanceTargetType, items: { id: number }[]) {
setPicked((prev) => {
const next = new Set(prev);
const all = items.every((i) => next.has(key(type, i.id)));
for (const i of items) (all ? next.delete(key(type, i.id)) : next.add(key(type, i.id)));
return next;
});
}
async function create(e: React.FormEvent) {
e.preventDefault();
setError(null);
setBusy(true);
try {
const chosen = [...picked].map((k) => {
const [type, id] = k.split(":");
return { type: type as MaintenanceTargetType, id: Number(id) };
});
const res = await api.maintenance.schedules.create({ name: name.trim(), reason: reason.trim() || undefined, targets: chosen, days, startTime, durationMinutes: minutes, timezone });
toast.success(`Saved. ${res.windowsMade} maintenance window${res.windowsMade === 1 ? "" : "s"} made for the coming week.`);
setName("");
setReason("");
setPicked(new Set());
setAdding(false);
load();
onChanged();
} catch (err) {
setError(readableError(err));
} finally {
setBusy(false);
}
}
async function toggle(s: MaintenanceSchedule) {
try {
await api.maintenance.schedules.setEnabled(s.id, !s.enabled);
load();
onChanged();
} catch (err) {
setError(readableError(err));
}
}
async function remove(s: MaintenanceSchedule) {
if (!(await confirmDialog({ title: "Remove recurring maintenance", message: `Remove "${s.name}"? Windows it made that haven't begun are taken back; one already under way ends on its own.`, confirmLabel: "Remove", danger: true }))) return;
try {
await api.maintenance.schedules.remove(s.id);
toast.success(`Removed ${s.name}.`);
load();
onChanged();
} catch (err) {
setError(readableError(err));
}
}
if (schedules !== null && schedules.length === 0 && !canEdit) return null;
return (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">Recurring</h3>
{canEdit && !adding && (
<div className="card-actions">
<button type="button" className="btn btn-sm btn-primary" onClick={() => setAdding(true)}>
Add recurring maintenance
</button>
</div>
)}
</div>
{adding && (
<form onSubmit={create}>
<div className="card-body row g-3">
<div className="col-md-4">
<label className="form-label" htmlFor="rec-name">
Name
</label>
<input id="rec-name" className="form-control" required maxLength={80} placeholder="e.g. Patch night" value={name} onChange={(e) => setName(e.target.value)} />
</div>
<div className="col-md-8">
<label className="form-label" htmlFor="rec-reason">
Reason (optional)
</label>
<input id="rec-reason" className="form-control" maxLength={200} placeholder="e.g. OS updates and restarts" value={reason} onChange={(e) => setReason(e.target.value)} />
</div>
<div className="col-12">
<div className="form-label" id="rec-days">
Days
</div>
<div className="btn-group" role="group" aria-labelledby="rec-days">
{DAYS.map((d) => {
const on = days.includes(d.value);
return (
<button key={d.value} type="button" className={`btn btn-sm ${on ? "btn-primary" : "btn-outline-secondary"}`} aria-pressed={on} onClick={() => setDays((prev) => (on ? prev.filter((x) => x !== d.value) : [...prev, d.value]))}>
{d.short}
</button>
);
})}
</div>
</div>
<div className="col-sm-6 col-md-3">
<label className="form-label" htmlFor="rec-start">
Starts at
</label>
<input id="rec-start" type="time" className="form-control" required value={startTime} onChange={(e) => setStartTime(e.target.value)} />
<div className="form-hint text-secondary small mt-1">Clock in {timezone}</div>
</div>
<div className="col-sm-6 col-md-3">
<label className="form-label" htmlFor="rec-length">
For
</label>
<select id="rec-length" className="form-select" value={minutes} onChange={(e) => setMinutes(Number(e.target.value))}>
{LENGTHS.map((l) => (
<option key={l.minutes} value={l.minutes}>
{l.label}
</option>
))}
</select>
</div>
<div className="col-12">
<div className="form-label">What it covers</div>
<div className="border rounded p-2" style={{ maxHeight: 220, overflowY: "auto" }}>
{groups.map(
(g) =>
g.items.length > 0 && (
<fieldset key={g.type} className="mb-2">
<legend className="form-label small text-secondary mb-1">
{g.label}{" "}
<button type="button" className="btn btn-link btn-sm p-0 ms-1" onClick={() => toggleGroup(g.type, g.items)}>
{g.items.every((i) => picked.has(key(g.type, i.id))) ? "none" : "all"}
</button>
</legend>
<div className="d-flex flex-wrap gap-3">
{g.items.map((i) => (
<label key={i.id} className="form-check mb-0">
<input
type="checkbox"
className="form-check-input"
checked={picked.has(key(g.type, i.id))}
onChange={(e) =>
setPicked((prev) => {
const next = new Set(prev);
if (e.target.checked) next.add(key(g.type, i.id));
else next.delete(key(g.type, i.id));
return next;
})
}
/>
<span className="form-check-label">{i.name}</span>
</label>
))}
</div>
</fieldset>
),
)}
</div>
</div>
<div className="col-12">
<ErrorNotice error={error} />
</div>
</div>
<div className="card-footer d-flex gap-2">
<button type="submit" className="btn btn-primary" disabled={busy || picked.size === 0 || days.length === 0 || name.trim() === ""}>
{busy ? "Saving…" : "Save"}
</button>
<button type="button" className="btn" onClick={() => setAdding(false)}>
Cancel
</button>
</div>
</form>
)}
{!adding && <ErrorNotice error={error} />}
<div className="table-responsive">
<table className="table table-vcenter card-table">
<thead>
<tr>
<th>Name</th>
<th>Covers</th>
<th>When</th>
<th>Next</th>
{canEdit && <th className="w-1">Actions</th>}
</tr>
</thead>
<tbody>
{schedules?.map((s) => (
<tr key={s.id} className={s.enabled ? undefined : "text-secondary"}>
<td>
{s.name}
{!s.enabled && <span className="badge bg-secondary-lt text-secondary ms-2">off</span>}
{s.reason && <div className="text-secondary small">{s.reason}</div>}
</td>
<td>{namesWords(s.targets.map((t) => t.name))}</td>
<td>
{daysWords(s.days)} {timeWords(s.startTime, s.durationMinutes)}
{s.timezone !== timezone && <span className="text-secondary small"> ({s.timezone})</span>}
</td>
<td>{s.nextStart ? <TimeAgo value={s.nextStart} /> : "—"}</td>
{canEdit && (
<td>
<div className="btn-list flex-nowrap">
<button type="button" className="btn btn-sm btn-outline-secondary" onClick={() => void toggle(s)}>
{s.enabled ? "Turn off" : "Turn on"}
</button>
<button type="button" className="btn btn-sm btn-outline-danger" onClick={() => void remove(s)}>
Remove
</button>
</div>
</td>
)}
</tr>
))}
{schedules?.length === 0 && (
<tr>
<td colSpan={canEdit ? 5 : 4} className="text-secondary text-center">
Nothing repeats yet. Add one for the night you patch, and its windows appear a week ahead.
</td>
</tr>
)}
</tbody>
</table>
</div>
</div>
);
}
+192
View File
@@ -0,0 +1,192 @@
import { useEffect, useRef, useState } from "react";
import { Link } from "react-router-dom";
import { api, type IntegrationSummary, type SemaphoreTask, type SemaphoreTemplate } from "../api/client";
import { confirmDialog } from "../utils/dialogs";
import { readableError } from "../utils/errors";
import { toast } from "../utils/toast";
import TimeAgo from "./TimeAgo";
import { BlockSkeleton } from "./Skeleton";
interface Loaded {
integration: IntegrationSummary;
templates: SemaphoreTemplate[];
failed: boolean;
}
const STATUS_WORDS: Record<SemaphoreTask["status"], { label: string; badge: string }> = {
waiting: { label: "Waiting to start", badge: "bg-secondary-lt text-secondary" },
starting: { label: "Starting", badge: "bg-blue-lt text-blue" },
running: { label: "Running", badge: "bg-blue-lt text-blue" },
waiting_confirmation: { label: "Waiting for approval in Semaphore", badge: "bg-yellow-lt text-yellow" },
confirmed: { label: "Approved", badge: "bg-blue-lt text-blue" },
rejected: { label: "Rejected in Semaphore", badge: "bg-red-lt text-red" },
stopping: { label: "Stopping", badge: "bg-yellow-lt text-yellow" },
stopped: { label: "Stopped", badge: "bg-secondary-lt text-secondary" },
success: { label: "Finished — success", badge: "bg-green-lt text-green" },
error: { label: "Finished — failed", badge: "bg-red-lt text-red" },
};
const FINISHED = new Set<SemaphoreTask["status"]>(["success", "error", "stopped", "rejected"]);
/**
* Starts a Semaphore template — optionally on this server only — and shows how the run gets on. It is the same "run" the Semaphore page
* has, with the one addition that matters from a server or an alert: ansible's limit, so "apply updates" touches the one machine and
* not the whole inventory. Asks first, and says plainly when there is no limit.
*/
export default function RunPlaybook({ serverName, defaultLimit }: { serverName: string; defaultLimit: string }) {
const [loaded, setLoaded] = useState<Loaded[] | null>(null);
const [failed, setFailed] = useState(false);
const [choice, setChoice] = useState("");
const [limitOn, setLimitOn] = useState(true);
const [limit, setLimit] = useState(defaultLimit);
const [busy, setBusy] = useState(false);
const [run, setRun] = useState<{ integrationId: number; projectId: number; name: string; limit: string; task: SemaphoreTask } | null>(null);
const alive = useRef(true);
useEffect(() => {
alive.current = true;
api.integrations
.list()
.then(async (res) => {
const servers = res.integrations.filter((i) => i.type === "semaphore" && i.enabled);
const all = await Promise.all(
servers.map((integration) =>
api.integrations.semaphore
.templates(integration.id)
.then((r): Loaded => ({ integration, templates: r.templates, failed: false }))
.catch((): Loaded => ({ integration, templates: [], failed: true })),
),
);
if (alive.current) setLoaded(all);
})
.catch(() => alive.current && setFailed(true));
return () => {
alive.current = false;
};
}, []);
// Watch a run until it is over.
useEffect(() => {
if (!run || FINISHED.has(run.task.status)) return;
const timer = setInterval(() => {
api.integrations.semaphore
.task(run.integrationId, run.projectId, run.task.id)
.then((res) => alive.current && setRun((r) => (r && r.task.id === res.task.id ? { ...r, task: res.task } : r)))
.catch(() => {
// Keep asking: one missed answer isn't the end of the run.
});
}, 3000);
return () => clearInterval(timer);
}, [run?.task.id, run?.task.status, run?.integrationId, run?.projectId]);
const options = (loaded ?? []).flatMap((l) =>
l.templates.map((t) => ({
key: `${l.integration.id}:${t.projectId}:${t.id}`,
integration: l.integration,
template: t,
label: `${t.name} — ${t.projectName}${(loaded ?? []).length > 1 ? ` (${l.integration.name})` : ""}`,
})),
);
const picked = options.find((o) => o.key === choice);
const limitText = limitOn ? limit.trim() : "";
async function start() {
if (!picked) return;
const where = limitText ? `on ${limitText} only` : "on everything the playbook covers — it isn't limited to one server";
if (
!(await confirmDialog({
title: `Run ${picked.template.name}?`,
message: `This starts "${picked.template.name}" in ${picked.integration.name} ${where}.`,
confirmLabel: "Run it",
danger: !limitText,
}))
)
return;
setBusy(true);
try {
const res = await api.integrations.semaphore.run(picked.integration.id, picked.template.projectId, picked.template.id, limitText ? { limit: limitText } : undefined);
setRun({ integrationId: picked.integration.id, projectId: picked.template.projectId, name: picked.template.name, limit: limitText, task: res.task });
toast.success(`Started ${picked.template.name}${limitText ? ` on ${limitText}` : ""}.`);
} catch (err) {
toast.error(readableError(err));
} finally {
setBusy(false);
}
}
if (failed) return <div className="text-secondary">Couldn't load the integrations.</div>;
if (loaded === null) return <BlockSkeleton lines={3} />;
if (loaded.length === 0) {
return (
<div className="text-secondary">
There is no Semaphore connected. Add one under <Link to="/integrations">Integrations</Link> to run playbooks from here.
</div>
);
}
return (
<div>
{loaded.some((l) => l.failed) && <div className="text-secondary small mb-2">Couldn't read the templates of {loaded.filter((l) => l.failed).map((l) => l.integration.name).join(", ")}.</div>}
{options.length === 0 ? (
<div className="text-secondary">No templates found in Semaphore.</div>
) : (
<>
<div className="mb-3">
<label className="form-label" htmlFor="run-template">
Template
</label>
<select id="run-template" className="form-select" value={choice} onChange={(e) => setChoice(e.target.value)}>
<option value="">Choose a template…</option>
{options.map((o) => (
<option key={o.key} value={o.key}>
{o.label}
</option>
))}
</select>
{picked?.template.lastTask && (
<div className="form-hint text-secondary small mt-1">
Last run here: {picked.template.lastTask.status}
{picked.template.lastTask.end ? (
<>
, <TimeAgo value={picked.template.lastTask.end} />
</>
) : null}
</div>
)}
</div>
<div className="mb-3">
<label className="form-check">
<input type="checkbox" className="form-check-input" checked={limitOn} onChange={(e) => setLimitOn(e.target.checked)} />
<span className="form-check-label">Run on {serverName} only</span>
</label>
{limitOn ? (
<div className="mt-2">
<label className="form-label small" htmlFor="run-limit">
Limit — the host name or pattern as it is in the Semaphore inventory
</label>
<input id="run-limit" className="form-control" value={limit} maxLength={200} onChange={(e) => setLimit(e.target.value)} placeholder="e.g. web01" />
</div>
) : (
<div className="text-danger small mt-1">Not limited: it runs on every host the playbook is set up for.</div>
)}
</div>
<button type="button" className="btn btn-primary" disabled={!picked || busy || (limitOn && limit.trim() === "")} onClick={() => void start()}>
{busy ? "Starting…" : "Run"}
</button>
</>
)}
{run && (
<div className="mt-3 border-top pt-3" role="status" aria-live="polite">
<div className="d-flex flex-wrap align-items-center gap-2">
<strong>{run.name}</strong>
{run.limit && <span className="text-secondary">on {run.limit}</span>}
<span className={`badge ${STATUS_WORDS[run.task.status]?.badge ?? "bg-secondary-lt text-secondary"}`}>{STATUS_WORDS[run.task.status]?.label ?? run.task.status}</span>
</div>
<div className="text-secondary small mt-1">
Run #{run.task.id} in Semaphore{FINISHED.has(run.task.status) ? "" : " — checking every few seconds"}. The output is in Semaphore.
</div>
</div>
)}
</div>
);
}
+20
View File
@@ -0,0 +1,20 @@
import { useState } from "react";
import DetailDrawer from "./DetailDrawer";
import RunPlaybook from "./RunPlaybook";
/** "Run…" on an alert about a server: opens a side panel to start a playbook on that server, without leaving the alerts. */
export default function RunPlaybookButton({ serverName }: { serverName: string }) {
const [open, setOpen] = useState(false);
return (
<>
<button type="button" className="btn btn-sm btn-outline-secondary" title={`Run a Semaphore playbook on ${serverName}`} onClick={() => setOpen(true)}>
Run…
</button>
{open && (
<DetailDrawer title="Run a playbook" subtitle={`On ${serverName}`} onClose={() => setOpen(false)}>
<RunPlaybook serverName={serverName} defaultLimit={serverName} />
</DetailDrawer>
)}
</>
);
}
+55
View File
@@ -0,0 +1,55 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type ServerRecord, type ServerRelation } from "../api/client";
import { readableError } from "../utils/errors";
import { toast } from "../utils/toast";
/**
* Which server this one runs on (a virtual machine on its host). When the host stops reporting, this server's own "offline" alert is
* folded under the host's instead of being a second one. Left alone it is worked out from the Proxmox node the server is linked to;
* an admin can say it by hand. Everyone sees it; only admins change it.
*/
export default function RunsOn({ serverId, parent, canEdit, onChanged }: { serverId: number; parent: ServerRelation | null; canEdit: boolean; onChanged: () => void }) {
const [servers, setServers] = useState<ServerRecord[] | null>(null);
const [busy, setBusy] = useState(false);
useEffect(() => {
if (!canEdit) return;
api.servers
.list()
.then((res) => setServers(res.servers))
.catch(() => setServers([]));
}, [canEdit]);
async function change(value: string) {
setBusy(true);
try {
await api.servers.update(serverId, { parentServerId: value === "" ? null : Number(value) });
toast.success(value === "" ? "Back to working it out from the Proxmox node." : "Saved.");
onChanged();
} catch (err) {
toast.error(readableError(err));
} finally {
setBusy(false);
}
}
if (!canEdit) {
return parent ? <Link to={`/servers/${parent.id}`}>{parent.name}</Link> : <span className="text-secondary">—</span>;
}
return (
<div>
<select className="form-select form-select-sm" aria-label="The server this one runs on" value={parent && !parent.auto ? String(parent.id) : ""} disabled={busy || servers === null} onChange={(e) => void change(e.target.value)}>
<option value="">{parent?.auto ? `Automatic: ${parent.name}` : "None (automatic)"}</option>
{(servers ?? [])
.filter((s) => s.id !== serverId)
.map((s) => (
<option key={s.id} value={s.id}>
{s.name}
</option>
))}
</select>
<div className="form-hint text-secondary small mt-1">Automatic means worked out from its Proxmox node. If it stops reporting, this server's offline alert is folded under its host's.</div>
</div>
);
}
+3
View File
@@ -27,10 +27,12 @@ export default function StatusStrip() {
.filter((a) => !a.silenced && a.severity !== "info")
.sort((a, b) => SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
const silenced = report.counts.silenced;
const grouped = report.counts.grouped ?? 0;
const blind = report.couldntCheck.length;
const notes: string[] = [];
if (silenced > 0) notes.push(`${silenced} silenced by maintenance`);
if (grouped > 0) notes.push(`${grouped} more explained by a server that is down`);
if (blind > 0 && badge) notes.push(`${blind} check${blind === 1 ? "" : "s"} couldn't run`);
if (!badge) {
@@ -50,6 +52,7 @@ export default function StatusStrip() {
<span className="text-secondary">No active alerts.</span>
)}
{silenced > 0 && <span className="text-secondary"> {silenced} silenced by maintenance.</span>}
{grouped > 0 && <span className="text-secondary"> {grouped} more explained by a server that is down.</span>}
</div>
</div>
</div>
+25
View File
@@ -0,0 +1,25 @@
import type { PendingUpdates } from "../api/client";
/**
* What a server has waiting: "Up to date", "5 waiting" or "5 waiting, 2 security" (red when any are security fixes), and a
* "restart needed" beside it. Says "not reported" when the agent hasn't said — never "up to date" for a server that simply didn't tell us.
*/
export default function UpdatesBadge({ updates }: { updates: PendingUpdates | null }) {
if (!updates) {
return (
<span className="badge bg-secondary-lt text-secondary" title="This server's agent hasn't reported updates — it may be an older agent, or its package manager isn't one it can ask.">
not reported
</span>
);
}
const security = updates.security ?? 0;
const level = updates.count === 0 ? "bg-green-lt text-green" : security > 0 ? "bg-red-lt text-red" : "bg-yellow-lt text-yellow";
return (
<span className="d-inline-flex flex-wrap align-items-center gap-1">
<span className={`badge ${level}`}>
{updates.count === 0 ? "Up to date" : `${updates.count} waiting${security > 0 ? `, ${security} security` : ""}`}
</span>
{updates.rebootRequired && <span className="badge bg-orange-lt text-orange">restart needed</span>}
</span>
);
}
+2
View File
@@ -22,6 +22,7 @@ import {
IconWorldWww,
IconListCheck,
IconRefresh,
IconDownload,
IconActivityHeartbeat,
IconShieldCheck,
IconTicket,
@@ -101,6 +102,7 @@ export const NAV: NavEntry[] = [
items: [
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
{ to: "/updates", label: "Updates", icon: <IconDownload size={20} /> },
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
{ to: "/osticket", label: "osTicket", icon: <IconTicket size={20} /> },
{ to: "/admin-links", label: "Admin Links", icon: <IconExternalLink size={20} /> },
+1
View File
@@ -13,6 +13,7 @@ export const GO_SHORTCUTS: GoShortcut[] = [
{ key: "a", to: "/alerts", label: "Alerts" },
{ key: "s", to: "/servers", label: "Servers" },
{ key: "m", to: "/maintenance", label: "Maintenance" },
{ key: "u", to: "/updates", label: "Updates" },
{ key: "i", to: "/integrations", label: "Integrations" },
{ key: "k", to: "/secrets", label: "Secrets" },
{ key: "p", to: "/map", label: "Map" },
+19 -3
View File
@@ -12,6 +12,7 @@ import { readableError } from "../utils/errors";
import { ErrorNotice } from "../components/Notice";
import TableToolbar from "../components/TableToolbar";
import SilenceButton from "../components/SilenceButton";
import RunPlaybookButton from "../components/RunPlaybookButton";
const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: number }> = {
critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 },
@@ -89,6 +90,7 @@ export default function Alerts({ user }: { user: CurrentUser }) {
{ label: "Warnings", value: counts?.warning, color: "text-yellow" },
{ label: "Info", value: counts?.info, color: "text-blue" },
{ label: "Silenced (maintenance)", value: counts?.silenced, color: "text-secondary" },
...((counts?.grouped ?? 0) > 0 ? [{ label: "Grouped under another", value: counts?.grouped, color: "text-secondary" }] : []),
];
return (
@@ -196,10 +198,10 @@ export default function Alerts({ user }: { user: CurrentUser }) {
</option>
))}
</select>
{(counts?.silenced ?? 0) > 0 && (
{(counts?.silenced ?? 0) + (counts?.grouped ?? 0) > 0 && (
<label className="form-check mb-0">
<input type="checkbox" className="form-check-input" checked={showSilenced} onChange={(e) => setShowSilenced(e.target.checked)} />
<span className="form-check-label">Show silenced</span>
<span className="form-check-label">{(counts?.grouped ?? 0) > 0 ? "Show silenced and grouped" : "Show silenced"}</span>
</label>
)}
</>
@@ -230,10 +232,21 @@ export default function Alerts({ user }: { user: CurrentUser }) {
<td>{a.link ? <Link to={a.link}>{a.source}</Link> : a.source}</td>
<td>
{a.message}
{a.silenced && (
{a.causedBy ? (
<span className="badge bg-secondary-lt text-secondary ms-2" title={`${a.causedBy.name} is down too, which explains this — notifications for this are held back. If ${a.causedBy.name} comes back and this stays down, it alerts then.`}>
caused by {a.causedBy.name}
</span>
) : (
a.silenced && (
<span className="badge bg-secondary-lt text-secondary ms-2" title="Under a maintenance window — notifications for this are held back">
silenced
</span>
)
)}
{a.affects && a.affects.length > 0 && (
<div className="text-secondary small">
Also takes down {a.affects.length} {a.affects.length === 1 ? "server" : "servers"}: {a.affects.join(", ")}
</div>
)}
{a.ticket &&
(() => {
@@ -252,7 +265,10 @@ export default function Alerts({ user }: { user: CurrentUser }) {
</td>
{canSilence && (
<td className="text-nowrap">
<div className="btn-list flex-nowrap">
<SilenceButton alert={a} onDone={() => void load(true)} />
{a.target?.type === "server" && !a.silenced && <RunPlaybookButton serverName={a.target.name} />}
</div>
</td>
)}
</tr>
+10
View File
@@ -18,6 +18,7 @@ import { ErrorNotice } from "../components/Notice";
import { toast } from "../utils/toast";
import { confirmDialog } from "../utils/dialogs";
import MaintenanceTimeline from "../components/MaintenanceTimeline";
import RecurringMaintenance from "../components/RecurringMaintenance";
const DURATIONS: { minutes: number; label: string }[] = [
{ minutes: 30, label: "30 minutes" },
@@ -326,6 +327,15 @@ export default function Maintenance({ user }: { user: CurrentUser }) {
{((windows?.length ?? 0) > 0 || upcoming.length > 0) && <MaintenanceTimeline windows={[...(windows ?? []), ...upcoming]} now={now} />}
<RecurringMaintenance
canEdit={canEdit}
targets={targets}
onChanged={() => {
load();
announceChange();
}}
/>
{canEdit && kumaIntegrations && kumaIntegrations.length > 0 && (
<div className="card mb-3">
<div className="card-header">
+43
View File
@@ -17,6 +17,9 @@ import ServerTrends from "../components/ServerTrends";
import ServerActivity from "../components/ServerActivity";
import TimeAgo from "../components/TimeAgo";
import UptimeRibbon from "../components/UptimeRibbon";
import RunsOn from "../components/RunsOn";
import RunPlaybook from "../components/RunPlaybook";
import UpdatesBadge from "../components/UpdatesBadge";
import ServerTags from "../components/ServerTags";
import ServerTaskTable, { SCHEDULE_TYPE_LABELS } from "../components/ServerTaskTable";
import { formatDateTime } from "../utils/date";
@@ -755,6 +758,36 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
<dd className="col-7">
<UptimeRibbon kind="servers" id={server.id} name={server.name} />
</dd>
<dt className="col-5">Runs on</dt>
<dd className="col-7">
<RunsOn serverId={server.id} parent={detail.relations.parent} canEdit={isAdmin} onChanged={() => void loadDetail()} />
</dd>
{detail.relations.children.length > 0 && (
<>
<dt className="col-5">Runs here</dt>
<dd className="col-7">
{detail.relations.children.map((c, i) => (
<span key={c.id}>
{i > 0 && ", "}
<Link to={`/servers/${c.id}`}>{c.name}</Link>
</span>
))}
</dd>
</>
)}
{server.lastSeenAt && (
<>
<dt className="col-5">Updates</dt>
<dd className="col-7">
<UpdatesBadge updates={server.pendingUpdates} />
{server.updatesPendingSince && server.pendingUpdates && server.pendingUpdates.count > 0 && (
<div className="text-secondary small">
waiting since <TimeAgo value={server.updatesPendingSince} />
</div>
)}
</dd>
</>
)}
</dl>
</div>
</div>
@@ -763,6 +796,16 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
{hardware.source === "agent" && <ServerTrends serverId={serverId} />}
{(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />}
{canEditTasks && (
<div className="card mb-3">
<div className="card-header">
<h3 className="card-title">Run a playbook</h3>
</div>
<div className="card-body">
<RunPlaybook serverName={server.name} defaultLimit={server.name} />
</div>
</div>
)}
<ServerPorts
serverId={serverId}
+244
View File
@@ -0,0 +1,244 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api, type DockhandContainer, type UpdatesServer } from "../api/client";
import SortableTh from "../components/SortableTh";
import TableToolbar from "../components/TableToolbar";
import TimeAgo from "../components/TimeAgo";
import UpdatesBadge from "../components/UpdatesBadge";
import { BlockSkeleton } from "../components/Skeleton";
import { ErrorNotice } from "../components/Notice";
import { useSortable } from "../hooks/useSortable";
import { useTableSearch } from "../hooks/useTableSearch";
import { downloadCsv } from "../utils/csv";
import { readableError } from "../utils/errors";
interface Row {
id: number;
name: string;
os: string;
manager: string;
/** Null: the agent hasn't said. */
count: number | null;
security: number | null;
restart: boolean;
/** ISO — when updates were first seen waiting. */
since: string | null;
reportedAt: string | null;
server: UpdatesServer;
}
function toRow(s: UpdatesServer): Row {
return {
id: s.id,
name: s.name,
os: s.osType,
manager: s.updates?.manager ?? "",
count: s.updates ? s.updates.count : null,
security: s.updates ? s.updates.security : null,
restart: s.updates?.rebootRequired ?? false,
since: s.pendingSince,
reportedAt: s.updates?.reportedAt || null,
server: s,
};
}
const DAY_MS = 86_400_000;
/**
* What is waiting to be installed, in one place: the updates each server's agent says are pending (how many, how many are security fixes,
* whether a restart is due, and how long they have been waiting) and the Docker containers that have a newer image. It only looks —
* nothing here installs anything.
*/
export default function Updates() {
const [servers, setServers] = useState<UpdatesServer[] | null>(null);
const [containers, setContainers] = useState<(DockhandContainer & { integrationName: string })[] | null>(null);
const [error, setError] = useState<string | null>(null);
const [onlyWaiting, setOnlyWaiting] = useState(false);
useEffect(() => {
let alive = true;
api.servers
.updates()
.then((res) => alive && setServers(res.servers))
.catch((err) => alive && setError(readableError(err)));
api.integrations
.list()
.then(async (res) => {
const docker = res.integrations.filter((i) => i.type === "dockhand" && i.enabled);
const all = await Promise.all(
docker.map((i) =>
api.integrations.dockhand
.containers(i.id)
.then((r) => r.containers.map((c) => ({ ...c, integrationName: i.name })))
.catch(() => []),
),
);
if (alive) setContainers(docker.length === 0 ? null : all.flat().filter((c) => c.updateAvailable === true));
})
.catch(() => {});
return () => {
alive = false;
};
}, []);
const rows = (servers ?? []).map(toRow);
const waiting = rows.filter((r) => (r.count ?? 0) > 0);
const tableSearch = useTableSearch(onlyWaiting ? waiting : rows);
const { sorted, sortKey, sortDir, requestSort } = useSortable(tableSearch.filtered, "security", "desc", { url: true });
const reporting = rows.filter((r) => r.count !== null).length;
const securityTotal = rows.reduce((n, r) => n + (r.security ?? 0), 0);
const restarts = rows.filter((r) => r.restart).length;
const oldest = waiting.map((r) => r.since).filter((s): s is string => !!s).sort()[0];
const notReporting = rows.length - reporting;
function exportCsv() {
if (!sorted) return;
downloadCsv(
"updates.csv",
["Server", "Package manager", "Waiting", "Security", "Restart needed", "Waiting since", "Reported"],
sorted.map((r) => [r.name, r.manager, r.count ?? "", r.security ?? "", r.restart ? "yes" : "no", r.since ?? "", r.reportedAt ?? ""]),
);
}
const cards: { label: string; value: string | number; note?: string; color: string }[] = [
{ label: "Servers with updates waiting", value: servers ? `${waiting.length} of ${reporting}` : "…", note: notReporting > 0 ? `${notReporting} haven't reported updates` : undefined, color: waiting.length > 0 ? "text-yellow" : "text-green" },
{ label: "Security updates waiting", value: servers ? securityTotal : "…", color: securityTotal > 0 ? "text-red" : "text-green" },
{ label: "Need a restart", value: servers ? restarts : "…", color: restarts > 0 ? "text-orange" : "text-green" },
{ label: "Longest wait", value: oldest ? `${Math.max(1, Math.floor((Date.now() - Date.parse(oldest)) / DAY_MS))} d` : servers ? "—" : "…", color: "text-secondary" },
];
return (
<>
<div className="d-flex align-items-center mb-3">
<h2 className="page-title mb-0">Updates</h2>
</div>
<p className="text-secondary">
What the agents say is waiting to be installed on each server, and the Docker containers that have a newer image. This page only looks — it installs nothing. To apply updates, run a playbook from the server's page.
</p>
<ErrorNotice error={error} />
<div className="row row-cards mb-3">
{cards.map((c) => (
<div className="col-6 col-lg-3" key={c.label}>
<div className="card card-sm">
<div className="card-body">
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
<div className="text-secondary">{c.label}</div>
{c.note && <div className="text-secondary small">{c.note}</div>}
</div>
</div>
</div>
))}
</div>
<div className="card mb-3">
<TableToolbar
search={tableSearch.query}
onSearch={tableSearch.setQuery}
placeholder="Search servers…"
filters={
<label className="form-check mb-0">
<input type="checkbox" className="form-check-input" checked={onlyWaiting} onChange={(e) => setOnlyWaiting(e.target.checked)} />
<span className="form-check-label">Only those with updates</span>
</label>
}
shown={sorted?.length}
total={onlyWaiting ? waiting.length : rows.length}
onExport={exportCsv}
exportDisabled={!sorted || sorted.length === 0}
/>
<div className="table-responsive">
<table className="table table-vcenter card-table">
<thead>
<tr>
<SortableTh<Row> label="Server" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Updates" sortKeyName="count" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Security" sortKeyName="security" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Waiting since" sortKeyName="since" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Reported" sortKeyName="reportedAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
</tr>
</thead>
<tbody>
{servers === null && !error && (
<tr>
<td colSpan={5}>
<BlockSkeleton lines={3} />
</td>
</tr>
)}
{sorted?.map((r) => (
<tr key={r.id}>
<td>
<Link to={`/servers/${r.id}`}>{r.name}</Link> <span className="text-secondary small">{r.manager}</span>
</td>
<td>
<UpdatesBadge updates={r.server.updates} />
</td>
<td>{r.security === null ? <span className="text-secondary">—</span> : r.security}</td>
<td className="text-secondary">{r.since ? <TimeAgo value={r.since} /> : "—"}</td>
<td className="text-secondary">{r.reportedAt ? <TimeAgo value={r.reportedAt} /> : "—"}</td>
</tr>
))}
{servers !== null && sorted?.length === 0 && (
<tr>
<td colSpan={5} className="text-secondary text-center">
{tableSearch.active || onlyWaiting ? "Nothing matches that." : "No server has reported yet."}
</td>
</tr>
)}
</tbody>
</table>
</div>
{notReporting > 0 && (
<div className="card-footer text-secondary small">
“Not reported” means that server's agent hasn't said — it may be an older agent (run its install command again, from Servers → Manage servers, to update it), or its package manager isn't apt, dnf, zypper or Windows Update.
</div>
)}
</div>
{containers !== null && (
<div className="card">
<div className="card-header">
<h3 className="card-title">Docker containers with a newer image</h3>
</div>
<div className="table-responsive">
<table className="table table-vcenter card-table">
<thead>
<tr>
<th>Container</th>
<th>Image</th>
<th>Newer version</th>
<th>Where</th>
<th>Checked</th>
</tr>
</thead>
<tbody>
{containers.map((c) => (
<tr key={`${c.integrationName}:${c.environmentId}:${c.id}`}>
<td>
<Link to="/docker">{c.name}</Link>
</td>
<td className="text-secondary">{c.image}</td>
<td>{c.newerVersion ?? "—"}</td>
<td className="text-secondary">{c.environmentName}</td>
<td className="text-secondary">
<TimeAgo value={c.checkedAt} />
</td>
</tr>
))}
{containers.length === 0 && (
<tr>
<td colSpan={5} className="text-secondary text-center">
Every container is on its latest image, as far as the last check could tell.
</td>
</tr>
)}
</tbody>
</table>
</div>
</div>
)}
</>
);
}
+5
View File
@@ -671,6 +671,11 @@ main:focus {
background-image: repeating-linear-gradient(135deg, rgba(255, 255, 255, 0.45) 0 3px, transparent 3px 6px);
}
/* Windows made from a recurring schedule get vertical stripes. */
.timeline-bar-recurring {
background-image: repeating-linear-gradient(90deg, rgba(255, 255, 255, 0.5) 0 2px, transparent 2px 5px);
}
/* ── The wall display: a full-screen status board, big enough to read across a room. ── */
.wall {
display: flex;