Fold alerts under their host, recurring maintenance, run playbooks from alerts, and an Updates page
- Servers can say what they run on (or it is worked out from the Proxmox node); when the host is down the guests' offline alerts fold under it: held back, not counted, not notified, still recorded. - Recurring maintenance schedules made into ordinary windows a week ahead (maintenance_schedules). - Run a Semaphore template limited to a server from its page or an alert, and watch the run. - Agents report pending updates (apt, dnf, zypper, Windows Update); an Updates page lists them. - Migrations 0024 and 0025. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
44e393f172
commit
171511c6db
45 files changed
+6769
-41
No files matched your search
+28
-5
@@ -23,8 +23,8 @@ what the app actually creates.
|
|||||||
|---|---|
|
|---|---|
|
||||||
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
|
||||||
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
|
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
|
||||||
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0023`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0025`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
||||||
| **Tables** | 25 application tables, plus drizzle's own `__drizzle_migrations`. |
|
| **Tables** | 26 application tables, plus drizzle's own `__drizzle_migrations`. |
|
||||||
|
|
||||||
**Not in the database:**
|
**Not in the database:**
|
||||||
|
|
||||||
@@ -62,9 +62,9 @@ erDiagram
|
|||||||
integrations ||--o{ alert_tickets : "osTicket used (set null)"
|
integrations ||--o{ alert_tickets : "osTicket used (set null)"
|
||||||
```
|
```
|
||||||
|
|
||||||
Nine tables stand alone, with no foreign keys: `settings`, `secrets`,
|
Ten tables stand alone, with no foreign keys: `settings`, `secrets`,
|
||||||
`ipam_entries`, `domains`, `diag_log`, `notification_queue`,
|
`ipam_entries`, `domains`, `diag_log`, `notification_queue`,
|
||||||
`consistency_ignores`, `tag_definitions`, `share_links`. `maintenance_windows` and `maintenance_aliases` also have no
|
`consistency_ignores`, `tag_definitions`, `share_links`, `maintenance_schedules`. `maintenance_windows` and `maintenance_aliases` also have no
|
||||||
foreign key — see [soft references](#soft-references-not-foreign-keys).
|
foreign key — see [soft references](#soft-references-not-foreign-keys).
|
||||||
|
|
||||||
## Tables
|
## Tables
|
||||||
@@ -145,7 +145,27 @@ required, so a forgotten window can't silence real problems forever.
|
|||||||
| `reason` | text | |
|
| `reason` | text | |
|
||||||
| `started_at`, `ends_at` | text NOT NULL | ISO. |
|
| `started_at`, `ends_at` | text NOT NULL | ISO. |
|
||||||
| `created_by` | text | Name of the person who opened it. `Uptime Kuma sync` for mirrored ones. |
|
| `created_by` | text | Name of the person who opened it. `Uptime Kuma sync` for mirrored ones. |
|
||||||
| `source` | text | `kuma:<integration id>:<maintenance id>` on windows mirrored from Uptime Kuma's schedule, so a later sync can find them again; NULL for ones started by hand. A window whose `started_at` is still in the future is *scheduled*: it silences nothing until it begins. A scheduled one that is cancelled has `ends_at` set equal to `started_at`. |
|
| `source` | text | `kuma:<integration id>:<maintenance id>` on windows mirrored from Uptime Kuma's schedule, so a later sync can find them again; `schedule:<schedule id>:<date>` on windows made from a recurring schedule (so one is made once, even if it was cancelled); NULL for ones started by hand. A window whose `started_at` is still in the future is *scheduled*: it silences nothing until it begins. A scheduled one that is cancelled has `ends_at` set equal to `started_at`. |
|
||||||
|
|
||||||
|
#### `maintenance_schedules`
|
||||||
|
|
||||||
|
A maintenance that repeats ("Sundays 02:00–04:00 on these servers"). It only says when:
|
||||||
|
the windows that silence alerts are ordinary `maintenance_windows` rows made a week
|
||||||
|
ahead and marked `schedule:<id>:<date>`.
|
||||||
|
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | integer PK | |
|
||||||
|
| `name` | text NOT NULL | e.g. "Patch night". |
|
||||||
|
| `reason` | text | |
|
||||||
|
| `targets` | text NOT NULL | JSON `[{"type":"server","id":3}, …]`. Polymorphic like `maintenance_windows`, so no foreign key; a deleted target is skipped. |
|
||||||
|
| `days` | text NOT NULL | JSON array of weekdays, 0 = Sunday … 6 = Saturday. |
|
||||||
|
| `start_time` | text NOT NULL | `HH:MM` on the clock of `timezone`. |
|
||||||
|
| `duration_minutes` | integer NOT NULL | |
|
||||||
|
| `timezone` | text NOT NULL | IANA name, so 02:00 stays 02:00 across summer and winter time. |
|
||||||
|
| `enabled` | bool NOT NULL, default 1 | Off makes no windows and takes back those that haven't begun. |
|
||||||
|
| `created_by` | text | |
|
||||||
|
| `created_at` | text NOT NULL | |
|
||||||
|
|
||||||
#### `maintenance_aliases`
|
#### `maintenance_aliases`
|
||||||
|
|
||||||
@@ -194,6 +214,9 @@ what it last reported.
|
|||||||
| `proxmox_guest_type` | text | `qemu` \| `lxc`. |
|
| `proxmox_guest_type` | text | `qemu` \| `lxc`. |
|
||||||
| `proxmox_vmid` | integer | The four `proxmox_*` columns are set together or cleared together (enforced by the API), by an admin — never by the agent. |
|
| `proxmox_vmid` | integer | The four `proxmox_*` columns are set together or cleared together (enforced by the API), by an admin — never by the agent. |
|
||||||
| `hide_proxmox_link` | bool NOT NULL, default 0 | Hides the "Proxmox link" card for servers that aren't Proxmox guests. Ignored while the server is actually linked. |
|
| `hide_proxmox_link` | bool NOT NULL, default 0 | Hides the "Proxmox link" card for servers that aren't Proxmox guests. Ignored while the server is actually linked. |
|
||||||
|
| `parent_server_id` | integer | The server this one runs on, set by an admin. **Not a foreign key** (a removed host's servers are cleared by the app). Null = work it out: a server linked to a Proxmox guest runs on the one server named after that guest's node. When the parent is down, this server's offline alert is folded under it. |
|
||||||
|
| `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. |
|
||||||
|
| `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. |
|
||||||
|
|
||||||
#### `scheduled_tasks`
|
#### `scheduled_tasks`
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -40,7 +40,7 @@ the dashboard views themselves load fine.
|
|||||||
|
|
||||||
- **Config fields:** Semaphore URL, API token
|
- **Config fields:** Semaphore URL, API token
|
||||||
- **Auth:** Bearer token
|
- **Auth:** Bearer token
|
||||||
- **Actions used:** list projects, list templates, run a template (creates a task)
|
- **Actions used:** list projects, list templates, run a template (creates a task — optionally with a `limit` of hosts and a note saying who started it from Homelab Manager), read a task's status
|
||||||
- **Required access:** A user API token with access to every project you want visible, and **permission to run templates** in those projects — a viewer/guest-level project role can list templates but will fail on "Run", so the account needs at least the operator-equivalent role Semaphore's own project permissions define.
|
- **Required access:** A user API token with access to every project you want visible, and **permission to run templates** in those projects — a viewer/guest-level project role can list templates but will fail on "Run", so the account needs at least the operator-equivalent role Semaphore's own project permissions define.
|
||||||
|
|
||||||
### Gitea
|
### Gitea
|
||||||
|
|||||||
@@ -400,6 +400,45 @@ the problems themselves, and each integration's status, refreshing every 30
|
|||||||
seconds with the screen kept awake where the browser allows. Esc returns to the
|
seconds with the screen kept awake where the browser allows. Esc returns to the
|
||||||
app.
|
app.
|
||||||
|
|
||||||
|
**Alerts folded under their host.** A server can say which server it runs on (a
|
||||||
|
virtual machine on its Proxmox host): on its page, "Runs on" — and left alone it
|
||||||
|
is worked out when the server is linked to a Proxmox guest and exactly one
|
||||||
|
server is named after that guest's node. When the host stops reporting, the
|
||||||
|
guests' own "server offline" alerts are folded under the host's ("also takes down
|
||||||
|
anna, elsa") instead of being one alert each: they are held back like a silenced
|
||||||
|
alert, not counted, left out of notifications and osTicket tickets, and labelled
|
||||||
|
"caused by pve1" on the Alerts page. They are still recorded as down (for the
|
||||||
|
uptime ribbons and "Since you last looked"), and if the host comes back while a
|
||||||
|
guest is still down, that guest alerts then as new — an outage is never swallowed.
|
||||||
|
Links that loop back on themselves cover nothing, and setting one that would is
|
||||||
|
refused.
|
||||||
|
|
||||||
|
**Recurring maintenance** (Maintenance → Recurring) repeats a window on the days
|
||||||
|
and clock time you choose — "Sundays 02:00–04:00 on these servers" — in the time
|
||||||
|
zone of the browser it was made in, so 02:00 stays 02:00 across summer and winter
|
||||||
|
time. Each time becomes an ordinary maintenance window a week ahead (shown on the
|
||||||
|
timeline with vertical stripes, and in Scheduled), so any single one can be
|
||||||
|
cancelled and stays cancelled. Turning a schedule off or removing it takes back
|
||||||
|
the windows that haven't begun.
|
||||||
|
|
||||||
|
**Running a playbook** from an alert or a server: a server's page has a "Run a
|
||||||
|
playbook" card, and alerts about a server have a "Run…" button, which pick a
|
||||||
|
Semaphore template and run it limited to that server (ansible's `--limit`, which
|
||||||
|
you can edit to match the host name in your inventory — or untick it to run on
|
||||||
|
everything, which asks more sternly). Semaphore is told who started it, the run
|
||||||
|
shows waiting → running → success or failed, and the audit log records what ran
|
||||||
|
and where. Operators and admins only.
|
||||||
|
|
||||||
|
**Updates** (Operations → Updates) shows what each agent says is waiting to be
|
||||||
|
installed — how many, how many are security fixes, whether a restart is due, and
|
||||||
|
how long updates have been waiting — plus the Docker containers with a newer image.
|
||||||
|
The Linux agent asks apt, dnf or zypper (only what they already know; nothing is
|
||||||
|
installed or refreshed) and the Windows agent asks Windows Update (at most every six
|
||||||
|
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
|
||||||
|
said — an older agent, or another package manager — shows "not reported", never "up
|
||||||
|
to date"; run its install command again to update its agent. It only looks: to apply
|
||||||
|
updates, run a playbook.
|
||||||
|
|
||||||
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
|
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
|
||||||
card earlier or later among its neighbours and hide the ones you don't use —
|
card earlier or later among its neighbours and hide the ones you don't use —
|
||||||
hidden cards stay listed, dimmed, while you arrange so they can be brought back,
|
hidden cards stay listed, dimmed, while you arrange so they can be brought back,
|
||||||
|
|||||||
@@ -182,6 +182,46 @@ collect_systemd_timers() {
|
|||||||
done < <(jq -c '.[]' <<<"$timers_json")
|
done < <(jq -c '.[]' <<<"$timers_json")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Updates waiting to be installed, as JSON in UPDATES_JSON ("null" when this host's package manager isn't one we know, or couldn't answer).
|
||||||
|
# Nothing is installed or refreshed: this reads what the package manager already knows about, so it is quick and changes nothing.
|
||||||
|
collect_updates() {
|
||||||
|
UPDATES_JSON="null"
|
||||||
|
local manager="" count="" security="null" reboot="false" out="" status=0
|
||||||
|
|
||||||
|
if command -v apt-get >/dev/null 2>&1; then
|
||||||
|
manager="apt"
|
||||||
|
# -s simulates: "Inst" lines are what an upgrade would install. Security ones come from a "-security" or "Security" archive.
|
||||||
|
if out=$(LC_ALL=C timeout 60 apt-get -s -o Debug::NoLocking=true upgrade 2>/dev/null); then
|
||||||
|
count=$(grep -c '^Inst ' <<<"$out" || true)
|
||||||
|
security=$(grep '^Inst ' <<<"$out" | grep -ci 'security' || true)
|
||||||
|
fi
|
||||||
|
[[ -f /var/run/reboot-required ]] && reboot="true"
|
||||||
|
elif command -v dnf >/dev/null 2>&1; then
|
||||||
|
manager="dnf"
|
||||||
|
# check-update exits 100 when there are updates, 0 when none, 1 on an error. --cacheonly: never goes to the network from here.
|
||||||
|
out=$(LC_ALL=C timeout 120 dnf -q check-update --cacheonly 2>/dev/null) || status=$?
|
||||||
|
if [[ "$status" -eq 0 || "$status" -eq 100 ]]; then
|
||||||
|
count=$(awk 'NF >= 3 && $1 ~ /^[A-Za-z0-9_.+-]+\.[A-Za-z0-9_]+$/ && /Obsoleting/ == 0 { n++ } /^Obsoleting/ { exit } END { print n + 0 }' <<<"$out")
|
||||||
|
security=$(LC_ALL=C timeout 120 dnf -q updateinfo list --security --cacheonly 2>/dev/null | grep -c '/Sec\.' || true)
|
||||||
|
fi
|
||||||
|
if command -v needs-restarting >/dev/null 2>&1; then
|
||||||
|
needs-restarting -r >/dev/null 2>&1 || reboot="true"
|
||||||
|
fi
|
||||||
|
elif command -v zypper >/dev/null 2>&1; then
|
||||||
|
manager="zypper"
|
||||||
|
if out=$(LC_ALL=C timeout 120 zypper -q --no-refresh list-updates 2>/dev/null); then
|
||||||
|
count=$(grep -c '^v |' <<<"$out" || true)
|
||||||
|
fi
|
||||||
|
[[ -f /run/reboot-needed ]] && reboot="true"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then
|
||||||
|
[[ "$security" =~ ^[0-9]+$ ]] || security="null"
|
||||||
|
UPDATES_JSON=$(jq -n --arg manager "$manager" --argjson count "$count" --argjson security "$security" --argjson reboot "$reboot" \
|
||||||
|
'{manager: $manager, count: $count, security: $security, reboot_required: $reboot}')
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
collect_system_info() {
|
collect_system_info() {
|
||||||
local ip_json="[]"
|
local ip_json="[]"
|
||||||
if command -v ip >/dev/null 2>&1; then
|
if command -v ip >/dev/null 2>&1; then
|
||||||
@@ -243,7 +283,12 @@ collect_system_info() {
|
|||||||
')
|
')
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Best-effort like the rest: a package manager that misbehaves must not cost the report.
|
||||||
|
UPDATES_JSON="null"
|
||||||
|
collect_updates || UPDATES_JSON="null"
|
||||||
|
|
||||||
SYSTEM_JSON=$(jq -n \
|
SYSTEM_JSON=$(jq -n \
|
||||||
|
--argjson updates "$UPDATES_JSON" \
|
||||||
--argjson ip_addresses "$ip_json" \
|
--argjson ip_addresses "$ip_json" \
|
||||||
--argjson listening_ports "$ports_json" \
|
--argjson listening_ports "$ports_json" \
|
||||||
--arg cpu_model "$cpu_model" \
|
--arg cpu_model "$cpu_model" \
|
||||||
@@ -257,7 +302,8 @@ collect_system_info() {
|
|||||||
cpu: { model: $cpu_model, cores: $cpu_cores, load_percent: $cpu_load_percent },
|
cpu: { model: $cpu_model, cores: $cpu_cores, load_percent: $cpu_load_percent },
|
||||||
memory: { total_bytes: $mem_total_bytes, used_bytes: $mem_used_bytes },
|
memory: { total_bytes: $mem_total_bytes, used_bytes: $mem_used_bytes },
|
||||||
disks: $disks,
|
disks: $disks,
|
||||||
listening_ports: $listening_ports
|
listening_ports: $listening_ports,
|
||||||
|
updates: $updates
|
||||||
}')
|
}')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ runs as a scheduled task (as SYSTEM, every 15 minutes) and pushes to `POST /api/
|
|||||||
the machine never needs to be reachable from Homelab Manager.
|
the machine never needs to be reachable from Homelab Manager.
|
||||||
|
|
||||||
|
|
||||||
|
Each report also says how many updates Windows Update has waiting (and how many are security or critical, and whether a restart is due). Searching Windows Update goes over the network and can take a minute or two, so it is done at most every six hours; the answer is kept in `%ProgramData%\HomelabManager\updates.json` and reused between. Nothing is downloaded or installed.
|
||||||
|
|
||||||
The installer limits `%ProgramData%\HomelabManager` to SYSTEM and Administrators. The agent runs as SYSTEM, so anyone who could change its script could run code as SYSTEM; the folder is not left to inherit ProgramData's default permissions. Re-running the install one-liner applies this to an existing install.
|
The installer limits `%ProgramData%\HomelabManager` to SYSTEM and Administrators. The agent runs as SYSTEM, so anyone who could change its script could run code as SYSTEM; the folder is not left to inherit ProgramData's default permissions. Re-running the install one-liner applies this to an existing install.
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
|
|||||||
@@ -220,6 +220,50 @@ function Get-ListeningPorts {
|
|||||||
return @($rows | Where-Object { $_.port -ge 1 -and $_.port -le 65535 } | Select-Object -First 2000)
|
return @($rows | Where-Object { $_.port -ge 1 -and $_.port -le 65535 } | Select-Object -First 2000)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ---- updates waiting to be installed ------------------------------------------------------------
|
||||||
|
$script:UpdatesCachePath = Join-Path $env:ProgramData 'HomelabManager\updates.json'
|
||||||
|
|
||||||
|
function Test-RebootRequired {
|
||||||
|
foreach ($key in @(
|
||||||
|
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired',
|
||||||
|
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending')) {
|
||||||
|
if (Test-Path $key) { return $true }
|
||||||
|
}
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
|
||||||
|
# Asks Windows Update what is waiting. That goes over the network and can take a minute, so it is done at most every six hours and the
|
||||||
|
# last answer is reused in between; whether a restart is due is cheap and always read fresh. Nothing is installed or downloaded.
|
||||||
|
function Get-PendingUpdates {
|
||||||
|
$result = $null
|
||||||
|
try {
|
||||||
|
if (Test-Path $script:UpdatesCachePath) {
|
||||||
|
$cached = Get-Content -Raw -Path $script:UpdatesCachePath | ConvertFrom-Json
|
||||||
|
$checked = [datetime]::Parse([string]$cached.checked_at, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::RoundtripKind)
|
||||||
|
if (([datetime]::UtcNow - $checked.ToUniversalTime()).TotalHours -lt 6) { $result = $cached }
|
||||||
|
}
|
||||||
|
} catch { $result = $null }
|
||||||
|
|
||||||
|
if (-not $result) {
|
||||||
|
$session = New-Object -ComObject Microsoft.Update.Session
|
||||||
|
$found = $session.CreateUpdateSearcher().Search("IsInstalled=0 and IsHidden=0 and Type='Software'")
|
||||||
|
$security = 0
|
||||||
|
foreach ($u in $found.Updates) {
|
||||||
|
foreach ($c in $u.Categories) {
|
||||||
|
if ($c.Name -eq 'Security Updates' -or $c.Name -eq 'Critical Updates') { $security++; break }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$result = [pscustomobject][ordered]@{ checked_at = [datetime]::UtcNow.ToString('o'); count = [int]$found.Updates.Count; security = [int]$security }
|
||||||
|
try {
|
||||||
|
$dir = Split-Path $script:UpdatesCachePath
|
||||||
|
if (-not (Test-Path $dir)) { $null = New-Item -ItemType Directory -Path $dir -Force }
|
||||||
|
ConvertTo-Json -InputObject $result | Set-Content -Path $script:UpdatesCachePath -Encoding UTF8
|
||||||
|
} catch { }
|
||||||
|
}
|
||||||
|
|
||||||
|
return [ordered]@{ manager = 'windows-update'; count = [int]$result.count; security = [int]$result.security; reboot_required = (Test-RebootRequired) }
|
||||||
|
}
|
||||||
|
|
||||||
function Get-SystemInfo {
|
function Get-SystemInfo {
|
||||||
$ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue |
|
$ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue |
|
||||||
Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } |
|
Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } |
|
||||||
@@ -240,7 +284,12 @@ function Get-SystemInfo {
|
|||||||
$disks += [pscustomobject][ordered]@{ mount = [string]$d.DeviceID; size_bytes = [int64]$d.Size; used_bytes = [int64]($d.Size - $d.FreeSpace) }
|
$disks += [pscustomobject][ordered]@{ mount = [string]$d.DeviceID; size_bytes = [int64]$d.Size; used_bytes = [int64]($d.Size - $d.FreeSpace) }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Best-effort: Windows Update being unreachable must not cost the rest of the report.
|
||||||
|
$updates = $null
|
||||||
|
try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))" }
|
||||||
|
|
||||||
return [ordered]@{
|
return [ordered]@{
|
||||||
|
updates = $updates
|
||||||
ip_addresses = @($ips)
|
ip_addresses = @($ips)
|
||||||
cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) }
|
cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) }
|
||||||
memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) }
|
memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) }
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
ALTER TABLE `servers` ADD `parent_server_id` integer;--> statement-breakpoint
|
||||||
|
ALTER TABLE `servers` ADD `pending_updates` text;--> statement-breakpoint
|
||||||
|
ALTER TABLE `servers` ADD `updates_pending_since` text;
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
CREATE TABLE `maintenance_schedules` (
|
||||||
|
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
|
||||||
|
`name` text NOT NULL,
|
||||||
|
`reason` text,
|
||||||
|
`targets` text NOT NULL,
|
||||||
|
`days` text NOT NULL,
|
||||||
|
`start_time` text NOT NULL,
|
||||||
|
`duration_minutes` integer NOT NULL,
|
||||||
|
`timezone` text NOT NULL,
|
||||||
|
`enabled` integer DEFAULT true NOT NULL,
|
||||||
|
`created_by` text,
|
||||||
|
`created_at` text DEFAULT (current_timestamp) NOT NULL
|
||||||
|
);
|
||||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -169,6 +169,20 @@
|
|||||||
"when": 1792181083723,
|
"when": 1792181083723,
|
||||||
"tag": "0023_premium_cyclops",
|
"tag": "0023_premium_cyclops",
|
||||||
"breakpoints": true
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 24,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1792191100783,
|
||||||
|
"tag": "0024_chubby_jackpot",
|
||||||
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 25,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1792191757544,
|
||||||
|
"tag": "0025_clean_the_hand",
|
||||||
|
"breakpoints": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -282,6 +282,15 @@ export const servers = sqliteTable("servers", {
|
|||||||
// card always shows) once a server IS actually linked, so unlinking stays
|
// card always shows) once a server IS actually linked, so unlinking stays
|
||||||
// reachable.
|
// reachable.
|
||||||
hideProxmoxLink: integer("hide_proxmox_link", { mode: "boolean" }).notNull().default(false),
|
hideProxmoxLink: integer("hide_proxmox_link", { mode: "boolean" }).notNull().default(false),
|
||||||
|
|
||||||
|
// The server this one runs on (a VM on its host), set by an admin. Null means "work it out": a guest linked to a Proxmox node runs on
|
||||||
|
// the server named after that node. When the parent stops reporting, this server's own offline alert is folded under the parent's.
|
||||||
|
parentServerId: integer("parent_server_id"),
|
||||||
|
|
||||||
|
// What the agent says is waiting to be installed, as JSON: {manager, count, security, rebootRequired, reportedAt}. Null until it reports.
|
||||||
|
pendingUpdates: text("pending_updates"),
|
||||||
|
// When updates were first seen waiting (and none were before) — how long the oldest has been waiting. Null when none are.
|
||||||
|
updatesPendingSince: text("updates_pending_since"),
|
||||||
});
|
});
|
||||||
|
|
||||||
export const scheduledTasks = sqliteTable("scheduled_tasks", {
|
export const scheduledTasks = sqliteTable("scheduled_tasks", {
|
||||||
@@ -505,3 +514,24 @@ export const shareLinks = sqliteTable("share_links", {
|
|||||||
expiresAt: text("expires_at"), // ISO; null = never
|
expiresAt: text("expires_at"), // ISO; null = never
|
||||||
lastUsedAt: text("last_used_at"), // ISO
|
lastUsedAt: text("last_used_at"), // ISO
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ─── Recurring maintenance ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// "Every Sunday 02:00–04:00 on these servers": a schedule kept here, turned into ordinary maintenance windows a week ahead (each marked
|
||||||
|
// `schedule:<id>:<date>` in maintenance_windows.source). The windows are what silence alerts; the schedule only says when they come.
|
||||||
|
export const maintenanceSchedules = sqliteTable("maintenance_schedules", {
|
||||||
|
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||||
|
name: text("name").notNull(), // e.g. "Patch night"
|
||||||
|
reason: text("reason"),
|
||||||
|
// JSON: {type: "server" | "integration" | "dns_provider", id: number}[] — polymorphic, so no foreign key; a deleted target is skipped.
|
||||||
|
targets: text("targets").notNull(),
|
||||||
|
days: text("days").notNull(), // JSON: weekdays it happens on, 0 = Sunday … 6 = Saturday
|
||||||
|
startTime: text("start_time").notNull(), // "HH:MM" on the clock of `timezone`
|
||||||
|
durationMinutes: integer("duration_minutes").notNull(),
|
||||||
|
timezone: text("timezone").notNull(), // IANA name, so "02:00" stays 02:00 across summer and winter time
|
||||||
|
enabled: integer("enabled", { mode: "boolean" }).notNull().default(true),
|
||||||
|
createdBy: text("created_by"),
|
||||||
|
createdAt: text("created_at")
|
||||||
|
.notNull()
|
||||||
|
.default(sql`(current_timestamp)`),
|
||||||
|
});
|
||||||
@@ -13,6 +13,7 @@ import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js";
|
|||||||
import { initHealthScheduler } from "./services/healthScheduler.js";
|
import { initHealthScheduler } from "./services/healthScheduler.js";
|
||||||
import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js";
|
import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js";
|
||||||
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
|
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
|
||||||
|
import { initMaintenanceScheduleScheduler } from "./services/maintenanceScheduleScheduler.js";
|
||||||
|
|
||||||
{
|
{
|
||||||
// Sessions are signed with this secret. Without one, the app would sign them with a value that's in the source code.
|
// Sessions are signed with this secret. Without one, the app would sign them with a value that's in the source code.
|
||||||
@@ -42,6 +43,7 @@ await initQuietHoursScheduler();
|
|||||||
await initHealthScheduler();
|
await initHealthScheduler();
|
||||||
await initAlertTicketScheduler();
|
await initAlertTicketScheduler();
|
||||||
await initKumaMaintenanceScheduler();
|
await initKumaMaintenanceScheduler();
|
||||||
|
await initMaintenanceScheduleScheduler();
|
||||||
|
|
||||||
createApp().listen(env.port, () => {
|
createApp().listen(env.port, () => {
|
||||||
console.log(`homelab-manager listening on port ${env.port}`);
|
console.log(`homelab-manager listening on port ${env.port}`);
|
||||||
|
|||||||
@@ -45,12 +45,20 @@ export interface SemaphoreTemplate {
|
|||||||
lastTask: SemaphoreTask | null;
|
lastTask: SemaphoreTask | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** What a run may be told: which hosts of the inventory to run on (ansible's --limit), and a note shown beside the run in Semaphore. */
|
||||||
|
export interface SemaphoreRunOptions {
|
||||||
|
limit?: string;
|
||||||
|
message?: string;
|
||||||
|
}
|
||||||
|
|
||||||
export interface SemaphoreAdapter {
|
export interface SemaphoreAdapter {
|
||||||
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
|
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
|
||||||
listTemplatesWithStatus(): Promise<SemaphoreTemplate[]>;
|
listTemplatesWithStatus(): Promise<SemaphoreTemplate[]>;
|
||||||
/** Like listTemplatesWithStatus, but says which projects couldn't be read — for callers that must not mistake "couldn't read" for "no templates". */
|
/** Like listTemplatesWithStatus, but says which projects couldn't be read — for callers that must not mistake "couldn't read" for "no templates". */
|
||||||
checkTemplates(): Promise<{ templates: SemaphoreTemplate[]; failedProjectIds: number[] }>;
|
checkTemplates(): Promise<{ templates: SemaphoreTemplate[]; failedProjectIds: number[] }>;
|
||||||
runTemplate(projectId: number, templateId: number): Promise<SemaphoreTask>;
|
runTemplate(projectId: number, templateId: number, options?: SemaphoreRunOptions): Promise<SemaphoreTask>;
|
||||||
|
/** How a run is getting on. */
|
||||||
|
getTask(projectId: number, taskId: number): Promise<SemaphoreTask>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapter {
|
export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapter {
|
||||||
@@ -134,11 +142,19 @@ export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapte
|
|||||||
return (await checkTemplates()).templates;
|
return (await checkTemplates()).templates;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function runTemplate(projectId: number, templateId: number): Promise<SemaphoreTask> {
|
async function runTemplate(projectId: number, templateId: number, options: SemaphoreRunOptions = {}): Promise<SemaphoreTask> {
|
||||||
const task = await api("POST", `/project/${projectId}/tasks`, { template_id: templateId });
|
const task = await api("POST", `/project/${projectId}/tasks`, {
|
||||||
|
template_id: templateId,
|
||||||
|
...(options.limit ? { limit: options.limit } : {}),
|
||||||
|
...(options.message ? { message: options.message } : {}),
|
||||||
|
});
|
||||||
return mapTask(task)!;
|
return mapTask(task)!;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function getTask(projectId: number, taskId: number): Promise<SemaphoreTask> {
|
||||||
|
return mapTask(await api("GET", `/project/${projectId}/tasks/${taskId}`))!;
|
||||||
|
}
|
||||||
|
|
||||||
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
|
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
|
||||||
const start = Date.now();
|
const start = Date.now();
|
||||||
try {
|
try {
|
||||||
@@ -149,5 +165,5 @@ export function createSemaphoreAdapter(config: SemaphoreConfig): SemaphoreAdapte
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return withDiagLogging("semaphore", { ping, listTemplatesWithStatus, checkTemplates, runTemplate });
|
return withDiagLogging("semaphore", { ping, listTemplatesWithStatus, checkTemplates, runTemplate, getTask });
|
||||||
}
|
}
|
||||||
@@ -23,7 +23,22 @@ const listeningPortSchema = z.object({
|
|||||||
process: text(100).optional(),
|
process: text(100).optional(),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// What is waiting to be installed. As lenient as the ports: a malformed answer is dropped, never allowed to cost the agent its report.
|
||||||
|
const updatesSchema = z.object({
|
||||||
|
manager: text(30).optional(),
|
||||||
|
count: z.number().int().min(0).max(100_000),
|
||||||
|
security: z.number().int().min(0).max(100_000).nullable().optional(),
|
||||||
|
reboot_required: z.boolean().optional(),
|
||||||
|
});
|
||||||
|
|
||||||
const systemSchema = z.object({
|
const systemSchema = z.object({
|
||||||
|
updates: z
|
||||||
|
.unknown()
|
||||||
|
.optional()
|
||||||
|
.transform((u) => {
|
||||||
|
const parsed = updatesSchema.safeParse(u);
|
||||||
|
return parsed.success ? parsed.data : undefined;
|
||||||
|
}),
|
||||||
ip_addresses: z.array(text(100)).transform(cap(200)).optional(),
|
ip_addresses: z.array(text(100)).transform(cap(200)).optional(),
|
||||||
cpu: z.object({ model: text(200).optional(), cores: z.number().optional(), load_percent: z.number().nullable().optional() }).optional(),
|
cpu: z.object({ model: text(200).optional(), cores: z.number().optional(), load_percent: z.number().nullable().optional() }).optional(),
|
||||||
memory: z.object({ total_bytes: z.number().optional(), used_bytes: z.number().optional() }).optional(),
|
memory: z.object({ total_bytes: z.number().optional(), used_bytes: z.number().optional() }).optional(),
|
||||||
|
|||||||
@@ -633,6 +633,14 @@ integrationsRouter.get("/:id/semaphore/templates", asyncHandler(async (req, res)
|
|||||||
}
|
}
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
// Which hosts a run may be limited to: names, addresses and ansible patterns (commas, colons, wildcards) — and nothing that could be
|
||||||
|
// taken for an option. Semaphore hands the value to ansible as an argument, but it is still only ever shown the characters a host has.
|
||||||
|
const runBodySchema = z
|
||||||
|
.object({
|
||||||
|
limit: z.union([z.literal(""), z.string().trim().regex(/^[A-Za-z0-9][A-Za-z0-9._,:*\-]{0,199}$/, "Use host names or patterns like web01,db*")]).optional(),
|
||||||
|
})
|
||||||
|
.default({});
|
||||||
|
|
||||||
integrationsRouter.post(
|
integrationsRouter.post(
|
||||||
"/:id/semaphore/projects/:projectId/templates/:templateId/run",
|
"/:id/semaphore/projects/:projectId/templates/:templateId/run",
|
||||||
requireRole("operator"),
|
requireRole("operator"),
|
||||||
@@ -645,16 +653,25 @@ integrationsRouter.post(
|
|||||||
if (!Number.isInteger(projectId) || !Number.isInteger(templateId)) {
|
if (!Number.isInteger(projectId) || !Number.isInteger(templateId)) {
|
||||||
return res.status(400).json({ error: "invalid_id" });
|
return res.status(400).json({ error: "invalid_id" });
|
||||||
}
|
}
|
||||||
|
const body = runBodySchema.safeParse(req.body ?? {});
|
||||||
|
if (!body.success) {
|
||||||
|
return res.status(400).json({ error: "invalid_body", message: "The hosts to limit it to can only be host names or patterns like web01,db*.", details: body.error.flatten() });
|
||||||
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const task = await found.adapter.runTemplate(projectId, templateId);
|
const actor = req.currentUser!;
|
||||||
|
const limit = body.data.limit || undefined;
|
||||||
|
const task = await found.adapter.runTemplate(projectId, templateId, {
|
||||||
|
limit,
|
||||||
|
message: `Started in Homelab Manager by ${actor.name ?? actor.email ?? actor.oidcSub}${limit ? ` (limited to ${limit})` : ""}`,
|
||||||
|
});
|
||||||
await recordAudit({
|
await recordAudit({
|
||||||
actor: req.currentUser!,
|
actor,
|
||||||
category: "integration",
|
category: "integration",
|
||||||
action: "run_template",
|
action: "run_template",
|
||||||
targetType: "semaphore_template",
|
targetType: "semaphore_template",
|
||||||
targetId: templateId,
|
targetId: templateId,
|
||||||
detail: { integrationId: found.integration.id, projectId, taskId: task.id },
|
detail: { integrationId: found.integration.id, projectId, taskId: task.id, ...(limit ? { limit } : {}) },
|
||||||
});
|
});
|
||||||
res.status(201).json({ task });
|
res.status(201).json({ task });
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
@@ -663,6 +680,20 @@ integrationsRouter.post(
|
|||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// How a run is getting on: waiting, running, success or error.
|
||||||
|
integrationsRouter.get("/:id/semaphore/projects/:projectId/tasks/:taskId", asyncHandler(async (req, res) => {
|
||||||
|
const found = await requireSemaphoreAdapter(req, res);
|
||||||
|
if (!found) return;
|
||||||
|
const projectId = Number(req.params.projectId);
|
||||||
|
const taskId = Number(req.params.taskId);
|
||||||
|
if (!Number.isInteger(projectId) || !Number.isInteger(taskId)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
try {
|
||||||
|
res.json({ task: await found.adapter.getTask(projectId, taskId) });
|
||||||
|
} catch (err) {
|
||||||
|
res.status(502).json({ error: err instanceof Error ? err.message : String(err) });
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
|
||||||
// ─── Proxmox ─────────────────────────────────────────────────────────────────
|
// ─── Proxmox ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
async function requireProxmoxAdapter(req: Request, res: Response) {
|
async function requireProxmoxAdapter(req: Request, res: Response) {
|
||||||
|
|||||||
@@ -2,7 +2,8 @@ import { Router } from "express";
|
|||||||
import { and, eq, inArray } from "drizzle-orm";
|
import { and, eq, inArray } from "drizzle-orm";
|
||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { dnsProviders, integrations, maintenanceAliases, maintenanceTargetTypes, maintenanceWindows, servers } from "../db/schema.js";
|
import { dnsProviders, integrations, maintenanceAliases, maintenanceSchedules, maintenanceTargetTypes, maintenanceWindows, servers } from "../db/schema.js";
|
||||||
|
import { deleteSchedule, isValidTimeZone, materializeSchedules, occurrencesBetween, removeFutureWindows, targetsOf } from "../services/maintenanceSchedules.js";
|
||||||
import { hostOfAddress } from "../services/kumaTargets.js";
|
import { hostOfAddress } from "../services/kumaTargets.js";
|
||||||
import { invalidateAlertsCache } from "../services/alerts.js";
|
import { invalidateAlertsCache } from "../services/alerts.js";
|
||||||
import { requireAuth, requireRole } from "../auth/middleware.js";
|
import { requireAuth, requireRole } from "../auth/middleware.js";
|
||||||
@@ -268,6 +269,105 @@ maintenanceRouter.post("/end-all", requireRole("operator"), asyncHandler(async (
|
|||||||
res.json({ ended: active.length });
|
res.json({ ended: active.length });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
// ─── Recurring schedules ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const scheduleSchema = z.object({
|
||||||
|
name: z.string().trim().min(1).max(80),
|
||||||
|
reason: z.string().trim().max(200).optional(),
|
||||||
|
targets: z.array(z.object({ type: z.enum(maintenanceTargetTypes), id: z.number().int() })).min(1).max(100),
|
||||||
|
days: z.array(z.number().int().min(0).max(6)).min(1).max(7),
|
||||||
|
startTime: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/, "Use a time like 02:00"),
|
||||||
|
durationMinutes: z.number().int().min(15).max(3 * 24 * 60),
|
||||||
|
timezone: z.string().min(1).max(64),
|
||||||
|
});
|
||||||
|
|
||||||
|
maintenanceRouter.get("/schedules", asyncHandler(async (_req, res) => {
|
||||||
|
const rows = await db.select().from(maintenanceSchedules).orderBy(maintenanceSchedules.name);
|
||||||
|
const now = new Date();
|
||||||
|
const out = [];
|
||||||
|
for (const s of rows) {
|
||||||
|
const targets = [];
|
||||||
|
for (const t of targetsOf(s)) {
|
||||||
|
const d = await describeTarget(t.type, t.id);
|
||||||
|
if (d) targets.push({ type: t.type, id: t.id, name: d.name, kind: d.kind });
|
||||||
|
}
|
||||||
|
const next = s.enabled ? occurrencesBetween(s, now, new Date(now.getTime() + 15 * 24 * 3600_000))[0] : undefined;
|
||||||
|
out.push({
|
||||||
|
id: s.id,
|
||||||
|
name: s.name,
|
||||||
|
reason: s.reason,
|
||||||
|
days: JSON.parse(s.days) as number[],
|
||||||
|
startTime: s.startTime,
|
||||||
|
durationMinutes: s.durationMinutes,
|
||||||
|
timezone: s.timezone,
|
||||||
|
enabled: s.enabled,
|
||||||
|
createdBy: s.createdBy,
|
||||||
|
targets,
|
||||||
|
nextStart: next ? next.start.toISOString() : null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
res.json({ schedules: out });
|
||||||
|
}));
|
||||||
|
|
||||||
|
maintenanceRouter.post("/schedules", requireRole("operator"), asyncHandler(async (req, res) => {
|
||||||
|
const parsed = scheduleSchema.safeParse(req.body);
|
||||||
|
if (!parsed.success) {
|
||||||
|
return res.status(400).json({ error: "invalid_body", message: "Name it, choose what it covers, the days, a start time and how long.", details: parsed.error.flatten() });
|
||||||
|
}
|
||||||
|
const data = parsed.data;
|
||||||
|
if (!isValidTimeZone(data.timezone)) return res.status(400).json({ error: "invalid_timezone", message: "That isn't a time zone this server knows." });
|
||||||
|
// Only things that exist, once each.
|
||||||
|
const seen = new Set<string>();
|
||||||
|
const targets = [];
|
||||||
|
for (const t of data.targets) {
|
||||||
|
const key = `${t.type}:${t.id}`;
|
||||||
|
if (seen.has(key)) continue;
|
||||||
|
seen.add(key);
|
||||||
|
if (!(await describeTarget(t.type, t.id))) return res.status(400).json({ error: "unknown_target", message: "One of the things chosen doesn't exist any more." });
|
||||||
|
targets.push(t);
|
||||||
|
}
|
||||||
|
const actor = req.currentUser!;
|
||||||
|
const [created] = await db
|
||||||
|
.insert(maintenanceSchedules)
|
||||||
|
.values({
|
||||||
|
name: data.name,
|
||||||
|
reason: data.reason || null,
|
||||||
|
targets: JSON.stringify(targets),
|
||||||
|
days: JSON.stringify([...new Set(data.days)].sort()),
|
||||||
|
startTime: data.startTime,
|
||||||
|
durationMinutes: data.durationMinutes,
|
||||||
|
timezone: data.timezone,
|
||||||
|
createdBy: actor.name ?? actor.email ?? actor.oidcSub,
|
||||||
|
})
|
||||||
|
.returning();
|
||||||
|
const made = await materializeSchedules(new Date(), undefined, created.id);
|
||||||
|
await recordAudit({ actor, category: "maintenance", action: "create_schedule", targetType: "maintenance_schedule", targetId: created.id, detail: { name: created.name, targets: targets.length, days: data.days, startTime: data.startTime, durationMinutes: data.durationMinutes } });
|
||||||
|
res.status(201).json({ id: created.id, windowsMade: made });
|
||||||
|
}));
|
||||||
|
|
||||||
|
maintenanceRouter.patch("/schedules/:id", requireRole("operator"), asyncHandler(async (req, res) => {
|
||||||
|
const id = Number(req.params.id);
|
||||||
|
const parsed = z.object({ enabled: z.boolean() }).safeParse(req.body);
|
||||||
|
if (!Number.isInteger(id) || !parsed.success) return res.status(400).json({ error: "invalid_body" });
|
||||||
|
const [row] = await db.select().from(maintenanceSchedules).where(eq(maintenanceSchedules.id, id)).limit(1);
|
||||||
|
if (!row) return res.status(404).json({ error: "not_found" });
|
||||||
|
await db.update(maintenanceSchedules).set({ enabled: parsed.data.enabled }).where(eq(maintenanceSchedules.id, id));
|
||||||
|
if (parsed.data.enabled) await materializeSchedules(new Date(), undefined, id);
|
||||||
|
else await removeFutureWindows(id);
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "maintenance", action: parsed.data.enabled ? "enable_schedule" : "disable_schedule", targetType: "maintenance_schedule", targetId: id, detail: { name: row.name } });
|
||||||
|
res.json({ enabled: parsed.data.enabled });
|
||||||
|
}));
|
||||||
|
|
||||||
|
maintenanceRouter.delete("/schedules/:id", requireRole("operator"), asyncHandler(async (req, res) => {
|
||||||
|
const id = Number(req.params.id);
|
||||||
|
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
const [row] = await db.select().from(maintenanceSchedules).where(eq(maintenanceSchedules.id, id)).limit(1);
|
||||||
|
if (!row) return res.status(404).json({ error: "not_found" });
|
||||||
|
await deleteSchedule(id);
|
||||||
|
await recordAudit({ actor: req.currentUser!, category: "maintenance", action: "delete_schedule", targetType: "maintenance_schedule", targetId: id, detail: { name: row.name } });
|
||||||
|
res.status(204).end();
|
||||||
|
}));
|
||||||
|
|
||||||
const syncUptimeKumaSchema = z.object({ integrationId: z.number().int().positive() });
|
const syncUptimeKumaSchema = z.object({ integrationId: z.number().int().positive() });
|
||||||
|
|
||||||
/** Reads Uptime Kuma's maintenance schedule now and brings the mirrored windows in line with it (the same pass that runs every 15 minutes when switched on). */
|
/** Reads Uptime Kuma's maintenance schedule now and brings the mirrored windows in line with it (the same pass that runs every 15 minutes when switched on). */
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import { servers, serverLinks, integrations, dnsRecordsCache, dnsZonesCache, dns
|
|||||||
import { requireAuth, requireRole } from "../auth/middleware.js";
|
import { requireAuth, requireRole } from "../auth/middleware.js";
|
||||||
import { generateApiToken } from "../services/tokens.js";
|
import { generateApiToken } from "../services/tokens.js";
|
||||||
import { recordAudit } from "../services/audit.js";
|
import { recordAudit } from "../services/audit.js";
|
||||||
|
import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js";
|
||||||
|
import { parsePendingUpdates } from "../services/pendingUpdates.js";
|
||||||
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
|
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
|
||||||
import { getServerActivity } from "../services/serverActivity.js";
|
import { getServerActivity } from "../services/serverActivity.js";
|
||||||
import { asyncHandler } from "../utils/asyncHandler.js";
|
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||||
@@ -21,9 +23,9 @@ import { getSettings } from "../services/settingsStore.js";
|
|||||||
export const serversRouter = Router();
|
export const serversRouter = Router();
|
||||||
|
|
||||||
/** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */
|
/** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */
|
||||||
function publicServer<T extends { apiTokenHash: string; tags: string | null }>(row: T) {
|
function publicServer<T extends { apiTokenHash: string; tags: string | null; pendingUpdates: string | null }>(row: T) {
|
||||||
const { apiTokenHash: _hash, tags, ...rest } = row;
|
const { apiTokenHash: _hash, tags, pendingUpdates, ...rest } = row;
|
||||||
return { ...rest, tags: parseTags(tags) };
|
return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates) };
|
||||||
}
|
}
|
||||||
|
|
||||||
serversRouter.use(requireAuth);
|
serversRouter.use(requireAuth);
|
||||||
@@ -113,6 +115,8 @@ const updateServerSchema = z
|
|||||||
proxmoxGuestType: z.enum(["qemu", "lxc"]).nullable().optional(),
|
proxmoxGuestType: z.enum(["qemu", "lxc"]).nullable().optional(),
|
||||||
proxmoxVmid: z.number().int().nullable().optional(),
|
proxmoxVmid: z.number().int().nullable().optional(),
|
||||||
hideProxmoxLink: z.boolean().optional(),
|
hideProxmoxLink: z.boolean().optional(),
|
||||||
|
// The server this one runs on. Null goes back to working it out from the Proxmox node.
|
||||||
|
parentServerId: z.number().int().nullable().optional(),
|
||||||
})
|
})
|
||||||
.refine(
|
.refine(
|
||||||
(data) => {
|
(data) => {
|
||||||
@@ -148,6 +152,17 @@ serversRouter.patch("/:id", requireRole("admin"), asyncHandler(async (req, res)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (parsed.data.parentServerId !== undefined && parsed.data.parentServerId !== null) {
|
||||||
|
const parentId = parsed.data.parentServerId;
|
||||||
|
const rows = await loadParentRows();
|
||||||
|
if (parentId === id || !rows.some((r) => r.id === parentId)) {
|
||||||
|
return res.status(400).json({ error: "invalid_parent", message: "Choose another server that exists." });
|
||||||
|
}
|
||||||
|
if (wouldLoop(rows, id, parentId)) {
|
||||||
|
return res.status(400).json({ error: "parent_loop", message: "That server already runs on this one, so each would be waiting for the other." });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const [updated] = await db.update(servers).set(parsed.data).where(eq(servers.id, id)).returning();
|
const [updated] = await db.update(servers).set(parsed.data).where(eq(servers.id, id)).returning();
|
||||||
|
|
||||||
await recordAudit({
|
await recordAudit({
|
||||||
@@ -156,7 +171,7 @@ serversRouter.patch("/:id", requireRole("admin"), asyncHandler(async (req, res)
|
|||||||
action: "update",
|
action: "update",
|
||||||
targetType: "server",
|
targetType: "server",
|
||||||
targetId: id,
|
targetId: id,
|
||||||
detail: { name: updated.name },
|
detail: { name: updated.name, ...(parsed.data.parentServerId !== undefined ? { parentServerId: parsed.data.parentServerId } : {}) },
|
||||||
});
|
});
|
||||||
|
|
||||||
const serverOut = publicServer(updated);
|
const serverOut = publicServer(updated);
|
||||||
@@ -173,6 +188,27 @@ serversRouter.get("/summary", asyncHandler(async (_req, res) => {
|
|||||||
res.json(buildServerSummary(rows, healthChecks, Date.now(), await activeSubjects()));
|
res.json(buildServerSummary(rows, healthChecks, Date.now(), await activeSubjects()));
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
// What every server that reports through an agent says is waiting to be installed, for the Updates page. Servers whose agent hasn't
|
||||||
|
// said (it predates this, or couldn't find out) are listed with no figures rather than left out.
|
||||||
|
serversRouter.get("/updates", asyncHandler(async (_req, res) => {
|
||||||
|
const rows = await db
|
||||||
|
.select({
|
||||||
|
id: servers.id,
|
||||||
|
name: servers.name,
|
||||||
|
osType: servers.osType,
|
||||||
|
lastSeenAt: servers.lastSeenAt,
|
||||||
|
pendingUpdates: servers.pendingUpdates,
|
||||||
|
updatesPendingSince: servers.updatesPendingSince,
|
||||||
|
})
|
||||||
|
.from(servers)
|
||||||
|
.orderBy(servers.name);
|
||||||
|
res.json({
|
||||||
|
servers: rows
|
||||||
|
.filter((r) => r.lastSeenAt !== null)
|
||||||
|
.map((r) => ({ id: r.id, name: r.name, osType: r.osType, lastSeenAt: r.lastSeenAt, updates: parsePendingUpdates(r.pendingUpdates), pendingSince: r.updatesPendingSince })),
|
||||||
|
});
|
||||||
|
}));
|
||||||
|
|
||||||
// For the Operations > Admin Links page — every server's admin bookmarks in one place, instead of visiting
|
// For the Operations > Admin Links page — every server's admin bookmarks in one place, instead of visiting
|
||||||
// each server's own detail page to find them.
|
// each server's own detail page to find them.
|
||||||
serversRouter.get("/links", asyncHandler(async (_req, res) => {
|
serversRouter.get("/links", asyncHandler(async (_req, res) => {
|
||||||
@@ -306,10 +342,21 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
|
|||||||
memUsedBytes: _memUsedBytes,
|
memUsedBytes: _memUsedBytes,
|
||||||
listeningPorts: _listeningPorts,
|
listeningPorts: _listeningPorts,
|
||||||
lastPortScan: _lastPortScan,
|
lastPortScan: _lastPortScan,
|
||||||
|
pendingUpdates: rawUpdates,
|
||||||
...serverOut
|
...serverOut
|
||||||
} = server;
|
} = server;
|
||||||
|
|
||||||
res.json({ server: { ...serverOut, tags: parseTags(rawTags) }, ipAddresses, hardware, dnsMatches, links });
|
// What this server runs on, and what runs on it (set by hand, or worked out from its Proxmox node).
|
||||||
|
const parentRows = await loadParentRows();
|
||||||
|
const links2 = resolveParents(parentRows);
|
||||||
|
const nameOf = new Map(parentRows.map((r) => [r.id, r.name]));
|
||||||
|
const parentLink = links2.get(id);
|
||||||
|
const relations = {
|
||||||
|
parent: parentLink ? { id: parentLink.parentId, name: nameOf.get(parentLink.parentId) ?? "", auto: parentLink.auto } : null,
|
||||||
|
children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)),
|
||||||
|
};
|
||||||
|
|
||||||
|
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates) }, ipAddresses, hardware, dnsMatches, links, relations });
|
||||||
}));
|
}));
|
||||||
|
|
||||||
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
|
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
|
||||||
@@ -440,6 +487,9 @@ serversRouter.delete("/:id", requireRole("admin"), asyncHandler(async (req, res)
|
|||||||
const id = Number(req.params.id);
|
const id = Number(req.params.id);
|
||||||
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
if (!Number.isInteger(id)) return res.status(400).json({ error: "invalid_id" });
|
||||||
|
|
||||||
|
// Servers that ran on this one go back to working out where they run (the link is a plain column, not a foreign key).
|
||||||
|
await db.update(servers).set({ parentServerId: null }).where(eq(servers.parentServerId, id));
|
||||||
|
|
||||||
const deleted = await db.delete(servers).where(eq(servers.id, id)).returning();
|
const deleted = await db.delete(servers).where(eq(servers.id, id)).returning();
|
||||||
if (deleted.length === 0) return res.status(404).json({ error: "not_found" });
|
if (deleted.length === 0) return res.status(404).json({ error: "not_found" });
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,10 @@ export interface Alert {
|
|||||||
link: string | null;
|
link: string | null;
|
||||||
/** Under an active maintenance window: still a real problem, but notifications for it are held back. */
|
/** Under an active maintenance window: still a real problem, but notifications for it are held back. */
|
||||||
silenced: boolean;
|
silenced: boolean;
|
||||||
|
/** Set when this problem is explained by another one (a guest down because its host is): it is held back like a silenced one, but the reason is this, not maintenance. */
|
||||||
|
causedBy?: { name: string };
|
||||||
|
/** On a host's alert: the servers on it whose own alerts are folded under this one. */
|
||||||
|
affects?: string[];
|
||||||
/** What a maintenance window would have to be on to silence it, when there is such a thing — which is what "Silence" acts on. */
|
/** What a maintenance window would have to be on to silence it, when there is such a thing — which is what "Silence" acts on. */
|
||||||
target: { type: "server" | "integration" | "dns_provider"; id: number; name: string } | null;
|
target: { type: "server" | "integration" | "dns_provider"; id: number; name: string } | null;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -28,12 +28,13 @@ import { computeSecretStatus } from "./secretStatus.js";
|
|||||||
import { getFailingSources } from "./integrationHealthMonitor.js";
|
import { getFailingSources } from "./integrationHealthMonitor.js";
|
||||||
import { getSettings } from "./settingsStore.js";
|
import { getSettings } from "./settingsStore.js";
|
||||||
import { sourceLabel } from "./notify.js";
|
import { sourceLabel } from "./notify.js";
|
||||||
|
import { coveredByParent, loadParentRows, resolveParents } from "./serverParents.js";
|
||||||
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
|
import type { Alert, AlertCategory, AlertSeverity, SourceFailure } from "./alertTypes.js";
|
||||||
|
|
||||||
export interface AlertsReport {
|
export interface AlertsReport {
|
||||||
alerts: Alert[];
|
alerts: Alert[];
|
||||||
/** Active problems by severity — those under a maintenance window are counted apart, not in these. */
|
/** Active problems by severity — those under a maintenance window are counted apart, not in these. */
|
||||||
counts: { critical: number; warning: number; info: number; silenced: number };
|
counts: { critical: number; warning: number; info: number; silenced: number; /** Held back because another problem explains them (a guest down because its host is). */ grouped: number };
|
||||||
/** Things that couldn't be checked, and which checks that leaves blind. */
|
/** Things that couldn't be checked, and which checks that leaves blind. */
|
||||||
couldntCheck: { name: string; error: string; affects: string[] }[];
|
couldntCheck: { name: string; error: string; affects: string[] }[];
|
||||||
/** Context worth knowing about how complete the picture is. */
|
/** Context worth knowing about how complete the picture is. */
|
||||||
@@ -115,7 +116,7 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
return row ? integrationTarget(row.id) : null;
|
return row ? integrationTarget(row.id) : null;
|
||||||
};
|
};
|
||||||
|
|
||||||
function push(a: { category: AlertCategory; severity: AlertSeverity; type: string; message: string; key: string; link?: string | null; silenced?: boolean; target?: Alert["target"] }) {
|
function push(a: { category: AlertCategory; severity: AlertSeverity; type: string; message: string; key: string; link?: string | null; silenced?: boolean; target?: Alert["target"]; causedBy?: Alert["causedBy"]; affects?: string[] }) {
|
||||||
const meta = SOURCES[a.type];
|
const meta = SOURCES[a.type];
|
||||||
alerts.push({
|
alerts.push({
|
||||||
id: `${a.category}:${a.key}`,
|
id: `${a.category}:${a.key}`,
|
||||||
@@ -124,8 +125,10 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
source: meta?.label ?? sourceLabel(a.type),
|
source: meta?.label ?? sourceLabel(a.type),
|
||||||
message: a.message,
|
message: a.message,
|
||||||
link: a.link === undefined ? (meta?.link ?? null) : a.link,
|
link: a.link === undefined ? (meta?.link ?? null) : a.link,
|
||||||
silenced: !!a.silenced,
|
silenced: !!a.silenced || !!a.causedBy,
|
||||||
target: a.target ?? null,
|
target: a.target ?? null,
|
||||||
|
...(a.causedBy ? { causedBy: a.causedBy } : {}),
|
||||||
|
...(a.affects?.length ? { affects: a.affects } : {}),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -160,10 +163,24 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
`Server-offline and server-disk checks are paused for another ${Math.ceil(grace / 60_000)} min after the app restarted, so agents get a chance to report before any server is judged.`,
|
`Server-offline and server-disk checks are paused for another ${Math.ceil(grace / 60_000)} min after the app restarted, so agents get a chance to report before any server is judged.`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
for (const c of evaluateHealth(snapshot, healthChecks, now, { skipServers: grace > 0 })) {
|
const conditions = evaluateHealth(snapshot, healthChecks, now, { skipServers: grace > 0 });
|
||||||
|
// A server that is down because the one it runs on is down is folded under that one.
|
||||||
|
const offlineIds = new Set<number>();
|
||||||
|
for (const c of conditions) {
|
||||||
|
const m = /^offline:server:(\d+)$/.exec(c.key);
|
||||||
|
if (m) offlineIds.add(Number(m[1]));
|
||||||
|
}
|
||||||
|
const cause = coveredByParent(offlineIds, resolveParents(await loadParentRows()));
|
||||||
|
const taken = new Map<number, string[]>();
|
||||||
|
for (const [child, parent] of cause) taken.set(parent, [...(taken.get(parent) ?? []), serverNames.get(child) ?? `#${child}`]);
|
||||||
|
for (const c of conditions) {
|
||||||
const origin = originOfConditionKey(c.key);
|
const origin = originOfConditionKey(c.key);
|
||||||
const subject = subjectOfConditionKey(c.key);
|
const subject = subjectOfConditionKey(c.key);
|
||||||
|
const offlineId = /^offline:server:(\d+)$/.exec(c.key)?.[1];
|
||||||
|
const parentOf = offlineId ? cause.get(Number(offlineId)) : undefined;
|
||||||
push({
|
push({
|
||||||
|
causedBy: parentOf !== undefined ? { name: serverNames.get(parentOf) ?? `#${parentOf}` } : undefined,
|
||||||
|
affects: offlineId ? taken.get(Number(offlineId))?.sort((a, b) => a.localeCompare(b)) : undefined,
|
||||||
category: origin.category,
|
category: origin.category,
|
||||||
severity: c.severity ?? "warning",
|
severity: c.severity ?? "warning",
|
||||||
type: origin.type,
|
type: origin.type,
|
||||||
@@ -403,13 +420,15 @@ export async function collectAlerts(): Promise<AlertsReport> {
|
|||||||
);
|
);
|
||||||
|
|
||||||
const active = alerts.filter((a) => !a.silenced);
|
const active = alerts.filter((a) => !a.silenced);
|
||||||
|
const grouped = alerts.filter((a) => a.causedBy).length;
|
||||||
return {
|
return {
|
||||||
alerts,
|
alerts,
|
||||||
counts: {
|
counts: {
|
||||||
critical: active.filter((a) => a.severity === "critical").length,
|
critical: active.filter((a) => a.severity === "critical").length,
|
||||||
warning: active.filter((a) => a.severity === "warning").length,
|
warning: active.filter((a) => a.severity === "warning").length,
|
||||||
info: active.filter((a) => a.severity === "info").length,
|
info: active.filter((a) => a.severity === "info").length,
|
||||||
silenced: alerts.length - active.length,
|
silenced: alerts.length - active.length - grouped,
|
||||||
|
grouped,
|
||||||
},
|
},
|
||||||
couldntCheck: [...blind.entries()].map(([name, v]) => ({ name, error: v.error, affects: [...v.affects] })),
|
couldntCheck: [...blind.entries()].map(([name, v]) => ({ name, error: v.error, affects: [...v.affects] })),
|
||||||
notes,
|
notes,
|
||||||
|
|||||||
@@ -8,10 +8,11 @@ import { notifyHealthIssues, notifyHealthRecovered } from "./notify.js";
|
|||||||
import { recordAudit } from "./audit.js";
|
import { recordAudit } from "./audit.js";
|
||||||
import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.js";
|
import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.js";
|
||||||
import { activeSubjects, subjectOfConditionKey } from "./maintenance.js";
|
import { activeSubjects, subjectOfConditionKey } from "./maintenance.js";
|
||||||
|
import { coveredByParent, loadParentRows, resolveParents } from "./serverParents.js";
|
||||||
|
|
||||||
const STATE_FLAG = "healthActiveConditions";
|
const STATE_FLAG = "healthActiveConditions";
|
||||||
/** After a restart the agents haven't had a chance to report yet (they run every 15 min), so server-derived conditions are held rather than judged. */
|
/** After a restart the agents haven't had a chance to report yet (they run every 15 min), so server-derived conditions are held rather than judged. */
|
||||||
const STARTUP_GRACE_MS = 20 * 60 * 1000;
|
const STARTUP_GRACE_MS = Number(process.env.HEALTH_STARTUP_GRACE_MS ?? 20 * 60 * 1000); // the environment variable is for tests
|
||||||
const PROCESS_START = Date.now();
|
const PROCESS_START = Date.now();
|
||||||
|
|
||||||
// ─── Types ──────────────────────────────────────────────────────────────────
|
// ─── Types ──────────────────────────────────────────────────────────────────
|
||||||
@@ -212,7 +213,8 @@ export function evaluateHealth(snapshot: HealthSnapshot, thresholds: Thresholds,
|
|||||||
|
|
||||||
// ─── State diff (pure) ──────────────────────────────────────────────────────
|
// ─── State diff (pure) ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
export type ActiveState = Record<string, { message: string; source: string }>;
|
/** `covered`: the problem is explained by another that is being reported (a guest down because its host is) — recorded, but not announced on its own. */
|
||||||
|
export type ActiveState = Record<string, { message: string; source: string; covered?: boolean }>;
|
||||||
|
|
||||||
/** `held` is a set of hierarchical sources that couldn't be read this run; matches the source itself or anything beneath it. */
|
/** `held` is a set of hierarchical sources that couldn't be read this run; matches the source itself or anything beneath it. */
|
||||||
function isHeld(source: string, held: Set<string>): boolean {
|
function isHeld(source: string, held: Set<string>): boolean {
|
||||||
@@ -226,6 +228,10 @@ function isHeld(source: string, held: Set<string>): boolean {
|
|||||||
* now is resolved — except when its source couldn't be read this run, in
|
* now is resolved — except when its source couldn't be read this run, in
|
||||||
* which case it's carried over untouched. Without that, one failed poll would
|
* which case it's carried over untouched. Without that, one failed poll would
|
||||||
* report a recovery and then re-alert the same problem on the next poll.
|
* report a recovery and then re-alert the same problem on the next poll.
|
||||||
|
*
|
||||||
|
* `covered` are keys of problems explained by another (a guest that is down because its host is). They are kept in the state and
|
||||||
|
* counted in `newlyDown` / `ended` (what happened, for the record), but are neither announced as new nor as resolved — until the
|
||||||
|
* explanation goes away while they are still there, when they are announced as new: an outage is never swallowed.
|
||||||
*/
|
*/
|
||||||
export function diffConditions(
|
export function diffConditions(
|
||||||
previous: ActiveState,
|
previous: ActiveState,
|
||||||
@@ -233,7 +239,16 @@ export function diffConditions(
|
|||||||
held: Set<string> = new Set(),
|
held: Set<string> = new Set(),
|
||||||
/** Subjects ("server:3", "integration:1") under an active maintenance window. */
|
/** Subjects ("server:3", "integration:1") under an active maintenance window. */
|
||||||
silenced: Set<string> = new Set(),
|
silenced: Set<string> = new Set(),
|
||||||
): { added: HealthCondition[]; resolved: { key: string; message: string }[]; next: ActiveState } {
|
covered: Set<string> = new Set(),
|
||||||
|
): {
|
||||||
|
added: HealthCondition[];
|
||||||
|
resolved: { key: string; message: string }[];
|
||||||
|
next: ActiveState;
|
||||||
|
/** Every problem that began, announced or not. */
|
||||||
|
newlyDown: HealthCondition[];
|
||||||
|
/** Every problem that ended, announced or not. */
|
||||||
|
ended: { key: string; message: string }[];
|
||||||
|
} {
|
||||||
const isSilenced = (key: string) => {
|
const isSilenced = (key: string) => {
|
||||||
const subject = subjectOfConditionKey(key);
|
const subject = subjectOfConditionKey(key);
|
||||||
return subject !== null && silenced.has(subject);
|
return subject !== null && silenced.has(subject);
|
||||||
@@ -243,20 +258,24 @@ export function diffConditions(
|
|||||||
const visible = current.filter((c) => !isSilenced(c.key));
|
const visible = current.filter((c) => !isSilenced(c.key));
|
||||||
|
|
||||||
const next: ActiveState = {};
|
const next: ActiveState = {};
|
||||||
for (const c of visible) next[c.key] = { message: c.message, source: c.source };
|
for (const c of visible) next[c.key] = covered.has(c.key) ? { message: c.message, source: c.source, covered: true } : { message: c.message, source: c.source };
|
||||||
|
|
||||||
const added = visible.filter((c) => !(c.key in previous));
|
const newlyDown = visible.filter((c) => !(c.key in previous));
|
||||||
|
// Announced: new and standing alone, or covered until now and no longer.
|
||||||
|
const added = visible.filter((c) => !covered.has(c.key) && (!(c.key in previous) || previous[c.key].covered));
|
||||||
const resolved: { key: string; message: string }[] = [];
|
const resolved: { key: string; message: string }[] = [];
|
||||||
|
const ended: { key: string; message: string }[] = [];
|
||||||
for (const [key, entry] of Object.entries(previous)) {
|
for (const [key, entry] of Object.entries(previous)) {
|
||||||
if (key in next) continue;
|
if (key in next) continue;
|
||||||
// Held (source unreadable) or silenced: leave it exactly as it was — neither cleared nor re-alerted.
|
// Held (source unreadable) or silenced: leave it exactly as it was — neither cleared nor re-alerted.
|
||||||
if (isHeld(entry.source, held) || isSilenced(key)) {
|
if (isHeld(entry.source, held) || isSilenced(key)) {
|
||||||
next[key] = entry;
|
next[key] = entry;
|
||||||
} else {
|
} else {
|
||||||
resolved.push({ key, message: entry.message });
|
ended.push({ key, message: entry.message });
|
||||||
|
if (!entry.covered) resolved.push({ key, message: entry.message });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return { added, resolved, next };
|
return { added, resolved, next, newlyDown, ended };
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── Collection (I/O) ───────────────────────────────────────────────────────
|
// ─── Collection (I/O) ───────────────────────────────────────────────────────
|
||||||
@@ -346,6 +365,14 @@ export async function recordServerTransitions(added: { key: string; message: str
|
|||||||
await write(resolved, { offline: "back_online", disk: "disk_ok" }, false);
|
await write(resolved, { offline: "back_online", disk: "disk_ok" }, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** "a, b and c", "a, b, c, d, e and 3 more". */
|
||||||
|
export function alsoList(names: string[]): string {
|
||||||
|
const shown = names.slice(0, 5);
|
||||||
|
const rest = names.length - shown.length;
|
||||||
|
const head = shown.length > 1 ? `${shown.slice(0, -1).join(", ")} and ${shown[shown.length - 1]}` : shown[0];
|
||||||
|
return rest > 0 ? `${shown.join(", ")} and ${rest} more` : head;
|
||||||
|
}
|
||||||
|
|
||||||
export async function runHealthCheck(now: number = Date.now()): Promise<{ added: number; resolved: number }> {
|
export async function runHealthCheck(now: number = Date.now()): Promise<{ added: number; resolved: number }> {
|
||||||
const { healthChecks } = await getSettings();
|
const { healthChecks } = await getSettings();
|
||||||
const { snapshot, held } = await collectSnapshot();
|
const { snapshot, held } = await collectSnapshot();
|
||||||
@@ -354,12 +381,30 @@ export async function runHealthCheck(now: number = Date.now()): Promise<{ added:
|
|||||||
if (inGrace) held.add("server");
|
if (inGrace) held.add("server");
|
||||||
|
|
||||||
const current = evaluateHealth(snapshot, healthChecks, now, { skipServers: inGrace });
|
const current = evaluateHealth(snapshot, healthChecks, now, { skipServers: inGrace });
|
||||||
const { added, resolved, next } = diffConditions(await loadState(), current, held, await activeSubjects(new Date(now)));
|
// A server that is down because the one it runs on is down is folded under that one: recorded, but not announced on its own.
|
||||||
|
const offlineIds = new Set<number>();
|
||||||
|
for (const c of current) {
|
||||||
|
const m = /^offline:server:(\d+)$/.exec(c.key);
|
||||||
|
if (m) offlineIds.add(Number(m[1]));
|
||||||
|
}
|
||||||
|
const cause = coveredByParent(offlineIds, resolveParents(await loadParentRows()));
|
||||||
|
const coveredKeys = new Set([...cause.keys()].map((id) => `offline:server:${id}`));
|
||||||
|
const { added: announced, resolved, next, newlyDown, ended } = diffConditions(await loadState(), current, held, await activeSubjects(new Date(now)), coveredKeys);
|
||||||
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
|
await setInternalFlag(STATE_FLAG, JSON.stringify(next));
|
||||||
|
|
||||||
|
// A host's message says what else it takes with it.
|
||||||
|
const names = new Map(snapshot.servers.map((s) => [s.id, s.name]));
|
||||||
|
const taken = new Map<number, string[]>();
|
||||||
|
for (const [child, parent] of cause) taken.set(parent, [...(taken.get(parent) ?? []), names.get(child) ?? `#${child}`]);
|
||||||
|
const added = announced.map((c) => {
|
||||||
|
const m = /^offline:server:(\d+)$/.exec(c.key);
|
||||||
|
const children = m ? taken.get(Number(m[1])) : undefined;
|
||||||
|
return children?.length ? { ...c, message: `${c.message} — also takes down ${alsoList(children)}` } : c;
|
||||||
|
});
|
||||||
|
|
||||||
// State is tracked even when the alert toggle is off (the notify functions check it),
|
// State is tracked even when the alert toggle is off (the notify functions check it),
|
||||||
// so turning alerts back on doesn't dump every long-standing condition at once.
|
// so turning alerts back on doesn't dump every long-standing condition at once.
|
||||||
await recordServerTransitions(added, resolved);
|
await recordServerTransitions(newlyDown, ended);
|
||||||
if (added.length > 0) await notifyHealthIssues(added);
|
if (added.length > 0) await notifyHealthIssues(added);
|
||||||
if (resolved.length > 0) await notifyHealthRecovered(resolved);
|
if (resolved.length > 0) await notifyHealthRecovered(resolved);
|
||||||
return { added: added.length, resolved: resolved.length };
|
return { added: added.length, resolved: resolved.length };
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
import { materializeSchedules } from "./maintenanceSchedules.js";
|
||||||
|
|
||||||
|
// Windows are made a week ahead, so a pass is only needed now and then: to make the next day's, and to pick up a server added to a
|
||||||
|
// schedule's list. Quarter-hourly keeps a restart from leaving a gap.
|
||||||
|
const INTERVAL_MS = 15 * 60 * 1000;
|
||||||
|
const FIRST_PASS_DELAY_MS = 30 * 1000;
|
||||||
|
|
||||||
|
let timer: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
||||||
|
async function pass(): Promise<void> {
|
||||||
|
try {
|
||||||
|
const made = await materializeSchedules();
|
||||||
|
if (made > 0) console.log(`[maintenanceSchedules] made ${made} maintenance window${made === 1 ? "" : "s"} from recurring schedules`);
|
||||||
|
} catch (err) {
|
||||||
|
console.error("[maintenanceSchedules]", err instanceof Error ? err.message : err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function initMaintenanceScheduleScheduler(): Promise<void> {
|
||||||
|
if (timer) return;
|
||||||
|
timer = setInterval(() => void pass(), INTERVAL_MS);
|
||||||
|
setTimeout(() => void pass(), FIRST_PASS_DELAY_MS);
|
||||||
|
console.log(`[maintenanceSchedules] making windows from recurring schedules a week ahead, checked every ${INTERVAL_MS / 60000} minutes`);
|
||||||
|
}
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
/**
|
||||||
|
* Recurring maintenance: "every Sunday 02:00–04:00 on these servers". A schedule only says when; the windows that silence alerts are
|
||||||
|
* ordinary maintenance windows made a week ahead (so they show on the timeline and can be cancelled one at a time, like any other).
|
||||||
|
* Each is marked `schedule:<id>:<date>`, which is how a later pass knows it has already made it — even if someone cancelled it.
|
||||||
|
*/
|
||||||
|
import { and, eq, gt, like } from "drizzle-orm";
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { maintenanceSchedules, maintenanceWindows, type MaintenanceTargetType } from "../db/schema.js";
|
||||||
|
import { invalidateAlertsCache } from "./alerts.js";
|
||||||
|
import { describeTarget } from "./maintenance.js";
|
||||||
|
|
||||||
|
export const HORIZON_DAYS = 7;
|
||||||
|
const DAY_MS = 24 * 3600_000;
|
||||||
|
|
||||||
|
export type ScheduleRow = typeof maintenanceSchedules.$inferSelect;
|
||||||
|
export interface ScheduleTarget {
|
||||||
|
type: MaintenanceTargetType;
|
||||||
|
id: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Occurrence {
|
||||||
|
start: Date;
|
||||||
|
end: Date;
|
||||||
|
/** The local calendar date it starts on, "2026-10-18" — unique per schedule. */
|
||||||
|
date: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Whether the system knows this time zone name. */
|
||||||
|
export function isValidTimeZone(tz: string): boolean {
|
||||||
|
try {
|
||||||
|
new Intl.DateTimeFormat("en-US", { timeZone: tz });
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const parts = (utcMs: number, tz: string) => {
|
||||||
|
const f = new Intl.DateTimeFormat("en-US", { timeZone: tz, hourCycle: "h23", year: "numeric", month: "2-digit", day: "2-digit", hour: "2-digit", minute: "2-digit", second: "2-digit" });
|
||||||
|
const out: Record<string, number> = {};
|
||||||
|
for (const p of f.formatToParts(new Date(utcMs))) if (p.type !== "literal") out[p.type] = Number(p.value);
|
||||||
|
return out;
|
||||||
|
};
|
||||||
|
|
||||||
|
/** How far the clock in `tz` is ahead of UTC at that moment, in ms. */
|
||||||
|
function offsetAt(utcMs: number, tz: string): number {
|
||||||
|
const p = parts(utcMs, tz);
|
||||||
|
return Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second) - Math.floor(utcMs / 1000) * 1000;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The instant a wall-clock time in `tz` happens. Winter/summer time is settled by asking the zone for its offset at the answer. */
|
||||||
|
export function zonedTime(year: number, month: number, day: number, hour: number, minute: number, tz: string): Date {
|
||||||
|
const naive = Date.UTC(year, month - 1, day, hour, minute);
|
||||||
|
let guess = naive - offsetAt(naive, tz);
|
||||||
|
guess = naive - offsetAt(guess, tz);
|
||||||
|
return new Date(guess);
|
||||||
|
}
|
||||||
|
|
||||||
|
const pad = (n: number) => String(n).padStart(2, "0");
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The times a schedule happens that touch [from, to]: those that have not ended by `from` and begin before `to`. A schedule that runs
|
||||||
|
* past midnight belongs to the day it starts on.
|
||||||
|
*/
|
||||||
|
export function occurrencesBetween(schedule: Pick<ScheduleRow, "days" | "startTime" | "durationMinutes" | "timezone">, from: Date, to: Date): Occurrence[] {
|
||||||
|
const days = new Set<number>(JSON.parse(schedule.days) as number[]);
|
||||||
|
const [hh, mm] = schedule.startTime.split(":").map(Number);
|
||||||
|
const out: Occurrence[] = [];
|
||||||
|
// Walk the calendar dates around the range (a day either side covers a schedule that started yesterday, or zone differences).
|
||||||
|
const first = parts(from.getTime() - DAY_MS, schedule.timezone);
|
||||||
|
let cursor = Date.UTC(first.year, first.month - 1, first.day);
|
||||||
|
const last = to.getTime() + DAY_MS;
|
||||||
|
for (; cursor <= last; cursor += DAY_MS) {
|
||||||
|
const d = new Date(cursor);
|
||||||
|
if (!days.has(d.getUTCDay())) continue;
|
||||||
|
const y = d.getUTCFullYear();
|
||||||
|
const m = d.getUTCMonth() + 1;
|
||||||
|
const day = d.getUTCDate();
|
||||||
|
const start = zonedTime(y, m, day, hh, mm, schedule.timezone);
|
||||||
|
const end = new Date(start.getTime() + schedule.durationMinutes * 60_000);
|
||||||
|
if (end > from && start < to) out.push({ start, end, date: `${y}-${pad(m)}-${pad(day)}` });
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function targetsOf(schedule: Pick<ScheduleRow, "targets">): ScheduleTarget[] {
|
||||||
|
try {
|
||||||
|
const v = JSON.parse(schedule.targets);
|
||||||
|
return Array.isArray(v) ? v.filter((t): t is ScheduleTarget => !!t && typeof t.id === "number" && typeof t.type === "string") : [];
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export const sourceOf = (scheduleId: number, date: string) => `schedule:${scheduleId}:${date}`;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Makes the windows for the coming week that don't exist yet. Safe to run as often as you like: a window already made (including one
|
||||||
|
* that was cancelled) is left as it is. Returns how many it made.
|
||||||
|
*/
|
||||||
|
export async function materializeSchedules(now: Date = new Date(), horizonDays: number = HORIZON_DAYS, onlyScheduleId?: number): Promise<number> {
|
||||||
|
const schedules = (await db.select().from(maintenanceSchedules)).filter((s) => s.enabled && (onlyScheduleId === undefined || s.id === onlyScheduleId));
|
||||||
|
const horizon = new Date(now.getTime() + horizonDays * DAY_MS);
|
||||||
|
let made = 0;
|
||||||
|
for (const schedule of schedules) {
|
||||||
|
const occurrences = occurrencesBetween(schedule, now, horizon);
|
||||||
|
if (occurrences.length === 0) continue;
|
||||||
|
const existing = await db.select({ source: maintenanceWindows.source, targetType: maintenanceWindows.targetType, targetId: maintenanceWindows.targetId }).from(maintenanceWindows).where(like(maintenanceWindows.source, `schedule:${schedule.id}:%`));
|
||||||
|
const have = new Set(existing.map((w) => `${w.source}|${w.targetType}:${w.targetId}`));
|
||||||
|
for (const target of targetsOf(schedule)) {
|
||||||
|
if (!(await describeTarget(target.type, target.id))) continue; // since removed: nothing left to silence
|
||||||
|
for (const occ of occurrences) {
|
||||||
|
const source = sourceOf(schedule.id, occ.date);
|
||||||
|
if (have.has(`${source}|${target.type}:${target.id}`)) continue;
|
||||||
|
await db.insert(maintenanceWindows).values({
|
||||||
|
targetType: target.type,
|
||||||
|
targetId: target.id,
|
||||||
|
reason: schedule.reason ? `${schedule.name}: ${schedule.reason}` : schedule.name,
|
||||||
|
startedAt: occ.start.toISOString(),
|
||||||
|
endsAt: occ.end.toISOString(),
|
||||||
|
createdBy: `Recurring: ${schedule.name}`,
|
||||||
|
source,
|
||||||
|
});
|
||||||
|
made++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (made > 0) invalidateAlertsCache();
|
||||||
|
return made;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Takes back the windows a schedule made that haven't begun — when it is switched off or removed. Ones under way end on their own. */
|
||||||
|
export async function removeFutureWindows(scheduleId: number, now: Date = new Date()): Promise<number> {
|
||||||
|
const gone = await db
|
||||||
|
.delete(maintenanceWindows)
|
||||||
|
.where(and(like(maintenanceWindows.source, `schedule:${scheduleId}:%`), gt(maintenanceWindows.startedAt, now.toISOString())))
|
||||||
|
.returning({ id: maintenanceWindows.id });
|
||||||
|
if (gone.length > 0) invalidateAlertsCache();
|
||||||
|
return gone.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function deleteSchedule(scheduleId: number): Promise<void> {
|
||||||
|
await removeFutureWindows(scheduleId);
|
||||||
|
await db.delete(maintenanceSchedules).where(eq(maintenanceSchedules.id, scheduleId));
|
||||||
|
}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
/**
|
||||||
|
* What each server says is waiting to be installed: how many updates, how many of them security fixes, whether a restart is due — and
|
||||||
|
* how long updates have been waiting, which the agent can't know and so is worked out here from one report to the next.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** What an agent reports about updates, as it arrives. */
|
||||||
|
export interface IncomingUpdates {
|
||||||
|
/** apt, dnf, zypper, windows-update … — what it asked. */
|
||||||
|
manager?: string;
|
||||||
|
count: number;
|
||||||
|
/** Null when this package manager can't say which are security fixes. */
|
||||||
|
security?: number | null;
|
||||||
|
reboot_required?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PendingUpdates {
|
||||||
|
manager: string;
|
||||||
|
count: number;
|
||||||
|
security: number | null;
|
||||||
|
rebootRequired: boolean;
|
||||||
|
/** ISO — when the agent last told us. */
|
||||||
|
reportedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parsePendingUpdates(stored: string | null): PendingUpdates | null {
|
||||||
|
if (!stored) return null;
|
||||||
|
try {
|
||||||
|
const v = JSON.parse(stored);
|
||||||
|
if (!v || typeof v !== "object" || typeof v.count !== "number") return null;
|
||||||
|
return {
|
||||||
|
manager: typeof v.manager === "string" ? v.manager : "",
|
||||||
|
count: v.count,
|
||||||
|
security: typeof v.security === "number" ? v.security : null,
|
||||||
|
rebootRequired: v.rebootRequired === true,
|
||||||
|
reportedAt: typeof v.reportedAt === "string" ? v.reportedAt : "",
|
||||||
|
};
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The columns to store after a report. Updates waiting since "since" keep that moment for as long as some are still waiting; when
|
||||||
|
* none are, it is cleared, and the next time some appear it starts again from then.
|
||||||
|
*/
|
||||||
|
export function nextUpdateState(
|
||||||
|
previous: { pendingUpdates: string | null; updatesPendingSince: string | null },
|
||||||
|
incoming: IncomingUpdates,
|
||||||
|
now: string,
|
||||||
|
): { pendingUpdates: string; updatesPendingSince: string | null } {
|
||||||
|
const before = parsePendingUpdates(previous.pendingUpdates);
|
||||||
|
const waiting = incoming.count > 0;
|
||||||
|
const since = waiting ? (before && before.count > 0 && previous.updatesPendingSince ? previous.updatesPendingSince : now) : null;
|
||||||
|
const record: PendingUpdates = {
|
||||||
|
manager: incoming.manager ?? "",
|
||||||
|
count: incoming.count,
|
||||||
|
security: typeof incoming.security === "number" ? incoming.security : null,
|
||||||
|
rebootRequired: incoming.reboot_required === true,
|
||||||
|
reportedAt: now,
|
||||||
|
};
|
||||||
|
return { pendingUpdates: JSON.stringify(record), updatesPendingSince: since };
|
||||||
|
}
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
/**
|
||||||
|
* Which server runs on which: a virtual machine on its Proxmox host, a container on a Docker box. When the host stops reporting, the
|
||||||
|
* things on it stop too, and seven alerts about seven guests only hide the one that matters — so those alerts are folded under the
|
||||||
|
* host's. The link is either set by hand ("runs on") or worked out: a server linked to a Proxmox guest runs on the server named after
|
||||||
|
* that guest's Proxmox node, when there is exactly one such server.
|
||||||
|
*/
|
||||||
|
import { db } from "../db/client.js";
|
||||||
|
import { servers } from "../db/schema.js";
|
||||||
|
|
||||||
|
export interface ParentRow {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
hostname: string | null;
|
||||||
|
parentServerId: number | null;
|
||||||
|
proxmoxNode: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParentLink {
|
||||||
|
parentId: number;
|
||||||
|
/** Worked out from the Proxmox node rather than set by hand. */
|
||||||
|
auto: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
const lc = (s: string | null | undefined) => (s ?? "").trim().toLowerCase();
|
||||||
|
|
||||||
|
/** Whether a server is the machine a Proxmox node name refers to: by its name, its hostname, or the first part of a fully qualified hostname. */
|
||||||
|
function isNode(row: ParentRow, node: string): boolean {
|
||||||
|
const n = lc(node);
|
||||||
|
if (!n) return false;
|
||||||
|
const host = lc(row.hostname);
|
||||||
|
return lc(row.name) === n || host === n || host.split(".")[0] === n;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Each server's parent, where it has one. A link set by hand wins; otherwise the unique server that is its Proxmox node. */
|
||||||
|
export function resolveParents(rows: ParentRow[]): Map<number, ParentLink> {
|
||||||
|
const byId = new Map(rows.map((r) => [r.id, r]));
|
||||||
|
const out = new Map<number, ParentLink>();
|
||||||
|
for (const row of rows) {
|
||||||
|
if (row.parentServerId !== null && row.parentServerId !== row.id && byId.has(row.parentServerId)) {
|
||||||
|
out.set(row.id, { parentId: row.parentServerId, auto: false });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (row.proxmoxNode) {
|
||||||
|
const candidates = rows.filter((r) => r.id !== row.id && isNode(r, row.proxmoxNode!));
|
||||||
|
if (candidates.length === 1) out.set(row.id, { parentId: candidates[0].id, auto: true });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Of the servers that are down, those whose cause is another server that is down too: each maps to the topmost ancestor that is down —
|
||||||
|
* the root cause. A chain (guest → VM host → the box it hangs off) is followed all the way up; a loop in the links covers nothing.
|
||||||
|
*/
|
||||||
|
export function coveredByParent(offline: Set<number>, parents: Map<number, ParentLink>): Map<number, number> {
|
||||||
|
const covered = new Map<number, number>();
|
||||||
|
for (const id of offline) {
|
||||||
|
const seen = new Set<number>([id]);
|
||||||
|
let root: number | null = null;
|
||||||
|
let looped = false;
|
||||||
|
let link = parents.get(id);
|
||||||
|
while (link && !seen.has(link.parentId)) {
|
||||||
|
if (offline.has(link.parentId)) root = link.parentId;
|
||||||
|
seen.add(link.parentId);
|
||||||
|
link = parents.get(link.parentId);
|
||||||
|
}
|
||||||
|
if (link && seen.has(link.parentId)) looped = true;
|
||||||
|
// A loop explains nothing (each would be waiting for the other), so its members stand alone and are announced.
|
||||||
|
if (root !== null && !looped) covered.set(id, root);
|
||||||
|
}
|
||||||
|
return covered;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Would making `parentId` the parent of `id` close a loop? True if `id` is `parentId` or already sits above it. */
|
||||||
|
export function wouldLoop(rows: ParentRow[], id: number, parentId: number): boolean {
|
||||||
|
const parents = resolveParents(rows);
|
||||||
|
const seen = new Set<number>();
|
||||||
|
let cur: number | undefined = parentId;
|
||||||
|
while (cur !== undefined && !seen.has(cur)) {
|
||||||
|
if (cur === id) return true;
|
||||||
|
seen.add(cur);
|
||||||
|
cur = parents.get(cur)?.parentId;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function loadParentRows(): Promise<ParentRow[]> {
|
||||||
|
return db
|
||||||
|
.select({ id: servers.id, name: servers.name, hostname: servers.hostname, parentServerId: servers.parentServerId, proxmoxNode: servers.proxmoxNode })
|
||||||
|
.from(servers);
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ import { and, eq, notInArray } from "drizzle-orm";
|
|||||||
import { db } from "../db/client.js";
|
import { db } from "../db/client.js";
|
||||||
import { scheduledTasks, servers } from "../db/schema.js";
|
import { scheduledTasks, servers } from "../db/schema.js";
|
||||||
import { recordServerMetrics } from "./serverMetrics.js";
|
import { recordServerMetrics } from "./serverMetrics.js";
|
||||||
|
import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js";
|
||||||
|
|
||||||
export interface IncomingTask {
|
export interface IncomingTask {
|
||||||
scheduleType: "cron" | "systemd_timer" | "windows_task";
|
scheduleType: "cron" | "systemd_timer" | "windows_task";
|
||||||
@@ -20,6 +21,8 @@ export interface IncomingSystemInfo {
|
|||||||
memory?: { total_bytes?: number; used_bytes?: number };
|
memory?: { total_bytes?: number; used_bytes?: number };
|
||||||
disks?: { mount: string; size_bytes: number; used_bytes: number }[];
|
disks?: { mount: string; size_bytes: number; used_bytes: number }[];
|
||||||
listening_ports?: { protocol: "tcp" | "udp"; port: number; address: string; process?: string }[];
|
listening_ports?: { protocol: "tcp" | "udp"; port: number; address: string; process?: string }[];
|
||||||
|
/** Updates waiting to be installed. Absent from an agent that doesn't report them (or couldn't find out this time). */
|
||||||
|
updates?: IncomingUpdates;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface AgentReport {
|
export interface AgentReport {
|
||||||
@@ -98,10 +101,13 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const system = report.system;
|
const system = report.system;
|
||||||
|
const [before] = await db.select({ pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince }).from(servers).where(eq(servers.id, serverId));
|
||||||
await db
|
await db
|
||||||
.update(servers)
|
.update(servers)
|
||||||
.set({
|
.set({
|
||||||
lastSeenAt: now,
|
lastSeenAt: now,
|
||||||
|
// Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before.
|
||||||
|
...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}),
|
||||||
...(report.hostname ? { hostname: report.hostname } : {}),
|
...(report.hostname ? { hostname: report.hostname } : {}),
|
||||||
// An agent knows what it runs on better than whoever registered the server did.
|
// An agent knows what it runs on better than whoever registered the server did.
|
||||||
...(report.osType === "linux" || report.osType === "windows" ? { osType: report.osType } : {}),
|
...(report.osType === "linux" || report.osType === "windows" ? { osType: report.osType } : {}),
|
||||||
|
|||||||
@@ -0,0 +1,202 @@
|
|||||||
|
/**
|
||||||
|
* Recurring maintenance: when a schedule happens (across summer and winter time), and the windows made from it.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import type { Server } from "node:http";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-sched-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers, maintenanceWindows, maintenanceSchedules, auditLog } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { maintenanceRouter } = await import("../src/routes/maintenance.js");
|
||||||
|
const { occurrencesBetween, zonedTime, materializeSchedules, isValidTimeZone } = await import("../src/services/maintenanceSchedules.js");
|
||||||
|
const { listActiveWindows } = await import("../src/services/maintenance.js");
|
||||||
|
const { eq, like } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
const sched = (days: number[], startTime: string, durationMinutes: number, timezone: string) => ({ days: JSON.stringify(days), startTime, durationMinutes, timezone });
|
||||||
|
const iso = (d: Date) => d.toISOString();
|
||||||
|
|
||||||
|
describe("when a schedule happens", () => {
|
||||||
|
test("a wall-clock time follows summer and winter time", () => {
|
||||||
|
// Sundays in Stockholm: 18 October is still summer time (UTC+2), 1 November is winter time (UTC+1).
|
||||||
|
const s = sched([0], "02:00", 120, "Europe/Stockholm");
|
||||||
|
const occ = occurrencesBetween(s, new Date("2026-10-15T00:00:00Z"), new Date("2026-11-03T00:00:00Z"));
|
||||||
|
const byDate = Object.fromEntries(occ.map((o) => [o.date, [iso(o.start), iso(o.end)]]));
|
||||||
|
assert.deepEqual(byDate["2026-10-18"], ["2026-10-18T00:00:00.000Z", "2026-10-18T02:00:00.000Z"]);
|
||||||
|
assert.deepEqual(byDate["2026-11-01"], ["2026-11-01T01:00:00.000Z", "2026-11-01T03:00:00.000Z"]);
|
||||||
|
assert.deepEqual(Object.keys(byDate).sort(), ["2026-10-18", "2026-10-25", "2026-11-01"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("it is the same wall-clock time in another zone, and a time that doesn't exist that night still happens", () => {
|
||||||
|
assert.equal(iso(zonedTime(2026, 7, 1, 2, 0, "America/New_York")), "2026-07-01T06:00:00.000Z");
|
||||||
|
assert.equal(iso(zonedTime(2026, 1, 1, 2, 0, "America/New_York")), "2026-01-01T07:00:00.000Z");
|
||||||
|
// 29 March 2026: clocks in Stockholm jump from 02:00 to 03:00, so 02:30 isn't a time that night.
|
||||||
|
const gap = zonedTime(2026, 3, 29, 2, 30, "Europe/Stockholm");
|
||||||
|
assert.ok(Math.abs(gap.getTime() - Date.parse("2026-03-29T01:00:00Z")) <= 3600_000);
|
||||||
|
assert.equal(zonedTime(2026, 10, 18, 0, 0, "UTC").toISOString(), "2026-10-18T00:00:00.000Z");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("one that runs past midnight belongs to the day it starts, and one under way counts", () => {
|
||||||
|
const s = sched([6], "23:00", 180, "UTC"); // Saturday 23:00 for three hours
|
||||||
|
const [occ] = occurrencesBetween(s, new Date("2026-10-18T00:30:00Z"), new Date("2026-10-18T06:00:00Z"));
|
||||||
|
assert.equal(occ.date, "2026-10-17");
|
||||||
|
assert.equal(iso(occ.start), "2026-10-17T23:00:00.000Z");
|
||||||
|
assert.equal(iso(occ.end), "2026-10-18T02:00:00.000Z");
|
||||||
|
assert.equal(occurrencesBetween(s, new Date("2026-10-18T02:00:00Z"), new Date("2026-10-18T06:00:00Z")).length, 0, "already over");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("every day, and only the days chosen", () => {
|
||||||
|
const week = occurrencesBetween(sched([0, 1, 2, 3, 4, 5, 6], "04:00", 60, "UTC"), new Date("2026-10-12T00:00:00Z"), new Date("2026-10-19T00:00:00Z"));
|
||||||
|
assert.equal(week.length, 7);
|
||||||
|
const weekdays = occurrencesBetween(sched([1, 3], "04:00", 60, "UTC"), new Date("2026-10-12T00:00:00Z"), new Date("2026-10-19T00:00:00Z"));
|
||||||
|
assert.deepEqual(weekdays.map((o) => o.date), ["2026-10-12", "2026-10-14"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("time zone names are checked", () => {
|
||||||
|
assert.equal(isValidTimeZone("Europe/Stockholm"), true);
|
||||||
|
assert.equal(isValidTimeZone("Mars/Olympus"), false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the schedules and their windows", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "operator";
|
||||||
|
let a = 0;
|
||||||
|
let b = 0;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "operator", name: "Olga", role: "operator" },
|
||||||
|
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
|
||||||
|
]);
|
||||||
|
const made = await db
|
||||||
|
.insert(servers)
|
||||||
|
.values([
|
||||||
|
{ name: "elsa", apiTokenHash: "a", apiTokenPrefix: "a" },
|
||||||
|
{ name: "anna", apiTokenHash: "b", apiTokenPrefix: "b" },
|
||||||
|
])
|
||||||
|
.returning({ id: servers.id });
|
||||||
|
a = made[0].id;
|
||||||
|
b = made[1].id;
|
||||||
|
|
||||||
|
const app0 = express();
|
||||||
|
app0.use(express.json());
|
||||||
|
app0.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
app0.use("/api/maintenance", maintenanceRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = app0.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/maintenance`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const call = (method: string, path: string, body?: unknown) =>
|
||||||
|
fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
const good = { name: "Patch night", reason: "OS updates", targets: [{ type: "server", id: 0 }], days: [0, 1, 2, 3, 4, 5, 6], startTime: "03:00", durationMinutes: 60, timezone: "UTC" };
|
||||||
|
const windowsOf = async (id: number) => db.select().from(maintenanceWindows).where(like(maintenanceWindows.source, `schedule:${id}:%`));
|
||||||
|
|
||||||
|
let scheduleId = 0;
|
||||||
|
|
||||||
|
test("making a schedule makes the coming week's windows, once each, for every server on it", async () => {
|
||||||
|
const res = await call("POST", "/schedules", { ...good, targets: [{ type: "server", id: a }, { type: "server", id: b }, { type: "server", id: a }] });
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
const body = await res.json();
|
||||||
|
scheduleId = body.id;
|
||||||
|
const windows = await windowsOf(scheduleId);
|
||||||
|
assert.ok(body.windowsMade === windows.length && (windows.length === 14 || windows.length === 16), `made ${windows.length}`);
|
||||||
|
assert.equal(new Set(windows.map((w) => `${w.targetId}|${w.source}`)).size, windows.length, "no duplicates");
|
||||||
|
assert.ok(windows.every((w) => w.reason === "Patch night: OS updates" && w.createdBy === "Recurring: Patch night"));
|
||||||
|
assert.ok(windows.every((w) => w.endsAt > w.startedAt));
|
||||||
|
|
||||||
|
// running again makes nothing more
|
||||||
|
assert.equal(await materializeSchedules(), 0);
|
||||||
|
assert.equal((await windowsOf(scheduleId)).length, windows.length);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a window cancelled by hand isn't made again, and the list says what is next", async () => {
|
||||||
|
const before = await windowsOf(scheduleId);
|
||||||
|
const one = before.find((w) => w.startedAt > new Date().toISOString())!;
|
||||||
|
await db.update(maintenanceWindows).set({ endsAt: one.startedAt }).where(eq(maintenanceWindows.id, one.id));
|
||||||
|
assert.equal(await materializeSchedules(), 0);
|
||||||
|
assert.equal((await windowsOf(scheduleId)).length, before.length);
|
||||||
|
|
||||||
|
const list = await (await call("GET", "/schedules")).json();
|
||||||
|
assert.equal(list.schedules.length, 1);
|
||||||
|
assert.equal(list.schedules[0].targets.length, 2);
|
||||||
|
assert.deepEqual(list.schedules[0].days, [0, 1, 2, 3, 4, 5, 6]);
|
||||||
|
assert.ok(Date.parse(list.schedules[0].nextStart) > 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a window made from a schedule silences while it runs", async () => {
|
||||||
|
const now = Date.now();
|
||||||
|
await db.insert(maintenanceWindows).values({ targetType: "server", targetId: a, reason: "x", startedAt: new Date(now - 60_000).toISOString(), endsAt: new Date(now + 600_000).toISOString(), source: `schedule:${scheduleId}:2000-01-01` });
|
||||||
|
assert.ok((await listActiveWindows()).some((w) => w.source === `schedule:${scheduleId}:2000-01-01`));
|
||||||
|
await db.delete(maintenanceWindows).where(eq(maintenanceWindows.source, `schedule:${scheduleId}:2000-01-01`));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("switching it off takes back what hasn't begun, and on makes it again", async () => {
|
||||||
|
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: false })).status, 200);
|
||||||
|
assert.equal((await windowsOf(scheduleId)).filter((w) => w.startedAt > new Date().toISOString()).length, 0);
|
||||||
|
assert.equal(await materializeSchedules(), 0, "off means nothing is made");
|
||||||
|
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: true })).status, 200);
|
||||||
|
assert.ok((await windowsOf(scheduleId)).length >= 12);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("only operators change schedules; a bad one is refused", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
assert.equal((await call("POST", "/schedules", { ...good, targets: [{ type: "server", id: a }] })).status, 403);
|
||||||
|
assert.equal((await call("PATCH", `/schedules/${scheduleId}`, { enabled: false })).status, 403);
|
||||||
|
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 403);
|
||||||
|
assert.equal((await call("GET", "/schedules")).status, 200, "anyone signed in can see them");
|
||||||
|
who = "operator";
|
||||||
|
const target = [{ type: "server", id: a }];
|
||||||
|
for (const bad of [
|
||||||
|
{ ...good, targets: target, name: "" },
|
||||||
|
{ ...good, targets: [] },
|
||||||
|
{ ...good, targets: target, days: [] },
|
||||||
|
{ ...good, targets: target, days: [7] },
|
||||||
|
{ ...good, targets: target, startTime: "25:00" },
|
||||||
|
{ ...good, targets: target, startTime: "3am" },
|
||||||
|
{ ...good, targets: target, durationMinutes: 5 },
|
||||||
|
{ ...good, targets: target, timezone: "Mars/Olympus" },
|
||||||
|
{ ...good, targets: [{ type: "server", id: 99999 }] },
|
||||||
|
]) {
|
||||||
|
assert.equal((await call("POST", "/schedules", bad)).status, 400, JSON.stringify(bad).slice(0, 80));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("removing a schedule takes back what hasn't begun and is in the audit log", async () => {
|
||||||
|
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 204);
|
||||||
|
assert.equal((await db.select().from(maintenanceSchedules)).length, 0);
|
||||||
|
assert.equal((await windowsOf(scheduleId)).filter((w) => w.startedAt > new Date().toISOString()).length, 0);
|
||||||
|
assert.equal((await call("DELETE", `/schedules/${scheduleId}`)).status, 404);
|
||||||
|
const actions = (await db.select().from(auditLog)).filter((e) => e.category === "maintenance").map((e) => e.action);
|
||||||
|
for (const wanted of ["create_schedule", "disable_schedule", "enable_schedule", "delete_schedule"]) assert.ok(actions.includes(wanted), wanted);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
/**
|
||||||
|
* Folding a guest's "offline" alert under its host's, and what servers say is waiting to be installed.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
import type { Server } from "node:http";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-parents-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
// No waiting after a restart before servers are judged, or nothing here could be "offline".
|
||||||
|
process.env.HEALTH_STARTUP_GRACE_MS = "0";
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, servers } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js");
|
||||||
|
const { diffConditions, alsoList } = await import("../src/services/healthMonitor.js");
|
||||||
|
const { resolveParents, coveredByParent, wouldLoop } = await import("../src/services/serverParents.js");
|
||||||
|
const { nextUpdateState, parsePendingUpdates } = await import("../src/services/pendingUpdates.js");
|
||||||
|
const { serversRouter } = await import("../src/routes/servers.js");
|
||||||
|
const { agentReportRouter } = await import("../src/routes/agentReport.js");
|
||||||
|
const { hashToken } = await import("../src/services/tokens.js");
|
||||||
|
const { eq } = await import("drizzle-orm");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
const row = (id: number, name: string, extra: Partial<{ hostname: string | null; parentServerId: number | null; proxmoxNode: string | null }> = {}) => ({
|
||||||
|
id,
|
||||||
|
name,
|
||||||
|
hostname: null,
|
||||||
|
parentServerId: null,
|
||||||
|
proxmoxNode: null,
|
||||||
|
...extra,
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("resolveParents", () => {
|
||||||
|
test("a link set by hand wins, and a Proxmox node that is exactly one server is found", () => {
|
||||||
|
const rows = [row(1, "pve1", { hostname: "pve1.lab.example" }), row(2, "web", { proxmoxNode: "PVE1" }), row(3, "db", { parentServerId: 2, proxmoxNode: "pve1" }), row(4, "lone")];
|
||||||
|
const parents = resolveParents(rows);
|
||||||
|
assert.deepEqual(parents.get(2), { parentId: 1, auto: true });
|
||||||
|
assert.deepEqual(parents.get(3), { parentId: 2, auto: false }, "by hand beats the node");
|
||||||
|
assert.equal(parents.get(4), undefined);
|
||||||
|
assert.equal(parents.get(1), undefined);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("two servers that could be the node, or none, give no parent; a dead link is ignored", () => {
|
||||||
|
const rows = [row(1, "pve1"), row(2, "pve1", { hostname: "other" }), row(3, "web", { proxmoxNode: "pve1" }), row(4, "x", { proxmoxNode: "nothing" }), row(5, "y", { parentServerId: 99 }), row(6, "z", { parentServerId: 6 })];
|
||||||
|
const parents = resolveParents(rows);
|
||||||
|
assert.equal(parents.get(3), undefined, "ambiguous");
|
||||||
|
assert.equal(parents.get(4), undefined);
|
||||||
|
assert.equal(parents.get(5), undefined);
|
||||||
|
assert.equal(parents.get(6), undefined, "not its own parent");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a server is not its own node", () => {
|
||||||
|
assert.equal(resolveParents([row(1, "pve1", { proxmoxNode: "pve1" })]).size, 0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("coveredByParent", () => {
|
||||||
|
const parents = resolveParents([row(1, "host"), row(2, "vm", { parentServerId: 1 }), row(3, "ct", { parentServerId: 2 }), row(4, "other", { parentServerId: 1 })]);
|
||||||
|
|
||||||
|
test("a guest is covered only while something above it is down, and by the topmost one", () => {
|
||||||
|
assert.deepEqual([...coveredByParent(new Set([1, 2, 3]), parents)].sort(), [[2, 1], [3, 1]]);
|
||||||
|
assert.deepEqual([...coveredByParent(new Set([2, 3]), parents)], [[3, 2]], "the host is up: the VM is the cause");
|
||||||
|
assert.deepEqual([...coveredByParent(new Set([3]), parents)], [], "nothing above it is down");
|
||||||
|
assert.deepEqual([...coveredByParent(new Set([1]), parents)], []);
|
||||||
|
assert.deepEqual([...coveredByParent(new Set([1, 3]), parents)], [[3, 1]], "an up VM in between doesn't hide the cause");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a loop covers nothing, so nothing is swallowed", () => {
|
||||||
|
const loop = resolveParents([row(1, "a", { parentServerId: 2 }), row(2, "b", { parentServerId: 1 }), row(3, "c", { parentServerId: 1 })]);
|
||||||
|
assert.equal(coveredByParent(new Set([1, 2, 3]), loop).size, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("wouldLoop spots a link that would close a loop", () => {
|
||||||
|
const rows = [row(1, "host"), row(2, "vm", { parentServerId: 1 })];
|
||||||
|
assert.equal(wouldLoop(rows, 1, 2), true);
|
||||||
|
assert.equal(wouldLoop(rows, 1, 1), true);
|
||||||
|
assert.equal(wouldLoop(rows, 2, 1), false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("diffConditions with covered problems", () => {
|
||||||
|
const c = (key: string) => ({ key, source: "server", message: key });
|
||||||
|
const names = (list: { key: string }[]) => list.map((x) => x.key).sort();
|
||||||
|
|
||||||
|
test("a problem explained by another is recorded but not announced, and neither is its end", () => {
|
||||||
|
const first = diffConditions({}, [c("offline:server:1"), c("offline:server:2")], new Set(), new Set(), new Set(["offline:server:2"]));
|
||||||
|
assert.deepEqual(names(first.added), ["offline:server:1"]);
|
||||||
|
assert.deepEqual(names(first.newlyDown), ["offline:server:1", "offline:server:2"], "both are down, for the record");
|
||||||
|
assert.equal(first.next["offline:server:2"].covered, true);
|
||||||
|
|
||||||
|
const later = diffConditions(first.next, [c("offline:server:1")], new Set(), new Set(), new Set());
|
||||||
|
assert.deepEqual(names(later.resolved), [], "the guest's end isn't announced either");
|
||||||
|
assert.deepEqual(names(later.ended), ["offline:server:2"], "but it did end");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("when the explanation goes away and the problem is still there, it is announced as new", () => {
|
||||||
|
const first = diffConditions({}, [c("offline:server:1"), c("offline:server:2")], new Set(), new Set(), new Set(["offline:server:2"]));
|
||||||
|
const hostBack = diffConditions(first.next, [c("offline:server:2")], new Set(), new Set(), new Set());
|
||||||
|
assert.deepEqual(names(hostBack.added), ["offline:server:2"]);
|
||||||
|
assert.deepEqual(names(hostBack.resolved), ["offline:server:1"]);
|
||||||
|
assert.deepEqual(names(hostBack.newlyDown), [], "it was already down; nothing began");
|
||||||
|
// and only once
|
||||||
|
const again = diffConditions(hostBack.next, [c("offline:server:2")], new Set(), new Set(), new Set());
|
||||||
|
assert.deepEqual(again.added, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("with nothing covered it works as before", () => {
|
||||||
|
const run = diffConditions({}, [c("a"), c("b")]);
|
||||||
|
assert.deepEqual(names(run.added), ["a", "b"]);
|
||||||
|
assert.deepEqual(names(run.newlyDown), ["a", "b"]);
|
||||||
|
const gone = diffConditions(run.next, [c("a")]);
|
||||||
|
assert.deepEqual(names(gone.resolved), ["b"]);
|
||||||
|
assert.deepEqual(names(gone.ended), ["b"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a list of names reads as words", () => {
|
||||||
|
assert.equal(alsoList(["a"]), "a");
|
||||||
|
assert.equal(alsoList(["a", "b", "c"]), "a, b and c");
|
||||||
|
assert.equal(alsoList(["a", "b", "c", "d", "e", "f", "g"]), "a, b, c, d, e and 2 more");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the alerts, folded", () => {
|
||||||
|
const hoursAgo = (n: number) => new Date(Date.now() - n * 3600_000).toISOString();
|
||||||
|
let hostId = 0;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
const made = await db
|
||||||
|
.insert(servers)
|
||||||
|
.values([
|
||||||
|
{ name: "pve1", apiTokenHash: "h", apiTokenPrefix: "h", lastSeenAt: hoursAgo(2) },
|
||||||
|
{ name: "web", apiTokenHash: "w", apiTokenPrefix: "w", lastSeenAt: hoursAgo(2), proxmoxNode: "pve1" },
|
||||||
|
{ name: "db", apiTokenHash: "d", apiTokenPrefix: "d", lastSeenAt: hoursAgo(2), proxmoxNode: "pve1" },
|
||||||
|
{ name: "elsewhere", apiTokenHash: "e", apiTokenPrefix: "e", lastSeenAt: hoursAgo(2) },
|
||||||
|
])
|
||||||
|
.returning({ id: servers.id });
|
||||||
|
hostId = made[0].id;
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the host's alert says what it takes with it, and the guests' are held back, not counted", async () => {
|
||||||
|
const report = await getAlerts(true);
|
||||||
|
const offline = report.alerts.filter((a) => a.category === "offline");
|
||||||
|
assert.equal(offline.length, 4);
|
||||||
|
const host = offline.find((a) => a.message.startsWith("pve1"))!;
|
||||||
|
assert.deepEqual(host.affects, ["db", "web"]);
|
||||||
|
assert.equal(host.silenced, false);
|
||||||
|
const web = offline.find((a) => a.message.startsWith("web"))!;
|
||||||
|
assert.deepEqual(web.causedBy, { name: "pve1" });
|
||||||
|
assert.equal(web.silenced, true, "held back like a silenced alert");
|
||||||
|
assert.equal(offline.find((a) => a.message.startsWith("elsewhere"))!.causedBy, undefined);
|
||||||
|
assert.equal(report.counts.critical, 2, "pve1 and the unrelated one");
|
||||||
|
assert.equal(report.counts.grouped, 2);
|
||||||
|
assert.equal(report.counts.silenced, 0, "none of this is maintenance");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("when the host reports again the guests that are still down stand alone", async () => {
|
||||||
|
await db.update(servers).set({ lastSeenAt: new Date().toISOString() }).where(eq(servers.id, hostId));
|
||||||
|
invalidateAlertsCache(); // a look under ten seconds old is reused otherwise
|
||||||
|
const report = await getAlerts(true);
|
||||||
|
assert.equal(report.counts.critical, 3);
|
||||||
|
assert.equal(report.counts.grouped, 0);
|
||||||
|
assert.ok(report.alerts.every((a) => !a.causedBy));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what servers say is waiting to be installed", () => {
|
||||||
|
test("how long updates have been waiting carries from report to report", () => {
|
||||||
|
const t0 = "2026-10-01T00:00:00.000Z";
|
||||||
|
const t1 = "2026-10-05T00:00:00.000Z";
|
||||||
|
const t2 = "2026-10-09T00:00:00.000Z";
|
||||||
|
const t3 = "2026-10-12T00:00:00.000Z";
|
||||||
|
const first = nextUpdateState({ pendingUpdates: null, updatesPendingSince: null }, { manager: "apt", count: 4, security: 1 }, t0);
|
||||||
|
assert.equal(first.updatesPendingSince, t0);
|
||||||
|
const second = nextUpdateState(first, { manager: "apt", count: 6, security: 2, reboot_required: true }, t1);
|
||||||
|
assert.equal(second.updatesPendingSince, t0, "still waiting: the clock keeps running");
|
||||||
|
assert.deepEqual(parsePendingUpdates(second.pendingUpdates), { manager: "apt", count: 6, security: 2, rebootRequired: true, reportedAt: t1 });
|
||||||
|
const cleared = nextUpdateState(second, { manager: "apt", count: 0, security: 0 }, t2);
|
||||||
|
assert.equal(cleared.updatesPendingSince, null);
|
||||||
|
assert.equal(nextUpdateState(cleared, { manager: "apt", count: 2 }, t3).updatesPendingSince, t3, "and starts again when some appear");
|
||||||
|
assert.equal(parsePendingUpdates(nextUpdateState(cleared, { count: 2 }, t3).pendingUpdates)?.security, null, "unknown security is unknown, not zero");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("garbage is no updates, not an error", () => {
|
||||||
|
assert.equal(parsePendingUpdates(null), null);
|
||||||
|
assert.equal(parsePendingUpdates("{"), null);
|
||||||
|
assert.equal(parsePendingUpdates(JSON.stringify({ count: "many" })), null);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the routes", () => {
|
||||||
|
let app: Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "admin";
|
||||||
|
let hostId = 0;
|
||||||
|
let guestId = 0;
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "admin", name: "Ada", role: "admin" },
|
||||||
|
{ oidcSub: "operator", name: "Olga", role: "operator" },
|
||||||
|
]);
|
||||||
|
const [host] = await db.insert(servers).values({ name: "rack", apiTokenHash: hashToken("host-token"), apiTokenPrefix: "ht", lastSeenAt: new Date().toISOString() }).returning();
|
||||||
|
const [guest] = await db.insert(servers).values({ name: "vm1", apiTokenHash: hashToken("guest-token"), apiTokenPrefix: "gt", lastSeenAt: new Date().toISOString() }).returning();
|
||||||
|
hostId = host.id;
|
||||||
|
guestId = guest.id;
|
||||||
|
|
||||||
|
const a = express();
|
||||||
|
a.use("/api/agent/report", agentReportRouter);
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/servers", serversRouter);
|
||||||
|
app = await new Promise<Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
app.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const call = (method: string, path: string, body?: unknown) =>
|
||||||
|
fetch(`${base}${path}`, { method, headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
const report = (token: string, system: unknown) =>
|
||||||
|
fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${token}` }, body: JSON.stringify({ hostname: "h", os_type: "linux", system, tasks: [] }) });
|
||||||
|
|
||||||
|
test("an agent's update counts are stored, a report without them keeps what was said, and a garbled answer costs nothing", async () => {
|
||||||
|
assert.equal((await report("guest-token", { updates: { manager: "apt", count: 7, security: 2, reboot_required: true } })).status, 202);
|
||||||
|
let [s] = await db.select().from(servers).where(eq(servers.id, guestId));
|
||||||
|
assert.deepEqual(parsePendingUpdates(s.pendingUpdates), { manager: "apt", count: 7, security: 2, rebootRequired: true, reportedAt: parsePendingUpdates(s.pendingUpdates)!.reportedAt });
|
||||||
|
const since = s.updatesPendingSince;
|
||||||
|
assert.ok(since);
|
||||||
|
|
||||||
|
assert.equal((await report("guest-token", { cpu: { cores: 2 } })).status, 202);
|
||||||
|
[s] = await db.select().from(servers).where(eq(servers.id, guestId));
|
||||||
|
assert.equal(parsePendingUpdates(s.pendingUpdates)?.count, 7, "an agent that said nothing doesn't wipe it");
|
||||||
|
|
||||||
|
assert.equal((await report("guest-token", { updates: { count: "lots" }, cpu: { cores: 4 } })).status, 202, "the report itself still goes through");
|
||||||
|
[s] = await db.select().from(servers).where(eq(servers.id, guestId));
|
||||||
|
assert.equal(s.cpuCores, 4);
|
||||||
|
assert.equal(parsePendingUpdates(s.pendingUpdates)?.count, 7);
|
||||||
|
assert.equal(s.updatesPendingSince, since);
|
||||||
|
|
||||||
|
assert.equal((await report("host-token", { updates: { manager: "dnf", count: 0 } })).status, 202);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the updates list shows every server that reports, with or without figures", async () => {
|
||||||
|
await db.insert(servers).values({ name: "silent", apiTokenHash: "s", apiTokenPrefix: "s" }); // never reported
|
||||||
|
const body = await (await call("GET", "/servers/updates")).json();
|
||||||
|
const byName = new Map(body.servers.map((x: { name: string }) => [x.name, x]));
|
||||||
|
assert.ok(!byName.has("silent"));
|
||||||
|
const vm = byName.get("vm1") as { updates: { count: number; security: number; rebootRequired: boolean }; pendingSince: string };
|
||||||
|
assert.equal(vm.updates.count, 7);
|
||||||
|
assert.equal(vm.updates.rebootRequired, true);
|
||||||
|
assert.ok(vm.pendingSince);
|
||||||
|
assert.equal((byName.get("rack") as { pendingSince: string | null }).pendingSince, null, "nothing waiting");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an admin says what a server runs on; a loop or a stranger is refused; it shows on both pages", async () => {
|
||||||
|
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: hostId })).status, 200);
|
||||||
|
const detail = await (await call("GET", `/servers/${guestId}/detail`)).json();
|
||||||
|
assert.deepEqual(detail.relations.parent, { id: hostId, name: "rack", auto: false });
|
||||||
|
const hostDetail = await (await call("GET", `/servers/${hostId}/detail`)).json();
|
||||||
|
assert.deepEqual(hostDetail.relations.children, [{ id: guestId, name: "vm1", auto: false }]);
|
||||||
|
assert.equal(detail.server.pendingUpdates.count, 7, "the detail shows what is waiting, as a figure rather than stored text");
|
||||||
|
|
||||||
|
const loop = await call("PATCH", `/servers/${hostId}`, { parentServerId: guestId });
|
||||||
|
assert.equal(loop.status, 400);
|
||||||
|
assert.equal((await loop.json()).error, "parent_loop");
|
||||||
|
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: guestId })).status, 400);
|
||||||
|
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: 99999 })).status, 400);
|
||||||
|
|
||||||
|
who = "operator";
|
||||||
|
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: null })).status, 403);
|
||||||
|
who = "admin";
|
||||||
|
assert.equal((await call("PATCH", `/servers/${guestId}`, { parentServerId: null })).status, 200);
|
||||||
|
assert.equal((await (await call("GET", `/servers/${guestId}/detail`)).json()).relations.parent, null, "back to working it out");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("removing the host sets the servers on it free", async () => {
|
||||||
|
await call("PATCH", `/servers/${guestId}`, { parentServerId: hostId });
|
||||||
|
assert.equal((await call("DELETE", `/servers/${hostId}`)).status, 204);
|
||||||
|
const [s] = await db.select().from(servers).where(eq(servers.id, guestId));
|
||||||
|
assert.equal(s.parentServerId, null);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
/**
|
||||||
|
* Running a Semaphore template from the app, limited to one host, and watching how the run is getting on.
|
||||||
|
*
|
||||||
|
* Run with: npm test --workspace server
|
||||||
|
*/
|
||||||
|
import { test, describe, before, after } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import http from "node:http";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { AddressInfo } from "node:net";
|
||||||
|
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "hm-sem-"));
|
||||||
|
process.env.DATABASE_PATH = join(dir, "test.sqlite");
|
||||||
|
process.env.SESSION_DIR = join(dir, "sessions");
|
||||||
|
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
|
||||||
|
|
||||||
|
const express = (await import("express")).default;
|
||||||
|
const { db } = await import("../src/db/client.js");
|
||||||
|
const { users, integrations, auditLog } = await import("../src/db/schema.js");
|
||||||
|
const { runMigrations } = await import("../src/db/migrate.js");
|
||||||
|
const { integrationsRouter } = await import("../src/routes/integrations.js");
|
||||||
|
|
||||||
|
await runMigrations();
|
||||||
|
|
||||||
|
describe("running a Semaphore template", () => {
|
||||||
|
let semaphore: http.Server;
|
||||||
|
let app: http.Server;
|
||||||
|
let base = "";
|
||||||
|
let who = "operator";
|
||||||
|
let integrationId = 0;
|
||||||
|
const received: { method: string; url: string; body: Record<string, unknown> | null; auth: string | undefined }[] = [];
|
||||||
|
|
||||||
|
before(async () => {
|
||||||
|
semaphore = await new Promise<http.Server>((resolve) => {
|
||||||
|
const s = http.createServer((req, res) => {
|
||||||
|
let raw = "";
|
||||||
|
req.on("data", (c) => (raw += c));
|
||||||
|
req.on("end", () => {
|
||||||
|
received.push({ method: req.method ?? "", url: req.url ?? "", body: raw ? JSON.parse(raw) : null, auth: req.headers.authorization });
|
||||||
|
res.setHeader("content-type", "application/json");
|
||||||
|
if (req.method === "POST" && req.url === "/api/project/3/tasks") return res.end(JSON.stringify({ id: 77, status: "waiting", created: "2026-10-17T10:00:00Z" }));
|
||||||
|
if (req.method === "GET" && req.url === "/api/project/3/tasks/77") return res.end(JSON.stringify({ id: 77, status: "running", start: "2026-10-17T10:00:05Z" }));
|
||||||
|
res.statusCode = 404;
|
||||||
|
res.end(JSON.stringify({ error: "not here" }));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
s.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
const url = `http://127.0.0.1:${(semaphore.address() as AddressInfo).port}`;
|
||||||
|
|
||||||
|
await db.insert(users).values([
|
||||||
|
{ oidcSub: "operator", name: "Olga", role: "operator" },
|
||||||
|
{ oidcSub: "viewer", name: "Vic", role: "viewer" },
|
||||||
|
]);
|
||||||
|
const [row] = await db.insert(integrations).values({ type: "semaphore", name: "Semaphore", baseUrl: url, config: JSON.stringify({ url, token: "secret-token" }) }).returning();
|
||||||
|
integrationId = row.id;
|
||||||
|
|
||||||
|
const a = express();
|
||||||
|
a.use(express.json());
|
||||||
|
a.use((req, _res, next) => {
|
||||||
|
(req as unknown as { session: unknown }).session = { user: { sub: who } };
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
a.use("/api/integrations", integrationsRouter);
|
||||||
|
app = await new Promise<http.Server>((resolve) => {
|
||||||
|
const s = a.listen(0, "127.0.0.1", () => resolve(s));
|
||||||
|
});
|
||||||
|
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/integrations/${integrationId}/semaphore`;
|
||||||
|
});
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
semaphore.close();
|
||||||
|
app.close();
|
||||||
|
try {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// Windows keeps the database file locked until the process exits.
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const run = (body?: unknown) =>
|
||||||
|
fetch(`${base}/projects/3/templates/9/run`, { method: "POST", headers: { "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||||
|
|
||||||
|
test("a run limited to a host tells Semaphore which, and who started it", async () => {
|
||||||
|
const res = await run({ limit: "elsa" });
|
||||||
|
assert.equal(res.status, 201);
|
||||||
|
assert.equal((await res.json()).task.id, 77);
|
||||||
|
const sent = received.find((r) => r.method === "POST")!;
|
||||||
|
assert.equal(sent.url, "/api/project/3/tasks");
|
||||||
|
assert.equal(sent.auth, "Bearer secret-token");
|
||||||
|
assert.equal(sent.body?.template_id, 9);
|
||||||
|
assert.equal(sent.body?.limit, "elsa");
|
||||||
|
assert.match(String(sent.body?.message), /Olga/);
|
||||||
|
assert.match(String(sent.body?.message), /limited to elsa/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a run with no limit sends none, and the audit log records what ran and where", async () => {
|
||||||
|
received.length = 0;
|
||||||
|
assert.equal((await run()).status, 201);
|
||||||
|
assert.equal((await run({})).status, 201);
|
||||||
|
for (const r of received) {
|
||||||
|
assert.equal(r.body?.limit, undefined);
|
||||||
|
assert.doesNotMatch(String(r.body?.message), /limited/);
|
||||||
|
}
|
||||||
|
const entries = (await db.select().from(auditLog)).filter((e) => e.action === "run_template");
|
||||||
|
assert.equal(entries.length, 3);
|
||||||
|
assert.match(entries[0].detail ?? "", /"limit":"elsa"/);
|
||||||
|
assert.doesNotMatch(entries[1].detail ?? "", /limit/);
|
||||||
|
assert.ok(entries.every((e) => !(e.detail ?? "").includes("secret-token")));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("anything that isn't a host or a pattern is refused before it reaches Semaphore", async () => {
|
||||||
|
received.length = 0;
|
||||||
|
for (const limit of ["--extra-vars x=1", "-e foo", "elsa; reboot", "$(id)", "a b", "", "x".repeat(201), "../etc"]) {
|
||||||
|
const res = await run({ limit });
|
||||||
|
assert.equal(res.status, limit === "" ? 201 : 400, JSON.stringify(limit).slice(0, 40));
|
||||||
|
}
|
||||||
|
// the empty one is "no limit"; nothing else was sent on
|
||||||
|
assert.equal(received.filter((r) => r.method === "POST").length, 1);
|
||||||
|
for (const ok of ["web01,db*", "elsa.lab.example", "10.0.0.5", "group:&web"]) {
|
||||||
|
if (ok.includes("&")) continue; // not a pattern this accepts
|
||||||
|
assert.equal((await run({ limit: ok })).status, 201, ok);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a viewer can't run anything", async () => {
|
||||||
|
who = "viewer";
|
||||||
|
received.length = 0;
|
||||||
|
assert.equal((await run({ limit: "elsa" })).status, 403);
|
||||||
|
assert.equal(received.length, 0);
|
||||||
|
who = "operator";
|
||||||
|
});
|
||||||
|
|
||||||
|
test("how a run is getting on can be asked for, and a missing run is reported", async () => {
|
||||||
|
const res = await fetch(`${base}/projects/3/tasks/77`);
|
||||||
|
assert.equal(res.status, 200);
|
||||||
|
assert.deepEqual((await res.json()).task, { id: 77, status: "running", created: null, start: "2026-10-17T10:00:05Z", end: null });
|
||||||
|
assert.equal((await fetch(`${base}/projects/3/tasks/999`)).status, 502);
|
||||||
|
assert.equal((await fetch(`${base}/projects/x/tasks/77`)).status, 400);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -28,6 +28,7 @@ import AdminLinks from "./pages/AdminLinks";
|
|||||||
import Alerts from "./pages/Alerts";
|
import Alerts from "./pages/Alerts";
|
||||||
import Generator from "./pages/Generator";
|
import Generator from "./pages/Generator";
|
||||||
import Maintenance from "./pages/Maintenance";
|
import Maintenance from "./pages/Maintenance";
|
||||||
|
import Updates from "./pages/Updates";
|
||||||
import Domains from "./pages/Domains";
|
import Domains from "./pages/Domains";
|
||||||
import Consistency from "./pages/Consistency";
|
import Consistency from "./pages/Consistency";
|
||||||
import Privacy from "./pages/Privacy";
|
import Privacy from "./pages/Privacy";
|
||||||
@@ -136,6 +137,7 @@ function Shelled({ user }: { user: CurrentUser }) {
|
|||||||
<Route path="/consistency" element={<Consistency user={user} />} />
|
<Route path="/consistency" element={<Consistency user={user} />} />
|
||||||
<Route path="/domains" element={<Domains user={user} />} />
|
<Route path="/domains" element={<Domains user={user} />} />
|
||||||
<Route path="/maintenance" element={<Maintenance user={user} />} />
|
<Route path="/maintenance" element={<Maintenance user={user} />} />
|
||||||
|
<Route path="/updates" element={<Updates />} />
|
||||||
<Route path="/docker" element={<Docker user={user} />} />
|
<Route path="/docker" element={<Docker user={user} />} />
|
||||||
<Route path="/tailscale" element={<Tailscale user={user} />} />
|
<Route path="/tailscale" element={<Tailscale user={user} />} />
|
||||||
<Route path="/semaphore" element={<Semaphore user={user} />} />
|
<Route path="/semaphore" element={<Semaphore user={user} />} />
|
||||||
|
|||||||
+78
-3
@@ -490,6 +490,38 @@ export interface ServerRecord {
|
|||||||
proxmoxVmid: number | null;
|
proxmoxVmid: number | null;
|
||||||
hideProxmoxLink: boolean;
|
hideProxmoxLink: boolean;
|
||||||
tags: string[];
|
tags: string[];
|
||||||
|
/** The server this one runs on, when an admin said so; null means it is worked out from the Proxmox node. */
|
||||||
|
parentServerId: number | null;
|
||||||
|
/** What the agent says is waiting to be installed; null until it has said. */
|
||||||
|
pendingUpdates: PendingUpdates | null;
|
||||||
|
/** When updates were first seen waiting (and none were before); null when none are. */
|
||||||
|
updatesPendingSince: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PendingUpdates {
|
||||||
|
manager: string;
|
||||||
|
count: number;
|
||||||
|
/** Null when the package manager can't say which are security fixes. */
|
||||||
|
security: number | null;
|
||||||
|
rebootRequired: boolean;
|
||||||
|
reportedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One server on the Updates page. */
|
||||||
|
export interface UpdatesServer {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
osType: string;
|
||||||
|
lastSeenAt: string | null;
|
||||||
|
updates: PendingUpdates | null;
|
||||||
|
pendingSince: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What a server runs on, and what runs on it. `auto`: worked out from the Proxmox node rather than set by hand. */
|
||||||
|
export interface ServerRelation {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
auto: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ServerHardware {
|
export interface ServerHardware {
|
||||||
@@ -531,6 +563,7 @@ export interface ServerDetail {
|
|||||||
hardware: ServerHardware;
|
hardware: ServerHardware;
|
||||||
dnsMatches: DnsMatch[];
|
dnsMatches: DnsMatch[];
|
||||||
links: ServerLink[];
|
links: ServerLink[];
|
||||||
|
relations: { parent: ServerRelation | null; children: ServerRelation[] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export type AlertSeverity = "critical" | "warning" | "info";
|
export type AlertSeverity = "critical" | "warning" | "info";
|
||||||
@@ -547,6 +580,10 @@ export interface AlertItem {
|
|||||||
silenced: boolean;
|
silenced: boolean;
|
||||||
/** What a maintenance window would go on to silence it — what the Silence button acts on. Null when nothing can silence it. */
|
/** What a maintenance window would go on to silence it — what the Silence button acts on. Null when nothing can silence it. */
|
||||||
target?: { type: MaintenanceTargetType; id: number; name: string } | null;
|
target?: { type: MaintenanceTargetType; id: number; name: string } | null;
|
||||||
|
/** Held back because another problem explains it (a guest down because its host is): the host's alert says so too. */
|
||||||
|
causedBy?: { name: string };
|
||||||
|
/** On a host's alert: the servers on it whose own alerts are folded under this one. */
|
||||||
|
affects?: string[];
|
||||||
/** The osTicket ticket opened for it, if one was. */
|
/** The osTicket ticket opened for it, if one was. */
|
||||||
ticket?: { number: string | null; url: string | null; grouped: boolean; openedAt: string | null } | null;
|
ticket?: { number: string | null; url: string | null; grouped: boolean; openedAt: string | null } | null;
|
||||||
}
|
}
|
||||||
@@ -590,7 +627,7 @@ export interface AlertTicketOverview {
|
|||||||
|
|
||||||
export interface AlertsReport {
|
export interface AlertsReport {
|
||||||
alerts: AlertItem[];
|
alerts: AlertItem[];
|
||||||
counts: { critical: number; warning: number; info: number; silenced: number };
|
counts: { critical: number; warning: number; info: number; silenced: number; grouped: number };
|
||||||
couldntCheck: { name: string; error: string; affects: string[] }[];
|
couldntCheck: { name: string; error: string; affects: string[] }[];
|
||||||
notes: string[];
|
notes: string[];
|
||||||
generatedAt: string;
|
generatedAt: string;
|
||||||
@@ -654,6 +691,34 @@ export interface ServerUpdateInput {
|
|||||||
proxmoxGuestType?: "qemu" | "lxc" | null;
|
proxmoxGuestType?: "qemu" | "lxc" | null;
|
||||||
proxmoxVmid?: number | null;
|
proxmoxVmid?: number | null;
|
||||||
hideProxmoxLink?: boolean;
|
hideProxmoxLink?: boolean;
|
||||||
|
/** The server this one runs on; null goes back to working it out from the Proxmox node. */
|
||||||
|
parentServerId?: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A maintenance that repeats: "every Sunday 02:00–04:00 on these servers". */
|
||||||
|
export interface MaintenanceSchedule {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
reason: string | null;
|
||||||
|
/** 0 = Sunday … 6 = Saturday. */
|
||||||
|
days: number[];
|
||||||
|
startTime: string;
|
||||||
|
durationMinutes: number;
|
||||||
|
timezone: string;
|
||||||
|
enabled: boolean;
|
||||||
|
createdBy: string | null;
|
||||||
|
targets: { type: MaintenanceTargetType; id: number; name: string; kind: string }[];
|
||||||
|
nextStart: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MaintenanceScheduleInput {
|
||||||
|
name: string;
|
||||||
|
reason?: string;
|
||||||
|
targets: { type: MaintenanceTargetType; id: number }[];
|
||||||
|
days: number[];
|
||||||
|
startTime: string;
|
||||||
|
durationMinutes: number;
|
||||||
|
timezone: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export type ScheduleType = "cron" | "systemd_timer" | "windows_task" | "docker" | "backup" | "update" | "n8n_workflow" | "manual";
|
export type ScheduleType = "cron" | "systemd_timer" | "windows_task" | "docker" | "backup" | "update" | "n8n_workflow" | "manual";
|
||||||
@@ -1230,6 +1295,12 @@ export const api = {
|
|||||||
startBulk: (data: { scope: MaintenanceBulkScope; minutes: number; reason?: string }) =>
|
startBulk: (data: { scope: MaintenanceBulkScope; minutes: number; reason?: string }) =>
|
||||||
request<{ started: number; extended: number; total: number }>("/api/maintenance/bulk", { method: "POST", body: JSON.stringify(data) }),
|
request<{ started: number; extended: number; total: number }>("/api/maintenance/bulk", { method: "POST", body: JSON.stringify(data) }),
|
||||||
endAll: () => request<{ ended: number }>("/api/maintenance/end-all", { method: "POST" }),
|
endAll: () => request<{ ended: number }>("/api/maintenance/end-all", { method: "POST" }),
|
||||||
|
schedules: {
|
||||||
|
list: () => request<{ schedules: MaintenanceSchedule[] }>("/api/maintenance/schedules"),
|
||||||
|
create: (data: MaintenanceScheduleInput) => request<{ id: number; windowsMade: number }>("/api/maintenance/schedules", { method: "POST", body: JSON.stringify(data) }),
|
||||||
|
setEnabled: (id: number, enabled: boolean) => request<{ enabled: boolean }>(`/api/maintenance/schedules/${id}`, { method: "PATCH", body: JSON.stringify({ enabled }) }),
|
||||||
|
remove: (id: number) => request<void>(`/api/maintenance/schedules/${id}`, { method: "DELETE" }),
|
||||||
|
},
|
||||||
syncUptimeKuma: (data: { integrationId: number }) =>
|
syncUptimeKuma: (data: { integrationId: number }) =>
|
||||||
request<UptimeKumaSyncResult>("/api/maintenance/sync/uptimekuma", { method: "POST", body: JSON.stringify(data) }),
|
request<UptimeKumaSyncResult>("/api/maintenance/sync/uptimekuma", { method: "POST", body: JSON.stringify(data) }),
|
||||||
importUptimeKuma: (data: { integrationId: number; minutes: number }) =>
|
importUptimeKuma: (data: { integrationId: number; minutes: number }) =>
|
||||||
@@ -1387,6 +1458,7 @@ export const api = {
|
|||||||
update: (id: number, data: ServerUpdateInput) =>
|
update: (id: number, data: ServerUpdateInput) =>
|
||||||
request<{ server: ServerRecord }>(`/api/servers/${id}`, { method: "PATCH", body: JSON.stringify(data) }),
|
request<{ server: ServerRecord }>(`/api/servers/${id}`, { method: "PATCH", body: JSON.stringify(data) }),
|
||||||
summary: () => request<ServerSummary>("/api/servers/summary"),
|
summary: () => request<ServerSummary>("/api/servers/summary"),
|
||||||
|
updates: () => request<{ servers: UpdatesServer[] }>("/api/servers/updates"),
|
||||||
detail: (id: number) => request<ServerDetail>(`/api/servers/${id}/detail`),
|
detail: (id: number) => request<ServerDetail>(`/api/servers/${id}/detail`),
|
||||||
activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`),
|
activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`),
|
||||||
metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`),
|
metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`),
|
||||||
@@ -1552,11 +1624,14 @@ export const api = {
|
|||||||
semaphore: {
|
semaphore: {
|
||||||
templates: (integrationId: number) =>
|
templates: (integrationId: number) =>
|
||||||
request<SemaphoreTemplatesResponse>(`/api/integrations/${integrationId}/semaphore/templates`),
|
request<SemaphoreTemplatesResponse>(`/api/integrations/${integrationId}/semaphore/templates`),
|
||||||
run: (integrationId: number, projectId: number, templateId: number) =>
|
/** `limit` runs it on those hosts only (ansible's --limit). */
|
||||||
|
run: (integrationId: number, projectId: number, templateId: number, options?: { limit?: string }) =>
|
||||||
request<{ task: SemaphoreTask }>(
|
request<{ task: SemaphoreTask }>(
|
||||||
`/api/integrations/${integrationId}/semaphore/projects/${projectId}/templates/${templateId}/run`,
|
`/api/integrations/${integrationId}/semaphore/projects/${projectId}/templates/${templateId}/run`,
|
||||||
{ method: "POST" },
|
{ method: "POST", body: JSON.stringify(options ?? {}) },
|
||||||
),
|
),
|
||||||
|
task: (integrationId: number, projectId: number, taskId: number) =>
|
||||||
|
request<{ task: SemaphoreTask }>(`/api/integrations/${integrationId}/semaphore/projects/${projectId}/tasks/${taskId}`),
|
||||||
},
|
},
|
||||||
proxmox: {
|
proxmox: {
|
||||||
guests: (integrationId: number) => request<ProxmoxGuestsResponse>(`/api/integrations/${integrationId}/proxmox/guests`),
|
guests: (integrationId: number) => request<ProxmoxGuestsResponse>(`/api/integrations/${integrationId}/proxmox/guests`),
|
||||||
|
|||||||
@@ -129,7 +129,7 @@ export default function MaintenanceTimeline({ windows, now }: { windows: Mainten
|
|||||||
return (
|
return (
|
||||||
<span
|
<span
|
||||||
key={w.id}
|
key={w.id}
|
||||||
className={`timeline-bar ${running ? "timeline-bar-running" : "timeline-bar-scheduled"} ${w.source ? "timeline-bar-mirrored" : ""}`}
|
className={`timeline-bar ${running ? "timeline-bar-running" : "timeline-bar-scheduled"} ${w.source?.startsWith("kuma:") ? "timeline-bar-mirrored" : ""} ${w.source?.startsWith("schedule:") ? "timeline-bar-recurring" : ""}`}
|
||||||
style={bar(from, to)}
|
style={bar(from, to)}
|
||||||
title={`${row.name}: ${formatDateTime(new Date(from))} – ${formatDateTime(new Date(to))}${w.reason ? ` · ${w.reason}` : ""}`}
|
title={`${row.name}: ${formatDateTime(new Date(from))} – ${formatDateTime(new Date(to))}${w.reason ? ` · ${w.reason}` : ""}`}
|
||||||
/>
|
/>
|
||||||
@@ -150,6 +150,9 @@ export default function MaintenanceTimeline({ windows, now }: { windows: Mainten
|
|||||||
<span>
|
<span>
|
||||||
<span className="timeline-key timeline-bar-mirrored" /> From Uptime Kuma
|
<span className="timeline-key timeline-bar-mirrored" /> From Uptime Kuma
|
||||||
</span>
|
</span>
|
||||||
|
<span>
|
||||||
|
<span className="timeline-key timeline-bar-recurring" /> Recurring
|
||||||
|
</span>
|
||||||
<span>The same windows are listed in the tables below.</span>
|
<span>The same windows are listed in the tables below.</span>
|
||||||
{further > 0 && <span>{further} further ahead than this range.</span>}
|
{further > 0 && <span>{further} further ahead than this range.</span>}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -0,0 +1,309 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { api, type MaintenanceSchedule, type MaintenanceTargetType, type MaintenanceTargets } from "../api/client";
|
||||||
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
import TimeAgo from "./TimeAgo";
|
||||||
|
import { ErrorNotice } from "./Notice";
|
||||||
|
|
||||||
|
// Monday first, as a week reads; the numbers are what the server keeps (0 = Sunday).
|
||||||
|
const DAYS: { value: number; short: string }[] = [
|
||||||
|
{ value: 1, short: "Mon" },
|
||||||
|
{ value: 2, short: "Tue" },
|
||||||
|
{ value: 3, short: "Wed" },
|
||||||
|
{ value: 4, short: "Thu" },
|
||||||
|
{ value: 5, short: "Fri" },
|
||||||
|
{ value: 6, short: "Sat" },
|
||||||
|
{ value: 0, short: "Sun" },
|
||||||
|
];
|
||||||
|
|
||||||
|
const LENGTHS: { minutes: number; label: string }[] = [
|
||||||
|
{ minutes: 30, label: "30 minutes" },
|
||||||
|
{ minutes: 60, label: "1 hour" },
|
||||||
|
{ minutes: 120, label: "2 hours" },
|
||||||
|
{ minutes: 180, label: "3 hours" },
|
||||||
|
{ minutes: 240, label: "4 hours" },
|
||||||
|
{ minutes: 360, label: "6 hours" },
|
||||||
|
{ minutes: 480, label: "8 hours" },
|
||||||
|
{ minutes: 720, label: "12 hours" },
|
||||||
|
{ minutes: 1440, label: "24 hours" },
|
||||||
|
];
|
||||||
|
|
||||||
|
function daysWords(days: number[]): string {
|
||||||
|
if (days.length === 7) return "Every day";
|
||||||
|
const set = new Set(days);
|
||||||
|
if ([1, 2, 3, 4, 5].every((d) => set.has(d)) && days.length === 5) return "Weekdays";
|
||||||
|
if (days.length === 2 && set.has(6) && set.has(0)) return "Weekends";
|
||||||
|
return DAYS.filter((d) => set.has(d.value)).map((d) => d.short).join(", ");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** "02:00–04:00", with "(+1 d)" when it runs into the next day. */
|
||||||
|
function timeWords(start: string, minutes: number): string {
|
||||||
|
const [h, m] = start.split(":").map(Number);
|
||||||
|
const end = h * 60 + m + minutes;
|
||||||
|
const clock = (t: number) => `${String(Math.floor((t % 1440) / 60)).padStart(2, "0")}:${String(t % 60).padStart(2, "0")}`;
|
||||||
|
return `${start}–${clock(end)}${end >= 1440 ? ` (+${Math.floor(end / 1440)} d)` : ""}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function namesWords(names: string[]): string {
|
||||||
|
return names.length <= 3 ? names.join(", ") : `${names.slice(0, 3).join(", ")} and ${names.length - 3} more`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Maintenance that comes back by itself — "every Sunday 02:00–04:00 on these servers". Each time is made into an ordinary maintenance
|
||||||
|
* window a week ahead, so it shows on the timeline and in Scheduled, silences alerts while it runs, and any single one can be cancelled.
|
||||||
|
*/
|
||||||
|
export default function RecurringMaintenance({ canEdit, targets, onChanged }: { canEdit: boolean; targets: MaintenanceTargets | null; onChanged: () => void }) {
|
||||||
|
const [schedules, setSchedules] = useState<MaintenanceSchedule[] | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [adding, setAdding] = useState(false);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
|
||||||
|
const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
|
||||||
|
const [name, setName] = useState("");
|
||||||
|
const [reason, setReason] = useState("");
|
||||||
|
const [days, setDays] = useState<number[]>([0]);
|
||||||
|
const [startTime, setStartTime] = useState("02:00");
|
||||||
|
const [minutes, setMinutes] = useState(120);
|
||||||
|
const [picked, setPicked] = useState<Set<string>>(new Set());
|
||||||
|
|
||||||
|
function load() {
|
||||||
|
api.maintenance.schedules
|
||||||
|
.list()
|
||||||
|
.then((res) => setSchedules(res.schedules))
|
||||||
|
.catch((err) => setError(readableError(err)));
|
||||||
|
}
|
||||||
|
useEffect(load, []);
|
||||||
|
|
||||||
|
const key = (type: MaintenanceTargetType, id: number) => `${type}:${id}`;
|
||||||
|
const groups: { label: string; type: MaintenanceTargetType; items: { id: number; name: string }[] }[] = [
|
||||||
|
{ label: "Servers", type: "server", items: targets?.servers ?? [] },
|
||||||
|
{ label: "Integrations", type: "integration", items: targets?.integrations ?? [] },
|
||||||
|
{ label: "DNS providers", type: "dns_provider", items: targets?.dnsProviders ?? [] },
|
||||||
|
];
|
||||||
|
|
||||||
|
function toggleGroup(type: MaintenanceTargetType, items: { id: number }[]) {
|
||||||
|
setPicked((prev) => {
|
||||||
|
const next = new Set(prev);
|
||||||
|
const all = items.every((i) => next.has(key(type, i.id)));
|
||||||
|
for (const i of items) (all ? next.delete(key(type, i.id)) : next.add(key(type, i.id)));
|
||||||
|
return next;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function create(e: React.FormEvent) {
|
||||||
|
e.preventDefault();
|
||||||
|
setError(null);
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
const chosen = [...picked].map((k) => {
|
||||||
|
const [type, id] = k.split(":");
|
||||||
|
return { type: type as MaintenanceTargetType, id: Number(id) };
|
||||||
|
});
|
||||||
|
const res = await api.maintenance.schedules.create({ name: name.trim(), reason: reason.trim() || undefined, targets: chosen, days, startTime, durationMinutes: minutes, timezone });
|
||||||
|
toast.success(`Saved. ${res.windowsMade} maintenance window${res.windowsMade === 1 ? "" : "s"} made for the coming week.`);
|
||||||
|
setName("");
|
||||||
|
setReason("");
|
||||||
|
setPicked(new Set());
|
||||||
|
setAdding(false);
|
||||||
|
load();
|
||||||
|
onChanged();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function toggle(s: MaintenanceSchedule) {
|
||||||
|
try {
|
||||||
|
await api.maintenance.schedules.setEnabled(s.id, !s.enabled);
|
||||||
|
load();
|
||||||
|
onChanged();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function remove(s: MaintenanceSchedule) {
|
||||||
|
if (!(await confirmDialog({ title: "Remove recurring maintenance", message: `Remove "${s.name}"? Windows it made that haven't begun are taken back; one already under way ends on its own.`, confirmLabel: "Remove", danger: true }))) return;
|
||||||
|
try {
|
||||||
|
await api.maintenance.schedules.remove(s.id);
|
||||||
|
toast.success(`Removed ${s.name}.`);
|
||||||
|
load();
|
||||||
|
onChanged();
|
||||||
|
} catch (err) {
|
||||||
|
setError(readableError(err));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (schedules !== null && schedules.length === 0 && !canEdit) return null;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Recurring</h3>
|
||||||
|
{canEdit && !adding && (
|
||||||
|
<div className="card-actions">
|
||||||
|
<button type="button" className="btn btn-sm btn-primary" onClick={() => setAdding(true)}>
|
||||||
|
Add recurring maintenance
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{adding && (
|
||||||
|
<form onSubmit={create}>
|
||||||
|
<div className="card-body row g-3">
|
||||||
|
<div className="col-md-4">
|
||||||
|
<label className="form-label" htmlFor="rec-name">
|
||||||
|
Name
|
||||||
|
</label>
|
||||||
|
<input id="rec-name" className="form-control" required maxLength={80} placeholder="e.g. Patch night" value={name} onChange={(e) => setName(e.target.value)} />
|
||||||
|
</div>
|
||||||
|
<div className="col-md-8">
|
||||||
|
<label className="form-label" htmlFor="rec-reason">
|
||||||
|
Reason (optional)
|
||||||
|
</label>
|
||||||
|
<input id="rec-reason" className="form-control" maxLength={200} placeholder="e.g. OS updates and restarts" value={reason} onChange={(e) => setReason(e.target.value)} />
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<div className="form-label" id="rec-days">
|
||||||
|
Days
|
||||||
|
</div>
|
||||||
|
<div className="btn-group" role="group" aria-labelledby="rec-days">
|
||||||
|
{DAYS.map((d) => {
|
||||||
|
const on = days.includes(d.value);
|
||||||
|
return (
|
||||||
|
<button key={d.value} type="button" className={`btn btn-sm ${on ? "btn-primary" : "btn-outline-secondary"}`} aria-pressed={on} onClick={() => setDays((prev) => (on ? prev.filter((x) => x !== d.value) : [...prev, d.value]))}>
|
||||||
|
{d.short}
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-sm-6 col-md-3">
|
||||||
|
<label className="form-label" htmlFor="rec-start">
|
||||||
|
Starts at
|
||||||
|
</label>
|
||||||
|
<input id="rec-start" type="time" className="form-control" required value={startTime} onChange={(e) => setStartTime(e.target.value)} />
|
||||||
|
<div className="form-hint text-secondary small mt-1">Clock in {timezone}</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-sm-6 col-md-3">
|
||||||
|
<label className="form-label" htmlFor="rec-length">
|
||||||
|
For
|
||||||
|
</label>
|
||||||
|
<select id="rec-length" className="form-select" value={minutes} onChange={(e) => setMinutes(Number(e.target.value))}>
|
||||||
|
{LENGTHS.map((l) => (
|
||||||
|
<option key={l.minutes} value={l.minutes}>
|
||||||
|
{l.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<div className="form-label">What it covers</div>
|
||||||
|
<div className="border rounded p-2" style={{ maxHeight: 220, overflowY: "auto" }}>
|
||||||
|
{groups.map(
|
||||||
|
(g) =>
|
||||||
|
g.items.length > 0 && (
|
||||||
|
<fieldset key={g.type} className="mb-2">
|
||||||
|
<legend className="form-label small text-secondary mb-1">
|
||||||
|
{g.label}{" "}
|
||||||
|
<button type="button" className="btn btn-link btn-sm p-0 ms-1" onClick={() => toggleGroup(g.type, g.items)}>
|
||||||
|
{g.items.every((i) => picked.has(key(g.type, i.id))) ? "none" : "all"}
|
||||||
|
</button>
|
||||||
|
</legend>
|
||||||
|
<div className="d-flex flex-wrap gap-3">
|
||||||
|
{g.items.map((i) => (
|
||||||
|
<label key={i.id} className="form-check mb-0">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
className="form-check-input"
|
||||||
|
checked={picked.has(key(g.type, i.id))}
|
||||||
|
onChange={(e) =>
|
||||||
|
setPicked((prev) => {
|
||||||
|
const next = new Set(prev);
|
||||||
|
if (e.target.checked) next.add(key(g.type, i.id));
|
||||||
|
else next.delete(key(g.type, i.id));
|
||||||
|
return next;
|
||||||
|
})
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
<span className="form-check-label">{i.name}</span>
|
||||||
|
</label>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</fieldset>
|
||||||
|
),
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="col-12">
|
||||||
|
<ErrorNotice error={error} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="card-footer d-flex gap-2">
|
||||||
|
<button type="submit" className="btn btn-primary" disabled={busy || picked.size === 0 || days.length === 0 || name.trim() === ""}>
|
||||||
|
{busy ? "Saving…" : "Save"}
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn" onClick={() => setAdding(false)}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{!adding && <ErrorNotice error={error} />}
|
||||||
|
<div className="table-responsive">
|
||||||
|
<table className="table table-vcenter card-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Name</th>
|
||||||
|
<th>Covers</th>
|
||||||
|
<th>When</th>
|
||||||
|
<th>Next</th>
|
||||||
|
{canEdit && <th className="w-1">Actions</th>}
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{schedules?.map((s) => (
|
||||||
|
<tr key={s.id} className={s.enabled ? undefined : "text-secondary"}>
|
||||||
|
<td>
|
||||||
|
{s.name}
|
||||||
|
{!s.enabled && <span className="badge bg-secondary-lt text-secondary ms-2">off</span>}
|
||||||
|
{s.reason && <div className="text-secondary small">{s.reason}</div>}
|
||||||
|
</td>
|
||||||
|
<td>{namesWords(s.targets.map((t) => t.name))}</td>
|
||||||
|
<td>
|
||||||
|
{daysWords(s.days)} {timeWords(s.startTime, s.durationMinutes)}
|
||||||
|
{s.timezone !== timezone && <span className="text-secondary small"> ({s.timezone})</span>}
|
||||||
|
</td>
|
||||||
|
<td>{s.nextStart ? <TimeAgo value={s.nextStart} /> : "—"}</td>
|
||||||
|
{canEdit && (
|
||||||
|
<td>
|
||||||
|
<div className="btn-list flex-nowrap">
|
||||||
|
<button type="button" className="btn btn-sm btn-outline-secondary" onClick={() => void toggle(s)}>
|
||||||
|
{s.enabled ? "Turn off" : "Turn on"}
|
||||||
|
</button>
|
||||||
|
<button type="button" className="btn btn-sm btn-outline-danger" onClick={() => void remove(s)}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
)}
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{schedules?.length === 0 && (
|
||||||
|
<tr>
|
||||||
|
<td colSpan={canEdit ? 5 : 4} className="text-secondary text-center">
|
||||||
|
Nothing repeats yet. Add one for the night you patch, and its windows appear a week ahead.
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
import { useEffect, useRef, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type IntegrationSummary, type SemaphoreTask, type SemaphoreTemplate } from "../api/client";
|
||||||
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
import TimeAgo from "./TimeAgo";
|
||||||
|
import { BlockSkeleton } from "./Skeleton";
|
||||||
|
|
||||||
|
interface Loaded {
|
||||||
|
integration: IntegrationSummary;
|
||||||
|
templates: SemaphoreTemplate[];
|
||||||
|
failed: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
const STATUS_WORDS: Record<SemaphoreTask["status"], { label: string; badge: string }> = {
|
||||||
|
waiting: { label: "Waiting to start", badge: "bg-secondary-lt text-secondary" },
|
||||||
|
starting: { label: "Starting", badge: "bg-blue-lt text-blue" },
|
||||||
|
running: { label: "Running", badge: "bg-blue-lt text-blue" },
|
||||||
|
waiting_confirmation: { label: "Waiting for approval in Semaphore", badge: "bg-yellow-lt text-yellow" },
|
||||||
|
confirmed: { label: "Approved", badge: "bg-blue-lt text-blue" },
|
||||||
|
rejected: { label: "Rejected in Semaphore", badge: "bg-red-lt text-red" },
|
||||||
|
stopping: { label: "Stopping", badge: "bg-yellow-lt text-yellow" },
|
||||||
|
stopped: { label: "Stopped", badge: "bg-secondary-lt text-secondary" },
|
||||||
|
success: { label: "Finished — success", badge: "bg-green-lt text-green" },
|
||||||
|
error: { label: "Finished — failed", badge: "bg-red-lt text-red" },
|
||||||
|
};
|
||||||
|
const FINISHED = new Set<SemaphoreTask["status"]>(["success", "error", "stopped", "rejected"]);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Starts a Semaphore template — optionally on this server only — and shows how the run gets on. It is the same "run" the Semaphore page
|
||||||
|
* has, with the one addition that matters from a server or an alert: ansible's limit, so "apply updates" touches the one machine and
|
||||||
|
* not the whole inventory. Asks first, and says plainly when there is no limit.
|
||||||
|
*/
|
||||||
|
export default function RunPlaybook({ serverName, defaultLimit }: { serverName: string; defaultLimit: string }) {
|
||||||
|
const [loaded, setLoaded] = useState<Loaded[] | null>(null);
|
||||||
|
const [failed, setFailed] = useState(false);
|
||||||
|
const [choice, setChoice] = useState("");
|
||||||
|
const [limitOn, setLimitOn] = useState(true);
|
||||||
|
const [limit, setLimit] = useState(defaultLimit);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [run, setRun] = useState<{ integrationId: number; projectId: number; name: string; limit: string; task: SemaphoreTask } | null>(null);
|
||||||
|
const alive = useRef(true);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
alive.current = true;
|
||||||
|
api.integrations
|
||||||
|
.list()
|
||||||
|
.then(async (res) => {
|
||||||
|
const servers = res.integrations.filter((i) => i.type === "semaphore" && i.enabled);
|
||||||
|
const all = await Promise.all(
|
||||||
|
servers.map((integration) =>
|
||||||
|
api.integrations.semaphore
|
||||||
|
.templates(integration.id)
|
||||||
|
.then((r): Loaded => ({ integration, templates: r.templates, failed: false }))
|
||||||
|
.catch((): Loaded => ({ integration, templates: [], failed: true })),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
if (alive.current) setLoaded(all);
|
||||||
|
})
|
||||||
|
.catch(() => alive.current && setFailed(true));
|
||||||
|
return () => {
|
||||||
|
alive.current = false;
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
// Watch a run until it is over.
|
||||||
|
useEffect(() => {
|
||||||
|
if (!run || FINISHED.has(run.task.status)) return;
|
||||||
|
const timer = setInterval(() => {
|
||||||
|
api.integrations.semaphore
|
||||||
|
.task(run.integrationId, run.projectId, run.task.id)
|
||||||
|
.then((res) => alive.current && setRun((r) => (r && r.task.id === res.task.id ? { ...r, task: res.task } : r)))
|
||||||
|
.catch(() => {
|
||||||
|
// Keep asking: one missed answer isn't the end of the run.
|
||||||
|
});
|
||||||
|
}, 3000);
|
||||||
|
return () => clearInterval(timer);
|
||||||
|
}, [run?.task.id, run?.task.status, run?.integrationId, run?.projectId]);
|
||||||
|
|
||||||
|
const options = (loaded ?? []).flatMap((l) =>
|
||||||
|
l.templates.map((t) => ({
|
||||||
|
key: `${l.integration.id}:${t.projectId}:${t.id}`,
|
||||||
|
integration: l.integration,
|
||||||
|
template: t,
|
||||||
|
label: `${t.name} — ${t.projectName}${(loaded ?? []).length > 1 ? ` (${l.integration.name})` : ""}`,
|
||||||
|
})),
|
||||||
|
);
|
||||||
|
const picked = options.find((o) => o.key === choice);
|
||||||
|
const limitText = limitOn ? limit.trim() : "";
|
||||||
|
|
||||||
|
async function start() {
|
||||||
|
if (!picked) return;
|
||||||
|
const where = limitText ? `on ${limitText} only` : "on everything the playbook covers — it isn't limited to one server";
|
||||||
|
if (
|
||||||
|
!(await confirmDialog({
|
||||||
|
title: `Run ${picked.template.name}?`,
|
||||||
|
message: `This starts "${picked.template.name}" in ${picked.integration.name} ${where}.`,
|
||||||
|
confirmLabel: "Run it",
|
||||||
|
danger: !limitText,
|
||||||
|
}))
|
||||||
|
)
|
||||||
|
return;
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
const res = await api.integrations.semaphore.run(picked.integration.id, picked.template.projectId, picked.template.id, limitText ? { limit: limitText } : undefined);
|
||||||
|
setRun({ integrationId: picked.integration.id, projectId: picked.template.projectId, name: picked.template.name, limit: limitText, task: res.task });
|
||||||
|
toast.success(`Started ${picked.template.name}${limitText ? ` on ${limitText}` : ""}.`);
|
||||||
|
} catch (err) {
|
||||||
|
toast.error(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (failed) return <div className="text-secondary">Couldn't load the integrations.</div>;
|
||||||
|
if (loaded === null) return <BlockSkeleton lines={3} />;
|
||||||
|
if (loaded.length === 0) {
|
||||||
|
return (
|
||||||
|
<div className="text-secondary">
|
||||||
|
There is no Semaphore connected. Add one under <Link to="/integrations">Integrations</Link> to run playbooks from here.
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div>
|
||||||
|
{loaded.some((l) => l.failed) && <div className="text-secondary small mb-2">Couldn't read the templates of {loaded.filter((l) => l.failed).map((l) => l.integration.name).join(", ")}.</div>}
|
||||||
|
{options.length === 0 ? (
|
||||||
|
<div className="text-secondary">No templates found in Semaphore.</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<div className="mb-3">
|
||||||
|
<label className="form-label" htmlFor="run-template">
|
||||||
|
Template
|
||||||
|
</label>
|
||||||
|
<select id="run-template" className="form-select" value={choice} onChange={(e) => setChoice(e.target.value)}>
|
||||||
|
<option value="">Choose a template…</option>
|
||||||
|
{options.map((o) => (
|
||||||
|
<option key={o.key} value={o.key}>
|
||||||
|
{o.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
{picked?.template.lastTask && (
|
||||||
|
<div className="form-hint text-secondary small mt-1">
|
||||||
|
Last run here: {picked.template.lastTask.status}
|
||||||
|
{picked.template.lastTask.end ? (
|
||||||
|
<>
|
||||||
|
, <TimeAgo value={picked.template.lastTask.end} />
|
||||||
|
</>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<div className="mb-3">
|
||||||
|
<label className="form-check">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={limitOn} onChange={(e) => setLimitOn(e.target.checked)} />
|
||||||
|
<span className="form-check-label">Run on {serverName} only</span>
|
||||||
|
</label>
|
||||||
|
{limitOn ? (
|
||||||
|
<div className="mt-2">
|
||||||
|
<label className="form-label small" htmlFor="run-limit">
|
||||||
|
Limit — the host name or pattern as it is in the Semaphore inventory
|
||||||
|
</label>
|
||||||
|
<input id="run-limit" className="form-control" value={limit} maxLength={200} onChange={(e) => setLimit(e.target.value)} placeholder="e.g. web01" />
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div className="text-danger small mt-1">Not limited: it runs on every host the playbook is set up for.</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<button type="button" className="btn btn-primary" disabled={!picked || busy || (limitOn && limit.trim() === "")} onClick={() => void start()}>
|
||||||
|
{busy ? "Starting…" : "Run"}
|
||||||
|
</button>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{run && (
|
||||||
|
<div className="mt-3 border-top pt-3" role="status" aria-live="polite">
|
||||||
|
<div className="d-flex flex-wrap align-items-center gap-2">
|
||||||
|
<strong>{run.name}</strong>
|
||||||
|
{run.limit && <span className="text-secondary">on {run.limit}</span>}
|
||||||
|
<span className={`badge ${STATUS_WORDS[run.task.status]?.badge ?? "bg-secondary-lt text-secondary"}`}>{STATUS_WORDS[run.task.status]?.label ?? run.task.status}</span>
|
||||||
|
</div>
|
||||||
|
<div className="text-secondary small mt-1">
|
||||||
|
Run #{run.task.id} in Semaphore{FINISHED.has(run.task.status) ? "" : " — checking every few seconds"}. The output is in Semaphore.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
import { useState } from "react";
|
||||||
|
import DetailDrawer from "./DetailDrawer";
|
||||||
|
import RunPlaybook from "./RunPlaybook";
|
||||||
|
|
||||||
|
/** "Run…" on an alert about a server: opens a side panel to start a playbook on that server, without leaving the alerts. */
|
||||||
|
export default function RunPlaybookButton({ serverName }: { serverName: string }) {
|
||||||
|
const [open, setOpen] = useState(false);
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<button type="button" className="btn btn-sm btn-outline-secondary" title={`Run a Semaphore playbook on ${serverName}`} onClick={() => setOpen(true)}>
|
||||||
|
Run…
|
||||||
|
</button>
|
||||||
|
{open && (
|
||||||
|
<DetailDrawer title="Run a playbook" subtitle={`On ${serverName}`} onClose={() => setOpen(false)}>
|
||||||
|
<RunPlaybook serverName={serverName} defaultLimit={serverName} />
|
||||||
|
</DetailDrawer>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type ServerRecord, type ServerRelation } from "../api/client";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
import { toast } from "../utils/toast";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Which server this one runs on (a virtual machine on its host). When the host stops reporting, this server's own "offline" alert is
|
||||||
|
* folded under the host's instead of being a second one. Left alone it is worked out from the Proxmox node the server is linked to;
|
||||||
|
* an admin can say it by hand. Everyone sees it; only admins change it.
|
||||||
|
*/
|
||||||
|
export default function RunsOn({ serverId, parent, canEdit, onChanged }: { serverId: number; parent: ServerRelation | null; canEdit: boolean; onChanged: () => void }) {
|
||||||
|
const [servers, setServers] = useState<ServerRecord[] | null>(null);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!canEdit) return;
|
||||||
|
api.servers
|
||||||
|
.list()
|
||||||
|
.then((res) => setServers(res.servers))
|
||||||
|
.catch(() => setServers([]));
|
||||||
|
}, [canEdit]);
|
||||||
|
|
||||||
|
async function change(value: string) {
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
await api.servers.update(serverId, { parentServerId: value === "" ? null : Number(value) });
|
||||||
|
toast.success(value === "" ? "Back to working it out from the Proxmox node." : "Saved.");
|
||||||
|
onChanged();
|
||||||
|
} catch (err) {
|
||||||
|
toast.error(readableError(err));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!canEdit) {
|
||||||
|
return parent ? <Link to={`/servers/${parent.id}`}>{parent.name}</Link> : <span className="text-secondary">—</span>;
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<div>
|
||||||
|
<select className="form-select form-select-sm" aria-label="The server this one runs on" value={parent && !parent.auto ? String(parent.id) : ""} disabled={busy || servers === null} onChange={(e) => void change(e.target.value)}>
|
||||||
|
<option value="">{parent?.auto ? `Automatic: ${parent.name}` : "None (automatic)"}</option>
|
||||||
|
{(servers ?? [])
|
||||||
|
.filter((s) => s.id !== serverId)
|
||||||
|
.map((s) => (
|
||||||
|
<option key={s.id} value={s.id}>
|
||||||
|
{s.name}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<div className="form-hint text-secondary small mt-1">Automatic means worked out from its Proxmox node. If it stops reporting, this server's offline alert is folded under its host's.</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -27,10 +27,12 @@ export default function StatusStrip() {
|
|||||||
.filter((a) => !a.silenced && a.severity !== "info")
|
.filter((a) => !a.silenced && a.severity !== "info")
|
||||||
.sort((a, b) => SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
|
.sort((a, b) => SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
|
||||||
const silenced = report.counts.silenced;
|
const silenced = report.counts.silenced;
|
||||||
|
const grouped = report.counts.grouped ?? 0;
|
||||||
const blind = report.couldntCheck.length;
|
const blind = report.couldntCheck.length;
|
||||||
|
|
||||||
const notes: string[] = [];
|
const notes: string[] = [];
|
||||||
if (silenced > 0) notes.push(`${silenced} silenced by maintenance`);
|
if (silenced > 0) notes.push(`${silenced} silenced by maintenance`);
|
||||||
|
if (grouped > 0) notes.push(`${grouped} more explained by a server that is down`);
|
||||||
if (blind > 0 && badge) notes.push(`${blind} check${blind === 1 ? "" : "s"} couldn't run`);
|
if (blind > 0 && badge) notes.push(`${blind} check${blind === 1 ? "" : "s"} couldn't run`);
|
||||||
|
|
||||||
if (!badge) {
|
if (!badge) {
|
||||||
@@ -50,6 +52,7 @@ export default function StatusStrip() {
|
|||||||
<span className="text-secondary">No active alerts.</span>
|
<span className="text-secondary">No active alerts.</span>
|
||||||
)}
|
)}
|
||||||
{silenced > 0 && <span className="text-secondary"> {silenced} silenced by maintenance.</span>}
|
{silenced > 0 && <span className="text-secondary"> {silenced} silenced by maintenance.</span>}
|
||||||
|
{grouped > 0 && <span className="text-secondary"> {grouped} more explained by a server that is down.</span>}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
import type { PendingUpdates } from "../api/client";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What a server has waiting: "Up to date", "5 waiting" or "5 waiting, 2 security" (red when any are security fixes), and a
|
||||||
|
* "restart needed" beside it. Says "not reported" when the agent hasn't said — never "up to date" for a server that simply didn't tell us.
|
||||||
|
*/
|
||||||
|
export default function UpdatesBadge({ updates }: { updates: PendingUpdates | null }) {
|
||||||
|
if (!updates) {
|
||||||
|
return (
|
||||||
|
<span className="badge bg-secondary-lt text-secondary" title="This server's agent hasn't reported updates — it may be an older agent, or its package manager isn't one it can ask.">
|
||||||
|
not reported
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const security = updates.security ?? 0;
|
||||||
|
const level = updates.count === 0 ? "bg-green-lt text-green" : security > 0 ? "bg-red-lt text-red" : "bg-yellow-lt text-yellow";
|
||||||
|
return (
|
||||||
|
<span className="d-inline-flex flex-wrap align-items-center gap-1">
|
||||||
|
<span className={`badge ${level}`}>
|
||||||
|
{updates.count === 0 ? "Up to date" : `${updates.count} waiting${security > 0 ? `, ${security} security` : ""}`}
|
||||||
|
</span>
|
||||||
|
{updates.rebootRequired && <span className="badge bg-orange-lt text-orange">restart needed</span>}
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -22,6 +22,7 @@ import {
|
|||||||
IconWorldWww,
|
IconWorldWww,
|
||||||
IconListCheck,
|
IconListCheck,
|
||||||
IconRefresh,
|
IconRefresh,
|
||||||
|
IconDownload,
|
||||||
IconActivityHeartbeat,
|
IconActivityHeartbeat,
|
||||||
IconShieldCheck,
|
IconShieldCheck,
|
||||||
IconTicket,
|
IconTicket,
|
||||||
@@ -101,6 +102,7 @@ export const NAV: NavEntry[] = [
|
|||||||
items: [
|
items: [
|
||||||
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
|
{ to: "/alerts", label: "Alerts", icon: <IconAlertTriangle size={20} /> },
|
||||||
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
|
{ to: "/maintenance", label: "Maintenance", icon: <IconTool size={20} /> },
|
||||||
|
{ to: "/updates", label: "Updates", icon: <IconDownload size={20} /> },
|
||||||
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
{ to: "/uptime-kuma", label: "Uptime Kuma", icon: <IconActivityHeartbeat size={20} /> },
|
||||||
{ to: "/osticket", label: "osTicket", icon: <IconTicket size={20} /> },
|
{ to: "/osticket", label: "osTicket", icon: <IconTicket size={20} /> },
|
||||||
{ to: "/admin-links", label: "Admin Links", icon: <IconExternalLink size={20} /> },
|
{ to: "/admin-links", label: "Admin Links", icon: <IconExternalLink size={20} /> },
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ export const GO_SHORTCUTS: GoShortcut[] = [
|
|||||||
{ key: "a", to: "/alerts", label: "Alerts" },
|
{ key: "a", to: "/alerts", label: "Alerts" },
|
||||||
{ key: "s", to: "/servers", label: "Servers" },
|
{ key: "s", to: "/servers", label: "Servers" },
|
||||||
{ key: "m", to: "/maintenance", label: "Maintenance" },
|
{ key: "m", to: "/maintenance", label: "Maintenance" },
|
||||||
|
{ key: "u", to: "/updates", label: "Updates" },
|
||||||
{ key: "i", to: "/integrations", label: "Integrations" },
|
{ key: "i", to: "/integrations", label: "Integrations" },
|
||||||
{ key: "k", to: "/secrets", label: "Secrets" },
|
{ key: "k", to: "/secrets", label: "Secrets" },
|
||||||
{ key: "p", to: "/map", label: "Map" },
|
{ key: "p", to: "/map", label: "Map" },
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import { readableError } from "../utils/errors";
|
|||||||
import { ErrorNotice } from "../components/Notice";
|
import { ErrorNotice } from "../components/Notice";
|
||||||
import TableToolbar from "../components/TableToolbar";
|
import TableToolbar from "../components/TableToolbar";
|
||||||
import SilenceButton from "../components/SilenceButton";
|
import SilenceButton from "../components/SilenceButton";
|
||||||
|
import RunPlaybookButton from "../components/RunPlaybookButton";
|
||||||
|
|
||||||
const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: number }> = {
|
const SEVERITY: Record<AlertSeverity, { label: string; badge: string; rank: number }> = {
|
||||||
critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 },
|
critical: { label: "Critical", badge: "bg-red-lt text-red", rank: 0 },
|
||||||
@@ -89,6 +90,7 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
{ label: "Warnings", value: counts?.warning, color: "text-yellow" },
|
{ label: "Warnings", value: counts?.warning, color: "text-yellow" },
|
||||||
{ label: "Info", value: counts?.info, color: "text-blue" },
|
{ label: "Info", value: counts?.info, color: "text-blue" },
|
||||||
{ label: "Silenced (maintenance)", value: counts?.silenced, color: "text-secondary" },
|
{ label: "Silenced (maintenance)", value: counts?.silenced, color: "text-secondary" },
|
||||||
|
...((counts?.grouped ?? 0) > 0 ? [{ label: "Grouped under another", value: counts?.grouped, color: "text-secondary" }] : []),
|
||||||
];
|
];
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -196,10 +198,10 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
</option>
|
</option>
|
||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
{(counts?.silenced ?? 0) > 0 && (
|
{(counts?.silenced ?? 0) + (counts?.grouped ?? 0) > 0 && (
|
||||||
<label className="form-check mb-0">
|
<label className="form-check mb-0">
|
||||||
<input type="checkbox" className="form-check-input" checked={showSilenced} onChange={(e) => setShowSilenced(e.target.checked)} />
|
<input type="checkbox" className="form-check-input" checked={showSilenced} onChange={(e) => setShowSilenced(e.target.checked)} />
|
||||||
<span className="form-check-label">Show silenced</span>
|
<span className="form-check-label">{(counts?.grouped ?? 0) > 0 ? "Show silenced and grouped" : "Show silenced"}</span>
|
||||||
</label>
|
</label>
|
||||||
)}
|
)}
|
||||||
</>
|
</>
|
||||||
@@ -230,10 +232,21 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
<td>{a.link ? <Link to={a.link}>{a.source}</Link> : a.source}</td>
|
<td>{a.link ? <Link to={a.link}>{a.source}</Link> : a.source}</td>
|
||||||
<td>
|
<td>
|
||||||
{a.message}
|
{a.message}
|
||||||
{a.silenced && (
|
{a.causedBy ? (
|
||||||
|
<span className="badge bg-secondary-lt text-secondary ms-2" title={`${a.causedBy.name} is down too, which explains this — notifications for this are held back. If ${a.causedBy.name} comes back and this stays down, it alerts then.`}>
|
||||||
|
caused by {a.causedBy.name}
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
a.silenced && (
|
||||||
<span className="badge bg-secondary-lt text-secondary ms-2" title="Under a maintenance window — notifications for this are held back">
|
<span className="badge bg-secondary-lt text-secondary ms-2" title="Under a maintenance window — notifications for this are held back">
|
||||||
silenced
|
silenced
|
||||||
</span>
|
</span>
|
||||||
|
)
|
||||||
|
)}
|
||||||
|
{a.affects && a.affects.length > 0 && (
|
||||||
|
<div className="text-secondary small">
|
||||||
|
Also takes down {a.affects.length} {a.affects.length === 1 ? "server" : "servers"}: {a.affects.join(", ")}
|
||||||
|
</div>
|
||||||
)}
|
)}
|
||||||
{a.ticket &&
|
{a.ticket &&
|
||||||
(() => {
|
(() => {
|
||||||
@@ -252,7 +265,10 @@ export default function Alerts({ user }: { user: CurrentUser }) {
|
|||||||
</td>
|
</td>
|
||||||
{canSilence && (
|
{canSilence && (
|
||||||
<td className="text-nowrap">
|
<td className="text-nowrap">
|
||||||
|
<div className="btn-list flex-nowrap">
|
||||||
<SilenceButton alert={a} onDone={() => void load(true)} />
|
<SilenceButton alert={a} onDone={() => void load(true)} />
|
||||||
|
{a.target?.type === "server" && !a.silenced && <RunPlaybookButton serverName={a.target.name} />}
|
||||||
|
</div>
|
||||||
</td>
|
</td>
|
||||||
)}
|
)}
|
||||||
</tr>
|
</tr>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ import { ErrorNotice } from "../components/Notice";
|
|||||||
import { toast } from "../utils/toast";
|
import { toast } from "../utils/toast";
|
||||||
import { confirmDialog } from "../utils/dialogs";
|
import { confirmDialog } from "../utils/dialogs";
|
||||||
import MaintenanceTimeline from "../components/MaintenanceTimeline";
|
import MaintenanceTimeline from "../components/MaintenanceTimeline";
|
||||||
|
import RecurringMaintenance from "../components/RecurringMaintenance";
|
||||||
|
|
||||||
const DURATIONS: { minutes: number; label: string }[] = [
|
const DURATIONS: { minutes: number; label: string }[] = [
|
||||||
{ minutes: 30, label: "30 minutes" },
|
{ minutes: 30, label: "30 minutes" },
|
||||||
@@ -326,6 +327,15 @@ export default function Maintenance({ user }: { user: CurrentUser }) {
|
|||||||
|
|
||||||
{((windows?.length ?? 0) > 0 || upcoming.length > 0) && <MaintenanceTimeline windows={[...(windows ?? []), ...upcoming]} now={now} />}
|
{((windows?.length ?? 0) > 0 || upcoming.length > 0) && <MaintenanceTimeline windows={[...(windows ?? []), ...upcoming]} now={now} />}
|
||||||
|
|
||||||
|
<RecurringMaintenance
|
||||||
|
canEdit={canEdit}
|
||||||
|
targets={targets}
|
||||||
|
onChanged={() => {
|
||||||
|
load();
|
||||||
|
announceChange();
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
|
||||||
{canEdit && kumaIntegrations && kumaIntegrations.length > 0 && (
|
{canEdit && kumaIntegrations && kumaIntegrations.length > 0 && (
|
||||||
<div className="card mb-3">
|
<div className="card mb-3">
|
||||||
<div className="card-header">
|
<div className="card-header">
|
||||||
|
|||||||
@@ -17,6 +17,9 @@ import ServerTrends from "../components/ServerTrends";
|
|||||||
import ServerActivity from "../components/ServerActivity";
|
import ServerActivity from "../components/ServerActivity";
|
||||||
import TimeAgo from "../components/TimeAgo";
|
import TimeAgo from "../components/TimeAgo";
|
||||||
import UptimeRibbon from "../components/UptimeRibbon";
|
import UptimeRibbon from "../components/UptimeRibbon";
|
||||||
|
import RunsOn from "../components/RunsOn";
|
||||||
|
import RunPlaybook from "../components/RunPlaybook";
|
||||||
|
import UpdatesBadge from "../components/UpdatesBadge";
|
||||||
import ServerTags from "../components/ServerTags";
|
import ServerTags from "../components/ServerTags";
|
||||||
import ServerTaskTable, { SCHEDULE_TYPE_LABELS } from "../components/ServerTaskTable";
|
import ServerTaskTable, { SCHEDULE_TYPE_LABELS } from "../components/ServerTaskTable";
|
||||||
import { formatDateTime } from "../utils/date";
|
import { formatDateTime } from "../utils/date";
|
||||||
@@ -755,6 +758,36 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
|
|||||||
<dd className="col-7">
|
<dd className="col-7">
|
||||||
<UptimeRibbon kind="servers" id={server.id} name={server.name} />
|
<UptimeRibbon kind="servers" id={server.id} name={server.name} />
|
||||||
</dd>
|
</dd>
|
||||||
|
<dt className="col-5">Runs on</dt>
|
||||||
|
<dd className="col-7">
|
||||||
|
<RunsOn serverId={server.id} parent={detail.relations.parent} canEdit={isAdmin} onChanged={() => void loadDetail()} />
|
||||||
|
</dd>
|
||||||
|
{detail.relations.children.length > 0 && (
|
||||||
|
<>
|
||||||
|
<dt className="col-5">Runs here</dt>
|
||||||
|
<dd className="col-7">
|
||||||
|
{detail.relations.children.map((c, i) => (
|
||||||
|
<span key={c.id}>
|
||||||
|
{i > 0 && ", "}
|
||||||
|
<Link to={`/servers/${c.id}`}>{c.name}</Link>
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</dd>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{server.lastSeenAt && (
|
||||||
|
<>
|
||||||
|
<dt className="col-5">Updates</dt>
|
||||||
|
<dd className="col-7">
|
||||||
|
<UpdatesBadge updates={server.pendingUpdates} />
|
||||||
|
{server.updatesPendingSince && server.pendingUpdates && server.pendingUpdates.count > 0 && (
|
||||||
|
<div className="text-secondary small">
|
||||||
|
waiting since <TimeAgo value={server.updatesPendingSince} />
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</dd>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
</dl>
|
</dl>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -763,6 +796,16 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
|
|||||||
|
|
||||||
{hardware.source === "agent" && <ServerTrends serverId={serverId} />}
|
{hardware.source === "agent" && <ServerTrends serverId={serverId} />}
|
||||||
{(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />}
|
{(user.role === "admin" || user.role === "operator") && <ServerActivity serverId={serverId} />}
|
||||||
|
{canEditTasks && (
|
||||||
|
<div className="card mb-3">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Run a playbook</h3>
|
||||||
|
</div>
|
||||||
|
<div className="card-body">
|
||||||
|
<RunPlaybook serverName={server.name} defaultLimit={server.name} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
<ServerPorts
|
<ServerPorts
|
||||||
serverId={serverId}
|
serverId={serverId}
|
||||||
|
|||||||
@@ -0,0 +1,244 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
import { api, type DockhandContainer, type UpdatesServer } from "../api/client";
|
||||||
|
import SortableTh from "../components/SortableTh";
|
||||||
|
import TableToolbar from "../components/TableToolbar";
|
||||||
|
import TimeAgo from "../components/TimeAgo";
|
||||||
|
import UpdatesBadge from "../components/UpdatesBadge";
|
||||||
|
import { BlockSkeleton } from "../components/Skeleton";
|
||||||
|
import { ErrorNotice } from "../components/Notice";
|
||||||
|
import { useSortable } from "../hooks/useSortable";
|
||||||
|
import { useTableSearch } from "../hooks/useTableSearch";
|
||||||
|
import { downloadCsv } from "../utils/csv";
|
||||||
|
import { readableError } from "../utils/errors";
|
||||||
|
|
||||||
|
interface Row {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
os: string;
|
||||||
|
manager: string;
|
||||||
|
/** Null: the agent hasn't said. */
|
||||||
|
count: number | null;
|
||||||
|
security: number | null;
|
||||||
|
restart: boolean;
|
||||||
|
/** ISO — when updates were first seen waiting. */
|
||||||
|
since: string | null;
|
||||||
|
reportedAt: string | null;
|
||||||
|
server: UpdatesServer;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toRow(s: UpdatesServer): Row {
|
||||||
|
return {
|
||||||
|
id: s.id,
|
||||||
|
name: s.name,
|
||||||
|
os: s.osType,
|
||||||
|
manager: s.updates?.manager ?? "",
|
||||||
|
count: s.updates ? s.updates.count : null,
|
||||||
|
security: s.updates ? s.updates.security : null,
|
||||||
|
restart: s.updates?.rebootRequired ?? false,
|
||||||
|
since: s.pendingSince,
|
||||||
|
reportedAt: s.updates?.reportedAt || null,
|
||||||
|
server: s,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const DAY_MS = 86_400_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What is waiting to be installed, in one place: the updates each server's agent says are pending (how many, how many are security fixes,
|
||||||
|
* whether a restart is due, and how long they have been waiting) and the Docker containers that have a newer image. It only looks —
|
||||||
|
* nothing here installs anything.
|
||||||
|
*/
|
||||||
|
export default function Updates() {
|
||||||
|
const [servers, setServers] = useState<UpdatesServer[] | null>(null);
|
||||||
|
const [containers, setContainers] = useState<(DockhandContainer & { integrationName: string })[] | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [onlyWaiting, setOnlyWaiting] = useState(false);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let alive = true;
|
||||||
|
api.servers
|
||||||
|
.updates()
|
||||||
|
.then((res) => alive && setServers(res.servers))
|
||||||
|
.catch((err) => alive && setError(readableError(err)));
|
||||||
|
api.integrations
|
||||||
|
.list()
|
||||||
|
.then(async (res) => {
|
||||||
|
const docker = res.integrations.filter((i) => i.type === "dockhand" && i.enabled);
|
||||||
|
const all = await Promise.all(
|
||||||
|
docker.map((i) =>
|
||||||
|
api.integrations.dockhand
|
||||||
|
.containers(i.id)
|
||||||
|
.then((r) => r.containers.map((c) => ({ ...c, integrationName: i.name })))
|
||||||
|
.catch(() => []),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
if (alive) setContainers(docker.length === 0 ? null : all.flat().filter((c) => c.updateAvailable === true));
|
||||||
|
})
|
||||||
|
.catch(() => {});
|
||||||
|
return () => {
|
||||||
|
alive = false;
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const rows = (servers ?? []).map(toRow);
|
||||||
|
const waiting = rows.filter((r) => (r.count ?? 0) > 0);
|
||||||
|
const tableSearch = useTableSearch(onlyWaiting ? waiting : rows);
|
||||||
|
const { sorted, sortKey, sortDir, requestSort } = useSortable(tableSearch.filtered, "security", "desc", { url: true });
|
||||||
|
|
||||||
|
const reporting = rows.filter((r) => r.count !== null).length;
|
||||||
|
const securityTotal = rows.reduce((n, r) => n + (r.security ?? 0), 0);
|
||||||
|
const restarts = rows.filter((r) => r.restart).length;
|
||||||
|
const oldest = waiting.map((r) => r.since).filter((s): s is string => !!s).sort()[0];
|
||||||
|
const notReporting = rows.length - reporting;
|
||||||
|
|
||||||
|
function exportCsv() {
|
||||||
|
if (!sorted) return;
|
||||||
|
downloadCsv(
|
||||||
|
"updates.csv",
|
||||||
|
["Server", "Package manager", "Waiting", "Security", "Restart needed", "Waiting since", "Reported"],
|
||||||
|
sorted.map((r) => [r.name, r.manager, r.count ?? "", r.security ?? "", r.restart ? "yes" : "no", r.since ?? "", r.reportedAt ?? ""]),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const cards: { label: string; value: string | number; note?: string; color: string }[] = [
|
||||||
|
{ label: "Servers with updates waiting", value: servers ? `${waiting.length} of ${reporting}` : "…", note: notReporting > 0 ? `${notReporting} haven't reported updates` : undefined, color: waiting.length > 0 ? "text-yellow" : "text-green" },
|
||||||
|
{ label: "Security updates waiting", value: servers ? securityTotal : "…", color: securityTotal > 0 ? "text-red" : "text-green" },
|
||||||
|
{ label: "Need a restart", value: servers ? restarts : "…", color: restarts > 0 ? "text-orange" : "text-green" },
|
||||||
|
{ label: "Longest wait", value: oldest ? `${Math.max(1, Math.floor((Date.now() - Date.parse(oldest)) / DAY_MS))} d` : servers ? "—" : "…", color: "text-secondary" },
|
||||||
|
];
|
||||||
|
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div className="d-flex align-items-center mb-3">
|
||||||
|
<h2 className="page-title mb-0">Updates</h2>
|
||||||
|
</div>
|
||||||
|
<p className="text-secondary">
|
||||||
|
What the agents say is waiting to be installed on each server, and the Docker containers that have a newer image. This page only looks — it installs nothing. To apply updates, run a playbook from the server's page.
|
||||||
|
</p>
|
||||||
|
<ErrorNotice error={error} />
|
||||||
|
|
||||||
|
<div className="row row-cards mb-3">
|
||||||
|
{cards.map((c) => (
|
||||||
|
<div className="col-6 col-lg-3" key={c.label}>
|
||||||
|
<div className="card card-sm">
|
||||||
|
<div className="card-body">
|
||||||
|
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
|
||||||
|
<div className="text-secondary">{c.label}</div>
|
||||||
|
{c.note && <div className="text-secondary small">{c.note}</div>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="card mb-3">
|
||||||
|
<TableToolbar
|
||||||
|
search={tableSearch.query}
|
||||||
|
onSearch={tableSearch.setQuery}
|
||||||
|
placeholder="Search servers…"
|
||||||
|
filters={
|
||||||
|
<label className="form-check mb-0">
|
||||||
|
<input type="checkbox" className="form-check-input" checked={onlyWaiting} onChange={(e) => setOnlyWaiting(e.target.checked)} />
|
||||||
|
<span className="form-check-label">Only those with updates</span>
|
||||||
|
</label>
|
||||||
|
}
|
||||||
|
shown={sorted?.length}
|
||||||
|
total={onlyWaiting ? waiting.length : rows.length}
|
||||||
|
onExport={exportCsv}
|
||||||
|
exportDisabled={!sorted || sorted.length === 0}
|
||||||
|
/>
|
||||||
|
<div className="table-responsive">
|
||||||
|
<table className="table table-vcenter card-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<SortableTh<Row> label="Server" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Updates" sortKeyName="count" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Security" sortKeyName="security" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Waiting since" sortKeyName="since" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
<SortableTh<Row> label="Reported" sortKeyName="reportedAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{servers === null && !error && (
|
||||||
|
<tr>
|
||||||
|
<td colSpan={5}>
|
||||||
|
<BlockSkeleton lines={3} />
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)}
|
||||||
|
{sorted?.map((r) => (
|
||||||
|
<tr key={r.id}>
|
||||||
|
<td>
|
||||||
|
<Link to={`/servers/${r.id}`}>{r.name}</Link> <span className="text-secondary small">{r.manager}</span>
|
||||||
|
</td>
|
||||||
|
<td>
|
||||||
|
<UpdatesBadge updates={r.server.updates} />
|
||||||
|
</td>
|
||||||
|
<td>{r.security === null ? <span className="text-secondary">—</span> : r.security}</td>
|
||||||
|
<td className="text-secondary">{r.since ? <TimeAgo value={r.since} /> : "—"}</td>
|
||||||
|
<td className="text-secondary">{r.reportedAt ? <TimeAgo value={r.reportedAt} /> : "—"}</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{servers !== null && sorted?.length === 0 && (
|
||||||
|
<tr>
|
||||||
|
<td colSpan={5} className="text-secondary text-center">
|
||||||
|
{tableSearch.active || onlyWaiting ? "Nothing matches that." : "No server has reported yet."}
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
{notReporting > 0 && (
|
||||||
|
<div className="card-footer text-secondary small">
|
||||||
|
“Not reported” means that server's agent hasn't said — it may be an older agent (run its install command again, from Servers → Manage servers, to update it), or its package manager isn't apt, dnf, zypper or Windows Update.
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{containers !== null && (
|
||||||
|
<div className="card">
|
||||||
|
<div className="card-header">
|
||||||
|
<h3 className="card-title">Docker containers with a newer image</h3>
|
||||||
|
</div>
|
||||||
|
<div className="table-responsive">
|
||||||
|
<table className="table table-vcenter card-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Container</th>
|
||||||
|
<th>Image</th>
|
||||||
|
<th>Newer version</th>
|
||||||
|
<th>Where</th>
|
||||||
|
<th>Checked</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{containers.map((c) => (
|
||||||
|
<tr key={`${c.integrationName}:${c.environmentId}:${c.id}`}>
|
||||||
|
<td>
|
||||||
|
<Link to="/docker">{c.name}</Link>
|
||||||
|
</td>
|
||||||
|
<td className="text-secondary">{c.image}</td>
|
||||||
|
<td>{c.newerVersion ?? "—"}</td>
|
||||||
|
<td className="text-secondary">{c.environmentName}</td>
|
||||||
|
<td className="text-secondary">
|
||||||
|
<TimeAgo value={c.checkedAt} />
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{containers.length === 0 && (
|
||||||
|
<tr>
|
||||||
|
<td colSpan={5} className="text-secondary text-center">
|
||||||
|
Every container is on its latest image, as far as the last check could tell.
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -671,6 +671,11 @@ main:focus {
|
|||||||
background-image: repeating-linear-gradient(135deg, rgba(255, 255, 255, 0.45) 0 3px, transparent 3px 6px);
|
background-image: repeating-linear-gradient(135deg, rgba(255, 255, 255, 0.45) 0 3px, transparent 3px 6px);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Windows made from a recurring schedule get vertical stripes. */
|
||||||
|
.timeline-bar-recurring {
|
||||||
|
background-image: repeating-linear-gradient(90deg, rgba(255, 255, 255, 0.5) 0 2px, transparent 2px 5px);
|
||||||
|
}
|
||||||
|
|
||||||
/* ── The wall display: a full-screen status board, big enough to read across a room. ── */
|
/* ── The wall display: a full-screen status board, big enough to read across a room. ── */
|
||||||
.wall {
|
.wall {
|
||||||
display: flex;
|
display: flex;
|
||||||
|
|||||||
Reference in new issue
Block a user