diff --git a/AGENT_UPDATE.md b/AGENT_UPDATE.md index c01b80a..4627d44 100644 --- a/AGENT_UPDATE.md +++ b/AGENT_UPDATE.md @@ -26,6 +26,7 @@ number when it is worth installing the agent again for. | 3 (Windows) | Can replace itself with the newer script when automatic updates are switched on (below). | | 4 (Linux) | Also reports failed services, disk health (SMART) and ZFS pool state; reads `pacman`; and can replace itself when automatic updates are switched on. | | 4 (Windows) | Also reports failed services, disk health and Storage Spaces pools (below), and what the machine is (maker, model, serial number). | +| 6 (Linux), 5 (Windows) | Also says **what it couldn't collect and why** (no `smartctl`, no systemd, a package manager that didn't answer, Windows Update that timed out …), shown on the server's page with what to do about it (below). | | 5 (Linux) | Also says what the machine is — maker, model and serial number, read from the firmware — so the inventory can fill itself in (below). | Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field @@ -208,10 +209,29 @@ The same three parts, from what Windows itself knows (the agent runs as SYSTEM, | What | Where it comes from | Notes | |---|---|---| -| **Failed services** | `Win32_Service` | A service that starts by itself, isn't running, and stopped with an error. Services that finished normally (exit code 0), that start on a trigger (1077) or that are starting up are not failures. Some services always do this (updaters, for one): list them under `ignoreServices` in `agent.json` (or `IGNORE_SERVICES` in the environment), as names or patterns like `gupdate*`. | +| **Failed services** | `Win32_Service` | A service that starts by itself, isn't running, and stopped with an error. Services that finished normally (exit code 0), that start on a trigger (1077) or that are starting up are not failures. Some services always do this (updaters, for one): list them under `ignoreServices` in `agent.json` (or `IGNORE_SERVICES` in the environment), as names or patterns like `gupdate*` — or, easier and for either system, on the server's page (below). | | **Disk health** | `Get-PhysicalDisk` and its reliability counters | Windows' own verdict: *Healthy* passes, *Warning* and *Unhealthy* fail, anything else says nothing. Temperature and wear come from the counters where the drive gives sane ones. Virtual disks are left out. | | **Storage pools** | `Get-StoragePool` (Storage Spaces) | *Healthy*, *Warning* and *Unhealthy* are reported as ONLINE, DEGRADED and FAULTED, with how full the pool is, and show beside ZFS pools. A machine with no pools reports an empty list. | +## Services to ignore, from the server's page + +A service that always shows as failed on one machine (an updater, a unit that is meant to be stopped) can be ignored there: on the server's **Host health** +card an administrator presses **Ignore** beside it, or types a name or a pattern with `*` (`gupdate*`, `snap.*`; case doesn't matter and `.service` is +optional). It stops counting as failed — no alert, not in `/metrics` — the moment it is saved, with no agent to update, and it still shows on the page, +marked *ignored*, with a button to stop ignoring it. The list belongs to that server (it is kept with its own alert limits), and works the same for Linux and +Windows; the Windows agent's own `ignoreServices` still works as well. + +## What the agent couldn't collect (Linux version 6, Windows version 5) + +Each agent notes, in every report, the checks it couldn't make: nothing there to read it with (*unavailable*) or it tried and couldn't (*failed*), why in a +word, and for a failure at most a line of what the tool said. The server's page turns that into a card, **What the agent couldn't collect**, with words and a +fix for each — "Disk health isn't reported: smartctl isn't installed → install smartmontools" — so the question *why is there no disk health here?* has an +answer on the page. A tip is simply how the machine is (a virtual machine with no serial number); *look at* is something to fix. A machine with no ZFS isn't +told it has no ZFS. The card also says when the agent is behind. The wording lives in the server, so it improves without installing anything again. + +On Linux, the step outside the sandbox (`--prepare`) leaves its notes with the other answers, and the report carries them; if that step didn't run at all +(an older systemd unit, or a copy run by hand) that is itself noted, because disk health and ZFS are only read there. `--check-updates` prints the same notes. + ## What the machine is (Linux version 5, Windows version 4) The agents say who made the machine, its model and its serial number (Linux reads `/sys/class/dmi/id`, where the serial is only diff --git a/DATABASE.md b/DATABASE.md index 6df3c7f..c07fb96 100644 --- a/DATABASE.md +++ b/DATABASE.md @@ -223,9 +223,10 @@ what it last reported. | `no_parent` | bool NOT NULL, default 0 | An admin said this server runs on nothing: no link, and no guess from its Proxmox node. Choosing a server by hand, or going back to automatic, clears it. | | `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. | | `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. | -| `thresholds` | text | This server's own alert limits as JSON: `{"serverOfflineMinutes":…,"diskUsagePercent":…,"diskForecastDays":…,"backupStaleHours":…,"mounts":{"/data":{"percent":95,"ignore":true}}}`. Anything left out follows Settings; null when it follows Settings entirely. Ranges are checked on the way in (offline 15–10080 min, disk 50–99 %, forecast 0–90 days, backup 0–2160 h; 0 means off). | +| `thresholds` | text | This server's own alert limits as JSON: `{"serverOfflineMinutes":…,"diskUsagePercent":…,"diskForecastDays":…,"backupStaleHours":…,"mounts":{"/data":{"percent":95,"ignore":true}},"ignoreServices":["gupdate*"]}`. `ignoreServices` are services never counted as failed here (names or patterns with `*`, up to 30). Anything left out follows Settings; null when it follows Settings entirely. Ranges are checked on the way in (offline 15–10080 min, disk 50–99 %, forecast 0–90 days, backup 0–2160 h; 0 means off). | | `host_health` | text | JSON `{"failedUnits":[{unit,description}],"smart":[{device,model,healthy,temperatureC,wearPercent}],"zfs":[{name,health,capacityPercent}],"reportedAt":"…"}`. Each part is present only if the agent reported it, and a report that leaves one out leaves it as it was. Null until an agent says anything. Judged by the health check and shown on the server's page. | | `hw_manufacturer`, `hw_model`, `hw_serial` | text | What the machine is, as its firmware says (the agent reports it; placeholders like "To Be Filled By O.E.M." are left out). Each is replaced only when a report has a value, so an agent that can't tell doesn't wipe what it said. Used to fill in the inventory. | +| `agent_diagnostics` | text | JSON `{"items":[{check,status,reason?,detail?}],"reportedAt":"…"}`: what the agent says it could not collect (`status` is `unavailable` or `failed`). Replaced by each report that carries the list (an empty list means nothing was missed) and kept when a report doesn't. Turned into words and fixes by the server; never sent to the browser as it is. | | `agent_version` | integer | Which version of the agent script runs on it, as the agent says in each report. Null until it says; an agent that predates the number counts as version 1. Compared with the scripts the app serves to say which agents are behind. | #### `scheduled_tasks` diff --git a/README.md b/README.md index 2567628..d8ddb0b 100644 --- a/README.md +++ b/README.md @@ -572,7 +572,9 @@ Proxmox guest) how old its backup may be — and per disk, either a limit or "no for this disk". Whatever is left empty follows Settings, so changing the global limit still reaches every server that hasn't said otherwise. They apply everywhere the global ones did: the alerts, the notifications, the dashboard, the map, the forecast, the -digest and `/metrics`. +digest and `/metrics`. The same place keeps the **services to ignore** on that server — an updater that always shows as failed, say — which can also be set from its +Host health card; and a card, **What the agent couldn't collect**, tells you why a figure is missing (no `smartctl`, an update check that didn't answer …) and what +to do. See AGENT_UPDATE.md. **Inventory** (Infrastructure → Inventory) keeps the hardware behind it all: name and kind, maker, model, serial, where it sits (and rack unit), when it was bought and for diff --git a/ROLES.md b/ROLES.md index 64b10bb..4d42fbc 100644 --- a/ROLES.md +++ b/ROLES.md @@ -105,7 +105,8 @@ even further, to admin only: - **Database backups** (Settings → Backup: making, downloading, uploading, checking, restoring and deleting them, and the schedule) are administrator-only: a backup holds the audit log, every server's address, the hashed agent tokens and all the stored credentials (encrypted). -- **Alert limits** on a server's page: everyone can see the ones a server has of its own; only an administrator changes them. +- **Alert limits** on a server's page: everyone can see the ones a server has of its own; only an administrator changes them. The same goes for the **services to + ignore** on its Host health card. Everyone sees *What the agent couldn't collect*. - **Host health** (failed services, disk health, ZFS pools) is shown to everyone on a server's page; **Update agents automatically** on the Servers page is admin-only. - **Sign-in groups, metrics tokens, event webhooks, share links (the wall display and the status pages, and what is on them) and the diff --git a/agent/linux/report-tasks.sh b/agent/linux/report-tasks.sh index 869b8a5..2913ea4 100644 --- a/agent/linux/report-tasks.sh +++ b/agent/linux/report-tasks.sh @@ -16,7 +16,8 @@ set -euo pipefail # 3 = the package manager is asked outside the sandbox (see --prepare), so dnf and zypper can answer there too # 4 = also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can update itself when told to # 5 = also says what the machine is (maker, model, serial number), from the firmware, for the inventory -AGENT_VERSION=5 +# 6 = also says what it could not collect and why (no smartctl, no systemd, a package manager that failed ...), so the server can say what to fix +AGENT_VERSION=6 ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}" if [[ -f "$ENV_FILE" ]]; then @@ -44,6 +45,8 @@ case "${1:-}" in esac UPDATES_FILE="${UPDATES_FILE:-/run/homelab-manager-agent/updates.json}" HEALTH_FILE="${HEALTH_FILE:-$(dirname "$UPDATES_FILE")/health.json}" +# What --prepare could not collect and why, left for the report to carry along. +DIAG_FILE="${DIAG_FILE:-$(dirname "$UPDATES_FILE")/diagnostics.json}" # Kept between runs (the unit's private directory is not): a SMART reading is only taken about once an hour, so sleeping disks are left alone. STATE_DIR="${STATE_DIR:-/var/lib/homelab-manager-agent}" # Where the package managers' own error output goes: nowhere, unless a person asked to see it. @@ -55,6 +58,23 @@ if [[ "$MODE" == "report" && ( -z "$API_URL" || -z "$API_TOKEN" ) ]]; then exit 1 fi +# Things that could not be collected, each a small JSON object: which check, whether it is "unavailable" (nothing to read it with, or nothing +# there to read) or "failed" (it tried and could not), why in a word, and at most a line of what the tool said. Only problems are noted: a +# check that is not mentioned went fine. The server turns these into what to do about them. +DIAG_ITEMS=() +diag() { + local check="$1" status="$2" reason="${3:-}" detail="${4:-}" + detail=$(printf '%s' "$detail" | tr -d '\000-\037' | head -c 160) + DIAG_ITEMS+=("$(jq -n -c --arg c "$check" --arg s "$status" --arg r "$reason" --arg d "$detail" \ + '{check: $c, status: $s} + (if $r != "" then {reason: $r} else {} end) + (if $d != "" then {detail: $d} else {} end)')") +} + +# The notes made so far as one JSON array ([] when there are none). +diag_json() { + if (( ${#DIAG_ITEMS[@]} == 0 )); then echo "[]"; return 0; fi + printf '%s\n' "${DIAG_ITEMS[@]}" | jq -s -c '.' +} + CURL_INSECURE_FLAG=() case "${API_INSECURE,,}" in 1|true|yes) CURL_INSECURE_FLAG=(-k) ;; @@ -253,9 +273,12 @@ collect_updates() { [[ -f /run/reboot-needed ]] && reboot="true" fi - if [[ -n "$manager" && ! "$count" =~ ^[0-9]+$ ]]; then + if [[ -z "$manager" ]]; then + diag updates unavailable no_package_manager + elif [[ ! "$count" =~ ^[0-9]+$ ]]; then # In the journal (systemctl status / journalctl -u homelab-manager-agent) so "not reported" has a reason to look for. echo "Note: couldn't read the pending updates from $manager - the report goes without them." >&2 + diag updates failed error "$manager did not answer" fi if [[ -n "$manager" && "$count" =~ ^[0-9]+$ ]]; then @@ -325,7 +348,10 @@ collect_hardware() { maker=$(dmi_value sys_vendor board_vendor) model=$(dmi_value product_name board_name) serial=$(dmi_value product_serial board_serial) - [[ -n "$maker$model$serial" ]] || return 0 + if [[ -z "$maker$model$serial" ]]; then + diag hardware unavailable not_found + return 0 + fi HARDWARE_JSON=$(jq -n -c --arg maker "$maker" --arg model "$model" --arg serial "$serial" \ '{} | (if $maker != "" then . + {manufacturer: $maker} else . end) | (if $model != "" then . + {model: $model} else . end) | (if $serial != "" then . + {serial: $serial} else . end)') || HARDWARE_JSON="null" } @@ -333,9 +359,15 @@ collect_hardware() { # Services (systemd units) that have failed. systemctl is asked the way the timers are, so this works inside the sandbox. collect_failed_units() { FAILED_UNITS_JSON="null" - command -v systemctl >/dev/null 2>&1 || return 0 + if ! command -v systemctl >/dev/null 2>&1; then + diag failed_units unavailable no_systemd + return 0 + fi local out - out=$(timeout 20 systemctl list-units --state=failed --plain --no-legend --no-pager --full 2>"$PM_ERR") || return 0 + if ! out=$(timeout 20 systemctl list-units --state=failed --plain --no-legend --no-pager --full 2>"$PM_ERR"); then + diag failed_units failed error "systemctl list-units did not answer" + return 0 + fi FAILED_UNITS_JSON=$(awk 'NF > 0 { unit = $1; $1 = $2 = $3 = $4 = ""; sub(/^ +/, ""); print unit "\t" $0 }' <<<"$out" \ | jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({unit: .[0], description: (.[1] // "")}) | .[0:100]') || FAILED_UNITS_JSON="null" } @@ -344,7 +376,14 @@ collect_failed_units() { # --prepare. Asked at most about once an hour, and with -n standby so a sleeping disk is not woken: what was last known of it is kept. collect_smart() { SMART_JSON="null" - command -v smartctl >/dev/null 2>&1 || return 0 + if ! command -v smartctl >/dev/null 2>&1; then + diag smart unavailable tool_missing smartctl + return 0 + fi + if [[ "$(id -u)" != "0" ]]; then + diag smart unavailable not_root + return 0 + fi local cache="$STATE_DIR/smart.json" now_s previous="[]" items="[]" dev out item scan checked now_s=$(date +%s) if [[ -r "$cache" ]]; then @@ -383,9 +422,15 @@ collect_smart() { # Each ZFS pool and whether it is ONLINE. Needs /dev/zfs, so it runs from --prepare too. collect_zfs() { ZFS_JSON="null" - command -v zpool >/dev/null 2>&1 || return 0 + if ! command -v zpool >/dev/null 2>&1; then + diag zfs unavailable tool_missing zpool + return 0 + fi local out - out=$(timeout 20 zpool list -H -o name,health,capacity 2>"$PM_ERR") || return 0 + if ! out=$(timeout 20 zpool list -H -o name,health,capacity 2>"$PM_ERR"); then + diag zfs failed error "zpool list did not answer" + return 0 + fi ZFS_JSON=$(jq -R -s -c 'split("\n") | map(select(length > 0) | split("\t")) | map({name: .[0], health: (.[1] // "UNKNOWN"), capacity_percent: ((.[2] // "" | rtrimstr("%") | tonumber?) // null)})' <<<"$out") || ZFS_JSON="null" } @@ -466,6 +511,22 @@ use_prepared_updates() { UPDATES_JSON=$json } +# What this report could not collect: the notes made here, then those --prepare left (it does the outside-the-sandbox part), one for each check. +# With nothing left by --prepare — on a real run, not a dry one — that step did not run, which is worth saying: disk health and ZFS are only read there. +collect_diagnostics() { + local from_prepare="[]" shaped + if [[ -r "$DIAG_FILE" && -n "$(find "$DIAG_FILE" -mmin -10 -print 2>/dev/null)" ]]; then + from_prepare=$(jq -c 'if type == "array" then [.[] | select(type == "object" and (.check | type) == "string" and (.status | type) == "string") + | {check: (.check | tostring), status: (.status | tostring)} + + (if ((.reason // "") | tostring) != "" then {reason: (.reason | tostring)} else {} end) + + (if ((.detail // "") | tostring) != "" then {detail: (.detail | tostring | .[0:160])} else {} end)] | .[0:20] else [] end' "$DIAG_FILE" 2>/dev/null) || from_prepare="[]" + elif [[ "$DRY_RUN" != "1" ]]; then + diag prepare unavailable step_missing + fi + shaped=$(jq -c -n --argjson here "$(diag_json)" --argjson there "$from_prepare" '($here + $there) | unique_by(.check) | .[0:20]') || shaped="[]" + DIAGNOSTICS_JSON=$shaped +} + collect_system_info() { local ip_json="[]" if command -v ip >/dev/null 2>&1; then @@ -543,6 +604,7 @@ collect_system_info() { fi HARDWARE_JSON="null" collect_hardware || HARDWARE_JSON="null" + collect_diagnostics HEALTH_JSON=$(jq -n -c --argjson failed "$FAILED_UNITS_JSON" --argjson smart "$SMART_JSON" --argjson zfs "$ZFS_JSON" \ '{} | (if $failed != null then . + {failed_units: $failed} else . end) | (if $smart != null then . + {smart: $smart} else . end) | (if $zfs != null then . + {zfs: $zfs} else . end)') || HEALTH_JSON="{}" [[ "$HEALTH_JSON" == "{}" ]] && HEALTH_JSON="null" @@ -551,6 +613,7 @@ collect_system_info() { --argjson updates "$UPDATES_JSON" \ --argjson health "$HEALTH_JSON" \ --argjson hardware "$HARDWARE_JSON" \ + --argjson diagnostics "$DIAGNOSTICS_JSON" \ --argjson ip_addresses "$ip_json" \ --argjson listening_ports "$ports_json" \ --arg cpu_model "$cpu_model" \ @@ -567,7 +630,8 @@ collect_system_info() { listening_ports: $listening_ports, updates: $updates, health: $health, - hardware: $hardware + hardware: $hardware, + diagnostics: $diagnostics }') } @@ -587,12 +651,17 @@ if [[ "$MODE" != "report" ]]; then jq . <<<"$SMART_JSON" echo "ZFS pools (null means there is no zpool here):" jq . <<<"$ZFS_JSON" + collect_hardware || true + echo "What could not be collected (empty means nothing was missed):" + diag_json | jq . self_update exit 0 fi # Leave the answers (or no file at all) for the report that follows. Each is written whole and moved into place, so a report never reads half of one. - rm -f "$UPDATES_FILE" "$HEALTH_FILE" + rm -f "$UPDATES_FILE" "$HEALTH_FILE" "$DIAG_FILE" mkdir -p "$(dirname "$UPDATES_FILE")" 2>/dev/null || exit 0 + # The notes always go, even when there are none: a missing file is how the report knows this step didn't run. + diag_json > "$DIAG_FILE.tmp" 2>/dev/null && mv -f "$DIAG_FILE.tmp" "$DIAG_FILE" 2>/dev/null || true if [[ "$UPDATES_JSON" != "null" ]]; then printf '%s\n' "$UPDATES_JSON" > "$UPDATES_FILE.tmp" 2>/dev/null && mv -f "$UPDATES_FILE.tmp" "$UPDATES_FILE" 2>/dev/null || true fi diff --git a/agent/windows/report-tasks.ps1 b/agent/windows/report-tasks.ps1 index 59c3169..0482133 100644 --- a/agent/windows/report-tasks.ps1 +++ b/agent/windows/report-tasks.ps1 @@ -35,10 +35,23 @@ $ErrorActionPreference = 'Stop' # change here is worth installing the agent again for. 1 = before the number was reported 2 = reports pending updates, and its own version # 3 = can replace itself with the newer script when an administrator has switched that on in Homelab Manager # 4 = also reports services that have failed, disk health, Storage Spaces pools and what the machine is (maker, model, serial) -$script:AgentVersion = 4 +# 5 = also says what it could not collect and why, so Homelab Manager can say what to fix +$script:AgentVersion = 5 $script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json' +# Things that could not be collected: which check, whether it is 'unavailable' (nothing there to read) or 'failed' (it tried and could not), why in a +# word, and at most a line of what Windows said. Only problems are noted; a check that is not mentioned went fine. +$script:Diag = New-Object System.Collections.ArrayList +function Add-Diag([string]$Check, [string]$Status, [string]$Reason = '', [string]$Detail = '') { + $item = [ordered]@{ check = $Check; status = $Status } + if ($Reason) { $item['reason'] = $Reason } + $clean = ($Detail -replace '[\x00-\x1f]', ' ').Trim() + if ($clean.Length -gt 160) { $clean = $clean.Substring(0, 160) } + if ($clean) { $item['detail'] = $clean } + [void]$script:Diag.Add([pscustomobject]$item) +} + # ---- configuration ------------------------------------------------------------ function Get-AgentConfig { @@ -287,7 +300,10 @@ function Get-PendingUpdates { } # No answer is no answer: the report then leaves updates out, and the server keeps what it was last told. - if ($cached.failed) { return $null } + if ($cached.failed) { + Add-Diag 'updates' 'failed' 'error' 'Windows Update did not answer; it is asked again in an hour' + return $null + } return [ordered]@{ manager = 'windows-update'; count = [int]$cached.count; security = [int]$cached.security; reboot_required = (Test-RebootRequired) } } @@ -355,7 +371,7 @@ function Get-HostHealth($Config = $null) { $health = [ordered]@{} $ignore = @() if ($Config -and $Config.IgnoreServices) { $ignore = @($Config.IgnoreServices) } - try { $health['failed_units'] = @(ConvertTo-FailedUnits @(Get-CimInstance -ClassName Win32_Service -ErrorAction Stop) $ignore) } catch { Write-Warning "Reading services failed. ($($_.Exception.Message))" } + try { $health['failed_units'] = @(ConvertTo-FailedUnits @(Get-CimInstance -ClassName Win32_Service -ErrorAction Stop) $ignore) } catch { Write-Warning "Reading services failed. ($($_.Exception.Message))"; Add-Diag 'failed_units' 'failed' 'error' $_.Exception.Message } try { $disks = @(Get-PhysicalDisk -ErrorAction Stop) $counters = @{} @@ -363,12 +379,12 @@ function Get-HostHealth($Config = $null) { try { $counters[[string]$d.DeviceId] = Get-StorageReliabilityCounter -PhysicalDisk $d -ErrorAction Stop } catch { } } $health['smart'] = @(ConvertTo-DiskHealth $disks $counters) - } catch { Write-Warning "Reading disk health failed. ($($_.Exception.Message))" } + } catch { Write-Warning "Reading disk health failed. ($($_.Exception.Message))"; Add-Diag 'smart' 'failed' 'error' $_.Exception.Message } try { $health['zfs'] = @(ConvertTo-PoolHealth @(Get-StoragePool -ErrorAction Stop | Where-Object { -not $_.IsPrimordial })) } catch { # A machine with no Storage Spaces pools gets an error, not an empty list: that is an answer ("no pools"), not a failure to read. - if ($_.FullyQualifiedErrorId -like '*NotFound*') { $health['zfs'] = @() } else { Write-Warning "Reading storage pools failed. ($($_.Exception.Message))" } + if ($_.FullyQualifiedErrorId -like '*NotFound*') { $health['zfs'] = @() } else { Write-Warning "Reading storage pools failed. ($($_.Exception.Message))"; Add-Diag 'zfs' 'failed' 'error' $_.Exception.Message } } if ($health.Count -eq 0) { return $null } return $health @@ -402,6 +418,7 @@ function Get-HardwareInfo { } function Get-SystemInfo($Config = $null) { + $script:Diag.Clear() $ips = @(Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue | Where-Object { $_.IPAddress -notlike '127.*' -and $_.IPAddress -notlike '169.254.*' -and $_.AddressState -eq 'Preferred' } | ForEach-Object { $_.IPAddress } | Select-Object -Unique) @@ -423,18 +440,22 @@ function Get-SystemInfo($Config = $null) { # Best-effort: Windows Update being unreachable must not cost the rest of the report. $updates = $null - try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))" } + try { $updates = Get-PendingUpdates } catch { Write-Warning "Checking for updates failed. ($($_.Exception.Message))"; Add-Diag 'updates' 'failed' 'error' $_.Exception.Message } # The same for what is wrong with the machine, and what it is: neither may cost the rest of the report. $health = $null - try { $health = Get-HostHealth $Config } catch { Write-Warning "Reading host health failed. ($($_.Exception.Message))" } + try { $health = Get-HostHealth $Config } catch { Write-Warning "Reading host health failed. ($($_.Exception.Message))"; Add-Diag 'host_health' 'failed' 'error' $_.Exception.Message } $hardware = $null - try { $hardware = Get-HardwareInfo } catch { Write-Warning "Reading the hardware identity failed. ($($_.Exception.Message))" } + try { + $hardware = Get-HardwareInfo + if (-not $hardware) { Add-Diag 'hardware' 'unavailable' 'not_found' } + } catch { Write-Warning "Reading the hardware identity failed. ($($_.Exception.Message))"; Add-Diag 'hardware' 'failed' 'error' $_.Exception.Message } return [ordered]@{ updates = $updates health = $health hardware = $hardware + diagnostics = @($script:Diag) ip_addresses = @($ips) cpu = [ordered]@{ model = [string]($cpus[0].Name).Trim(); cores = [int]$cores; load_percent = $(if ($null -ne $load) { [math]::Round([double]$load, 1) } else { $null }) } memory = [ordered]@{ total_bytes = $totalBytes; used_bytes = ($totalBytes - $freeBytes) } diff --git a/server/drizzle/0038_wise_malcolm_colcord.sql b/server/drizzle/0038_wise_malcolm_colcord.sql new file mode 100644 index 0000000..f9ee182 --- /dev/null +++ b/server/drizzle/0038_wise_malcolm_colcord.sql @@ -0,0 +1 @@ +ALTER TABLE `servers` ADD `agent_diagnostics` text; \ No newline at end of file diff --git a/server/drizzle/meta/0038_snapshot.json b/server/drizzle/meta/0038_snapshot.json new file mode 100644 index 0000000..434120a --- /dev/null +++ b/server/drizzle/meta/0038_snapshot.json @@ -0,0 +1,3384 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "11cfbabb-2d91-4446-8b26-80c1b0be834e", + "prevId": "493b72d0-43ed-46c5-9b61-d8063227bf2e", + "tables": { + "alert_acks": { + "name": "alert_acks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "assignee_user_id": { + "name": "assignee_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "assignee_name": { + "name": "assignee_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "acked_by": { + "name": "acked_by", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "acked_at": { + "name": "acked_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "until": { + "name": "until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "alert_acks_key_idx": { + "name": "alert_acks_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": true + } + }, + "foreignKeys": { + "alert_acks_assignee_user_id_users_id_fk": { + "name": "alert_acks_assignee_user_id_users_id_fk", + "tableFrom": "alert_acks", + "tableTo": "users", + "columnsFrom": [ + "assignee_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_incidents": { + "name": "alert_incidents", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "peak_severity": { + "name": "peak_severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "link": { + "name": "link", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_name": { + "name": "target_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "caused_by": { + "name": "caused_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "opened_at": { + "name": "opened_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resolved_at": { + "name": "resolved_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "missing_since": { + "name": "missing_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "escalated_step": { + "name": "escalated_step", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "escalated_at": { + "name": "escalated_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_incidents_key_idx": { + "name": "alert_incidents_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + }, + "alert_incidents_opened_idx": { + "name": "alert_incidents_opened_idx", + "columns": [ + "opened_at" + ], + "isUnique": false + }, + "alert_incidents_target_idx": { + "name": "alert_incidents_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "alert_tickets": { + "name": "alert_tickets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "alert_key": { + "name": "alert_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "seen_count": { + "name": "seen_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ticket_number": { + "name": "ticket_number", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "grouped": { + "name": "grouped", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ticketed_at": { + "name": "ticketed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cleared_at": { + "name": "cleared_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "alert_tickets_key_idx": { + "name": "alert_tickets_key_idx", + "columns": [ + "alert_key" + ], + "isUnique": false + } + }, + "foreignKeys": { + "alert_tickets_integration_id_integrations_id_fk": { + "name": "alert_tickets_integration_id_integrations_id_fk", + "tableFrom": "alert_tickets", + "tableTo": "integrations", + "columnsFrom": [ + "integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assets": { + "name": "assets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'other'" + }, + "manufacturer": { + "name": "manufacturer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "serial": { + "name": "serial", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_date": { + "name": "purchase_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_price": { + "name": "purchase_price", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warranty_until": { + "name": "warranty_until", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rack_unit": { + "name": "rack_unit", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "power_watts": { + "name": "power_watts", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_log": { + "name": "audit_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "actor_label": { + "name": "actor_label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "audit_log_actor_user_id_users_id_fk": { + "name": "audit_log_actor_user_id_users_id_fk", + "tableFrom": "audit_log", + "tableTo": "users", + "columnsFrom": [ + "actor_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "consistency_ignores": { + "name": "consistency_ignores", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "consistency_ignores_key_unique": { + "name": "consistency_ignores_key_unique", + "columns": [ + "key" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dependencies": { + "name": "dependencies", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "from_type": { + "name": "from_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "from_id": { + "name": "from_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "to_type": { + "name": "to_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "to_id": { + "name": "to_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "dependencies_edge_idx": { + "name": "dependencies_edge_idx", + "columns": [ + "from_type", + "from_id", + "to_type", + "to_id" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "diag_log": { + "name": "diag_log", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "operation": { + "name": "operation", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ok": { + "name": "ok", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "latency_ms": { + "name": "latency_ms", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_providers": { + "name": "dns_providers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_type": { + "name": "provider_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "dns_providers_credential_id_integration_credentials_id_fk": { + "name": "dns_providers_credential_id_integration_credentials_id_fk", + "tableFrom": "dns_providers", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_records_cache": { + "name": "dns_records_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "record_id": { + "name": "record_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ttl": { + "name": "ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "priority": { + "name": "priority", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxied": { + "name": "proxied", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "dns_records_cache_provider_id_dns_providers_id_fk": { + "name": "dns_records_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_records_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "dns_zones_cache": { + "name": "dns_zones_cache", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider_id": { + "name": "provider_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "zone_name": { + "name": "zone_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "synced_at": { + "name": "synced_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "dns_zones_cache_provider_zone_idx": { + "name": "dns_zones_cache_provider_zone_idx", + "columns": [ + "provider_id", + "zone_id" + ], + "isUnique": true + } + }, + "foreignKeys": { + "dns_zones_cache_provider_id_dns_providers_id_fk": { + "name": "dns_zones_cache_provider_id_dns_providers_id_fk", + "tableFrom": "dns_zones_cache", + "tableTo": "dns_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "domains": { + "name": "domains", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'manual'" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "registrar": { + "name": "registrar", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "lookup_source": { + "name": "lookup_source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_ok_at": { + "name": "last_checked_ok_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "domains_name_unique": { + "name": "domains_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "event_webhooks": { + "name": "event_webhooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "secret": { + "name": "secret", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "events": { + "name": "events", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_attempt_at": { + "name": "last_attempt_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_result": { + "name": "last_result", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "incident_notes": { + "name": "incident_notes", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "incident_id": { + "name": "incident_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "author": { + "name": "author", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "incident_notes_incident_idx": { + "name": "incident_notes_incident_idx", + "columns": [ + "incident_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "incident_notes_incident_id_alert_incidents_id_fk": { + "name": "incident_notes_incident_id_alert_incidents_id_fk", + "tableFrom": "incident_notes", + "tableTo": "alert_incidents", + "columnsFrom": [ + "incident_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integration_credentials": { + "name": "integration_credentials", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "encrypted_secret": { + "name": "encrypted_secret", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "integrations": { + "name": "integrations", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "credential_id": { + "name": "credential_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "integrations_credential_id_integration_credentials_id_fk": { + "name": "integrations_credential_id_integration_credentials_id_fk", + "tableFrom": "integrations", + "tableTo": "integration_credentials", + "columnsFrom": [ + "credential_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "ipam_entries": { + "name": "ipam_entries", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "vendor": { + "name": "vendor", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location": { + "name": "location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "ipam_entries_ip_address_unique": { + "name": "ipam_entries_ip_address_unique", + "columns": [ + "ip_address" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_aliases": { + "name": "maintenance_aliases", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "address": { + "name": "address", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "maintenance_aliases_unique_idx": { + "name": "maintenance_aliases_unique_idx", + "columns": [ + "address", + "target_type", + "target_id" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_schedules": { + "name": "maintenance_schedules", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "targets": { + "name": "targets", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "days": { + "name": "days", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "start_time": { + "name": "start_time", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_minutes": { + "name": "duration_minutes", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timezone": { + "name": "timezone", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "maintenance_windows": { + "name": "maintenance_windows", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ends_at": { + "name": "ends_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "metrics_tokens": { + "name": "metrics_tokens", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "metrics_tokens_token_hash_unique": { + "name": "metrics_tokens_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "notification_queue": { + "name": "notification_queue", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "audience": { + "name": "audience", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "port_forwards": { + "name": "port_forwards", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "external_port": { + "name": "external_port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination": { + "name": "destination", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "internal_port": { + "name": "internal_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "port_forwards_server_id_servers_id_fk": { + "name": "port_forwards_server_id_servers_id_fk", + "tableFrom": "port_forwards", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "push_subscriptions": { + "name": "push_subscriptions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "endpoint": { + "name": "endpoint", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "p256dh": { + "name": "p256dh", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "auth": { + "name": "auth", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_success_at": { + "name": "last_success_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "min_severity": { + "name": "min_severity", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "categories": { + "name": "categories", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "push_subscriptions_endpoint_unique": { + "name": "push_subscriptions_endpoint_unique", + "columns": [ + "endpoint" + ], + "isUnique": true + } + }, + "foreignKeys": { + "push_subscriptions_user_id_users_id_fk": { + "name": "push_subscriptions_user_id_users_id_fk", + "tableFrom": "push_subscriptions", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "runbooks": { + "name": "runbooks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "target_type": { + "name": "target_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "category": { + "name": "category", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "runbooks_target_idx": { + "name": "runbooks_target_idx", + "columns": [ + "target_type", + "target_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "scheduled_tasks": { + "name": "scheduled_tasks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "schedule_type": { + "name": "schedule_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'agent'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "command": { + "name": "command", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "schedule_expression": { + "name": "schedule_expression", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "next_run_at": { + "name": "next_run_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "raw_metadata": { + "name": "raw_metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_stale": { + "name": "is_stale", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "first_seen_at": { + "name": "first_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "scheduled_tasks_server_id_servers_id_fk": { + "name": "scheduled_tasks_server_id_servers_id_fk", + "tableFrom": "scheduled_tasks", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "secrets": { + "name": "secrets", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'generic'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expiry_date": { + "name": "expiry_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "warn_days": { + "name": "warn_days", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 30 + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_host": { + "name": "check_host", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "check_port": { + "name": "check_port", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_check_error": { + "name": "last_check_error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_links": { + "name": "server_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "server_links_server_id_servers_id_fk": { + "name": "server_links_server_id_servers_id_fk", + "tableFrom": "server_links", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_metrics": { + "name": "server_metrics", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metric": { + "name": "metric", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "real", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "recorded_at": { + "name": "recorded_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "server_metrics_lookup_idx": { + "name": "server_metrics_lookup_idx", + "columns": [ + "server_id", + "metric", + "recorded_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "server_metrics_server_id_servers_id_fk": { + "name": "server_metrics_server_id_servers_id_fk", + "tableFrom": "server_metrics", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "server_ports": { + "name": "server_ports", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "port": { + "name": "port", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "protocol": { + "name": "protocol", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'tcp'" + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "comment": { + "name": "comment", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "open": { + "name": "open", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "last_seen_open_at": { + "name": "last_seen_open_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "server_ports_unique": { + "name": "server_ports_unique", + "columns": [ + "server_id", + "port", + "protocol" + ], + "isUnique": true + } + }, + "foreignKeys": { + "server_ports_server_id_servers_id_fk": { + "name": "server_ports_server_id_servers_id_fk", + "tableFrom": "server_ports", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "servers": { + "name": "servers", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "hostname": { + "name": "hostname", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "os_type": { + "name": "os_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'linux'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "api_token_hash": { + "name": "api_token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "api_token_prefix": { + "name": "api_token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ip_addresses": { + "name": "ip_addresses", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_model": { + "name": "cpu_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_cores": { + "name": "cpu_cores", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpu_load_percent": { + "name": "cpu_load_percent", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_total_bytes": { + "name": "mem_total_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mem_used_bytes": { + "name": "mem_used_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disks": { + "name": "disks", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "listening_ports": { + "name": "listening_ports", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_port_scan": { + "name": "last_port_scan", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_integration_id": { + "name": "proxmox_integration_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_node": { + "name": "proxmox_node", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_guest_type": { + "name": "proxmox_guest_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "proxmox_vmid": { + "name": "proxmox_vmid", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hide_proxmox_link": { + "name": "hide_proxmox_link", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "parent_server_id": { + "name": "parent_server_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "no_parent": { + "name": "no_parent", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "pending_updates": { + "name": "pending_updates", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_pending_since": { + "name": "updates_pending_since", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_health": { + "name": "host_health", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_diagnostics": { + "name": "agent_diagnostics", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_manufacturer": { + "name": "hw_manufacturer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_model": { + "name": "hw_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "hw_serial": { + "name": "hw_serial", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "thresholds": { + "name": "thresholds", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_version": { + "name": "agent_version", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "servers_proxmox_integration_id_integrations_id_fk": { + "name": "servers_proxmox_integration_id_integrations_id_fk", + "tableFrom": "servers", + "tableTo": "integrations", + "columnsFrom": [ + "proxmox_integration_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "settings": { + "name": "settings", + "columns": { + "key": { + "name": "key", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "share_links": { + "name": "share_links", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token_prefix": { + "name": "token_prefix", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "detail": { + "name": "detail", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'summary'" + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'wall'" + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "expires_at": { + "name": "expires_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_used_at": { + "name": "last_used_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "share_links_token_hash_unique": { + "name": "share_links_token_hash_unique", + "columns": [ + "token_hash" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "status_items": { + "name": "status_items", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "link_id": { + "name": "link_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "position": { + "name": "position", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_id": { + "name": "target_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "monitor_id": { + "name": "monitor_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "status_items_link_idx": { + "name": "status_items_link_idx", + "columns": [ + "link_id", + "position" + ], + "isUnique": false + } + }, + "foreignKeys": { + "status_items_link_id_share_links_id_fk": { + "name": "status_items_link_id_share_links_id_fk", + "tableFrom": "status_items", + "tableTo": "share_links", + "columnsFrom": [ + "link_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "tag_definitions": { + "name": "tag_definitions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "tag_definitions_name_unique": { + "name": "tag_definitions_name_unique", + "columns": [ + "name" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "update_runs": { + "name": "update_runs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "server_id": { + "name": "server_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "integration_id": { + "name": "integration_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_id": { + "name": "template_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "template_name": { + "name": "template_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "host_limit": { + "name": "host_limit", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_id": { + "name": "task_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "task_status": { + "name": "task_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_id": { + "name": "window_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "window_created": { + "name": "window_created", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_by": { + "name": "started_by", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_ended_at": { + "name": "task_ended_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "host_reported_at": { + "name": "host_reported_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finished_at": { + "name": "finished_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_before": { + "name": "updates_before", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updates_after": { + "name": "updates_after", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reboot_required": { + "name": "reboot_required", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "poll_errors": { + "name": "poll_errors", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + } + }, + "indexes": { + "update_runs_server_idx": { + "name": "update_runs_server_idx", + "columns": [ + "server_id", + "started_at" + ], + "isUnique": false + }, + "update_runs_status_idx": { + "name": "update_runs_status_idx", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "update_runs_server_id_servers_id_fk": { + "name": "update_runs_server_id_servers_id_fk", + "tableFrom": "update_runs", + "tableTo": "servers", + "columnsFrom": [ + "server_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "users": { + "name": "users", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "oidc_sub": { + "name": "oidc_sub", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'viewer'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "last_login_at": { + "name": "last_login_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "favorites": { + "name": "favorites", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_looked_at": { + "name": "last_looked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "groups": { + "name": "groups", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dashboard_layout": { + "name": "dashboard_layout", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "users_oidc_sub_unique": { + "name": "users_oidc_sub_unique", + "columns": [ + "oidc_sub" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/server/drizzle/meta/_journal.json b/server/drizzle/meta/_journal.json index c197714..62e6b7d 100644 --- a/server/drizzle/meta/_journal.json +++ b/server/drizzle/meta/_journal.json @@ -267,6 +267,13 @@ "when": 1792344782975, "tag": "0037_past_magik", "breakpoints": true + }, + { + "idx": 38, + "version": "6", + "when": 1792354148200, + "tag": "0038_wise_malcolm_colcord", + "breakpoints": true } ] } \ No newline at end of file diff --git a/server/src/db/schema.ts b/server/src/db/schema.ts index 39f1915..fb0b21e 100644 --- a/server/src/db/schema.ts +++ b/server/src/db/schema.ts @@ -341,6 +341,9 @@ export const servers = sqliteTable("servers", { // What the agent says about the machine's own health, as JSON: {failedUnits, smart, zfs, reportedAt} — see services/hostHealth.ts. Null until it says. hostHealth: text("host_health"), + // What the agent says it could not collect, as JSON: {items: [{check, status, reason?, detail?}], reportedAt} — see services/agentDiagnostics.ts. Null until an agent that can say so has. + agentDiagnostics: text("agent_diagnostics"), + // What the machine is, as its firmware says (agents report it): maker, model and serial number. Used to fill in the inventory. hwManufacturer: text("hw_manufacturer"), hwModel: text("hw_model"), diff --git a/server/src/routes/agentReport.ts b/server/src/routes/agentReport.ts index bbe1c66..f814dfb 100644 --- a/server/src/routes/agentReport.ts +++ b/server/src/routes/agentReport.ts @@ -59,7 +59,23 @@ const hardwareSchema = z.object({ serial: text(100).optional().catch(undefined), }); +// What the agent could not collect: leniently read, each entry on its own, so one odd one never costs the report. +const diagnosticsSchema = z.unknown().transform((v) => { + if (!Array.isArray(v)) return undefined; + const out: { check: string; status: "unavailable" | "failed"; reason?: string; detail?: string }[] = []; + for (const i of v.slice(0, 20)) { + if (!i || typeof i !== "object") continue; + const e = i as Record; + const clean = (x: unknown, max: number) => (typeof x === "string" ? x.replace(/[\u0000-\u001f]/g, " ").trim().slice(0, max) : ""); + const check = clean(e.check, 40); + if (!check || (e.status !== "unavailable" && e.status !== "failed")) continue; + out.push({ check, status: e.status, ...(clean(e.reason, 40) ? { reason: clean(e.reason, 40) } : {}), ...(clean(e.detail, 160) ? { detail: clean(e.detail, 160) } : {}) }); + } + return out; +}); + const systemSchema = z.object({ + diagnostics: diagnosticsSchema.optional(), hardware: z .unknown() .optional() diff --git a/server/src/routes/servers.ts b/server/src/routes/servers.ts index 965921b..940f66a 100644 --- a/server/src/routes/servers.ts +++ b/server/src/routes/servers.ts @@ -8,7 +8,8 @@ import { generateApiToken } from "../services/tokens.js"; import { recordAudit } from "../services/audit.js"; import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js"; import { parsePendingUpdates } from "../services/pendingUpdates.js"; -import { parseHostHealth } from "../services/hostHealth.js"; +import { markIgnoredServices, parseHostHealth } from "../services/hostHealth.js"; +import { describeDiagnostics, parseDiagnostics } from "../services/agentDiagnostics.js"; import { checkThresholds, parseThresholds } from "../services/serverThresholds.js"; import { latestAgentVersions } from "../services/agentVersions.js"; import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js"; @@ -27,8 +28,8 @@ import { getSettings, updateSettings } from "../services/settingsStore.js"; export const serversRouter = Router(); /** A server row as the API returns it: no token hash, and tags as an array rather than the stored JSON. */ -function publicServer(row: T) { - const { apiTokenHash: _hash, tags, pendingUpdates, hostHealth, thresholds, ...rest } = row; +function publicServer(row: T) { + const { apiTokenHash: _hash, tags, pendingUpdates, hostHealth, thresholds, agentDiagnostics: _diagnostics, ...rest } = row; return { ...rest, tags: parseTags(tags), pendingUpdates: parsePendingUpdates(pendingUpdates), hostHealth: parseHostHealth(hostHealth), thresholds: parseThresholds(thresholds) }; } @@ -382,6 +383,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => { pendingUpdates: rawUpdates, hostHealth: rawHealth, thresholds: rawThresholds, + agentDiagnostics: rawDiagnostics, ...serverOut } = server; @@ -395,7 +397,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => { children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)), }; - res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates), hostHealth: parseHostHealth(rawHealth), thresholds: parseThresholds(rawThresholds) }, thresholdDefaults: (await getSettings()).healthChecks, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() }); + res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates), hostHealth: markIgnoredServices(parseHostHealth(rawHealth), parseThresholds(rawThresholds)?.ignoreServices), thresholds: parseThresholds(rawThresholds) }, diagnostics: describeDiagnostics(parseDiagnostics(rawDiagnostics), { osType: server.osType, lastSeenAt: server.lastSeenAt, agentVersion: server.agentVersion, latest: latestAgentVersions() }), thresholdDefaults: (await getSettings()).healthChecks, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() }); })); const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) }); diff --git a/server/src/services/agentDiagnostics.ts b/server/src/services/agentDiagnostics.ts new file mode 100644 index 0000000..40ce94a --- /dev/null +++ b/server/src/services/agentDiagnostics.ts @@ -0,0 +1,130 @@ +/** + * What an agent could not collect, and what to do about it. Agents (Linux version 6, Windows version 5) note each check they couldn't make — + * with whether nothing was there to read it with ("unavailable") or it tried and couldn't ("failed"), why in a word and at most a line of + * what the tool said. They say only problems; a check not mentioned went fine. The words and the fixes live here, not in the agents, so they + * can be improved without installing anything again. + */ +import { isAgentOutdated, type LatestAgentVersions } from "./agentVersions.js"; + +export interface AgentDiagnostic { + check: string; + status: "unavailable" | "failed"; + reason?: string; + detail?: string; +} + +export interface StoredDiagnostics { + items: AgentDiagnostic[]; + /** ISO — the report these came from. */ + reportedAt: string; +} + +export interface DiagnosticNote { + check: string; + /** "warning": something to look at. "info": a tip, or something that is simply how this machine is. */ + level: "warning" | "info"; + title: string; + fix: string | null; + /** What the agent said about it, when it said anything. */ + detail: string | null; +} + +const clean = (v: unknown, max: number) => (typeof v === "string" ? v.replace(/[\u0000-\u001f]/g, " ").trim().slice(0, max) : ""); + +/** Reads what is stored, leniently. Null when the agent has never said (an agent that predates this, or one that has nothing to report on). */ +export function parseDiagnostics(stored: string | null | undefined): StoredDiagnostics | null { + if (!stored) return null; + try { + const v = JSON.parse(stored); + if (!v || typeof v !== "object" || !Array.isArray(v.items)) return null; + const items: AgentDiagnostic[] = []; + for (const i of v.items.slice(0, 20)) { + if (!i || typeof i !== "object") continue; + const check = clean(i.check, 40); + if (!check || (i.status !== "unavailable" && i.status !== "failed")) continue; + const item: AgentDiagnostic = { check, status: i.status }; + const reason = clean(i.reason, 40); + const detail = clean(i.detail, 160); + if (reason) item.reason = reason; + if (detail) item.detail = detail; + items.push(item); + } + return { items, reportedAt: typeof v.reportedAt === "string" ? v.reportedAt : "" }; + } catch { + return null; + } +} + +/** What to store after a report that carried notes (an empty list is an answer: nothing was missed). */ +export function nextDiagnostics(incoming: AgentDiagnostic[], now: string): string { + return JSON.stringify({ items: incoming, reportedAt: now } satisfies StoredDiagnostics); +} + +type Os = "linux" | "windows"; + +/** One note for what the agent said, or null when it is not worth a person's attention (no ZFS on a machine that has none). */ +function describe(d: AgentDiagnostic, os: Os): Omit | null { + const linux = os === "linux"; + const key = `${d.check}/${d.status}${d.reason ? `/${d.reason}` : ""}`; + switch (key) { + case "prepare/unavailable/step_missing": + return { + check: d.check, + level: "warning", + title: "The agent's step outside its sandbox didn't run", + fix: "Disk health and ZFS pools are only read in that step. Install the agent again (Servers → Manage servers has the command) so its systemd unit has it; if it is installed, look at `journalctl -u homelab-manager-agent` on the host.", + }; + case "updates/unavailable/no_package_manager": + return { check: d.check, level: "info", title: "Pending updates can't be read here", fix: "The agent understands apt, dnf, zypper and pacman, and Windows Update. This machine has none of them." }; + case "failed_units/unavailable/no_systemd": + return { check: d.check, level: "info", title: "Failed services can't be read: this machine doesn't run systemd", fix: null }; + case "smart/unavailable/tool_missing": + return { check: d.check, level: "info", title: "Disk health isn't reported: smartctl isn't installed", fix: "Install smartmontools on the host (apt install smartmontools, or dnf install smartmontools); the next report includes it." }; + case "smart/unavailable/not_root": + return { check: d.check, level: "warning", title: "Disk health needs the agent to run as root", fix: "The install script's systemd unit runs it as root. A copy started by hand as another user can't read the disks." }; + case "zfs/unavailable/tool_missing": + return null; // most machines have no ZFS, and that is not a problem + case "hardware/unavailable/not_found": + return { check: d.check, level: "info", title: "No maker, model or serial number found", fix: "Many virtual machines and single-board computers don't have them. Type them into the Inventory instead." }; + } + if (d.status === "failed") { + const what: Record = { updates: "pending updates", failed_units: "services", smart: "disk health", zfs: "storage pools", host_health: "the machine's health", hardware: "what the machine is" }; + const fixes: Record = linux + ? { + updates: "Run `sudo /usr/local/bin/homelab-manager-agent.sh --check-updates` on the host to see what the package manager said (dnf and zypper answer from their caches: `sudo dnf makecache`).", + failed_units: "`systemctl list-units --state=failed` didn't answer in time on that host.", + zfs: "`zpool list` didn't answer; check that the ZFS kernel module is loaded.", + } + : { + updates: "Windows Update didn't answer (the search is tried again after an hour). Check the Windows Update service and that the machine can reach Microsoft.", + failed_units: "Reading services through WMI failed. Check that the Windows Management Instrumentation service is running.", + smart: "Get-PhysicalDisk failed. In an elevated PowerShell, run `Get-PhysicalDisk` to see why.", + zfs: "Get-StoragePool failed. In an elevated PowerShell, run `Get-StoragePool` to see why.", + host_health: "Reading the machine's health failed; the detail says why.", + hardware: "Reading the system's own details through WMI failed; the detail says why.", + }; + return { check: d.check, level: "warning", title: `Reading ${what[d.check] ?? d.check} failed`, fix: fixes[d.check] ?? null }; + } + return { check: d.check, level: "info", title: `${d.check.replace(/_/g, " ")} isn't available${d.reason ? ` (${d.reason.replace(/_/g, " ")})` : ""}`, fix: null }; +} + +/** What to tell a person about one server's agent: what it couldn't collect (with what to do) and, when it is behind, that. Warnings first. */ +export function describeDiagnostics(stored: StoredDiagnostics | null, context: { osType: string; lastSeenAt: string | null; agentVersion: number | null; latest: LatestAgentVersions }): DiagnosticNote[] { + const os: Os = context.osType === "windows" ? "windows" : "linux"; + const notes: DiagnosticNote[] = []; + for (const d of stored?.items ?? []) { + const note = describe(d, os); + if (note) notes.push({ ...note, detail: d.status === "failed" ? (d.detail ?? null) : null }); // what the agent said is only worth showing for a failure + } + if (isAgentOutdated({ osType: context.osType, lastSeenAt: context.lastSeenAt, agentVersion: context.agentVersion }, context.latest)) { + const current = os === "windows" ? context.latest.windows : context.latest.linux; + notes.push({ + check: "agent_behind", + level: "info", + title: `This agent is version ${context.agentVersion ?? 1}; ${current} is current`, + fix: "A newer agent reports more and says more about what it couldn't read. Update it from the Updates page, or switch on automatic updates for agents in Settings.", + detail: null, + }); + } + return notes.sort((a, b) => Number(b.level === "warning") - Number(a.level === "warning")); +} diff --git a/server/src/services/agentVersions.ts b/server/src/services/agentVersions.ts index 322436e..f783578 100644 --- a/server/src/services/agentVersions.ts +++ b/server/src/services/agentVersions.ts @@ -11,6 +11,7 @@ * 4 — Linux: also reports failed services, disk health (SMART) and ZFS pools, reads pacman, and can replace itself too. * Windows: also reports failed services, disk health and Storage Spaces pools, and what the machine is (maker, model, serial) * 5 — Linux: also says what the machine is (maker, model, serial), for the inventory + * 6 — Linux (and Windows 5): also says what it could not collect and why, so the server can say what to fix */ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; diff --git a/server/src/services/healthMonitor.ts b/server/src/services/healthMonitor.ts index 75ae345..89e23f6 100644 --- a/server/src/services/healthMonitor.ts +++ b/server/src/services/healthMonitor.ts @@ -10,7 +10,7 @@ import { getInternalFlag, getSettings, setInternalFlag } from "./settingsStore.j import { activeSubjects, subjectOfConditionKey } from "./maintenance.js"; import { dependencyCover, HEALTH_STATE_FLAG } from "./dependencies.js"; import { getFailingSources } from "./integrationHealthMonitor.js"; -import { hostHealthConditions, parseHostHealth, type HostHealth } from "./hostHealth.js"; +import { hostHealthConditions, parseHostHealth, withoutIgnoredServices, type HostHealth } from "./hostHealth.js"; import { diskLimit, parseThresholds, type ServerThresholds } from "./serverThresholds.js"; const STATE_FLAG = HEALTH_STATE_FLAG; @@ -306,7 +306,7 @@ export async function collectSnapshot(): Promise<{ snapshot: HealthSnapshot; hel } catch { // a malformed report shouldn't stop every other server being checked } - snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: parseHostHealth(s.hostHealth), thresholds: parseThresholds(s.thresholds) }); + snapshot.servers.push({ id: s.id, name: s.name, lastSeenAt: s.lastSeenAt, disks, health: withoutIgnoredServices(parseHostHealth(s.hostHealth), parseThresholds(s.thresholds)?.ignoreServices), thresholds: parseThresholds(s.thresholds) }); } const rows = await db diff --git a/server/src/services/hostHealth.ts b/server/src/services/hostHealth.ts index 80bd657..c08ba97 100644 --- a/server/src/services/hostHealth.ts +++ b/server/src/services/hostHealth.ts @@ -8,6 +8,8 @@ export interface FailedUnit { unit: string; description: string; + /** Set where it is shown, for a unit this server has been told to ignore. It never counts as failed. */ + ignored?: boolean; } export interface SmartDisk { device: string; @@ -82,6 +84,31 @@ export function nextHostHealth(previous: string | null, incoming: IncomingHostHe return any ? JSON.stringify(next) : previous; } +const globToRegex = (pattern: string) => new RegExp(`^${pattern.trim().toLowerCase().replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*")}$`); + +/** Whether a unit is one this server is told to ignore: a name, or a pattern with * ("gupdate*"); case doesn't matter and ".service" may be left off. */ +export function isServiceIgnored(unit: string, patterns: string[] | undefined): boolean { + if (!patterns || patterns.length === 0) return false; + const u = unit.toLowerCase(); + const bare = u.replace(/\.service$/, ""); + return patterns.some((p) => { + const re = globToRegex(p); + return re.test(u) || re.test(bare); + }); +} + +/** The health with the ignored services taken out: what alerts and figures are worked from. */ +export function withoutIgnoredServices(health: HostHealth | null, patterns: string[] | undefined): HostHealth | null { + if (!health?.failedUnits || !patterns || patterns.length === 0) return health; + return { ...health, failedUnits: health.failedUnits.filter((u) => !isServiceIgnored(u.unit, patterns)) }; +} + +/** The health with the ignored services marked, not removed: what the server's page shows, so they can be seen and taken off the list. */ +export function markIgnoredServices(health: HostHealth | null, patterns: string[] | undefined): HostHealth | null { + if (!health?.failedUnits || !patterns || patterns.length === 0) return health; + return { ...health, failedUnits: health.failedUnits.map((u) => (isServiceIgnored(u.unit, patterns) ? { ...u, ignored: true } : u)) }; +} + /** A pool in any state but ONLINE has lost something (DEGRADED has lost redundancy, the rest are worse). */ export const poolIsFine = (p: ZfsPool) => p.health.toUpperCase() === "ONLINE"; diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 4e67668..b09197a 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -10,7 +10,8 @@ import { getAlerts } from "./alerts.js"; import { isAgentOutdated, latestAgentVersions } from "./agentVersions.js"; import { getSourceStatus } from "./integrationHealthMonitor.js"; import { parsePendingUpdates } from "./pendingUpdates.js"; -import { parseHostHealth, poolIsFine } from "./hostHealth.js"; +import { parseHostHealth, poolIsFine, withoutIgnoredServices } from "./hostHealth.js"; +import { parseThresholds as parseServerThresholds } from "./serverThresholds.js"; import { parseThresholds } from "./serverThresholds.js"; import { getCachedBackupCoverage } from "./backupCoverage.js"; import { lastAttempt as lastSelfBackup, listSnapshots } from "./selfBackup.js"; @@ -88,7 +89,7 @@ export async function buildMetrics(now: Date = new Date()): Promise { if (updates.security !== null) out.add("homelab_server_pending_security_updates", "gauge", "Of those, the ones that are security fixes.", updates.security, { server }); out.add("homelab_server_reboot_required", "gauge", "1 if a restart is due.", updates.rebootRequired ? 1 : 0, { server }); } - const health = parseHostHealth(s.hostHealth); + const health = withoutIgnoredServices(parseHostHealth(s.hostHealth), parseServerThresholds(s.thresholds)?.ignoreServices); if (health?.failedUnits) out.add("homelab_server_failed_units", "gauge", "Services (systemd units) that have failed on the server.", health.failedUnits.length, { server }); for (const d of health?.smart ?? []) { if (d.healthy !== null) out.add("homelab_server_disk_healthy", "gauge", "1 if the disk's own SMART self-assessment passes, 0 if it fails.", d.healthy ? 1 : 0, { server, device: d.device }); diff --git a/server/src/services/serverThresholds.ts b/server/src/services/serverThresholds.ts index b0c3c68..d55fa4c 100644 --- a/server/src/services/serverThresholds.ts +++ b/server/src/services/serverThresholds.ts @@ -21,6 +21,8 @@ export interface ServerThresholds { /** 0 turns the stale-backup alert off for this server's guest. */ backupStaleHours?: number; mounts?: Record; + /** Services (units) never to count as failed on this server: names or patterns like "gupdate*" (case does not matter, ".service" is optional). */ + ignoreServices?: string[]; } /** The global values these override. */ @@ -38,6 +40,8 @@ export const LIMITS = { backupStaleHours: { min: 0, max: 2160 }, } as const; export const MAX_MOUNT_RULES = 50; +export const MAX_IGNORED_SERVICES = 30; +const MAX_SERVICE_PATTERN = 100; const inRange = (v: unknown, key: keyof typeof LIMITS): v is number => typeof v === "number" && Number.isInteger(v) && v >= LIMITS[key].min && v <= LIMITS[key].max; @@ -61,6 +65,11 @@ export function parseThresholds(stored: string | null | undefined): ServerThresh } if (Object.keys(mounts).length > 0) out.mounts = mounts; } + if (Array.isArray(v.ignoreServices)) { + const strings: string[] = (v.ignoreServices as unknown[]).filter((p): p is string => typeof p === "string"); + const list = [...new Set(strings.map((p) => p.trim()).filter((p) => p.length > 0 && p.length <= MAX_SERVICE_PATTERN))].slice(0, MAX_IGNORED_SERVICES); + if (list.length > 0) out.ignoreServices = list; + } return Object.keys(out).length > 0 ? out : null; } catch { return null; @@ -103,6 +112,20 @@ export function checkThresholds(input: unknown): { ok: true; value: ServerThresh } if (Object.keys(mounts).length > 0) out.mounts = mounts; } + if (v.ignoreServices !== undefined && v.ignoreServices !== null) { + if (!Array.isArray(v.ignoreServices)) return { ok: false, message: "The services to ignore should be a list." }; + const list: string[] = []; + for (const raw of v.ignoreServices) { + if (typeof raw !== "string") return { ok: false, message: "Each service to ignore should be a name or a pattern." }; + const p = raw.trim(); + if (p.length === 0) continue; + // eslint-disable-next-line no-control-regex + if (p.length > MAX_SERVICE_PATTERN || /[\u0000-\u001f]/.test(p)) return { ok: false, message: `"${p.slice(0, 30)}…" isn't a service name or pattern (up to ${MAX_SERVICE_PATTERN} characters, no line breaks).` }; + if (!list.includes(p)) list.push(p); + } + if (list.length > MAX_IGNORED_SERVICES) return { ok: false, message: `At most ${MAX_IGNORED_SERVICES} services can be ignored on one server.` }; + if (list.length > 0) out.ignoreServices = list; + } return { ok: true, value: Object.keys(out).length > 0 ? out : null }; } diff --git a/server/src/services/taskSync.ts b/server/src/services/taskSync.ts index b3d8d35..88d0967 100644 --- a/server/src/services/taskSync.ts +++ b/server/src/services/taskSync.ts @@ -6,6 +6,7 @@ import { nextUpdateState, type IncomingUpdates } from "./pendingUpdates.js"; import { nextHostHealth, type IncomingHostHealth } from "./hostHealth.js"; import { recordAudit } from "./audit.js"; import { fillInventoryFromServer } from "./inventoryAutofill.js"; +import { nextDiagnostics, type AgentDiagnostic } from "./agentDiagnostics.js"; export interface IncomingTask { scheduleType: "cron" | "systemd_timer" | "windows_task"; @@ -30,6 +31,8 @@ export interface IncomingSystemInfo { health?: IncomingHostHealth; /** What the machine is, from its firmware: each part present only if it says anything. */ hardware?: { manufacturer?: string; model?: string; serial?: string }; + /** What the agent could not collect; an empty list says nothing was missed. */ + diagnostics?: AgentDiagnostic[]; } export interface AgentReport { @@ -118,6 +121,7 @@ export async function syncServerTasks(serverId: number, report: AgentReport) { // Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before. ...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}), ...(system?.health && before ? { hostHealth: nextHostHealth(before.hostHealth, system.health, now) } : {}), + ...(system?.diagnostics ? { agentDiagnostics: nextDiagnostics(system.diagnostics, now) } : {}), ...(system?.hardware?.manufacturer ? { hwManufacturer: system.hardware.manufacturer } : {}), ...(system?.hardware?.model ? { hwModel: system.hardware.model } : {}), ...(system?.hardware?.serial ? { hwSerial: system.hardware.serial } : {}), diff --git a/server/test/agentDiagnostics.test.ts b/server/test/agentDiagnostics.test.ts new file mode 100644 index 0000000..f42e8f3 --- /dev/null +++ b/server/test/agentDiagnostics.test.ts @@ -0,0 +1,207 @@ +/** + * What an agent could not collect and what to do about it, and the services a server is told to ignore. + * + * Run with: npm test --workspace server + */ +import { test, describe, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import type { Server } from "node:http"; + +const dir = mkdtempSync(join(tmpdir(), "hm-diag-")); +process.env.DATABASE_PATH = join(dir, "test.sqlite"); +process.env.SESSION_DIR = join(dir, "sessions"); +process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32); +process.env.HEALTH_STARTUP_GRACE_MS = "0"; + +const express = (await import("express")).default; +const { db } = await import("../src/db/client.js"); +const { users, servers } = await import("../src/db/schema.js"); +const { runMigrations } = await import("../src/db/migrate.js"); +const { describeDiagnostics, parseDiagnostics } = await import("../src/services/agentDiagnostics.js"); +const { isServiceIgnored, withoutIgnoredServices, markIgnoredServices } = await import("../src/services/hostHealth.js"); +const { checkThresholds, parseThresholds } = await import("../src/services/serverThresholds.js"); +const { getAlerts, invalidateAlertsCache } = await import("../src/services/alerts.js"); +const { buildMetrics } = await import("../src/services/metrics.js"); +const { serversRouter } = await import("../src/routes/servers.js"); +const { agentReportRouter } = await import("../src/routes/agentReport.js"); +const { hashToken } = await import("../src/services/tokens.js"); + +await runMigrations(); + +const latest = { linux: 6, windows: 5 }; +const ctx = (extra: Partial<{ osType: string; lastSeenAt: string | null; agentVersion: number | null }> = {}) => ({ osType: "linux", lastSeenAt: "2026-10-18T10:00:00Z", agentVersion: 6, latest, ...extra }); +const stored = (items: unknown[]) => JSON.stringify({ items, reportedAt: "2026-10-18T10:00:00Z" }); + +describe("saying what to do about what couldn't be collected", () => { + test("a missing tool is a tip with the fix; ZFS that isn't there is not worth saying", () => { + const notes = describeDiagnostics( + parseDiagnostics(stored([ + { check: "smart", status: "unavailable", reason: "tool_missing", detail: "smartctl" }, + { check: "zfs", status: "unavailable", reason: "tool_missing", detail: "zpool" }, + { check: "hardware", status: "unavailable", reason: "not_found" }, + ])), + ctx(), + ); + assert.deepEqual(notes.map((n) => n.check), ["smart", "hardware"], "no word about ZFS"); + assert.equal(notes[0].level, "info"); + assert.match(notes[0].fix!, /smartmontools/); + }); + + test("a failure is a warning that says what the agent said, with a fix for the system it is on", () => { + const failed = stored([{ check: "updates", status: "failed", reason: "error", detail: "apt did not answer" }]); + const linux = describeDiagnostics(parseDiagnostics(failed), ctx())[0]; + assert.equal(linux.level, "warning"); + assert.equal(linux.detail, "apt did not answer"); + assert.match(linux.fix!, /--check-updates/); + const windows = describeDiagnostics(parseDiagnostics(failed), ctx({ osType: "windows", agentVersion: 5 }))[0]; + assert.match(windows.fix!, /Windows Update/); + }); + + test("a step that didn't run is a warning, and warnings come first", () => { + const notes = describeDiagnostics(parseDiagnostics(stored([{ check: "hardware", status: "unavailable", reason: "not_found" }, { check: "prepare", status: "unavailable", reason: "step_missing" }])), ctx()); + assert.deepEqual(notes.map((n) => [n.check, n.level]), [["prepare", "warning"], ["hardware", "info"]]); + }); + + test("something it has no words for is still said, rather than dropped", () => { + const notes = describeDiagnostics(parseDiagnostics(stored([{ check: "gpu_temps", status: "failed", reason: "error" }, { check: "fans", status: "unavailable", reason: "no_sensor" }])), ctx()); + assert.equal(notes[0].level, "warning"); + assert.match(notes[0].title, /gpu_temps/); + assert.match(notes[1].title, /fans isn't available \(no sensor\)/); + }); + + test("an agent that is behind is told so, whether or not it said anything", () => { + const notes = describeDiagnostics(null, ctx({ agentVersion: 3 })); + assert.equal(notes.length, 1); + assert.match(notes[0].title, /version 3; 6 is current/); + assert.deepEqual(describeDiagnostics(null, ctx()), [], "an agent that is current and silent has nothing to say"); + assert.deepEqual(describeDiagnostics(null, ctx({ lastSeenAt: null, agentVersion: 3 })), [], "a server that has never reported has no agent to update"); + }); + + test("what is stored is read leniently", () => { + assert.equal(parseDiagnostics(null), null); + assert.equal(parseDiagnostics("not json"), null); + const read = parseDiagnostics(stored([{ check: "ok", status: "fine" }, "x", { check: "smart", status: "failed", detail: "a\nb" }]))!; + assert.deepEqual(read.items, [{ check: "smart", status: "failed", detail: "a b" }]); + }); +}); + +describe("services a server is told to ignore", () => { + test("names and patterns, in any case, with or without .service", () => { + assert.equal(isServiceIgnored("gupdate.service", ["gupdate"]), true); + assert.equal(isServiceIgnored("GUpdate", ["gupdate*"]), true); + assert.equal(isServiceIgnored("snap.lxd.activate.service", ["snap.*"]), true); + assert.equal(isServiceIgnored("nginx.service", ["gupdate*", "snap.*"]), false); + assert.equal(isServiceIgnored("nginx.service", ["nginx.service"]), true); + assert.equal(isServiceIgnored("mynginx.service", ["nginx"]), false, "a name is the whole name"); + assert.equal(isServiceIgnored("a+b.service", ["a+b"]), true, "characters that mean something to a regex are only characters here"); + assert.equal(isServiceIgnored("anything", []), false); + assert.equal(isServiceIgnored("anything", undefined), false); + }); + + test("left out for what is judged, marked for what is shown", () => { + const health = { failedUnits: [{ unit: "nginx.service", description: "" }, { unit: "gupdate", description: "Google Update" }], reportedAt: "t" }; + assert.deepEqual(withoutIgnoredServices(health, ["gupdate*"])!.failedUnits!.map((u) => u.unit), ["nginx.service"]); + assert.deepEqual(markIgnoredServices(health, ["gupdate*"])!.failedUnits!.map((u) => [u.unit, u.ignored ?? false]), [["nginx.service", false], ["gupdate", true]]); + assert.equal(withoutIgnoredServices(null, ["x"]), null); + assert.deepEqual(withoutIgnoredServices(health, []), health); + }); + + test("the list is part of a server's own thresholds: kept tidy, and a wrong one is refused", () => { + const ok = checkThresholds({ ignoreServices: [" gupdate* ", "gupdate*", "", "snap.*"] }); + assert.deepEqual(ok.ok && ok.value, { ignoreServices: ["gupdate*", "snap.*"] }); + assert.equal(checkThresholds({ ignoreServices: "gupdate" }).ok, false); + assert.equal(checkThresholds({ ignoreServices: [3] }).ok, false); + assert.equal(checkThresholds({ ignoreServices: ["x".repeat(101)] }).ok, false); + assert.equal(checkThresholds({ ignoreServices: Array.from({ length: 31 }, (_, i) => `s${i}`) }).ok, false); + assert.equal(checkThresholds({ ignoreServices: [] }).ok && (checkThresholds({ ignoreServices: [] }) as { value: unknown }).value, null, "an empty list is no thresholds at all"); + assert.deepEqual(parseThresholds(JSON.stringify({ ignoreServices: ["a", 3, "", "a"] })), { ignoreServices: ["a"] }); + }); +}); + +describe("through a report, the page, the alerts and the figures", () => { + let app: Server; + let base = ""; + let id = 0; + + before(async () => { + await db.insert(users).values([{ oidcSub: "admin", name: "Ada", role: "admin" }]); + const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("tok"), apiTokenPrefix: "t", lastSeenAt: new Date().toISOString(), agentVersion: 6 }).returning(); + id = s.id; + const a = express(); + a.use("/api/agent/report", agentReportRouter); + a.use(express.json()); + a.use((req, _res, next) => { + (req as unknown as { session: unknown }).session = { user: { sub: "admin" } }; + next(); + }); + a.use("/api/servers", serversRouter); + app = await new Promise((resolve) => { + const x = a.listen(0, "127.0.0.1", () => resolve(x)); + }); + base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`; + }); + + after(() => { + app.close(); + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + // Windows keeps the database file locked until the process exits. + } + }); + + const report = (system: unknown) => + fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer tok" }, body: JSON.stringify({ hostname: "h", os_type: "linux", agent_version: 6, tasks: [], system }) }); + const detail = async () => await (await fetch(`${base}/servers/${id}/detail`)).json(); + const patch = (body: unknown) => fetch(`${base}/servers/${id}`, { method: "PATCH", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }); + + test("what the agent says it couldn't collect is shown with what to do; garbage entries are dropped; silence keeps what was said", async () => { + assert.equal( + (await report({ diagnostics: [{ check: "smart", status: "unavailable", reason: "tool_missing", detail: "smartctl" }, { check: "x", status: "bogus" }, "junk", { status: "failed" }] })).status, + 202, + ); + let d = (await detail()).diagnostics; + assert.deepEqual(d.map((n: { check: string }) => n.check), ["smart"]); + assert.match(d[0].fix, /smartmontools/); + + assert.equal((await report({ cpu: { cores: 2 } })).status, 202); + assert.equal((await detail()).diagnostics.length, 1, "a report that doesn't say keeps what was said"); + assert.equal((await report({ diagnostics: "garbage" })).status, 202); + assert.equal((await detail()).diagnostics.length, 1, "and so does one that isn't a list"); + + assert.equal((await report({ diagnostics: [] })).status, 202); + assert.deepEqual((await detail()).diagnostics, [], "an empty list says nothing was missed"); + const row = (await (await fetch(`${base}/servers`)).json()).servers.find((x: { id: number }) => x.id === id); + assert.equal(row.agentDiagnostics, undefined, "never as raw stored text"); + }); + + test("a service the server is told to ignore stops being an alert and a figure, still shows on the page marked, and can be un-ignored", async () => { + assert.equal((await report({ health: { failed_units: [{ unit: "nginx.service", description: "web" }, { unit: "gupdate.service", description: "Google Update" }] } })).status, 202); + invalidateAlertsCache(); + const failing = async () => (await getAlerts(true)).alerts.find((a) => a.id.startsWith("services:failed-units") && a.id.includes(`:server:${id}`)); + assert.match((await failing())!.message, /2 services have failed/); + + assert.equal((await patch({ thresholds: { ignoreServices: ["gupdate*"] } })).status, 200); + assert.match((await failing())!.message, /1 service has failed — nginx\.service/, "the alert cache was dropped when the list changed"); + assert.match(await buildMetrics(), new RegExp(`homelab_server_failed_units\\{server="elsa"[^}]*\\} 1`)); + + const units = (await detail()).server.hostHealth.failedUnits; + assert.deepEqual(units.map((u: { unit: string; ignored?: boolean }) => [u.unit, u.ignored ?? false]), [["nginx.service", false], ["gupdate.service", true]]); + + assert.equal((await patch({ thresholds: { ignoreServices: ["gupdate*", "nginx"] } })).status, 200); + assert.equal(await failing(), undefined, "with every failed service ignored there is nothing to alert about"); + + assert.equal((await patch({ thresholds: null })).status, 200); + assert.match((await failing())!.message, /2 services have failed/); + }); + + test("a list that isn't one is refused with a message", async () => { + const res = await patch({ thresholds: { ignoreServices: "gupdate" } }); + assert.equal(res.status, 400); + assert.match((await res.json()).message, /should be a list/); + }); +}); diff --git a/server/test/windowsAgentHealth.test.ts b/server/test/windowsAgentHealth.test.ts index 6ac6528..b82c0d1 100644 --- a/server/test/windowsAgentHealth.test.ts +++ b/server/test/windowsAgentHealth.test.ts @@ -32,9 +32,9 @@ function run(snippet: string): unknown { } describe("the Windows agent's health parts", { skip: !havePwsh }, () => { - test("it says it is version 4, and stays plain ASCII", () => { + test("it says it is version 5, and stays plain ASCII", () => { const text = readFileSync(script, "utf8"); - assert.match(text, /^\$script:AgentVersion = 4\s*$/m); + assert.match(text, /^\$script:AgentVersion = 5\s*$/m); assert.ok(!/[^\x00-\x7f]/.test(text)); }); @@ -103,6 +103,22 @@ describe("the Windows agent's health parts", { skip: !havePwsh }, () => { assert.deepEqual(out.map((p) => [p.name, p.health, p.capacity_percent]), [["Media", "ONLINE", 25], ["Backups", "DEGRADED", 90], ["Broken", "FAULTED", null]]); }); + test("what could not be read is noted, with a word for why and at most a line of what Windows said", () => { + const out = run(` + function Get-CimInstance { throw "WMI is unhappy\`r\`nsecond line" } + $WarningPreference = 'SilentlyContinue' + $null = Get-HostHealth @{ IgnoreServices = @() } + Add-Diag 'hardware' 'unavailable' 'not_found' + Add-Diag 'updates' 'failed' 'error' ('x' * 400) + ConvertTo-Json -InputObject @{ items = @($script:Diag) } -Depth 4 -Compress + `) as { items: { check: string; status: string; reason?: string; detail?: string }[] }; + const byCheck = Object.fromEntries(out.items.map((i) => [i.check, i])); + assert.deepEqual([byCheck.failed_units.status, byCheck.failed_units.reason], ["failed", "error"]); + assert.equal(byCheck.failed_units.detail, "WMI is unhappy second line", "line breaks are flattened"); + assert.deepEqual(byCheck.hardware, { check: "hardware", status: "unavailable", reason: "not_found" }, "nothing to say about it, no detail"); + assert.equal(byCheck.updates.detail!.length, 160, "cut to a line"); + }); + test("a maker's placeholders aren't a model or a serial number", () => { const out = run(` $t = @('To Be Filled By O.E.M.', 'Default string', 'System Serial Number', 'System Product Name', '0000000', ' ', 'None', 'Unknown', 'ASUS', 'ROG STRIX B560-F', 'PF2ABCDE', '0A1B') diff --git a/web/src/api/client.ts b/web/src/api/client.ts index 4cfa21d..1960aad 100644 --- a/web/src/api/client.ts +++ b/web/src/api/client.ts @@ -632,7 +632,7 @@ export interface AgentLatest { /** What a Linux agent says about the machine itself. Each part is there only if the agent could find it out. */ export interface HostHealth { - failedUnits?: { unit: string; description: string }[]; + failedUnits?: { unit: string; description: string; /** This server is told to ignore it: it doesn't count as failed. */ ignored?: boolean }[]; smart?: { device: string; model: string; healthy: boolean | null; temperatureC: number | null; wearPercent: number | null }[]; zfs?: { name: string; health: string; capacityPercent: number | null }[]; reportedAt: string; @@ -655,6 +655,17 @@ export interface ServerThresholds { diskForecastDays?: number; backupStaleHours?: number; mounts?: Record; + /** Services never to count as failed here: names or patterns like "gupdate*". */ + ignoreServices?: string[]; +} + +/** One thing worth telling a person about a server's agent: what it couldn't collect (and what to do), or that it is behind. */ +export interface AgentDiagnosticNote { + check: string; + level: "warning" | "info"; + title: string; + fix: string | null; + detail: string | null; } export interface UpdateRun { @@ -746,6 +757,8 @@ export interface ServerDetail { links: ServerLink[]; relations: { parent: ServerRelation | null; children: ServerRelation[] }; agentLatest: AgentLatest; + /** What its agent couldn't collect, with what to do about it. */ + diagnostics: AgentDiagnosticNote[]; /** The limits in Settings, which this server follows wherever it has none of its own. */ thresholdDefaults: HealthCheckSettings; } diff --git a/web/src/components/AgentDiagnostics.tsx b/web/src/components/AgentDiagnostics.tsx new file mode 100644 index 0000000..48d7dce --- /dev/null +++ b/web/src/components/AgentDiagnostics.tsx @@ -0,0 +1,39 @@ +import type { AgentDiagnosticNote } from "../api/client"; + +/** The fixes put commands between backticks: those are shown as code. */ +const withCode = (text: string) => text.split("`").map((part, i) => (i % 2 === 1 ? {part} : part)); + +/** + * What the server's agent couldn't collect, and what to do about it — the answer to "why is there no disk health here?". A tip is how this + * machine is (a virtual machine with no serial number); a warning is something to look at. Nothing to say means no card. + */ +export default function AgentDiagnostics({ notes }: { notes: AgentDiagnosticNote[] }) { + if (notes.length === 0) return null; + const warnings = notes.filter((n) => n.level === "warning").length; + return ( +
+
+

What the agent couldn’t collect

+
{warnings > 0 ? {warnings} to look at : tips}
+
+
+ {notes.map((n) => ( +
+
+ {n.level === "warning" ? "look at" : "tip"} +
+
{n.title}
+ {n.detail && ( +
+ It said: {n.detail} +
+ )} + {n.fix &&
{withCode(n.fix)}
} +
+
+
+ ))} +
+
+ ); +} diff --git a/web/src/components/HostHealthCard.tsx b/web/src/components/HostHealthCard.tsx index 1f7c512..a68b742 100644 --- a/web/src/components/HostHealthCard.tsx +++ b/web/src/components/HostHealthCard.tsx @@ -1,3 +1,4 @@ +import { useState } from "react"; import type { HostHealth } from "../api/client"; import TimeAgo from "./TimeAgo"; @@ -9,18 +10,41 @@ const poolFine = (health: string) => health.toUpperCase() === "ONLINE"; * whether each storage pool (ZFS, or Storage Spaces on Windows) is online. Each part appears only if the agent reported it (no smartctl or no ZFS on the host means no part), * and the card is left out when it has nothing to say. */ -export default function HostHealthCard({ health }: { health: HostHealth | null }) { - if (!health) return null; - const { failedUnits, smart, zfs } = health; +export default function HostHealthCard({ + health, + ignored = [], + onIgnoredChange, +}: { + health: HostHealth | null; + /** The services this server is told to ignore (names or patterns). */ + ignored?: string[]; + /** Given when the person can change that list; it is saved by whoever holds it. */ + onIgnoredChange?: (list: string[]) => Promise | void; +}) { + const [pattern, setPattern] = useState(""); + const [busy, setBusy] = useState(false); + if (!health && ignored.length === 0) return null; + const { failedUnits, smart, zfs } = health ?? { failedUnits: undefined, smart: undefined, zfs: undefined, reportedAt: "" }; + + async function change(list: string[]) { + if (!onIgnoredChange) return; + setBusy(true); + try { + await onIgnoredChange(list); + setPattern(""); + } finally { + setBusy(false); + } + } const hasSmart = !!smart && smart.length > 0; const hasZfs = !!zfs && zfs.length > 0; - if (failedUnits === undefined && !hasSmart && !hasZfs) return null; + if (failedUnits === undefined && !hasSmart && !hasZfs && ignored.length === 0) return null; return (

Host health

- {health.reportedAt && ( + {health?.reportedAt && (
reported
@@ -30,15 +54,21 @@ export default function HostHealthCard({ health }: { health: HostHealth | null } {failedUnits !== undefined && (
Services
- {failedUnits.length === 0 ? ( + {failedUnits.filter((u) => !u.ignored).length === 0 ? ( none have failed - ) : ( + ) : null} + {failedUnits.length > 0 && (
    {failedUnits.map((u) => ( -
  • - failed +
  • + {u.ignored ? ignored : failed} {u.unit} {u.description && {u.description}} + {onIgnoredChange && !u.ignored && ( + + )}
  • ))}
@@ -46,6 +76,38 @@ export default function HostHealthCard({ health }: { health: HostHealth | null }
)} + {(ignored.length > 0 || onIgnoredChange) && ( +
+
Services this server ignores
+ {ignored.length === 0 &&
None. A service that always fails here (an updater, say) can be ignored, so it stops raising an alert.
} +
+ {ignored.map((p) => ( + + {p} + {onIgnoredChange && ( +
+ {onIgnoredChange && ( +
{ + e.preventDefault(); + const p = pattern.trim(); + if (p && !ignored.includes(p)) void change([...ignored, p]); + }} + > + setPattern(e.target.value)} /> + +
+ )} +
+ )} + {hasSmart && (
Disks (SMART)
diff --git a/web/src/components/ServerThresholds.tsx b/web/src/components/ServerThresholds.tsx index a786138..e7f5b0e 100644 --- a/web/src/components/ServerThresholds.tsx +++ b/web/src/components/ServerThresholds.tsx @@ -13,7 +13,7 @@ const FIELDS: { key: Field; label: string; unit: string; min: number; max: numbe { key: "backupStaleHours", label: "Its backup is too old after", unit: "hours", min: 0, max: 2160, hint: "For the Proxmox guest this server is. 0 means it is never called stale.", onlyForGuests: true }, ]; -const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts") || Object.keys(t.mounts ?? {}).length > 0); +const has = (t: Thresholds | null) => !!t && (Object.keys(t).some((k) => k !== "mounts" && k !== "ignoreServices") || Object.keys(t.mounts ?? {}).length > 0); /** * This server's own alert limits. Anything left empty follows Settings → Notifications, so changing the global limit still reaches every @@ -53,6 +53,8 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC if (rule.percent !== undefined || rule.ignore) rules[m] = rule; } if (Object.keys(rules).length > 0) out.mounts = rules; + // The services to ignore are edited on the health card; saving limits here must not lose them. + if (own?.ignoreServices?.length) out.ignoreServices = own.ignoreServices; return Object.keys(out).length > 0 ? out : null; } @@ -195,7 +197,7 @@ export default function ServerThresholds({ server, defaults, disks, canEdit, onC {busy ? "Saving…" : "Save"} {has(own) && ( - )} diff --git a/web/src/pages/ServerDetail.tsx b/web/src/pages/ServerDetail.tsx index bfdb4f6..9755506 100644 --- a/web/src/pages/ServerDetail.tsx +++ b/web/src/pages/ServerDetail.tsx @@ -19,6 +19,9 @@ import TimeAgo from "../components/TimeAgo"; import UptimeRibbon from "../components/UptimeRibbon"; import RunsOn from "../components/RunsOn"; import HostHealthCard from "../components/HostHealthCard"; +import AgentDiagnostics from "../components/AgentDiagnostics"; +import { toast } from "../utils/toast"; +import { readableError } from "../utils/errors"; import ServerThresholds from "../components/ServerThresholds"; import RunPlaybook from "../components/RunPlaybook"; import RunbookPanel from "../components/RunbookPanel"; @@ -809,7 +812,25 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
- + { + try { + // The list is part of the server's own limits; the rest of them are carried over as they are. + await api.servers.update(server.id, { thresholds: { ...(server.thresholds ?? {}), ignoreServices: list.length > 0 ? list : undefined } }); + toast.success(list.length > (server.thresholds?.ignoreServices?.length ?? 0) ? "Ignored. It no longer counts as failed on this server." : "Saved."); + void loadDetail(); + } catch (err) { + toast.error(readableError(err)); + } + } + : undefined + } + /> + {hardware.source === "agent" && } d.mount)} canEdit={isAdmin} onChanged={() => void loadDetail()} /> {(user.role === "admin" || user.role === "operator") && }