Add a weekly digest and report which version of the agent each server runs

- Weekly digest through the notification channels: alerts now, a week of uptime, what happened, disks
  heading for full, expiring secrets and domains, updates waiting, planned maintenance. Off until switched
  on in Settings -> Notifications; Preview and Send it now there. Setting key 'digest'.
- Agents say their version in every report (Linux and Windows scripts are version 2); the server compares
  it with the scripts it serves and marks agents that are behind on the Servers, server and Updates pages.
  servers.agent_version, migration 0026.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-17 19:49:36 +02:00
1 parent 3da5c639b1
commit aa94d84f6c
28 files changed
+3179 -19

No files matched your search

+16 -4
View File
@@ -14,12 +14,24 @@ This page is about how to do that, safely, and how to tell it worked.
| a version that only changed the app | No. Older agents keep working: a report is read the same way as before. | | a version that only changed the app | No. Older agents keep working: a report is read the same way as before. |
| the version that added **pending updates** (the Updates page) | Yes, to see updates. Until then the server's row says *not reported*. Everything else about it carries on as before. | | the version that added **pending updates** (the Updates page) | Yes, to see updates. Until then the server's row says *not reported*. Everything else about it carries on as before. |
Each agent script carries a version number and sends it in every report; Homelab Manager compares it with the
scripts it serves itself, so it can tell you which agents are behind. A change to an agent is only given a new
number when it is worth installing the agent again for.
| Agent version | What it added |
|---|---|
| 1 | Everything before the number was reported — so an agent that says no version is version 1. |
| 2 | Reports pending updates, and its own version. |
Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field Reports are always read leniently — a field an older agent doesn't send is simply left alone, and a field
an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of an agent sends that the server doesn't know yet is ignored — so there is never a rush, and a mixed fleet of
old and new agents is fine. old and new agents is fine.
**How to tell which agents are old.** The agent doesn't report a version number. The visible sign is the **How to tell which agents are old.** Homelab Manager says so: a server whose agent is behind shows an
**Updates** page showing *not reported* for a server that has an agent. On the machine itself: *agent behind* badge on the **Servers** page, *update available* beside **Agent** on its own page, and *behind* in the
**Agent** column of the **Updates** page, which also counts them under *Agents to update*. (An agent that was
updated before version numbers existed reads as "v1 or older" until it has been updated once more.) On the
machine itself:
```bash ```bash
# Linux — prints 1 or more if the script knows about updates, 0 if it doesn't # Linux — prints 1 or more if the script knows about updates, 0 if it doesn't
@@ -122,8 +134,8 @@ playbook*) — a good way to try it on one machine first, then run it on the res
## Checking it worked ## Checking it worked
1. **The Updates page.** The server's row changes from *not reported* to its figures within one reporting 1. **The Updates page.** The server's row changes from *not reported* to its figures, and its **Agent** to the
interval (15 minutes by default; on Windows the installer reports straight away, and on Linux you can current version, within one reporting interval (15 minutes by default; on Windows the installer reports straight away, and on Linux you can
start one now — below). start one now — below).
2. **Last seen** on the server's page moves to *just now*. 2. **Last seen** on the server's page moves to *just now*.
+3 -1
View File
@@ -23,7 +23,7 @@ what the app actually creates.
|---|---| |---|---|
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. | | **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. | | **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0025`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. | | **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0026`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
| **Tables** | 26 application tables, plus drizzle's own `__drizzle_migrations`. | | **Tables** | 26 application tables, plus drizzle's own `__drizzle_migrations`. |
**Not in the database:** **Not in the database:**
@@ -217,6 +217,7 @@ what it last reported.
| `parent_server_id` | integer | The server this one runs on, set by an admin. **Not a foreign key** (a removed host's servers are cleared by the app). Null = work it out: a server linked to a Proxmox guest runs on the one server named after that guest's node. When the parent is down, this server's offline alert is folded under it. | | `parent_server_id` | integer | The server this one runs on, set by an admin. **Not a foreign key** (a removed host's servers are cleared by the app). Null = work it out: a server linked to a Proxmox guest runs on the one server named after that guest's node. When the parent is down, this server's offline alert is folded under it. |
| `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. | | `pending_updates` | text | JSON `{"manager":"apt","count":5,"security":2,"rebootRequired":false,"reportedAt":"…"}` from the agent; null until it has said. `security` is null when the package manager can't tell. |
| `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. | | `updates_pending_since` | text | ISO; when updates were first seen waiting (and none were before). Null when none are waiting. |
| `agent_version` | integer | Which version of the agent script runs on it, as the agent says in each report. Null until it says; an agent that predates the number counts as version 1. Compared with the scripts the app serves to say which agents are behind. |
#### `scheduled_tasks` #### `scheduled_tasks`
@@ -538,6 +539,7 @@ that were never changed aren't stored; the app fills in defaults when reading.
| `gotify`, `ntfy`, `smtp`, `webhook` | The four notification channels: enabled, address, and credentials (token / password / secret). The credential fields are stored encrypted (prefix `enc:v1:`) — see [Retention and backups](#retention-and-backups). | | `gotify`, `ntfy`, `smtp`, `webhook` | The four notification channels: enabled, address, and credentials (token / password / secret). The credential fields are stored encrypted (prefix `enc:v1:`) — see [Retention and backups](#retention-and-backups). |
| `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). | | `notifications` | Which events notify (`dnsAdd`, `healthAlerts`, …), the daily-reminder time (`secretCheckTime`, default `08:00`) and `timezone` (default `UTC`), and the integration-failure threshold (default 3). |
| `quietHours` | `enabled`, `start`, `end`. | | `quietHours` | `enabled`, `start`, `end`. |
| `digest` | The weekly digest: `enabled` (false), `day` (0 = Sunday … 6; default 1, Monday) and `time` (`HH:MM`, default `08:00`), in the notification time zone. |
| `healthChecks` | `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page). | | `healthChecks` | `serverOfflineMinutes` (60), `diskUsagePercent` (90), `domainWarnDays` (30), `diskForecastDays` (14; 0 = off — how far ahead a filling disk is flagged on the Alerts page). |
| `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). | | `logRetention` | `enabled`, `retentionDays` (90), `intervalHours` (24). |
| `display` | `dateFormat`, `timeFormat`, `pageSize`. | | `display` | `dateFormat`, `timeFormat`, `pageSize`. |
+14
View File
@@ -89,6 +89,20 @@ schedule.
|---|---| |---|---|
| "Notifications from quiet hours" | Once, at quiet hours' configured end time, **only if enabled and only if at least one notification was held** during the window. Bundles every held notification's title and message into one message, then clears the queue. | | "Notifications from quiet hours" | Once, at quiet hours' configured end time, **only if enabled and only if at least one notification was held** during the window. Bundles every held notification's title and message into one message, then clears the queue. |
## Weekly digest
| Notification | Fires when |
|---|---|
| "Homelab weekly digest" | Once a week, on the day and at the time chosen under Settings → Notifications (in the notification time zone), **only if switched on** — it is off until then. One plain-text message through every channel that is on; quiet hours hold it like any other notification. |
It gathers what the app already knows into a few lines per topic, and leaves out any topic with nothing to say:
what is wrong **right now**; **uptime** over the last seven days (what was down and for how long, with the percentage);
**what happened** (servers that stopped reporting and came back, services that stopped answering, maintenance);
**disks heading for full**; secrets and domains that are **expiring** or have expired; **updates waiting** (how many
servers, how many security fixes, who needs a restart, the longest wait); and **planned maintenance** in the next seven
days. A week with nothing in it says "A quiet week: nothing needs attention." On the settings page, *Preview* shows
what it would say right now and *Send it now* sends it at once (through the channels that are on, whatever the time).
## The Alerts page ## The Alerts page
**Operations → Alerts** shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and **Operations → Alerts** shows what these notifications are about — the problems that exist *right now* — as a list you can look at, filter, and
+12
View File
@@ -440,6 +440,18 @@ to date"; run its install command again to update its agent (how, and how to
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)). It only looks: to apply
updates, run a playbook. updates, run a playbook.
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
week through your notification channels: what is wrong now, what was down and for how
long, what happened, disks heading for full, things about to expire, updates waiting
and maintenance coming up — each topic only if it has something to say, and "a quiet
week" when nothing does. "Preview" shows what it would say right now and "Send it now"
sends it at once. Details in [NOTIFICATIONS.md](NOTIFICATIONS.md).
**Agent versions.** Each agent script says its version in every report, and Homelab
Manager compares it with the scripts it serves, so a server whose agent is behind is
marked on the Servers page, on its own page and on the Updates page (which counts them).
How to update one: [AGENT_UPDATE.md](AGENT_UPDATE.md).
**Customising the dashboard.** Customise (top of the dashboard) lets you move each **Customising the dashboard.** Customise (top of the dashboard) lets you move each
card earlier or later among its neighbours and hide the ones you don't use — card earlier or later among its neighbours and hide the ones you don't use —
hidden cards stay listed, dimmed, while you arrange so they can be brought back, hidden cards stay listed, dimmed, while you arrange so they can be brought back,
+7 -1
View File
@@ -10,6 +10,11 @@
# #
set -euo pipefail set -euo pipefail
# Which version of this script this is, sent with every report so Homelab Manager can tell which agents are behind. Raise it whenever a
# change here is worth installing the agent again for (and add it to the list at the top of server/src/services/agentVersions.ts).
# 1 = before the number was reported 2 = reports pending updates, and its own version
AGENT_VERSION=2
ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}" ENV_FILE="${ENV_FILE:-/etc/homelab-manager-agent.env}"
if [[ -f "$ENV_FILE" ]]; then if [[ -f "$ENV_FILE" ]]; then
# shellcheck disable=SC1090 # shellcheck disable=SC1090
@@ -333,9 +338,10 @@ PAYLOAD=$(jq -n \
--arg hostname "$HOSTNAME_VALUE" \ --arg hostname "$HOSTNAME_VALUE" \
--arg os_type "linux" \ --arg os_type "linux" \
--arg reported_at "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \ --arg reported_at "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
--argjson agent_version "$AGENT_VERSION" \
--argjson system "$SYSTEM_JSON" \ --argjson system "$SYSTEM_JSON" \
--argjson tasks "$TASKS_JSON" \ --argjson tasks "$TASKS_JSON" \
'{hostname: $hostname, os_type: $os_type, reported_at: $reported_at, system: $system, tasks: $tasks}') '{hostname: $hostname, os_type: $os_type, reported_at: $reported_at, agent_version: $agent_version, system: $system, tasks: $tasks}')
if [[ "$DRY_RUN" == "1" ]]; then if [[ "$DRY_RUN" == "1" ]]; then
echo "$PAYLOAD" | jq . echo "$PAYLOAD" | jq .
+5
View File
@@ -31,6 +31,10 @@ param(
$ErrorActionPreference = 'Stop' $ErrorActionPreference = 'Stop'
# Which version of this script this is, sent with every report so Homelab Manager can tell which agents are behind. Raise it whenever a
# change here is worth installing the agent again for. 1 = before the number was reported 2 = reports pending updates, and its own version
$script:AgentVersion = 2
$script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json' $script:DefaultConfigPath = Join-Path $env:ProgramData 'HomelabManager\agent.json'
# ---- configuration ------------------------------------------------------------ # ---- configuration ------------------------------------------------------------
@@ -374,6 +378,7 @@ function Invoke-Agent {
hostname = Get-Fqdn hostname = Get-Fqdn
os_type = 'windows' os_type = 'windows'
reported_at = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd'T'HH:mm:ss'Z'") reported_at = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd'T'HH:mm:ss'Z'")
agent_version = $script:AgentVersion
system = $system system = $system
tasks = @($tasks) tasks = @($tasks)
} }
@@ -0,0 +1 @@
ALTER TABLE `servers` ADD `agent_version` integer;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -183,6 +183,13 @@
"when": 1792191757544, "when": 1792191757544,
"tag": "0025_clean_the_hand", "tag": "0025_clean_the_hand",
"breakpoints": true "breakpoints": true
},
{
"idx": 26,
"version": "6",
"when": 1792258587758,
"tag": "0026_reflective_northstar",
"breakpoints": true
} }
] ]
} }
+4
View File
@@ -291,6 +291,10 @@ export const servers = sqliteTable("servers", {
pendingUpdates: text("pending_updates"), pendingUpdates: text("pending_updates"),
// When updates were first seen waiting (and none were before) — how long the oldest has been waiting. Null when none are. // When updates were first seen waiting (and none were before) — how long the oldest has been waiting. Null when none are.
updatesPendingSince: text("updates_pending_since"), updatesPendingSince: text("updates_pending_since"),
// Which version of the agent script is running on it, as the agent says in each report (see agent/*/report-tasks.*). Null until it says:
// an agent that predates the number being reported is version 1.
agentVersion: integer("agent_version"),
}); });
export const scheduledTasks = sqliteTable("scheduled_tasks", { export const scheduledTasks = sqliteTable("scheduled_tasks", {
+2
View File
@@ -10,6 +10,7 @@ import { initDockerUpdateScheduler } from "./services/dockerUpdateScheduler.js";
import { initProxmoxBackupScheduler } from "./services/proxmoxBackupScheduler.js"; import { initProxmoxBackupScheduler } from "./services/proxmoxBackupScheduler.js";
import { initPbsVerificationScheduler } from "./services/pbsVerificationScheduler.js"; import { initPbsVerificationScheduler } from "./services/pbsVerificationScheduler.js";
import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js"; import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js";
import { initWeeklyDigestScheduler } from "./services/weeklyDigestScheduler.js";
import { initHealthScheduler } from "./services/healthScheduler.js"; import { initHealthScheduler } from "./services/healthScheduler.js";
import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js"; import { initAlertTicketScheduler } from "./services/alertTicketScheduler.js";
import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js"; import { initKumaMaintenanceScheduler } from "./services/kumaMaintenanceScheduler.js";
@@ -40,6 +41,7 @@ await initDockerUpdateScheduler();
await initProxmoxBackupScheduler(); await initProxmoxBackupScheduler();
await initPbsVerificationScheduler(); await initPbsVerificationScheduler();
await initQuietHoursScheduler(); await initQuietHoursScheduler();
await initWeeklyDigestScheduler();
await initHealthScheduler(); await initHealthScheduler();
await initAlertTicketScheduler(); await initAlertTicketScheduler();
await initKumaMaintenanceScheduler(); await initKumaMaintenanceScheduler();
+6
View File
@@ -65,6 +65,11 @@ const reportSchema = z.object({
hostname: text(255).optional(), hostname: text(255).optional(),
os_type: text(50).optional(), os_type: text(50).optional(),
reported_at: text(50).optional(), reported_at: text(50).optional(),
// Lenient like the rest: a number that makes no sense is dropped, never allowed to cost the agent its report.
agent_version: z
.unknown()
.optional()
.transform((v) => (typeof v === "number" && Number.isInteger(v) && v >= 1 && v <= 100_000 ? v : undefined)),
system: systemSchema.nullable().optional(), system: systemSchema.nullable().optional(),
tasks: z tasks: z
.array( .array(
@@ -111,6 +116,7 @@ agentReportRouter.post("/", asyncHandler(authenticateAgent), express.json({ limi
await syncServerTasks(server.id, { await syncServerTasks(server.id, {
hostname: parsed.data.hostname, hostname: parsed.data.hostname,
osType: parsed.data.os_type, osType: parsed.data.os_type,
agentVersion: parsed.data.agent_version,
system: parsed.data.system, system: parsed.data.system,
tasks: parsed.data.tasks.map((t) => ({ tasks: parsed.data.tasks.map((t) => ({
scheduleType: t.schedule_type, scheduleType: t.schedule_type,
+7 -2
View File
@@ -8,6 +8,7 @@ import { generateApiToken } from "../services/tokens.js";
import { recordAudit } from "../services/audit.js"; import { recordAudit } from "../services/audit.js";
import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js"; import { loadParentRows, resolveParents, wouldLoop } from "../services/serverParents.js";
import { parsePendingUpdates } from "../services/pendingUpdates.js"; import { parsePendingUpdates } from "../services/pendingUpdates.js";
import { latestAgentVersions } from "../services/agentVersions.js";
import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js"; import { getServerMetrics, RETENTION_DAYS } from "../services/serverMetrics.js";
import { getServerActivity } from "../services/serverActivity.js"; import { getServerActivity } from "../services/serverActivity.js";
import { asyncHandler } from "../utils/asyncHandler.js"; import { asyncHandler } from "../utils/asyncHandler.js";
@@ -42,6 +43,8 @@ serversRouter.get("/", asyncHandler(async (_req, res) => {
const rows = await db.query.servers.findMany({ orderBy: (s, { asc }) => [asc(s.name)] }); const rows = await db.query.servers.findMany({ orderBy: (s, { asc }) => [asc(s.name)] });
res.json({ res.json({
servers: rows.map(publicServer), servers: rows.map(publicServer),
// What a new install would be, so the page can say which agents are behind.
agentLatest: latestAgentVersions(),
}); });
})); }));
@@ -199,13 +202,15 @@ serversRouter.get("/updates", asyncHandler(async (_req, res) => {
lastSeenAt: servers.lastSeenAt, lastSeenAt: servers.lastSeenAt,
pendingUpdates: servers.pendingUpdates, pendingUpdates: servers.pendingUpdates,
updatesPendingSince: servers.updatesPendingSince, updatesPendingSince: servers.updatesPendingSince,
agentVersion: servers.agentVersion,
}) })
.from(servers) .from(servers)
.orderBy(servers.name); .orderBy(servers.name);
res.json({ res.json({
servers: rows servers: rows
.filter((r) => r.lastSeenAt !== null) .filter((r) => r.lastSeenAt !== null)
.map((r) => ({ id: r.id, name: r.name, osType: r.osType, lastSeenAt: r.lastSeenAt, updates: parsePendingUpdates(r.pendingUpdates), pendingSince: r.updatesPendingSince })), .map((r) => ({ id: r.id, name: r.name, osType: r.osType, lastSeenAt: r.lastSeenAt, updates: parsePendingUpdates(r.pendingUpdates), pendingSince: r.updatesPendingSince, agentVersion: r.agentVersion })),
agentLatest: latestAgentVersions(),
}); });
})); }));
@@ -356,7 +361,7 @@ serversRouter.get("/:id/detail", asyncHandler(async (req, res) => {
children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)), children: [...links2.entries()].filter(([, l]) => l.parentId === id).map(([childId, l]) => ({ id: childId, name: nameOf.get(childId) ?? "", auto: l.auto })).sort((a, b) => a.name.localeCompare(b.name)),
}; };
res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates) }, ipAddresses, hardware, dnsMatches, links, relations }); res.json({ server: { ...serverOut, tags: parseTags(rawTags), pendingUpdates: parsePendingUpdates(rawUpdates) }, ipAddresses, hardware, dnsMatches, links, relations, agentLatest: latestAgentVersions() });
})); }));
const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) }); const tagsSchema = z.object({ tags: z.array(z.string().max(100)).max(50) });
+27
View File
@@ -12,6 +12,9 @@ import { schedulePbsVerificationCheck } from "../services/pbsVerificationSchedul
import { scheduleLogRetentionPurge } from "../services/logRetentionScheduler.js"; import { scheduleLogRetentionPurge } from "../services/logRetentionScheduler.js";
import { purgeOldLogs } from "../services/logRetention.js"; import { purgeOldLogs } from "../services/logRetention.js";
import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js"; import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js";
import { scheduleWeeklyDigest } from "../services/weeklyDigestScheduler.js";
import { buildDigest, sendDigest } from "../services/weeklyDigest.js";
import { enabledChannelCount } from "../services/notify.js";
import { testGotify, testNtfy, testSmtp, testWebhook, flushQuietHoursQueue } from "../services/notify.js"; import { testGotify, testNtfy, testSmtp, testWebhook, flushQuietHoursQueue } from "../services/notify.js";
import { listQueuedNotifications } from "../services/notificationQueue.js"; import { listQueuedNotifications } from "../services/notificationQueue.js";
import { buildExportPayload, encryptExport, decryptExport, applyImportPayload, type EncryptedExportFile } from "../services/configBackup.js"; import { buildExportPayload, encryptExport, decryptExport, applyImportPayload, type EncryptedExportFile } from "../services/configBackup.js";
@@ -99,6 +102,10 @@ const updateSchema = z.object({
.object({ enabled: z.boolean(), start: z.string().regex(/^\d{2}:\d{2}$/), end: z.string().regex(/^\d{2}:\d{2}$/) }) .object({ enabled: z.boolean(), start: z.string().regex(/^\d{2}:\d{2}$/), end: z.string().regex(/^\d{2}:\d{2}$/) })
.partial() .partial()
.optional(), .optional(),
digest: z
.object({ enabled: z.boolean(), day: z.number().int().min(0).max(6), time: z.string().regex(/^([01]\d|2[0-3]):[0-5]\d$/) })
.partial()
.optional(),
}); });
settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => { settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => {
@@ -123,6 +130,9 @@ settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => {
if (parsed.data.quietHours || parsed.data.notifications) { if (parsed.data.quietHours || parsed.data.notifications) {
await scheduleQuietHoursFlush(); await scheduleQuietHoursFlush();
} }
if (parsed.data.digest || parsed.data.notifications) {
await scheduleWeeklyDigest();
}
await recordAudit({ await recordAudit({
actor: req.currentUser!, actor: req.currentUser!,
@@ -203,6 +213,23 @@ settingsRouter.post("/purge-logs", requireRole("admin"), asyncHandler(async (req
res.json(result); res.json(result);
})); }));
// What the weekly digest would say right now, so it can be read before it is switched on.
settingsRouter.get("/digest/preview", requireRole("admin"), asyncHandler(async (_req, res) => {
const digest = await buildDigest();
res.json({ title: digest.title, message: digest.message, quiet: digest.quiet, channels: await enabledChannelCount() });
}));
// Sends it now, through whatever channels are switched on, at whatever time it is.
settingsRouter.post("/digest/send", requireRole("admin"), asyncHandler(async (req, res) => {
const channels = await enabledChannelCount();
if (channels === 0) {
return res.status(400).json({ error: "no_channel", message: "No notification channel is switched on, so there is nowhere to send it. Set one up above first." });
}
const digest = await sendDigest({ immediately: true });
await recordAudit({ actor: req.currentUser!, category: "settings", action: "send_digest", detail: { channels, quiet: digest.quiet } });
res.json({ sent: true, channels });
}));
settingsRouter.get("/quiet-hours-queue", requireRole("admin"), asyncHandler(async (_req, res) => { settingsRouter.get("/quiet-hours-queue", requireRole("admin"), asyncHandler(async (_req, res) => {
const items = await listQueuedNotifications(); const items = await listQueuedNotifications();
res.json({ count: items.length, items }); res.json({ count: items.length, items });
+62
View File
@@ -0,0 +1,62 @@
/**
* Which version of the agent each server runs, and which version this Homelab Manager would give a new install. An agent is a copy of
* a script that was served at install time, so after the app is updated the agents out on the servers can be behind. Each script carries
* its version as a number (`AGENT_VERSION=2` / `$script:AgentVersion = 2`), says it in every report, and the "latest" is read from the very
* scripts this app serves — so the two can never disagree about what "current" means.
*
* 1 — before the number was reported (so an agent that says nothing is version 1)
* 2 — reports pending updates, and its own version
*/
import { readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
export type AgentOs = "linux" | "windows";
export interface LatestAgentVersions {
linux: number | null;
windows: number | null;
}
// server/src/services → the repository's agent folder (the same place the app serves /agent from; in the image it sits beside server/dist).
const here = dirname(fileURLToPath(import.meta.url));
export const AGENT_DIR = join(here, "..", "..", "..", "agent");
const FILES: Record<AgentOs, string> = { linux: join("linux", "report-tasks.sh"), windows: join("windows", "report-tasks.ps1") };
const PATTERNS: Record<AgentOs, RegExp> = {
linux: /^AGENT_VERSION=(\d+)\s*$/m,
windows: /^\$script:AgentVersion\s*=\s*(\d+)\s*$/m,
};
/** The version a script says it is, or null if it doesn't say. */
export function parseAgentVersion(text: string, os: AgentOs): number | null {
const m = PATTERNS[os].exec(text.replace(/\r\n/g, "\n"));
return m ? Number(m[1]) : null;
}
let cached: LatestAgentVersions | null = null;
/** What the scripts served from here say. Read once (they don't change while the app runs); null for one that can't be read. */
export function latestAgentVersions(dir: string = AGENT_DIR): LatestAgentVersions {
if (cached && dir === AGENT_DIR) return cached;
const read = (os: AgentOs): number | null => {
try {
return parseAgentVersion(readFileSync(join(dir, FILES[os]), "utf8"), os);
} catch {
return null;
}
};
const result = { linux: read("linux"), windows: read("windows") };
if (dir === AGENT_DIR) cached = result;
return result;
}
/**
* Whether a server's agent is behind. Only a server that reports through an agent can be: one that never has (a Proxmox guest tracked
* through its host, say) has none to update. An agent that reports no version is version 1.
*/
export function isAgentOutdated(row: { osType: string; lastSeenAt: string | null; agentVersion: number | null }, latest: LatestAgentVersions): boolean {
if (!row.lastSeenAt) return false;
const newest = row.osType === "windows" ? latest.windows : row.osType === "linux" ? latest.linux : null;
if (newest === null) return false;
return (row.agentVersion ?? 1) < newest;
}
+16
View File
@@ -104,6 +104,22 @@ async function isQuietHoursActive(): Promise<boolean> {
return startMin < endMin ? nowMin >= startMin && nowMin < endMin : nowMin >= startMin || nowMin < endMin; return startMin < endMin ? nowMin >= startMin && nowMin < endMin : nowMin >= startMin || nowMin < endMin;
} }
/** Sends at once through every channel that is switched on, whatever the time — for something a person just asked for. */
export async function notifyNow(title: string, message: string): Promise<void> {
await sendToChannels(title, message);
}
/** How many channels are switched on and filled in enough to send, so a "send it now" can say when there is nowhere to send to. */
export async function enabledChannelCount(): Promise<number> {
const { gotify, ntfy, smtp, webhook } = await getSettings();
return [
gotify.enabled && gotify.url && gotify.token,
ntfy.enabled && ntfy.url && ntfy.topic,
smtp.enabled && smtp.host && smtp.to && smtp.from,
webhook.enabled && webhook.url,
].filter(Boolean).length;
}
export async function notify(title: string, message: string): Promise<void> { export async function notify(title: string, message: string): Promise<void> {
if (await isQuietHoursActive()) { if (await isQuietHoursActive()) {
await enqueueNotification(title, message); await enqueueNotification(title, message);
+11
View File
@@ -99,6 +99,15 @@ export interface HealthCheckSettings {
diskForecastDays: number; diskForecastDays: number;
} }
/** The weekly digest: one message a week through the notification channels. Off until an admin switches it on under Settings → Notifications. */
export interface DigestSettings {
enabled: boolean;
/** 0 = Sunday … 6 = Saturday, in notifications.timezone. */
day: number;
/** "HH:MM", in notifications.timezone. */
time: string;
}
export interface ConsistencySettings { export interface ConsistencySettings {
/** CIDR ranges and single addresses the consistency report ignores entirely — e.g. Docker networks that repeat on many hosts. */ /** CIDR ranges and single addresses the consistency report ignores entirely — e.g. Docker networks that repeat on many hosts. */
excludedRanges: string[]; excludedRanges: string[];
@@ -150,6 +159,7 @@ export interface AppSettings {
logRetention: LogRetentionSettings; logRetention: LogRetentionSettings;
quietHours: QuietHoursSettings; quietHours: QuietHoursSettings;
healthChecks: HealthCheckSettings; healthChecks: HealthCheckSettings;
digest: DigestSettings;
consistency: ConsistencySettings; consistency: ConsistencySettings;
nameGenerator: NameGeneratorSettings; nameGenerator: NameGeneratorSettings;
alertTickets: AlertTicketSettings; alertTickets: AlertTicketSettings;
@@ -182,6 +192,7 @@ const DEFAULTS: AppSettings = {
display: { dateFormat: "ymd", timeFormat: "24h", pageSize: 20 }, display: { dateFormat: "ymd", timeFormat: "24h", pageSize: 20 },
logRetention: { enabled: false, retentionDays: 90, intervalHours: 24 }, logRetention: { enabled: false, retentionDays: 90, intervalHours: 24 },
quietHours: { enabled: false, start: "22:00", end: "07:00" }, quietHours: { enabled: false, start: "22:00", end: "07:00" },
digest: { enabled: false, day: 1, time: "08:00" },
healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14 }, healthChecks: { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14 },
// Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every // Docker's default bridge (172.17.0.0/16) and the pool it hands custom networks from (172.18–31) live here, and every
// Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN. // Docker host repeats them. Shown on the Consistency page, where it can be removed if 172.16/12 is used as a real LAN.
+3
View File
@@ -29,6 +29,8 @@ export interface AgentReport {
hostname?: string; hostname?: string;
/** What the agent says it runs on. Only "linux" and "windows" are recorded; anything else is ignored. */ /** What the agent says it runs on. Only "linux" and "windows" are recorded; anything else is ignored. */
osType?: string; osType?: string;
/** Which version of the agent script is running. Absent from an agent that predates the number. */
agentVersion?: number;
system?: IncomingSystemInfo | null; system?: IncomingSystemInfo | null;
tasks: IncomingTask[]; tasks: IncomingTask[];
} }
@@ -109,6 +111,7 @@ export async function syncServerTasks(serverId: number, report: AgentReport) {
// Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before. // Only stored when the agent reported it, so an agent that predates this (or couldn't tell this time) doesn't wipe what it said before.
...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}), ...(system?.updates && before ? nextUpdateState(before, system.updates, now) : {}),
...(report.hostname ? { hostname: report.hostname } : {}), ...(report.hostname ? { hostname: report.hostname } : {}),
...(report.agentVersion !== undefined ? { agentVersion: report.agentVersion } : {}),
// An agent knows what it runs on better than whoever registered the server did. // An agent knows what it runs on better than whoever registered the server did.
...(report.osType === "linux" || report.osType === "windows" ? { osType: report.osType } : {}), ...(report.osType === "linux" || report.osType === "windows" ? { osType: report.osType } : {}),
...(system?.ip_addresses ? { ipAddresses: JSON.stringify(system.ip_addresses) } : {}), ...(system?.ip_addresses ? { ipAddresses: JSON.stringify(system.ip_addresses) } : {}),
+229
View File
@@ -0,0 +1,229 @@
/**
* The weekly digest: one message, sent through the notification channels, that says how the week went and what is coming — alerts now,
* what was down and for how long, what happened, disks heading for full, things about to expire, updates waiting, planned maintenance.
* Everything in it is already known to the app (the alerts, the uptime record, the audit log, the agents' reports); this only gathers it
* into a few lines per topic and leaves out any topic with nothing to say. Plain text, so it reads in a phone notification and an email alike.
*/
import { db } from "../db/client.js";
import { integrations, secrets, servers } from "../db/schema.js";
import { env } from "../env.js";
import { getAlerts } from "./alerts.js";
import { getChangesSince } from "./changesSince.js";
import { collectDomainAlerts } from "./domainMonitor.js";
import { describeTarget, listUpcomingWindows } from "./maintenance.js";
import { notify, notifyNow } from "./notify.js";
import { parsePendingUpdates } from "./pendingUpdates.js";
import { computeSecretStatus } from "./secretStatus.js";
import { disksFillingSoon, getServerMetrics } from "./serverMetrics.js";
import { getSettings } from "./settingsStore.js";
import { getUptime } from "./uptimeHistory.js";
export const DIGEST_DAYS = 7;
const DAY_MS = 24 * 3600_000;
const MAX_LINES = 8;
export interface Digest {
title: string;
message: string;
/** Nothing in it needs anyone: the quiet week. */
quiet: boolean;
}
interface Section {
heading: string;
lines: string[];
/** A section that only reassures ("every server stayed up") doesn't make the week un-quiet. */
attention: boolean;
}
/** "2 h 40 min", "35 min", "3 d 4 h". */
export function spanWords(minutes: number): string {
const m = Math.max(1, Math.round(minutes));
if (m < 60) return `${m} min`;
const h = Math.floor(m / 60);
if (h < 24) return m % 60 && h < 10 ? `${h} h ${m % 60} min` : `${h} h`;
const d = Math.floor(h / 24);
return h % 24 ? `${d} d ${h % 24} h` : `${d} d`;
}
const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`;
function listWithMore(lines: string[], max = MAX_LINES): string[] {
return lines.length > max ? [...lines.slice(0, max), `…and ${lines.length - max} more`] : lines;
}
function whenWords(date: Date, timezone: string): string {
return new Intl.DateTimeFormat("en-GB", { timeZone: timezone, weekday: "short", day: "numeric", month: "short", hour: "2-digit", minute: "2-digit", hourCycle: "h23" }).format(date);
}
function dayWords(date: Date, timezone: string): string {
return new Intl.DateTimeFormat("en-GB", { timeZone: timezone, weekday: "short", day: "numeric", month: "short", year: "numeric" }).format(date);
}
export async function buildDigest(now: Date = new Date()): Promise<Digest> {
const settings = await getSettings();
const tz = settings.notifications.timezone || "UTC";
const sections: Section[] = [];
const serverRows = await db
.select({ id: servers.id, name: servers.name, lastSeenAt: servers.lastSeenAt, pendingUpdates: servers.pendingUpdates, updatesPendingSince: servers.updatesPendingSince })
.from(servers);
const serverName = new Map(serverRows.map((s) => [s.id, s.name]));
// ── Right now ──
try {
const report = await getAlerts(false);
const active = report.alerts.filter((a) => !a.silenced && a.severity !== "info");
const lines: string[] = [];
if (active.length === 0) lines.push(report.couldntCheck.length > 0 ? `No active alerts, but ${plural(report.couldntCheck.length, "check")} couldn't run.` : "No active alerts.");
else {
lines.push(`${report.counts.critical} critical, ${report.counts.warning} warning`);
// Things about to expire have a section of their own, with dates; they are counted here but not listed twice.
const listed = active.filter((a) => a.category !== "expiry");
lines.push(...listWithMore(listed.map((a) => `- ${a.message}`), 5));
if (listed.length < active.length) lines.push(`(${plural(active.length - listed.length, "more is", "more are")} about things expiring, below)`);
}
const held: string[] = [];
if (report.counts.silenced > 0) held.push(`${report.counts.silenced} silenced by maintenance`);
if (report.counts.grouped > 0) held.push(`${report.counts.grouped} folded under a server that is down`);
if (held.length > 0) lines.push(`(${held.join(", ")})`);
sections.push({ heading: "Right now", lines, attention: active.length > 0 || report.couldntCheck.length > 0 });
} catch (err) {
console.error("[digest] couldn't read the alerts:", err);
}
// ── Uptime ──
try {
const uptime = await getUptime(now, DIGEST_DAYS);
const integrationName = new Map((await db.select({ id: integrations.id, name: integrations.name }).from(integrations)).map((i) => [i.id, i.name]));
const rows: { name: string; down: number; percent: number }[] = [];
let measured = 0;
for (const [id, series] of Object.entries(uptime.servers)) {
measured++;
const down = series.buckets.reduce((n, b) => n + b.d, 0);
if (down > 0) rows.push({ name: serverName.get(Number(id)) ?? `server ${id}`, down, percent: series.percent });
}
for (const [id, series] of Object.entries(uptime.integrations)) {
measured++;
const down = series.buckets.reduce((n, b) => n + b.d, 0);
if (down > 0) rows.push({ name: integrationName.get(Number(id)) ?? `integration ${id}`, down, percent: series.percent });
}
if (measured > 0) {
rows.sort((a, b) => b.down - a.down);
sections.push({
heading: `Uptime, last ${DIGEST_DAYS} days`,
lines: rows.length === 0 ? ["Everything stayed up."] : listWithMore(rows.map((r) => `- ${r.name}: ${r.percent}% (${spanWords(r.down)} down)`)),
attention: rows.length > 0,
});
}
} catch (err) {
console.error("[digest] couldn't read the uptime:", err);
}
// ── What happened ──
try {
const changes = await getChangesSince({ since: new Date(now.getTime() - DIGEST_DAYS * DAY_MS), now, actorId: 0, canSeeAudit: false });
if (changes.events.length > 0) {
sections.push({ heading: "What happened", lines: listWithMore(changes.events.map((e) => `- ${e.title}`)), attention: changes.events.some((e) => e.tone === "bad") });
}
} catch (err) {
console.error("[digest] couldn't read what happened:", err);
}
// ── Disks heading for full ──
try {
const { diskForecastDays, diskUsagePercent } = settings.healthChecks;
if (diskForecastDays > 0) {
const lines: string[] = [];
for (const s of serverRows) {
for (const f of disksFillingSoon(await getServerMetrics(s.id, 14, now), diskForecastDays, diskUsagePercent)) {
lines.push(`- ${s.name}: ${f.mount} is ${Math.round(f.latest)}% full and filling by about ${f.perDay.toFixed(1)} points a day — full in about ${plural(f.daysToFull, "day")}`);
}
}
if (lines.length > 0) sections.push({ heading: "Disks heading for full", lines: listWithMore(lines), attention: true });
}
} catch (err) {
console.error("[digest] couldn't read the disk forecasts:", err);
}
// ── About to expire ──
try {
const expiring: { daysLeft: number; text: string }[] = [];
for (const s of await db.select().from(secrets)) {
const status = computeSecretStatus(s.expiryDate, s.warnDays, now);
if (status.status === "expired") expiring.push({ daysLeft: status.daysLeft, text: `Secret "${s.name}" expired ${plural(Math.abs(status.daysLeft), "day")} ago` });
else if (status.status === "expiring") expiring.push({ daysLeft: status.daysLeft, text: `Secret "${s.name}" expires in ${plural(status.daysLeft, "day")} (${s.expiryDate})` });
}
for (const d of (await collectDomainAlerts(now)).expiring) {
expiring.push({ daysLeft: d.daysLeft, text: d.daysLeft < 0 ? `Domain ${d.name} expired ${plural(Math.abs(d.daysLeft), "day")} ago` : `Domain ${d.name} expires in ${plural(d.daysLeft, "day")} (${d.expiresAt})` });
}
if (expiring.length > 0) {
expiring.sort((a, b) => a.daysLeft - b.daysLeft);
sections.push({ heading: "Expiring", lines: listWithMore(expiring.map((e) => `- ${e.text}`)), attention: true });
}
} catch (err) {
console.error("[digest] couldn't read what is expiring:", err);
}
// ── Updates waiting ──
try {
const waiting = serverRows
.map((s) => ({ name: s.name, updates: parsePendingUpdates(s.pendingUpdates), since: s.updatesPendingSince }))
.filter((s) => s.updates && s.updates.count > 0);
if (waiting.length > 0) {
const security = waiting.reduce((n, s) => n + (s.updates!.security ?? 0), 0);
const restarts = waiting.filter((s) => s.updates!.rebootRequired).length;
const oldest = waiting.filter((s) => s.since).sort((a, b) => Date.parse(a.since!) - Date.parse(b.since!))[0];
const lines = [`${plural(waiting.length, "server")} ${waiting.length === 1 ? "has" : "have"} updates waiting${security > 0 ? ` (${plural(security, "security fix", "security fixes")} among them)` : ""}.`];
if (restarts > 0) lines.push(`${plural(restarts, "server")} ${restarts === 1 ? "needs" : "need"} a restart.`);
if (oldest) lines.push(`Longest wait: ${oldest.name}, ${spanWords((now.getTime() - Date.parse(oldest.since!)) / 60_000)}.`);
const withSecurity = waiting.filter((s) => (s.updates!.security ?? 0) > 0).sort((a, b) => (b.updates!.security ?? 0) - (a.updates!.security ?? 0));
lines.push(...listWithMore(withSecurity.map((s) => `- ${s.name}: ${s.updates!.count} waiting, ${s.updates!.security} security`), 5));
sections.push({ heading: "Updates waiting", lines, attention: security > 0 });
}
} catch (err) {
console.error("[digest] couldn't read the pending updates:", err);
}
// ── Planned ──
try {
const horizon = now.getTime() + DIGEST_DAYS * DAY_MS;
const groups = new Map<string, { start: string; end: string; reason: string | null; names: string[] }>();
for (const w of await listUpcomingWindows(now)) {
if (Date.parse(w.startedAt) > horizon) continue;
const target = await describeTarget(w.targetType, w.targetId);
if (!target) continue;
const key = `${w.startedAt}|${w.endsAt}|${w.reason ?? ""}`;
const g = groups.get(key) ?? { start: w.startedAt, end: w.endsAt, reason: w.reason, names: [] };
g.names.push(target.name);
groups.set(key, g);
}
if (groups.size > 0) {
const lines = [...groups.values()]
.sort((a, b) => a.start.localeCompare(b.start))
.map((g) => {
const sorted = [...g.names].sort((a, b) => a.localeCompare(b));
const names = sorted.length <= 3 ? sorted.join(", ") : `${sorted.slice(0, 3).join(", ")} and ${sorted.length - 3} more`;
return `- ${whenWords(new Date(g.start), tz)}, ${spanWords((Date.parse(g.end) - Date.parse(g.start)) / 60_000)}: ${names}${g.reason ? ` — ${g.reason}` : ""}`;
});
sections.push({ heading: "Planned maintenance", lines: listWithMore(lines, 6), attention: false });
}
} catch (err) {
console.error("[digest] couldn't read the planned maintenance:", err);
}
const quiet = !sections.some((s) => s.attention);
const envName = env.environment?.name;
const title = `Homelab weekly digest${envName ? ` (${envName})` : ""}`;
const body = sections.map((s) => `${s.heading.toUpperCase()}\n${s.lines.join("\n")}`).join("\n\n");
const message = `${dayWords(now, tz)}\n${quiet ? "A quiet week: nothing needs attention.\n" : ""}\n${body}`.trim();
return { title, message, quiet };
}
/** Builds the digest and sends it through the channels — held with other notifications during quiet hours, unless `immediately`. */
export async function sendDigest(options: { immediately?: boolean; now?: Date } = {}): Promise<Digest> {
const digest = await buildDigest(options.now);
if (options.immediately) await notifyNow(digest.title, digest.message);
else await notify(digest.title, digest.message);
return digest;
}
@@ -0,0 +1,33 @@
import schedule from "node-schedule";
import { getSettings } from "./settingsStore.js";
import { sendDigest } from "./weeklyDigest.js";
let currentJob: schedule.Job | null = null;
/** The cron rule for "this weekday (0 = Sunday) at HH:MM". */
export function cronForDigest(day: number, time: string): string {
const [h, m] = time.split(":").map(Number);
const d = Number.isInteger(day) && day >= 0 && day <= 6 ? day : 1;
return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} * * ${d}`;
}
/** (Re)schedules the weekly digest per the current settings. Call again after they change. */
export async function scheduleWeeklyDigest(): Promise<void> {
if (currentJob) {
currentJob.cancel();
currentJob = null;
}
const { digest, notifications } = await getSettings();
if (!digest.enabled) {
console.log("[digest] weekly digest is off, nothing scheduled");
return;
}
currentJob = schedule.scheduleJob({ rule: cronForDigest(digest.day, digest.time), tz: notifications.timezone }, () => {
sendDigest().catch((err) => console.error("[digest] couldn't send the weekly digest:", err));
});
console.log(`Weekly digest scheduled: ${cronForDigest(digest.day, digest.time)} (${notifications.timezone})`);
}
export async function initWeeklyDigestScheduler(): Promise<void> {
await scheduleWeeklyDigest();
}
+138
View File
@@ -0,0 +1,138 @@
/**
* Which version of the agent each server runs, and which agents are behind the scripts this app serves.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
import type { Server } from "node:http";
const dir = mkdtempSync(join(tmpdir(), "hm-agentver-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { parseAgentVersion, latestAgentVersions, isAgentOutdated } = await import("../src/services/agentVersions.js");
const { serversRouter } = await import("../src/routes/servers.js");
const { agentReportRouter } = await import("../src/routes/agentReport.js");
const { hashToken } = await import("../src/services/tokens.js");
const { eq } = await import("drizzle-orm");
await runMigrations();
describe("reading a version out of a script", () => {
test("each script says what it is, in the form the server looks for", () => {
assert.equal(parseAgentVersion("#!/usr/bin/env bash\nset -e\n\nAGENT_VERSION=7\nENV_FILE=x\n", "linux"), 7);
assert.equal(parseAgentVersion("$script:AgentVersion = 12\r\n$script:Other = 1\r\n", "windows"), 12);
assert.equal(parseAgentVersion("# AGENT_VERSION=3 in a comment\n", "linux"), null, "only a line that is the assignment counts");
assert.equal(parseAgentVersion("no version here", "windows"), null);
});
test("the scripts this app really serves each carry a version", () => {
const latest = latestAgentVersions();
assert.ok(Number.isInteger(latest.linux) && latest.linux! >= 2, `linux ${latest.linux}`);
assert.ok(Number.isInteger(latest.windows) && latest.windows! >= 2, `windows ${latest.windows}`);
});
test("a folder with nothing in it gives no version, not an error", () => {
const empty = mkdtempSync(join(tmpdir(), "hm-noagent-"));
assert.deepEqual(latestAgentVersions(empty), { linux: null, windows: null });
mkdirSync(join(empty, "linux"));
writeFileSync(join(empty, "linux", "report-tasks.sh"), "AGENT_VERSION=5\n");
assert.deepEqual(latestAgentVersions(empty), { linux: 5, windows: null });
rmSync(empty, { recursive: true, force: true });
});
});
describe("which agents are behind", () => {
const latest = { linux: 3, windows: 2 };
const seen = "2026-10-17T00:00:00Z";
test("an agent that says nothing is version 1, and older than the latest is behind", () => {
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: seen, agentVersion: null }, latest), true);
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: seen, agentVersion: 2 }, latest), true);
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: seen, agentVersion: 3 }, latest), false);
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: seen, agentVersion: 4 }, latest), false, "ahead of this app is not behind");
assert.equal(isAgentOutdated({ osType: "windows", lastSeenAt: seen, agentVersion: 2 }, latest), false, "each system has its own latest");
});
test("a server with no agent has nothing to update, and an unknown latest flags nothing", () => {
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: null, agentVersion: null }, latest), false);
assert.equal(isAgentOutdated({ osType: "linux", lastSeenAt: seen, agentVersion: null }, { linux: null, windows: null }), false);
assert.equal(isAgentOutdated({ osType: "plan9", lastSeenAt: seen, agentVersion: null }, latest), false);
});
});
describe("the version in reports", () => {
let app: Server;
let base = "";
let id = 0;
before(async () => {
await db.insert(users).values({ oidcSub: "admin", name: "Ada", role: "admin" });
const [s] = await db.insert(servers).values({ name: "elsa", apiTokenHash: hashToken("tok"), apiTokenPrefix: "t" }).returning();
id = s.id;
const a = express();
a.use("/api/agent/report", agentReportRouter);
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "admin" } };
next();
});
a.use("/api/servers", serversRouter);
app = await new Promise<Server>((resolve) => {
const x = a.listen(0, "127.0.0.1", () => resolve(x));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api`;
});
after(() => {
app.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
const report = (extra: Record<string, unknown>) =>
fetch(`${base}/agent/report`, { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer tok" }, body: JSON.stringify({ hostname: "h", os_type: "linux", tasks: [], ...extra }) });
test("an agent's version is stored; a report without one, or with nonsense, leaves it as it was", async () => {
assert.equal((await report({})).status, 202);
let [s] = await db.select().from(servers).where(eq(servers.id, id));
assert.equal(s.agentVersion, null, "an old agent says nothing");
assert.equal((await report({ agent_version: 2 })).status, 202);
[s] = await db.select().from(servers).where(eq(servers.id, id));
assert.equal(s.agentVersion, 2);
for (const odd of ["two", 0, -1, 1.5, 999999999, null, {}]) {
assert.equal((await report({ agent_version: odd })).status, 202, `${JSON.stringify(odd)} must not cost the report`);
}
[s] = await db.select().from(servers).where(eq(servers.id, id));
assert.equal(s.agentVersion, 2);
});
test("the lists say what the latest is, and the server's own row says what it runs", async () => {
const list = await (await fetch(`${base}/servers`)).json();
assert.ok(list.agentLatest.linux >= 2 && list.agentLatest.windows >= 2);
assert.equal(list.servers.find((x: { id: number }) => x.id === id).agentVersion, 2);
const detail = await (await fetch(`${base}/servers/${id}/detail`)).json();
assert.equal(detail.server.agentVersion, 2);
assert.ok(detail.agentLatest.linux >= 2);
const updates = await (await fetch(`${base}/servers/updates`)).json();
assert.equal(updates.servers.find((x: { id: number }) => x.id === id).agentVersion, 2);
assert.ok(updates.agentLatest.windows >= 2);
});
});
+202
View File
@@ -0,0 +1,202 @@
/**
* The weekly digest: what it gathers, what it leaves out, and sending it.
*
* Run with: npm test --workspace server
*/
import { test, describe, before, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { AddressInfo } from "node:net";
const dir = mkdtempSync(join(tmpdir(), "hm-digest-"));
process.env.DATABASE_PATH = join(dir, "test.sqlite");
process.env.SESSION_DIR = join(dir, "sessions");
process.env.CREDENTIALS_ENCRYPTION_KEY = "ab".repeat(32);
process.env.HEALTH_STARTUP_GRACE_MS = "0";
const express = (await import("express")).default;
const { db } = await import("../src/db/client.js");
const { users, servers, secrets, auditLog, maintenanceWindows } = await import("../src/db/schema.js");
const { runMigrations } = await import("../src/db/migrate.js");
const { buildDigest, spanWords } = await import("../src/services/weeklyDigest.js");
const { cronForDigest } = await import("../src/services/weeklyDigestScheduler.js");
const { settingsRouter } = await import("../src/routes/settings.js");
const { updateSettings, getSettings } = await import("../src/services/settingsStore.js");
const { invalidateAlertsCache } = await import("../src/services/alerts.js");
await runMigrations();
const HOUR = 3600_000;
const DAY = 24 * HOUR;
const NOW = new Date(Math.floor(Date.now() / 1000) * 1000);
const dbTime = (ms: number) => new Date(ms).toISOString().slice(0, 19).replace("T", " ");
describe("small pieces", () => {
test("spans read as words", () => {
assert.equal(spanWords(35), "35 min");
assert.equal(spanWords(160), "2 h 40 min");
assert.equal(spanWords(60 * 30), "1 d 6 h");
assert.equal(spanWords(60 * 9), "9 h");
assert.equal(spanWords(60 * 24 * 3 + 240), "3 d 4 h");
});
test("the schedule is the chosen weekday and time", () => {
assert.equal(cronForDigest(1, "08:00"), "0 8 * * 1");
assert.equal(cronForDigest(0, "21:30"), "30 21 * * 0");
assert.equal(cronForDigest(9, "nonsense"), "0 8 * * 1", "a bad value falls back rather than scheduling nonsense");
});
});
describe("the quiet week", () => {
test("with nothing wrong it says so, and leaves out topics with nothing to say", async () => {
const digest = await buildDigest(NOW);
assert.equal(digest.quiet, true);
assert.match(digest.message, /A quiet week: nothing needs attention\./);
assert.match(digest.message, /RIGHT NOW\nNo active alerts\./);
for (const heading of ["UPTIME", "WHAT HAPPENED", "DISKS HEADING FOR FULL", "EXPIRING", "UPDATES WAITING", "PLANNED MAINTENANCE"]) {
assert.ok(!digest.message.includes(heading), `no ${heading} section when there's nothing in it`);
}
assert.match(digest.title, /^Homelab weekly digest/);
});
});
describe("a week with things in it", () => {
let app: http.Server;
let base = "";
let webhook: http.Server;
const sent: { content: string }[] = [];
before(async () => {
await db.insert(users).values({ oidcSub: "admin", name: "Ada", role: "admin" });
const made = await db
.insert(servers)
.values([
{ name: "elsa", apiTokenHash: "a", apiTokenPrefix: "a", createdAt: dbTime(NOW.getTime() - 40 * DAY), lastSeenAt: new Date(NOW.getTime() - 5 * HOUR).toISOString() },
{ name: "anna", apiTokenHash: "b", apiTokenPrefix: "b", createdAt: dbTime(NOW.getTime() - 40 * DAY), lastSeenAt: NOW.toISOString(), pendingUpdates: JSON.stringify({ manager: "apt", count: 9, security: 3, rebootRequired: true, reportedAt: NOW.toISOString() }), updatesPendingSince: new Date(NOW.getTime() - 12 * DAY).toISOString() },
{ name: "zoe", apiTokenHash: "c", apiTokenPrefix: "c", createdAt: dbTime(NOW.getTime() - 40 * DAY), lastSeenAt: NOW.toISOString(), pendingUpdates: JSON.stringify({ manager: "apt", count: 0, security: 0, rebootRequired: false, reportedAt: NOW.toISOString() }) },
])
.returning({ id: servers.id });
const [elsa, anna] = made.map((m) => m.id);
// elsa was down for 2 hours three days ago, and has now stopped reporting again.
await db.insert(auditLog).values([
{ category: "server", action: "went_offline", targetType: "server", targetId: String(elsa), actorLabel: "system", createdAt: dbTime(NOW.getTime() - 3 * DAY) },
{ category: "server", action: "back_online", targetType: "server", targetId: String(elsa), actorLabel: "system", createdAt: dbTime(NOW.getTime() - 3 * DAY + 2 * HOUR) },
]);
const date = (days: number) => new Date(NOW.getTime() + days * DAY).toISOString().slice(0, 10);
await db.insert(secrets).values([
{ name: "VPN key", type: "api_token", expiryDate: date(9), warnDays: 30 },
{ name: "Old token", type: "api_token", expiryDate: date(-4), warnDays: 30 },
{ name: "Fine for years", type: "api_token", expiryDate: date(900), warnDays: 30 },
]);
await db.insert(maintenanceWindows).values([elsa, anna].map((id) => ({
targetType: "server" as const,
targetId: id,
reason: "Patch night",
startedAt: new Date(NOW.getTime() + 2 * DAY).toISOString(),
endsAt: new Date(NOW.getTime() + 2 * DAY + 2 * HOUR).toISOString(),
createdBy: "Ada",
})));
webhook = await new Promise<http.Server>((resolve) => {
const s = http.createServer((req, res) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
sent.push(JSON.parse(raw));
res.end("ok");
});
});
s.listen(0, "127.0.0.1", () => resolve(s));
});
const a = express();
a.use(express.json());
a.use((req, _res, next) => {
(req as unknown as { session: unknown }).session = { user: { sub: "admin" } };
next();
});
a.use("/api/settings", settingsRouter);
app = await new Promise<http.Server>((resolve) => {
const s = a.listen(0, "127.0.0.1", () => resolve(s));
});
base = `http://127.0.0.1:${(app.address() as AddressInfo).port}/api/settings`;
invalidateAlertsCache();
});
after(async () => {
// Saving the schedule started a cron job, which would keep this process alive.
await (await import("node-schedule")).default.gracefulShutdown();
app.close();
webhook.close();
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// Windows keeps the database file locked until the process exits.
}
});
test("it gathers each topic into a few lines", async () => {
const digest = await buildDigest(NOW);
assert.equal(digest.quiet, false);
const m = digest.message;
assert.match(m, /RIGHT NOW\n.*critical/, m);
assert.ok(!/RIGHT NOW[^]*Secret "Old token"[^]*EXPIRING/.test(m), "an expiring secret is in its own section, not listed twice");
assert.match(m, /\(1 more is about things expiring, below\)|\(2 more are about things expiring, below\)/);
assert.match(m, /elsa hasn't reported for 5 h/);
assert.match(m, /UPTIME, LAST 7 DAYS\n- elsa: .*% \(.*down\)/s, "elsa's downtime is told with a figure");
assert.match(m, /WHAT HAPPENED\n- elsa was offline for 2 h/);
assert.match(m, /EXPIRING\n- Secret "Old token" expired 4 days ago\n- Secret "VPN key" expires in 9 days/);
assert.ok(!m.includes("Fine for years"));
assert.match(m, /UPDATES WAITING\n1 server has updates waiting \(3 security fixes among them\)\.\n1 server needs a restart\.\nLongest wait: anna, 12 d\.\n- anna: 9 waiting, 3 security/);
assert.match(m, /PLANNED MAINTENANCE\n- .*2 h: anna, elsa — Patch night/);
});
test("preview needs an administrator's route and shows the same text; sending needs somewhere to send to", async () => {
const preview = await (await fetch(`${base}/digest/preview`)).json();
assert.match(preview.title, /^Homelab weekly digest/);
assert.match(preview.message, /UPDATES WAITING/);
assert.equal(preview.channels, 0);
const none = await fetch(`${base}/digest/send`, { method: "POST" });
assert.equal(none.status, 400);
assert.equal((await none.json()).error, "no_channel");
assert.equal(sent.length, 0);
});
test("sending goes out through a switched-on channel at once, and is in the audit log", async () => {
const url = `http://127.0.0.1:${(webhook.address() as AddressInfo).port}/hook`;
await updateSettings({ webhook: { enabled: true, url }, quietHours: { enabled: true, start: "00:00", end: "23:59" } });
const res = await fetch(`${base}/digest/send`, { method: "POST" });
assert.equal(res.status, 200);
assert.deepEqual(await res.json(), { sent: true, channels: 1 });
assert.equal(sent.length, 1, "sent at once even inside quiet hours: a person asked for it now");
assert.match(sent[0].content, /^\*\*Homelab weekly digest/);
assert.match(sent[0].content, /UPDATES WAITING/);
const entries = (await db.select().from(auditLog)).filter((e) => e.action === "send_digest");
assert.equal(entries.length, 1);
await updateSettings({ quietHours: { enabled: false } });
});
test("the schedule is saved, and a day or time that makes no sense is refused", async () => {
const put = (digest: unknown) => fetch(`${base}/`, { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ digest }) });
assert.equal((await put({ enabled: true, day: 0, time: "21:30" })).status, 200);
assert.deepEqual((await getSettings()).digest, { enabled: true, day: 0, time: "21:30" });
for (const bad of [{ day: 7 }, { day: -1 }, { day: 1.5 }, { time: "25:00" }, { time: "8am" }, { enabled: "yes" }]) {
assert.equal((await put(bad)).status, 400, JSON.stringify(bad));
}
assert.deepEqual((await getSettings()).digest, { enabled: true, day: 0, time: "21:30" }, "nothing was half-saved");
});
test("it is off until switched on", async () => {
const fresh = mkdtempSync(join(tmpdir(), "x-"));
rmSync(fresh, { recursive: true, force: true });
const { digest } = await getSettings();
assert.equal(typeof digest.enabled, "boolean");
});
});
+24 -2
View File
@@ -401,6 +401,15 @@ export interface QuietHoursSettings {
end: string; end: string;
} }
/** The weekly digest: one message a week through the notification channels. */
export interface DigestSettings {
enabled: boolean;
/** 0 = Sunday … 6 = Saturday, in the notification time zone. */
day: number;
/** "HH:MM", in the notification time zone. */
time: string;
}
export interface AppSettings { export interface AppSettings {
gotify: GotifySettings; gotify: GotifySettings;
ntfy: NtfySettings; ntfy: NtfySettings;
@@ -413,6 +422,7 @@ export interface AppSettings {
logRetention: LogRetentionSettings; logRetention: LogRetentionSettings;
quietHours: QuietHoursSettings; quietHours: QuietHoursSettings;
healthChecks: HealthCheckSettings; healthChecks: HealthCheckSettings;
digest: DigestSettings;
} }
export type AppSettingsPatch = { [K in keyof AppSettings]?: Partial<AppSettings[K]> }; export type AppSettingsPatch = { [K in keyof AppSettings]?: Partial<AppSettings[K]> };
@@ -496,6 +506,14 @@ export interface ServerRecord {
pendingUpdates: PendingUpdates | null; pendingUpdates: PendingUpdates | null;
/** When updates were first seen waiting (and none were before); null when none are. */ /** When updates were first seen waiting (and none were before); null when none are. */
updatesPendingSince: string | null; updatesPendingSince: string | null;
/** Which version of the agent script it runs, as the agent says; null from one that predates the number (version 1). */
agentVersion: number | null;
}
/** The agent version a new install would be, per system; null when this app can't tell. */
export interface AgentLatest {
linux: number | null;
windows: number | null;
} }
export interface PendingUpdates { export interface PendingUpdates {
@@ -515,6 +533,7 @@ export interface UpdatesServer {
lastSeenAt: string | null; lastSeenAt: string | null;
updates: PendingUpdates | null; updates: PendingUpdates | null;
pendingSince: string | null; pendingSince: string | null;
agentVersion: number | null;
} }
/** What a server runs on, and what runs on it. `auto`: worked out from the Proxmox node rather than set by hand. */ /** What a server runs on, and what runs on it. `auto`: worked out from the Proxmox node rather than set by hand. */
@@ -564,6 +583,7 @@ export interface ServerDetail {
dnsMatches: DnsMatch[]; dnsMatches: DnsMatch[];
links: ServerLink[]; links: ServerLink[];
relations: { parent: ServerRelation | null; children: ServerRelation[] }; relations: { parent: ServerRelation | null; children: ServerRelation[] };
agentLatest: AgentLatest;
} }
export type AlertSeverity = "critical" | "warning" | "info"; export type AlertSeverity = "critical" | "warning" | "info";
@@ -1445,7 +1465,7 @@ export const api = {
clearCache: () => request<{ ok: true }>("/api/dns/cache/clear", { method: "POST" }), clearCache: () => request<{ ok: true }>("/api/dns/cache/clear", { method: "POST" }),
}, },
servers: { servers: {
list: () => request<{ servers: ServerRecord[] }>("/api/servers"), list: () => request<{ servers: ServerRecord[]; agentLatest: AgentLatest }>("/api/servers"),
create: (data: { name: string; hostname?: string; description?: string; osType?: "linux" | "windows" }) => create: (data: { name: string; hostname?: string; description?: string; osType?: "linux" | "windows" }) =>
request<{ server: ServerRecord; token: string }>("/api/servers", { request<{ server: ServerRecord; token: string }>("/api/servers", {
method: "POST", method: "POST",
@@ -1458,7 +1478,7 @@ export const api = {
update: (id: number, data: ServerUpdateInput) => update: (id: number, data: ServerUpdateInput) =>
request<{ server: ServerRecord }>(`/api/servers/${id}`, { method: "PATCH", body: JSON.stringify(data) }), request<{ server: ServerRecord }>(`/api/servers/${id}`, { method: "PATCH", body: JSON.stringify(data) }),
summary: () => request<ServerSummary>("/api/servers/summary"), summary: () => request<ServerSummary>("/api/servers/summary"),
updates: () => request<{ servers: UpdatesServer[] }>("/api/servers/updates"), updates: () => request<{ servers: UpdatesServer[]; agentLatest: AgentLatest }>("/api/servers/updates"),
detail: (id: number) => request<ServerDetail>(`/api/servers/${id}/detail`), detail: (id: number) => request<ServerDetail>(`/api/servers/${id}/detail`),
activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`), activity: (id: number) => request<{ events: ServerActivityEvent[] }>(`/api/servers/${id}/activity`),
metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`), metrics: (id: number, days: number) => request<{ days: number; series: ServerMetricSeries[] }>(`/api/servers/${id}/metrics?days=${days}`),
@@ -1677,5 +1697,7 @@ export const api = {
"/api/settings/quiet-hours-queue", "/api/settings/quiet-hours-queue",
), ),
flushQuietHours: () => request<{ flushed: number }>("/api/settings/flush-quiet-hours", { method: "POST" }), flushQuietHours: () => request<{ flushed: number }>("/api/settings/flush-quiet-hours", { method: "POST" }),
digestPreview: () => request<{ title: string; message: string; quiet: boolean; channels: number }>("/api/settings/digest/preview"),
digestSend: () => request<{ sent: boolean; channels: number }>("/api/settings/digest/send", { method: "POST" }),
}, },
}; };
+12
View File
@@ -20,6 +20,7 @@ import UptimeRibbon from "../components/UptimeRibbon";
import RunsOn from "../components/RunsOn"; import RunsOn from "../components/RunsOn";
import RunPlaybook from "../components/RunPlaybook"; import RunPlaybook from "../components/RunPlaybook";
import UpdatesBadge from "../components/UpdatesBadge"; import UpdatesBadge from "../components/UpdatesBadge";
import { agentState, agentVersionWords } from "../utils/agent";
import ServerTags from "../components/ServerTags"; import ServerTags from "../components/ServerTags";
import ServerTaskTable, { SCHEDULE_TYPE_LABELS } from "../components/ServerTaskTable"; import ServerTaskTable, { SCHEDULE_TYPE_LABELS } from "../components/ServerTaskTable";
import { formatDateTime } from "../utils/date"; import { formatDateTime } from "../utils/date";
@@ -777,6 +778,17 @@ export default function ServerDetail({ user }: { user: CurrentUser }) {
)} )}
{server.lastSeenAt && ( {server.lastSeenAt && (
<> <>
<dt className="col-5">Agent</dt>
<dd className="col-7">
{agentVersionWords(server.agentVersion)}{" "}
{agentState(server, detail.agentLatest) === "behind" ? (
<span className="badge bg-yellow-lt text-yellow" title="Run its install command again to update it (Servers → Manage servers).">
update available
</span>
) : (
<span className="badge bg-green-lt text-green">current</span>
)}
</dd>
<dt className="col-5">Updates</dt> <dt className="col-5">Updates</dt>
<dd className="col-7"> <dd className="col-7">
<UpdatesBadge updates={server.pendingUpdates} /> <UpdatesBadge updates={server.pendingUpdates} />
+19 -3
View File
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useState, type CSSProperties } from "react"; import { useCallback, useEffect, useState, type CSSProperties } from "react";
import { Link, useSearchParams } from "react-router-dom"; import { Link, useSearchParams } from "react-router-dom";
import { api, type CurrentUser, type ServerRecord } from "../api/client"; import { api, type AgentLatest, type CurrentUser, type ServerRecord } from "../api/client";
import CopyButton from "../components/CopyButton"; import CopyButton from "../components/CopyButton";
import { formatDateTime } from "../utils/date"; import { formatDateTime } from "../utils/date";
import TimeAgo from "../components/TimeAgo"; import TimeAgo from "../components/TimeAgo";
@@ -19,6 +19,7 @@ import { toast } from "../utils/toast";
import { SearchBox } from "../components/TableToolbar"; import { SearchBox } from "../components/TableToolbar";
import { useTableSearch } from "../hooks/useTableSearch"; import { useTableSearch } from "../hooks/useTableSearch";
import SaveViewButton from "../components/SaveViewButton"; import SaveViewButton from "../components/SaveViewButton";
import { agentState } from "../utils/agent";
function typeBadgeStyle(colors: Record<string, string>, type: string): CSSProperties { function typeBadgeStyle(colors: Record<string, string>, type: string): CSSProperties {
const color = colors[type]; const color = colors[type];
@@ -31,6 +32,7 @@ export default function Servers({ user }: { user: CurrentUser }) {
const isAdmin = user.role === "admin"; const isAdmin = user.role === "admin";
const [servers, setServers] = useState<ServerRecord[]>([]); const [servers, setServers] = useState<ServerRecord[]>([]);
const [agentLatest, setAgentLatest] = useState<AgentLatest | null>(null);
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const [managingServers, setManagingServers] = useState(false); const [managingServers, setManagingServers] = useState(false);
const [integrationColors, setIntegrationColors] = useState<Record<string, string>>({}); const [integrationColors, setIntegrationColors] = useState<Record<string, string>>({});
@@ -68,7 +70,10 @@ export default function Servers({ user }: { user: CurrentUser }) {
const loadServers = useCallback(() => { const loadServers = useCallback(() => {
return api.servers return api.servers
.list() .list()
.then((res) => setServers(res.servers)) .then((res) => {
setServers(res.servers);
setAgentLatest(res.agentLatest);
})
.catch((err) => setError(String(err))); .catch((err) => setError(String(err)));
}, []); }, []);
@@ -184,7 +189,13 @@ export default function Servers({ user }: { user: CurrentUser }) {
Proxmox Proxmox
</span> </span>
) : s.lastSeenAt ? ( ) : s.lastSeenAt ? (
<span className="badge bg-green-lt text-green">Agent</span> agentState(s, agentLatest) === "behind" ? (
<span className="badge bg-yellow-lt text-yellow" title="Its agent is older than the one this Homelab Manager would install now. Run its install command again to update it.">
Agent behind
</span>
) : (
<span className="badge bg-green-lt text-green">Agent</span>
)
) : ( ) : (
<span className="badge bg-secondary-lt text-secondary">No data</span> <span className="badge bg-secondary-lt text-secondary">No data</span>
)} )}
@@ -385,6 +396,11 @@ export default function Servers({ user }: { user: CurrentUser }) {
<tr key={s.id}> <tr key={s.id}>
<td> <td>
<Link to={`/servers/${s.id}`}>{s.name}</Link> <Link to={`/servers/${s.id}`}>{s.name}</Link>
{agentState(s, agentLatest) === "behind" && (
<span className="badge bg-yellow-lt text-yellow ms-2" title="Its agent is older than the one this Homelab Manager would install now. Run its install command again to update it.">
agent behind
</span>
)}
</td> </td>
<td>{s.hostname ?? "—"}</td> <td>{s.hostname ?? "—"}</td>
<td> <td>
+24 -5
View File
@@ -1,6 +1,7 @@
import { useEffect, useState } from "react"; import { useEffect, useState } from "react";
import { Link } from "react-router-dom"; import { Link } from "react-router-dom";
import { api, type DockhandContainer, type UpdatesServer } from "../api/client"; import { api, type AgentLatest, type DockhandContainer, type UpdatesServer } from "../api/client";
import { agentState, agentVersionWords } from "../utils/agent";
import SortableTh from "../components/SortableTh"; import SortableTh from "../components/SortableTh";
import TableToolbar from "../components/TableToolbar"; import TableToolbar from "../components/TableToolbar";
import TimeAgo from "../components/TimeAgo"; import TimeAgo from "../components/TimeAgo";
@@ -24,6 +25,7 @@ interface Row {
/** ISO — when updates were first seen waiting. */ /** ISO — when updates were first seen waiting. */
since: string | null; since: string | null;
reportedAt: string | null; reportedAt: string | null;
agentVersion: number | null;
server: UpdatesServer; server: UpdatesServer;
} }
@@ -38,6 +40,7 @@ function toRow(s: UpdatesServer): Row {
restart: s.updates?.rebootRequired ?? false, restart: s.updates?.rebootRequired ?? false,
since: s.pendingSince, since: s.pendingSince,
reportedAt: s.updates?.reportedAt || null, reportedAt: s.updates?.reportedAt || null,
agentVersion: s.agentVersion,
server: s, server: s,
}; };
} }
@@ -51,6 +54,7 @@ const DAY_MS = 86_400_000;
*/ */
export default function Updates() { export default function Updates() {
const [servers, setServers] = useState<UpdatesServer[] | null>(null); const [servers, setServers] = useState<UpdatesServer[] | null>(null);
const [agentLatest, setAgentLatest] = useState<AgentLatest | null>(null);
const [containers, setContainers] = useState<(DockhandContainer & { integrationName: string })[] | null>(null); const [containers, setContainers] = useState<(DockhandContainer & { integrationName: string })[] | null>(null);
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const [onlyWaiting, setOnlyWaiting] = useState(false); const [onlyWaiting, setOnlyWaiting] = useState(false);
@@ -59,7 +63,11 @@ export default function Updates() {
let alive = true; let alive = true;
api.servers api.servers
.updates() .updates()
.then((res) => alive && setServers(res.servers)) .then((res) => {
if (!alive) return;
setServers(res.servers);
setAgentLatest(res.agentLatest);
})
.catch((err) => alive && setError(readableError(err))); .catch((err) => alive && setError(readableError(err)));
api.integrations api.integrations
.list() .list()
@@ -91,6 +99,7 @@ export default function Updates() {
const restarts = rows.filter((r) => r.restart).length; const restarts = rows.filter((r) => r.restart).length;
const oldest = waiting.map((r) => r.since).filter((s): s is string => !!s).sort()[0]; const oldest = waiting.map((r) => r.since).filter((s): s is string => !!s).sort()[0];
const notReporting = rows.length - reporting; const notReporting = rows.length - reporting;
const behind = rows.filter((r) => agentState(r.server, agentLatest) === "behind").length;
function exportCsv() { function exportCsv() {
if (!sorted) return; if (!sorted) return;
@@ -105,6 +114,7 @@ export default function Updates() {
{ label: "Servers with updates waiting", value: servers ? `${waiting.length} of ${reporting}` : "…", note: notReporting > 0 ? `${notReporting} haven't reported updates` : undefined, color: waiting.length > 0 ? "text-yellow" : "text-green" }, { label: "Servers with updates waiting", value: servers ? `${waiting.length} of ${reporting}` : "…", note: notReporting > 0 ? `${notReporting} haven't reported updates` : undefined, color: waiting.length > 0 ? "text-yellow" : "text-green" },
{ label: "Security updates waiting", value: servers ? securityTotal : "…", color: securityTotal > 0 ? "text-red" : "text-green" }, { label: "Security updates waiting", value: servers ? securityTotal : "…", color: securityTotal > 0 ? "text-red" : "text-green" },
{ label: "Need a restart", value: servers ? restarts : "…", color: restarts > 0 ? "text-orange" : "text-green" }, { label: "Need a restart", value: servers ? restarts : "…", color: restarts > 0 ? "text-orange" : "text-green" },
{ label: "Agents to update", value: servers ? behind : "…", note: behind > 0 ? "run their install command again" : undefined, color: behind > 0 ? "text-yellow" : "text-green" },
{ label: "Longest wait", value: oldest ? `${Math.max(1, Math.floor((Date.now() - Date.parse(oldest)) / DAY_MS))} d` : servers ? "—" : "…", color: "text-secondary" }, { label: "Longest wait", value: oldest ? `${Math.max(1, Math.floor((Date.now() - Date.parse(oldest)) / DAY_MS))} d` : servers ? "—" : "…", color: "text-secondary" },
]; ];
@@ -120,7 +130,7 @@ export default function Updates() {
<div className="row row-cards mb-3"> <div className="row row-cards mb-3">
{cards.map((c) => ( {cards.map((c) => (
<div className="col-6 col-lg-3" key={c.label}> <div className="col-6 col-lg" key={c.label}>
<div className="card card-sm"> <div className="card card-sm">
<div className="card-body"> <div className="card-body">
<div className={`h2 mb-0 ${c.color}`}>{c.value}</div> <div className={`h2 mb-0 ${c.color}`}>{c.value}</div>
@@ -157,12 +167,13 @@ export default function Updates() {
<SortableTh<Row> label="Security" sortKeyName="security" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="Security" sortKeyName="security" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Waiting since" sortKeyName="since" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="Waiting since" sortKeyName="since" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Reported" sortKeyName="reportedAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> <SortableTh<Row> label="Reported" sortKeyName="reportedAt" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
<SortableTh<Row> label="Agent" sortKeyName="agentVersion" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
{servers === null && !error && ( {servers === null && !error && (
<tr> <tr>
<td colSpan={5}> <td colSpan={6}>
<BlockSkeleton lines={3} /> <BlockSkeleton lines={3} />
</td> </td>
</tr> </tr>
@@ -178,11 +189,19 @@ export default function Updates() {
<td>{r.security === null ? <span className="text-secondary">—</span> : r.security}</td> <td>{r.security === null ? <span className="text-secondary">—</span> : r.security}</td>
<td className="text-secondary">{r.since ? <TimeAgo value={r.since} /> : "—"}</td> <td className="text-secondary">{r.since ? <TimeAgo value={r.since} /> : "—"}</td>
<td className="text-secondary">{r.reportedAt ? <TimeAgo value={r.reportedAt} /> : "—"}</td> <td className="text-secondary">{r.reportedAt ? <TimeAgo value={r.reportedAt} /> : "—"}</td>
<td>
{agentVersionWords(r.agentVersion)}{" "}
{agentState(r.server, agentLatest) === "behind" && (
<span className="badge bg-yellow-lt text-yellow" title="Older than the agent this Homelab Manager would install now.">
behind
</span>
)}
</td>
</tr> </tr>
))} ))}
{servers !== null && sorted?.length === 0 && ( {servers !== null && sorted?.length === 0 && (
<tr> <tr>
<td colSpan={5} className="text-secondary text-center"> <td colSpan={6} className="text-secondary text-center">
{tableSearch.active || onlyWaiting ? "Nothing matches that." : "No server has reported yet."} {tableSearch.active || onlyWaiting ? "Nothing matches that." : "No server has reported yet."}
</td> </td>
</tr> </tr>
+102 -1
View File
@@ -7,9 +7,11 @@ import {
type SmtpSettings, type SmtpSettings,
type WebhookSettings, type WebhookSettings,
type NotificationEvents, type NotificationEvents,
type DigestSettings,
type QuietHoursSettings, type QuietHoursSettings,
type HealthCheckSettings, type HealthCheckSettings,
} from "../../api/client"; } from "../../api/client";
import { readableError } from "../../utils/errors";
import { ErrorNotice } from "../../components/Notice"; import { ErrorNotice } from "../../components/Notice";
import { PageSkeleton } from "../../components/Skeleton"; import { PageSkeleton } from "../../components/Skeleton";
@@ -58,6 +60,17 @@ const DEFAULT_NOTIFICATIONS: NotificationEvents = {
integrationFailureThreshold: 3, integrationFailureThreshold: 3,
}; };
const DEFAULT_QUIET_HOURS: QuietHoursSettings = { enabled: false, start: "22:00", end: "07:00" }; const DEFAULT_QUIET_HOURS: QuietHoursSettings = { enabled: false, start: "22:00", end: "07:00" };
const DEFAULT_DIGEST: DigestSettings = { enabled: false, day: 1, time: "08:00" };
// Monday first, as a week reads; the numbers are what the server keeps (0 = Sunday).
const DIGEST_DAYS: { value: number; label: string }[] = [
{ value: 1, label: "Monday" },
{ value: 2, label: "Tuesday" },
{ value: 3, label: "Wednesday" },
{ value: 4, label: "Thursday" },
{ value: 5, label: "Friday" },
{ value: 6, label: "Saturday" },
{ value: 0, label: "Sunday" },
];
const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14 }; const DEFAULT_HEALTH_CHECKS: HealthCheckSettings = { serverOfflineMinutes: 60, diskUsagePercent: 90, domainWarnDays: 30, diskForecastDays: 14 };
type TestResult = { ok: boolean; message: string } | null; type TestResult = { ok: boolean; message: string } | null;
@@ -85,6 +98,10 @@ export default function NotificationSettings() {
const [notifications, setNotifications] = useState<NotificationEvents>(DEFAULT_NOTIFICATIONS); const [notifications, setNotifications] = useState<NotificationEvents>(DEFAULT_NOTIFICATIONS);
const [quietHours, setQuietHours] = useState<QuietHoursSettings>(DEFAULT_QUIET_HOURS); const [quietHours, setQuietHours] = useState<QuietHoursSettings>(DEFAULT_QUIET_HOURS);
const [healthChecks, setHealthChecks] = useState<HealthCheckSettings>(DEFAULT_HEALTH_CHECKS); const [healthChecks, setHealthChecks] = useState<HealthCheckSettings>(DEFAULT_HEALTH_CHECKS);
const [digest, setDigest] = useState<DigestSettings>(DEFAULT_DIGEST);
const [digestPreview, setDigestPreview] = useState<{ title: string; message: string } | null>(null);
const [digestBusy, setDigestBusy] = useState<"preview" | "send" | null>(null);
const [digestResult, setDigestResult] = useState<TestResult>(null);
const [queuedCount, setQueuedCount] = useState<number | null>(null); const [queuedCount, setQueuedCount] = useState<number | null>(null);
const [flushing, setFlushing] = useState(false); const [flushing, setFlushing] = useState(false);
const [flushResult, setFlushResult] = useState<string | null>(null); const [flushResult, setFlushResult] = useState<string | null>(null);
@@ -109,6 +126,7 @@ export default function NotificationSettings() {
setNotifications(res.settings.notifications); setNotifications(res.settings.notifications);
setQuietHours(res.settings.quietHours); setQuietHours(res.settings.quietHours);
setHealthChecks(res.settings.healthChecks); setHealthChecks(res.settings.healthChecks);
setDigest(res.settings.digest ?? DEFAULT_DIGEST);
}) })
.catch((err) => setLoadError(err instanceof Error ? err.message : String(err))) .catch((err) => setLoadError(err instanceof Error ? err.message : String(err)))
.finally(() => setLoading(false)); .finally(() => setLoading(false));
@@ -123,7 +141,7 @@ export default function NotificationSettings() {
setSaveError(null); setSaveError(null);
setSaved(false); setSaved(false);
try { try {
await api.settings.update({ gotify, ntfy, smtp, webhook, notifications, quietHours, healthChecks }); await api.settings.update({ gotify, ntfy, smtp, webhook, notifications, quietHours, healthChecks, digest });
setSaved(true); setSaved(true);
setTimeout(() => setSaved(false), 3000); setTimeout(() => setSaved(false), 3000);
} catch (err) { } catch (err) {
@@ -133,6 +151,32 @@ export default function NotificationSettings() {
} }
} }
async function handleDigestPreview() {
setDigestBusy("preview");
setDigestResult(null);
try {
const res = await api.settings.digestPreview();
setDigestPreview({ title: res.title, message: res.message });
} catch (err) {
setDigestResult({ ok: false, message: err instanceof Error ? err.message : String(err) });
} finally {
setDigestBusy(null);
}
}
async function handleDigestSend() {
setDigestBusy("send");
setDigestResult(null);
try {
const res = await api.settings.digestSend();
setDigestResult({ ok: true, message: `Sent through ${res.channels} channel${res.channels === 1 ? "" : "s"}.` });
} catch (err) {
setDigestResult({ ok: false, message: readableError(err) });
} finally {
setDigestBusy(null);
}
}
async function handleFlushNow() { async function handleFlushNow() {
setFlushing(true); setFlushing(true);
setFlushResult(null); setFlushResult(null);
@@ -740,6 +784,63 @@ export default function NotificationSettings() {
</div> </div>
</div> </div>
</div> </div>
<div className="col-md-6">
<div className="card">
<div className="card-header">
<h3 className="card-title">Weekly digest</h3>
</div>
<div className="card-body">
<p className="text-secondary">
One message a week through the channels above, in the time zone set above: what is wrong now, what was down and for how long, what happened,
disks heading for full, things about to expire, updates waiting and maintenance coming up. A topic with nothing to say is left out, and a
quiet week says so in one line. Quiet hours hold it like any other notification.
</p>
<label className="form-check mb-3">
<input type="checkbox" className="form-check-input" checked={digest.enabled} onChange={(e) => setDigest((d) => ({ ...d, enabled: e.target.checked }))} />
<span className="form-check-label">Send the weekly digest</span>
</label>
<div className="row g-2 mb-3">
<div className="col-6">
<label className="form-label" htmlFor="digest-day">
Day
</label>
<select id="digest-day" className="form-select" value={digest.day} disabled={!digest.enabled} onChange={(e) => setDigest((d) => ({ ...d, day: Number(e.target.value) }))}>
{DIGEST_DAYS.map((d) => (
<option key={d.value} value={d.value}>
{d.label}
</option>
))}
</select>
</div>
<div className="col-6">
<label className="form-label" htmlFor="digest-time">
Time
</label>
<input id="digest-time" type="time" className="form-control" value={digest.time} disabled={!digest.enabled} onChange={(e) => setDigest((d) => ({ ...d, time: e.target.value }))} />
</div>
</div>
<div className="btn-list">
<button type="button" className="btn btn-outline-secondary" onClick={handleDigestPreview} disabled={digestBusy !== null}>
{digestBusy === "preview" ? "Gathering…" : "Preview"}
</button>
<button type="button" className="btn btn-outline-secondary" onClick={handleDigestSend} disabled={digestBusy !== null}>
{digestBusy === "send" ? "Sending…" : "Send it now"}
</button>
</div>
<div className="form-hint mt-2">Settings above are saved with the Save button; the preview and "send it now" don't depend on them.</div>
<TestBadge result={digestResult} />
{digestPreview && (
<div className="mt-3">
<div className="fw-bold">{digestPreview.title}</div>
<pre className="border rounded p-2 mt-1 mb-0" style={{ whiteSpace: "pre-wrap", fontSize: "0.8125rem" }}>
{digestPreview.message}
</pre>
</div>
)}
</div>
</div>
</div>
</div> </div>
</> </>
); );
+20
View File
@@ -0,0 +1,20 @@
import type { AgentLatest } from "../api/client";
/** What an agent's version means for a server, for the badges: not on an agent at all, current, or behind. */
export type AgentState = "none" | "current" | "behind";
/**
* Whether a server's agent is behind what this Homelab Manager would install today. The same rule the server uses: only a server that
* reports through an agent has one, and an agent that says no version is version 1.
*/
export function agentState(server: { osType: string; lastSeenAt: string | null; agentVersion: number | null }, latest: AgentLatest | null): AgentState {
if (!server.lastSeenAt) return "none";
const newest = latest ? (server.osType === "windows" ? latest.windows : server.osType === "linux" ? latest.linux : null) : null;
if (newest === null || newest === undefined) return "current"; // nothing to compare with: don't cry wolf
return (server.agentVersion ?? 1) < newest ? "behind" : "current";
}
/** "v2", or "v1 or older" for one that has never said. */
export function agentVersionWords(version: number | null): string {
return version === null ? "v1 or older" : `v${version}`;
}