diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md index 41d1f4d..2311150 100644 --- a/INTEGRATIONS.md +++ b/INTEGRATIONS.md @@ -79,6 +79,31 @@ the dashboard views themselves load fine. metrics endpoint only exposes current status, not a monitor's scheduled start/end time, so this reflects what's in maintenance right now rather than mirroring Uptime Kuma's own schedule. +### Proxmox Backup Server + +- **Config fields:** Proxmox Backup Server URL, API token ID, API token secret, allow self-signed certificate +- **Auth:** `PBSAPIToken=:` header — note the **colon** between the token id and secret; Proxmox + VE's own token header uses `=` there instead, so a PVE token/secret pair copied verbatim into this integration's + fields will still format correctly (the adapter supplies the colon itself) as long as the id/secret values + themselves are right. +- **Actions used:** read-only — list datastores and their usage, read every stored snapshot's verification status, + read the PBS host's own CPU/RAM/disk. No writes; nothing here can prune, delete, or re-verify a backup. +- **Required access:** A dedicated API token (Configuration → Access Control → API Token) belonging to a user with + **Datastore.Audit** on the datastore(s) to show, and **Sys.Audit** for the node status card to populate. A token + scoped to just `Datastore.Audit` on one datastore will still work — other datastores it can't read are shown with + an error rather than failing the whole page. +- **What you get, and why it's separate from the Proxmox VE integration:** Proxmox VE (the "Proxmox" integration + above) already shows whether the last `vzdump` push to PBS succeeded, but has no visibility at all into PBS's own + backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by + PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot), + and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed + verification or a datastore couldn't be read at all — the "Notify on" section in Settings → Notifications has a + dedicated toggle for it, sharing the same daily reminder time as the other daily checks. +- **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the + `PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run + it against a real Proxmox Backup Server yet. If a datastore comes back empty or with the wrong fields, tell us + what your instance actually returned and we'll adjust. + ### phpIPAM - **Config fields:** phpIPAM URL, API app ID, App token diff --git a/README.md b/README.md index 0bfca89..5399abd 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ Repository: `git@10.200.5.13:bobban/Homelab-manager.git` ([gitea.labsconnect.se/bobban/Homelab-manager](https://gitea.labsconnect.se/bobban/Homelab-manager) externally). A single dashboard for a homelab: Proxmox, Synology DSM, Semaphore, Tailscale, -Gitea, Dockhand/Docker, and Uptime Kuma status and basic actions, plus DNS record +Gitea, Dockhand/Docker, Uptime Kuma, and Proxmox Backup Server status and basic +actions, plus DNS record management, an IP address inventory (IPAM), and a secret-expiry tracker (ported from [Sloth Manager](../Sloth%20manager)) and scheduled-task tracking across Debian/Raspbian hosts (ported from @@ -37,9 +38,11 @@ All modules from the original plan are built: couldn't be refreshed. The Uptime Kuma widget shows the monitor count, how many are down, and how many are matched to one of your servers. + The Proxmox Backup Server widget shows the datastore count and how many + stored snapshots have failed verification or were never verified. - **Diagnostic Log** (admin-only) — every call this app makes to a DNS provider or integration (Tailscale, Proxmox, Synology, Semaphore, Gitea, - Dockhand, Uptime Kuma), success or failure, with latency and the error message if it + Dockhand, Uptime Kuma, Proxmox Backup Server), success or failure, with latency and the error message if it failed — the last 500 calls, filterable by source/result, for troubleshooting connectivity issues (ported from Sloth Manager's provider-diagnostics log, generalized to cover every integration this app @@ -80,9 +83,9 @@ All modules from the original plan are built: link options") — it stays visible regardless once a server actually is linked, so unlinking is always reachable. - **Tailscale**, **Proxmox**, **Synology**, **Semaphore**, **Gitea**, - **Docker**, and **Uptime Kuma** each get their own top-level page (backed - by the matching integration) instead of living inside a shared Integrations - browsing view: + **Docker**, **Uptime Kuma**, and **Proxmox Backup Server** each get their + own top-level page (backed by the matching integration) instead of living + inside a shared Integrations browsing view: - **Tailscale** — device list with online/authorized status, and authorize/deauthorize/remove actions; a live device-count widget. - **Proxmox** — VM/LXC status across every node in the cluster, with @@ -114,11 +117,18 @@ All modules from the original plan are built: Uptime Kuma has no conventional REST API (the dashboard talks to it over Socket.IO), so this reads its Prometheus `/metrics` endpoint instead and parses that itself; read-only, no actions. + - **Proxmox Backup Server** — every configured datastore's usage and + snapshot count, the PBS host's own CPU/RAM/disk, and each stored + snapshot's verification status, since Proxmox VE only knows whether a + backup *ran*, never whether PBS's own verify pass on the stored data + still passes; a daily check alerts on any snapshot that's failed + verification or a datastore that couldn't be read. Read-only, no + actions (nothing here can prune, delete, or trigger a re-verify). The Integrations page itself is now just a list of configured integrations (name/type/status, visible to every role) with an admin-only "Add integration" button and edit/enable/disable/delete - actions per row — the seven dedicated pages above are where you + actions per row — the eight dedicated pages above are where you actually use each one. - Every table in the app is click-to-sort on any column (numbers, booleans, and dates/text sort correctly regardless of how the column formats them) @@ -154,9 +164,10 @@ assumed HTTPS-only (the NAS is reached over plain HTTP), and the Tailscale adapter read `online`/`isExitNode` fields that don't actually exist in the real API response (fixed to derive them from `connectedToControl` and `enabledRoutes`). See the git log for the full verification notes per -integration. (Uptime Kuma, added later, is not part of that "six" — see -its own git log entry for what was and wasn't verified against a real -instance.) +integration. (Uptime Kuma and Proxmox Backup Server, added later, are not +part of that "six" — see their own git log entries, and +[INTEGRATIONS.md](INTEGRATIONS.md), for what was and wasn't verified +against a real instance.) Server and storage health is watched every 15 minutes: a server whose agent stops reporting, a server disk / Proxmox storage / Synology volume passing a @@ -225,7 +236,8 @@ workflow or branch, the same as the Gitea page shows. **Maintenance mode** silences alerts about one server, integration, or DNS provider while you work on it (server offline / disk, storage and Synology -health, Proxmox backup alerts, and "integration down" for that service type). +health, Proxmox backup alerts, Proxmox Backup Server verification alerts, and +"integration down" for that service type). Every window has a fixed end (5 minutes to 7 days) and expires on its own, and a problem that began during a window and is still present when it ends alerts then — a forgotten window can't hide an outage. A banner shows what's currently diff --git a/server/src/db/schema.ts b/server/src/db/schema.ts index a03dede..78929ff 100644 --- a/server/src/db/schema.ts +++ b/server/src/db/schema.ts @@ -288,6 +288,7 @@ export const integrationTypes = [ "dockhand", "uptimekuma", "phpipam", + "pbs", ] as const; export type IntegrationType = (typeof integrationTypes)[number]; diff --git a/server/src/index.ts b/server/src/index.ts index 4207158..5b093a0 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -33,6 +33,7 @@ import { initTailscaleKeyExpiryScheduler } from "./services/tailscaleKeyExpirySc import { initLogRetentionScheduler } from "./services/logRetentionScheduler.js"; import { initDockerUpdateScheduler } from "./services/dockerUpdateScheduler.js"; import { initProxmoxBackupScheduler } from "./services/proxmoxBackupScheduler.js"; +import { initPbsVerificationScheduler } from "./services/pbsVerificationScheduler.js"; import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js"; import { initHealthScheduler } from "./services/healthScheduler.js"; @@ -43,6 +44,7 @@ await initTailscaleKeyExpiryScheduler(); await initLogRetentionScheduler(); await initDockerUpdateScheduler(); await initProxmoxBackupScheduler(); +await initPbsVerificationScheduler(); await initQuietHoursScheduler(); await initHealthScheduler(); diff --git a/server/src/integrations/fieldSchemas.ts b/server/src/integrations/fieldSchemas.ts index 91b1810..f1c9580 100644 --- a/server/src/integrations/fieldSchemas.ts +++ b/server/src/integrations/fieldSchemas.ts @@ -56,6 +56,12 @@ export const INTEGRATION_FIELDS: Partial:` — note the colon, not the `=` PVE + * uses between the id and the secret; the id itself is the same shape either product uses + * ("user@realm!tokenname"). See https://pbs.proxmox.com/docs/user-management.html. + * + * PBS's dashboard/API is HTTPS-only (default port 8007) and, like Proxmox VE, commonly runs with + * a self-signed certificate in a homelab — hence the same "insecure" opt-out via node:https. + * + * There is no single "is this backup okay" flag anywhere in Proxmox VE — vzdump only reports that + * the push to the datastore finished, never whether the stored data still verifies. This adapter + * reads that directly from PBS: GET /admin/datastore lists the configured datastores, GET + * /admin/datastore/{store}/status gives its usage, and GET /admin/datastore/{store}/snapshots + * lists every stored backup with its own verification state — read from the snapshot data + * itself rather than by trying to correlate verify-job schedules with task-log entries, since the + * snapshot's own state is the ground truth and doesn't depend on guessing a task "worker type" + * string. GET /nodes/localhost/status gives the server's own CPU/RAM/disk — PBS is a single + * node, and "localhost" is the documented way to address it without needing its real hostname. + * + * Endpoints, the token header format, and the datastore/snapshot field names are cross-checked + * against PBS's own published documentation and API-derived community write-ups, but this has + * not been run against a live instance. Every field is read defensively (optional, independently + * type-checked), so a field PBS renames or omits in some version leaves that value blank rather + * than breaking the whole read. + */ +import * as https from "node:https"; +import { withDiagLogging } from "../../services/diagLog.js"; + +export interface PbsConfig { + url: string; + tokenId: string; + tokenSecret: string; + insecure?: boolean; +} + +export interface PbsFailedSnapshot { + backupType: string; // "vm" | "ct" | "host" + backupId: string; + /** Unix seconds. */ + backupTime: number; +} + +export interface PbsDatastore { + name: string; + comment: string | null; + totalBytes: number | null; + usedBytes: number | null; + availBytes: number | null; + /** null when the datastore couldn't be read at all (e.g. this token lacks Datastore.Audit on it) — distinct from "0 snapshots". */ + error: string | null; + snapshotCount: number; + /** Verified and found bad. */ + failedCount: number; + /** Present in the datastore but never checked by a verify job. */ + unverifiedCount: number; + /** Newest snapshot across the whole datastore, if any (unix seconds). */ + latestSnapshotAt: number | null; + /** Up to 20 of the most recent verification failures, newest first. */ + recentFailures: PbsFailedSnapshot[]; +} + +export interface PbsNodeStatus { + cpuUsagePercent: number | null; + cpuCores: number | null; + memTotalBytes: number | null; + memUsedBytes: number | null; + rootfsTotalBytes: number | null; + rootfsUsedBytes: number | null; + uptime: number | null; +} + +export interface PbsAdapter { + ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>; + listDatastores(): Promise; + getNodeStatus(): Promise; +} + +interface RawResponse { + status: number; + text: () => string; +} + +function request(url: string, insecure: boolean, headers: Record): Promise { + return new Promise((resolve, reject) => { + const parsed = new URL(url); + const req = https.request( + { + hostname: parsed.hostname, + port: parsed.port || 8007, + path: parsed.pathname + parsed.search, + method: "GET", + headers, + rejectUnauthorized: !insecure, + }, + (res) => { + let body = ""; + res.setEncoding("utf8"); + res.on("data", (chunk) => { + body += chunk; + }); + res.on("end", () => resolve({ status: res.statusCode ?? 0, text: () => body })); + }, + ); + req.on("error", reject); + req.end(); + }); +} + +const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null); +const str = (v: unknown): string | null => (typeof v === "string" && v !== "" ? v : null); + +export function createPbsAdapter(config: PbsConfig): PbsAdapter { + const insecure = config.insecure === true; + + function base() { + return config.url.replace(/\/$/, ""); + } + + function headers() { + return { Authorization: `PBSAPIToken=${config.tokenId}:${config.tokenSecret}`, Accept: "application/json" }; + } + + async function api(path: string): Promise { + const res = await request(`${base()}/api2/json${path}`, insecure, headers()); + let data: any = null; + try { + data = res.text() ? JSON.parse(res.text()) : null; + } catch { + // non-JSON error page + } + if (res.status < 200 || res.status >= 300) { + const message = data?.errors ? JSON.stringify(data.errors) : data?.message; + throw new Error(message || `Proxmox Backup Server API error: HTTP ${res.status}`); + } + return data?.data; + } + + async function listDatastoreNames(): Promise<{ name: string; comment: string | null }[]> { + const data = await api("/admin/datastore"); + return (Array.isArray(data) ? data : []) + .map((d: any) => ({ name: str(d?.name ?? d?.store), comment: str(d?.comment) })) + .filter((d: { name: string | null }): d is { name: string; comment: string | null } => d.name !== null); + } + + async function getDatastoreStatus(name: string): Promise<{ total: number | null; used: number | null; avail: number | null }> { + const data = await api(`/admin/datastore/${encodeURIComponent(name)}/status`); + return { total: num(data?.total), used: num(data?.used), avail: num(data?.avail) }; + } + + async function getSnapshotSummary(name: string) { + const data = await api(`/admin/datastore/${encodeURIComponent(name)}/snapshots`); + const snapshots = Array.isArray(data) ? data : []; + + let failedCount = 0; + let unverifiedCount = 0; + let latestSnapshotAt: number | null = null; + const failures: PbsFailedSnapshot[] = []; + + for (const s of snapshots) { + const backupTime = num(s?.["backup-time"]); + if (backupTime !== null && (latestSnapshotAt === null || backupTime > latestSnapshotAt)) latestSnapshotAt = backupTime; + + const state = str(s?.verification?.state)?.toLowerCase() ?? null; + if (state === "failed") { + failedCount++; + const backupType = str(s?.["backup-type"]); + const backupId = str(s?.["backup-id"]); + if (backupType && backupId && backupTime !== null) failures.push({ backupType, backupId, backupTime }); + } else if (state === null) { + unverifiedCount++; + } + } + + failures.sort((a, b) => b.backupTime - a.backupTime); + return { snapshotCount: snapshots.length, failedCount, unverifiedCount, latestSnapshotAt, recentFailures: failures.slice(0, 20) }; + } + + async function listDatastores(): Promise { + const names = await listDatastoreNames(); + return Promise.all( + names.map(async ({ name, comment }): Promise => { + try { + const [status, snapshots] = await Promise.all([getDatastoreStatus(name), getSnapshotSummary(name)]); + return { + name, + comment, + totalBytes: status.total, + usedBytes: status.used, + availBytes: status.avail, + error: null, + ...snapshots, + }; + } catch (err) { + return { + name, + comment, + totalBytes: null, + usedBytes: null, + availBytes: null, + error: err instanceof Error ? err.message : String(err), + snapshotCount: 0, + failedCount: 0, + unverifiedCount: 0, + latestSnapshotAt: null, + recentFailures: [], + }; + } + }), + ); + } + + async function getNodeStatus(): Promise { + const data = await api("/nodes/localhost/status"); + return { + cpuUsagePercent: num(data?.cpu) !== null ? num(data.cpu)! * 100 : null, + cpuCores: num(data?.cpuinfo?.cpus), + memTotalBytes: num(data?.memory?.total), + memUsedBytes: num(data?.memory?.used), + rootfsTotalBytes: num(data?.root?.total), + rootfsUsedBytes: num(data?.root?.used), + uptime: num(data?.uptime), + }; + } + + async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> { + const start = Date.now(); + try { + await api("/admin/datastore"); + return { ok: true, latencyMs: Date.now() - start }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : String(err) }; + } + } + + return withDiagLogging("pbs", { ping, listDatastores, getNodeStatus }); +} diff --git a/server/src/integrations/registry.ts b/server/src/integrations/registry.ts index 4211a4a..f76b686 100644 --- a/server/src/integrations/registry.ts +++ b/server/src/integrations/registry.ts @@ -8,6 +8,7 @@ import { createProxmoxAdapter } from "./proxmox/adapter.js"; import { createSynologyAdapter } from "./synology/adapter.js"; import { createUptimeKumaAdapter } from "./uptimekuma/adapter.js"; import { createPhpIpamAdapter } from "./phpipam/adapter.js"; +import { createPbsAdapter } from "./pbs/adapter.js"; export interface PingableAdapter { ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>; @@ -38,6 +39,8 @@ export function createIntegrationAdapter(type: IntegrationType, config: Integrat return createUptimeKumaAdapter(config as any); case "phpipam": return createPhpIpamAdapter(config as any); + case "pbs": + return createPbsAdapter(config as any); default: throw new Error(`Integration type "${type}" is not implemented yet`); } diff --git a/server/src/routes/integrations.ts b/server/src/routes/integrations.ts index 7a2feec..42821bb 100644 --- a/server/src/routes/integrations.ts +++ b/server/src/routes/integrations.ts @@ -21,6 +21,7 @@ import { createSemaphoreAdapter } from "../integrations/semaphore/adapter.js"; import { createProxmoxAdapter, guestsWithoutBackupCoverage } from "../integrations/proxmox/adapter.js"; import { createSynologyAdapter } from "../integrations/synology/adapter.js"; import { createUptimeKumaAdapter } from "../integrations/uptimekuma/adapter.js"; +import { createPbsAdapter } from "../integrations/pbs/adapter.js"; import { attachMatchedServers, summarizeMonitors, toMatchableServers } from "../services/uptimeKumaMatch.js"; import { asyncHandler } from "../utils/asyncHandler.js"; @@ -805,3 +806,55 @@ integrationsRouter.get("/:id/uptimekuma/monitors", asyncHandler(async (req, res) res.status(502).json({ error: err instanceof Error ? err.message : String(err) }); } })); + +// ─── Proxmox Backup Server ─────────────────────────────────────────────────── +// Read-only — no destructive actions (pruning/GC/deleting snapshots) are exposed. + +async function requirePbsAdapter(req: Request, res: Response) { + const id = Number(req.params.id); + const loaded = await loadIntegrationConfig(id); + if (!loaded) { + res.status(404).json({ error: "not_found" }); + return null; + } + if (loaded.integration.type !== "pbs") { + res.status(400).json({ error: "wrong_type" }); + return null; + } + if (!loaded.integration.enabled) { + res.status(400).json({ error: "integration_disabled" }); + return null; + } + return { integration: loaded.integration, adapter: createPbsAdapter(loaded.config as any) }; +} + +integrationsRouter.get("/:id/pbs/datastores", asyncHandler(async (req, res) => { + const found = await requirePbsAdapter(req, res); + if (!found) return; + + try { + const datastores = await found.adapter.listDatastores(); + res.json({ + datastores, + summary: { + datastoreCount: datastores.length, + failedSnapshotCount: datastores.reduce((sum, d) => sum + d.failedCount, 0), + unverifiedSnapshotCount: datastores.reduce((sum, d) => sum + d.unverifiedCount, 0), + }, + }); + } catch (err) { + res.status(502).json({ error: err instanceof Error ? err.message : String(err) }); + } +})); + +integrationsRouter.get("/:id/pbs/status", asyncHandler(async (req, res) => { + const found = await requirePbsAdapter(req, res); + if (!found) return; + + try { + const status = await found.adapter.getNodeStatus(); + res.json(status); + } catch (err) { + res.status(502).json({ error: err instanceof Error ? err.message : String(err) }); + } +})); diff --git a/server/src/routes/settings.ts b/server/src/routes/settings.ts index 798c864..15abc11 100644 --- a/server/src/routes/settings.ts +++ b/server/src/routes/settings.ts @@ -7,6 +7,7 @@ import { scheduleSecretExpiryCheck } from "../services/secretExpiryScheduler.js" import { scheduleTailscaleKeyExpiryCheck } from "../services/tailscaleKeyExpiryScheduler.js"; import { scheduleDockerUpdateCheck } from "../services/dockerUpdateScheduler.js"; import { scheduleProxmoxBackupCheck } from "../services/proxmoxBackupScheduler.js"; +import { schedulePbsVerificationCheck } from "../services/pbsVerificationScheduler.js"; import { scheduleLogRetentionPurge } from "../services/logRetentionScheduler.js"; import { purgeOldLogs } from "../services/logRetention.js"; import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js"; @@ -68,6 +69,7 @@ const updateSchema = z.object({ tailscaleKeyCheck: z.boolean(), dockerUpdateCheck: z.boolean(), proxmoxBackupCheck: z.boolean(), + pbsVerificationCheck: z.boolean(), healthAlerts: z.boolean(), automationAlerts: z.boolean(), domainExpiryCheck: z.boolean(), @@ -111,6 +113,7 @@ settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => { await scheduleTailscaleKeyExpiryCheck(); await scheduleDockerUpdateCheck(); await scheduleProxmoxBackupCheck(); + await schedulePbsVerificationCheck(); } if (parsed.data.logRetention) { await scheduleLogRetentionPurge(); diff --git a/server/src/services/notify.ts b/server/src/services/notify.ts index a02269c..5a20bc7 100644 --- a/server/src/services/notify.ts +++ b/server/src/services/notify.ts @@ -277,6 +277,22 @@ export async function notifyProxmoxUncoveredGuests( ); } +export async function notifyPbsVerificationFailed( + failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[], +): Promise { + if (failures.length === 0) return; + if (!(await eventEnabled("pbsVerificationCheck"))) return; + const lines = failures.map((f) => + f.error + ? `${f.datastore} [${f.integrationName}]: couldn't be read — ${f.error}` + : `${f.datastore} [${f.integrationName}]: ${f.failedCount} snapshot${f.failedCount !== 1 ? "s" : ""} failed verification`, + ); + await notify( + "Homelab Manager — Proxmox Backup Server Verification Failed", + `${failures.length} datastore${failures.length !== 1 ? "s have" : " has"} a problem:\n\n${lines.join("\n")}`, + ); +} + export async function notifyHealthIssues(issues: { message: string }[]): Promise { if (issues.length === 0) return; if (!(await eventEnabled("healthAlerts"))) return; diff --git a/server/src/services/pbsVerificationScheduler.ts b/server/src/services/pbsVerificationScheduler.ts new file mode 100644 index 0000000..f566b92 --- /dev/null +++ b/server/src/services/pbsVerificationScheduler.ts @@ -0,0 +1,80 @@ +import schedule from "node-schedule"; +import { and, eq } from "drizzle-orm"; +import { db } from "../db/client.js"; +import { integrations } from "../db/schema.js"; +import { loadIntegrationConfig } from "../integrations/loadIntegration.js"; +import { createPbsAdapter } from "../integrations/pbs/adapter.js"; +import { notifyPbsVerificationFailed } from "./notify.js"; +import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js"; +import { isInMaintenance } from "./maintenance.js"; + +const LAST_RUN_FLAG = "pbsVerificationCheckLastRunDate"; + +async function checkPbsVerification(): Promise { + const rows = await db + .select({ id: integrations.id, name: integrations.name }) + .from(integrations) + .where(and(eq(integrations.type, "pbs"), eq(integrations.enabled, true))); + + const failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[] = []; + + for (const row of rows) { + // A PBS host being worked on can't be reached reliably; this check is daily, so tomorrow's pass covers it. + if (await isInMaintenance("integration", row.id)) continue; + try { + const loaded = await loadIntegrationConfig(row.id); + if (!loaded) continue; + const adapter = createPbsAdapter(loaded.config as any); + const datastores = await adapter.listDatastores(); + + for (const d of datastores) { + if (d.error) { + failures.push({ integrationName: row.name, datastore: d.name, failedCount: 0, error: d.error }); + } else if (d.failedCount > 0) { + failures.push({ integrationName: row.name, datastore: d.name, failedCount: d.failedCount, error: null }); + } + } + } catch (err) { + console.error(`[pbsVerification] check failed for integration ${row.id}:`, err); + } + } + + await notifyPbsVerificationFailed(failures); +} + +async function checkPbsVerificationOnce(): Promise { + const today = new Date().toDateString(); + const lastRun = await getInternalFlag(LAST_RUN_FLAG); + if (lastRun === today) return; + await setInternalFlag(LAST_RUN_FLAG, today); + await checkPbsVerification(); +} + +function cronFromTime(time: string): string { + const [h, m] = time.split(":").map(Number); + return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} * * *`; +} + +let currentJob: schedule.Job | null = null; + +/** (Re)schedules the daily PBS verification-failure check per the current notification settings. Call again after settings change. */ +export async function schedulePbsVerificationCheck(): Promise { + if (currentJob) { + currentJob.cancel(); + currentJob = null; + } + const { notifications } = await getSettings(); + currentJob = schedule.scheduleJob({ rule: cronFromTime(notifications.secretCheckTime), tz: notifications.timezone }, () => { + setInternalFlag(LAST_RUN_FLAG, "").catch(() => {}); + getSettings().then(({ notifications: n }) => { + if (n.pbsVerificationCheck) checkPbsVerification().catch((err) => console.error("[pbsVerification] check failed:", err)); + }); + }); + console.log(`PBS verification check scheduled at ${notifications.secretCheckTime} (${notifications.timezone})`); +} + +/** Runs once at startup (skipped if already run today), then arms the daily schedule. */ +export async function initPbsVerificationScheduler(): Promise { + await checkPbsVerificationOnce(); + await schedulePbsVerificationCheck(); +} diff --git a/server/src/services/settingsStore.ts b/server/src/services/settingsStore.ts index 6d88330..7c62049 100644 --- a/server/src/services/settingsStore.ts +++ b/server/src/services/settingsStore.ts @@ -42,6 +42,8 @@ export interface NotificationEvents { tailscaleKeyCheck: boolean; dockerUpdateCheck: boolean; proxmoxBackupCheck: boolean; + /** Daily reminder while a Proxmox Backup Server snapshot has failed verification, or a datastore couldn't be read at all. */ + pbsVerificationCheck: boolean; healthAlerts: boolean; /** A Semaphore template or Gitea repo whose latest run failed, and when it succeeds again. */ automationAlerts: boolean; @@ -125,6 +127,7 @@ const DEFAULTS: AppSettings = { tailscaleKeyCheck: true, dockerUpdateCheck: true, proxmoxBackupCheck: true, + pbsVerificationCheck: true, healthAlerts: true, automationAlerts: true, domainExpiryCheck: true, diff --git a/web/src/App.tsx b/web/src/App.tsx index 78830bd..e9f14fa 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -20,6 +20,7 @@ import Gitea from "./pages/Gitea"; import Proxmox from "./pages/Proxmox"; import Synology from "./pages/Synology"; import UptimeKuma from "./pages/UptimeKuma"; +import PbsBackup from "./pages/PbsBackup"; import Generator from "./pages/Generator"; import Maintenance from "./pages/Maintenance"; import Domains from "./pages/Domains"; @@ -105,6 +106,7 @@ export default function App() { } /> } /> } /> + } /> request(`/api/integrations/${integrationId}/uptimekuma/monitors`), }, + pbs: { + datastores: (integrationId: number) => + request(`/api/integrations/${integrationId}/pbs/datastores`), + status: (integrationId: number) => + request(`/api/integrations/${integrationId}/pbs/status`), + }, semaphore: { templates: (integrationId: number) => request(`/api/integrations/${integrationId}/semaphore/templates`), diff --git a/web/src/components/CommandPalette.tsx b/web/src/components/CommandPalette.tsx index 18e8115..145d821 100644 --- a/web/src/components/CommandPalette.tsx +++ b/web/src/components/CommandPalette.tsx @@ -19,6 +19,7 @@ const INTEGRATION_TYPE_LABELS: Record = { dockhand: "Dockhand", uptimekuma: "Uptime Kuma", phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", }; const DNS_PROVIDER_LABELS: Record = { diff --git a/web/src/components/IntegrationEditForm.tsx b/web/src/components/IntegrationEditForm.tsx index 25e9d48..191125f 100644 --- a/web/src/components/IntegrationEditForm.tsx +++ b/web/src/components/IntegrationEditForm.tsx @@ -10,6 +10,7 @@ const TYPE_LABELS: Record = { dockhand: "Dockhand", uptimekuma: "Uptime Kuma", phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", }; export default function IntegrationEditForm({ diff --git a/web/src/components/IntegrationForm.tsx b/web/src/components/IntegrationForm.tsx index d16ef0b..b09f29c 100644 --- a/web/src/components/IntegrationForm.tsx +++ b/web/src/components/IntegrationForm.tsx @@ -10,6 +10,7 @@ const TYPE_LABELS: Record = { dockhand: "Dockhand", uptimekuma: "Uptime Kuma", phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", }; export default function IntegrationForm({ onCreated, onCancel }: { onCreated: () => void; onCancel: () => void }) { diff --git a/web/src/layout/AppShell.tsx b/web/src/layout/AppShell.tsx index 7cea473..5baa0ff 100644 --- a/web/src/layout/AppShell.tsx +++ b/web/src/layout/AppShell.tsx @@ -28,6 +28,7 @@ import { IconShieldLock, IconRefresh, IconActivityHeartbeat, + IconShieldCheck, } from "@tabler/icons-react"; import { api, type CurrentUser, type MaintenanceWindow } from "../api/client"; import { formatRemaining } from "../utils/duration"; @@ -65,6 +66,7 @@ const NAV: NavEntry[] = [ { to: "/servers", label: "Servers", icon: }, { to: "/proxmox", label: "Proxmox", icon: }, { to: "/synology", label: "Synology", icon: }, + { to: "/pbs", label: "Proxmox Backup", icon: }, { to: "/docker", label: "Docker", icon: }, { to: "/tailscale", label: "Tailscale", icon: }, ], diff --git a/web/src/pages/Dashboard.tsx b/web/src/pages/Dashboard.tsx index f8d1644..4d862d7 100644 --- a/web/src/pages/Dashboard.tsx +++ b/web/src/pages/Dashboard.tsx @@ -6,6 +6,7 @@ import { type DomainList, type IntegrationSummary, type IntegrationType, + type PbsDatastoresResponse, type SecretRecord, type ServerSummary, type TailscaleDevicesResponse, @@ -202,6 +203,7 @@ const WIDGETS: { type: IntegrationType; label: string }[] = [ { type: "gitea", label: "Gitea" }, { type: "dockhand", label: "Dockhand / Docker" }, { type: "uptimekuma", label: "Uptime Kuma" }, + { type: "pbs", label: "Proxmox Backup Server" }, ]; interface UptimeKumaSummary { @@ -212,6 +214,12 @@ interface UptimeKumaSummary { statusBreakdown: { label: string; count: number }[]; } +interface PbsSummary { + datastoreCount: number; + failedSnapshotCount: number; + unverifiedSnapshotCount: number; +} + interface GiteaSummary { repoCount: number; privateCount: number; @@ -277,6 +285,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) { const [proxmoxSummary, setProxmoxSummary] = useState(null); const [synologySummary, setSynologySummary] = useState(null); const [uptimeKumaSummary, setUptimeKumaSummary] = useState(null); + const [pbsSummary, setPbsSummary] = useState(null); const [dnsStats, setDnsStats] = useState> | null>(null); const [dnsError, setDnsError] = useState(null); @@ -460,6 +469,24 @@ export default function Dashboard({ user }: { user: CurrentUser }) { .catch(() => setUptimeKumaSummary(null)); }, [integrations]); + useEffect(() => { + const pbs = integrations?.find((i) => i.type === "pbs" && i.enabled); + if (!pbs) { + setPbsSummary(null); + return; + } + api.integrations.pbs + .datastores(pbs.id) + .then((res: PbsDatastoresResponse) => + setPbsSummary({ + datastoreCount: res.summary.datastoreCount, + failedSnapshotCount: res.summary.failedSnapshotCount, + unverifiedSnapshotCount: res.summary.unverifiedSnapshotCount, + }), + ) + .catch(() => setPbsSummary(null)); + }, [integrations]); + const domains = domainList?.domains ?? []; const expiredDomains = domains.filter((d) => d.status === "expired"); const expiringDomains = domains.filter((d) => d.status === "expiring"); @@ -764,7 +791,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
{WIDGETS.map(({ type, label }) => { @@ -776,8 +803,16 @@ export default function Dashboard({ user }: { user: CurrentUser }) { const isLiveProxmox = type === "proxmox" && integration && proxmoxSummary; const isLiveSynology = type === "synology" && integration && synologySummary; const isLiveUptimeKuma = type === "uptimekuma" && integration && uptimeKumaSummary; + const isLivePbs = type === "pbs" && integration && pbsSummary; const isLive = - isLiveTailscale || isLiveGitea || isLiveDockhand || isLiveSemaphore || isLiveProxmox || isLiveSynology || isLiveUptimeKuma; + isLiveTailscale || + isLiveGitea || + isLiveDockhand || + isLiveSemaphore || + isLiveProxmox || + isLiveSynology || + isLiveUptimeKuma || + isLivePbs; return ( )} + ) : isLivePbs ? ( +
+
+ +
+
+ 0 ? "#ef4444" : undefined} + /> +
+
+ 0 ? "#f59e0b" : undefined} + /> +
+
) : (
{integration ? ( diff --git a/web/src/pages/Integrations.tsx b/web/src/pages/Integrations.tsx index a05ede9..590ae4b 100644 --- a/web/src/pages/Integrations.tsx +++ b/web/src/pages/Integrations.tsx @@ -16,6 +16,7 @@ const TYPE_LABELS: Record = { dockhand: "Dockhand", uptimekuma: "Uptime Kuma", phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", }; function typeBadgeStyle(colors: Record, type: IntegrationType): CSSProperties { diff --git a/web/src/pages/Maintenance.tsx b/web/src/pages/Maintenance.tsx index f86b52a..42a9327 100644 --- a/web/src/pages/Maintenance.tsx +++ b/web/src/pages/Maintenance.tsx @@ -351,6 +351,7 @@ export default function Maintenance({ user }: { user: CurrentUser }) {
  • Server offline and disk-full alerts (for a server).
  • Storage, volume and disk-health alerts (for a Proxmox or Synology integration).
  • Backup-failure and uncovered-guest alerts (for a Proxmox integration).
    • +
  • Backup-verification-failure alerts (for a Proxmox Backup Server integration).
  • Failed-run alerts for the Semaphore or Gitea integration you choose.
  • "Integration down" alerts for that type of service — these are tracked per type (e.g. all diff --git a/web/src/pages/PbsBackup.tsx b/web/src/pages/PbsBackup.tsx new file mode 100644 index 0000000..72d5edf --- /dev/null +++ b/web/src/pages/PbsBackup.tsx @@ -0,0 +1,331 @@ +import { useEffect, useState } from "react"; +import { Link } from "react-router-dom"; +import { + api, + type CurrentUser, + type IntegrationSummary, + type PbsDatastore, + type PbsDatastoresResponse, + type PbsNodeStatus, +} from "../api/client"; +import { useSortable } from "../hooks/useSortable"; +import SortableTh from "../components/SortableTh"; +import { downloadCsv } from "../utils/csv"; + +function formatBytes(bytes: number | null): string { + if (bytes === null) return "—"; + const units = ["B", "KB", "MB", "GB", "TB", "PB"]; + let value = bytes; + let unit = 0; + while (value >= 1024 && unit < units.length - 1) { + value /= 1024; + unit++; + } + return `${value.toFixed(1)} ${units[unit]}`; +} + +function formatDate(unixSeconds: number | null): string { + if (unixSeconds === null) return "—"; + return new Date(unixSeconds * 1000).toLocaleString(); +} + +function formatUptime(seconds: number | null): string { + if (seconds === null) return "—"; + const days = Math.floor(seconds / 86400); + const hours = Math.floor((seconds % 86400) / 3600); + const minutes = Math.floor((seconds % 3600) / 60); + const parts: string[] = []; + if (days) parts.push(`${days}d`); + if (days || hours) parts.push(`${hours}h`); + parts.push(`${minutes}m`); + return parts.join(" "); +} + +function verificationBadge(d: PbsDatastore) { + if (d.error) return Unreadable; + if (d.failedCount > 0) return {d.failedCount} failed; + if (d.unverifiedCount > 0) return {d.unverifiedCount} unverified; + return All verified; +} + +export default function PbsBackup({ user: _user }: { user: CurrentUser }) { + const [integrations, setIntegrations] = useState(null); + const [selectedId, setSelectedId] = useState(null); + const [error, setError] = useState(null); + + const [data, setData] = useState(null); + const [loading, setLoading] = useState(false); + + const [status, setStatus] = useState(null); + const [statusError, setStatusError] = useState(null); + const [loadingStatus, setLoadingStatus] = useState(false); + + useEffect(() => { + api.integrations + .list() + .then((res) => { + const pbsIntegrations = res.integrations.filter((i) => i.type === "pbs"); + setIntegrations(pbsIntegrations); + if (!selectedId && pbsIntegrations.length > 0) setSelectedId(pbsIntegrations[0].id); + }) + .catch((err) => setError(err instanceof Error ? err.message : String(err))); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + const selected = integrations?.find((i) => i.id === selectedId) ?? null; + + function loadDatastores(id: number) { + setLoading(true); + setError(null); + api.integrations.pbs + .datastores(id) + .then((res) => setData(res)) + .catch((err) => setError(err instanceof Error ? err.message : String(err))) + .finally(() => setLoading(false)); + } + + function loadStatus(id: number) { + setLoadingStatus(true); + setStatusError(null); + api.integrations.pbs + .status(id) + .then((res) => setStatus(res)) + .catch((err) => setStatusError(err instanceof Error ? err.message : String(err))) + .finally(() => setLoadingStatus(false)); + } + + useEffect(() => { + if (selected?.enabled) { + loadDatastores(selected.id); + loadStatus(selected.id); + } else { + setData(null); + setStatus(null); + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [selectedId]); + + const { sorted: sortedDatastores, sortKey, sortDir, requestSort } = useSortable(data?.datastores); + + function exportCsv() { + if (!sortedDatastores) return; + downloadCsv( + "pbs-datastores.csv", + ["Datastore", "Used", "Total", "Snapshots", "Failed", "Unverified", "Latest snapshot"], + sortedDatastores.map((d) => [ + d.name, + formatBytes(d.usedBytes), + formatBytes(d.totalBytes), + d.snapshotCount, + d.failedCount, + d.unverifiedCount, + formatDate(d.latestSnapshotAt), + ]), + ); + } + + return ( + <> +

    Proxmox Backup Server

    + {error &&
    {error}
    } + + {integrations?.length === 0 ? ( +
    +
    + No Proxmox Backup Server integration configured yet. Add one under Integrations. +
    +
    + ) : ( + <> + {integrations && integrations.length > 1 && ( +
    + +
    + )} + + {selected && !selected.enabled ? ( +
    +
    + "{selected.name}" is disabled. Enable it under Integrations → Manage integrations to see its + datastores. +
    +
    + ) : ( +
    +
    +
    +
    +

    Node

    +
    + +
    +
    +
    + {statusError ? ( +
    {statusError}
    + ) : ( +
    +
    +
    Uptime
    +
    {formatUptime(status?.uptime ?? null)}
    +
    +
    +
    CPU
    +
    + {status?.cpuUsagePercent !== null && status?.cpuUsagePercent !== undefined + ? `${status.cpuUsagePercent.toFixed(0)}%` + : "—"} + {status?.cpuCores ? ({status.cpuCores} cores) : null} +
    +
    +
    +
    Memory
    +
    + {formatBytes(status?.memUsedBytes ?? null)} / {formatBytes(status?.memTotalBytes ?? null)} +
    +
    +
    +
    Root filesystem
    +
    + {formatBytes(status?.rootfsUsedBytes ?? null)} / {formatBytes(status?.rootfsTotalBytes ?? null)} +
    +
    +
    + )} +
    +
    +
    +
    +
    +
    +

    Datastores

    +
    + + +
    +
    +
    + + + + label="Datastore" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + label="Used" sortKeyName="usedBytes" activeKey={sortKey} direction={sortDir} onSort={requestSort} /> + + label="Snapshots" + sortKeyName="snapshotCount" + activeKey={sortKey} + direction={sortDir} + onSort={requestSort} + /> + + label="Latest snapshot" + sortKeyName="latestSnapshotAt" + activeKey={sortKey} + direction={sortDir} + onSort={requestSort} + /> + + label="Verification" + sortKeyName="failedCount" + activeKey={sortKey} + direction={sortDir} + onSort={requestSort} + /> + + + + {sortedDatastores?.map((d) => ( + + + + + + + + ))} + {sortedDatastores?.length === 0 && ( + + + + )} + +
    + {d.name} + {d.comment &&
    {d.comment}
    } +     		+ {formatBytes(d.usedBytes)} / {formatBytes(d.totalBytes)} + {d.snapshotCount}{formatDate(d.latestSnapshotAt)}{verificationBadge(d)}
    + No datastores found. +
    +
    +
    +
    + {sortedDatastores?.some((d) => d.recentFailures.length > 0) && ( +
    +
    +
    +

    Recent verification failures

    +
    +
    + + + + + + + + + + {sortedDatastores + .flatMap((d) => d.recentFailures.map((f) => ({ datastore: d.name, ...f }))) + .map((f, idx) => ( + + + + + + ))} + +
    DatastoreBackupWhen
    {f.datastore} + {f.backupType}/{f.backupId} + {formatDate(f.backupTime)}
    +
    +
    +
    + )} +
    + )} + + )} + + ); +} diff --git a/web/src/pages/settings/BadgeSettings.tsx b/web/src/pages/settings/BadgeSettings.tsx index ca56fa3..c82b725 100644 --- a/web/src/pages/settings/BadgeSettings.tsx +++ b/web/src/pages/settings/BadgeSettings.tsx @@ -19,6 +19,7 @@ const INTEGRATION_NAMES: Record = { dockhand: "Dockhand", uptimekuma: "Uptime Kuma", phpipam: "phpIPAM", + pbs: "Proxmox Backup Server", }; function ColorList({ diff --git a/web/src/pages/settings/NotificationSettings.tsx b/web/src/pages/settings/NotificationSettings.tsx index c748eb1..ca3f840 100644 --- a/web/src/pages/settings/NotificationSettings.tsx +++ b/web/src/pages/settings/NotificationSettings.tsx @@ -46,6 +46,7 @@ const DEFAULT_NOTIFICATIONS: NotificationEvents = { tailscaleKeyCheck: true, dockerUpdateCheck: true, proxmoxBackupCheck: true, + pbsVerificationCheck: true, healthAlerts: true, automationAlerts: true, domainExpiryCheck: true, @@ -527,6 +528,7 @@ export default function NotificationSettings() { { key: "tailscaleKeyCheck" as const, label: "Tailscale key expiry reminder" }, { key: "dockerUpdateCheck" as const, label: "Docker image update available" }, { key: "proxmoxBackupCheck" as const, label: "Proxmox backup failed or a guest has no coverage" }, + { key: "pbsVerificationCheck" as const, label: "Proxmox Backup Server snapshot failed verification" }, { key: "healthAlerts" as const, label: "Server offline, disk nearly full, or Synology volume/disk problem" }, { key: "domainExpiryCheck" as const, label: "Domain registration expiring or expired (daily reminder)" }, { key: "automationAlerts" as const, label: "Semaphore template or Gitea workflow run failed" }, @@ -618,6 +620,7 @@ export default function NotificationSettings() { notifications.tailscaleKeyCheck || notifications.dockerUpdateCheck || notifications.proxmoxBackupCheck || + notifications.pbsVerificationCheck || notifications.domainExpiryCheck; return (