Add a Proxmox Backup Server integration: datastore/snapshot verification status
Proxmox VE already shows whether the last vzdump push to PBS succeeded, but has no visibility into PBS's own backup verification, GC/prune health, or host status. This adds PBS as its own integration (own adapter, page, nav entry, and Dashboard widget) that reads datastore usage and, for every stored snapshot, its verification state directly from PBS. A new daily check (mirroring the existing Proxmox backup-failure check) notifies when a snapshot has failed verification or a datastore couldn't be read, with its own toggle in Settings -> Notifications and its own maintenance-window silencing. Not verified against a live PBS instance — built from PBS's published API docs and a scratch test against a mocked PBS server exercising the adapter's parsing and auth-header format (PBSAPIToken uses a colon separator, unlike PVE's PVEAPIToken which uses =). See INTEGRATIONS.md for details and the "not verified" caveat. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
70ba60c7da
commit
df2a5ce42b
24 files changed
+897
-13
No files matched your search
@@ -79,6 +79,31 @@ the dashboard views themselves load fine.
|
||||
metrics endpoint only exposes current status, not a monitor's scheduled start/end time, so this reflects
|
||||
what's in maintenance right now rather than mirroring Uptime Kuma's own schedule.
|
||||
|
||||
### Proxmox Backup Server
|
||||
|
||||
- **Config fields:** Proxmox Backup Server URL, API token ID, API token secret, allow self-signed certificate
|
||||
- **Auth:** `PBSAPIToken=<tokenId>:<tokenSecret>` header — note the **colon** between the token id and secret; Proxmox
|
||||
VE's own token header uses `=` there instead, so a PVE token/secret pair copied verbatim into this integration's
|
||||
fields will still format correctly (the adapter supplies the colon itself) as long as the id/secret values
|
||||
themselves are right.
|
||||
- **Actions used:** read-only — list datastores and their usage, read every stored snapshot's verification status,
|
||||
read the PBS host's own CPU/RAM/disk. No writes; nothing here can prune, delete, or re-verify a backup.
|
||||
- **Required access:** A dedicated API token (Configuration → Access Control → API Token) belonging to a user with
|
||||
**Datastore.Audit** on the datastore(s) to show, and **Sys.Audit** for the node status card to populate. A token
|
||||
scoped to just `Datastore.Audit` on one datastore will still work — other datastores it can't read are shown with
|
||||
an error rather than failing the whole page.
|
||||
- **What you get, and why it's separate from the Proxmox VE integration:** Proxmox VE (the "Proxmox" integration
|
||||
above) already shows whether the last `vzdump` push to PBS succeeded, but has no visibility at all into PBS's own
|
||||
backup **verification** — whether the data PBS actually stored still passes an integrity check, run separately by
|
||||
PBS's verify jobs. This integration reads that directly from PBS (`verification.state` on each stored snapshot),
|
||||
and a daily check (mirroring the Proxmox backup-failure check) sends a notification when a snapshot has failed
|
||||
verification or a datastore couldn't be read at all — the "Notify on" section in Settings → Notifications has a
|
||||
dedicated toggle for it, sharing the same daily reminder time as the other daily checks.
|
||||
- **Not verified against a live instance** — built from PBS's own published API documentation (endpoints, the
|
||||
`PBSAPIToken` header format, and the datastore/snapshot field names all cross-checked there), but nobody has run
|
||||
it against a real Proxmox Backup Server yet. If a datastore comes back empty or with the wrong fields, tell us
|
||||
what your instance actually returned and we'll adjust.
|
||||
|
||||
### phpIPAM
|
||||
|
||||
- **Config fields:** phpIPAM URL, API app ID, App token
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
Repository: `git@10.200.5.13:bobban/Homelab-manager.git` ([gitea.labsconnect.se/bobban/Homelab-manager](https://gitea.labsconnect.se/bobban/Homelab-manager) externally).
|
||||
|
||||
A single dashboard for a homelab: Proxmox, Synology DSM, Semaphore, Tailscale,
|
||||
Gitea, Dockhand/Docker, and Uptime Kuma status and basic actions, plus DNS record
|
||||
Gitea, Dockhand/Docker, Uptime Kuma, and Proxmox Backup Server status and basic
|
||||
actions, plus DNS record
|
||||
management, an IP address inventory (IPAM), and a secret-expiry tracker
|
||||
(ported from [Sloth Manager](../Sloth%20manager)) and scheduled-task tracking
|
||||
across Debian/Raspbian hosts (ported from
|
||||
@@ -37,9 +38,11 @@ All modules from the original plan are built:
|
||||
couldn't be refreshed.
|
||||
The Uptime Kuma widget shows the monitor count, how many are down, and how
|
||||
many are matched to one of your servers.
|
||||
The Proxmox Backup Server widget shows the datastore count and how many
|
||||
stored snapshots have failed verification or were never verified.
|
||||
- **Diagnostic Log** (admin-only) — every call this app makes to a DNS
|
||||
provider or integration (Tailscale, Proxmox, Synology, Semaphore, Gitea,
|
||||
Dockhand, Uptime Kuma), success or failure, with latency and the error message if it
|
||||
Dockhand, Uptime Kuma, Proxmox Backup Server), success or failure, with latency and the error message if it
|
||||
failed — the last 500 calls, filterable by source/result, for
|
||||
troubleshooting connectivity issues (ported from Sloth Manager's
|
||||
provider-diagnostics log, generalized to cover every integration this app
|
||||
@@ -80,9 +83,9 @@ All modules from the original plan are built:
|
||||
link options") — it stays visible regardless once a server actually is
|
||||
linked, so unlinking is always reachable.
|
||||
- **Tailscale**, **Proxmox**, **Synology**, **Semaphore**, **Gitea**,
|
||||
**Docker**, and **Uptime Kuma** each get their own top-level page (backed
|
||||
by the matching integration) instead of living inside a shared Integrations
|
||||
browsing view:
|
||||
**Docker**, **Uptime Kuma**, and **Proxmox Backup Server** each get their
|
||||
own top-level page (backed by the matching integration) instead of living
|
||||
inside a shared Integrations browsing view:
|
||||
- **Tailscale** — device list with online/authorized status, and
|
||||
authorize/deauthorize/remove actions; a live device-count widget.
|
||||
- **Proxmox** — VM/LXC status across every node in the cluster, with
|
||||
@@ -114,11 +117,18 @@ All modules from the original plan are built:
|
||||
Uptime Kuma has no conventional REST API (the dashboard talks to it over
|
||||
Socket.IO), so this reads its Prometheus `/metrics` endpoint instead and
|
||||
parses that itself; read-only, no actions.
|
||||
- **Proxmox Backup Server** — every configured datastore's usage and
|
||||
snapshot count, the PBS host's own CPU/RAM/disk, and each stored
|
||||
snapshot's verification status, since Proxmox VE only knows whether a
|
||||
backup *ran*, never whether PBS's own verify pass on the stored data
|
||||
still passes; a daily check alerts on any snapshot that's failed
|
||||
verification or a datastore that couldn't be read. Read-only, no
|
||||
actions (nothing here can prune, delete, or trigger a re-verify).
|
||||
|
||||
The Integrations page itself is now just a list of configured
|
||||
integrations (name/type/status, visible to every role) with an
|
||||
admin-only "Add integration" button and edit/enable/disable/delete
|
||||
actions per row — the seven dedicated pages above are where you
|
||||
actions per row — the eight dedicated pages above are where you
|
||||
actually use each one.
|
||||
- Every table in the app is click-to-sort on any column (numbers, booleans,
|
||||
and dates/text sort correctly regardless of how the column formats them)
|
||||
@@ -154,9 +164,10 @@ assumed HTTPS-only (the NAS is reached over plain HTTP), and the Tailscale
|
||||
adapter read `online`/`isExitNode` fields that don't actually exist in the
|
||||
real API response (fixed to derive them from `connectedToControl` and
|
||||
`enabledRoutes`). See the git log for the full verification notes per
|
||||
integration. (Uptime Kuma, added later, is not part of that "six" — see
|
||||
its own git log entry for what was and wasn't verified against a real
|
||||
instance.)
|
||||
integration. (Uptime Kuma and Proxmox Backup Server, added later, are not
|
||||
part of that "six" — see their own git log entries, and
|
||||
[INTEGRATIONS.md](INTEGRATIONS.md), for what was and wasn't verified
|
||||
against a real instance.)
|
||||
|
||||
Server and storage health is watched every 15 minutes: a server whose agent
|
||||
stops reporting, a server disk / Proxmox storage / Synology volume passing a
|
||||
@@ -225,7 +236,8 @@ workflow or branch, the same as the Gitea page shows.
|
||||
|
||||
**Maintenance mode** silences alerts about one server, integration, or DNS
|
||||
provider while you work on it (server offline / disk, storage and Synology
|
||||
health, Proxmox backup alerts, and "integration down" for that service type).
|
||||
health, Proxmox backup alerts, Proxmox Backup Server verification alerts, and
|
||||
"integration down" for that service type).
|
||||
Every window has a fixed end (5 minutes to 7 days) and expires on its own, and
|
||||
a problem that began during a window and is still present when it ends alerts
|
||||
then — a forgotten window can't hide an outage. A banner shows what's currently
|
||||
|
||||
@@ -288,6 +288,7 @@ export const integrationTypes = [
|
||||
"dockhand",
|
||||
"uptimekuma",
|
||||
"phpipam",
|
||||
"pbs",
|
||||
] as const;
|
||||
export type IntegrationType = (typeof integrationTypes)[number];
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ import { initTailscaleKeyExpiryScheduler } from "./services/tailscaleKeyExpirySc
|
||||
import { initLogRetentionScheduler } from "./services/logRetentionScheduler.js";
|
||||
import { initDockerUpdateScheduler } from "./services/dockerUpdateScheduler.js";
|
||||
import { initProxmoxBackupScheduler } from "./services/proxmoxBackupScheduler.js";
|
||||
import { initPbsVerificationScheduler } from "./services/pbsVerificationScheduler.js";
|
||||
import { initQuietHoursScheduler } from "./services/quietHoursScheduler.js";
|
||||
import { initHealthScheduler } from "./services/healthScheduler.js";
|
||||
|
||||
@@ -43,6 +44,7 @@ await initTailscaleKeyExpiryScheduler();
|
||||
await initLogRetentionScheduler();
|
||||
await initDockerUpdateScheduler();
|
||||
await initProxmoxBackupScheduler();
|
||||
await initPbsVerificationScheduler();
|
||||
await initQuietHoursScheduler();
|
||||
await initHealthScheduler();
|
||||
|
||||
|
||||
@@ -56,6 +56,12 @@ export const INTEGRATION_FIELDS: Partial<Record<IntegrationType, IntegrationFiel
|
||||
{ key: "token", label: "App token (API code)", secret: true, type: "password" },
|
||||
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
|
||||
],
|
||||
pbs: [
|
||||
{ key: "url", label: "Proxmox Backup Server URL", secret: false, placeholder: "https://pbs.example.lan:8007" },
|
||||
{ key: "tokenId", label: "API token ID", secret: false, placeholder: "root@pam!homelab-manager" },
|
||||
{ key: "tokenSecret", label: "API token secret", secret: true, type: "password" },
|
||||
{ key: "insecure", label: "Allow self-signed certificate", secret: false, type: "checkbox" },
|
||||
],
|
||||
};
|
||||
|
||||
/** Fixed base URL per integration type, stored on the row for display/reference. */
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
/**
|
||||
* Proxmox Backup Server adapter — uses PBS's REST API (api2/json), the same overall shape as
|
||||
* Proxmox VE's (both are built on the same Rust API framework), but a distinct product with its
|
||||
* own auth scheme and endpoints. Requires config: url, tokenId, tokenSecret; optional: insecure
|
||||
*
|
||||
* Auth: `Authorization: PBSAPIToken=<tokenId>:<tokenSecret>` — note the colon, not the `=` PVE
|
||||
* uses between the id and the secret; the id itself is the same shape either product uses
|
||||
* ("user@realm!tokenname"). See https://pbs.proxmox.com/docs/user-management.html.
|
||||
*
|
||||
* PBS's dashboard/API is HTTPS-only (default port 8007) and, like Proxmox VE, commonly runs with
|
||||
* a self-signed certificate in a homelab — hence the same "insecure" opt-out via node:https.
|
||||
*
|
||||
* There is no single "is this backup okay" flag anywhere in Proxmox VE — vzdump only reports that
|
||||
* the push to the datastore finished, never whether the stored data still verifies. This adapter
|
||||
* reads that directly from PBS: GET /admin/datastore lists the configured datastores, GET
|
||||
* /admin/datastore/{store}/status gives its usage, and GET /admin/datastore/{store}/snapshots
|
||||
* lists every stored backup with its own verification state — read from the snapshot data
|
||||
* itself rather than by trying to correlate verify-job schedules with task-log entries, since the
|
||||
* snapshot's own state is the ground truth and doesn't depend on guessing a task "worker type"
|
||||
* string. GET /nodes/localhost/status gives the server's own CPU/RAM/disk — PBS is a single
|
||||
* node, and "localhost" is the documented way to address it without needing its real hostname.
|
||||
*
|
||||
* Endpoints, the token header format, and the datastore/snapshot field names are cross-checked
|
||||
* against PBS's own published documentation and API-derived community write-ups, but this has
|
||||
* not been run against a live instance. Every field is read defensively (optional, independently
|
||||
* type-checked), so a field PBS renames or omits in some version leaves that value blank rather
|
||||
* than breaking the whole read.
|
||||
*/
|
||||
import * as https from "node:https";
|
||||
import { withDiagLogging } from "../../services/diagLog.js";
|
||||
|
||||
export interface PbsConfig {
|
||||
url: string;
|
||||
tokenId: string;
|
||||
tokenSecret: string;
|
||||
insecure?: boolean;
|
||||
}
|
||||
|
||||
export interface PbsFailedSnapshot {
|
||||
backupType: string; // "vm" | "ct" | "host"
|
||||
backupId: string;
|
||||
/** Unix seconds. */
|
||||
backupTime: number;
|
||||
}
|
||||
|
||||
export interface PbsDatastore {
|
||||
name: string;
|
||||
comment: string | null;
|
||||
totalBytes: number | null;
|
||||
usedBytes: number | null;
|
||||
availBytes: number | null;
|
||||
/** null when the datastore couldn't be read at all (e.g. this token lacks Datastore.Audit on it) — distinct from "0 snapshots". */
|
||||
error: string | null;
|
||||
snapshotCount: number;
|
||||
/** Verified and found bad. */
|
||||
failedCount: number;
|
||||
/** Present in the datastore but never checked by a verify job. */
|
||||
unverifiedCount: number;
|
||||
/** Newest snapshot across the whole datastore, if any (unix seconds). */
|
||||
latestSnapshotAt: number | null;
|
||||
/** Up to 20 of the most recent verification failures, newest first. */
|
||||
recentFailures: PbsFailedSnapshot[];
|
||||
}
|
||||
|
||||
export interface PbsNodeStatus {
|
||||
cpuUsagePercent: number | null;
|
||||
cpuCores: number | null;
|
||||
memTotalBytes: number | null;
|
||||
memUsedBytes: number | null;
|
||||
rootfsTotalBytes: number | null;
|
||||
rootfsUsedBytes: number | null;
|
||||
uptime: number | null;
|
||||
}
|
||||
|
||||
export interface PbsAdapter {
|
||||
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
|
||||
listDatastores(): Promise<PbsDatastore[]>;
|
||||
getNodeStatus(): Promise<PbsNodeStatus>;
|
||||
}
|
||||
|
||||
interface RawResponse {
|
||||
status: number;
|
||||
text: () => string;
|
||||
}
|
||||
|
||||
function request(url: string, insecure: boolean, headers: Record<string, string>): Promise<RawResponse> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const parsed = new URL(url);
|
||||
const req = https.request(
|
||||
{
|
||||
hostname: parsed.hostname,
|
||||
port: parsed.port || 8007,
|
||||
path: parsed.pathname + parsed.search,
|
||||
method: "GET",
|
||||
headers,
|
||||
rejectUnauthorized: !insecure,
|
||||
},
|
||||
(res) => {
|
||||
let body = "";
|
||||
res.setEncoding("utf8");
|
||||
res.on("data", (chunk) => {
|
||||
body += chunk;
|
||||
});
|
||||
res.on("end", () => resolve({ status: res.statusCode ?? 0, text: () => body }));
|
||||
},
|
||||
);
|
||||
req.on("error", reject);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
|
||||
const str = (v: unknown): string | null => (typeof v === "string" && v !== "" ? v : null);
|
||||
|
||||
export function createPbsAdapter(config: PbsConfig): PbsAdapter {
|
||||
const insecure = config.insecure === true;
|
||||
|
||||
function base() {
|
||||
return config.url.replace(/\/$/, "");
|
||||
}
|
||||
|
||||
function headers() {
|
||||
return { Authorization: `PBSAPIToken=${config.tokenId}:${config.tokenSecret}`, Accept: "application/json" };
|
||||
}
|
||||
|
||||
async function api(path: string): Promise<any> {
|
||||
const res = await request(`${base()}/api2/json${path}`, insecure, headers());
|
||||
let data: any = null;
|
||||
try {
|
||||
data = res.text() ? JSON.parse(res.text()) : null;
|
||||
} catch {
|
||||
// non-JSON error page
|
||||
}
|
||||
if (res.status < 200 || res.status >= 300) {
|
||||
const message = data?.errors ? JSON.stringify(data.errors) : data?.message;
|
||||
throw new Error(message || `Proxmox Backup Server API error: HTTP ${res.status}`);
|
||||
}
|
||||
return data?.data;
|
||||
}
|
||||
|
||||
async function listDatastoreNames(): Promise<{ name: string; comment: string | null }[]> {
|
||||
const data = await api("/admin/datastore");
|
||||
return (Array.isArray(data) ? data : [])
|
||||
.map((d: any) => ({ name: str(d?.name ?? d?.store), comment: str(d?.comment) }))
|
||||
.filter((d: { name: string | null }): d is { name: string; comment: string | null } => d.name !== null);
|
||||
}
|
||||
|
||||
async function getDatastoreStatus(name: string): Promise<{ total: number | null; used: number | null; avail: number | null }> {
|
||||
const data = await api(`/admin/datastore/${encodeURIComponent(name)}/status`);
|
||||
return { total: num(data?.total), used: num(data?.used), avail: num(data?.avail) };
|
||||
}
|
||||
|
||||
async function getSnapshotSummary(name: string) {
|
||||
const data = await api(`/admin/datastore/${encodeURIComponent(name)}/snapshots`);
|
||||
const snapshots = Array.isArray(data) ? data : [];
|
||||
|
||||
let failedCount = 0;
|
||||
let unverifiedCount = 0;
|
||||
let latestSnapshotAt: number | null = null;
|
||||
const failures: PbsFailedSnapshot[] = [];
|
||||
|
||||
for (const s of snapshots) {
|
||||
const backupTime = num(s?.["backup-time"]);
|
||||
if (backupTime !== null && (latestSnapshotAt === null || backupTime > latestSnapshotAt)) latestSnapshotAt = backupTime;
|
||||
|
||||
const state = str(s?.verification?.state)?.toLowerCase() ?? null;
|
||||
if (state === "failed") {
|
||||
failedCount++;
|
||||
const backupType = str(s?.["backup-type"]);
|
||||
const backupId = str(s?.["backup-id"]);
|
||||
if (backupType && backupId && backupTime !== null) failures.push({ backupType, backupId, backupTime });
|
||||
} else if (state === null) {
|
||||
unverifiedCount++;
|
||||
}
|
||||
}
|
||||
|
||||
failures.sort((a, b) => b.backupTime - a.backupTime);
|
||||
return { snapshotCount: snapshots.length, failedCount, unverifiedCount, latestSnapshotAt, recentFailures: failures.slice(0, 20) };
|
||||
}
|
||||
|
||||
async function listDatastores(): Promise<PbsDatastore[]> {
|
||||
const names = await listDatastoreNames();
|
||||
return Promise.all(
|
||||
names.map(async ({ name, comment }): Promise<PbsDatastore> => {
|
||||
try {
|
||||
const [status, snapshots] = await Promise.all([getDatastoreStatus(name), getSnapshotSummary(name)]);
|
||||
return {
|
||||
name,
|
||||
comment,
|
||||
totalBytes: status.total,
|
||||
usedBytes: status.used,
|
||||
availBytes: status.avail,
|
||||
error: null,
|
||||
...snapshots,
|
||||
};
|
||||
} catch (err) {
|
||||
return {
|
||||
name,
|
||||
comment,
|
||||
totalBytes: null,
|
||||
usedBytes: null,
|
||||
availBytes: null,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
snapshotCount: 0,
|
||||
failedCount: 0,
|
||||
unverifiedCount: 0,
|
||||
latestSnapshotAt: null,
|
||||
recentFailures: [],
|
||||
};
|
||||
}
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function getNodeStatus(): Promise<PbsNodeStatus> {
|
||||
const data = await api("/nodes/localhost/status");
|
||||
return {
|
||||
cpuUsagePercent: num(data?.cpu) !== null ? num(data.cpu)! * 100 : null,
|
||||
cpuCores: num(data?.cpuinfo?.cpus),
|
||||
memTotalBytes: num(data?.memory?.total),
|
||||
memUsedBytes: num(data?.memory?.used),
|
||||
rootfsTotalBytes: num(data?.root?.total),
|
||||
rootfsUsedBytes: num(data?.root?.used),
|
||||
uptime: num(data?.uptime),
|
||||
};
|
||||
}
|
||||
|
||||
async function ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
await api("/admin/datastore");
|
||||
return { ok: true, latencyMs: Date.now() - start };
|
||||
} catch (err) {
|
||||
return { ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
return withDiagLogging("pbs", { ping, listDatastores, getNodeStatus });
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import { createProxmoxAdapter } from "./proxmox/adapter.js";
|
||||
import { createSynologyAdapter } from "./synology/adapter.js";
|
||||
import { createUptimeKumaAdapter } from "./uptimekuma/adapter.js";
|
||||
import { createPhpIpamAdapter } from "./phpipam/adapter.js";
|
||||
import { createPbsAdapter } from "./pbs/adapter.js";
|
||||
|
||||
export interface PingableAdapter {
|
||||
ping(): Promise<{ ok: boolean; latencyMs?: number; error?: string }>;
|
||||
@@ -38,6 +39,8 @@ export function createIntegrationAdapter(type: IntegrationType, config: Integrat
|
||||
return createUptimeKumaAdapter(config as any);
|
||||
case "phpipam":
|
||||
return createPhpIpamAdapter(config as any);
|
||||
case "pbs":
|
||||
return createPbsAdapter(config as any);
|
||||
default:
|
||||
throw new Error(`Integration type "${type}" is not implemented yet`);
|
||||
}
|
||||
|
||||
@@ -21,6 +21,7 @@ import { createSemaphoreAdapter } from "../integrations/semaphore/adapter.js";
|
||||
import { createProxmoxAdapter, guestsWithoutBackupCoverage } from "../integrations/proxmox/adapter.js";
|
||||
import { createSynologyAdapter } from "../integrations/synology/adapter.js";
|
||||
import { createUptimeKumaAdapter } from "../integrations/uptimekuma/adapter.js";
|
||||
import { createPbsAdapter } from "../integrations/pbs/adapter.js";
|
||||
import { attachMatchedServers, summarizeMonitors, toMatchableServers } from "../services/uptimeKumaMatch.js";
|
||||
import { asyncHandler } from "../utils/asyncHandler.js";
|
||||
|
||||
@@ -805,3 +806,55 @@ integrationsRouter.get("/:id/uptimekuma/monitors", asyncHandler(async (req, res)
|
||||
res.status(502).json({ error: err instanceof Error ? err.message : String(err) });
|
||||
}
|
||||
}));
|
||||
|
||||
// ─── Proxmox Backup Server ───────────────────────────────────────────────────
|
||||
// Read-only — no destructive actions (pruning/GC/deleting snapshots) are exposed.
|
||||
|
||||
async function requirePbsAdapter(req: Request, res: Response) {
|
||||
const id = Number(req.params.id);
|
||||
const loaded = await loadIntegrationConfig(id);
|
||||
if (!loaded) {
|
||||
res.status(404).json({ error: "not_found" });
|
||||
return null;
|
||||
}
|
||||
if (loaded.integration.type !== "pbs") {
|
||||
res.status(400).json({ error: "wrong_type" });
|
||||
return null;
|
||||
}
|
||||
if (!loaded.integration.enabled) {
|
||||
res.status(400).json({ error: "integration_disabled" });
|
||||
return null;
|
||||
}
|
||||
return { integration: loaded.integration, adapter: createPbsAdapter(loaded.config as any) };
|
||||
}
|
||||
|
||||
integrationsRouter.get("/:id/pbs/datastores", asyncHandler(async (req, res) => {
|
||||
const found = await requirePbsAdapter(req, res);
|
||||
if (!found) return;
|
||||
|
||||
try {
|
||||
const datastores = await found.adapter.listDatastores();
|
||||
res.json({
|
||||
datastores,
|
||||
summary: {
|
||||
datastoreCount: datastores.length,
|
||||
failedSnapshotCount: datastores.reduce((sum, d) => sum + d.failedCount, 0),
|
||||
unverifiedSnapshotCount: datastores.reduce((sum, d) => sum + d.unverifiedCount, 0),
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
res.status(502).json({ error: err instanceof Error ? err.message : String(err) });
|
||||
}
|
||||
}));
|
||||
|
||||
integrationsRouter.get("/:id/pbs/status", asyncHandler(async (req, res) => {
|
||||
const found = await requirePbsAdapter(req, res);
|
||||
if (!found) return;
|
||||
|
||||
try {
|
||||
const status = await found.adapter.getNodeStatus();
|
||||
res.json(status);
|
||||
} catch (err) {
|
||||
res.status(502).json({ error: err instanceof Error ? err.message : String(err) });
|
||||
}
|
||||
}));
|
||||
@@ -7,6 +7,7 @@ import { scheduleSecretExpiryCheck } from "../services/secretExpiryScheduler.js"
|
||||
import { scheduleTailscaleKeyExpiryCheck } from "../services/tailscaleKeyExpiryScheduler.js";
|
||||
import { scheduleDockerUpdateCheck } from "../services/dockerUpdateScheduler.js";
|
||||
import { scheduleProxmoxBackupCheck } from "../services/proxmoxBackupScheduler.js";
|
||||
import { schedulePbsVerificationCheck } from "../services/pbsVerificationScheduler.js";
|
||||
import { scheduleLogRetentionPurge } from "../services/logRetentionScheduler.js";
|
||||
import { purgeOldLogs } from "../services/logRetention.js";
|
||||
import { scheduleQuietHoursFlush } from "../services/quietHoursScheduler.js";
|
||||
@@ -68,6 +69,7 @@ const updateSchema = z.object({
|
||||
tailscaleKeyCheck: z.boolean(),
|
||||
dockerUpdateCheck: z.boolean(),
|
||||
proxmoxBackupCheck: z.boolean(),
|
||||
pbsVerificationCheck: z.boolean(),
|
||||
healthAlerts: z.boolean(),
|
||||
automationAlerts: z.boolean(),
|
||||
domainExpiryCheck: z.boolean(),
|
||||
@@ -111,6 +113,7 @@ settingsRouter.put("/", requireRole("admin"), asyncHandler(async (req, res) => {
|
||||
await scheduleTailscaleKeyExpiryCheck();
|
||||
await scheduleDockerUpdateCheck();
|
||||
await scheduleProxmoxBackupCheck();
|
||||
await schedulePbsVerificationCheck();
|
||||
}
|
||||
if (parsed.data.logRetention) {
|
||||
await scheduleLogRetentionPurge();
|
||||
|
||||
@@ -277,6 +277,22 @@ export async function notifyProxmoxUncoveredGuests(
|
||||
);
|
||||
}
|
||||
|
||||
export async function notifyPbsVerificationFailed(
|
||||
failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[],
|
||||
): Promise<void> {
|
||||
if (failures.length === 0) return;
|
||||
if (!(await eventEnabled("pbsVerificationCheck"))) return;
|
||||
const lines = failures.map((f) =>
|
||||
f.error
|
||||
? `${f.datastore} [${f.integrationName}]: couldn't be read — ${f.error}`
|
||||
: `${f.datastore} [${f.integrationName}]: ${f.failedCount} snapshot${f.failedCount !== 1 ? "s" : ""} failed verification`,
|
||||
);
|
||||
await notify(
|
||||
"Homelab Manager — Proxmox Backup Server Verification Failed",
|
||||
`${failures.length} datastore${failures.length !== 1 ? "s have" : " has"} a problem:\n\n${lines.join("\n")}`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function notifyHealthIssues(issues: { message: string }[]): Promise<void> {
|
||||
if (issues.length === 0) return;
|
||||
if (!(await eventEnabled("healthAlerts"))) return;
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
import schedule from "node-schedule";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { db } from "../db/client.js";
|
||||
import { integrations } from "../db/schema.js";
|
||||
import { loadIntegrationConfig } from "../integrations/loadIntegration.js";
|
||||
import { createPbsAdapter } from "../integrations/pbs/adapter.js";
|
||||
import { notifyPbsVerificationFailed } from "./notify.js";
|
||||
import { getSettings, getInternalFlag, setInternalFlag } from "./settingsStore.js";
|
||||
import { isInMaintenance } from "./maintenance.js";
|
||||
|
||||
const LAST_RUN_FLAG = "pbsVerificationCheckLastRunDate";
|
||||
|
||||
async function checkPbsVerification(): Promise<void> {
|
||||
const rows = await db
|
||||
.select({ id: integrations.id, name: integrations.name })
|
||||
.from(integrations)
|
||||
.where(and(eq(integrations.type, "pbs"), eq(integrations.enabled, true)));
|
||||
|
||||
const failures: { integrationName: string; datastore: string; failedCount: number; error: string | null }[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
// A PBS host being worked on can't be reached reliably; this check is daily, so tomorrow's pass covers it.
|
||||
if (await isInMaintenance("integration", row.id)) continue;
|
||||
try {
|
||||
const loaded = await loadIntegrationConfig(row.id);
|
||||
if (!loaded) continue;
|
||||
const adapter = createPbsAdapter(loaded.config as any);
|
||||
const datastores = await adapter.listDatastores();
|
||||
|
||||
for (const d of datastores) {
|
||||
if (d.error) {
|
||||
failures.push({ integrationName: row.name, datastore: d.name, failedCount: 0, error: d.error });
|
||||
} else if (d.failedCount > 0) {
|
||||
failures.push({ integrationName: row.name, datastore: d.name, failedCount: d.failedCount, error: null });
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`[pbsVerification] check failed for integration ${row.id}:`, err);
|
||||
}
|
||||
}
|
||||
|
||||
await notifyPbsVerificationFailed(failures);
|
||||
}
|
||||
|
||||
async function checkPbsVerificationOnce(): Promise<void> {
|
||||
const today = new Date().toDateString();
|
||||
const lastRun = await getInternalFlag(LAST_RUN_FLAG);
|
||||
if (lastRun === today) return;
|
||||
await setInternalFlag(LAST_RUN_FLAG, today);
|
||||
await checkPbsVerification();
|
||||
}
|
||||
|
||||
function cronFromTime(time: string): string {
|
||||
const [h, m] = time.split(":").map(Number);
|
||||
return `${Number.isFinite(m) ? m : 0} ${Number.isFinite(h) ? h : 8} * * *`;
|
||||
}
|
||||
|
||||
let currentJob: schedule.Job | null = null;
|
||||
|
||||
/** (Re)schedules the daily PBS verification-failure check per the current notification settings. Call again after settings change. */
|
||||
export async function schedulePbsVerificationCheck(): Promise<void> {
|
||||
if (currentJob) {
|
||||
currentJob.cancel();
|
||||
currentJob = null;
|
||||
}
|
||||
const { notifications } = await getSettings();
|
||||
currentJob = schedule.scheduleJob({ rule: cronFromTime(notifications.secretCheckTime), tz: notifications.timezone }, () => {
|
||||
setInternalFlag(LAST_RUN_FLAG, "").catch(() => {});
|
||||
getSettings().then(({ notifications: n }) => {
|
||||
if (n.pbsVerificationCheck) checkPbsVerification().catch((err) => console.error("[pbsVerification] check failed:", err));
|
||||
});
|
||||
});
|
||||
console.log(`PBS verification check scheduled at ${notifications.secretCheckTime} (${notifications.timezone})`);
|
||||
}
|
||||
|
||||
/** Runs once at startup (skipped if already run today), then arms the daily schedule. */
|
||||
export async function initPbsVerificationScheduler(): Promise<void> {
|
||||
await checkPbsVerificationOnce();
|
||||
await schedulePbsVerificationCheck();
|
||||
}
|
||||
@@ -42,6 +42,8 @@ export interface NotificationEvents {
|
||||
tailscaleKeyCheck: boolean;
|
||||
dockerUpdateCheck: boolean;
|
||||
proxmoxBackupCheck: boolean;
|
||||
/** Daily reminder while a Proxmox Backup Server snapshot has failed verification, or a datastore couldn't be read at all. */
|
||||
pbsVerificationCheck: boolean;
|
||||
healthAlerts: boolean;
|
||||
/** A Semaphore template or Gitea repo whose latest run failed, and when it succeeds again. */
|
||||
automationAlerts: boolean;
|
||||
@@ -125,6 +127,7 @@ const DEFAULTS: AppSettings = {
|
||||
tailscaleKeyCheck: true,
|
||||
dockerUpdateCheck: true,
|
||||
proxmoxBackupCheck: true,
|
||||
pbsVerificationCheck: true,
|
||||
healthAlerts: true,
|
||||
automationAlerts: true,
|
||||
domainExpiryCheck: true,
|
||||
|
||||
@@ -20,6 +20,7 @@ import Gitea from "./pages/Gitea";
|
||||
import Proxmox from "./pages/Proxmox";
|
||||
import Synology from "./pages/Synology";
|
||||
import UptimeKuma from "./pages/UptimeKuma";
|
||||
import PbsBackup from "./pages/PbsBackup";
|
||||
import Generator from "./pages/Generator";
|
||||
import Maintenance from "./pages/Maintenance";
|
||||
import Domains from "./pages/Domains";
|
||||
@@ -105,6 +106,7 @@ export default function App() {
|
||||
<Route path="/proxmox" element={<Proxmox user={user} />} />
|
||||
<Route path="/synology" element={<Synology user={user} />} />
|
||||
<Route path="/uptime-kuma" element={<UptimeKuma user={user} />} />
|
||||
<Route path="/pbs" element={<PbsBackup user={user} />} />
|
||||
<Route
|
||||
path="/users"
|
||||
element={
|
||||
|
||||
+43
-1
@@ -182,6 +182,7 @@ export interface NotificationEvents {
|
||||
tailscaleKeyCheck: boolean;
|
||||
dockerUpdateCheck: boolean;
|
||||
proxmoxBackupCheck: boolean;
|
||||
pbsVerificationCheck: boolean;
|
||||
healthAlerts: boolean;
|
||||
automationAlerts: boolean;
|
||||
domainExpiryCheck: boolean;
|
||||
@@ -428,7 +429,7 @@ export interface ManualTaskInput {
|
||||
enabled?: boolean;
|
||||
}
|
||||
|
||||
export type IntegrationType = "proxmox" | "synology" | "semaphore" | "tailscale" | "gitea" | "dockhand" | "uptimekuma" | "phpipam";
|
||||
export type IntegrationType = "proxmox" | "synology" | "semaphore" | "tailscale" | "gitea" | "dockhand" | "uptimekuma" | "phpipam" | "pbs";
|
||||
|
||||
export interface IntegrationField {
|
||||
key: string;
|
||||
@@ -635,6 +636,41 @@ export interface UptimeKumaImportResult {
|
||||
monitorsInMaintenance: number;
|
||||
}
|
||||
|
||||
export interface PbsFailedSnapshot {
|
||||
backupType: string;
|
||||
backupId: string;
|
||||
backupTime: number;
|
||||
}
|
||||
|
||||
export interface PbsDatastore {
|
||||
name: string;
|
||||
comment: string | null;
|
||||
totalBytes: number | null;
|
||||
usedBytes: number | null;
|
||||
availBytes: number | null;
|
||||
error: string | null;
|
||||
snapshotCount: number;
|
||||
failedCount: number;
|
||||
unverifiedCount: number;
|
||||
latestSnapshotAt: number | null;
|
||||
recentFailures: PbsFailedSnapshot[];
|
||||
}
|
||||
|
||||
export interface PbsDatastoresResponse {
|
||||
datastores: PbsDatastore[];
|
||||
summary: { datastoreCount: number; failedSnapshotCount: number; unverifiedSnapshotCount: number };
|
||||
}
|
||||
|
||||
export interface PbsNodeStatus {
|
||||
cpuUsagePercent: number | null;
|
||||
cpuCores: number | null;
|
||||
memTotalBytes: number | null;
|
||||
memUsedBytes: number | null;
|
||||
rootfsTotalBytes: number | null;
|
||||
rootfsUsedBytes: number | null;
|
||||
uptime: number | null;
|
||||
}
|
||||
|
||||
export type SemaphoreTaskStatus =
|
||||
| "waiting"
|
||||
| "starting"
|
||||
@@ -1115,6 +1151,12 @@ export const api = {
|
||||
monitors: (integrationId: number) =>
|
||||
request<UptimeKumaMonitorsResponse>(`/api/integrations/${integrationId}/uptimekuma/monitors`),
|
||||
},
|
||||
pbs: {
|
||||
datastores: (integrationId: number) =>
|
||||
request<PbsDatastoresResponse>(`/api/integrations/${integrationId}/pbs/datastores`),
|
||||
status: (integrationId: number) =>
|
||||
request<PbsNodeStatus>(`/api/integrations/${integrationId}/pbs/status`),
|
||||
},
|
||||
semaphore: {
|
||||
templates: (integrationId: number) =>
|
||||
request<SemaphoreTemplatesResponse>(`/api/integrations/${integrationId}/semaphore/templates`),
|
||||
|
||||
@@ -19,6 +19,7 @@ const INTEGRATION_TYPE_LABELS: Record<IntegrationType, string> = {
|
||||
dockhand: "Dockhand",
|
||||
uptimekuma: "Uptime Kuma",
|
||||
phpipam: "phpIPAM",
|
||||
pbs: "Proxmox Backup Server",
|
||||
};
|
||||
|
||||
const DNS_PROVIDER_LABELS: Record<DnsProviderType, string> = {
|
||||
|
||||
@@ -10,6 +10,7 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
|
||||
dockhand: "Dockhand",
|
||||
uptimekuma: "Uptime Kuma",
|
||||
phpipam: "phpIPAM",
|
||||
pbs: "Proxmox Backup Server",
|
||||
};
|
||||
|
||||
export default function IntegrationEditForm({
|
||||
|
||||
@@ -10,6 +10,7 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
|
||||
dockhand: "Dockhand",
|
||||
uptimekuma: "Uptime Kuma",
|
||||
phpipam: "phpIPAM",
|
||||
pbs: "Proxmox Backup Server",
|
||||
};
|
||||
|
||||
export default function IntegrationForm({ onCreated, onCancel }: { onCreated: () => void; onCancel: () => void }) {
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
IconShieldLock,
|
||||
IconRefresh,
|
||||
IconActivityHeartbeat,
|
||||
IconShieldCheck,
|
||||
} from "@tabler/icons-react";
|
||||
import { api, type CurrentUser, type MaintenanceWindow } from "../api/client";
|
||||
import { formatRemaining } from "../utils/duration";
|
||||
@@ -65,6 +66,7 @@ const NAV: NavEntry[] = [
|
||||
{ to: "/servers", label: "Servers", icon: <IconServer2 size={20} /> },
|
||||
{ to: "/proxmox", label: "Proxmox", icon: <IconServerCog size={20} /> },
|
||||
{ to: "/synology", label: "Synology", icon: <IconDatabase size={20} /> },
|
||||
{ to: "/pbs", label: "Proxmox Backup", icon: <IconShieldCheck size={20} /> },
|
||||
{ to: "/docker", label: "Docker", icon: <IconBrandDocker size={20} /> },
|
||||
{ to: "/tailscale", label: "Tailscale", icon: <IconAffiliate size={20} /> },
|
||||
],
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
type DomainList,
|
||||
type IntegrationSummary,
|
||||
type IntegrationType,
|
||||
type PbsDatastoresResponse,
|
||||
type SecretRecord,
|
||||
type ServerSummary,
|
||||
type TailscaleDevicesResponse,
|
||||
@@ -202,6 +203,7 @@ const WIDGETS: { type: IntegrationType; label: string }[] = [
|
||||
{ type: "gitea", label: "Gitea" },
|
||||
{ type: "dockhand", label: "Dockhand / Docker" },
|
||||
{ type: "uptimekuma", label: "Uptime Kuma" },
|
||||
{ type: "pbs", label: "Proxmox Backup Server" },
|
||||
];
|
||||
|
||||
interface UptimeKumaSummary {
|
||||
@@ -212,6 +214,12 @@ interface UptimeKumaSummary {
|
||||
statusBreakdown: { label: string; count: number }[];
|
||||
}
|
||||
|
||||
interface PbsSummary {
|
||||
datastoreCount: number;
|
||||
failedSnapshotCount: number;
|
||||
unverifiedSnapshotCount: number;
|
||||
}
|
||||
|
||||
interface GiteaSummary {
|
||||
repoCount: number;
|
||||
privateCount: number;
|
||||
@@ -277,6 +285,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
||||
const [proxmoxSummary, setProxmoxSummary] = useState<ProxmoxSummary | null>(null);
|
||||
const [synologySummary, setSynologySummary] = useState<SynologySummary | null>(null);
|
||||
const [uptimeKumaSummary, setUptimeKumaSummary] = useState<UptimeKumaSummary | null>(null);
|
||||
const [pbsSummary, setPbsSummary] = useState<PbsSummary | null>(null);
|
||||
|
||||
const [dnsStats, setDnsStats] = useState<Awaited<ReturnType<typeof api.dns.stats>> | null>(null);
|
||||
const [dnsError, setDnsError] = useState<string | null>(null);
|
||||
@@ -460,6 +469,24 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
||||
.catch(() => setUptimeKumaSummary(null));
|
||||
}, [integrations]);
|
||||
|
||||
useEffect(() => {
|
||||
const pbs = integrations?.find((i) => i.type === "pbs" && i.enabled);
|
||||
if (!pbs) {
|
||||
setPbsSummary(null);
|
||||
return;
|
||||
}
|
||||
api.integrations.pbs
|
||||
.datastores(pbs.id)
|
||||
.then((res: PbsDatastoresResponse) =>
|
||||
setPbsSummary({
|
||||
datastoreCount: res.summary.datastoreCount,
|
||||
failedSnapshotCount: res.summary.failedSnapshotCount,
|
||||
unverifiedSnapshotCount: res.summary.unverifiedSnapshotCount,
|
||||
}),
|
||||
)
|
||||
.catch(() => setPbsSummary(null));
|
||||
}, [integrations]);
|
||||
|
||||
const domains = domainList?.domains ?? [];
|
||||
const expiredDomains = domains.filter((d) => d.status === "expired");
|
||||
const expiringDomains = domains.filter((d) => d.status === "expiring");
|
||||
@@ -764,7 +791,7 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
||||
|
||||
<SectionHeader
|
||||
title="Integrations"
|
||||
hint="Live status widgets for Proxmox, Synology, Semaphore, Tailscale, Gitea, Dockhand, and Uptime Kuma will appear here as each integration is connected."
|
||||
hint="Live status widgets for Proxmox, Synology, Semaphore, Tailscale, Gitea, Dockhand, Uptime Kuma, and Proxmox Backup Server will appear here as each integration is connected."
|
||||
/>
|
||||
<div className="row row-cards">
|
||||
{WIDGETS.map(({ type, label }) => {
|
||||
@@ -776,8 +803,16 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
||||
const isLiveProxmox = type === "proxmox" && integration && proxmoxSummary;
|
||||
const isLiveSynology = type === "synology" && integration && synologySummary;
|
||||
const isLiveUptimeKuma = type === "uptimekuma" && integration && uptimeKumaSummary;
|
||||
const isLivePbs = type === "pbs" && integration && pbsSummary;
|
||||
const isLive =
|
||||
isLiveTailscale || isLiveGitea || isLiveDockhand || isLiveSemaphore || isLiveProxmox || isLiveSynology || isLiveUptimeKuma;
|
||||
isLiveTailscale ||
|
||||
isLiveGitea ||
|
||||
isLiveDockhand ||
|
||||
isLiveSemaphore ||
|
||||
isLiveProxmox ||
|
||||
isLiveSynology ||
|
||||
isLiveUptimeKuma ||
|
||||
isLivePbs;
|
||||
|
||||
return (
|
||||
<WidgetCard
|
||||
@@ -953,6 +988,26 @@ export default function Dashboard({ user }: { user: CurrentUser }) {
|
||||
<BreakdownBar data={uptimeKumaSummary!.statusBreakdown} colors={MONITOR_STATE_COLORS} compact />
|
||||
)}
|
||||
</>
|
||||
) : isLivePbs ? (
|
||||
<div className="row g-2 mb-3">
|
||||
<div className="col-4">
|
||||
<MiniStat label="Datastores" value={pbsSummary!.datastoreCount} />
|
||||
</div>
|
||||
<div className="col-4">
|
||||
<MiniStat
|
||||
label="Failed"
|
||||
value={pbsSummary!.failedSnapshotCount}
|
||||
accent={pbsSummary!.failedSnapshotCount > 0 ? "#ef4444" : undefined}
|
||||
/>
|
||||
</div>
|
||||
<div className="col-4">
|
||||
<MiniStat
|
||||
label="Unverified"
|
||||
value={pbsSummary!.unverifiedSnapshotCount}
|
||||
accent={pbsSummary!.unverifiedSnapshotCount > 0 ? "#f59e0b" : undefined}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div className="text-secondary mt-2">
|
||||
{integration ? (
|
||||
|
||||
@@ -16,6 +16,7 @@ const TYPE_LABELS: Record<IntegrationType, string> = {
|
||||
dockhand: "Dockhand",
|
||||
uptimekuma: "Uptime Kuma",
|
||||
phpipam: "phpIPAM",
|
||||
pbs: "Proxmox Backup Server",
|
||||
};
|
||||
|
||||
function typeBadgeStyle(colors: Record<string, string>, type: IntegrationType): CSSProperties {
|
||||
|
||||
@@ -351,6 +351,7 @@ export default function Maintenance({ user }: { user: CurrentUser }) {
|
||||
<li>Server offline and disk-full alerts (for a server).</li>
|
||||
<li>Storage, volume and disk-health alerts (for a Proxmox or Synology integration).</li>
|
||||
<li>Backup-failure and uncovered-guest alerts (for a Proxmox integration).</li>
|
||||
<li>Backup-verification-failure alerts (for a Proxmox Backup Server integration).</li>
|
||||
<li>Failed-run alerts for the Semaphore or Gitea integration you choose.</li>
|
||||
<li>
|
||||
"Integration down" alerts for that <em>type</em> of service — these are tracked per type (e.g. all
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { Link } from "react-router-dom";
|
||||
import {
|
||||
api,
|
||||
type CurrentUser,
|
||||
type IntegrationSummary,
|
||||
type PbsDatastore,
|
||||
type PbsDatastoresResponse,
|
||||
type PbsNodeStatus,
|
||||
} from "../api/client";
|
||||
import { useSortable } from "../hooks/useSortable";
|
||||
import SortableTh from "../components/SortableTh";
|
||||
import { downloadCsv } from "../utils/csv";
|
||||
|
||||
function formatBytes(bytes: number | null): string {
|
||||
if (bytes === null) return "—";
|
||||
const units = ["B", "KB", "MB", "GB", "TB", "PB"];
|
||||
let value = bytes;
|
||||
let unit = 0;
|
||||
while (value >= 1024 && unit < units.length - 1) {
|
||||
value /= 1024;
|
||||
unit++;
|
||||
}
|
||||
return `${value.toFixed(1)} ${units[unit]}`;
|
||||
}
|
||||
|
||||
function formatDate(unixSeconds: number | null): string {
|
||||
if (unixSeconds === null) return "—";
|
||||
return new Date(unixSeconds * 1000).toLocaleString();
|
||||
}
|
||||
|
||||
function formatUptime(seconds: number | null): string {
|
||||
if (seconds === null) return "—";
|
||||
const days = Math.floor(seconds / 86400);
|
||||
const hours = Math.floor((seconds % 86400) / 3600);
|
||||
const minutes = Math.floor((seconds % 3600) / 60);
|
||||
const parts: string[] = [];
|
||||
if (days) parts.push(`${days}d`);
|
||||
if (days || hours) parts.push(`${hours}h`);
|
||||
parts.push(`${minutes}m`);
|
||||
return parts.join(" ");
|
||||
}
|
||||
|
||||
function verificationBadge(d: PbsDatastore) {
|
||||
if (d.error) return <span className="badge bg-red-lt text-red">Unreadable</span>;
|
||||
if (d.failedCount > 0) return <span className="badge bg-red-lt text-red">{d.failedCount} failed</span>;
|
||||
if (d.unverifiedCount > 0) return <span className="badge bg-yellow-lt text-yellow">{d.unverifiedCount} unverified</span>;
|
||||
return <span className="badge bg-green-lt text-green">All verified</span>;
|
||||
}
|
||||
|
||||
export default function PbsBackup({ user: _user }: { user: CurrentUser }) {
|
||||
const [integrations, setIntegrations] = useState<IntegrationSummary[] | null>(null);
|
||||
const [selectedId, setSelectedId] = useState<number | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const [data, setData] = useState<PbsDatastoresResponse | null>(null);
|
||||
const [loading, setLoading] = useState(false);
|
||||
|
||||
const [status, setStatus] = useState<PbsNodeStatus | null>(null);
|
||||
const [statusError, setStatusError] = useState<string | null>(null);
|
||||
const [loadingStatus, setLoadingStatus] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
api.integrations
|
||||
.list()
|
||||
.then((res) => {
|
||||
const pbsIntegrations = res.integrations.filter((i) => i.type === "pbs");
|
||||
setIntegrations(pbsIntegrations);
|
||||
if (!selectedId && pbsIntegrations.length > 0) setSelectedId(pbsIntegrations[0].id);
|
||||
})
|
||||
.catch((err) => setError(err instanceof Error ? err.message : String(err)));
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, []);
|
||||
|
||||
const selected = integrations?.find((i) => i.id === selectedId) ?? null;
|
||||
|
||||
function loadDatastores(id: number) {
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
api.integrations.pbs
|
||||
.datastores(id)
|
||||
.then((res) => setData(res))
|
||||
.catch((err) => setError(err instanceof Error ? err.message : String(err)))
|
||||
.finally(() => setLoading(false));
|
||||
}
|
||||
|
||||
function loadStatus(id: number) {
|
||||
setLoadingStatus(true);
|
||||
setStatusError(null);
|
||||
api.integrations.pbs
|
||||
.status(id)
|
||||
.then((res) => setStatus(res))
|
||||
.catch((err) => setStatusError(err instanceof Error ? err.message : String(err)))
|
||||
.finally(() => setLoadingStatus(false));
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
if (selected?.enabled) {
|
||||
loadDatastores(selected.id);
|
||||
loadStatus(selected.id);
|
||||
} else {
|
||||
setData(null);
|
||||
setStatus(null);
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [selectedId]);
|
||||
|
||||
const { sorted: sortedDatastores, sortKey, sortDir, requestSort } = useSortable(data?.datastores);
|
||||
|
||||
function exportCsv() {
|
||||
if (!sortedDatastores) return;
|
||||
downloadCsv(
|
||||
"pbs-datastores.csv",
|
||||
["Datastore", "Used", "Total", "Snapshots", "Failed", "Unverified", "Latest snapshot"],
|
||||
sortedDatastores.map((d) => [
|
||||
d.name,
|
||||
formatBytes(d.usedBytes),
|
||||
formatBytes(d.totalBytes),
|
||||
d.snapshotCount,
|
||||
d.failedCount,
|
||||
d.unverifiedCount,
|
||||
formatDate(d.latestSnapshotAt),
|
||||
]),
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<>
|
||||
<h2 className="page-title mb-3">Proxmox Backup Server</h2>
|
||||
{error && <div className="alert alert-danger">{error}</div>}
|
||||
|
||||
{integrations?.length === 0 ? (
|
||||
<div className="card">
|
||||
<div className="card-body text-secondary">
|
||||
No Proxmox Backup Server integration configured yet. <Link to="/integrations">Add one under Integrations</Link>.
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{integrations && integrations.length > 1 && (
|
||||
<div className="mb-3" style={{ maxWidth: 320 }}>
|
||||
<select
|
||||
className="form-select"
|
||||
value={selectedId ?? ""}
|
||||
onChange={(e) => setSelectedId(Number(e.target.value))}
|
||||
>
|
||||
{integrations.map((i) => (
|
||||
<option key={i.id} value={i.id} disabled={!i.enabled}>
|
||||
{i.name}
|
||||
{!i.enabled ? " — disabled" : ""}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{selected && !selected.enabled ? (
|
||||
<div className="card">
|
||||
<div className="card-body text-secondary">
|
||||
"{selected.name}" is disabled. Enable it under Integrations → Manage integrations to see its
|
||||
datastores.
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div className="row g-3">
|
||||
<div className="col-12">
|
||||
<div className="card">
|
||||
<div className="card-header">
|
||||
<h3 className="card-title">Node</h3>
|
||||
<div className="card-actions">
|
||||
<button
|
||||
className="btn btn-sm btn-outline-secondary"
|
||||
onClick={() => selected && loadStatus(selected.id)}
|
||||
disabled={loadingStatus}
|
||||
>
|
||||
{loadingStatus ? "Refreshing…" : "Refresh"}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div className="card-body">
|
||||
{statusError ? (
|
||||
<div className="text-danger">{statusError}</div>
|
||||
) : (
|
||||
<div className="row g-3">
|
||||
<div className="col-6 col-md-3">
|
||||
<div className="text-secondary">Uptime</div>
|
||||
<div>{formatUptime(status?.uptime ?? null)}</div>
|
||||
</div>
|
||||
<div className="col-6 col-md-3">
|
||||
<div className="text-secondary">CPU</div>
|
||||
<div>
|
||||
{status?.cpuUsagePercent !== null && status?.cpuUsagePercent !== undefined
|
||||
? `${status.cpuUsagePercent.toFixed(0)}%`
|
||||
: "—"}
|
||||
{status?.cpuCores ? <span className="text-secondary"> ({status.cpuCores} cores)</span> : null}
|
||||
</div>
|
||||
</div>
|
||||
<div className="col-6 col-md-3">
|
||||
<div className="text-secondary">Memory</div>
|
||||
<div>
|
||||
{formatBytes(status?.memUsedBytes ?? null)} / {formatBytes(status?.memTotalBytes ?? null)}
|
||||
</div>
|
||||
</div>
|
||||
<div className="col-6 col-md-3">
|
||||
<div className="text-secondary">Root filesystem</div>
|
||||
<div>
|
||||
{formatBytes(status?.rootfsUsedBytes ?? null)} / {formatBytes(status?.rootfsTotalBytes ?? null)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div className="col-12">
|
||||
<div className="card">
|
||||
<div className="card-header">
|
||||
<h3 className="card-title">Datastores</h3>
|
||||
<div className="card-actions">
|
||||
<button
|
||||
className="btn btn-sm btn-outline-secondary"
|
||||
onClick={exportCsv}
|
||||
disabled={!sortedDatastores || sortedDatastores.length === 0}
|
||||
>
|
||||
Export CSV
|
||||
</button>
|
||||
<button
|
||||
className="btn btn-sm btn-outline-secondary"
|
||||
onClick={() => selected && loadDatastores(selected.id)}
|
||||
disabled={loading}
|
||||
>
|
||||
{loading ? "Refreshing…" : "Refresh"}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div className="table-responsive">
|
||||
<table className="table table-vcenter card-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<SortableTh<PbsDatastore> label="Datastore" sortKeyName="name" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||
<SortableTh<PbsDatastore> label="Used" sortKeyName="usedBytes" activeKey={sortKey} direction={sortDir} onSort={requestSort} />
|
||||
<SortableTh<PbsDatastore>
|
||||
label="Snapshots"
|
||||
sortKeyName="snapshotCount"
|
||||
activeKey={sortKey}
|
||||
direction={sortDir}
|
||||
onSort={requestSort}
|
||||
/>
|
||||
<SortableTh<PbsDatastore>
|
||||
label="Latest snapshot"
|
||||
sortKeyName="latestSnapshotAt"
|
||||
activeKey={sortKey}
|
||||
direction={sortDir}
|
||||
onSort={requestSort}
|
||||
/>
|
||||
<SortableTh<PbsDatastore>
|
||||
label="Verification"
|
||||
sortKeyName="failedCount"
|
||||
activeKey={sortKey}
|
||||
direction={sortDir}
|
||||
onSort={requestSort}
|
||||
/>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{sortedDatastores?.map((d) => (
|
||||
<tr key={d.name}>
|
||||
<td>
|
||||
{d.name}
|
||||
{d.comment && <div className="text-secondary small">{d.comment}</div>}
|
||||
</td>
|
||||
<td className="text-secondary">
|
||||
{formatBytes(d.usedBytes)} / {formatBytes(d.totalBytes)}
|
||||
</td>
|
||||
<td className="text-secondary">{d.snapshotCount}</td>
|
||||
<td className="text-secondary">{formatDate(d.latestSnapshotAt)}</td>
|
||||
<td>{verificationBadge(d)}</td>
|
||||
</tr>
|
||||
))}
|
||||
{sortedDatastores?.length === 0 && (
|
||||
<tr>
|
||||
<td colSpan={5} className="text-secondary text-center">
|
||||
No datastores found.
|
||||
</td>
|
||||
</tr>
|
||||
)}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{sortedDatastores?.some((d) => d.recentFailures.length > 0) && (
|
||||
<div className="col-12">
|
||||
<div className="card">
|
||||
<div className="card-header">
|
||||
<h3 className="card-title">Recent verification failures</h3>
|
||||
</div>
|
||||
<div className="table-responsive">
|
||||
<table className="table table-vcenter card-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Datastore</th>
|
||||
<th>Backup</th>
|
||||
<th>When</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{sortedDatastores
|
||||
.flatMap((d) => d.recentFailures.map((f) => ({ datastore: d.name, ...f })))
|
||||
.map((f, idx) => (
|
||||
<tr key={`${f.datastore}-${f.backupType}-${f.backupId}-${f.backupTime}-${idx}`}>
|
||||
<td>{f.datastore}</td>
|
||||
<td className="text-secondary">
|
||||
{f.backupType}/{f.backupId}
|
||||
</td>
|
||||
<td className="text-secondary">{formatDate(f.backupTime)}</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -19,6 +19,7 @@ const INTEGRATION_NAMES: Record<string, string> = {
|
||||
dockhand: "Dockhand",
|
||||
uptimekuma: "Uptime Kuma",
|
||||
phpipam: "phpIPAM",
|
||||
pbs: "Proxmox Backup Server",
|
||||
};
|
||||
|
||||
function ColorList({
|
||||
|
||||
@@ -46,6 +46,7 @@ const DEFAULT_NOTIFICATIONS: NotificationEvents = {
|
||||
tailscaleKeyCheck: true,
|
||||
dockerUpdateCheck: true,
|
||||
proxmoxBackupCheck: true,
|
||||
pbsVerificationCheck: true,
|
||||
healthAlerts: true,
|
||||
automationAlerts: true,
|
||||
domainExpiryCheck: true,
|
||||
@@ -527,6 +528,7 @@ export default function NotificationSettings() {
|
||||
{ key: "tailscaleKeyCheck" as const, label: "Tailscale key expiry reminder" },
|
||||
{ key: "dockerUpdateCheck" as const, label: "Docker image update available" },
|
||||
{ key: "proxmoxBackupCheck" as const, label: "Proxmox backup failed or a guest has no coverage" },
|
||||
{ key: "pbsVerificationCheck" as const, label: "Proxmox Backup Server snapshot failed verification" },
|
||||
{ key: "healthAlerts" as const, label: "Server offline, disk nearly full, or Synology volume/disk problem" },
|
||||
{ key: "domainExpiryCheck" as const, label: "Domain registration expiring or expired (daily reminder)" },
|
||||
{ key: "automationAlerts" as const, label: "Semaphore template or Gitea workflow run failed" },
|
||||
@@ -618,6 +620,7 @@ export default function NotificationSettings() {
|
||||
notifications.tailscaleKeyCheck ||
|
||||
notifications.dockerUpdateCheck ||
|
||||
notifications.proxmoxBackupCheck ||
|
||||
notifications.pbsVerificationCheck ||
|
||||
notifications.domainExpiryCheck;
|
||||
return (
|
||||
<div className="row g-2 mt-2">
|
||||
|
||||
Reference in new issue
Block a user