Duplicati: know when a backup failed or has gone quiet
Duplicati reports each operation to a secret address (its own send-http-url option), so nothing calls Duplicati and no password for it is kept. Each machine is a source; each job is learned by name and shows its last result, duration, size, changes and a history. A failed run is a warning (critical from the second in a row); one that has not succeeded for longer than expected (per job, 2 days by default) is a warning, critical at three times that. Maintenance on the machine's server silences them; /metrics series added. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
12f2662465
commit
2f082ca5c2
23 files changed
+8047
-4
No files matched your search
+48
-3
@@ -23,8 +23,8 @@ what the app actually creates.
|
||||
|---|---|
|
||||
| **File** | `DATABASE_PATH`, default `../data/homelab-manager.sqlite` (relative to `server/`; `/app/data/...` in Docker). The folder is created if missing. |
|
||||
| **Foreign keys** | Switched on for every connection (`PRAGMA foreign_keys = ON`), so the `ON DELETE` rules below are enforced. |
|
||||
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0055`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
||||
| **Tables** | 65 application tables, plus drizzle's own `__drizzle_migrations`. |
|
||||
| **Migrations** | SQL files in [`server/drizzle/`](server/drizzle) (`0000` … `0056`), applied automatically when the server starts. Which ones have run is recorded in the table `__drizzle_migrations`. |
|
||||
| **Tables** | 68 application tables, plus drizzle's own `__drizzle_migrations`. |
|
||||
|
||||
**Not in the database, but beside it in the data folder:** the backups of the database itself (`backups/homelab-manager-….sqlite`, taken daily and on request; see [BACKUP.md](BACKUP.md)), and, only while a restore is waiting or has just happened, `restore-pending.sqlite` / `restore-pending.json` and `restore-result.json`.
|
||||
|
||||
@@ -129,7 +129,7 @@ Who changed what. Written by the app on every change (see
|
||||
| `id` | integer PK | |
|
||||
| `actor_user_id` | integer → `users.id`, **set null** on delete | Null for automatic actions, and for entries whose user was deleted. |
|
||||
| `actor_label` | text | The user's name/email as it was at the time (or `system`), so an entry still reads correctly after the account is gone. |
|
||||
| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `dependency` (`create`, `delete` — what needs what), `endpoint` (`create`, `update`, `delete`), `restore_test` (`record`, `delete`), `push` (`subscribe`, `unsubscribe` — a browser added or removed, never its address or keys), `share_link` (a link made or removed, and `update_status_page` when what a status page shows is changed — never the secret), `runbook`, `alert` (`ack`, `unack` — with the alert's id; on the thing it is about, so they show on a server's Activity card), `incident` (`note`, `note_removed`), `db_backup` (`create`, `upload`, `download`, `delete`, `restore_staged`, `restore_cancelled`, `settings`), `away` (`start`, `end` — with `automatic: true` when its time ran out), `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. |
|
||||
| `category` | text NOT NULL | Free text. In use: `server` (including the health check's own `went_offline`, `back_online`, `disk_full` and `disk_ok`, and `agent_updated` when a server's agent reports a higher version than before, written as `system` and shown on a server's Activity card), `integration` (including the health check's `went_failing` / `recovered` for a kind of service, with `target_type` `integration_type` and the service type as `target_id`, which the dashboard summary and the uptime ribbons read), `dns`, `settings` (including `agent_auto_update`), `ipam`, `secret`, `domain`, `tag`, `task`, `session`, `network`, `maintenance`, `consistency`, `user`, `privacy`, `diag_log`, `alert_ticket`, `dependency` (`create`, `delete` — what needs what), `endpoint` (`create`, `update`, `delete`), `restore_test` (`record`, `delete`), `push` (`subscribe`, `unsubscribe` — a browser added or removed, never its address or keys), `share_link` (a link made or removed, and `update_status_page` when what a status page shows is changed — never the secret), `runbook`, `alert` (`ack`, `unack` — with the alert's id; on the thing it is about, so they show on a server's Activity card), `incident` (`note`, `note_removed`), `duplicati` (`create_source`, `update_source`, `delete_source`, `update_backup`, `forget_backup`), `db_backup` (`create`, `upload`, `download`, `delete`, `restore_staged`, `restore_cancelled`, `settings`), `away` (`start`, `end` — with `automatic: true` when its time ran out), `update_run` (`start`, `finished`, `failed`, `cancelled` — shown on the server's Activity card), `inventory`, `metrics_token` and `webhook` (made, changed or removed — never a secret or an address). `user` also records `role_from_groups` when a sign-in changed someone's role. |
|
||||
| `action` | text NOT NULL | `create`, `update`, `delete`, `start`, `stop`, … — free text. |
|
||||
| `target_type`, `target_id` | text | What it happened to. `target_id` is text so it can hold any kind of ID; there's no foreign key. |
|
||||
| `detail` | text | JSON, free-form context (what changed, names, counts). Secrets are never put here. |
|
||||
@@ -862,6 +862,50 @@ A service-level target (Monitoring → Service levels): what a thing is held to,
|
||||
|
||||
Nothing about how a target is doing is stored: it is worked out when asked, from `endpoint_daily` and the audit log.
|
||||
|
||||
#### `duplicati_sources`
|
||||
|
||||
A machine running Duplicati that reports to this app. `token_hash` is the SHA-256 of the secret in its address (shown once, never stored); `server_id` is a plain reference to the server it runs on (maintenance on it silences the alerts).
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer PK | |
|
||||
| `name` | text NOT NULL | |
|
||||
| `token_hash` | text NOT NULL, unique | |
|
||||
| `token_prefix` | text NOT NULL | The first characters, to tell addresses apart. |
|
||||
| `server_id` | integer | |
|
||||
| `created_by`, `created_at` | text | |
|
||||
| `last_report_at` | text | ISO; the last time anything arrived. |
|
||||
|
||||
#### `duplicati_backups`
|
||||
|
||||
One backup job of one source, as its last report left it; learned from a first report, by name (unique with `source_id`). `source_id` is a foreign key with **cascade**.
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer PK | |
|
||||
| `source_id`, `name` | integer, text | |
|
||||
| `last_report_at`, `last_begin`, `last_end` | text | ISO. |
|
||||
| `last_result` | text | `Success`, `Warning`, `Error` or `Fatal`. |
|
||||
| `duration_seconds`, `examined_files`, `size_bytes`, `added_files`, `modified_files`, `deleted_files` | integer | As the last report said; null when it did not. |
|
||||
| `warnings`, `errors` | integer NOT NULL, default 0 | |
|
||||
| `message` | text | The first error, or warning. |
|
||||
| `last_success_at` | text | ISO; a run with warnings counts. |
|
||||
| `consecutive_failures` | integer NOT NULL, default 0 | Critical from 2. |
|
||||
| `expected_hours` | integer NOT NULL, default 48 | How long it may go without succeeding. |
|
||||
| `alert` | bool NOT NULL, default 1 | |
|
||||
|
||||
#### `duplicati_runs`
|
||||
|
||||
Each report of a backup, for its history: 400 days and at most 400 per job. `backup_id` is a foreign key with **cascade**; (`backup_id`, `at`) is indexed.
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer PK | |
|
||||
| `at` | text NOT NULL | ISO; when the backup finished. |
|
||||
| `result` | text NOT NULL | |
|
||||
| `duration_seconds`, `size_bytes`, `files_changed` | integer | |
|
||||
| `warnings`, `errors` | integer NOT NULL, default 0 | |
|
||||
|
||||
#### `alert_rules`
|
||||
|
||||
Limits of the administrator's own (Monitoring → Alert rules), judged for the servers they cover on the health pass; what they find is ordinary health conditions keyed `rule:<id>:server:<server>[:<disk>]`.
|
||||
@@ -1238,6 +1282,7 @@ are deleted), and `notification_queue` is emptied each time the quiet-hours dige
|
||||
| `storage_samples` | 400 days (a pool not reported for 30 days is deleted with its readings) |
|
||||
| `ha_readings` | 35 days |
|
||||
| `ups_readings` | 35 days |
|
||||
| `duplicati_runs` | 400 days, and at most 400 per backup job |
|
||||
| `heartbeat_pings` | 14 days |
|
||||
| `alert_incidents`, `incident_notes` | 400 days after the incident resolved |
|
||||
| `webhook_deliveries` | the newest 100 per webhook, and 14 days |
|
||||
|
||||
Reference in new issue
Block a user