Duplicati: know when a backup failed or has gone quiet

Duplicati reports each operation to a secret address (its own send-http-url option), so
nothing calls Duplicati and no password for it is kept. Each machine is a source; each job
is learned by name and shows its last result, duration, size, changes and a history.
A failed run is a warning (critical from the second in a row); one that has not succeeded
for longer than expected (per job, 2 days by default) is a warning, critical at three
times that. Maintenance on the machine's server silences them; /metrics series added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-24 21:36:13 +02:00
1 parent 12f2662465
commit 2f082ca5c2
23 files changed
+8047 -4

No files matched your search

+19
View File
@@ -12,6 +12,7 @@ This page explains what Homelab Manager watches, how often, how a problem become
- [What is normal for the time](#what-is-normal-for-the-time)
- [Service levels](#service-levels)
- [Your own alert rules](#your-own-alert-rules)
- [Duplicati backups](#duplicati-backups)
- [Docker containers that are broken](#docker-containers-that-are-broken)
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
- [Integrations](#integrations)
@@ -128,6 +129,7 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Duplicati backup failed / overdue** | A backup's last run failed (critical from the second in a row), or it has not succeeded for longer than it should | per job (2 days by default) | Duplicati page; [below](#duplicati-backups) |
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
@@ -192,6 +194,23 @@ A rule's finding is an **ordinary alert** with the key `rule:<rule id>:server:<s
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is
offline or has not reported for 45 minutes is not judged.
## Duplicati backups
Duplicati tells this app how each backup went ([INTEGRATIONS.md](../INTEGRATIONS.md#duplicati) says how to set it up), and **Backups → Duplicati** shows every job: when it last succeeded, how the last run ended, how long it took, how big the source is, how many files changed, and where it stands.
| State | Meaning | Alert |
|---|---|---|
| **In order** | The last run worked and it is not overdue | — |
| **Warnings** | The last run finished with warnings (a locked file, say). It still counts as a backup | none |
| **Failed** | The last run ended in `Error` or `Fatal` | warning; **critical from the second failure in a row** |
| **Overdue** | It has not succeeded for longer than it is **expected** to — 2 days unless you set it per job (6 hours to 90 days) | warning; **critical at three times that** |
The alert says which job, on which machine, why (the first error Duplicati reported) and when the last good backup was. Only the *Backup* operation counts: a compact, a repair or a verification does not say whether the data is safe. A job can be **muted** (no alert; it is still listed),
and one that no longer exists can be **forgotten** (with its history; if Duplicati reports it again it is learned again). Each machine can be tied to a server, so **maintenance on that server silences its backups' alerts**.
What is kept: the last result of each job, and each report for 400 days or 400 reports. `/metrics` has `homelab_duplicati_backup_ok`, `homelab_duplicati_last_success_age_seconds` and `homelab_duplicati_source_size_bytes` for each job. A job is only learned from its first report, so a job that has
*never* reported is not known to be missing; and because the reports are pushed, **the app being down when a backup finishes** loses that report (the backup shows as overdue only once it is).
## Docker containers that are broken
If you use Dockhand, the health pass (every 15 minutes) also reads each container's **state** and **status text** — the same text the Docker page shows — and raises an alert (category *Failing services*, source *Docker*) for:
+2
View File
@@ -294,6 +294,8 @@ All gauges. The same table is checked by a test against what `/metrics` really s
| `homelab_endpoint_certificate_days_left` | `name`, `kind` | Days until the certificate it shows expires (negative once it has). |
| `homelab_service_level_target_percent` | `name`, `subject`, `type` | The availability a service-level target asks for. |
| `homelab_service_level_availability_percent`, `homelab_service_level_error_budget_remaining_minutes` | `name`, `subject`, `type` | How available the thing has been so far this calendar month, and the downtime still allowed before the target is missed (negative once it is). Left out for a target with nothing recorded this month. |
| `homelab_duplicati_backup_ok` | `backup`, `source` | 1 if the Duplicati backup's last run worked and it is not overdue, else 0. |
| `homelab_duplicati_last_success_age_seconds`, `homelab_duplicati_source_size_bytes` | `backup`, `source` | Seconds since it last succeeded (a run with warnings counts), and the size of what its last report examined. |
| `homelab_ups_on_battery` | `name` | 1 if the UPS has lost mains power and runs on its battery, else 0; left out for a UPS that has not been read in the last five minutes. |
| `homelab_ups_battery_charge_percent`, `homelab_ups_runtime_seconds`, `homelab_ups_load_percent`, `homelab_ups_input_volts` | `name` | How full the battery is, how long it would last at the present load, how much of its capacity the load uses, and the mains voltage it sees — each left out when the UPS does not say. |
| `homelab_heartbeat_up` | `name` | 1 if the job has reported in time and not said it failed, else 0; switched-off heartbeats are left out. |