Broken Docker containers are alerts

From Dockhand, on the health pass: a container that keeps restarting (critical, with
its exit code), one whose own health check fails, one that stopped unexpectedly, one
created and never started for an hour, and a Docker host Dockhand cannot reach. A stop
is only news when it was seen happen, so stacks switched off long ago stay quiet.
A Leave alone button on the Docker page and a list in Settings; on by default.

Also fixes a date-dependent storage-forecast test that pinned its clock while the code
under test reads the real one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-10-24 21:03:56 +02:00
1 parent 16b24f5436
commit 0fa50c2d5a
20 files changed
+659 -17

No files matched your search

+22
View File
@@ -12,6 +12,7 @@ This page explains what Homelab Manager watches, how often, how a problem become
- [What is normal for the time](#what-is-normal-for-the-time)
- [Service levels](#service-levels)
- [Your own alert rules](#your-own-alert-rules)
- [Docker containers that are broken](#docker-containers-that-are-broken)
- [Endpoint checks and heartbeats](#endpoint-checks-and-heartbeats)
- [Integrations](#integrations)
- [Forecasts](#forecasts)
@@ -127,6 +128,7 @@ Every threshold here is a default; where it can be changed, the place is named.
| **Restore test** | A restore test failed, or none has been done within the window | failure always; age off by default | Settings → Notifications |
| **Expiry** | A secret, certificate, domain or Tailscale key is within its warning window or has expired | per item (domains 30 days) | the item; Settings → Notifications |
| **Docker updates** | A container image has a newer version | | Settings → Notifications |
| **Docker container broken** | A container keeps restarting (critical), is unhealthy, stopped unexpectedly, or was created and never started; or Dockhand cannot reach a Docker host | on | Settings → Notifications; [below](#docker-containers-that-are-broken) |
| **Automation failed** | A Semaphore template's or Gitea repository's latest run failed (a run still going or cancelled does not count) | | not configurable |
| **Integration failing** | An integration's calls have failed several times in a row ([Integrations](#integrations)) | 3 calls | Settings → Notifications |
| **Endpoint check failing / certificate running out** | See [below](#endpoint-checks-and-heartbeats) | | per check |
@@ -190,6 +192,26 @@ A rule's finding is an **ordinary alert** with the key `rule:<rule id>:server:<s
server, damped when flapping, routed by the notification rules, and an event for webhooks. Rules are looked at on the 15-minute health pass, so "for 15 minutes" in practice means 15 to 30. A server that is
offline or has not reported for 45 minutes is not judged.
## Docker containers that are broken
If you use Dockhand, the health pass (every 15 minutes) also reads each container's **state** and **status text** — the same text the Docker page shows — and raises an alert (category *Failing services*, source *Docker*) for:
| What | When | Severity |
|---|---|---|
| **Restart loop** | Docker says the container is restarting ("Restarting (127) 39 seconds ago") | critical |
| **Unhealthy** | The container's own health check fails ("Up 6 days (unhealthy)") | warning |
| **Stopped unexpectedly** | It is exited or dead, and it was **running at the last look**, or it has never been seen and stopped **within the last day** (from "Exited (0) 15 minutes ago") | warning |
| **Never started** | It was created and has stayed "Created" for **an hour** | warning |
| **Docker host unreachable** | Dockhand could not read one of its environments (a Hawser or Docker agent that is down) | warning |
The hard part is *stopped*: many containers are stopped on purpose (a stack that is switched off, a one-shot job, a watchtower that ran and exited). So a stop is only news when it was **seen happen** — a container that has been down all along (a stack
that exited two weeks ago, one that exited three months ago) is left alone. Once raised, a stop stays raised until the container runs again, is removed, or is left alone. Only what the app has seen is judged, so after switching it on nothing alerts for what was already
long stopped. The scanner containers Dockhand starts (`dockhand-grype-*`, `dockhand-trivy-*`) are always left alone.
**Leaving a container alone:** the *Leave alone* button on the Docker page (administrators; *Watch again* undoes it), or the list under Settings → Notifications → What to send → *Containers to leave alone* — one per line, a name or `host/name`, `*` for anything.
Maintenance on the Docker integration holds these alerts like its other ones. If Dockhand itself cannot be read the containers' alerts are held rather than cleared (and the integration is raised as failing, as any other). Switch the whole thing off with *Warn when a
Docker container is broken*. What the app has seen of each container is kept between looks (in `settings`, key `_internal:containerMemory`).
## Endpoint checks and heartbeats
**Endpoint checks** (Monitoring → Endpoints) are for the simple "is it up?" that does not need a monitoring server of its own. Three kinds: