Add Hetzner Robot: dedicated servers and storage boxes, with shutting-down, locked and rescue-left-on alerts

Read every 15 minutes (Robot allows 200 requests an hour; read-only): each server's product, addresses, status, rescue system and paid-until, and the storage boxes. Adapter, sync service, hetzner_robot_state table, /api/hetzner-robot, Infrastructure → Hetzner Robot page, metrics and docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bobbanandClaude Sonnet 5.5 committed 2026-11-04 23:26:26 +01:00
1 parent 7618bf0f0a
commit 01acbaf831
31 files changed
+10105 -10

No files matched your search

+14 -1
View File
@@ -15,7 +15,7 @@ the dashboard views themselves load fine. (UniFi, Nginx Proxy Manager, Home Assi
### Checking what an integration can read
On the Integrations page an administrator has an **Inspect** button on each integration that reads through the shared HTTP helper (Authentik, Immich, Jellyfin, NetBox, Nextcloud, Paperless-ngx, Prometheus, Hetzner Cloud, Speedtest Tracker, Fragnet, Sonarr, Radarr, Prowlarr, Lidarr, Readarr, qBittorrent, SABnzbd, AdGuard Home, Technitium DNS and Grafana). It runs the integration's reader once and shows:
On the Integrations page an administrator has an **Inspect** button on each integration that reads through the shared HTTP helper (Authentik, Immich, Jellyfin, NetBox, Nextcloud, Paperless-ngx, Prometheus, Hetzner Cloud, Hetzner Robot, Speedtest Tracker, Fragnet, Sonarr, Radarr, Prowlarr, Lidarr, Readarr, qBittorrent, SABnzbd, AdGuard Home, Technitium DNS and Grafana). It runs the integration's reader once and shows:
- **what was asked** — each request, its path (with the *names* of the query parameters, not their values), the status, how long it took and how many bytes;
- **what came back** — the *shape* of each JSON answer to three levels: the names of the fields and the kind of each (`string`, `number`, `array(12) of { … }`), with a `?` on a field that only some items of a list have. Never a value, never a token, so the report can be copied into a message;
@@ -432,6 +432,19 @@ osTicket's API cannot close a ticket, so this is the one place in the app that *
- **A wrong password is retried every five minutes**, and qBittorrent bans an address for an hour after a few wrong ones (*Web UI → Ban client after consecutive failures*). If the integration says it was refused with HTTP 403, fix the password and wait out the ban (or restart qBittorrent).
- **Not verified against live installs** — the endpoints and fields are as the two applications' published API documentation has them, and the reader is tried against pretend ones. qBittorrent 5 renamed some states (`pausedDL` became `stoppedDL`); both are understood. SABnzbd's figures arrive as text, in megabytes and gigabytes, and are converted. Torrents are not paused, removed or added from here, and no NZB is added.
### Hetzner Robot
- **Config fields:** the **webservice user name** (it looks like `#ws+abc123…`) and its **password** (stored encrypted), made in Hetzner Robot → Settings → Webservice and app settings. That is not the login to Robot itself. There is no address to give: Robot is always `https://robot-ws.your-server.de`. This is the *dedicated servers and storage boxes* side of Hetzner; the cloud servers are the **Hetzner Cloud** integration, with a project token.
- **Required access:** the webservice user, reachable from the app over HTTPS. HTTP Basic is sent with every request, and the app only ever sends GET requests.
- **What it does:** every **fifteen minutes** it reads `GET /server` (number, name, product, data centre, addresses, traffic plan, status, cancelled, paid until), `GET /server/{number}` for each server (whether its **rescue system** is on — it is not in the list) and `GET /storagebox` (product, location, cancelled, locked, paid until, the linked server). **Infrastructure → Hetzner Robot** shows them. Nothing is reset, nothing is activated, ordered or cancelled.
- **Fifteen minutes, and 25 servers:** Robot allows **200 requests an hour**, and a read is a list, one request per server and a list of storage boxes — so ten servers is 12 requests a read, 48 an hour. Only the first 25 servers (by number) are asked about their rescue system; any more are listed, counted on the page, and left without a rescue state.
- **Alerts** (source *Hetzner Robot*; held back by maintenance on the integration): a **server or storage box that is cancelled and ends within 30 days** (critical within 7; the date is the last day it runs, and the data goes with it); a **storage box that is locked** (critical); a **rescue system on for two hours** (the server's next boot is into the rescue system); a server **still "in process" a day** after the app first saw it so. See [MONITORING.md](docs/MONITORING.md#hetzner-robot).
- "Since when" (rescue on, not ready) is what this app saw: the time of the first read that showed it.
- If the account cannot be read for 45 minutes its alerts are neither raised nor cleared, the Alerts page says it could not check, and the integration is raised as failing like any other. A part that could not be read (the storage boxes, a server's rescue state) is shown as *could not be read*, never as none or off.
- `/metrics` has `homelab_hetzner_robot_servers`, `homelab_hetzner_robot_rescue_active`, `homelab_hetzner_robot_days_left` and `homelab_hetzner_robot_storage_box_locked`.
- **A wrong password is retried every fifteen minutes**, and Robot blocks an address for a while after a few wrong ones (HTTP 403). If the integration says it was refused, fix the password and wait.
- **Not verified against a live account** — the endpoints and fields are as Hetzner's published webservice documentation has them, and the reader is tried against a pretend one. An account with no servers (or no storage boxes) is answered with a 404 and a code ending in `NOT_FOUND`, read as none. Whether Robot switches the rescue system off by itself after a while is not known here, so the alert is for one that has been on two hours; traffic used is not read (Robot gives it per address, and the plan only).
### Hetzner Cloud
- **Config fields:** the project's **API token** (Hetzner Console → the project → Security → API tokens → *Generate API token*; stored encrypted). **Read** permission is enough — and recommended: the app only ever sends GET requests. A token belongs to **one project**, so add one integration for each project you want to watch.