Files
Homelab-manager/README.md
T
bobbanandClaude Sonnet 5.5 06ac120a1d Services: opt-in monitoring with a per-server watch list
A failed service now raises an alert (and counts in /metrics and the public API) only when the server is told to watch it, by name or pattern, or * for all. Failed services that are not watched still show on the Host health card with a Watch button. Replaces the ignore list.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-16 21:48:32 +01:00

1220 lines
146 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Homelab Manager
Repository: `git@10.200.5.13:bobban/Homelab-manager.git` ([gitea.labsconnect.se/bobban/Homelab-manager](https://gitea.labsconnect.se/bobban/Homelab-manager) externally).
A single dashboard for a homelab: Proxmox, Synology DSM, Semaphore, Tailscale,
Gitea, Dockhand/Docker, Uptime Kuma, Proxmox Backup Server, and osTicket status
and basic actions, plus DNS record
management, an IP address inventory (IPAM), and a secret-expiry tracker
(ported from [Sloth Manager](../Sloth%20manager)) and scheduled-task tracking
across Debian/Raspbian hosts (ported from
[Schedule Task Manager](../ScheduleTaskManager)). Looks and feels like a
[Tabler](https://tabler.io) admin dashboard. Sign-in is delegated to
Authentik (OIDC), with local admin/operator/viewer roles.
## Status
All modules from the original plan are built:
- Monorepo scaffold, Tabler-themed app shell with a grouped sidebar (Infrastructure, Apps, Backups, Network, Monitoring, Automation, Operations, Records, Reference, Administration; groups open on demand, the one holding the current page is always open, and what you leave open is remembered)
- Authentik OIDC login, roles (first user to sign in becomes admin), audit log
(changes made in the app, sign-ins and sign-outs with the IP they came from, new accounts, and the log's own
automatic trimming — attributed to "system"; settings changes show what changed, but never credentials)
- **Dashboard** — opens with a status banner that answers "is it fine?": one sentence ("All clear", "3 things need attention"), the root cause when one
problem explains others, which kinds of thing are wrong (each links to those alerts) and the worst few. Below it, a card whose status badge is red or yellow
comes first in its row, with a coloured top edge (switchable under Customise, remembered in that browser; your own arrangement is untouched underneath).
The rest is an overview of every system this app tracks, all sharing
one widget-card design (label + status badge, a small stat row, then its
own breakdown): DNS (domain/record counts per provider, cached records by
type), Secrets (monitored/expiring/expired, by type), and one widget per
integration — Tailscale by OS, Proxmox by node (VM/LXC counts too),
Dockhand by container state (plus host count), Semaphore and Gitea by
last-run status (plus private-repo count), and Synology's CPU/RAM
alongside its disk-health breakdown. The integration widgets are in sections that follow the sidebar's groups (Infrastructure, Apps, Network and so on) and show only
the groups you have something connected in; an integration with no widget of its own (Jellyfin, Immich, Nextcloud, Hetzner, Speedtest …) gets a small tile: whether it works,
how many alerts are open for it, and a link to its page. Breakdowns render as a stacked
proportion bar with a legend — no charting
library, matching the rest of the app's plain-Tabler-CSS approach.
The Servers widget shows how many servers are online, offline, or have never
reported, which are offline and for how long, and any disk at or above the
usage threshold — decided by the same rules as the health alerts, so the
widget and the notifications always agree, and following the thresholds in
Settings even for viewers who can't open them.
The Domains widget shows how many registrations are tracked, which are
expired or expiring (soonest first), the next one to expire, and how many
couldn't be refreshed.
The Uptime Kuma widget shows the monitor count, how many are down, and how
many are matched to one of your servers.
The Proxmox Backup Server widget shows the datastore count and how many
stored snapshots have failed verification or were never verified.
The osTicket widget shows how many tickets are open, overdue, and awaiting
a staff reply.
The Technitium DNS widget (in the Network section) shows whether DNS answers and how fast, the last day's queries and the share blocked, a bar of how they were answered
(cache, resolved, its own zones, blocked, dropped) and what is wrong with it; it is refreshed every minute and can be hidden under Customise like the rest.
- **Diagnostic Log** (admin-only; Administration → Integrations → *Call log*) — every call this app makes to a DNS
provider or integration (Tailscale, Proxmox, Synology, Semaphore, Gitea,
Dockhand, Uptime Kuma, Proxmox Backup Server, osTicket), success or failure, with latency and the error message if it
failed — the last 500 calls, filterable by source/result, for
troubleshooting connectivity issues (ported from Sloth Manager's
provider-diagnostics log, generalized to cover every integration this app
has, not just DNS)
- **Secrets** — expiry tracking for API tokens/certs/passwords. An SSL
certificate can optionally be given a host:port to watch: the app opens a
real TLS connection (daily, and on demand via "Check now"), reads the
certificate's actual expiry, and keeps the date current — so a renewed cert
is picked up automatically and an unreachable host is flagged instead of
silently going stale. A secret that doesn't expire (a break-glass password,
a key with no end date) can be ticked "Never expires": it is listed with the
status "no expiry" and never raises an alert or reminder
- **IP Addresses (IPAM)** — inventory of IPs across vendors/locations,
with "Sync from Tailscale", "Sync from Proxmox", and "Sync from phpIPAM"
actions to pull in tailnet device IPs, VM/LXC IPs, and phpIPAM's own
addresses (never overwrites a manually-entered IP, or one a different sync
owns), and each entry now shows its matching DNS record(s) from the DNS
module's cache. Shares the Consistency page's excluded-ranges setting (see
below) so addresses that aren't interesting to track — a Docker bridge
network repeating on every host, say — can be hidden here too, with a
"Hide excluded addresses" toggle and a per-entry "Exclude…" shortcut to add
a range on the spot; managing ranges from either page updates the other
- **DNS** — zone/record management across Cloudflare, Loopia, Pi-hole, Azure
DNS, cPanel, and Technitium; providers are configured in-app (not via env
vars) and their credentials are encrypted at rest. On a Technitium provider an
administrator can also **make a zone** (Primary, Secondary or Forwarder — a name, or a
network for a reverse zone) **edit** one (switch it off and on; who may ask it, copy it, and be told when it changes) and **delete** one (its name typed to confirm; every record in it goes), and
Technitium's own refusal (a zone that exists, a token that may not delete) is shown as it was said
- **Servers** — cron/systemd tracking across Debian/Raspbian servers via a
lightweight push agent (`agent/linux/`), Windows scheduled-task tracking
through a PowerShell agent (`agent/windows/`, see its README), plus manual
entries for things an agent can't see (Docker jobs, backups). The Servers page itself just lists
registered servers and (admin-only) adds new ones / issues agent tokens;
a server's **hostname** — which an agent replaces with the machine's own at every report — can be
changed by an administrator on its page (*Identity → Change*): it is then **kept** even when the
agent reports another (marked *set by hand*, with what the agent reports shown beside it), until
*Use what the agent reports* gives it up. It is the name this app has for the server, used to
match it to DNS names and monitors and to reach it; it does not rename the machine itself;
clicking a server opens its detail page with CPU/RAM/disk status, IP
addresses, matching DNS names (looked up from the DNS module's cache), and
its scheduled tasks — live hardware from Proxmox for VM/LXC-backed
servers, or from the agent's own hardware report for everything else,
both showing the same per-disk usage breakdown (an LXC's root filesystem
read straight from the host; a QEMU VM's actual mounts via its guest
agent, alongside the allocated size Proxmox already knew about without
one). Servers that report through the agent also get a **Trends** card: small
lines for CPU load, memory and each disk over the last 24 hours, 7, 30 or 90 days or a year
(a sample is kept about every ten minutes, for 35 days; hours and days beyond that, up to a year), and for a disk that is
filling, its rate of growth and roughly how many days until it is full — shown
only with at least a day of history. Proxmox-linked servers also get
start/stop/restart buttons right on the detail page. The detail page also has an **Admin Links** section
(operator/admin to add/edit/remove) for bookmarking that server's own
admin UIs — Dockge, Webmin, Cockpit, Portainer, or anything else reachable
by URL. **Reference → Admin Links** summarizes every server's admin links
in one sortable, searchable table (server, label, URL, an "Open" link, and
the same add/edit/delete as the per-server section — adding one here just
asks which server it belongs to), so finding or managing one doesn't mean
visiting each server's own page. Since not every server is a Proxmox VM, an admin can hide the
"Proxmox link" card per server ("Not a VM? Hide this" / "+ Show Proxmox
link options") — it stays visible regardless once a server actually is
linked, so unlinking is always reachable.
- **Tailscale**, **Proxmox**, **Synology**, **Semaphore**, **Gitea**,
**Docker**, **Uptime Kuma**, **Proxmox Backup Server**, and **osTicket**
each get their own top-level page (backed by the matching integration)
instead of living inside a shared Integrations browsing view:
- **Tailscale** — device list with online/authorized status, and
authorize/deauthorize/remove actions; a live device-count widget.
- **Proxmox** — VM/LXC status across every node in the cluster, with
start/restart/shutdown/stop actions; a live running/total widget. Each
online node also gets its own host-stats card — uptime, CPU usage/cores/
load average, RAM and swap usage, and per-storage usage (local, LVM-thin,
ZFS, NFS, etc). Supports self-signed certificates (common in homelab
setups).
- **Synology** — volume and disk health (read-only by design).
Supports self-signed certificates.
- **Semaphore** — Ansible run status per template across every
project, with a "Run" action to trigger a template; a live
template-count widget (with a last-failed warning).
- **Gitea** — repo list with each repo's last CI run status, and
re-running just the failed jobs in a run; a live repo-count widget
(with a failing-build warning).
- **Docker** — container status across every Docker host Dockhand
manages (one credential covers all of them), with
start/stop/restart actions, a host filter, image-update status per
container (from Dockhand's own cached update check, plus a button
to trigger a fresh one), a live running/total widget (with an
updates-available count), and the **vulnerabilities** Dockhand's own
scans found per host (critical ones listed; a warning is raised for the critical ones — and, if you give a number, for hosts with that many high ones — and
it can be switched off). A number of days you choose also makes running images built longer ago than that a warning — scanned or not (off by default).
- **Uptime Kuma** — every monitor's status (up/down/pending/maintenance),
response time, and 24h/30d uptime, with certificate days remaining where
applicable. Each monitor is matched to one of your servers when its
target (an IP, or a TCP/HTTP hostname) lines up with that server's own
address or hostname, linking straight to it — so you can see what's
actually being watched on each box, not just a flat monitor list.
Uptime Kuma has no conventional REST API (the dashboard talks to it over
Socket.IO), so this reads its Prometheus `/metrics` endpoint instead and
parses that itself; read-only, no actions.
- **Proxmox Backup Server** — every configured datastore's usage and
snapshot count, the PBS host's own CPU/RAM/disk, and each stored
snapshot's verification status, since Proxmox VE only knows whether a
backup *ran*, never whether PBS's own verify pass on the stored data
still passes; a daily check alerts on any snapshot that's failed
verification or a datastore that couldn't be read. Read-only, no
actions (nothing here can prune, delete, or trigger a re-verify).
- **osTicket** — every currently open ticket, with its status, priority,
department, assigned staff member or team, requester, and two flags
worth a glance on their own: **overdue** and **awaiting our reply**.
osTicket's own REST API only supports *creating* tickets, not listing
them, so this reads osTicket's MySQL/MariaDB database directly with a
read-only user — the only integration here that isn't a REST API.
Read-only, no actions.
The Integrations page itself is a list of configured
integrations (name/type/status, visible to every role), grouped the way the sidebar
groups their pages (Infrastructure, Apps, Backups, Network, Monitoring, Automation),
with an admin-only "Add integration" button — its kind picker is the same groups,
with a search box — and edit/enable/disable/delete actions per row. The dedicated
pages are where you actually use each one. A **Cards / Table** switch beside the search box
shows the same integrations as a table instead — name, kind, group, address, status, the
last 30 days and the same buttons — sortable on any column (Status puts the failing ones
first); the choice is in the address (`?layout=table`, with `?sort=`), so it can be linked or
saved as a view, and this browser remembers it for the next visit. Search and the status
filter work in both.
- Every table in the app is click-to-sort on any column (numbers, booleans,
and dates/text sort correctly regardless of how the column formats them)
and has an "Export CSV" button next to it that exports whatever's
currently sorted/filtered. Tables that can realistically grow large
(DNS zones/records, IP Addresses, Secrets, Servers, Audit Log, and each
integration's device/container/guest/repo/template list) are paginated,
20 rows per page by default — adjustable under Settings → Display — and
CSV export still covers every sorted/filtered row, not just the current
page.
The few small tables that have no search or sorting of their own (the
maintenance lists, tags, share links, API tokens and webhooks, the
Backup page's job and run lists, the comparison table) have an Export CSV
too, which writes the table as it is on screen — times as exact timestamps,
action buttons left out.
- **Alert settings** (Monitoring → Alert settings) — what decides what is an alert, on one page with three tabs: *Alert rules*, *Service levels* and *Alert tickets* (`/alert-settings?view=rules|levels|tickets`). Everyone signed in sees the first two; *Alert tickets* is administrators' (it used to be under Settings, and `/alert-rules`, `/service-levels` and `/settings/alert-tickets` still lead to the right tab). Notifications — channels, what is sent, who hears what, quiet hours — serve more than alerts, so they stay under **Settings → Notifications**, with a link to them at the top of this page for administrators. Maintenance stays on its own page.
- **Settings** (admin-only) — notification channels (Gotify, ntfy, Telegram,
Pushover, SMTP, generic webhook, and push to phones and browsers) with per-channel test buttons, per-event toggles (DNS
record added/updated/deleted, daily secret-expiry reminder and daily
Tailscale key-expiry reminder — both sharing one configurable
time/timezone), badge-color customization for both DNS
providers and integration types, and a **Display** tab (date order,
12/24-hour clock, and rows-per-page for every paginated table) applied
consistently across the app.
All six integrations follow the same config-in-UI + encrypted-credentials
pattern, added (and edited — e.g. to rotate an expired API token without
recreating the whole integration) through **Integrations → Manage
integrations**. See [INTEGRATIONS.md](INTEGRATIONS.md) for exactly what
credential to create and what access it needs in each target system,
for every integration and DNS provider.
**Verified for real, end to end**: every module above — including all six
integrations, both their read-only views and their write actions
(start/stop/restart, trigger-a-run, authorize/deauthorize) — has been
exercised against the user's actual live homelab, not just built against
specs. That pass also found and fixed two real bugs: the Synology adapter
assumed HTTPS-only (the NAS is reached over plain HTTP), and the Tailscale
adapter read `online`/`isExitNode` fields that don't actually exist in the
real API response (fixed to derive them from `connectedToControl` and
`enabledRoutes`). See the git log for the full verification notes per
integration. (Uptime Kuma, Proxmox Backup Server, and osTicket, added
later, are not part of that "six" — see their own git log entries, and
[INTEGRATIONS.md](INTEGRATIONS.md), for what was and wasn't verified
against a real instance.)
Server and storage health is watched every 15 minutes: a server whose agent
stops reporting, a server disk / Proxmox storage / Synology volume passing a
usage threshold, and a Synology volume or disk that's degraded or failing each
raise one notification when the problem starts and one when it clears (both
thresholds are set under Settings → Notifications). Active problems are
remembered across restarts, so a rebuild doesn't re-alert them.
**Tags** — servers can be tagged (prod, media, rack-1, …) from their detail
page by operators and admins. Tags show as coloured chips on the Servers page,
which can be filtered by one or several tags (the filter is in the URL, so a
tag on a server's page links to everything sharing it), and they're searchable
from the global search box.
Admins manage tags under Settings → Tags: add tags before any server uses them (they're
offered as one-click suggestions when tagging), give any tag a colour of your choosing
(or leave it on the automatic one), rename tags, and delete them. Renaming to a name
that already exists merges the two, and both rename and delete rewrite every server
that carries the tag.
**Name generator** — Reference → Generator suggests server names (and usernames and
passwords, which are made in your browser and never stored). Server names are picked from
name lists: Swedish girl and boy names, Disney and Pixar characters, Norse mythology and
Astrid Lindgren to start with, or "Mixed" for all of them together. A name already used
by a server isn't suggested. "How many" makes up to 20 different names at once (handy when
setting up several machines), each with its own Copy button and a "Copy all" that puts
them one per line on the clipboard. Admins edit the lists under Settings → Names — add and remove
names, rename, delete or create lists, put a built-in list back as it was — and can
import the most common Swedish names from Skatteverket's open name statistics for
girls or boys over the latest one to five years. The import is shown to you first and only
changes a list once you add it and save. (Statistics Sweden used to publish this but
stopped after 2023.) Names are kept to letters, digits and hyphens so they work as
hostnames; å, ä and ö become a, a and o.
**Agents and what the server connects to** — the agents only push: the server replies "received" and nothing more, so it has no way to
make a server run anything. Each server has its own token (kept only as a hash), which can do one thing: replace that server's own report.
The Linux agent hands its token to curl on standard input and keeps its payload in a private temporary directory, so neither is visible
to other users on the machine, and its systemd unit runs with the usual restrictions (no new privileges, read-only system, private /tmp) and keeps only the three
root powers the agent uses — reading every user's crontab, switching user to list it, and seeing which program owns each socket.
On Windows the agent's folder is limited to SYSTEM and Administrators. The agent endpoint checks the token before it reads a report, takes
reports of several MB, and trims overlong text rather than rejecting the whole report. **After upgrading, re-run the install one-liner on
each server** to pick this up — an installed agent keeps its own copy of the script and doesn't update itself.
A few features connect to a host somebody typed in: the certificate check on a secret, the WHOIS server a registry names, and the port
scanner. These refuse this server itself (loopback), link-local addresses and the cloud metadata addresses (169.254.169.254 and its
relatives), and connect to the address they checked rather than looking the name up a second time. LAN hosts are fine for certificate
checks and scans; a WHOIS server must be public. IDs in a request (a device, a node, a DNS record) are checked against what a real ID
looks like before they go into a call to a connected system. `npm test --workspace server` covers all of this.
**Signing in** — through Authentik (OpenID Connect) with PKCE and a random `state`. The ID token's signature is checked against Authentik's
published keys, not just its contents, so the login doesn't rest on the connection to Authentik alone. A new session id is issued at sign-in. A
request that changes something is refused when it comes from another origin: the session cookie is SameSite=Lax, which doesn't stop another app on
the same parent domain, so the server also checks where each such request came from. A failed sign-in shows a plain message and is written to the
audit log (as `session` / `login_failed`, with the address and a short reason), and sign-in is limited to 20 attempts a minute per address.
Every response carries `frame-ancestors 'none'`, `X-Frame-Options`, `nosniff` and a no-referrer-to-other-sites policy. In production the app
refuses to start without a `SESSION_SECRET`, and warns if it is still the placeholder from `.env.example`.
Things to know: the first account to sign in becomes admin, so sign in yourself first on a new install. A session lasts up to 7 days after last use,
and disabling someone in Authentik does not end it — end the session under Users → *Signed in now* and it ends at once (accounts themselves are not deleted in the app).
If Authentik uses a certificate from an internal CA, point Node at the CA with `NODE_EXTRA_CA_CERTS=/path/to/ca.pem` rather than turning verification
off with `NODE_TLS_REJECT_UNAUTHORIZED=0`.
**Privacy** — a page every signed-in user can open that says what this
installation stores (accounts, sign-in sessions with their IP and browser, the audit
and diagnostic logs, server reports, the secrets tracker, credentials), where data
goes (Authentik, your integrations and DNS providers, the notification channels
that are switched on, domain registries), what's kept in the browser, who can see
what, and how to limit or remove data. Retention, integrations and channels are
read live from the installation; channel addresses are shown to admins only. Each
user sees their own account and sign-ins there and can download their own data
(account, sign-ins, audit-log entries) as a JSON file.
**Consistency** — a report of where IPAM, DNS and your servers disagree about
an address: the same address on two servers, a DNS record named after a server
that points somewhere it isn't, an IPAM entry labelled with a server's name at
the wrong address, addresses in use that IPAM doesn't list (with an "Add to
IPAM" button), and server addresses no DNS record points at. It only compares
data the app already holds — nothing is fetched when you open it — so it says how
many servers reported addresses and how fresh the synced DNS zones are. Only
private addresses are compared; ranges you exclude (Docker's, which repeat the same
subnet on many hosts — 172.16.0.0/12 is excluded by default, remove it if that's a
real LAN for you) are left out of every source; anything that's fine on purpose
can be ignored with a reason, and stays ignored. The excluded-ranges list is the
same one the IP Addresses page manages — edit it from either page and both
reflect the change.
**Domains** (Network → DNS → *Domain names*) — when each domain registration expires, read from the registry
itself. The domains behind your DNS zones are picked up automatically (a zone
like `lab.example.se` resolves to the `example.se` registration that actually
expires); others can be added by hand. Each is looked up daily over RDAP where
the TLD offers it, and otherwise over WHOIS via IANA's referral — which is what
makes `.se`, `.nu` and `.io` work, since those aren't in the RDAP bootstrap.
You're reminded daily from N days before expiry (Settings → Notifications,
default 30) until it's renewed, and told when an expiry date couldn't be
refreshed for several days. Registries that don't publish an expiry (`.de`,
`.eu`) can be tracked but have no date to warn about. Private zones (`.lan`,
`.local`) are skipped.
**Automation failures** — every 15 minutes the app looks at the latest run of
each Semaphore template and each Gitea repo's latest workflow run. A failed one
raises a single notification (with the project/template or repo, run number, and
for Gitea the run's link), and another when a later run succeeds. It's
state-based, so a job that fails every night alerts on the first failure rather
than every night. A run that's still going, was cancelled, or was stopped by
hand leaves things as they were, and anything that couldn't be read (a Semaphore
project or Gitea repo that errored, or an integration that's down) is neither
cleared nor re-announced. The first check after upgrading only records what's
already failing, so old failures aren't announced. Toggle it under Settings →
Notifications. For Gitea this follows the repo's most recent run on any
workflow or branch, the same as the Gitea page shows.
**Alerts** (near the top of the sidebar, visible to every role) opens with one line on where things stand — how many problems need attention and, when some are
held back, which problem explains how many ("nasbox accounts for 3") — and marks each row with a coloured edge by severity. Held-back alerts are nested under
the alert that explains them (switch "Group under their cause" off for the flat list; the choice is remembered): the cause says "Show the 3 alerts held back
under this" and opens them. It lists everything that's
wrong right now in one place, instead of waiting for a notification or visiting
each page: servers that stopped reporting, full or nearly full disks and volumes
(critical from 95%), Synology volume/disk problems, failed or uncovered Proxmox
backups, failed Proxmox Backup Server verifications, container image updates,
secrets, domains and Tailscale keys that are expired or about to be, failed
Semaphore/Gitea runs, Uptime Kuma monitors that are down, overdue osTicket
tickets, and integrations whose calls keep failing. It runs the same checks that
send the notifications — so the two can't disagree — but ignores the on/off
toggles, since it's for looking at rather than being interrupted by. Problems
under a maintenance window stay listed, marked silenced and counted separately.
It also looks ahead: from an agent server's own history (see its Trends card), a
disk that is on course to be full within 14 days (Settings → Notifications → "Warn
when a disk will fill within"; 0 turns it off) is flagged with its rate of growth
and the days left — critical when that is three days or less. It needs at least
three days of steady growth to say anything, skips a disk that jumped once or
wobbles, and leaves out one already past the usual disk threshold, which has its
own alert. These forecasts appear on the page and the dashboard (and in alert
tickets if you've chosen disk problems); they are not sent as notifications. On
any alert that a maintenance window can silence, operators and admins get a
**Silence** button (1, 4 or 24 hours): it starts a maintenance window on that
server or integration — so everything about it goes quiet, which the toast says —
and it can be ended early under Maintenance.
It checks live (a recent result is reused for a minute; "Check now" forces a
fresh one), and anything it couldn't read is called out at the top rather than
quietly treated as fine.
**Alert tickets** (Alert settings → Alert tickets, admin only) opens a ticket in osTicket for an alert, through osTicket's own API. Every 15 minutes the
problems on the Alerts page are looked at, and one that is new and has lasted two checks in a row gets one ticket — never a second for the same problem,
and never one for something under a maintenance window or about osTicket itself. You choose which severities and kinds count (critical and warning by
default), and the Alerts page shows the ticket number on each alert that has one. If many problems appear at once — a network drop that makes thirty
things look down — they become one summary ticket instead of thirty. Problems already there when you switch it on are left alone unless you ask for
tickets for them too. osTicket's API can't close or update a ticket, so once it's open it's yours to close — in osTicket, or, if you give the integration a **separate database login that may update tickets**, from the Alerts page: the Close drawer then offers *Also close the osTicket ticket* — for one alert, or for several picked together (the tickets are closed one after another; Undo opens the alerts again but not the tickets) (it sets the ticket to Closed and writes a note in its thread saying who closed it and why;
this writes to osTicket's database, so it checks the tables first, does it all in one transaction, and **Test closing** tries every write and rolls it back). It needs an API key made in osTicket (tied to
this server's IP address, allowed to create tickets), entered on the osTicket integration; "Send a test ticket" checks the whole chain.
**Maintenance mode** silences alerts about one server, integration, or DNS
provider while you work on it (server offline / disk, storage and Synology
health, Proxmox backup alerts, Proxmox Backup Server verification alerts, and
"integration down" for that service type).
For a power cut or a rack reboot, the "Many at once" choices at the top of the
list start a window on every server, every (enabled) integration, every
(enabled) DNS provider, or all of them in one go — asking first before
"Everything" — and "End all" on the Active list closes them together. Each is
still its own window, so any can be ended alone. Every window has a fixed end (5 minutes to 7 days) and expires on its own, and
a problem that began during a window and is still present when it ends alerts
then — a forgotten window can't hide an outage. A banner shows what's currently
silenced to every signed-in user. If you also run Uptime Kuma, the Maintenance
page can take its maintenance windows over: "Sync schedule" reads each window
(daily, weekly, one-off, cron or switched on by hand), finds the servers,
integrations and DNS providers its monitors are the address of, and makes a
window here for each, for the slot running now and the next
one — listed under Scheduled until it starts, then silencing alerts like any
other. Tick "Keep maintenance windows in step" on the Uptime Kuma integration
and this happens by itself every 15 minutes, so a daily window needs no
clicking at all; a window changed, switched off or deleted in Uptime Kuma is
followed, and one you end or cancel here by hand stays ended. Reading the
schedule needs a login on the Uptime Kuma integration (see INTEGRATIONS.md).
"Import now" is the older, simpler route: it only looks at what is in
maintenance this moment and starts a window for a duration you choose.
**Favourites and a narrow sidebar.** Hover (or tab to) a page in the sidebar and
click its star to pin it: pinned pages are listed under "Favourites" at the top
of the menu, are offered first in the search box, and follow you to any
browser, because they are saved with your user (the search box can also pin or
unpin the page you are on). A table's search words, status, tag and sort live
in the address (`/servers?status=offline`), so they survive a reload and can be
bookmarked; once something narrows the table a **Save view** button pins that
view too, listed under Favourites by its filters ("Servers not reporting"). A
button at the foot of the sidebar — or the `[` key
— narrows it to a strip of icons for more room on a laptop screen: a group's
pages then open in a small menu beside its icon, the alert and maintenance
counts stay visible, and your choice is remembered in that browser. On a phone
the sidebar is the same slide-out menu as ever, and the pages people reach from a notification or a printed label are made for one hand: on Alerts,
Updates and Inventory each row becomes a compact card (worst first, with its buttons underneath), the buttons are finger-sized, the
acknowledge panel fills the screen, and the Alerts page leaves out its row of figures — the headline already says them. Scanning an
inventory label opens that piece of hardware at the top of the page, with an Edit button for operators.
**Getting around.** Ctrl/Cmd+K opens a search that also runs actions: go to any
page, switch light/dark or the density, start (or end) a maintenance window on
a server, integration or DNS provider ("silence elsa"), open the wall display,
sign out. It also does the things you do most often: **acknowledge** one of the worst alerts nobody is on (for 4 hours, handed to you) or all of them
at once, **update** a server that has updates waiting (it opens the update panel, which still asks before anything runs), **print the label** for a piece of
hardware or open it in the inventory, and ask **"What depends on …?"** of a server or service (it opens the dependency map with that box picked). "g" then a letter goes to a page (`g d` dashboard, `g a` alerts,
`g s` servers, `g m` maintenance, `g i` integrations, `g k` secrets, `g p` map,
`g w` wall display), "/" jumps to the page's table search, and "?" lists them
all, and "j" / "k" move down and up the rows of a table that opens a panel (Enter
opens the row you are on). Shortcuts for pages your role can't open are ignored,
and none fire while you are typing. On a table of Proxmox guests, Docker containers, Tailscale
devices, Uptime Kuma monitors, Gitea repositories or Semaphore templates, a
click (or Enter) on a row opens a panel with everything known about it, without
leaving the table.
**Map** (Reference → Map) has two views, *Connections* and *Dependencies* (see below). *Connections* draws how things connect: which Proxmox host each
linked server runs on, which DNS names point at it (an A or AAAA record holding
one of the server's own addresses — nothing is guessed from names), and which
Uptime Kuma monitors watch it. Hover or tab to a box to light up its
connections; each box links to its page. A monitor Uptime Kuma can't be read for
leaves a note rather than a silent gap. Both views have a **Lines** switch above the boxes: *Curved* (the default) or *Right-angled* — out from a box, across, up or down
and into the next, with the corners softly rounded, like a wiring diagram. It changes only how the lines are drawn, is remembered in this browser, and is shared by both views.
The Map's second view, **Dependencies**, draws what needs what with the foundations on the left (the servers and services that take part in
a dependency or a "runs on" link, each with its status). Pick a box and what would go down with it turns red and what it relies on blue,
with a sentence underneath — "If it goes down, 4 things go with it: …" — so the answer to "what breaks if I reboot this?" is there before you do it.
**Named wall displays** (Settings → Share links → *Wall displays*, administrators). One board for every screen is rarely right: the hall TV wants the overview and what needs attention, the office monitor the apps and the network. A display is a **name**, an **address of its own** (`/wall/hall-tv`, made from the name unless you give one — it stays put if you later correct the name, so a bookmark on a screen keeps working), **which pages it rotates through and in what order** (the overview, *Needs attention*, and a page for each group of integrations; none ticked is every page) and **how fast**. A page that has nothing to show just now (*Needs attention* with nothing wrong, a group with no integrations) is left out, and a display none of whose pages exist shows the overview. The signed-in wall opens one from its address and has a *Display* choice in its footer; the footer says which display it is; an unknown address says so and shows everything. How fast is decided by the address (`?rotate=45`) first, then the display, then what this browser last chose. A **wall share link** can open a display too (choose it when making the link): the public board then rotates through that display's pages at its speed; remove the display and the link goes on opening the whole board. Up to twenty displays; each change is in the audit log and the change log. **A night schedule** can dim or blank a display: from a time to a time on **the screen's own clock** (every day; an end earlier than the start runs past midnight), the board is either **dimmed** (the same picture at about a quarter of its brightness) or **blank** (black, with a faint note; the screen is no longer held awake, so a tablet or computer may go to sleep). A page cannot switch a TV off, so these are the two things it can do. By default **a critical alert brings the board back** whatever the hour (switch that off per display), and **a touch or a key press brings it back for two minutes**. It works the same for a share link that opens the display. A display only chooses among the board's pages — what the board shows (and, for a share link, how much: numbers only or with names) is unchanged.
**Wall display** (`/wall`, also on the dashboard and the user menu) is a
full-screen status board for a spare monitor or tablet: the overall state in
large type, servers online, alerts, maintenance windows and failing integrations — and, for what is connected, how many Hetzner servers and game servers are running, how many critical vulnerabilities and stuck downloads there are, and one line for each group of the menu (Infrastructure, Apps …) saying whether it is well —
the problems themselves, and each integration's status, refreshing every 30
seconds with the screen kept awake where the browser allows. Esc returns to the
app.
**Rotating the wall display.** A spare screen can show more than one board: choose **Rotate → every 15 / 30 / 60 / 120 s** in the footer (or open `/wall?rotate=45`; `?rotate=on` is 30 s, `?rotate=off` stops it) and the lower half of the board changes page by page while the status line, the clock and the figures at the top stay put. The pages are the **Overview** (as before), **Needs attention** (the open alerts and what is in maintenance, in full — only while there is something), and one page for each group of the menu that has integrations (Infrastructure, Apps …) listing them, the failing ones first. The footer says which page it is on and **Pause** / **Resume** holds one on screen; the left and right arrow keys step by hand. The choice is in the address, so a TV's bookmark carries it, and this browser remembers the last one chosen on the board itself; without either the wall stays on the overview, as it always was. A share link's wall rotates the same way (add `?rotate=30` to its address) when it shows names; one that shows numbers only has a single page and offers nothing to rotate.
**Alerts folded under their host.** A server can say which server it runs on (a
virtual machine on its Proxmox host): on its page, "Runs on" — and left alone it
is worked out when the server is linked to a Proxmox guest and exactly one
server is named after that guest's node. When the host stops reporting, the
guests' own "server offline" alerts are folded under the host's ("also takes down
anna, elsa") instead of being one alert each: they are held back like a silenced
alert, not counted, left out of notifications and osTicket tickets, and labelled
"caused by pve1" on the Alerts page. They are still recorded as down (for the
uptime ribbons and "Since you last looked"), and if the host comes back while a
guest is still down, that guest alerts then as new — an outage is never swallowed.
Links that loop back on themselves cover nothing, and setting one that would is
refused. "Nothing — it runs on its own" stops the guess for a server that isn't a
guest (a physical machine that happens to share a Proxmox node's name), so its
alerts always stand alone.
**Restore tests** (on the Backup coverage page). A backup nobody has restored is a hope, not a backup: when you try one, record it — which guest (or anything else: a
share, a database), the day, how it went (restored fine, partly, failed, or *no need to test this one*) and a note. Each guest's newest test sits beside its backups.
A **failed** test is always an alert (critical) until a newer one passes. Under Settings → Notifications → "A restore should have been tested within" you can also
ask to be reminded: a guest that has backups but no passing test within that many days (or ever) becomes a warning, and so does anything else you chose to track;
0, the default, leaves the reminder off. A guest marked *no need to test* never is. Operators record tests, administrators can remove a mistaken one; the history is
kept, `/metrics` has each subject's age and result, and everything is in the change log.
**The change log** (Records → Change log, operators and administrators) is one list of what has changed, newest first and grouped by day: agents updated,
settings changed (with what to what), maintenance started and ended, dependencies edited, hardware added, endpoint checks and restore tests, alerts raised and cleared —
each as a sentence. Narrow it by kind of change, by server, by how far back (a day to everything kept), or by a word; "Show older" goes on from where it stopped; CSV
export. A server's Activity card links to its slice of it. Sign-ins and people's own data exports aren't changes and are left out. The raw audit log is the other tab of the same page (Change log → *Raw audit log*).
**Dependencies beyond servers** (Reference → Map → *Dependencies*, which draws the graph with the editor under it). Say that Gitea needs the server it is installed on, or that a server
needs Proxmox to answer. **Internet** is on the list too, as something a server, a service or a network device can need (a VPS, Hetzner, the router): it is drawn on the map and answers "what depends on Internet?", but nothing watches it, so it is never down and holds no alert back. It is only ever needed — it can't be made to need anything. When what is needed is down, the alert of what needs it is folded under the first one exactly like a
guest under its host — shown, labelled "caused by …", left out of notifications and tickets — and the root cause says what else it
takes down. It follows chains and the "runs on" links of servers; something needing two things is explained by whichever is down;
a loop explains nothing and one that would be made is refused. An integration is judged per kind of service (as its health is),
so it is folded only when every enabled instance of that kind is explained. A service that starts failing while the server it needs
is offline isn't announced; if it is still failing once the server is back, the next failed call alerts. (DNS providers raise no
alerts of their own, so they aren't part of this.)
**Who hears what** (Settings → Notifications → Who hears what). *Routing rules* send a kind of notification, of at least a severity, about servers with
certain tags, to the channels you choose — the first matching rule decides, none matching means every channel that is on, and a rule with no channels tells
nobody. Each phone or browser can also say it only wants warnings or critical alerts, and only some kinds. *Escalation* tells again, through louder channels
and perhaps an extra email address, about an alert nobody has acknowledged after a time you set (up to three steps, each once; an acknowledgement stops the
rest). See NOTIFICATIONS.md.
**Seeing the impact before maintenance.** Choosing a server or integration under Maintenance → Start maintenance lists what depends on it and
will look down while it is — and offers to put those under maintenance too, for the same time, with the reason "Depends on …" (ticked by default;
each is its own window, listed and ended like any other, and audited). The recurring-schedule form shows the same note for what is picked and can add
the dependents to the selection. The "Update this server now" drawer shows what depends on the server and may look down while it restarts; it doesn't
silence them, because their windows would outlast the run — use Maintenance first if you want that.
**Recurring maintenance** (Maintenance → Recurring) repeats a window on the days
and clock time you choose — "Sundays 02:00–04:00 on these servers" — in the time
zone of the browser it was made in, so 02:00 stays 02:00 across summer and winter
time. Each time becomes an ordinary maintenance window a week ahead (shown on the
timeline with vertical stripes, and in Scheduled), so any single one can be
cancelled and stays cancelled. Turning a schedule off or removing it takes back
the windows that haven't begun. Edit changes a schedule's name, days, time, length
or what it covers: the windows that haven't begun are made again from the new
settings (a single one cancelled by hand comes back too), and one already under
way keeps its times and ends on its own.
**Running a playbook** from an alert or a server: a server's page has a "Run a
playbook" card, and alerts about a server have a "Run…" button, which pick a
Semaphore template and run it limited to that server (ansible's `--limit`, which
you can edit to match the host name in your inventory — or untick it to run on
everything, which asks more sternly). Semaphore is told who started it, the run
shows waiting → running → success or failed, and the audit log records what ran
and where. Operators and admins only.
**Updates** (Operations → Updates) shows what each agent says is waiting to be
installed — how many, how many are security fixes, whether a restart is due, and
how long updates have been waiting — plus the Docker containers with a newer image.
The Linux agent asks apt, dnf or zypper (only what they already know; nothing is
installed or refreshed — that one question is asked outside the agent's sandbox, which `dnf` and `zypper` need) and the Windows agent asks Windows Update (at most every six
hours, cached under `%ProgramData%\HomelabManager`). A server whose agent hasn't
said — an older agent, or another package manager — shows "not reported", never "up
to date"; run its install command again to update its agent (how, and how to
tell it worked: [AGENT_UPDATE.md](AGENT_UPDATE.md)).
**Acknowledging and handing over alerts.** On the Alerts page, **Acknowledge…** (operators) says
"somebody is on this": an optional note, who has it (an administrator or operator), and how long it should
be left alone (an hour to a week, or until the alert is gone). It changes nothing about the problem — the
alert is still raised, still counted by `/metrics` and still on the wall display — but the alert shows who
has it and why, the sidebar count, the dashboard strip and the Alerts cards count only what nobody is on,
and the list can be filtered to "nobody is on it" or "handed to me". An acknowledgement that runs out puts
the alert back among the unattended; it goes by itself once the alert has been gone for half an hour, so
the same problem returning next month is not "still being dealt with". It is in the audit log and in the
server's history.
**Incidents** (Alerts → **History**; the Alerts page has two views, *Now* and *History*, and an incident's own page is under it). Each time an alert is raised and later goes away is recorded as an
incident, with when it began and ended and how bad it got, so "what happened on Tuesday night?" has an answer
after the alert is gone. Open one for the whole story in order: the alert, **who was told** (each notification that was about it — a new alert, a recovery notice, a reminder, an escalation — through which channels, which refused and what they said, or why it was held for quiet hours or not sent because a routing rule says nobody), who acknowledged it and why, the
maintenance windows, what was changed on the server, update runs, other alerts about the same thing that
began around the same time, and the notes people wrote (operators can add them; the author or an
administrator can remove one). Incidents are recorded from the alerts as they are collected and from a look
every five minutes, so one that comes and goes between two looks may be missed. An alert is only called
resolved after it has been missing for a couple of minutes in a row — thirty when a check couldn't run, and
never in the first minutes after the app starts, when server alerts are held back on purpose. Resolved
incidents are kept for 400 days. The weekly digest has an *Incidents* section.
**How big the data is** (Settings → Data → Storage). The size of the database file, its backups and the signed-in sessions, and every table by the room it takes (with its indexes), its rows, how many it gains a day, how long it is
meant to be kept and how far back it actually reaches. A history that holds far more than its limit is flagged **not trimmed** (the trimming is not running), the logs are flagged **no limit** until purging is switched on
(Settings → Data → Logs), and a history that has not yet grown to its limit is *filling up*. Administrators only; it only measures.
**Duplicati** (Backups → Duplicati). If you run Duplicati, the app now tells you when a backup **failed** or **has not succeeded for longer than it should** — the usual way a backup tool goes wrong is silently. Duplicati reports to the app (its own `send-http-url` option): you add each machine, get a secret
address once, and paste three options into Duplicati's default options; no password for Duplicati is kept and the app never calls it. Every job then appears with when it last succeeded, how the last run went, how long it took, how big the source is and what changed, with a history; a failed run is a warning (critical from the
second in a row), an overdue one (2 days by default, per job) a warning (critical at three times that), and maintenance on the machine's server silences them. A run that ends with **warnings** still counts as a good backup, and the page shows **what the warnings said** (on the job and on each run in its history); a job can be set to **alert on warnings** too (off until you switch it on). See INTEGRATIONS.md and docs/MONITORING.md.
**Nextcloud** (Apps → Nextcloud). Add a Nextcloud integration (its address; an administrator's app password adds more) and the app tells you when people **cannot use it**: **critical** when its database needs upgrading, a warning when it is stuck in **maintenance mode**, and a warning (critical under 5 GB) when the **data disk is low**.
The page also shows the users, the files and the app updates waiting. Whether its **background jobs (cron) are running** is read from Nextcloud's own setup checks (Nextcloud 28 or later), and a warning is raised when they are not — a stopped cron would otherwise go unnoticed for weeks. See INTEGRATIONS.md and docs/MONITORING.md.
**History of integration figures.** The numbers an integration reports — AdGuard's and Technitium's queries and blocked share, the queue and missing items of Sonarr and Radarr, download speeds, Immich's photos and library size, Paperless's documents, Nextcloud's users and free space, n8n's failed runs, Grafana's firing rules, Prometheus's targets down, Authentik's failed sign-ins, Hetzner's monthly cost — are kept every quarter of an hour (and as hours and days beyond five weeks, up to about two years), and each integration's card has **Show the history**: a chart for each figure, 24 hours to a year, with how it changed and its lowest and highest. A stretch when the integration could not be read is a gap, not a flat line. See INTEGRATIONS.md ("History of the figures").
**Hetzner Cloud** (Infrastructure → Hetzner Cloud). Add a project's read-only API token and the app lists its servers with their status, this month's traffic against the quota, backups, volumes and snapshots, and roughly what it costs a month. It warns when a server that was running is **off**, when one has used **80% of its traffic** (critical at 100%), when **rescue mode** is left on (the next reboot would not come back up), and when **backups** are enabled but late. A server that is also one the app knows by its agent gets a Hetzner card on its own page. It also lists **load balancers** (with the health of their targets), **floating IPs** and **certificates**, warns when a target is unhealthy, a certificate is about to end or a floating IP points at nothing, and **forecasts the month's traffic** ("at this rate web will pass its quota on 24 June"). Nothing at Hetzner is ever changed. See INTEGRATIONS.md and docs/MONITORING.md.
**Hetzner Robot** (Infrastructure → Hetzner Robot). Hetzner's dedicated servers and storage boxes — Robot is the other Hetzner, next to the Cloud. Add a webservice user (Robot → Settings → Webservice and app settings) and its password and the app lists the account's dedicated servers (product, data centre, addresses, traffic plan, status, whether the rescue system is on, and what is paid until or when a cancelled one shuts down) and its storage boxes. It warns when **a server or storage box is cancelled and about to shut down** (within 30 days; critical within 7 — the data goes with it), when **a storage box is locked**, when **a rescue system has been left on** (two hours — the next boot is into it) and when **a server is still "in process"** a day after it was first seen so. Nothing is reset, ordered or cancelled. See INTEGRATIONS.md and docs/MONITORING.md.
**Game servers** (Infrastructure → Game servers, for Fragnet's game panel). Add an account's API key and the app lists its game servers: the game, the IP addresses and ports (click one to copy it), whether each is running, its memory and disk against the plan, its backups, schedules and databases. It warns when a server's **newest backup is late** (7 days by default, or none at all — a server can be left out by name) and when **disk or memory is nearly full** (90% by default); a server that is merely stopped is not an alert, since game servers are often stopped on purpose. Operators can **start, stop, restart and kill** a server from the page, and **back it up now** (asked first, written to the audit log) — the panel makes the backup in the background and the page shows it as being made, then done; nothing else at Fragnet is changed. Fragnet's panel is a fork of Pterodactyl, so the same integration reads any Pterodactyl panel (give its address). See INTEGRATIONS.md and docs/MONITORING.md.
**Media managers** (Apps → Media managers). Add Sonarr, Radarr, Prowlarr, Lidarr and Readarr with their address and API key and the app shows, for each, what the application says is wrong with itself, its download queue (what is stuck or failed, and why), the disks it fills, how many wanted items have no file and, for Prowlarr, which indexers have failed. It warns when an application reports a **warning or error** (a download client that cannot be reached, indexers switched off), when **downloads have been stuck for two hours**, and when a **disk is nearly full**. Nothing is searched, grabbed or changed. See INTEGRATIONS.md and docs/MONITORING.md.
**Download clients** (Apps → Download clients). Add qBittorrent (its Web UI address, and the user name and password if it asks for them) or SABnzbd (its address and API key) and the app shows, for each, how fast it is downloading, what is in its queue and in what state, how much room its download folder has, what failed in the last day and SABnzbd's own warnings. It warns when **downloads are stuck** (in an error state, or stalled with nothing moving for a day), when qBittorrent has **no connection to the network** for 15 minutes (a VPN that dropped), when SABnzbd's queue is **paused and forgotten** (six hours, with downloads waiting) or **downloads nothing** (an hour, with work waiting — a news server that does not answer), and when a **download disk is nearly full**. Nothing is added, paused or removed. See INTEGRATIONS.md and docs/MONITORING.md.
**AdGuard Home** (Network → AdGuard Home). Add its address (and sign-in, if it has one) and the app reads it every two minutes — and each time asks its DNS server a real question over UDP, because the web interface can be up while DNS is not. It shows whether DNS answers (and how fast), whether protection is on, the queries and how many were blocked, the names blocked most and the clients asking most, and its filter lists with when each was updated. It raises **DNS that has stopped answering** (ten minutes of it, critical), **a DNS server that is not running** (critical), **protection left off for an hour** and **filter lists that have not updated for a week**. Nothing is changed. See INTEGRATIONS.md and docs/MONITORING.md.
**Technitium DNS** (Network → Technitium DNS). Add the address of its web console and an API token (a non-expiring one, made under Administration → Sessions) and the app reads it every two minutes — and, like AdGuard Home, asks its DNS server a real question over UDP each time, because the console can be up while DNS is not. It shows whether DNS answers (and how fast), whether blocking is on, how long the server has run, the last day's queries and how they ended (from the cache, resolved, from its own zones, blocked, dropped, failed), the clients and names asked about most and blocked most, the cache against its limit, its block lists and when they are next updated, how its zones are (an expired or unsynced secondary zone is named) and its DHCP leases. It raises **DNS that has stopped answering** (ten minutes of it, critical), **a secondary zone that has expired** (critical) or **cannot sync** (warning), **blocking left off for an hour** and **block lists that are overdue for an update**. Nothing is changed. Technitium's API has no figures for the machine's CPU, memory or disk — a server agent on that machine reports those. Zones and records are connected separately, as a DNS provider. See INTEGRATIONS.md and docs/MONITORING.md.
**Grafana** (Monitoring → Grafana). Add its address and a service account token (the Viewer role is enough) and the app shows which alert rules are firing or pending, which cannot be evaluated, and which data sources fail Grafana's own health check. It raises **a rule that is firing** (critical when it is labelled critical or page), **a rule that cannot be evaluated** and **a data source that fails its check**. Nothing is silenced or changed. See INTEGRATIONS.md and docs/MONITORING.md.
**App versions** (Operations → App versions). The version each connected application runs — as its integration last read it — against the newest release of it, with how far behind it is (a patch, a minor update, a major version) and how long the newer release has been out. It looks the newest releases up on GitHub once a day, and **that is the only thing this app asks of an address that is not yours, so it is switched off until an administrator switches it on**. A number of days (0 = never) makes a newer release that has been out that long a warning. Sonarr, Radarr, Prowlarr, Lidarr, Readarr, AdGuard Home, Grafana, Jellyfin, Immich, Paperless-ngx, Nextcloud, NetBox, Authentik and Prometheus are compared. See docs/MONITORING.md.
**Read now** (Integrations → *Read now* on each integration, and on the card of each integration on its own page — Media managers, Download clients, Hetzner Robot, Hetzner Cloud, Game servers, AdGuard Home, Technitium DNS, Grafana and Speedtest Tracker; operators and administrators). Reads one integration at once instead of waiting for its next turn, and says how it went. An integration that is read on a schedule and stored (most of them) is read the way the schedule reads it, and what it says is judged again at once, so an alert that has cleared clears and a new one appears without waiting. One that is read live (Proxmox, Synology, Tailscale, Gitea, Semaphore, Uptime Kuma, Proxmox Backup Server, osTicket, phpIPAM, Nginx Proxy Manager) is asked whether it answers — the question *Test* asks — which also brings its Working/Failing mark up to date. The same integration is not read again within 20 seconds, since some systems limit how often they may be asked (Hetzner Robot allows 200 requests an hour). See INTEGRATIONS.md.
**Inspect an integration** (Integrations → *Inspect*, administrators). Runs the integration's reader once, now, and shows what each request asked and got back — the **shape** of every answer (the names and kinds of its fields, never the values) — beside what the reader made of it and the fields that came out empty. It is how an integration written from documentation is checked against the real system: an answer with a field the reader did not expect shows up as an empty field. *Copy the report* gives plain text that can be pasted into a message. Each inspection is written to the audit log; nothing is stored or changed.
**Grouping alerts.** On the Alerts page a *Group by* choice puts the alerts under a header for each source (Hetzner, Sonarr …), kind of problem (Disk, Backups …) or type of alert (a Hetzner certificate about to end …) — worst group first, each header showing how many and how serious, each group folding away, and a tick on the header picking everything in it for a bulk action. Remembered per browser; not grouped is the default.
**Speedtest** (Monitoring → Speedtest). Add a Speedtest Tracker integration (its address and an API token with the *results:read* ability) and the app charts how fast the internet has been — download, upload and ping over up to 90 days — and, if you tell it what the line is sold at, warns when **the middle of the last three tests is under 60%** of it, and when **the tests keep failing**. One slow test is not an alert. The app never starts a test. On a status page the integration reads as the internet. See INTEGRATIONS.md and docs/MONITORING.md.
**NetBox** (Monitoring → NetBox). Add a NetBox integration (its address and a read-only token): a warning when **no background worker** is running (they are a separate container in the Docker image, and can be down while the pages load), and a list on the page of **the servers you run that have no address in NetBox**. That list is
not an alert; documentation that is behind is not an outage. See INTEGRATIONS.md and docs/MONITORING.md.
**n8n** (Automation → n8n). Add an n8n integration (its address and an API key): a warning for each **workflow that is switched on and failed on its last run** — an automation that has quietly stopped — which clears when it next succeeds. The page lists the workflows with their last run and the failures of the last day. See
INTEGRATIONS.md and docs/MONITORING.md.
**Immich** (Apps → Immich). Add an Immich integration (its address and a read-only API key) and the app watches the thing you cannot recreate: it warns when the **disk under the photo library** is filling (critical at 95 %) and when **background jobs have failed** — the thumbnails and metadata that Immich
quietly never made. The page shows the photo and video counts, what the library takes and each job queue. See INTEGRATIONS.md and docs/MONITORING.md.
**Paperless-ngx** (Apps → Paperless). Add a Paperless-ngx integration (its address and a superuser's API token) and the app tells you when it has stopped taking documents in: **critical** when the database, Redis or the **worker that consumes documents** is in error, and a warning for **documents that failed to import**
(named, with the error), a failing index or sanity check, and a filling disk. The page shows the documents, the inbox and Paperless's own system status. See INTEGRATIONS.md and docs/MONITORING.md.
**Prometheus** (Monitoring → Prometheus). Add a Prometheus integration (just its address) and what you already measure there joins the one list of what is wrong: a **scrape target that stops answering** (five minutes, grouped by job — a node exporter on each host, say) and **alerts that Prometheus's own rules are firing**
(critical when labelled so). The page shows targets by job and what is down or firing. Alerts managed by Grafana itself are not seen. See INTEGRATIONS.md and docs/MONITORING.md.
**Authentik** (Apps → Authentik). Add an Authentik integration (its address and an API token) and the app watches the sign-in in front of everything else: it raises a **critical** alert when an **outpost** that guards applications stops reporting, and warns when a **certificate** or an **API token** is
about to run out (the token it reads Authentik with included), when **no worker** is connected, and when **sign-ins are failing** or flows are misconfigured. The page lists the outposts, what runs out soonest and the failed sign-ins of the last day. See INTEGRATIONS.md and docs/MONITORING.md.
An administrator also has a second tab there, **Users and groups**, and can **make a user** on it — with a second, optional token for it — and gets a **one-time recovery link** to send them, so they choose their own password; no password is ever set or seen here, and the link is shown once and not kept. The same page lists the users and lets an administrator **edit** one (name, email, groups), **switch one off** (and on again) or **delete** one (the user name has to be typed); service accounts, the account the token belongs to and the last active administrator are refused. **Groups** can be made (empty, and never as a group that makes administrators), renamed, deleted (the name has to be typed; never one that makes administrators) and have users put in and taken out (whether a group makes administrators is not changed here); a group that decides roles in this app is marked and asks first.
**Jellyfin** (Apps → Jellyfin). Add a Jellyfin integration (its address and an API key) and the app shows whether it is up, **who is streaming what** and whether it is being transcoded, the scheduled tasks that failed, and a chart of how many watched at once; it warns when Jellyfin **wants a restart**, a
**task failed** or **sign-ins are failing**. And when the Jellyfin runs on a server the app manages, **an update run asks first if people are watching**, and a patch plan **waits for them (up to two hours)** instead of restarting the machine in the middle of a film. See INTEGRATIONS.md and docs/MONITORING.md.
**Database backups** (Settings → Data → Backup). A copy of Homelab Manager's own database is taken every day (03:30
in the notification time zone, the newest 14 kept; on unless you turn it off), checked as it is made, and
kept in `data/backups/`. The page lists them with **Download**, **Check**, **Restore…** and **Delete**,
makes one now (**Back up now**), takes one from somewhere else (**Upload a backup…**) and sets the schedule.
A restore is checked first, then staged, and takes effect the next time the app starts — restart it
(`docker compose restart homelab-manager`); the database that was in use is kept as a "before a restore"
backup, so a restore can be undone. A backup made by a *newer* version than the one running is refused.
If the newest backup gets more than two days old, or a scheduled one fails, it is an alert (and a message).
**A backup does not contain `CREDENTIALS_ENCRYPTION_KEY`**, which decrypts the credentials stored in it,
and it is still on the same machine until you copy it away: see [BACKUP.md](BACKUP.md).
**Patch plans** (Updates → *Patch plans*). "Sundays at 02:00, the servers tagged `lab`, one at a time, stop at the first failure." An administrator makes a plan: which days
and what time (on the clock of a time zone, so 02:00 stays 02:00 in summer and winter), which servers (everything with any of the tags chosen, and any servers named), and the Semaphore
playbook that applies updates (and, if you like, a second one that restarts a server which says a restart is due once its updates are in — see below). When it is due — or an operator presses **Run now** — it goes down the list of servers by name, **one at a time**: each is updated by an ordinary update run
(below), and the plan waits for it to finish before it starts the next. A server with nothing waiting is left alone (unless the plan is told not to mind), and what **depends on** a server
(a guest that runs on it, a service that needs it) is silenced for the same time and let go when it is done. If a server's update fails and the plan says to stop — the default — the servers
after it are left untouched and the run says where it stopped; otherwise it carries on and says at the end that something failed. One message goes to your channels when a run ends, with a line for
each server. A plan starts once a day at most (the day it last started is kept), up to half an hour late if the app was down at its time — later than that, that day is skipped rather than
surprising anyone mid-morning — and never overlaps itself. A run that is going is followed through a restart of this app, and can be stopped (the servers not yet started are left alone; a playbook
already started keeps running in Semaphore). Plans appear in **Coming up** as maintenance, with how many servers they update, and the runs in the change log and the monthly report. Everyone can see
plans and how they went; administrators make, change and remove them; operators run and stop them.
**Update runs** (Updates → Update…, operators). Apply a server's waiting updates in
one go: pick the Semaphore playbook that does it, and Homelab Manager puts the server
under maintenance (so nothing alerts while it works and restarts), runs the playbook
limited to that one server, follows it, waits for the server's agent to report again,
and then says how many updates are still waiting and whether a restart is due before
ending the window it made. The runs are listed on the Updates page and in the server's
history, survive a restart of this app, and end as *failed* (with the window ended, so
the alerts can speak) if the playbook fails, Semaphore stops answering, the server
doesn't report again within 30 minutes, or it takes more than three hours. "Stop
following" ends the window and stops watching; it can't stop a playbook already running
in Semaphore. A message goes to your channels when a run ends (a toggle under *What to
send*).
**Restarting after updates.** The update drawer has a second choice, *Restart it afterwards, if it says a restart is due, with* — a Semaphore playbook that only restarts the
server (in the same Semaphore as the update's). When the updates are in and the server reports again saying a restart is due, that playbook is run for the same one server, the maintenance
window is kept up for it (at least 45 more minutes), and the server is **waited for again and checked**: it must have started again, which is read from the uptime its agent reports (its start is
the report's time minus its uptime; a start a couple of minutes before the restart began still counts) — or, where an agent reports no uptime, from its *restart needed* flag clearing. A playbook that
finished while the server did not restart is a *failed* run that says so ("check that the playbook really restarts it"); a restart playbook that fails, or can't be started, ends the run saying the
updates are in and the server still needs a restart. Without a second playbook nothing changes: the run says a restart is still needed, and an update playbook may restart the machine itself, as before.
A server that only needs a restart (nothing waiting, *restart needed* showing) has a **Restart…** button on the Updates page that does the restart on its own, with the same checks. A **patch plan** can
carry the restart playbook too: each server is updated and, if it then says so, restarted before the plan goes on to the next; a server with nothing waiting that still needs a restart is restarted instead
of left alone; the plan's maintenance time counts for the update and the restart together. Operators start them; the restart is in the server's history and the audit log, and the message at the end
says "the server was restarted and is back".
**Host health and automatic agent updates.** The Linux agent (version 4) also reports services that have
failed, each disk's SMART verdict, temperature and wear, and ZFS pool state. They show on a server's page
(*Host health*) and become alerts — a failed service is a warning under its own kind, a failing disk or a pool that isn't
ONLINE is critical — with the same maintenance silencing, notifications and runbook links as other alerts. Disk space
heading for full was already forecast from each server's history (alert, weekly digest, trend note); `/metrics`
now carries it too as `homelab_server_disk_days_to_full`. **Servers → Update agents automatically** (off by default) lets
an agent that is behind download the newer script from here, check it against a SHA-256 the server gives, and replace itself.
How it works and what it can't do: [AGENT_UPDATE.md](AGENT_UPDATE.md).
**Runbooks.** Write down what to do when something is wrong — "if elsa is down,
check the UPS first, then…" — on a server's page (the Runbooks card) or an
integration's card. A note can be for anything about the thing, or for one kind of
alert (a disk filling up). On the Alerts page an alert about a server or integration
shows a Runbook link when a note fits it, and opens it in a side panel with the notes
that fit that alert first. A note can also be written once for a whole **kind** of alert — the *What to do* link beside an alert, "when a Hetzner certificate is about to end: renew it in the console, then…", with an optional page to open — and then shows beside every alert of that kind, whatever it is about. Everyone signed in reads them; operators write them. The
text is plain: a blank line is a new paragraph, `- ` a list, ``` around lines a block
of commands, backticks inline code; links become clickable, and nothing typed in a
note can run as HTML.
**Endpoint checks** (Monitoring → Endpoints). The app asks a **web address**, a **TCP port** or a **TLS certificate** whether it is fine, every few minutes (each
check has its own interval, from 1 minute), and remembers how it went: the answer time, uptime over 24 hours and 7 days, a row of the recent results, and — for https
addresses and certificate checks — when the certificate runs out. A web check wants a good status (200–399 unless you say otherwise, like `200-204, 401`) and can
also want a word in the page, or **a value in a JSON answer within its limit** — `power.now` below 5000, `state` equal to playing, a Prometheus query's result above 0 — with an optional header (a token) sent with the request;
the value read is shown, charted under *History…*, and in `/metrics`; a certificate that isn't trusted (self-signed, a private authority) fails unless you tick *Accept a certificate that isn't trusted*, and
its expiry is watched either way. A check that has **failed several times in a row** (2 by default, so a blip isn't) is a *Monitoring* alert and one notification, with
another when it answers again; a certificate within its warning time (14 days by default; critical in the last three) is an *Expiry* alert. A check can be linked to a
server, so maintenance on it silences the check's alerts and the server's tags can route them. Everyone can look; administrators set checks up (each one makes this
server connect somewhere — never to itself or the cloud metadata address, though your own network is fine) and operators can ask *Check now*. They also appear on the
dashboard and in `/metrics`. Results are kept for two weeks, then as hours (four months) and days (years); "History…" on a check draws them.
**Suggested checks** (on the Endpoints page, administrators). From what the app already knows it offers checks you could add with a click: a **web port a server's agent
sees open** (80, 443, 8080, 8443 — when a server has both the plain and the secure one, the secure one, since it also reads the certificate), under the name DNS gives that
server if there is one (an A or AAAA record with one of its addresses, or an alias to it), otherwise its own hostname or address; and a **secret that already reads a
certificate**, since a check also hears when the service stops answering; and, with Dockhand connected, **the apps in your containers**, recognised from their images and checked on each one's own health address with what a good
answer looks like — Immich's ping must say `pong`, Nextcloud's `status.php` must not be in maintenance mode, Gitea's health must say `pass`, Grafana's database must be `ok`, Authentik's health endpoint must answer, and so on for about forty
apps (the ones with no open health endpoint get a plain "the page answers"). They are asked on the Docker host's address (the server the app knows by that name) at the port the container publishes. Each is linked to its server. Something a check already covers is never offered, **Not for me**
stops one being offered (and they can be brought back), and nothing is asked until you add it. Uptime Kuma's monitors are not offered: they are already being watched, and
the adapter only keeps a host and port for them, not the address.
**Heartbeats** (Monitoring → Heartbeats). "Did my 3 a.m. job actually run?" An administrator makes a heartbeat for a job — how often it should report, and how late is
still fine — and gets an address with a long random token in it. The job calls it when it finishes (`curl -fsS -m 10 -o /dev/null <address>` after the backup in cron);
the app writes that down and shows when the next call is due, the run times and the last runs as a row of ticks. A call that does **not come in time** is a *warning*
alert and one notification (the same bookkeeping as everything else: said once, again as cleared when the call comes); a job that calls `<address>/fail` — it ran and went
wrong — is a *critical* alert at once, with what it said. `/start` before the job lets the finishing call say how long it took; anything sent with a call
(`--data-raw "3.2 GB copied"`, or `?msg=`) is kept as a note, up to 200 characters. A heartbeat can be linked to the server the job runs on, so maintenance there silences
it and the server's tags route it. Everyone can look; only administrators see the addresses, make, change and remove heartbeats, or give one a **new address** if it got out.
The address needs no sign-in (it is a token, like an agent's), and all it can do is say "I'm alive": it can't read anything. Calls are limited per address, and an
unknown token and a wrong path are answered alike. They also appear on the dashboard and in `/metrics`; the last two weeks of calls are kept. The format is in [docs/HEARTBEATS-AND-CALENDAR.md](docs/HEARTBEATS-AND-CALENDAR.md).
**Coming up** (Operations → Coming up, and a card on the dashboard). Everything with a date ahead in one list, nearest first: secrets, domains and certificates that
expire (including the ones endpoint checks see), Tailscale keys that are expiring, hardware warranties and end of support, planned and **recurring** maintenance, restore
tests that are due (when a time limit is set), the day a **bill** falls (Costs), and — as one line — updates that have been waiting. What has already passed in the last 30 days is shown as past, under
*Needs attention now*. Narrow it by kind, server and how far ahead (up to a year); the filters are in the address, so a view can be linked. A schedule that repeats weekly is one line (its next time) unless
you ask for every one. An administrator can make a **calendar feed** (Settings → Share links → *A calendar feed*): an address for a phone's calendar to subscribe to, refreshed
every few hours, with each day as an all-day event and maintenance with its start and end. A feed made with *kinds only* says only "A secret expires", with no names, so it is fine on a shared
calendar; *with names* also gives the name and a link back. Like any share link the address is shown once and can be removed. Tailscale keys are asked for at most every
ten minutes, and only those the integration reports as expiring soon are listed.
**Proxmox backups** (Backups → Proxmox backups) is one page with two tabs: *Backup coverage* and
*Proxmox Backup Server* (the integration's datastores and verification; the old `/pbs` address goes to
that tab). **Backup coverage** answers "what isn't backed
up?" for every Proxmox guest: is a backup job covering it, when was it last backed up
(from the newest snapshot on a Proxmox Backup Server, or the last successful Proxmox
backup task that names the guest), and did that go well — Recent backup, Stale (older
than 1, 2, 3, 7 or 14 days, your choice), Failed, No backup, or Unknown (a job covers it
but nothing says when it last ran, typical when backups go somewhere other than a
Backup Server — never guessed at). Worst first, searchable, with CSV, and the weekly
digest lists what needs a look. A guest whose newest backup is older than **Settings →
Notifications → "A guest's backup is too old after"** (48 hours by default; 0 is off) is
also an alert of the Backups kind — a warning, critical at three times the limit — with a
runbook link like any other, sent once a day with the other backup messages (under the
Proxmox backup toggle) and counted in `/metrics` (`homelab_backup_guests`,
`homelab_backup_last_age_seconds`). A server that is a guest can have its own limit
(a weekly backup is fine at eight days). Failed backups already raise their own alerts —
the latest failing run on a node, and snapshots that fail verification — so those aren't
repeated per guest. The page's own "Stale after" selector is only a view.
**Alert limits per server.** On a server's page an administrator can give it limits of
its own — offline after, disk usage percentage, the disk-forecast window, and (for a
Proxmox guest) how old its backup may be — and per disk, either a limit or "no alerts
for this disk". Whatever is left empty follows Settings, so changing the global limit
still reaches every server that hasn't said otherwise. They apply everywhere the global
ones did: the alerts, the notifications, the dashboard, the map, the forecast, the
digest and `/metrics`. The same place keeps the **services to watch** on that server — only a service you list there raises an alert when it has failed, so the many that don't matter stay quiet; the list can also be set from its
Host health card (a failed service that isn't watched shows there with a **Watch** button; `*` watches them all); and a card, **What the agent couldn't collect**, tells you why a figure is missing (no `smartctl`, an update check that didn't answer …) and what
to do. See AGENT_UPDATE.md.
**Inventory** (Reference → Inventory) keeps the hardware behind it all: name and
kind, maker, model, serial, where it sits (and rack unit), when it was bought and for
how much, when its warranty ends, its typical power draw, and which server it is.
Summary cards show how much is under warranty (and what ends within 90 days), the total
draw, and — with a price per kWh set at the foot of the page — a rough yearly running
cost. Everyone signed in can look; operators keep it up to date; the weekly digest
mentions a warranty ending within 60 days, or support ending within 90.
**Lifecycle planning** (Inventory → Lifecycle). Give a piece of hardware an **end of support** (when the maker stops patching it, or when you plan to retire it) and a
**replacement cost**, and the Lifecycle page lists what ages out, year by year for the next 3, 5, 7 or 10 years, with what replacing each year's hardware is expected to
cost (the total says how many have no figure, so it is "at least" that). What is already past its date is listed first, and what is further off, or has no date, is kept
apart. Until a piece has an end of support, the end of its warranty stands in — and the page says so — and until it has a replacement cost, what it was bought for does.
CSV export; the weekly digest also mentions support ending within 90 days.
**Empty and loading lists.** A list with nothing in it says what is missing, why that is, and — when you can — the one thing to do about it (a button that opens the right form, or
a link to the page that fills it). A list that has things but not the ones you searched for says so, with a button that clears the search and filters. While a list is on its way
it shows grey placeholder lines rather than looking empty.
Entries that are a server also pick up what its agent says the machine is: when you link an entry to a server (or the
agent reports later), a blank maker, model or serial number is filled in from the firmware — never over what a person wrote.
When the agent says something different, the row offers **Refresh from agent**, which asks before replacing. **Print labels**
(on the Inventory page) makes a sheet of stickers — a QR code, the name, what it is, its serial number and where it sits, in
three sizes — and scanning a code opens that entry (`/inventory?asset=…`) for whoever is signed in. The codes are drawn in the
browser; nothing about them leaves the page.
**Roles from groups** (Settings → Sign-in, off until switched on): name the groups
that mean administrator and operator, and at each sign-in a person's role follows the
groups the identity provider (Authentik) reports; everyone else becomes a viewer, or
keeps what they have. "Check" shows what each person would become from the groups they
had last time, before you switch it on. Safeguards: a sign-in with no groups at all (a
wrong claim name) leaves roles alone, the only administrator is never demoted by a
group rule, the first person ever is still the administrator, and while it is on the
Users page shows roles instead of letting you set them. The Users page also shows the
groups each person had at their last sign-in.
**Profile pictures** (Settings → Sign-in, both sources off until switched on). Without one a person is their initials. An administrator can allow the picture the
identity provider names in its `picture` claim (kept at each sign-in; only an address on Authentik's own host or on Gravatar is ever used, so a claim can't make the
server fetch just anywhere) and/or **Gravatar** (by a SHA-256 hash of the person's email — never the address itself). The picture from the provider is tried first.
Either way it is **the server** that asks: it checks what comes back (a small PNG, JPEG, WebP or GIF — never SVG), keeps it for a day, and serves it from `/api/avatars/<id>`
to anyone signed in, so people's browsers never contact Gravatar or Authentik for it, and a person with no picture, or a host that can't be reached, simply shows initials.
The Privacy page says when it is on. The picture shows in the user menu and on the Users page.
**Prometheus and webhooks** (Settings → API and webhooks). `/metrics` answers in
Prometheus's text format — servers up or not, load, memory, disks, waiting updates,
agent versions, alert counts, which integrations work, maintenance, what is about to
expire — to a request with a metrics token (a bearer token made on that page, shown
once, only its hash kept). Event webhooks send a signed JSON message to a URL when an
alert begins or ends or maintenance starts or finishes, for Home Assistant, n8n or a
status board; **Sent** on a webhook lists the last 100 messages to it (none older than 14 days) with how each went and the message itself, and **Send again** sends one exactly as it was, same id and signature. Both are described, with the exact series and message formats, in
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) and [docs/WEBHOOKS.md](docs/WEBHOOKS.md).
**Service levels** (Alert settings → Service levels). Say how available something should be — **99.9 % for Gitea** — and see what is left of this month's **error budget**: the downtime the target allows (99.9 % of a 31-day
month allows about 45 minutes; 99 % about 7 hours). A target is set on an **endpoint check** (measured from its answers, day by day), a **server** or an **integration** (measured from when the audit log recorded it going away and
coming back, to the minute), and is measured over calendar months in the notification time zone. Each card shows the figure so far, the downtime used, what is allowed and what is left, how much of the budget is used (and where the
pace so far would end the month), and how each of the last six months went. When **80 %** of the budget is used it is a warning, and when it is overspent a critical alert, until the month turns over — switchable per target. Planned maintenance is
**not** taken out: down is down. A target with nothing recorded for the period says so rather than guessing. `/metrics` has the target, the figure and the minutes left; administrators set targets, everyone sees them.
**Monthly, quarterly and annual reports** (Records → Reports; and, if switched on, a message on a day of the month). How a calendar month went, on one page that can be read, printed and copied as text:
**availability** (servers, from when each stopped reporting and reported again; endpoints, from every check's answers, kept day by day for years; integrations), **incidents** (how many began and
ended, how long they took, the longest, by kind, and which are still open), **updates** (update runs and patch plan runs, how many updates were installed, by server), **restore tests** recorded in the
month and the **backups as they are now**, and what was **renewed** (a certificate or a domain whose expiry moved clearly later — noticed when the app re-reads the date). The month is on the clock of the notification time zone;
the current month is "so far". It can only say what was recorded: endpoint availability starts from the last two weeks of results when the daily figures were added, renewals from when the app began noticing them,
and the backups are a snapshot, not a history. The same page also gives a **quarter** (Q3 2026) or a **year** — the same figures over longer, with a **Service levels** section saying how each target did against the downtime it allowed over the whole stretch — chosen with the *Month · Quarter · Year* buttons. Settings → Notifications → *Monthly report* sends last month's report on a day (1–28) and time of your choice (off until switched on), with *Preview* and *Send now*; two more boxes add the **quarterly** report after the month that ends a quarter (March, June, September, December) and the **annual** one after December.
**Weekly digest** (Settings → Notifications, off until switched on) sends one message a
week through your notification channels: what is wrong now, what was down and for how
long, what happened, disks heading for full, things about to expire, updates waiting
and maintenance coming up — each topic only if it has something to say, and "a quiet
week" when nothing does. "Preview" shows what it would say right now and "Send it now"
sends it at once. Details in [NOTIFICATIONS.md](NOTIFICATIONS.md).
**Agent versions.** Each agent script says its version in every report, and Homelab
Manager compares it with the scripts it serves, so a server whose agent is behind is
marked on the Servers page, on its own page and on the Updates page (which counts them).
How to update one: [AGENT_UPDATE.md](AGENT_UPDATE.md).
**Customising the dashboard.** Customise (top of the dashboard) lets you move each
Overview card earlier or later among its neighbours and hide the ones you don't use —
hidden cards stay listed, dimmed, while you arrange so they can be brought back,
and Reset returns the default. The cards at the top (*Needs attention*, *My work*,
*Since you last looked*, *Getting started* and the *To do list*) can be hidden the same
way, each under a bar with its name and a Hide / Show button (they are not moved: the top
is two columns). Hide every card on one side of the top and the other takes the full width;
hide all of them and the row is gone and the Overview moves up. It is saved with your user,
so it follows you to any browser, and it is yours alone.
**Comparing servers.** On the Servers page, Compare lets you pick two to four
servers and open their load, memory or a disk on one chart (24 hours, 7 or 30
days), each line with its own colour and dash pattern and a table of now, average
and peak beside it. Servers that don't report through an agent have no history
to draw. The Servers page also has a search box and a "not reporting /
reporting" filter, and the dashboard links to the not-reporting ones.
**Share links** (Settings → Share links, administrators only) make an address
that opens a read-only wall display **without signing in**, for a TV or someone
who only wants to know if things are up. Each link has a name, how long it works
(until removed, 7 days … a year) and how much it shows: *numbers only* (servers
online, alert counts, what is in maintenance, integrations failing — never a
name) or *with names* (also the text of the alerts, what is in maintenance and
the integrations' names — never addresses, tokens or settings). The picture is
built on the server, so nothing else leaves it. The secret in the address is
shown once; only a hash is stored, a link that never existed, ended or was
removed answers the same way, and making and removing links are in the audit log
(without the secret). It is as private as the address: remove it when it isn't
needed.
**Costs** (Records → Costs). What the homelab costs, in one place: the domains, VPN, licences and subscriptions you type in (an amount per month or per year, a currency, a kind, and optionally the day it is billed) and a Hetzner project's estimate, read by itself. Each currency has its own monthly and yearly total and a split by kind — none is converted into another. A day a bill falls on is an anchor: the next time it falls (the same day next month, or next year) is worked out and shows in **Coming up** as *Renewals*. Everyone signed in sees it; an administrator changes it.
**Electricity** (the *Electricity* card on Records → Costs). The typical power draw you enter for each piece of hardware in the **Inventory**, the **price of a kWh** and the **currency** set under Inventory (Running cost) give an estimate: the draws added up, run all day every day (730 hours a month), at one price — shown as watts, kWh and a cost per month and per year, with **each device’s own cost and share, biggest first**, and how many pieces of hardware have no draw entered. When there is a draw, a price and a currency that is a **three-letter code** (SEK, EUR …) it is also **counted in the totals** above under its own *Electricity* category, listed with the other things that are read, and kept in the cost history; with a currency typed as “kr” it is worked out and shown but not counted (the totals are per ISO currency), and without a price the card says what to set. It is an estimate of what the hardware draws, not a reading from the meter.
**Cost history** (the *History* card at the bottom of Records → Costs). What the homelab cost **in each past month**, per currency (6 to 36 months back): a bar for each month — the items you typed in, and the darker part for Hetzner's estimate — and beneath it the figure for each month and how much it moved from the month before. A month's figure is the monthly amount of everything that was in force at the month's end (a yearly bill counts a twelfth, as the totals do; the month that is not over is as of now), so an item that was dearer last spring, or has since been removed, is in the months it was there. Under it, **what changed**: items added, changed (*what it was → what it is*, and a rename or a new kind) and removed, and the day a Hetzner estimate moved from one figure to another. It starts when the app began keeping it: an item that was there before is shown at its current amount from the day it was added (what it was earlier is not known), and Hetzner's estimate is written down once a day, so it is kept from the first day it was. Each item's earlier amounts are kept even after the item is removed. Everyone signed in sees it.
**Status pages** are the other thing a share link can open: a page for family or housemates
that answers "is the movie server up?" without a sign-in and without anything that isn't
theirs to see. Pick **A status page** when making a link, then choose what is on it and what
each thing is called ("Movies and TV", "Family photos"): each is a server (reporting, or not
— judged by its own limits), an integration (its latest calls work, or don't) or an Uptime
Kuma monitor. It says *Working*, *Having trouble* (it answers, but has an alert open that nobody has silenced, snoozed or closed — a disk filling, a stuck queue, an internet line slower than it is sold at; the alert itself is never shown), *Not working*, *Being worked on* (under a maintenance
window) or *Can't tell right now*, with the 30-day figure where one is known, a heading and
an optional message ("Movies are being updated tonight"). The page follows the device's
light or dark setting and looks again every 30 seconds. Whoever has the address is given only
the names you chose and those states — no addresses, no ids, no alert text, nothing about
what the thing really is — and a status page's address doesn't open the wall display (or the
other way round). Uptime Kuma is asked at most every 30 seconds however many people look. Up
to 30 things per page.
**A Grafana dashboard** for the `/metrics` series: Settings → API and webhooks → *Download
the dashboard*, then in Grafana Dashboards → New → Import and pick your Prometheus when
asked. It has rows for what needs attention, servers (with a server picker), disks (usage,
filling up, SMART health, SSD wear, temperature, ZFS pools), backups (guests by state, age
of each newest backup, the app's own backup) and secrets, power and the app itself. It is
built from the same code as `/metrics`, and a test checks that it only asks for series that
are really served. See [docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md).
**Since you last looked.** The dashboard opens with a card on what happened
since you last pressed "Got it" (the first time, the last day; never more than
two weeks): servers that stopped reporting or filled a disk, services that
stopped answering, and maintenance that began or ended — each problem and its
recovery folded into one line ("elsa was offline for 12 min — reporting again";
a problem that hasn't ended says "still offline"). Operators and admins also see
how many changes other people made, with a link to the audit log. Nothing to say
means no card. The mark is stored on your user, so it follows you to any browser.
**Uptime ribbons.** Servers (list, cards and the server's page) and integration
cards show a strip of one bar per day for the last 30 days — green for a day with
nothing wrong, amber for some time down, red for half a day or more, grey where
nothing is known — and the share of time up. It is worked out from the moments
the health checks write in the audit log (a server stopped reporting / reported
again; an integration's calls started failing — the same threshold as the failure
alert — / worked again), so it can only be as old as those records: servers go
back to when they were added, integrations to the first time this version ran.
Integrations are tracked per kind of service, as the health checks are.
**Maintenance timeline.** The Maintenance page draws the running and scheduled
windows along a time axis (the next 24 hours, 7 days or 30 days), one row per
server, integration or DNS provider and a top row for "anything in
maintenance", so overlaps and gaps are visible. Windows mirrored from Uptime
Kuma are striped. The tables below still list the same windows.
**Times as people say them.** "Last seen", sync times, session activity and
similar columns read "5 min ago" and show the exact date and time, in the format
chosen under Settings → Display, on hover. Exports and the audit log keep exact
times.
**Activity.** Operators and admins see an Activity card on each server's page:
what changed on it and who changed it, the moments the health check saw it stop
reporting (and report again) or a disk fill (and recover) — now written to the
audit log as they happen — and its maintenance windows, including ones set up by
the Uptime Kuma sync. **Getting started**: until the basics are in place, an
admin's dashboard shows a checklist (an integration, a server and agent, DNS, a
secret, notifications, a second user) whose ticks come from what actually
exists; it can be hidden, and brought back from the search box.
**Proxmox snapshots.** Click a guest on Infrastructure → Proxmox and its drawer has a **Snapshots** section: what there is (oldest first, a branch indented, *with memory* marked), **Take a snapshot** for operators (a name or one from the time, a description, and for a running VM whether to keep its memory), and **Roll back…** and **Delete…** for administrators, each behind a confirmation that says what is lost. Proxmox does each as a task, followed for up to 20 seconds; each is in the audit log and the change log. The API token needs VM.Snapshot and VM.Snapshot.Rollback (the PVEVMAdmin role has them). The guest actions (start, shut down, restart, stop) were there already.
**Small trends in lists.** The rows of three lists carry a small line for which way things are going: each **server's card** (Infrastructure → Servers) has its load and memory over the last 24 hours and its **fullest disk over the last week** (a disk moves too slowly to show in a day; the line says which mount it is, and turns orange from 80% and red from 90%), and the admin table behind *Manage servers* has the same three lines side by side in a *Load and disk* column (load and memory turn orange from 75% and red from 90%), each **endpoint check** has how long it took to answer over the last day, with a red tick under each stretch in which a check failed, and each **integration** that keeps a history (Integrations, as cards or table, with a *Last week* column) has the figure it is known for over the last week — DNS queries for AdGuard, the queue for Sonarr, photos for Immich, the monthly cost for Hetzner. A gap in the data is a gap in the line, the newest reading is a dot, and each line is a picture with a sentence for a screen reader (*CPU load, last 24 hours: 12% now, between 3% and 40%*). They are drawn from the history the app already keeps (nothing new is stored), come from one request for the whole list, refresh every five minutes and are simply left out where there is no history. A line fitted to its own data makes a small change look steep — the sentence and the numbers beside it say how big it really is. The full charts are one click away on each page.
**On a phone, under the thumb.** Below tablet width a **bar along the bottom** holds the places used most — Dashboard, Alerts (with how many are active, red when one is critical), Servers — and **Menu**, which opens the full menu; it keeps clear of the home bar and a page you have hidden is not offered. On the Alerts page an operator can **acknowledge with one tap** (*✓ I'm on it*: acknowledged until the alert is gone, no note) or **snooze for an hour** (*Snooze 1 h*), and **swipe a row**: to the right acknowledges, to the left snoozes — the row follows the finger, says what letting go will do, and does it only once it has been dragged far enough, and a drag that is mostly up or down (a scroll) never does anything. Each of these has an **Undo** in the toast that follows, nothing can be done only by swiping (the buttons are there too, and *Acknowledge…* and *Snooze…* open the full forms for a note, a person or another time), and an alert that is silenced, snoozed, closed or already taken is not offered the same again.
**On a phone.** Below desktop width the top bar (logo and menu button) stays at the top of the screen so the menu is always within reach, with a menu button of at least 44 px; an open menu scrolls by itself (it no longer pushes the page down), its rows are finger-sized and the favourite stars are always visible; text is a little larger on a small screen, toasts run across the width and drawers fill it. The page asks for the whole screen (`viewport-fit=cover`) and keeps clear of the notch, the rounded corners and the home bar, which matters when it is installed to the home screen (the status bar then sits over the app). Tables turn into stacked cards, form fields are 16 px so a phone does not zoom in on them, and what is pressed does not wait for a second tap. Checked at 375 px: no page scrolls sideways.
**Swedish interface.** Press *Svenska* under *Language* in the user menu and the whole screen is drawn again in Swedish at once; the choice is kept on your account and follows you to any browser (without a choice the browser's language decides, and that guess is not stored). Swedish covers the menu, the shared parts (search, tables, dialogs, "5 min ago"), the sign-in page, and the headings, buttons, table columns, filters, status words, empty states and short messages of every page — the integration pages and the settings included. Text with no Swedish yet is shown in English, so nothing looks broken. Not yet translated: what the server writes (alert messages, notifications, digests, reports, e-mail), the longer explanatory paragraphs and sentences with numbers or names built into them. How to add more is in docs/LANGUAGES.md.
**Accessibility.** There is a skip-to-content link, labelled landmarks, visible
keyboard focus, and page changes are announced to screen readers; sortable table
headings are real buttons; every form field is tied to its label; status is
never colour alone; the search box and dialogs are proper dialogs (keyboard
trapped and restored); and the text colours were checked against the contrast
minimum in both themes. Motion is turned down for those who ask the system to.
**New ports.** When an agent first reports what a server is listening on, that is taken as normal (the baseline). After that, a **TCP port that begins listening**, that is reachable from outside the machine, and that
nobody has noted is a warning on the Alerts page (kind *Monitoring*) and a notification, with what is listening on it: "nas: a new port is listening — 8080/tcp (node)". UDP (clients and discovery open them all the time), ports from 49152 up
(where programs pick a free one) and anything bound to localhost only are left out. The port's row on the server's **Ports** card says *New* and has **Mark as expected**, which notes it and ends the alert; so does a label or comment you write on
it yourself. **Network → Ports** shows the same across every server: a *New* badge on each such port, a notice that counts them (with **Show only new**), **Mark as expected** on a row and **Mark all as expected** (which asks first, naming the servers) — so there is no need to open each server. It also ends when the port stops listening, and maintenance on the server holds it back like its other alerts. It can be switched off under Settings → Notifications (*Servers and disks*); servers get their baseline from their next report after an upgrade, so nothing alerts for what was already there.
**Broken containers.** With Dockhand connected, the app now says when a **container is broken**, not only when an image has an update: one that **keeps restarting** (critical, with its exit code), one whose **own health check fails**, one that **stopped when nobody meant it to**, one **created and
never started**, and a **Docker host Dockhand can no longer reach**. A stop is only news when it was seen happen — a stack switched off weeks ago stays quiet — and an alert stays until the container runs again. The Docker page marks each problem and has a **Leave alone** button;
the same list is under Settings → Notifications (names, `host/name`, `*`). See docs/MONITORING.md.
**What is normal for the time.** For each server's CPU load and memory use the app learns what is usual for **the same hour of the same weekday** — "a Tuesday at 03:00" — from that server's own history (the median of that hour's
averages over up to ten weeks, with a robust spread; it needs **four weeks** of that hour before it says anything, and the hours are those of the notification time zone). A server's *History → Trends* then draws a grey
**usual** line with a band around it behind the real one, and says "usually about 8 % now". It is **quiet on purpose**: only a *high* reading is ever a warning, and only when the last hour averaged at least 50 % CPU (70 % memory), at least
30 points (15) above the usual and four spreads above it, and **never dropped back toward normal during that hour** — so a spike, a job that always runs a bit late, or a server that is simply always busy at that hour is not an alert. The
warning ("wiki-host: CPU load is unusually high for a Tuesday at 03:00 — it averaged 79 % over the last hour, where it is normally about 8 % (from 6 weeks of this hour)") is an ordinary *Monitoring* alert and notification: held back by
maintenance on the server, snoozed, flapping-damped and routed like the others, and left to the server's offline alert when it has stopped reporting. Switch it off under Settings → Notifications (*Servers and disks*).
**A year of history.** The figures behind the trend lines and the answer-time charts used to reach back two weeks or five. A server's load, memory and disks are still kept as a sample every ten minutes for 35 days, and now also as an hour's and a day's average, lowest and highest (hours for four
months, days for over two years); a check's answers are kept as hours for four months and, as before, days for years. A chart asks for 24 hours, 7, 30 or 90 days or a year and is drawn from whichever is fine enough. A disk that two weeks call flat is looked at again over ninety days, so a slow, steady climb is forecast too
(on the Alerts page, the weekly digest and in `/metrics`). The first start after the upgrade builds the hours and days from the samples already kept, once.
**One chart.** Server trends, comparing servers, a check's availability and answer times, and a server's availability all use the same chart and the same ranges: hover (or press the arrow keys once it has focus) to read the values at any moment, a shaded band shows the lowest and highest where a point
stands for an hour or a day, a gap in the data is a gap in the line, and the range you pick is remembered for the next chart. **Endpoints → History…** opens a check's charts; a server's **History** section has its availability (30 days to a year) above its trends.
**API tokens.** The user menu → **API tokens** (for everyone signed in): make a read-only token for a script, Home Assistant or a dashboard — shown once, 30, 90 or 365 days or no end, ten at most, removed at once on request and with the person — and use it with `GET /api/v1/…` for
servers, alerts, incidents, inventory, network equipment, endpoint checks and maintenance. The addresses, what each answers and an example are in [docs/API.md](docs/API.md).
**UniFi network equipment.** Add a UniFi controller under Integrations (an API key, or a read-only local account; see INTEGRATIONS.md) and **Network → Network devices** lists its switches, access points and
gateways: whether each is up (and since when it was not), IP and MAC address, uptime, firmware (with *update available*), the device it is connected through, how many things depend on it, and — on the *Networks & Wi-Fi* tab —
its networks (VLAN, address range) and Wi-Fi names (security, which network). It is read every five minutes and on **Read now**; nothing is ever written to the controller, and Wi-Fi passphrases are never read. The devices
join the **dependency map** (Map → Dependencies): what each is connected through becomes a dependency by itself, you can write others ("the NAS needs the rack switch"), and when a device is down, what needs it is folded under
its alert — "Hallway AP has been offline for 45 min — also takes down pi". A device offline for longer than a set time (Settings → Notifications, 10 minutes) is a critical *Server down* alert, can be silenced with maintenance
(also in the Maintenance page and the command palette), and can have runbooks. Not yet tried against a real controller — see INTEGRATIONS.md.
**Which switch a server is plugged into.** Every read also asks the controller what is connected, and a client whose IP address is exactly one of a server's own (and only one server's) is taken to be that server: the server is then
*connected through* that switch or access point (and port, where the controller says; Wi-Fi if it is on an access point). That is a dependency made by itself — when the switch is down, the server's offline alert is folded under
the switch's. It shows on the server's page (Network section), on the Network devices page (*Plugged in:* under each device), and on **Map → Connections**, where a switch or access point with servers on it is drawn next to them.
**IP Addresses → Sync from UniFi** adds the devices' addresses (label from the device name, the model and MAC in the notes); an entry made by hand or by another sync is left alone.
**My work** (the menu, for operators and administrators; also a line on the Dashboard when something wants a look). Everything with your name on it: the alerts you acknowledged or were handed (what runs out first at the top, with *Change…* to
edit the acknowledgement), the maintenance windows you started (*+1 h*, *+4 h*, *End now* with an Undo), and the update runs you started that are still going or finished in the last day. What runs out within a day (acknowledgements), ends within two
hours (maintenance), is handed to you or is still going is counted in the badge beside the menu entry.
**A server's page in sections.** Overview (links, hardware, identity, runbooks — fold a card by clicking its header, and *Arrange cards* puts them in the order you like, for every server), Health, Network (what it is connected through, ports), History (trends, activity), Tasks (scheduled tasks, run a playbook) and Settings (Proxmox link, its own alert limits),
switched by `?view=`, so a section can be linked to: an alert about a failing service opens the Health section, a new-port alert the Network section.
**The status strip.** Under a server's name, on every section of its page, a row of figures says how it is at a glance: **Status** (online, with when it last reported — or *offline*, with when it did, and a note when Proxmox says the guest is still running, so a stopped agent is not read as a dead machine; for a server with no agent report, what Proxmox says, or that nothing has reported), **Maintenance** (only while a window on this server is on, with when it ends and why), **Alerts** (how many are open about this server and how bad — what a maintenance window, a snooze or a closing has put aside is not counted, but said — and a link to the Alerts page), **Updates** (what is waiting, how many of them security fixes, whether a reboot is needed), **Disk** (the fullest, against its own limit: green, yellow within ten points of it, red at it; a mount with its own limit is judged by that, and an ignored mount is left out), **CPU · memory** and **how long it has run** (for a guest Proxmox reads, or from an agent of version 8 on Linux, 7 on Windows, which say it in every report; left out while the server is offline, since it is then no longer how long it has been up). A figure nothing is known for is left out or says so — it is never filled in. It uses what the page already has plus the alerts and the maintenance windows, read again whenever the page is (the *Refresh* button). Everyone signed in sees it.
**Accessibility.** The main pages were checked against WCAG 2.2 AA with axe-core in both themes and now have no violations: text in the status colours is a shade that reads on the light theme, the tabs, pills, switch buttons and info alerts read on the dark one, a link in a sentence is underlined, a link that is a whole line is at least 24 px tall, the controls that had no name have one, and the map’s nodes are named by what is written on them. What was found, what was done and how to run the check again are in docs/ACCESSIBILITY.md.
**Brand colours and logos.** Every kind of integration and DNS provider has its brand's colour and, where the open icon set [Simple Icons](https://simpleicons.org) (CC0) has one, its logo — Proxmox, Synology, Tailscale, Gitea, Jellyfin, Immich, Nextcloud, Grafana, Prometheus, Home Assistant, Sonarr, Radarr, qBittorrent, Hetzner, Cloudflare, Pi-hole and more. The kind **tags** on the Integrations page (cards and table), the Servers page and the DNS page are in that colour with the logo, made readable on the light and the dark theme alike (the lightness is held where the text reads, the hue kept; every tag is at least 4.9:1 on dark and 5.2:1 on light); the dashboard's integration cards have the logo beside their title; the kind picker has it on each choice; and **the menu** has a small icon before each page in its groups — the logo for an integration's page, the page's own icon for the rest — which **Settings → Display → Menu → Show an icon before each page** switches off for this browser. A few kinds have no open logo (NetBox, Prowlarr, Lidarr, Readarr, SABnzbd, Technitium, osTicket, phpIPAM, Fragnet, Dockhand, Semaphore, UPS): they keep their generic icon and an approximate colour. **Settings → Display → Badge colours** shows each tag as it looks now, lets you choose another colour for any of them, and **Use the brand's** goes back. Logos and names belong to their owners and are used only to tell the systems apart.
**Shorter page introductions.** The few lines under a page’s title that say what the page is are folded to two lines when they are longer than that, with **Show more** under them, so the page’s own content starts higher up. What you choose is remembered for that page in this browser; a short one is shown as it is, a link inside the folded part opens it when you tab to it, and a printout shows all of it.
**Notification history.** Records → Notifications lists every notification that was sent, with how it went through each channel (or why it was not sent: held for quiet hours, a routing rule said nobody, no channel switched on), the error a channel gave when it refused, a filter and a search, and **Send again**. Kept 30 days. See NOTIFICATIONS.md.
**A shorter menu for a smaller homelab.** The menu has a page for every kind of integration the app can read, whether or not you run it. **Settings → Display → Menu → Hide the pages of integrations I have not added** (off until you switch it on, the same for everyone) leaves out of the menu, the command palette and the `g`-shortcuts the pages whose integrations are not in **Administration → Integrations**: Gitea, n8n, Grafana … as the list under the switch says (a disabled integration still counts as added; any one of Sonarr, Radarr, Lidarr, Readarr or Prowlarr keeps *Media managers*, and a Proxmox or a Backup Server keeps *Proxmox backups*). Pages that are useful without an integration — Servers, Alerts, DNS, Endpoints, Secrets, Inventory and the rest — always stay, a group with nothing left disappears, and a hidden page is still there by its address. Adding an integration brings its page back at once.
**Away mode, and critical alerts through the night.** For a one-person homelab, "on call" is about *when you can be reached*. **Settings → Notifications → When → Away** is a one-off "I'm away until Sunday evening": notifications are held while it lasts
(a banner on every page says so and counts them) and sent as **one digest** when it ends — at the time you gave, or when you press *I'm back* — so a week away is one message to read, not a buzzing phone. **Critical** alerts (a server down, a disk full, mains
power lost) can still come through at once, which is the default; untick it for true silence. You can plan one ahead (start later) and add a note. **Quiet hours** got the same option, *Still send critical alerts*, off by default: a full disk at 03:00 is
sent, the warnings wait for the morning. Escalation steps marked *send even during quiet hours* are sent while away too. Held messages only: the alerts themselves are still on the Alerts page, in webhooks and in `/metrics`. See NOTIFICATIONS.md.
**Finding a setting, and undo instead of asking.** Settings has a *Find a setting* box (and the command palette lists every setting too) that goes straight to where it is set — it knows what each is for, so "quiet", "prometheus" or "disk full"
all find something. Removing a dependency, an admin link, a runbook, a hand-made IP address (one or several), a port opening or a restore test no longer asks first: it is gone at once, with an **Undo** on the message for fifteen seconds that makes it again from
what was on screen. Things that could not come back as they were — a secret, a check with its history, a share link or token, a DNS record, a hardware entry that has labels printed with its number — still ask first.
**Flapping alerts and the noise report.** An alert that is raised and clears again and again is more tiring than one that stays: each raise and each recovery is a message. When the same
alert has been **raised three times within an hour** (Settings → Notifications → *Servers and disks* → *Call an alert flapping after*; 0 turns it off) it is called **flapping**: one message says
so — "raised 3 times within an hour" — and its further raise and recovery messages are held back. The alert itself is still on the Alerts page, with a *flapping* badge, and keeps counting for the badge in the menu and for /metrics.
Once it has been calm (no raise) for an hour, one message says whether it is **still active** or **cleared**, and it is treated as an ordinary alert again. The webhooks that receive every event (Settings → API → event webhooks)
still hear every raise and recovery; only the human-facing messages are held. The state is kept in the database, so a restart does not forget what is flapping, and the start and end of each spell are in the change log.
**Alerts → Noise** shows which alerts keep coming back over the last 7, 30, 90 or 180 days, worked out from the incident record: how many times each was raised, how long it lasted in total, on average
and at the longest, the most times it came within a single hour, whether it flapped or is flapping now, and a line on what usually helps (a longer grace time, another threshold, a look at the thing itself).
Only alerts raised at least twice are listed; the list can be searched, sorted and exported as CSV. Everyone signed in can see it.
**Bulk actions.** On the lists where it is a chore to do the same to many, tick the rows (or *Pick all N shown*) and a bar appears at the foot of the page. Each action works on the whole selection and then says how many were
changed and how many were **left alone, and why** (a server already tagged, a loop that a *runs on* choice would make, a date that is not a date). Every one has an **Undo** button on the message that stays for fifteen seconds and puts back exactly
what was there before (the earlier acknowledgement text, the earlier tags, the earlier maintenance end times); an undo that cannot be done in full says so. One entry goes in the change log per action, not one per row, and at most 200 are done at once.
- **Alerts** (operators): *Acknowledge…* (a note, who has them, for how long), *Remove acknowledgement*, *Snooze…* (hide them all until a time you choose — an hour, tomorrow at 09:00, next Monday, or your own; each comes back once, if it is still there) and *Wake* (end the snoozes of those picked), *Close…* (one reason for all of them, written on each incident; each comes back only if it gets worse) and *Open again* (for closed ones), and *Silence…* (maintenance on what each alert is about, for 5 minutes to a week). Undoing a snooze puts back each alert's earlier snooze exactly; one that would already have run out is dropped rather than announced at once.
- **Servers** (operators; press **Select** beside *Compare*): *Tag…* (add a tag to, or take one off, all of them), *Silence…* (maintenance, optionally with what depends on them too), and *Runs on…* (administrators — which server they run on, or back to working it out from Proxmox).
- **Inventory** (operators): *Change where / dates…* (where it is, warranty until, end of support — each left as it is, set, or cleared) and *Print labels* for just those pieces.
**Operating-system support.** The agents now report which operating system a server runs (Linux agent 7, Windows agent 6; re-run the install one-liner to update). A server's page shows it in a *System* row with a
badge — *supported*, *ends in N days* or *support ended* — worked out from a table of end-of-life dates for Ubuntu, Debian, CentOS, RHEL and its rebuilds, Fedora, Alpine, openSUSE Leap, Proxmox VE and Windows
(client and server). A server whose system is past its end, or ends within **Settings → Notifications → *Flag an operating system whose support ends within*** (90 days; 0 turns it off) is on the Alerts page in the *Updates* category with a link to the server — a warning once
support has ended, a note while it is only coming. The dates are a static table in the app, not looked up: a release it does not know is shown without a badge, never guessed at. The read API gives it as `operatingSystem`.
**Suggested dependencies.** Under **Map → Dependencies**, a *Suggested* card offers what the app can work out without being told: a service (Gitea, Semaphore, Uptime Kuma, phpIPAM, Home Assistant…) that is reached at a
server's own address most likely runs there, so it needs that server. *Add* writes it as an ordinary dependency; *Not that* hides the suggestion (*Show the N you said no to* brings them back). Nothing is written until you add it.
**Notes and a to-do list.** Every server's page has a *To do* card (the open jobs about it, with a line to add another and a box to tick one off) and a *Notes* card, and every network device a *Notes* button (operators; its drawer shows the device's jobs too), for what is worth remembering: free text with the same small formatting as runbooks
(bold, lists, links). **Operations → To do** is a list of the small jobs — "replace the UPS battery", "move the NAS to the new switch" — each with a day it is due, a person it is handed to and a server or device it is
about. A job with a day shows in **Coming up**, a job handed to someone in their **My work** (and its badge), and ticking it off, editing and removing it go in the change log. Operators and administrators write,
everyone signed in reads. The **dashboard** shows every open job in a **To do** card down its right-hand side (beside the attention, since-you-last-looked and getting-started cards on a wide screen, under them on a narrow one): soonest due first, overdue ones in red, a box to tick one off (with an undo) and, for operators, a line to add another; it scrolls when the list is long and is read again every minute.
**Photos on a note.** A note can carry up to **12 photos** — the rack as it is, a cable's label, the back of the NAS: *Add photo* on the Notes card (from the camera on a phone, or from the files), a click to see one large (arrow keys move between them), a caption, and *Remove*. The browser makes each one
small first (1600 pixels on the long side, as a JPEG — a few hundred kilobytes; a small PNG or a GIF is sent as it is) and draws it again, which also leaves out where the photo was taken. The server takes only a **JPEG, PNG, GIF or WebP, told by the picture's own first bytes** (an SVG, which can carry script, is refused
whatever the upload calls itself), at most 4 MB, and serves it with `nosniff` and a policy that lets it run nothing. Photos are kept **in the database** (so a database backup has them and no extra volume is needed) and go with the server or device when it is removed. Operators add, caption and remove; everyone signed in sees them; adding
and removing are in the change log.
**Racks and a network diagram.** **Inventory → Racks** draws each rack from the hardware's *where* (the rack's name) and *rack unit* ("U12", or "U12-13" for a tall piece), from the bottom up, with the units used,
the power drawn and a piece that clashes with another in red; hardware that names a rack but no unit is listed beside it. It can be printed. **Network devices → Diagram** draws the UniFi gateway, switches and access
points as a tree by what each is connected through, with the servers plugged into each, a red box for a device that is down, and a *Print* button.
**Nginx Proxy Manager.** Add it under Integrations (a user that may only *view* proxy hosts and certificates is enough; see INTEGRATIONS.md) and **Network → Proxy hosts** lists each name it serves, where it forwards to
(and which server that is, when the address is one the app knows), its certificate and whether NPM could set the host up. A certificate in use that ends within 14 days (critical in the last three), or has ended, and a
host NPM itself says is not working, are alerts; a host that is switched off is left alone. The hosts it serves are also offered as endpoint checks (Endpoints → Suggested) on the name people really use, linked to the
server it forwards to. It only reads — unless you give the integration a second user that may *manage* proxy hosts: then an administrator can make, change, switch off and delete hosts from that page (only what differs is sent, a host's access list and custom configuration are never touched, deleting asks for the host's first name, and every change is in the audit log).
**Home Assistant.** Add it under Integrations with a long-lived access token (made at the bottom of your profile page in Home Assistant). **Apps → Home Assistant** is where you *choose* the few entities worth an
alert — a UPS battery, a freezer's temperature, a pump — not all of them. Each is read every five minutes; a number keeps a history (35 days, with a chart), and you can set a limit under and/or over which it is an alert.
An entity that has been *unavailable* (or has gone missing) for 15 minutes is an alert too. If Home Assistant cannot be read for twenty minutes its entities are neither raised nor cleared, and the Alerts page says it
could not check. Nothing is ever sent to Home Assistant. Neither integration has been tried against the real thing — see INTEGRATIONS.md.
**A UPS** (Monitoring → UPS). Add an integration of type *UPS*: the address of the machine running NUT (`upsd`), or of the UPS's network card with *SNMP* ticked (version 2c, the standard UPS-MIB). Every minute the app reads it — battery
charge, how long it would last, the load, the mains voltage — and the UPS page draws them over time and lists when mains power was lost. **Mains power lost** is a critical alert and a notification (so is a battery nearly empty;
a battery to be replaced and a load of 90 % are warnings). The UPS is also a **node in the dependency map**: say what it powers, and while it runs on battery it counts as *down*, so the servers that go offline with it are
folded under its alert ("also takes down …") instead of each being announced; the page lists what is on it in the **order to shut it down**, so nothing is switched off while something that needs it is still running. The app only
reads — it never sends a command to the UPS. Not tried against a real UPS; see INTEGRATIONS.md.
**Alert rules of your own.** **Alert settings → Alert rules** (administrators write them, everyone sees them) is for limits the built-in checks do not have: *CPU above 90% for 15 minutes*, *memory above 85%*,
*a disk above 80%* (every disk, or the one you name), *a disk warmer than 55 °C*. A rule says what to watch, above or below what, for how long (a reading in the history every 10–15 minutes; every reading in that time has
to be past the limit, so a dip starts the count again), how serious it is and which servers it covers (all, one, or those with a tag). **Try it now** shows which servers it would be raised on right now before it is saved.
What a rule finds is an ordinary alert on the Alerts page, in the *Monitoring* category with the rule's own severity — announced, recovered, held back while flapping, silenced by maintenance on the server and
routed by the notification rules like the rest. A server that is not reporting is not judged (its offline alert says so). Rules are judged **as soon as the server reports** — right after its report is stored, so one that has been past its limit long enough is raised in seconds rather than at the next 15-minute pass — and again on every pass. A temperature is only known as of
the disk's last report, so it is judged right now and has no time.
**Forecasts for Proxmox storage and Synology volumes.** The disk forecast used to work only for servers with an agent. Each health pass now also writes down the figures it has just read from Proxmox (each node's
storages and root filesystem, a shared storage once) and Synology (each volume) — one reading an hour, kept 400 days — and the same straight-line forecast is made from them: *full in about 40 days*. It is shown beside
each storage on the **Proxmox** page and each volume on the **Synology** page (click it for the history chart), is an alert in the *Disk* category when it is within the forecast window
(Settings → Notifications → *Warn when a disk will fill within*, 14 days — the same setting as for server disks) and not yet over the usual "disk nearly full" limit, is a line in the weekly digest, and is
`homelab_storage_days_to_full` in /metrics. It needs a few days of readings and growth that is really steady; until then the page says it is collecting history.
**Snooze.** An alert on the Alerts page has **Snooze…** (operators): an hour, four hours, tomorrow at 09:00, in three days, next Monday, or a time of your own, up to 90 days. A snoozed alert is hidden like one under
maintenance — it is not counted in the menu badge, the dashboard or the digest, and the raise and recovery messages of the health and automation checks for it are not sent — but it is not maintenance: nothing is
promised about the thing. When the time comes an alert that is **still there is announced again, once** ("Snooze ended — an alert you snoozed is still there"); one that went away is simply gone. *Wake now* ends it early.
A snooze, like an acknowledgement, is dropped once its alert has been gone for half an hour. The summaries that other checks send (stale backups, expiry, updates) are not held, only what is announced per alert.
**Close.** An alert also has **Close…** (operators), for one that has been **dealt with or is accepted** — fixed outside the app, or a warning you have decided to live with. It is hidden exactly like a snoozed one (not counted, its messages not sent) but **with no end**: it comes back only if it **gets worse** (a more serious
severity than it had when you closed it — "Closed alert got worse" is said once), or if the problem goes away for half an hour and returns, which makes it a new one. A number in its message changing (a disk at 91 %, then 92 %) does not open it. The reason you give is **written on its incident**, so the history says why, and
it shows beside the alert (*closed by Olga*; hover for the note). *Open again* brings it back at once. Closing is not maintenance: nothing is promised about the thing, use *Silence* for planned work. Closing, and opening again, are in the change log and the audit log.
**What changed before this?** Every incident page has a card with that name: what the app knows happened in the time before the alert began (30 minutes, 2, 6 or 24 hours), **ranked by how
likely each is to be connected**. It looks through the audit log (settings, DNS, dependencies, a server being changed or going down), maintenance windows that began or ended, update runs and patch plans, other
alerts raised first, and a step in the server's own figures (a disk that jumped from 50% to 73%). Each is scored from how *related* it is (the same thing; something it needs, such as the host it runs on or the
switch it is plugged into; something that needs it; a change that touches everything), what *kind* it is (a maintenance window ending or a host going down is likelier than a runbook being edited) and how
*close* it was. They are shown as *Likely*, *Possible* or *Background* (the last behind a button). It is a list of suspects, not a diagnosis: the app can say what happened and when, not why, and it cannot see a
change made on the machine itself. Everyone signed in can see it; it is also `GET /api/v1/incidents/:id/before` in the read API.
**Buttons on notifications.** With **Settings → Notifications → Who hears what → Buttons on notifications** switched on (off by default), a notification can be dealt with from the message itself, without opening the app
or signing in: **Acknowledge**, **Silence for an hour** (maintenance on what the alert is about) and **Snooze until tomorrow at 09:00** (in the notification time zone). A message about **one** alert gets them as
buttons — three on an ntfy message, two on a phone push (the browser's own notification), one button on a Telegram message and one link on a Pushover message, with the rest on the page a tap opens — and a message about several gets one link to a page that lists each with its buttons;
Gotify, email and the chat webhook get that link as text (and Gotify's tap opens it). Escalation messages carry them too. Each link is a **signed token for the alerts in that one message only**, good for **24 hours**, and it
can do just those three things — it cannot read anything or change a setting. **Opening a link never does anything** (so a preview or a mail scanner is harmless); only pressing a button acts, and it is written to the audit
log as *from a notification link*, with the address it came from. The price is that **anyone who can read the notification can press its buttons** for those hours: fine for a private ntfy topic, your own Gotify or inbox, not for
a public ntfy.sh topic. Changing `SESSION_SECRET` or switching the setting off ends every link. A message held for quiet hours goes out without them.
**Ports** — each server's detail page has a Ports card for finding free ports
and remembering what each one is for. "Scan…" runs a TCP scan of a port range
from the app against the server's address (private addresses only, up to
20,000 ports at a time) and lists what's open plus the ranges that were
confirmed free; a free range can be clicked to reserve a port. Any port can
carry a service name and a comment, and a port with a note counts as taken
even when nothing is listening. The agent also reports what is bound on the
host (`ss`), which catches services listening on localhost only — a scan from
elsewhere can't see those, so they'd otherwise look free. Re-run the agent
install one-liner on a host to pick that up.
**Ports named by their containers.** When a server's Docker host is in Dockhand (the Docker integration), the ports its running containers publish are named after the container — 8096 → *jellyfin* — on the server's Ports card, in the agent table on the Network → Ports page (hover for the image) and in the new-port alert ("a new port is listening — 8096/tcp jellyfin (container)"). It is automatic: nothing to set up beyond the Docker host in Dockhand having the same name as the server (the short name, the hostname or an address — the match Uptime Kuma targets use), and it is only a name beside the port. A label you wrote wins; a container's name wins over a port preset's; it is never written onto the port notes, so a published port is still a new port to be noted, and a container that starts publishing something is still news — now with its name. It is read on the health pass from the container list Dockhand already gives (every 15 minutes, and at start-up, whether or not the container alerts are on); a Docker host Dockhand can't reach keeps its names, and names not refreshed for a day are dropped. Containers on the host network publish nothing Docker lists, so their ports aren't named.
**Port presets** (Network → Ports → the *Presets* tab). A preset is a named list of ports written once — *Remote access* (SSH, RDP, VNC, WinRM), *Databases* (MySQL, PostgreSQL, Redis, MongoDB …), *Web*, *File sharing*, *Homelab tools* come ready-made, and are ordinary presets you can change or delete. Add your own by typing one port to a line (`3389 RDP`, `53/udp DNS`). A preset does three things: it **names ports** — a port in any preset shows its name (3389 → RDP) on every server's Ports card, in the agent table and in the new-port alert ("a new port is listening — 3389/tcp RDP"), without a label typed for each host, and a name you write yourself always wins; it **checks hosts** — pick any number of servers and see in one table which of the preset's TCP ports are open on each, from a scan from the app (private addresses only, at the server's first known address; what it finds is remembered like any scan); and it **names the open ports on all of them** in one go, keeping names somebody already wrote unless you say to replace them. Note that a port with a name counts as noted, so writing names also quiets the new-port alert for those ports. A scan only looks at TCP; a UDP port in a preset is only given its name. Operators change presets; everyone sees them.
**Network → Ports** is the cross-server counterpart: one page listing every
port every agent currently reports as listening, across all servers at once
(protocol, address, process, last report time), each linking back to its
server. Below that, a separate, manually-maintained table is for the ports
this app can't see on its own — a router's port forward, an edge firewall
rule, a cloud security group — the same reason people keep a spreadsheet of
"what did I open and why." Each entry has a label, the external port/protocol,
an optional link to a tracked server (plus its own internal port, when NAT
changes it) or a freeform destination, a free-text "source" (which
router/firewall/service it's actually configured on — this app doesn't talk
to any firewall, so it can't manage or verify the rule, only record it), and
a comment. Viewers can see both tables; adding, editing, or deleting a manual
entry needs operator or admin.
The app is installable as a PWA — "Install app" / "Add to Home Screen" from the
browser gives it its own icon and a standalone window on phone or desktop. This
needs the site to be served over HTTPS (browsers only offer install on secure
origins; `localhost` also counts). The bundled service worker deliberately
caches nothing, so an installed copy always shows the current build.
**More documentation** — the guides are in [docs/](docs/README.md):
[docs/SSO.md](docs/SSO.md) — single sign-on with Authentik, what the setup needs, roles from groups, troubleshooting;
[docs/API.md](docs/API.md) — the read API (`/api/v1`) and the other machine-facing addresses;
[docs/PROMETHEUS-GRAFANA.md](docs/PROMETHEUS-GRAFANA.md) — `/metrics`, Prometheus, the Grafana dashboard and every series;
[docs/WEBHOOKS.md](docs/WEBHOOKS.md) — event webhooks (signed JSON) and the notification webhook;
[docs/HEARTBEATS-AND-CALENDAR.md](docs/HEARTBEATS-AND-CALENDAR.md) — cron-job heartbeats and the calendar feed;
[docs/MONITORING.md](docs/MONITORING.md) — how monitoring works, from a reading to an alert.
The references: [ROLES.md](ROLES.md) — who can see and do what;
[NOTIFICATIONS.md](NOTIFICATIONS.md) — every notification the app sends and when;
[INTEGRATIONS.md](INTEGRATIONS.md) — the credentials each integration needs;
[DATABASE.md](DATABASE.md) — the database tables, columns and relationships;
[BACKUP.md](BACKUP.md) — what to back up, how (also while it runs), how to restore, and how to check it.
## Requirements
- Node.js 20+
- An Authentik instance reachable from wherever this app runs
## 1. Set up an Authentik application
1. Create an **OAuth2/OpenID Provider**:
- Redirect URI: `<APP_BASE_URL>/auth/callback`
- Scopes: `openid`, `email`, `profile`
2. Create an **Application** using that provider, and assign the users/groups
who should be able to sign in — Authentik controls who can authenticate;
the app's own admin/operator/viewer roles control what they can do once in.
3. Copy the provider's issuer URL, client ID, and client secret into `.env`.
## 2. Local development
_Tests:_ `npm test --workspace server` runs the server's regression tests (Node's built-in runner, no extra setup) — among them a check that an ID in a
request can't be turned into a different call to a connected system (Tailscale, Proxmox, Gitea, Dockhand, Cloudflare, Azure).
```bash
cp .env.example .env # fill in AUTHENTIK_*, SESSION_SECRET, CREDENTIALS_ENCRYPTION_KEY
npm install
npm run dev:server # http://localhost:3000 (API)
npm run dev:web # http://localhost:5173 (Vite dev server, proxies /api and /auth to :3000)
```
Visit `http://localhost:5173` during development. Database migrations run
automatically on server start. SQLite data lands in `./data` (gitignored).
Generate `SESSION_SECRET` and `CREDENTIALS_ENCRYPTION_KEY` with:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
## 3. Run with Docker
```bash
cp .env.example .env
# edit .env
docker compose -f docker-compose.dev.yml up -d --build # build locally
# or, once an image is published to your registry:
docker compose up -d
```
The app listens on `HOST_PORT` (default `3000`); SQLite data persists in
`./data` on the host.
**Telling a dev copy from production:** set `ENVIRONMENT_NAME` (e.g. `Dev`) and
optionally `ENVIRONMENT_COLOR` in that instance's `.env` (or under `environment:`
in its compose file). The sidebar then takes the colour, a badge with the name
sits next to the title and on the sign-in page, and the browser tab reads
`[Dev] Homelab Manager`. The colour is a name (`red`, `orange`, `amber`, `green`,
`teal`, `blue`, `purple`, `pink`, `gray`) or a hex value such as `#b45309`; a
name alone gets orange. Leave both unset on production and nothing changes.
Restart the container after changing them.
### arm64 (Raspberry Pi, Apple silicon, ARM servers)
Two compose files, one to build the image and one to run the published one:
```bash
# On any machine with Docker: build the arm64 image and push it to the registry
docker compose -f docker-compose.arm64.build.yml build
docker compose -f docker-compose.arm64.build.yml push
# On the arm64 machine: pull that image and run it (no build there)
docker compose -f docker-compose.arm64.yml pull
docker compose -f docker-compose.arm64.yml up -d
```
`docker-compose.arm64.build.yml` can also build and run right on an arm64 machine
(`up -d --build`). Building on x86 goes through QEMU emulation — slower, and it needs
a one-time `docker run --privileged --rm tonistiigi/binfmt --install arm64`.
Both files use the image `gitea.labsconnect.se/bobban/homelabmanager-homelab-manager:arm64`.
Set `IMAGE_REPO` and/or `ARM64_TAG` in `.env` to use another registry or to pin a release
tag (e.g. `ARM64_TAG=1.4.0-arm64`). The arm64 tag is kept separate from the default image,
so the existing `docker-compose.yml` is unaffected. A `.dockerignore` keeps host
`node_modules`, `.env` and `data/` out of the build.