docs: 2026-05-28 ops coverage audit + WAL-G monitor fix

- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
  cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
  Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
  plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
  timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
  to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
  SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
  noted litellm decommissioned but DB retained).
This commit is contained in:
2026-05-28 21:20:35 +02:00
parent b26796e30c
commit 9cbd02e3ff
2 changed files with 36 additions and 1 deletions
+5 -1
View File
@@ -62,7 +62,9 @@ Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmi
- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
- Daily `wal-g backup-push` cron at 02:00
- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder`
- DBs hosted: `vaultwarden`, `paperless`, `litellm` (decommissioned but DB retained), `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux`
- `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy
- WAL-G monitor (CT 103) checks this via `pg_stat_archiver`
Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup``jottacloud:WAL-G/`).
@@ -194,6 +196,8 @@ Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell re
> **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running.
> **Initial false-positive (2026-05-28 16:00)**: First scheduled run alerted "CT 113 query failed". Cause: CT 103 had no SSH access to CT 113 (host key + authorized_keys never deployed); `2>/dev/null` in the script hid the underlying `Host key verification failed` error. Fix: deployed CT 103 pub key to CT 113 + `ssh-keyscan`; added retry loop (2 attempts, 5s gap, 10s ConnectTimeout) for transient blips.
**Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`.
### Applying a new config