docs: 2026-05-28 ops coverage audit + WAL-G monitor fix

- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
  cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
  Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
  plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
  timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
  to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
  SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
  noted litellm decommissioned but DB retained).

services/backrest.md

@@ -62,7 +62,9 @@ Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmi
 
 - `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
 - Daily `wal-g backup-push` cron at 02:00
-- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder`
+- DBs hosted: `vaultwarden`, `paperless`, `litellm` (decommissioned but DB retained), `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux`
+- `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy
+- WAL-G monitor (CT 103) checks this via `pg_stat_archiver`
 
 Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`).
 
@@ -194,6 +196,8 @@ Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell re
 
 > **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running.
 
+> **Initial false-positive (2026-05-28 16:00)**: First scheduled run alerted "CT 113 query failed". Cause: CT 103 had no SSH access to CT 113 (host key + authorized_keys never deployed); `2>/dev/null` in the script hid the underlying `Host key verification failed` error. Fix: deployed CT 103 pub key to CT 113 + `ssh-keyscan`; added retry loop (2 attempts, 5s gap, 10s ConnectTimeout) for transient blips.
+
 **Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`.
 
 ### Applying a new config

services/homelab-architecture.md

@@ -308,6 +308,37 @@ Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
   kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
 - Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.
 
+## Ops coverage matrix (verified 2026-05-28)
+
+| Capability | Coverage |
+|---|---|
+| **Log aggregation** (Loki @ CT 109) | All 8 hosts shipping via Alloy: CT 101, 102, 103 (newly enabled today), 104, 105, 109, 113, nuc. Last 5m: 3.6k lines from CT 104, 2.1k from CT 113, 100s on quieter hosts. |
+| **Metrics** (Prometheus @ CT 109) | Scrapes node-exporter on CT 104/109, pve-exporter, traefik, home-assistant prom-endpoint, WAL-G textfile collector |
+| **Backups** (Backrest @ CT 103 → JottaCloud) | 14 services paths + HA tars + Immich tars + Nextcloud Borg + paperless media + Garage S3 + n8n local data. All 16 plan paths verified present. 3 pre-stage hooks (pocketid, garage, n8n). Success/error hooks → Gotify on all 3 plans. |
+| **Postgres archiving** (WAL-G → Garage S3 → JottaCloud) | CT 113 main pg17 (9 DBs incl. owui/miniflux) ✓, CT 104 immich_postgres ✓ (fixed today). Hourly staleness monitor on CT 103 with SSH+psql probe. |
+| **Image-update notifications** (Diun @ CT 109) | Watches CT 104 docker (67 images), Mondays 08:00 → Gotify. CT 109/113/101 own containers not watched (future work). |
+| **Config-to-git** (Phase 3) | Per-host scripts: CT 102 AdGuard, CT 103 Backrest. Central `sync-config-repos` on PVE 03:15 covers CT 103/104/109/113 stacks (auto-discovers CT 104+109 stacks). HAOS gap: Git Pull addon needs UI install. |
+| **Offsite personal data** (Phase 1a rclone sync) | Saturdays 01:00 on CT 103. Immich originals → JottaCloud Photos; UNAS personal → JottaCloud UNAS. Last run 2026-05-27 13h32m / 160k files. |
+| **Gotify apps** | Diun (#1), Backrest (#2), WAL-G monitor (#3), Gitea digest (#4) |
+| **systemd timers active** | CT 103: walg-staleness (hourly). nuc: gitea-digest (08:00), vm100-watchdog (1m), unas-nfs-watchdog (15m), protect-haos-kvm (5m), homelab-health (1h). |
+| **Cron jobs active** | nuc 03:15 sync-config-repos. CT 103 Sat 01:00 unas-sync, 02:30 walg-offsite-sync, 03:00 ct103-conf-backup. CT 102 daily adguard-conf-backup. |
+
+### Recent ops debt cleared (2026-05-28)
+- Vaultwarden data migrated CT 104 local → UNAS (was unrecoverable on disk loss)
+- Backrest false-positive: CT 103 → CT 113 SSH keys never deployed; fixed + retry added to monitor
+- Immich WAL-G push broken for 42h (garage container on wrong Docker network post-incident) — fixed
+- Diun, Backrest hooks, WAL-G monitor, Gitea digest deployed
+- Alloy enabled on CT 103
+- Stale CT 110/111/112 references removed from `sync-config-repos`
+
+### Still open
+- HAOS Git Pull addon (UI-only install)
+- Vaultwarden Borg passphrase storage (UI-only — Vaultwarden web UI)
+- n8n-flows issues #1, #3, #4, #7, #8, #9, #10
+- Diun second instance for CT 109/113/101 own containers
+- DAVx5 → HonorOS `org.tasks` battery-whitelist (user phone, not server)
+- Shared-password rotation (`tapirnase` reuse — deferred)
+
 ## Open / roadmap
 mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
 expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers