docs: 2026-05-28 ops coverage audit + WAL-G monitor fix

- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
  cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
  Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
  plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
  timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
  to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
  SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
  noted litellm decommissioned but DB retained).
This commit is contained in:
2026-05-28 21:20:35 +02:00
parent b26796e30c
commit 9cbd02e3ff
2 changed files with 36 additions and 1 deletions
+31
View File
@@ -308,6 +308,37 @@ Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
- Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.
## Ops coverage matrix (verified 2026-05-28)
| Capability | Coverage |
|---|---|
| **Log aggregation** (Loki @ CT 109) | All 8 hosts shipping via Alloy: CT 101, 102, 103 (newly enabled today), 104, 105, 109, 113, nuc. Last 5m: 3.6k lines from CT 104, 2.1k from CT 113, 100s on quieter hosts. |
| **Metrics** (Prometheus @ CT 109) | Scrapes node-exporter on CT 104/109, pve-exporter, traefik, home-assistant prom-endpoint, WAL-G textfile collector |
| **Backups** (Backrest @ CT 103 → JottaCloud) | 14 services paths + HA tars + Immich tars + Nextcloud Borg + paperless media + Garage S3 + n8n local data. All 16 plan paths verified present. 3 pre-stage hooks (pocketid, garage, n8n). Success/error hooks → Gotify on all 3 plans. |
| **Postgres archiving** (WAL-G → Garage S3 → JottaCloud) | CT 113 main pg17 (9 DBs incl. owui/miniflux) ✓, CT 104 immich_postgres ✓ (fixed today). Hourly staleness monitor on CT 103 with SSH+psql probe. |
| **Image-update notifications** (Diun @ CT 109) | Watches CT 104 docker (67 images), Mondays 08:00 → Gotify. CT 109/113/101 own containers not watched (future work). |
| **Config-to-git** (Phase 3) | Per-host scripts: CT 102 AdGuard, CT 103 Backrest. Central `sync-config-repos` on PVE 03:15 covers CT 103/104/109/113 stacks (auto-discovers CT 104+109 stacks). HAOS gap: Git Pull addon needs UI install. |
| **Offsite personal data** (Phase 1a rclone sync) | Saturdays 01:00 on CT 103. Immich originals → JottaCloud Photos; UNAS personal → JottaCloud UNAS. Last run 2026-05-27 13h32m / 160k files. |
| **Gotify apps** | Diun (#1), Backrest (#2), WAL-G monitor (#3), Gitea digest (#4) |
| **systemd timers active** | CT 103: walg-staleness (hourly). nuc: gitea-digest (08:00), vm100-watchdog (1m), unas-nfs-watchdog (15m), protect-haos-kvm (5m), homelab-health (1h). |
| **Cron jobs active** | nuc 03:15 sync-config-repos. CT 103 Sat 01:00 unas-sync, 02:30 walg-offsite-sync, 03:00 ct103-conf-backup. CT 102 daily adguard-conf-backup. |
### Recent ops debt cleared (2026-05-28)
- Vaultwarden data migrated CT 104 local → UNAS (was unrecoverable on disk loss)
- Backrest false-positive: CT 103 → CT 113 SSH keys never deployed; fixed + retry added to monitor
- Immich WAL-G push broken for 42h (garage container on wrong Docker network post-incident) — fixed
- Diun, Backrest hooks, WAL-G monitor, Gitea digest deployed
- Alloy enabled on CT 103
- Stale CT 110/111/112 references removed from `sync-config-repos`
### Still open
- HAOS Git Pull addon (UI-only install)
- Vaultwarden Borg passphrase storage (UI-only — Vaultwarden web UI)
- n8n-flows issues #1, #3, #4, #7, #8, #9, #10
- Diun second instance for CT 109/113/101 own containers
- DAVx5 → HonorOS `org.tasks` battery-whitelist (user phone, not server)
- Shared-password rotation (`tapirnase` reuse — deferred)
## Open / roadmap
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers