From 9cbd02e3ff132bcb0ef81705562e5a3808b267a9 Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Thu, 28 May 2026 21:20:35 +0200 Subject: [PATCH] docs: 2026-05-28 ops coverage audit + WAL-G monitor fix - homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts), Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3 plans have hooks), Diun, config-to-git, offsite sync, Gotify apps, timers and crons across the fleet. - backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access to CT 113; 2>/dev/null hid the host-key error). Documented retry loop + SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux; noted litellm decommissioned but DB retained). --- services/backrest.md | 6 +++++- services/homelab-architecture.md | 31 +++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/services/backrest.md b/services/backrest.md index a68bc57..57f8ec6 100644 --- a/services/backrest.md +++ b/services/backrest.md @@ -62,7 +62,9 @@ Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmi - `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` - Daily `wal-g backup-push` cron at 02:00 -- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder` +- DBs hosted: `vaultwarden`, `paperless`, `litellm` (decommissioned but DB retained), `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux` +- `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy +- WAL-G monitor (CT 103) checks this via `pg_stat_archiver` Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`). @@ -194,6 +196,8 @@ Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell re > **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running. +> **Initial false-positive (2026-05-28 16:00)**: First scheduled run alerted "CT 113 query failed". Cause: CT 103 had no SSH access to CT 113 (host key + authorized_keys never deployed); `2>/dev/null` in the script hid the underlying `Host key verification failed` error. Fix: deployed CT 103 pub key to CT 113 + `ssh-keyscan`; added retry loop (2 attempts, 5s gap, 10s ConnectTimeout) for transient blips. + **Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`. ### Applying a new config diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index ad2bbb5..36aa801 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -308,6 +308,37 @@ Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg), kept as rollback). Audit other SQLite stacks for NFS-backed data dirs. - Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26. +## Ops coverage matrix (verified 2026-05-28) + +| Capability | Coverage | +|---|---| +| **Log aggregation** (Loki @ CT 109) | All 8 hosts shipping via Alloy: CT 101, 102, 103 (newly enabled today), 104, 105, 109, 113, nuc. Last 5m: 3.6k lines from CT 104, 2.1k from CT 113, 100s on quieter hosts. | +| **Metrics** (Prometheus @ CT 109) | Scrapes node-exporter on CT 104/109, pve-exporter, traefik, home-assistant prom-endpoint, WAL-G textfile collector | +| **Backups** (Backrest @ CT 103 → JottaCloud) | 14 services paths + HA tars + Immich tars + Nextcloud Borg + paperless media + Garage S3 + n8n local data. All 16 plan paths verified present. 3 pre-stage hooks (pocketid, garage, n8n). Success/error hooks → Gotify on all 3 plans. | +| **Postgres archiving** (WAL-G → Garage S3 → JottaCloud) | CT 113 main pg17 (9 DBs incl. owui/miniflux) ✓, CT 104 immich_postgres ✓ (fixed today). Hourly staleness monitor on CT 103 with SSH+psql probe. | +| **Image-update notifications** (Diun @ CT 109) | Watches CT 104 docker (67 images), Mondays 08:00 → Gotify. CT 109/113/101 own containers not watched (future work). | +| **Config-to-git** (Phase 3) | Per-host scripts: CT 102 AdGuard, CT 103 Backrest. Central `sync-config-repos` on PVE 03:15 covers CT 103/104/109/113 stacks (auto-discovers CT 104+109 stacks). HAOS gap: Git Pull addon needs UI install. | +| **Offsite personal data** (Phase 1a rclone sync) | Saturdays 01:00 on CT 103. Immich originals → JottaCloud Photos; UNAS personal → JottaCloud UNAS. Last run 2026-05-27 13h32m / 160k files. | +| **Gotify apps** | Diun (#1), Backrest (#2), WAL-G monitor (#3), Gitea digest (#4) | +| **systemd timers active** | CT 103: walg-staleness (hourly). nuc: gitea-digest (08:00), vm100-watchdog (1m), unas-nfs-watchdog (15m), protect-haos-kvm (5m), homelab-health (1h). | +| **Cron jobs active** | nuc 03:15 sync-config-repos. CT 103 Sat 01:00 unas-sync, 02:30 walg-offsite-sync, 03:00 ct103-conf-backup. CT 102 daily adguard-conf-backup. | + +### Recent ops debt cleared (2026-05-28) +- Vaultwarden data migrated CT 104 local → UNAS (was unrecoverable on disk loss) +- Backrest false-positive: CT 103 → CT 113 SSH keys never deployed; fixed + retry added to monitor +- Immich WAL-G push broken for 42h (garage container on wrong Docker network post-incident) — fixed +- Diun, Backrest hooks, WAL-G monitor, Gitea digest deployed +- Alloy enabled on CT 103 +- Stale CT 110/111/112 references removed from `sync-config-repos` + +### Still open +- HAOS Git Pull addon (UI-only install) +- Vaultwarden Borg passphrase storage (UI-only — Vaultwarden web UI) +- n8n-flows issues #1, #3, #4, #7, #8, #9, #10 +- Diun second instance for CT 109/113/101 own containers +- DAVx5 → HonorOS `org.tasks` battery-whitelist (user phone, not server) +- Shared-password rotation (`tapirnase` reuse — deferred) + ## Open / roadmap mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration + expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers