fkrebs
9cbd02e3ff
docs: 2026-05-28 ops coverage audit + WAL-G monitor fix
...
- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
noted litellm decommissioned but DB retained).
2026-05-28 21:20:35 +02:00
fkrebs
b26796e30c
ops: Loki/Alloy fleet enabled; Backrest+WAL-G+Gitea Gotify hooks; Immich WAL fix
...
- Alloy now active on CT 102, 103, 104, 109, 101, 105, 113, nuc (mix of
Docker container + systemd binary); all hosts shipping journal/docker
logs to Loki on CT 109. CT 103 alloy newly enabled.
- Backrest plans: actionGotify hooks on SUCCESS (prio 3) + ERROR/WARNING
(prio 8). Gotify app 'Backrest', token AtIMGOYcZsb5HfO.
- WAL-G staleness monitor on CT 103 (hourly): pg_stat_archiver SSH/exec
against CT 113 + CT 104 immich_postgres. Found Immich WAL push failing
for 42h — root cause: garage container on bridge network not
shared_backend (Garage 2026-05-28 incident aftermath). Reconnected to
shared_backend; image pinned v0.9.4 in compose to match running.
- Gitea daily digest at 08:00 on PVE host nuc — yesterday's commits.
- Closes fkrebs/n8n-flows #2 , #5 , #6 .
2026-05-28 15:19:40 +02:00
fkrebs
c05f38d008
docs: Diun deployed on CT 109; doc-sync Phase 1a/3 actual state
...
- Diun (Docker image update notifier) on CT 109, watching CT 104 via
socket-proxy, Mondays 08:00 → Gotify app "Diun" (token A34U_7k3biVI_po)
- backrest.md: Phase 1a (rclone sync) confirmed ACTIVE since 2026-05-22
(Saturdays 01:00). Phase 3 (config-to-git) confirmed ACTIVE across
CT 102 / CT 103 / PVE host; sync-config-repos rebuilt to drop dead
CT 110/111/112 refs, auto-discover CT 104 stacks, add CT 109 stacks
- Coverage map updated to reflect actual state (no longer "not deployed")
- Open Phase 3 gap: HAOS Git Pull addon — requires HAOS UI install
- services-overview.md: Diun row added; Traefik replaces Zoraxy
2026-05-28 10:03:39 +02:00
fkrebs
f1ed462c91
fix: resolve 7 of 8 backup blind spots (2026-05-28)
...
- Vaultwarden data moved to UNAS /services/vaultwarden (CT 104 local → NFS)
- Garage S3 pre-backup rsync hook (garage-prestage.sh) + path added to plan
- n8n pre-backup rsync hook (n8n-prestage.sh) + confirm path covered
- Paperless-ngx media path added to services-backup-plan
- Stale paths (arcane, arr-stack, gluetun) removed from plan
- Repo passwords confirmed rotated (not tapirnase); LiteLLM blind spot N/A
- CT 103 SSH key deployed to CT 104 + CT 109 for prestage hooks
- pocketid-prestage.sh updated from CT 110 (destroyed) → CT 109
- Coverage map updated with new entries
2026-05-28 01:35:31 +02:00
fkrebs
24c7d56a1b
backup: update coverage map and blind spots (2026-05-28)
...
- CT 108 Zoraxy decommissioned -> Traefik on CT 109
- Add critical blind spots: Vaultwarden (local FS, not backed up),
Garage S3 data (WAL-G archive store, not backed up), Paperless media
- Fix services plan path list to match actual Backrest config.json
- Mark stale plan paths: arcane, arr-stack, gluetun
- Note Garage config incident: garage.toml became a directory after
Portainer redeploy; fixed 2026-05-28 by manual replace + layout apply
2026-05-28 01:15:29 +02:00
fkrebs
24d99d18fc
backup: fix CT 109 coverage gaps; archive dead Gitea repos
...
- ops-backup.timer on CT 109: daily 01:30 tars Pocket-ID data +
Infisical pg_dump → Garage S3 ct109-portainer-backup (ops-*.tar.gz)
→ already offsite via walg-offsite-sync
- backrest.md: coverage map updated for CT 110/111/112 decommission,
CT 109 Pocket-ID + Infisical now marked covered
- services plan paths: removed services/pocketid (no longer on UNAS)
- Gitea: archived ct110-pocket-id, ct111-dev, mcp-shepard
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-05-26 22:41:18 +02:00
fkrebs
f65ceef90a
Add Portainer BE doc; update backup coverage for CT 109
...
- New services/portainer.md: BE license key, agent inventory, backup
setup (Garage S3 + JottaCloud offsite), OIDC pending steps
- backrest.md: add CT 109 Portainer to coverage map, remove stale
services/arcane path from plan, note Arcane decommission
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-05-26 17:12:36 +02:00
fkrebs
0d83f83bc0
docs: MCP gateway git/gitlab/paper-search; CT 109 plan; new backrest/db/arcane docs
...
- mcp-gateway.md: 26 servers (added git, gitlab, paper-search catalog bridge);
document --pass-environment requirement for env var passthrough in DinD bridges
- ct-inventory.md + homelab-architecture.md: CT 109 IP conflict, CT 113 live
- proxmox-state.md §16: Loki, sshwifty, Alloy, IP conflict note, sidecar table
expanded to CT 103/111/113; build order updated
- infra/portmap.md: CT 113 db section; QNAP TS-251D (Klipper/Mainsail)
- services/backrest.md: full backup strategy doc (new)
- services/databases.md: CT 113 postgres/WAL-G/pgAdmin doc (new)
- services/arcane.md: Docker management IDE doc (new)
- CHANGELOG.md: 2026-05-22 entry
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-05-22 09:35:07 +02:00
fkrebs