fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consistency pass 2026-05-23 (session ×4)

Browse Source 
- zoraxy.md: fix arcane upstream to CT109 (192.168.1.8:10002), add
  SkipWebSocketOriginCheck column, note missing routes (dozzle, etc.)
- arcane.md: reflect CT109 migration complete, expand agent table to
  all 8 environments, fix MANAGER_API_URL and DB paths
- ct-inventory.md: CT103 RAM 512→4096+swap; CT109 footnote complete
- proxmox-memory-audit.md: CT103 bump, add CT109+CT113 rows, fix sum
- portmap.md: add Wetty row, fix arcane backend, WAL-G scrape target,
  homepage decommissioned, dozzle LAN-only note
- mcp-gateway.md: add gitea/paperless/proxmox, count 26→29 servers
- RESUME.md: check off WAL-G, Loki, Zoraxy audit; CT109 in key state
  table; session ×4 completed items block

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Florian Krebs
2026-05-23 16:42:45 +02:00
parent 14137c32a7
commit 895658a115
7 changed files with 84 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESUME.md
+20 -7
View File
@@ -1,6 +1,6 @@
# Session Resume
Last updated: 2026-05-23 (session continued ×3).
Last updated: 2026-05-23 (session continued ×4).
## Open items (urgency order)
@@ -13,12 +13,12 @@ Last updated: 2026-05-23 (session continued ×3).
- [x] ~~**Arcane OIDC secret**~~ — fixed 2026-05-23. New secret generated, Pocket-ID bcrypt updated, Arcane compose updated, force-recreated. Login confirmed working.
- [ ] **Backrest media-repo** — `video-projects-plan` has NEVER completed a successful snapshot. `media-backup-plan` last OK 2026-05-21; failed 2026-05-22 01:30; orphaned restic run in progress since 19:10 (Backrest restarted 21:23, orphaned it). Both plans next scheduled 2026-05-25 02:00/01:30. Monitor for completion; if they fail again, investigate rclone:jottacloud:media auth.
- [ ] **AdGuard split-horizon DNS** — internal clients still resolve to public IPs. Rewrite rules audit. See `services/adguard-dns.md`.
- [ ] **WAL-G monitoring** — silent stall went undetected for 13 h. Add textfile collector + alert for hung archiver. See `services/homelab-architecture.md` roadmap.
- [ ] **Zoraxy audit** — CT 108 routes: WS headers, ACME coverage, decommissioned routes, auth consistency. See `services/zoraxy.md`.
- [x] ~~**WAL-G monitoring**~~ — done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`.
- [x] ~~**Zoraxy audit**~~ — done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`.
### 🟡 Medium — incomplete migrations / cleanup debt
- [ ] **Docker disk reclaim** — ~17 GB reclaimable on CT 104 (10.3 GB unused images, 6.9 GB build cache). `docker system prune -a` after confirming no needed images. See `infra/proxmox-state.md` §10c.
- [ ] **Dormant stacks audit** — ~10 stacks defined but not running (homepage, dozzle, arr-stack, qdrant, etc.). Document: parked vs. abandoned. See `infra/proxmox-state.md` §10c.
- [ ] **Dormant stacks audit** — 4 decommissioned 2026-05-23 (daytona, homepage, proxy, streamio — composes renamed `.DECOMMISSIONED`). Remaining ~6 still to audit: dozzle (CT 104 copy), arr-stack, qdrant, etc. See `infra/proxmox-state.md` §10c.
- [x] ~~**Config-to-git**~~ — deployed 2026-05-23. Scripts + daily 03:00 crons on CT 108, CT 102, PVE. Pushes to `fkrebs/zoraxy-conf`, `fkrebs/adguard-conf`, `fkrebs/pve-conf` (force-push). HA (`fkrebs/ha-config`) still pending — needs Git Pull addon on HAOS.
- [ ] **D-Link hardening** — admin UI open to full LAN, SNMP unverified, HTTP-only. Enable trusted-host allowlist, harden SNMP, add TLS. See `services/homelab-architecture.md`.
- [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22.
@@ -27,7 +27,7 @@ Last updated: 2026-05-23 (session continued ×3).
- [ ] **Infisical OIDC config** — Pocket-ID client created (`b2069075-ede2-4251-ad1f-9a62e6a188b3`). Manual step needed: enter OIDC settings in Infisical admin at `http://192.168.1.7:8200/admin` → Settings → OIDC. Issuer: `https://id.nuclide.systems`, ClientID: `b2069075`, Secret: `qsANw95zsza0_tzJR9SzSp4Z6iEIhidgJPHpDih1ffE`.
- [ ] **Infisical secrets migration** — Phase 2: import CT104 `.env` keys. Use `infisical import`. See `services/secrets-manager.md` Phase 2.
- [ ] **Arcane auto-update notifications** — ntfy.nuclide.systems provider configured; auto-update daily 3am set. Gotify Apprise incompatible (payload format). Workaround: configure via Arcane UI or add shoutrrr plugin. Gotify app token: `AQnDNB2Y6JlOewc` (app: arcane-updates).
- [ ] **Loki dashboard broken in Grafana** — Loki is healthy (labels returning). Likely dashboard query issue. Needs investigation.
- [x] ~~**Loki dashboard broken in Grafana**~~ — fixed 2026-05-23. K8s labels (namespace/pod) replaced with Docker labels (host/container/job) in 14 places across template variables and panel LogQL expressions. Provisioning file updated + pushed via Grafana API.
- [ ] **Grafana HAOS dashboard** — HA recorder reconfiguration planned (exclude Prometheus-covered metrics). Reduce recorder include list.
- [ ] **HAOS SSH key** — manual step: paste nuc public key in HA terminal.
@@ -42,10 +42,22 @@ Last updated: 2026-05-23 (session continued ×3).
- [ ] **ZFS ARC tuning** — raise `zfs_arc_max` from 6.2 GiB → 16 GiB once CT 101 memory right-sized. See `infra/proxmox-state.md` §3.
- [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`.
- [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`.
- [ ] **Additional MCP servers** — community MCPs exist for Gitea, Paperless, Karakeep, Vaultwarden, Proxmox-VE, Audiobookshelf. Evaluate + wire in.
- [ ] **Additional MCP servers** — Gitea ✓, Paperless ✓, Proxmox-VE ✓ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf.
- [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts with unique tokens per environment. 8 environments total: NUC, db, shepard, docker, nextcloud, id, dev, secrets. Auto-update daily 3am. ntfy notifications configured.
- [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers.
## Recently completed (2026-05-23, session continued ×4)
- **Docs consistency pass**: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state.
- **Wetty deployed** on CT 109 (:4090, LAN-only). `jump-menu.sh` on nuc → SSH jump menu to all 10 hosts.
- **MCP servers added**: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers.
- **Loki dashboard fixed**: K8s labels replaced with Docker labels in 14 places.
- **WAL-G monitoring**: textfile collector on CT 113 + Prometheus alert rules on CT 109.
- **CT 103 RAM**: 2 GB → 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup).
- **Zoraxy WS**: `SkipWebSocketOriginCheck` enabled on 13 routes. Arcane upstream corrected to CT 109.
- **CT 104 cleanup**: daytona, homepage, proxy, streamio decommissioned.
- **Paperless-ngx**: USERMAP_UID=997 + file permissions fixed, container stable.
## Recently completed (2026-05-23, session continued ×3)
- **Homarr board deployed**: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: `x_offset`/`y_offset` on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board `nuclide` set as home board.
@@ -100,7 +112,8 @@ Last updated: 2026-05-23 (session continued ×3).
| Host | IP | Role |
|---|---|---|
| nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs |
| nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs; jump-menu.sh |
| CT 109 ops | 192.168.1.8 | Monitoring — Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty |
| CT 104 docker | 192.168.1.40 | Main Docker host — AI/ML, media, ~65 containers |
| CT 113 db | 192.168.1.6 | Shared Postgres 17 + WAL-G → Garage S3 (cron 02:00) |
| CT 110 id | 192.168.1.5 | Pocket-ID OIDC IdP |