# Session Resume Last updated: 2026-05-23 (session continued Γ—4). ## Open items (urgency order) ### πŸ”΄ Critical β€” security risk or unrecoverable data loss - [ ] **Password rotation: `tapirnase`** β€” shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` Β§operational rules. - [ ] **UNAS personal data off-host** β€” ~820 GB Immich library partially covered by `media-backup-plan` in Backrest (last OK 2026-05-21 but failing/orphaned). `video-projects-plan` never completed. Backrest is the right vehicle β€” fix reliability first (see 🟠 Backrest item), then verify JottaCloud coverage. See `services/backrest.md` phase 1a. - [ ] **Nextcloud Borg passphrase β†’ Vaultwarden** β€” passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See `services/backrest.md` blind spot #4b. ### 🟠 High β€” known broken / verification needed - [x] ~~**Arcane OIDC secret**~~ β€” fixed 2026-05-23. New secret generated, Pocket-ID bcrypt updated, Arcane compose updated, force-recreated. Login confirmed working. - [ ] **Backrest media-repo** β€” `video-projects-plan` has NEVER completed a successful snapshot. `media-backup-plan` last OK 2026-05-21; failed 2026-05-22 01:30; orphaned restic run in progress since 19:10 (Backrest restarted 21:23, orphaned it). Both plans next scheduled 2026-05-25 02:00/01:30. Monitor for completion; if they fail again, investigate rclone:jottacloud:media auth. - [x] ~~**AdGuard split-horizon DNS**~~ β€” done 2026-05-23. Wildcard rewrite `*.nuclide.systems β†’ 192.168.1.4` added to AdGuardHome.yaml; `rewrites_enabled: true` set. All internal clients now resolve to Zoraxy directly. - [x] ~~**WAL-G monitoring**~~ β€” done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`. - [x] ~~**Zoraxy audit**~~ β€” done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`. ### 🟑 Medium β€” incomplete migrations / cleanup debt - [ ] **Docker disk reclaim** β€” ~17 GB reclaimable on CT 104 (10.3 GB unused images, 6.9 GB build cache). `docker system prune -a` after confirming no needed images. See `infra/proxmox-state.md` Β§10c. - [ ] **Dormant stacks audit** β€” 4 decommissioned 2026-05-23 (daytona, homepage, proxy, streamio β€” composes renamed `.DECOMMISSIONED`). Remaining ~6 still to audit: dozzle (CT 104 copy), arr-stack, qdrant, etc. See `infra/proxmox-state.md` Β§10c. - [x] ~~**Config-to-git**~~ β€” deployed 2026-05-23. Scripts + daily 03:00 crons on CT 108, CT 102, PVE. Pushes to `fkrebs/zoraxy-conf`, `fkrebs/adguard-conf`, `fkrebs/pve-conf` (force-push). HA (`fkrebs/ha-config`) still pending β€” needs Git Pull addon on HAOS. - [ ] **D-Link hardening** β€” admin UI open to full LAN, SNMP unverified, HTTP-only. Enable trusted-host allowlist, harden SNMP, add TLS. See `services/homelab-architecture.md`. - [ ] **Vaultwarden OIDC SSO** β€” Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22. ### 🟑 Medium β€” incomplete migrations / cleanup debt - [ ] **Infisical OIDC config** β€” Pocket-ID client created (`b2069075-ede2-4251-ad1f-9a62e6a188b3`). Manual step needed: enter OIDC settings in Infisical admin at `http://192.168.1.7:8200/admin` β†’ Settings β†’ OIDC. Issuer: `https://id.nuclide.systems`, ClientID: `b2069075`, Secret: `qsANw95zsza0_tzJR9SzSp4Z6iEIhidgJPHpDih1ffE`. - [ ] **Infisical secrets migration** β€” Phase 2: import CT104 `.env` keys. Use `infisical import`. See `services/secrets-manager.md` Phase 2. - [ ] **Arcane auto-update notifications** β€” ntfy.nuclide.systems provider configured; auto-update daily 3am set. Gotify Apprise incompatible (payload format). Workaround: configure via Arcane UI or add shoutrrr plugin. Gotify app token: `AQnDNB2Y6JlOewc` (app: arcane-updates). - [x] ~~**Loki dashboard broken in Grafana**~~ β€” fixed 2026-05-23. K8s labels (namespace/pod) replaced with Docker labels (host/container/job) in 14 places across template variables and panel LogQL expressions. Provisioning file updated + pushed via Grafana API. - [ ] **Grafana HAOS dashboard** β€” HA recorder reconfiguration planned (exclude Prometheus-covered metrics). Reduce recorder include list. - [ ] **HAOS SSH key** β€” manual step: paste nuc public key in HA terminal. ### πŸ”΅ Planned β€” requires infrastructure or significant effort - [ ] **Second NVMe** β€” needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items. - [ ] **Infisical secrets migration** β€” CT 112 deployed (`192.168.1.7:8200`, LAN-only). Phase 1 done. Remaining: - **Phase 2**: create project `homelab/ct104`, import `ai/.env` (~55 keys) via `infisical import`, add agent sidecars to compose stacks - **Phase 3**: replace hardcoded env vars in Coder `main.tf` (committed to Gitea β€” security risk) - **Phase 4**: OIDC SSO via Pocket-ID (low priority β€” convenience) See `services/secrets-manager.md`. - [x] ~~**Consolidate syncstack β†’ MCP gateway**~~ β€” done 2026-05-23. `litellm_sync.py` ported from syncstack, mounted into gateway. Gateway runs `run_sync()` every 15 min (120s startup delay). `/etc/cron.d/syncstack` deleted; `Dockerfile.syncstack` β†’ `.DECOMMISSIONED-2026-05-23`; `/opt/stacks/CLAUDE.md` updated. - [ ] **ZFS ARC tuning** β€” raise `zfs_arc_max` from 6.2 GiB β†’ 16 GiB once CT 101 memory right-sized. See `infra/proxmox-state.md` Β§3. - [ ] **Document ingestion n8n workflow** β€” Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless β†’ Docling β†’ embeddings β†’ LobeChat KB. See `services/doc-ingestion.md`. - [ ] **ComfyUI async queue** β€” MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`. - [ ] **Additional MCP servers** β€” Gitea βœ“, Paperless βœ“, Proxmox-VE βœ“ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf. - [x] ~~**Arcane + Dozzle β†’ CT 109**~~ β€” done 2026-05-23. Headless agents on all 7 Docker hosts with unique tokens per environment. 8 environments total: NUC, db, shepard, docker, nextcloud, id, dev, secrets. Auto-update daily 3am. ntfy notifications configured. - [x] ~~**crawl4ai SSE StreamConsumed**~~ β€” resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers. ## Recently completed (2026-05-23, session continued Γ—4) - **Docs consistency pass**: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state. - **Wetty deployed** on CT 109 (:4090, LAN-only). `jump-menu.sh` on nuc β†’ SSH jump menu to all 10 hosts. - **MCP servers added**: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers. - **Loki dashboard fixed**: K8s labels replaced with Docker labels in 14 places. - **WAL-G monitoring**: textfile collector on CT 113 + Prometheus alert rules on CT 109. - **CT 103 RAM**: 2 GB β†’ 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup). - **Zoraxy WS**: `SkipWebSocketOriginCheck` enabled on 13 routes. Arcane upstream corrected to CT 109. - **CT 104 cleanup**: daytona, homepage, proxy, streamio decommissioned. - **Paperless-ngx**: USERMAP_UID=997 + file permissions fixed, container stable. ## Recently completed (2026-05-23, session continued Γ—3) - **Homarr board deployed**: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: `x_offset`/`y_offset` on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board `nuclide` set as home board. - **Arcane environments fixed**: deleted stale shared-token environments; created 6 unique edge environments (shepard, docker, nextcloud, id, dev, secrets) via API with `useApiKey=true`. Each agent updated with unique token. All 8 environments online. - **Arcane API key seeded**: `ADMIN_STATIC_API_KEY` from compose was never seeded (DB pre-existed). Inserted via argon2id hash. Key `arc_d3357a65` now functional for automation. - **Infisical SITE_URL fixed**: changed from `https://secrets.nuclide.systems` to `http://192.168.1.7:8200` (LAN-only, no Zoraxy route). - **Infisical Pocket-ID client created**: client `b2069075-ede2-4251-ad1f-9a62e6a188b3`, callback `http://192.168.1.7:8200/api/v1/sso/oidc/callback`. Manual OIDC config entry still needed in Infisical admin UI. ## Recently completed (2026-05-23, session continued Γ—2) - **PVE LXC startup order fixed**: CT103/104/105/112/113 had no startup order. Set: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full deterministic boot sequence now in place. Also enabled CT109 onboot=1 (was missing). - **KSM enabled on PVE host**: kernel samepage merging active + persisted via `ksm-enable.service`. Zero-cost memory deduplication across LXC containers. - **5-minute critical watchdog deployed**: `homelab-health-quick.timer` on nuc; `QUICK=1` mode checks LXC status + critical containers every 5 min. Auto-starts stopped LXCs, notifies Gotify for HAOS KVM issues. - **PVE backlog issues filed**: `fkrebs/docs` #2 (NFS v3β†’v4), #3 (ZFS ARC 16 GiB), #4 (hugepages for KVM). ## Recently completed (2026-05-23, session continued) - **HAOS KVM OOM root-caused and fixed**: karakeep_chrome on CT 104 triggered host OOM β†’ killed HAOS KVM twice. Applied: `shm_size: '512m'` to chrome, `oom_score_adj=-300` on KVM PID, systemd timer `protect-haos-kvm.timer` (every 5 min) for persistence. - **Arcane + Dozzle migrated to CT 109**: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080). - **Homarr + Grafana OIDC wired**: clients created via Pocket-ID API (no manual UI steps). Homarr client `63a94e30`, Grafana client `92d987d5` (PKCE). Both SSO-active. - **n8n flows tracked in Gitea**: repo `fkrebs/n8n-flows` with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n). - **CT 104 ai repo cleaned**: `lobehub/data/` untracked from git, `.gitignore` fixed, litellm-config + mcp-gateway changes committed. - **zoraxy-conf repo**: arcane proxy update committed and pushed (CT 108). ## Recently completed (2026-05-23) - **Config-to-git crons**: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy β†’ `zoraxy-conf`), CT 102 (AdGuard β†’ `adguard-conf`), PVE (rsync `/etc/pve` β†’ `pve-conf`). Force-push. All three verified with initial push. - **Syncstack β†’ MCP gateway**: `litellm_sync.py` (ported from syncstack.py) mounted into gateway container. `_litellm_sync_loop()` runs every 15 min in gateway's asyncio loop. Cron `/etc/cron.d/syncstack` deleted. `Dockerfile.syncstack` decommissioned. `/opt/stacks/CLAUDE.md` updated. ## Recently completed (session continued Γ—2, 2026-05-22) - **crawl4ai MCP fixed**: was returning 404 on `/mcp` (streamable-HTTP). Current 0.8.6 image already has MCP over SSE at `/mcp/sse`. Added SSE transport support to `mcp-gateway/server.py`: per-request SSE round-trip with auto-initialize handshake; `transport: sse` in hardcoded SERVERS + config.json. Gateway health now shows crawl4ai ok; lobe-sync shows 27/27 servers. - **syncstack cron fixed**: missing `cd /opt/stacks/ai &&` prefix caused "no configuration file" every 15 min since ~May 18. Fixed. Manual run confirmed 60 models including 5 Claude models; LobeChat recreated. ## Recently completed (session continued, 2026-05-22) - **claude-max-bridge `/v1/responses` endpoint**: implemented OpenAI Responses API with `previous_response_id` chaining. Root cause of "session already in use": Claude CLI leaves session JSONL files in an un-resumable "dequeued" state after each `--print` run. Fix: maintain conversation history server-side as formatted text; inject into system prompt for continuations; always use `--no-session-persistence`. Also fixed: assistant turns in `--input-format stream-json` require `content` as array-of-blocks not string (JS error otherwise). Assistant content array format also fixed in `_parse_messages` / `_parse_responses_input`. - **LiteLLM pass-through for `/v1/responses`**: added `pass_through_endpoints` entry in `/opt/stacks/ai/litellm-config/config.yaml` β†’ `http://claude-max-bridge:8000/v1/responses`. LiteLLM appends its token hash to the response `id`, so `previous_response_id` chaining only works when calling the bridge directly (not through LiteLLM). ## Recently completed (this session, 2026-05-22) - **Pocket-ID backup**: pre-hook on CT 103 runs `pocket-id export` + scp keys β†’ UNAS staging; `services-backup-plan` snapshots β†’ jottacloud. Verified. - **Stale NFS vaultwarden dir deleted**: `/mnt/pve/unas/services/vaultwarden/` removed. - **Nextcloud CIFS β†’ NFS**: crash-loop fixed; CT 105 mp0 migrated from `/mnt/pve/unas_smb` to `/mnt/pve/unas`; CIFS fstab entry removed. - **Vaultwarden β†’ local zfs**: `/mnt/pve/unas/services/vaultwarden` β†’ `/opt/stacks/vaultwarden/data`; stale `db.sqlite3` deleted; DB already on CT 113 postgres. - **WAL-G fixed**: `immich_postgres` and `lobe-postgres` backups now succeed. Fixed: missing `-u postgres`, missing data path, `set -e` abort, dead `shared-postgres` reference. - **ComfyUI `--async-offload` removed**: caused CPU fallback + pure-noise output on GGUF img2img with XPU. - **apps/ decommissioned**: dead duplicate compose files in `/opt/stacks/apps/` renamed `.DECOMMISSIONED-2026-05-22`. - **Docs consolidated**: storage.md, volumes.md, homelab-architecture.md, proxmox-state.md, proxmox-memory-audit.md, ct-inventory.md, README.md all reconciled. ## Key system state | Host | IP | Role | |---|---|---| | nuc (PVE) | 192.168.1.20 | Proxmox host β€” SSH gateway to all CTs; jump-menu.sh | | CT 109 ops | 192.168.1.8 | Monitoring β€” Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty | | CT 104 docker | 192.168.1.40 | Main Docker host β€” AI/ML, media, ~65 containers | | CT 113 db | 192.168.1.6 | Shared Postgres 17 + WAL-G β†’ Garage S3 (cron 02:00) | | CT 110 id | 192.168.1.5 | Pocket-ID OIDC IdP | | CT 103 backrest | 192.168.1.3 | Backrest β€” jottacloud via rclone; UNAS at `/mnt/pve/unas` | | CT 111 dev | 192.168.1.42 | Coder + Gitea | | CT 108 zoraxy | 192.168.1.4 | Reverse proxy + ACME β€” **always confirm before changes** | ## Constraints to remember - Zoraxy (CT 108): always get explicit user confirmation before any config change. - Infisical (CT 112): LAN-only, no Zoraxy route β€” must not be internet-exposed. - CT 104 has a separate agent reconciling compose files β€” rename decommissioned files (don't just stop), update `/opt/stacks/CLAUDE.md`. - ComfyUI container limit: 24 G. Do not raise without measuring host impact. - Pocket-ID is SQLite-only β€” no Postgres migration possible.