docs/RESUME.md

Session Resume

Last updated: 2026-05-26.

Open items (urgency order)

🔴 Critical — security risk or unrecoverable data loss

• Password rotation: tapirnase — shared across WiFi PSK, LiteLLM root key (sk-tapirnase), D-Link admin, Backrest repo password, HAOS Terminal & SSH addon password. Single sniff = broad blast radius. Rotate per-service. See services/homelab-architecture.md §operational rules.
• Nextcloud Borg passphrase → Vaultwarden — passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See services/backrest.md blind spot #4b.
• Vault secrets sweep — 32 plaintext secrets in Obsidian vault notes (Anthropic key, Gemini, NC AIO passphrase, etc.). Tracker at Home/Homelab/Secrets to Move.md in vault. Migrate to Vaultwarden, redact notes, rotate the high-risk ones.

🟠 High — known broken / verification needed

• Arcane OIDC secret — fixed 2026-05-23.
• Backrest media-repo / video-projects — all three plans succeeded 2026-05-26 (media 01:30, services 01:04, video-projects 04:00). video-projects-plan had never completed before this. ✅
• AdGuard split-horizon DNS — done 2026-05-23.
• WAL-G monitoring — done 2026-05-23. Textfile collector (/usr/local/bin/walg-metrics.sh) on CT 113 emits walg_last_success_timestamp_seconds + walg_archive_status to node-exporter textfile. Prometheus alert rules deployed on CT 109: WalgArchiveStale (>24h) + WalgArchiveFailed.
• Zoraxy audit — done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; SkipWebSocketOriginCheck enabled on 13 routes; decommissioned routes noted. Missing: dozzle.nuclide.systems route (never created), immich-tools, paperless, paperless-ai. See services/zoraxy.md.

🟡 Medium — incomplete migrations / cleanup debt

• Docker disk reclaim — ~17 GB reclaimable on CT 104.
• Dormant stacks audit — ~6 still to audit (dozzle CT 104 copy, arr-stack, qdrant, etc.). 4 decommissioned 2026-05-23.
• Config-to-git fleet — 8 repos as of 2026-05-24. PVE/CT 108/CT 102/HA/CT 103/CT 109/CT 113/obsidian-vault/obsidian-config. Full table in infra/config-to-git.md. Manual step pending: HA Terminal & SSH addon init_commands for cron persistence.
• D-Link hardening — admin UI at http://dlink.nuclide.lan (192.168.1.10). Trusted-host allowlist + SNMP + TLS.
• Vaultwarden OIDC SSO — Pocket-ID client ready, env vars not yet set in /opt/stacks/vaultwarden/.env.
• Infisical OIDC config — manual step in Infisical admin at http://secrets.nuclide.lan:8200/admin → Settings → OIDC. Client b2069075-…, secret qsANw95zsza0_….
• Infisical secrets migration Phase 2 — import CT104 .env keys via infisical import.
• Grafana HAOS dashboard — HA recorder reconfiguration (exclude Prometheus-covered metrics).
• HAOS SSH key — done 2026-05-27.
• Obsidian config drift — removed obsidian42-brat, obsidian-tasks-plugin, tasks-caldav-sync from community-plugins.json 2026-05-27.
• Memos Sync plugin install — config pushed to obsidian-config, plugin binary needs Community-Plugins install + paste of pre-baked data.json.
• Immich HTTP/2 Keep-Alive — root-caused 2026-05-24. Workaround live (per-SSID LAN URL). Permanent fix: strip header at Zoraxy. Needs Zoraxy confirmation.

🔵 Planned — requires infrastructure or significant effort

• Second NVMe — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items.
• Infisical secrets migration — CT 112 deployed (192.168.1.7:8200, LAN-only). Phase 1 done. Remaining:
  • Phase 2: create project homelab/ct104, import ai/.env (~55 keys) via infisical import, add agent sidecars to compose stacks
  • Phase 3: replace hardcoded env vars in Coder main.tf (committed to Gitea — security risk)
  • Phase 4: OIDC SSO via Pocket-ID (low priority — convenience) See services/secrets-manager.md.
• Consolidate syncstack → MCP gateway — done 2026-05-23. litellm_sync.py ported from syncstack, mounted into gateway. Gateway runs run_sync() every 15 min (120s startup delay). /etc/cron.d/syncstack deleted; Dockerfile.syncstack → .DECOMMISSIONED-2026-05-23; /opt/stacks/CLAUDE.md updated.
• ZFS ARC tuning — raise zfs_arc_max from 6.2 GiB → 16 GiB once CT 101 memory right-sized. See infra/proxmox-state.md §3.
• Document ingestion n8n workflow — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See services/doc-ingestion.md.
• ComfyUI async queue — MCP tool blocks 90–300 s. Job-queue pattern: generate_image() returns ID; get_job_status() polls. See services/comfyui.md.
• Additional MCP servers — Gitea ✓, Paperless ✓, Proxmox-VE ✓ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf.
• Arcane + Dozzle → CT 109 — done 2026-05-23.
• crawl4ai SSE StreamConsumed — resolved 2026-05-22.

Recently completed (2026-05-26)

• Backrest 3-for-3 success: all plans now completing on schedule. video-projects-plan recovered from never-completed state.

Recently completed (2026-05-24)

Vault + Obsidian

• Vault dedup: Work 1/ promoted → Work/, old Work/ parked as Work.stale-backup-2026-05-24/ (1-week safety net). Rescued unique Journal/Journal.md before swap. Killed dup Home/BrainBox.md, Home/Templates/, Willkommen.md ×2, .caldav-sync/.
• 234 asset-link rewrites across 41 files — collapsed assets/, ../assets/, ../../foo.pdf → .assets/…. Broken links 242 → 9 (rest non-issues: tel:, about:reader?, siyuan://).
• Home/Homelab/Secrets to Move.md — 32 plaintext secrets inventoried.
• Home/Homelab/Obsidian Plugin Audit.md — 17 installed plugins keep/watch/re-evaluate, 3 config-drift items, 10 new candidates ranked for the stack.
• Home/Homelab/App Endpoint Checklist.md — per-service LAN + external URLs + apps-to-update list. Gotchas: clients sticking to old IP, Immich HTTP/2 Keep-Alive bug.
• Daily-note merge policy: Claude edits land in Work/Journal/YYYY/MM/YYYY-MM-DD.md under ## 🤖 Claude edits (not parallel folder).
• Obsidian Git plugin: wired end-to-end against fkrebs/obsidian-config. Auto-commit + push every 60 min, pull on boot, PAT gitignored. Verified working.
• Streamlined-ui CSS snippet pushed to obsidian-config (hides ribbon icons for non-clickable plugins, compact status bar, tighter file tree).
• Memos Sync plugin config pre-pushed (binary install pending user).

Config-to-git fleet expanded to 8 repos

• New: fkrebs/ct103-conf, fkrebs/ct109-conf, fkrebs/ct113-conf, fkrebs/home-assistant-config, fkrebs/obsidian-vault (rolling 2-commit window), fkrebs/obsidian-config.
• All daily 03:00 / 04:00. Pattern documented in infra/config-to-git.md.
• Rule articulated: every host with hand-edited config must push daily.

Networking — *.nuclide.lan zone

• 30 AdGuard A-records on CT 102 — per-host (pve, nas, unifi, dlink, backrest, etc.) + per-service aliases (immich, vault, karakeep, grafana, prometheus, …).
• Documented in services/adguard-dns.md.
• Solves "client registered to old IP" gotcha — when service moves CT, edit one AdGuard rewrite instead of N apps.
• D-Link IP captured: 192.168.1.10.
• Filed feedback memory: always use *.nuclide.lan aliases in URLs, never bare IPs.

Homarr

• 3 new boards built via SQL: home (morning routine, default), admin (by-tier), command (live-ops iframes + grid). Old "nuclide" board was lost when OIDC user was recreated for password reset.
• Boards owned by fkrebs@nucli.de, is_public=1. 33 unique apps with ping status.
• Integrations not yet wired (Proxmox / Docker / AdGuard / Gotify widgets — need tokens).

Pocket-ID

• LAN callback URLs added for Homarr, Grafana, Infisical, Proxmox VE so OIDC works via both bare-IP and .nuclide.lan alias paths.

Other

• Immich Android upload bug diagnosed: HTTP/2 + Keep-Alive header conflict. Workaround live (per-SSID LAN URL). Permanent fix pending Zoraxy header strip.
• Nexa #1 filed at fkrebs/nexa#1 — 9 design patterns from Obsidian + Claude session as input for Nexa Phase 1–2.
• Wi-Fi SSID captured: nuclide. Used for per-network app routing (Immich, HA Companion).

Recently completed (2026-05-23, session continued ×4)

• Docs consistency pass: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state.
• Wetty deployed on CT 109 (:4090, LAN-only). jump-menu.sh on nuc → SSH jump menu to all 10 hosts.
• MCP servers added: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers.
• Loki dashboard fixed: K8s labels replaced with Docker labels in 14 places.
• WAL-G monitoring: textfile collector on CT 113 + Prometheus alert rules on CT 109.
• CT 103 RAM: 2 GB → 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup).
• Zoraxy WS: SkipWebSocketOriginCheck enabled on 13 routes. Arcane upstream corrected to CT 109.
• CT 104 cleanup: daytona, homepage, proxy, streamio decommissioned.
• Paperless-ngx: USERMAP_UID=997 + file permissions fixed, container stable.

Recently completed (2026-05-23, session continued ×3)

• Homarr board deployed: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: x_offset/y_offset on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board nuclide set as home board.
• Arcane environments fixed: deleted stale shared-token environments; created 6 unique edge environments (shepard, docker, nextcloud, id, dev, secrets) via API with useApiKey=true. Each agent updated with unique token. All 8 environments online.
• Arcane API key seeded: ADMIN_STATIC_API_KEY from compose was never seeded (DB pre-existed). Inserted via argon2id hash. Key arc_d3357a65 now functional for automation.
• Infisical SITE_URL fixed: changed from https://secrets.nuclide.systems to http://192.168.1.7:8200 (LAN-only, no Zoraxy route).
• Infisical Pocket-ID client created: client b2069075-ede2-4251-ad1f-9a62e6a188b3, callback http://192.168.1.7:8200/api/v1/sso/oidc/callback. Manual OIDC config entry still needed in Infisical admin UI.

Recently completed (2026-05-23, session continued ×2)

• PVE LXC startup order fixed: CT103/104/105/112/113 had no startup order. Set: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full deterministic boot sequence now in place. Also enabled CT109 onboot=1 (was missing).
• KSM enabled on PVE host: kernel samepage merging active + persisted via ksm-enable.service. Zero-cost memory deduplication across LXC containers.
• 5-minute critical watchdog deployed: homelab-health-quick.timer on nuc; QUICK=1 mode checks LXC status + critical containers every 5 min. Auto-starts stopped LXCs, notifies Gotify for HAOS KVM issues.
• PVE backlog issues filed: fkrebs/docs #2 (NFS v3→v4), #3 (ZFS ARC 16 GiB), #4 (hugepages for KVM).

Recently completed (2026-05-23, session continued)

• HAOS KVM OOM root-caused and fixed: karakeep_chrome on CT 104 triggered host OOM → killed HAOS KVM twice. Applied: shm_size: '512m' to chrome, oom_score_adj=-300 on KVM PID, systemd timer protect-haos-kvm.timer (every 5 min) for persistence.
• Arcane + Dozzle migrated to CT 109: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080).
• Homarr + Grafana OIDC wired: clients created via Pocket-ID API (no manual UI steps). Homarr client 63a94e30, Grafana client 92d987d5 (PKCE). Both SSO-active.
• n8n flows tracked in Gitea: repo fkrebs/n8n-flows with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n).
• CT 104 ai repo cleaned: lobehub/data/ untracked from git, .gitignore fixed, litellm-config + mcp-gateway changes committed.
• zoraxy-conf repo: arcane proxy update committed and pushed (CT 108).

Recently completed (2026-05-23)

• Config-to-git crons: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy → zoraxy-conf), CT 102 (AdGuard → adguard-conf), PVE (rsync /etc/pve → pve-conf). Force-push. All three verified with initial push.
• Syncstack → MCP gateway: litellm_sync.py (ported from syncstack.py) mounted into gateway container. _litellm_sync_loop() runs every 15 min in gateway's asyncio loop. Cron /etc/cron.d/syncstack deleted. Dockerfile.syncstack decommissioned. /opt/stacks/CLAUDE.md updated.

Recently completed (session continued ×2, 2026-05-22)

• crawl4ai MCP fixed: was returning 404 on /mcp (streamable-HTTP). Current 0.8.6 image already has MCP over SSE at /mcp/sse. Added SSE transport support to mcp-gateway/server.py: per-request SSE round-trip with auto-initialize handshake; transport: sse in hardcoded SERVERS + config.json. Gateway health now shows crawl4ai ok; lobe-sync shows 27/27 servers.
• syncstack cron fixed: missing cd /opt/stacks/ai && prefix caused "no configuration file" every 15 min since ~May 18. Fixed. Manual run confirmed 60 models including 5 Claude models; LobeChat recreated.

Recently completed (session continued, 2026-05-22)

• claude-max-bridge /v1/responses endpoint: implemented OpenAI Responses API with previous_response_id chaining. Root cause of "session already in use": Claude CLI leaves session JSONL files in an un-resumable "dequeued" state after each --print run. Fix: maintain conversation history server-side as formatted text; inject into system prompt for continuations; always use --no-session-persistence. Also fixed: assistant turns in --input-format stream-json require content as array-of-blocks not string (JS error otherwise). Assistant content array format also fixed in _parse_messages / _parse_responses_input.
• LiteLLM pass-through for /v1/responses: added pass_through_endpoints entry in /opt/stacks/ai/litellm-config/config.yaml → http://claude-max-bridge:8000/v1/responses. LiteLLM appends its token hash to the response id, so previous_response_id chaining only works when calling the bridge directly (not through LiteLLM).

Recently completed (this session, 2026-05-22)

• Pocket-ID backup: pre-hook on CT 103 runs pocket-id export + scp keys → UNAS staging; services-backup-plan snapshots → jottacloud. Verified.
• Stale NFS vaultwarden dir deleted: /mnt/pve/unas/services/vaultwarden/ removed.
• Nextcloud CIFS → NFS: crash-loop fixed; CT 105 mp0 migrated from /mnt/pve/unas_smb to /mnt/pve/unas; CIFS fstab entry removed.
• Vaultwarden → local zfs: /mnt/pve/unas/services/vaultwarden → /opt/stacks/vaultwarden/data; stale db.sqlite3 deleted; DB already on CT 113 postgres.
• WAL-G fixed: immich_postgres and lobe-postgres backups now succeed. Fixed: missing -u postgres, missing data path, set -e abort, dead shared-postgres reference.
• ComfyUI --async-offload removed: caused CPU fallback + pure-noise output on GGUF img2img with XPU.
• apps/ decommissioned: dead duplicate compose files in /opt/stacks/apps/ renamed .DECOMMISSIONED-2026-05-22.
• Docs consolidated: storage.md, volumes.md, homelab-architecture.md, proxmox-state.md, proxmox-memory-audit.md, ct-inventory.md, README.md all reconciled.

Key system state

Host	IP	Role
nuc (PVE)	192.168.1.20	Proxmox host — SSH gateway to all CTs; jump-menu.sh
CT 109 ops	192.168.1.8	Monitoring — Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty
CT 104 docker	192.168.1.40	Main Docker host — AI/ML, media, ~65 containers
CT 113 db	192.168.1.6	Shared Postgres 17 + WAL-G → Garage S3 (cron 02:00)
CT 110 id	192.168.1.5	Pocket-ID OIDC IdP
CT 103 backrest	192.168.1.3	Backrest — jottacloud via rclone; UNAS at /mnt/pve/unas
CT 111 dev	192.168.1.42	Coder + Gitea
CT 108 zoraxy	192.168.1.4	Reverse proxy + ACME — always confirm before changes

Constraints to remember

• Zoraxy (CT 108): always get explicit user confirmation before any config change.
• Infisical (CT 112): LAN-only, no Zoraxy route — must not be internet-exposed.
• CT 104 has a separate agent reconciling compose files — rename decommissioned files (don't just stop), update /opt/stacks/CLAUDE.md.
• ComfyUI container limit: 24 G. Do not raise without measuring host impact.
• Pocket-ID is SQLite-only — no Postgres migration possible.
• Use *.nuclide.lan aliases when presenting URLs to the user (never bare IPs). Wi-Fi SSID is nuclide.
• Vault edits append to Work/Journal/YYYY/MM/YYYY-MM-DD.md under ## 🤖 Claude edits; never create a parallel folder.