Skip to content

# Session Resume

Last updated: 2026-05-26.

## Open items (urgency order)

### 🔴 Critical — security risk or unrecoverable data loss

- [ ] Password rotation: `tapirnase` — shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password, HAOS Terminal & SSH addon password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` §operational rules.
- [ ] Nextcloud Borg passphrase → Vaultwarden — passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See `services/backrest.md` blind spot #4b.
- [ ] Vault secrets sweep — 32 plaintext secrets in Obsidian vault notes (Anthropic key, Gemini, NC AIO passphrase, etc.). Tracker at `Home/Homelab/Secrets to Move.md` in vault. Migrate to Vaultwarden, redact notes, rotate the high-risk ones.

### 🟠 High — known broken / verification needed

- [x] ~~Arcane OIDC secret~~ — fixed 2026-05-23.
- [x] ~~Backrest media-repo / video-projects~~ — all three plans succeeded 2026-05-26 (media 01:30, services 01:04, video-projects 04:00). `video-projects-plan` had never completed before this. ✅
- [x] ~~AdGuard split-horizon DNS~~ — done 2026-05-23.
- [x] ~~WAL-G monitoring~~ — done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`.
- [x] ~~Zoraxy audit~~ — done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`.

### 🟡 Medium — incomplete migrations / cleanup debt

- [ ] Docker disk reclaim — ~17 GB reclaimable on CT 104.
- [ ] Dormant stacks audit — ~6 still to audit (dozzle CT 104 copy, arr-stack, qdrant, etc.). 4 decommissioned 2026-05-23.
- [x] ~~Config-to-git fleet~~ — 8 repos as of 2026-05-24. PVE/CT 108/CT 102/HA/CT 103/CT 109/CT 113/obsidian-vault/obsidian-config. Full table in `infra/config-to-git.md`. Manual step pending: HA Terminal & SSH addon `init_commands` for cron persistence.
- [ ] D-Link hardening — admin UI at `http://dlink.nuclide.lan` (`192.168.1.10`). Trusted-host allowlist + SNMP + TLS.
- [ ] Vaultwarden OIDC SSO — Pocket-ID client ready, env vars not yet set in `/opt/stacks/vaultwarden/.env`.
- [ ] Infisical OIDC config — manual step in Infisical admin at `http://secrets.nuclide.lan:8200/admin` → Settings → OIDC. Client `b2069075-…`, secret `qsANw95zsza0_…`.
- [ ] Infisical secrets migration Phase 2 — import CT104 `.env` keys via `infisical import`.
- [ ] Grafana HAOS dashboard — HA recorder reconfiguration (exclude Prometheus-covered metrics).
- [ ] HAOS SSH key — paste nuc public key in HA terminal.
- [ ] Obsidian config drift — 3 enabled-but-missing plugins (`obsidian42-brat`, `obsidian-tasks-plugin`, `tasks-caldav-sync`). Reinstall or remove from `community-plugins.json`. See `Home/Homelab/Obsidian Plugin Audit.md` in vault.
- [ ] Memos Sync plugin install — config pushed to `obsidian-config`, plugin binary needs Community-Plugins install + paste of pre-baked `data.json`.
- [ ] Immich HTTP/2 Keep-Alive — root-caused 2026-05-24. Workaround live (per-SSID LAN URL). Permanent fix: strip header at Zoraxy. Needs Zoraxy confirmation.

### 🔵 Planned — requires infrastructure or significant effort

- [ ] Second NVMe — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items.
- [ ] Infisical secrets migration — CT 112 deployed (`192.168.1.7:8200`, LAN-only). Phase 1 done. Remaining:
- Phase 2: create project `homelab/ct104`, import `ai/.env` (~55 keys) via `infisical import`, add agent sidecars to compose stacks
- Phase 3: replace hardcoded env vars in Coder `main.tf` (committed to Gitea — security risk)
- Phase 4: OIDC SSO via Pocket-ID (low priority — convenience) See `services/secrets-manager.md`.
- [x] ~~Consolidate syncstack → MCP gateway~~ — done 2026-05-23. `litellm_sync.py` ported from syncstack, mounted into gateway. Gateway runs `run_sync()` every 15 min (120s startup delay). `/etc/cron.d/syncstack` deleted; `Dockerfile.syncstack` → `.DECOMMISSIONED-2026-05-23`; `/opt/stacks/CLAUDE.md` updated.
- [ ] ZFS ARC tuning — raise `zfs_arc_max` from 6.2 GiB → 16 GiB once CT 101 memory right-sized. See `infra/proxmox-state.md` §3.
- [ ] Document ingestion n8n workflow — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`.
- [ ] ComfyUI async queue — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`.
- [ ] Additional MCP servers — Gitea ✓, Paperless ✓, Proxmox-VE ✓ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf.
- [x] ~~Arcane + Dozzle → CT 109~~ — done 2026-05-23.
- [x] ~~crawl4ai SSE StreamConsumed~~ — resolved 2026-05-22.

## Recently completed (2026-05-26)

- Backrest 3-for-3 success: all plans now completing on schedule. `video-projects-plan` recovered from never-completed state.

## Recently completed (2026-05-24)

### Vault + Obsidian

- Vault dedup: `Work 1/` promoted → `Work/`, old `Work/` parked as `Work.stale-backup-2026-05-24/` (1-week safety net). Rescued unique `Journal/Journal.md` before swap. Killed dup `Home/BrainBox.md`, `Home/Templates/`, `Willkommen.md` ×2, `.caldav-sync/`.
- 234 asset-link rewrites across 41 files — collapsed `assets/`, `../assets/`, `../../foo.pdf` → `.assets/…`. Broken links 242 → 9 (rest non-issues: `tel:`, `about:reader?`, `siyuan://`).
- `Home/Homelab/Secrets to Move.md` — 32 plaintext secrets inventoried.
- `Home/Homelab/Obsidian Plugin Audit.md` — 17 installed plugins keep/watch/re-evaluate, 3 config-drift items, 10 new candidates ranked for the stack.
- `Home/Homelab/App Endpoint Checklist.md` — per-service LAN + external URLs + apps-to-update list. Gotchas: clients sticking to old IP, Immich HTTP/2 Keep-Alive bug.
- Daily-note merge policy: Claude edits land in `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## 🤖 Claude edits` (not parallel folder).
- Obsidian Git plugin: wired end-to-end against `fkrebs/obsidian-config`. Auto-commit + push every 60 min, pull on boot, PAT gitignored. Verified working.
- Streamlined-ui CSS snippet pushed to `obsidian-config` (hides ribbon icons for non-clickable plugins, compact status bar, tighter file tree).
- Memos Sync plugin config pre-pushed (binary install pending user).

### Config-to-git fleet expanded to 8 repos

- New: `fkrebs/ct103-conf`, `fkrebs/ct109-conf`, `fkrebs/ct113-conf`, `fkrebs/home-assistant-config`, `fkrebs/obsidian-vault` (rolling 2-commit window), `fkrebs/obsidian-config`.
- All daily 03:00 / 04:00. Pattern documented in `infra/config-to-git.md`.
- Rule articulated: every host with hand-edited config must push daily.

### Networking — `*.nuclide.lan` zone

- 30 AdGuard A-records on CT 102 — per-host (`pve`, `nas`, `unifi`, `dlink`, `backrest`, etc.) + per-service aliases (`immich`, `vault`, `karakeep`, `grafana`, `prometheus`, …).
- Documented in `services/adguard-dns.md`.
- Solves "client registered to old IP" gotcha — when service moves CT, edit one AdGuard rewrite instead of N apps.
- D-Link IP captured: `192.168.1.10`.
- Filed feedback memory: always use `*.nuclide.lan` aliases in URLs, never bare IPs.

### Homarr

- 3 new boards built via SQL: `home` (morning routine, default), `admin` (by-tier), `command` (live-ops iframes + grid). Old "nuclide" board was lost when OIDC user was recreated for password reset.
- Boards owned by `fkrebs@nucli.de`, `is_public=1`. 33 unique apps with ping status.
- Integrations not yet wired (Proxmox / Docker / AdGuard / Gotify widgets — need tokens).

### Pocket-ID

- LAN callback URLs added for Homarr, Grafana, Infisical, Proxmox VE so OIDC works via both bare-IP and `.nuclide.lan` alias paths.

### Other

- Immich Android upload bug diagnosed: HTTP/2 + `Keep-Alive` header conflict. Workaround live (per-SSID LAN URL). Permanent fix pending Zoraxy header strip.
- Nexa #1 filed at https://git.nuclide.systems/fkrebs/nexa/issues/1 — 9 design patterns from Obsidian + Claude session as input for Nexa Phase 1–2.
- Wi-Fi SSID captured: `nuclide`. Used for per-network app routing (Immich, HA Companion).

## Recently completed (2026-05-23, session continued ×4)

- Docs consistency pass: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state.
- Wetty deployed on CT 109 (:4090, LAN-only). `jump-menu.sh` on nuc → SSH jump menu to all 10 hosts.
- MCP servers added: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers.
- Loki dashboard fixed: K8s labels replaced with Docker labels in 14 places.
- WAL-G monitoring: textfile collector on CT 113 + Prometheus alert rules on CT 109.
- CT 103 RAM: 2 GB → 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup).
- Zoraxy WS: `SkipWebSocketOriginCheck` enabled on 13 routes. Arcane upstream corrected to CT 109.
- CT 104 cleanup: daytona, homepage, proxy, streamio decommissioned.
- Paperless-ngx: USERMAP_UID=997 + file permissions fixed, container stable.

## Recently completed (2026-05-23, session continued ×3)

- Homarr board deployed: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: `x_offset`/`y_offset` on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board `nuclide` set as home board.
- Arcane environments fixed: deleted stale shared-token environments; created 6 unique edge environments (shepard, docker, nextcloud, id, dev, secrets) via API with `useApiKey=true`. Each agent updated with unique token. All 8 environments online.
- Arcane API key seeded: `ADMIN_STATIC_API_KEY` from compose was never seeded (DB pre-existed). Inserted via argon2id hash. Key `arc_d3357a65` now functional for automation.
- Infisical SITE_URL fixed: changed from `https://secrets.nuclide.systems` to `http://192.168.1.7:8200` (LAN-only, no Zoraxy route).
- Infisical Pocket-ID client created: client `b2069075-ede2-4251-ad1f-9a62e6a188b3`, callback `http://192.168.1.7:8200/api/v1/sso/oidc/callback`. Manual OIDC config entry still needed in Infisical admin UI.

## Recently completed (2026-05-23, session continued ×2)

- PVE LXC startup order fixed: CT103/104/105/112/113 had no startup order. Set: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full deterministic boot sequence now in place. Also enabled CT109 onboot=1 (was missing).
- KSM enabled on PVE host: kernel samepage merging active + persisted via `ksm-enable.service`. Zero-cost memory deduplication across LXC containers.
- 5-minute critical watchdog deployed: `homelab-health-quick.timer` on nuc; `QUICK=1` mode checks LXC status + critical containers every 5 min. Auto-starts stopped LXCs, notifies Gotify for HAOS KVM issues.
- PVE backlog issues filed: `fkrebs/docs` #2 (NFS v3→v4), #3 (ZFS ARC 16 GiB), #4 (hugepages for KVM).

## Recently completed (2026-05-23, session continued)

- HAOS KVM OOM root-caused and fixed: karakeep_chrome on CT 104 triggered host OOM → killed HAOS KVM twice. Applied: `shm_size: '512m'` to chrome, `oom_score_adj=-300` on KVM PID, systemd timer `protect-haos-kvm.timer` (every 5 min) for persistence.
- Arcane + Dozzle migrated to CT 109: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080).
- Homarr + Grafana OIDC wired: clients created via Pocket-ID API (no manual UI steps). Homarr client `63a94e30`, Grafana client `92d987d5` (PKCE). Both SSO-active.
- n8n flows tracked in Gitea: repo `fkrebs/n8n-flows` with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n).
- CT 104 ai repo cleaned: `lobehub/data/` untracked from git, `.gitignore` fixed, litellm-config + mcp-gateway changes committed.
- zoraxy-conf repo: arcane proxy update committed and pushed (CT 108).

## Recently completed (2026-05-23)

- Config-to-git crons: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy → `zoraxy-conf`), CT 102 (AdGuard → `adguard-conf`), PVE (rsync `/etc/pve` → `pve-conf`). Force-push. All three verified with initial push.
- Syncstack → MCP gateway: `litellm_sync.py` (ported from syncstack.py) mounted into gateway container. `_litellm_sync_loop()` runs every 15 min in gateway's asyncio loop. Cron `/etc/cron.d/syncstack` deleted. `Dockerfile.syncstack` decommissioned. `/opt/stacks/CLAUDE.md` updated.

## Recently completed (session continued ×2, 2026-05-22)

- crawl4ai MCP fixed: was returning 404 on `/mcp` (streamable-HTTP). Current 0.8.6 image already has MCP over SSE at `/mcp/sse`. Added SSE transport support to `mcp-gateway/server.py`: per-request SSE round-trip with auto-initialize handshake; `transport: sse` in hardcoded SERVERS + config.json. Gateway health now shows crawl4ai ok; lobe-sync shows 27/27 servers.
- syncstack cron fixed: missing `cd /opt/stacks/ai &&` prefix caused "no configuration file" every 15 min since ~May 18. Fixed. Manual run confirmed 60 models including 5 Claude models; LobeChat recreated.

## Recently completed (session continued, 2026-05-22)

- claude-max-bridge `/v1/responses` endpoint: implemented OpenAI Responses API with `previous_response_id` chaining. Root cause of "session already in use": Claude CLI leaves session JSONL files in an un-resumable "dequeued" state after each `--print` run. Fix: maintain conversation history server-side as formatted text; inject into system prompt for continuations; always use `--no-session-persistence`. Also fixed: assistant turns in `--input-format stream-json` require `content` as array-of-blocks not string (JS error otherwise). Assistant content array format also fixed in `_parse_messages` / `_parse_responses_input`.
- LiteLLM pass-through for `/v1/responses`: added `pass_through_endpoints` entry in `/opt/stacks/ai/litellm-config/config.yaml` → `http://claude-max-bridge:8000/v1/responses`. LiteLLM appends its token hash to the response `id`, so `previous_response_id` chaining only works when calling the bridge directly (not through LiteLLM).

## Recently completed (this session, 2026-05-22)

- Pocket-ID backup: pre-hook on CT 103 runs `pocket-id export` + scp keys → UNAS staging; `services-backup-plan` snapshots → jottacloud. Verified.
- Stale NFS vaultwarden dir deleted: `/mnt/pve/unas/services/vaultwarden/` removed.
- Nextcloud CIFS → NFS: crash-loop fixed; CT 105 mp0 migrated from `/mnt/pve/unas_smb` to `/mnt/pve/unas`; CIFS fstab entry removed.
- Vaultwarden → local zfs: `/mnt/pve/unas/services/vaultwarden` → `/opt/stacks/vaultwarden/data`; stale `db.sqlite3` deleted; DB already on CT 113 postgres.
- WAL-G fixed: `immich_postgres` and `lobe-postgres` backups now succeed. Fixed: missing `-u postgres`, missing data path, `set -e` abort, dead `shared-postgres` reference.
- ComfyUI `--async-offload` removed: caused CPU fallback + pure-noise output on GGUF img2img with XPU.
- apps/ decommissioned: dead duplicate compose files in `/opt/stacks/apps/` renamed `.DECOMMISSIONED-2026-05-22`.
- Docs consolidated: storage.md, volumes.md, homelab-architecture.md, proxmox-state.md, proxmox-memory-audit.md, ct-inventory.md, README.md all reconciled.

## Key system state

Host | IP | Role
--- | --- | ---
nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs; jump-menu.sh
CT 109 ops | 192.168.1.8 | Monitoring — Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty
CT 104 docker | 192.168.1.40 | Main Docker host — AI/ML, media, ~65 containers
CT 113 db | 192.168.1.6 | Shared Postgres 17 + WAL-G → Garage S3 (cron 02:00)
CT 110 id | 192.168.1.5 | Pocket-ID OIDC IdP
CT 103 backrest | 192.168.1.3 | Backrest — jottacloud via rclone; UNAS at `/mnt/pve/unas`
CT 111 dev | 192.168.1.42 | Coder + Gitea
CT 108 zoraxy | 192.168.1.4 | Reverse proxy + ACME — always confirm before changes

## Constraints to remember

- Zoraxy (CT 108): always get explicit user confirmation before any config change.
- Infisical (CT 112): LAN-only, no Zoraxy route — must not be internet-exposed.
- CT 104 has a separate agent reconciling compose files — rename decommissioned files (don't just stop), update `/opt/stacks/CLAUDE.md`.
- ComfyUI container limit: 24 G. Do not raise without measuring host impact.
- Pocket-ID is SQLite-only — no Postgres migration possible.
- Use `*.nuclide.lan` aliases when presenting URLs to the user (never bare IPs). Wi-Fi SSID is `nuclide`.
- Vault edits append to `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## 🤖 Claude edits`; never create a parallel folder.