docs/CHANGELOG.md

Changelog

All notable infrastructure / service / doc changes. Newest first.

2026-07-15

Immich outage — CT 104 root filesystem full (200 G / 200 G, 0 avail)

Cascading failure: Immich 502 from Traefik. Root cause CT 104 root FS at 100% — Postgres crash-looping ("Disk quota exceeded" on pg_stat_tmp), Valkey blocking writes ("MISCONF stop-writes-on-bgsave-error"), immich_server not serving on :12000.

• Freed 89 GB via docker system prune -af --volumes on CT 104. Post-prune: 183 G / 200 G (92%).
• Docker network state got wedged (orphan endpoint after prune blocked recreate); resolved with systemctl restart docker on CT 104 → all 79 containers auto-restored via restart: unless-stopped.
• Immich stack recreated via docker compose up -d --force-recreate; postgres data preserved on bind mount /opt/stacks/immich/postgres/. All 4 containers healthy; external route returns HTTP/2 200.

Top disk consumers identified (post-cleanup):

• mcp-crawl4ai container writable layer: 40.7 GB — crawler cache not persisted to a volume. Needs remediation (bind mount or periodic reset).
• /opt/stacks/shared-db/garage: 82 GB (Garage S3 data on root FS).
• /opt/stacks/arr-stack/media: 27 GB (dormant per CLAUDE.md — safe to delete).
• Docker daemon logs are already rotated (10 MB × 3 per container in /etc/docker/daemon.json).

Alerting added on CT 109:

• Prometheus rules filesystem.yml (Warning >85% / Critical >95% / Full >99%) at /opt/stacks/monitoring/prometheus/rules/.
• Grafana-managed alert rule Filesystem >85% (uid cfs5tiefwl3b4a, folder Alerts), routes to existing Gotify contact point via root notification policy. First delivery expected in ~10 min while CT 104 sits at 92%.

2026-07-10

Karakeep + Memos SUNSET (full wipe)

Both services no longer in use. Wiped end-to-end.

• CT 104: docker-compose.yml in /opt/stacks/karakeep/ and /opt/stacks/memos/ renamed to .sunset-2026-07-10. Containers stopped + removed (karakeep, karakeep_chrome, karakeep_meilisearch, memos). Local data dir /opt/stacks/karakeep/localdata/ (279 MB) deleted.
• CT 104 MCP: mcp-memos (docker-run) container removed. karakeep-mcp.yml compose renamed to .sunset-2026-07-10 (container was already stopped).
• UNAS: /mnt/pve/unas/services/karakeep (472 MB) and /mnt/pve/unas/services/memos (896 KB) deleted.
• CT 113 Postgres: memos DB and role dropped.
• Garage S3: memos bucket key access revoked (read/write/owner=false). Bucket delete blocked by an internal quorum decode error — 3.5 MiB / 4 objects orphaned on the internal ID df4c6b31.... Flagged for a separate Garage repair pass.
• CT 103 Backrest: both paths removed from services-backup-plan. Restic rewrite --forget --exclude applied to jottacloud:services repo: 13 snapshots modified, 591 packs deleted, ~7 GiB freed on JottaCloud.
• CT 109 Traefik: hoarder and memos router+service blocks removed from remote.yml (file provider auto-reloaded).
• CT 110 Pocket-ID: OIDC clients karakeep (d92f8...) and memos (62bf4...) deleted.
• Cloudflare DNS: left untouched (wildcard *.nuclide.systems covers).
• Docs: rows removed from services-overview.md.

2026-05-23

Observability stack (CT 109 "ops")

• CT 109 ("ops") fully provisioned: Debian 13, Docker 29.5.2. Stack /opt/stacks/monitoring/ running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in ct-inventory.md.

• Loki deployed on CT 109: log aggregation at :3100, 30-day retention, TSDB v13 schema, filesystem storage at ./loki/data/. limits_config set reject_old_samples_max_age: 168h (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via provisioning/datasources/loki.yml. Grafana Loki datasource UID: P8E80F9AEF21F6940.

• Prometheus remote_write receiver enabled: --web.enable-remote-write-receiver added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.

• prometheus-pve-exporter deployed: prompve/prometheus-pve-exporter:latest on CT 109; credentials in ./pve-exporter/pve.yml (chmod 644 — container runs non-root). PVE monitor@pve service user + API token c4be45dc-... created with PVEAuditor role on /. Prometheus scrapes :9221/pve?target=192.168.1.20. Metrics: per-VM/CT CPU, memory, disk, network I/O.

• HA prometheus integration added: prometheus: block in HA configuration.yaml; HA exposes ~9211 hass_* entity-level metrics at :8123/api/prometheus. CT 109 Prometheus scrapes with bearer token (HA long-lived token), scrape_interval: 60s. This is distinct from Alloy host-level metrics — both needed.

• Grafana Alloy deployed across all 12 hosts: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to http://192.168.1.8:3100/loki/api/v1/push. Deployment breakdown:

  • Docker Compose (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, group_add: ["999"] for journal access.
  • Binary + systemd (CT 102, 103, 108, nuc): apt install alloy, config at /etc/alloy/config.alloy, systemctl enable --now alloy. Journal-only (no Docker socket on these hosts).
  • HA add-on (VM 100): wymangr/hassos-addons Grafana Alloy v0.0.8 — config fills in prometheus_remote_write to CT 109 + Loki endpoint. Distinct host label homeassistant.
  • All configs stored in /opt/homelab-configs/alloy/ (see homelab-configs repo below).

• 5 Grafana dashboards provisioned:

  • Node Exporter Full (Grafana ID 1860) — imported for all hosts
  • LiteLLM (custom-built) — /d/.../litellm; panels: request rate, error rate, token throughput, model split, p95 latency
  • Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
  • Loki / log explorer — imported
  • Prometheus target health — imported

• 3 Grafana alert rules created via /api/v1/provisioning/alert-rules (folder alerts-folder):

  • Node disk > 85% — per-instance mountpoint disk usage, 5m window
  • Prometheus target down — count(up == 0) > 0, 5m window
  • LiteLLM error rate > 5% — 10m rate window, 5m pending

Homarr (CT 109)

• Homarr v1 deployed: ghcr.io/homarr-labs/homarr:latest on CT 109 :7575. Data at ./data/, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).

• homelab-configs Gitea repo: private repo fkrebs/homelab-configs at git.nuclide.systems; cloned to /opt/homelab-configs on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at /usr/local/bin/sync-homelab-configs. Commit + push on any diff.

Pocket-ID OIDC (wired via API)

• Homarr OIDC: client 63a94e30 created via Pocket-ID API. Redirect URI http://192.168.1.8:7575/api/auth/callback/oidc. Homarr compose updated (AUTH_PROVIDER=oidc, AUTH_OIDC_CLIENT_ID, AUTH_OIDC_CLIENT_SECRET, AUTH_OIDC_ISSUER, etc.), force-recreated. SSO active 2026-05-23.

• Grafana OIDC: client 92d987d5 created via Pocket-ID API with PKCE enabled. Redirect URI http://192.168.1.8:3000/login/generic_oauth. Grafana compose updated with GF_AUTH_GENERIC_OAUTH_* vars, force-recreated. SSO active 2026-05-23; role mapping: admins group → Admin, else Viewer.

Arcane + Dozzle migrated to CT 109; headless agents on all hosts

• Arcane server moved CT 104 → CT 109: stack at /opt/stacks/arcane/docker-compose.yml on CT 109, APP_URL=http://192.168.1.8:10002. Zoraxy proxy route for arcane.nuclide.systems updated to 192.168.1.8:10002. Old CT 104 arcane container stopped (renamed decommissioned).
• Dozzle server moved CT 104 → CT 109: stack at /opt/stacks/dozzle/docker-compose.yml (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).
• Headless agents deployed on CT 101, 104, 105, 110, 111, 112 via /opt/stacks/ops-agents/docker-compose.yml (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated MANAGER_API_URL to CT 109 and added Dozzle agent.
• docs-server migrated CT 111 → CT 109: mkdocs.yml site_url fixed, nav expanded. docs-server now at http://192.168.1.8:13080.

HAOS KVM OOM protection

• Root cause identified: karakeep_chrome (CT 104, gcr.io/zenika-hub/alpine-chrome:123) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally.
• shm_size: '512m' added to karakeep chrome service and force-recreated to reduce Chrome shared memory pressure.
• oom_score_adj=-300 set on HAOS KVM PID (protects it from OOM killer — minimum score = hardest to kill).
• Persistence via systemd timer (protect-haos-kvm.timer on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.

UNAS Pro config tracking

• SSH key auth established: Added nuc's RSA pubkey to UNAS Pro (192.168.1.31) /root/.ssh/authorized_keys via password SSH. Key-based auth now works from nuc without password.
• fkrebs/unas-conf repo created: Daily cron (03:20 on nuc) syncs: NFS exports, shares JSON, active exportfs, disk usage, cron config, OS version. Script at /usr/local/bin/sync-unas-conf.
• NFSv4 status: UNAS kernel already supports NFSv4 (+4 +4.1 +4.2) but UniFi OS doesn't configure an fsid=0 root export. NFSv4 upgrade (docs #2) remains backlog — would require adding an unmanaged drop-in export that firmware updates could overwrite.

PVE host optimizations

• LXC startup order set for all CTs: CT103 (backrest), CT104 (docker), CT105 (nextcloud), CT112 (infisical), CT113 (db) had no startup order defined. Assigned: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full boot order now: DNS(1) → Zoraxy(2) → Pocket-ID(3) → DB(4) → Ops(5) → Infisical(6) → Docker(7) → Backrest(8) → Nextcloud(9) → HAOS(10) → Dev(20). Also set CT109 onboot=1 (was missing).
• KSM (Kernel Samepage Merging) enabled: echo 1 > /sys/kernel/mm/ksm/run active now; persisted via /etc/systemd/system/ksm-enable.service. Deduplicates identical pages across LXC containers — effective for shared libc/runtime pages. Zero-risk, online change.
• 5-minute critical watchdog timer added: homelab-health-quick.timer on nuc runs QUICK=1 homelab-health every 5 min. Checks: all LXC status (auto-starts if stopped), HAOS KVM (notify only), critical containers (Pocket-ID, Prometheus, Grafana, LiteLLM, MCP gateway). Complements the hourly full sweep.
• Gitea issues filed for backlog PVE opts (fkrebs/docs #2-4): NFS v3→v4 upgrade, ZFS ARC increase to 16 GiB (blocked on CT101 right-sizing), hugepages for HAOS KVM.

n8n flow tracking

• Gitea repo fkrebs/n8n-flows created: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via n8n export:workflow --all → split → commit + push.
• IDEAS.md: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).
• 10 Gitea issues filed on fkrebs/n8n-flows for each idea — tracked in memory.
• Homelab health check (bash/systemd): interim implementation on nuc (/usr/local/bin/homelab-health, homelab-health.timer, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: docker start for stopped containers, docker prune for CT104 /tmp >80%. Escalates to Gotify (homelab-health app, token Az9NpC-m1jjf571) for crash loops/disk critical. Migration to n8n tracked as issue #1.

CT 104 ai repo cleanup

• lobehub/data/ untracked: Postgres data dir was accidentally tracked in git. git rm --cached -r lobehub/data/ + added to .gitignore. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).

Docs

• README.md expanded: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).
• infra/portmap.md updated: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
• services/homelab-architecture.md CT 109 description updated to reflect full stack.
• services/pocket-id.md created: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.

2026-05-22

• Pocket-ID backup wired into Backrest: CT 110 SQLite-only (no Postgres). Pre-backup hook script at /opt/backrest/scripts/pocketid-prestage.sh on CT 103: SSHs to CT 110, runs pocket-id export inside container, docker cps ZIP to host, scp's ZIP + signing keys to /mnt/pve/unas/services/pocketid-backup/. services-backup-plan snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.

• Vaultwarden data → local zfs: Moved /mnt/pve/unas/services/vaultwarden to /opt/stacks/vaultwarden/data (local NVMe). DB was already on CT 113 postgres; only rsa_key.pem, config.json, attachments/ remain in the data dir. Stale db.sqlite3 deleted. NFS copy at services/vaultwarden/ is now orphaned and can be cleaned up.

• WAL-G backup script fixed (CT 104): All nightly postgres backups were failing silently. Three bugs fixed: (1) set -e causing script abort when shared-postgres container not found (it was migrated to CT 113); (2) missing -u postgres on all docker exec calls — WAL-G requires the postgres role; (3) missing explicit data path /var/lib/postgresql/data on backup-push. shared-postgres commented out (CT 113 has its own WAL-G cron). immich_postgres and lobe-postgres now back up successfully to Garage S3.

• apps/ directory decommissioned (CT 104): /opt/stacks/apps/ contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db — all without .env files, never running. Renamed to .DECOMMISSIONED-2026-05-22 following established pattern. The canonical stacks at /opt/stacks/<service>/ are unaffected.

• ComfyUI --async-offload removed: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from ai/comfyui.yml. Current args: --listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16. See infra/proxmox-memory-audit.md.

• CT 105 Nextcloud CIFS → NFS: Nextcloud was crash-looping ("Appdata is not present") because unas_smb CIFS mount (//192.168.1.31/storage) was not mounted on the host. Root cause: fstab entry was missing _netdev; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 mp0 from /mnt/pve/unas_smb to /mnt/pve/unas (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart.

• arr stack started: vpn_gluetun, shelfarr (:13004), rdtclient (via VPN, :13001), prowlarr (:13002), audiobookshelf (:13003). Fixed empty labels: in VPN compose.

• claude-max-bridge: new service on CT 104 (ai-internal, no host port). FastAPI shim wrapping claude CLI as an OpenAI-compatible /v1/chat/completions endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: --strict-mcp-config (no MCP servers), --input-format stream-json (multi-turn), --include-partial-messages (real streaming deltas). Bind-mounts /usr/local/bin/claude + /root/.claude (rw for OAuth token refresh). Source at git.nuclide.systems/fkrebs/claude-max-bridge.

• LiteLLM Claude models: claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5 + short aliases (sonnet, opus, haiku) + effort variants (claude-sonnet-4-6-high, claude-opus-4-7-high) added via openai/ provider pointing at claude-max-bridge:8000/v1. Priced at Anthropic list rates for cost-visibility — actual cost $0 (Max). Bridge features: temperature→effort mapping, extra_body.effort override, extra_body.fallback_model, x_claude_cost_usd and x_claude_rate_limit in responses.

• claude-max-bridge v2: additional CLI parameter mappings: extra_body.system_prompt_mode (replace/append → --system-prompt/--append-system-prompt), extra_body.max_turns → --max-turns, extra_body.max_budget_usd → --max-budget-usd, response_format.json_schema → --json-schema, extra_body.exclude_dynamic_system_prompt_sections → --exclude-dynamic-system-prompt-sections. Build context moved from /tmp/ to /opt/stacks/ai/claude-max-bridge/ (survives reboots). Source committed to Gitea at a781abb.

• Shepard MongoDB pinned to 8.0.4: mongo:8.0 (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's 7.0.2-5-pve kernel string (parsed as major version 7 ≥ 6.19). Pinned to mongo:8.0.4 in /opt/shepard/infrastructure/docker-compose.yml. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning.

• ComfyUI XPU optimisation: removed --lowvram + --reserve-vram 1.0; added --async-offload (Intel-patched XPU streams, ~10% speedup) + --force-fp16; memory limit 20 G → 24 G (FLUX peaks ~22 GB in shared XPU RAM). --disable-smart-memory kept since get_free_memory() returns the full 58 GB shared pool on Arc — smart eviction is blind to the cgroup limit. See infra/proxmox-memory-audit.md.

• WaveSpeed FBC workflow: ApplyFBCacheOnModel(residual_diff_threshold=0.12) + FluxGuidance(guidance=3.5) + BasicGuider + SamplerCustomAdvanced added to FLUX.1-schnell pipeline. Workflow saved at /mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json. Estimated 30–50% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); torch.compile node must NOT be used on XPU.

• LiteLLM benchmark: bench.py at /opt/stacks/ai/benchmark/ measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (http://kroki:8000). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101–143 ms. See services/llm-benchmark.md.

• MCP gateway: added git, gitlab (DLR, --pass-environment fix for mcp-proxy env passthrough bug), paper-search (Docker catalog image replaces inline python:3.12-slim); UNPAYWALL_EMAIL wired into paper-search. Removed duplicate papersearch container. Updated services/mcp-gateway.md.

• Nextcloud MCP re-enabled: cbcoutinho/nextcloud-mcp-server running in single_user_basic mode with NEXTCLOUD_VERIFY_SSL=false. App-password generated non-interactively via occ user:add-app-password. Gateway patched: (1) NEXTCLOUD_VERIFY_SSL / MCP_DEPLOYMENT_MODE now baked into hardcoded SERVERS env; (2) startup overlay extended to persist enabled flag; (3) proxy route bypasses per-user credential provisioning when MCP_DEPLOYMENT_MODE=single_user_basic. 26 servers healthy.

• CT 109 renamed observe → ops: updated ct-inventory.md, homelab-architecture.md, proxmox-state.md (IP changed to .8 after conflict resolution).

• CT 101 Shepard disk expanded: 100 GiB → 500 GiB (ZFS online resize, no restart).

• CT 112 Infisical marked live: ct-inventory.md and portmap.md updated; LAN-only, no Zoraxy route.

• litellm DB password rotated: placeholder litellm_password_here replaced in both ai/.env and litellm-config/config.yaml (config.yaml takes precedence over env var). Container restarted, healthy.

• UNAS migration dumps cleaned: 7 × *-migration-20260521.sql (~492 MB) deleted from /mnt/pve/unas/dump/.

• Arcane OIDC secret rotated: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and /opt/stacks/arcane/docker-compose.yml. Container restarted.

• n8n OIDC secret rotated: Pocket-ID DB bcrypt update; secret applied to /opt/stacks/n8n/docker-compose.yml (authoritative stack). /opt/stacks/apps/n8n/ was a dead duplicate (no .env, never ran) — compose renamed .DECOMMISSIONED-2026-05-22.

• n8n encryption key rotated: N8N_ENCRYPTION_KEY + N8N_USER_MANAGEMENT_JWT_SECRET replaced with 32-byte hex values in /opt/stacks/n8n/.env and /opt/stacks/n8n/data/config. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered.

• Backrest repo passwords rotated: tapirnase (weak placeholder) replaced on both media-repo and services-repo (JottaCloud). Used restic key add + restic key remove; config.json updated; daemon restarted.

• Obsidian MCP (mcpvault): new server added. bitbonsai/mcpvault bridged via mcp-proxy stdio→HTTP in a custom mcp-obsidian-bridge image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud Notizen/ folder at /mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users.

• LobeHub MCP sync: all 27 gateway servers synced into user_installed_plugins for both users; stale papersearch entry removed; all manifests updated to long-lived mcp_* gateway token. Skipped: crawl4ai (no /mcp endpoint in unclecode/crawl4ai:latest), n8n (uses own auth).

• CT 109 plan updated: IP conflict noted (.6 taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.

• Portmap: CT 113 (db, .6) section added; QNAP TS-251D (Klipper/Mainsail) section added.

• New docs: services/backrest.md, services/databases.md, services/arcane.md.

2026-05-21

Docs & docs infrastructure

• Mermaid diagrams now rendering in mkdocs Material site — was missing custom_fences config in pymdownx.superfences. Fixed in docs-server/mkdocs.yml and rebuilt container.
• site_url corrected to http://192.168.1.111:13080 (docs are LAN-only; no Zoraxy route exists).
• Docs restructured: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to infra/ section. TODO items migrated into CHANGELOG + ideas.
• Secrets manager design doc added at services/secrets-manager.md — recommends Infisical on a new CT 112.

Storage / S3

• s3.nuclide.systems Zoraxy route fixed — EnableWebsocketCustomHeaders, DisableHopByHopHeaderRemoval, EnableAutoHTTPS all set. Garage S3 API now reachable externally.
• chat-artifacts Garage S3 bucket created — public-read via https://chat-artifacts.s3.nuclide.systems/<key>. Credentials at /root/garage-chat-artifacts.creds (not git-tracked).

MCP surface

• upload-artifact MCP server deployed (port 18011, CT 104). Tools: upload_text, upload_base64, list_artifacts, get_url, delete_artifact. Registered in gateway (now 24 servers). Source: git.nuclide.systems/fkrebs/mcp-upload-artifact.
• ComfyUI MCP now auto-uploads completed images to chat-artifacts S3 and includes the public URL alongside the inline base64 image.
• DAYTONA_API_KEY removed from CT 104 ai/.env (Daytona decommissioned 2026-05-20).

Dev environment

• savefig dotfiles helper added to fkrebs/dotfiles — uploads any file to chat-artifacts using uv run --with boto3; reads credentials from env vars.
• Coder python-uv template updated with CHAT_ARTIFACTS_* env vars baked in; template pushed to Coder server.

Gitea repos

• New repos created: mcp-comfyui, mcp-docling, mcp-shepard, mcp-upload-artifact, home-assistant-config — all at git.nuclide.systems/fkrebs/.
• HAOS config (192.168.1.60, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see services/secrets-manager.md for SSH access notes).

Planned

• CT 112 "secrets" — Infisical deployment (see services/secrets-manager.md)
• Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt)
• Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)
• LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred)

2026-05-20

Auth / Identity (single sign-on lockdown)

• Pocket-ID migrated CT 104 → CT 110 "id" (192.168.1.5:11000). Authoritative writes now on CT 110; old data dir on CT 104 is a stale snapshot. Compose on CT 104 renamed .MIGRATED-TO-CT110-2026-05-20. Zoraxy upstream for id.nuclide.systems flipped.
• Pocket-ID 2.7.0 uses bcrypt for OIDC client secrets — previously some clients (Gitea, Coder) had raw 64-char SHA-256 hex in oidc_clients.secret, which silently failed every token exchange. Regenerated both with proper bcrypt hashes; audit also surfaced an orphaned Daytona client and the empty-secret vscode row.
• Gitea SSO via Pocket-ID wired with bcrypt fix + correct Scopes (openid,profile,email) + two_factor_policy=skip + ACCOUNT_LINKING=auto + ENABLE_AUTO_REGISTRATION=true + USERNAME=preferred_username. Local password sign-in form disabled (GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM=false); legacy OpenID 2.0 button hidden (GITEA__openid__ENABLE_OPENID_SIGNIN=false).
• Coder SSO via Pocket-ID wired. User row flipped login_type=password → oidc. CODER_DISABLE_PASSWORD_AUTH=true + CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE=false. Workspace-side GitHub external_auth left on (for repo cloning).

Dev environment (new CT 111 "dev", 192.168.1.42)

• New LXC stood up with Coder + Gitea + their Postgres companions.
• Templates pushed to Coder: python-uv (persistent, GPU, code-server, baked LiteLLM + Anthropic env, Claude Code auto-install) and mcp-sandbox (ephemeral, sci stack pre-baked).
• GPU passthrough confirmed: /dev/dri/by-path/pci-0000:00:02.0-render exposed; CT 111-level symlink at /dev/dri/renderD128 (systemd-tmpfiles) so the Docker provider's path parser is happy.
• dotfiles repo created at fkrebs/dotfiles on Gitea — oh-my-zsh, LiteLLM model picker (models.env), docker alias set, ~/.claude/settings.json permissions allowlist, global Python CLAUDE.md, .gitignore_global, VSCodium settings + extensions.txt.
• mkproj helper appends [tool.coder] workspace+owner block to new pyproject.toml, copies pre-commit config + .gitignore, installs dev deps, initial commit.

Reverse proxy

• Zoraxy WebSocket forwarding enabled for dev., git., mcp.nuclide.systems (EnableWebsocketCustomHeaders=true + HeaderRewriteRules.DisableHopByHopHeaderRemoval=true). Coder agent / Gitea live updates / streamable-MCP all need this. Backups at /tmp/zoraxy-backup-2026-05-20/ on CT 108.

MCP gateway

• daytona removed, coder added in /opt/stacks/ai/mcp-gateway/config.json. The morning-briefing scheduled agent's server list updated. Built a small coder-mcp image (multi-stage from ghcr.io/coder/coder:latest + uv:bookworm-slim + uv tool install mcp-proxy) that coder logins once then runs coder exp mcp server behind streamable-HTTP.
• Coder MCP exposes tools coder_create_task + workspace lifecycle from Coder Agent v1.27.1.

Decommissions

• Daytona (/opt/stacks/daytona/ + /opt/stacks/ai/daytona-mcp.yml on CT 104) — containers stopped + removed; compose files renamed .DECOMMISSIONED-2026-05-20. Coder replaces it.

Docs / governance

• /CLAUDE.md created on host — operator doctrine for any agent SSHing in: keep things running, enforce consistency, plan rollback at medium-risk+, services AI-accessible by default, test before declaring done, generate slash commands for recurring tasks, inventory skills/commands/MCPs at session start.
• /docs is now a git repo, baseline at 04d54ac.
• Doc reorg into infra/ services/ history/ ideas/. Traefik guides moved to history/ (Zoraxy is permanent). mcp-gateway-requirements.md and scrubbing-list-2026-05-17.md also archived.

2026-05-19

• Garage moved off NFS-on-UNAS to local zfs on CT 104 (/opt/stacks/shared-db/garage/{meta,data}) after a WAL-G outage. Tier-1 (SQLite) databases off NFS as a follow-up; UNAS reserved for bulk media + workspace home dirs.
• n8n reverted to local SQLite after a Postgres migration attempt failed.

2026-05-17

• MCP gateway design frozen. Implementation in services/mcp-gateway.md; design rationale archived as history/mcp-gateway-requirements.md.
• Per-stack scrubbing pass (see history/scrubbing-list-2026-05-17.md for the inventory at that snapshot).

2026-05-16

• Traefik abandoned, Zoraxy is the production reverse proxy. Two migration guides preserved in history/ for context only.