docs/CHANGELOG.md

# Changelog

All notable infrastructure / service / doc changes. Newest first.

## 2026-07-15

### Immich outage — CT 104 root filesystem full (200 G / 200 G, 0 avail)

Cascading failure: Immich 502 from Traefik. Root cause CT 104 root FS at 100% — Postgres crash-looping ("Disk quota exceeded" on pg_stat_tmp), Valkey blocking writes ("MISCONF stop-writes-on-bgsave-error"), immich_server not serving on :12000.

- **Freed 89 GB** via `docker system prune -af --volumes` on CT 104. Post-prune: 183 G / 200 G (92%).
- Docker network state got wedged (orphan endpoint after prune blocked recreate); resolved with `systemctl restart docker` on CT 104 → all 79 containers auto-restored via `restart: unless-stopped`.
- Immich stack recreated via `docker compose up -d --force-recreate`; postgres data preserved on bind mount `/opt/stacks/immich/postgres/`. All 4 containers healthy; external route returns HTTP/2 200.

**Top disk consumers identified (post-cleanup):**
- `mcp-crawl4ai` container writable layer: **40.7 GB** — crawler cache not persisted to a volume. Needs remediation (bind mount or periodic reset).
- `/opt/stacks/shared-db/garage`: 82 GB (Garage S3 data on root FS).
- `/opt/stacks/arr-stack/media`: 27 GB (dormant per CLAUDE.md — safe to delete).
- Docker daemon logs are already rotated (10 MB × 3 per container in `/etc/docker/daemon.json`).

**Alerting added on CT 109:**
- Prometheus rules `filesystem.yml` (Warning >85% / Critical >95% / Full >99%) at `/opt/stacks/monitoring/prometheus/rules/`.
- Grafana-managed alert rule `Filesystem >85%` (uid `cfs5tiefwl3b4a`, folder `Alerts`), routes to existing Gotify contact point via root notification policy. First delivery expected in ~10 min while CT 104 sits at 92%.

## 2026-07-10

### Karakeep + Memos SUNSET (full wipe)

Both services no longer in use. Wiped end-to-end.

- **CT 104**: `docker-compose.yml` in `/opt/stacks/karakeep/` and `/opt/stacks/memos/` renamed to `.sunset-2026-07-10`. Containers stopped + removed (karakeep, karakeep_chrome, karakeep_meilisearch, memos). Local data dir `/opt/stacks/karakeep/localdata/` (279 MB) deleted.
- **CT 104 MCP**: `mcp-memos` (docker-run) container removed. `karakeep-mcp.yml` compose renamed to `.sunset-2026-07-10` (container was already stopped).
- **UNAS**: `/mnt/pve/unas/services/karakeep` (472 MB) and `/mnt/pve/unas/services/memos` (896 KB) deleted.
- **CT 113 Postgres**: `memos` DB and role dropped.
- **Garage S3**: `memos` bucket key access revoked (read/write/owner=false). Bucket delete blocked by an internal quorum decode error — 3.5 MiB / 4 objects orphaned on the internal ID `df4c6b31...`. Flagged for a separate Garage repair pass.
- **CT 103 Backrest**: both paths removed from `services-backup-plan`. Restic `rewrite --forget --exclude` applied to `jottacloud:services` repo: 13 snapshots modified, 591 packs deleted, ~7 GiB freed on JottaCloud.
- **CT 109 Traefik**: `hoarder` and `memos` router+service blocks removed from `remote.yml` (file provider auto-reloaded).
- **CT 110 Pocket-ID**: OIDC clients `karakeep` (`d92f8...`) and `memos` (`62bf4...`) deleted.
- **Cloudflare DNS**: left untouched (wildcard `*.nuclide.systems` covers).
- Docs: rows removed from `services-overview.md`.

## 2026-05-23

### Observability stack (CT 109 "ops")

- **CT 109 ("ops") fully provisioned**: Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring/` running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in `ct-inventory.md`.

- **Loki deployed on CT 109**: log aggregation at `:3100`, 30-day retention, TSDB v13 schema, filesystem storage at `./loki/data/`. `limits_config` set `reject_old_samples_max_age: 168h` (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via `provisioning/datasources/loki.yml`. Grafana Loki datasource UID: `P8E80F9AEF21F6940`.

- **Prometheus remote_write receiver enabled**: `--web.enable-remote-write-receiver` added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.

- **prometheus-pve-exporter deployed**: `prompve/prometheus-pve-exporter:latest` on CT 109; credentials in `./pve-exporter/pve.yml` (chmod 644 — container runs non-root). PVE `monitor@pve` service user + API token `c4be45dc-...` created with `PVEAuditor` role on `/`. Prometheus scrapes `:9221/pve?target=192.168.1.20`. Metrics: per-VM/CT CPU, memory, disk, network I/O.

- **HA prometheus integration added**: `prometheus:` block in HA `configuration.yaml`; HA exposes ~9211 `hass_*` entity-level metrics at `:8123/api/prometheus`. CT 109 Prometheus scrapes with bearer token (HA long-lived token), `scrape_interval: 60s`. This is distinct from Alloy host-level metrics — both needed.

- **Grafana Alloy deployed across all 12 hosts**: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to `http://192.168.1.8:3100/loki/api/v1/push`. Deployment breakdown:
  - **Docker Compose** (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, `group_add: ["999"]` for journal access.
  - **Binary + systemd** (CT 102, 103, 108, nuc): apt install `alloy`, config at `/etc/alloy/config.alloy`, `systemctl enable --now alloy`. Journal-only (no Docker socket on these hosts).
  - **HA add-on** (VM 100): `wymangr/hassos-addons` Grafana Alloy v0.0.8 — config fills in `prometheus_remote_write` to CT 109 + Loki endpoint. Distinct host label `homeassistant`.
  - All configs stored in `/opt/homelab-configs/alloy/` (see homelab-configs repo below).

- **5 Grafana dashboards provisioned**:
  - Node Exporter Full (Grafana ID 1860) — imported for all hosts
  - LiteLLM (custom-built) — `/d/.../litellm`; panels: request rate, error rate, token throughput, model split, p95 latency
  - Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
  - Loki / log explorer — imported
  - Prometheus target health — imported

- **3 Grafana alert rules created** via `/api/v1/provisioning/alert-rules` (folder `alerts-folder`):
  - `Node disk > 85%` — per-instance mountpoint disk usage, 5m window
  - `Prometheus target down` — `count(up == 0) > 0`, 5m window
  - `LiteLLM error rate > 5%` — 10m rate window, 5m pending

### Homarr (CT 109)

- **Homarr v1 deployed**: `ghcr.io/homarr-labs/homarr:latest` on CT 109 `:7575`. Data at `./data/`, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).

- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.

### Pocket-ID OIDC (wired via API)

- **Homarr OIDC**: client `63a94e30` created via Pocket-ID API. Redirect URI `http://192.168.1.8:7575/api/auth/callback/oidc`. Homarr compose updated (`AUTH_PROVIDER=oidc`, `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET`, `AUTH_OIDC_ISSUER`, etc.), force-recreated. SSO active 2026-05-23.

- **Grafana OIDC**: client `92d987d5` created via Pocket-ID API with PKCE enabled. Redirect URI `http://192.168.1.8:3000/login/generic_oauth`. Grafana compose updated with `GF_AUTH_GENERIC_OAUTH_*` vars, force-recreated. SSO active 2026-05-23; role mapping: `admins` group → Admin, else Viewer.

### Arcane + Dozzle migrated to CT 109; headless agents on all hosts

- **Arcane server moved CT 104 → CT 109**: stack at `/opt/stacks/arcane/docker-compose.yml` on CT 109, `APP_URL=http://192.168.1.8:10002`. Zoraxy proxy route for `arcane.nuclide.systems` updated to `192.168.1.8:10002`. Old CT 104 arcane container stopped (renamed decommissioned).
- **Dozzle server moved CT 104 → CT 109**: stack at `/opt/stacks/dozzle/docker-compose.yml` (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).
- **Headless agents deployed on CT 101, 104, 105, 110, 111, 112** via `/opt/stacks/ops-agents/docker-compose.yml` (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated `MANAGER_API_URL` to CT 109 and added Dozzle agent.
- **docs-server migrated CT 111 → CT 109**: `mkdocs.yml` site_url fixed, nav expanded. docs-server now at `http://192.168.1.8:13080`.

### HAOS KVM OOM protection

- **Root cause identified**: karakeep_chrome (CT 104, `gcr.io/zenika-hub/alpine-chrome:123`) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally.
- **shm_size: '512m' added to karakeep chrome service** and force-recreated to reduce Chrome shared memory pressure.
- **oom_score_adj=-300 set on HAOS KVM PID** (protects it from OOM killer — minimum score = hardest to kill).
- **Persistence via systemd timer** (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.

### UNAS Pro config tracking

- **SSH key auth established**: Added nuc's RSA pubkey to UNAS Pro (`192.168.1.31`) `/root/.ssh/authorized_keys` via password SSH. Key-based auth now works from nuc without password.
- **`fkrebs/unas-conf` repo created**: Daily cron (03:20 on nuc) syncs: NFS exports, shares JSON, active exportfs, disk usage, cron config, OS version. Script at `/usr/local/bin/sync-unas-conf`.
- **NFSv4 status**: UNAS kernel already supports NFSv4 (`+4 +4.1 +4.2`) but UniFi OS doesn't configure an `fsid=0` root export. NFSv4 upgrade (docs #2) remains backlog — would require adding an unmanaged drop-in export that firmware updates could overwrite.

### PVE host optimizations

- **LXC startup order set for all CTs**: CT103 (backrest), CT104 (docker), CT105 (nextcloud), CT112 (infisical), CT113 (db) had no startup order defined. Assigned: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full boot order now: DNS(1) → Zoraxy(2) → Pocket-ID(3) → DB(4) → Ops(5) → Infisical(6) → Docker(7) → Backrest(8) → Nextcloud(9) → HAOS(10) → Dev(20). Also set CT109 `onboot=1` (was missing).
- **KSM (Kernel Samepage Merging) enabled**: `echo 1 > /sys/kernel/mm/ksm/run` active now; persisted via `/etc/systemd/system/ksm-enable.service`. Deduplicates identical pages across LXC containers — effective for shared libc/runtime pages. Zero-risk, online change.
- **5-minute critical watchdog timer added**: `homelab-health-quick.timer` on nuc runs `QUICK=1 homelab-health` every 5 min. Checks: all LXC status (auto-starts if stopped), HAOS KVM (notify only), critical containers (Pocket-ID, Prometheus, Grafana, LiteLLM, MCP gateway). Complements the hourly full sweep.
- **Gitea issues filed for backlog PVE opts** (`fkrebs/docs` #2-4): NFS v3→v4 upgrade, ZFS ARC increase to 16 GiB (blocked on CT101 right-sizing), hugepages for HAOS KVM.

### n8n flow tracking

- **Gitea repo `fkrebs/n8n-flows` created**: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push.
- **IDEAS.md**: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).
- **10 Gitea issues filed** on `fkrebs/n8n-flows` for each idea — tracked in memory.
- **Homelab health check (bash/systemd)**: interim implementation on nuc (`/usr/local/bin/homelab-health`, `homelab-health.timer`, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: `docker start` for stopped containers, `docker prune` for CT104 /tmp >80%. Escalates to Gotify (`homelab-health` app, token `Az9NpC-m1jjf571`) for crash loops/disk critical. Migration to n8n tracked as issue #1.

### CT 104 ai repo cleanup

- **lobehub/data/ untracked**: Postgres data dir was accidentally tracked in git. `git rm --cached -r lobehub/data/` + added to `.gitignore`. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).

### Docs

- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).
- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.
- **`services/pocket-id.md` created**: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.

## 2026-05-22

- **Pocket-ID backup wired into Backrest**: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.

- **Vaultwarden data → local zfs**: Moved `/mnt/pve/unas/services/vaultwarden` to `/opt/stacks/vaultwarden/data` (local NVMe). DB was already on CT 113 postgres; only `rsa_key.pem`, `config.json`, `attachments/` remain in the data dir. Stale `db.sqlite3` deleted. NFS copy at `services/vaultwarden/` is now orphaned and can be cleaned up.

- **WAL-G backup script fixed (CT 104)**: All nightly postgres backups were failing silently. Three bugs fixed: (1) `set -e` causing script abort when `shared-postgres` container not found (it was migrated to CT 113); (2) missing `-u postgres` on all `docker exec` calls — WAL-G requires the postgres role; (3) missing explicit data path `/var/lib/postgresql/data` on `backup-push`. `shared-postgres` commented out (CT 113 has its own WAL-G cron). `immich_postgres` and `lobe-postgres` now back up successfully to Garage S3.

- **apps/ directory decommissioned (CT 104)**: `/opt/stacks/apps/` contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db — all without `.env` files, never running. Renamed to `.DECOMMISSIONED-2026-05-22` following established pattern. The canonical stacks at `/opt/stacks/<service>/` are unaffected.

- **ComfyUI `--async-offload` removed**: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from `ai/comfyui.yml`. Current args: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16`. See `infra/proxmox-memory-audit.md`.

- **CT 105 Nextcloud CIFS → NFS**: Nextcloud was crash-looping ("Appdata is not present") because `unas_smb` CIFS mount (`//192.168.1.31/storage`) was not mounted on the host. Root cause: fstab entry was missing `_netdev`; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 `mp0` from `/mnt/pve/unas_smb` to `/mnt/pve/unas` (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart.

- **arr stack started**: `vpn_gluetun`, `shelfarr` (`:13004`), `rdtclient` (via VPN, `:13001`), `prowlarr` (`:13002`), `audiobookshelf` (`:13003`). Fixed empty `labels:` in VPN compose.
- **claude-max-bridge**: new service on CT 104 (`ai-internal`, no host port). FastAPI shim wrapping `claude` CLI as an OpenAI-compatible `/v1/chat/completions` endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: `--strict-mcp-config` (no MCP servers), `--input-format stream-json` (multi-turn), `--include-partial-messages` (real streaming deltas). Bind-mounts `/usr/local/bin/claude` + `/root/.claude` (rw for OAuth token refresh). Source at `git.nuclide.systems/fkrebs/claude-max-bridge`.
- **LiteLLM Claude models**: `claude-sonnet-4-6`, `claude-opus-4-7`, `claude-haiku-4-5` + short aliases (`sonnet`, `opus`, `haiku`) + effort variants (`claude-sonnet-4-6-high`, `claude-opus-4-7-high`) added via `openai/` provider pointing at `claude-max-bridge:8000/v1`. Priced at Anthropic list rates for cost-visibility — actual cost $0 (Max). Bridge features: temperature→effort mapping, `extra_body.effort` override, `extra_body.fallback_model`, `x_claude_cost_usd` and `x_claude_rate_limit` in responses.
- **claude-max-bridge v2**: additional CLI parameter mappings: `extra_body.system_prompt_mode` (`replace`/`append` → `--system-prompt`/`--append-system-prompt`), `extra_body.max_turns` → `--max-turns`, `extra_body.max_budget_usd` → `--max-budget-usd`, `response_format.json_schema` → `--json-schema`, `extra_body.exclude_dynamic_system_prompt_sections` → `--exclude-dynamic-system-prompt-sections`. Build context moved from `/tmp/` to `/opt/stacks/ai/claude-max-bridge/` (survives reboots). Source committed to Gitea at `a781abb`.
- **Shepard MongoDB pinned to 8.0.4**: `mongo:8.0` (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's `7.0.2-5-pve` kernel string (parsed as major version 7 ≥ 6.19). Pinned to `mongo:8.0.4` in `/opt/shepard/infrastructure/docker-compose.yml`. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning.
- **ComfyUI XPU optimisation**: removed `--lowvram` + `--reserve-vram 1.0`; added `--async-offload` (Intel-patched XPU streams, ~10% speedup) + `--force-fp16`; memory limit 20 G → 24 G (FLUX peaks ~22 GB in shared XPU RAM). `--disable-smart-memory` kept since `get_free_memory()` returns the full 58 GB shared pool on Arc — smart eviction is blind to the cgroup limit. See `infra/proxmox-memory-audit.md`.
- **WaveSpeed FBC workflow**: `ApplyFBCacheOnModel(residual_diff_threshold=0.12)` + `FluxGuidance(guidance=3.5)` + `BasicGuider` + `SamplerCustomAdvanced` added to FLUX.1-schnell pipeline. Workflow saved at `/mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json`. Estimated 30–50% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); `torch.compile` node must NOT be used on XPU.
- **LiteLLM benchmark**: `bench.py` at `/opt/stacks/ai/benchmark/` measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (`http://kroki:8000`). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101–143 ms. See `services/llm-benchmark.md`.

- **MCP gateway**: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`.
- **Nextcloud MCP re-enabled**: `cbcoutinho/nextcloud-mcp-server` running in `single_user_basic` mode with `NEXTCLOUD_VERIFY_SSL=false`. App-password generated non-interactively via `occ user:add-app-password`. Gateway patched: (1) `NEXTCLOUD_VERIFY_SSL` / `MCP_DEPLOYMENT_MODE` now baked into hardcoded SERVERS env; (2) startup overlay extended to persist `enabled` flag; (3) proxy route bypasses per-user credential provisioning when `MCP_DEPLOYMENT_MODE=single_user_basic`. 26 servers healthy.
- **CT 109 renamed `observe` → `ops`**: updated `ct-inventory.md`, `homelab-architecture.md`, `proxmox-state.md` (IP changed to `.8` after conflict resolution).
- **CT 101 Shepard disk expanded**: 100 GiB → 500 GiB (ZFS online resize, no restart).
- **CT 112 Infisical marked live**: `ct-inventory.md` and `portmap.md` updated; LAN-only, no Zoraxy route.
- **litellm DB password rotated**: placeholder `litellm_password_here` replaced in both `ai/.env` and `litellm-config/config.yaml` (config.yaml takes precedence over env var). Container restarted, healthy.
- **UNAS migration dumps cleaned**: 7 × `*-migration-20260521.sql` (~492 MB) deleted from `/mnt/pve/unas/dump/`.
- **Arcane OIDC secret rotated**: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and `/opt/stacks/arcane/docker-compose.yml`. Container restarted.
- **n8n OIDC secret rotated**: Pocket-ID DB bcrypt update; secret applied to `/opt/stacks/n8n/docker-compose.yml` (authoritative stack). `/opt/stacks/apps/n8n/` was a dead duplicate (no `.env`, never ran) — compose renamed `.DECOMMISSIONED-2026-05-22`.
- **n8n encryption key rotated**: `N8N_ENCRYPTION_KEY` + `N8N_USER_MANAGEMENT_JWT_SECRET` replaced with 32-byte hex values in `/opt/stacks/n8n/.env` and `/opt/stacks/n8n/data/config`. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered.
- **Backrest repo passwords rotated**: `tapirnase` (weak placeholder) replaced on both `media-repo` and `services-repo` (JottaCloud). Used `restic key add` + `restic key remove`; `config.json` updated; daemon restarted.
- **Obsidian MCP (`mcpvault`)**: new server added. `bitbonsai/mcpvault` bridged via `mcp-proxy` stdio→HTTP in a custom `mcp-obsidian-bridge` image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud `Notizen/` folder at `/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen`, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users.
- **LobeHub MCP sync**: all 27 gateway servers synced into `user_installed_plugins` for both users; stale `papersearch` entry removed; all manifests updated to long-lived `mcp_*` gateway token. Skipped: `crawl4ai` (no `/mcp` endpoint in `unclecode/crawl4ai:latest`), `n8n` (uses own auth).
- **CT 109 plan updated**: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.
- **Portmap**: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added.
- **New docs**: `services/backrest.md`, `services/databases.md`, `services/arcane.md`.

## 2026-05-21

### Docs & docs infrastructure
- **Mermaid diagrams now rendering** in mkdocs Material site — was missing `custom_fences` config in `pymdownx.superfences`. Fixed in `docs-server/mkdocs.yml` and rebuilt container.
- **`site_url` corrected** to `http://192.168.1.111:13080` (docs are LAN-only; no Zoraxy route exists).
- **Docs restructured**: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to `infra/` section. TODO items migrated into CHANGELOG + ideas.
- **Secrets manager design doc** added at `services/secrets-manager.md` — recommends Infisical on a new CT 112.

### Storage / S3
- **`s3.nuclide.systems` Zoraxy route fixed** — `EnableWebsocketCustomHeaders`, `DisableHopByHopHeaderRemoval`, `EnableAutoHTTPS` all set. Garage S3 API now reachable externally.
- **`chat-artifacts` Garage S3 bucket created** — public-read via `https://chat-artifacts.s3.nuclide.systems/<key>`. Credentials at `/root/garage-chat-artifacts.creds` (not git-tracked).

### MCP surface
- **`upload-artifact` MCP server** deployed (port 18011, CT 104). Tools: `upload_text`, `upload_base64`, `list_artifacts`, `get_url`, `delete_artifact`. Registered in gateway (now 24 servers). Source: `git.nuclide.systems/fkrebs/mcp-upload-artifact`.
- **ComfyUI MCP** now auto-uploads completed images to `chat-artifacts` S3 and includes the public URL alongside the inline base64 image.
- **DAYTONA_API_KEY** removed from CT 104 `ai/.env` (Daytona decommissioned 2026-05-20).

### Dev environment
- **`savefig` dotfiles helper** added to `fkrebs/dotfiles` — uploads any file to `chat-artifacts` using `uv run --with boto3`; reads credentials from env vars.
- **Coder `python-uv` template** updated with `CHAT_ARTIFACTS_*` env vars baked in; template pushed to Coder server.

### Gitea repos
- New repos created: `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact`, `home-assistant-config` — all at `git.nuclide.systems/fkrebs/`.
- **HAOS config** (`192.168.1.60`, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see `services/secrets-manager.md` for SSH access notes).

### Planned
- CT 112 "secrets" — Infisical deployment (see `services/secrets-manager.md`)
- Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt)
- Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)
- LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred)

## 2026-05-20

### Auth / Identity (single sign-on lockdown)
- **Pocket-ID migrated CT 104 → CT 110 "id"** (`192.168.1.5:11000`). Authoritative writes now on CT 110; old data dir on CT 104 is a stale snapshot. Compose on CT 104 renamed `.MIGRATED-TO-CT110-2026-05-20`. Zoraxy upstream for `id.nuclide.systems` flipped.
- **Pocket-ID 2.7.0 uses bcrypt for OIDC client secrets** — previously some clients (Gitea, Coder) had raw 64-char SHA-256 hex in `oidc_clients.secret`, which silently failed every token exchange. Regenerated both with proper bcrypt hashes; audit also surfaced an orphaned Daytona client and the empty-secret `vscode` row.
- **Gitea SSO via Pocket-ID** wired with bcrypt fix + correct `Scopes` (`openid,profile,email`) + `two_factor_policy=skip` + `ACCOUNT_LINKING=auto` + `ENABLE_AUTO_REGISTRATION=true` + `USERNAME=preferred_username`. Local password sign-in form disabled (`GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM=false`); legacy OpenID 2.0 button hidden (`GITEA__openid__ENABLE_OPENID_SIGNIN=false`).
- **Coder SSO via Pocket-ID** wired. User row flipped `login_type=password → oidc`. `CODER_DISABLE_PASSWORD_AUTH=true` + `CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE=false`. Workspace-side GitHub external_auth left on (for repo cloning).

### Dev environment (new CT 111 "dev", `192.168.1.42`)
- New LXC stood up with Coder + Gitea + their Postgres companions.
- Templates pushed to Coder: **`python-uv`** (persistent, GPU, code-server, baked LiteLLM + Anthropic env, Claude Code auto-install) and **`mcp-sandbox`** (ephemeral, sci stack pre-baked).
- GPU passthrough confirmed: `/dev/dri/by-path/pci-0000:00:02.0-render` exposed; CT 111-level symlink at `/dev/dri/renderD128` (systemd-tmpfiles) so the Docker provider's path parser is happy.
- `dotfiles` repo created at `fkrebs/dotfiles` on Gitea — oh-my-zsh, LiteLLM model picker (`models.env`), docker alias set, `~/.claude/settings.json` permissions allowlist, global Python `CLAUDE.md`, `.gitignore_global`, VSCodium settings + extensions.txt.
- `mkproj` helper appends `[tool.coder]` workspace+owner block to new `pyproject.toml`, copies pre-commit config + `.gitignore`, installs dev deps, initial commit.

### Reverse proxy
- **Zoraxy WebSocket forwarding enabled** for `dev.`, `git.`, `mcp.nuclide.systems` (`EnableWebsocketCustomHeaders=true` + `HeaderRewriteRules.DisableHopByHopHeaderRemoval=true`). Coder agent / Gitea live updates / streamable-MCP all need this. Backups at `/tmp/zoraxy-backup-2026-05-20/` on CT 108.

### MCP gateway
- **`daytona` removed**, **`coder` added** in `/opt/stacks/ai/mcp-gateway/config.json`. The `morning-briefing` scheduled agent's server list updated. Built a small `coder-mcp` image (multi-stage from `ghcr.io/coder/coder:latest` + `uv:bookworm-slim` + `uv tool install mcp-proxy`) that `coder login`s once then runs `coder exp mcp server` behind streamable-HTTP.
- Coder MCP exposes tools `coder_create_task` + workspace lifecycle from `Coder Agent v1.27.1`.

### Decommissions
- **Daytona** (`/opt/stacks/daytona/` + `/opt/stacks/ai/daytona-mcp.yml` on CT 104) — containers stopped + removed; compose files renamed `.DECOMMISSIONED-2026-05-20`. Coder replaces it.

### Docs / governance
- **`/CLAUDE.md` created on host** — operator doctrine for any agent SSHing in: keep things running, enforce consistency, plan rollback at medium-risk+, services AI-accessible by default, test before declaring done, generate slash commands for recurring tasks, inventory skills/commands/MCPs at session start.
- **`/docs` is now a git repo**, baseline at `04d54ac`.
- **Doc reorg** into `infra/ services/ history/ ideas/`. Traefik guides moved to `history/` (Zoraxy is permanent). `mcp-gateway-requirements.md` and `scrubbing-list-2026-05-17.md` also archived.

## 2026-05-19

- **Garage moved off NFS-on-UNAS to local zfs** on CT 104 (`/opt/stacks/shared-db/garage/{meta,data}`) after a WAL-G outage. Tier-1 (SQLite) databases off NFS as a follow-up; UNAS reserved for bulk media + workspace home dirs.
- **n8n reverted to local SQLite** after a Postgres migration attempt failed.

## 2026-05-17

- MCP gateway design frozen. Implementation in `services/mcp-gateway.md`; design rationale archived as `history/mcp-gateway-requirements.md`.
- Per-stack scrubbing pass (see `history/scrubbing-list-2026-05-17.md` for the inventory at that snapshot).

## 2026-05-16

- **Traefik abandoned**, **Zoraxy is the production reverse proxy**. Two migration guides preserved in `history/` for context only.