Skip to content

# Changelog

All notable infrastructure / service / doc changes. Newest first.

## 2026-05-23

### Observability stack (CT 109 "ops")

- CT 109 ("ops") fully provisioned: Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring/` running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in `ct-inventory.md`.

- Loki deployed on CT 109: log aggregation at `:3100`, 30-day retention, TSDB v13 schema, filesystem storage at `./loki/data/`. `limits_config` set `reject_old_samples_max_age: 168h` (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via `provisioning/datasources/loki.yml`. Grafana Loki datasource UID: `P8E80F9AEF21F6940`.

- Prometheus remote_write receiver enabled: `--web.enable-remote-write-receiver` added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.

- prometheus-pve-exporter deployed: `prompve/prometheus-pve-exporter:latest` on CT 109; credentials in `./pve-exporter/pve.yml` (chmod 644 — container runs non-root). PVE `monitor@pve` service user + API token `c4be45dc-...` created with `PVEAuditor` role on `/`. Prometheus scrapes `:9221/pve?target=192.168.1.20`. Metrics: per-VM/CT CPU, memory, disk, network I/O.

- HA prometheus integration added: `prometheus:` block in HA `configuration.yaml`; HA exposes ~9211 `hass_*` entity-level metrics at `:8123/api/prometheus`. CT 109 Prometheus scrapes with bearer token (HA long-lived token), `scrape_interval: 60s`. This is distinct from Alloy host-level metrics — both needed.

- Grafana Alloy deployed across all 12 hosts: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to `http://192.168.1.8:3100/loki/api/v1/push`. Deployment breakdown:

- Docker Compose (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, `group_add: ["999"]` for journal access.
- Binary + systemd (CT 102, 103, 108, nuc): apt install `alloy`, config at `/etc/alloy/config.alloy`, `systemctl enable --now alloy`. Journal-only (no Docker socket on these hosts).
- HA add-on (VM 100): `wymangr/hassos-addons` Grafana Alloy v0.0.8 — config fills in `prometheus_remote_write` to CT 109 + Loki endpoint. Distinct host label `homeassistant`.

- All configs stored in `/opt/homelab-configs/alloy/` (see homelab-configs repo below).

- 5 Grafana dashboards provisioned:

- Node Exporter Full (Grafana ID 1860) — imported for all hosts
- LiteLLM (custom-built) — `/d/.../litellm`; panels: request rate, error rate, token throughput, model split, p95 latency
- Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
- Loki / log explorer — imported

- Prometheus target health — imported

- 3 Grafana alert rules created via `/api/v1/provisioning/alert-rules` (folder `alerts-folder`):

- `Node disk > 85%` — per-instance mountpoint disk usage, 5m window
- `Prometheus target down` — `count(up == 0) > 0`, 5m window
- `LiteLLM error rate > 5%` — 10m rate window, 5m pending

### Homarr (CT 109)

- Homarr v1 deployed: `ghcr.io/homarr-labs/homarr:latest` on CT 109 `:7575`. Data at `./data/`, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).

- homelab-configs Gitea repo: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.

### Pocket-ID OIDC (wired via API)

- Homarr OIDC: client `63a94e30` created via Pocket-ID API. Redirect URI `http://192.168.1.8:7575/api/auth/callback/oidc`. Homarr compose updated (`AUTH_PROVIDER=oidc`, `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET`, `AUTH_OIDC_ISSUER`, etc.), force-recreated. SSO active 2026-05-23.

- Grafana OIDC: client `92d987d5` created via Pocket-ID API with PKCE enabled. Redirect URI `http://192.168.1.8:3000/login/generic_oauth`. Grafana compose updated with `GF_AUTH_GENERIC_OAUTH_*` vars, force-recreated. SSO active 2026-05-23; role mapping: `admins` group → Admin, else Viewer.

### Arcane + Dozzle migrated to CT 109; headless agents on all hosts

- Arcane server moved CT 104 → CT 109: stack at `/opt/stacks/arcane/docker-compose.yml` on CT 109, `APP_URL=http://192.168.1.8:10002`. Zoraxy proxy route for `arcane.nuclide.systems` updated to `192.168.1.8:10002`. Old CT 104 arcane container stopped (renamed decommissioned).
- Dozzle server moved CT 104 → CT 109: stack at `/opt/stacks/dozzle/docker-compose.yml` (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).
- Headless agents deployed on CT 101, 104, 105, 110, 111, 112 via `/opt/stacks/ops-agents/docker-compose.yml` (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated `MANAGER_API_URL` to CT 109 and added Dozzle agent.
- docs-server migrated CT 111 → CT 109: `mkdocs.yml` site_url fixed, nav expanded. docs-server now at `http://192.168.1.8:13080`.

### HAOS KVM OOM protection

- Root cause identified: karakeep_chrome (CT 104, `gcr.io/zenika-hub/alpine-chrome:123`) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally.
- shm_size: '512m' added to karakeep chrome service and force-recreated to reduce Chrome shared memory pressure.
- oom_score_adj=-300 set on HAOS KVM PID (protects it from OOM killer — minimum score = hardest to kill).
- Persistence via systemd timer (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.

### UNAS Pro config tracking

- SSH key auth established: Added nuc's RSA pubkey to UNAS Pro (`192.168.1.31`) `/root/.ssh/authorized_keys` via password SSH. Key-based auth now works from nuc without password.
- `fkrebs/unas-conf` repo created: Daily cron (03:20 on nuc) syncs: NFS exports, shares JSON, active exportfs, disk usage, cron config, OS version. Script at `/usr/local/bin/sync-unas-conf`.
- NFSv4 status: UNAS kernel already supports NFSv4 (`+4 +4.1 +4.2`) but UniFi OS doesn't configure an `fsid=0` root export. NFSv4 upgrade (docs #2) remains backlog — would require adding an unmanaged drop-in export that firmware updates could overwrite.

### PVE host optimizations

- LXC startup order set for all CTs: CT103 (backrest), CT104 (docker), CT105 (nextcloud), CT112 (infisical), CT113 (db) had no startup order defined. Assigned: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full boot order now: DNS(1) → Zoraxy(2) → Pocket-ID(3) → DB(4) → Ops(5) → Infisical(6) → Docker(7) → Backrest(8) → Nextcloud(9) → HAOS(10) → Dev(20). Also set CT109 `onboot=1` (was missing).
- KSM (Kernel Samepage Merging) enabled: `echo 1 > /sys/kernel/mm/ksm/run` active now; persisted via `/etc/systemd/system/ksm-enable.service`. Deduplicates identical pages across LXC containers — effective for shared libc/runtime pages. Zero-risk, online change.
- 5-minute critical watchdog timer added: `homelab-health-quick.timer` on nuc runs `QUICK=1 homelab-health` every 5 min. Checks: all LXC status (auto-starts if stopped), HAOS KVM (notify only), critical containers (Pocket-ID, Prometheus, Grafana, LiteLLM, MCP gateway). Complements the hourly full sweep.
- Gitea issues filed for backlog PVE opts (`fkrebs/docs` #2-4): NFS v3→v4 upgrade, ZFS ARC increase to 16 GiB (blocked on CT101 right-sizing), hugepages for HAOS KVM.

### n8n flow tracking

- Gitea repo `fkrebs/n8n-flows` created: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push.
- IDEAS.md: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).
- 10 Gitea issues filed on `fkrebs/n8n-flows` for each idea — tracked in memory.
- Homelab health check (bash/systemd): interim implementation on nuc (`/usr/local/bin/homelab-health`, `homelab-health.timer`, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: `docker start` for stopped containers, `docker prune` for CT104 /tmp >80%. Escalates to Gotify (`homelab-health` app, token `Az9NpC-m1jjf571`) for crash loops/disk critical. Migration to n8n tracked as issue #1.

### CT 104 ai repo cleanup

- lobehub/data/ untracked: Postgres data dir was accidentally tracked in git. `git rm --cached -r lobehub/data/` + added to `.gitignore`. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).

### Docs

- `README.md` expanded: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).
- `infra/portmap.md` updated: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- `services/homelab-architecture.md` CT 109 description updated to reflect full stack.
- `services/pocket-id.md` created: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.

## 2026-05-22

- Pocket-ID backup wired into Backrest: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.

- Vaultwarden data → local zfs: Moved `/mnt/pve/unas/services/vaultwarden` to `/opt/stacks/vaultwarden/data` (local NVMe). DB was already on CT 113 postgres; only `rsa_key.pem`, `config.json`, `attachments/` remain in the data dir. Stale `db.sqlite3` deleted. NFS copy at `services/vaultwarden/` is now orphaned and can be cleaned up.

- WAL-G backup script fixed (CT 104): All nightly postgres backups were failing silently. Three bugs fixed: (1) `set -e` causing script abort when `shared-postgres` container not found (it was migrated to CT 113); (2) missing `-u postgres` on all `docker exec` calls — WAL-G requires the postgres role; (3) missing explicit data path `/var/lib/postgresql/data` on `backup-push`. `shared-postgres` commented out (CT 113 has its own WAL-G cron). `immich_postgres` and `lobe-postgres` now back up successfully to Garage S3.

- apps/ directory decommissioned (CT 104): `/opt/stacks/apps/` contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db — all without `.env` files, never running. Renamed to `.DECOMMISSIONED-2026-05-22` following established pattern. The canonical stacks at `/opt/stacks/<service>/` are unaffected.

- ComfyUI `--async-offload` removed: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from `ai/comfyui.yml`. Current args: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16`. See `infra/proxmox-memory-audit.md`.

- CT 105 Nextcloud CIFS → NFS: Nextcloud was crash-looping ("Appdata is not present") because `unas_smb` CIFS mount (`//192.168.1.31/storage`) was not mounted on the host. Root cause: fstab entry was missing `_netdev`; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 `mp0` from `/mnt/pve/unas_smb` to `/mnt/pve/unas` (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart.

- arr stack started: `vpn_gluetun`, `shelfarr` (`:13004`), `rdtclient` (via VPN, `:13001`), `prowlarr` (`:13002`), `audiobookshelf` (`:13003`). Fixed empty `labels:` in VPN compose.

- claude-max-bridge: new service on CT 104 (`ai-internal`, no host port). FastAPI shim wrapping `claude` CLI as an OpenAI-compatible `/v1/chat/completions` endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: `--strict-mcp-config` (no MCP servers), `--input-format stream-json` (multi-turn), `--include-partial-messages` (real streaming deltas). Bind-mounts `/usr/local/bin/claude` + `/root/.claude` (rw for OAuth token refresh). Source at `git.nuclide.systems/fkrebs/claude-max-bridge`.
- LiteLLM Claude models: `claude-sonnet-4-6`, `claude-opus-4-7`, `claude-haiku-4-5` + short aliases (`sonnet`, `opus`, `haiku`) + effort variants (`claude-sonnet-4-6-high`, `claude-opus-4-7-high`) added via `openai/` provider pointing at `claude-max-bridge:8000/v1`. Priced at Anthropic list rates for cost-visibility — actual cost $0 (Max). Bridge features: temperature→effort mapping, `extra_body.effort` override, `extra_body.fallback_model`, `x_claude_cost_usd` and `x_claude_rate_limit` in responses.
- claude-max-bridge v2: additional CLI parameter mappings: `extra_body.system_prompt_mode` (`replace`/`append` → `--system-prompt`/`--append-system-prompt`), `extra_body.max_turns` → `--max-turns`, `extra_body.max_budget_usd` → `--max-budget-usd`, `response_format.json_schema` → `--json-schema`, `extra_body.exclude_dynamic_system_prompt_sections` → `--exclude-dynamic-system-prompt-sections`. Build context moved from `/tmp/` to `/opt/stacks/ai/claude-max-bridge/` (survives reboots). Source committed to Gitea at `a781abb`.
- Shepard MongoDB pinned to 8.0.4: `mongo:8.0` (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's `7.0.2-5-pve` kernel string (parsed as major version 7 ≥ 6.19). Pinned to `mongo:8.0.4` in `/opt/shepard/infrastructure/docker-compose.yml`. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning.
- ComfyUI XPU optimisation: removed `--lowvram` + `--reserve-vram 1.0`; added `--async-offload` (Intel-patched XPU streams, ~10% speedup) + `--force-fp16`; memory limit 20 G → 24 G (FLUX peaks ~22 GB in shared XPU RAM). `--disable-smart-memory` kept since `get_free_memory()` returns the full 58 GB shared pool on Arc — smart eviction is blind to the cgroup limit. See `infra/proxmox-memory-audit.md`.
- WaveSpeed FBC workflow: `ApplyFBCacheOnModel(residual_diff_threshold=0.12)` + `FluxGuidance(guidance=3.5)` + `BasicGuider` + `SamplerCustomAdvanced` added to FLUX.1-schnell pipeline. Workflow saved at `/mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json`. Estimated 30–50% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); `torch.compile` node must NOT be used on XPU.

- LiteLLM benchmark: `bench.py` at `/opt/stacks/ai/benchmark/` measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (`http://kroki:8000`). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101–143 ms. See `services/llm-benchmark.md`.

- MCP gateway: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`.

- Nextcloud MCP re-enabled: `cbcoutinho/nextcloud-mcp-server` running in `single_user_basic` mode with `NEXTCLOUD_VERIFY_SSL=false`. App-password generated non-interactively via `occ user:add-app-password`. Gateway patched: (1) `NEXTCLOUD_VERIFY_SSL` / `MCP_DEPLOYMENT_MODE` now baked into hardcoded SERVERS env; (2) startup overlay extended to persist `enabled` flag; (3) proxy route bypasses per-user credential provisioning when `MCP_DEPLOYMENT_MODE=single_user_basic`. 26 servers healthy.
- CT 109 renamed `observe` → `ops`: updated `ct-inventory.md`, `homelab-architecture.md`, `proxmox-state.md` (IP changed to `.8` after conflict resolution).
- CT 101 Shepard disk expanded: 100 GiB → 500 GiB (ZFS online resize, no restart).
- CT 112 Infisical marked live: `ct-inventory.md` and `portmap.md` updated; LAN-only, no Zoraxy route.
- litellm DB password rotated: placeholder `litellm_password_here` replaced in both `ai/.env` and `litellm-config/config.yaml` (config.yaml takes precedence over env var). Container restarted, healthy.
- UNAS migration dumps cleaned: 7 × `*-migration-20260521.sql` (~492 MB) deleted from `/mnt/pve/unas/dump/`.
- Arcane OIDC secret rotated: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and `/opt/stacks/arcane/docker-compose.yml`. Container restarted.
- n8n OIDC secret rotated: Pocket-ID DB bcrypt update; secret applied to `/opt/stacks/n8n/docker-compose.yml` (authoritative stack). `/opt/stacks/apps/n8n/` was a dead duplicate (no `.env`, never ran) — compose renamed `.DECOMMISSIONED-2026-05-22`.
- n8n encryption key rotated: `N8N_ENCRYPTION_KEY` + `N8N_USER_MANAGEMENT_JWT_SECRET` replaced with 32-byte hex values in `/opt/stacks/n8n/.env` and `/opt/stacks/n8n/data/config`. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered.
- Backrest repo passwords rotated: `tapirnase` (weak placeholder) replaced on both `media-repo` and `services-repo` (JottaCloud). Used `restic key add` + `restic key remove`; `config.json` updated; daemon restarted.
- Obsidian MCP (`mcpvault`): new server added. `bitbonsai/mcpvault` bridged via `mcp-proxy` stdio→HTTP in a custom `mcp-obsidian-bridge` image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud `Notizen/` folder at `/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen`, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users.
- LobeHub MCP sync: all 27 gateway servers synced into `user_installed_plugins` for both users; stale `papersearch` entry removed; all manifests updated to long-lived `mcp_*` gateway token. Skipped: `crawl4ai` (no `/mcp` endpoint in `unclecode/crawl4ai:latest`), `n8n` (uses own auth).
- CT 109 plan updated: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.
- Portmap: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added.
- New docs: `services/backrest.md`, `services/databases.md`, `services/arcane.md`.

## 2026-05-21

### Docs & docs infrastructure

- Mermaid diagrams now rendering in mkdocs Material site — was missing `custom_fences` config in `pymdownx.superfences`. Fixed in `docs-server/mkdocs.yml` and rebuilt container.
- `site_url` corrected to `http://192.168.1.111:13080` (docs are LAN-only; no Zoraxy route exists).
- Docs restructured: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to `infra/` section. TODO items migrated into CHANGELOG + ideas.
- Secrets manager design doc added at `services/secrets-manager.md` — recommends Infisical on a new CT 112.

### Storage / S3

- `s3.nuclide.systems` Zoraxy route fixed — `EnableWebsocketCustomHeaders`, `DisableHopByHopHeaderRemoval`, `EnableAutoHTTPS` all set. Garage S3 API now reachable externally.
- `chat-artifacts` Garage S3 bucket created — public-read via `https://chat-artifacts.s3.nuclide.systems/<key>`. Credentials at `/root/garage-chat-artifacts.creds` (not git-tracked).

### MCP surface

- `upload-artifact` MCP server deployed (port 18011, CT 104). Tools: `upload_text`, `upload_base64`, `list_artifacts`, `get_url`, `delete_artifact`. Registered in gateway (now 24 servers). Source: `git.nuclide.systems/fkrebs/mcp-upload-artifact`.
- ComfyUI MCP now auto-uploads completed images to `chat-artifacts` S3 and includes the public URL alongside the inline base64 image.
- DAYTONA_API_KEY removed from CT 104 `ai/.env` (Daytona decommissioned 2026-05-20).

### Dev environment

- `savefig` dotfiles helper added to `fkrebs/dotfiles` — uploads any file to `chat-artifacts` using `uv run --with boto3`; reads credentials from env vars.
- Coder `python-uv` template updated with `CHAT_ARTIFACTS_*` env vars baked in; template pushed to Coder server.

### Gitea repos

- New repos created: `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact`, `home-assistant-config` — all at `git.nuclide.systems/fkrebs/`.
- HAOS config (`192.168.1.60`, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see `services/secrets-manager.md` for SSH access notes).

### Planned

- CT 112 "secrets" — Infisical deployment (see `services/secrets-manager.md`)
- Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt)
- Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)
- LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred)

## 2026-05-20

### Auth / Identity (single sign-on lockdown)

- Pocket-ID migrated CT 104 → CT 110 "id" (`192.168.1.5:11000`). Authoritative writes now on CT 110; old data dir on CT 104 is a stale snapshot. Compose on CT 104 renamed `.MIGRATED-TO-CT110-2026-05-20`. Zoraxy upstream for `id.nuclide.systems` flipped.
- Pocket-ID 2.7.0 uses bcrypt for OIDC client secrets — previously some clients (Gitea, Coder) had raw 64-char SHA-256 hex in `oidc_clients.secret`, which silently failed every token exchange. Regenerated both with proper bcrypt hashes; audit also surfaced an orphaned Daytona client and the empty-secret `vscode` row.
- Gitea SSO via Pocket-ID wired with bcrypt fix + correct `Scopes` (`openid,profile,email`) + `two_factor_policy=skip` + `ACCOUNT_LINKING=auto` + `ENABLE_AUTO_REGISTRATION=true` + `USERNAME=preferred_username`. Local password sign-in form disabled (`GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM=false`); legacy OpenID 2.0 button hidden (`GITEA__openid__ENABLE_OPENID_SIGNIN=false`).
- Coder SSO via Pocket-ID wired. User row flipped `login_type=password → oidc`. `CODER_DISABLE_PASSWORD_AUTH=true` + `CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE=false`. Workspace-side GitHub external_auth left on (for repo cloning).

### Dev environment (new CT 111 "dev", `192.168.1.42`)

- New LXC stood up with Coder + Gitea + their Postgres companions.
- Templates pushed to Coder: `python-uv` (persistent, GPU, code-server, baked LiteLLM + Anthropic env, Claude Code auto-install) and `mcp-sandbox` (ephemeral, sci stack pre-baked).
- GPU passthrough confirmed: `/dev/dri/by-path/pci-0000:00:02.0-render` exposed; CT 111-level symlink at `/dev/dri/renderD128` (systemd-tmpfiles) so the Docker provider's path parser is happy.
- `dotfiles` repo created at `fkrebs/dotfiles` on Gitea — oh-my-zsh, LiteLLM model picker (`models.env`), docker alias set, `~/.claude/settings.json` permissions allowlist, global Python `CLAUDE.md`, `.gitignore_global`, VSCodium settings + extensions.txt.
- `mkproj` helper appends `[tool.coder]` workspace+owner block to new `pyproject.toml`, copies pre-commit config + `.gitignore`, installs dev deps, initial commit.

### Reverse proxy

- Zoraxy WebSocket forwarding enabled for `dev.`, `git.`, `mcp.nuclide.systems` (`EnableWebsocketCustomHeaders=true` + `HeaderRewriteRules.DisableHopByHopHeaderRemoval=true`). Coder agent / Gitea live updates / streamable-MCP all need this. Backups at `/tmp/zoraxy-backup-2026-05-20/` on CT 108.

### MCP gateway

- `daytona` removed, `coder` added in `/opt/stacks/ai/mcp-gateway/config.json`. The `morning-briefing` scheduled agent's server list updated. Built a small `coder-mcp` image (multi-stage from `ghcr.io/coder/coder:latest` + `uv:bookworm-slim` + `uv tool install mcp-proxy`) that `coder login`s once then runs `coder exp mcp server` behind streamable-HTTP.
- Coder MCP exposes tools `coder_create_task` + workspace lifecycle from `Coder Agent v1.27.1`.

### Decommissions

- Daytona (`/opt/stacks/daytona/` + `/opt/stacks/ai/daytona-mcp.yml` on CT 104) — containers stopped + removed; compose files renamed `.DECOMMISSIONED-2026-05-20`. Coder replaces it.

### Docs / governance

- `/CLAUDE.md` created on host — operator doctrine for any agent SSHing in: keep things running, enforce consistency, plan rollback at medium-risk+, services AI-accessible by default, test before declaring done, generate slash commands for recurring tasks, inventory skills/commands/MCPs at session start.
- `/docs` is now a git repo, baseline at `04d54ac`.
- Doc reorg into `infra/ services/ history/ ideas/`. Traefik guides moved to `history/` (Zoraxy is permanent). `mcp-gateway-requirements.md` and `scrubbing-list-2026-05-17.md` also archived.

## 2026-05-19

- Garage moved off NFS-on-UNAS to local zfs on CT 104 (`/opt/stacks/shared-db/garage/{meta,data}`) after a WAL-G outage. Tier-1 (SQLite) databases off NFS as a follow-up; UNAS reserved for bulk media + workspace home dirs.
- n8n reverted to local SQLite after a Postgres migration attempt failed.

## 2026-05-17

- MCP gateway design frozen. Implementation in `services/mcp-gateway.md`; design rationale archived as `history/mcp-gateway-requirements.md`.
- Per-stack scrubbing pass (see `history/scrubbing-list-2026-05-17.md` for the inventory at that snapshot).

## 2026-05-16

- Traefik abandoned, Zoraxy is the production reverse proxy. Two migration guides preserved in `history/` for context only.