docs/CHANGELOG.md

# Changelog

All notable infrastructure / service / doc changes. Newest first.

## 2026-05-23

### Observability stack (CT 109 "ops")

- **CT 109 ("ops") fully provisioned**: Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring/` running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in `ct-inventory.md`.

- **Loki deployed on CT 109**: log aggregation at `:3100`, 30-day retention, TSDB v13 schema, filesystem storage at `./loki/data/`. `limits_config` set `reject_old_samples_max_age: 168h` (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via `provisioning/datasources/loki.yml`. Grafana Loki datasource UID: `P8E80F9AEF21F6940`.

- **Prometheus remote_write receiver enabled**: `--web.enable-remote-write-receiver` added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.

- **prometheus-pve-exporter deployed**: `prompve/prometheus-pve-exporter:latest` on CT 109; credentials in `./pve-exporter/pve.yml` (chmod 644 — container runs non-root). PVE `monitor@pve` service user + API token `c4be45dc-...` created with `PVEAuditor` role on `/`. Prometheus scrapes `:9221/pve?target=192.168.1.20`. Metrics: per-VM/CT CPU, memory, disk, network I/O.

- **HA prometheus integration added**: `prometheus:` block in HA `configuration.yaml`; HA exposes ~9211 `hass_*` entity-level metrics at `:8123/api/prometheus`. CT 109 Prometheus scrapes with bearer token (HA long-lived token), `scrape_interval: 60s`. This is distinct from Alloy host-level metrics — both needed.

- **Grafana Alloy deployed across all 12 hosts**: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to `http://192.168.1.8:3100/loki/api/v1/push`. Deployment breakdown:
  - **Docker Compose** (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, `group_add: ["999"]` for journal access.
  - **Binary + systemd** (CT 102, 103, 108, nuc): apt install `alloy`, config at `/etc/alloy/config.alloy`, `systemctl enable --now alloy`. Journal-only (no Docker socket on these hosts).
  - **HA add-on** (VM 100): `wymangr/hassos-addons` Grafana Alloy v0.0.8 — config fills in `prometheus_remote_write` to CT 109 + Loki endpoint. Distinct host label `homeassistant`.
  - All configs stored in `/opt/homelab-configs/alloy/` (see homelab-configs repo below).

- **5 Grafana dashboards provisioned**:
  - Node Exporter Full (Grafana ID 1860) — imported for all hosts
  - LiteLLM (custom-built) — `/d/.../litellm`; panels: request rate, error rate, token throughput, model split, p95 latency
  - Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
  - Loki / log explorer — imported
  - Prometheus target health — imported

- **3 Grafana alert rules created** via `/api/v1/provisioning/alert-rules` (folder `alerts-folder`):
  - `Node disk > 85%` — per-instance mountpoint disk usage, 5m window
  - `Prometheus target down` — `count(up == 0) > 0`, 5m window
  - `LiteLLM error rate > 5%` — 10m rate window, 5m pending

### Homarr (CT 109)

- **Homarr v1 deployed**: `ghcr.io/homarr-labs/homarr:latest` on CT 109 `:7575`. Data at `./data/`, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).

- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.

### Pocket-ID OIDC (pending user action)

- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate.

- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars.

### Docs

- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config).
- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.

## 2026-05-22

- **Pocket-ID backup wired into Backrest**: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.

- **Vaultwarden data → local zfs**: Moved `/mnt/pve/unas/services/vaultwarden` to `/opt/stacks/vaultwarden/data` (local NVMe). DB was already on CT 113 postgres; only `rsa_key.pem`, `config.json`, `attachments/` remain in the data dir. Stale `db.sqlite3` deleted. NFS copy at `services/vaultwarden/` is now orphaned and can be cleaned up.

- **WAL-G backup script fixed (CT 104)**: All nightly postgres backups were failing silently. Three bugs fixed: (1) `set -e` causing script abort when `shared-postgres` container not found (it was migrated to CT 113); (2) missing `-u postgres` on all `docker exec` calls — WAL-G requires the postgres role; (3) missing explicit data path `/var/lib/postgresql/data` on `backup-push`. `shared-postgres` commented out (CT 113 has its own WAL-G cron). `immich_postgres` and `lobe-postgres` now back up successfully to Garage S3.

- **apps/ directory decommissioned (CT 104)**: `/opt/stacks/apps/` contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db — all without `.env` files, never running. Renamed to `.DECOMMISSIONED-2026-05-22` following established pattern. The canonical stacks at `/opt/stacks/<service>/` are unaffected.

- **ComfyUI `--async-offload` removed**: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from `ai/comfyui.yml`. Current args: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16`. See `infra/proxmox-memory-audit.md`.

- **CT 105 Nextcloud CIFS → NFS**: Nextcloud was crash-looping ("Appdata is not present") because `unas_smb` CIFS mount (`//192.168.1.31/storage`) was not mounted on the host. Root cause: fstab entry was missing `_netdev`; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 `mp0` from `/mnt/pve/unas_smb` to `/mnt/pve/unas` (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart.

- **arr stack started**: `vpn_gluetun`, `shelfarr` (`:13004`), `rdtclient` (via VPN, `:13001`), `prowlarr` (`:13002`), `audiobookshelf` (`:13003`). Fixed empty `labels:` in VPN compose.
- **claude-max-bridge**: new service on CT 104 (`ai-internal`, no host port). FastAPI shim wrapping `claude` CLI as an OpenAI-compatible `/v1/chat/completions` endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: `--strict-mcp-config` (no MCP servers), `--input-format stream-json` (multi-turn), `--include-partial-messages` (real streaming deltas). Bind-mounts `/usr/local/bin/claude` + `/root/.claude` (rw for OAuth token refresh). Source at `git.nuclide.systems/fkrebs/claude-max-bridge`.
- **LiteLLM Claude models**: `claude-sonnet-4-6`, `claude-opus-4-7`, `claude-haiku-4-5` + short aliases (`sonnet`, `opus`, `haiku`) + effort variants (`claude-sonnet-4-6-high`, `claude-opus-4-7-high`) added via `openai/` provider pointing at `claude-max-bridge:8000/v1`. Priced at Anthropic list rates for cost-visibility — actual cost $0 (Max). Bridge features: temperature→effort mapping, `extra_body.effort` override, `extra_body.fallback_model`, `x_claude_cost_usd` and `x_claude_rate_limit` in responses.
- **claude-max-bridge v2**: additional CLI parameter mappings: `extra_body.system_prompt_mode` (`replace`/`append` → `--system-prompt`/`--append-system-prompt`), `extra_body.max_turns` → `--max-turns`, `extra_body.max_budget_usd` → `--max-budget-usd`, `response_format.json_schema` → `--json-schema`, `extra_body.exclude_dynamic_system_prompt_sections` → `--exclude-dynamic-system-prompt-sections`. Build context moved from `/tmp/` to `/opt/stacks/ai/claude-max-bridge/` (survives reboots). Source committed to Gitea at `a781abb`.
- **Shepard MongoDB pinned to 8.0.4**: `mongo:8.0` (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's `7.0.2-5-pve` kernel string (parsed as major version 7 ≥ 6.19). Pinned to `mongo:8.0.4` in `/opt/shepard/infrastructure/docker-compose.yml`. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning.
- **ComfyUI XPU optimisation**: removed `--lowvram` + `--reserve-vram 1.0`; added `--async-offload` (Intel-patched XPU streams, ~10% speedup) + `--force-fp16`; memory limit 20 G → 24 G (FLUX peaks ~22 GB in shared XPU RAM). `--disable-smart-memory` kept since `get_free_memory()` returns the full 58 GB shared pool on Arc — smart eviction is blind to the cgroup limit. See `infra/proxmox-memory-audit.md`.
- **WaveSpeed FBC workflow**: `ApplyFBCacheOnModel(residual_diff_threshold=0.12)` + `FluxGuidance(guidance=3.5)` + `BasicGuider` + `SamplerCustomAdvanced` added to FLUX.1-schnell pipeline. Workflow saved at `/mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json`. Estimated 30–50% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); `torch.compile` node must NOT be used on XPU.
- **LiteLLM benchmark**: `bench.py` at `/opt/stacks/ai/benchmark/` measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (`http://kroki:8000`). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101–143 ms. See `services/llm-benchmark.md`.

- **MCP gateway**: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`.
- **Nextcloud MCP re-enabled**: `cbcoutinho/nextcloud-mcp-server` running in `single_user_basic` mode with `NEXTCLOUD_VERIFY_SSL=false`. App-password generated non-interactively via `occ user:add-app-password`. Gateway patched: (1) `NEXTCLOUD_VERIFY_SSL` / `MCP_DEPLOYMENT_MODE` now baked into hardcoded SERVERS env; (2) startup overlay extended to persist `enabled` flag; (3) proxy route bypasses per-user credential provisioning when `MCP_DEPLOYMENT_MODE=single_user_basic`. 26 servers healthy.
- **CT 109 renamed `observe` → `ops`**: updated `ct-inventory.md`, `homelab-architecture.md`, `proxmox-state.md` (IP changed to `.8` after conflict resolution).
- **CT 101 Shepard disk expanded**: 100 GiB → 500 GiB (ZFS online resize, no restart).
- **CT 112 Infisical marked live**: `ct-inventory.md` and `portmap.md` updated; LAN-only, no Zoraxy route.
- **litellm DB password rotated**: placeholder `litellm_password_here` replaced in both `ai/.env` and `litellm-config/config.yaml` (config.yaml takes precedence over env var). Container restarted, healthy.
- **UNAS migration dumps cleaned**: 7 × `*-migration-20260521.sql` (~492 MB) deleted from `/mnt/pve/unas/dump/`.
- **Arcane OIDC secret rotated**: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and `/opt/stacks/arcane/docker-compose.yml`. Container restarted.
- **n8n OIDC secret rotated**: Pocket-ID DB bcrypt update; secret applied to `/opt/stacks/n8n/docker-compose.yml` (authoritative stack). `/opt/stacks/apps/n8n/` was a dead duplicate (no `.env`, never ran) — compose renamed `.DECOMMISSIONED-2026-05-22`.
- **n8n encryption key rotated**: `N8N_ENCRYPTION_KEY` + `N8N_USER_MANAGEMENT_JWT_SECRET` replaced with 32-byte hex values in `/opt/stacks/n8n/.env` and `/opt/stacks/n8n/data/config`. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered.
- **Backrest repo passwords rotated**: `tapirnase` (weak placeholder) replaced on both `media-repo` and `services-repo` (JottaCloud). Used `restic key add` + `restic key remove`; `config.json` updated; daemon restarted.
- **Obsidian MCP (`mcpvault`)**: new server added. `bitbonsai/mcpvault` bridged via `mcp-proxy` stdio→HTTP in a custom `mcp-obsidian-bridge` image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud `Notizen/` folder at `/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen`, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users.
- **LobeHub MCP sync**: all 27 gateway servers synced into `user_installed_plugins` for both users; stale `papersearch` entry removed; all manifests updated to long-lived `mcp_*` gateway token. Skipped: `crawl4ai` (no `/mcp` endpoint in `unclecode/crawl4ai:latest`), `n8n` (uses own auth).
- **CT 109 plan updated**: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.
- **Portmap**: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added.
- **New docs**: `services/backrest.md`, `services/databases.md`, `services/arcane.md`.

## 2026-05-21

### Docs & docs infrastructure
- **Mermaid diagrams now rendering** in mkdocs Material site — was missing `custom_fences` config in `pymdownx.superfences`. Fixed in `docs-server/mkdocs.yml` and rebuilt container.
- **`site_url` corrected** to `http://192.168.1.111:13080` (docs are LAN-only; no Zoraxy route exists).
- **Docs restructured**: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to `infra/` section. TODO items migrated into CHANGELOG + ideas.
- **Secrets manager design doc** added at `services/secrets-manager.md` — recommends Infisical on a new CT 112.

### Storage / S3
- **`s3.nuclide.systems` Zoraxy route fixed** — `EnableWebsocketCustomHeaders`, `DisableHopByHopHeaderRemoval`, `EnableAutoHTTPS` all set. Garage S3 API now reachable externally.
- **`chat-artifacts` Garage S3 bucket created** — public-read via `https://chat-artifacts.s3.nuclide.systems/<key>`. Credentials at `/root/garage-chat-artifacts.creds` (not git-tracked).

### MCP surface
- **`upload-artifact` MCP server** deployed (port 18011, CT 104). Tools: `upload_text`, `upload_base64`, `list_artifacts`, `get_url`, `delete_artifact`. Registered in gateway (now 24 servers). Source: `git.nuclide.systems/fkrebs/mcp-upload-artifact`.
- **ComfyUI MCP** now auto-uploads completed images to `chat-artifacts` S3 and includes the public URL alongside the inline base64 image.
- **DAYTONA_API_KEY** removed from CT 104 `ai/.env` (Daytona decommissioned 2026-05-20).

### Dev environment
- **`savefig` dotfiles helper** added to `fkrebs/dotfiles` — uploads any file to `chat-artifacts` using `uv run --with boto3`; reads credentials from env vars.
- **Coder `python-uv` template** updated with `CHAT_ARTIFACTS_*` env vars baked in; template pushed to Coder server.

### Gitea repos
- New repos created: `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact`, `home-assistant-config` — all at `git.nuclide.systems/fkrebs/`.
- **HAOS config** (`192.168.1.60`, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see `services/secrets-manager.md` for SSH access notes).

### Planned
- CT 112 "secrets" — Infisical deployment (see `services/secrets-manager.md`)
- Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt)
- Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)
- LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred)

## 2026-05-20

### Auth / Identity (single sign-on lockdown)
- **Pocket-ID migrated CT 104 → CT 110 "id"** (`192.168.1.5:11000`). Authoritative writes now on CT 110; old data dir on CT 104 is a stale snapshot. Compose on CT 104 renamed `.MIGRATED-TO-CT110-2026-05-20`. Zoraxy upstream for `id.nuclide.systems` flipped.
- **Pocket-ID 2.7.0 uses bcrypt for OIDC client secrets** — previously some clients (Gitea, Coder) had raw 64-char SHA-256 hex in `oidc_clients.secret`, which silently failed every token exchange. Regenerated both with proper bcrypt hashes; audit also surfaced an orphaned Daytona client and the empty-secret `vscode` row.
- **Gitea SSO via Pocket-ID** wired with bcrypt fix + correct `Scopes` (`openid,profile,email`) + `two_factor_policy=skip` + `ACCOUNT_LINKING=auto` + `ENABLE_AUTO_REGISTRATION=true` + `USERNAME=preferred_username`. Local password sign-in form disabled (`GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM=false`); legacy OpenID 2.0 button hidden (`GITEA__openid__ENABLE_OPENID_SIGNIN=false`).
- **Coder SSO via Pocket-ID** wired. User row flipped `login_type=password → oidc`. `CODER_DISABLE_PASSWORD_AUTH=true` + `CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE=false`. Workspace-side GitHub external_auth left on (for repo cloning).

### Dev environment (new CT 111 "dev", `192.168.1.42`)
- New LXC stood up with Coder + Gitea + their Postgres companions.
- Templates pushed to Coder: **`python-uv`** (persistent, GPU, code-server, baked LiteLLM + Anthropic env, Claude Code auto-install) and **`mcp-sandbox`** (ephemeral, sci stack pre-baked).
- GPU passthrough confirmed: `/dev/dri/by-path/pci-0000:00:02.0-render` exposed; CT 111-level symlink at `/dev/dri/renderD128` (systemd-tmpfiles) so the Docker provider's path parser is happy.
- `dotfiles` repo created at `fkrebs/dotfiles` on Gitea — oh-my-zsh, LiteLLM model picker (`models.env`), docker alias set, `~/.claude/settings.json` permissions allowlist, global Python `CLAUDE.md`, `.gitignore_global`, VSCodium settings + extensions.txt.
- `mkproj` helper appends `[tool.coder]` workspace+owner block to new `pyproject.toml`, copies pre-commit config + `.gitignore`, installs dev deps, initial commit.

### Reverse proxy
- **Zoraxy WebSocket forwarding enabled** for `dev.`, `git.`, `mcp.nuclide.systems` (`EnableWebsocketCustomHeaders=true` + `HeaderRewriteRules.DisableHopByHopHeaderRemoval=true`). Coder agent / Gitea live updates / streamable-MCP all need this. Backups at `/tmp/zoraxy-backup-2026-05-20/` on CT 108.

### MCP gateway
- **`daytona` removed**, **`coder` added** in `/opt/stacks/ai/mcp-gateway/config.json`. The `morning-briefing` scheduled agent's server list updated. Built a small `coder-mcp` image (multi-stage from `ghcr.io/coder/coder:latest` + `uv:bookworm-slim` + `uv tool install mcp-proxy`) that `coder login`s once then runs `coder exp mcp server` behind streamable-HTTP.
- Coder MCP exposes tools `coder_create_task` + workspace lifecycle from `Coder Agent v1.27.1`.

### Decommissions
- **Daytona** (`/opt/stacks/daytona/` + `/opt/stacks/ai/daytona-mcp.yml` on CT 104) — containers stopped + removed; compose files renamed `.DECOMMISSIONED-2026-05-20`. Coder replaces it.

### Docs / governance
- **`/CLAUDE.md` created on host** — operator doctrine for any agent SSHing in: keep things running, enforce consistency, plan rollback at medium-risk+, services AI-accessible by default, test before declaring done, generate slash commands for recurring tasks, inventory skills/commands/MCPs at session start.
- **`/docs` is now a git repo**, baseline at `04d54ac`.
- **Doc reorg** into `infra/ services/ history/ ideas/`. Traefik guides moved to `history/` (Zoraxy is permanent). `mcp-gateway-requirements.md` and `scrubbing-list-2026-05-17.md` also archived.

## 2026-05-19

- **Garage moved off NFS-on-UNAS to local zfs** on CT 104 (`/opt/stacks/shared-db/garage/{meta,data}`) after a WAL-G outage. Tier-1 (SQLite) databases off NFS as a follow-up; UNAS reserved for bulk media + workspace home dirs.
- **n8n reverted to local SQLite** after a Postgres migration attempt failed.

## 2026-05-17

- MCP gateway design frozen. Implementation in `services/mcp-gateway.md`; design rationale archived as `history/mcp-gateway-requirements.md`.
- Per-stack scrubbing pass (see `history/scrubbing-list-2026-05-17.md` for the inventory at that snapshot).

## 2026-05-16

- **Traefik abandoned**, **Zoraxy is the production reverse proxy**. Two migration guides preserved in `history/` for context only.