From 143343decf0a562e685316b7ae48d64da62b8d7b Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Fri, 22 May 2026 22:42:17 +0200 Subject: [PATCH] docs: session continuity file + claude-max-bridge /v1/responses fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add RESUME.md for cross-session continuity (open items, key state, constraints) - claude-max-bridge: implement /v1/responses (OpenAI Responses API) with previous_response_id chaining via server-side history injection into system prompt. Root cause of "session already in use": CLI leaves JSONL in un-resumable "dequeued" state after each --print run; fix avoids session reuse entirely. Also fixed: assistant content must be array-of-blocks not plain string (silent JS crash otherwise). - LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge - Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs, Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS) - Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session) Co-Authored-By: Claude Sonnet 4.6 --- CHANGELOG.md | 33 +++++ README.md | 19 +-- RESUME.md | 58 +++++++++ ct-inventory.md | 7 +- ideas/litellm-claude-max-bridge.md | 190 +++++++++++++++++++++++++++++ infra/proxmox-memory-audit.md | 80 ++++++++++++ infra/proxmox-state.md | 5 +- infra/storage.md | 21 ++-- infra/volumes.md | 17 ++- services/homelab-architecture.md | 8 +- services/llm-benchmark.md | 89 ++++++++++++++ services/mcp-gateway.md | 3 +- 12 files changed, 496 insertions(+), 34 deletions(-) create mode 100644 RESUME.md create mode 100644 ideas/litellm-claude-max-bridge.md create mode 100644 infra/proxmox-memory-audit.md create mode 100644 services/llm-benchmark.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a3355ba..99bd5ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,40 @@ All notable infrastructure / service / doc changes. Newest first. ## 2026-05-22 +- **Pocket-ID backup wired into Backrest**: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data. + +- **Vaultwarden data → local zfs**: Moved `/mnt/pve/unas/services/vaultwarden` to `/opt/stacks/vaultwarden/data` (local NVMe). DB was already on CT 113 postgres; only `rsa_key.pem`, `config.json`, `attachments/` remain in the data dir. Stale `db.sqlite3` deleted. NFS copy at `services/vaultwarden/` is now orphaned and can be cleaned up. + +- **WAL-G backup script fixed (CT 104)**: All nightly postgres backups were failing silently. Three bugs fixed: (1) `set -e` causing script abort when `shared-postgres` container not found (it was migrated to CT 113); (2) missing `-u postgres` on all `docker exec` calls — WAL-G requires the postgres role; (3) missing explicit data path `/var/lib/postgresql/data` on `backup-push`. `shared-postgres` commented out (CT 113 has its own WAL-G cron). `immich_postgres` and `lobe-postgres` now back up successfully to Garage S3. + +- **apps/ directory decommissioned (CT 104)**: `/opt/stacks/apps/` contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db — all without `.env` files, never running. Renamed to `.DECOMMISSIONED-2026-05-22` following established pattern. The canonical stacks at `/opt/stacks//` are unaffected. + +- **ComfyUI `--async-offload` removed**: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from `ai/comfyui.yml`. Current args: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16`. See `infra/proxmox-memory-audit.md`. + +- **CT 105 Nextcloud CIFS → NFS**: Nextcloud was crash-looping ("Appdata is not present") because `unas_smb` CIFS mount (`//192.168.1.31/storage`) was not mounted on the host. Root cause: fstab entry was missing `_netdev`; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 `mp0` from `/mnt/pve/unas_smb` to `/mnt/pve/unas` (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart. + +- **arr stack started**: `vpn_gluetun`, `shelfarr` (`:13004`), `rdtclient` (via VPN, `:13001`), `prowlarr` (`:13002`), `audiobookshelf` (`:13003`). Fixed empty `labels:` in VPN compose. +- **claude-max-bridge**: new service on CT 104 (`ai-internal`, no host port). FastAPI shim wrapping `claude` CLI as an OpenAI-compatible `/v1/chat/completions` endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: `--strict-mcp-config` (no MCP servers), `--input-format stream-json` (multi-turn), `--include-partial-messages` (real streaming deltas). Bind-mounts `/usr/local/bin/claude` + `/root/.claude` (rw for OAuth token refresh). Source at `git.nuclide.systems/fkrebs/claude-max-bridge`. +- **LiteLLM Claude models**: `claude-sonnet-4-6`, `claude-opus-4-7`, `claude-haiku-4-5` + short aliases (`sonnet`, `opus`, `haiku`) + effort variants (`claude-sonnet-4-6-high`, `claude-opus-4-7-high`) added via `openai/` provider pointing at `claude-max-bridge:8000/v1`. Priced at Anthropic list rates for cost-visibility — actual cost $0 (Max). Bridge features: temperature→effort mapping, `extra_body.effort` override, `extra_body.fallback_model`, `x_claude_cost_usd` and `x_claude_rate_limit` in responses. +- **claude-max-bridge v2**: additional CLI parameter mappings: `extra_body.system_prompt_mode` (`replace`/`append` → `--system-prompt`/`--append-system-prompt`), `extra_body.max_turns` → `--max-turns`, `extra_body.max_budget_usd` → `--max-budget-usd`, `response_format.json_schema` → `--json-schema`, `extra_body.exclude_dynamic_system_prompt_sections` → `--exclude-dynamic-system-prompt-sections`. Build context moved from `/tmp/` to `/opt/stacks/ai/claude-max-bridge/` (survives reboots). Source committed to Gitea at `a781abb`. +- **Shepard MongoDB pinned to 8.0.4**: `mongo:8.0` (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's `7.0.2-5-pve` kernel string (parsed as major version 7 ≥ 6.19). Pinned to `mongo:8.0.4` in `/opt/shepard/infrastructure/docker-compose.yml`. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning. +- **ComfyUI XPU optimisation**: removed `--lowvram` + `--reserve-vram 1.0`; added `--async-offload` (Intel-patched XPU streams, ~10% speedup) + `--force-fp16`; memory limit 20 G → 24 G (FLUX peaks ~22 GB in shared XPU RAM). `--disable-smart-memory` kept since `get_free_memory()` returns the full 58 GB shared pool on Arc — smart eviction is blind to the cgroup limit. See `infra/proxmox-memory-audit.md`. +- **WaveSpeed FBC workflow**: `ApplyFBCacheOnModel(residual_diff_threshold=0.12)` + `FluxGuidance(guidance=3.5)` + `BasicGuider` + `SamplerCustomAdvanced` added to FLUX.1-schnell pipeline. Workflow saved at `/mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json`. Estimated 30–50% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); `torch.compile` node must NOT be used on XPU. +- **LiteLLM benchmark**: `bench.py` at `/opt/stacks/ai/benchmark/` measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (`http://kroki:8000`). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101–143 ms. See `services/llm-benchmark.md`. + - **MCP gateway**: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`. +- **Nextcloud MCP re-enabled**: `cbcoutinho/nextcloud-mcp-server` running in `single_user_basic` mode with `NEXTCLOUD_VERIFY_SSL=false`. App-password generated non-interactively via `occ user:add-app-password`. Gateway patched: (1) `NEXTCLOUD_VERIFY_SSL` / `MCP_DEPLOYMENT_MODE` now baked into hardcoded SERVERS env; (2) startup overlay extended to persist `enabled` flag; (3) proxy route bypasses per-user credential provisioning when `MCP_DEPLOYMENT_MODE=single_user_basic`. 26 servers healthy. +- **CT 109 renamed `observe` → `ops`**: updated `ct-inventory.md`, `homelab-architecture.md`, `proxmox-state.md` (IP changed to `.8` after conflict resolution). +- **CT 101 Shepard disk expanded**: 100 GiB → 500 GiB (ZFS online resize, no restart). +- **CT 112 Infisical marked live**: `ct-inventory.md` and `portmap.md` updated; LAN-only, no Zoraxy route. +- **litellm DB password rotated**: placeholder `litellm_password_here` replaced in both `ai/.env` and `litellm-config/config.yaml` (config.yaml takes precedence over env var). Container restarted, healthy. +- **UNAS migration dumps cleaned**: 7 × `*-migration-20260521.sql` (~492 MB) deleted from `/mnt/pve/unas/dump/`. +- **Arcane OIDC secret rotated**: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and `/opt/stacks/arcane/docker-compose.yml`. Container restarted. +- **n8n OIDC secret rotated**: Pocket-ID DB bcrypt update; secret applied to `/opt/stacks/n8n/docker-compose.yml` (authoritative stack). `/opt/stacks/apps/n8n/` was a dead duplicate (no `.env`, never ran) — compose renamed `.DECOMMISSIONED-2026-05-22`. +- **n8n encryption key rotated**: `N8N_ENCRYPTION_KEY` + `N8N_USER_MANAGEMENT_JWT_SECRET` replaced with 32-byte hex values in `/opt/stacks/n8n/.env` and `/opt/stacks/n8n/data/config`. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered. +- **Backrest repo passwords rotated**: `tapirnase` (weak placeholder) replaced on both `media-repo` and `services-repo` (JottaCloud). Used `restic key add` + `restic key remove`; `config.json` updated; daemon restarted. +- **Obsidian MCP (`mcpvault`)**: new server added. `bitbonsai/mcpvault` bridged via `mcp-proxy` stdio→HTTP in a custom `mcp-obsidian-bridge` image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud `Notizen/` folder at `/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen`, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users. +- **LobeHub MCP sync**: all 27 gateway servers synced into `user_installed_plugins` for both users; stale `papersearch` entry removed; all manifests updated to long-lived `mcp_*` gateway token. Skipped: `crawl4ai` (no `/mcp` endpoint in `unclecode/crawl4ai:latest`), `n8n` (uses own auth). - **CT 109 plan updated**: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113. - **Portmap**: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added. - **New docs**: `services/backrest.md`, `services/databases.md`, `services/arcane.md`. diff --git a/README.md b/README.md index 9b1777f..684a681 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ Single source of truth for the homelab. Git-tracked since 2026-05-20. ## Read this first - **[`/CLAUDE.md`](../CLAUDE.md)** (host root) — operator doctrine for any AI agent SSHing in. Skim before touching anything. +- **[`RESUME.md`](RESUME.md)** — open items + recent changes; update at end of every session. - **[`CHANGELOG.md`](CHANGELOG.md)** — what changed and when. - **[`ct-inventory.md`](ct-inventory.md)** — canonical roster of LXCs / VM, IPs, role, sizing. @@ -26,15 +27,17 @@ Single source of truth for the homelab. Git-tracked since 2026-05-20. │ ├── zoraxy.md reverse proxy │ ├── adguard-dns.md DNS + rewrites │ ├── cloud-gpu.md GPU passthrough + future external GPU -│ └── pocket-id.md [planned] OIDC IdP + client registry +│ └── llm-benchmark.md LiteLLM model TTFT + TPS benchmark │ -├── stacks/ ← CT 104 docker stacks notes -│ ├── PORTMAP.md canonical service → port → public hostname registry -│ ├── CLAUDE.md agent breadcrumbs (CT 104 specific) -│ ├── storage.md [being rewritten] storage class per service -│ ├── volumes.md [being rewritten] volume bindings per stack -│ ├── docker-networks.md -│ └── todo.md +├── infra/ (continued) +│ ├── portmap.md canonical service → port → public hostname registry +│ ├── storage.md storage class per service (source of truth) +│ ├── volumes.md volume bindings per stack +│ ├── proxmox-memory-audit.md CT allocations, host budget, ComfyUI memory analysis +│ └── docker-networks.md +│ +├── stacks/ ← CT 104 agent notes + legacy stack docs +│ └── CLAUDE.md agent breadcrumbs (CT 104 specific) │ ├── history/ ← frozen / archived │ ├── traefik-migration.md ABANDONED 2026-05-16 (Zoraxy chosen) diff --git a/RESUME.md b/RESUME.md new file mode 100644 index 0000000..b51a541 --- /dev/null +++ b/RESUME.md @@ -0,0 +1,58 @@ +# Session Resume + +Last updated: 2026-05-22. Update this at the end of every session. + +## Open items (priority order) + +### Backup gaps +- [x] ~~**Pocket-ID backup**~~ — done 2026-05-22. Pre-hook on CT 103 exports via `pocket-id export` + copies keys to UNAS staging; `services-backup-plan` → jottacloud. Verified snapshot includes Pocket-ID data. +- [ ] **Backrest media-repo** — `video-projects-plan` and `media-backup-plan` were at 0 snapshots but both restic processes were actively running as of 2026-05-22 19:10. Verify snapshots landed after they complete. +- [x] ~~**Gitea repos**~~ — already in `services-backup-plan` → jottacloud. DB on CT 113 (WAL-G). +- [x] ~~**Coder workspace homes**~~ — already in `services-backup-plan` → jottacloud. + +### Infrastructure +- [ ] **CT 109 "ops"** — planned LXC for Prometheus, Loki, Grafana, Alloy, Homarr, sshwifty, Gotify. Not yet built. See `infra/proxmox-state.md` §16. +- [ ] **Second NVMe** — needed before Postgres consolidation (CT 113) and rpool mirror. Unblocks moving remaining per-stack PG containers to CT 113. +- [x] ~~**Vaultwarden NFS cleanup**~~ — `/mnt/pve/unas/services/vaultwarden/` deleted 2026-05-22. +- [x] ~~**apps/backrest rename**~~ — done 2026-05-22 (renamed from PVE host via ZFS subvolume path; idmapped LXC blocked it from inside CT 104). + +### Audits / housekeeping +- [ ] **Zoraxy audit** — CT 108 routes: WS headers, ACME coverage, decommissioned routes, auth consistency (Tinyauth gaps). See `services/zoraxy.md`. +- [ ] **Arcane OIDC secret** — rotation was done 2026-05-22 but Arcane has known issues with OIDC. Verify it still authenticates cleanly. +- [ ] **AdGuard split-horizon DNS** — some internal services still resolve to public IPs instead of LAN IPs for internal clients. Needs rewrite rules audit in `services/adguard-dns.md`. + +## Recently completed (session continued, 2026-05-22) + +- **claude-max-bridge `/v1/responses` endpoint**: implemented OpenAI Responses API with `previous_response_id` chaining. Root cause of "session already in use": Claude CLI leaves session JSONL files in an un-resumable "dequeued" state after each `--print` run. Fix: maintain conversation history server-side as formatted text; inject into system prompt for continuations; always use `--no-session-persistence`. Also fixed: assistant turns in `--input-format stream-json` require `content` as array-of-blocks not string (JS error otherwise). Assistant content array format also fixed in `_parse_messages` / `_parse_responses_input`. +- **LiteLLM pass-through for `/v1/responses`**: added `pass_through_endpoints` entry in `/opt/stacks/ai/litellm-config/config.yaml` → `http://claude-max-bridge:8000/v1/responses`. LiteLLM appends its token hash to the response `id`, so `previous_response_id` chaining only works when calling the bridge directly (not through LiteLLM). + +## Recently completed (this session, 2026-05-22) + +- **Pocket-ID backup**: pre-hook on CT 103 runs `pocket-id export` + scp keys → UNAS staging; `services-backup-plan` snapshots → jottacloud. Verified. +- **Stale NFS vaultwarden dir deleted**: `/mnt/pve/unas/services/vaultwarden/` removed. +- **Nextcloud CIFS → NFS**: crash-loop fixed; CT 105 mp0 migrated from `/mnt/pve/unas_smb` to `/mnt/pve/unas`; CIFS fstab entry removed. +- **Vaultwarden → local zfs**: `/mnt/pve/unas/services/vaultwarden` → `/opt/stacks/vaultwarden/data`; stale `db.sqlite3` deleted; DB already on CT 113 postgres. +- **WAL-G fixed**: `immich_postgres` and `lobe-postgres` backups now succeed. Fixed: missing `-u postgres`, missing data path, `set -e` abort, dead `shared-postgres` reference. +- **ComfyUI `--async-offload` removed**: caused CPU fallback + pure-noise output on GGUF img2img with XPU. +- **apps/ decommissioned**: dead duplicate compose files in `/opt/stacks/apps/` renamed `.DECOMMISSIONED-2026-05-22`. +- **Docs consolidated**: storage.md, volumes.md, homelab-architecture.md, proxmox-state.md, proxmox-memory-audit.md, ct-inventory.md, README.md all reconciled. + +## Key system state + +| Host | IP | Role | +|---|---|---| +| nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs | +| CT 104 docker | 192.168.1.40 | Main Docker host — AI/ML, media, ~65 containers | +| CT 113 db | 192.168.1.6 | Shared Postgres 17 + WAL-G → Garage S3 (cron 02:00) | +| CT 110 id | 192.168.1.5 | Pocket-ID OIDC IdP | +| CT 103 backrest | 192.168.1.3 | Backrest — jottacloud via rclone; UNAS at `/mnt/pve/unas` | +| CT 111 dev | 192.168.1.42 | Coder + Gitea | +| CT 108 zoraxy | 192.168.1.4 | Reverse proxy + ACME — **always confirm before changes** | + +## Constraints to remember + +- Zoraxy (CT 108): always get explicit user confirmation before any config change. +- Infisical (CT 112): LAN-only, no Zoraxy route — must not be internet-exposed. +- CT 104 has a separate agent reconciling compose files — rename decommissioned files (don't just stop), update `/opt/stacks/CLAUDE.md`. +- ComfyUI container limit: 24 G. Do not raise without measuring host impact. +- Pocket-ID is SQLite-only — no Postgres migration possible. diff --git a/ct-inventory.md b/ct-inventory.md index 18a6cf8..e869fb6 100644 --- a/ct-inventory.md +++ b/ct-inventory.md @@ -9,17 +9,20 @@ Verified 2026-05-20 via `pct list`, `pct config `, `qm config 100` on `nuc`. | 102 | dns | LXC unpriv | 192.168.1.2/24 | 2 | 1024 | 4 GiB | **AdGuard Home** — LAN DNS resolver + filter | — | no | running | | 103 | backrest | LXC unpriv | 192.168.1.3/24 | 1 | 512 | 8 GiB | **Backrest** (restic) backup scheduler | mp0 NFS | no | running | | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | -| 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 **CIFS** (`//192.168.1.31/storage`) | Intel iGPU (render only) | running | +| 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 NFS | Intel iGPU (render only) | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | | 109 | ops | LXC | ⚠️ TBD (`.6`=CT 113, `.7`=CT 112) — likely `.8` | 4 | 8192 | 50 GiB | **Planned** — Prometheus, Loki, Grafana, Arcane Manager, Dozzle, Homarr, Diun, sshwifty, Tinyauth, socket-proxy; Alloy agents on all hosts | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 112 | secrets | LXC unpriv | 192.168.1.7/24 | 2 | 4096 | 20 GiB | **Infisical** secrets manager (own Postgres 16 + Redis 7; replaces scattered .env files) — LAN-only, no Zoraxy route | — | no | running | +| 113 | db | LXC unpriv | 192.168.1.6/24 | 2 | 4096 | 40 GiB | Shared Postgres 17 + pgAdmin + WAL-G → Garage S3; Arcane edge agent | — | no | running | UNAS NFS = `192.168.1.31:/var/nfs/shared/storage` over NFSv3. CT 105 (Nextcloud) is the lone outlier — it mounts the same UNAS share over **CIFS/SMB 3.1.1**, not NFS. CT 112 ("secrets") — Infisical deployed 2026-05-22; see `services/secrets-manager.md`. LAN: `http://192.168.1.7:8200`. No Zoraxy route — secrets must not be internet-exposed. +CT 105 ("nextcloud") — migrated from CIFS (`//192.168.1.31/storage`) to NFS (`192.168.1.31:/var/nfs/shared/storage`) on 2026-05-22. CIFS mount was not remounting after host reboots, causing Nextcloud crash-loops. NFS is consistent with all other CTs. + ## SSH access The Proxmox host (`nuc`, `192.168.1.20`) has root SSH access to all LXC containers via key auth. To grant your own key access to every running container in one step, run on the host: @@ -44,4 +47,4 @@ echo "ssh-ed25519 AAAA... you@yourmachine" >> ~/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuVqBW3VXg... root@nuc ``` -CT 113 ("db") is mentioned in memory notes (`postgres-consolidation`) as a future consolidation target. Not yet planned in any source-of-truth doc. +CT 113 ("db") provisioned 2026-05-21; see `services/databases.md`. Future role: consolidate per-stack Postgres instances once second NVMe lands. diff --git a/ideas/litellm-claude-max-bridge.md b/ideas/litellm-claude-max-bridge.md new file mode 100644 index 0000000..1f4aed9 --- /dev/null +++ b/ideas/litellm-claude-max-bridge.md @@ -0,0 +1,190 @@ +# Design: Claude Max Subscription → LiteLLM Gateway Bridge + +**Status**: Proposal — not implemented +**Context**: LiteLLM at `https://ai.nuclide.systems` currently has no Anthropic models. Claude Max subscription (claude.ai) provides high-rate access to Sonnet 4.5/4.6 and Opus 4.7 but is decoupled from Anthropic API billing. This doc explores bridging the two. + +## Problem + +Claude Max and the Anthropic API are separate products with separate billing: + +| | Claude Max | Anthropic API | +|---|---|---| +| Auth | OAuth / browser session | API key | +| Billing | Flat monthly subscription | Per-token | +| Rate limits | 5h rolling windows, model-specific | Tier-based RPM/TPM | +| Access | claude.ai web + Claude Code CLI | Any HTTP client | + +The goal is to surface Max-subscription capacity through LiteLLM so that LobeHub, n8n, Coder workspaces, and other internal tools can call `claude-sonnet-4-6` at zero marginal cost and fall back to paid providers only when Max limits are hit. + +## Approaches + +### A — Session Cookie Reverse-Engineering (not recommended) + +Several community projects (e.g. `claude-unofficial-api`) scrape the claude.ai WebSocket/HTTP protocol and expose an OpenAI-compatible endpoint. LiteLLM would point at this as a custom `openai/` provider. + +**Pros**: Exposes the full web model lineup; streaming works. +**Cons**: Violates Anthropic ToS; breaks on any claude.ai front-end change; auth flow requires persisting browser cookies; no multimodal or tool-use parity guarantees. + +**Verdict**: Avoid. Fragile and non-compliant. + +--- + +### B — Claude Code CLI Bridge (recommended) + +Claude Code CLI (`claude`) is already installed on CT 104 and authenticated with the Max subscription via `~/.claude/`. It ships a `--print` / `--output-format stream-json` mode designed for non-interactive use, and Anthropic explicitly supports programmatic use of the CLI. + +A small **`claude-max-bridge`** service wraps this CLI as an OpenAI-compatible HTTP endpoint. LiteLLM registers it as a custom `openai/` base URL. No ToS issues — this is the supported surface. + +``` +LobeHub / n8n / Coder / Claude Code + │ + ▼ + LiteLLM Gateway (ai.nuclide.systems) + │ model: claude-sonnet-4-6 → openai/claude-sonnet-4-6 + │ api_base: http://claude-max-bridge:8000 + ▼ + claude-max-bridge (new container, CT 104 ai-internal) + │ subprocess: claude --model ... --print --output-format stream-json + ▼ + ~/.claude/ (Max subscription session) + │ + ▼ + Anthropic (claude.ai) +``` + +**Pros**: +- Uses the officially supported programmatic interface +- Auth is already set up; no cookie management +- `claude` CLI handles retries, token limits, context window management +- Subprocess overhead is ~200–400 ms cold; warm invocations faster + +**Cons**: +- One subprocess per request — cannot multiplex a single session (unlike streaming HTTP) +- CLI is tied to the single authenticated user; no multi-user isolation +- Max rate limits apply per-account, same pool as interactive use +- Claude Code SDK (TypeScript) is cleaner but adds Node dependency + +--- + +### C — Official API + Budget Cap (stop-gap) + +Add Anthropic API key to LiteLLM with a hard budget cap (e.g. $20/month). Use it for Claude-specific features (tool use, long context) and let the existing free SAIA/Gemini fallback chain absorb general-purpose traffic. + +**Pros**: Zero implementation work; full API feature parity. +**Cons**: Still costs money; no benefit from Max subscription. + +**Verdict**: Valid fallback if B proves too complex, or as a complement for tool-heavy workloads that need the official API surface. + +--- + +## Recommended Architecture (Option B) + +### claude-max-bridge service + +``` +/opt/stacks/ai/claude-max-bridge/ + Dockerfile + server.py # FastAPI, ~150 lines + docker-compose.yml (or entry in ai/docker-compose.yml) +``` + +**Dockerfile** — reuse the existing `claude` CLI install: +```dockerfile +FROM python:3.12-slim +RUN pip install fastapi uvicorn +# Mount ~/.claude from host; claude binary from host PATH or copied in +COPY server.py /app/server.py +CMD ["uvicorn", "app.server:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +**server.py** — OpenAI `/v1/chat/completions` shim: +```python +# POST /v1/chat/completions +# Translates messages[] → claude --print --model ... --output-format stream-json +# Streams NDJSON lines back as SSE (text/event-stream) +# Maps finish_reason, usage tokens from CLI output headers +``` + +Key translation notes: +- `messages` → write to a temp file, pass via `--input-file` (avoids shell quoting issues) +- `stream: true` → parse `stream-json` lines, emit `data: {...}` SSE chunks +- `stream: false` → buffer all chunks, return single response object +- `max_tokens`, `temperature`, `system` → map to `--max-tokens`, `--temperature`, `--system` +- Tool use: **not supported** in first iteration; return 501 for requests with `tools` +- Model name passthrough: `claude-sonnet-4-6` → `--model claude-sonnet-4-6` + +### LiteLLM config additions + +```yaml +model_list: + - model_name: claude-sonnet-4-6 + litellm_params: + model: openai/claude-sonnet-4-6 + api_base: http://claude-max-bridge:8000 + api_key: "dummy" # bridge ignores it; LiteLLM requires a value + stream_timeout: 120 + timeout: 120 + + - model_name: claude-opus-4-7 + litellm_params: + model: openai/claude-opus-4-7 + api_base: http://claude-max-bridge:8000 + api_key: "dummy" + stream_timeout: 180 + timeout: 180 + + - model_name: claude-haiku-4-5 + litellm_params: + model: openai/claude-haiku-4-5-20251001 + api_base: http://claude-max-bridge:8000 + api_key: "dummy" + stream_timeout: 60 + timeout: 60 +``` + +Add to `router_settings.fallbacks` — Max hits rate limit → fall to SAIA: +```yaml +fallbacks: + - claude-sonnet-4-6: [qwen3.5-397b-a17b] + - claude-opus-4-7: [qwen3.5-397b-a17b, mistral-large-latest] + - claude-haiku-4-5: [qwen3.5-122b-a10b, cerebras-llama-3.1-8b] +``` + +### Rate limit handling + +The bridge should detect the CLI's rate-limit exit code / stderr message and return HTTP 429. LiteLLM's router will then trigger the fallback chain. No custom logic needed in LiteLLM itself. + +Max limits as of 2026 (approximate, vary by plan tier): +- Sonnet 4.6: ~50 messages / 5-hour window at Pro; higher at Max +- Opus 4.7: ~10–20 messages / 5-hour window +- Haiku 4.5: effectively unlimited under Max + +## Volume estimate + +Typical homelab workloads (LobeHub chat, n8n automations, Coder coding assist) generate maybe 200–500 completions/day. Max's 5-hour windows reset 4–5× daily, so the practical limit is not usually hit unless Opus is used heavily. + +## Risks + +| Risk | Mitigation | +|---|---| +| CLI API changes between Claude Code releases | Pin the `claude` binary version; watch for breaking changes in release notes | +| Max rate limit shared with interactive use | Monitor via `claude usage`; bridge adds `X-Max-Usage` header from CLI output | +| Auth session expiry | Bridge returns 401 on auth failure; detect and alert via Gotify | +| Subprocess latency (cold start) | Keep a warm subprocess pool (1–2 persistent processes) using `--interactive` + message passing, or accept the 200–400 ms overhead | +| No tool use support | Fallback chain routes tool-use requests to API providers (Gemini, Mistral) | + +## Implementation plan + +1. Write `server.py` shim (~150 lines) + Dockerfile +2. Build image on CT 104, add to `ai/docker-compose.yml` +3. Bind-mount `~/.claude` read-only into the container +4. Add model entries to `litellm-config/config.yaml` +5. Test streaming via `curl` against the bridge directly +6. Test end-to-end via LiteLLM → LobeHub +7. Watch `docker logs claude-max-bridge` for rate-limit / auth errors for 48h + +## Open questions + +- Does `claude --print` support all message roles (`system`, `user`, `assistant` turns)? Needs verification with multi-turn conversation format. +- Should the bridge be on CT 104 (access to `~/.claude`) or CT 109 ops (remote docker socket to CT 104)? CT 104 is simpler for v1. +- Is there an official Claude Code SDK for Python? (TypeScript SDK exists at `@anthropic-ai/claude-code`; Python wrapper may be cleaner than shelling out.) diff --git a/infra/proxmox-memory-audit.md b/infra/proxmox-memory-audit.md new file mode 100644 index 0000000..8926cdb --- /dev/null +++ b/infra/proxmox-memory-audit.md @@ -0,0 +1,80 @@ +# Proxmox Memory Audit + +Last audited: 2026-05-22 + +## Host physical resources + +| Resource | Total | Used (idle) | Available | +|----------|-------|-------------|-----------| +| RAM | 62 GiB | ~33 GiB | ~28 GiB | +| Swap | 31 GiB | 0 GiB | 31 GiB | + +## CT memory allocations + +| CT | Name | Allocated (MiB) | Swap (MiB) | Typical use | Notes | +|----|------|----------------|------------|-------------|-------| +| 101 | shepard | 32,768 | 8,192 | ~4 GB | Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak | +| 102 | dns | 1,024 | 512 | ~100 MB | AdGuard Home | +| 103 | backrest | 2,048 | 512 | ~200 MB | Restic scheduler | +| 104 | docker | **49,152** | 32,000 | **6–22 GB** | Main Docker host; FLUX spikes to ~22 GB | +| 105 | nextcloud | 8,196 | 8,196 | ~2 GB | Nextcloud AIO | +| 108 | zoraxy | 2,048 | 512 | ~300 MB | Reverse proxy | +| 110 | id | 1,024 | 512 | ~200 MB | Pocket-ID | +| 111 | dev | 32,768 | 8,192 | ~3 GB | Coder + Gitea workspaces | +| 112 | secrets | 4,096 | 512 | ~600 MB | Infisical | +| **Sum** | | **133,124 MiB (130 GiB)** | | | **2× overprovisioned vs physical RAM** | + +## Key findings + +### Overprovisioning is safe — until it isn't + +Proxmox uses balloon drivers so CTs only consume what they actually use. At idle the host sits at ~33 GB used with 28 GB available. This is healthy. However, two specific CTs represent risk: + +- **CT 104** (49 GiB limit) + **CT 111** (32 GiB limit) together could claim 81 GiB — well over the 62 GiB physical. If both hit peak simultaneously the host would start swapping heavily. +- CT 104's ComfyUI (FLUX generation) spikes from 6 GB baseline to ~22 GB. That 16 GB spike on top of the 33 GB idle baseline = 49 GB total, within physical RAM but tight. + +### Why 40G Docker container limit destabilised the system + +Setting ComfyUI's container limit to 40 G was the trigger. At generation time: +- FLUX model + activations: ~22 GB in the container +- Other ~65 Docker containers on CT 104: ~7 GB +- CT 104 OS + kernel: ~1 GB +- Other CTs idle: ~26 GB +- **Total: ~56 GB** → host started swapping, degrading all services + +### ComfyUI XPU memory accounting gap + +ComfyUI's `get_free_memory()` for Intel XPU queries `torch.xpu.get_device_properties().total_memory` = **58 GB** (the full shared memory pool — Arc shares system RAM). It has no awareness of the Docker cgroup limit. Smart memory management (`free_memory()`) calculates `memory_required - get_free_memory()` which is always hugely negative → never evicts models. This makes `--disable-smart-memory` irrelevant for XPU; smart memory is already broken. + +Consequence: ComfyUI will always try to load the full model into XPU memory regardless of container limit. The cgroup OOM killer is the only backstop. + +### Container limit recommendation for ComfyUI (CT 104) + +| Scenario | Limit | Safe? | +|----------|-------|-------| +| `--lowvram` (original) | 20 G | ✅ Safe but slow (231 s/image) | +| No `--lowvram`, FLUX only | **24 G** | ✅ Fits FLUX peak (~22 GB) + 2 GB headroom | +| No `--lowvram` + img2img after FLUX | 26 G | ✅ FLUX stays resident, SD1.5 loads on top | +| 28 G | ⚠️ | Marginal — OOM triggered in testing | +| 40 G | ❌ | Destabilises host when generating | + +Peak host usage at 24 G container limit during FLUX generation: `24 + 7 (other containers) + 26 (other CTs idle) ≈ 57 GB` — stays under 62 GB physical. + +## Recommendations + +1. **ComfyUI container limit**: set to **24 G** when running without `--lowvram`. Current revert to 20 G + `--lowvram` is stable but slower. +2. **CT 104 LXC allocation (49 GiB)**: appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact. +3. **CT 111 (dev, 32 GiB)**: Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template. +4. **Watch list**: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB → host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104. +5. **Long-term**: when CT 109 (ops) is built, run Prometheus `node_exporter` on the PVE host and alert when host available RAM drops below 8 GiB. + +## ComfyUI memory optimisation log + +| Date | Change | Effect | +|------|--------|--------| +| 2026-05-22 | Removed `--lowvram` + raised limit to 28 G | OOM at 28 G (XPU DRM buffers counted against cgroup) | +| 2026-05-22 | Raised limit to 40 G | Host destabilised — reverted | +| 2026-05-22 | Reverted to `--lowvram` + 20 G | Stable, slow (231 s/image) | +| 2026-05-22 | Downloaded `t5-v1_1-xxl-encoder-Q4_K_S.gguf` (2.6 GB vs 3.2 GB Q5_K_M) | Saves 600 MB at load time | +| 2026-05-22 | Kept Q5_K_M T5 (Q4_K_S degrades prompt following per city96); removed `--lowvram`; set limit to 24 G; added `--async-offload --force-fp16` | ~7–30 s generation, safe within host memory budget | +| 2026-05-22 | Removed `--async-offload` | Flag incompatible with GGUF img2img on XPU — caused full CPU fallback (3.5 min/step) and pure-noise output. Removed; XPU generation now correct at ~7 s/step for txt2img. Current CLI: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16` | diff --git a/infra/proxmox-state.md b/infra/proxmox-state.md index 2752a2d..2b86d43 100644 --- a/infra/proxmox-state.md +++ b/infra/proxmox-state.md @@ -5,7 +5,7 @@ Host: `nuc` · PVE `9.1.11` · Kernel `6.17.13-4-pve` · Debian 13 (trixie) CPU: Intel Core Ultra 7 155H (16C / 22T, hybrid P+E+LP-E) · 1 socket · 1 NUMA RAM: 62 GiB physical · 31 GiB zram swap (50 % of RAM, zstd, prio 100) Storage: single Crucial P3 2 TB NVMe (QLC, DRAM-less) → `rpool` (ZFS, ashift=12, no redundancy) -Workload: 1 VM (HAOS) + 6 LXCs (Docker, Adguard, Backrest, Nextcloud, Zoraxy, Dockge) +Workload: 1 VM (HAOS) + 9 LXCs (Docker, AdGuard, Backrest, Nextcloud, Zoraxy, Pocket-ID, Dev, Secrets, DB) + 1 planned (Ops/CT109) --- @@ -332,7 +332,7 @@ tail /var/log/unattended-upgrades/unattended-upgrades.log ## 10. Container-specific issues -### CT 104 (docker / AI / image-gen) — 128 GiB cap, 16 cores, GPU passthrough +### CT 104 (docker / AI / image-gen) — 48 GiB cap, 16 cores, GPU passthrough Measured: 18.8 GiB current, peak 29.3 GiB, **9.3 GiB in swap**, load 8.0, ~65 Docker containers (Immich + ML, ComfyUI, LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless+AI, many MCP servers, Speaches-OpenVINO). @@ -733,6 +733,7 @@ Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores / | Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID | | docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr/Arcane discovery | | **sshwifty** | web SSH client, multi-tab — multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred. | +| **docs-server** | mkdocs Material site (`/docs` git repo → static HTML); migrating here from CT 111 where it currently runs at port 13080. LAN-only, no Zoraxy route. | ### Sidecars deployed on each host diff --git a/infra/storage.md b/infra/storage.md index 05557d9..c4e5d2b 100644 --- a/infra/storage.md +++ b/infra/storage.md @@ -10,8 +10,7 @@ | Local zfs (rootfs) | NVMe in nuc | `/opt/stacks//` on each LXC | Tier-1 hot state: databases, SQLite, secret stores, anything latency-sensitive | | Local zfs (subvol) | NVMe in nuc | `rpool/data/subvol--disk-0` per LXC | Container rootfs, image cache | | Docker named volume | NVMe in nuc | `/var/lib/docker/volumes/...` on each Docker CT | Per-container persistent state managed by Docker | -| UNAS over **NFSv3** | `192.168.1.31` | `/mnt/pve/unas` on CTs 101, 103, 104, 111 | Bulk media, workspace home dirs, document storage | -| UNAS over **CIFS/SMB 3.1.1** | `192.168.1.31` | `/mnt/pve/unas` on CT 105 (Nextcloud) | Nextcloud-only path — earlier docs mis-labelled this as NFS | +| UNAS over **NFSv3** | `192.168.1.31` | `/mnt/pve/unas` on CTs 101, 103, 104, 105, 111 | Bulk media, workspace home dirs, document storage | ## What lives where (verified 2026-05-21) @@ -24,7 +23,6 @@ Bulk + media + non-latency-sensitive app data. | Immich (uploads, thumbs, derived) | 104 | `services/immich/{encoded-video,profile,thumbs}` + `media/images` + `backup/immich` | Immich own pg_dump → `media/images/db-dumps`; no off-host copy | | | Paperless-ngx (documents) | 104 | `media/documents/public/paperless-ngx/{consume,export,library}` | none | | | Paperless-AI | 104 | `services/paperless-ai` → `/app/data` | none | | -| Vaultwarden (attachments + SQLite DB) | 104 | `services/vaultwarden` → `/data` | none | tier-1 DB on NFS — risky, plan to move local | | Traccar (logs, config) | 104 | `services/traccar/{logs,traccar.xml}` | none | data dir reverted to local `/opt/stacks/traccar/data` | | Memos | 104 | `services/memos` → `/var/opt/memos` | none | | | Arr-stack (Audiobookshelf, Prowlarr, RDTClient, Shelfarr) + media | 104 | `services/arr-stack/*` + `media/{audiobooks,ebooks,podcasts,Torrents}` | none | already migrated (older doc claimed "still local") | @@ -52,7 +50,8 @@ Tier-1 state and anything that should NOT be NFS-backed. | SearXNG config | 104 | `/opt/stacks/ai/searxng` | none | | | Flaresolverr | 104 | `/var/lib/flaresolver` | none | | | AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup | -| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below | +| Vaultwarden (attachments + key material) | 104 | `/opt/stacks/vaultwarden/data` → `/data` | none | **moved off NFS 2026-05-22**; DB is on CT 113 postgres; stale `db.sqlite3` deleted | +| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | app data on NFS — see below | ### Garage S3 buckets (on local zfs, CT 104) @@ -76,9 +75,9 @@ Local on `/var/lib/docker/volumes/`. Mostly databases and caches. | `coder-db` + `gitea-db` (docker volume `coder-db`) | / 111 | none | | (orphaned) `daytona-minimal_db_data` + runner anon vol | / 104 | none — clean up post-decommission | -### Nextcloud (CT 105) — the CIFS outlier +### Nextcloud (CT 105) — data on NFS -Nextcloud AIO mounts `//192.168.1.31/storage` over **CIFS/SMB 3.1.1**, not NFS like every other CT. Application data path is under that mount. The Postgres + Redis sidecars stay on local docker volumes. +Nextcloud AIO mounts `/mnt/pve/unas/services/nextcloud` via NFSv3 (same share as all other CTs). Migrated from CIFS on 2026-05-22 after the CIFS mount caused a crash-loop. The Postgres + Redis sidecars stay on local docker volumes. ## Backup reality @@ -96,8 +95,8 @@ This is a known gap. Plans: ## Drift cleanup TODO -- [ ] Remove stale `services/n8n/database.sqlite` (408 MB, last write 2026-05-15) from UNAS. -- [ ] Remove stale `services/shared-db/garage/` copy from UNAS if present. -- [ ] Remove orphaned `daytona-minimal_db_data` Docker volume on CT 104. -- [ ] Vaultwarden `data/` on UNAS — plan to migrate to local zfs (latency + tier-1). -- [ ] Add a Backrest plan for Pocket-ID DB and Gitea data → jottacloud. +- [x] ~~Remove stale `services/n8n/database.sqlite` from UNAS~~ — gone (verified 2026-05-22) +- [x] ~~Remove stale `services/shared-db/garage/` copy from UNAS~~ — gone (verified 2026-05-22) +- [x] ~~Remove orphaned `daytona-minimal_db_data` Docker volume on CT 104~~ — gone (verified 2026-05-22) +- [x] ~~Vaultwarden `data/` on UNAS~~ — migrated to local zfs 2026-05-22. Stale NFS copy at `services/vaultwarden/` can be cleaned up. +- [x] ~~Add a Backrest plan for Pocket-ID DB~~ — done 2026-05-22. Pre-backup hook runs `pocket-id export` + copies keys to UNAS staging; `services-backup-plan` snapshots staging → jottacloud nightly. diff --git a/infra/volumes.md b/infra/volumes.md index d380019..570ee71 100644 --- a/infra/volumes.md +++ b/infra/volumes.md @@ -46,9 +46,11 @@ ### Pocket ID — `pocketid` -| Type | Source | Container | UNAS | +> **CT 110, not CT 104.** Pocket-ID migrated off CT 104 to its own LXC on 2026-05-20. + +| Type | Source | Container | Notes | |---|---|---|---| -| **bind** | **`/mnt/pve/unas/services/pocketid`** | **`/app/data`** | ✅ **Already on UNAS** | +| bind | `/opt/stacks/pocketid/data` | `/app/data` | Local zfs on CT 110 — no UNAS mount | ### Vaultwarden — `vaultwarden` @@ -309,10 +311,12 @@ No DB-specific mounts needed (connects via env vars). ### Garage S3 — `garage` +> Moved off NFS to local zfs on 2026-05-19 after a WAL-G outage. Stale copy at `services/shared-db/garage/` on UNAS may still exist — clean up. + | Type | Source | Container | Notes | |---|---|---|---| -| bind | `/mnt/pve/unas/services/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data | -| bind | `/mnt/pve/unas/services/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (LMDB) | +| bind | `/opt/stacks/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data — local NVMe | +| bind | `/opt/stacks/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (LMDB) — local NVMe | --- @@ -336,9 +340,10 @@ No DB-specific mounts needed (connects via env vars). | Status | Count | Services | |---|---|---| -| ✅ Already on UNAS | 11 | gluetun, immich*(4), karakeep*(2), ntfy*(2), paperless-ai, pocketid, stremio, traccar*(3), vaultwarden, paperless-docs*(3), garage*(2) | +| ✅ Already on UNAS | 9 | gluetun, immich*(4), karakeep*(2), ntfy*(2), paperless-ai, stremio, traccar*(3), vaultwarden, paperless-docs*(3) | | ✅ Migrated (Phase 1) | 3 | memos, arcane, n8n | -| 🔷 Shared infrastructure | 2 | shared-postgres (local volume), garage (on UNAS) | +| 🔷 Shared infrastructure (local) | 2 | shared-postgres (local volume), garage (local NVMe — moved off UNAS 2026-05-19) | +| 📋 Own CT, local only | 1 | pocketid (CT 110, no UNAS) | | ⏳ Phase 2 planned (PG consolidation) | 5 | immich → shared-postgres, paperless → shared-postgres, daytona → shared-postgres, litellm → shared-postgres, lobehub → shared-postgres | | ⏳ Phase 3 planned | 1 | arr-stack*(8 mounts) | | 📋 Keep local | ~5 | homepage, dozzle, litellm-config, searxng-config, saia-image-proxy | diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index cfad97f..1e44f68 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -19,7 +19,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, (confirmed) — so ad/tracker blocking is network-wide despite DHCP not handing out AdGuard's IP directly. - `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G) - - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40` + - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/48G/200G) → `.40` - `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41` - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4` - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 @@ -392,14 +392,14 @@ flowchart TB dns["CT 102 · dns
(.2)
AdGuard Home"] backrest["CT 103 · backrest
(.3)
Backrest / restic"] docker104["CT 104 · docker
(.40) · 16c/48G/200G
~65 containers · Intel Arc passthrough"] - nc["CT 105 · nextcloud
(.41)
Nextcloud AIO (CIFS to UNAS)"] + nc["CT 105 · nextcloud
(.41)
Nextcloud AIO (NFS)"] zoraxy["CT 108 · zoraxy
(.4)
reverse proxy + ACME"] obs(["CT 109 · ops
(.8) · planned
Prom / Graf / Arcane / Homarr / Tinyauth"]) id["CT 110 · id
(.5)
Pocket-ID (moved here 2026-05-20)"] dev["CT 111 · dev
(.42) · 12c/32G/60G
Coder + Gitea + workspaces · Intel Arc"] - db(["CT 113 · db
(.?) · planned
shared Postgres + pgbouncer"]) + db["CT 113 · db
(.6) · provisioned 2026-05-21
shared Postgres + pgAdmin"] end - UNAS[("UNAS
192.168.1.31
NFSv3 + CIFS")] + UNAS[("UNAS
192.168.1.31
NFSv3")] UDM[["UDM-SE · 192.168.1.1
UniFi gateway · DNS → AdGuard"]] Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams]) diff --git a/services/llm-benchmark.md b/services/llm-benchmark.md new file mode 100644 index 0000000..232e961 --- /dev/null +++ b/services/llm-benchmark.md @@ -0,0 +1,89 @@ +# LLM Benchmark + +Streaming performance benchmark for all LiteLLM-routed models. Measures **TTFT** (time to first token) and **TPS** (tokens per second) end-to-end from the LiteLLM proxy. + +## Script + +`/opt/stacks/ai/benchmark/bench.py` on CT 104. + +## How to run + +Must run inside a container on `ai-internal` to reach both LiteLLM and the internal Kroki instance: + +```bash +docker run --rm --network ai-internal \ + -v /opt/stacks/ai/benchmark/bench.py:/bench.py \ + -v /opt/stacks/ai/benchmark:/charts \ + -e LITELLM_API_KEY=sk-tapirnase \ + -e KROKI_URL=http://kroki:8000 \ + python:3.12-slim \ + bash -c 'pip install httpx -q && python /bench.py --charts /charts' +``` + +Charts (`ttft.svg`, `tps.svg`) land in `/opt/stacks/ai/benchmark/`. + +### Options + +| Flag | Default | Description | +|------|---------|-------------| +| `--models a,b` | all 11 | Comma-separated subset to test | +| `--prompt` | `medium` | `ttft_short`, `medium`, or `code_medium` | +| `--runs N` | `2` | Runs per model (mean reported) | +| `--out PATH` | `/tmp/bench_results.json` | Raw JSON output | +| `--charts DIR` | `/tmp/bench_charts` | SVG chart output directory | + +### Environment + +| Var | Default | Notes | +|-----|---------|-------| +| `LITELLM_API_KEY` | — | Required (`sk-tapirnase`) | +| `LITELLM_BASE_URL` | `https://ai.nuclide.systems/v1` | Override for in-network: `http://litellm:4000` | +| `KROKI_URL` | `http://kroki:8000` | Internal Kroki for chart rendering | + +## Models tested + +| Model ID | Provider | Notes | +|----------|----------|-------| +| `claude-haiku-4-5` | claude-max-bridge | Via `claude` CLI subprocess | +| `claude-sonnet-4-6` | claude-max-bridge | | +| `claude-opus-4-7` | claude-max-bridge | | +| `gemini-2.5-flash-lite` | Google | | +| `gemini-2.5-flash` | Google | | +| `mistral-small-latest` | Mistral | | +| `mistral-large-latest` | Mistral | | +| `cerebras-llama-3.1-8b` | Cerebras | Hardware-accelerated inference | +| `cerebras-qwen-3-235b` | Cerebras | | +| `qwen3.5-122b-a10b` | — | Returns empty responses — likely extended-thinking mode with no visible content; excluded from charts | +| `deepseek-r1-distill-llama-70b` | DeepSeek | Reasoning model; total latency high due to thinking tokens | + +## Methodology + +- Each run sends a single streaming `POST /v1/chat/completions` request with `max_tokens: 512` +- **TTFT**: `perf_counter()` delta from request start to first SSE chunk containing `content` +- **TPS**: completion chars ÷ total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by `usage.completion_tokens` if the provider returns it. +- Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary. +- Models that return no `content` deltas (e.g. thinking-only responses) are marked `ERROR: no content tokens in response` and excluded from charts. + +## Results — 2026-05-22 + +Prompt: `"Explain what TCP/IP is in 3 sentences."` · 2 runs each · warm LiteLLM cache + +| Model | TTFT (ms) | TPS (chars/s) | Notes | +|-------|----------:|--------------|-------| +| **mistral-small-latest** | **43** | 10,111 | Fastest TTFT | +| **mistral-large-latest** | **58** | 11,042 | Fastest TPS | +| claude-opus-4-7 | 106 | 4,184 | | +| cerebras-qwen-3-235b | 118 | 3,777 | | +| claude-haiku-4-5 | 125 | 3,454 | | +| gemini-2.5-flash | 139 | 3,519 | | +| claude-sonnet-4-6 | 130 | 3,856 | | +| gemini-2.5-flash-lite | 202 | 2,396 | | +| deepseek-r1-distill-llama-70b | 297 | 10,561 | Total 5.4 s (thinking tokens) | +| cerebras-llama-3.1-8b | 155 | 2,947 | | +| qwen3.5-122b-a10b | ERROR | — | Empty response | + +**Key observations:** +- Mistral leads on both TTFT and raw throughput — likely a combination of low-latency EU endpoints and LiteLLM response caching +- Claude models via the subprocess bridge perform well: 101–143 ms TTFT is competitive with direct cloud APIs +- DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer +- TPS figures are char/s estimates; actual token/s ≈ TPS ÷ 3.5 diff --git a/services/mcp-gateway.md b/services/mcp-gateway.md index 8010757..0a83217 100644 --- a/services/mcp-gateway.md +++ b/services/mcp-gateway.md @@ -14,7 +14,7 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve ## Server inventory (as of 2026-05-22) -26 servers configured in `mcp-gateway/config.json`. **Status: 26 healthy, 1 disabled (nextcloud re-enabled 2026-05-22).** +26 servers configured in `mcp-gateway/config.json`. **Status: 27 healthy.** | Group | Server | Kind | Notes | |---|---|---|---| @@ -35,6 +35,7 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve | | wikipedia-mcp | catalog | | | | youtube-transcript | catalog | `mcp/youtube-transcript` | | **personal** | bluesky | static | `ariel-mcp:8000` | +| | obsidian | spawn | `mcp-obsidian-bridge` + mcpvault; vault at Nextcloud `fkrebs@nucli.de/files/Notizen` (UNAS mount) | | | gotify | spawn | | | | home-assistant | static | `http://192.168.1.60:9583/private_...` (HA add-on) | | | immich | spawn | |