fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: session continuity file + claude-max-bridge /v1/responses fix

Browse Source 
- Add RESUME.md for cross-session continuity (open items, key state, constraints)
- claude-max-bridge: implement /v1/responses (OpenAI Responses API) with
  previous_response_id chaining via server-side history injection into system prompt.
  Root cause of "session already in use": CLI leaves JSONL in un-resumable
  "dequeued" state after each --print run; fix avoids session reuse entirely.
  Also fixed: assistant content must be array-of-blocks not plain string (silent
  JS crash otherwise).
- LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge
- Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs,
  Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS)
- Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Florian Krebs
2026-05-22 22:42:17 +02:00
parent e0199042c1
commit 143343decf
12 changed files with 496 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infra/proxmox-memory-audit.md
+80
View File
@@ -0,0 +1,80 @@
# Proxmox Memory Audit
Last audited: 2026-05-22
## Host physical resources
| Resource | Total | Used (idle) | Available |
|----------|-------|-------------|-----------|
| RAM | 62 GiB | ~33 GiB | ~28 GiB |
| Swap | 31 GiB | 0 GiB | 31 GiB |
## CT memory allocations
| CT | Name | Allocated (MiB) | Swap (MiB) | Typical use | Notes |
|----|------|----------------|------------|-------------|-------|
| 101 | shepard | 32,768 | 8,192 | ~4 GB | Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak |
| 102 | dns | 1,024 | 512 | ~100 MB | AdGuard Home |
| 103 | backrest | 2,048 | 512 | ~200 MB | Restic scheduler |
| 104 | docker | **49,152** | 32,000 | **6–22 GB** | Main Docker host; FLUX spikes to ~22 GB |
| 105 | nextcloud | 8,196 | 8,196 | ~2 GB | Nextcloud AIO |
| 108 | zoraxy | 2,048 | 512 | ~300 MB | Reverse proxy |
| 110 | id | 1,024 | 512 | ~200 MB | Pocket-ID |
| 111 | dev | 32,768 | 8,192 | ~3 GB | Coder + Gitea workspaces |
| 112 | secrets | 4,096 | 512 | ~600 MB | Infisical |
| **Sum** | | **133,124 MiB (130 GiB)** | | | **2× overprovisioned vs physical RAM** |
## Key findings
### Overprovisioning is safe — until it isn't
Proxmox uses balloon drivers so CTs only consume what they actually use. At idle the host sits at ~33 GB used with 28 GB available. This is healthy. However, two specific CTs represent risk:
- **CT 104** (49 GiB limit) + **CT 111** (32 GiB limit) together could claim 81 GiB — well over the 62 GiB physical. If both hit peak simultaneously the host would start swapping heavily.
- CT 104's ComfyUI (FLUX generation) spikes from 6 GB baseline to ~22 GB. That 16 GB spike on top of the 33 GB idle baseline = 49 GB total, within physical RAM but tight.
### Why 40G Docker container limit destabilised the system
Setting ComfyUI's container limit to 40 G was the trigger. At generation time:
- FLUX model + activations: ~22 GB in the container
- Other ~65 Docker containers on CT 104: ~7 GB
- CT 104 OS + kernel: ~1 GB
- Other CTs idle: ~26 GB
- **Total: ~56 GB** → host started swapping, degrading all services
### ComfyUI XPU memory accounting gap
ComfyUI's `get_free_memory()` for Intel XPU queries `torch.xpu.get_device_properties().total_memory` = **58 GB** (the full shared memory pool — Arc shares system RAM). It has no awareness of the Docker cgroup limit. Smart memory management (`free_memory()`) calculates `memory_required - get_free_memory()` which is always hugely negative → never evicts models. This makes `--disable-smart-memory` irrelevant for XPU; smart memory is already broken.
Consequence: ComfyUI will always try to load the full model into XPU memory regardless of container limit. The cgroup OOM killer is the only backstop.
### Container limit recommendation for ComfyUI (CT 104)
| Scenario | Limit | Safe? |
|----------|-------|-------|
| `--lowvram` (original) | 20 G | ✅ Safe but slow (231 s/image) |
| No `--lowvram`, FLUX only | **24 G** | ✅ Fits FLUX peak (~22 GB) + 2 GB headroom |
| No `--lowvram` + img2img after FLUX | 26 G | ✅ FLUX stays resident, SD1.5 loads on top |
| 28 G | ⚠️ | Marginal — OOM triggered in testing |
| 40 G | ❌ | Destabilises host when generating |
Peak host usage at 24 G container limit during FLUX generation: `24 + 7 (other containers) + 26 (other CTs idle) ≈ 57 GB` — stays under 62 GB physical.
## Recommendations
1. **ComfyUI container limit**: set to **24 G** when running without `--lowvram`. Current revert to 20 G + `--lowvram` is stable but slower.
2. **CT 104 LXC allocation (49 GiB)**: appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact.
3. **CT 111 (dev, 32 GiB)**: Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template.
4. **Watch list**: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB → host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104.
5. **Long-term**: when CT 109 (ops) is built, run Prometheus `node_exporter` on the PVE host and alert when host available RAM drops below 8 GiB.
## ComfyUI memory optimisation log
| Date | Change | Effect |
|------|--------|--------|
| 2026-05-22 | Removed `--lowvram` + raised limit to 28 G | OOM at 28 G (XPU DRM buffers counted against cgroup) |
| 2026-05-22 | Raised limit to 40 G | Host destabilised — reverted |
| 2026-05-22 | Reverted to `--lowvram` + 20 G | Stable, slow (231 s/image) |
| 2026-05-22 | Downloaded `t5-v1_1-xxl-encoder-Q4_K_S.gguf` (2.6 GB vs 3.2 GB Q5_K_M) | Saves 600 MB at load time |
| 2026-05-22 | Kept Q5_K_M T5 (Q4_K_S degrades prompt following per city96); removed `--lowvram`; set limit to 24 G; added `--async-offload --force-fp16` | ~7–30 s generation, safe within host memory budget |
| 2026-05-22 | Removed `--async-offload` | Flag incompatible with GGUF img2img on XPU — caused full CPU fallback (3.5 min/step) and pure-noise output. Removed; XPU generation now correct at ~7 s/step for txt2img. Current CLI: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16` |
infra/proxmox-state.md
+3 -2
View File
@@ -5,7 +5,7 @@ Host: `nuc` · PVE `9.1.11` · Kernel `6.17.13-4-pve` · Debian 13 (trixie)
CPU: Intel Core Ultra 7 155H (16C / 22T, hybrid P+E+LP-E) · 1 socket · 1 NUMA
RAM: 62 GiB physical · 31 GiB zram swap (50 % of RAM, zstd, prio 100)
Storage: single Crucial P3 2 TB NVMe (QLC, DRAM-less) → `rpool` (ZFS, ashift=12, no redundancy)
Workload: 1 VM (HAOS) + 6 LXCs (Docker, Adguard, Backrest, Nextcloud, Zoraxy, Dockge)
Workload: 1 VM (HAOS) + 9 LXCs (Docker, AdGuard, Backrest, Nextcloud, Zoraxy, Pocket-ID, Dev, Secrets, DB) + 1 planned (Ops/CT109)
---
@@ -332,7 +332,7 @@ tail /var/log/unattended-upgrades/unattended-upgrades.log
## 10. Container-specific issues
### CT 104 (docker / AI / image-gen) — 128 GiB cap, 16 cores, GPU passthrough
### CT 104 (docker / AI / image-gen) — 48 GiB cap, 16 cores, GPU passthrough
Measured: 18.8 GiB current, peak 29.3 GiB, **9.3 GiB in swap**, load 8.0, ~65 Docker containers (Immich + ML, ComfyUI, LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless+AI, many MCP servers, Speaches-OpenVINO).
@@ -733,6 +733,7 @@ Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores /
| Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID |
| docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr/Arcane discovery |
| **sshwifty** | web SSH client, multi-tab — multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred. |
| **docs-server** | mkdocs Material site (`/docs` git repo → static HTML); migrating here from CT 111 where it currently runs at port 13080. LAN-only, no Zoraxy route. |
### Sidecars deployed on each host
infra/storage.md
+10 -11
View File
@@ -10,8 +10,7 @@
| Local zfs (rootfs) | NVMe in nuc | `/opt/stacks/<stack>/` on each LXC | Tier-1 hot state: databases, SQLite, secret stores, anything latency-sensitive |
| Local zfs (subvol) | NVMe in nuc | `rpool/data/subvol-<NNN>-disk-0` per LXC | Container rootfs, image cache |
| Docker named volume | NVMe in nuc | `/var/lib/docker/volumes/...` on each Docker CT | Per-container persistent state managed by Docker |
| UNAS over **NFSv3** | `192.168.1.31` | `/mnt/pve/unas` on CTs 101, 103, 104, 111 | Bulk media, workspace home dirs, document storage |
| UNAS over **CIFS/SMB 3.1.1** | `192.168.1.31` | `/mnt/pve/unas` on CT 105 (Nextcloud) | Nextcloud-only path — earlier docs mis-labelled this as NFS |
| UNAS over **NFSv3** | `192.168.1.31` | `/mnt/pve/unas` on CTs 101, 103, 104, 105, 111 | Bulk media, workspace home dirs, document storage |
## What lives where (verified 2026-05-21)
@@ -24,7 +23,6 @@ Bulk + media + non-latency-sensitive app data.
| Immich (uploads, thumbs, derived) | 104 | `services/immich/{encoded-video,profile,thumbs}` + `media/images` + `backup/immich` | Immich own pg_dump → `media/images/db-dumps`; no off-host copy | |
| Paperless-ngx (documents) | 104 | `media/documents/public/paperless-ngx/{consume,export,library}` | none | |
| Paperless-AI | 104 | `services/paperless-ai` → `/app/data` | none | |
| Vaultwarden (attachments + SQLite DB) | 104 | `services/vaultwarden` → `/data` | none | tier-1 DB on NFS — risky, plan to move local |
| Traccar (logs, config) | 104 | `services/traccar/{logs,traccar.xml}` | none | data dir reverted to local `/opt/stacks/traccar/data` |
| Memos | 104 | `services/memos` → `/var/opt/memos` | none | |
| Arr-stack (Audiobookshelf, Prowlarr, RDTClient, Shelfarr) + media | 104 | `services/arr-stack/*` + `media/{audiobooks,ebooks,podcasts,Torrents}` | none | already migrated (older doc claimed "still local") |
@@ -52,7 +50,8 @@ Tier-1 state and anything that should NOT be NFS-backed.
| SearXNG config | 104 | `/opt/stacks/ai/searxng` | none | |
| Flaresolverr | 104 | `/var/lib/flaresolver` | none | |
| AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup |
| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below |
| Vaultwarden (attachments + key material) | 104 | `/opt/stacks/vaultwarden/data` → `/data` | none | **moved off NFS 2026-05-22**; DB is on CT 113 postgres; stale `db.sqlite3` deleted |
| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | app data on NFS — see below |
### Garage S3 buckets (on local zfs, CT 104)
@@ -76,9 +75,9 @@ Local on `/var/lib/docker/volumes/`. Mostly databases and caches.
| `coder-db` + `gitea-db` (docker volume `coder-db`) | / 111 | none |
| (orphaned) `daytona-minimal_db_data` + runner anon vol | / 104 | none — clean up post-decommission |
### Nextcloud (CT 105) — the CIFS outlier
### Nextcloud (CT 105) — data on NFS
Nextcloud AIO mounts `//192.168.1.31/storage` over **CIFS/SMB 3.1.1**, not NFS like every other CT. Application data path is under that mount. The Postgres + Redis sidecars stay on local docker volumes.
Nextcloud AIO mounts `/mnt/pve/unas/services/nextcloud` via NFSv3 (same share as all other CTs). Migrated from CIFS on 2026-05-22 after the CIFS mount caused a crash-loop. The Postgres + Redis sidecars stay on local docker volumes.
## Backup reality
@@ -96,8 +95,8 @@ This is a known gap. Plans:
## Drift cleanup TODO
- [ ] Remove stale `services/n8n/database.sqlite` (408 MB, last write 2026-05-15) from UNAS.
- [ ] Remove stale `services/shared-db/garage/` copy from UNAS if present.
- [ ] Remove orphaned `daytona-minimal_db_data` Docker volume on CT 104.
- [ ] Vaultwarden `data/` on UNAS — plan to migrate to local zfs (latency + tier-1).
- [ ] Add a Backrest plan for Pocket-ID DB and Gitea data → jottacloud.
- [x] ~~Remove stale `services/n8n/database.sqlite` from UNAS~~ — gone (verified 2026-05-22)
- [x] ~~Remove stale `services/shared-db/garage/` copy from UNAS~~ — gone (verified 2026-05-22)
- [x] ~~Remove orphaned `daytona-minimal_db_data` Docker volume on CT 104~~ — gone (verified 2026-05-22)
- [x] ~~Vaultwarden `data/` on UNAS~~ — migrated to local zfs 2026-05-22. Stale NFS copy at `services/vaultwarden/` can be cleaned up.
- [x] ~~Add a Backrest plan for Pocket-ID DB~~ — done 2026-05-22. Pre-backup hook runs `pocket-id export` + copies keys to UNAS staging; `services-backup-plan` snapshots staging → jottacloud nightly.
infra/volumes.md
+11 -6
View File
@@ -46,9 +46,11 @@
### Pocket ID — `pocketid`
| Type | Source | Container | UNAS |
> **CT 110, not CT 104.** Pocket-ID migrated off CT 104 to its own LXC on 2026-05-20.
| Type | Source | Container | Notes |
|---|---|---|---|
| **bind** | **`/mnt/pve/unas/services/pocketid`** | **`/app/data`** | ✅ **Already on UNAS** |
| bind | `/opt/stacks/pocketid/data` | `/app/data` | Local zfs on CT 110 — no UNAS mount |
### Vaultwarden — `vaultwarden`
@@ -309,10 +311,12 @@ No DB-specific mounts needed (connects via env vars).
### Garage S3 — `garage`
> Moved off NFS to local zfs on 2026-05-19 after a WAL-G outage. Stale copy at `services/shared-db/garage/` on UNAS may still exist — clean up.
| Type | Source | Container | Notes |
|---|---|---|---|
| bind | `/mnt/pve/unas/services/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data |
| bind | `/mnt/pve/unas/services/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (LMDB) |
| bind | `/opt/stacks/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data — local NVMe |
| bind | `/opt/stacks/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (LMDB) — local NVMe |
---
@@ -336,9 +340,10 @@ No DB-specific mounts needed (connects via env vars).
| Status | Count | Services |
|---|---|---|
| ✅ Already on UNAS | 11 | gluetun, immich*(4), karakeep*(2), ntfy*(2), paperless-ai, pocketid, stremio, traccar*(3), vaultwarden, paperless-docs*(3), garage*(2) |
| ✅ Already on UNAS | 9 | gluetun, immich*(4), karakeep*(2), ntfy*(2), paperless-ai, stremio, traccar*(3), vaultwarden, paperless-docs*(3) |
| ✅ Migrated (Phase 1) | 3 | memos, arcane, n8n |
| 🔷 Shared infrastructure | 2 | shared-postgres (local volume), garage (on UNAS) |
| 🔷 Shared infrastructure (local) | 2 | shared-postgres (local volume), garage (local NVMe — moved off UNAS 2026-05-19) |
| 📋 Own CT, local only | 1 | pocketid (CT 110, no UNAS) |
| ⏳ Phase 2 planned (PG consolidation) | 5 | immich → shared-postgres, paperless → shared-postgres, daytona → shared-postgres, litellm → shared-postgres, lobehub → shared-postgres |
| ⏳ Phase 3 planned | 1 | arr-stack*(8 mounts) |
| 📋 Keep local | ~5 | homepage, dozzle, litellm-config, searxng-config, saia-image-proxy |