From 12ed4426e056d3332f252fcd1652c85c1bf9010c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:10:51 +0000 Subject: [PATCH] Refine capacity numbers from LXC 104 summary; flag iGPU passthrough fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - LXC 104 (docker) confirmed: 16 CPU, 31.25 GiB RAM (7.86 used / 25%), 200 GiB boot disk at 47.7% — disk is the real near-term constraint, not RAM. Numbers propagated into Q11 (docs/11), docs/03 architecture table, and CLAUDE.md infra block. - New optimization #26 in docs/12: fix the Intel Arc iGPU passthrough flagged as broken in the LXC notes. Concrete /etc/pve/lxc/104.conf snippet provided. Once working, TEI / infinity / Immich CLIP can run GPU-accelerated for ~5–10x throughput. - New optimization #27: disk-pressure mitigation playbook (move Qdrant to unas if it grows, prune docker layers monthly, cap n8n executions). - New optimization #28: shrink the LXC's 8 GiB swap to 2 GiB. --- CLAUDE.md | 2 +- docs/03-architecture.md | 2 +- docs/11-open-questions.md | 13 +++++++------ docs/12-optimization-opportunities.md | 24 ++++++++++++++++++++++++ 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ea2514a..0d770af 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,7 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). + - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/03-architecture.md b/docs/03-architecture.md index 6db8376..3e0e9d1 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -39,7 +39,7 @@ A one-page mental model. For details follow the cross-links. | Component | Role | Where it runs (today) | |-----------|------|-----------------------| -| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | +| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` | | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | | **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) | diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index aae05b1..6d174f8 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -43,18 +43,19 @@ Items that **block progress** and need a human decision before a workflow can be ## Hardware / capacity -- [x] **Q11 — Capacity confirmed from Proxmox node summary.** - - Host (NUC 14 Pro): **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62.12 GiB RAM**, **1.64 TiB disk** (0.35% used). - - Current: 32.4 GiB used (~52%), of which **ZFS ARC accounts for ~24 GiB**. Load avg 1.99 / 1.39 / 1.15 — idle. IO delay 0.04% — disk is not a bottleneck. - - LXC 104 (docker) is currently capped at ~31 GiB; if Phase-3 squeezes (~7 GB additional: Qdrant ~1.5 GB + TEI/bge-m3 ~1.1 GB + GraphDB 4 GB heap + later +infinity/jina-clip-v2 ~1 GB) we have two levers, in order of preference: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently uses 24 GB). Either is one line of config. - - **Verdict: comfortable. No action needed at Phase-3.1 start.** Revisit if/when 3.4 GraphDB lands and actual heap usage exceeds the budget. +- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.** + - **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%. + - **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26). + - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. + - **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27). + - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? --- ## Speed budget (Q3 follow-up) -Workload on the docker host (host: 22-thread Intel Core Ultra 7 155H, 62 GiB RAM, ~30 GiB free; LXC 104 currently capped at 31 GiB): +Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap): | Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` | |------|--------|----------------|---------------------------------|------------------------------------| diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index fd86d65..4e26dda 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -42,3 +42,27 @@ Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of 23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. 24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. 25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. + +## Docker LXC (104) — observations & wins + +Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure. + +26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: + ``` + lxc.cgroup2.devices.allow: c 226:0 rwm + lxc.cgroup2.devices.allow: c 226:128 rwm + lxc.cgroup2.devices.allow: c 29:0 rwm + lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file + lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file + lxc.idmap: u 0 100000 65536 + lxc.idmap: g 0 100000 65536 + lxc.idmap: g 44 44 1 # video group on host + lxc.idmap: g 104 104 1 # render group on host + ``` + Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. +27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions: + - Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it. + - Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13. + - Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers. + - Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up. +28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.