Refine capacity numbers from LXC 104 summary; flag iGPU passthrough fix
- LXC 104 (docker) confirmed: 16 CPU, 31.25 GiB RAM (7.86 used / 25%), 200 GiB boot disk at 47.7% — disk is the real near-term constraint, not RAM. Numbers propagated into Q11 (docs/11), docs/03 architecture table, and CLAUDE.md infra block. - New optimization #26 in docs/12: fix the Intel Arc iGPU passthrough flagged as broken in the LXC notes. Concrete /etc/pve/lxc/104.conf snippet provided. Once working, TEI / infinity / Immich CLIP can run GPU-accelerated for ~5–10x throughput. - New optimization #27: disk-pressure mitigation playbook (move Qdrant to unas if it grows, prune docker layers monthly, cap n8n executions). - New optimization #28: shrink the LXC's 8 GiB swap to 2 GiB.
This commit is contained in:
Claude
@@ -15,7 +15,7 @@ This file tells future automated runs what they need to know about this repo.
|
||||
- Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
|
||||
- LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
|
||||
- LXC 103 backrest — backup orchestration.
|
||||
- LXC 104 docker — main docker host at `192.168.1.40` (40 containers).
|
||||
- LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26).
|
||||
- LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`.
|
||||
- LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13).
|
||||
- LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`.
|
||||
|
||||
@@ -39,7 +39,7 @@ A one-page mental model. For details follow the cross-links.
|
||||
|
||||
| Component | Role | Where it runs (today) |
|
||||
|-----------|------|-----------------------|
|
||||
| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` |
|
||||
| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` |
|
||||
| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` |
|
||||
| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) |
|
||||
| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) |
|
||||
|
||||
@@ -43,18 +43,19 @@ Items that **block progress** and need a human decision before a workflow can be
|
||||
|
||||
## Hardware / capacity
|
||||
|
||||
- [x] **Q11 — Capacity confirmed from Proxmox node summary.**
|
||||
- Host (NUC 14 Pro): **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62.12 GiB RAM**, **1.64 TiB disk** (0.35% used).
|
||||
- Current: 32.4 GiB used (~52%), of which **ZFS ARC accounts for ~24 GiB**. Load avg 1.99 / 1.39 / 1.15 — idle. IO delay 0.04% — disk is not a bottleneck.
|
||||
- LXC 104 (docker) is currently capped at ~31 GiB; if Phase-3 squeezes (~7 GB additional: Qdrant ~1.5 GB + TEI/bge-m3 ~1.1 GB + GraphDB 4 GB heap + later +infinity/jina-clip-v2 ~1 GB) we have two levers, in order of preference: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently uses 24 GB). Either is one line of config.
|
||||
- **Verdict: comfortable. No action needed at Phase-3.1 start.** Revisit if/when 3.4 GraphDB lands and actual heap usage exceeds the budget.
|
||||
- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.**
|
||||
- **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
|
||||
- **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
|
||||
- **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
|
||||
- **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27).
|
||||
- **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
|
||||
- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports?
|
||||
|
||||
---
|
||||
|
||||
## Speed budget (Q3 follow-up)
|
||||
|
||||
Workload on the docker host (host: 22-thread Intel Core Ultra 7 155H, 62 GiB RAM, ~30 GiB free; LXC 104 currently capped at 31 GiB):
|
||||
Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap):
|
||||
|
||||
| Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` |
|
||||
|------|--------|----------------|---------------------------------|------------------------------------|
|
||||
|
||||
@@ -42,3 +42,27 @@ Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of
|
||||
23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
|
||||
24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
|
||||
25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
|
||||
|
||||
## Docker LXC (104) — observations & wins
|
||||
|
||||
Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure.
|
||||
|
||||
26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
|
||||
```
|
||||
lxc.cgroup2.devices.allow: c 226:0 rwm
|
||||
lxc.cgroup2.devices.allow: c 226:128 rwm
|
||||
lxc.cgroup2.devices.allow: c 29:0 rwm
|
||||
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
|
||||
lxc.idmap: u 0 100000 65536
|
||||
lxc.idmap: g 0 100000 65536
|
||||
lxc.idmap: g 44 44 1 # video group on host
|
||||
lxc.idmap: g 104 104 1 # render group on host
|
||||
```
|
||||
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
|
||||
27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions:
|
||||
- Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it.
|
||||
- Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13.
|
||||
- Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers.
|
||||
- Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up.
|
||||
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
|
||||
|
||||
Reference in New Issue
Block a user