diff --git a/CLAUDE.md b/CLAUDE.md index 0d770af..d2fb3d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,9 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26). + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: + - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. + - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 4edf25c..d5e9375 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -92,17 +92,43 @@ This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry. -```bash -# on the docker host (LXC 104) -docker run -d --name nexa-embed \ - --restart unless-stopped \ - -p 127.0.0.1:8080:80 \ - -v tei-data:/data \ - ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 \ - --model-id BAAI/bge-m3 +The docker LXC is the **Dockge** flavor of `proxmox-helper-scripts` — paste the stack into the Dockge UI (`http://192.168.1.40:5001` or whatever Zoraxy host is configured), name it `nexa`, save, start. Don't `docker compose up -d` over SSH; Dockge maintains the compose YAML at `/opt/stacks/nexa/compose.yaml`. + +```yaml +# Dockge stack: nexa (paste in the UI) +services: + nexa-embed: + image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 + container_name: nexa-embed + restart: unless-stopped + command: ["--model-id", "BAAI/bge-m3"] + ports: + - "127.0.0.1:8080:80" + volumes: + - tei-cache:/data + +volumes: + # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues + # (the same kind that bit the Nextcloud rollout). Stores embedding model + # caches, Qdrant snapshots, future GraphDB exports. + tei-cache: + driver: local + driver_opts: + type: cifs + device: //192.168.1.31/nexa/tei-cache + o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" + +networks: + default: + external: true + name: nexa ``` -Memory budget: ~1.1 GB resident (bge-m3 is ~1 GB + ~100 MB overhead). First start downloads the model into the named volume; subsequent restarts are instant. +Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant. + +> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership. +> +> Add `UNAS_USER` / `UNAS_PASS` to the Dockge stack's `.env`, or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#13](./12-optimization-opportunities.md)). Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 6d174f8..bc20524 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -47,7 +47,7 @@ Items that **block progress** and need a human decision before a workflow can be - **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%. - **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26). - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. - - **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27). + - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27). - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 4e26dda..f301290 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -60,9 +60,16 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions: - - Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it. - - Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13. - - Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers. - - Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up. +27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets: + - **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs. + - **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow. + - **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files. + - **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant//` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy. + - **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. + - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. + + Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths. +28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. +29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. +30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). 28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.