fkrebs/nexa
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

SMB-default for Nexa volumes (user confirmed NFS+unpriv-LXC writes break)

Browse Source 
User's Nextcloud rollout already hit the classic trap of unprivileged-LXC
write attempts against an NFS share, so don't repeat that pattern for
Nexa. Updated docs to:

- Use SMB volumes mounted directly as docker volumes (driver: local,
  type: cifs) with explicit username/uid/gid — bypasses LXC uid-mapping
  entirely. Concrete YAML in docs/09 step 3 against
  //192.168.1.31/nexa/<volume>.
- Keep the host-side /mnt/pve/unas NFS mount for admin-only use (rsync,
  manual snapshot copies). Don't bind-mount it into containers.
- docs/12 #27 reordered to lead with SMB targets (qdrant, graphdb,
  tei-cache, snapshots) and only mention NFS for host admin.
- CLAUDE.md storage block flipped to "SMB = default per-container
  storage; NFS = host-side admin only".
- Also captured: deployments go through the Dockge UI (LXC is
  the Dockge flavor of helper-scripts), not raw `docker compose up`.
- New optimization #30 about the local mail spool nag on the docker LXC
  ("You have new mail." at login).
This commit is contained in:
Claude
2026-05-04 22:14:12 +00:00
parent 12ed4426e0
commit a0a67eefab
4 changed files with 51 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -1
View File
@@ -15,7 +15,9 @@ This file tells future automated runs what they need to know about this repo.
- Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
- LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
- LXC 103 backrest — backup orchestration.
- LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26).
- LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways:
- **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts.
- **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26).
- LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`.
- LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13).
- LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`.
docs/09-deployment.md
+35 -9
View File
@@ -92,17 +92,43 @@ This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB
Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.
```bash
# on the docker host (LXC 104)
docker run -d --name nexa-embed \
--restart unless-stopped \
-p 127.0.0.1:8080:80 \
-v tei-data:/data \
ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 \
--model-id BAAI/bge-m3
The docker LXC is the **Dockge** flavor of `proxmox-helper-scripts` — paste the stack into the Dockge UI (`http://192.168.1.40:5001` or whatever Zoraxy host is configured), name it `nexa`, save, start. Don't `docker compose up -d` over SSH; Dockge maintains the compose YAML at `/opt/stacks/nexa/compose.yaml`.
```yaml
# Dockge stack: nexa (paste in the UI)
services:
nexa-embed:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
container_name: nexa-embed
restart: unless-stopped
command: ["--model-id", "BAAI/bge-m3"]
ports:
- "127.0.0.1:8080:80"
volumes:
- tei-cache:/data
volumes:
# SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues
# (the same kind that bit the Nextcloud rollout). Stores embedding model
# caches, Qdrant snapshots, future GraphDB exports.
tei-cache:
driver: local
driver_opts:
type: cifs
device: //192.168.1.31/nexa/tei-cache
o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8"
networks:
default:
external: true
name: nexa
```
Memory budget: ~1.1 GB resident (bge-m3 is ~1 GB + ~100 MB overhead). First start downloads the model into the named volume; subsequent restarts are instant.
Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant.
> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership.
>
> Add `UNAS_USER` / `UNAS_PASS` to the Dockge stack's `.env`, or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#13](./12-optimization-opportunities.md)).
Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter:
docs/11-open-questions.md
+1 -1
View File
@@ -47,7 +47,7 @@ Items that **block progress** and need a human decision before a workflow can be
- **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
- **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
- **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
- **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27).
- **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27).
- **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports?
docs/12-optimization-opportunities.md
+12 -5
View File
@@ -60,9 +60,16 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i
lxc.idmap: g 104 104 1 # render group on host
```
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions:
- Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it.
- Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13.
- Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers.
- Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up.
27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets:
- **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs.
- **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow.
- **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files.
- **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant/<date>/` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy.
- **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
- **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths.
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH.
30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.