fkrebs/nexa
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

SMB-default for Nexa volumes (user confirmed NFS+unpriv-LXC writes break)

Browse Source 
User's Nextcloud rollout already hit the classic trap of unprivileged-LXC
write attempts against an NFS share, so don't repeat that pattern for
Nexa. Updated docs to:

- Use SMB volumes mounted directly as docker volumes (driver: local,
  type: cifs) with explicit username/uid/gid — bypasses LXC uid-mapping
  entirely. Concrete YAML in docs/09 step 3 against
  //192.168.1.31/nexa/<volume>.
- Keep the host-side /mnt/pve/unas NFS mount for admin-only use (rsync,
  manual snapshot copies). Don't bind-mount it into containers.
- docs/12 #27 reordered to lead with SMB targets (qdrant, graphdb,
  tei-cache, snapshots) and only mention NFS for host admin.
- CLAUDE.md storage block flipped to "SMB = default per-container
  storage; NFS = host-side admin only".
- Also captured: deployments go through the Dockge UI (LXC is
  the Dockge flavor of helper-scripts), not raw `docker compose up`.
- New optimization #30 about the local mail spool nag on the docker LXC
  ("You have new mail." at login).
This commit is contained in:
Claude
2026-05-04 22:14:12 +00:00
parent 12ed4426e0
commit a0a67eefab
4 changed files with 51 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/12-optimization-opportunities.md
+12 -5
View File
@@ -60,9 +60,16 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i
lxc.idmap: g 104 104 1 # render group on host
```
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions:
- Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it.
- Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13.
- Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers.
- Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up.
27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets:
- **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs.
- **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow.
- **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files.
- **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant/<date>/` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy.
- **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
- **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths.
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH.
30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.