fkrebs/nexa
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Pin homelab-wide UNAS storage convention; add Phase 6 + info wishlist

Browse Source 
- UNAS layout decoded from the user's tree: services/<svc>/ is the
  homelab-wide docker config store (every container follows the same
  pattern — immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid,
  shelfmark, stremio, traccar, vaultwarden, gluetun). Nexa MUST follow
  the same pattern at services/nexa/. backup/<svc>/ for snapshots.
  Pinned in CLAUDE.md and docs/12 #31.
- Stale services flagged for retirement: services/siyuan/ (migrated to
  Obsidian) and services/open-webui/ (unused, only LobeHub is alive).
  Drops the "two LLM UIs" item (#15) — it's now "retire open-webui".
- Hard-blocklist for any Nexa indexer pinned in CLAUDE.md:
  _sortMe/wallet/**, *.gpg/asc/key/pem/kdbx/credentials/secret, plus
  the Nextcloud appdata dir.
- Three new "future user-facing wins" surfaced from the tree:
  Paperless-AI as a Phase-2.x triage helper for _sortMe/Downloads/,
  media/Recipes/ (~300 entries) as the showcase RAG corpus, and
  Paperless-AI's existing ChromaDB as a potential read-from source
  rather than re-embedding scanned docs.
- Housekeeping campaign in docs/12 §35-37: consolidate postgres,
  audit UNAS-everywhere, S3 archive tier on s3.nuclide.systems.
- Phase 6 added to the roadmap: "Nexa as homelab steward". Polls
  Dockge/docker/Proxmox, diffs vs documented state, emits a daily
  drift report. Most of the housekeeping campaign becomes
  semi-automatic once 6.1-6.3 ship.
- New docs/13-information-wishlist.md packages the still-needed
  inventory as 5-6 paste-and-run tasks, each with explicit "📍 Where"
  markers (which shell or which UI) and what it unblocks. Highest
  leverage = Task 1 (read an existing compose stack to lock Q19).
- docs/index.md TOC extended to 13 rows.

Also fixed numbering drift in docs/12 (duplicate #28, missing #17/#18)
and added Q19 to docs/11 covering the SMB-share verification step.
This commit is contained in:
Claude
2026-05-04 22:24:24 +00:00
parent a0a67eefab
commit 250a5cb2a6
7 changed files with 207 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/12-optimization-opportunities.md
+64 -24
View File
@@ -20,34 +20,45 @@ Observations from the running infrastructure. Each item is independent — accep
11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk.
12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.
13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded.
14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later.
15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation:
- **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
- **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A.
- **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus.
Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.
15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.
16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).
17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.
20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.
## Proxmox host (NUC 14 Pro) tuning
Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:
17. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):
21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):
```
echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB
update-initramfs -u
```
Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live.
18. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
19. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
20. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
21. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
22. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
## Docker LXC (104) — observations & wins
Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure.
26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
```
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
@@ -60,16 +71,45 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i
lxc.idmap: g 104 104 1 # render group on host
```
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets:
- **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs.
- **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow.
- **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files.
- **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant/<date>/` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy.
- **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services/<svc>/` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets:
- `services/nexa/qdrant/` — Qdrant data dir (move before the visual collection lands).
- `services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk).
- `services/nexa/graphdb/` — Phase-3.4 RDF store.
- `backup/nexa/snapshots/qdrant/<date>/` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup/<svc>/` pattern used by Home Assistant / Immich.
- **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
- **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths.
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH.
30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19).
Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths.
32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
33. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH.
34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
## Housekeeping campaign (cross-cutting, do together)
These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.
35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Dockge: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths:
- **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron.
- **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.
Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running.
36. **UNAS-integration audit.** `services/<svc>/` on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the `volumes:` block. Suspected non-compliant (need verification):
- `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection.
- `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state.
- `memos` — the user-facing notes app; loss = data loss.
- `n8n` — workflows + executions DB.
- `audiobookshelf` — listening progress + library metadata.
- the `*-mcp` containers — likely stateless (cache only), low priority.
Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.
37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup/<svc>/` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier:
- **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination.
- **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore).
- **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars.
- Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target.
This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs.