nexa/docs/12-optimization-opportunities.md

12 — Optimization Opportunities

Observations from the running infrastructure. Each item is independent — accept, defer, or reject.

For Nexa directly

1. Reuse, don't redeploy. The earlier DEPLOYMENT.md would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new 09-deployment treats these as pre-existing — keeps the config minimal and avoids port collisions.
2. Use LiteLLM virtual keys per logical caller. Today there's one SAIA key. Issuing one key per workflow (nexa-router, nexa-embed, nexa-digest) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything.
3. Use n8n's credential objects, never inline secrets. The current workflows under nexa-core/n8n-workflows/phase-1/*.json should be reviewed — if any header Authorization is hardcoded, replace with credential references before importing.
4. Centralise system alerts on a single ntfy topic (nexa.system). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them.
5. Defer graph DB until Phase 3.4. Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries.
6. Auto-export n8n workflows. nexa-core/scripts/backup_workflows.sh already exists. Schedule it inside the n8n container (cron) and let it git commit && git push — this is the cheapest disaster recovery.

For the wider homelab (out of scope but worth noting)

7. AI gateway naming. ai.nuclide.systems currently proxies LobeHub (a chat UI on :3210), while the LiteLLM API lives on :4000. For Nexa, point n8n directly at LiteLLM (http://192.168.1.40:4000 over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts.

8. MCP servers consolidation. Dozzle shows crawl4ai-mcp, markitdown-mcp, papersearch-mcp running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding).

9. Backup the n8n SQLite file — Backrest covers /home/node/.n8n if added; today the only "backup" is the workflow JSON which omits credentials and execution history.

10. Pocket-ID SSO in front of n8n would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win.

11. Vaultwarden as the secret store for Nexa secrets (SAIA_API_KEY, MEMOS_API_KEY, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a .env on disk.

12. AdGuard as DNS-based control plane. Since AdGuard is the resolver for the LAN, you can rewrite *.nuclide.systems to 192.168.1.4 (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.

13. Disk usage on LXC 104 is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting EXECUTIONS_DATA_PRUNE=true and EXECUTIONS_DATA_MAX_AGE=168 (7 days) on n8n keeps it bounded.

14. Vector-store sprawl in the homelab — three competing indexes today. Nexa is about to be the fourth. Track for eventual consolidation:

    • Obsidian Copilot (.copilot, .copilot-index) and Smart Connections / Smart Composer (.smart-env, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
    • Paperless-AI ChromaDB (services/paperless-ai/chromadb/) — embeds scanned documents from Paperless-ngx for Q&A.
    • Nexa Qdrant (nexa_knowledge_text, soon _visual) — embeds the cross-source corpus.

    Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa read from Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.

15. Retire open-webui. Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (ai.nuclide.systems). Stop the container, tar services/open-webui/ into backup/open-webui/, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.

16. services/siyuan/workspace/ is dead data. SiYuan retired, content migrated to Obsidian (Nextcloud Notizen/). Keep a final tar in backup/siyuan/, then rm -rf services/siyuan/. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).

17. Real-time Obsidian sync via notify_push. Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's notify_push app for sub-second propagation. One-line workflow change in n8n.

18. assets/ is 186 MB of binaries in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.

19. Paperless-AI as a Phase-2.x triage helper for _sortMe/Downloads/. UNAS shows ~230 PDFs/docs in _sortMe/Downloads/ plus another batch under _sortMe/Anne/. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to delegate — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.

20. media/Recipes/ has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for #nexa:ask (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.

Proxmox host (NUC 14 Pro) tuning

Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:

21. Cap ZFS ARC. Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):

    echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf   # 8 GiB
    update-initramfs -u
    

    Reboot or echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max to apply live.
22. Enable KSM (Kernel Same-page Merging). With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently KSM sharing: 0 B in the summary. PVE has ksmtuned available — systemctl enable --now ksmtuned.
23. Suppress the pve-no-subscription repository warning — either accept it (it's a homelab) and apply the pve-no-subscription-warning polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
24. Swap is 31 GiB on a 62 GiB box with ZFS root — almost certainly oversized. Drop vm.swappiness to 10 (sysctl -w vm.swappiness=10 + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
25. Verify scheduled ZFS scrub is enabled. PVE ships zfs-scrub-monthly@.timer — systemctl list-timers | grep zfs to confirm. Cheap insurance on a 1.6 TiB pool.
26. SMART monitoring on the NVMe. smartctl -a /dev/nvme0 should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing nexa.system ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
27. NTP source via AdGuard. AdGuard already resolves DNS for the LAN; pointing the host's systemd-timesyncd at pool.ntp.org resolved through AdGuard avoids any external dependency for time. One-line change in /etc/systemd/timesyncd.conf.
28. fstrim.timer enabled for the SSD pool — verify with systemctl status fstrim.timer. Default-on in modern PVE, but quick to confirm.
29. Watchdog config is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.

Docker LXC (104) — observations & wins

Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), 200 GiB boot disk at 47.7% used — disk pressure outranks RAM pressure.

30. Fix the Intel iGPU passthrough. The container's own notes flag the binding as "likely failing". Once /dev/dri/{card0,renderD128} is visible inside the LXC, both TEI and infinity can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically 5–10× faster than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in /etc/pve/lxc/104.conf:

    lxc.cgroup2.devices.allow: c 226:0 rwm
    lxc.cgroup2.devices.allow: c 226:128 rwm
    lxc.cgroup2.devices.allow: c 29:0 rwm
    lxc.mount.entry: /dev/dri/card0       dev/dri/card0       none bind,optional,create=file
    lxc.mount.entry: /dev/dri/renderD128  dev/dri/renderD128  none bind,optional,create=file
    lxc.idmap: u 0 100000 65536
    lxc.idmap: g 0 100000 65536
    lxc.idmap: g 44 44 1            # video group on host
    lxc.idmap: g 104 104 1          # render group on host
    

    Then inside the container: usermod -aG video,render <docker-user> and run TEI with --device cuda replaced by the OpenVINO build (text-embeddings-inference:cpu-1.5-openvino). Not needed for Phase-3.1's volume but a clean upgrade path.

31. Match the homelab-wide UNAS storage convention. Every existing docker stack on LXC 104 already binds its persistent data into services/<svc>/ on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa must follow the same pattern at services/nexa/. Do not invent a parallel layout. Targets:

    • services/nexa/qdrant/ — Qdrant data dir (move before the visual collection lands).
    • services/nexa/tei-cache/ — embedding model cache (keeps multi-GB models off the boot disk).
    • services/nexa/graphdb/ — Phase-3.4 RDF store.
    • backup/nexa/snapshots/qdrant/<date>/ — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the backup/<svc>/ pattern used by Home Assistant / Immich.
    • n8n executions retention: EXECUTIONS_DATA_PRUNE=true, EXECUTIONS_DATA_MAX_AGE=168.
    • Monthly docker image prune --all --filter "until=720h" to clear dangling layers.

    Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver local, type cifs) with explicit username=/uid=/gid= bypasses LXC uid-mapping entirely. Pre-deploy step: read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as Q19.

    Credentials (UNAS_USER / UNAS_PASS) go in Vaultwarden (12/#11), injected at deploy. The host-side /mnt/pve/unas NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths.

32. The LXC has its own 8 GiB swap. Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (pct set 104 -swap 2048) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.

33. Stacks live in Dockge, not raw compose. The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named nexa in the Dockge UI; the compose YAML is stored under /opt/stacks/nexa/ (Dockge default) and edited from the web UI. This means docs/09 "deploy" snippets translate to "paste this into Dockge → save → start", not docker compose up -d over SSH.

34. Untriaged local mail on the LXC. Console shows You have new mail. at login — the system mail spool on /var/mail/root has unread messages, almost always cron job failures. mailx or mutt to inspect, then either fix the failing job or send the spool to nexa.system ntfy via a tiny aliases entry (root: |/usr/local/bin/spool-to-ntfy.sh).

Housekeeping campaign (cross-cutting, do together)

These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto Phase 6 (Nexa as homelab steward): once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.

35. Consolidate Postgres instances. Today there are at least four independent Postgres containers running — visible from Dockge: immich_postgres, lobe-postgres, litellm_db, paperless-ngx-db-1 (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths:

    • Single shared cluster (pgsql container with one role per app, one database per app). Most modern apps support DATABASE_URL → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron.
    • Or migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.

    Pre-step: list every running container with docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql' to inventory exactly what's running.

36. UNAS-integration audit. services/<svc>/ on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the volumes: block. Suspected non-compliant (need verification):

    • qdrant_scientific — vector data possibly on local boot disk; critical to verify before Phase-3.1 piles on a nexa_knowledge_text collection.
    • litellm_db, lobe-postgres, lobe-redis — DB containers, persistent state.
    • memos — the user-facing notes app; loss = data loss.
    • n8n — workflows + executions DB.
    • audiobookshelf — listening progress + library metadata.
    • the *-mcp containers — likely stateless (cache only), low priority.

    Output: a one-page table service | persistent? | currently bound to | should be bound to. Then migrate the non-compliant ones one-by-one (stop → rsync data to services/<svc>/ on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.

37. S3 archive tier (s3.nuclide.systems). Currently UNAS holds backups in backup/<svc>/ and there's no cold offsite. Make s3.nuclide.systems the canonical cold tier:

    • Restic / borg repos targeting S3 for: backup/home-assistant/, backup/immich/, backup/nextcloud/, future backup/nexa/. Backrest already orchestrates Borg — just add an S3 destination.
    • Qdrant + GraphDB snapshots flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore).
    • Immich photo originals to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its IMMICH_* env vars.
    • Resolves Q12 — once the bucket name + access key are set, every other archive just adds a target.

    This is the right time to do it because we're about to add Nexa snapshots; designing the path for all services rather than just Nexa avoids one-offs.