Skip to content

# 12 — Optimization Opportunities

Observations from the running infrastructure. Each item is independent — accept, defer, or reject.

## For Nexa directly

1. Reuse, don't redeploy. The earlier `DEPLOYMENT.md` would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new 09-deployment treats these as pre-existing — keeps the config minimal and avoids port collisions.
2. Use LiteLLM virtual keys per logical caller. Today there's one SAIA key. Issuing one key per workflow (`nexa-router`, `nexa-embed`, `nexa-digest`) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything.
3. Use n8n's credential objects, never inline secrets. The current workflows under `nexa-core/n8n-workflows/phase-1/*.json` should be reviewed — if any header `Authorization` is hardcoded, replace with credential references before importing.
4. Centralise system alerts on a single ntfy topic (`nexa.system`). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them.
5. Defer graph DB until Phase 3.4. Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries.
6. Auto-export n8n workflows. `nexa-core/scripts/backup_workflows.sh` already exists. Schedule it inside the n8n container (cron) and let it `git commit && git push` — this is the cheapest disaster recovery.

## For the wider homelab (out of scope but worth noting)

1. AI gateway naming. `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts.
2. MCP servers consolidation. Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding).
3. Backup the n8n SQLite file — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history.
4. Pocket-ID SSO in front of n8n would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win.
5. Vaultwarden as the secret store for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk.
6. AdGuard as DNS-based control plane. Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.
7. Disk usage on LXC 104 is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded.

8. Vector-store sprawl in the homelab — three competing indexes today. Nexa is about to be the fourth. Track for eventual consolidation:

   - Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
   - Paperless-AI ChromaDB (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A.
   - Nexa Qdrant (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus.

   Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa read from Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.

9. Retire `open-webui`. Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.

10. `services/siyuan/workspace/` is dead data. SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).

11. Real-time Obsidian sync via `notify_push`. Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
12. `assets/` is 186 MB of binaries in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
13. Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`. UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to delegate — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.
14. `media/Recipes/` has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.

## Proxmox host (NUC 14 Pro) tuning

Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:

1. Cap ZFS ARC. Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):

       echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf   # 8 GiB
       update-initramfs -u

   Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live.
2. Enable KSM (Kernel Same-page Merging). With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
3. Suppress the `pve-no-subscription` repository warning — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
4. Swap is 31 GiB on a 62 GiB box with ZFS root — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
5. Verify scheduled ZFS scrub is enabled. PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
6. SMART monitoring on the NVMe. `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
7. NTP source via AdGuard. AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
8. `fstrim.timer` enabled for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
9. Watchdog config is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.

## Docker LXC (104) — observations & wins

Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), 200 GiB boot disk at 47.7% used — disk pressure outranks RAM pressure.

1. Fix the Intel iGPU passthrough. The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically 5–10× faster than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:

       lxc.cgroup2.devices.allow: c 226:0 rwm
       lxc.cgroup2.devices.allow: c 226:128 rwm
       lxc.cgroup2.devices.allow: c 29:0 rwm
       lxc.mount.entry: /dev/dri/card0       dev/dri/card0       none bind,optional,create=file
       lxc.mount.entry: /dev/dri/renderD128  dev/dri/renderD128  none bind,optional,create=file
       lxc.idmap: u 0 100000 65536
       lxc.idmap: g 0 100000 65536
       lxc.idmap: g 44 44 1            # video group on host
       lxc.idmap: g 104 104 1          # render group on host

   Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.

2. Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`. Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services/<svc>/...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets:

   - `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir.
   - `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk).
   - `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store.
   - `/mnt/pve/unas/backup/nexa/snapshots/qdrant/<date>/` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup/<svc>/` pattern used by Home Assistant / Immich.
   - n8n executions retention: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
   - Monthly `docker image prune --all --filter "until=720h"` to clear dangling layers.

   Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/<vol>:/<container-path>`. Concrete example in 09-deployment §Step 3.

   Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets (12/#11) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is withdrawn (12/#27) — it was an over-fitting to the Nextcloud-specific NFS issue. 32. The LXC has its own 8 GiB swap. Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. 33. Stacks are managed in Arcane; Dockge is stale. The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to Arcane as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. Dozzle stays — it's the log viewer, not a manager, so it isn't redundant with Arcane. 34. Untriaged local mail on the LXC. Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).

## Housekeeping campaign (cross-cutting, do together)

These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto Phase 6 (Nexa as homelab steward): once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.

1. Consolidate Postgres instances. Today there are at least four independent Postgres containers running — visible from Arcane: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths:

   - Single shared cluster (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron.
   - Or migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.

   Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running.

2. UNAS-integration audit. `/mnt/pve/unas/services/<svc>/` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification):

   - `qdrant_scientific` — vector data possibly on local boot disk; critical to verify before Phase-3.1 piles on a `nexa_knowledge_text` collection.
   - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state.
   - `memos` — the user-facing notes app; loss = data loss.
   - `n8n` — workflows + executions DB.
   - `audiobookshelf` — listening progress + library metadata.
   - the `*-mcp` containers — likely stateless (cache only), low priority.

   Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.

3. 3-2-1 backup tiering: warm on-site (S3) + cold off-site. `s3.nuclide.systems` is already up but it's on-site — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a warm tier, not a disaster-recovery copy. The full design is two tiers:

   Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket): - Restic / borg repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination. - Qdrant + GraphDB snapshots flow UNAS → S3 daily. - Resolves Q12 once the bucket name + access key are set.

   Cold tier — off-site provider (decision pending — see Q20): - Jottacloud "Unlimited" (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`. - Hetzner Storage Box BX21 (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative. - Backblaze B2 (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during full restores. - rsync.net (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication. - Storj DCS ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem.

   Always encrypt before upload regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.

   Tiering & cadence:

       [Source]   UNAS RAID-6 + native snapshots (#38)
                    │ daily restic/borg
                    ▼
       [Warm]     s3.nuclide.systems (on-site, fast restore)
                    │ weekly rclone copy + crypt
                    ▼
       [Cold]     Off-site (Jottacloud / B2 / Hetzner)  ← satisfies the "1" in 3-2-1

   What goes off-site (priority order): 1. Immich photo originals — irreplaceable. 2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable. 3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo. 4. Vaultwarden DB — cryptographically sensitive but small; explicitly include.

   What does not need off-site: movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.

   Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it.

4. Configure UNAS Pool snapshots. The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. not configured. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. Highest-leverage data-protection change in the homelab right now — costs nothing, recovers everything.

5. Use UNAS native snapshots for fast Nexa rollback once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.