From 250a5cb2a6d3e452586a26197f588196a6e30177 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:24:24 +0000 Subject: [PATCH] Pin homelab-wide UNAS storage convention; add Phase 6 + info wishlist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - UNAS layout decoded from the user's tree: services// is the homelab-wide docker config store (every container follows the same pattern โ€” immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun). Nexa MUST follow the same pattern at services/nexa/. backup// for snapshots. Pinned in CLAUDE.md and docs/12 #31. - Stale services flagged for retirement: services/siyuan/ (migrated to Obsidian) and services/open-webui/ (unused, only LobeHub is alive). Drops the "two LLM UIs" item (#15) โ€” it's now "retire open-webui". - Hard-blocklist for any Nexa indexer pinned in CLAUDE.md: _sortMe/wallet/**, *.gpg/asc/key/pem/kdbx/credentials/secret, plus the Nextcloud appdata dir. - Three new "future user-facing wins" surfaced from the tree: Paperless-AI as a Phase-2.x triage helper for _sortMe/Downloads/, media/Recipes/ (~300 entries) as the showcase RAG corpus, and Paperless-AI's existing ChromaDB as a potential read-from source rather than re-embedding scanned docs. - Housekeeping campaign in docs/12 ยง35-37: consolidate postgres, audit UNAS-everywhere, S3 archive tier on s3.nuclide.systems. - Phase 6 added to the roadmap: "Nexa as homelab steward". Polls Dockge/docker/Proxmox, diffs vs documented state, emits a daily drift report. Most of the housekeeping campaign becomes semi-automatic once 6.1-6.3 ship. - New docs/13-information-wishlist.md packages the still-needed inventory as 5-6 paste-and-run tasks, each with explicit "๐Ÿ“ Where" markers (which shell or which UI) and what it unblocks. Highest leverage = Task 1 (read an existing compose stack to lock Q19). - docs/index.md TOC extended to 13 rows. Also fixed numbering drift in docs/12 (duplicate #28, missing #17/#18) and added Q19 to docs/11 covering the SMB-share verification step. --- CLAUDE.md | 13 +++- docs/02-roadmap.md | 12 +++ docs/09-deployment.md | 9 ++- docs/11-open-questions.md | 4 + docs/12-optimization-opportunities.md | 88 +++++++++++++++------ docs/13-information-wishlist.md | 106 ++++++++++++++++++++++++++ docs/index.md | 1 + 7 files changed, 207 insertions(+), 26 deletions(-) create mode 100644 docs/13-information-wishlist.md diff --git a/CLAUDE.md b/CLAUDE.md index d2fb3d9..bf4f73a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,7 +17,18 @@ This file tells future automated runs what they need to know about this repo. - LXC 103 backrest โ€” backup orchestration. - LXC 104 docker โ€” main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` โ€” stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) โ€” fine for host-side admin tasks and read-mostly mounts. - - **SMB** โ€” **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` โ€” sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* โ€” see [docs/12 #26](./docs/12-optimization-opportunities.md#26). + - **SMB** โ€” **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` โ€” sidesteps uid-mapping entirely. + + **UNAS layout conventions** (homelab-wide, all docker containers follow them): + - **`services//` is the general docker config store** โ€” every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian) and `services/open-webui/` (unused). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. + - `backup//` โ€” per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots โ†’ `backup/nexa/`.** + - `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` โ€” user data, not Nexa's concern. + - **Before deploying any Nexa container, READ AN EXISTING STACK** (e.g. `services/karakeep/` or `services/immich/`'s compose under Dockge) to confirm the exact mount syntax in use โ€” driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD โ€” see Q19** in `docs/11`. + +**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob): + - `_sortMe/wallet/**` โ€” contains PGP keys + bitcoin wallet files. + - Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`. + - The Nextcloud appdata dir (`services/nextcloud/appdata_*`) โ€” Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* โ€” see [docs/12 #26](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud โ€” Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint โ€” currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy โ€” reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index 1aae00c..7ebd58a 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -46,3 +46,15 @@ Iterative build-out, value-first. Each phase is shippable on its own. - [ ] **5.1 HA Voice** โ€” Wake-word integration, Wyoming protocol. - [ ] **5.2 System monitoring** โ€” Proxmox / Backrest alerts into the Memos system feed via ntfy. + +## Phase 6 โ€” Nexa as homelab steward + +**Focus:** Nexa actively maintains the homelab inventory instead of being told manually. +**Milestone:** Nexa runs a daily *drift report* โ€” what changed, what's stale, what's missing โ€” and turns it into actionable comments under a pinned `[STEWARD]` memo. + +- [ ] **6.1 Inventory sync** โ€” Nexa polls Dockge / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, โ€ฆ). +- [ ] **6.2 Drift detection** โ€” diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35โ€“37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago โ€” retire?"*, *"Container `foo` has no `services/foo/` bind โ€” schedule UNAS migration?"*. +- [ ] **6.3 Steward commands** โ€” `#nexa:retire ` (tar to `backup/`, stop, archive the stack), `#nexa:document ` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed). +- [ ] **6.4 Self-update of docs** โ€” when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface). + +This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) โ€” most of those items become semi-automatic once 6.1โ€“6.3 ship. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index d5e9375..49d2eae 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -111,11 +111,18 @@ volumes: # SMB volume on UNAS โ€” sidesteps NFS+unprivileged-LXC permission issues # (the same kind that bit the Nextcloud rollout). Stores embedding model # caches, Qdrant snapshots, future GraphDB exports. + # + # The exact mount block (driver, share name, credential injection) MUST + # be copied verbatim from an existing stack on the same docker host โ€” + # services// on UNAS is the homelab-wide docker config store and + # every container already follows the same SMB pattern. See Q19 in + # docs/11. The block below is the *shape* but the share name "" + # is a placeholder until verified against e.g. services/karakeep/compose. tei-cache: driver: local driver_opts: type: cifs - device: //192.168.1.31/nexa/tei-cache + device: //192.168.1.31//services/nexa/tei-cache o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" networks: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index bc20524..97b2083 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -51,6 +51,10 @@ Items that **block progress** and need a human decision before a workflow can be - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 โ€” S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? +## Verification needed before Phase-3.1 deploy + +- [ ] **Q19 โ€” Confirm the SMB share name + mount syntax** by reading an existing Dockge stack (recommended: `services/karakeep/` or `services/immich/`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Dockge, copy its `volumes:` block verbatim, swap `karakeep` โ†’ `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** โ€” same driver, same credentials secret, same uid/gid โ€” and avoids reinventing storage just for Nexa. + --- ## Speed budget (Q3 follow-up) diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index f301290..4a91abb 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -20,34 +20,45 @@ Observations from the running infrastructure. Each item is independent โ€” accep 11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, โ€ฆ) โ€” read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. 12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN โ€” already the case if AdGuard rewrite rules are set, worth verifying. 13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. -14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores โ€” three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. -15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. -16. **`assets/` is 186 MB of binaries** in the Obsidian vault โ€” worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +14. **Vector-store sprawl in the homelab โ€” three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation: + - **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) โ€” embed the Obsidian vault into two separate vector stores inside the vault. + - **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) โ€” embeds scanned documents from Paperless-ngx for Q&A. + - **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) โ€” embeds the cross-source corpus. + + Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet. + +15. **Retire `open-webui`.** Confirmed stale by the user โ€” only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache. + +16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path). +17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. +18. **`assets/` is 186 MB of binaries** in the Obsidian vault โ€” worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* โ€” wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos. +20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win. ## Proxmox host (NUC 14 Pro) tuning Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order: -17. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8โ€“12 GiB frees ~12โ€“16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used โ€” there's nothing hot to cache): +21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8โ€“12 GiB frees ~12โ€“16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used โ€” there's nothing hot to cache): ``` echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB update-initramfs -u ``` Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live. -18. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1โ€“3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available โ€” `systemctl enable --now ksmtuned`. -19. **Suppress the `pve-no-subscription` repository warning** โ€” either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. -20. **Swap is 31 GiB on a 62 GiB box with ZFS root** โ€” almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. -21. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` โ€” `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. -22. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. -23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. -24. **`fstrim.timer` enabled** for the SSD pool โ€” verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. -25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. +22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1โ€“3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available โ€” `systemctl enable --now ksmtuned`. +23. **Suppress the `pve-no-subscription` repository warning** โ€” either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. +24. **Swap is 31 GiB on a 62 GiB box with ZFS root** โ€” almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. +25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` โ€” `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. +26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. +27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. +28. **`fstrim.timer` enabled** for the SSD pool โ€” verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. +29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. ## Docker LXC (104) โ€” observations & wins Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** โ€” disk pressure outranks RAM pressure. -26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM โ€” typically **5โ€“10ร— faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: +30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM โ€” typically **5โ€“10ร— faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: ``` lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm @@ -60,16 +71,45 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` โ€” that bypasses LXC uid-mapping entirely. Targets: - - **Qdrant data dir** (`qdrant_scientific`) โ†’ SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs. - - **GraphDB repo** (Phase 3.4) โ†’ `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow. - - **Embedding model caches** (TEI/infinity) โ†’ `//192.168.1.31/nexa/tei-cache` โ€” keeps the boot disk free of multi-GB model files. - - **Daily Qdrant snapshots** โ†’ `//192.168.1.31/nexa/snapshots/qdrant//` before rsync to S3 (Phase 3.3 โ€” Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy. - - **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. +31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services//` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets: + - `services/nexa/qdrant/` โ€” Qdrant data dir (move before the visual collection lands). + - `services/nexa/tei-cache/` โ€” embedding model cache (keeps multi-GB models off the boot disk). + - `services/nexa/graphdb/` โ€” Phase-3.4 RDF store. + - `backup/nexa/snapshots/qdrant//` โ€” daily Qdrant snapshots before rsync to S3 (Phase 3.3 โ€” Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. + - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. - Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) โ€” just not for per-container write paths. -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1โ€“2 GiB (`pct set 104 -swap 2048`) โ€” frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. -29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge โ†’ save โ†’ start", not `docker compose up -d` over SSH. -30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login โ€” the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1โ€“2 GiB (`pct set 104 -swap 2048`) โ€” frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. + Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim โ€” this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19). + + Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) โ€” not for per-container write paths. +32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1โ€“2 GiB (`pct set 104 -swap 2048`) โ€” frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. +33. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge โ†’ save โ†’ start", not `docker compose up -d` over SSH. +34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login โ€” the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). + +## Housekeeping campaign (cross-cutting, do together) + +These three are inter-related โ€” picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35โ€“37 become semi-automatic โ€” Nexa proposes the migrations rather than us hunting them down. + +35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running โ€” visible from Dockge: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100โ€“300 MB RAM and has its own backup story. Two paths: + - **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` โ†’ just point them at the shared instance. Saves ~600 MBโ€“1 GB RAM and consolidates backups to one pg_dump cron. + - **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk. + + Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. + +36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the `volumes:` block. Suspected non-compliant (need verification): + - `qdrant_scientific` โ€” vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. + - `litellm_db`, `lobe-postgres`, `lobe-redis` โ€” DB containers, persistent state. + - `memos` โ€” the user-facing notes app; loss = data loss. + - `n8n` โ€” workflows + executions DB. + - `audiobookshelf` โ€” listening progress + library metadata. + - the `*-mcp` containers โ€” likely stateless (cache only), low priority. + + Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop โ†’ rsync data to `services//` on UNAS โ†’ re-create stack with the new bind โ†’ verify โ†’ keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward. + +37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup//` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier: + - **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg โ€” just add an S3 destination. + - **Qdrant + GraphDB snapshots** flow UNAS โ†’ S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore). + - **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars. + - Resolves [Q12](./11-open-questions.md#q12) โ€” once the bucket name + access key are set, every other archive just adds a target. + + This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md new file mode 100644 index 0000000..8787bab --- /dev/null +++ b/docs/13-information-wishlist.md @@ -0,0 +1,106 @@ +# 13 โ€” Information Wishlist + +What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent โ€” run any subset, in any order. + +Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/-.txt` โ€” whichever is easiest. + +--- + +## Task 1 โ€” Stack inventory + reference compose **(highest leverage)** + +๐Ÿ“ **Where:** Proxmox UI โ†’ Datacenter โ†’ `nuc` โ†’ LXC `104 (docker)` โ†’ **Console** (or `ssh root@192.168.1.40` from your PC). + +Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. + +```bash +docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort +ls /opt/stacks/ +cat /opt/stacks/karakeep/compose.yaml # or any clean small stack +``` + +**Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. + +--- + +## Task 2 โ€” Storage map (Proxmox + docker) + +๐Ÿ“ **Where:** *Two shells* โ€” first the **Proxmox host** (Datacenter โ†’ `nuc` โ†’ **Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**. + +Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB. + +```bash +# (a) on the Proxmox host (192.168.1.20): +cat /etc/pve/storage.cfg +cat /etc/pve/lxc/104.conf +``` + +```bash +# (b) on LXC 104 (192.168.1.40): +docker volume ls +ls -l /dev/dri/ # confirms whether iGPU passthrough actually works +``` + +**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17. + +--- + +## Task 3 โ€” Postgres / DB workload inventory + +๐Ÿ“ **Where:** **LXC 104 console** (same shell as Task 1). + +Direct input to housekeeping #35 (consolidate postgres instances). + +```bash +for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do + echo "=== $c ===" + docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null' +done +``` + +**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision). + +--- + +## Task 4 โ€” LiteLLM model list for the Nexa key + +๐Ÿ“ **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported. + +- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) โ†’ Settings โ†’ Model List, filtered to the Nexa virtual key. Screenshot the model rows. +- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set: + ```bash + curl -sH "Authorization: Bearer $SAIA_API_KEY" \ + https://ai.nuclide.systems/v1/models | jq '.data[].id' + ``` + +**Unblocks:** Q18 follow-up โ€” which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI. + +--- + +## Task 5 โ€” S3 archive credentials + +๐Ÿ“ **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy โ€” same login as the rest of the homelab). + +Walk to: **Buckets** โ†’ either pick an existing Nexa-suitable bucket or create one called `nexa` โ†’ note the bucket name. Then **Access Keys** โ†’ create a key named `nexa-snapshots` with read/write on that bucket โ†’ drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden). + +**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3. + +--- + +## Task 6 โ€” Reverse-proxy + DNS authority *(only if needed)* + +๐Ÿ“ **Where:** two browser UIs. + +- **Zoraxy** (`http://192.168.1.4:8000`) โ†’ HTTP Proxy โ†’ either click "Export" if available, or take a full screenshot of the table. +- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) โ†’ Filters โ†’ DNS Rewrites โ†’ screenshot. + +Skip unless we hit a routing surprise during Phase 1. + +--- + +## When something else is needed + +The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "๐Ÿ“ Where" framing. + +## Self-serve once Phase 6 ships + +Once [Phase 6 โ€” Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 1โ€“3 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember. diff --git a/docs/index.md b/docs/index.md index 37c2bba..d7d42c0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,6 +22,7 @@ This is the documentation entry point. Read top-to-bottom for first-time setup, | 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. | | 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. | | 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. | +| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. | ---