diff --git a/CLAUDE.md b/CLAUDE.md index d2fb3d9..bf4f73a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,7 +17,18 @@ This file tells future automated runs what they need to know about this repo. - LXC 103 backrest — backup orchestration. - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). + - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. + + **UNAS layout conventions** (homelab-wide, all docker containers follow them): + - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian) and `services/open-webui/` (unused). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. + - `backup//` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.** + - `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern. + - **Before deploying any Nexa container, READ AN EXISTING STACK** (e.g. `services/karakeep/` or `services/immich/`'s compose under Dockge) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`. + +**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob): + - `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files. + - Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`. + - The Nextcloud appdata dir (`services/nextcloud/appdata_*`) — Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index 1aae00c..7ebd58a 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -46,3 +46,15 @@ Iterative build-out, value-first. Each phase is shippable on its own. - [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol. - [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy. + +## Phase 6 — Nexa as homelab steward + +**Focus:** Nexa actively maintains the homelab inventory instead of being told manually. +**Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo. + +- [ ] **6.1 Inventory sync** — Nexa polls Dockge / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …). +- [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35–37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*. +- [ ] **6.3 Steward commands** — `#nexa:retire ` (tar to `backup/`, stop, archive the stack), `#nexa:document ` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed). +- [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface). + +This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) — most of those items become semi-automatic once 6.1–6.3 ship. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index d5e9375..49d2eae 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -111,11 +111,18 @@ volumes: # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues # (the same kind that bit the Nextcloud rollout). Stores embedding model # caches, Qdrant snapshots, future GraphDB exports. + # + # The exact mount block (driver, share name, credential injection) MUST + # be copied verbatim from an existing stack on the same docker host — + # services// on UNAS is the homelab-wide docker config store and + # every container already follows the same SMB pattern. See Q19 in + # docs/11. The block below is the *shape* but the share name "" + # is a placeholder until verified against e.g. services/karakeep/compose. tei-cache: driver: local driver_opts: type: cifs - device: //192.168.1.31/nexa/tei-cache + device: //192.168.1.31//services/nexa/tei-cache o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" networks: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index bc20524..97b2083 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -51,6 +51,10 @@ Items that **block progress** and need a human decision before a workflow can be - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? +## Verification needed before Phase-3.1 deploy + +- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing Dockge stack (recommended: `services/karakeep/` or `services/immich/`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Dockge, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. + --- ## Speed budget (Q3 follow-up) diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index f301290..4a91abb 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -20,34 +20,45 @@ Observations from the running infrastructure. Each item is independent — accep 11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. 12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. 13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. -14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. -15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. -16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation: + - **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault. + - **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A. + - **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus. + + Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet. + +15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache. + +16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path). +17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. +18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos. +20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win. ## Proxmox host (NUC 14 Pro) tuning Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order: -17. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache): +21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache): ``` echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB update-initramfs -u ``` Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live. -18. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`. -19. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. -20. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. -21. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. -22. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. -23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. -24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. -25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. +22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`. +23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. +24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. +25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. +26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. +27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. +28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. +29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. ## Docker LXC (104) — observations & wins Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure. -26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: +30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: ``` lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm @@ -60,16 +71,45 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets: - - **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs. - - **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow. - - **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files. - - **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant//` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy. - - **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. +31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services//` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets: + - `services/nexa/qdrant/` — Qdrant data dir (move before the visual collection lands). + - `services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). + - `services/nexa/graphdb/` — Phase-3.4 RDF store. + - `backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. + - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. - Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths. -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. -29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. -30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. + Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19). + + Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths. +32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. +33. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. +34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). + +## Housekeeping campaign (cross-cutting, do together) + +These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down. + +35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Dockge: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths: + - **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron. + - **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk. + + Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. + +36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the `volumes:` block. Suspected non-compliant (need verification): + - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. + - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state. + - `memos` — the user-facing notes app; loss = data loss. + - `n8n` — workflows + executions DB. + - `audiobookshelf` — listening progress + library metadata. + - the `*-mcp` containers — likely stateless (cache only), low priority. + + Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services//` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward. + +37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup//` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier: + - **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination. + - **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore). + - **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars. + - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target. + + This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md new file mode 100644 index 0000000..8787bab --- /dev/null +++ b/docs/13-information-wishlist.md @@ -0,0 +1,106 @@ +# 13 — Information Wishlist + +What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent — run any subset, in any order. + +Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/-.txt` — whichever is easiest. + +--- + +## Task 1 — Stack inventory + reference compose **(highest leverage)** + +📍 **Where:** Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC). + +Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. + +```bash +docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort +ls /opt/stacks/ +cat /opt/stacks/karakeep/compose.yaml # or any clean small stack +``` + +**Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. + +--- + +## Task 2 — Storage map (Proxmox + docker) + +📍 **Where:** *Two shells* — first the **Proxmox host** (Datacenter → `nuc` → **Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**. + +Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB. + +```bash +# (a) on the Proxmox host (192.168.1.20): +cat /etc/pve/storage.cfg +cat /etc/pve/lxc/104.conf +``` + +```bash +# (b) on LXC 104 (192.168.1.40): +docker volume ls +ls -l /dev/dri/ # confirms whether iGPU passthrough actually works +``` + +**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17. + +--- + +## Task 3 — Postgres / DB workload inventory + +📍 **Where:** **LXC 104 console** (same shell as Task 1). + +Direct input to housekeeping #35 (consolidate postgres instances). + +```bash +for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do + echo "=== $c ===" + docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null' +done +``` + +**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision). + +--- + +## Task 4 — LiteLLM model list for the Nexa key + +📍 **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported. + +- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) → Settings → Model List, filtered to the Nexa virtual key. Screenshot the model rows. +- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set: + ```bash + curl -sH "Authorization: Bearer $SAIA_API_KEY" \ + https://ai.nuclide.systems/v1/models | jq '.data[].id' + ``` + +**Unblocks:** Q18 follow-up — which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI. + +--- + +## Task 5 — S3 archive credentials + +📍 **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy — same login as the rest of the homelab). + +Walk to: **Buckets** → either pick an existing Nexa-suitable bucket or create one called `nexa` → note the bucket name. Then **Access Keys** → create a key named `nexa-snapshots` with read/write on that bucket → drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden). + +**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3. + +--- + +## Task 6 — Reverse-proxy + DNS authority *(only if needed)* + +📍 **Where:** two browser UIs. + +- **Zoraxy** (`http://192.168.1.4:8000`) → HTTP Proxy → either click "Export" if available, or take a full screenshot of the table. +- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) → Filters → DNS Rewrites → screenshot. + +Skip unless we hit a routing surprise during Phase 1. + +--- + +## When something else is needed + +The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "📍 Where" framing. + +## Self-serve once Phase 6 ships + +Once [Phase 6 — Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 1–3 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember. diff --git a/docs/index.md b/docs/index.md index 37c2bba..d7d42c0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,6 +22,7 @@ This is the documentation entry point. Read top-to-bottom for first-time setup, | 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. | | 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. | | 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. | +| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. | ---