From a8446f83717917025f70f910666c50dce464196e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 5 May 2026 05:01:48 +0000 Subject: [PATCH] Q19 resolved: storage convention is plain host bind-mount, not SMB volume MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Karakeep compose user shared confirms every container in LXC 104 uses host bind-mounts of /mnt/pve/unas/services// directly. No driver_opts, no CIFS, no per-volume credentials. The earlier SMB-as-docker-volume proposal in this repo was over-fitting to the Nextcloud-specific NFS issue and is withdrawn. The Nextcloud failure was Nextcloud-specific: its setup tooling chowns the data dir to www-data, which fails against root_squash exports. Normal containers don't trigger that path, so plain bind-mounts work fine. Changes: - docs/09 §Step 3: compose block dropped from ~30 lines (volume, driver_opts, CIFS device, credentials env) to a one-line bind-mount + env_file. Added a pre-deploy `mkdir -p` for the UNAS service + backup paths. Explanatory note retained for future agents. - docs/12 #27 marked withdrawn pointing to #31. #31 rewritten with the bind-mount pattern (was SMB-default). #36 (UNAS audit) now looks for `- /var/lib/docker/...` or anonymous volumes as the non-compliant signal. - CLAUDE.md storage block: bind-mount is the standard, SMB is documented escape-hatch only. - docs/11: Q19 moved into Resolved with the new framing. Q12 + Q20 grouped under a new "## Backups" section header (the now-empty "Verification needed" section removed). - docs/13 Task 1 struck through with done-marker pointing to docs/09. Net: Phase-3.1 deployment YAML is now actually deployable as written. --- CLAUDE.md | 9 ++--- docs/09-deployment.md | 48 +++++++++++---------------- docs/11-open-questions.md | 8 ++--- docs/12-optimization-opportunities.md | 16 ++++----- docs/13-information-wishlist.md | 16 ++------- 5 files changed, 38 insertions(+), 59 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4fc9215..036eb22 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,10 +15,11 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). Reachable two ways: - - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - - **SMB** — **default for Nexa per-container volumes** (Ubiquiti also recommends SMB as the modern path; the URL is `smb://192.168.1.31/` macOS / `\\192.168.1.31\` Windows). The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. - - **Storage-layer snapshots are NOT configured** on the UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, but not "the whole pool, last Tuesday". See [docs/12 #38](./docs/12-optimization-opportunities.md#38). + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at `/var/nfs/shared/storage`, mounted by Proxmox at `/mnt/pve/unas`, also exposed via SMB at `smb://192.168.1.31/` (Mac) / `\\192.168.1.31\` (Win). + + **Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of `/mnt/pve/unas/services//` from inside LXC 104.** No `driver_opts`, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an **escape hatch only** ([docs/12 #27](./docs/12-optimization-opportunities.md#27)) for services that hit Nextcloud-style NFS issues — Nexa doesn't, so we don't use it. + + **Storage-layer snapshots are NOT configured** on UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, not "the whole pool last Tuesday". Highest-leverage fix in the homelab right now ([docs/12 #38](./docs/12-optimization-opportunities.md#38)). **UNAS layout conventions** (homelab-wide, all docker containers follow them): - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index cbebddd..16c865d 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -95,7 +95,11 @@ Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB im The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle. ```yaml -# Nexa stack — paste into Arcane +# Nexa stack — paste into Arcane. +# Storage convention matches the rest of the homelab (verified against +# the running Karakeep stack, Q19): host bind-mount of +# /mnt/pve/unas/services//. No volume-driver, no CIFS, no +# credentials in the compose — the LXC's NFS mount is already there. services: nexa-embed: image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 @@ -105,37 +109,25 @@ services: ports: - "127.0.0.1:8080:80" volumes: - - tei-cache:/data + - /mnt/pve/unas/services/nexa/tei-cache:/data + env_file: + - .env -volumes: - # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues - # (the same kind that bit the Nextcloud rollout). Stores embedding model - # caches, Qdrant snapshots, future GraphDB exports. - # - # The exact mount block (driver, share name, credential injection) MUST - # be copied verbatim from an existing stack on the same docker host — - # services// on UNAS is the homelab-wide docker config store and - # every container already follows the same SMB pattern. See Q19 in - # docs/11. The block below is the *shape* but the share name "" - # is a placeholder until verified against e.g. services/karakeep/compose. - tei-cache: - driver: local - driver_opts: - type: cifs - device: //192.168.1.31//services/nexa/tei-cache - o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" - -networks: - default: - external: true - name: nexa +networks: {} ``` -Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant. +Pre-deploy step on the docker LXC (one-time): -> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership. -> -> Add `UNAS_USER` / `UNAS_PASS` to the stack's `.env` (Arcane provides an editor for this), or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#11](./12-optimization-opportunities.md#11)). +```bash +mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb} +mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb} +``` + +Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into `/mnt/pve/unas/services/nexa/tei-cache/`; subsequent restarts are instant. + +Secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, `QDRANT_API_KEY`, `NC_APP_PASSWORD`) go in the stack's `.env` next to the compose — same pattern Karakeep uses (`env_file: .env`). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term ([12/#11](./12-optimization-opportunities.md#11)) but isn't required for the first cut. + +> **Why bind-mount and not SMB?** Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's "had it with Nextcloud" experience. The Karakeep stack confirms the actual convention is the simpler one: **host bind-mount of the LXC's existing `/mnt/pve/unas` NFS mount**. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to `www-data`, which fails against `root_squash` exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in [12/#27](./12-optimization-opportunities.md#27) only as an escape hatch *if* a future service hits Nextcloud-style issues — Nexa doesn't, so we don't use it. Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index e3364fd..593d868 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -17,6 +17,7 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. - [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich. - [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant). +- [x] **Q19 — Storage convention is plain host bind-mount of `/mnt/pve/unas/services//`.** Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via `env_file: .env` next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn ([12/#27](./12-optimization-opportunities.md#27)) — it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to `www-data` and fails on `root_squash`-style exports; that doesn't apply to normal containers). Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). - [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files//Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index): - `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache. - `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB). @@ -49,12 +50,9 @@ Items that **block progress** and need a human decision before a workflow can be - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27). - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. -- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? - -## Verification needed before Phase-3.1 deploy - -- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. +## Backups +- [ ] **Q12 — S3 archive bucket** (warm tier). `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots, GraphDB exports, and as the rsync source for the cold-tier copy. - [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end. --- diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 56b6e41..816f444 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -71,17 +71,17 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services//` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets: - - `services/nexa/qdrant/` — Qdrant data dir (move before the visual collection lands). - - `services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). - - `services/nexa/graphdb/` — Phase-3.4 RDF store. - - `backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. +31. **Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services//`.** Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services//...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets: + - `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir. + - `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). + - `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store. + - `/mnt/pve/unas/backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. - Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19). + Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/:/`. Concrete example in [09-deployment §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). - Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths. + Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets ([12/#11](#11)) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is **withdrawn** ([12/#27](#27)) — it was an over-fitting to the Nextcloud-specific NFS issue. 32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. 33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane. 34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). @@ -96,7 +96,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. -36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Arcane stack and check the `volumes:` block. Suspected non-compliant (need verification): +36. **UNAS-integration audit.** `/mnt/pve/unas/services//` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification): - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state. - `memos` — the user-facing notes app; loss = data loss. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md index dc34d92..7999bba 100644 --- a/docs/13-information-wishlist.md +++ b/docs/13-information-wishlist.md @@ -6,21 +6,9 @@ Each task header tells you **exactly where to run it** (which shell or which UI) --- -## Task 1 — Stack inventory + reference compose **(highest leverage)** +## ~~Task 1 — Stack inventory + reference compose~~ ✅ DONE -📍 **Two halves — the canonical compose lives in the Arcane UI; the running-container view comes from a shell.** - -Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. - -**(a) Reference compose** — open **Arcane** in your browser → pick a clean small stack (Karakeep is a good choice) → screenshot or copy the **compose.yaml** *and* the **`.env`** (or whatever credential file is referenced). Just the `volumes:` block + the volume credential pattern is enough; everything else is bonus. - -**(b) Running-container view** — Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC): - -```bash -docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort -``` - -**Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. +User pasted the Karakeep compose. Q19 resolved → host bind-mount of `/mnt/pve/unas/services//` is the convention. Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). The `docker ps -a` half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) — but that's not blocking now. ---