fkrebs/nexa
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
250a5cb2a6d3e452586a26197f588196a6e30177
nexa/docs/11-open-questions.md
T
Claude 250a5cb2a6 Pin homelab-wide UNAS storage convention; add Phase 6 + info wishlist 
- UNAS layout decoded from the user's tree: services/<svc>/ is the
  homelab-wide docker config store (every container follows the same
  pattern — immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid,
  shelfmark, stremio, traccar, vaultwarden, gluetun). Nexa MUST follow
  the same pattern at services/nexa/. backup/<svc>/ for snapshots.
  Pinned in CLAUDE.md and docs/12 #31.
- Stale services flagged for retirement: services/siyuan/ (migrated to
  Obsidian) and services/open-webui/ (unused, only LobeHub is alive).
  Drops the "two LLM UIs" item (#15) — it's now "retire open-webui".
- Hard-blocklist for any Nexa indexer pinned in CLAUDE.md:
  _sortMe/wallet/**, *.gpg/asc/key/pem/kdbx/credentials/secret, plus
  the Nextcloud appdata dir.
- Three new "future user-facing wins" surfaced from the tree:
  Paperless-AI as a Phase-2.x triage helper for _sortMe/Downloads/,
  media/Recipes/ (~300 entries) as the showcase RAG corpus, and
  Paperless-AI's existing ChromaDB as a potential read-from source
  rather than re-embedding scanned docs.
- Housekeeping campaign in docs/12 §35-37: consolidate postgres,
  audit UNAS-everywhere, S3 archive tier on s3.nuclide.systems.
- Phase 6 added to the roadmap: "Nexa as homelab steward". Polls
  Dockge/docker/Proxmox, diffs vs documented state, emits a daily
  drift report. Most of the housekeeping campaign becomes
  semi-automatic once 6.1-6.3 ship.
- New docs/13-information-wishlist.md packages the still-needed
  inventory as 5-6 paste-and-run tasks, each with explicit "📍 Where"
  markers (which shell or which UI) and what it unblocks. Highest
  leverage = Task 1 (read an existing compose stack to lock Q19).
- docs/index.md TOC extended to 13 rows.

Also fixed numbering drift in docs/12 (duplicate #28, missing #17/#18)
and added Q19 to docs/11 covering the SMB-share verification step.
2026-05-04 22:24:24 +00:00

72 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 11 — Open Questions (user-info-required)
Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide.
## Resolved
- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly.
- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container.
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below.
- [x] **Q5 — Display name → Karakeep.** The Zoraxy host alias `hoarder.nuclide.systems` is legacy — keep it for compatibility, but all docs, prompts and new workflow nodes use "Karakeep".
- [x] **Q13 — Octoprint container is *intentionally* temporary.** Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching `octoprint*` (or any container tagged `proxmox-he 3d-printing`).
- [x] **Q14 — Homepage Zoraxy widget → no such widget.** Config error in `homepage/services.yaml`; cosmetic, not Nexa-related.
- [x] **Q15 — Embeddings staging plan → A now, C prepared.**
- **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus.
- **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16).
- All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json).
- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme.
- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich.
- [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant).
- [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files/<user>/Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index):
- `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache.
- `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB).
- `.caldav-sync/` — calendar sync, not notes.
- `assets/` — 186 MB of binaries; routed through the Phase-3.2 visual queue (`nexa:pendingVisualIndex`), not the text path.
- `Templates/` — empty templates, low semantic value.
- `BMO/`, `Excalidraw/` — plugin folders.
Anything else under `Notizen/**/*.md` is fair game.
- [x] **Q6 + Q7 — Lists & calendars discovered by name, not ID. Self-healing.**
Confirmed names from screenshots:
- **Tasks**: `Persönlich` (14), `DLR` (13), `Einkaufsliste` (13), `Wunschliste` (9).
- **Calendars**: `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name).
- Mapping in classification: `DLR` = Work-Kontext, `Persönlich` = Personal-Kontext, `Einkaufsliste` = Shopping, `Wunschliste` = Wishes.
**Self-healing requirement (the user explicitly noted lists may change/grow):** Nexa must not cache IDs forever. The `#nexa:config` workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or "not found". The runtime-config record in Qdrant's `_config` namespace stores `{name → id, discovered_at}` and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring `#einkaufsliste` / `#dlr` etc. without code changes.
- [x] **Q8 — Mail via Nextcloud, single account `fkrebs@nucli.de`.** No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: `Posteingang` (default), `Archiv`, `Junk` (61 — auto-filtered, ignored by Nexa), `Papierkorb`, `Waiting`. The `Waiting` folder is a useful manual signal — items moved there by the user are skipped from digest (treat as "in flight").
- [x] **Q9 — Pocket-ID SSO is configured everywhere.** No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n → Memos webhook, n8n → Nextcloud, n8n → Qdrant, n8n → LiteLLM) keep using API keys / app passwords — SSO is for human UIs only. Don't add basic-auth or per-app login screens.
This **resolves Q10 too** (the proposed "Pocket-ID SSO in front of n8n / Memos" is already done; Nexa just inherits it).
- [x] **Q10 — folded into Q9.**
## Hardware / capacity
- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.**
- **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
- **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
- **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
- **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27).
- **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports?
## Verification needed before Phase-3.1 deploy
- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing Dockge stack (recommended: `services/karakeep/` or `services/immich/`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `<unas-share>` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Dockge, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa.
---
## Speed budget (Q3 follow-up)
Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap):
| Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` |
|------|--------|----------------|---------------------------------|------------------------------------|
| Real-time memo embed | 1 doc | <500 ms incl. n8n round-trip | ✅ ~50–100 ms | ✅ ~20 ms |
| Daily ingest | ~70 docs | <60 s | ✅ ~5–10 s | ✅ ~2 s |
| Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~2–4 min | ✅ <1 min |
| RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms |
Conclusion: CPU-only TEI is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to **10 req/min** which would block real-time embed; self-hosted TEI side-steps that completely.
Reference in New Issue View Git Blame Copy Permalink