diff --git a/CLAUDE.md b/CLAUDE.md index 3054bab..335bb5d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,9 @@ This file tells future automated runs what they need to know about this repo. - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. - VM 100 haos — Home Assistant. - **Already-running services on docker host** (don't redeploy): - - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). + - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). +- **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state. +- **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`. - **Decided for Nexa** (don't re-litigate without user input): - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive. diff --git a/docs/04-integration-matrix.md b/docs/04-integration-matrix.md index 2fab535..4ae6799 100644 --- a/docs/04-integration-matrix.md +++ b/docs/04-integration-matrix.md @@ -59,8 +59,10 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari * Wichtige private Termine werden in den privaten Kalender extrahiert. * Newsletter landen als Zusammenfassung in Memos. -### Wissens-Management (Obsidian & Qdrant) -* Obsidian-Notizen (Arbeit & Privat) werden in Qdrant indiziert. +### Wissens-Management (Obsidian via Nextcloud & Qdrant) +* Der Obsidian-Vault liegt im Nextcloud-Ordner `Notizen/` (Multi-Device-Sync via Nextcloud-Client). Nexa liest **read-only über WebDAV** (`/remote.php/dav/files//Notizen/`) — kein Mount, kein zweiter Sync-Mechanismus. +* Markdown-Dateien (`Notizen/**/*.md`) werden in Qdrant (`nexa_knowledge_text`) indiziert; Bilder unter `assets/` landen im Phase-3.2-Visual-Queue (siehe [docs/11/Q4](./11-open-questions.md)). +* Plugin-Internals (`.copilot/`, `.smart-env/`, `.caldav-sync/`, `Templates/`, `BMO/`, `Excalidraw/`) sind explizit von der Indizierung ausgeschlossen. * Nexa nutzt dieses Wissen, um Arbeits-Tasks besser zu beschreiben, auch wenn sie keinen Zugriff auf die Arbeits-E-Mails hat. --- diff --git a/docs/09-deployment.md b/docs/09-deployment.md index e196f99..a884f14 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -13,7 +13,7 @@ Surveyed from Homepage / Dozzle / Proxmox / Zoraxy: | LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) | | Nextcloud | LXC 105 | `https://nc.nuclide.systems` | | ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` | -| Karakeep (Hoarder) | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` | +| Karakeep | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` (legacy host alias kept for compatibility) | | Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` | | Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` | | Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` | @@ -145,8 +145,8 @@ done Inside n8n, attach credentials to the imported nodes: - **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY` -- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` -- **Nextcloud** → app password +- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` (chat + `nexa-embed`) +- **Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under `Notizen/`)** → one app password (`$NC_APP_PASSWORD`), reused across all three node types - **Qdrant** → header `api-key: $QDRANT_API_KEY` Activate each workflow individually after smoke-test. diff --git a/docs/10-operations.md b/docs/10-operations.md index 4c69609..c5fe9da 100644 --- a/docs/10-operations.md +++ b/docs/10-operations.md @@ -18,7 +18,7 @@ Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items | Signal | Source | Sink | |--------|--------|------| -| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks | Memos system feed via ntfy | +| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. **Skip names matching `octoprint*`** — that container is intentionally powered down most of the time. | Memos system feed via ntfy | | n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic | | Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest | | LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest | diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 7d60c48..bd4ca39 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -7,18 +7,24 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly. - [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container. - [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below. +- [x] **Q5 — Display name → Karakeep.** The Zoraxy host alias `hoarder.nuclide.systems` is legacy — keep it for compatibility, but all docs, prompts and new workflow nodes use "Karakeep". +- [x] **Q13 — Octoprint container is *intentionally* temporary.** Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching `octoprint*` (or any container tagged `proxmox-he 3d-printing`). +- [x] **Q14 — Homepage Zoraxy widget → no such widget.** Config error in `homepage/services.yaml`; cosmetic, not Nexa-related. - [x] **Q15 — Embeddings staging plan → A now, C prepared.** - **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus. - **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16). - All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json). - -## Architectural decisions - -- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. -- [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. -- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. Sub-question on whether SAIA already proxies an embedding model is still worth a 1-line check in the LiteLLM admin UI before deploying TEI — moved to Q18. -- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich; if a Nexa workflow ever needs photo context it will go through a Memos-mediated handoff rather than a direct API. -- [ ] **Q18 — Does SAIA already proxy an embedding model?** 1-line check in the LiteLLM admin UI: list models on the Nexa virtual key. If the SAIA backend offers e.g. `mistral-embed`, we can skip TEI in Phase 3.1 entirely. If not, deploy TEI as planned. +- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. +- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich. +- [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant). +- [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files//Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index): + - `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache. + - `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB). + - `.caldav-sync/` — calendar sync, not notes. + - `assets/` — 186 MB of binaries; routed through the Phase-3.2 visual queue (`nexa:pendingVisualIndex`), not the text path. + - `Templates/` — empty templates, low semantic value. + - `BMO/`, `Excalidraw/` — plugin folders. + Anything else under `Notizen/**/*.md` is fair game. ## Identifiers needed (auto-discoverable, but list now if known) @@ -30,14 +36,9 @@ Items that **block progress** and need a human decision before a workflow can be ## Hardware / capacity -- [ ] **Q11 — RAM headroom on docker host.** 29.5 GiB free / ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing + TEI/`bge-m3` (~1.1 GB) + Ontotext GraphDB (~4 GB heap) ⇒ ~14 GB used worst case, still ample. Confirm acceptable. +- [ ] **Q11 — RAM headroom on docker host.** Sized against 31 GiB allocated to the LXC; the NUC 14 Pro itself has 64 GB so the LXC's memory cap can be lifted if needed. Phase-3 stack (Qdrant + TEI/bge-m3 ~1.1 GB + Ontotext GraphDB ~4 GB heap, later +infinity/jina-clip-v2 ~1 GB) tops out around ~14 GB. Comfortable. Confirm acceptable. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? -## Process - -- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but Phase-5 monitoring would alert on it. Suppress or is it intentional? -- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage. Cosmetic, unrelated to Nexa. - --- ## Speed budget (Q3 follow-up) @@ -51,4 +52,4 @@ Workload on the docker host (16 CPU, ~30 GB free RAM): | Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~2–4 min | ✅ <1 min | | RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms | -Conclusion: CPU-only Ollama is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. +Conclusion: CPU-only TEI is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to **10 req/min** which would block real-time embed; self-hosted TEI side-steps that completely. diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 77c5f28..d70d7ff 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -13,13 +13,13 @@ Observations from the running infrastructure. Each item is independent — accep ## For the wider homelab (out of scope but worth noting) -7. **Octoprint `EXITED`** — either remove the Homepage entry or fix the container; right now it permanently shows red. -8. **Zoraxy widget missing** on Homepage. Homepage v1.x has no built-in Zoraxy widget, but you can use the generic `customapi` widget against `/api/stats/summary` to surface request counts. -9. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. -10. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). -11. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. -12. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. -13. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. -14. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. -15. **Immich + Qdrant.** Immich is already running and embeds photos with CLIP. If you ever want photo-aware Nexa answers, you can mount Immich's existing search via its API — no second embedding pass. -16. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. +7. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. +8. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). +9. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. +10. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. +11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. +12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. +13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. +14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. +15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. +16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.