From 17c7f5033fd8aef15daa5c51b102f19a5a33ea0f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:33:59 +0000 Subject: [PATCH] Stage embeddings: ship Path A (text) now, prepare Path C (text+visual) additively MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging. --- CLAUDE.md | 6 +- docs/02-roadmap.md | 12 ++-- docs/03-architecture.md | 6 +- docs/08-graphrag-architecture.md | 20 +++++-- docs/09-deployment.md | 68 ++++++++++++++++++++-- docs/11-open-questions.md | 16 ++--- nexa-core/config/qdrant_schema.json | 22 +++++-- nexa-core/config/qdrant_schema_visual.json | 26 +++++++++ 8 files changed, 142 insertions(+), 34 deletions(-) create mode 100644 nexa-core/config/qdrant_schema_visual.json diff --git a/CLAUDE.md b/CLAUDE.md index 95bd968..3054bab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,9 +21,9 @@ This file tells future automated runs what they need to know about this repo. - **Already-running services on docker host** (don't redeploy): - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). - **Decided for Nexa** (don't re-litigate without user input): - - Vector store: **reuse `qdrant_scientific`** with `nexa_*` collection prefix. - - Embeddings: **TEI** (HF text-embeddings-inference), self-hosted, `bge-m3` candidate (Q15 still pending the model name). - - Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. + - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). + - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive. + - Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. RDF schema in docs/08 already includes `nexa:modality` / `nexa:mediaUri` / `nexa:vectorCollection` / `nexa:pendingVisualIndex`. - Chat model: SAIA via LiteLLM virtual key. ## When working on Nexa diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index fc59f97..1aae00c 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -22,13 +22,13 @@ Iterative build-out, value-first. Each phase is shippable on its own. ## Phase 3 — Memory (Qdrant & RAG) -**Focus:** Qdrant + (optional) graph DB for retrieval-augmented answers. -**Milestone:** RAG works — Nexa answers from archived notes. +**Focus:** Qdrant + graph layer for retrieval-augmented answers. +**Milestone:** RAG works — Nexa answers from archived notes; images captured today are queued for visual indexing later. -- [ ] **3.1 Embedding pipeline** — automatic vectorization of Memos & Obsidian. -- [ ] **3.2 Context retrieval** — `#nexa:ask` reads Qdrant before answering. -- [ ] **3.3 S3 long-term archive** — Qdrant snapshots to the S3 server. -- [ ] **3.4 Graph layer (deferred decision — see [11-open-questions](./11-open-questions.md))** — Neo4j *or* Ontotext GraphDB for structural queries. +- [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet. +- [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32). +- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. +- [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md). ## Phase 4 — Motor (organization & action) diff --git a/docs/03-architecture.md b/docs/03-architecture.md index d796f8f..6db8376 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -42,9 +42,9 @@ A one-page mental model. For details follow the cross-links. | **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | -| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse existing `qdrant_scientific` with `nexa_*` collection prefix | -| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-graph-db-phase-34) | -| **TEI** (HF text-embeddings-inference) | Self-hosted embeddings (`bge-m3` / TBD — see [11/Q15](./11-open-questions.md)) | docker host LXC 104, CPU only — single-binary embedding server | +| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) | +| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-ontotext-graphdb-phase-34) | +| **TEI** (HF text-embeddings-inference) | Self-hosted text embeddings, `BAAI/bge-m3`, Phase 3.1 | docker host LXC 104, CPU only — swapped for `infinity` in Phase 3.2 to add `jina-clip-v2` | | **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` | | **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` | | **Backrest** | Backup orchestration | LXC 103 | diff --git a/docs/08-graphrag-architecture.md b/docs/08-graphrag-architecture.md index 5cd1ddd..110b24c 100644 --- a/docs/08-graphrag-architecture.md +++ b/docs/08-graphrag-architecture.md @@ -45,14 +45,22 @@ nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range n nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime . # Datatype properties -nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done" -nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal" -nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5 -nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID — bridges the two pillars -nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup +nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done" +nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal" +nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5 +nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup + +# Cross-pillar / multimodality +nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image" +nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://… +nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual" +nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID +nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them ``` -The `nexa:vectorId` property is the **bridge** between graph and vector store. Every node that has semantic content carries the Qdrant point ID, so a SPARQL hit can trigger a vector lookup and vice versa. +`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way. + +`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases. --- diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 298e327..e196f99 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -49,20 +49,44 @@ $EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KE The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead. -## Step 2 — Qdrant collection +## Step 2 — Qdrant collection (`nexa_knowledge_text`) + +Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring. ```bash # adjust QDRANT_HOST in .env first source nexa-core/.env -curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \ +# create the text collection from the schema file +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \ -H "Content-Type: application/json" \ -H "api-key: $QDRANT_API_KEY" \ -d @nexa-core/config/qdrant_schema.json ``` +The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)): + +| Field | Why it's there now | +|-------|--------------------| +| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. | +| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. | +| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. | +| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. | +| `content_hash` | de-dup. | +| `context` | `work` / `personal`. | + > Targets the existing `qdrant_scientific` instance — just an extra collection, no new container. -> The schema file's `vectors.size` must match the embedding model picked in [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`. +> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`. + +### Image attachments today (queue them) + +Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later: + +- Memo with an image → text body still goes into `nexa_knowledge_text`. +- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`). +- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection. + +This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself. ## Step 3 — Self-hosted embeddings (TEI) @@ -228,8 +252,44 @@ For schema and example queries: [08-graphrag-architecture](./08-graphrag-archite --- +## Phase add-on: visual collection (Phase 3.2) + +Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`. + +```bash +# (1) replace TEI with infinity (or run alongside) for CLIP-family support +docker rm -f nexa-embed +docker run -d --name nexa-embed \ + --restart unless-stopped \ + -p 127.0.0.1:8080:80 \ + -v infinity-data:/app/.cache \ + michaelf34/infinity:latest \ + v2 \ + --model-id BAAI/bge-m3 \ + --model-id jinaai/jina-clip-v2 \ + --port 80 + +# (2) create the visual collection +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \ + -H "Content-Type: application/json" \ + -H "api-key: $QDRANT_API_KEY" \ + -d @nexa-core/config/qdrant_schema_visual.json + +# (3) register the second model in LiteLLM as `nexa-embed-visual` +# (same OpenAI-compatible route, different model id) + +# (4) backfill queued images: +# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri } +# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection, +# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex. +``` + +n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt. + +--- + ## Step-back / rollback - **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops. -- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`. +- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`. - **Re-discover** — `#nexa:reset-config` then `#nexa:config`. diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index e4ea61f..03eca23 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -6,18 +6,20 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly. - [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container. -- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below; concrete model still open as **Q15**. +- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below. +- [x] **Q15 — Embeddings staging plan → A now, C prepared.** + - **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus. + - **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16). + - All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json). ## Architectural decisions - [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. - [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. -- [ ] **Q15 — Embedding model (served via TEI).** Two viable options on the docker host (CPU only — see speed budget): - - **`BAAI/bge-m3`** (568 M params, ~1 GB RAM, multilingual incl. German, 1024-dim, ~10–20 docs/s on CPU). **Recommended** because the corpus is DE/EN-mixed. Requires `qdrant_schema.json` `vectors.size = 1024`. - - **`nomic-ai/nomic-embed-text-v1.5`** (137 M, ~250 MB RAM, EN-leaning, 768-dim, ~50 docs/s CPU). Lighter / faster but weaker on German. - - **(maybe) does SAIA already proxy any embedding model?** If the SAIA backend offers e.g. `mistral-embed` for free, we can skip TEI entirely. Worth a 1-line check in the LiteLLM admin UI before deploying TEI. - - Decide before Phase 3.1. +- [ ] **Q16 — Image-attachment queue ergonomics (Phase 3.2).** The agreed plan is: in 3.1 we record images as `nexa:Note` with `nexa:pendingVisualIndex true` but don't embed them. Open sub-questions for when 3.2 lands: + - Where to store image bytes between capture and indexing? Three options: (a) leave them in their source (Memos attachments dir / Nextcloud / Obsidian), reference by `media_uri`; (b) copy to a staging area on the docker host; (c) push to `s3.nuclide.systems` immediately. **Recommendation: (a)** — zero copy, smallest blast radius. + - Does SAIA already proxy any embedding model? If the SAIA backend offers e.g. `mistral-embed` we could simplify 3.1 by skipping TEI. Worth a 1-line check in the LiteLLM admin UI. +- [ ] **Q17 — Reuse Immich's CLIP for photo-library queries?** Immich already runs CLIP server-side on the photo library. For images that live *in Immich*, querying its smart-search API is cheaper than re-embedding. Is the Immich API key OK to add to the n8n workflow, or do we treat Immich as out-of-band? ## Identifiers needed (auto-discoverable, but list now if known) diff --git a/nexa-core/config/qdrant_schema.json b/nexa-core/config/qdrant_schema.json index 94a04c8..f692473 100644 --- a/nexa-core/config/qdrant_schema.json +++ b/nexa-core/config/qdrant_schema.json @@ -1,12 +1,24 @@ { - "collection_name": "nexa_knowledge", + "collection_name": "nexa_knowledge_text", "vector_config": { - "size": 1536, + "size": 1024, "distance": "Cosine" }, "payload_schema": { - "source": "keyword", - "tags": "keyword", - "created_at": "datetime" + "source": "keyword", + "source_type": "keyword", + "modality": "keyword", + "mime_type": "keyword", + "media_uri": "keyword", + "context": "keyword", + "tags": "keyword", + "graph_iri": "keyword", + "content_hash":"keyword", + "created_at": "datetime" + }, + "_notes": { + "vector_size_rationale": "1024 = bge-m3 output. If Q15 resolves to nomic-embed-text-v1.5 instead, rebuild this collection with size: 768.", + "modality_field": "Always 'text' in this collection. Forward-compatibility for Phase 3.2 (see qdrant_schema_visual.json).", + "graph_iri_field": "IRI of the corresponding nexa:Note in GraphDB. The same value is stored on the GraphDB node as nexa:vectorId pointing back here. This is the cross-pillar bridge." } } diff --git a/nexa-core/config/qdrant_schema_visual.json b/nexa-core/config/qdrant_schema_visual.json new file mode 100644 index 0000000..325190f --- /dev/null +++ b/nexa-core/config/qdrant_schema_visual.json @@ -0,0 +1,26 @@ +{ + "collection_name": "nexa_knowledge_visual", + "vector_config": { + "size": 768, + "distance": "Cosine" + }, + "payload_schema": { + "source": "keyword", + "source_type": "keyword", + "modality": "keyword", + "mime_type": "keyword", + "media_uri": "keyword", + "context": "keyword", + "tags": "keyword", + "graph_iri": "keyword", + "content_hash":"keyword", + "created_at": "datetime" + }, + "_notes": { + "status": "PLACEHOLDER — not deployed in Phase 3.1. Created in Phase 3.2.", + "vector_size_rationale": "768 = jina-clip-v2 joint text/image space. The same model embeds both text queries and images, so semantic queries against this collection are simple kNN.", + "served_by": "Will be served by `infinity` (https://github.com/michaelfeil/infinity) — drop-in replacement for TEI that supports CLIP-family models with the same OpenAI-compatible API.", + "modality_field": "'image' for indexed images, 'text' if a text query has been embedded into the visual space for retrieval (rare — usually we embed the text query at search time and don't store it).", + "media_uri_field": "Required for image rows. Examples: 'memos://memos/123/attachment/4', 'nextcloud://files/photos/whiteboard-2025-09.jpg', 'obsidian://attachments/diagram.png'." + } +}