Stage embeddings: ship Path A (text) now, prepare Path C (text+visual) additively
Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging.
This commit is contained in:
Claude
+6
-6
@@ -22,13 +22,13 @@ Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
|
||||
## Phase 3 — Memory (Qdrant & RAG)
|
||||
|
||||
**Focus:** Qdrant + (optional) graph DB for retrieval-augmented answers.
|
||||
**Milestone:** RAG works — Nexa answers from archived notes.
|
||||
**Focus:** Qdrant + graph layer for retrieval-augmented answers.
|
||||
**Milestone:** RAG works — Nexa answers from archived notes; images captured today are queued for visual indexing later.
|
||||
|
||||
- [ ] **3.1 Embedding pipeline** — automatic vectorization of Memos & Obsidian.
|
||||
- [ ] **3.2 Context retrieval** — `#nexa:ask` reads Qdrant before answering.
|
||||
- [ ] **3.3 S3 long-term archive** — Qdrant snapshots to the S3 server.
|
||||
- [ ] **3.4 Graph layer (deferred decision — see [11-open-questions](./11-open-questions.md))** — Neo4j *or* Ontotext GraphDB for structural queries.
|
||||
- [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet.
|
||||
- [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32).
|
||||
- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server.
|
||||
- [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md).
|
||||
|
||||
## Phase 4 — Motor (organization & action)
|
||||
|
||||
|
||||
@@ -42,9 +42,9 @@ A one-page mental model. For details follow the cross-links.
|
||||
| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` |
|
||||
| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` |
|
||||
| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) |
|
||||
| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse existing `qdrant_scientific` with `nexa_*` collection prefix |
|
||||
| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-graph-db-phase-34) |
|
||||
| **TEI** (HF text-embeddings-inference) | Self-hosted embeddings (`bge-m3` / TBD — see [11/Q15](./11-open-questions.md)) | docker host LXC 104, CPU only — single-binary embedding server |
|
||||
| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) |
|
||||
| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-ontotext-graphdb-phase-34) |
|
||||
| **TEI** (HF text-embeddings-inference) | Self-hosted text embeddings, `BAAI/bge-m3`, Phase 3.1 | docker host LXC 104, CPU only — swapped for `infinity` in Phase 3.2 to add `jina-clip-v2` |
|
||||
| **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` |
|
||||
| **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` |
|
||||
| **Backrest** | Backup orchestration | LXC 103 |
|
||||
|
||||
@@ -45,14 +45,22 @@ nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range n
|
||||
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
|
||||
|
||||
# Datatype properties
|
||||
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
|
||||
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
|
||||
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5
|
||||
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID — bridges the two pillars
|
||||
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
|
||||
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
|
||||
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
|
||||
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5
|
||||
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
|
||||
|
||||
# Cross-pillar / multimodality
|
||||
nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image"
|
||||
nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://…
|
||||
nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual"
|
||||
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID
|
||||
nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them
|
||||
```
|
||||
|
||||
The `nexa:vectorId` property is the **bridge** between graph and vector store. Every node that has semantic content carries the Qdrant point ID, so a SPARQL hit can trigger a vector lookup and vice versa.
|
||||
`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way.
|
||||
|
||||
`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+64
-4
@@ -49,20 +49,44 @@ $EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KE
|
||||
|
||||
The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead.
|
||||
|
||||
## Step 2 — Qdrant collection
|
||||
## Step 2 — Qdrant collection (`nexa_knowledge_text`)
|
||||
|
||||
Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring.
|
||||
|
||||
```bash
|
||||
# adjust QDRANT_HOST in .env first
|
||||
source nexa-core/.env
|
||||
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \
|
||||
# create the text collection from the schema file
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema.json
|
||||
```
|
||||
|
||||
The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)):
|
||||
|
||||
| Field | Why it's there now |
|
||||
|-------|--------------------|
|
||||
| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. |
|
||||
| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. |
|
||||
| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. |
|
||||
| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. |
|
||||
| `content_hash` | de-dup. |
|
||||
| `context` | `work` / `personal`. |
|
||||
|
||||
> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container.
|
||||
> The schema file's `vectors.size` must match the embedding model picked in [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
|
||||
> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
|
||||
|
||||
### Image attachments today (queue them)
|
||||
|
||||
Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later:
|
||||
|
||||
- Memo with an image → text body still goes into `nexa_knowledge_text`.
|
||||
- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`).
|
||||
- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection.
|
||||
|
||||
This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself.
|
||||
|
||||
## Step 3 — Self-hosted embeddings (TEI)
|
||||
|
||||
@@ -228,8 +252,44 @@ For schema and example queries: [08-graphrag-architecture](./08-graphrag-archite
|
||||
|
||||
---
|
||||
|
||||
## Phase add-on: visual collection (Phase 3.2)
|
||||
|
||||
Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`.
|
||||
|
||||
```bash
|
||||
# (1) replace TEI with infinity (or run alongside) for CLIP-family support
|
||||
docker rm -f nexa-embed
|
||||
docker run -d --name nexa-embed \
|
||||
--restart unless-stopped \
|
||||
-p 127.0.0.1:8080:80 \
|
||||
-v infinity-data:/app/.cache \
|
||||
michaelf34/infinity:latest \
|
||||
v2 \
|
||||
--model-id BAAI/bge-m3 \
|
||||
--model-id jinaai/jina-clip-v2 \
|
||||
--port 80
|
||||
|
||||
# (2) create the visual collection
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema_visual.json
|
||||
|
||||
# (3) register the second model in LiteLLM as `nexa-embed-visual`
|
||||
# (same OpenAI-compatible route, different model id)
|
||||
|
||||
# (4) backfill queued images:
|
||||
# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }
|
||||
# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,
|
||||
# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.
|
||||
```
|
||||
|
||||
n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt.
|
||||
|
||||
---
|
||||
|
||||
## Step-back / rollback
|
||||
|
||||
- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops.
|
||||
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`.
|
||||
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`.
|
||||
- **Re-discover** — `#nexa:reset-config` then `#nexa:config`.
|
||||
|
||||
@@ -6,18 +6,20 @@ Items that **block progress** and need a human decision before a workflow can be
|
||||
|
||||
- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly.
|
||||
- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container.
|
||||
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below; concrete model still open as **Q15**.
|
||||
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below.
|
||||
- [x] **Q15 — Embeddings staging plan → A now, C prepared.**
|
||||
- **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus.
|
||||
- **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16).
|
||||
- All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json).
|
||||
|
||||
## Architectural decisions
|
||||
|
||||
- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable.
|
||||
- [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts.
|
||||
- [ ] **Q15 — Embedding model (served via TEI).** Two viable options on the docker host (CPU only — see speed budget):
|
||||
- **`BAAI/bge-m3`** (568 M params, ~1 GB RAM, multilingual incl. German, 1024-dim, ~10–20 docs/s on CPU). **Recommended** because the corpus is DE/EN-mixed. Requires `qdrant_schema.json` `vectors.size = 1024`.
|
||||
- **`nomic-ai/nomic-embed-text-v1.5`** (137 M, ~250 MB RAM, EN-leaning, 768-dim, ~50 docs/s CPU). Lighter / faster but weaker on German.
|
||||
- **(maybe) does SAIA already proxy any embedding model?** If the SAIA backend offers e.g. `mistral-embed` for free, we can skip TEI entirely. Worth a 1-line check in the LiteLLM admin UI before deploying TEI.
|
||||
|
||||
Decide before Phase 3.1.
|
||||
- [ ] **Q16 — Image-attachment queue ergonomics (Phase 3.2).** The agreed plan is: in 3.1 we record images as `nexa:Note` with `nexa:pendingVisualIndex true` but don't embed them. Open sub-questions for when 3.2 lands:
|
||||
- Where to store image bytes between capture and indexing? Three options: (a) leave them in their source (Memos attachments dir / Nextcloud / Obsidian), reference by `media_uri`; (b) copy to a staging area on the docker host; (c) push to `s3.nuclide.systems` immediately. **Recommendation: (a)** — zero copy, smallest blast radius.
|
||||
- Does SAIA already proxy any embedding model? If the SAIA backend offers e.g. `mistral-embed` we could simplify 3.1 by skipping TEI. Worth a 1-line check in the LiteLLM admin UI.
|
||||
- [ ] **Q17 — Reuse Immich's CLIP for photo-library queries?** Immich already runs CLIP server-side on the photo library. For images that live *in Immich*, querying its smart-search API is cheaper than re-embedding. Is the Immich API key OK to add to the n8n workflow, or do we treat Immich as out-of-band?
|
||||
|
||||
## Identifiers needed (auto-discoverable, but list now if known)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user