Stage embeddings: ship Path A (text) now, prepare Path C (text+visual) additively
Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging.
This commit is contained in:
Claude
+64
-4
@@ -49,20 +49,44 @@ $EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KE
|
||||
|
||||
The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead.
|
||||
|
||||
## Step 2 — Qdrant collection
|
||||
## Step 2 — Qdrant collection (`nexa_knowledge_text`)
|
||||
|
||||
Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring.
|
||||
|
||||
```bash
|
||||
# adjust QDRANT_HOST in .env first
|
||||
source nexa-core/.env
|
||||
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \
|
||||
# create the text collection from the schema file
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema.json
|
||||
```
|
||||
|
||||
The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)):
|
||||
|
||||
| Field | Why it's there now |
|
||||
|-------|--------------------|
|
||||
| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. |
|
||||
| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. |
|
||||
| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. |
|
||||
| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. |
|
||||
| `content_hash` | de-dup. |
|
||||
| `context` | `work` / `personal`. |
|
||||
|
||||
> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container.
|
||||
> The schema file's `vectors.size` must match the embedding model picked in [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
|
||||
> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
|
||||
|
||||
### Image attachments today (queue them)
|
||||
|
||||
Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later:
|
||||
|
||||
- Memo with an image → text body still goes into `nexa_knowledge_text`.
|
||||
- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`).
|
||||
- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection.
|
||||
|
||||
This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself.
|
||||
|
||||
## Step 3 — Self-hosted embeddings (TEI)
|
||||
|
||||
@@ -228,8 +252,44 @@ For schema and example queries: [08-graphrag-architecture](./08-graphrag-archite
|
||||
|
||||
---
|
||||
|
||||
## Phase add-on: visual collection (Phase 3.2)
|
||||
|
||||
Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`.
|
||||
|
||||
```bash
|
||||
# (1) replace TEI with infinity (or run alongside) for CLIP-family support
|
||||
docker rm -f nexa-embed
|
||||
docker run -d --name nexa-embed \
|
||||
--restart unless-stopped \
|
||||
-p 127.0.0.1:8080:80 \
|
||||
-v infinity-data:/app/.cache \
|
||||
michaelf34/infinity:latest \
|
||||
v2 \
|
||||
--model-id BAAI/bge-m3 \
|
||||
--model-id jinaai/jina-clip-v2 \
|
||||
--port 80
|
||||
|
||||
# (2) create the visual collection
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema_visual.json
|
||||
|
||||
# (3) register the second model in LiteLLM as `nexa-embed-visual`
|
||||
# (same OpenAI-compatible route, different model id)
|
||||
|
||||
# (4) backfill queued images:
|
||||
# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }
|
||||
# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,
|
||||
# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.
|
||||
```
|
||||
|
||||
n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt.
|
||||
|
||||
---
|
||||
|
||||
## Step-back / rollback
|
||||
|
||||
- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops.
|
||||
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`.
|
||||
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`.
|
||||
- **Re-discover** — `#nexa:reset-config` then `#nexa:config`.
|
||||
|
||||
Reference in New Issue
Block a user