Stage embeddings: ship Path A (text) now, prepare Path C (text+visual) additively
Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging.
This commit is contained in:
@@ -6,18 +6,20 @@ Items that **block progress** and need a human decision before a workflow can be
|
||||
|
||||
- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly.
|
||||
- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container.
|
||||
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below; concrete model still open as **Q15**.
|
||||
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below.
|
||||
- [x] **Q15 — Embeddings staging plan → A now, C prepared.**
|
||||
- **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus.
|
||||
- **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16).
|
||||
- All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json).
|
||||
|
||||
## Architectural decisions
|
||||
|
||||
- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable.
|
||||
- [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts.
|
||||
- [ ] **Q15 — Embedding model (served via TEI).** Two viable options on the docker host (CPU only — see speed budget):
|
||||
- **`BAAI/bge-m3`** (568 M params, ~1 GB RAM, multilingual incl. German, 1024-dim, ~10–20 docs/s on CPU). **Recommended** because the corpus is DE/EN-mixed. Requires `qdrant_schema.json` `vectors.size = 1024`.
|
||||
- **`nomic-ai/nomic-embed-text-v1.5`** (137 M, ~250 MB RAM, EN-leaning, 768-dim, ~50 docs/s CPU). Lighter / faster but weaker on German.
|
||||
- **(maybe) does SAIA already proxy any embedding model?** If the SAIA backend offers e.g. `mistral-embed` for free, we can skip TEI entirely. Worth a 1-line check in the LiteLLM admin UI before deploying TEI.
|
||||
|
||||
Decide before Phase 3.1.
|
||||
- [ ] **Q16 — Image-attachment queue ergonomics (Phase 3.2).** The agreed plan is: in 3.1 we record images as `nexa:Note` with `nexa:pendingVisualIndex true` but don't embed them. Open sub-questions for when 3.2 lands:
|
||||
- Where to store image bytes between capture and indexing? Three options: (a) leave them in their source (Memos attachments dir / Nextcloud / Obsidian), reference by `media_uri`; (b) copy to a staging area on the docker host; (c) push to `s3.nuclide.systems` immediately. **Recommendation: (a)** — zero copy, smallest blast radius.
|
||||
- Does SAIA already proxy any embedding model? If the SAIA backend offers e.g. `mistral-embed` we could simplify 3.1 by skipping TEI. Worth a 1-line check in the LiteLLM admin UI.
|
||||
- [ ] **Q17 — Reuse Immich's CLIP for photo-library queries?** Immich already runs CLIP server-side on the photo library. For images that live *in Immich*, querying its smart-search API is cheaper than re-embedding. Is the Immich API key OK to add to the n8n workflow, or do we treat Immich as out-of-band?
|
||||
|
||||
## Identifiers needed (auto-discoverable, but list now if known)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user