Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging.
8.4 KiB
08 — GraphRAG: Structural Knowledge & Relations
Decision: graph layer = Ontotext GraphDB with SPARQL (resolved in 11/Q1). Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.
Two-pillar memory
| Pillar | Question it answers | Backed by |
|---|---|---|
| Qdrant (vectors) | "What is similar / relevant?" | Cosine search over embeddings |
| GraphDB (RDF) | "What is connected? What depends on what? Who is involved?" | SPARQL over a typed graph |
Both pillars are queried in parallel for #nexa:ask and merged before SAIA generates the final answer.
RDF schema
Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.
@prefix nexa: <https://nuclide.systems/nexa/ontology#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix prov: <http://www.w3.org/ns/prov#> .
# Classes
nexa:Project a rdfs:Class .
nexa:Task a rdfs:Class .
nexa:Person a rdfs:Class .
nexa:Technology a rdfs:Class .
nexa:Topic a rdfs:Class .
nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests
nexa:File a rdfs:Class .
# Properties
nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .
nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .
nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .
nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .
nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
# Datatype properties
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
# Cross-pillar / multimodality
nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image"
nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://…
nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual"
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID
nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them
nexa:vectorId + nexa:vectorCollection together are the bridge between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's graph_iri field walks back the other way.
nexa:modality, nexa:mediaUri and nexa:pendingVisualIndex exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as nexa:Note with modality "image" and pendingVisualIndex true, then picked up by the Phase-3.2 backfill workflow — no data loss across phases.
Sync flows
1. Memos → GraphDB (real-time)
Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren"
│
▼ SAIA extracts entities + relations as JSON
│ { tasks: [{title, urgency}], technologies: [...],
│ relations: [{type:"uses", from:..., to:...}] }
│
▼ n8n turns JSON into a SPARQL UPDATE
│
└──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge"
2. Obsidian → GraphDB (#nexa:sync-obsidian)
For each Obsidian note: parse front-matter + headings → emit nexa:Project, nexa:Task, nexa:Note triples; nexa:mentions for [[wikilinks]].
3. Nextcloud Tasks ↔ GraphDB (bidirectional)
n8n trigger on Nextcloud CalDAV/Tasks change → INSERT/DELETE DATA to keep nexa:status and nexa:scheduledFor in sync.
Example SPARQL queries
Q1 — All open tasks involving JWT, by urgency
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?urgency ?projectName
WHERE {
?tech rdfs:label "JWT" .
?task nexa:uses ?tech ;
rdfs:label ?taskTitle ;
nexa:status ?status ;
nexa:urgency ?urgency .
FILTER (?status IN ("needs-action", "in-progress"))
OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }
}
ORDER BY DESC(?urgency)
Q2 — What does Auth-Service transitively depend on?
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT DISTINCT ?dep ?label
WHERE {
?root rdfs:label "Auth-Service" .
?root nexa:dependsOn+ ?dep .
?dep rdfs:label ?label .
}
(+ is SPARQL property-paths — transitive closure, free.)
Q3 — Topics with the most note-mentions in the last day
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?topic (COUNT(?note) AS ?n)
WHERE {
?note a nexa:Note ;
prov:generatedAtTime ?ts ;
nexa:mentions ?topic .
FILTER (?ts > NOW() - "P1D"^^xsd:duration)
}
GROUP BY ?topic
ORDER BY DESC(?n)
LIMIT 10
Q4 — Cross-pillar: "find vectors for tasks blocking project X"
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?vectorId
WHERE {
?proj rdfs:label "Nexa" .
?task nexa:childOf ?proj ;
nexa:status "in-progress" ;
nexa:vectorId ?vectorId ;
rdfs:label ?taskTitle .
}
n8n then takes each ?vectorId, fetches the embedding from Qdrant, and runs a "more like this" search for richer context.
GraphRAG answer pipeline (#nexa:ask)
#nexa:ask <question>
│
┌──────┴──────┐
▼ ▼
[Qdrant] [GraphDB]
semantic structural
top-k SPARQL — auto-generated
notes paths / dependencies
│ │
└──────┬──────┘
▼
merge + rank
│
▼
SAIA prompt:
"Given these passages and these relations, answer …"
│
▼
comment under the original memo
Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a SELECT/CONSTRUCT query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.
n8n integration sketch
Workflow: Graph-Sync Trigger
[Memos Webhook]
│
[Parse Content]
│
[SAIA: Extract entities + relations as JSON]
│
[Build SPARQL UPDATE INSERT DATA { ... }]
│
[HTTP POST → /repositories/nexa_knowledge/statements]
│
[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]
Workflow: Question Router
[#nexa:ask Query]
│
┌─┴────────────────┐
▼ ▼
[SAIA: NL → SPARQL] [Qdrant: kNN]
│ │
[POST → SPARQL endpoint]
│ │
└──────┬───────────┘
▼
rank + merge → SAIA answer
Graph-management commands
#nexa:graph-status
Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL SELECT (COUNT).
#nexa:graph-trace [entity]
Returns the 1-hop (and optionally 2-hop) neighbourhood — a DESCRIBE <iri> plus a templated outgoing/incoming query.
#nexa:graph-rebuild
Clears the named graph and replays Obsidian + Memos. SPARQL: CLEAR GRAPH <https://nuclide.systems/nexa/runtime> followed by the import workflow.
Why two stores
| Scenario | Qdrant | GraphDB | Best |
|---|---|---|---|
| "Which note was similar to this one?" | ✅ | ❌ | Qdrant |
| "What blocks this task?" | ❌ | ✅ | GraphDB |
| "Explain this project" | ✅ context | ✅ structure | both |
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
Combined: complete understanding rather than a search index or a structure index.