Files
nexa/docs/08-graphrag-architecture.md
T
Claude a1e14c64c3 Apply Q1–Q3 decisions: GraphDB/SPARQL, reuse Qdrant, self-host embeddings via TEI
- Q1 → Ontotext GraphDB (SPARQL). docs/08 fully rewritten with RDF schema,
  example SPARQL queries (transitive deps via property paths, time-windowed
  topic counts, cross-pillar joins via nexa:vectorId).
- Q2 → reuse qdrant_scientific with nexa_* collection prefix; docs/09 step 2
  now points there explicitly.
- Q3 → no OpenAI embeddings. Self-host on the docker host. Use TEI
  (HuggingFace text-embeddings-inference) — single Rust binary, ~500 MB image,
  OpenAI-compatible — instead of Ollama, since we only need embeddings.
- docs/09 Phase-3.4 add-on simplified to a single Ontotext compose snippet
  (Neo4j option dropped) plus repo creation curl.
- docs/11 Q3 marked resolved; new Q15 picks the model (bge-m3 vs nomic-embed)
  and adds the open question of whether SAIA already proxies an embedding
  model that would let us skip TEI entirely.
- docs/03 + CLAUDE.md updated with the new decisions so future runs don't
  re-litigate.
2026-05-04 21:26:08 +00:00

247 lines
7.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 08 — GraphRAG: Structural Knowledge & Relations
Decision: graph layer = **Ontotext GraphDB** with **SPARQL** (resolved in [11/Q1](./11-open-questions.md)).
Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.
---
## Two-pillar memory
| Pillar | Question it answers | Backed by |
|--------|---------------------|-----------|
| **Qdrant** (vectors) | *"What is similar / relevant?"* | Cosine search over embeddings |
| **GraphDB** (RDF) | *"What is connected? What depends on what? Who is involved?"* | SPARQL over a typed graph |
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer.
---
## RDF schema
Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.
```turtle
@prefix nexa: <https://nuclide.systems/nexa/ontology#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix prov: <http://www.w3.org/ns/prov#> .
# Classes
nexa:Project a rdfs:Class .
nexa:Task a rdfs:Class .
nexa:Person a rdfs:Class .
nexa:Technology a rdfs:Class .
nexa:Topic a rdfs:Class .
nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests
nexa:File a rdfs:Class .
# Properties
nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .
nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .
nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .
nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .
nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
# Datatype properties
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 15
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID — bridges the two pillars
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
```
The `nexa:vectorId` property is the **bridge** between graph and vector store. Every node that has semantic content carries the Qdrant point ID, so a SPARQL hit can trigger a vector lookup and vice versa.
---
## Sync flows
### 1. Memos → GraphDB (real-time)
```
Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren"
▼ SAIA extracts entities + relations as JSON
│ { tasks: [{title, urgency}], technologies: [...],
│ relations: [{type:"uses", from:..., to:...}] }
▼ n8n turns JSON into a SPARQL UPDATE
└──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge"
```
### 2. Obsidian → GraphDB (`#nexa:sync-obsidian`)
For each Obsidian note: parse front-matter + headings → emit `nexa:Project`, `nexa:Task`, `nexa:Note` triples; `nexa:mentions` for `[[wikilinks]]`.
### 3. Nextcloud Tasks ↔ GraphDB (bidirectional)
n8n trigger on Nextcloud CalDAV/Tasks change → `INSERT/DELETE DATA` to keep `nexa:status` and `nexa:scheduledFor` in sync.
---
## Example SPARQL queries
### Q1 — All open tasks involving JWT, by urgency
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?urgency ?projectName
WHERE {
?tech rdfs:label "JWT" .
?task nexa:uses ?tech ;
rdfs:label ?taskTitle ;
nexa:status ?status ;
nexa:urgency ?urgency .
FILTER (?status IN ("needs-action", "in-progress"))
OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }
}
ORDER BY DESC(?urgency)
```
### Q2 — What does Auth-Service transitively depend on?
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT DISTINCT ?dep ?label
WHERE {
?root rdfs:label "Auth-Service" .
?root nexa:dependsOn+ ?dep .
?dep rdfs:label ?label .
}
```
(`+` is SPARQL property-paths — transitive closure, free.)
### Q3 — Topics with the most note-mentions in the last day
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?topic (COUNT(?note) AS ?n)
WHERE {
?note a nexa:Note ;
prov:generatedAtTime ?ts ;
nexa:mentions ?topic .
FILTER (?ts > NOW() - "P1D"^^xsd:duration)
}
GROUP BY ?topic
ORDER BY DESC(?n)
LIMIT 10
```
### Q4 — Cross-pillar: "find vectors for tasks blocking project X"
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?vectorId
WHERE {
?proj rdfs:label "Nexa" .
?task nexa:childOf ?proj ;
nexa:status "in-progress" ;
nexa:vectorId ?vectorId ;
rdfs:label ?taskTitle .
}
```
n8n then takes each `?vectorId`, fetches the embedding from Qdrant, and runs a "more like this" search for richer context.
---
## GraphRAG answer pipeline (`#nexa:ask`)
```
#nexa:ask <question>
┌──────┴──────┐
▼ ▼
[Qdrant] [GraphDB]
semantic structural
top-k SPARQL — auto-generated
notes paths / dependencies
│ │
└──────┬──────┘
merge + rank
SAIA prompt:
"Given these passages and these relations, answer …"
comment under the original memo
```
Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a `SELECT`/`CONSTRUCT` query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.
---
## n8n integration sketch
### Workflow: Graph-Sync Trigger
```
[Memos Webhook]
[Parse Content]
[SAIA: Extract entities + relations as JSON]
[Build SPARQL UPDATE INSERT DATA { ... }]
[HTTP POST → /repositories/nexa_knowledge/statements]
[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]
```
### Workflow: Question Router
```
[#nexa:ask Query]
┌─┴────────────────┐
▼ ▼
[SAIA: NL → SPARQL] [Qdrant: kNN]
│ │
[POST → SPARQL endpoint]
│ │
└──────┬───────────┘
rank + merge → SAIA answer
```
---
## Graph-management commands
### `#nexa:graph-status`
Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL `SELECT (COUNT)`.
### `#nexa:graph-trace [entity]`
Returns the 1-hop (and optionally 2-hop) neighbourhood — a `DESCRIBE <iri>` plus a templated outgoing/incoming query.
### `#nexa:graph-rebuild`
Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https://nuclide.systems/nexa/runtime>` followed by the import workflow.
---
## Why two stores
| Scenario | Qdrant | GraphDB | Best |
|----------|--------|---------|------|
| "Which note was similar to this one?" | ✅ | ❌ | Qdrant |
| "What blocks this task?" | ❌ | ✅ | GraphDB |
| "Explain this project" | ✅ context | ✅ structure | both |
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
Combined: **complete understanding** rather than a search index *or* a structure index.