Claude
17c7f5033f
Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging.
255 lines
8.4 KiB
Markdown
255 lines
8.4 KiB
Markdown
# 08 — GraphRAG: Structural Knowledge & Relations
|
||
|
||
Decision: graph layer = **Ontotext GraphDB** with **SPARQL** (resolved in [11/Q1](./11-open-questions.md)).
|
||
Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.
|
||
|
||
---
|
||
|
||
## Two-pillar memory
|
||
|
||
| Pillar | Question it answers | Backed by |
|
||
|--------|---------------------|-----------|
|
||
| **Qdrant** (vectors) | *"What is similar / relevant?"* | Cosine search over embeddings |
|
||
| **GraphDB** (RDF) | *"What is connected? What depends on what? Who is involved?"* | SPARQL over a typed graph |
|
||
|
||
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer.
|
||
|
||
---
|
||
|
||
## RDF schema
|
||
|
||
Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.
|
||
|
||
```turtle
|
||
@prefix nexa: <https://nuclide.systems/nexa/ontology#> .
|
||
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||
@prefix prov: <http://www.w3.org/ns/prov#> .
|
||
|
||
# Classes
|
||
nexa:Project a rdfs:Class .
|
||
nexa:Task a rdfs:Class .
|
||
nexa:Person a rdfs:Class .
|
||
nexa:Technology a rdfs:Class .
|
||
nexa:Topic a rdfs:Class .
|
||
nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests
|
||
nexa:File a rdfs:Class .
|
||
|
||
# Properties
|
||
nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .
|
||
nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .
|
||
nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .
|
||
nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .
|
||
nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .
|
||
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
|
||
|
||
# Datatype properties
|
||
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
|
||
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
|
||
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5
|
||
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
|
||
|
||
# Cross-pillar / multimodality
|
||
nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image"
|
||
nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://…
|
||
nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual"
|
||
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID
|
||
nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them
|
||
```
|
||
|
||
`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way.
|
||
|
||
`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases.
|
||
|
||
---
|
||
|
||
## Sync flows
|
||
|
||
### 1. Memos → GraphDB (real-time)
|
||
|
||
```
|
||
Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren"
|
||
│
|
||
▼ SAIA extracts entities + relations as JSON
|
||
│ { tasks: [{title, urgency}], technologies: [...],
|
||
│ relations: [{type:"uses", from:..., to:...}] }
|
||
│
|
||
▼ n8n turns JSON into a SPARQL UPDATE
|
||
│
|
||
└──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge"
|
||
```
|
||
|
||
### 2. Obsidian → GraphDB (`#nexa:sync-obsidian`)
|
||
|
||
For each Obsidian note: parse front-matter + headings → emit `nexa:Project`, `nexa:Task`, `nexa:Note` triples; `nexa:mentions` for `[[wikilinks]]`.
|
||
|
||
### 3. Nextcloud Tasks ↔ GraphDB (bidirectional)
|
||
|
||
n8n trigger on Nextcloud CalDAV/Tasks change → `INSERT/DELETE DATA` to keep `nexa:status` and `nexa:scheduledFor` in sync.
|
||
|
||
---
|
||
|
||
## Example SPARQL queries
|
||
|
||
### Q1 — All open tasks involving JWT, by urgency
|
||
|
||
```sparql
|
||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||
|
||
SELECT ?taskTitle ?urgency ?projectName
|
||
WHERE {
|
||
?tech rdfs:label "JWT" .
|
||
?task nexa:uses ?tech ;
|
||
rdfs:label ?taskTitle ;
|
||
nexa:status ?status ;
|
||
nexa:urgency ?urgency .
|
||
FILTER (?status IN ("needs-action", "in-progress"))
|
||
OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }
|
||
}
|
||
ORDER BY DESC(?urgency)
|
||
```
|
||
|
||
### Q2 — What does Auth-Service transitively depend on?
|
||
|
||
```sparql
|
||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||
|
||
SELECT DISTINCT ?dep ?label
|
||
WHERE {
|
||
?root rdfs:label "Auth-Service" .
|
||
?root nexa:dependsOn+ ?dep .
|
||
?dep rdfs:label ?label .
|
||
}
|
||
```
|
||
|
||
(`+` is SPARQL property-paths — transitive closure, free.)
|
||
|
||
### Q3 — Topics with the most note-mentions in the last day
|
||
|
||
```sparql
|
||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
|
||
|
||
SELECT ?topic (COUNT(?note) AS ?n)
|
||
WHERE {
|
||
?note a nexa:Note ;
|
||
prov:generatedAtTime ?ts ;
|
||
nexa:mentions ?topic .
|
||
FILTER (?ts > NOW() - "P1D"^^xsd:duration)
|
||
}
|
||
GROUP BY ?topic
|
||
ORDER BY DESC(?n)
|
||
LIMIT 10
|
||
```
|
||
|
||
### Q4 — Cross-pillar: "find vectors for tasks blocking project X"
|
||
|
||
```sparql
|
||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||
|
||
SELECT ?taskTitle ?vectorId
|
||
WHERE {
|
||
?proj rdfs:label "Nexa" .
|
||
?task nexa:childOf ?proj ;
|
||
nexa:status "in-progress" ;
|
||
nexa:vectorId ?vectorId ;
|
||
rdfs:label ?taskTitle .
|
||
}
|
||
```
|
||
|
||
n8n then takes each `?vectorId`, fetches the embedding from Qdrant, and runs a "more like this" search for richer context.
|
||
|
||
---
|
||
|
||
## GraphRAG answer pipeline (`#nexa:ask`)
|
||
|
||
```
|
||
#nexa:ask <question>
|
||
│
|
||
┌──────┴──────┐
|
||
▼ ▼
|
||
[Qdrant] [GraphDB]
|
||
semantic structural
|
||
top-k SPARQL — auto-generated
|
||
notes paths / dependencies
|
||
│ │
|
||
└──────┬──────┘
|
||
▼
|
||
merge + rank
|
||
│
|
||
▼
|
||
SAIA prompt:
|
||
"Given these passages and these relations, answer …"
|
||
│
|
||
▼
|
||
comment under the original memo
|
||
```
|
||
|
||
Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a `SELECT`/`CONSTRUCT` query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.
|
||
|
||
---
|
||
|
||
## n8n integration sketch
|
||
|
||
### Workflow: Graph-Sync Trigger
|
||
|
||
```
|
||
[Memos Webhook]
|
||
│
|
||
[Parse Content]
|
||
│
|
||
[SAIA: Extract entities + relations as JSON]
|
||
│
|
||
[Build SPARQL UPDATE INSERT DATA { ... }]
|
||
│
|
||
[HTTP POST → /repositories/nexa_knowledge/statements]
|
||
│
|
||
[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]
|
||
```
|
||
|
||
### Workflow: Question Router
|
||
|
||
```
|
||
[#nexa:ask Query]
|
||
│
|
||
┌─┴────────────────┐
|
||
▼ ▼
|
||
[SAIA: NL → SPARQL] [Qdrant: kNN]
|
||
│ │
|
||
[POST → SPARQL endpoint]
|
||
│ │
|
||
└──────┬───────────┘
|
||
▼
|
||
rank + merge → SAIA answer
|
||
```
|
||
|
||
---
|
||
|
||
## Graph-management commands
|
||
|
||
### `#nexa:graph-status`
|
||
|
||
Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL `SELECT (COUNT)`.
|
||
|
||
### `#nexa:graph-trace [entity]`
|
||
|
||
Returns the 1-hop (and optionally 2-hop) neighbourhood — a `DESCRIBE <iri>` plus a templated outgoing/incoming query.
|
||
|
||
### `#nexa:graph-rebuild`
|
||
|
||
Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https://nuclide.systems/nexa/runtime>` followed by the import workflow.
|
||
|
||
---
|
||
|
||
## Why two stores
|
||
|
||
| Scenario | Qdrant | GraphDB | Best |
|
||
|----------|--------|---------|------|
|
||
| "Which note was similar to this one?" | ✅ | ❌ | Qdrant |
|
||
| "What blocks this task?" | ❌ | ✅ | GraphDB |
|
||
| "Explain this project" | ✅ context | ✅ structure | both |
|
||
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
|
||
|
||
Combined: **complete understanding** rather than a search index *or* a structure index.
|