Merge pull request #1 from noheton/claude/organize-docs-deployment-7N4v2
Reorganize documentation and consolidate deployment guides
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Instructions for Claude (and other agents)
|
||||
|
||||
This file tells future automated runs what they need to know about this repo.
|
||||
|
||||
## Repo conventions
|
||||
|
||||
- **Documentation:** all docs live in `/docs/` and are numbered. Entry point is `docs/index.md`. When you add a doc, give it the next free `NN-` prefix and add a row to the index TOC.
|
||||
- **Runtime artifacts:** live in `nexa-core/` (workflows, prompts, configs, scripts). Don't put `.md` documentation in there — link from `/docs/` instead.
|
||||
- **Source-of-truth:** if a doc duplicates content from `nexa-core/config/*.md`, **delete the duplicate**. Single source of truth.
|
||||
|
||||
## Real infrastructure (verified from screenshots, May 2026)
|
||||
|
||||
- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9, kernel 6.17.13-4-pve, EFI).
|
||||
- Hardware: **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62 GiB RAM**, **1.64 TiB disk**.
|
||||
- Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
|
||||
- LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
|
||||
- LXC 103 backrest — backup orchestration.
|
||||
- LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at `/var/nfs/shared/storage`, mounted by Proxmox at `/mnt/pve/unas`, also exposed via SMB at `smb://192.168.1.31/<share>` (Mac) / `\\192.168.1.31\<share>` (Win).
|
||||
|
||||
**Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` from inside LXC 104.** No `driver_opts`, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an **escape hatch only** ([docs/12 #27](./docs/12-optimization-opportunities.md#27)) for services that hit Nextcloud-style NFS issues — Nexa doesn't, so we don't use it.
|
||||
|
||||
**Storage-layer snapshots are NOT configured** on UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, not "the whole pool last Tuesday". Highest-leverage fix in the homelab right now ([docs/12 #38](./docs/12-optimization-opportunities.md#38)).
|
||||
|
||||
**UNAS layout conventions** (homelab-wide, all docker containers follow them):
|
||||
- **`services/<svc>/` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout.
|
||||
- `backup/<svc>/` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.**
|
||||
- `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern.
|
||||
- **Before deploying any Nexa container, READ AN EXISTING STACK** in **Arcane** (e.g. `karakeep` or `immich`) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`.
|
||||
|
||||
**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob):
|
||||
- `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files.
|
||||
- Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`.
|
||||
- The Nextcloud appdata dir (`services/nextcloud/appdata_*`) — Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26).
|
||||
- LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`.
|
||||
- LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13).
|
||||
- LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`.
|
||||
- VM 100 haos — Home Assistant.
|
||||
- **Already-running services on docker host** (don't redeploy):
|
||||
- Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`).
|
||||
- **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state.
|
||||
- **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`.
|
||||
- **Nextcloud Tasks lists & calendars** (German names, may grow over time):
|
||||
- `Persönlich` → Personal context.
|
||||
- `DLR` → Work context (DLR is the user's employer).
|
||||
- `Einkaufsliste` → Shopping.
|
||||
- `Wunschliste` → Wishes.
|
||||
Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh.
|
||||
- **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests.
|
||||
- **Backup model is 3-2-1**: UNAS native snapshots → `s3.nuclide.systems` (warm, on-site) → encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate — see [Q20](./docs/11-open-questions.md#q20)). Always `restic`/`rclone-crypt` before upload — third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first.
|
||||
- **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords.
|
||||
- **Decided for Nexa** (don't re-litigate without user input):
|
||||
- Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`).
|
||||
- Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive.
|
||||
- Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. RDF schema in docs/08 already includes `nexa:modality` / `nexa:mediaUri` / `nexa:vectorCollection` / `nexa:pendingVisualIndex`.
|
||||
- Chat model: SAIA via LiteLLM virtual key.
|
||||
|
||||
## When working on Nexa
|
||||
|
||||
1. **Read `docs/index.md` first** — it's the navigator.
|
||||
2. **Open questions first.** Before writing code or workflow JSON, scan `docs/11-open-questions.md`. If your task touches an unanswered Q, **stop and ask** rather than picking a default. Append new blockers to that doc as `[ ] Q-NN`.
|
||||
3. **Optimization findings.** When you spot infrastructure improvements, add them to `docs/12-optimization-opportunities.md` as a numbered bullet — don't just mention them in commit messages.
|
||||
4. **Never inline secrets** in workflow JSON or `.env` committed to git. Use n8n credentials, LiteLLM virtual keys, or (longer term) Vaultwarden.
|
||||
5. **Keep deployment minimal.** The default answer to *"do we need a new container?"* is **no** — the existing stack covers most needs.
|
||||
|
||||
## Branch policy
|
||||
|
||||
- This branch is `claude/organize-docs-deployment-7N4v2`. Push only here unless told otherwise.
|
||||
- New work for an unrelated feature → new branch under `claude/<topic>`.
|
||||
|
||||
## Quick links
|
||||
|
||||
- [docs/index.md](./docs/index.md)
|
||||
- [docs/09-deployment.md](./docs/09-deployment.md) — most-touched file during bring-up
|
||||
- [docs/11-open-questions.md](./docs/11-open-questions.md) — read **before** assuming defaults
|
||||
@@ -1 +1,17 @@
|
||||
# nexa
|
||||
|
||||
**Neural Nexus for Information & Automation** — central nervous system for a personal IT setup.
|
||||
|
||||
> Memos = voice & ear · n8n = reflexes · SAIA (LiteLLM) = brain · Qdrant (+ optional graph DB) = memory · Nextcloud = hands.
|
||||
|
||||
## Start here
|
||||
|
||||
📖 **[docs/index.md](./docs/index.md)** — TOC, reading paths, repo layout.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
nexa/
|
||||
├── docs/ ← all documentation, numbered for reading order
|
||||
└── nexa-core/ ← runtime: n8n workflows, configs, prompts, scripts
|
||||
```
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# 01 — Vision & Scope
|
||||
|
||||
## Vision
|
||||
|
||||
Nexa is the **central nervous system** of a personal IT setup: an intelligent middleware sitting between input sources (Memos, e-mail, RSS, Karakeep, Bluesky), knowledge stores (Obsidian, Qdrant, Nextcloud) and organization tools (Nextcloud Calendar/Tasks).
|
||||
|
||||
### Core goals
|
||||
|
||||
- **Cognitive offload** — Nexa sorts, filters and proposes; it doesn't just store.
|
||||
- **Context separation** — clean AI-driven split between *Work* and *Personal*.
|
||||
- **Single point of interaction** — Memos is the primary surface ("voice & ear").
|
||||
- **Knowledge synergy** — link ephemeral memos to deep notes via semantic search.
|
||||
|
||||
## Scope
|
||||
|
||||
### In-scope
|
||||
|
||||
- Triaging ~20 personal e-mails per day.
|
||||
- Routing tasks to the right Nextcloud list (Work vs. Personal).
|
||||
- Suggesting time-boxes in the work calendar.
|
||||
- Long-term memory in Qdrant (and optionally a graph DB).
|
||||
- Aggregating critical IT alerts (Proxmox, Backrest) into a Memos system feed.
|
||||
|
||||
### Out-of-scope
|
||||
|
||||
- Direct access to work IT (work mail, work servers).
|
||||
- Active document editing (metadata extraction only).
|
||||
- Replacing specialist UIs (Arcane / Dozzle / Portainer for containers).
|
||||
|
||||
## Success metrics
|
||||
|
||||
- **Reduced mail interaction** — less time in the mail app, more on the Nexa digest.
|
||||
- **Calendar fill-rate** — focus slots auto-blocked on the work calendar.
|
||||
- **Search latency** — old memos / Obsidian notes resurface instantly via `#nexa:ask`.
|
||||
@@ -0,0 +1,62 @@
|
||||
# 02 — Roadmap & Phases
|
||||
|
||||
Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
|
||||
## Phase 1 — The Spine (connectivity)
|
||||
|
||||
**Focus:** datapath between Memos, n8n and SAIA.
|
||||
**Milestone:** Nexa replies on Memos and answers simple questions.
|
||||
|
||||
- [ ] **1.1 Memos ↔ n8n bridge** — webhook ingress, content filter, comment-back.
|
||||
- [ ] **1.2 SAIA integration** — call LiteLLM proxy from n8n; chat via Memos comments.
|
||||
- [ ] **1.3 Git-sync** — periodic export of n8n workflows into this repo.
|
||||
|
||||
## Phase 2 — Senses (input channels)
|
||||
|
||||
**Focus:** e-mail filter and the Work-vs-Personal router.
|
||||
**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
|
||||
|
||||
- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
|
||||
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
|
||||
- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
|
||||
|
||||
## Phase 3 — Memory (Qdrant & RAG)
|
||||
|
||||
**Focus:** Qdrant + graph layer for retrieval-augmented answers.
|
||||
**Milestone:** RAG works — Nexa answers from archived notes; images captured today are queued for visual indexing later.
|
||||
|
||||
- [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet.
|
||||
- [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32).
|
||||
- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. **Deferred** until [Q12 + Q20](./11-open-questions.md#backups-deferred) are decided. Until then, snapshots stay on UNAS at `backup/nexa/snapshots/qdrant/<date>/`; UNAS RAID 6 + native pool snapshots ([12/#38](./12-optimization-opportunities.md#38)) cover the data-loss scenarios that matter most for Phase 3.1.
|
||||
- [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md).
|
||||
|
||||
## Phase 4 — Motor (organization & action)
|
||||
|
||||
**Focus:** calendar integration and time-boxing.
|
||||
**Milestone:** Nexa proactively proposes morning focus slots in the work calendar.
|
||||
|
||||
- [ ] **4.1 Nextcloud Tasks sync** — bidirectional status sync (Memos ↔ NC Tasks).
|
||||
- [ ] **4.2 Time-boxing agent** — calendar-aware slot suggestions.
|
||||
- [ ] **4.3 Karakeep curation** — automated review cycle for saved links.
|
||||
|
||||
## Phase 5 — Daily integration (polish)
|
||||
|
||||
**Focus:** Home Assistant Voice and monitoring.
|
||||
**Milestone:** Nexa speaks via HA-Voice and surfaces critical system states proactively.
|
||||
|
||||
- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
|
||||
- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
|
||||
- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
|
||||
|
||||
## Phase 6 — Nexa as homelab steward
|
||||
|
||||
**Focus:** Nexa actively maintains the homelab inventory instead of being told manually.
|
||||
**Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo.
|
||||
|
||||
- [ ] **6.1 Inventory sync** — Nexa polls Arcane / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …).
|
||||
- [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35–37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*.
|
||||
- [ ] **6.3 Steward commands** — `#nexa:retire <svc>` (tar to `backup/`, stop, archive the stack), `#nexa:document <svc>` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed).
|
||||
- [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface).
|
||||
|
||||
This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) — most of those items become semi-automatic once 6.1–6.3 ship.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 03 — Architecture Overview
|
||||
|
||||
A one-page mental model. For details follow the cross-links.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
┌────────────────┐
|
||||
voice / typing ──────▶│ Memos │◀──── Nexa replies as comments
|
||||
│ (interface) │
|
||||
└───────┬────────┘
|
||||
│ webhook (- [ ] / #nexa:*)
|
||||
▼
|
||||
┌────────────────┐
|
||||
IMAP / RSS / NC ────▶│ n8n │◀──── workflows live in
|
||||
Karakeep / Bluesky │ (logic) │ ./nexa-core/n8n-workflows
|
||||
└───┬────────┬───┘
|
||||
classify ▲ │ │ ▲ retrieve
|
||||
│ ▼ ▼ │
|
||||
┌──────────┐ ┌──────────┐
|
||||
│ SAIA │ │ Qdrant │
|
||||
│ LiteLLM │ │ vector │
|
||||
└──────────┘ └────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌──────────┐
|
||||
│ GraphDB │ Ontotext, SPARQL
|
||||
│ (RDF) │ Phase 3.4
|
||||
└──────────┘
|
||||
│
|
||||
▼ writes
|
||||
┌──────────────────────────────┐
|
||||
│ Nextcloud (Tasks, Calendar, │
|
||||
│ Mail, Files / Obsidian) │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Role | Where it runs (today) |
|
||||
|-----------|------|-----------------------|
|
||||
| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` |
|
||||
| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` |
|
||||
| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) |
|
||||
| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) |
|
||||
| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-ontotext-graphdb-phase-34) |
|
||||
| **TEI** (HF text-embeddings-inference) | Self-hosted text embeddings, `BAAI/bge-m3`, Phase 3.1 | docker host LXC 104, CPU only — swapped for `infinity` in Phase 3.2 to add `jina-clip-v2` |
|
||||
| **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` |
|
||||
| **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` |
|
||||
| **Backrest** | Backup orchestration | LXC 103 |
|
||||
| **Zoraxy** | Reverse proxy + TLS | LXC 108 (`192.168.1.4:8000`) |
|
||||
| **AdGuard DNS** | Internal name resolution | LXC 102 |
|
||||
| **Home Assistant** | Voice + house automation | VM 100 (HAOS) |
|
||||
|
||||
## Two-pillar memory
|
||||
|
||||
- **Qdrant** answers *"what is similar / relevant?"* (cosine search over embeddings).
|
||||
- **Ontotext GraphDB** answers *"who, what depends on what, how is it structured?"* (SPARQL over RDF).
|
||||
|
||||
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. See [08 — GraphRAG architecture](./08-graphrag-architecture.md).
|
||||
|
||||
## Dual-context routing
|
||||
|
||||
Every input is classified `work` or `personal` before any side effect (task creation, calendar write). See [04 — Integration matrix](./04-integration-matrix.md) and [06 — Classification logic](./06-classification-logic.md).
|
||||
@@ -1,4 +1,4 @@
|
||||
# 🔗 Nexa Integrations-Matrix (V2)
|
||||
# 04 — Integrations-Matrix
|
||||
|
||||
Diese Matrix definiert die logische Trennung zwischen privaten und beruflichen Datenströmen sowie die Anbindung der Infrastruktur.
|
||||
|
||||
@@ -10,17 +10,15 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari
|
||||
### A. Bereich: ARBEIT (Work)
|
||||
* **Sichtbarkeit:** Eingeschränkt (kein Zugriff auf Arbeits-E-Mails oder Firmen-Server).
|
||||
* **Datenquellen:**
|
||||
* Separate Nextcloud Task-Liste (`Work_Tasks`).
|
||||
* Separater Nextcloud Kalender (`Work_Calendar`).
|
||||
* Nextcloud Task-Liste **`DLR`** (sowie der gleichnamige Kalender).
|
||||
* Memos mit Tag `#work` oder semantischer Erkennung.
|
||||
* **Nexa-Fokus:** Timeblocking, Fokus-Zeiten, Vorbereitung von Meetings basierend auf Obsidian-Notizen.
|
||||
|
||||
### B. Bereich: PRIVAT (Personal)
|
||||
* **Sichtbarkeit:** Vollständig.
|
||||
* **Datenquellen:**
|
||||
* Nextcloud Task-Liste (`Personal_Tasks`).
|
||||
* Haupt-Kalender.
|
||||
* E-Mails (IMAP - 20 Mails/Tag).
|
||||
* Nextcloud Task-Liste **`Persönlich`** und gleichnamiger Kalender.
|
||||
* Nextcloud Mail (`fkrebs@nucli.de`, einziges Konto). Ordner `Waiting` ist ein manuelles Signal — Mails dort werden vom Digest ausgeschlossen.
|
||||
* Bluesky, RSS, Karakeep.
|
||||
* **Nexa-Fokus:** Automatisierung des Alltags, Kuratierung von Wissen, E-Mail-Management.
|
||||
|
||||
@@ -59,8 +57,10 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari
|
||||
* Wichtige private Termine werden in den privaten Kalender extrahiert.
|
||||
* Newsletter landen als Zusammenfassung in Memos.
|
||||
|
||||
### Wissens-Management (Obsidian & Qdrant)
|
||||
* Obsidian-Notizen (Arbeit & Privat) werden in Qdrant indiziert.
|
||||
### Wissens-Management (Obsidian via Nextcloud & Qdrant)
|
||||
* Der Obsidian-Vault liegt im Nextcloud-Ordner `Notizen/` (Multi-Device-Sync via Nextcloud-Client). Nexa liest **read-only über WebDAV** (`/remote.php/dav/files/<user>/Notizen/`) — kein Mount, kein zweiter Sync-Mechanismus.
|
||||
* Markdown-Dateien (`Notizen/**/*.md`) werden in Qdrant (`nexa_knowledge_text`) indiziert; Bilder unter `assets/` landen im Phase-3.2-Visual-Queue (siehe [docs/11/Q4](./11-open-questions.md)).
|
||||
* Plugin-Internals (`.copilot/`, `.smart-env/`, `.caldav-sync/`, `Templates/`, `BMO/`, `Excalidraw/`) sind explizit von der Indizierung ausgeschlossen.
|
||||
* Nexa nutzt dieses Wissen, um Arbeits-Tasks besser zu beschreiben, auch wenn sie keinen Zugriff auf die Arbeits-E-Mails hat.
|
||||
|
||||
---
|
||||
@@ -0,0 +1,176 @@
|
||||
# 05 — Command System (`#nexa:*`)
|
||||
|
||||
Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag:
|
||||
|
||||
## 🔧 Konfigurations-Kommandos
|
||||
|
||||
### #nexa:config
|
||||
- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud Listen/Kalender, Qdrant Collections, LiteLLM-Modelle, Mail-Konto).
|
||||
- **Läuft auch automatisch**:
|
||||
- Täglich (cron in n8n) — neue Listen / Kalender werden ohne manuelles Eingreifen aufgenommen.
|
||||
- Bei Cache-Miss — wenn ein Workflow eine Liste/ID nicht mehr findet, läuft Discovery einmal, dann Retry.
|
||||
- **Antwort**: Postet Report mit den aktuellen Konfigurationen als Memos-Kommentar.
|
||||
- **Speichert**: Konfiguration in der Qdrant `_config` Namespace (nicht in `.env`).
|
||||
|
||||
```
|
||||
#nexa:config
|
||||
```
|
||||
|
||||
**Beispiel-Response:**
|
||||
```
|
||||
✅ Autodiscovery abgeschlossen (2026-05-04T23:59):
|
||||
- Nextcloud Listen: Persönlich (id=14), DLR (id=dlr-1), Einkaufsliste (id=…), Wunschliste (id=…)
|
||||
- Nextcloud Kalender: Persönlich, DLR, Einkaufsliste, Wunschliste
|
||||
- Mail-Konto: fkrebs@nucli.de (Posteingang, Archiv, Junk, Waiting)
|
||||
- Qdrant Collection: nexa_knowledge_text (1024 dims, Cosine, 0 Punkte)
|
||||
- LiteLLM-Modelle: nexa-chat, nexa-embed
|
||||
```
|
||||
|
||||
### #nexa:status
|
||||
- **Beschreibung**: Zeigt den aktuellen System-Status
|
||||
- **Antwort**: Uptime, verbundene Systeme, Memory-Stats
|
||||
|
||||
```
|
||||
#nexa:status
|
||||
```
|
||||
|
||||
### #nexa:sync-obsidian
|
||||
- **Beschreibung**: Triggert sofortige Synchronisierung mit Obsidian Vault
|
||||
- **Optional Parameter**: `--vault=/path/to/vault` oder `--force` für Neuindexierung
|
||||
|
||||
```
|
||||
#nexa:sync-obsidian --force
|
||||
```
|
||||
|
||||
## 📊 Memory & RAG Kommandos
|
||||
|
||||
### #nexa:ask [question]
|
||||
- **Beschreibung**: Semantische Suche in Qdrant (RAG) basierend auf Obsidian + Memos
|
||||
- **Antwort**: Top-3 Treffer mit Quellen
|
||||
|
||||
```
|
||||
#nexa:ask Wie implementierten wir das JWT-Middleware-Pattern?
|
||||
```
|
||||
|
||||
### #nexa:learn [topic]
|
||||
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
|
||||
|
||||
```
|
||||
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
|
||||
```
|
||||
|
||||
### #nexa:forget [query-or-iri]
|
||||
- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
|
||||
- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
|
||||
|
||||
### #nexa:retain [source_type] [days]
|
||||
- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
|
||||
### #nexa:ask [question] [--web]
|
||||
- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
|
||||
|
||||
## 🎯 Workflow-Kommandos
|
||||
|
||||
### #nexa:route-test [text]
|
||||
- **Beschreibung**: Testet die Klassifizierung (Work/Personal) ohne Task zu erstellen
|
||||
|
||||
```
|
||||
#nexa:route-test Muss morgen die Präsentation für den Client fertigstellen
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```
|
||||
💬 Klassifizierung (Test-Mode):
|
||||
- Kontext: WORK
|
||||
- Vertrauen: 0.95
|
||||
- Begründung: "Client-Präsentation → professioneller Kontext"
|
||||
```
|
||||
|
||||
### #nexa:email-digest
|
||||
- **Beschreibung**: Triggert sofortige Email-Zusammenfassung (sonst tägl. 7 Uhr)
|
||||
|
||||
```
|
||||
#nexa:email-digest
|
||||
```
|
||||
|
||||
### #nexa:digest
|
||||
- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
|
||||
1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
|
||||
2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
|
||||
3. Drift-Report aus Phase 6 (sobald aktiv).
|
||||
4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
|
||||
|
||||
#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
|
||||
Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
|
||||
|
||||
| Mal | Wartezeit bis zur nächsten Frage |
|
||||
|-----|----------------------------------|
|
||||
| 1 → 2 | 3 Tage |
|
||||
| 2 → 3 | 7 Tage |
|
||||
| 3 → 4 | 21 Tage |
|
||||
| ≥ 5 | 60 Tage |
|
||||
|
||||
Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
|
||||
- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
|
||||
- `nextAskAt <= today`.
|
||||
- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
|
||||
|
||||
Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
|
||||
|
||||
### #nexa:remind <question-id>
|
||||
- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
|
||||
|
||||
### #nexa:answered <question-id> <text>
|
||||
- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
|
||||
|
||||
## 🔐 Admin-Kommandos
|
||||
|
||||
### #nexa:reset-config
|
||||
- **Beschreibung**: Löscht gespeicherte Konfiguration, triggert neuen Autodiscovery
|
||||
- **Warnung**: Alle benutzerdefinierten Settings werden vergessen
|
||||
|
||||
```
|
||||
#nexa:reset-config
|
||||
```
|
||||
|
||||
### #nexa:export-state
|
||||
- **Beschreibung**: Exportiert aktuellen State als JSON (für Backups)
|
||||
|
||||
```
|
||||
#nexa:export-state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Implementierung in n8n
|
||||
|
||||
Ein **Command Parser** läuft immer mit:
|
||||
|
||||
1. **Memos Webhook** empfängt alle Memos
|
||||
2. **Regex Check**: Sucht nach `#nexa:command`
|
||||
3. **Router**: Versendet an entsprechenden n8n-Workflow
|
||||
4. **Antwort**: Postet Reply als Kommentar/Edit
|
||||
|
||||
**Command Parser Regex:**
|
||||
```
|
||||
^#nexa:(\w+)(?:\s+([^\n]*?))?(?:$|\s*--)
|
||||
```
|
||||
|
||||
Extrahiert: `[command, parameters]`
|
||||
|
||||
---
|
||||
|
||||
## 🎛️ Dynamische Konfigurationsspeicherung
|
||||
|
||||
Statt .env zu editieren:
|
||||
|
||||
1. **First Run**: `#nexa:config` speichert zu lokalen Metadata
|
||||
2. **Speich-Ziel**:
|
||||
- **Primary**: Qdrant Metadata (als `_config` Namespace)
|
||||
- **Fallback**: `nexa-core/config/runtime_config.json` (gitignored)
|
||||
- **Notfall**: .env (nur initial)
|
||||
|
||||
3. **Reload-Logik**: Bei jedem Workflow-Start werden Settings aus Qdrant geladen
|
||||
|
||||
Das macht Nexa vollständig selbstständig nach dem initialem Setup!
|
||||
@@ -1,4 +1,4 @@
|
||||
# Nexa Klassifizierungs-Logik
|
||||
# 06 — Klassifizierungs-Logik
|
||||
|
||||
Dieser Fragebogen dient der Feinabstimmung von SAIA, um Tasks korrekt zu routen.
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# 07 — Workflow Spec: Phase-2 Task-Router
|
||||
|
||||
## 1. Trigger
|
||||
* **Typ:** Webhook (Memos)
|
||||
* **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`.
|
||||
|
||||
## 2. Intelligence Node (SAIA)
|
||||
* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: `{ "context": "work" | "personal" | "shopping" | "wishes", "summary": "string", "urgency": 1-5 }`."
|
||||
* **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.
|
||||
|
||||
## 3. List Resolver (vor dem Switch)
|
||||
* Lese den Cache `nexa._config.lists` aus Qdrant: `{ "Persönlich": <id>, "DLR": <id>, "Einkaufsliste": <id>, "Wunschliste": <id>, discovered_at: <ts> }`.
|
||||
* Bei Cache-Miss oder 404 vom nachfolgenden Create-Task → triggere `#nexa:config` (Discovery), update den Cache, retry **einmal**.
|
||||
|
||||
## 4. Switch Node (4-Wege)
|
||||
* **Route 1:** `context == "work"` → Nextcloud Create Task auf Liste `DLR`.
|
||||
* **Route 2:** `context == "personal"` → Nextcloud Create Task auf Liste `Persönlich`.
|
||||
* **Route 3:** `context == "shopping"` → Nextcloud Create Task auf Liste `Einkaufsliste`.
|
||||
* **Route 4:** `context == "wishes"` → Nextcloud Create Task auf Liste `Wunschliste`.
|
||||
* **Default (Konfidenz < 0.7):** `Persönlich` (siehe [06](./06-classification-logic.md)).
|
||||
|
||||
## 5. Feedback Loop
|
||||
* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
|
||||
|
||||
## 6. Listen-Diskoverability
|
||||
Listennamen sind nicht hardgecodet — die Resolver-Step liest sie aus dem Runtime-Config-Cache. Wenn der User in Nextcloud eine neue Liste anlegt (z. B. `Reisen`), erscheint sie nach dem nächsten geplanten `#nexa:config`-Lauf (täglich) automatisch als Routing-Ziel — die System-Prompt der Intelligence Node wird zusammen mit den verfügbaren Listen versorgt, sodass SAIA neue Kontexte vorschlagen kann.
|
||||
@@ -0,0 +1,286 @@
|
||||
# 08 — GraphRAG: Structural Knowledge & Relations
|
||||
|
||||
Decision: graph layer = **Ontotext GraphDB** with **SPARQL** (resolved in [11/Q1](./11-open-questions.md)).
|
||||
Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.
|
||||
|
||||
---
|
||||
|
||||
## Two-pillar memory
|
||||
|
||||
| Pillar | Question it answers | Backed by |
|
||||
|--------|---------------------|-----------|
|
||||
| **Qdrant** (vectors) | *"What is similar / relevant?"* | Cosine search over embeddings |
|
||||
| **GraphDB** (RDF) | *"What is connected? What depends on what? Who is involved?"* | SPARQL over a typed graph |
|
||||
|
||||
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer.
|
||||
|
||||
---
|
||||
|
||||
## RDF schema
|
||||
|
||||
Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.
|
||||
|
||||
```turtle
|
||||
@prefix nexa: <https://nuclide.systems/nexa/ontology#> .
|
||||
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||||
@prefix prov: <http://www.w3.org/ns/prov#> .
|
||||
|
||||
# Classes
|
||||
nexa:Project a rdfs:Class .
|
||||
nexa:Task a rdfs:Class .
|
||||
nexa:Person a rdfs:Class .
|
||||
nexa:Technology a rdfs:Class .
|
||||
nexa:Topic a rdfs:Class .
|
||||
nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests
|
||||
nexa:File a rdfs:Class .
|
||||
|
||||
# Properties
|
||||
nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .
|
||||
nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .
|
||||
nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .
|
||||
nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .
|
||||
nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .
|
||||
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
|
||||
|
||||
# Datatype properties
|
||||
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
|
||||
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
|
||||
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5
|
||||
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
|
||||
|
||||
# Cross-pillar / multimodality
|
||||
nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image"
|
||||
nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://…
|
||||
nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual"
|
||||
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID
|
||||
nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them
|
||||
```
|
||||
|
||||
`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way.
|
||||
|
||||
`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases.
|
||||
|
||||
---
|
||||
|
||||
## Sync flows
|
||||
|
||||
### 1. Memos → GraphDB (real-time)
|
||||
|
||||
```
|
||||
Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren"
|
||||
│
|
||||
▼ SAIA extracts entities + relations as JSON
|
||||
│ { tasks: [{title, urgency}], technologies: [...],
|
||||
│ relations: [{type:"uses", from:..., to:...}] }
|
||||
│
|
||||
▼ n8n turns JSON into a SPARQL UPDATE
|
||||
│
|
||||
└──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge"
|
||||
```
|
||||
|
||||
### 2. Obsidian → GraphDB (`#nexa:sync-obsidian`)
|
||||
|
||||
For each Obsidian note: parse front-matter + headings → emit `nexa:Project`, `nexa:Task`, `nexa:Note` triples; `nexa:mentions` for `[[wikilinks]]`.
|
||||
|
||||
### 3. Nextcloud Tasks ↔ GraphDB (bidirectional)
|
||||
|
||||
n8n trigger on Nextcloud CalDAV/Tasks change → `INSERT/DELETE DATA` to keep `nexa:status` and `nexa:scheduledFor` in sync.
|
||||
|
||||
---
|
||||
|
||||
## Example SPARQL queries
|
||||
|
||||
### Q1 — All open tasks involving JWT, by urgency
|
||||
|
||||
```sparql
|
||||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||||
|
||||
SELECT ?taskTitle ?urgency ?projectName
|
||||
WHERE {
|
||||
?tech rdfs:label "JWT" .
|
||||
?task nexa:uses ?tech ;
|
||||
rdfs:label ?taskTitle ;
|
||||
nexa:status ?status ;
|
||||
nexa:urgency ?urgency .
|
||||
FILTER (?status IN ("needs-action", "in-progress"))
|
||||
OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }
|
||||
}
|
||||
ORDER BY DESC(?urgency)
|
||||
```
|
||||
|
||||
### Q2 — What does Auth-Service transitively depend on?
|
||||
|
||||
```sparql
|
||||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||||
|
||||
SELECT DISTINCT ?dep ?label
|
||||
WHERE {
|
||||
?root rdfs:label "Auth-Service" .
|
||||
?root nexa:dependsOn+ ?dep .
|
||||
?dep rdfs:label ?label .
|
||||
}
|
||||
```
|
||||
|
||||
(`+` is SPARQL property-paths — transitive closure, free.)
|
||||
|
||||
### Q3 — Topics with the most note-mentions in the last day
|
||||
|
||||
```sparql
|
||||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||||
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
|
||||
|
||||
SELECT ?topic (COUNT(?note) AS ?n)
|
||||
WHERE {
|
||||
?note a nexa:Note ;
|
||||
prov:generatedAtTime ?ts ;
|
||||
nexa:mentions ?topic .
|
||||
FILTER (?ts > NOW() - "P1D"^^xsd:duration)
|
||||
}
|
||||
GROUP BY ?topic
|
||||
ORDER BY DESC(?n)
|
||||
LIMIT 10
|
||||
```
|
||||
|
||||
### Q4 — Cross-pillar: "find vectors for tasks blocking project X"
|
||||
|
||||
```sparql
|
||||
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
|
||||
|
||||
SELECT ?taskTitle ?vectorId
|
||||
WHERE {
|
||||
?proj rdfs:label "Nexa" .
|
||||
?task nexa:childOf ?proj ;
|
||||
nexa:status "in-progress" ;
|
||||
nexa:vectorId ?vectorId ;
|
||||
rdfs:label ?taskTitle .
|
||||
}
|
||||
```
|
||||
|
||||
n8n then takes each `?vectorId`, fetches the embedding from Qdrant, and runs a "more like this" search for richer context.
|
||||
|
||||
---
|
||||
|
||||
## GraphRAG answer pipeline (`#nexa:ask`)
|
||||
|
||||
```
|
||||
#nexa:ask <question>
|
||||
│
|
||||
┌──────┴──────┐
|
||||
▼ ▼
|
||||
[Qdrant] [GraphDB]
|
||||
semantic structural
|
||||
top-k SPARQL — auto-generated
|
||||
notes paths / dependencies
|
||||
│ │
|
||||
└──────┬──────┘
|
||||
▼
|
||||
merge + rank
|
||||
│
|
||||
▼
|
||||
SAIA prompt:
|
||||
"Given these passages and these relations, answer …"
|
||||
│
|
||||
▼
|
||||
comment under the original memo
|
||||
```
|
||||
|
||||
Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a `SELECT`/`CONSTRUCT` query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.
|
||||
|
||||
---
|
||||
|
||||
## n8n integration sketch
|
||||
|
||||
### Workflow: Graph-Sync Trigger
|
||||
|
||||
```
|
||||
[Memos Webhook]
|
||||
│
|
||||
[Parse Content]
|
||||
│
|
||||
[SAIA: Extract entities + relations as JSON]
|
||||
│
|
||||
[Build SPARQL UPDATE INSERT DATA { ... }]
|
||||
│
|
||||
[HTTP POST → /repositories/nexa_knowledge/statements]
|
||||
│
|
||||
[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]
|
||||
```
|
||||
|
||||
### Workflow: Question Router
|
||||
|
||||
```
|
||||
[#nexa:ask Query]
|
||||
│
|
||||
┌─┴────────────────┐
|
||||
▼ ▼
|
||||
[SAIA: NL → SPARQL] [Qdrant: kNN]
|
||||
│ │
|
||||
[POST → SPARQL endpoint]
|
||||
│ │
|
||||
└──────┬───────────┘
|
||||
▼
|
||||
rank + merge → SAIA answer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Graph-management commands
|
||||
|
||||
### `#nexa:graph-status`
|
||||
|
||||
Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL `SELECT (COUNT)`.
|
||||
|
||||
### `#nexa:graph-trace [entity]`
|
||||
|
||||
Returns the 1-hop (and optionally 2-hop) neighbourhood — a `DESCRIBE <iri>` plus a templated outgoing/incoming query.
|
||||
|
||||
### `#nexa:graph-rebuild`
|
||||
|
||||
Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https://nuclide.systems/nexa/runtime>` followed by the import workflow.
|
||||
|
||||
---
|
||||
|
||||
## Why two stores
|
||||
|
||||
| Scenario | Qdrant | GraphDB | Best |
|
||||
|----------|--------|---------|------|
|
||||
| "Which note was similar to this one?" | ✅ | ❌ | Qdrant |
|
||||
| "What blocks this task?" | ❌ | ✅ | GraphDB |
|
||||
| "Explain this project" | ✅ context | ✅ structure | both |
|
||||
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
|
||||
|
||||
Combined: **complete understanding** rather than a search index *or* a structure index.
|
||||
|
||||
---
|
||||
|
||||
## Memory sources & retention
|
||||
|
||||
Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
|
||||
|
||||
| `source_type` | Where it comes from | Default TTL | Rationale |
|
||||
|---------------|--------------------|-------------|-----------|
|
||||
| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
|
||||
| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
|
||||
| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
|
||||
| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
|
||||
| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
|
||||
| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
|
||||
| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
|
||||
| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
|
||||
| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
|
||||
|
||||
**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
|
||||
|
||||
### Web search loop
|
||||
|
||||
`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
|
||||
|
||||
This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
|
||||
|
||||
### Manual overrides
|
||||
|
||||
- `#nexa:learn <text> --permanent` overrides the default TTL.
|
||||
- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
|
||||
- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).
|
||||
@@ -0,0 +1,326 @@
|
||||
# 09 — Deployment
|
||||
|
||||
Pragmatic deployment guide that **assumes the existing homelab** and adds only what's missing.
|
||||
|
||||
## What's already running (no action required)
|
||||
|
||||
Surveyed from Homepage / Dozzle / Proxmox / Zoraxy:
|
||||
|
||||
| Service | Host / port | URL |
|
||||
|---------|-------------|-----|
|
||||
| Memos | docker LXC 104 → `:5230` | `https://memos.nuclide.systems` |
|
||||
| n8n | docker LXC 104 → `:5678` | `https://n8n.nuclide.systems` |
|
||||
| LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) |
|
||||
| Nextcloud | LXC 105 | `https://nc.nuclide.systems` |
|
||||
| ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` |
|
||||
| Karakeep | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` (legacy host alias kept for compatibility) |
|
||||
| Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` |
|
||||
| Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` |
|
||||
| Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` |
|
||||
| Backrest | LXC 103 | (internal) |
|
||||
| AdGuard DNS | LXC 102 | (internal) |
|
||||
| Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems |
|
||||
| `qdrant_scientific` (existing) | docker LXC 104 | **reused** — Nexa uses `nexa_*` collections in this instance |
|
||||
|
||||
The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**.
|
||||
|
||||
## What's missing for Nexa
|
||||
|
||||
1. **Qdrant collection** for Nexa (`nexa_knowledge`) inside the existing `qdrant_scientific` instance — vector dim follows Q15 (1024 for `bge-m3`, 768 for `nomic-embed-text`).
|
||||
2. **TEI** (HF text-embeddings-inference) on the docker host for self-hosted embeddings (LiteLLM key is **not** authorised for OpenAI embeddings — see [11/Q3+Q15](./11-open-questions.md)). Lighter than Ollama: single Rust binary, ~500 MB image, no LLM runtime.
|
||||
3. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n.
|
||||
4. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`).
|
||||
5. **Memos webhook → n8n** wired through the Memos config.
|
||||
6. **LiteLLM virtual key** for the `nexa` user with chat-only access (no embeddings — handled by Ollama).
|
||||
7. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied.
|
||||
8. **(Phase 3.4) Ontotext GraphDB** for the SPARQL pillar — see add-on at the bottom of this doc.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Secrets
|
||||
|
||||
Copy `nexa-core/.env.example` → `nexa-core/.env` and fill **only** the secrets:
|
||||
|
||||
```bash
|
||||
cd nexa-core
|
||||
cp .env.example .env
|
||||
$EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY
|
||||
```
|
||||
|
||||
The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead.
|
||||
|
||||
## Step 2 — Qdrant collection (`nexa_knowledge_text`)
|
||||
|
||||
Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring.
|
||||
|
||||
```bash
|
||||
# adjust QDRANT_HOST in .env first
|
||||
source nexa-core/.env
|
||||
|
||||
# create the text collection from the schema file
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema.json
|
||||
```
|
||||
|
||||
The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)):
|
||||
|
||||
| Field | Why it's there now |
|
||||
|-------|--------------------|
|
||||
| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. |
|
||||
| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. |
|
||||
| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. |
|
||||
| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. |
|
||||
| `content_hash` | de-dup. |
|
||||
| `context` | `work` / `personal`. |
|
||||
|
||||
> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container.
|
||||
> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
|
||||
|
||||
### Image attachments today (queue them)
|
||||
|
||||
Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later:
|
||||
|
||||
- Memo with an image → text body still goes into `nexa_knowledge_text`.
|
||||
- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`).
|
||||
- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection.
|
||||
|
||||
This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself.
|
||||
|
||||
## Step 3 — Self-hosted embeddings (TEI)
|
||||
|
||||
Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.
|
||||
|
||||
The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle.
|
||||
|
||||
```yaml
|
||||
# Nexa stack — paste into Arcane.
|
||||
# Storage convention matches the rest of the homelab (verified against
|
||||
# the running Karakeep stack, Q19): host bind-mount of
|
||||
# /mnt/pve/unas/services/<svc>/<vol>. No volume-driver, no CIFS, no
|
||||
# credentials in the compose — the LXC's NFS mount is already there.
|
||||
services:
|
||||
nexa-embed:
|
||||
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
|
||||
container_name: nexa-embed
|
||||
restart: unless-stopped
|
||||
command: ["--model-id", "BAAI/bge-m3"]
|
||||
ports:
|
||||
- "127.0.0.1:8080:80"
|
||||
volumes:
|
||||
- /mnt/pve/unas/services/nexa/tei-cache:/data
|
||||
env_file:
|
||||
- .env
|
||||
|
||||
networks: {}
|
||||
```
|
||||
|
||||
Pre-deploy step on the docker LXC (one-time):
|
||||
|
||||
```bash
|
||||
mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb}
|
||||
mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}
|
||||
```
|
||||
|
||||
Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into `/mnt/pve/unas/services/nexa/tei-cache/`; subsequent restarts are instant.
|
||||
|
||||
Secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, `QDRANT_API_KEY`, `NC_APP_PASSWORD`) go in the stack's `.env` next to the compose — same pattern Karakeep uses (`env_file: .env`). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term ([12/#11](./12-optimization-opportunities.md#11)) but isn't required for the first cut.
|
||||
|
||||
> **Why bind-mount and not SMB?** Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's "had it with Nextcloud" experience. The Karakeep stack confirms the actual convention is the simpler one: **host bind-mount of the LXC's existing `/mnt/pve/unas` NFS mount**. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to `www-data`, which fails against `root_squash` exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in [12/#27](./12-optimization-opportunities.md#27) only as an escape hatch *if* a future service hits Nextcloud-style issues — Nexa doesn't, so we don't use it.
|
||||
|
||||
Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter:
|
||||
|
||||
- model name: `nexa-embed`
|
||||
- provider: `openai`
|
||||
- model: `bge-m3`
|
||||
- api_base: `http://nexa-embed:80/v1`
|
||||
- api_key: any non-empty string (TEI ignores it)
|
||||
|
||||
Now n8n only ever talks to LiteLLM and the model is swappable without touching workflows.
|
||||
|
||||
## Step 4 — LiteLLM virtual key
|
||||
|
||||
In the LiteLLM admin UI (`ai.nuclide.systems`):
|
||||
|
||||
1. Create user `nexa`.
|
||||
2. Issue a virtual key with access to:
|
||||
- one chat model (already-available model from your SAIA gateway).
|
||||
- the `nexa-embed` model from Step 3.
|
||||
3. Paste the key into `SAIA_API_KEY` in `.env`.
|
||||
|
||||
## Step 4 — n8n workflows
|
||||
|
||||
Import the JSON exports — credentials are filled inside n8n, not in the JSON:
|
||||
|
||||
```bash
|
||||
# n8n personal access token from the n8n UI: Settings → API
|
||||
N8N_URL=https://n8n.nuclide.systems
|
||||
N8N_TOKEN=... # from the n8n UI
|
||||
|
||||
for f in nexa-core/n8n-workflows/phase-1/*.json \
|
||||
nexa-core/n8n-workflows/phase-2/*.json; do
|
||||
curl -X POST "$N8N_URL/api/v1/workflows" \
|
||||
-H "X-N8N-API-KEY: $N8N_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data-binary "@$f"
|
||||
done
|
||||
```
|
||||
|
||||
Inside n8n, attach credentials to the imported nodes:
|
||||
|
||||
- **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY`
|
||||
- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` (chat + `nexa-embed`)
|
||||
- **Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under `Notizen/`)** → one app password (`$NC_APP_PASSWORD`), reused across all three node types
|
||||
- **Qdrant** → header `api-key: $QDRANT_API_KEY`
|
||||
|
||||
Activate each workflow individually after smoke-test.
|
||||
|
||||
## Step 5 — Memos webhook
|
||||
|
||||
In the Memos admin UI, set the webhook URL to the production address of the discovery workflow:
|
||||
|
||||
```
|
||||
https://n8n.nuclide.systems/webhook/memos
|
||||
```
|
||||
|
||||
The same URL is the one the workflow exposes; verify with:
|
||||
|
||||
```bash
|
||||
curl -i https://n8n.nuclide.systems/webhook/memos
|
||||
# expect 200 / 405, never 404
|
||||
```
|
||||
|
||||
## Step 6 — Bootstrap commands via Memos
|
||||
|
||||
Create a memo with body `#nexa:config` — the discovery workflow:
|
||||
|
||||
1. Lists Nextcloud Tasks lists, looking for `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (and any new lists added later — see [05](./05-command-system.md)). Caches `name → id`.
|
||||
2. Lists Nextcloud Calendars by the same names; caches IDs.
|
||||
3. Verifies the Nextcloud Mail account `fkrebs@nucli.de` and the folder list (`Posteingang`, `Archiv`, `Junk`, `Waiting`).
|
||||
4. Counts existing Qdrant points in `nexa_knowledge_text`.
|
||||
5. Verifies LiteLLM reachability + lists available models for the Nexa virtual key.
|
||||
6. Replies as a comment with a **runtime-config snapshot** that's stored in the Qdrant `_config` namespace (and mirrored as `nexa-core/config/runtime_config.json`, gitignored).
|
||||
|
||||
The same workflow is also triggered by:
|
||||
- A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention).
|
||||
- Cache-miss in the task-router — if a list ID 404s, the router fires `#nexa:config` once and retries.
|
||||
|
||||
After this point, `.env` is read-once. Subsequent runs read config from Qdrant.
|
||||
|
||||
## Step 7 — Smoke tests
|
||||
|
||||
```bash
|
||||
# (1) Memos round-trip — should produce a comment within ~5 s
|
||||
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
||||
-H "Authorization: Bearer $MEMOS_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"content":"- [ ] testing the router #nexa"}'
|
||||
|
||||
# (2) Classification dry-run
|
||||
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
||||
-H "Authorization: Bearer $MEMOS_API_KEY" \
|
||||
-d '{"content":"#nexa:route-test buy milk"}'
|
||||
|
||||
# (3) RAG test (requires at least one indexed memo/note)
|
||||
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
||||
-H "Authorization: Bearer $MEMOS_API_KEY" \
|
||||
-d '{"content":"#nexa:ask what is the goal of nexa?"}'
|
||||
```
|
||||
|
||||
## Step 8 — Reverse proxy
|
||||
|
||||
Already done — Zoraxy at `192.168.1.4:8000` terminates TLS for `*.nuclide.systems` and forwards to docker LXC 104 (`192.168.1.40`). **No new entry is required for Nexa**: every service Nexa talks to already has a host entry.
|
||||
|
||||
## Step 9 — Backups
|
||||
|
||||
Already covered by Backrest (LXC 103). Add:
|
||||
|
||||
- **n8n workflows** → `nexa-core/scripts/backup_workflows.sh` (already present) into a Backrest schedule.
|
||||
- **Qdrant snapshots** → schedule a daily `POST /collections/nexa_knowledge/snapshots` and rsync to S3 (`s3.nuclide.systems`). Add as a Backrest pre-hook on the docker host.
|
||||
|
||||
For deeper detail: [10 — Operations](./10-operations.md).
|
||||
|
||||
---
|
||||
|
||||
## Phase add-on: Ontotext GraphDB (Phase 3.4)
|
||||
|
||||
Defer until 3.1–3.3 ship.
|
||||
|
||||
```yaml
|
||||
# nexa-core/docker-compose.graph.yml
|
||||
services:
|
||||
graphdb:
|
||||
image: ontotext/graphdb:10.7.0
|
||||
container_name: nexa-graphdb
|
||||
ports: ["127.0.0.1:7200:7200"]
|
||||
environment:
|
||||
GDB_JAVA_OPTS: "-Xmx4g -Xms1g"
|
||||
volumes:
|
||||
- ./data/graphdb:/opt/graphdb/home
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
After first start, create the repository (one-time):
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7200/rest/repositories \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"id": "nexa_knowledge",
|
||||
"title": "Nexa Knowledge Graph",
|
||||
"type": "graphdb",
|
||||
"params": {
|
||||
"ruleset": {"value": "rdfsplus-optimized"},
|
||||
"baseURL": {"value": "https://nuclide.systems/nexa/"}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Optional Zoraxy entry `graph.nuclide.systems` → `192.168.1.40:7200` if you want the SPARQL Workbench in a browser; otherwise n8n talks to it on the docker network at `http://nexa-graphdb:7200`.
|
||||
|
||||
For schema and example queries: [08-graphrag-architecture](./08-graphrag-architecture.md).
|
||||
|
||||
---
|
||||
|
||||
## Phase add-on: visual collection (Phase 3.2)
|
||||
|
||||
Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`.
|
||||
|
||||
```bash
|
||||
# (1) replace TEI with infinity (or run alongside) for CLIP-family support
|
||||
docker rm -f nexa-embed
|
||||
docker run -d --name nexa-embed \
|
||||
--restart unless-stopped \
|
||||
-p 127.0.0.1:8080:80 \
|
||||
-v infinity-data:/app/.cache \
|
||||
michaelf34/infinity:latest \
|
||||
v2 \
|
||||
--model-id BAAI/bge-m3 \
|
||||
--model-id jinaai/jina-clip-v2 \
|
||||
--port 80
|
||||
|
||||
# (2) create the visual collection
|
||||
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: $QDRANT_API_KEY" \
|
||||
-d @nexa-core/config/qdrant_schema_visual.json
|
||||
|
||||
# (3) register the second model in LiteLLM as `nexa-embed-visual`
|
||||
# (same OpenAI-compatible route, different model id)
|
||||
|
||||
# (4) backfill queued images:
|
||||
# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }
|
||||
# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,
|
||||
# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.
|
||||
```
|
||||
|
||||
n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt.
|
||||
|
||||
---
|
||||
|
||||
## Step-back / rollback
|
||||
|
||||
- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops.
|
||||
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`.
|
||||
- **Re-discover** — `#nexa:reset-config` then `#nexa:config`.
|
||||
@@ -0,0 +1,101 @@
|
||||
# 10 — Operations
|
||||
|
||||
Day-2 concerns. Backups, monitoring, troubleshooting.
|
||||
|
||||
## Backups
|
||||
|
||||
**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)):
|
||||
1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable).
|
||||
2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103).
|
||||
3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext.
|
||||
|
||||
Nexa-specific items inside this pipeline:
|
||||
|
||||
| What | How | Frequency |
|
||||
|------|-----|-----------|
|
||||
| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
|
||||
| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) |
|
||||
| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily |
|
||||
| Memos DB | Backrest snapshot of Memos data dir | already covered |
|
||||
| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered |
|
||||
| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
|
||||
|
||||
## Monitoring
|
||||
|
||||
| Signal | Source | Sink |
|
||||
|--------|--------|------|
|
||||
| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. **Skip names matching `octoprint*`** — that container is intentionally powered down most of the time. | Memos system feed via ntfy |
|
||||
| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic |
|
||||
| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest |
|
||||
| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest |
|
||||
| Backrest run status | Backrest webhook | ntfy |
|
||||
|
||||
A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Memos webhook not firing
|
||||
|
||||
```bash
|
||||
# is the Memos webhook config still pointing at n8n?
|
||||
curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \
|
||||
https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'
|
||||
|
||||
# does n8n still expose the path?
|
||||
curl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404
|
||||
```
|
||||
|
||||
If 404 → workflow is inactive in n8n. Activate.
|
||||
|
||||
### LiteLLM 401 / 429
|
||||
|
||||
- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential.
|
||||
- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.
|
||||
|
||||
### Qdrant `nexa_knowledge` empty
|
||||
|
||||
```bash
|
||||
# is the indexing workflow active?
|
||||
docker logs nexa-n8n 2>&1 | grep memos_bridge | tail
|
||||
|
||||
# manual index test
|
||||
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
||||
-H "Authorization: Bearer $MEMOS_API_KEY" \
|
||||
-d '{"content":"manual probe #nexa"}'
|
||||
|
||||
# point count
|
||||
curl -s -H "api-key: $QDRANT_API_KEY" \
|
||||
$QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'
|
||||
```
|
||||
|
||||
### Wrong list routing (Work vs Personal)
|
||||
|
||||
1. Run `#nexa:route-test <text>` and check the confidence value.
|
||||
2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)).
|
||||
3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change.
|
||||
|
||||
### "Where did Nexa store this?"
|
||||
|
||||
```bash
|
||||
# state dump
|
||||
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
||||
-d '{"content":"#nexa:export-state"}'
|
||||
# returns runtime_config + counters as a JSON memo
|
||||
```
|
||||
|
||||
## Log locations
|
||||
|
||||
| Component | Where |
|
||||
|-----------|-------|
|
||||
| Memos | `docker logs nexa-memos` (LXC 104) |
|
||||
| n8n | `docker logs nexa-n8n` |
|
||||
| LiteLLM | `docker logs litellm` |
|
||||
| Qdrant | `docker logs qdrant_scientific` |
|
||||
| Zoraxy | LXC 108 web UI → Statistical Analysis |
|
||||
| Backrest | LXC 103 web UI |
|
||||
|
||||
For a one-shot dump:
|
||||
|
||||
```bash
|
||||
ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# 11 — Open Questions (user-info-required)
|
||||
|
||||
Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide.
|
||||
|
||||
## Resolved
|
||||
|
||||
- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly.
|
||||
- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container.
|
||||
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below.
|
||||
- [x] **Q5 — Display name → Karakeep.** The Zoraxy host alias `hoarder.nuclide.systems` is legacy — keep it for compatibility, but all docs, prompts and new workflow nodes use "Karakeep".
|
||||
- [x] **Q13 — Octoprint container is *intentionally* temporary.** Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching `octoprint*` (or any container tagged `proxmox-he 3d-printing`).
|
||||
- [x] **Q14 — Homepage Zoraxy widget → no such widget.** Config error in `homepage/services.yaml`; cosmetic, not Nexa-related.
|
||||
- [x] **Q15 — Embeddings staging plan → A now, C prepared.**
|
||||
- **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus.
|
||||
- **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16).
|
||||
- All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json).
|
||||
- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme.
|
||||
- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich.
|
||||
- [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant).
|
||||
- [x] **Q19 — Storage convention is plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via `env_file: .env` next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn ([12/#27](./12-optimization-opportunities.md#27)) — it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to `www-data` and fails on `root_squash`-style exports; that doesn't apply to normal containers). Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
|
||||
- [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files/<user>/Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index):
|
||||
- `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache.
|
||||
- `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB).
|
||||
- `.caldav-sync/` — calendar sync, not notes.
|
||||
- `assets/` — 186 MB of binaries; routed through the Phase-3.2 visual queue (`nexa:pendingVisualIndex`), not the text path.
|
||||
- `Templates/` — empty templates, low semantic value.
|
||||
- `BMO/`, `Excalidraw/` — plugin folders.
|
||||
Anything else under `Notizen/**/*.md` is fair game.
|
||||
- [x] **Q6 + Q7 — Lists & calendars discovered by name, not ID. Self-healing.**
|
||||
Confirmed names from screenshots:
|
||||
- **Tasks**: `Persönlich` (14), `DLR` (13), `Einkaufsliste` (13), `Wunschliste` (9).
|
||||
- **Calendars**: `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name).
|
||||
- Mapping in classification: `DLR` = Work-Kontext, `Persönlich` = Personal-Kontext, `Einkaufsliste` = Shopping, `Wunschliste` = Wishes.
|
||||
|
||||
**Self-healing requirement (the user explicitly noted lists may change/grow):** Nexa must not cache IDs forever. The `#nexa:config` workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or "not found". The runtime-config record in Qdrant's `_config` namespace stores `{name → id, discovered_at}` and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring `#einkaufsliste` / `#dlr` etc. without code changes.
|
||||
|
||||
- [x] **Q8 — Mail via Nextcloud, single account `fkrebs@nucli.de`.** No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: `Posteingang` (default), `Archiv`, `Junk` (61 — auto-filtered, ignored by Nexa), `Papierkorb`, `Waiting`. The `Waiting` folder is a useful manual signal — items moved there by the user are skipped from digest (treat as "in flight").
|
||||
|
||||
- [x] **Q9 — Pocket-ID SSO is configured everywhere.** No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n → Memos webhook, n8n → Nextcloud, n8n → Qdrant, n8n → LiteLLM) keep using API keys / app passwords — SSO is for human UIs only. Don't add basic-auth or per-app login screens.
|
||||
|
||||
This **resolves Q10 too** (the proposed "Pocket-ID SSO in front of n8n / Memos" is already done; Nexa just inherits it).
|
||||
|
||||
- [x] **Q10 — folded into Q9.**
|
||||
|
||||
## Hardware / capacity
|
||||
|
||||
- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.**
|
||||
- **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
|
||||
- **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
|
||||
- **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
|
||||
- **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27).
|
||||
- **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
|
||||
## Backups (deferred)
|
||||
|
||||
Backup-tier decisions are intentionally parked while Phase-3.1/3.2/3.4 are built. UNAS RAID 6 + Backrest already cover file-level recovery; pool-snapshot configuration ([12/#38](./12-optimization-opportunities.md#38)) remains the single highest-leverage data-protection action and doesn't depend on either question below. Re-open both when [Phase 3.3](./02-roadmap.md#phase-3--memory-qdrant--rag) becomes the next-up item.
|
||||
|
||||
- [ ] **Q12 — *deferred.*** S3 archive bucket on `s3.nuclide.systems` (warm tier). Pick when Phase 3.3 is up next.
|
||||
- [ ] **Q20 — *deferred.*** Off-site cold-tier provider (Jottacloud is the leading candidate). Pick when Phase 3.3 is up next.
|
||||
|
||||
---
|
||||
|
||||
## Speed budget (Q3 follow-up)
|
||||
|
||||
Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap):
|
||||
|
||||
| Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` |
|
||||
|------|--------|----------------|---------------------------------|------------------------------------|
|
||||
| Real-time memo embed | 1 doc | <500 ms incl. n8n round-trip | ✅ ~50–100 ms | ✅ ~20 ms |
|
||||
| Daily ingest | ~70 docs | <60 s | ✅ ~5–10 s | ✅ ~2 s |
|
||||
| Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~2–4 min | ✅ <1 min |
|
||||
| RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms |
|
||||
|
||||
Conclusion: CPU-only TEI is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to **10 req/min** which would block real-time embed; self-hosted TEI side-steps that completely.
|
||||
@@ -0,0 +1,149 @@
|
||||
# 12 — Optimization Opportunities
|
||||
|
||||
Observations from the running infrastructure. Each item is independent — accept, defer, or reject.
|
||||
|
||||
## For Nexa directly
|
||||
|
||||
1. **Reuse, don't redeploy.** The earlier `DEPLOYMENT.md` would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new [09-deployment](./09-deployment.md) treats these as pre-existing — keeps the config minimal and avoids port collisions.
|
||||
2. **Use LiteLLM virtual keys per logical caller.** Today there's one SAIA key. Issuing one key per workflow (`nexa-router`, `nexa-embed`, `nexa-digest`) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything.
|
||||
3. **Use n8n's *credential* objects, never inline secrets.** The current workflows under `nexa-core/n8n-workflows/phase-1/*.json` should be reviewed — if any header `Authorization` is hardcoded, replace with credential references before importing.
|
||||
4. **Centralise system alerts on a single ntfy topic** (`nexa.system`). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them.
|
||||
5. **Defer graph DB until Phase 3.4.** Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries.
|
||||
6. **Auto-export n8n workflows.** `nexa-core/scripts/backup_workflows.sh` already exists. Schedule it inside the n8n container (cron) and let it `git commit && git push` — this is the cheapest disaster recovery.
|
||||
|
||||
## For the wider homelab (out of scope but worth noting)
|
||||
|
||||
7. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts.
|
||||
8. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding).
|
||||
9. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history.
|
||||
10. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win.
|
||||
11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk.
|
||||
12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.
|
||||
13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded.
|
||||
14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation:
|
||||
- **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
|
||||
- **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A.
|
||||
- **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus.
|
||||
|
||||
Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.
|
||||
|
||||
15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.
|
||||
|
||||
16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).
|
||||
17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
|
||||
18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
|
||||
19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.
|
||||
20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.
|
||||
|
||||
## Proxmox host (NUC 14 Pro) tuning
|
||||
|
||||
Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:
|
||||
|
||||
21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):
|
||||
```
|
||||
echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB
|
||||
update-initramfs -u
|
||||
```
|
||||
Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live.
|
||||
22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
|
||||
23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
|
||||
24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
|
||||
25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
|
||||
26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
|
||||
27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
|
||||
28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
|
||||
29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
|
||||
|
||||
## Docker LXC (104) — observations & wins
|
||||
|
||||
Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure.
|
||||
|
||||
30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
|
||||
```
|
||||
lxc.cgroup2.devices.allow: c 226:0 rwm
|
||||
lxc.cgroup2.devices.allow: c 226:128 rwm
|
||||
lxc.cgroup2.devices.allow: c 29:0 rwm
|
||||
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
|
||||
lxc.idmap: u 0 100000 65536
|
||||
lxc.idmap: g 0 100000 65536
|
||||
lxc.idmap: g 44 44 1 # video group on host
|
||||
lxc.idmap: g 104 104 1 # render group on host
|
||||
```
|
||||
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
|
||||
31. **Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services/<svc>/...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets:
|
||||
- `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir.
|
||||
- `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk).
|
||||
- `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store.
|
||||
- `/mnt/pve/unas/backup/nexa/snapshots/qdrant/<date>/` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup/<svc>/` pattern used by Home Assistant / Immich.
|
||||
- **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
|
||||
- **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
|
||||
|
||||
Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/<vol>:/<container-path>`. Concrete example in [09-deployment §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
|
||||
|
||||
Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets ([12/#11](#11)) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is **withdrawn** ([12/#27](#27)) — it was an over-fitting to the Nextcloud-specific NFS issue.
|
||||
32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
|
||||
33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane.
|
||||
34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
|
||||
|
||||
## Housekeeping campaign (cross-cutting, do together)
|
||||
|
||||
These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.
|
||||
|
||||
35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Arcane: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths:
|
||||
- **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron.
|
||||
- **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.
|
||||
|
||||
Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running.
|
||||
|
||||
36. **UNAS-integration audit.** `/mnt/pve/unas/services/<svc>/` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification):
|
||||
- `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection.
|
||||
- `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state.
|
||||
- `memos` — the user-facing notes app; loss = data loss.
|
||||
- `n8n` — workflows + executions DB.
|
||||
- `audiobookshelf` — listening progress + library metadata.
|
||||
- the `*-mcp` containers — likely stateless (cache only), low priority.
|
||||
|
||||
Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.
|
||||
|
||||
37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.**
|
||||
`s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers:
|
||||
|
||||
**Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):**
|
||||
- **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination.
|
||||
- **Qdrant + GraphDB snapshots** flow UNAS → S3 daily.
|
||||
- Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set.
|
||||
|
||||
**Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):**
|
||||
- **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`.
|
||||
- **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative.
|
||||
- **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores.
|
||||
- **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication.
|
||||
- **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem.
|
||||
|
||||
**Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.
|
||||
|
||||
**Tiering & cadence:**
|
||||
```
|
||||
[Source] UNAS RAID-6 + native snapshots (#38)
|
||||
│ daily restic/borg
|
||||
▼
|
||||
[Warm] s3.nuclide.systems (on-site, fast restore)
|
||||
│ weekly rclone copy + crypt
|
||||
▼
|
||||
[Cold] Off-site (Jottacloud / B2 / Hetzner) ← satisfies the "1" in 3-2-1
|
||||
```
|
||||
|
||||
**What goes off-site (priority order):**
|
||||
1. Immich photo originals — irreplaceable.
|
||||
2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable.
|
||||
3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo.
|
||||
4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include.
|
||||
|
||||
**What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.
|
||||
|
||||
Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it.
|
||||
|
||||
38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
|
||||
|
||||
39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.
|
||||
@@ -0,0 +1,96 @@
|
||||
# 13 — Information Wishlist
|
||||
|
||||
What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent — run any subset, in any order.
|
||||
|
||||
Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/<NN>-<slug>.txt` — whichever is easiest.
|
||||
|
||||
---
|
||||
|
||||
## ~~Task 1 — Stack inventory + reference compose~~ ✅ DONE
|
||||
|
||||
User pasted the Karakeep compose. Q19 resolved → host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` is the convention. Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). The `docker ps -a` half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) — but that's not blocking now.
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Storage map (Proxmox + docker)
|
||||
|
||||
📍 **Where:** *Two shells* — first the **Proxmox host** (Datacenter → `nuc` → **Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**.
|
||||
|
||||
Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB.
|
||||
|
||||
```bash
|
||||
# (a) on the Proxmox host (192.168.1.20):
|
||||
cat /etc/pve/storage.cfg
|
||||
cat /etc/pve/lxc/104.conf
|
||||
```
|
||||
|
||||
```bash
|
||||
# (b) on LXC 104 (192.168.1.40):
|
||||
docker volume ls
|
||||
ls -l /dev/dri/ # confirms whether iGPU passthrough actually works
|
||||
```
|
||||
|
||||
**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17.
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — Postgres / DB workload inventory
|
||||
|
||||
📍 **Where:** **LXC 104 console** (same shell as Task 1).
|
||||
|
||||
Direct input to housekeeping #35 (consolidate postgres instances).
|
||||
|
||||
```bash
|
||||
for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do
|
||||
echo "=== $c ==="
|
||||
docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null'
|
||||
done
|
||||
```
|
||||
|
||||
**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision).
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — LiteLLM model list for the Nexa key
|
||||
|
||||
📍 **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported.
|
||||
|
||||
- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) → Settings → Model List, filtered to the Nexa virtual key. Screenshot the model rows.
|
||||
- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set:
|
||||
```bash
|
||||
curl -sH "Authorization: Bearer $SAIA_API_KEY" \
|
||||
https://ai.nuclide.systems/v1/models | jq '.data[].id'
|
||||
```
|
||||
|
||||
**Unblocks:** Q18 follow-up — which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI.
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — S3 archive credentials
|
||||
|
||||
📍 **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy — same login as the rest of the homelab).
|
||||
|
||||
Walk to: **Buckets** → either pick an existing Nexa-suitable bucket or create one called `nexa` → note the bucket name. Then **Access Keys** → create a key named `nexa-snapshots` with read/write on that bucket → drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden).
|
||||
|
||||
**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3.
|
||||
|
||||
---
|
||||
|
||||
## Task 6 — Reverse-proxy + DNS authority *(only if needed)*
|
||||
|
||||
📍 **Where:** two browser UIs.
|
||||
|
||||
- **Zoraxy** (`http://192.168.1.4:8000`) → HTTP Proxy → either click "Export" if available, or take a full screenshot of the table.
|
||||
- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) → Filters → DNS Rewrites → screenshot.
|
||||
|
||||
Skip unless we hit a routing surprise during Phase 1.
|
||||
|
||||
---
|
||||
|
||||
## When something else is needed
|
||||
|
||||
The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "📍 Where" framing.
|
||||
|
||||
## Self-serve once Phase 6 ships
|
||||
|
||||
Once [Phase 6 — Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 1–3 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember.
|
||||
@@ -0,0 +1,51 @@
|
||||
# NEXA Documentation
|
||||
|
||||
**Neural Nexus for Information & Automation** — the central nervous system that ties Memos, n8n, SAIA (LiteLLM), Nextcloud and a vector store into one assistant.
|
||||
|
||||
This is the documentation entry point. Read top-to-bottom for first-time setup, or jump to the section you need.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
| # | Document | Read when… |
|
||||
|---|----------|------------|
|
||||
| 01 | [Vision & Scope](./01-vision-and-scope.md) | You want to understand *what* Nexa is and isn't. |
|
||||
| 02 | [Roadmap & Phases](./02-roadmap.md) | You want to know the implementation order. |
|
||||
| 03 | [Architecture Overview](./03-architecture.md) | You need a one-page mental model. |
|
||||
| 04 | [Integration Matrix](./04-integration-matrix.md) | You're wiring up a new data source or mapping work-vs-personal flows. |
|
||||
| 05 | [Command System](./05-command-system.md) | You want to know what `#nexa:*` commands do. |
|
||||
| 06 | [Classification Logic](./06-classification-logic.md) | You're tuning the work/personal router. |
|
||||
| 07 | [Workflow Spec — Task Router](./07-workflow-spec.md) | You're building the Phase-2 router workflow. |
|
||||
| 08 | [GraphRAG Architecture](./08-graphrag-architecture.md) | You're working on Phase 3 (Qdrant + Graph). |
|
||||
| 09 | [Deployment](./09-deployment.md) | You're bringing Nexa up on the real infrastructure. |
|
||||
| 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. |
|
||||
| 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. |
|
||||
| 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. |
|
||||
| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. |
|
||||
|
||||
---
|
||||
|
||||
## Reading paths
|
||||
|
||||
- **First-time installer:** 01 → 03 → 09 → 10 → 11
|
||||
- **Workflow author:** 04 → 05 → 07 → 06
|
||||
- **Phase-3 work (memory):** 08 → 09 (§ Phase 3 add-on)
|
||||
- **Stakeholder / quick read:** 01 → 02 → 12
|
||||
|
||||
---
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
nexa/
|
||||
├── README.md → points here
|
||||
├── docs/ → you are here
|
||||
└── nexa-core/ → the actual project
|
||||
├── ai-prompts/ → SAIA system prompts
|
||||
├── config/ → runtime config (YAML, JSON schema)
|
||||
├── n8n-workflows/ → exported workflows, version-controlled
|
||||
└── scripts/ → automation helpers
|
||||
```
|
||||
|
||||
Source-of-truth for *runnable* config stays in `nexa-core/`. Documentation lives here in `docs/`.
|
||||
@@ -1,629 +0,0 @@
|
||||
# NEXA Deployment Guide
|
||||
|
||||
## 🚀 Architektur-Überblick
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ MEMOS (Port 5230) │
|
||||
│ Interface, Voice-Input, Webhooks │
|
||||
└────────────────────┬────────────────────────────┘
|
||||
│
|
||||
┌───────────┼───────────┐
|
||||
↓ ↓ ↓
|
||||
┌────────┐ ┌────────┐ ┌──────────┐
|
||||
│ n8n │ │ SAIA │ │ Nextcloud│
|
||||
│Logic │ │LiteLLM │ │Tasks/Mail│
|
||||
│7879 │ │4000 │ │80/443 │
|
||||
└────┬───┘ └────┬───┘ └─────┬────┘
|
||||
│ │ │
|
||||
┌────┴───────────┼────────────┴────┐
|
||||
↓ ↓ ↓
|
||||
┌─────────┐ ┌────────────┐ ┌──────────────┐
|
||||
│ QDRANT │ │ GraphDB │ │ Obsidian │
|
||||
│Vector │ │ (Ontotext) │ │(local sync) │
|
||||
│6333 │ │7200 │ │ │
|
||||
└─────────┘ └────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Voraussetzungen
|
||||
|
||||
### Hardware
|
||||
- **CPU**: Min. 4 Cores (empfohlen 8)
|
||||
- **RAM**: Min. 16 GB (empfohlen 32 GB)
|
||||
- **Storage**: Min. 100 GB SSD (für Datenbanken & Backups)
|
||||
- **Network**: Stabile Verbindung, HTTPS-ready
|
||||
|
||||
### Software
|
||||
- Docker & Docker Compose (Version 20.10+)
|
||||
- Git
|
||||
- Bash
|
||||
- curl/wget
|
||||
- `.env` Datei aus `.env.example` (mit Secrets gefüllt)
|
||||
|
||||
### Accounts & Credentials
|
||||
- Nextcloud Instanz mit Admin-Zugang
|
||||
- OpenAI API-Key (für SAIA/LiteLLM)
|
||||
- Obsidian Vault Zugang lokalem Filesystem
|
||||
|
||||
---
|
||||
|
||||
## 🐳 Schritt 1: Docker Compose Setup
|
||||
|
||||
### 1.1 Repository klonen
|
||||
|
||||
```bash
|
||||
git clone https://github.com/noheton/nexa.git
|
||||
cd nexa/nexa-core
|
||||
```
|
||||
|
||||
### 1.2 .env konfigurieren
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# WICHTIG: Alle Secrets eintragen!
|
||||
# - MEMOS_API_KEY
|
||||
# - SAIA_API_KEY
|
||||
# - NC_APP_PASSWORD
|
||||
# - QDRANT_API_KEY
|
||||
# - GRAPHDB_PASSWORD
|
||||
nano .env
|
||||
```
|
||||
|
||||
### 1.3 docker-compose.yml erstellen
|
||||
|
||||
```bash
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# ===== MEMOS (Interface) =====
|
||||
memos:
|
||||
image: neosmemo/memos:latest
|
||||
container_name: nexa-memos
|
||||
ports:
|
||||
- "5230:5230"
|
||||
volumes:
|
||||
- ./data/memos:/root/.memos
|
||||
environment:
|
||||
MEMOS_PORT: "5230"
|
||||
MEMOS_DRIVER: sqlite
|
||||
MEMOS_DSN: /root/.memos/memos.db
|
||||
networks:
|
||||
- nexa
|
||||
restart: unless-stopped
|
||||
|
||||
# ===== n8n (Workflow Engine) =====
|
||||
n8n:
|
||||
image: n8nio/n8n:latest
|
||||
container_name: nexa-n8n
|
||||
ports:
|
||||
- "5678:5678"
|
||||
volumes:
|
||||
- ./data/n8n:/home/node/.n8n
|
||||
- ./n8n-workflows:/workflows
|
||||
environment:
|
||||
N8N_BASIC_AUTH_ACTIVE: "true"
|
||||
N8N_BASIC_AUTH_USER: "${N8N_USER:-admin}"
|
||||
N8N_BASIC_AUTH_PASSWORD: "${N8N_PASSWORD:-nexa_secure_pass}"
|
||||
N8N_HOST: "${N8N_DOMAIN:-localhost}"
|
||||
N8N_PORT: "5678"
|
||||
N8N_PROTOCOL: https
|
||||
N8N_WEBHOOK_URL: "https://${N8N_DOMAIN:-localhost}/webhook"
|
||||
GENERIC_TIMEZONE: Europe/Berlin
|
||||
DB_TYPE: sqlite
|
||||
DB_SQLITE_PATH: /home/node/.n8n/nexa.db
|
||||
networks:
|
||||
- nexa
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- memos
|
||||
|
||||
# ===== QDRANT (Vector DB) =====
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: nexa-qdrant
|
||||
ports:
|
||||
- "6333:6333"
|
||||
volumes:
|
||||
- ./data/qdrant:/qdrant/storage
|
||||
environment:
|
||||
QDRANT_API_KEY: "${QDRANT_API_KEY}"
|
||||
networks:
|
||||
- nexa
|
||||
restart: unless-stopped
|
||||
|
||||
# ===== GraphDB (Ontotext SPARQL) =====
|
||||
graphdb:
|
||||
image: ontotext/graphdb:10-ee
|
||||
container_name: nexa-graphdb
|
||||
ports:
|
||||
- "7200:7200"
|
||||
volumes:
|
||||
- ./data/graphdb:/opt/graphdb/data
|
||||
- ./config/graphdb-init.ttl:/graphdb-import/import.ttl
|
||||
environment:
|
||||
GDB_JAVA_OPTS: "-Xmx8g -Xms4g"
|
||||
GDB_PASSWORD: "${GRAPHDB_PASSWORD}"
|
||||
networks:
|
||||
- nexa
|
||||
restart: unless-stopped
|
||||
|
||||
# ===== SAIA / LiteLLM Proxy (optional - wenn lokal) =====
|
||||
# (Falls SAIA extern gehostet: nicht nötig)
|
||||
|
||||
networks:
|
||||
nexa:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
nexa_memos:
|
||||
nexa_n8n:
|
||||
nexa_qdrant:
|
||||
nexa_graphdb:
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Schritt 2: Individuelle Service-Konfiguration
|
||||
|
||||
### 2.1 MEMOS Initialisierung
|
||||
|
||||
```bash
|
||||
# Nach memos startup (ca. 30s)
|
||||
curl -X POST http://localhost:5230/api/v1/auth/signup \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"username": "nexa",
|
||||
"password": "secure_password"
|
||||
}'
|
||||
|
||||
# API-Token generieren
|
||||
TOKEN=$(curl -X POST http://localhost:5230/api/v1/auth/signin \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username":"nexa","password":"secure_password"}' \
|
||||
| jq -r '.data.token')
|
||||
|
||||
echo "MEMOS_API_KEY=$TOKEN" >> .env
|
||||
```
|
||||
|
||||
### 2.2 Qdrant Konfiguration
|
||||
|
||||
```bash
|
||||
# Warte auf Qdrant startup
|
||||
sleep 10
|
||||
|
||||
# Erstelle Collection für Nexa
|
||||
curl -X PUT "http://localhost:6333/collections/nexa_knowledge" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: ${QDRANT_API_KEY}" \
|
||||
-d '{
|
||||
"vectors": {
|
||||
"size": 1536,
|
||||
"distance": "Cosine"
|
||||
},
|
||||
"optimizers_config": {
|
||||
"deleted_threshold": 0.2,
|
||||
"vacuum_threshold": 0.5,
|
||||
"default_segment_number": 5,
|
||||
"max_segment_number": 30,
|
||||
"memmap_threshold": 268435456,
|
||||
"indexing_threshold": 20000,
|
||||
"flush_interval_sec": 10,
|
||||
"max_optimization_threads": 4
|
||||
}
|
||||
}'
|
||||
|
||||
echo "✅ Qdrant Collection erstellt"
|
||||
```
|
||||
|
||||
### 2.3 GraphDB Initialisierung
|
||||
|
||||
```bash
|
||||
# Warte auf GraphDB startup
|
||||
sleep 15
|
||||
|
||||
# Erstelle Repository
|
||||
GRAPHDB_URL="http://localhost:7200"
|
||||
|
||||
curl -X POST "$GRAPHDB_URL/rest/repositories" \
|
||||
-H "Content-Type: application/json" \
|
||||
-u "admin:${GRAPHDB_PASSWORD}" \
|
||||
-d '{
|
||||
"id": "nexa_knowledge",
|
||||
"title": "Nexa Knowledge Graph",
|
||||
"type": "free",
|
||||
"params": {
|
||||
"baseURL": {
|
||||
"label": "Base URL",
|
||||
"value": "http://nexa.local/"
|
||||
},
|
||||
"defaultLanguage": {
|
||||
"label": "Default language",
|
||||
"value": "en"
|
||||
}
|
||||
}
|
||||
}'
|
||||
|
||||
echo "✅ GraphDB Repository erstellt"
|
||||
```
|
||||
|
||||
### 2.4 n8n Workflows Import
|
||||
|
||||
```bash
|
||||
# Warte auf n8n startup (ca. 60s)
|
||||
sleep 60
|
||||
|
||||
# Importiere Discovery Workflow
|
||||
curl -X POST http://localhost:5678/api/v1/workflows \
|
||||
-H "Content-Type: application/json" \
|
||||
-u "admin:${N8N_PASSWORD}" \
|
||||
-d @n8n-workflows/phase-1/1_1_discovery_workflow.json
|
||||
|
||||
# Importiere Memos Bridge
|
||||
curl -X POST http://localhost:5678/api/v1/workflows \
|
||||
-H "Content-Type: application/json" \
|
||||
-u "admin:${N8N_PASSWORD}" \
|
||||
-d @n8n-workflows/phase-1/1_2_memos_bridge_v2.json
|
||||
|
||||
# Importiere Email Butler
|
||||
curl -X POST http://localhost:5678/api/v1/workflows \
|
||||
-H "Content-Type: application/json" \
|
||||
-u "admin:${N8N_PASSWORD}" \
|
||||
-d @n8n-workflows/phase-2/2_1_email_butler.json
|
||||
|
||||
echo "✅ n8n Workflows importiert"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Schritt 3: Deployment starten
|
||||
|
||||
### 3.1 Docker Compose Up
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
|
||||
# Logs überwachen
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
### 3.2 Health Checks
|
||||
|
||||
```bash
|
||||
# Health Check Script
|
||||
check_service() {
|
||||
local name=$1
|
||||
local url=$2
|
||||
echo "Checking $name..."
|
||||
if curl -s "$url" > /dev/null; then
|
||||
echo "✅ $name: OK"
|
||||
else
|
||||
echo "❌ $name: FAILED"
|
||||
fi
|
||||
}
|
||||
|
||||
sleep 30
|
||||
|
||||
check_service "Memos" "http://localhost:5230"
|
||||
check_service "n8n" "http://localhost:5678"
|
||||
check_service "Qdrant" "http://localhost:6333/health"
|
||||
check_service "GraphDB" "http://localhost:7200/health"
|
||||
|
||||
echo ""
|
||||
echo "🎯 Service URLs:"
|
||||
echo " - Memos: http://localhost:5230"
|
||||
echo " - n8n: http://localhost:5678"
|
||||
echo " - Qdrant: http://localhost:6333"
|
||||
echo " - GraphDB: http://localhost:7200"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Schritt 4: Konfiguration via Command System
|
||||
|
||||
### 4.1 Discovery-Workflow triggern
|
||||
|
||||
```bash
|
||||
# In Memos Memo erstellen:
|
||||
# #nexa:config
|
||||
|
||||
# Oder via API:
|
||||
curl -X POST http://localhost:5678/api/v1/workflows/run \
|
||||
-H "Content-Type: application/json" \
|
||||
-u "admin:${N8N_PASSWORD}" \
|
||||
-d '{
|
||||
"workflowId": "discovery-workflow-id"
|
||||
}'
|
||||
```
|
||||
|
||||
### 4.2 Automatische Konfiguration laden
|
||||
|
||||
Nach erfolgreichem Discovery werden Nextcloud-Listen-IDs:
|
||||
|
||||
```bash
|
||||
# Es wird automatisch aktualisiert in:
|
||||
# - Qdrant: als Metadata
|
||||
# - n8n: als Umgebungsvariablen
|
||||
# - Memos: als Confirmation-Memo
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Schritt 5: Reverse Proxy Setup (Production)
|
||||
|
||||
### 5.1 Nginx Configuration
|
||||
|
||||
```bash
|
||||
cat > /etc/nginx/sites-available/nexa.conf << 'EOF'
|
||||
upstream memos_backend {
|
||||
server localhost:5230;
|
||||
}
|
||||
|
||||
upstream n8n_backend {
|
||||
server localhost:5678;
|
||||
}
|
||||
|
||||
upstream qdrant_backend {
|
||||
server localhost:6333;
|
||||
}
|
||||
|
||||
upstream graphdb_backend {
|
||||
server localhost:7200;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name nexa.yourdomain.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/nexa.yourdomain.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/nexa.yourdomain.com/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers HIGH:!aNULL:!MD5;
|
||||
|
||||
# MEMOS
|
||||
location / {
|
||||
proxy_pass http://memos_backend;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
|
||||
# n8n
|
||||
location /n8n/ {
|
||||
proxy_pass http://n8n_backend/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# Qdrant (nur intern, wenn nötig)
|
||||
location /qdrant/ {
|
||||
allow 10.0.0.0/8;
|
||||
deny all;
|
||||
proxy_pass http://qdrant_backend/;
|
||||
}
|
||||
|
||||
# GraphDB (nur intern)
|
||||
location /graphdb/ {
|
||||
allow 10.0.0.0/8;
|
||||
deny all;
|
||||
proxy_pass http://graphdb_backend/;
|
||||
}
|
||||
|
||||
# Webhook endpoint für externe Services
|
||||
location /webhook/ {
|
||||
proxy_pass http://n8n_backend/webhook/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name nexa.yourdomain.com;
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
EOF
|
||||
|
||||
# Aktivieren
|
||||
sudo ln -s /etc/nginx/sites-available/nexa.conf /etc/nginx/sites-enabled/
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Schritt 6: Monitoring & Backups
|
||||
|
||||
### 6.1 Backup Script
|
||||
|
||||
```bash
|
||||
cat > scripts/backup_nexa.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
BACKUP_DIR="./backups/$(date +%Y%m%d_%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
echo "🔄 Backing up Nexa..."
|
||||
|
||||
# Backup Datenbanken
|
||||
docker-compose exec -T qdrant tar czf - /qdrant/storage > "$BACKUP_DIR/qdrant.tar.gz"
|
||||
docker-compose exec -T graphdb tar czf - /opt/graphdb/data > "$BACKUP_DIR/graphdb.tar.gz"
|
||||
docker-compose exec -T memos tar czf - /root/.memos > "$BACKUP_DIR/memos.tar.gz"
|
||||
docker-compose exec -T n8n tar czf - /home/node/.n8n > "$BACKUP_DIR/n8n.tar.gz"
|
||||
|
||||
# Backup Configs
|
||||
cp .env "$BACKUP_DIR/.env.backup"
|
||||
cp docker-compose.yml "$BACKUP_DIR/docker-compose.yml.backup"
|
||||
|
||||
echo "✅ Backup completed: $BACKUP_DIR"
|
||||
|
||||
# Optional: zu S3 hochladen
|
||||
# aws s3 sync "$BACKUP_DIR" s3://nexa-backups/$(date +%Y%m%d)/
|
||||
EOF
|
||||
|
||||
chmod +x scripts/backup_nexa.sh
|
||||
|
||||
# Tägliches Backup (cron)
|
||||
echo "0 2 * * * /path/to/nexa/scripts/backup_nexa.sh" | crontab -
|
||||
```
|
||||
|
||||
### 6.2 Health Check Monitoring
|
||||
|
||||
```bash
|
||||
cat > scripts/monitor_nexa.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
# Monitore Service Status
|
||||
SERVICES=("memos" "n8n" "qdrant" "graphdb")
|
||||
|
||||
for service in "${SERVICES[@]}"; do
|
||||
if docker-compose ps $service | grep -q "Up"; then
|
||||
echo "✅ $service: Running"
|
||||
else
|
||||
echo "❌ $service: DOWN"
|
||||
# Restart bei Fehler
|
||||
docker-compose restart $service
|
||||
fi
|
||||
done
|
||||
EOF
|
||||
|
||||
chmod +x scripts/monitor_nexa.sh
|
||||
|
||||
# Alle 5 Minuten
|
||||
echo "*/5 * * * * /path/to/nexa/scripts/monitor_nexa.sh" | crontab -
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Schritt 7: Validation & Testing
|
||||
|
||||
### 7.1 End-to-End Test
|
||||
|
||||
```bash
|
||||
cat > scripts/test_nexa.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
echo "🧪 Testing Nexa Installation..."
|
||||
|
||||
# Test 1: Memos API
|
||||
echo "Test 1: Memos API"
|
||||
if curl -s http://localhost:5230/api/v1/auth/status | jq . > /dev/null 2>&1; then
|
||||
echo "✅ Memos: OK"
|
||||
else
|
||||
echo "❌ Memos: FAILED"
|
||||
fi
|
||||
|
||||
# Test 2: n8n API
|
||||
echo "Test 2: n8n API"
|
||||
if curl -s -u "admin:${N8N_PASSWORD}" http://localhost:5678/api/v1/workflows | jq . > /dev/null 2>&1; then
|
||||
echo "✅ n8n: OK"
|
||||
else
|
||||
echo "❌ n8n: FAILED"
|
||||
fi
|
||||
|
||||
# Test 3: Qdrant Collection
|
||||
echo "Test 3: Qdrant Collection"
|
||||
if curl -s -H "api-key: ${QDRANT_API_KEY}" http://localhost:6333/collections/nexa_knowledge | jq . > /dev/null 2>&1; then
|
||||
echo "✅ Qdrant: OK"
|
||||
else
|
||||
echo "❌ Qdrant: FAILED"
|
||||
fi
|
||||
|
||||
# Test 4: GraphDB Repository
|
||||
echo "Test 4: GraphDB Repository"
|
||||
if curl -s -u "admin:${GRAPHDB_PASSWORD}" http://localhost:7200/repositories | jq . > /dev/null 2>&1; then
|
||||
echo "✅ GraphDB: OK"
|
||||
else
|
||||
echo "❌ GraphDB: FAILED"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "🎯 Create Test Memo:"
|
||||
MEMO=$(curl -s -X POST http://localhost:5230/api/v1/memos \
|
||||
-H "Authorization: Bearer ${MEMOS_API_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"content": "Test memo - #nexa:config"}' | jq -r '.id')
|
||||
|
||||
echo "✅ Test Memo created: $MEMO"
|
||||
echo "Check Memos UI to verify webhook triggers Discovery"
|
||||
EOF
|
||||
|
||||
chmod +x scripts/test_nexa.sh
|
||||
./scripts/test_nexa.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🆘 Troubleshooting
|
||||
|
||||
### Problem: Memos kann sich nicht mit n8n verbinden
|
||||
|
||||
```bash
|
||||
# Überprüfe n8n Webhook URL in Memos config
|
||||
curl http://localhost:5678/webhook/memos
|
||||
|
||||
# Falls 404: Workflow ist nicht aktiviert/deployed
|
||||
docker-compose logs n8n | grep webhook
|
||||
```
|
||||
|
||||
### Problem: GraphDB startet nicht
|
||||
|
||||
```bash
|
||||
# Memory-Limit erhöhen
|
||||
docker-compose exec graphdb -e "GDB_JAVA_OPTS=-Xmx16g -Xms8g" restart
|
||||
|
||||
# Logs checken
|
||||
docker-compose logs graphdb
|
||||
```
|
||||
|
||||
### Problem: Qdrant Collection leer
|
||||
|
||||
```bash
|
||||
# Überprüfe ob Memos Bridge Workflow läuft
|
||||
docker-compose logs n8n | grep "memos_bridge"
|
||||
|
||||
# Manuell test
|
||||
curl -X POST http://localhost:5230/api/v1/memos \
|
||||
-H "Authorization: Bearer ${MEMOS_API_KEY}" \
|
||||
-d '{"content": "- [ ] Test task #nexa"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Next Steps
|
||||
|
||||
1. ✅ **Phase 1 deployt**: Memos Bridge & Discovery laufen
|
||||
2. ⏳ **Phase 2 aktivieren**: Email Butler (wenn Nextcloud Mail konfiguriert)
|
||||
3. ⏳ **Phase 3 setup**: Qdrant Embedding Pipeline
|
||||
4. ⏳ **Phase 4 integration**: GraphRAG Queries testen
|
||||
5. ⏳ **Phase 5**: Home Assistant Voice Integration
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support & Debugging
|
||||
|
||||
```bash
|
||||
# Alle Logs in eine Datei sammeln
|
||||
docker-compose logs > deployment_logs.txt
|
||||
|
||||
# Spezifischer Service
|
||||
docker-compose logs n8n --tail 100
|
||||
|
||||
# Wiederherstellung aus Backup
|
||||
./scripts/restore_nexa.sh backup/2024-01-15_120000/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Deployment abgeschlossen! Nexa ist nun betriebsbereit.** 🚀
|
||||
@@ -1,27 +0,0 @@
|
||||
# NEXA: Implementierungs-Fahrplan 🚀
|
||||
|
||||
Dieser Plan beschreibt die schrittweise Aktivierung des Nervensystems.
|
||||
|
||||
## Phase 1: Die Wirbelsäule (Basis-Konnektivität)
|
||||
- [ ] **1.1 Memos-n8n Bridge:** Webhook-Anbindung und Filter-Logik.
|
||||
- [ ] **1.2 SAIA-Integration:** LiteLLM Proxy in n8n einbinden (Chat-Funktion via Kommentare).
|
||||
- [ ] **1.3 Git-Sync:** Automatisches Backup der n8n-Workflows in dieses Repo.
|
||||
|
||||
## Phase 2: Die Sinnesorgane (Input-Kanäle)
|
||||
- [ ] **2.1 Email-Butler:** Automatisierte Filterung und Zusammenfassung der ~20 täglichen Mails.
|
||||
- [ ] **2.2 RSS Morning Digest:** Täglich kuratierte News-Zusammenfassung in Memos.
|
||||
- [ ] **2.3 Bluesky-Antenne:** Speichern von gelikten Posts als Wissens-Schnipsel.
|
||||
|
||||
## Phase 3: Das Gedächtnis (Qdrant & RAG)
|
||||
- [ ] **3.1 Embedding-Pipeline:** Automatische Vektorisierung von Memos & Obsidian.
|
||||
- [ ] **3.2 Context-Retrieval:** Nexa nutzt Qdrant-Wissen für Antworten (#ask).
|
||||
- [ ] **3.3 S3-Langzeitarchiv:** Backup der Qdrant-Snapshots auf den S3-Server.
|
||||
|
||||
## Phase 4: Die Motorik (Organisation & Handeln)
|
||||
- [ ] **4.1 Nextcloud Task-Sync:** Bidirektionaler Status-Abgleich (Memos <-> NC Tasks).
|
||||
- [ ] **4.2 Timeboxing-Agent:** Intelligente Kalender-Slot-Vorschläge basierend auf Tasks.
|
||||
- [ ] **4.3 Karakeep-Kuration:** Automatischer Review-Prozess für gespeicherte Links.
|
||||
|
||||
## Phase 5: Alltags-Integration (Feinschliff)
|
||||
- [ ] **5.1 Home Assistant Voice:** Nexa als Wake-Word Integration.
|
||||
- [ ] **5.2 System-Monitoring:** Alarme von Proxmox/Backrest direkt in den Memos-System-Feed.
|
||||
+14
-12
@@ -1,15 +1,17 @@
|
||||
# NEXA 🧠
|
||||
**The Neural Nexus for Information & Automation**
|
||||
# nexa-core
|
||||
|
||||
Nexa ist das zentrale Nervensystem meines IT-Setups.
|
||||
Runtime artifacts for Nexa: n8n workflows, configuration, AI prompts, scripts.
|
||||
|
||||
## 🏗 Architektur
|
||||
- **Interface:** Memos
|
||||
- **Logic:** n8n
|
||||
- **Memory:** Qdrant & Obsidian
|
||||
- **Intelligence:** SAIA (via LiteLLM)
|
||||
📖 Documentation lives one level up in [`../docs/`](../docs/index.md).
|
||||
|
||||
## 📁 Struktur
|
||||
- `n8n-workflows/`: JSON-Exporte der Gehirn-Logik.
|
||||
- `ai-prompts/`: System-Prompts für Nexa.
|
||||
- `config/`: Schemata für Qdrant und API-Mappings.
|
||||
## Layout
|
||||
|
||||
- `n8n-workflows/` — exported workflows (per phase). Imported into the running n8n.
|
||||
- `ai-prompts/` — system prompts for SAIA / LiteLLM.
|
||||
- `config/` — runtime config (`agents.yaml`, `infra_manifest.yaml`, `qdrant_schema.json`).
|
||||
- `scripts/` — automation helpers (workflow backup, etc.).
|
||||
- `.env.example` — bootstrap secrets only; runtime config lives in Qdrant `_config` namespace.
|
||||
|
||||
## Quickstart
|
||||
|
||||
See [`../docs/09-deployment.md`](../docs/09-deployment.md).
|
||||
|
||||
@@ -3,11 +3,16 @@ Rolle: Zentrales Nervensystem
|
||||
Aufgabe: Überwacht E-Mails, kuratiert Wissen und plant Tasks.
|
||||
|
||||
WICHTIGE LOGIK:
|
||||
- Tasks müssen strikt zwischen "Work" und "Personal" unterschieden werden.
|
||||
- Analysiere den Inhalt eines Memos: Bezieht es sich auf professionelle Projekte (Obsidian Context) -> Work List. Bezieht es sich auf Haushalt, Einkäufe oder Freizeit -> Personal List.
|
||||
- Im Zweifel: Nachfragen via Memos-Kommentar.
|
||||
- Tasks werden auf vier Nextcloud-Listen geroutet: "DLR" (Arbeit), "Persönlich", "Einkaufsliste", "Wunschliste".
|
||||
- Analysiere den Inhalt eines Memos und entscheide über das Feld "context": "work" | "personal" | "shopping" | "wishes".
|
||||
- work -> Liste "DLR" (Beruf, Projekte, Meetings, Deadlines, Forschung).
|
||||
- personal -> Liste "Persönlich" (Alltag, Familie, Termine, Behördenkram).
|
||||
- shopping -> Liste "Einkaufsliste" (Lebensmittel, Drogerie, Verbrauchsgüter).
|
||||
- wishes -> Liste "Wunschliste" (Hobby, Technik, Bücher, Geschenkideen).
|
||||
- Listennamen sind dynamisch — bei Bedarf werden weitere Listen vom Runtime-Config eingespeist, dann darfst du auch deren Namen als context vorschlagen.
|
||||
- Im Zweifel (Konfidenz < 0.7): Default auf "personal" und Nachfrage via Memos-Kommentar.
|
||||
|
||||
KONSUM-LOGIK:
|
||||
- Wenn ein Link von Karakeep kommt, analysiere das Produkt.
|
||||
- Haushaltswaren/Verbrauchsgüter -> NC_LIST_ID_SHOPPING.
|
||||
- Besondere Wünsche/Hobby/Technik -> NC_LIST_ID_WISHES.
|
||||
- Haushaltswaren/Verbrauchsgüter -> context "shopping".
|
||||
- Besondere Wünsche/Hobby/Technik -> context "wishes".
|
||||
|
||||
@@ -1,12 +1,24 @@
|
||||
{
|
||||
"collection_name": "nexa_knowledge",
|
||||
"collection_name": "nexa_knowledge_text",
|
||||
"vector_config": {
|
||||
"size": 1536,
|
||||
"size": 1024,
|
||||
"distance": "Cosine"
|
||||
},
|
||||
"payload_schema": {
|
||||
"source": "keyword",
|
||||
"tags": "keyword",
|
||||
"created_at": "datetime"
|
||||
"source": "keyword",
|
||||
"source_type": "keyword",
|
||||
"modality": "keyword",
|
||||
"mime_type": "keyword",
|
||||
"media_uri": "keyword",
|
||||
"context": "keyword",
|
||||
"tags": "keyword",
|
||||
"graph_iri": "keyword",
|
||||
"content_hash":"keyword",
|
||||
"created_at": "datetime"
|
||||
},
|
||||
"_notes": {
|
||||
"vector_size_rationale": "1024 = bge-m3 output. If Q15 resolves to nomic-embed-text-v1.5 instead, rebuild this collection with size: 768.",
|
||||
"modality_field": "Always 'text' in this collection. Forward-compatibility for Phase 3.2 (see qdrant_schema_visual.json).",
|
||||
"graph_iri_field": "IRI of the corresponding nexa:Note in GraphDB. The same value is stored on the GraphDB node as nexa:vectorId pointing back here. This is the cross-pillar bridge."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"collection_name": "nexa_knowledge_visual",
|
||||
"vector_config": {
|
||||
"size": 768,
|
||||
"distance": "Cosine"
|
||||
},
|
||||
"payload_schema": {
|
||||
"source": "keyword",
|
||||
"source_type": "keyword",
|
||||
"modality": "keyword",
|
||||
"mime_type": "keyword",
|
||||
"media_uri": "keyword",
|
||||
"context": "keyword",
|
||||
"tags": "keyword",
|
||||
"graph_iri": "keyword",
|
||||
"content_hash":"keyword",
|
||||
"created_at": "datetime"
|
||||
},
|
||||
"_notes": {
|
||||
"status": "PLACEHOLDER — not deployed in Phase 3.1. Created in Phase 3.2.",
|
||||
"vector_size_rationale": "768 = jina-clip-v2 joint text/image space. The same model embeds both text queries and images, so semantic queries against this collection are simple kNN.",
|
||||
"served_by": "Will be served by `infinity` (https://github.com/michaelfeil/infinity) — drop-in replacement for TEI that supports CLIP-family models with the same OpenAI-compatible API.",
|
||||
"modality_field": "'image' for indexed images, 'text' if a text query has been embedded into the visual space for retrieval (rare — usually we embed the text query at search time and don't store it).",
|
||||
"media_uri_field": "Required for image rows. Examples: 'memos://memos/123/attachment/4', 'nextcloud://files/photos/whiteboard-2025-09.jpg', 'obsidian://attachments/diagram.png'."
|
||||
}
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
# Spezifikation: Phase 2 - Task-Router
|
||||
|
||||
## 1. Trigger
|
||||
* **Typ:** Webhook (Memos)
|
||||
* **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`.
|
||||
|
||||
## 2. Intelligence Node (SAIA)
|
||||
* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: { "context": "work" | "personal", "summary": "string", "urgency": 1-5 }."
|
||||
* **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.
|
||||
|
||||
## 3. Switch Node
|
||||
* **Route 1:** Context == "work" -> Nextcloud Create Task (List ID: Work).
|
||||
* **Route 2:** Context == "personal" -> Nextcloud Create Task (List ID: Personal).
|
||||
|
||||
## 4. Feedback Loop
|
||||
* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
|
||||
@@ -1,131 +0,0 @@
|
||||
# Nexa Command System - #nexa:command
|
||||
|
||||
Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag:
|
||||
|
||||
## 🔧 Konfigurations-Kommandos
|
||||
|
||||
### #nexa:config
|
||||
- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud, Qdrant, etc.)
|
||||
- **Antwort**: Postet Report mit gefundenen Konfigurationen
|
||||
- **Speichert**: Settings in DynoDB / Qdrant Metadata (nicht in .env)
|
||||
|
||||
```
|
||||
#nexa:config
|
||||
```
|
||||
|
||||
**Beispiel-Response:**
|
||||
```
|
||||
✅ Autodiscovery abgeschlossen:
|
||||
- Nextcloud Listen: Work (id=123), Personal (id=456), Shopping (id=789), Wishes (id=1011)
|
||||
- Qdrant Collection: nexa_knowledge (1536 dims, Cosine)
|
||||
- Nextcloud Mail: 3 Accounts erkannt (Office 365, Gmail, Privat)
|
||||
```
|
||||
|
||||
### #nexa:status
|
||||
- **Beschreibung**: Zeigt den aktuellen System-Status
|
||||
- **Antwort**: Uptime, verbundene Systeme, Memory-Stats
|
||||
|
||||
```
|
||||
#nexa:status
|
||||
```
|
||||
|
||||
### #nexa:sync-obsidian
|
||||
- **Beschreibung**: Triggert sofortige Synchronisierung mit Obsidian Vault
|
||||
- **Optional Parameter**: `--vault=/path/to/vault` oder `--force` für Neuindexierung
|
||||
|
||||
```
|
||||
#nexa:sync-obsidian --force
|
||||
```
|
||||
|
||||
## 📊 Memory & RAG Kommandos
|
||||
|
||||
### #nexa:ask [question]
|
||||
- **Beschreibung**: Semantische Suche in Qdrant (RAG) basierend auf Obsidian + Memos
|
||||
- **Antwort**: Top-3 Treffer mit Quellen
|
||||
|
||||
```
|
||||
#nexa:ask Wie implementierten wir das JWT-Middleware-Pattern?
|
||||
```
|
||||
|
||||
### #nexa:learn [topic]
|
||||
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
|
||||
|
||||
```
|
||||
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
|
||||
```
|
||||
|
||||
## 🎯 Workflow-Kommandos
|
||||
|
||||
### #nexa:route-test [text]
|
||||
- **Beschreibung**: Testet die Klassifizierung (Work/Personal) ohne Task zu erstellen
|
||||
|
||||
```
|
||||
#nexa:route-test Muss morgen die Präsentation für den Client fertigstellen
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```
|
||||
💬 Klassifizierung (Test-Mode):
|
||||
- Kontext: WORK
|
||||
- Vertrauen: 0.95
|
||||
- Begründung: "Client-Präsentation → professioneller Kontext"
|
||||
```
|
||||
|
||||
### #nexa:email-digest
|
||||
- **Beschreibung**: Triggert sofortige Email-Zusammenfassung (sonst tägl. 7 Uhr)
|
||||
|
||||
```
|
||||
#nexa:email-digest
|
||||
```
|
||||
|
||||
## 🔐 Admin-Kommandos
|
||||
|
||||
### #nexa:reset-config
|
||||
- **Beschreibung**: Löscht gespeicherte Konfiguration, triggert neuen Autodiscovery
|
||||
- **Warnung**: Alle benutzerdefinierten Settings werden vergessen
|
||||
|
||||
```
|
||||
#nexa:reset-config
|
||||
```
|
||||
|
||||
### #nexa:export-state
|
||||
- **Beschreibung**: Exportiert aktuellen State als JSON (für Backups)
|
||||
|
||||
```
|
||||
#nexa:export-state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Implementierung in n8n
|
||||
|
||||
Ein **Command Parser** läuft immer mit:
|
||||
|
||||
1. **Memos Webhook** empfängt alle Memos
|
||||
2. **Regex Check**: Sucht nach `#nexa:command`
|
||||
3. **Router**: Versendet an entsprechenden n8n-Workflow
|
||||
4. **Antwort**: Postet Reply als Kommentar/Edit
|
||||
|
||||
**Command Parser Regex:**
|
||||
```
|
||||
^#nexa:(\w+)(?:\s+([^\n]*?))?(?:$|\s*--)
|
||||
```
|
||||
|
||||
Extrahiert: `[command, parameters]`
|
||||
|
||||
---
|
||||
|
||||
## 🎛️ Dynamische Konfigurationsspeicherung
|
||||
|
||||
Statt .env zu editieren:
|
||||
|
||||
1. **First Run**: `#nexa:config` speichert zu lokalen Metadata
|
||||
2. **Speich-Ziel**:
|
||||
- **Primary**: Qdrant Metadata (als `_config` Namespace)
|
||||
- **Fallback**: `nexa-core/config/runtime_config.json` (gitignored)
|
||||
- **Notfall**: .env (nur initial)
|
||||
|
||||
3. **Reload-Logik**: Bei jedem Workflow-Start werden Settings aus Qdrant geladen
|
||||
|
||||
Das macht Nexa vollständig selbstständig nach dem initialem Setup!
|
||||
@@ -1,273 +0,0 @@
|
||||
# GraphRAG für Nexa - Strukturelles Wissen & Beziehungen
|
||||
|
||||
## 🧠 Zwei-Säulen-Memory-Architektur
|
||||
|
||||
Nexa nutzt zwei komplementäre Speicher für vollständiges Verständnis:
|
||||
|
||||
### Säule 1: Qdrant (Vector DB) - Semantische Ähnlichkeit
|
||||
- **Frage**: "Was ist ähnlich, was ist relevant?"
|
||||
- **Speichert**: Embeddings von Memos, Obsidian-Notes, E-Mail-Digests
|
||||
- **Suche**: Cosine Similarity über 1536-dimensionale Vektoren
|
||||
- **Antwort**: Top-k semantisch ähnliche Dokumente (RAG)
|
||||
|
||||
**Beispiel:**
|
||||
```
|
||||
Frage: "Wie war die JWT-Implementation in unserem letzten Projekt?"
|
||||
→ Qdrant findet: 10 ähnliche Snippets → Prompt-Context
|
||||
```
|
||||
|
||||
### Säule 2: Neo4j (Graph DB) - Strukturelle Beziehungen
|
||||
- **Frage**: "Wer ist beteiligt? Was hängt davon ab? Wie ist es organisiert?"
|
||||
- **Speichert**:
|
||||
- **Knoten (Nodes)**: Projects, Tasks, People, Technologies, Dates
|
||||
- **Kanten (Relationships)**: `USES`, `OWNS`, `DEPENDS_ON`, `MENTIONS`, `CREATED_BY`
|
||||
- **Suche**: Cypher-Queries über Netzwerk-Topologie
|
||||
- **Antwort**: Pfade, zentrale Knoten, Abhängigkeitsanalyse
|
||||
|
||||
**Beispiel:**
|
||||
```cypher
|
||||
MATCH (tech:Technology)-[USES*]->*(service:Service),
|
||||
(tech)-[:MENTIONED_IN]->(task:Task)
|
||||
WHERE tech.name = "JWT" AND task.status = "DONE"
|
||||
RETURN tech, service, task
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Neo4j Schema für Nexa
|
||||
|
||||
### Node-Typen
|
||||
|
||||
```cypher
|
||||
(Project {name, startDate, status, priority})
|
||||
(Task {title, description, status, urgency: 1-5, context: "work"|"personal"})
|
||||
(Person {name, role, email})
|
||||
(Technology {name, category: "framework"|"language"|"tool"})
|
||||
(Topic {name, domain})
|
||||
(File {path, type, lastModified})
|
||||
(Note {id, content_hash, created_at})
|
||||
```
|
||||
|
||||
### Relationship-Typen
|
||||
|
||||
| Beziehung | Von | Zu | Beschreibung |
|
||||
|-----------|-----|-----------|-------------|
|
||||
| `OWNS` | Person | Task/Project | Wer ist verantwortlich |
|
||||
| `USES` | Task/Project | Technology | Welche Tech wird verwendet |
|
||||
| `DEPENDS_ON` | Task/Project | Task/Project | Hängt davon ab |
|
||||
| `CREATED_BY` | Note | Person | Autor einer Notiz |
|
||||
| `MENTIONS` | Note/Task | Topic/Technology | Verweist auf |
|
||||
| `RELATED_TO` | Task | Task | Zusammenhängend |
|
||||
| `CHILD_OF` | Task | Project | Task gehört zu Project |
|
||||
| `SCHEDULED_FOR` | Task | Date | Zeitliche Zuordnung |
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Synchronisierungs-Flows
|
||||
|
||||
### 1. **Memos → Neo4j** (automatisch beim Erstellen)
|
||||
|
||||
Wenn ein Memo mit `- [ ]` erstellt wird:
|
||||
|
||||
```
|
||||
Memo-Content: "Muss JWT-Middleware für Auth-Service refaktorieren"
|
||||
↓
|
||||
SAIA extrahiert Entitäten:
|
||||
- Technologie: "JWT-Middleware", "Auth-Service"
|
||||
- Kontext: "work"
|
||||
- Urgenz: 3
|
||||
↓
|
||||
Erstelle in Neo4j:
|
||||
- Task node (title="JWT-Middleware refaktorieren")
|
||||
- Technology nodes: JWT, Auth-Service
|
||||
- Relationships: Task -[USES]-> JWT, Task -[USES]-> Auth-Service
|
||||
- Task -[CREATED_BY]-> Me (Person node)
|
||||
```
|
||||
|
||||
### 2. **Obsidian → Neo4j** (#nexa:sync-obsidian)
|
||||
|
||||
Beim Sync von Obsidian-Vault:
|
||||
|
||||
```
|
||||
File: obsidian/projects/nexa-deployment.md
|
||||
↓
|
||||
Header Struktur wird zu Graph:
|
||||
- Project "Nexa Deployment"
|
||||
- Technologies: Docker, n8n, Nextcloud
|
||||
- Dependencies: "requires infrastructure setup"
|
||||
↓
|
||||
Erstelle:
|
||||
- Project nodes mit Metadata
|
||||
- Hierarchie: Project -[CONTAINS]-> Task -[USES]-> Technology
|
||||
```
|
||||
|
||||
### 3. **Nextcloud Tasks → Neo4j** (bidirektional)
|
||||
|
||||
Wenn Task in Nextcloud erstellt/gelöcht:
|
||||
|
||||
```
|
||||
Nextcloud Task: "Review JWT implementation" (in Work-List)
|
||||
↓
|
||||
n8n Trigger erkennt Änderung
|
||||
↓
|
||||
Update Neo4j:
|
||||
- Task node status = "active"
|
||||
- Task -[DEPENDS_ON]-> related Task (falls Beziehung existiert)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🤖 GraphRAG Queries - Praktische Anwendungen
|
||||
|
||||
### Query 1: "Zeige alle offenen Tasks, die JWT betreffen"
|
||||
|
||||
```cypher
|
||||
MATCH (tech:Technology {name:"JWT"}),
|
||||
(tech)-[:MENTIONED_IN|USES]-(task:Task),
|
||||
(task)-[:CHILD_OF]->(project:Project)
|
||||
WHERE task.status IN ["NEEDS-ACTION", "IN-PROGRESS"]
|
||||
RETURN project.name, task.title, task.urgency
|
||||
ORDER BY task.urgency DESC
|
||||
```
|
||||
|
||||
### Query 2: "Von welchen Tasks/Projects hängt Auth-Service ab?"
|
||||
|
||||
```cypher
|
||||
MATCH path = (dependent)-[:DEPENDS_ON*]->(service:Technology {name:"Auth-Service"})
|
||||
RETURN [x IN nodes(path) | x.name], length(path)
|
||||
ORDER BY length(path) ASC
|
||||
```
|
||||
|
||||
### Query 3: "Wer arbeitet an diesem Projekt und auf welchen Technologien?"
|
||||
|
||||
```cypher
|
||||
MATCH (person:Person)-[:OWNS]->(task:Task)-[:CHILD_OF]->(project:Project {name:"Nexa"}),
|
||||
(task)-[:USES]->(tech:Technology)
|
||||
RETURN person.name, collect(task.title), collect(tech.name)
|
||||
```
|
||||
|
||||
### Query 4: "Was ist seit gestern neu an Technologien/Themen gelernt?"
|
||||
|
||||
```cypher
|
||||
MATCH (note:Note)-[:MENTIONS]->(topic:Topic)
|
||||
WHERE note.created_at > datetime.now() - duration('P1D')
|
||||
RETURN DISTINCT topic.name, count(note) as mentions
|
||||
ORDER BY mentions DESC
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 GraphRAG-Enhanced Answer Generation
|
||||
|
||||
Wenn Nutzer `#nexa:ask` fragt:
|
||||
|
||||
```
|
||||
#nexa:ask Wie stehen die JWT-Tasks zur Deadline?
|
||||
↓
|
||||
┌─────────────┴──────────────┐
|
||||
↓ ↓
|
||||
[Qdrant] [Neo4j]
|
||||
Semantic Structural
|
||||
Search Search
|
||||
↓ ↓
|
||||
Top-3 Notes (JWT)-[DEPENDS_ON]
|
||||
mit ähnlichem → Connected Tasks
|
||||
Kontext → Critical Path
|
||||
↓ ↓
|
||||
Combine: Merge + Rank
|
||||
↓
|
||||
System Prompt (SAIA):
|
||||
"Basierend auf diesem Wissen und diesen Abhängigkeiten, beantworte:"
|
||||
↓
|
||||
Response mit:
|
||||
- Semantisch relevanter Information (Qdrant)
|
||||
- Strukturellem Kontext (Neo4j)
|
||||
- Abhängigkeitsanalyse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ n8n Integration (Phase 3)
|
||||
|
||||
### Workflow: Graph-Sync Trigger
|
||||
|
||||
```
|
||||
[Memos Webhook]
|
||||
↓
|
||||
[Parse Content]
|
||||
↓
|
||||
[SAIA: Extract Entities]
|
||||
├→ Tasks, Projects, Technologies, People
|
||||
└→ Relationships (uses, depends_on, mentions)
|
||||
↓
|
||||
[Neo4j: Create/Update Nodes & Relationships]
|
||||
↓
|
||||
[Index in Qdrant Metadata: Add graph_node_id]
|
||||
```
|
||||
|
||||
### Workflow: Question Router
|
||||
|
||||
```
|
||||
[#nexa:ask Query]
|
||||
↓
|
||||
[Parallel Search]
|
||||
├→ [Qdrant: Vector Similarity]
|
||||
├→ [Neo4j: Graph Query (auto-generated Cypher)]
|
||||
└→ [Rank & Merge Results]
|
||||
↓
|
||||
[SAIA: Generate Answer with Context]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Commands für Graph-Management
|
||||
|
||||
### #nexa:graph-status
|
||||
Zeigt Statistiken der Graph-DB
|
||||
```
|
||||
#nexa:graph-status
|
||||
```
|
||||
|
||||
Response:
|
||||
```
|
||||
📊 Neo4j Graph Status:
|
||||
- Nodes: 342 (Tasks: 145, Projects: 12, Technologies: 52, Notes: 133)
|
||||
- Relationships: 1,247
|
||||
- Density: 0.34
|
||||
- Most Connected: "JWT-Middleware" (27 connections)
|
||||
```
|
||||
|
||||
### #nexa:graph-trace [entity]
|
||||
Zeigt alle Verbindungen für eine Entität
|
||||
```
|
||||
#nexa:graph-trace JWT-Middleware
|
||||
```
|
||||
|
||||
### #nexa:graph-rebuild
|
||||
Rekonstruiert Graph aus Obsidian + Memos (einmalig, danach inkrementell)
|
||||
```
|
||||
#nexa:graph-rebuild --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Phase 3 Roadmap (Graph Integration)
|
||||
|
||||
- [ ] **3.1 Neo4j Setup**: Collection, Constraints, Indexes
|
||||
- [ ] **3.2 Graph-Sync Workflow**: Automatische Entity Extraction
|
||||
- [ ] **3.3 GraphRAG Pipeline**: Kombinierte Qdrant + Neo4j Queries
|
||||
- [ ] **3.4 Dependency Tracker**: Warnt vor zirkulären Dependencies
|
||||
- [ ] **3.5 Critical Path Analysis**: Berechnet längsten Weg im Graph
|
||||
|
||||
---
|
||||
|
||||
## 💡 Warum zwei DBs?
|
||||
|
||||
| Szenario | Qdrant | Neo4j | Kombi |
|
||||
|----------|--------|-------|-------|
|
||||
| "Welche Note war ähnlich?" | ✅ Best | ❌ | Qdrant |
|
||||
| "Welche Tasks blockieren diesen Task?" | ❌ | ✅ Best | Neo4j |
|
||||
| "Erkläre mir dieses Projekt umfassend" | ✅ Context | ✅ Structure | **Beide!** |
|
||||
| "Finde alle JWT-bezogenen offenen Work" | ✅ Semantic | ✅ Crisp | **Beide!** |
|
||||
|
||||
Kombiniert: **Vollständiges Verständnis** statt nur Suchindex oder nur Struktur.
|
||||
@@ -1,80 +0,0 @@
|
||||
📑 Projekt Nexa: Scoping & Vision
|
||||
1. Vision & Zielsetzung
|
||||
Nexa ist als "Zentrales Nervensystem" konzipiert. Das Ziel ist die Schaffung einer intelligenten Middleware zwischen Input-Quellen (Memos, E-Mail, RSS), Wissensspeichern (Obsidian, Qdrant) und Organisations-Tools (Nextcloud Calendar/Tasks).
|
||||
|
||||
Kernziele:
|
||||
Kognitive Entlastung: Nexa sortiert, filtert und schlägt vor, anstatt nur zu speichern.
|
||||
|
||||
Kontext-Trennung: Saubere, KI-gestützte Unterscheidung zwischen Privat und Arbeit.
|
||||
|
||||
Single Point of Interaction: Memos fungiert als primäre Schnittstelle ("The Voice & Ear").
|
||||
|
||||
Wissens-Synergie: Verknüpfung von flüchtigen Memos mit tiefem Wissen in Obsidian via semantischer Suche.
|
||||
|
||||
2. Abgrenzung (Scope)
|
||||
Um die Komplexität beherrschbar zu halten, wird das Projekt wie folgt abgegrenzt:
|
||||
|
||||
In-Scope (Was Nexa tut):
|
||||
Triaging: Klassifizierung von 20 Mails/Tag (Privat).
|
||||
|
||||
Task-Routing: Verteilung von Aufgaben auf die richtige Nextcloud-Liste (Work vs. Personal).
|
||||
|
||||
Scheduling: Vorschlagen von Timeboxen im Arbeitskalender.
|
||||
|
||||
Memory: Aufbau eines Langzeitgedächtnisses in Qdrant.
|
||||
|
||||
Monitoring: Aggregation kritischer IT-Alarme (Proxmox, Backrest).
|
||||
|
||||
Out-of-Scope (Was Nexa NICHT tut):
|
||||
Kein direkter Zugriff auf Arbeits-IT (E-Mail/Server).
|
||||
|
||||
Keine aktive Bearbeitung von Dokumenten (außer Metadaten-Extraktion).
|
||||
|
||||
Keine Ersetzung von Spezial-UIs (z.B. Arcane oder Portainer für Container-Management).
|
||||
|
||||
3. Phasenmodell der Entwicklung
|
||||
Der Aufbau erfolgt iterativ, um frühzeitig Mehrwert zu generieren.
|
||||
|
||||
Phase 1: Die Wirbelsäule (Konnektivität)
|
||||
Fokus: Datenfluss zwischen Memos, n8n und SAIA.
|
||||
|
||||
Meilenstein: Nexa antwortet auf Memos und kann einfache Fragen beantworten.
|
||||
|
||||
Technik: Webhooks, REST-API, LiteLLM-Proxy.
|
||||
|
||||
Phase 2: Sensorik & Klassifizierung (The Senses)
|
||||
Fokus: E-Mail-Filterung und das Routing-System.
|
||||
|
||||
Meilenstein: Nexa erkennt den Unterschied zwischen "Arbeits-Task" und "Privat-Task" ohne manuelle Tags.
|
||||
|
||||
Technik: IMAP-Node, n8n-Logic-Router, System-Prompts für Kontext-Analyse.
|
||||
|
||||
Phase 3: Das semantische Gedächtnis (The Brain)
|
||||
Fokus: Anbindung von Qdrant und Obsidian.
|
||||
|
||||
Meilenstein: RAG (Retrieval Augmented Generation). Nexa beantwortet Fragen basierend auf deinen archivierten Notizen.
|
||||
|
||||
Technik: Embeddings-Pipeline, Qdrant-Vektor-Suche.
|
||||
|
||||
Phase 4: Proaktive Motorik (The Assistant)
|
||||
Fokus: Kalender-Integration und Timeboxing.
|
||||
|
||||
Meilenstein: Nexa schlägt morgens proaktiv Fokus-Zeiten im Arbeitskalender vor.
|
||||
|
||||
Technik: Nextcloud CalDAV/CardDAV Integration, Optimierungs-Logik für freie Slots.
|
||||
|
||||
Phase 5: Physische Präsenz (Voice & Alarms)
|
||||
Fokus: Home Assistant Voice & Monitoring-Integration.
|
||||
|
||||
Meilenstein: Nexa spricht über HA-Voice und meldet kritische Systemzustände proaktiv.
|
||||
|
||||
Technik: Home Assistant API, Wyoming Protocol, ntfy/Memos-Push.
|
||||
|
||||
4. Erfolgsmetriken
|
||||
Woran merken wir, dass Nexa funktioniert?
|
||||
|
||||
Reduktion der Mail-Interaktion: Weniger Zeit in der Mail-App, mehr Fokus auf den Nexa-Digest.
|
||||
|
||||
Kalender-Füllrate: Fokus-Slots für Arbeit sind automatisch geblockt.
|
||||
|
||||
Such-Geschwindigkeit: Informationen aus alten Memos/Obsidian werden via #ask sofort gefunden.
|
||||
Reference in New Issue
Block a user