Merge pull request #1 from noheton/claude/organize-docs-deployment-7N4v2

Reorganize documentation and consolidate deployment guides
This commit is contained in:
Florian Krebs
2026-05-05 07:22:55 +02:00
committed by GitHub
26 changed files with 1611 additions and 1188 deletions
+74
View File
@@ -0,0 +1,74 @@
# Instructions for Claude (and other agents)
This file tells future automated runs what they need to know about this repo.
## Repo conventions
- **Documentation:** all docs live in `/docs/` and are numbered. Entry point is `docs/index.md`. When you add a doc, give it the next free `NN-` prefix and add a row to the index TOC.
- **Runtime artifacts:** live in `nexa-core/` (workflows, prompts, configs, scripts). Don't put `.md` documentation in there — link from `/docs/` instead.
- **Source-of-truth:** if a doc duplicates content from `nexa-core/config/*.md`, **delete the duplicate**. Single source of truth.
## Real infrastructure (verified from screenshots, May 2026)
- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9, kernel 6.17.13-4-pve, EFI).
- Hardware: **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62 GiB RAM**, **1.64 TiB disk**.
- Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
- LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
- LXC 103 backrest — backup orchestration.
- LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at `/var/nfs/shared/storage`, mounted by Proxmox at `/mnt/pve/unas`, also exposed via SMB at `smb://192.168.1.31/<share>` (Mac) / `\\192.168.1.31\<share>` (Win).
**Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` from inside LXC 104.** No `driver_opts`, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an **escape hatch only** ([docs/12 #27](./docs/12-optimization-opportunities.md#27)) for services that hit Nextcloud-style NFS issues — Nexa doesn't, so we don't use it.
**Storage-layer snapshots are NOT configured** on UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, not "the whole pool last Tuesday". Highest-leverage fix in the homelab right now ([docs/12 #38](./docs/12-optimization-opportunities.md#38)).
**UNAS layout conventions** (homelab-wide, all docker containers follow them):
- **`services/<svc>/` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout.
- `backup/<svc>/` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.**
- `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern.
- **Before deploying any Nexa container, READ AN EXISTING STACK** in **Arcane** (e.g. `karakeep` or `immich`) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`.
**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob):
- `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files.
- Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`.
- The Nextcloud appdata dir (`services/nextcloud/appdata_*`) — Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26).
- LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`.
- LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13).
- LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`.
- VM 100 haos — Home Assistant.
- **Already-running services on docker host** (don't redeploy):
- Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`).
- **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state.
- **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`.
- **Nextcloud Tasks lists & calendars** (German names, may grow over time):
- `Persönlich` → Personal context.
- `DLR` → Work context (DLR is the user's employer).
- `Einkaufsliste` → Shopping.
- `Wunschliste` → Wishes.
Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh.
- **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests.
- **Backup model is 3-2-1**: UNAS native snapshots → `s3.nuclide.systems` (warm, on-site) → encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate — see [Q20](./docs/11-open-questions.md#q20)). Always `restic`/`rclone-crypt` before upload — third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first.
- **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords.
- **Decided for Nexa** (don't re-litigate without user input):
- Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`).
- Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive.
- Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. RDF schema in docs/08 already includes `nexa:modality` / `nexa:mediaUri` / `nexa:vectorCollection` / `nexa:pendingVisualIndex`.
- Chat model: SAIA via LiteLLM virtual key.
## When working on Nexa
1. **Read `docs/index.md` first** — it's the navigator.
2. **Open questions first.** Before writing code or workflow JSON, scan `docs/11-open-questions.md`. If your task touches an unanswered Q, **stop and ask** rather than picking a default. Append new blockers to that doc as `[ ] Q-NN`.
3. **Optimization findings.** When you spot infrastructure improvements, add them to `docs/12-optimization-opportunities.md` as a numbered bullet — don't just mention them in commit messages.
4. **Never inline secrets** in workflow JSON or `.env` committed to git. Use n8n credentials, LiteLLM virtual keys, or (longer term) Vaultwarden.
5. **Keep deployment minimal.** The default answer to *"do we need a new container?"* is **no** — the existing stack covers most needs.
## Branch policy
- This branch is `claude/organize-docs-deployment-7N4v2`. Push only here unless told otherwise.
- New work for an unrelated feature → new branch under `claude/<topic>`.
## Quick links
- [docs/index.md](./docs/index.md)
- [docs/09-deployment.md](./docs/09-deployment.md) — most-touched file during bring-up
- [docs/11-open-questions.md](./docs/11-open-questions.md) — read **before** assuming defaults
+17 -1
View File
@@ -1 +1,17 @@
# nexa
# nexa
**Neural Nexus for Information & Automation** — central nervous system for a personal IT setup.
> Memos = voice & ear · n8n = reflexes · SAIA (LiteLLM) = brain · Qdrant (+ optional graph DB) = memory · Nextcloud = hands.
## Start here
📖 **[docs/index.md](./docs/index.md)** — TOC, reading paths, repo layout.
## Repository layout
```
nexa/
├── docs/ ← all documentation, numbered for reading order
└── nexa-core/ ← runtime: n8n workflows, configs, prompts, scripts
```
+34
View File
@@ -0,0 +1,34 @@
# 01 — Vision & Scope
## Vision
Nexa is the **central nervous system** of a personal IT setup: an intelligent middleware sitting between input sources (Memos, e-mail, RSS, Karakeep, Bluesky), knowledge stores (Obsidian, Qdrant, Nextcloud) and organization tools (Nextcloud Calendar/Tasks).
### Core goals
- **Cognitive offload** — Nexa sorts, filters and proposes; it doesn't just store.
- **Context separation** — clean AI-driven split between *Work* and *Personal*.
- **Single point of interaction** — Memos is the primary surface ("voice & ear").
- **Knowledge synergy** — link ephemeral memos to deep notes via semantic search.
## Scope
### In-scope
- Triaging ~20 personal e-mails per day.
- Routing tasks to the right Nextcloud list (Work vs. Personal).
- Suggesting time-boxes in the work calendar.
- Long-term memory in Qdrant (and optionally a graph DB).
- Aggregating critical IT alerts (Proxmox, Backrest) into a Memos system feed.
### Out-of-scope
- Direct access to work IT (work mail, work servers).
- Active document editing (metadata extraction only).
- Replacing specialist UIs (Arcane / Dozzle / Portainer for containers).
## Success metrics
- **Reduced mail interaction** — less time in the mail app, more on the Nexa digest.
- **Calendar fill-rate** — focus slots auto-blocked on the work calendar.
- **Search latency** — old memos / Obsidian notes resurface instantly via `#nexa:ask`.
+62
View File
@@ -0,0 +1,62 @@
# 02 — Roadmap & Phases
Iterative build-out, value-first. Each phase is shippable on its own.
## Phase 1 — The Spine (connectivity)
**Focus:** datapath between Memos, n8n and SAIA.
**Milestone:** Nexa replies on Memos and answers simple questions.
- [ ] **1.1 Memos ↔ n8n bridge** — webhook ingress, content filter, comment-back.
- [ ] **1.2 SAIA integration** — call LiteLLM proxy from n8n; chat via Memos comments.
- [ ] **1.3 Git-sync** — periodic export of n8n workflows into this repo.
## Phase 2 — Senses (input channels)
**Focus:** e-mail filter and the Work-vs-Personal router.
**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
- [ ] **2.4 On-demand web fetch**`#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
## Phase 3 — Memory (Qdrant & RAG)
**Focus:** Qdrant + graph layer for retrieval-augmented answers.
**Milestone:** RAG works — Nexa answers from archived notes; images captured today are queued for visual indexing later.
- [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet.
- [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32).
- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. **Deferred** until [Q12 + Q20](./11-open-questions.md#backups-deferred) are decided. Until then, snapshots stay on UNAS at `backup/nexa/snapshots/qdrant/<date>/`; UNAS RAID 6 + native pool snapshots ([12/#38](./12-optimization-opportunities.md#38)) cover the data-loss scenarios that matter most for Phase 3.1.
- [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md).
## Phase 4 — Motor (organization & action)
**Focus:** calendar integration and time-boxing.
**Milestone:** Nexa proactively proposes morning focus slots in the work calendar.
- [ ] **4.1 Nextcloud Tasks sync** — bidirectional status sync (Memos ↔ NC Tasks).
- [ ] **4.2 Time-boxing agent** — calendar-aware slot suggestions.
- [ ] **4.3 Karakeep curation** — automated review cycle for saved links.
## Phase 5 — Daily integration (polish)
**Focus:** Home Assistant Voice and monitoring.
**Milestone:** Nexa speaks via HA-Voice and surfaces critical system states proactively.
- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
## Phase 6 — Nexa as homelab steward
**Focus:** Nexa actively maintains the homelab inventory instead of being told manually.
**Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo.
- [ ] **6.1 Inventory sync** — Nexa polls Arcane / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …).
- [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #3537). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*.
- [ ] **6.3 Steward commands**`#nexa:retire <svc>` (tar to `backup/`, stop, archive the stack), `#nexa:document <svc>` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed).
- [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface).
This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) — most of those items become semi-automatic once 6.16.3 ship.
+64
View File
@@ -0,0 +1,64 @@
# 03 — Architecture Overview
A one-page mental model. For details follow the cross-links.
## Topology
```
┌────────────────┐
voice / typing ──────▶│ Memos │◀──── Nexa replies as comments
│ (interface) │
└───────┬────────┘
│ webhook (- [ ] / #nexa:*)
┌────────────────┐
IMAP / RSS / NC ────▶│ n8n │◀──── workflows live in
Karakeep / Bluesky │ (logic) │ ./nexa-core/n8n-workflows
└───┬────────┬───┘
classify ▲ │ │ ▲ retrieve
│ ▼ ▼ │
┌──────────┐ ┌──────────┐
│ SAIA │ │ Qdrant │
│ LiteLLM │ │ vector │
└──────────┘ └────┬─────┘
┌──────────┐
│ GraphDB │ Ontotext, SPARQL
│ (RDF) │ Phase 3.4
└──────────┘
▼ writes
┌──────────────────────────────┐
│ Nextcloud (Tasks, Calendar, │
│ Mail, Files / Obsidian) │
└──────────────────────────────┘
```
## Components
| Component | Role | Where it runs (today) |
|-----------|------|-----------------------|
| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` |
| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` |
| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) |
| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) |
| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-ontotext-graphdb-phase-34) |
| **TEI** (HF text-embeddings-inference) | Self-hosted text embeddings, `BAAI/bge-m3`, Phase 3.1 | docker host LXC 104, CPU only — swapped for `infinity` in Phase 3.2 to add `jina-clip-v2` |
| **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` |
| **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` |
| **Backrest** | Backup orchestration | LXC 103 |
| **Zoraxy** | Reverse proxy + TLS | LXC 108 (`192.168.1.4:8000`) |
| **AdGuard DNS** | Internal name resolution | LXC 102 |
| **Home Assistant** | Voice + house automation | VM 100 (HAOS) |
## Two-pillar memory
- **Qdrant** answers *"what is similar / relevant?"* (cosine search over embeddings).
- **Ontotext GraphDB** answers *"who, what depends on what, how is it structured?"* (SPARQL over RDF).
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. See [08 — GraphRAG architecture](./08-graphrag-architecture.md).
## Dual-context routing
Every input is classified `work` or `personal` before any side effect (task creation, calendar write). See [04 — Integration matrix](./04-integration-matrix.md) and [06 — Classification logic](./06-classification-logic.md).
@@ -1,4 +1,4 @@
# 🔗 Nexa Integrations-Matrix (V2)
# 04 — Integrations-Matrix
Diese Matrix definiert die logische Trennung zwischen privaten und beruflichen Datenströmen sowie die Anbindung der Infrastruktur.
@@ -10,17 +10,15 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari
### A. Bereich: ARBEIT (Work)
* **Sichtbarkeit:** Eingeschränkt (kein Zugriff auf Arbeits-E-Mails oder Firmen-Server).
* **Datenquellen:**
* Separate Nextcloud Task-Liste (`Work_Tasks`).
* Separater Nextcloud Kalender (`Work_Calendar`).
* Nextcloud Task-Liste **`DLR`** (sowie der gleichnamige Kalender).
* Memos mit Tag `#work` oder semantischer Erkennung.
* **Nexa-Fokus:** Timeblocking, Fokus-Zeiten, Vorbereitung von Meetings basierend auf Obsidian-Notizen.
### B. Bereich: PRIVAT (Personal)
* **Sichtbarkeit:** Vollständig.
* **Datenquellen:**
* Nextcloud Task-Liste (`Personal_Tasks`).
* Haupt-Kalender.
* E-Mails (IMAP - 20 Mails/Tag).
* Nextcloud Task-Liste **`Persönlich`** und gleichnamiger Kalender.
* Nextcloud Mail (`fkrebs@nucli.de`, einziges Konto). Ordner `Waiting` ist ein manuelles Signal — Mails dort werden vom Digest ausgeschlossen.
* Bluesky, RSS, Karakeep.
* **Nexa-Fokus:** Automatisierung des Alltags, Kuratierung von Wissen, E-Mail-Management.
@@ -59,8 +57,10 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari
* Wichtige private Termine werden in den privaten Kalender extrahiert.
* Newsletter landen als Zusammenfassung in Memos.
### Wissens-Management (Obsidian & Qdrant)
* Obsidian-Notizen (Arbeit & Privat) werden in Qdrant indiziert.
### Wissens-Management (Obsidian via Nextcloud & Qdrant)
* Der Obsidian-Vault liegt im Nextcloud-Ordner `Notizen/` (Multi-Device-Sync via Nextcloud-Client). Nexa liest **read-only über WebDAV** (`/remote.php/dav/files/<user>/Notizen/`) — kein Mount, kein zweiter Sync-Mechanismus.
* Markdown-Dateien (`Notizen/**/*.md`) werden in Qdrant (`nexa_knowledge_text`) indiziert; Bilder unter `assets/` landen im Phase-3.2-Visual-Queue (siehe [docs/11/Q4](./11-open-questions.md)).
* Plugin-Internals (`.copilot/`, `.smart-env/`, `.caldav-sync/`, `Templates/`, `BMO/`, `Excalidraw/`) sind explizit von der Indizierung ausgeschlossen.
* Nexa nutzt dieses Wissen, um Arbeits-Tasks besser zu beschreiben, auch wenn sie keinen Zugriff auf die Arbeits-E-Mails hat.
---
+176
View File
@@ -0,0 +1,176 @@
# 05 — Command System (`#nexa:*`)
Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag:
## 🔧 Konfigurations-Kommandos
### #nexa:config
- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud Listen/Kalender, Qdrant Collections, LiteLLM-Modelle, Mail-Konto).
- **Läuft auch automatisch**:
- Täglich (cron in n8n) — neue Listen / Kalender werden ohne manuelles Eingreifen aufgenommen.
- Bei Cache-Miss — wenn ein Workflow eine Liste/ID nicht mehr findet, läuft Discovery einmal, dann Retry.
- **Antwort**: Postet Report mit den aktuellen Konfigurationen als Memos-Kommentar.
- **Speichert**: Konfiguration in der Qdrant `_config` Namespace (nicht in `.env`).
```
#nexa:config
```
**Beispiel-Response:**
```
✅ Autodiscovery abgeschlossen (2026-05-04T23:59):
- Nextcloud Listen: Persönlich (id=14), DLR (id=dlr-1), Einkaufsliste (id=…), Wunschliste (id=…)
- Nextcloud Kalender: Persönlich, DLR, Einkaufsliste, Wunschliste
- Mail-Konto: fkrebs@nucli.de (Posteingang, Archiv, Junk, Waiting)
- Qdrant Collection: nexa_knowledge_text (1024 dims, Cosine, 0 Punkte)
- LiteLLM-Modelle: nexa-chat, nexa-embed
```
### #nexa:status
- **Beschreibung**: Zeigt den aktuellen System-Status
- **Antwort**: Uptime, verbundene Systeme, Memory-Stats
```
#nexa:status
```
### #nexa:sync-obsidian
- **Beschreibung**: Triggert sofortige Synchronisierung mit Obsidian Vault
- **Optional Parameter**: `--vault=/path/to/vault` oder `--force` für Neuindexierung
```
#nexa:sync-obsidian --force
```
## 📊 Memory & RAG Kommandos
### #nexa:ask [question]
- **Beschreibung**: Semantische Suche in Qdrant (RAG) basierend auf Obsidian + Memos
- **Antwort**: Top-3 Treffer mit Quellen
```
#nexa:ask Wie implementierten wir das JWT-Middleware-Pattern?
```
### #nexa:learn [topic]
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
```
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
```
### #nexa:forget [query-or-iri]
- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
### #nexa:retain [source_type] [days]
- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
### #nexa:ask [question] [--web]
- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
## 🎯 Workflow-Kommandos
### #nexa:route-test [text]
- **Beschreibung**: Testet die Klassifizierung (Work/Personal) ohne Task zu erstellen
```
#nexa:route-test Muss morgen die Präsentation für den Client fertigstellen
```
**Response:**
```
💬 Klassifizierung (Test-Mode):
- Kontext: WORK
- Vertrauen: 0.95
- Begründung: "Client-Präsentation → professioneller Kontext"
```
### #nexa:email-digest
- **Beschreibung**: Triggert sofortige Email-Zusammenfassung (sonst tägl. 7 Uhr)
```
#nexa:email-digest
```
### #nexa:digest
- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
3. Drift-Report aus Phase 6 (sobald aktiv).
4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
| Mal | Wartezeit bis zur nächsten Frage |
|-----|----------------------------------|
| 1 → 2 | 3 Tage |
| 2 → 3 | 7 Tage |
| 3 → 4 | 21 Tage |
| ≥ 5 | 60 Tage |
Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
- `nextAskAt <= today`.
- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
### #nexa:remind <question-id>
- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
### #nexa:answered <question-id> <text>
- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
## 🔐 Admin-Kommandos
### #nexa:reset-config
- **Beschreibung**: Löscht gespeicherte Konfiguration, triggert neuen Autodiscovery
- **Warnung**: Alle benutzerdefinierten Settings werden vergessen
```
#nexa:reset-config
```
### #nexa:export-state
- **Beschreibung**: Exportiert aktuellen State als JSON (für Backups)
```
#nexa:export-state
```
---
## 📝 Implementierung in n8n
Ein **Command Parser** läuft immer mit:
1. **Memos Webhook** empfängt alle Memos
2. **Regex Check**: Sucht nach `#nexa:command`
3. **Router**: Versendet an entsprechenden n8n-Workflow
4. **Antwort**: Postet Reply als Kommentar/Edit
**Command Parser Regex:**
```
^#nexa:(\w+)(?:\s+([^\n]*?))?(?:$|\s*--)
```
Extrahiert: `[command, parameters]`
---
## 🎛️ Dynamische Konfigurationsspeicherung
Statt .env zu editieren:
1. **First Run**: `#nexa:config` speichert zu lokalen Metadata
2. **Speich-Ziel**:
- **Primary**: Qdrant Metadata (als `_config` Namespace)
- **Fallback**: `nexa-core/config/runtime_config.json` (gitignored)
- **Notfall**: .env (nur initial)
3. **Reload-Logik**: Bei jedem Workflow-Start werden Settings aus Qdrant geladen
Das macht Nexa vollständig selbstständig nach dem initialem Setup!
@@ -1,4 +1,4 @@
# Nexa Klassifizierungs-Logik
# 06 — Klassifizierungs-Logik
Dieser Fragebogen dient der Feinabstimmung von SAIA, um Tasks korrekt zu routen.
+26
View File
@@ -0,0 +1,26 @@
# 07 — Workflow Spec: Phase-2 Task-Router
## 1. Trigger
* **Typ:** Webhook (Memos)
* **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`.
## 2. Intelligence Node (SAIA)
* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: `{ "context": "work" | "personal" | "shopping" | "wishes", "summary": "string", "urgency": 1-5 }`."
* **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.
## 3. List Resolver (vor dem Switch)
* Lese den Cache `nexa._config.lists` aus Qdrant: `{ "Persönlich": <id>, "DLR": <id>, "Einkaufsliste": <id>, "Wunschliste": <id>, discovered_at: <ts> }`.
* Bei Cache-Miss oder 404 vom nachfolgenden Create-Task → triggere `#nexa:config` (Discovery), update den Cache, retry **einmal**.
## 4. Switch Node (4-Wege)
* **Route 1:** `context == "work"` → Nextcloud Create Task auf Liste `DLR`.
* **Route 2:** `context == "personal"` → Nextcloud Create Task auf Liste `Persönlich`.
* **Route 3:** `context == "shopping"` → Nextcloud Create Task auf Liste `Einkaufsliste`.
* **Route 4:** `context == "wishes"` → Nextcloud Create Task auf Liste `Wunschliste`.
* **Default (Konfidenz < 0.7):** `Persönlich` (siehe [06](./06-classification-logic.md)).
## 5. Feedback Loop
* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
## 6. Listen-Diskoverability
Listennamen sind nicht hardgecodet — die Resolver-Step liest sie aus dem Runtime-Config-Cache. Wenn der User in Nextcloud eine neue Liste anlegt (z. B. `Reisen`), erscheint sie nach dem nächsten geplanten `#nexa:config`-Lauf (täglich) automatisch als Routing-Ziel — die System-Prompt der Intelligence Node wird zusammen mit den verfügbaren Listen versorgt, sodass SAIA neue Kontexte vorschlagen kann.
+286
View File
@@ -0,0 +1,286 @@
# 08 — GraphRAG: Structural Knowledge & Relations
Decision: graph layer = **Ontotext GraphDB** with **SPARQL** (resolved in [11/Q1](./11-open-questions.md)).
Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.
---
## Two-pillar memory
| Pillar | Question it answers | Backed by |
|--------|---------------------|-----------|
| **Qdrant** (vectors) | *"What is similar / relevant?"* | Cosine search over embeddings |
| **GraphDB** (RDF) | *"What is connected? What depends on what? Who is involved?"* | SPARQL over a typed graph |
Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer.
---
## RDF schema
Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.
```turtle
@prefix nexa: <https://nuclide.systems/nexa/ontology#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix prov: <http://www.w3.org/ns/prov#> .
# Classes
nexa:Project a rdfs:Class .
nexa:Task a rdfs:Class .
nexa:Person a rdfs:Class .
nexa:Technology a rdfs:Class .
nexa:Topic a rdfs:Class .
nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests
nexa:File a rdfs:Class .
# Properties
nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .
nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .
nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .
nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .
nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .
nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .
# Datatype properties
nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done"
nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal"
nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 15
nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup
# Cross-pillar / multimodality
nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image"
nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://…
nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual"
nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID
nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them
```
`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way.
`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases.
---
## Sync flows
### 1. Memos → GraphDB (real-time)
```
Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren"
▼ SAIA extracts entities + relations as JSON
│ { tasks: [{title, urgency}], technologies: [...],
│ relations: [{type:"uses", from:..., to:...}] }
▼ n8n turns JSON into a SPARQL UPDATE
└──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge"
```
### 2. Obsidian → GraphDB (`#nexa:sync-obsidian`)
For each Obsidian note: parse front-matter + headings → emit `nexa:Project`, `nexa:Task`, `nexa:Note` triples; `nexa:mentions` for `[[wikilinks]]`.
### 3. Nextcloud Tasks ↔ GraphDB (bidirectional)
n8n trigger on Nextcloud CalDAV/Tasks change → `INSERT/DELETE DATA` to keep `nexa:status` and `nexa:scheduledFor` in sync.
---
## Example SPARQL queries
### Q1 — All open tasks involving JWT, by urgency
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?urgency ?projectName
WHERE {
?tech rdfs:label "JWT" .
?task nexa:uses ?tech ;
rdfs:label ?taskTitle ;
nexa:status ?status ;
nexa:urgency ?urgency .
FILTER (?status IN ("needs-action", "in-progress"))
OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }
}
ORDER BY DESC(?urgency)
```
### Q2 — What does Auth-Service transitively depend on?
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT DISTINCT ?dep ?label
WHERE {
?root rdfs:label "Auth-Service" .
?root nexa:dependsOn+ ?dep .
?dep rdfs:label ?label .
}
```
(`+` is SPARQL property-paths — transitive closure, free.)
### Q3 — Topics with the most note-mentions in the last day
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?topic (COUNT(?note) AS ?n)
WHERE {
?note a nexa:Note ;
prov:generatedAtTime ?ts ;
nexa:mentions ?topic .
FILTER (?ts > NOW() - "P1D"^^xsd:duration)
}
GROUP BY ?topic
ORDER BY DESC(?n)
LIMIT 10
```
### Q4 — Cross-pillar: "find vectors for tasks blocking project X"
```sparql
PREFIX nexa: <https://nuclide.systems/nexa/ontology#>
SELECT ?taskTitle ?vectorId
WHERE {
?proj rdfs:label "Nexa" .
?task nexa:childOf ?proj ;
nexa:status "in-progress" ;
nexa:vectorId ?vectorId ;
rdfs:label ?taskTitle .
}
```
n8n then takes each `?vectorId`, fetches the embedding from Qdrant, and runs a "more like this" search for richer context.
---
## GraphRAG answer pipeline (`#nexa:ask`)
```
#nexa:ask <question>
┌──────┴──────┐
▼ ▼
[Qdrant] [GraphDB]
semantic structural
top-k SPARQL — auto-generated
notes paths / dependencies
│ │
└──────┬──────┘
merge + rank
SAIA prompt:
"Given these passages and these relations, answer …"
comment under the original memo
```
Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a `SELECT`/`CONSTRUCT` query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.
---
## n8n integration sketch
### Workflow: Graph-Sync Trigger
```
[Memos Webhook]
[Parse Content]
[SAIA: Extract entities + relations as JSON]
[Build SPARQL UPDATE INSERT DATA { ... }]
[HTTP POST → /repositories/nexa_knowledge/statements]
[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]
```
### Workflow: Question Router
```
[#nexa:ask Query]
┌─┴────────────────┐
▼ ▼
[SAIA: NL → SPARQL] [Qdrant: kNN]
│ │
[POST → SPARQL endpoint]
│ │
└──────┬───────────┘
rank + merge → SAIA answer
```
---
## Graph-management commands
### `#nexa:graph-status`
Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL `SELECT (COUNT)`.
### `#nexa:graph-trace [entity]`
Returns the 1-hop (and optionally 2-hop) neighbourhood — a `DESCRIBE <iri>` plus a templated outgoing/incoming query.
### `#nexa:graph-rebuild`
Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https://nuclide.systems/nexa/runtime>` followed by the import workflow.
---
## Why two stores
| Scenario | Qdrant | GraphDB | Best |
|----------|--------|---------|------|
| "Which note was similar to this one?" | ✅ | ❌ | Qdrant |
| "What blocks this task?" | ❌ | ✅ | GraphDB |
| "Explain this project" | ✅ context | ✅ structure | both |
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
Combined: **complete understanding** rather than a search index *or* a structure index.
---
## Memory sources & retention
Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
| `source_type` | Where it comes from | Default TTL | Rationale |
|---------------|--------------------|-------------|-----------|
| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
### Web search loop
`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
### Manual overrides
- `#nexa:learn <text> --permanent` overrides the default TTL.
- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).
+326
View File
@@ -0,0 +1,326 @@
# 09 — Deployment
Pragmatic deployment guide that **assumes the existing homelab** and adds only what's missing.
## What's already running (no action required)
Surveyed from Homepage / Dozzle / Proxmox / Zoraxy:
| Service | Host / port | URL |
|---------|-------------|-----|
| Memos | docker LXC 104 → `:5230` | `https://memos.nuclide.systems` |
| n8n | docker LXC 104 → `:5678` | `https://n8n.nuclide.systems` |
| LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) |
| Nextcloud | LXC 105 | `https://nc.nuclide.systems` |
| ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` |
| Karakeep | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` (legacy host alias kept for compatibility) |
| Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` |
| Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` |
| Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` |
| Backrest | LXC 103 | (internal) |
| AdGuard DNS | LXC 102 | (internal) |
| Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems |
| `qdrant_scientific` (existing) | docker LXC 104 | **reused** — Nexa uses `nexa_*` collections in this instance |
The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**.
## What's missing for Nexa
1. **Qdrant collection** for Nexa (`nexa_knowledge`) inside the existing `qdrant_scientific` instance — vector dim follows Q15 (1024 for `bge-m3`, 768 for `nomic-embed-text`).
2. **TEI** (HF text-embeddings-inference) on the docker host for self-hosted embeddings (LiteLLM key is **not** authorised for OpenAI embeddings — see [11/Q3+Q15](./11-open-questions.md)). Lighter than Ollama: single Rust binary, ~500 MB image, no LLM runtime.
3. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n.
4. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`).
5. **Memos webhook → n8n** wired through the Memos config.
6. **LiteLLM virtual key** for the `nexa` user with chat-only access (no embeddings — handled by Ollama).
7. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied.
8. **(Phase 3.4) Ontotext GraphDB** for the SPARQL pillar — see add-on at the bottom of this doc.
---
## Step 1 — Secrets
Copy `nexa-core/.env.example``nexa-core/.env` and fill **only** the secrets:
```bash
cd nexa-core
cp .env.example .env
$EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY
```
The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead.
## Step 2 — Qdrant collection (`nexa_knowledge_text`)
Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring.
```bash
# adjust QDRANT_HOST in .env first
source nexa-core/.env
# create the text collection from the schema file
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \
-H "Content-Type: application/json" \
-H "api-key: $QDRANT_API_KEY" \
-d @nexa-core/config/qdrant_schema.json
```
The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)):
| Field | Why it's there now |
|-------|--------------------|
| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. |
| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. |
| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. |
| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. |
| `content_hash` | de-dup. |
| `context` | `work` / `personal`. |
> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container.
> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.
### Image attachments today (queue them)
Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later:
- Memo with an image → text body still goes into `nexa_knowledge_text`.
- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`).
- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection.
This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself.
## Step 3 — Self-hosted embeddings (TEI)
Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.
The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle.
```yaml
# Nexa stack — paste into Arcane.
# Storage convention matches the rest of the homelab (verified against
# the running Karakeep stack, Q19): host bind-mount of
# /mnt/pve/unas/services/<svc>/<vol>. No volume-driver, no CIFS, no
# credentials in the compose — the LXC's NFS mount is already there.
services:
nexa-embed:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
container_name: nexa-embed
restart: unless-stopped
command: ["--model-id", "BAAI/bge-m3"]
ports:
- "127.0.0.1:8080:80"
volumes:
- /mnt/pve/unas/services/nexa/tei-cache:/data
env_file:
- .env
networks: {}
```
Pre-deploy step on the docker LXC (one-time):
```bash
mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb}
mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}
```
Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into `/mnt/pve/unas/services/nexa/tei-cache/`; subsequent restarts are instant.
Secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, `QDRANT_API_KEY`, `NC_APP_PASSWORD`) go in the stack's `.env` next to the compose — same pattern Karakeep uses (`env_file: .env`). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term ([12/#11](./12-optimization-opportunities.md#11)) but isn't required for the first cut.
> **Why bind-mount and not SMB?** Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's "had it with Nextcloud" experience. The Karakeep stack confirms the actual convention is the simpler one: **host bind-mount of the LXC's existing `/mnt/pve/unas` NFS mount**. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to `www-data`, which fails against `root_squash` exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in [12/#27](./12-optimization-opportunities.md#27) only as an escape hatch *if* a future service hits Nextcloud-style issues — Nexa doesn't, so we don't use it.
Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter:
- model name: `nexa-embed`
- provider: `openai`
- model: `bge-m3`
- api_base: `http://nexa-embed:80/v1`
- api_key: any non-empty string (TEI ignores it)
Now n8n only ever talks to LiteLLM and the model is swappable without touching workflows.
## Step 4 — LiteLLM virtual key
In the LiteLLM admin UI (`ai.nuclide.systems`):
1. Create user `nexa`.
2. Issue a virtual key with access to:
- one chat model (already-available model from your SAIA gateway).
- the `nexa-embed` model from Step 3.
3. Paste the key into `SAIA_API_KEY` in `.env`.
## Step 4 — n8n workflows
Import the JSON exports — credentials are filled inside n8n, not in the JSON:
```bash
# n8n personal access token from the n8n UI: Settings → API
N8N_URL=https://n8n.nuclide.systems
N8N_TOKEN=... # from the n8n UI
for f in nexa-core/n8n-workflows/phase-1/*.json \
nexa-core/n8n-workflows/phase-2/*.json; do
curl -X POST "$N8N_URL/api/v1/workflows" \
-H "X-N8N-API-KEY: $N8N_TOKEN" \
-H "Content-Type: application/json" \
--data-binary "@$f"
done
```
Inside n8n, attach credentials to the imported nodes:
- **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY`
- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` (chat + `nexa-embed`)
- **Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under `Notizen/`)** → one app password (`$NC_APP_PASSWORD`), reused across all three node types
- **Qdrant** → header `api-key: $QDRANT_API_KEY`
Activate each workflow individually after smoke-test.
## Step 5 — Memos webhook
In the Memos admin UI, set the webhook URL to the production address of the discovery workflow:
```
https://n8n.nuclide.systems/webhook/memos
```
The same URL is the one the workflow exposes; verify with:
```bash
curl -i https://n8n.nuclide.systems/webhook/memos
# expect 200 / 405, never 404
```
## Step 6 — Bootstrap commands via Memos
Create a memo with body `#nexa:config` — the discovery workflow:
1. Lists Nextcloud Tasks lists, looking for `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (and any new lists added later — see [05](./05-command-system.md)). Caches `name → id`.
2. Lists Nextcloud Calendars by the same names; caches IDs.
3. Verifies the Nextcloud Mail account `fkrebs@nucli.de` and the folder list (`Posteingang`, `Archiv`, `Junk`, `Waiting`).
4. Counts existing Qdrant points in `nexa_knowledge_text`.
5. Verifies LiteLLM reachability + lists available models for the Nexa virtual key.
6. Replies as a comment with a **runtime-config snapshot** that's stored in the Qdrant `_config` namespace (and mirrored as `nexa-core/config/runtime_config.json`, gitignored).
The same workflow is also triggered by:
- A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention).
- Cache-miss in the task-router — if a list ID 404s, the router fires `#nexa:config` once and retries.
After this point, `.env` is read-once. Subsequent runs read config from Qdrant.
## Step 7 — Smoke tests
```bash
# (1) Memos round-trip — should produce a comment within ~5 s
curl -X POST https://memos.nuclide.systems/api/v1/memos \
-H "Authorization: Bearer $MEMOS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"content":"- [ ] testing the router #nexa"}'
# (2) Classification dry-run
curl -X POST https://memos.nuclide.systems/api/v1/memos \
-H "Authorization: Bearer $MEMOS_API_KEY" \
-d '{"content":"#nexa:route-test buy milk"}'
# (3) RAG test (requires at least one indexed memo/note)
curl -X POST https://memos.nuclide.systems/api/v1/memos \
-H "Authorization: Bearer $MEMOS_API_KEY" \
-d '{"content":"#nexa:ask what is the goal of nexa?"}'
```
## Step 8 — Reverse proxy
Already done — Zoraxy at `192.168.1.4:8000` terminates TLS for `*.nuclide.systems` and forwards to docker LXC 104 (`192.168.1.40`). **No new entry is required for Nexa**: every service Nexa talks to already has a host entry.
## Step 9 — Backups
Already covered by Backrest (LXC 103). Add:
- **n8n workflows** → `nexa-core/scripts/backup_workflows.sh` (already present) into a Backrest schedule.
- **Qdrant snapshots** → schedule a daily `POST /collections/nexa_knowledge/snapshots` and rsync to S3 (`s3.nuclide.systems`). Add as a Backrest pre-hook on the docker host.
For deeper detail: [10 — Operations](./10-operations.md).
---
## Phase add-on: Ontotext GraphDB (Phase 3.4)
Defer until 3.13.3 ship.
```yaml
# nexa-core/docker-compose.graph.yml
services:
graphdb:
image: ontotext/graphdb:10.7.0
container_name: nexa-graphdb
ports: ["127.0.0.1:7200:7200"]
environment:
GDB_JAVA_OPTS: "-Xmx4g -Xms1g"
volumes:
- ./data/graphdb:/opt/graphdb/home
restart: unless-stopped
```
After first start, create the repository (one-time):
```bash
curl -X POST http://localhost:7200/rest/repositories \
-H 'Content-Type: application/json' \
-d '{
"id": "nexa_knowledge",
"title": "Nexa Knowledge Graph",
"type": "graphdb",
"params": {
"ruleset": {"value": "rdfsplus-optimized"},
"baseURL": {"value": "https://nuclide.systems/nexa/"}
}
}'
```
Optional Zoraxy entry `graph.nuclide.systems``192.168.1.40:7200` if you want the SPARQL Workbench in a browser; otherwise n8n talks to it on the docker network at `http://nexa-graphdb:7200`.
For schema and example queries: [08-graphrag-architecture](./08-graphrag-architecture.md).
---
## Phase add-on: visual collection (Phase 3.2)
Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`.
```bash
# (1) replace TEI with infinity (or run alongside) for CLIP-family support
docker rm -f nexa-embed
docker run -d --name nexa-embed \
--restart unless-stopped \
-p 127.0.0.1:8080:80 \
-v infinity-data:/app/.cache \
michaelf34/infinity:latest \
v2 \
--model-id BAAI/bge-m3 \
--model-id jinaai/jina-clip-v2 \
--port 80
# (2) create the visual collection
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \
-H "Content-Type: application/json" \
-H "api-key: $QDRANT_API_KEY" \
-d @nexa-core/config/qdrant_schema_visual.json
# (3) register the second model in LiteLLM as `nexa-embed-visual`
# (same OpenAI-compatible route, different model id)
# (4) backfill queued images:
# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }
# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,
# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.
```
n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt.
---
## Step-back / rollback
- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops.
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`.
- **Re-discover** — `#nexa:reset-config` then `#nexa:config`.
+101
View File
@@ -0,0 +1,101 @@
# 10 — Operations
Day-2 concerns. Backups, monitoring, troubleshooting.
## Backups
**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)):
1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable).
2. **Warm tier**`s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103).
3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext.
Nexa-specific items inside this pipeline:
| What | How | Frequency |
|------|-----|-----------|
| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) |
| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily |
| Memos DB | Backrest snapshot of Memos data dir | already covered |
| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered |
| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
## Monitoring
| Signal | Source | Sink |
|--------|--------|------|
| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. **Skip names matching `octoprint*`** — that container is intentionally powered down most of the time. | Memos system feed via ntfy |
| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic |
| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest |
| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest |
| Backrest run status | Backrest webhook | ntfy |
A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo.
## Troubleshooting
### Memos webhook not firing
```bash
# is the Memos webhook config still pointing at n8n?
curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \
https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'
# does n8n still expose the path?
curl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404
```
If 404 → workflow is inactive in n8n. Activate.
### LiteLLM 401 / 429
- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential.
- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.
### Qdrant `nexa_knowledge` empty
```bash
# is the indexing workflow active?
docker logs nexa-n8n 2>&1 | grep memos_bridge | tail
# manual index test
curl -X POST https://memos.nuclide.systems/api/v1/memos \
-H "Authorization: Bearer $MEMOS_API_KEY" \
-d '{"content":"manual probe #nexa"}'
# point count
curl -s -H "api-key: $QDRANT_API_KEY" \
$QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'
```
### Wrong list routing (Work vs Personal)
1. Run `#nexa:route-test <text>` and check the confidence value.
2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)).
3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change.
### "Where did Nexa store this?"
```bash
# state dump
curl -X POST https://memos.nuclide.systems/api/v1/memos \
-d '{"content":"#nexa:export-state"}'
# returns runtime_config + counters as a JSON memo
```
## Log locations
| Component | Where |
|-----------|-------|
| Memos | `docker logs nexa-memos` (LXC 104) |
| n8n | `docker logs nexa-n8n` |
| LiteLLM | `docker logs litellm` |
| Qdrant | `docker logs qdrant_scientific` |
| Zoraxy | LXC 108 web UI → Statistical Analysis |
| Backrest | LXC 103 web UI |
For a one-shot dump:
```bash
ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log
```
+73
View File
@@ -0,0 +1,73 @@
# 11 — Open Questions (user-info-required)
Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide.
## Resolved
- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly.
- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container.
- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below.
- [x] **Q5 — Display name → Karakeep.** The Zoraxy host alias `hoarder.nuclide.systems` is legacy — keep it for compatibility, but all docs, prompts and new workflow nodes use "Karakeep".
- [x] **Q13 — Octoprint container is *intentionally* temporary.** Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching `octoprint*` (or any container tagged `proxmox-he 3d-printing`).
- [x] **Q14 — Homepage Zoraxy widget → no such widget.** Config error in `homepage/services.yaml`; cosmetic, not Nexa-related.
- [x] **Q15 — Embeddings staging plan → A now, C prepared.**
- **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus.
- **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16).
- All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json).
- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme.
- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich.
- [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant).
- [x] **Q19 — Storage convention is plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via `env_file: .env` next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn ([12/#27](./12-optimization-opportunities.md#27)) — it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to `www-data` and fails on `root_squash`-style exports; that doesn't apply to normal containers). Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
- [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files/<user>/Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index):
- `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache.
- `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB).
- `.caldav-sync/` — calendar sync, not notes.
- `assets/` — 186 MB of binaries; routed through the Phase-3.2 visual queue (`nexa:pendingVisualIndex`), not the text path.
- `Templates/` — empty templates, low semantic value.
- `BMO/`, `Excalidraw/` — plugin folders.
Anything else under `Notizen/**/*.md` is fair game.
- [x] **Q6 + Q7 — Lists & calendars discovered by name, not ID. Self-healing.**
Confirmed names from screenshots:
- **Tasks**: `Persönlich` (14), `DLR` (13), `Einkaufsliste` (13), `Wunschliste` (9).
- **Calendars**: `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name).
- Mapping in classification: `DLR` = Work-Kontext, `Persönlich` = Personal-Kontext, `Einkaufsliste` = Shopping, `Wunschliste` = Wishes.
**Self-healing requirement (the user explicitly noted lists may change/grow):** Nexa must not cache IDs forever. The `#nexa:config` workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or "not found". The runtime-config record in Qdrant's `_config` namespace stores `{name → id, discovered_at}` and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring `#einkaufsliste` / `#dlr` etc. without code changes.
- [x] **Q8 — Mail via Nextcloud, single account `fkrebs@nucli.de`.** No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: `Posteingang` (default), `Archiv`, `Junk` (61 — auto-filtered, ignored by Nexa), `Papierkorb`, `Waiting`. The `Waiting` folder is a useful manual signal — items moved there by the user are skipped from digest (treat as "in flight").
- [x] **Q9 — Pocket-ID SSO is configured everywhere.** No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n → Memos webhook, n8n → Nextcloud, n8n → Qdrant, n8n → LiteLLM) keep using API keys / app passwords — SSO is for human UIs only. Don't add basic-auth or per-app login screens.
This **resolves Q10 too** (the proposed "Pocket-ID SSO in front of n8n / Memos" is already done; Nexa just inherits it).
- [x] **Q10 — folded into Q9.**
## Hardware / capacity
- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.**
- **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
- **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
- **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
- **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27).
- **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
## Backups (deferred)
Backup-tier decisions are intentionally parked while Phase-3.1/3.2/3.4 are built. UNAS RAID 6 + Backrest already cover file-level recovery; pool-snapshot configuration ([12/#38](./12-optimization-opportunities.md#38)) remains the single highest-leverage data-protection action and doesn't depend on either question below. Re-open both when [Phase 3.3](./02-roadmap.md#phase-3--memory-qdrant--rag) becomes the next-up item.
- [ ] **Q12 — *deferred.*** S3 archive bucket on `s3.nuclide.systems` (warm tier). Pick when Phase 3.3 is up next.
- [ ] **Q20 — *deferred.*** Off-site cold-tier provider (Jottacloud is the leading candidate). Pick when Phase 3.3 is up next.
---
## Speed budget (Q3 follow-up)
Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap):
| Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` |
|------|--------|----------------|---------------------------------|------------------------------------|
| Real-time memo embed | 1 doc | <500 ms incl. n8n round-trip | ✅ ~50100 ms | ✅ ~20 ms |
| Daily ingest | ~70 docs | <60 s | ✅ ~510 s | ✅ ~2 s |
| Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~24 min | ✅ <1 min |
| RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms |
Conclusion: CPU-only TEI is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to **10 req/min** which would block real-time embed; self-hosted TEI side-steps that completely.
+149
View File
@@ -0,0 +1,149 @@
# 12 — Optimization Opportunities
Observations from the running infrastructure. Each item is independent — accept, defer, or reject.
## For Nexa directly
1. **Reuse, don't redeploy.** The earlier `DEPLOYMENT.md` would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new [09-deployment](./09-deployment.md) treats these as pre-existing — keeps the config minimal and avoids port collisions.
2. **Use LiteLLM virtual keys per logical caller.** Today there's one SAIA key. Issuing one key per workflow (`nexa-router`, `nexa-embed`, `nexa-digest`) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything.
3. **Use n8n's *credential* objects, never inline secrets.** The current workflows under `nexa-core/n8n-workflows/phase-1/*.json` should be reviewed — if any header `Authorization` is hardcoded, replace with credential references before importing.
4. **Centralise system alerts on a single ntfy topic** (`nexa.system`). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them.
5. **Defer graph DB until Phase 3.4.** Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries.
6. **Auto-export n8n workflows.** `nexa-core/scripts/backup_workflows.sh` already exists. Schedule it inside the n8n container (cron) and let it `git commit && git push` — this is the cheapest disaster recovery.
## For the wider homelab (out of scope but worth noting)
7. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts.
8. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding).
9. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history.
10. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win.
11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk.
12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.
13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded.
14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation:
- **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
- **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A.
- **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus.
Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.
15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.
16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).
17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.
20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.
## Proxmox host (NUC 14 Pro) tuning
Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:
21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 812 GiB frees ~1216 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):
```
echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB
update-initramfs -u
```
Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live.
22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 13 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
## Docker LXC (104) — observations & wins
Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure.
30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **510× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
```
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 65536
lxc.idmap: g 44 44 1 # video group on host
lxc.idmap: g 104 104 1 # render group on host
```
Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
31. **Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services/<svc>/...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets:
- `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir.
- `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk).
- `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store.
- `/mnt/pve/unas/backup/nexa/snapshots/qdrant/<date>/` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup/<svc>/` pattern used by Home Assistant / Immich.
- **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
- **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/<vol>:/<container-path>`. Concrete example in [09-deployment §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets ([12/#11](#11)) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is **withdrawn** ([12/#27](#27)) — it was an over-fitting to the Nextcloud-specific NFS issue.
32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 12 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane.
34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
## Housekeeping campaign (cross-cutting, do together)
These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #3537 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.
35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Arcane: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100300 MB RAM and has its own backup story. Two paths:
- **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB1 GB RAM and consolidates backups to one pg_dump cron.
- **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.
Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running.
36. **UNAS-integration audit.** `/mnt/pve/unas/services/<svc>/` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification):
- `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection.
- `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state.
- `memos` — the user-facing notes app; loss = data loss.
- `n8n` — workflows + executions DB.
- `audiobookshelf` — listening progress + library metadata.
- the `*-mcp` containers — likely stateless (cache only), low priority.
Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.
37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.**
`s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers:
**Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):**
- **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination.
- **Qdrant + GraphDB snapshots** flow UNAS → S3 daily.
- Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set.
**Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):**
- **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`.
- **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative.
- **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores.
- **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication.
- **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem.
**Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.
**Tiering & cadence:**
```
[Source] UNAS RAID-6 + native snapshots (#38)
│ daily restic/borg
[Warm] s3.nuclide.systems (on-site, fast restore)
│ weekly rclone copy + crypt
[Cold] Off-site (Jottacloud / B2 / Hetzner) ← satisfies the "1" in 3-2-1
```
**What goes off-site (priority order):**
1. Immich photo originals — irreplaceable.
2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable.
3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo.
4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include.
**What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.
Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it.
38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.
+96
View File
@@ -0,0 +1,96 @@
# 13 — Information Wishlist
What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent — run any subset, in any order.
Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/<NN>-<slug>.txt` — whichever is easiest.
---
## ~~Task 1 — Stack inventory + reference compose~~ ✅ DONE
User pasted the Karakeep compose. Q19 resolved → host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` is the convention. Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). The `docker ps -a` half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) — but that's not blocking now.
---
## Task 2 — Storage map (Proxmox + docker)
📍 **Where:** *Two shells* — first the **Proxmox host** (Datacenter → `nuc`**Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**.
Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB.
```bash
# (a) on the Proxmox host (192.168.1.20):
cat /etc/pve/storage.cfg
cat /etc/pve/lxc/104.conf
```
```bash
# (b) on LXC 104 (192.168.1.40):
docker volume ls
ls -l /dev/dri/ # confirms whether iGPU passthrough actually works
```
**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17.
---
## Task 3 — Postgres / DB workload inventory
📍 **Where:** **LXC 104 console** (same shell as Task 1).
Direct input to housekeeping #35 (consolidate postgres instances).
```bash
for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do
echo "=== $c ==="
docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null'
done
```
**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision).
---
## Task 4 — LiteLLM model list for the Nexa key
📍 **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported.
- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) → Settings → Model List, filtered to the Nexa virtual key. Screenshot the model rows.
- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set:
```bash
curl -sH "Authorization: Bearer $SAIA_API_KEY" \
https://ai.nuclide.systems/v1/models | jq '.data[].id'
```
**Unblocks:** Q18 follow-up — which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI.
---
## Task 5 — S3 archive credentials
📍 **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy — same login as the rest of the homelab).
Walk to: **Buckets** → either pick an existing Nexa-suitable bucket or create one called `nexa` → note the bucket name. Then **Access Keys** → create a key named `nexa-snapshots` with read/write on that bucket → drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden).
**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3.
---
## Task 6 — Reverse-proxy + DNS authority *(only if needed)*
📍 **Where:** two browser UIs.
- **Zoraxy** (`http://192.168.1.4:8000`) → HTTP Proxy → either click "Export" if available, or take a full screenshot of the table.
- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) → Filters → DNS Rewrites → screenshot.
Skip unless we hit a routing surprise during Phase 1.
---
## When something else is needed
The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "📍 Where" framing.
## Self-serve once Phase 6 ships
Once [Phase 6 — Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 13 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember.
+51
View File
@@ -0,0 +1,51 @@
# NEXA Documentation
**Neural Nexus for Information & Automation** — the central nervous system that ties Memos, n8n, SAIA (LiteLLM), Nextcloud and a vector store into one assistant.
This is the documentation entry point. Read top-to-bottom for first-time setup, or jump to the section you need.
---
## Table of Contents
| # | Document | Read when… |
|---|----------|------------|
| 01 | [Vision & Scope](./01-vision-and-scope.md) | You want to understand *what* Nexa is and isn't. |
| 02 | [Roadmap & Phases](./02-roadmap.md) | You want to know the implementation order. |
| 03 | [Architecture Overview](./03-architecture.md) | You need a one-page mental model. |
| 04 | [Integration Matrix](./04-integration-matrix.md) | You're wiring up a new data source or mapping work-vs-personal flows. |
| 05 | [Command System](./05-command-system.md) | You want to know what `#nexa:*` commands do. |
| 06 | [Classification Logic](./06-classification-logic.md) | You're tuning the work/personal router. |
| 07 | [Workflow Spec — Task Router](./07-workflow-spec.md) | You're building the Phase-2 router workflow. |
| 08 | [GraphRAG Architecture](./08-graphrag-architecture.md) | You're working on Phase 3 (Qdrant + Graph). |
| 09 | [Deployment](./09-deployment.md) | You're bringing Nexa up on the real infrastructure. |
| 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. |
| 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. |
| 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. |
| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. |
---
## Reading paths
- **First-time installer:** 01 → 03 → 09 → 10 → 11
- **Workflow author:** 04 → 05 → 07 → 06
- **Phase-3 work (memory):** 08 → 09 (§ Phase 3 add-on)
- **Stakeholder / quick read:** 01 → 02 → 12
---
## Repository layout
```
nexa/
├── README.md → points here
├── docs/ → you are here
└── nexa-core/ → the actual project
├── ai-prompts/ → SAIA system prompts
├── config/ → runtime config (YAML, JSON schema)
├── n8n-workflows/ → exported workflows, version-controlled
└── scripts/ → automation helpers
```
Source-of-truth for *runnable* config stays in `nexa-core/`. Documentation lives here in `docs/`.
-629
View File
@@ -1,629 +0,0 @@
# NEXA Deployment Guide
## 🚀 Architektur-Überblick
```
┌─────────────────────────────────────────────────┐
│ MEMOS (Port 5230) │
│ Interface, Voice-Input, Webhooks │
└────────────────────┬────────────────────────────┘
┌───────────┼───────────┐
↓ ↓ ↓
┌────────┐ ┌────────┐ ┌──────────┐
│ n8n │ │ SAIA │ │ Nextcloud│
│Logic │ │LiteLLM │ │Tasks/Mail│
│7879 │ │4000 │ │80/443 │
└────┬───┘ └────┬───┘ └─────┬────┘
│ │ │
┌────┴───────────┼────────────┴────┐
↓ ↓ ↓
┌─────────┐ ┌────────────┐ ┌──────────────┐
│ QDRANT │ │ GraphDB │ │ Obsidian │
│Vector │ │ (Ontotext) │ │(local sync) │
│6333 │ │7200 │ │ │
└─────────┘ └────────────┘ └──────────────┘
```
---
## 📋 Voraussetzungen
### Hardware
- **CPU**: Min. 4 Cores (empfohlen 8)
- **RAM**: Min. 16 GB (empfohlen 32 GB)
- **Storage**: Min. 100 GB SSD (für Datenbanken & Backups)
- **Network**: Stabile Verbindung, HTTPS-ready
### Software
- Docker & Docker Compose (Version 20.10+)
- Git
- Bash
- curl/wget
- `.env` Datei aus `.env.example` (mit Secrets gefüllt)
### Accounts & Credentials
- Nextcloud Instanz mit Admin-Zugang
- OpenAI API-Key (für SAIA/LiteLLM)
- Obsidian Vault Zugang lokalem Filesystem
---
## 🐳 Schritt 1: Docker Compose Setup
### 1.1 Repository klonen
```bash
git clone https://github.com/noheton/nexa.git
cd nexa/nexa-core
```
### 1.2 .env konfigurieren
```bash
cp .env.example .env
# WICHTIG: Alle Secrets eintragen!
# - MEMOS_API_KEY
# - SAIA_API_KEY
# - NC_APP_PASSWORD
# - QDRANT_API_KEY
# - GRAPHDB_PASSWORD
nano .env
```
### 1.3 docker-compose.yml erstellen
```bash
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
# ===== MEMOS (Interface) =====
memos:
image: neosmemo/memos:latest
container_name: nexa-memos
ports:
- "5230:5230"
volumes:
- ./data/memos:/root/.memos
environment:
MEMOS_PORT: "5230"
MEMOS_DRIVER: sqlite
MEMOS_DSN: /root/.memos/memos.db
networks:
- nexa
restart: unless-stopped
# ===== n8n (Workflow Engine) =====
n8n:
image: n8nio/n8n:latest
container_name: nexa-n8n
ports:
- "5678:5678"
volumes:
- ./data/n8n:/home/node/.n8n
- ./n8n-workflows:/workflows
environment:
N8N_BASIC_AUTH_ACTIVE: "true"
N8N_BASIC_AUTH_USER: "${N8N_USER:-admin}"
N8N_BASIC_AUTH_PASSWORD: "${N8N_PASSWORD:-nexa_secure_pass}"
N8N_HOST: "${N8N_DOMAIN:-localhost}"
N8N_PORT: "5678"
N8N_PROTOCOL: https
N8N_WEBHOOK_URL: "https://${N8N_DOMAIN:-localhost}/webhook"
GENERIC_TIMEZONE: Europe/Berlin
DB_TYPE: sqlite
DB_SQLITE_PATH: /home/node/.n8n/nexa.db
networks:
- nexa
restart: unless-stopped
depends_on:
- memos
# ===== QDRANT (Vector DB) =====
qdrant:
image: qdrant/qdrant:latest
container_name: nexa-qdrant
ports:
- "6333:6333"
volumes:
- ./data/qdrant:/qdrant/storage
environment:
QDRANT_API_KEY: "${QDRANT_API_KEY}"
networks:
- nexa
restart: unless-stopped
# ===== GraphDB (Ontotext SPARQL) =====
graphdb:
image: ontotext/graphdb:10-ee
container_name: nexa-graphdb
ports:
- "7200:7200"
volumes:
- ./data/graphdb:/opt/graphdb/data
- ./config/graphdb-init.ttl:/graphdb-import/import.ttl
environment:
GDB_JAVA_OPTS: "-Xmx8g -Xms4g"
GDB_PASSWORD: "${GRAPHDB_PASSWORD}"
networks:
- nexa
restart: unless-stopped
# ===== SAIA / LiteLLM Proxy (optional - wenn lokal) =====
# (Falls SAIA extern gehostet: nicht nötig)
networks:
nexa:
driver: bridge
volumes:
nexa_memos:
nexa_n8n:
nexa_qdrant:
nexa_graphdb:
EOF
```
---
## 🔧 Schritt 2: Individuelle Service-Konfiguration
### 2.1 MEMOS Initialisierung
```bash
# Nach memos startup (ca. 30s)
curl -X POST http://localhost:5230/api/v1/auth/signup \
-H "Content-Type: application/json" \
-d '{
"username": "nexa",
"password": "secure_password"
}'
# API-Token generieren
TOKEN=$(curl -X POST http://localhost:5230/api/v1/auth/signin \
-H "Content-Type: application/json" \
-d '{"username":"nexa","password":"secure_password"}' \
| jq -r '.data.token')
echo "MEMOS_API_KEY=$TOKEN" >> .env
```
### 2.2 Qdrant Konfiguration
```bash
# Warte auf Qdrant startup
sleep 10
# Erstelle Collection für Nexa
curl -X PUT "http://localhost:6333/collections/nexa_knowledge" \
-H "Content-Type: application/json" \
-H "api-key: ${QDRANT_API_KEY}" \
-d '{
"vectors": {
"size": 1536,
"distance": "Cosine"
},
"optimizers_config": {
"deleted_threshold": 0.2,
"vacuum_threshold": 0.5,
"default_segment_number": 5,
"max_segment_number": 30,
"memmap_threshold": 268435456,
"indexing_threshold": 20000,
"flush_interval_sec": 10,
"max_optimization_threads": 4
}
}'
echo "✅ Qdrant Collection erstellt"
```
### 2.3 GraphDB Initialisierung
```bash
# Warte auf GraphDB startup
sleep 15
# Erstelle Repository
GRAPHDB_URL="http://localhost:7200"
curl -X POST "$GRAPHDB_URL/rest/repositories" \
-H "Content-Type: application/json" \
-u "admin:${GRAPHDB_PASSWORD}" \
-d '{
"id": "nexa_knowledge",
"title": "Nexa Knowledge Graph",
"type": "free",
"params": {
"baseURL": {
"label": "Base URL",
"value": "http://nexa.local/"
},
"defaultLanguage": {
"label": "Default language",
"value": "en"
}
}
}'
echo "✅ GraphDB Repository erstellt"
```
### 2.4 n8n Workflows Import
```bash
# Warte auf n8n startup (ca. 60s)
sleep 60
# Importiere Discovery Workflow
curl -X POST http://localhost:5678/api/v1/workflows \
-H "Content-Type: application/json" \
-u "admin:${N8N_PASSWORD}" \
-d @n8n-workflows/phase-1/1_1_discovery_workflow.json
# Importiere Memos Bridge
curl -X POST http://localhost:5678/api/v1/workflows \
-H "Content-Type: application/json" \
-u "admin:${N8N_PASSWORD}" \
-d @n8n-workflows/phase-1/1_2_memos_bridge_v2.json
# Importiere Email Butler
curl -X POST http://localhost:5678/api/v1/workflows \
-H "Content-Type: application/json" \
-u "admin:${N8N_PASSWORD}" \
-d @n8n-workflows/phase-2/2_1_email_butler.json
echo "✅ n8n Workflows importiert"
```
---
## 🚀 Schritt 3: Deployment starten
### 3.1 Docker Compose Up
```bash
docker-compose up -d
# Logs überwachen
docker-compose logs -f
```
### 3.2 Health Checks
```bash
# Health Check Script
check_service() {
local name=$1
local url=$2
echo "Checking $name..."
if curl -s "$url" > /dev/null; then
echo "$name: OK"
else
echo "$name: FAILED"
fi
}
sleep 30
check_service "Memos" "http://localhost:5230"
check_service "n8n" "http://localhost:5678"
check_service "Qdrant" "http://localhost:6333/health"
check_service "GraphDB" "http://localhost:7200/health"
echo ""
echo "🎯 Service URLs:"
echo " - Memos: http://localhost:5230"
echo " - n8n: http://localhost:5678"
echo " - Qdrant: http://localhost:6333"
echo " - GraphDB: http://localhost:7200"
```
---
## ⚙️ Schritt 4: Konfiguration via Command System
### 4.1 Discovery-Workflow triggern
```bash
# In Memos Memo erstellen:
# #nexa:config
# Oder via API:
curl -X POST http://localhost:5678/api/v1/workflows/run \
-H "Content-Type: application/json" \
-u "admin:${N8N_PASSWORD}" \
-d '{
"workflowId": "discovery-workflow-id"
}'
```
### 4.2 Automatische Konfiguration laden
Nach erfolgreichem Discovery werden Nextcloud-Listen-IDs:
```bash
# Es wird automatisch aktualisiert in:
# - Qdrant: als Metadata
# - n8n: als Umgebungsvariablen
# - Memos: als Confirmation-Memo
```
---
## 🔐 Schritt 5: Reverse Proxy Setup (Production)
### 5.1 Nginx Configuration
```bash
cat > /etc/nginx/sites-available/nexa.conf << 'EOF'
upstream memos_backend {
server localhost:5230;
}
upstream n8n_backend {
server localhost:5678;
}
upstream qdrant_backend {
server localhost:6333;
}
upstream graphdb_backend {
server localhost:7200;
}
server {
listen 443 ssl http2;
server_name nexa.yourdomain.com;
ssl_certificate /etc/letsencrypt/live/nexa.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nexa.yourdomain.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# MEMOS
location / {
proxy_pass http://memos_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# n8n
location /n8n/ {
proxy_pass http://n8n_backend/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# Qdrant (nur intern, wenn nötig)
location /qdrant/ {
allow 10.0.0.0/8;
deny all;
proxy_pass http://qdrant_backend/;
}
# GraphDB (nur intern)
location /graphdb/ {
allow 10.0.0.0/8;
deny all;
proxy_pass http://graphdb_backend/;
}
# Webhook endpoint für externe Services
location /webhook/ {
proxy_pass http://n8n_backend/webhook/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
server {
listen 80;
server_name nexa.yourdomain.com;
return 301 https://$server_name$request_uri;
}
EOF
# Aktivieren
sudo ln -s /etc/nginx/sites-available/nexa.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```
---
## 📊 Schritt 6: Monitoring & Backups
### 6.1 Backup Script
```bash
cat > scripts/backup_nexa.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="./backups/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
echo "🔄 Backing up Nexa..."
# Backup Datenbanken
docker-compose exec -T qdrant tar czf - /qdrant/storage > "$BACKUP_DIR/qdrant.tar.gz"
docker-compose exec -T graphdb tar czf - /opt/graphdb/data > "$BACKUP_DIR/graphdb.tar.gz"
docker-compose exec -T memos tar czf - /root/.memos > "$BACKUP_DIR/memos.tar.gz"
docker-compose exec -T n8n tar czf - /home/node/.n8n > "$BACKUP_DIR/n8n.tar.gz"
# Backup Configs
cp .env "$BACKUP_DIR/.env.backup"
cp docker-compose.yml "$BACKUP_DIR/docker-compose.yml.backup"
echo "✅ Backup completed: $BACKUP_DIR"
# Optional: zu S3 hochladen
# aws s3 sync "$BACKUP_DIR" s3://nexa-backups/$(date +%Y%m%d)/
EOF
chmod +x scripts/backup_nexa.sh
# Tägliches Backup (cron)
echo "0 2 * * * /path/to/nexa/scripts/backup_nexa.sh" | crontab -
```
### 6.2 Health Check Monitoring
```bash
cat > scripts/monitor_nexa.sh << 'EOF'
#!/bin/bash
# Monitore Service Status
SERVICES=("memos" "n8n" "qdrant" "graphdb")
for service in "${SERVICES[@]}"; do
if docker-compose ps $service | grep -q "Up"; then
echo "✅ $service: Running"
else
echo "❌ $service: DOWN"
# Restart bei Fehler
docker-compose restart $service
fi
done
EOF
chmod +x scripts/monitor_nexa.sh
# Alle 5 Minuten
echo "*/5 * * * * /path/to/nexa/scripts/monitor_nexa.sh" | crontab -
```
---
## 🧪 Schritt 7: Validation & Testing
### 7.1 End-to-End Test
```bash
cat > scripts/test_nexa.sh << 'EOF'
#!/bin/bash
echo "🧪 Testing Nexa Installation..."
# Test 1: Memos API
echo "Test 1: Memos API"
if curl -s http://localhost:5230/api/v1/auth/status | jq . > /dev/null 2>&1; then
echo "✅ Memos: OK"
else
echo "❌ Memos: FAILED"
fi
# Test 2: n8n API
echo "Test 2: n8n API"
if curl -s -u "admin:${N8N_PASSWORD}" http://localhost:5678/api/v1/workflows | jq . > /dev/null 2>&1; then
echo "✅ n8n: OK"
else
echo "❌ n8n: FAILED"
fi
# Test 3: Qdrant Collection
echo "Test 3: Qdrant Collection"
if curl -s -H "api-key: ${QDRANT_API_KEY}" http://localhost:6333/collections/nexa_knowledge | jq . > /dev/null 2>&1; then
echo "✅ Qdrant: OK"
else
echo "❌ Qdrant: FAILED"
fi
# Test 4: GraphDB Repository
echo "Test 4: GraphDB Repository"
if curl -s -u "admin:${GRAPHDB_PASSWORD}" http://localhost:7200/repositories | jq . > /dev/null 2>&1; then
echo "✅ GraphDB: OK"
else
echo "❌ GraphDB: FAILED"
fi
echo ""
echo "🎯 Create Test Memo:"
MEMO=$(curl -s -X POST http://localhost:5230/api/v1/memos \
-H "Authorization: Bearer ${MEMOS_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"content": "Test memo - #nexa:config"}' | jq -r '.id')
echo "✅ Test Memo created: $MEMO"
echo "Check Memos UI to verify webhook triggers Discovery"
EOF
chmod +x scripts/test_nexa.sh
./scripts/test_nexa.sh
```
---
## 🆘 Troubleshooting
### Problem: Memos kann sich nicht mit n8n verbinden
```bash
# Überprüfe n8n Webhook URL in Memos config
curl http://localhost:5678/webhook/memos
# Falls 404: Workflow ist nicht aktiviert/deployed
docker-compose logs n8n | grep webhook
```
### Problem: GraphDB startet nicht
```bash
# Memory-Limit erhöhen
docker-compose exec graphdb -e "GDB_JAVA_OPTS=-Xmx16g -Xms8g" restart
# Logs checken
docker-compose logs graphdb
```
### Problem: Qdrant Collection leer
```bash
# Überprüfe ob Memos Bridge Workflow läuft
docker-compose logs n8n | grep "memos_bridge"
# Manuell test
curl -X POST http://localhost:5230/api/v1/memos \
-H "Authorization: Bearer ${MEMOS_API_KEY}" \
-d '{"content": "- [ ] Test task #nexa"}'
```
---
## 📈 Next Steps
1.**Phase 1 deployt**: Memos Bridge & Discovery laufen
2.**Phase 2 aktivieren**: Email Butler (wenn Nextcloud Mail konfiguriert)
3.**Phase 3 setup**: Qdrant Embedding Pipeline
4.**Phase 4 integration**: GraphRAG Queries testen
5.**Phase 5**: Home Assistant Voice Integration
---
## 📞 Support & Debugging
```bash
# Alle Logs in eine Datei sammeln
docker-compose logs > deployment_logs.txt
# Spezifischer Service
docker-compose logs n8n --tail 100
# Wiederherstellung aus Backup
./scripts/restore_nexa.sh backup/2024-01-15_120000/
```
---
**Deployment abgeschlossen! Nexa ist nun betriebsbereit.** 🚀
-27
View File
@@ -1,27 +0,0 @@
# NEXA: Implementierungs-Fahrplan 🚀
Dieser Plan beschreibt die schrittweise Aktivierung des Nervensystems.
## Phase 1: Die Wirbelsäule (Basis-Konnektivität)
- [ ] **1.1 Memos-n8n Bridge:** Webhook-Anbindung und Filter-Logik.
- [ ] **1.2 SAIA-Integration:** LiteLLM Proxy in n8n einbinden (Chat-Funktion via Kommentare).
- [ ] **1.3 Git-Sync:** Automatisches Backup der n8n-Workflows in dieses Repo.
## Phase 2: Die Sinnesorgane (Input-Kanäle)
- [ ] **2.1 Email-Butler:** Automatisierte Filterung und Zusammenfassung der ~20 täglichen Mails.
- [ ] **2.2 RSS Morning Digest:** Täglich kuratierte News-Zusammenfassung in Memos.
- [ ] **2.3 Bluesky-Antenne:** Speichern von gelikten Posts als Wissens-Schnipsel.
## Phase 3: Das Gedächtnis (Qdrant & RAG)
- [ ] **3.1 Embedding-Pipeline:** Automatische Vektorisierung von Memos & Obsidian.
- [ ] **3.2 Context-Retrieval:** Nexa nutzt Qdrant-Wissen für Antworten (#ask).
- [ ] **3.3 S3-Langzeitarchiv:** Backup der Qdrant-Snapshots auf den S3-Server.
## Phase 4: Die Motorik (Organisation & Handeln)
- [ ] **4.1 Nextcloud Task-Sync:** Bidirektionaler Status-Abgleich (Memos <-> NC Tasks).
- [ ] **4.2 Timeboxing-Agent:** Intelligente Kalender-Slot-Vorschläge basierend auf Tasks.
- [ ] **4.3 Karakeep-Kuration:** Automatischer Review-Prozess für gespeicherte Links.
## Phase 5: Alltags-Integration (Feinschliff)
- [ ] **5.1 Home Assistant Voice:** Nexa als Wake-Word Integration.
- [ ] **5.2 System-Monitoring:** Alarme von Proxmox/Backrest direkt in den Memos-System-Feed.
+14 -12
View File
@@ -1,15 +1,17 @@
# NEXA 🧠
**The Neural Nexus for Information & Automation**
# nexa-core
Nexa ist das zentrale Nervensystem meines IT-Setups.
Runtime artifacts for Nexa: n8n workflows, configuration, AI prompts, scripts.
## 🏗 Architektur
- **Interface:** Memos
- **Logic:** n8n
- **Memory:** Qdrant & Obsidian
- **Intelligence:** SAIA (via LiteLLM)
📖 Documentation lives one level up in [`../docs/`](../docs/index.md).
## 📁 Struktur
- `n8n-workflows/`: JSON-Exporte der Gehirn-Logik.
- `ai-prompts/`: System-Prompts für Nexa.
- `config/`: Schemata für Qdrant und API-Mappings.
## Layout
- `n8n-workflows/` — exported workflows (per phase). Imported into the running n8n.
- `ai-prompts/` — system prompts for SAIA / LiteLLM.
- `config/` — runtime config (`agents.yaml`, `infra_manifest.yaml`, `qdrant_schema.json`).
- `scripts/` — automation helpers (workflow backup, etc.).
- `.env.example` — bootstrap secrets only; runtime config lives in Qdrant `_config` namespace.
## Quickstart
See [`../docs/09-deployment.md`](../docs/09-deployment.md).
+10 -5
View File
@@ -3,11 +3,16 @@ Rolle: Zentrales Nervensystem
Aufgabe: Überwacht E-Mails, kuratiert Wissen und plant Tasks.
WICHTIGE LOGIK:
- Tasks müssen strikt zwischen "Work" und "Personal" unterschieden werden.
- Analysiere den Inhalt eines Memos: Bezieht es sich auf professionelle Projekte (Obsidian Context) -> Work List. Bezieht es sich auf Haushalt, Einkäufe oder Freizeit -> Personal List.
- Im Zweifel: Nachfragen via Memos-Kommentar.
- Tasks werden auf vier Nextcloud-Listen geroutet: "DLR" (Arbeit), "Persönlich", "Einkaufsliste", "Wunschliste".
- Analysiere den Inhalt eines Memos und entscheide über das Feld "context": "work" | "personal" | "shopping" | "wishes".
- work -> Liste "DLR" (Beruf, Projekte, Meetings, Deadlines, Forschung).
- personal -> Liste "Persönlich" (Alltag, Familie, Termine, Behördenkram).
- shopping -> Liste "Einkaufsliste" (Lebensmittel, Drogerie, Verbrauchsgüter).
- wishes -> Liste "Wunschliste" (Hobby, Technik, Bücher, Geschenkideen).
- Listennamen sind dynamisch — bei Bedarf werden weitere Listen vom Runtime-Config eingespeist, dann darfst du auch deren Namen als context vorschlagen.
- Im Zweifel (Konfidenz < 0.7): Default auf "personal" und Nachfrage via Memos-Kommentar.
KONSUM-LOGIK:
- Wenn ein Link von Karakeep kommt, analysiere das Produkt.
- Haushaltswaren/Verbrauchsgüter -> NC_LIST_ID_SHOPPING.
- Besondere Wünsche/Hobby/Technik -> NC_LIST_ID_WISHES.
- Haushaltswaren/Verbrauchsgüter -> context "shopping".
- Besondere Wünsche/Hobby/Technik -> context "wishes".
+17 -5
View File
@@ -1,12 +1,24 @@
{
"collection_name": "nexa_knowledge",
"collection_name": "nexa_knowledge_text",
"vector_config": {
"size": 1536,
"size": 1024,
"distance": "Cosine"
},
"payload_schema": {
"source": "keyword",
"tags": "keyword",
"created_at": "datetime"
"source": "keyword",
"source_type": "keyword",
"modality": "keyword",
"mime_type": "keyword",
"media_uri": "keyword",
"context": "keyword",
"tags": "keyword",
"graph_iri": "keyword",
"content_hash":"keyword",
"created_at": "datetime"
},
"_notes": {
"vector_size_rationale": "1024 = bge-m3 output. If Q15 resolves to nomic-embed-text-v1.5 instead, rebuild this collection with size: 768.",
"modality_field": "Always 'text' in this collection. Forward-compatibility for Phase 3.2 (see qdrant_schema_visual.json).",
"graph_iri_field": "IRI of the corresponding nexa:Note in GraphDB. The same value is stored on the GraphDB node as nexa:vectorId pointing back here. This is the cross-pillar bridge."
}
}
@@ -0,0 +1,26 @@
{
"collection_name": "nexa_knowledge_visual",
"vector_config": {
"size": 768,
"distance": "Cosine"
},
"payload_schema": {
"source": "keyword",
"source_type": "keyword",
"modality": "keyword",
"mime_type": "keyword",
"media_uri": "keyword",
"context": "keyword",
"tags": "keyword",
"graph_iri": "keyword",
"content_hash":"keyword",
"created_at": "datetime"
},
"_notes": {
"status": "PLACEHOLDER — not deployed in Phase 3.1. Created in Phase 3.2.",
"vector_size_rationale": "768 = jina-clip-v2 joint text/image space. The same model embeds both text queries and images, so semantic queries against this collection are simple kNN.",
"served_by": "Will be served by `infinity` (https://github.com/michaelfeil/infinity) — drop-in replacement for TEI that supports CLIP-family models with the same OpenAI-compatible API.",
"modality_field": "'image' for indexed images, 'text' if a text query has been embedded into the visual space for retrieval (rare — usually we embed the text query at search time and don't store it).",
"media_uri_field": "Required for image rows. Examples: 'memos://memos/123/attachment/4', 'nextcloud://files/photos/whiteboard-2025-09.jpg', 'obsidian://attachments/diagram.png'."
}
}
-16
View File
@@ -1,16 +0,0 @@
# Spezifikation: Phase 2 - Task-Router
## 1. Trigger
* **Typ:** Webhook (Memos)
* **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`.
## 2. Intelligence Node (SAIA)
* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: { "context": "work" | "personal", "summary": "string", "urgency": 1-5 }."
* **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.
## 3. Switch Node
* **Route 1:** Context == "work" -> Nextcloud Create Task (List ID: Work).
* **Route 2:** Context == "personal" -> Nextcloud Create Task (List ID: Personal).
## 4. Feedback Loop
* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
-131
View File
@@ -1,131 +0,0 @@
# Nexa Command System - #nexa:command
Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag:
## 🔧 Konfigurations-Kommandos
### #nexa:config
- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud, Qdrant, etc.)
- **Antwort**: Postet Report mit gefundenen Konfigurationen
- **Speichert**: Settings in DynoDB / Qdrant Metadata (nicht in .env)
```
#nexa:config
```
**Beispiel-Response:**
```
✅ Autodiscovery abgeschlossen:
- Nextcloud Listen: Work (id=123), Personal (id=456), Shopping (id=789), Wishes (id=1011)
- Qdrant Collection: nexa_knowledge (1536 dims, Cosine)
- Nextcloud Mail: 3 Accounts erkannt (Office 365, Gmail, Privat)
```
### #nexa:status
- **Beschreibung**: Zeigt den aktuellen System-Status
- **Antwort**: Uptime, verbundene Systeme, Memory-Stats
```
#nexa:status
```
### #nexa:sync-obsidian
- **Beschreibung**: Triggert sofortige Synchronisierung mit Obsidian Vault
- **Optional Parameter**: `--vault=/path/to/vault` oder `--force` für Neuindexierung
```
#nexa:sync-obsidian --force
```
## 📊 Memory & RAG Kommandos
### #nexa:ask [question]
- **Beschreibung**: Semantische Suche in Qdrant (RAG) basierend auf Obsidian + Memos
- **Antwort**: Top-3 Treffer mit Quellen
```
#nexa:ask Wie implementierten wir das JWT-Middleware-Pattern?
```
### #nexa:learn [topic]
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
```
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
```
## 🎯 Workflow-Kommandos
### #nexa:route-test [text]
- **Beschreibung**: Testet die Klassifizierung (Work/Personal) ohne Task zu erstellen
```
#nexa:route-test Muss morgen die Präsentation für den Client fertigstellen
```
**Response:**
```
💬 Klassifizierung (Test-Mode):
- Kontext: WORK
- Vertrauen: 0.95
- Begründung: "Client-Präsentation → professioneller Kontext"
```
### #nexa:email-digest
- **Beschreibung**: Triggert sofortige Email-Zusammenfassung (sonst tägl. 7 Uhr)
```
#nexa:email-digest
```
## 🔐 Admin-Kommandos
### #nexa:reset-config
- **Beschreibung**: Löscht gespeicherte Konfiguration, triggert neuen Autodiscovery
- **Warnung**: Alle benutzerdefinierten Settings werden vergessen
```
#nexa:reset-config
```
### #nexa:export-state
- **Beschreibung**: Exportiert aktuellen State als JSON (für Backups)
```
#nexa:export-state
```
---
## 📝 Implementierung in n8n
Ein **Command Parser** läuft immer mit:
1. **Memos Webhook** empfängt alle Memos
2. **Regex Check**: Sucht nach `#nexa:command`
3. **Router**: Versendet an entsprechenden n8n-Workflow
4. **Antwort**: Postet Reply als Kommentar/Edit
**Command Parser Regex:**
```
^#nexa:(\w+)(?:\s+([^\n]*?))?(?:$|\s*--)
```
Extrahiert: `[command, parameters]`
---
## 🎛️ Dynamische Konfigurationsspeicherung
Statt .env zu editieren:
1. **First Run**: `#nexa:config` speichert zu lokalen Metadata
2. **Speich-Ziel**:
- **Primary**: Qdrant Metadata (als `_config` Namespace)
- **Fallback**: `nexa-core/config/runtime_config.json` (gitignored)
- **Notfall**: .env (nur initial)
3. **Reload-Logik**: Bei jedem Workflow-Start werden Settings aus Qdrant geladen
Das macht Nexa vollständig selbstständig nach dem initialem Setup!
-273
View File
@@ -1,273 +0,0 @@
# GraphRAG für Nexa - Strukturelles Wissen & Beziehungen
## 🧠 Zwei-Säulen-Memory-Architektur
Nexa nutzt zwei komplementäre Speicher für vollständiges Verständnis:
### Säule 1: Qdrant (Vector DB) - Semantische Ähnlichkeit
- **Frage**: "Was ist ähnlich, was ist relevant?"
- **Speichert**: Embeddings von Memos, Obsidian-Notes, E-Mail-Digests
- **Suche**: Cosine Similarity über 1536-dimensionale Vektoren
- **Antwort**: Top-k semantisch ähnliche Dokumente (RAG)
**Beispiel:**
```
Frage: "Wie war die JWT-Implementation in unserem letzten Projekt?"
→ Qdrant findet: 10 ähnliche Snippets → Prompt-Context
```
### Säule 2: Neo4j (Graph DB) - Strukturelle Beziehungen
- **Frage**: "Wer ist beteiligt? Was hängt davon ab? Wie ist es organisiert?"
- **Speichert**:
- **Knoten (Nodes)**: Projects, Tasks, People, Technologies, Dates
- **Kanten (Relationships)**: `USES`, `OWNS`, `DEPENDS_ON`, `MENTIONS`, `CREATED_BY`
- **Suche**: Cypher-Queries über Netzwerk-Topologie
- **Antwort**: Pfade, zentrale Knoten, Abhängigkeitsanalyse
**Beispiel:**
```cypher
MATCH (tech:Technology)-[USES*]->*(service:Service),
(tech)-[:MENTIONED_IN]->(task:Task)
WHERE tech.name = "JWT" AND task.status = "DONE"
RETURN tech, service, task
```
---
## 📊 Neo4j Schema für Nexa
### Node-Typen
```cypher
(Project {name, startDate, status, priority})
(Task {title, description, status, urgency: 1-5, context: "work"|"personal"})
(Person {name, role, email})
(Technology {name, category: "framework"|"language"|"tool"})
(Topic {name, domain})
(File {path, type, lastModified})
(Note {id, content_hash, created_at})
```
### Relationship-Typen
| Beziehung | Von | Zu | Beschreibung |
|-----------|-----|-----------|-------------|
| `OWNS` | Person | Task/Project | Wer ist verantwortlich |
| `USES` | Task/Project | Technology | Welche Tech wird verwendet |
| `DEPENDS_ON` | Task/Project | Task/Project | Hängt davon ab |
| `CREATED_BY` | Note | Person | Autor einer Notiz |
| `MENTIONS` | Note/Task | Topic/Technology | Verweist auf |
| `RELATED_TO` | Task | Task | Zusammenhängend |
| `CHILD_OF` | Task | Project | Task gehört zu Project |
| `SCHEDULED_FOR` | Task | Date | Zeitliche Zuordnung |
---
## 🔄 Synchronisierungs-Flows
### 1. **Memos → Neo4j** (automatisch beim Erstellen)
Wenn ein Memo mit `- [ ]` erstellt wird:
```
Memo-Content: "Muss JWT-Middleware für Auth-Service refaktorieren"
SAIA extrahiert Entitäten:
- Technologie: "JWT-Middleware", "Auth-Service"
- Kontext: "work"
- Urgenz: 3
Erstelle in Neo4j:
- Task node (title="JWT-Middleware refaktorieren")
- Technology nodes: JWT, Auth-Service
- Relationships: Task -[USES]-> JWT, Task -[USES]-> Auth-Service
- Task -[CREATED_BY]-> Me (Person node)
```
### 2. **Obsidian → Neo4j** (#nexa:sync-obsidian)
Beim Sync von Obsidian-Vault:
```
File: obsidian/projects/nexa-deployment.md
Header Struktur wird zu Graph:
- Project "Nexa Deployment"
- Technologies: Docker, n8n, Nextcloud
- Dependencies: "requires infrastructure setup"
Erstelle:
- Project nodes mit Metadata
- Hierarchie: Project -[CONTAINS]-> Task -[USES]-> Technology
```
### 3. **Nextcloud Tasks → Neo4j** (bidirektional)
Wenn Task in Nextcloud erstellt/gelöcht:
```
Nextcloud Task: "Review JWT implementation" (in Work-List)
n8n Trigger erkennt Änderung
Update Neo4j:
- Task node status = "active"
- Task -[DEPENDS_ON]-> related Task (falls Beziehung existiert)
```
---
## 🤖 GraphRAG Queries - Praktische Anwendungen
### Query 1: "Zeige alle offenen Tasks, die JWT betreffen"
```cypher
MATCH (tech:Technology {name:"JWT"}),
(tech)-[:MENTIONED_IN|USES]-(task:Task),
(task)-[:CHILD_OF]->(project:Project)
WHERE task.status IN ["NEEDS-ACTION", "IN-PROGRESS"]
RETURN project.name, task.title, task.urgency
ORDER BY task.urgency DESC
```
### Query 2: "Von welchen Tasks/Projects hängt Auth-Service ab?"
```cypher
MATCH path = (dependent)-[:DEPENDS_ON*]->(service:Technology {name:"Auth-Service"})
RETURN [x IN nodes(path) | x.name], length(path)
ORDER BY length(path) ASC
```
### Query 3: "Wer arbeitet an diesem Projekt und auf welchen Technologien?"
```cypher
MATCH (person:Person)-[:OWNS]->(task:Task)-[:CHILD_OF]->(project:Project {name:"Nexa"}),
(task)-[:USES]->(tech:Technology)
RETURN person.name, collect(task.title), collect(tech.name)
```
### Query 4: "Was ist seit gestern neu an Technologien/Themen gelernt?"
```cypher
MATCH (note:Note)-[:MENTIONS]->(topic:Topic)
WHERE note.created_at > datetime.now() - duration('P1D')
RETURN DISTINCT topic.name, count(note) as mentions
ORDER BY mentions DESC
```
---
## 🔗 GraphRAG-Enhanced Answer Generation
Wenn Nutzer `#nexa:ask` fragt:
```
#nexa:ask Wie stehen die JWT-Tasks zur Deadline?
┌─────────────┴──────────────┐
↓ ↓
[Qdrant] [Neo4j]
Semantic Structural
Search Search
↓ ↓
Top-3 Notes (JWT)-[DEPENDS_ON]
mit ähnlichem → Connected Tasks
Kontext → Critical Path
↓ ↓
Combine: Merge + Rank
System Prompt (SAIA):
"Basierend auf diesem Wissen und diesen Abhängigkeiten, beantworte:"
Response mit:
- Semantisch relevanter Information (Qdrant)
- Strukturellem Kontext (Neo4j)
- Abhängigkeitsanalyse
```
---
## 🛠️ n8n Integration (Phase 3)
### Workflow: Graph-Sync Trigger
```
[Memos Webhook]
[Parse Content]
[SAIA: Extract Entities]
├→ Tasks, Projects, Technologies, People
└→ Relationships (uses, depends_on, mentions)
[Neo4j: Create/Update Nodes & Relationships]
[Index in Qdrant Metadata: Add graph_node_id]
```
### Workflow: Question Router
```
[#nexa:ask Query]
[Parallel Search]
├→ [Qdrant: Vector Similarity]
├→ [Neo4j: Graph Query (auto-generated Cypher)]
└→ [Rank & Merge Results]
[SAIA: Generate Answer with Context]
```
---
## 📋 Commands für Graph-Management
### #nexa:graph-status
Zeigt Statistiken der Graph-DB
```
#nexa:graph-status
```
Response:
```
📊 Neo4j Graph Status:
- Nodes: 342 (Tasks: 145, Projects: 12, Technologies: 52, Notes: 133)
- Relationships: 1,247
- Density: 0.34
- Most Connected: "JWT-Middleware" (27 connections)
```
### #nexa:graph-trace [entity]
Zeigt alle Verbindungen für eine Entität
```
#nexa:graph-trace JWT-Middleware
```
### #nexa:graph-rebuild
Rekonstruiert Graph aus Obsidian + Memos (einmalig, danach inkrementell)
```
#nexa:graph-rebuild --force
```
---
## 🚀 Phase 3 Roadmap (Graph Integration)
- [ ] **3.1 Neo4j Setup**: Collection, Constraints, Indexes
- [ ] **3.2 Graph-Sync Workflow**: Automatische Entity Extraction
- [ ] **3.3 GraphRAG Pipeline**: Kombinierte Qdrant + Neo4j Queries
- [ ] **3.4 Dependency Tracker**: Warnt vor zirkulären Dependencies
- [ ] **3.5 Critical Path Analysis**: Berechnet längsten Weg im Graph
---
## 💡 Warum zwei DBs?
| Szenario | Qdrant | Neo4j | Kombi |
|----------|--------|-------|-------|
| "Welche Note war ähnlich?" | ✅ Best | ❌ | Qdrant |
| "Welche Tasks blockieren diesen Task?" | ❌ | ✅ Best | Neo4j |
| "Erkläre mir dieses Projekt umfassend" | ✅ Context | ✅ Structure | **Beide!** |
| "Finde alle JWT-bezogenen offenen Work" | ✅ Semantic | ✅ Crisp | **Beide!** |
Kombiniert: **Vollständiges Verständnis** statt nur Suchindex oder nur Struktur.
-80
View File
@@ -1,80 +0,0 @@
📑 Projekt Nexa: Scoping & Vision
1. Vision & Zielsetzung
Nexa ist als "Zentrales Nervensystem" konzipiert. Das Ziel ist die Schaffung einer intelligenten Middleware zwischen Input-Quellen (Memos, E-Mail, RSS), Wissensspeichern (Obsidian, Qdrant) und Organisations-Tools (Nextcloud Calendar/Tasks).
Kernziele:
Kognitive Entlastung: Nexa sortiert, filtert und schlägt vor, anstatt nur zu speichern.
Kontext-Trennung: Saubere, KI-gestützte Unterscheidung zwischen Privat und Arbeit.
Single Point of Interaction: Memos fungiert als primäre Schnittstelle ("The Voice & Ear").
Wissens-Synergie: Verknüpfung von flüchtigen Memos mit tiefem Wissen in Obsidian via semantischer Suche.
2. Abgrenzung (Scope)
Um die Komplexität beherrschbar zu halten, wird das Projekt wie folgt abgegrenzt:
In-Scope (Was Nexa tut):
Triaging: Klassifizierung von 20 Mails/Tag (Privat).
Task-Routing: Verteilung von Aufgaben auf die richtige Nextcloud-Liste (Work vs. Personal).
Scheduling: Vorschlagen von Timeboxen im Arbeitskalender.
Memory: Aufbau eines Langzeitgedächtnisses in Qdrant.
Monitoring: Aggregation kritischer IT-Alarme (Proxmox, Backrest).
Out-of-Scope (Was Nexa NICHT tut):
Kein direkter Zugriff auf Arbeits-IT (E-Mail/Server).
Keine aktive Bearbeitung von Dokumenten (außer Metadaten-Extraktion).
Keine Ersetzung von Spezial-UIs (z.B. Arcane oder Portainer für Container-Management).
3. Phasenmodell der Entwicklung
Der Aufbau erfolgt iterativ, um frühzeitig Mehrwert zu generieren.
Phase 1: Die Wirbelsäule (Konnektivität)
Fokus: Datenfluss zwischen Memos, n8n und SAIA.
Meilenstein: Nexa antwortet auf Memos und kann einfache Fragen beantworten.
Technik: Webhooks, REST-API, LiteLLM-Proxy.
Phase 2: Sensorik & Klassifizierung (The Senses)
Fokus: E-Mail-Filterung und das Routing-System.
Meilenstein: Nexa erkennt den Unterschied zwischen "Arbeits-Task" und "Privat-Task" ohne manuelle Tags.
Technik: IMAP-Node, n8n-Logic-Router, System-Prompts für Kontext-Analyse.
Phase 3: Das semantische Gedächtnis (The Brain)
Fokus: Anbindung von Qdrant und Obsidian.
Meilenstein: RAG (Retrieval Augmented Generation). Nexa beantwortet Fragen basierend auf deinen archivierten Notizen.
Technik: Embeddings-Pipeline, Qdrant-Vektor-Suche.
Phase 4: Proaktive Motorik (The Assistant)
Fokus: Kalender-Integration und Timeboxing.
Meilenstein: Nexa schlägt morgens proaktiv Fokus-Zeiten im Arbeitskalender vor.
Technik: Nextcloud CalDAV/CardDAV Integration, Optimierungs-Logik für freie Slots.
Phase 5: Physische Präsenz (Voice & Alarms)
Fokus: Home Assistant Voice & Monitoring-Integration.
Meilenstein: Nexa spricht über HA-Voice und meldet kritische Systemzustände proaktiv.
Technik: Home Assistant API, Wyoming Protocol, ntfy/Memos-Push.
4. Erfolgsmetriken
Woran merken wir, dass Nexa funktioniert?
Reduktion der Mail-Interaktion: Weniger Zeit in der Mail-App, mehr Fokus auf den Nexa-Digest.
Kalender-Füllrate: Fokus-Slots für Arbeit sind automatisch geblockt.
Such-Geschwindigkeit: Informationen aus alten Memos/Obsidian werden via #ask sofort gefunden.