From 9b18e2710d4a586c1c5940ed8a89559b5624f373 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:18:28 +0000 Subject: [PATCH] Reorganize /docs with numbered TOC; deployment rewrite for actual homelab - /docs/ now contains 12 numbered guides + index.md (navigator). - Old duplicates removed: nexa-core/doc/, DEPLOYMENT.md, PLAN.md, config/{classification_logic,workflows_spec}.md. - 09-deployment.md targets the running infra (LXC 104 docker host, Zoraxy at 192.168.1.4, existing Memos/n8n/LiteLLM/Nextcloud/Qdrant) rather than spinning a parallel stack; minimal-config approach. - 11-open-questions.md tracks user-info-required blockers. - 12-optimization-opportunities.md captures homelab tweaks. - CLAUDE.md added so future agent runs know repo conventions and infra. --- CLAUDE.md | 41 ++ README.md | 18 +- docs/01-vision-and-scope.md | 34 + docs/02-roadmap.md | 48 ++ docs/03-architecture.md | 63 ++ .../04-integration-matrix.md | 2 +- .../05-command-system.md | 2 +- .../06-classification-logic.md | 2 +- .../07-workflow-spec.md | 2 +- .../08-graphrag-architecture.md | 6 +- docs/09-deployment.md | 193 ++++++ docs/10-operations.md | 95 +++ docs/11-open-questions.md | 29 + docs/12-optimization-opportunities.md | 25 + docs/index.md | 50 ++ nexa-core/DEPLOYMENT.md | 629 ------------------ nexa-core/PLAN.md | 27 - nexa-core/README.md | 26 +- nexa-core/doc/scope.md | 80 --- 19 files changed, 618 insertions(+), 754 deletions(-) create mode 100644 CLAUDE.md create mode 100644 docs/01-vision-and-scope.md create mode 100644 docs/02-roadmap.md create mode 100644 docs/03-architecture.md rename nexa-core/doc/integration_matrix.md => docs/04-integration-matrix.md (98%) rename nexa-core/doc/command_system.md => docs/05-command-system.md (98%) rename nexa-core/config/classification_logic.md => docs/06-classification-logic.md (95%) rename nexa-core/config/workflows_spec.md => docs/07-workflow-spec.md (93%) rename nexa-core/doc/graphrag_architecture.md => docs/08-graphrag-architecture.md (95%) create mode 100644 docs/09-deployment.md create mode 100644 docs/10-operations.md create mode 100644 docs/11-open-questions.md create mode 100644 docs/12-optimization-opportunities.md create mode 100644 docs/index.md delete mode 100644 nexa-core/DEPLOYMENT.md delete mode 100644 nexa-core/PLAN.md delete mode 100644 nexa-core/doc/scope.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..95fbf1f --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,41 @@ +# Instructions for Claude (and other agents) + +This file tells future automated runs what they need to know about this repo. + +## Repo conventions + +- **Documentation:** all docs live in `/docs/` and are numbered. Entry point is `docs/index.md`. When you add a doc, give it the next free `NN-` prefix and add a row to the index TOC. +- **Runtime artifacts:** live in `nexa-core/` (workflows, prompts, configs, scripts). Don't put `.md` documentation in there — link from `/docs/` instead. +- **Source-of-truth:** if a doc duplicates content from `nexa-core/config/*.md`, **delete the duplicate**. Single source of truth. + +## Real infrastructure (verified from screenshots, May 2026) + +- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9). + - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. + - LXC 103 backrest — backup orchestration. + - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). + - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. + - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). + - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. + - VM 100 haos — Home Assistant. +- **Already-running services on docker host** (don't redeploy): + - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). + +## When working on Nexa + +1. **Read `docs/index.md` first** — it's the navigator. +2. **Open questions first.** Before writing code or workflow JSON, scan `docs/11-open-questions.md`. If your task touches an unanswered Q, **stop and ask** rather than picking a default. Append new blockers to that doc as `[ ] Q-NN`. +3. **Optimization findings.** When you spot infrastructure improvements, add them to `docs/12-optimization-opportunities.md` as a numbered bullet — don't just mention them in commit messages. +4. **Never inline secrets** in workflow JSON or `.env` committed to git. Use n8n credentials, LiteLLM virtual keys, or (longer term) Vaultwarden. +5. **Keep deployment minimal.** The default answer to *"do we need a new container?"* is **no** — the existing stack covers most needs. + +## Branch policy + +- This branch is `claude/organize-docs-deployment-7N4v2`. Push only here unless told otherwise. +- New work for an unrelated feature → new branch under `claude/`. + +## Quick links + +- [docs/index.md](./docs/index.md) +- [docs/09-deployment.md](./docs/09-deployment.md) — most-touched file during bring-up +- [docs/11-open-questions.md](./docs/11-open-questions.md) — read **before** assuming defaults diff --git a/README.md b/README.md index f77a05e..fdb68b0 100644 --- a/README.md +++ b/README.md @@ -1 +1,17 @@ -# nexa \ No newline at end of file +# nexa + +**Neural Nexus for Information & Automation** — central nervous system for a personal IT setup. + +> Memos = voice & ear · n8n = reflexes · SAIA (LiteLLM) = brain · Qdrant (+ optional graph DB) = memory · Nextcloud = hands. + +## Start here + +📖 **[docs/index.md](./docs/index.md)** — TOC, reading paths, repo layout. + +## Repository layout + +``` +nexa/ +├── docs/ ← all documentation, numbered for reading order +└── nexa-core/ ← runtime: n8n workflows, configs, prompts, scripts +``` diff --git a/docs/01-vision-and-scope.md b/docs/01-vision-and-scope.md new file mode 100644 index 0000000..d6d9366 --- /dev/null +++ b/docs/01-vision-and-scope.md @@ -0,0 +1,34 @@ +# 01 — Vision & Scope + +## Vision + +Nexa is the **central nervous system** of a personal IT setup: an intelligent middleware sitting between input sources (Memos, e-mail, RSS, Karakeep, Bluesky), knowledge stores (Obsidian, Qdrant, Nextcloud) and organization tools (Nextcloud Calendar/Tasks). + +### Core goals + +- **Cognitive offload** — Nexa sorts, filters and proposes; it doesn't just store. +- **Context separation** — clean AI-driven split between *Work* and *Personal*. +- **Single point of interaction** — Memos is the primary surface ("voice & ear"). +- **Knowledge synergy** — link ephemeral memos to deep notes via semantic search. + +## Scope + +### In-scope + +- Triaging ~20 personal e-mails per day. +- Routing tasks to the right Nextcloud list (Work vs. Personal). +- Suggesting time-boxes in the work calendar. +- Long-term memory in Qdrant (and optionally a graph DB). +- Aggregating critical IT alerts (Proxmox, Backrest) into a Memos system feed. + +### Out-of-scope + +- Direct access to work IT (work mail, work servers). +- Active document editing (metadata extraction only). +- Replacing specialist UIs (Arcane / Dozzle / Portainer for containers). + +## Success metrics + +- **Reduced mail interaction** — less time in the mail app, more on the Nexa digest. +- **Calendar fill-rate** — focus slots auto-blocked on the work calendar. +- **Search latency** — old memos / Obsidian notes resurface instantly via `#nexa:ask`. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md new file mode 100644 index 0000000..fc59f97 --- /dev/null +++ b/docs/02-roadmap.md @@ -0,0 +1,48 @@ +# 02 — Roadmap & Phases + +Iterative build-out, value-first. Each phase is shippable on its own. + +## Phase 1 — The Spine (connectivity) + +**Focus:** datapath between Memos, n8n and SAIA. +**Milestone:** Nexa replies on Memos and answers simple questions. + +- [ ] **1.1 Memos ↔ n8n bridge** — webhook ingress, content filter, comment-back. +- [ ] **1.2 SAIA integration** — call LiteLLM proxy from n8n; chat via Memos comments. +- [ ] **1.3 Git-sync** — periodic export of n8n workflows into this repo. + +## Phase 2 — Senses (input channels) + +**Focus:** e-mail filter and the Work-vs-Personal router. +**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags. + +- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails. +- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. +- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets. + +## Phase 3 — Memory (Qdrant & RAG) + +**Focus:** Qdrant + (optional) graph DB for retrieval-augmented answers. +**Milestone:** RAG works — Nexa answers from archived notes. + +- [ ] **3.1 Embedding pipeline** — automatic vectorization of Memos & Obsidian. +- [ ] **3.2 Context retrieval** — `#nexa:ask` reads Qdrant before answering. +- [ ] **3.3 S3 long-term archive** — Qdrant snapshots to the S3 server. +- [ ] **3.4 Graph layer (deferred decision — see [11-open-questions](./11-open-questions.md))** — Neo4j *or* Ontotext GraphDB for structural queries. + +## Phase 4 — Motor (organization & action) + +**Focus:** calendar integration and time-boxing. +**Milestone:** Nexa proactively proposes morning focus slots in the work calendar. + +- [ ] **4.1 Nextcloud Tasks sync** — bidirectional status sync (Memos ↔ NC Tasks). +- [ ] **4.2 Time-boxing agent** — calendar-aware slot suggestions. +- [ ] **4.3 Karakeep curation** — automated review cycle for saved links. + +## Phase 5 — Daily integration (polish) + +**Focus:** Home Assistant Voice and monitoring. +**Milestone:** Nexa speaks via HA-Voice and surfaces critical system states proactively. + +- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol. +- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy. diff --git a/docs/03-architecture.md b/docs/03-architecture.md new file mode 100644 index 0000000..544df46 --- /dev/null +++ b/docs/03-architecture.md @@ -0,0 +1,63 @@ +# 03 — Architecture Overview + +A one-page mental model. For details follow the cross-links. + +## Topology + +``` + ┌────────────────┐ + voice / typing ──────▶│ Memos │◀──── Nexa replies as comments + │ (interface) │ + └───────┬────────┘ + │ webhook (- [ ] / #nexa:*) + ▼ + ┌────────────────┐ + IMAP / RSS / NC ────▶│ n8n │◀──── workflows live in + Karakeep / Bluesky │ (logic) │ ./nexa-core/n8n-workflows + └───┬────────┬───┘ + classify ▲ │ │ ▲ retrieve + │ ▼ ▼ │ + ┌──────────┐ ┌──────────┐ + │ SAIA │ │ Qdrant │ + │ LiteLLM │ │ vector │ + └──────────┘ └────┬─────┘ + │ optional + ▼ + ┌──────────┐ + │ Graph │ (Neo4j or Ontotext — + │ DB │ see open questions) + └──────────┘ + │ + ▼ writes + ┌──────────────────────────────┐ + │ Nextcloud (Tasks, Calendar, │ + │ Mail, Files / Obsidian) │ + └──────────────────────────────┘ +``` + +## Components + +| Component | Role | Where it runs (today) | +|-----------|------|-----------------------| +| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | +| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | +| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | +| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 (existing `qdrant_scientific` may be reused — see [11](./11-open-questions.md)) | +| **Graph DB** | Structural memory (deferred) | not yet deployed | +| **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` | +| **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` | +| **Backrest** | Backup orchestration | LXC 103 | +| **Zoraxy** | Reverse proxy + TLS | LXC 108 (`192.168.1.4:8000`) | +| **AdGuard DNS** | Internal name resolution | LXC 102 | +| **Home Assistant** | Voice + house automation | VM 100 (HAOS) | + +## Two-pillar memory + +- **Qdrant** answers *"what is similar / relevant?"* (cosine search over embeddings). +- **Graph DB** answers *"who, what depends on what, how is it structured?"* (Cypher / SPARQL). + +Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. See [08 — GraphRAG architecture](./08-graphrag-architecture.md). + +## Dual-context routing + +Every input is classified `work` or `personal` before any side effect (task creation, calendar write). See [04 — Integration matrix](./04-integration-matrix.md) and [06 — Classification logic](./06-classification-logic.md). diff --git a/nexa-core/doc/integration_matrix.md b/docs/04-integration-matrix.md similarity index 98% rename from nexa-core/doc/integration_matrix.md rename to docs/04-integration-matrix.md index dfc7da7..2fab535 100644 --- a/nexa-core/doc/integration_matrix.md +++ b/docs/04-integration-matrix.md @@ -1,4 +1,4 @@ -# 🔗 Nexa Integrations-Matrix (V2) +# 04 — Integrations-Matrix Diese Matrix definiert die logische Trennung zwischen privaten und beruflichen Datenströmen sowie die Anbindung der Infrastruktur. diff --git a/nexa-core/doc/command_system.md b/docs/05-command-system.md similarity index 98% rename from nexa-core/doc/command_system.md rename to docs/05-command-system.md index d497fc9..4d5143d 100644 --- a/nexa-core/doc/command_system.md +++ b/docs/05-command-system.md @@ -1,4 +1,4 @@ -# Nexa Command System - #nexa:command +# 05 — Command System (`#nexa:*`) Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag: diff --git a/nexa-core/config/classification_logic.md b/docs/06-classification-logic.md similarity index 95% rename from nexa-core/config/classification_logic.md rename to docs/06-classification-logic.md index 3e8321c..4a5890f 100644 --- a/nexa-core/config/classification_logic.md +++ b/docs/06-classification-logic.md @@ -1,4 +1,4 @@ -# Nexa Klassifizierungs-Logik +# 06 — Klassifizierungs-Logik Dieser Fragebogen dient der Feinabstimmung von SAIA, um Tasks korrekt zu routen. diff --git a/nexa-core/config/workflows_spec.md b/docs/07-workflow-spec.md similarity index 93% rename from nexa-core/config/workflows_spec.md rename to docs/07-workflow-spec.md index 17b1b26..2960f43 100644 --- a/nexa-core/config/workflows_spec.md +++ b/docs/07-workflow-spec.md @@ -1,4 +1,4 @@ -# Spezifikation: Phase 2 - Task-Router +# 07 — Workflow Spec: Phase-2 Task-Router ## 1. Trigger * **Typ:** Webhook (Memos) diff --git a/nexa-core/doc/graphrag_architecture.md b/docs/08-graphrag-architecture.md similarity index 95% rename from nexa-core/doc/graphrag_architecture.md rename to docs/08-graphrag-architecture.md index 40e4184..430b9fa 100644 --- a/nexa-core/doc/graphrag_architecture.md +++ b/docs/08-graphrag-architecture.md @@ -1,4 +1,8 @@ -# GraphRAG für Nexa - Strukturelles Wissen & Beziehungen +# 08 — GraphRAG: Structural Knowledge & Relations + +> ⚠️ **Open decision (see [11-open-questions](./11-open-questions.md))**: this document describes the design with **Neo4j**. The earlier deployment draft referenced **Ontotext GraphDB** (SPARQL). Pick one before Phase 3.4. + +## GraphRAG für Nexa — Strukturelles Wissen & Beziehungen ## 🧠 Zwei-Säulen-Memory-Architektur diff --git a/docs/09-deployment.md b/docs/09-deployment.md new file mode 100644 index 0000000..7358e64 --- /dev/null +++ b/docs/09-deployment.md @@ -0,0 +1,193 @@ +# 09 — Deployment + +Pragmatic deployment guide that **assumes the existing homelab** and adds only what's missing. + +## What's already running (no action required) + +Surveyed from Homepage / Dozzle / Proxmox / Zoraxy: + +| Service | Host / port | URL | +|---------|-------------|-----| +| Memos | docker LXC 104 → `:5230` | `https://memos.nuclide.systems` | +| n8n | docker LXC 104 → `:5678` | `https://n8n.nuclide.systems` | +| LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) | +| Nextcloud | LXC 105 | `https://nc.nuclide.systems` | +| ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` | +| Karakeep (Hoarder) | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` | +| Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` | +| Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` | +| Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` | +| Backrest | LXC 103 | (internal) | +| AdGuard DNS | LXC 102 | (internal) | +| Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems | +| `qdrant_scientific` (existing) | docker LXC 104 | reuse — see [11](./11-open-questions.md) | + +The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**. + +## What's missing for Nexa + +1. **Qdrant collection** for Nexa (`nexa_knowledge`) — either on the existing `qdrant_scientific` or a dedicated container. +2. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n. +3. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`). +4. **Memos webhook → n8n** wired through the Memos config. +5. **LiteLLM virtual key** for the `nexa` user with embedding + chat models. +6. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied. +7. **(Phase 3.4)** decision on graph DB (Neo4j vs. Ontotext GraphDB) — see [11](./11-open-questions.md). + +--- + +## Step 1 — Secrets + +Copy `nexa-core/.env.example` → `nexa-core/.env` and fill **only** the secrets: + +```bash +cd nexa-core +cp .env.example .env +$EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY +``` + +The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead. + +## Step 2 — Qdrant collection + +```bash +# adjust QDRANT_HOST in .env first +source nexa-core/.env + +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \ + -H "Content-Type: application/json" \ + -H "api-key: $QDRANT_API_KEY" \ + -d @nexa-core/config/qdrant_schema.json +``` + +> If reusing `qdrant_scientific`, this just adds another collection; no new container needed. + +## Step 3 — LiteLLM virtual key + +In the LiteLLM admin UI (`ai.nuclide.systems`): + +1. Create user `nexa`. +2. Issue a virtual key with access to one chat model (e.g. `gpt-4o-mini` / a local Sonnet equivalent) and one embedding model (`text-embedding-3-small`). +3. Paste the key into `SAIA_API_KEY` in `.env`. + +## Step 4 — n8n workflows + +Import the JSON exports — credentials are filled inside n8n, not in the JSON: + +```bash +# n8n personal access token from the n8n UI: Settings → API +N8N_URL=https://n8n.nuclide.systems +N8N_TOKEN=... # from the n8n UI + +for f in nexa-core/n8n-workflows/phase-1/*.json \ + nexa-core/n8n-workflows/phase-2/*.json; do + curl -X POST "$N8N_URL/api/v1/workflows" \ + -H "X-N8N-API-KEY: $N8N_TOKEN" \ + -H "Content-Type: application/json" \ + --data-binary "@$f" +done +``` + +Inside n8n, attach credentials to the imported nodes: + +- **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY` +- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` +- **Nextcloud** → app password +- **Qdrant** → header `api-key: $QDRANT_API_KEY` + +Activate each workflow individually after smoke-test. + +## Step 5 — Memos webhook + +In the Memos admin UI, set the webhook URL to the production address of the discovery workflow: + +``` +https://n8n.nuclide.systems/webhook/memos +``` + +The same URL is the one the workflow exposes; verify with: + +```bash +curl -i https://n8n.nuclide.systems/webhook/memos +# expect 200 / 405, never 404 +``` + +## Step 6 — Bootstrap commands via Memos + +Create a memo with body `#nexa:config` — the discovery workflow: + +1. Lists Nextcloud Tasks lists, picks Work / Personal / Shopping / Wishes by name. +2. Counts existing Qdrant points in `nexa_knowledge`. +3. Verifies LiteLLM reachability + lists available models. +4. Replies as a comment with a **runtime-config snapshot** that's stored as Qdrant metadata (`_config` namespace) and as `nexa-core/config/runtime_config.json` (gitignored). + +After this point, `.env` is read-once. Subsequent runs read config from Qdrant. + +## Step 7 — Smoke tests + +```bash +# (1) Memos round-trip — should produce a comment within ~5 s +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"content":"- [ ] testing the router #nexa"}' + +# (2) Classification dry-run +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"#nexa:route-test buy milk"}' + +# (3) RAG test (requires at least one indexed memo/note) +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"#nexa:ask what is the goal of nexa?"}' +``` + +## Step 8 — Reverse proxy + +Already done — Zoraxy at `192.168.1.4:8000` terminates TLS for `*.nuclide.systems` and forwards to docker LXC 104 (`192.168.1.40`). **No new entry is required for Nexa**: every service Nexa talks to already has a host entry. + +## Step 9 — Backups + +Already covered by Backrest (LXC 103). Add: + +- **n8n workflows** → `nexa-core/scripts/backup_workflows.sh` (already present) into a Backrest schedule. +- **Qdrant snapshots** → schedule a daily `POST /collections/nexa_knowledge/snapshots` and rsync to S3 (`s3.nuclide.systems`). Add as a Backrest pre-hook on the docker host. + +For deeper detail: [10 — Operations](./10-operations.md). + +--- + +## Phase add-on: graph DB (Phase 3.4) + +Defer until 3.1–3.3 ship. When needed, two minimal options — pick one: + +### Option A — Neo4j (lighter, popular for graph-RAG) + +```yaml +# nexa-core/docker-compose.graph.yml +services: + neo4j: + image: neo4j:5-community + container_name: nexa-neo4j + ports: ["7687:7687", "7474:7474"] + environment: + NEO4J_AUTH: "neo4j/${NEO4J_PASSWORD}" + NEO4J_server_memory_heap_max__size: 2G + volumes: ["./data/neo4j:/data"] + restart: unless-stopped +``` + +### Option B — Ontotext GraphDB (SPARQL, RDF native) + +Keep the snippet from the original `DEPLOYMENT.md` Schritt 1.3 for `graphdb`. Heavier (~4 GB heap), only justified if you want SHACL / OWL reasoning. + +Add a Zoraxy host entry `graph.nuclide.systems` → `192.168.1.40:7474` (Neo4j) or `:7200` (GraphDB) **only if** browser access is desired; otherwise n8n talks to it on the docker network. + +--- + +## Step-back / rollback + +- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops. +- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`. +- **Re-discover** — `#nexa:reset-config` then `#nexa:config`. diff --git a/docs/10-operations.md b/docs/10-operations.md new file mode 100644 index 0000000..4c69609 --- /dev/null +++ b/docs/10-operations.md @@ -0,0 +1,95 @@ +# 10 — Operations + +Day-2 concerns. Backups, monitoring, troubleshooting. + +## Backups + +Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items: + +| What | How | Frequency | +|------|-----|-----------| +| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container | +| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) | +| Memos DB | Backrest snapshot of Memos data dir | already covered | +| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered | +| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | + +## Monitoring + +| Signal | Source | Sink | +|--------|--------|------| +| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks | Memos system feed via ntfy | +| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic | +| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest | +| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest | +| Backrest run status | Backrest webhook | ntfy | + +A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo. + +## Troubleshooting + +### Memos webhook not firing + +```bash +# is the Memos webhook config still pointing at n8n? +curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \ + https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks' + +# does n8n still expose the path? +curl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404 +``` + +If 404 → workflow is inactive in n8n. Activate. + +### LiteLLM 401 / 429 + +- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential. +- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk. + +### Qdrant `nexa_knowledge` empty + +```bash +# is the indexing workflow active? +docker logs nexa-n8n 2>&1 | grep memos_bridge | tail + +# manual index test +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"manual probe #nexa"}' + +# point count +curl -s -H "api-key: $QDRANT_API_KEY" \ + $QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count' +``` + +### Wrong list routing (Work vs Personal) + +1. Run `#nexa:route-test ` and check the confidence value. +2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)). +3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change. + +### "Where did Nexa store this?" + +```bash +# state dump +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -d '{"content":"#nexa:export-state"}' +# returns runtime_config + counters as a JSON memo +``` + +## Log locations + +| Component | Where | +|-----------|-------| +| Memos | `docker logs nexa-memos` (LXC 104) | +| n8n | `docker logs nexa-n8n` | +| LiteLLM | `docker logs litellm` | +| Qdrant | `docker logs qdrant_scientific` | +| Zoraxy | LXC 108 web UI → Statistical Analysis | +| Backrest | LXC 103 web UI | + +For a one-shot dump: + +```bash +ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log +``` diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md new file mode 100644 index 0000000..ff5e7c8 --- /dev/null +++ b/docs/11-open-questions.md @@ -0,0 +1,29 @@ +# 11 — Open Questions (user-info-required) + +Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide. + +## Architectural decisions + +- [ ] **Q1 — Graph DB choice.** Phase 3.4 says *Neo4j* in [08](./08-graphrag-architecture.md) but the original deployment draft used *Ontotext GraphDB*. **Recommendation:** Neo4j (smaller footprint, Cypher is friendlier for n8n HTTP nodes, no SHACL/OWL needs identified). Confirm or override. +- [ ] **Q2 — Reuse `qdrant_scientific` or run a dedicated `nexa-qdrant`?** Reuse keeps memory usage flat (~670 MB seen on Homepage); dedicated gives clean isolation and independent snapshot lifecycle. **Recommendation:** reuse with a `nexa_*` collection prefix. +- [ ] **Q3 — Embeddings model.** Default in `infra_manifest.yaml` is `text-embedding-3-small` (OpenAI). Is the LiteLLM virtual key authorised to call it, or should we route to a self-hosted alternative (e.g. `bge-m3` via Ollama on the same host)? +- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. +- [ ] **Q5 — Karakeep vs. Hoarder naming.** The Zoraxy host is `hoarder.nuclide.systems` but the container is `karakeep-*` and Homepage labels it Karakeep. They're the same product (rename in 2024). Pick one display name to use in docs and prompts. + +## Identifiers needed (auto-discoverable, but list now if known) + +- [ ] **Q6 — Nextcloud Tasks list IDs** for: `Work_Tasks`, `Personal_Tasks`, `Shopping`, `Wishes`. Discovery via `#nexa:config` will fill these — confirm names match. +- [ ] **Q7 — Nextcloud Calendar IDs** for: `Work_Calendar`, primary personal calendar. +- [ ] **Q8 — IMAP credentials for the personal mail account.** Can n8n reuse a Nextcloud Mail account (preferred — no extra password) or must we add a dedicated IMAP entry? +- [ ] **Q9 — ntfy topic name** for `nexa.system`. Is the topic public on `ntfy.nuclide.systems` or should it be authenticated? +- [ ] **Q10 — Pocket-ID role.** `id.nuclide.systems` is running. Do we want SSO in front of the n8n / Memos UIs, or skip for now? + +## Hardware / capacity + +- [ ] **Q11 — RAM headroom on docker host.** Homepage shows 29.5 GiB free out of ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing of Obsidian + Memos history is fine here, but a Phase-3.4 GraphDB choice changes this (Neo4j ~2 GB heap, Ontotext ~4 GB). Confirm acceptable. +- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots? + +## Process + +- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but flagging it because Phase 5 monitoring would alert on it. Suppress, or is it intentional? +- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage — Homepage doesn't have a Zoraxy widget yet. Cosmetic, unrelated to Nexa. diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md new file mode 100644 index 0000000..77c5f28 --- /dev/null +++ b/docs/12-optimization-opportunities.md @@ -0,0 +1,25 @@ +# 12 — Optimization Opportunities + +Observations from the running infrastructure. Each item is independent — accept, defer, or reject. + +## For Nexa directly + +1. **Reuse, don't redeploy.** The earlier `DEPLOYMENT.md` would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new [09-deployment](./09-deployment.md) treats these as pre-existing — keeps the config minimal and avoids port collisions. +2. **Use LiteLLM virtual keys per logical caller.** Today there's one SAIA key. Issuing one key per workflow (`nexa-router`, `nexa-embed`, `nexa-digest`) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything. +3. **Use n8n's *credential* objects, never inline secrets.** The current workflows under `nexa-core/n8n-workflows/phase-1/*.json` should be reviewed — if any header `Authorization` is hardcoded, replace with credential references before importing. +4. **Centralise system alerts on a single ntfy topic** (`nexa.system`). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them. +5. **Defer graph DB until Phase 3.4.** Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries. +6. **Auto-export n8n workflows.** `nexa-core/scripts/backup_workflows.sh` already exists. Schedule it inside the n8n container (cron) and let it `git commit && git push` — this is the cheapest disaster recovery. + +## For the wider homelab (out of scope but worth noting) + +7. **Octoprint `EXITED`** — either remove the Homepage entry or fix the container; right now it permanently shows red. +8. **Zoraxy widget missing** on Homepage. Homepage v1.x has no built-in Zoraxy widget, but you can use the generic `customapi` widget against `/api/stats/summary` to surface request counts. +9. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. +10. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). +11. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. +12. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. +13. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. +14. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. +15. **Immich + Qdrant.** Immich is already running and embeds photos with CLIP. If you ever want photo-aware Nexa answers, you can mount Immich's existing search via its API — no second embedding pass. +16. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..37c2bba --- /dev/null +++ b/docs/index.md @@ -0,0 +1,50 @@ +# NEXA Documentation + +**Neural Nexus for Information & Automation** — the central nervous system that ties Memos, n8n, SAIA (LiteLLM), Nextcloud and a vector store into one assistant. + +This is the documentation entry point. Read top-to-bottom for first-time setup, or jump to the section you need. + +--- + +## Table of Contents + +| # | Document | Read when… | +|---|----------|------------| +| 01 | [Vision & Scope](./01-vision-and-scope.md) | You want to understand *what* Nexa is and isn't. | +| 02 | [Roadmap & Phases](./02-roadmap.md) | You want to know the implementation order. | +| 03 | [Architecture Overview](./03-architecture.md) | You need a one-page mental model. | +| 04 | [Integration Matrix](./04-integration-matrix.md) | You're wiring up a new data source or mapping work-vs-personal flows. | +| 05 | [Command System](./05-command-system.md) | You want to know what `#nexa:*` commands do. | +| 06 | [Classification Logic](./06-classification-logic.md) | You're tuning the work/personal router. | +| 07 | [Workflow Spec — Task Router](./07-workflow-spec.md) | You're building the Phase-2 router workflow. | +| 08 | [GraphRAG Architecture](./08-graphrag-architecture.md) | You're working on Phase 3 (Qdrant + Graph). | +| 09 | [Deployment](./09-deployment.md) | You're bringing Nexa up on the real infrastructure. | +| 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. | +| 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. | +| 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. | + +--- + +## Reading paths + +- **First-time installer:** 01 → 03 → 09 → 10 → 11 +- **Workflow author:** 04 → 05 → 07 → 06 +- **Phase-3 work (memory):** 08 → 09 (§ Phase 3 add-on) +- **Stakeholder / quick read:** 01 → 02 → 12 + +--- + +## Repository layout + +``` +nexa/ +├── README.md → points here +├── docs/ → you are here +└── nexa-core/ → the actual project + ├── ai-prompts/ → SAIA system prompts + ├── config/ → runtime config (YAML, JSON schema) + ├── n8n-workflows/ → exported workflows, version-controlled + └── scripts/ → automation helpers +``` + +Source-of-truth for *runnable* config stays in `nexa-core/`. Documentation lives here in `docs/`. diff --git a/nexa-core/DEPLOYMENT.md b/nexa-core/DEPLOYMENT.md deleted file mode 100644 index 6a3b458..0000000 --- a/nexa-core/DEPLOYMENT.md +++ /dev/null @@ -1,629 +0,0 @@ -# NEXA Deployment Guide - -## 🚀 Architektur-Überblick - -``` -┌─────────────────────────────────────────────────┐ -│ MEMOS (Port 5230) │ -│ Interface, Voice-Input, Webhooks │ -└────────────────────┬────────────────────────────┘ - │ - ┌───────────┼───────────┐ - ↓ ↓ ↓ - ┌────────┐ ┌────────┐ ┌──────────┐ - │ n8n │ │ SAIA │ │ Nextcloud│ - │Logic │ │LiteLLM │ │Tasks/Mail│ - │7879 │ │4000 │ │80/443 │ - └────┬───┘ └────┬───┘ └─────┬────┘ - │ │ │ - ┌────┴───────────┼────────────┴────┐ - ↓ ↓ ↓ -┌─────────┐ ┌────────────┐ ┌──────────────┐ -│ QDRANT │ │ GraphDB │ │ Obsidian │ -│Vector │ │ (Ontotext) │ │(local sync) │ -│6333 │ │7200 │ │ │ -└─────────┘ └────────────┘ └──────────────┘ -``` - ---- - -## 📋 Voraussetzungen - -### Hardware -- **CPU**: Min. 4 Cores (empfohlen 8) -- **RAM**: Min. 16 GB (empfohlen 32 GB) -- **Storage**: Min. 100 GB SSD (für Datenbanken & Backups) -- **Network**: Stabile Verbindung, HTTPS-ready - -### Software -- Docker & Docker Compose (Version 20.10+) -- Git -- Bash -- curl/wget -- `.env` Datei aus `.env.example` (mit Secrets gefüllt) - -### Accounts & Credentials -- Nextcloud Instanz mit Admin-Zugang -- OpenAI API-Key (für SAIA/LiteLLM) -- Obsidian Vault Zugang lokalem Filesystem - ---- - -## 🐳 Schritt 1: Docker Compose Setup - -### 1.1 Repository klonen - -```bash -git clone https://github.com/noheton/nexa.git -cd nexa/nexa-core -``` - -### 1.2 .env konfigurieren - -```bash -cp .env.example .env -# WICHTIG: Alle Secrets eintragen! -# - MEMOS_API_KEY -# - SAIA_API_KEY -# - NC_APP_PASSWORD -# - QDRANT_API_KEY -# - GRAPHDB_PASSWORD -nano .env -``` - -### 1.3 docker-compose.yml erstellen - -```bash -cat > docker-compose.yml << 'EOF' -version: '3.8' - -services: - # ===== MEMOS (Interface) ===== - memos: - image: neosmemo/memos:latest - container_name: nexa-memos - ports: - - "5230:5230" - volumes: - - ./data/memos:/root/.memos - environment: - MEMOS_PORT: "5230" - MEMOS_DRIVER: sqlite - MEMOS_DSN: /root/.memos/memos.db - networks: - - nexa - restart: unless-stopped - - # ===== n8n (Workflow Engine) ===== - n8n: - image: n8nio/n8n:latest - container_name: nexa-n8n - ports: - - "5678:5678" - volumes: - - ./data/n8n:/home/node/.n8n - - ./n8n-workflows:/workflows - environment: - N8N_BASIC_AUTH_ACTIVE: "true" - N8N_BASIC_AUTH_USER: "${N8N_USER:-admin}" - N8N_BASIC_AUTH_PASSWORD: "${N8N_PASSWORD:-nexa_secure_pass}" - N8N_HOST: "${N8N_DOMAIN:-localhost}" - N8N_PORT: "5678" - N8N_PROTOCOL: https - N8N_WEBHOOK_URL: "https://${N8N_DOMAIN:-localhost}/webhook" - GENERIC_TIMEZONE: Europe/Berlin - DB_TYPE: sqlite - DB_SQLITE_PATH: /home/node/.n8n/nexa.db - networks: - - nexa - restart: unless-stopped - depends_on: - - memos - - # ===== QDRANT (Vector DB) ===== - qdrant: - image: qdrant/qdrant:latest - container_name: nexa-qdrant - ports: - - "6333:6333" - volumes: - - ./data/qdrant:/qdrant/storage - environment: - QDRANT_API_KEY: "${QDRANT_API_KEY}" - networks: - - nexa - restart: unless-stopped - - # ===== GraphDB (Ontotext SPARQL) ===== - graphdb: - image: ontotext/graphdb:10-ee - container_name: nexa-graphdb - ports: - - "7200:7200" - volumes: - - ./data/graphdb:/opt/graphdb/data - - ./config/graphdb-init.ttl:/graphdb-import/import.ttl - environment: - GDB_JAVA_OPTS: "-Xmx8g -Xms4g" - GDB_PASSWORD: "${GRAPHDB_PASSWORD}" - networks: - - nexa - restart: unless-stopped - - # ===== SAIA / LiteLLM Proxy (optional - wenn lokal) ===== - # (Falls SAIA extern gehostet: nicht nötig) - -networks: - nexa: - driver: bridge - -volumes: - nexa_memos: - nexa_n8n: - nexa_qdrant: - nexa_graphdb: -EOF -``` - ---- - -## 🔧 Schritt 2: Individuelle Service-Konfiguration - -### 2.1 MEMOS Initialisierung - -```bash -# Nach memos startup (ca. 30s) -curl -X POST http://localhost:5230/api/v1/auth/signup \ - -H "Content-Type: application/json" \ - -d '{ - "username": "nexa", - "password": "secure_password" - }' - -# API-Token generieren -TOKEN=$(curl -X POST http://localhost:5230/api/v1/auth/signin \ - -H "Content-Type: application/json" \ - -d '{"username":"nexa","password":"secure_password"}' \ - | jq -r '.data.token') - -echo "MEMOS_API_KEY=$TOKEN" >> .env -``` - -### 2.2 Qdrant Konfiguration - -```bash -# Warte auf Qdrant startup -sleep 10 - -# Erstelle Collection für Nexa -curl -X PUT "http://localhost:6333/collections/nexa_knowledge" \ - -H "Content-Type: application/json" \ - -H "api-key: ${QDRANT_API_KEY}" \ - -d '{ - "vectors": { - "size": 1536, - "distance": "Cosine" - }, - "optimizers_config": { - "deleted_threshold": 0.2, - "vacuum_threshold": 0.5, - "default_segment_number": 5, - "max_segment_number": 30, - "memmap_threshold": 268435456, - "indexing_threshold": 20000, - "flush_interval_sec": 10, - "max_optimization_threads": 4 - } - }' - -echo "✅ Qdrant Collection erstellt" -``` - -### 2.3 GraphDB Initialisierung - -```bash -# Warte auf GraphDB startup -sleep 15 - -# Erstelle Repository -GRAPHDB_URL="http://localhost:7200" - -curl -X POST "$GRAPHDB_URL/rest/repositories" \ - -H "Content-Type: application/json" \ - -u "admin:${GRAPHDB_PASSWORD}" \ - -d '{ - "id": "nexa_knowledge", - "title": "Nexa Knowledge Graph", - "type": "free", - "params": { - "baseURL": { - "label": "Base URL", - "value": "http://nexa.local/" - }, - "defaultLanguage": { - "label": "Default language", - "value": "en" - } - } - }' - -echo "✅ GraphDB Repository erstellt" -``` - -### 2.4 n8n Workflows Import - -```bash -# Warte auf n8n startup (ca. 60s) -sleep 60 - -# Importiere Discovery Workflow -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-1/1_1_discovery_workflow.json - -# Importiere Memos Bridge -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-1/1_2_memos_bridge_v2.json - -# Importiere Email Butler -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-2/2_1_email_butler.json - -echo "✅ n8n Workflows importiert" -``` - ---- - -## 🚀 Schritt 3: Deployment starten - -### 3.1 Docker Compose Up - -```bash -docker-compose up -d - -# Logs überwachen -docker-compose logs -f -``` - -### 3.2 Health Checks - -```bash -# Health Check Script -check_service() { - local name=$1 - local url=$2 - echo "Checking $name..." - if curl -s "$url" > /dev/null; then - echo "✅ $name: OK" - else - echo "❌ $name: FAILED" - fi -} - -sleep 30 - -check_service "Memos" "http://localhost:5230" -check_service "n8n" "http://localhost:5678" -check_service "Qdrant" "http://localhost:6333/health" -check_service "GraphDB" "http://localhost:7200/health" - -echo "" -echo "🎯 Service URLs:" -echo " - Memos: http://localhost:5230" -echo " - n8n: http://localhost:5678" -echo " - Qdrant: http://localhost:6333" -echo " - GraphDB: http://localhost:7200" -``` - ---- - -## ⚙️ Schritt 4: Konfiguration via Command System - -### 4.1 Discovery-Workflow triggern - -```bash -# In Memos Memo erstellen: -# #nexa:config - -# Oder via API: -curl -X POST http://localhost:5678/api/v1/workflows/run \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d '{ - "workflowId": "discovery-workflow-id" - }' -``` - -### 4.2 Automatische Konfiguration laden - -Nach erfolgreichem Discovery werden Nextcloud-Listen-IDs: - -```bash -# Es wird automatisch aktualisiert in: -# - Qdrant: als Metadata -# - n8n: als Umgebungsvariablen -# - Memos: als Confirmation-Memo -``` - ---- - -## 🔐 Schritt 5: Reverse Proxy Setup (Production) - -### 5.1 Nginx Configuration - -```bash -cat > /etc/nginx/sites-available/nexa.conf << 'EOF' -upstream memos_backend { - server localhost:5230; -} - -upstream n8n_backend { - server localhost:5678; -} - -upstream qdrant_backend { - server localhost:6333; -} - -upstream graphdb_backend { - server localhost:7200; -} - -server { - listen 443 ssl http2; - server_name nexa.yourdomain.com; - - ssl_certificate /etc/letsencrypt/live/nexa.yourdomain.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/nexa.yourdomain.com/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - ssl_ciphers HIGH:!aNULL:!MD5; - - # MEMOS - location / { - proxy_pass http://memos_backend; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_read_timeout 3600s; - proxy_send_timeout 3600s; - } - - # n8n - location /n8n/ { - proxy_pass http://n8n_backend/; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - } - - # Qdrant (nur intern, wenn nötig) - location /qdrant/ { - allow 10.0.0.0/8; - deny all; - proxy_pass http://qdrant_backend/; - } - - # GraphDB (nur intern) - location /graphdb/ { - allow 10.0.0.0/8; - deny all; - proxy_pass http://graphdb_backend/; - } - - # Webhook endpoint für externe Services - location /webhook/ { - proxy_pass http://n8n_backend/webhook/; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - } -} - -server { - listen 80; - server_name nexa.yourdomain.com; - return 301 https://$server_name$request_uri; -} -EOF - -# Aktivieren -sudo ln -s /etc/nginx/sites-available/nexa.conf /etc/nginx/sites-enabled/ -sudo nginx -t -sudo systemctl reload nginx -``` - ---- - -## 📊 Schritt 6: Monitoring & Backups - -### 6.1 Backup Script - -```bash -cat > scripts/backup_nexa.sh << 'EOF' -#!/bin/bash - -BACKUP_DIR="./backups/$(date +%Y%m%d_%H%M%S)" -mkdir -p "$BACKUP_DIR" - -echo "🔄 Backing up Nexa..." - -# Backup Datenbanken -docker-compose exec -T qdrant tar czf - /qdrant/storage > "$BACKUP_DIR/qdrant.tar.gz" -docker-compose exec -T graphdb tar czf - /opt/graphdb/data > "$BACKUP_DIR/graphdb.tar.gz" -docker-compose exec -T memos tar czf - /root/.memos > "$BACKUP_DIR/memos.tar.gz" -docker-compose exec -T n8n tar czf - /home/node/.n8n > "$BACKUP_DIR/n8n.tar.gz" - -# Backup Configs -cp .env "$BACKUP_DIR/.env.backup" -cp docker-compose.yml "$BACKUP_DIR/docker-compose.yml.backup" - -echo "✅ Backup completed: $BACKUP_DIR" - -# Optional: zu S3 hochladen -# aws s3 sync "$BACKUP_DIR" s3://nexa-backups/$(date +%Y%m%d)/ -EOF - -chmod +x scripts/backup_nexa.sh - -# Tägliches Backup (cron) -echo "0 2 * * * /path/to/nexa/scripts/backup_nexa.sh" | crontab - -``` - -### 6.2 Health Check Monitoring - -```bash -cat > scripts/monitor_nexa.sh << 'EOF' -#!/bin/bash - -# Monitore Service Status -SERVICES=("memos" "n8n" "qdrant" "graphdb") - -for service in "${SERVICES[@]}"; do - if docker-compose ps $service | grep -q "Up"; then - echo "✅ $service: Running" - else - echo "❌ $service: DOWN" - # Restart bei Fehler - docker-compose restart $service - fi -done -EOF - -chmod +x scripts/monitor_nexa.sh - -# Alle 5 Minuten -echo "*/5 * * * * /path/to/nexa/scripts/monitor_nexa.sh" | crontab - -``` - ---- - -## 🧪 Schritt 7: Validation & Testing - -### 7.1 End-to-End Test - -```bash -cat > scripts/test_nexa.sh << 'EOF' -#!/bin/bash - -echo "🧪 Testing Nexa Installation..." - -# Test 1: Memos API -echo "Test 1: Memos API" -if curl -s http://localhost:5230/api/v1/auth/status | jq . > /dev/null 2>&1; then - echo "✅ Memos: OK" -else - echo "❌ Memos: FAILED" -fi - -# Test 2: n8n API -echo "Test 2: n8n API" -if curl -s -u "admin:${N8N_PASSWORD}" http://localhost:5678/api/v1/workflows | jq . > /dev/null 2>&1; then - echo "✅ n8n: OK" -else - echo "❌ n8n: FAILED" -fi - -# Test 3: Qdrant Collection -echo "Test 3: Qdrant Collection" -if curl -s -H "api-key: ${QDRANT_API_KEY}" http://localhost:6333/collections/nexa_knowledge | jq . > /dev/null 2>&1; then - echo "✅ Qdrant: OK" -else - echo "❌ Qdrant: FAILED" -fi - -# Test 4: GraphDB Repository -echo "Test 4: GraphDB Repository" -if curl -s -u "admin:${GRAPHDB_PASSWORD}" http://localhost:7200/repositories | jq . > /dev/null 2>&1; then - echo "✅ GraphDB: OK" -else - echo "❌ GraphDB: FAILED" -fi - -echo "" -echo "🎯 Create Test Memo:" -MEMO=$(curl -s -X POST http://localhost:5230/api/v1/memos \ - -H "Authorization: Bearer ${MEMOS_API_KEY}" \ - -H "Content-Type: application/json" \ - -d '{"content": "Test memo - #nexa:config"}' | jq -r '.id') - -echo "✅ Test Memo created: $MEMO" -echo "Check Memos UI to verify webhook triggers Discovery" -EOF - -chmod +x scripts/test_nexa.sh -./scripts/test_nexa.sh -``` - ---- - -## 🆘 Troubleshooting - -### Problem: Memos kann sich nicht mit n8n verbinden - -```bash -# Überprüfe n8n Webhook URL in Memos config -curl http://localhost:5678/webhook/memos - -# Falls 404: Workflow ist nicht aktiviert/deployed -docker-compose logs n8n | grep webhook -``` - -### Problem: GraphDB startet nicht - -```bash -# Memory-Limit erhöhen -docker-compose exec graphdb -e "GDB_JAVA_OPTS=-Xmx16g -Xms8g" restart - -# Logs checken -docker-compose logs graphdb -``` - -### Problem: Qdrant Collection leer - -```bash -# Überprüfe ob Memos Bridge Workflow läuft -docker-compose logs n8n | grep "memos_bridge" - -# Manuell test -curl -X POST http://localhost:5230/api/v1/memos \ - -H "Authorization: Bearer ${MEMOS_API_KEY}" \ - -d '{"content": "- [ ] Test task #nexa"}' -``` - ---- - -## 📈 Next Steps - -1. ✅ **Phase 1 deployt**: Memos Bridge & Discovery laufen -2. ⏳ **Phase 2 aktivieren**: Email Butler (wenn Nextcloud Mail konfiguriert) -3. ⏳ **Phase 3 setup**: Qdrant Embedding Pipeline -4. ⏳ **Phase 4 integration**: GraphRAG Queries testen -5. ⏳ **Phase 5**: Home Assistant Voice Integration - ---- - -## 📞 Support & Debugging - -```bash -# Alle Logs in eine Datei sammeln -docker-compose logs > deployment_logs.txt - -# Spezifischer Service -docker-compose logs n8n --tail 100 - -# Wiederherstellung aus Backup -./scripts/restore_nexa.sh backup/2024-01-15_120000/ -``` - ---- - -**Deployment abgeschlossen! Nexa ist nun betriebsbereit.** 🚀 diff --git a/nexa-core/PLAN.md b/nexa-core/PLAN.md deleted file mode 100644 index 06a14a6..0000000 --- a/nexa-core/PLAN.md +++ /dev/null @@ -1,27 +0,0 @@ -# NEXA: Implementierungs-Fahrplan 🚀 - -Dieser Plan beschreibt die schrittweise Aktivierung des Nervensystems. - -## Phase 1: Die Wirbelsäule (Basis-Konnektivität) -- [ ] **1.1 Memos-n8n Bridge:** Webhook-Anbindung und Filter-Logik. -- [ ] **1.2 SAIA-Integration:** LiteLLM Proxy in n8n einbinden (Chat-Funktion via Kommentare). -- [ ] **1.3 Git-Sync:** Automatisches Backup der n8n-Workflows in dieses Repo. - -## Phase 2: Die Sinnesorgane (Input-Kanäle) -- [ ] **2.1 Email-Butler:** Automatisierte Filterung und Zusammenfassung der ~20 täglichen Mails. -- [ ] **2.2 RSS Morning Digest:** Täglich kuratierte News-Zusammenfassung in Memos. -- [ ] **2.3 Bluesky-Antenne:** Speichern von gelikten Posts als Wissens-Schnipsel. - -## Phase 3: Das Gedächtnis (Qdrant & RAG) -- [ ] **3.1 Embedding-Pipeline:** Automatische Vektorisierung von Memos & Obsidian. -- [ ] **3.2 Context-Retrieval:** Nexa nutzt Qdrant-Wissen für Antworten (#ask). -- [ ] **3.3 S3-Langzeitarchiv:** Backup der Qdrant-Snapshots auf den S3-Server. - -## Phase 4: Die Motorik (Organisation & Handeln) -- [ ] **4.1 Nextcloud Task-Sync:** Bidirektionaler Status-Abgleich (Memos <-> NC Tasks). -- [ ] **4.2 Timeboxing-Agent:** Intelligente Kalender-Slot-Vorschläge basierend auf Tasks. -- [ ] **4.3 Karakeep-Kuration:** Automatischer Review-Prozess für gespeicherte Links. - -## Phase 5: Alltags-Integration (Feinschliff) -- [ ] **5.1 Home Assistant Voice:** Nexa als Wake-Word Integration. -- [ ] **5.2 System-Monitoring:** Alarme von Proxmox/Backrest direkt in den Memos-System-Feed. diff --git a/nexa-core/README.md b/nexa-core/README.md index ea1e6c0..1a53502 100644 --- a/nexa-core/README.md +++ b/nexa-core/README.md @@ -1,15 +1,17 @@ -# NEXA 🧠 -**The Neural Nexus for Information & Automation** +# nexa-core -Nexa ist das zentrale Nervensystem meines IT-Setups. +Runtime artifacts for Nexa: n8n workflows, configuration, AI prompts, scripts. -## 🏗 Architektur -- **Interface:** Memos -- **Logic:** n8n -- **Memory:** Qdrant & Obsidian -- **Intelligence:** SAIA (via LiteLLM) +📖 Documentation lives one level up in [`../docs/`](../docs/index.md). -## 📁 Struktur -- `n8n-workflows/`: JSON-Exporte der Gehirn-Logik. -- `ai-prompts/`: System-Prompts für Nexa. -- `config/`: Schemata für Qdrant und API-Mappings. +## Layout + +- `n8n-workflows/` — exported workflows (per phase). Imported into the running n8n. +- `ai-prompts/` — system prompts for SAIA / LiteLLM. +- `config/` — runtime config (`agents.yaml`, `infra_manifest.yaml`, `qdrant_schema.json`). +- `scripts/` — automation helpers (workflow backup, etc.). +- `.env.example` — bootstrap secrets only; runtime config lives in Qdrant `_config` namespace. + +## Quickstart + +See [`../docs/09-deployment.md`](../docs/09-deployment.md). diff --git a/nexa-core/doc/scope.md b/nexa-core/doc/scope.md deleted file mode 100644 index 925ad72..0000000 --- a/nexa-core/doc/scope.md +++ /dev/null @@ -1,80 +0,0 @@ -📑 Projekt Nexa: Scoping & Vision -1. Vision & Zielsetzung -Nexa ist als "Zentrales Nervensystem" konzipiert. Das Ziel ist die Schaffung einer intelligenten Middleware zwischen Input-Quellen (Memos, E-Mail, RSS), Wissensspeichern (Obsidian, Qdrant) und Organisations-Tools (Nextcloud Calendar/Tasks). - -Kernziele: -Kognitive Entlastung: Nexa sortiert, filtert und schlägt vor, anstatt nur zu speichern. - -Kontext-Trennung: Saubere, KI-gestützte Unterscheidung zwischen Privat und Arbeit. - -Single Point of Interaction: Memos fungiert als primäre Schnittstelle ("The Voice & Ear"). - -Wissens-Synergie: Verknüpfung von flüchtigen Memos mit tiefem Wissen in Obsidian via semantischer Suche. - -2. Abgrenzung (Scope) -Um die Komplexität beherrschbar zu halten, wird das Projekt wie folgt abgegrenzt: - -In-Scope (Was Nexa tut): -Triaging: Klassifizierung von 20 Mails/Tag (Privat). - -Task-Routing: Verteilung von Aufgaben auf die richtige Nextcloud-Liste (Work vs. Personal). - -Scheduling: Vorschlagen von Timeboxen im Arbeitskalender. - -Memory: Aufbau eines Langzeitgedächtnisses in Qdrant. - -Monitoring: Aggregation kritischer IT-Alarme (Proxmox, Backrest). - -Out-of-Scope (Was Nexa NICHT tut): -Kein direkter Zugriff auf Arbeits-IT (E-Mail/Server). - -Keine aktive Bearbeitung von Dokumenten (außer Metadaten-Extraktion). - -Keine Ersetzung von Spezial-UIs (z.B. Arcane oder Portainer für Container-Management). - -3. Phasenmodell der Entwicklung -Der Aufbau erfolgt iterativ, um frühzeitig Mehrwert zu generieren. - -Phase 1: Die Wirbelsäule (Konnektivität) -Fokus: Datenfluss zwischen Memos, n8n und SAIA. - -Meilenstein: Nexa antwortet auf Memos und kann einfache Fragen beantworten. - -Technik: Webhooks, REST-API, LiteLLM-Proxy. - -Phase 2: Sensorik & Klassifizierung (The Senses) -Fokus: E-Mail-Filterung und das Routing-System. - -Meilenstein: Nexa erkennt den Unterschied zwischen "Arbeits-Task" und "Privat-Task" ohne manuelle Tags. - -Technik: IMAP-Node, n8n-Logic-Router, System-Prompts für Kontext-Analyse. - -Phase 3: Das semantische Gedächtnis (The Brain) -Fokus: Anbindung von Qdrant und Obsidian. - -Meilenstein: RAG (Retrieval Augmented Generation). Nexa beantwortet Fragen basierend auf deinen archivierten Notizen. - -Technik: Embeddings-Pipeline, Qdrant-Vektor-Suche. - -Phase 4: Proaktive Motorik (The Assistant) -Fokus: Kalender-Integration und Timeboxing. - -Meilenstein: Nexa schlägt morgens proaktiv Fokus-Zeiten im Arbeitskalender vor. - -Technik: Nextcloud CalDAV/CardDAV Integration, Optimierungs-Logik für freie Slots. - -Phase 5: Physische Präsenz (Voice & Alarms) -Fokus: Home Assistant Voice & Monitoring-Integration. - -Meilenstein: Nexa spricht über HA-Voice und meldet kritische Systemzustände proaktiv. - -Technik: Home Assistant API, Wyoming Protocol, ntfy/Memos-Push. - -4. Erfolgsmetriken -Woran merken wir, dass Nexa funktioniert? - -Reduktion der Mail-Interaktion: Weniger Zeit in der Mail-App, mehr Fokus auf den Nexa-Digest. - -Kalender-Füllrate: Fokus-Slots für Arbeit sind automatisch geblockt. - -Such-Geschwindigkeit: Informationen aus alten Memos/Obsidian werden via #ask sofort gefunden. \ No newline at end of file