From 9b18e2710d4a586c1c5940ed8a89559b5624f373 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:18:28 +0000 Subject: [PATCH 01/14] Reorganize /docs with numbered TOC; deployment rewrite for actual homelab - /docs/ now contains 12 numbered guides + index.md (navigator). - Old duplicates removed: nexa-core/doc/, DEPLOYMENT.md, PLAN.md, config/{classification_logic,workflows_spec}.md. - 09-deployment.md targets the running infra (LXC 104 docker host, Zoraxy at 192.168.1.4, existing Memos/n8n/LiteLLM/Nextcloud/Qdrant) rather than spinning a parallel stack; minimal-config approach. - 11-open-questions.md tracks user-info-required blockers. - 12-optimization-opportunities.md captures homelab tweaks. - CLAUDE.md added so future agent runs know repo conventions and infra. --- CLAUDE.md | 41 ++ README.md | 18 +- docs/01-vision-and-scope.md | 34 + docs/02-roadmap.md | 48 ++ docs/03-architecture.md | 63 ++ .../04-integration-matrix.md | 2 +- .../05-command-system.md | 2 +- .../06-classification-logic.md | 2 +- .../07-workflow-spec.md | 2 +- .../08-graphrag-architecture.md | 6 +- docs/09-deployment.md | 193 ++++++ docs/10-operations.md | 95 +++ docs/11-open-questions.md | 29 + docs/12-optimization-opportunities.md | 25 + docs/index.md | 50 ++ nexa-core/DEPLOYMENT.md | 629 ------------------ nexa-core/PLAN.md | 27 - nexa-core/README.md | 26 +- nexa-core/doc/scope.md | 80 --- 19 files changed, 618 insertions(+), 754 deletions(-) create mode 100644 CLAUDE.md create mode 100644 docs/01-vision-and-scope.md create mode 100644 docs/02-roadmap.md create mode 100644 docs/03-architecture.md rename nexa-core/doc/integration_matrix.md => docs/04-integration-matrix.md (98%) rename nexa-core/doc/command_system.md => docs/05-command-system.md (98%) rename nexa-core/config/classification_logic.md => docs/06-classification-logic.md (95%) rename nexa-core/config/workflows_spec.md => docs/07-workflow-spec.md (93%) rename nexa-core/doc/graphrag_architecture.md => docs/08-graphrag-architecture.md (95%) create mode 100644 docs/09-deployment.md create mode 100644 docs/10-operations.md create mode 100644 docs/11-open-questions.md create mode 100644 docs/12-optimization-opportunities.md create mode 100644 docs/index.md delete mode 100644 nexa-core/DEPLOYMENT.md delete mode 100644 nexa-core/PLAN.md delete mode 100644 nexa-core/doc/scope.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..95fbf1f --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,41 @@ +# Instructions for Claude (and other agents) + +This file tells future automated runs what they need to know about this repo. + +## Repo conventions + +- **Documentation:** all docs live in `/docs/` and are numbered. Entry point is `docs/index.md`. When you add a doc, give it the next free `NN-` prefix and add a row to the index TOC. +- **Runtime artifacts:** live in `nexa-core/` (workflows, prompts, configs, scripts). Don't put `.md` documentation in there — link from `/docs/` instead. +- **Source-of-truth:** if a doc duplicates content from `nexa-core/config/*.md`, **delete the duplicate**. Single source of truth. + +## Real infrastructure (verified from screenshots, May 2026) + +- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9). + - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. + - LXC 103 backrest — backup orchestration. + - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). + - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. + - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). + - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. + - VM 100 haos — Home Assistant. +- **Already-running services on docker host** (don't redeploy): + - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). + +## When working on Nexa + +1. **Read `docs/index.md` first** — it's the navigator. +2. **Open questions first.** Before writing code or workflow JSON, scan `docs/11-open-questions.md`. If your task touches an unanswered Q, **stop and ask** rather than picking a default. Append new blockers to that doc as `[ ] Q-NN`. +3. **Optimization findings.** When you spot infrastructure improvements, add them to `docs/12-optimization-opportunities.md` as a numbered bullet — don't just mention them in commit messages. +4. **Never inline secrets** in workflow JSON or `.env` committed to git. Use n8n credentials, LiteLLM virtual keys, or (longer term) Vaultwarden. +5. **Keep deployment minimal.** The default answer to *"do we need a new container?"* is **no** — the existing stack covers most needs. + +## Branch policy + +- This branch is `claude/organize-docs-deployment-7N4v2`. Push only here unless told otherwise. +- New work for an unrelated feature → new branch under `claude/`. + +## Quick links + +- [docs/index.md](./docs/index.md) +- [docs/09-deployment.md](./docs/09-deployment.md) — most-touched file during bring-up +- [docs/11-open-questions.md](./docs/11-open-questions.md) — read **before** assuming defaults diff --git a/README.md b/README.md index f77a05e..fdb68b0 100644 --- a/README.md +++ b/README.md @@ -1 +1,17 @@ -# nexa \ No newline at end of file +# nexa + +**Neural Nexus for Information & Automation** — central nervous system for a personal IT setup. + +> Memos = voice & ear · n8n = reflexes · SAIA (LiteLLM) = brain · Qdrant (+ optional graph DB) = memory · Nextcloud = hands. + +## Start here + +📖 **[docs/index.md](./docs/index.md)** — TOC, reading paths, repo layout. + +## Repository layout + +``` +nexa/ +├── docs/ ← all documentation, numbered for reading order +└── nexa-core/ ← runtime: n8n workflows, configs, prompts, scripts +``` diff --git a/docs/01-vision-and-scope.md b/docs/01-vision-and-scope.md new file mode 100644 index 0000000..d6d9366 --- /dev/null +++ b/docs/01-vision-and-scope.md @@ -0,0 +1,34 @@ +# 01 — Vision & Scope + +## Vision + +Nexa is the **central nervous system** of a personal IT setup: an intelligent middleware sitting between input sources (Memos, e-mail, RSS, Karakeep, Bluesky), knowledge stores (Obsidian, Qdrant, Nextcloud) and organization tools (Nextcloud Calendar/Tasks). + +### Core goals + +- **Cognitive offload** — Nexa sorts, filters and proposes; it doesn't just store. +- **Context separation** — clean AI-driven split between *Work* and *Personal*. +- **Single point of interaction** — Memos is the primary surface ("voice & ear"). +- **Knowledge synergy** — link ephemeral memos to deep notes via semantic search. + +## Scope + +### In-scope + +- Triaging ~20 personal e-mails per day. +- Routing tasks to the right Nextcloud list (Work vs. Personal). +- Suggesting time-boxes in the work calendar. +- Long-term memory in Qdrant (and optionally a graph DB). +- Aggregating critical IT alerts (Proxmox, Backrest) into a Memos system feed. + +### Out-of-scope + +- Direct access to work IT (work mail, work servers). +- Active document editing (metadata extraction only). +- Replacing specialist UIs (Arcane / Dozzle / Portainer for containers). + +## Success metrics + +- **Reduced mail interaction** — less time in the mail app, more on the Nexa digest. +- **Calendar fill-rate** — focus slots auto-blocked on the work calendar. +- **Search latency** — old memos / Obsidian notes resurface instantly via `#nexa:ask`. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md new file mode 100644 index 0000000..fc59f97 --- /dev/null +++ b/docs/02-roadmap.md @@ -0,0 +1,48 @@ +# 02 — Roadmap & Phases + +Iterative build-out, value-first. Each phase is shippable on its own. + +## Phase 1 — The Spine (connectivity) + +**Focus:** datapath between Memos, n8n and SAIA. +**Milestone:** Nexa replies on Memos and answers simple questions. + +- [ ] **1.1 Memos ↔ n8n bridge** — webhook ingress, content filter, comment-back. +- [ ] **1.2 SAIA integration** — call LiteLLM proxy from n8n; chat via Memos comments. +- [ ] **1.3 Git-sync** — periodic export of n8n workflows into this repo. + +## Phase 2 — Senses (input channels) + +**Focus:** e-mail filter and the Work-vs-Personal router. +**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags. + +- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails. +- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. +- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets. + +## Phase 3 — Memory (Qdrant & RAG) + +**Focus:** Qdrant + (optional) graph DB for retrieval-augmented answers. +**Milestone:** RAG works — Nexa answers from archived notes. + +- [ ] **3.1 Embedding pipeline** — automatic vectorization of Memos & Obsidian. +- [ ] **3.2 Context retrieval** — `#nexa:ask` reads Qdrant before answering. +- [ ] **3.3 S3 long-term archive** — Qdrant snapshots to the S3 server. +- [ ] **3.4 Graph layer (deferred decision — see [11-open-questions](./11-open-questions.md))** — Neo4j *or* Ontotext GraphDB for structural queries. + +## Phase 4 — Motor (organization & action) + +**Focus:** calendar integration and time-boxing. +**Milestone:** Nexa proactively proposes morning focus slots in the work calendar. + +- [ ] **4.1 Nextcloud Tasks sync** — bidirectional status sync (Memos ↔ NC Tasks). +- [ ] **4.2 Time-boxing agent** — calendar-aware slot suggestions. +- [ ] **4.3 Karakeep curation** — automated review cycle for saved links. + +## Phase 5 — Daily integration (polish) + +**Focus:** Home Assistant Voice and monitoring. +**Milestone:** Nexa speaks via HA-Voice and surfaces critical system states proactively. + +- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol. +- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy. diff --git a/docs/03-architecture.md b/docs/03-architecture.md new file mode 100644 index 0000000..544df46 --- /dev/null +++ b/docs/03-architecture.md @@ -0,0 +1,63 @@ +# 03 — Architecture Overview + +A one-page mental model. For details follow the cross-links. + +## Topology + +``` + ┌────────────────┐ + voice / typing ──────▶│ Memos │◀──── Nexa replies as comments + │ (interface) │ + └───────┬────────┘ + │ webhook (- [ ] / #nexa:*) + ▼ + ┌────────────────┐ + IMAP / RSS / NC ────▶│ n8n │◀──── workflows live in + Karakeep / Bluesky │ (logic) │ ./nexa-core/n8n-workflows + └───┬────────┬───┘ + classify ▲ │ │ ▲ retrieve + │ ▼ ▼ │ + ┌──────────┐ ┌──────────┐ + │ SAIA │ │ Qdrant │ + │ LiteLLM │ │ vector │ + └──────────┘ └────┬─────┘ + │ optional + ▼ + ┌──────────┐ + │ Graph │ (Neo4j or Ontotext — + │ DB │ see open questions) + └──────────┘ + │ + ▼ writes + ┌──────────────────────────────┐ + │ Nextcloud (Tasks, Calendar, │ + │ Mail, Files / Obsidian) │ + └──────────────────────────────┘ +``` + +## Components + +| Component | Role | Where it runs (today) | +|-----------|------|-----------------------| +| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | +| **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | +| **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | +| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 (existing `qdrant_scientific` may be reused — see [11](./11-open-questions.md)) | +| **Graph DB** | Structural memory (deferred) | not yet deployed | +| **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` | +| **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` | +| **Backrest** | Backup orchestration | LXC 103 | +| **Zoraxy** | Reverse proxy + TLS | LXC 108 (`192.168.1.4:8000`) | +| **AdGuard DNS** | Internal name resolution | LXC 102 | +| **Home Assistant** | Voice + house automation | VM 100 (HAOS) | + +## Two-pillar memory + +- **Qdrant** answers *"what is similar / relevant?"* (cosine search over embeddings). +- **Graph DB** answers *"who, what depends on what, how is it structured?"* (Cypher / SPARQL). + +Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. See [08 — GraphRAG architecture](./08-graphrag-architecture.md). + +## Dual-context routing + +Every input is classified `work` or `personal` before any side effect (task creation, calendar write). See [04 — Integration matrix](./04-integration-matrix.md) and [06 — Classification logic](./06-classification-logic.md). diff --git a/nexa-core/doc/integration_matrix.md b/docs/04-integration-matrix.md similarity index 98% rename from nexa-core/doc/integration_matrix.md rename to docs/04-integration-matrix.md index dfc7da7..2fab535 100644 --- a/nexa-core/doc/integration_matrix.md +++ b/docs/04-integration-matrix.md @@ -1,4 +1,4 @@ -# 🔗 Nexa Integrations-Matrix (V2) +# 04 — Integrations-Matrix Diese Matrix definiert die logische Trennung zwischen privaten und beruflichen Datenströmen sowie die Anbindung der Infrastruktur. diff --git a/nexa-core/doc/command_system.md b/docs/05-command-system.md similarity index 98% rename from nexa-core/doc/command_system.md rename to docs/05-command-system.md index d497fc9..4d5143d 100644 --- a/nexa-core/doc/command_system.md +++ b/docs/05-command-system.md @@ -1,4 +1,4 @@ -# Nexa Command System - #nexa:command +# 05 — Command System (`#nexa:*`) Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag: diff --git a/nexa-core/config/classification_logic.md b/docs/06-classification-logic.md similarity index 95% rename from nexa-core/config/classification_logic.md rename to docs/06-classification-logic.md index 3e8321c..4a5890f 100644 --- a/nexa-core/config/classification_logic.md +++ b/docs/06-classification-logic.md @@ -1,4 +1,4 @@ -# Nexa Klassifizierungs-Logik +# 06 — Klassifizierungs-Logik Dieser Fragebogen dient der Feinabstimmung von SAIA, um Tasks korrekt zu routen. diff --git a/nexa-core/config/workflows_spec.md b/docs/07-workflow-spec.md similarity index 93% rename from nexa-core/config/workflows_spec.md rename to docs/07-workflow-spec.md index 17b1b26..2960f43 100644 --- a/nexa-core/config/workflows_spec.md +++ b/docs/07-workflow-spec.md @@ -1,4 +1,4 @@ -# Spezifikation: Phase 2 - Task-Router +# 07 — Workflow Spec: Phase-2 Task-Router ## 1. Trigger * **Typ:** Webhook (Memos) diff --git a/nexa-core/doc/graphrag_architecture.md b/docs/08-graphrag-architecture.md similarity index 95% rename from nexa-core/doc/graphrag_architecture.md rename to docs/08-graphrag-architecture.md index 40e4184..430b9fa 100644 --- a/nexa-core/doc/graphrag_architecture.md +++ b/docs/08-graphrag-architecture.md @@ -1,4 +1,8 @@ -# GraphRAG für Nexa - Strukturelles Wissen & Beziehungen +# 08 — GraphRAG: Structural Knowledge & Relations + +> ⚠️ **Open decision (see [11-open-questions](./11-open-questions.md))**: this document describes the design with **Neo4j**. The earlier deployment draft referenced **Ontotext GraphDB** (SPARQL). Pick one before Phase 3.4. + +## GraphRAG für Nexa — Strukturelles Wissen & Beziehungen ## 🧠 Zwei-Säulen-Memory-Architektur diff --git a/docs/09-deployment.md b/docs/09-deployment.md new file mode 100644 index 0000000..7358e64 --- /dev/null +++ b/docs/09-deployment.md @@ -0,0 +1,193 @@ +# 09 — Deployment + +Pragmatic deployment guide that **assumes the existing homelab** and adds only what's missing. + +## What's already running (no action required) + +Surveyed from Homepage / Dozzle / Proxmox / Zoraxy: + +| Service | Host / port | URL | +|---------|-------------|-----| +| Memos | docker LXC 104 → `:5230` | `https://memos.nuclide.systems` | +| n8n | docker LXC 104 → `:5678` | `https://n8n.nuclide.systems` | +| LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) | +| Nextcloud | LXC 105 | `https://nc.nuclide.systems` | +| ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` | +| Karakeep (Hoarder) | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` | +| Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` | +| Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` | +| Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` | +| Backrest | LXC 103 | (internal) | +| AdGuard DNS | LXC 102 | (internal) | +| Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems | +| `qdrant_scientific` (existing) | docker LXC 104 | reuse — see [11](./11-open-questions.md) | + +The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**. + +## What's missing for Nexa + +1. **Qdrant collection** for Nexa (`nexa_knowledge`) — either on the existing `qdrant_scientific` or a dedicated container. +2. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n. +3. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`). +4. **Memos webhook → n8n** wired through the Memos config. +5. **LiteLLM virtual key** for the `nexa` user with embedding + chat models. +6. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied. +7. **(Phase 3.4)** decision on graph DB (Neo4j vs. Ontotext GraphDB) — see [11](./11-open-questions.md). + +--- + +## Step 1 — Secrets + +Copy `nexa-core/.env.example` → `nexa-core/.env` and fill **only** the secrets: + +```bash +cd nexa-core +cp .env.example .env +$EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY +``` + +The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead. + +## Step 2 — Qdrant collection + +```bash +# adjust QDRANT_HOST in .env first +source nexa-core/.env + +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \ + -H "Content-Type: application/json" \ + -H "api-key: $QDRANT_API_KEY" \ + -d @nexa-core/config/qdrant_schema.json +``` + +> If reusing `qdrant_scientific`, this just adds another collection; no new container needed. + +## Step 3 — LiteLLM virtual key + +In the LiteLLM admin UI (`ai.nuclide.systems`): + +1. Create user `nexa`. +2. Issue a virtual key with access to one chat model (e.g. `gpt-4o-mini` / a local Sonnet equivalent) and one embedding model (`text-embedding-3-small`). +3. Paste the key into `SAIA_API_KEY` in `.env`. + +## Step 4 — n8n workflows + +Import the JSON exports — credentials are filled inside n8n, not in the JSON: + +```bash +# n8n personal access token from the n8n UI: Settings → API +N8N_URL=https://n8n.nuclide.systems +N8N_TOKEN=... # from the n8n UI + +for f in nexa-core/n8n-workflows/phase-1/*.json \ + nexa-core/n8n-workflows/phase-2/*.json; do + curl -X POST "$N8N_URL/api/v1/workflows" \ + -H "X-N8N-API-KEY: $N8N_TOKEN" \ + -H "Content-Type: application/json" \ + --data-binary "@$f" +done +``` + +Inside n8n, attach credentials to the imported nodes: + +- **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY` +- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` +- **Nextcloud** → app password +- **Qdrant** → header `api-key: $QDRANT_API_KEY` + +Activate each workflow individually after smoke-test. + +## Step 5 — Memos webhook + +In the Memos admin UI, set the webhook URL to the production address of the discovery workflow: + +``` +https://n8n.nuclide.systems/webhook/memos +``` + +The same URL is the one the workflow exposes; verify with: + +```bash +curl -i https://n8n.nuclide.systems/webhook/memos +# expect 200 / 405, never 404 +``` + +## Step 6 — Bootstrap commands via Memos + +Create a memo with body `#nexa:config` — the discovery workflow: + +1. Lists Nextcloud Tasks lists, picks Work / Personal / Shopping / Wishes by name. +2. Counts existing Qdrant points in `nexa_knowledge`. +3. Verifies LiteLLM reachability + lists available models. +4. Replies as a comment with a **runtime-config snapshot** that's stored as Qdrant metadata (`_config` namespace) and as `nexa-core/config/runtime_config.json` (gitignored). + +After this point, `.env` is read-once. Subsequent runs read config from Qdrant. + +## Step 7 — Smoke tests + +```bash +# (1) Memos round-trip — should produce a comment within ~5 s +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"content":"- [ ] testing the router #nexa"}' + +# (2) Classification dry-run +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"#nexa:route-test buy milk"}' + +# (3) RAG test (requires at least one indexed memo/note) +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"#nexa:ask what is the goal of nexa?"}' +``` + +## Step 8 — Reverse proxy + +Already done — Zoraxy at `192.168.1.4:8000` terminates TLS for `*.nuclide.systems` and forwards to docker LXC 104 (`192.168.1.40`). **No new entry is required for Nexa**: every service Nexa talks to already has a host entry. + +## Step 9 — Backups + +Already covered by Backrest (LXC 103). Add: + +- **n8n workflows** → `nexa-core/scripts/backup_workflows.sh` (already present) into a Backrest schedule. +- **Qdrant snapshots** → schedule a daily `POST /collections/nexa_knowledge/snapshots` and rsync to S3 (`s3.nuclide.systems`). Add as a Backrest pre-hook on the docker host. + +For deeper detail: [10 — Operations](./10-operations.md). + +--- + +## Phase add-on: graph DB (Phase 3.4) + +Defer until 3.1–3.3 ship. When needed, two minimal options — pick one: + +### Option A — Neo4j (lighter, popular for graph-RAG) + +```yaml +# nexa-core/docker-compose.graph.yml +services: + neo4j: + image: neo4j:5-community + container_name: nexa-neo4j + ports: ["7687:7687", "7474:7474"] + environment: + NEO4J_AUTH: "neo4j/${NEO4J_PASSWORD}" + NEO4J_server_memory_heap_max__size: 2G + volumes: ["./data/neo4j:/data"] + restart: unless-stopped +``` + +### Option B — Ontotext GraphDB (SPARQL, RDF native) + +Keep the snippet from the original `DEPLOYMENT.md` Schritt 1.3 for `graphdb`. Heavier (~4 GB heap), only justified if you want SHACL / OWL reasoning. + +Add a Zoraxy host entry `graph.nuclide.systems` → `192.168.1.40:7474` (Neo4j) or `:7200` (GraphDB) **only if** browser access is desired; otherwise n8n talks to it on the docker network. + +--- + +## Step-back / rollback + +- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops. +- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`. +- **Re-discover** — `#nexa:reset-config` then `#nexa:config`. diff --git a/docs/10-operations.md b/docs/10-operations.md new file mode 100644 index 0000000..4c69609 --- /dev/null +++ b/docs/10-operations.md @@ -0,0 +1,95 @@ +# 10 — Operations + +Day-2 concerns. Backups, monitoring, troubleshooting. + +## Backups + +Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items: + +| What | How | Frequency | +|------|-----|-----------| +| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container | +| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) | +| Memos DB | Backrest snapshot of Memos data dir | already covered | +| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered | +| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | + +## Monitoring + +| Signal | Source | Sink | +|--------|--------|------| +| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks | Memos system feed via ntfy | +| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic | +| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest | +| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest | +| Backrest run status | Backrest webhook | ntfy | + +A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo. + +## Troubleshooting + +### Memos webhook not firing + +```bash +# is the Memos webhook config still pointing at n8n? +curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \ + https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks' + +# does n8n still expose the path? +curl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404 +``` + +If 404 → workflow is inactive in n8n. Activate. + +### LiteLLM 401 / 429 + +- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential. +- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk. + +### Qdrant `nexa_knowledge` empty + +```bash +# is the indexing workflow active? +docker logs nexa-n8n 2>&1 | grep memos_bridge | tail + +# manual index test +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -H "Authorization: Bearer $MEMOS_API_KEY" \ + -d '{"content":"manual probe #nexa"}' + +# point count +curl -s -H "api-key: $QDRANT_API_KEY" \ + $QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count' +``` + +### Wrong list routing (Work vs Personal) + +1. Run `#nexa:route-test ` and check the confidence value. +2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)). +3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change. + +### "Where did Nexa store this?" + +```bash +# state dump +curl -X POST https://memos.nuclide.systems/api/v1/memos \ + -d '{"content":"#nexa:export-state"}' +# returns runtime_config + counters as a JSON memo +``` + +## Log locations + +| Component | Where | +|-----------|-------| +| Memos | `docker logs nexa-memos` (LXC 104) | +| n8n | `docker logs nexa-n8n` | +| LiteLLM | `docker logs litellm` | +| Qdrant | `docker logs qdrant_scientific` | +| Zoraxy | LXC 108 web UI → Statistical Analysis | +| Backrest | LXC 103 web UI | + +For a one-shot dump: + +```bash +ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log +``` diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md new file mode 100644 index 0000000..ff5e7c8 --- /dev/null +++ b/docs/11-open-questions.md @@ -0,0 +1,29 @@ +# 11 — Open Questions (user-info-required) + +Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide. + +## Architectural decisions + +- [ ] **Q1 — Graph DB choice.** Phase 3.4 says *Neo4j* in [08](./08-graphrag-architecture.md) but the original deployment draft used *Ontotext GraphDB*. **Recommendation:** Neo4j (smaller footprint, Cypher is friendlier for n8n HTTP nodes, no SHACL/OWL needs identified). Confirm or override. +- [ ] **Q2 — Reuse `qdrant_scientific` or run a dedicated `nexa-qdrant`?** Reuse keeps memory usage flat (~670 MB seen on Homepage); dedicated gives clean isolation and independent snapshot lifecycle. **Recommendation:** reuse with a `nexa_*` collection prefix. +- [ ] **Q3 — Embeddings model.** Default in `infra_manifest.yaml` is `text-embedding-3-small` (OpenAI). Is the LiteLLM virtual key authorised to call it, or should we route to a self-hosted alternative (e.g. `bge-m3` via Ollama on the same host)? +- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. +- [ ] **Q5 — Karakeep vs. Hoarder naming.** The Zoraxy host is `hoarder.nuclide.systems` but the container is `karakeep-*` and Homepage labels it Karakeep. They're the same product (rename in 2024). Pick one display name to use in docs and prompts. + +## Identifiers needed (auto-discoverable, but list now if known) + +- [ ] **Q6 — Nextcloud Tasks list IDs** for: `Work_Tasks`, `Personal_Tasks`, `Shopping`, `Wishes`. Discovery via `#nexa:config` will fill these — confirm names match. +- [ ] **Q7 — Nextcloud Calendar IDs** for: `Work_Calendar`, primary personal calendar. +- [ ] **Q8 — IMAP credentials for the personal mail account.** Can n8n reuse a Nextcloud Mail account (preferred — no extra password) or must we add a dedicated IMAP entry? +- [ ] **Q9 — ntfy topic name** for `nexa.system`. Is the topic public on `ntfy.nuclide.systems` or should it be authenticated? +- [ ] **Q10 — Pocket-ID role.** `id.nuclide.systems` is running. Do we want SSO in front of the n8n / Memos UIs, or skip for now? + +## Hardware / capacity + +- [ ] **Q11 — RAM headroom on docker host.** Homepage shows 29.5 GiB free out of ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing of Obsidian + Memos history is fine here, but a Phase-3.4 GraphDB choice changes this (Neo4j ~2 GB heap, Ontotext ~4 GB). Confirm acceptable. +- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots? + +## Process + +- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but flagging it because Phase 5 monitoring would alert on it. Suppress, or is it intentional? +- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage — Homepage doesn't have a Zoraxy widget yet. Cosmetic, unrelated to Nexa. diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md new file mode 100644 index 0000000..77c5f28 --- /dev/null +++ b/docs/12-optimization-opportunities.md @@ -0,0 +1,25 @@ +# 12 — Optimization Opportunities + +Observations from the running infrastructure. Each item is independent — accept, defer, or reject. + +## For Nexa directly + +1. **Reuse, don't redeploy.** The earlier `DEPLOYMENT.md` would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new [09-deployment](./09-deployment.md) treats these as pre-existing — keeps the config minimal and avoids port collisions. +2. **Use LiteLLM virtual keys per logical caller.** Today there's one SAIA key. Issuing one key per workflow (`nexa-router`, `nexa-embed`, `nexa-digest`) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything. +3. **Use n8n's *credential* objects, never inline secrets.** The current workflows under `nexa-core/n8n-workflows/phase-1/*.json` should be reviewed — if any header `Authorization` is hardcoded, replace with credential references before importing. +4. **Centralise system alerts on a single ntfy topic** (`nexa.system`). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them. +5. **Defer graph DB until Phase 3.4.** Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries. +6. **Auto-export n8n workflows.** `nexa-core/scripts/backup_workflows.sh` already exists. Schedule it inside the n8n container (cron) and let it `git commit && git push` — this is the cheapest disaster recovery. + +## For the wider homelab (out of scope but worth noting) + +7. **Octoprint `EXITED`** — either remove the Homepage entry or fix the container; right now it permanently shows red. +8. **Zoraxy widget missing** on Homepage. Homepage v1.x has no built-in Zoraxy widget, but you can use the generic `customapi` widget against `/api/stats/summary` to surface request counts. +9. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. +10. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). +11. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. +12. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. +13. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. +14. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. +15. **Immich + Qdrant.** Immich is already running and embeds photos with CLIP. If you ever want photo-aware Nexa answers, you can mount Immich's existing search via its API — no second embedding pass. +16. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..37c2bba --- /dev/null +++ b/docs/index.md @@ -0,0 +1,50 @@ +# NEXA Documentation + +**Neural Nexus for Information & Automation** — the central nervous system that ties Memos, n8n, SAIA (LiteLLM), Nextcloud and a vector store into one assistant. + +This is the documentation entry point. Read top-to-bottom for first-time setup, or jump to the section you need. + +--- + +## Table of Contents + +| # | Document | Read when… | +|---|----------|------------| +| 01 | [Vision & Scope](./01-vision-and-scope.md) | You want to understand *what* Nexa is and isn't. | +| 02 | [Roadmap & Phases](./02-roadmap.md) | You want to know the implementation order. | +| 03 | [Architecture Overview](./03-architecture.md) | You need a one-page mental model. | +| 04 | [Integration Matrix](./04-integration-matrix.md) | You're wiring up a new data source or mapping work-vs-personal flows. | +| 05 | [Command System](./05-command-system.md) | You want to know what `#nexa:*` commands do. | +| 06 | [Classification Logic](./06-classification-logic.md) | You're tuning the work/personal router. | +| 07 | [Workflow Spec — Task Router](./07-workflow-spec.md) | You're building the Phase-2 router workflow. | +| 08 | [GraphRAG Architecture](./08-graphrag-architecture.md) | You're working on Phase 3 (Qdrant + Graph). | +| 09 | [Deployment](./09-deployment.md) | You're bringing Nexa up on the real infrastructure. | +| 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. | +| 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. | +| 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. | + +--- + +## Reading paths + +- **First-time installer:** 01 → 03 → 09 → 10 → 11 +- **Workflow author:** 04 → 05 → 07 → 06 +- **Phase-3 work (memory):** 08 → 09 (§ Phase 3 add-on) +- **Stakeholder / quick read:** 01 → 02 → 12 + +--- + +## Repository layout + +``` +nexa/ +├── README.md → points here +├── docs/ → you are here +└── nexa-core/ → the actual project + ├── ai-prompts/ → SAIA system prompts + ├── config/ → runtime config (YAML, JSON schema) + ├── n8n-workflows/ → exported workflows, version-controlled + └── scripts/ → automation helpers +``` + +Source-of-truth for *runnable* config stays in `nexa-core/`. Documentation lives here in `docs/`. diff --git a/nexa-core/DEPLOYMENT.md b/nexa-core/DEPLOYMENT.md deleted file mode 100644 index 6a3b458..0000000 --- a/nexa-core/DEPLOYMENT.md +++ /dev/null @@ -1,629 +0,0 @@ -# NEXA Deployment Guide - -## 🚀 Architektur-Überblick - -``` -┌─────────────────────────────────────────────────┐ -│ MEMOS (Port 5230) │ -│ Interface, Voice-Input, Webhooks │ -└────────────────────┬────────────────────────────┘ - │ - ┌───────────┼───────────┐ - ↓ ↓ ↓ - ┌────────┐ ┌────────┐ ┌──────────┐ - │ n8n │ │ SAIA │ │ Nextcloud│ - │Logic │ │LiteLLM │ │Tasks/Mail│ - │7879 │ │4000 │ │80/443 │ - └────┬───┘ └────┬───┘ └─────┬────┘ - │ │ │ - ┌────┴───────────┼────────────┴────┐ - ↓ ↓ ↓ -┌─────────┐ ┌────────────┐ ┌──────────────┐ -│ QDRANT │ │ GraphDB │ │ Obsidian │ -│Vector │ │ (Ontotext) │ │(local sync) │ -│6333 │ │7200 │ │ │ -└─────────┘ └────────────┘ └──────────────┘ -``` - ---- - -## 📋 Voraussetzungen - -### Hardware -- **CPU**: Min. 4 Cores (empfohlen 8) -- **RAM**: Min. 16 GB (empfohlen 32 GB) -- **Storage**: Min. 100 GB SSD (für Datenbanken & Backups) -- **Network**: Stabile Verbindung, HTTPS-ready - -### Software -- Docker & Docker Compose (Version 20.10+) -- Git -- Bash -- curl/wget -- `.env` Datei aus `.env.example` (mit Secrets gefüllt) - -### Accounts & Credentials -- Nextcloud Instanz mit Admin-Zugang -- OpenAI API-Key (für SAIA/LiteLLM) -- Obsidian Vault Zugang lokalem Filesystem - ---- - -## 🐳 Schritt 1: Docker Compose Setup - -### 1.1 Repository klonen - -```bash -git clone https://github.com/noheton/nexa.git -cd nexa/nexa-core -``` - -### 1.2 .env konfigurieren - -```bash -cp .env.example .env -# WICHTIG: Alle Secrets eintragen! -# - MEMOS_API_KEY -# - SAIA_API_KEY -# - NC_APP_PASSWORD -# - QDRANT_API_KEY -# - GRAPHDB_PASSWORD -nano .env -``` - -### 1.3 docker-compose.yml erstellen - -```bash -cat > docker-compose.yml << 'EOF' -version: '3.8' - -services: - # ===== MEMOS (Interface) ===== - memos: - image: neosmemo/memos:latest - container_name: nexa-memos - ports: - - "5230:5230" - volumes: - - ./data/memos:/root/.memos - environment: - MEMOS_PORT: "5230" - MEMOS_DRIVER: sqlite - MEMOS_DSN: /root/.memos/memos.db - networks: - - nexa - restart: unless-stopped - - # ===== n8n (Workflow Engine) ===== - n8n: - image: n8nio/n8n:latest - container_name: nexa-n8n - ports: - - "5678:5678" - volumes: - - ./data/n8n:/home/node/.n8n - - ./n8n-workflows:/workflows - environment: - N8N_BASIC_AUTH_ACTIVE: "true" - N8N_BASIC_AUTH_USER: "${N8N_USER:-admin}" - N8N_BASIC_AUTH_PASSWORD: "${N8N_PASSWORD:-nexa_secure_pass}" - N8N_HOST: "${N8N_DOMAIN:-localhost}" - N8N_PORT: "5678" - N8N_PROTOCOL: https - N8N_WEBHOOK_URL: "https://${N8N_DOMAIN:-localhost}/webhook" - GENERIC_TIMEZONE: Europe/Berlin - DB_TYPE: sqlite - DB_SQLITE_PATH: /home/node/.n8n/nexa.db - networks: - - nexa - restart: unless-stopped - depends_on: - - memos - - # ===== QDRANT (Vector DB) ===== - qdrant: - image: qdrant/qdrant:latest - container_name: nexa-qdrant - ports: - - "6333:6333" - volumes: - - ./data/qdrant:/qdrant/storage - environment: - QDRANT_API_KEY: "${QDRANT_API_KEY}" - networks: - - nexa - restart: unless-stopped - - # ===== GraphDB (Ontotext SPARQL) ===== - graphdb: - image: ontotext/graphdb:10-ee - container_name: nexa-graphdb - ports: - - "7200:7200" - volumes: - - ./data/graphdb:/opt/graphdb/data - - ./config/graphdb-init.ttl:/graphdb-import/import.ttl - environment: - GDB_JAVA_OPTS: "-Xmx8g -Xms4g" - GDB_PASSWORD: "${GRAPHDB_PASSWORD}" - networks: - - nexa - restart: unless-stopped - - # ===== SAIA / LiteLLM Proxy (optional - wenn lokal) ===== - # (Falls SAIA extern gehostet: nicht nötig) - -networks: - nexa: - driver: bridge - -volumes: - nexa_memos: - nexa_n8n: - nexa_qdrant: - nexa_graphdb: -EOF -``` - ---- - -## 🔧 Schritt 2: Individuelle Service-Konfiguration - -### 2.1 MEMOS Initialisierung - -```bash -# Nach memos startup (ca. 30s) -curl -X POST http://localhost:5230/api/v1/auth/signup \ - -H "Content-Type: application/json" \ - -d '{ - "username": "nexa", - "password": "secure_password" - }' - -# API-Token generieren -TOKEN=$(curl -X POST http://localhost:5230/api/v1/auth/signin \ - -H "Content-Type: application/json" \ - -d '{"username":"nexa","password":"secure_password"}' \ - | jq -r '.data.token') - -echo "MEMOS_API_KEY=$TOKEN" >> .env -``` - -### 2.2 Qdrant Konfiguration - -```bash -# Warte auf Qdrant startup -sleep 10 - -# Erstelle Collection für Nexa -curl -X PUT "http://localhost:6333/collections/nexa_knowledge" \ - -H "Content-Type: application/json" \ - -H "api-key: ${QDRANT_API_KEY}" \ - -d '{ - "vectors": { - "size": 1536, - "distance": "Cosine" - }, - "optimizers_config": { - "deleted_threshold": 0.2, - "vacuum_threshold": 0.5, - "default_segment_number": 5, - "max_segment_number": 30, - "memmap_threshold": 268435456, - "indexing_threshold": 20000, - "flush_interval_sec": 10, - "max_optimization_threads": 4 - } - }' - -echo "✅ Qdrant Collection erstellt" -``` - -### 2.3 GraphDB Initialisierung - -```bash -# Warte auf GraphDB startup -sleep 15 - -# Erstelle Repository -GRAPHDB_URL="http://localhost:7200" - -curl -X POST "$GRAPHDB_URL/rest/repositories" \ - -H "Content-Type: application/json" \ - -u "admin:${GRAPHDB_PASSWORD}" \ - -d '{ - "id": "nexa_knowledge", - "title": "Nexa Knowledge Graph", - "type": "free", - "params": { - "baseURL": { - "label": "Base URL", - "value": "http://nexa.local/" - }, - "defaultLanguage": { - "label": "Default language", - "value": "en" - } - } - }' - -echo "✅ GraphDB Repository erstellt" -``` - -### 2.4 n8n Workflows Import - -```bash -# Warte auf n8n startup (ca. 60s) -sleep 60 - -# Importiere Discovery Workflow -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-1/1_1_discovery_workflow.json - -# Importiere Memos Bridge -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-1/1_2_memos_bridge_v2.json - -# Importiere Email Butler -curl -X POST http://localhost:5678/api/v1/workflows \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d @n8n-workflows/phase-2/2_1_email_butler.json - -echo "✅ n8n Workflows importiert" -``` - ---- - -## 🚀 Schritt 3: Deployment starten - -### 3.1 Docker Compose Up - -```bash -docker-compose up -d - -# Logs überwachen -docker-compose logs -f -``` - -### 3.2 Health Checks - -```bash -# Health Check Script -check_service() { - local name=$1 - local url=$2 - echo "Checking $name..." - if curl -s "$url" > /dev/null; then - echo "✅ $name: OK" - else - echo "❌ $name: FAILED" - fi -} - -sleep 30 - -check_service "Memos" "http://localhost:5230" -check_service "n8n" "http://localhost:5678" -check_service "Qdrant" "http://localhost:6333/health" -check_service "GraphDB" "http://localhost:7200/health" - -echo "" -echo "🎯 Service URLs:" -echo " - Memos: http://localhost:5230" -echo " - n8n: http://localhost:5678" -echo " - Qdrant: http://localhost:6333" -echo " - GraphDB: http://localhost:7200" -``` - ---- - -## ⚙️ Schritt 4: Konfiguration via Command System - -### 4.1 Discovery-Workflow triggern - -```bash -# In Memos Memo erstellen: -# #nexa:config - -# Oder via API: -curl -X POST http://localhost:5678/api/v1/workflows/run \ - -H "Content-Type: application/json" \ - -u "admin:${N8N_PASSWORD}" \ - -d '{ - "workflowId": "discovery-workflow-id" - }' -``` - -### 4.2 Automatische Konfiguration laden - -Nach erfolgreichem Discovery werden Nextcloud-Listen-IDs: - -```bash -# Es wird automatisch aktualisiert in: -# - Qdrant: als Metadata -# - n8n: als Umgebungsvariablen -# - Memos: als Confirmation-Memo -``` - ---- - -## 🔐 Schritt 5: Reverse Proxy Setup (Production) - -### 5.1 Nginx Configuration - -```bash -cat > /etc/nginx/sites-available/nexa.conf << 'EOF' -upstream memos_backend { - server localhost:5230; -} - -upstream n8n_backend { - server localhost:5678; -} - -upstream qdrant_backend { - server localhost:6333; -} - -upstream graphdb_backend { - server localhost:7200; -} - -server { - listen 443 ssl http2; - server_name nexa.yourdomain.com; - - ssl_certificate /etc/letsencrypt/live/nexa.yourdomain.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/nexa.yourdomain.com/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - ssl_ciphers HIGH:!aNULL:!MD5; - - # MEMOS - location / { - proxy_pass http://memos_backend; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_read_timeout 3600s; - proxy_send_timeout 3600s; - } - - # n8n - location /n8n/ { - proxy_pass http://n8n_backend/; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - } - - # Qdrant (nur intern, wenn nötig) - location /qdrant/ { - allow 10.0.0.0/8; - deny all; - proxy_pass http://qdrant_backend/; - } - - # GraphDB (nur intern) - location /graphdb/ { - allow 10.0.0.0/8; - deny all; - proxy_pass http://graphdb_backend/; - } - - # Webhook endpoint für externe Services - location /webhook/ { - proxy_pass http://n8n_backend/webhook/; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - } -} - -server { - listen 80; - server_name nexa.yourdomain.com; - return 301 https://$server_name$request_uri; -} -EOF - -# Aktivieren -sudo ln -s /etc/nginx/sites-available/nexa.conf /etc/nginx/sites-enabled/ -sudo nginx -t -sudo systemctl reload nginx -``` - ---- - -## 📊 Schritt 6: Monitoring & Backups - -### 6.1 Backup Script - -```bash -cat > scripts/backup_nexa.sh << 'EOF' -#!/bin/bash - -BACKUP_DIR="./backups/$(date +%Y%m%d_%H%M%S)" -mkdir -p "$BACKUP_DIR" - -echo "🔄 Backing up Nexa..." - -# Backup Datenbanken -docker-compose exec -T qdrant tar czf - /qdrant/storage > "$BACKUP_DIR/qdrant.tar.gz" -docker-compose exec -T graphdb tar czf - /opt/graphdb/data > "$BACKUP_DIR/graphdb.tar.gz" -docker-compose exec -T memos tar czf - /root/.memos > "$BACKUP_DIR/memos.tar.gz" -docker-compose exec -T n8n tar czf - /home/node/.n8n > "$BACKUP_DIR/n8n.tar.gz" - -# Backup Configs -cp .env "$BACKUP_DIR/.env.backup" -cp docker-compose.yml "$BACKUP_DIR/docker-compose.yml.backup" - -echo "✅ Backup completed: $BACKUP_DIR" - -# Optional: zu S3 hochladen -# aws s3 sync "$BACKUP_DIR" s3://nexa-backups/$(date +%Y%m%d)/ -EOF - -chmod +x scripts/backup_nexa.sh - -# Tägliches Backup (cron) -echo "0 2 * * * /path/to/nexa/scripts/backup_nexa.sh" | crontab - -``` - -### 6.2 Health Check Monitoring - -```bash -cat > scripts/monitor_nexa.sh << 'EOF' -#!/bin/bash - -# Monitore Service Status -SERVICES=("memos" "n8n" "qdrant" "graphdb") - -for service in "${SERVICES[@]}"; do - if docker-compose ps $service | grep -q "Up"; then - echo "✅ $service: Running" - else - echo "❌ $service: DOWN" - # Restart bei Fehler - docker-compose restart $service - fi -done -EOF - -chmod +x scripts/monitor_nexa.sh - -# Alle 5 Minuten -echo "*/5 * * * * /path/to/nexa/scripts/monitor_nexa.sh" | crontab - -``` - ---- - -## 🧪 Schritt 7: Validation & Testing - -### 7.1 End-to-End Test - -```bash -cat > scripts/test_nexa.sh << 'EOF' -#!/bin/bash - -echo "🧪 Testing Nexa Installation..." - -# Test 1: Memos API -echo "Test 1: Memos API" -if curl -s http://localhost:5230/api/v1/auth/status | jq . > /dev/null 2>&1; then - echo "✅ Memos: OK" -else - echo "❌ Memos: FAILED" -fi - -# Test 2: n8n API -echo "Test 2: n8n API" -if curl -s -u "admin:${N8N_PASSWORD}" http://localhost:5678/api/v1/workflows | jq . > /dev/null 2>&1; then - echo "✅ n8n: OK" -else - echo "❌ n8n: FAILED" -fi - -# Test 3: Qdrant Collection -echo "Test 3: Qdrant Collection" -if curl -s -H "api-key: ${QDRANT_API_KEY}" http://localhost:6333/collections/nexa_knowledge | jq . > /dev/null 2>&1; then - echo "✅ Qdrant: OK" -else - echo "❌ Qdrant: FAILED" -fi - -# Test 4: GraphDB Repository -echo "Test 4: GraphDB Repository" -if curl -s -u "admin:${GRAPHDB_PASSWORD}" http://localhost:7200/repositories | jq . > /dev/null 2>&1; then - echo "✅ GraphDB: OK" -else - echo "❌ GraphDB: FAILED" -fi - -echo "" -echo "🎯 Create Test Memo:" -MEMO=$(curl -s -X POST http://localhost:5230/api/v1/memos \ - -H "Authorization: Bearer ${MEMOS_API_KEY}" \ - -H "Content-Type: application/json" \ - -d '{"content": "Test memo - #nexa:config"}' | jq -r '.id') - -echo "✅ Test Memo created: $MEMO" -echo "Check Memos UI to verify webhook triggers Discovery" -EOF - -chmod +x scripts/test_nexa.sh -./scripts/test_nexa.sh -``` - ---- - -## 🆘 Troubleshooting - -### Problem: Memos kann sich nicht mit n8n verbinden - -```bash -# Überprüfe n8n Webhook URL in Memos config -curl http://localhost:5678/webhook/memos - -# Falls 404: Workflow ist nicht aktiviert/deployed -docker-compose logs n8n | grep webhook -``` - -### Problem: GraphDB startet nicht - -```bash -# Memory-Limit erhöhen -docker-compose exec graphdb -e "GDB_JAVA_OPTS=-Xmx16g -Xms8g" restart - -# Logs checken -docker-compose logs graphdb -``` - -### Problem: Qdrant Collection leer - -```bash -# Überprüfe ob Memos Bridge Workflow läuft -docker-compose logs n8n | grep "memos_bridge" - -# Manuell test -curl -X POST http://localhost:5230/api/v1/memos \ - -H "Authorization: Bearer ${MEMOS_API_KEY}" \ - -d '{"content": "- [ ] Test task #nexa"}' -``` - ---- - -## 📈 Next Steps - -1. ✅ **Phase 1 deployt**: Memos Bridge & Discovery laufen -2. ⏳ **Phase 2 aktivieren**: Email Butler (wenn Nextcloud Mail konfiguriert) -3. ⏳ **Phase 3 setup**: Qdrant Embedding Pipeline -4. ⏳ **Phase 4 integration**: GraphRAG Queries testen -5. ⏳ **Phase 5**: Home Assistant Voice Integration - ---- - -## 📞 Support & Debugging - -```bash -# Alle Logs in eine Datei sammeln -docker-compose logs > deployment_logs.txt - -# Spezifischer Service -docker-compose logs n8n --tail 100 - -# Wiederherstellung aus Backup -./scripts/restore_nexa.sh backup/2024-01-15_120000/ -``` - ---- - -**Deployment abgeschlossen! Nexa ist nun betriebsbereit.** 🚀 diff --git a/nexa-core/PLAN.md b/nexa-core/PLAN.md deleted file mode 100644 index 06a14a6..0000000 --- a/nexa-core/PLAN.md +++ /dev/null @@ -1,27 +0,0 @@ -# NEXA: Implementierungs-Fahrplan 🚀 - -Dieser Plan beschreibt die schrittweise Aktivierung des Nervensystems. - -## Phase 1: Die Wirbelsäule (Basis-Konnektivität) -- [ ] **1.1 Memos-n8n Bridge:** Webhook-Anbindung und Filter-Logik. -- [ ] **1.2 SAIA-Integration:** LiteLLM Proxy in n8n einbinden (Chat-Funktion via Kommentare). -- [ ] **1.3 Git-Sync:** Automatisches Backup der n8n-Workflows in dieses Repo. - -## Phase 2: Die Sinnesorgane (Input-Kanäle) -- [ ] **2.1 Email-Butler:** Automatisierte Filterung und Zusammenfassung der ~20 täglichen Mails. -- [ ] **2.2 RSS Morning Digest:** Täglich kuratierte News-Zusammenfassung in Memos. -- [ ] **2.3 Bluesky-Antenne:** Speichern von gelikten Posts als Wissens-Schnipsel. - -## Phase 3: Das Gedächtnis (Qdrant & RAG) -- [ ] **3.1 Embedding-Pipeline:** Automatische Vektorisierung von Memos & Obsidian. -- [ ] **3.2 Context-Retrieval:** Nexa nutzt Qdrant-Wissen für Antworten (#ask). -- [ ] **3.3 S3-Langzeitarchiv:** Backup der Qdrant-Snapshots auf den S3-Server. - -## Phase 4: Die Motorik (Organisation & Handeln) -- [ ] **4.1 Nextcloud Task-Sync:** Bidirektionaler Status-Abgleich (Memos <-> NC Tasks). -- [ ] **4.2 Timeboxing-Agent:** Intelligente Kalender-Slot-Vorschläge basierend auf Tasks. -- [ ] **4.3 Karakeep-Kuration:** Automatischer Review-Prozess für gespeicherte Links. - -## Phase 5: Alltags-Integration (Feinschliff) -- [ ] **5.1 Home Assistant Voice:** Nexa als Wake-Word Integration. -- [ ] **5.2 System-Monitoring:** Alarme von Proxmox/Backrest direkt in den Memos-System-Feed. diff --git a/nexa-core/README.md b/nexa-core/README.md index ea1e6c0..1a53502 100644 --- a/nexa-core/README.md +++ b/nexa-core/README.md @@ -1,15 +1,17 @@ -# NEXA 🧠 -**The Neural Nexus for Information & Automation** +# nexa-core -Nexa ist das zentrale Nervensystem meines IT-Setups. +Runtime artifacts for Nexa: n8n workflows, configuration, AI prompts, scripts. -## 🏗 Architektur -- **Interface:** Memos -- **Logic:** n8n -- **Memory:** Qdrant & Obsidian -- **Intelligence:** SAIA (via LiteLLM) +📖 Documentation lives one level up in [`../docs/`](../docs/index.md). -## 📁 Struktur -- `n8n-workflows/`: JSON-Exporte der Gehirn-Logik. -- `ai-prompts/`: System-Prompts für Nexa. -- `config/`: Schemata für Qdrant und API-Mappings. +## Layout + +- `n8n-workflows/` — exported workflows (per phase). Imported into the running n8n. +- `ai-prompts/` — system prompts for SAIA / LiteLLM. +- `config/` — runtime config (`agents.yaml`, `infra_manifest.yaml`, `qdrant_schema.json`). +- `scripts/` — automation helpers (workflow backup, etc.). +- `.env.example` — bootstrap secrets only; runtime config lives in Qdrant `_config` namespace. + +## Quickstart + +See [`../docs/09-deployment.md`](../docs/09-deployment.md). diff --git a/nexa-core/doc/scope.md b/nexa-core/doc/scope.md deleted file mode 100644 index 925ad72..0000000 --- a/nexa-core/doc/scope.md +++ /dev/null @@ -1,80 +0,0 @@ -📑 Projekt Nexa: Scoping & Vision -1. Vision & Zielsetzung -Nexa ist als "Zentrales Nervensystem" konzipiert. Das Ziel ist die Schaffung einer intelligenten Middleware zwischen Input-Quellen (Memos, E-Mail, RSS), Wissensspeichern (Obsidian, Qdrant) und Organisations-Tools (Nextcloud Calendar/Tasks). - -Kernziele: -Kognitive Entlastung: Nexa sortiert, filtert und schlägt vor, anstatt nur zu speichern. - -Kontext-Trennung: Saubere, KI-gestützte Unterscheidung zwischen Privat und Arbeit. - -Single Point of Interaction: Memos fungiert als primäre Schnittstelle ("The Voice & Ear"). - -Wissens-Synergie: Verknüpfung von flüchtigen Memos mit tiefem Wissen in Obsidian via semantischer Suche. - -2. Abgrenzung (Scope) -Um die Komplexität beherrschbar zu halten, wird das Projekt wie folgt abgegrenzt: - -In-Scope (Was Nexa tut): -Triaging: Klassifizierung von 20 Mails/Tag (Privat). - -Task-Routing: Verteilung von Aufgaben auf die richtige Nextcloud-Liste (Work vs. Personal). - -Scheduling: Vorschlagen von Timeboxen im Arbeitskalender. - -Memory: Aufbau eines Langzeitgedächtnisses in Qdrant. - -Monitoring: Aggregation kritischer IT-Alarme (Proxmox, Backrest). - -Out-of-Scope (Was Nexa NICHT tut): -Kein direkter Zugriff auf Arbeits-IT (E-Mail/Server). - -Keine aktive Bearbeitung von Dokumenten (außer Metadaten-Extraktion). - -Keine Ersetzung von Spezial-UIs (z.B. Arcane oder Portainer für Container-Management). - -3. Phasenmodell der Entwicklung -Der Aufbau erfolgt iterativ, um frühzeitig Mehrwert zu generieren. - -Phase 1: Die Wirbelsäule (Konnektivität) -Fokus: Datenfluss zwischen Memos, n8n und SAIA. - -Meilenstein: Nexa antwortet auf Memos und kann einfache Fragen beantworten. - -Technik: Webhooks, REST-API, LiteLLM-Proxy. - -Phase 2: Sensorik & Klassifizierung (The Senses) -Fokus: E-Mail-Filterung und das Routing-System. - -Meilenstein: Nexa erkennt den Unterschied zwischen "Arbeits-Task" und "Privat-Task" ohne manuelle Tags. - -Technik: IMAP-Node, n8n-Logic-Router, System-Prompts für Kontext-Analyse. - -Phase 3: Das semantische Gedächtnis (The Brain) -Fokus: Anbindung von Qdrant und Obsidian. - -Meilenstein: RAG (Retrieval Augmented Generation). Nexa beantwortet Fragen basierend auf deinen archivierten Notizen. - -Technik: Embeddings-Pipeline, Qdrant-Vektor-Suche. - -Phase 4: Proaktive Motorik (The Assistant) -Fokus: Kalender-Integration und Timeboxing. - -Meilenstein: Nexa schlägt morgens proaktiv Fokus-Zeiten im Arbeitskalender vor. - -Technik: Nextcloud CalDAV/CardDAV Integration, Optimierungs-Logik für freie Slots. - -Phase 5: Physische Präsenz (Voice & Alarms) -Fokus: Home Assistant Voice & Monitoring-Integration. - -Meilenstein: Nexa spricht über HA-Voice und meldet kritische Systemzustände proaktiv. - -Technik: Home Assistant API, Wyoming Protocol, ntfy/Memos-Push. - -4. Erfolgsmetriken -Woran merken wir, dass Nexa funktioniert? - -Reduktion der Mail-Interaktion: Weniger Zeit in der Mail-App, mehr Fokus auf den Nexa-Digest. - -Kalender-Füllrate: Fokus-Slots für Arbeit sind automatisch geblockt. - -Such-Geschwindigkeit: Informationen aus alten Memos/Obsidian werden via #ask sofort gefunden. \ No newline at end of file From a1e14c64c3beba10a0ce81c505185f715cc4b35c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:26:08 +0000 Subject: [PATCH 02/14] =?UTF-8?q?Apply=20Q1=E2=80=93Q3=20decisions:=20Grap?= =?UTF-8?q?hDB/SPARQL,=20reuse=20Qdrant,=20self-host=20embeddings=20via=20?= =?UTF-8?q?TEI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Q1 → Ontotext GraphDB (SPARQL). docs/08 fully rewritten with RDF schema, example SPARQL queries (transitive deps via property paths, time-windowed topic counts, cross-pillar joins via nexa:vectorId). - Q2 → reuse qdrant_scientific with nexa_* collection prefix; docs/09 step 2 now points there explicitly. - Q3 → no OpenAI embeddings. Self-host on the docker host. Use TEI (HuggingFace text-embeddings-inference) — single Rust binary, ~500 MB image, OpenAI-compatible — instead of Ollama, since we only need embeddings. - docs/09 Phase-3.4 add-on simplified to a single Ontotext compose snippet (Neo4j option dropped) plus repo creation curl. - docs/11 Q3 marked resolved; new Q15 picks the model (bge-m3 vs nomic-embed) and adds the open question of whether SAIA already proxies an embedding model that would let us skip TEI entirely. - docs/03 + CLAUDE.md updated with the new decisions so future runs don't re-litigate. --- CLAUDE.md | 5 + docs/03-architecture.md | 13 +- docs/08-graphrag-architecture.md | 405 ++++++++++++++----------------- docs/09-deployment.md | 92 +++++-- docs/11-open-questions.md | 40 ++- 5 files changed, 298 insertions(+), 257 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 95fbf1f..95bd968 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,6 +20,11 @@ This file tells future automated runs what they need to know about this repo. - VM 100 haos — Home Assistant. - **Already-running services on docker host** (don't redeploy): - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). +- **Decided for Nexa** (don't re-litigate without user input): + - Vector store: **reuse `qdrant_scientific`** with `nexa_*` collection prefix. + - Embeddings: **TEI** (HF text-embeddings-inference), self-hosted, `bge-m3` candidate (Q15 still pending the model name). + - Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. + - Chat model: SAIA via LiteLLM virtual key. ## When working on Nexa diff --git a/docs/03-architecture.md b/docs/03-architecture.md index 544df46..d796f8f 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -21,11 +21,11 @@ A one-page mental model. For details follow the cross-links. │ SAIA │ │ Qdrant │ │ LiteLLM │ │ vector │ └──────────┘ └────┬─────┘ - │ optional + │ ▼ ┌──────────┐ - │ Graph │ (Neo4j or Ontotext — - │ DB │ see open questions) + │ GraphDB │ Ontotext, SPARQL + │ (RDF) │ Phase 3.4 └──────────┘ │ ▼ writes @@ -42,8 +42,9 @@ A one-page mental model. For details follow the cross-links. | **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | -| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 (existing `qdrant_scientific` may be reused — see [11](./11-open-questions.md)) | -| **Graph DB** | Structural memory (deferred) | not yet deployed | +| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse existing `qdrant_scientific` with `nexa_*` collection prefix | +| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-graph-db-phase-34) | +| **TEI** (HF text-embeddings-inference) | Self-hosted embeddings (`bge-m3` / TBD — see [11/Q15](./11-open-questions.md)) | docker host LXC 104, CPU only — single-binary embedding server | | **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` | | **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` | | **Backrest** | Backup orchestration | LXC 103 | @@ -54,7 +55,7 @@ A one-page mental model. For details follow the cross-links. ## Two-pillar memory - **Qdrant** answers *"what is similar / relevant?"* (cosine search over embeddings). -- **Graph DB** answers *"who, what depends on what, how is it structured?"* (Cypher / SPARQL). +- **Ontotext GraphDB** answers *"who, what depends on what, how is it structured?"* (SPARQL over RDF). Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. See [08 — GraphRAG architecture](./08-graphrag-architecture.md). diff --git a/docs/08-graphrag-architecture.md b/docs/08-graphrag-architecture.md index 430b9fa..5cd1ddd 100644 --- a/docs/08-graphrag-architecture.md +++ b/docs/08-graphrag-architecture.md @@ -1,277 +1,246 @@ # 08 — GraphRAG: Structural Knowledge & Relations -> ⚠️ **Open decision (see [11-open-questions](./11-open-questions.md))**: this document describes the design with **Neo4j**. The earlier deployment draft referenced **Ontotext GraphDB** (SPARQL). Pick one before Phase 3.4. - -## GraphRAG für Nexa — Strukturelles Wissen & Beziehungen - -## 🧠 Zwei-Säulen-Memory-Architektur - -Nexa nutzt zwei komplementäre Speicher für vollständiges Verständnis: - -### Säule 1: Qdrant (Vector DB) - Semantische Ähnlichkeit -- **Frage**: "Was ist ähnlich, was ist relevant?" -- **Speichert**: Embeddings von Memos, Obsidian-Notes, E-Mail-Digests -- **Suche**: Cosine Similarity über 1536-dimensionale Vektoren -- **Antwort**: Top-k semantisch ähnliche Dokumente (RAG) - -**Beispiel:** -``` -Frage: "Wie war die JWT-Implementation in unserem letzten Projekt?" -→ Qdrant findet: 10 ähnliche Snippets → Prompt-Context -``` - -### Säule 2: Neo4j (Graph DB) - Strukturelle Beziehungen -- **Frage**: "Wer ist beteiligt? Was hängt davon ab? Wie ist es organisiert?" -- **Speichert**: - - **Knoten (Nodes)**: Projects, Tasks, People, Technologies, Dates - - **Kanten (Relationships)**: `USES`, `OWNS`, `DEPENDS_ON`, `MENTIONS`, `CREATED_BY` -- **Suche**: Cypher-Queries über Netzwerk-Topologie -- **Antwort**: Pfade, zentrale Knoten, Abhängigkeitsanalyse - -**Beispiel:** -```cypher -MATCH (tech:Technology)-[USES*]->*(service:Service), - (tech)-[:MENTIONED_IN]->(task:Task) -WHERE tech.name = "JWT" AND task.status = "DONE" -RETURN tech, service, task -``` +Decision: graph layer = **Ontotext GraphDB** with **SPARQL** (resolved in [11/Q1](./11-open-questions.md)). +Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later. --- -## 📊 Neo4j Schema für Nexa +## Two-pillar memory -### Node-Typen +| Pillar | Question it answers | Backed by | +|--------|---------------------|-----------| +| **Qdrant** (vectors) | *"What is similar / relevant?"* | Cosine search over embeddings | +| **GraphDB** (RDF) | *"What is connected? What depends on what? Who is involved?"* | SPARQL over a typed graph | -```cypher -(Project {name, startDate, status, priority}) -(Task {title, description, status, urgency: 1-5, context: "work"|"personal"}) -(Person {name, role, email}) -(Technology {name, category: "framework"|"language"|"tool"}) -(Topic {name, domain}) -(File {path, type, lastModified}) -(Note {id, content_hash, created_at}) -``` - -### Relationship-Typen - -| Beziehung | Von | Zu | Beschreibung | -|-----------|-----|-----------|-------------| -| `OWNS` | Person | Task/Project | Wer ist verantwortlich | -| `USES` | Task/Project | Technology | Welche Tech wird verwendet | -| `DEPENDS_ON` | Task/Project | Task/Project | Hängt davon ab | -| `CREATED_BY` | Note | Person | Autor einer Notiz | -| `MENTIONS` | Note/Task | Topic/Technology | Verweist auf | -| `RELATED_TO` | Task | Task | Zusammenhängend | -| `CHILD_OF` | Task | Project | Task gehört zu Project | -| `SCHEDULED_FOR` | Task | Date | Zeitliche Zuordnung | +Both pillars are queried in parallel for `#nexa:ask` and merged before SAIA generates the final answer. --- -## 🔄 Synchronisierungs-Flows +## RDF schema -### 1. **Memos → Neo4j** (automatisch beim Erstellen) +Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain. -Wenn ein Memo mit `- [ ]` erstellt wird: +```turtle +@prefix nexa: . +@prefix rdf: . +@prefix rdfs: . +@prefix xsd: . +@prefix prov: . -``` -Memo-Content: "Muss JWT-Middleware für Auth-Service refaktorieren" - ↓ -SAIA extrahiert Entitäten: - - Technologie: "JWT-Middleware", "Auth-Service" - - Kontext: "work" - - Urgenz: 3 - ↓ -Erstelle in Neo4j: - - Task node (title="JWT-Middleware refaktorieren") - - Technology nodes: JWT, Auth-Service - - Relationships: Task -[USES]-> JWT, Task -[USES]-> Auth-Service - - Task -[CREATED_BY]-> Me (Person node) +# Classes +nexa:Project a rdfs:Class . +nexa:Task a rdfs:Class . +nexa:Person a rdfs:Class . +nexa:Technology a rdfs:Class . +nexa:Topic a rdfs:Class . +nexa:Note a rdfs:Class . # Memos / Obsidian / mail digests +nexa:File a rdfs:Class . + +# Properties +nexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task . +nexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology . +nexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task . +nexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project . +nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic . +nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime . + +# Datatype properties +nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done" +nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal" +nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5 +nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID — bridges the two pillars +nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup ``` -### 2. **Obsidian → Neo4j** (#nexa:sync-obsidian) - -Beim Sync von Obsidian-Vault: - -``` -File: obsidian/projects/nexa-deployment.md - ↓ -Header Struktur wird zu Graph: - - Project "Nexa Deployment" - - Technologies: Docker, n8n, Nextcloud - - Dependencies: "requires infrastructure setup" - ↓ -Erstelle: - - Project nodes mit Metadata - - Hierarchie: Project -[CONTAINS]-> Task -[USES]-> Technology -``` - -### 3. **Nextcloud Tasks → Neo4j** (bidirektional) - -Wenn Task in Nextcloud erstellt/gelöcht: - -``` -Nextcloud Task: "Review JWT implementation" (in Work-List) - ↓ -n8n Trigger erkennt Änderung - ↓ -Update Neo4j: - - Task node status = "active" - - Task -[DEPENDS_ON]-> related Task (falls Beziehung existiert) -``` +The `nexa:vectorId` property is the **bridge** between graph and vector store. Every node that has semantic content carries the Qdrant point ID, so a SPARQL hit can trigger a vector lookup and vice versa. --- -## 🤖 GraphRAG Queries - Praktische Anwendungen +## Sync flows -### Query 1: "Zeige alle offenen Tasks, die JWT betreffen" +### 1. Memos → GraphDB (real-time) -```cypher -MATCH (tech:Technology {name:"JWT"}), - (tech)-[:MENTIONED_IN|USES]-(task:Task), - (task)-[:CHILD_OF]->(project:Project) -WHERE task.status IN ["NEEDS-ACTION", "IN-PROGRESS"] -RETURN project.name, task.title, task.urgency -ORDER BY task.urgency DESC +``` +Memo content: "Muss JWT-Middleware für Auth-Service refaktorieren" + │ + ▼ SAIA extracts entities + relations as JSON + │ { tasks: [{title, urgency}], technologies: [...], + │ relations: [{type:"uses", from:..., to:...}] } + │ + ▼ n8n turns JSON into a SPARQL UPDATE + │ + └──▶ INSERT DATA { ... } against GraphDB repo "nexa_knowledge" ``` -### Query 2: "Von welchen Tasks/Projects hängt Auth-Service ab?" +### 2. Obsidian → GraphDB (`#nexa:sync-obsidian`) -```cypher -MATCH path = (dependent)-[:DEPENDS_ON*]->(service:Technology {name:"Auth-Service"}) -RETURN [x IN nodes(path) | x.name], length(path) -ORDER BY length(path) ASC -``` +For each Obsidian note: parse front-matter + headings → emit `nexa:Project`, `nexa:Task`, `nexa:Note` triples; `nexa:mentions` for `[[wikilinks]]`. -### Query 3: "Wer arbeitet an diesem Projekt und auf welchen Technologien?" +### 3. Nextcloud Tasks ↔ GraphDB (bidirectional) -```cypher -MATCH (person:Person)-[:OWNS]->(task:Task)-[:CHILD_OF]->(project:Project {name:"Nexa"}), - (task)-[:USES]->(tech:Technology) -RETURN person.name, collect(task.title), collect(tech.name) -``` - -### Query 4: "Was ist seit gestern neu an Technologien/Themen gelernt?" - -```cypher -MATCH (note:Note)-[:MENTIONS]->(topic:Topic) -WHERE note.created_at > datetime.now() - duration('P1D') -RETURN DISTINCT topic.name, count(note) as mentions -ORDER BY mentions DESC -``` +n8n trigger on Nextcloud CalDAV/Tasks change → `INSERT/DELETE DATA` to keep `nexa:status` and `nexa:scheduledFor` in sync. --- -## 🔗 GraphRAG-Enhanced Answer Generation +## Example SPARQL queries -Wenn Nutzer `#nexa:ask` fragt: +### Q1 — All open tasks involving JWT, by urgency +```sparql +PREFIX nexa: + +SELECT ?taskTitle ?urgency ?projectName +WHERE { + ?tech rdfs:label "JWT" . + ?task nexa:uses ?tech ; + rdfs:label ?taskTitle ; + nexa:status ?status ; + nexa:urgency ?urgency . + FILTER (?status IN ("needs-action", "in-progress")) + OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . } +} +ORDER BY DESC(?urgency) ``` -#nexa:ask Wie stehen die JWT-Tasks zur Deadline? - ↓ - ┌─────────────┴──────────────┐ - ↓ ↓ - [Qdrant] [Neo4j] - Semantic Structural - Search Search - ↓ ↓ - Top-3 Notes (JWT)-[DEPENDS_ON] - mit ähnlichem → Connected Tasks - Kontext → Critical Path - ↓ ↓ - Combine: Merge + Rank - ↓ - System Prompt (SAIA): - "Basierend auf diesem Wissen und diesen Abhängigkeiten, beantworte:" - ↓ - Response mit: - - Semantisch relevanter Information (Qdrant) - - Strukturellem Kontext (Neo4j) - - Abhängigkeitsanalyse + +### Q2 — What does Auth-Service transitively depend on? + +```sparql +PREFIX nexa: + +SELECT DISTINCT ?dep ?label +WHERE { + ?root rdfs:label "Auth-Service" . + ?root nexa:dependsOn+ ?dep . + ?dep rdfs:label ?label . +} ``` +(`+` is SPARQL property-paths — transitive closure, free.) + +### Q3 — Topics with the most note-mentions in the last day + +```sparql +PREFIX nexa: +PREFIX xsd: + +SELECT ?topic (COUNT(?note) AS ?n) +WHERE { + ?note a nexa:Note ; + prov:generatedAtTime ?ts ; + nexa:mentions ?topic . + FILTER (?ts > NOW() - "P1D"^^xsd:duration) +} +GROUP BY ?topic +ORDER BY DESC(?n) +LIMIT 10 +``` + +### Q4 — Cross-pillar: "find vectors for tasks blocking project X" + +```sparql +PREFIX nexa: + +SELECT ?taskTitle ?vectorId +WHERE { + ?proj rdfs:label "Nexa" . + ?task nexa:childOf ?proj ; + nexa:status "in-progress" ; + nexa:vectorId ?vectorId ; + rdfs:label ?taskTitle . +} +``` + +n8n then takes each `?vectorId`, fetches the embedding from Qdrant, and runs a "more like this" search for richer context. + --- -## 🛠️ n8n Integration (Phase 3) +## GraphRAG answer pipeline (`#nexa:ask`) + +``` + #nexa:ask + │ + ┌──────┴──────┐ + ▼ ▼ + [Qdrant] [GraphDB] + semantic structural + top-k SPARQL — auto-generated + notes paths / dependencies + │ │ + └──────┬──────┘ + ▼ + merge + rank + │ + ▼ + SAIA prompt: + "Given these passages and these relations, answer …" + │ + ▼ + comment under the original memo +``` + +Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a `SELECT`/`CONSTRUCT` query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure. + +--- + +## n8n integration sketch ### Workflow: Graph-Sync Trigger ``` -[Memos Webhook] - ↓ +[Memos Webhook] + │ [Parse Content] - ↓ -[SAIA: Extract Entities] - ├→ Tasks, Projects, Technologies, People - └→ Relationships (uses, depends_on, mentions) - ↓ -[Neo4j: Create/Update Nodes & Relationships] - ↓ -[Index in Qdrant Metadata: Add graph_node_id] + │ +[SAIA: Extract entities + relations as JSON] + │ +[Build SPARQL UPDATE INSERT DATA { ... }] + │ +[HTTP POST → /repositories/nexa_knowledge/statements] + │ +[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE] ``` ### Workflow: Question Router ``` [#nexa:ask Query] - ↓ -[Parallel Search] - ├→ [Qdrant: Vector Similarity] - ├→ [Neo4j: Graph Query (auto-generated Cypher)] - └→ [Rank & Merge Results] - ↓ -[SAIA: Generate Answer with Context] + │ + ┌─┴────────────────┐ + ▼ ▼ +[SAIA: NL → SPARQL] [Qdrant: kNN] + │ │ +[POST → SPARQL endpoint] + │ │ + └──────┬───────────┘ + ▼ + rank + merge → SAIA answer ``` --- -## 📋 Commands für Graph-Management +## Graph-management commands -### #nexa:graph-status -Zeigt Statistiken der Graph-DB -``` -#nexa:graph-status -``` +### `#nexa:graph-status` -Response: -``` -📊 Neo4j Graph Status: -- Nodes: 342 (Tasks: 145, Projects: 12, Technologies: 52, Notes: 133) -- Relationships: 1,247 -- Density: 0.34 -- Most Connected: "JWT-Middleware" (27 connections) -``` +Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL `SELECT (COUNT)`. -### #nexa:graph-trace [entity] -Zeigt alle Verbindungen für eine Entität -``` -#nexa:graph-trace JWT-Middleware -``` +### `#nexa:graph-trace [entity]` -### #nexa:graph-rebuild -Rekonstruiert Graph aus Obsidian + Memos (einmalig, danach inkrementell) -``` -#nexa:graph-rebuild --force -``` +Returns the 1-hop (and optionally 2-hop) neighbourhood — a `DESCRIBE ` plus a templated outgoing/incoming query. + +### `#nexa:graph-rebuild` + +Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH ` followed by the import workflow. --- -## 🚀 Phase 3 Roadmap (Graph Integration) +## Why two stores -- [ ] **3.1 Neo4j Setup**: Collection, Constraints, Indexes -- [ ] **3.2 Graph-Sync Workflow**: Automatische Entity Extraction -- [ ] **3.3 GraphRAG Pipeline**: Kombinierte Qdrant + Neo4j Queries -- [ ] **3.4 Dependency Tracker**: Warnt vor zirkulären Dependencies -- [ ] **3.5 Critical Path Analysis**: Berechnet längsten Weg im Graph +| Scenario | Qdrant | GraphDB | Best | +|----------|--------|---------|------| +| "Which note was similar to this one?" | ✅ | ❌ | Qdrant | +| "What blocks this task?" | ❌ | ✅ | GraphDB | +| "Explain this project" | ✅ context | ✅ structure | both | +| "All JWT-related open work" | ✅ semantic | ✅ crisp | both | ---- - -## 💡 Warum zwei DBs? - -| Szenario | Qdrant | Neo4j | Kombi | -|----------|--------|-------|-------| -| "Welche Note war ähnlich?" | ✅ Best | ❌ | Qdrant | -| "Welche Tasks blockieren diesen Task?" | ❌ | ✅ Best | Neo4j | -| "Erkläre mir dieses Projekt umfassend" | ✅ Context | ✅ Structure | **Beide!** | -| "Finde alle JWT-bezogenen offenen Work" | ✅ Semantic | ✅ Crisp | **Beide!** | - -Kombiniert: **Vollständiges Verständnis** statt nur Suchindex oder nur Struktur. +Combined: **complete understanding** rather than a search index *or* a structure index. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 7358e64..298e327 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -20,19 +20,20 @@ Surveyed from Homepage / Dozzle / Proxmox / Zoraxy: | Backrest | LXC 103 | (internal) | | AdGuard DNS | LXC 102 | (internal) | | Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems | -| `qdrant_scientific` (existing) | docker LXC 104 | reuse — see [11](./11-open-questions.md) | +| `qdrant_scientific` (existing) | docker LXC 104 | **reused** — Nexa uses `nexa_*` collections in this instance | The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**. ## What's missing for Nexa -1. **Qdrant collection** for Nexa (`nexa_knowledge`) — either on the existing `qdrant_scientific` or a dedicated container. -2. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n. -3. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`). -4. **Memos webhook → n8n** wired through the Memos config. -5. **LiteLLM virtual key** for the `nexa` user with embedding + chat models. -6. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied. -7. **(Phase 3.4)** decision on graph DB (Neo4j vs. Ontotext GraphDB) — see [11](./11-open-questions.md). +1. **Qdrant collection** for Nexa (`nexa_knowledge`) inside the existing `qdrant_scientific` instance — vector dim follows Q15 (1024 for `bge-m3`, 768 for `nomic-embed-text`). +2. **TEI** (HF text-embeddings-inference) on the docker host for self-hosted embeddings (LiteLLM key is **not** authorised for OpenAI embeddings — see [11/Q3+Q15](./11-open-questions.md)). Lighter than Ollama: single Rust binary, ~500 MB image, no LLM runtime. +3. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n. +4. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`). +5. **Memos webhook → n8n** wired through the Memos config. +6. **LiteLLM virtual key** for the `nexa` user with chat-only access (no embeddings — handled by Ollama). +7. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied. +8. **(Phase 3.4) Ontotext GraphDB** for the SPARQL pillar — see add-on at the bottom of this doc. --- @@ -60,14 +61,43 @@ curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \ -d @nexa-core/config/qdrant_schema.json ``` -> If reusing `qdrant_scientific`, this just adds another collection; no new container needed. +> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container. +> The schema file's `vectors.size` must match the embedding model picked in [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`. -## Step 3 — LiteLLM virtual key +## Step 3 — Self-hosted embeddings (TEI) + +Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry. + +```bash +# on the docker host (LXC 104) +docker run -d --name nexa-embed \ + --restart unless-stopped \ + -p 127.0.0.1:8080:80 \ + -v tei-data:/data \ + ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 \ + --model-id BAAI/bge-m3 +``` + +Memory budget: ~1.1 GB resident (bge-m3 is ~1 GB + ~100 MB overhead). First start downloads the model into the named volume; subsequent restarts are instant. + +Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: + +- model name: `nexa-embed` +- provider: `openai` +- model: `bge-m3` +- api_base: `http://nexa-embed:80/v1` +- api_key: any non-empty string (TEI ignores it) + +Now n8n only ever talks to LiteLLM and the model is swappable without touching workflows. + +## Step 4 — LiteLLM virtual key In the LiteLLM admin UI (`ai.nuclide.systems`): 1. Create user `nexa`. -2. Issue a virtual key with access to one chat model (e.g. `gpt-4o-mini` / a local Sonnet equivalent) and one embedding model (`text-embedding-3-small`). +2. Issue a virtual key with access to: + - one chat model (already-available model from your SAIA gateway). + - the `nexa-embed` model from Step 3. 3. Paste the key into `SAIA_API_KEY` in `.env`. ## Step 4 — n8n workflows @@ -158,31 +188,43 @@ For deeper detail: [10 — Operations](./10-operations.md). --- -## Phase add-on: graph DB (Phase 3.4) +## Phase add-on: Ontotext GraphDB (Phase 3.4) -Defer until 3.1–3.3 ship. When needed, two minimal options — pick one: - -### Option A — Neo4j (lighter, popular for graph-RAG) +Defer until 3.1–3.3 ship. ```yaml # nexa-core/docker-compose.graph.yml services: - neo4j: - image: neo4j:5-community - container_name: nexa-neo4j - ports: ["7687:7687", "7474:7474"] + graphdb: + image: ontotext/graphdb:10.7.0 + container_name: nexa-graphdb + ports: ["127.0.0.1:7200:7200"] environment: - NEO4J_AUTH: "neo4j/${NEO4J_PASSWORD}" - NEO4J_server_memory_heap_max__size: 2G - volumes: ["./data/neo4j:/data"] + GDB_JAVA_OPTS: "-Xmx4g -Xms1g" + volumes: + - ./data/graphdb:/opt/graphdb/home restart: unless-stopped ``` -### Option B — Ontotext GraphDB (SPARQL, RDF native) +After first start, create the repository (one-time): -Keep the snippet from the original `DEPLOYMENT.md` Schritt 1.3 for `graphdb`. Heavier (~4 GB heap), only justified if you want SHACL / OWL reasoning. +```bash +curl -X POST http://localhost:7200/rest/repositories \ + -H 'Content-Type: application/json' \ + -d '{ + "id": "nexa_knowledge", + "title": "Nexa Knowledge Graph", + "type": "graphdb", + "params": { + "ruleset": {"value": "rdfsplus-optimized"}, + "baseURL": {"value": "https://nuclide.systems/nexa/"} + } + }' +``` -Add a Zoraxy host entry `graph.nuclide.systems` → `192.168.1.40:7474` (Neo4j) or `:7200` (GraphDB) **only if** browser access is desired; otherwise n8n talks to it on the docker network. +Optional Zoraxy entry `graph.nuclide.systems` → `192.168.1.40:7200` if you want the SPARQL Workbench in a browser; otherwise n8n talks to it on the docker network at `http://nexa-graphdb:7200`. + +For schema and example queries: [08-graphrag-architecture](./08-graphrag-architecture.md). --- diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index ff5e7c8..e4ea61f 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -2,13 +2,22 @@ Items that **block progress** and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide. +## Resolved + +- [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly. +- [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container. +- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below; concrete model still open as **Q15**. + ## Architectural decisions -- [ ] **Q1 — Graph DB choice.** Phase 3.4 says *Neo4j* in [08](./08-graphrag-architecture.md) but the original deployment draft used *Ontotext GraphDB*. **Recommendation:** Neo4j (smaller footprint, Cypher is friendlier for n8n HTTP nodes, no SHACL/OWL needs identified). Confirm or override. -- [ ] **Q2 — Reuse `qdrant_scientific` or run a dedicated `nexa-qdrant`?** Reuse keeps memory usage flat (~670 MB seen on Homepage); dedicated gives clean isolation and independent snapshot lifecycle. **Recommendation:** reuse with a `nexa_*` collection prefix. -- [ ] **Q3 — Embeddings model.** Default in `infra_manifest.yaml` is `text-embedding-3-small` (OpenAI). Is the LiteLLM virtual key authorised to call it, or should we route to a self-hosted alternative (e.g. `bge-m3` via Ollama on the same host)? - [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. -- [ ] **Q5 — Karakeep vs. Hoarder naming.** The Zoraxy host is `hoarder.nuclide.systems` but the container is `karakeep-*` and Homepage labels it Karakeep. They're the same product (rename in 2024). Pick one display name to use in docs and prompts. +- [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. +- [ ] **Q15 — Embedding model (served via TEI).** Two viable options on the docker host (CPU only — see speed budget): + - **`BAAI/bge-m3`** (568 M params, ~1 GB RAM, multilingual incl. German, 1024-dim, ~10–20 docs/s on CPU). **Recommended** because the corpus is DE/EN-mixed. Requires `qdrant_schema.json` `vectors.size = 1024`. + - **`nomic-ai/nomic-embed-text-v1.5`** (137 M, ~250 MB RAM, EN-leaning, 768-dim, ~50 docs/s CPU). Lighter / faster but weaker on German. + - **(maybe) does SAIA already proxy any embedding model?** If the SAIA backend offers e.g. `mistral-embed` for free, we can skip TEI entirely. Worth a 1-line check in the LiteLLM admin UI before deploying TEI. + + Decide before Phase 3.1. ## Identifiers needed (auto-discoverable, but list now if known) @@ -20,10 +29,25 @@ Items that **block progress** and need a human decision before a workflow can be ## Hardware / capacity -- [ ] **Q11 — RAM headroom on docker host.** Homepage shows 29.5 GiB free out of ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing of Obsidian + Memos history is fine here, but a Phase-3.4 GraphDB choice changes this (Neo4j ~2 GB heap, Ontotext ~4 GB). Confirm acceptable. -- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots? +- [ ] **Q11 — RAM headroom on docker host.** 29.5 GiB free / ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing + TEI/`bge-m3` (~1.1 GB) + Ontotext GraphDB (~4 GB heap) ⇒ ~14 GB used worst case, still ample. Confirm acceptable. +- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? ## Process -- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but flagging it because Phase 5 monitoring would alert on it. Suppress, or is it intentional? -- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage — Homepage doesn't have a Zoraxy widget yet. Cosmetic, unrelated to Nexa. +- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but Phase-5 monitoring would alert on it. Suppress or is it intentional? +- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage. Cosmetic, unrelated to Nexa. + +--- + +## Speed budget (Q3 follow-up) + +Workload on the docker host (16 CPU, ~30 GB free RAM): + +| Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` | +|------|--------|----------------|---------------------------------|------------------------------------| +| Real-time memo embed | 1 doc | <500 ms incl. n8n round-trip | ✅ ~50–100 ms | ✅ ~20 ms | +| Daily ingest | ~70 docs | <60 s | ✅ ~5–10 s | ✅ ~2 s | +| Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~2–4 min | ✅ <1 min | +| RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms | + +Conclusion: CPU-only Ollama is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. From 17c7f5033fd8aef15daa5c51b102f19a5a33ea0f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:33:59 +0000 Subject: [PATCH 03/14] Stage embeddings: ship Path A (text) now, prepare Path C (text+visual) additively MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decision (Q15 resolved): start with text-only via TEI + bge-m3 in Phase 3.1, prepare data shapes so Phase 3.2 (visual collection via infinity + jina-clip-v2) is a pure additive operation — no rename, no schema migration, no n8n rewiring. Concretely: - Qdrant collection renamed nexa_knowledge → nexa_knowledge_text (1024-dim for bge-m3) with modality-aware payload (modality, source_type, media_uri, graph_iri, content_hash, context). Visual placeholder schema committed alongside (qdrant_schema_visual.json, 768-dim, jina-clip-v2). - Image attachments captured in 3.1 are recorded in GraphDB as nexa:Note with nexa:modality "image" + nexa:pendingVisualIndex true; the 3.2 backfill workflow picks them up and embeds. No data lost between phases — the queue is the GraphDB itself. - RDF schema (docs/08) gains nexa:modality, nexa:mediaUri, nexa:vectorCollection, nexa:pendingVisualIndex from day one. - docs/02 roadmap split: 3.1 = text RAG (Path A), 3.2 = visual collection (Path C), 3.4 = Ontotext GraphDB. - docs/09 grows a "Phase add-on: visual collection (Phase 3.2)" section with the TEI→infinity swap, second collection create, LiteLLM second model registration, and the SPARQL-driven backfill query. - New open questions: Q16 (queue ergonomics + does SAIA already proxy an embed model?), Q17 (reuse Immich's CLIP for photo-library queries?). - docs/03 + CLAUDE.md updated so future runs use the new collection names and don't re-decide the staging. --- CLAUDE.md | 6 +- docs/02-roadmap.md | 12 ++-- docs/03-architecture.md | 6 +- docs/08-graphrag-architecture.md | 20 +++++-- docs/09-deployment.md | 68 ++++++++++++++++++++-- docs/11-open-questions.md | 16 ++--- nexa-core/config/qdrant_schema.json | 22 +++++-- nexa-core/config/qdrant_schema_visual.json | 26 +++++++++ 8 files changed, 142 insertions(+), 34 deletions(-) create mode 100644 nexa-core/config/qdrant_schema_visual.json diff --git a/CLAUDE.md b/CLAUDE.md index 95bd968..3054bab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,9 +21,9 @@ This file tells future automated runs what they need to know about this repo. - **Already-running services on docker host** (don't redeploy): - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). - **Decided for Nexa** (don't re-litigate without user input): - - Vector store: **reuse `qdrant_scientific`** with `nexa_*` collection prefix. - - Embeddings: **TEI** (HF text-embeddings-inference), self-hosted, `bge-m3` candidate (Q15 still pending the model name). - - Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. + - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). + - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive. + - Graph store: **Ontotext GraphDB** (SPARQL/RDF), Phase 3.4. RDF schema in docs/08 already includes `nexa:modality` / `nexa:mediaUri` / `nexa:vectorCollection` / `nexa:pendingVisualIndex`. - Chat model: SAIA via LiteLLM virtual key. ## When working on Nexa diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index fc59f97..1aae00c 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -22,13 +22,13 @@ Iterative build-out, value-first. Each phase is shippable on its own. ## Phase 3 — Memory (Qdrant & RAG) -**Focus:** Qdrant + (optional) graph DB for retrieval-augmented answers. -**Milestone:** RAG works — Nexa answers from archived notes. +**Focus:** Qdrant + graph layer for retrieval-augmented answers. +**Milestone:** RAG works — Nexa answers from archived notes; images captured today are queued for visual indexing later. -- [ ] **3.1 Embedding pipeline** — automatic vectorization of Memos & Obsidian. -- [ ] **3.2 Context retrieval** — `#nexa:ask` reads Qdrant before answering. -- [ ] **3.3 S3 long-term archive** — Qdrant snapshots to the S3 server. -- [ ] **3.4 Graph layer (deferred decision — see [11-open-questions](./11-open-questions.md))** — Neo4j *or* Ontotext GraphDB for structural queries. +- [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet. +- [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32). +- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. +- [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md). ## Phase 4 — Motor (organization & action) diff --git a/docs/03-architecture.md b/docs/03-architecture.md index d796f8f..6db8376 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -42,9 +42,9 @@ A one-page mental model. For details follow the cross-links. | **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | -| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse existing `qdrant_scientific` with `nexa_*` collection prefix | -| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-graph-db-phase-34) | -| **TEI** (HF text-embeddings-inference) | Self-hosted embeddings (`bge-m3` / TBD — see [11/Q15](./11-open-questions.md)) | docker host LXC 104, CPU only — single-binary embedding server | +| **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) | +| **Ontotext GraphDB** | Structural memory via SPARQL (Phase 3.4) | not yet deployed; see [09-deployment](./09-deployment.md#phase-add-on-ontotext-graphdb-phase-34) | +| **TEI** (HF text-embeddings-inference) | Self-hosted text embeddings, `BAAI/bge-m3`, Phase 3.1 | docker host LXC 104, CPU only — swapped for `infinity` in Phase 3.2 to add `jina-clip-v2` | | **Nextcloud** | Tasks, calendar, mail, files | dedicated LXC 105 → `nc.nuclide.systems` | | **ntfy** | Push channel for system alerts | docker host → `ntfy.nuclide.systems` | | **Backrest** | Backup orchestration | LXC 103 | diff --git a/docs/08-graphrag-architecture.md b/docs/08-graphrag-architecture.md index 5cd1ddd..110b24c 100644 --- a/docs/08-graphrag-architecture.md +++ b/docs/08-graphrag-architecture.md @@ -45,14 +45,22 @@ nexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range n nexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime . # Datatype properties -nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done" -nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal" -nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5 -nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID — bridges the two pillars -nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup +nexa:status a rdf:Property ; rdfs:range xsd:string . # "needs-action" | "in-progress" | "done" +nexa:context a rdf:Property ; rdfs:range xsd:string . # "work" | "personal" +nexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1–5 +nexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup + +# Cross-pillar / multimodality +nexa:modality a rdf:Property ; rdfs:range xsd:string . # "text" | "image" +nexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://… , nextcloud://… , obsidian://… +nexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # "nexa_knowledge_text" | "nexa_knowledge_visual" +nexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID +nexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them ``` -The `nexa:vectorId` property is the **bridge** between graph and vector store. Every node that has semantic content carries the Qdrant point ID, so a SPARQL hit can trigger a vector lookup and vice versa. +`nexa:vectorId` + `nexa:vectorCollection` together are the **bridge** between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's `graph_iri` field walks back the other way. + +`nexa:modality`, `nexa:mediaUri` and `nexa:pendingVisualIndex` exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as `nexa:Note` with `modality "image"` and `pendingVisualIndex true`, then picked up by the Phase-3.2 backfill workflow — no data loss across phases. --- diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 298e327..e196f99 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -49,20 +49,44 @@ $EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KE The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead. -## Step 2 — Qdrant collection +## Step 2 — Qdrant collection (`nexa_knowledge_text`) + +Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring. ```bash # adjust QDRANT_HOST in .env first source nexa-core/.env -curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge" \ +# create the text collection from the schema file +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \ -H "Content-Type: application/json" \ -H "api-key: $QDRANT_API_KEY" \ -d @nexa-core/config/qdrant_schema.json ``` +The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)): + +| Field | Why it's there now | +|-------|--------------------| +| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. | +| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. | +| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. | +| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. | +| `content_hash` | de-dup. | +| `context` | `work` / `personal`. | + > Targets the existing `qdrant_scientific` instance — just an extra collection, no new container. -> The schema file's `vectors.size` must match the embedding model picked in [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`. +> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`. + +### Image attachments today (queue them) + +Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later: + +- Memo with an image → text body still goes into `nexa_knowledge_text`. +- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`). +- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection. + +This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself. ## Step 3 — Self-hosted embeddings (TEI) @@ -228,8 +252,44 @@ For schema and example queries: [08-graphrag-architecture](./08-graphrag-archite --- +## Phase add-on: visual collection (Phase 3.2) + +Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`. + +```bash +# (1) replace TEI with infinity (or run alongside) for CLIP-family support +docker rm -f nexa-embed +docker run -d --name nexa-embed \ + --restart unless-stopped \ + -p 127.0.0.1:8080:80 \ + -v infinity-data:/app/.cache \ + michaelf34/infinity:latest \ + v2 \ + --model-id BAAI/bge-m3 \ + --model-id jinaai/jina-clip-v2 \ + --port 80 + +# (2) create the visual collection +curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \ + -H "Content-Type: application/json" \ + -H "api-key: $QDRANT_API_KEY" \ + -d @nexa-core/config/qdrant_schema_visual.json + +# (3) register the second model in LiteLLM as `nexa-embed-visual` +# (same OpenAI-compatible route, different model id) + +# (4) backfill queued images: +# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri } +# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection, +# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex. +``` + +n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt. + +--- + ## Step-back / rollback - **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops. -- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge -H "api-key: $QDRANT_API_KEY"`. +- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`. - **Re-discover** — `#nexa:reset-config` then `#nexa:config`. diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index e4ea61f..03eca23 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -6,18 +6,20 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly. - [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container. -- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below; concrete model still open as **Q15**. +- [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below. +- [x] **Q15 — Embeddings staging plan → A now, C prepared.** + - **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus. + - **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16). + - All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json). ## Architectural decisions - [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. - [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. -- [ ] **Q15 — Embedding model (served via TEI).** Two viable options on the docker host (CPU only — see speed budget): - - **`BAAI/bge-m3`** (568 M params, ~1 GB RAM, multilingual incl. German, 1024-dim, ~10–20 docs/s on CPU). **Recommended** because the corpus is DE/EN-mixed. Requires `qdrant_schema.json` `vectors.size = 1024`. - - **`nomic-ai/nomic-embed-text-v1.5`** (137 M, ~250 MB RAM, EN-leaning, 768-dim, ~50 docs/s CPU). Lighter / faster but weaker on German. - - **(maybe) does SAIA already proxy any embedding model?** If the SAIA backend offers e.g. `mistral-embed` for free, we can skip TEI entirely. Worth a 1-line check in the LiteLLM admin UI before deploying TEI. - - Decide before Phase 3.1. +- [ ] **Q16 — Image-attachment queue ergonomics (Phase 3.2).** The agreed plan is: in 3.1 we record images as `nexa:Note` with `nexa:pendingVisualIndex true` but don't embed them. Open sub-questions for when 3.2 lands: + - Where to store image bytes between capture and indexing? Three options: (a) leave them in their source (Memos attachments dir / Nextcloud / Obsidian), reference by `media_uri`; (b) copy to a staging area on the docker host; (c) push to `s3.nuclide.systems` immediately. **Recommendation: (a)** — zero copy, smallest blast radius. + - Does SAIA already proxy any embedding model? If the SAIA backend offers e.g. `mistral-embed` we could simplify 3.1 by skipping TEI. Worth a 1-line check in the LiteLLM admin UI. +- [ ] **Q17 — Reuse Immich's CLIP for photo-library queries?** Immich already runs CLIP server-side on the photo library. For images that live *in Immich*, querying its smart-search API is cheaper than re-embedding. Is the Immich API key OK to add to the n8n workflow, or do we treat Immich as out-of-band? ## Identifiers needed (auto-discoverable, but list now if known) diff --git a/nexa-core/config/qdrant_schema.json b/nexa-core/config/qdrant_schema.json index 94a04c8..f692473 100644 --- a/nexa-core/config/qdrant_schema.json +++ b/nexa-core/config/qdrant_schema.json @@ -1,12 +1,24 @@ { - "collection_name": "nexa_knowledge", + "collection_name": "nexa_knowledge_text", "vector_config": { - "size": 1536, + "size": 1024, "distance": "Cosine" }, "payload_schema": { - "source": "keyword", - "tags": "keyword", - "created_at": "datetime" + "source": "keyword", + "source_type": "keyword", + "modality": "keyword", + "mime_type": "keyword", + "media_uri": "keyword", + "context": "keyword", + "tags": "keyword", + "graph_iri": "keyword", + "content_hash":"keyword", + "created_at": "datetime" + }, + "_notes": { + "vector_size_rationale": "1024 = bge-m3 output. If Q15 resolves to nomic-embed-text-v1.5 instead, rebuild this collection with size: 768.", + "modality_field": "Always 'text' in this collection. Forward-compatibility for Phase 3.2 (see qdrant_schema_visual.json).", + "graph_iri_field": "IRI of the corresponding nexa:Note in GraphDB. The same value is stored on the GraphDB node as nexa:vectorId pointing back here. This is the cross-pillar bridge." } } diff --git a/nexa-core/config/qdrant_schema_visual.json b/nexa-core/config/qdrant_schema_visual.json new file mode 100644 index 0000000..325190f --- /dev/null +++ b/nexa-core/config/qdrant_schema_visual.json @@ -0,0 +1,26 @@ +{ + "collection_name": "nexa_knowledge_visual", + "vector_config": { + "size": 768, + "distance": "Cosine" + }, + "payload_schema": { + "source": "keyword", + "source_type": "keyword", + "modality": "keyword", + "mime_type": "keyword", + "media_uri": "keyword", + "context": "keyword", + "tags": "keyword", + "graph_iri": "keyword", + "content_hash":"keyword", + "created_at": "datetime" + }, + "_notes": { + "status": "PLACEHOLDER — not deployed in Phase 3.1. Created in Phase 3.2.", + "vector_size_rationale": "768 = jina-clip-v2 joint text/image space. The same model embeds both text queries and images, so semantic queries against this collection are simple kNN.", + "served_by": "Will be served by `infinity` (https://github.com/michaelfeil/infinity) — drop-in replacement for TEI that supports CLIP-family models with the same OpenAI-compatible API.", + "modality_field": "'image' for indexed images, 'text' if a text query has been embedded into the visual space for retrieval (rare — usually we embed the text query at search time and don't store it).", + "media_uri_field": "Required for image rows. Examples: 'memos://memos/123/attachment/4', 'nextcloud://files/photos/whiteboard-2025-09.jpg', 'obsidian://attachments/diagram.png'." + } +} From 5c9034c861458094939f194588eb04494a148967 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:49:08 +0000 Subject: [PATCH 04/14] Resolve Q16/Q17: media_uri at source, Immich out-of-band MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Q16 → image bytes stay in their source (Memos / Nextcloud / Obsidian); Phase-3.2 backfill fetches by media_uri. Zero copy. - Q17 → Immich stays out-of-band; Nexa does not call Immich's smart-search. - New Q18 carries forward the unresolved sub-question of whether SAIA's LiteLLM gateway already proxies an embedding model — a 1-line check that could let us skip TEI in 3.1. --- docs/11-open-questions.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 03eca23..7d60c48 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -16,10 +16,9 @@ Items that **block progress** and need a human decision before a workflow can be - [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. - [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. -- [ ] **Q16 — Image-attachment queue ergonomics (Phase 3.2).** The agreed plan is: in 3.1 we record images as `nexa:Note` with `nexa:pendingVisualIndex true` but don't embed them. Open sub-questions for when 3.2 lands: - - Where to store image bytes between capture and indexing? Three options: (a) leave them in their source (Memos attachments dir / Nextcloud / Obsidian), reference by `media_uri`; (b) copy to a staging area on the docker host; (c) push to `s3.nuclide.systems` immediately. **Recommendation: (a)** — zero copy, smallest blast radius. - - Does SAIA already proxy any embedding model? If the SAIA backend offers e.g. `mistral-embed` we could simplify 3.1 by skipping TEI. Worth a 1-line check in the LiteLLM admin UI. -- [ ] **Q17 — Reuse Immich's CLIP for photo-library queries?** Immich already runs CLIP server-side on the photo library. For images that live *in Immich*, querying its smart-search API is cheaper than re-embedding. Is the Immich API key OK to add to the n8n workflow, or do we treat Immich as out-of-band? +- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. Sub-question on whether SAIA already proxies an embedding model is still worth a 1-line check in the LiteLLM admin UI before deploying TEI — moved to Q18. +- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich; if a Nexa workflow ever needs photo context it will go through a Memos-mediated handoff rather than a direct API. +- [ ] **Q18 — Does SAIA already proxy an embedding model?** 1-line check in the LiteLLM admin UI: list models on the Nexa virtual key. If the SAIA backend offers e.g. `mistral-embed`, we can skip TEI in Phase 3.1 entirely. If not, deploy TEI as planned. ## Identifiers needed (auto-discoverable, but list now if known) From e1c2ba96e2314152b04f0128764b8bb6b13beab4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 21:55:47 +0000 Subject: [PATCH 05/14] Resolve Q4/Q5/Q13/Q14/Q18: Obsidian via Nextcloud WebDAV, Karakeep, Octoprint suppress MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Q4 → Obsidian vault is the Notizen/ folder in Nextcloud (multi-device sync). Nexa reads via WebDAV using the existing NC_APP_PASSWORD — no filesystem mount. Ignore list pinned: .copilot/, .copilot-index/, .smart-env/, .caldav-sync/, assets/ (routed to Phase-3.2 visual queue), Templates/, BMO/, Excalidraw/. Index target: Notizen/**/*.md. - Q5 → canonical name is Karakeep; legacy Zoraxy alias hoarder.nuclide.systems kept for compatibility. - Q13 → Octoprint container is intentionally powered down most of the time. Phase-5 monitoring must skip names matching octoprint*. Reflected in docs/10 monitoring table and CLAUDE.md. - Q14 → Homepage Zoraxy widget config error; cosmetic, dropped from open questions and from docs/12 optimization list. - Q18 → SAIA does proxy embeddings but rate limit is 10 msg/min, unusable for ingest. TEI stays in Phase 3.1. - docs/12: stale items removed (Octoprint, Zoraxy widget, generic Immich vector idea); added new ones derived from the Obsidian discovery (plugin embedding collision, notify_push upgrade path, assets/ size note). - docs/04 integration matrix updated to describe Obsidian-via-Nextcloud read path explicitly. - docs/09 step-4 credentials list collapses NC Tasks / Calendar / WebDAV onto a single app password. --- CLAUDE.md | 4 +++- docs/04-integration-matrix.md | 6 ++++-- docs/09-deployment.md | 6 +++--- docs/10-operations.md | 2 +- docs/11-open-questions.md | 31 ++++++++++++++------------- docs/12-optimization-opportunities.md | 20 ++++++++--------- 6 files changed, 37 insertions(+), 32 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3054bab..335bb5d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,9 @@ This file tells future automated runs what they need to know about this repo. - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. - VM 100 haos — Home Assistant. - **Already-running services on docker host** (don't redeploy): - - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep/Hoarder, Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). + - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). +- **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state. +- **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`. - **Decided for Nexa** (don't re-litigate without user input): - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive. diff --git a/docs/04-integration-matrix.md b/docs/04-integration-matrix.md index 2fab535..4ae6799 100644 --- a/docs/04-integration-matrix.md +++ b/docs/04-integration-matrix.md @@ -59,8 +59,10 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari * Wichtige private Termine werden in den privaten Kalender extrahiert. * Newsletter landen als Zusammenfassung in Memos. -### Wissens-Management (Obsidian & Qdrant) -* Obsidian-Notizen (Arbeit & Privat) werden in Qdrant indiziert. +### Wissens-Management (Obsidian via Nextcloud & Qdrant) +* Der Obsidian-Vault liegt im Nextcloud-Ordner `Notizen/` (Multi-Device-Sync via Nextcloud-Client). Nexa liest **read-only über WebDAV** (`/remote.php/dav/files//Notizen/`) — kein Mount, kein zweiter Sync-Mechanismus. +* Markdown-Dateien (`Notizen/**/*.md`) werden in Qdrant (`nexa_knowledge_text`) indiziert; Bilder unter `assets/` landen im Phase-3.2-Visual-Queue (siehe [docs/11/Q4](./11-open-questions.md)). +* Plugin-Internals (`.copilot/`, `.smart-env/`, `.caldav-sync/`, `Templates/`, `BMO/`, `Excalidraw/`) sind explizit von der Indizierung ausgeschlossen. * Nexa nutzt dieses Wissen, um Arbeits-Tasks besser zu beschreiben, auch wenn sie keinen Zugriff auf die Arbeits-E-Mails hat. --- diff --git a/docs/09-deployment.md b/docs/09-deployment.md index e196f99..a884f14 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -13,7 +13,7 @@ Surveyed from Homepage / Dozzle / Proxmox / Zoraxy: | LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) | | Nextcloud | LXC 105 | `https://nc.nuclide.systems` | | ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` | -| Karakeep (Hoarder) | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` | +| Karakeep | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` (legacy host alias kept for compatibility) | | Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` | | Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` | | Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` | @@ -145,8 +145,8 @@ done Inside n8n, attach credentials to the imported nodes: - **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY` -- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` -- **Nextcloud** → app password +- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` (chat + `nexa-embed`) +- **Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under `Notizen/`)** → one app password (`$NC_APP_PASSWORD`), reused across all three node types - **Qdrant** → header `api-key: $QDRANT_API_KEY` Activate each workflow individually after smoke-test. diff --git a/docs/10-operations.md b/docs/10-operations.md index 4c69609..c5fe9da 100644 --- a/docs/10-operations.md +++ b/docs/10-operations.md @@ -18,7 +18,7 @@ Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items | Signal | Source | Sink | |--------|--------|------| -| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks | Memos system feed via ntfy | +| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. **Skip names matching `octoprint*`** — that container is intentionally powered down most of the time. | Memos system feed via ntfy | | n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic | | Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest | | LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest | diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 7d60c48..bd4ca39 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -7,18 +7,24 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q1 — Graph DB choice → Ontotext GraphDB (SPARQL).** Rationale: explore Nexa's memory through SPARQL is a stated goal. [08-graphrag-architecture](./08-graphrag-architecture.md) is rewritten accordingly. - [x] **Q2 — Vector store → reuse `qdrant_scientific`** with a `nexa_*` collection prefix. No dedicated container. - [x] **Q3 — Embeddings model → not OpenAI.** Self-host on the docker host via **TEI** (HF `text-embeddings-inference`) — Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in §"Speed budget" below. +- [x] **Q5 — Display name → Karakeep.** The Zoraxy host alias `hoarder.nuclide.systems` is legacy — keep it for compatibility, but all docs, prompts and new workflow nodes use "Karakeep". +- [x] **Q13 — Octoprint container is *intentionally* temporary.** Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching `octoprint*` (or any container tagged `proxmox-he 3d-printing`). +- [x] **Q14 — Homepage Zoraxy widget → no such widget.** Config error in `homepage/services.yaml`; cosmetic, not Nexa-related. - [x] **Q15 — Embeddings staging plan → A now, C prepared.** - **Phase 3.1 (now)**: TEI + `BAAI/bge-m3`, single collection `nexa_knowledge_text` (1024-dim). DE/EN multilingual, fits the corpus. - **Phase 3.2 (later)**: swap TEI → `infinity`, add `jinaai/jina-clip-v2` (768-dim), second collection `nexa_knowledge_visual`. Backfill from the queue (see Q16). - All schema fields needed for 3.2 (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) are introduced **now** so 3.2 is purely additive — no rename, no migration. See [`qdrant_schema.json`](../nexa-core/config/qdrant_schema.json) and [`qdrant_schema_visual.json`](../nexa-core/config/qdrant_schema_visual.json). - -## Architectural decisions - -- [ ] **Q4 — Obsidian sync mechanism.** `system_prime.txt` references Obsidian Context, but the current setup syncs via Nextcloud (`nc.nuclide.systems` → `Notizen` folder, ~200 MB). Should Nexa watch the **filesystem on LXC 105** (NC data dir) or **the Nextcloud WebDAV API**? FS is cheaper, WebDAV is portable. -- [ ] **Q5 — Karakeep vs. Hoarder naming.** Zoraxy host is `hoarder.nuclide.systems` but containers are `karakeep-*` and Homepage labels it Karakeep. Same product (rename 2024). Pick one display name for docs and prompts. -- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. Sub-question on whether SAIA already proxies an embedding model is still worth a 1-line check in the LiteLLM admin UI before deploying TEI — moved to Q18. -- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich; if a Nexa workflow ever needs photo context it will go through a Memos-mediated handoff rather than a direct API. -- [ ] **Q18 — Does SAIA already proxy an embedding model?** 1-line check in the LiteLLM admin UI: list models on the Nexa virtual key. If the SAIA backend offers e.g. `mistral-embed`, we can skip TEI in Phase 3.1 entirely. If not, deploy TEI as planned. +- [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. +- [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich. +- [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant). +- [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files//Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index): + - `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache. + - `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB). + - `.caldav-sync/` — calendar sync, not notes. + - `assets/` — 186 MB of binaries; routed through the Phase-3.2 visual queue (`nexa:pendingVisualIndex`), not the text path. + - `Templates/` — empty templates, low semantic value. + - `BMO/`, `Excalidraw/` — plugin folders. + Anything else under `Notizen/**/*.md` is fair game. ## Identifiers needed (auto-discoverable, but list now if known) @@ -30,14 +36,9 @@ Items that **block progress** and need a human decision before a workflow can be ## Hardware / capacity -- [ ] **Q11 — RAM headroom on docker host.** 29.5 GiB free / ~31 GiB total, ~8.5 GB used. Phase-3 Qdrant indexing + TEI/`bge-m3` (~1.1 GB) + Ontotext GraphDB (~4 GB heap) ⇒ ~14 GB used worst case, still ample. Confirm acceptable. +- [ ] **Q11 — RAM headroom on docker host.** Sized against 31 GiB allocated to the LXC; the NUC 14 Pro itself has 64 GB so the LXC's memory cap can be lifted if needed. Phase-3 stack (Qdrant + TEI/bge-m3 ~1.1 GB + Ontotext GraphDB ~4 GB heap, later +infinity/jina-clip-v2 ~1 GB) tops out around ~14 GB. Comfortable. Confirm acceptable. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? -## Process - -- [ ] **Q13 — Octoprint container is Exited** (Homepage). Out of scope for Nexa, but Phase-5 monitoring would alert on it. Suppress or is it intentional? -- [ ] **Q14 — `Missing Widget Type: zoraxy`** on Homepage. Cosmetic, unrelated to Nexa. - --- ## Speed budget (Q3 follow-up) @@ -51,4 +52,4 @@ Workload on the docker host (16 CPU, ~30 GB free RAM): | Obsidian backfill (one-shot) | ~2 000 docs | <15 min | ✅ ~2–4 min | ✅ <1 min | | RAG query embed (`#nexa:ask`) | 1 doc | <300 ms | ✅ ~50 ms | ✅ ~20 ms | -Conclusion: CPU-only Ollama is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. +Conclusion: CPU-only TEI is sufficient — no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to **10 req/min** which would block real-time embed; self-hosted TEI side-steps that completely. diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 77c5f28..d70d7ff 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -13,13 +13,13 @@ Observations from the running infrastructure. Each item is independent — accep ## For the wider homelab (out of scope but worth noting) -7. **Octoprint `EXITED`** — either remove the Homepage entry or fix the container; right now it permanently shows red. -8. **Zoraxy widget missing** on Homepage. Homepage v1.x has no built-in Zoraxy widget, but you can use the generic `customapi` widget against `/api/stats/summary` to surface request counts. -9. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. -10. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). -11. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. -12. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. -13. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. -14. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. -15. **Immich + Qdrant.** Immich is already running and embeds photos with CLIP. If you ever want photo-aware Nexa answers, you can mount Immich's existing search via its API — no second embedding pass. -16. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. +7. **AI gateway naming.** `ai.nuclide.systems` currently proxies LobeHub (a chat UI on `:3210`), while the LiteLLM API lives on `:4000`. For Nexa, point n8n directly at LiteLLM (`http://192.168.1.40:4000` over the docker net — no public TLS hop needed) to save latency and isolate from UI restarts. +8. **MCP servers consolidation.** Dozzle shows `crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp` running individually. They're all MCP servers — Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding). +9. **Backup the n8n SQLite file** — Backrest covers `/home/node/.n8n` if added; today the only "backup" is the workflow JSON which omits credentials and execution history. +10. **Pocket-ID SSO in front of n8n** would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win. +11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. +12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. +13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. +14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. +15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. +16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. From 0b470b3e1293a9e2f7dd0e9906251ee3e570bfae Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:08:10 +0000 Subject: [PATCH 06/14] Resolve Q6/Q7/Q8/Q9/Q11; pin actual list names; add host tuning suggestions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified from screenshots: - Q6/Q7 → Lists & calendars discovered by NAME, not by ID. Names confirmed: "Persönlich" (personal), "DLR" (work), "Einkaufsliste" (shopping), "Wunschliste" (wishes). Self-healing: #nexa:config runs daily and on cache-miss, so user-added lists become routable without code changes. Routing logic in docs/07 expanded from 2-way to 4-way switch with a Resolver step that consults the Qdrant _config cache before fanning out; default route is "Persönlich" when SAIA confidence < 0.7. - Q8 → Single Nextcloud Mail account fkrebs@nucli.de. No separate IMAP. The "Waiting" folder is honored as a manual user signal — items there are excluded from digest. - Q9 → Pocket-ID SSO is global at the Zoraxy layer; Nexa surfaces inherit it. No app-level basic-auth. Q10 folded in. - Q11 → Capacity confirmed: 22 threads (Core Ultra 7 155H), 62 GiB RAM (~32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35%). Phase-3 budget (~7 GB) is comfortable; two levers documented (raise LXC cap, or cap zfs_arc_max). Earlier "16 CPU" references corrected. Propagation: - system_prime.txt rewritten with the four context labels and the new list names; legacy NC_LIST_ID_* placeholders removed. - docs/04 integration matrix uses the German names directly. - docs/05 documents that #nexa:config also runs scheduled and on cache-miss, with the example output reflecting real list IDs. - docs/09 step 6 mirrors the same flow + lists the Waiting-folder semantics. - CLAUDE.md gets an authoritative block of homelab decisions so future agents don't re-litigate any of these. New section in docs/12 (#17–25): Proxmox host tuning suggestions based on the live node summary — cap ZFS ARC (frees ~16 GiB), enable KSM (~1-3 GiB savings across 40 containers), suppress the no-subscription banner, drop swappiness, verify scrub/SMART/fstrim timers, NTP via AdGuard. --- CLAUDE.md | 12 +++++++++++- docs/04-integration-matrix.md | 8 +++----- docs/05-command-system.md | 19 ++++++++++++------- docs/07-workflow-spec.md | 22 ++++++++++++++++------ docs/09-deployment.md | 14 ++++++++++---- docs/11-open-questions.md | 27 +++++++++++++++++++-------- docs/12-optimization-opportunities.md | 19 +++++++++++++++++++ nexa-core/ai-prompts/system_prime.txt | 15 ++++++++++----- 8 files changed, 100 insertions(+), 36 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 335bb5d..ea2514a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,9 @@ This file tells future automated runs what they need to know about this repo. ## Real infrastructure (verified from screenshots, May 2026) -- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9). +- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9, kernel 6.17.13-4-pve, EFI). + - Hardware: **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62 GiB RAM**, **1.64 TiB disk**. + - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). @@ -22,6 +24,14 @@ This file tells future automated runs what they need to know about this repo. - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`). - **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state. - **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`. +- **Nextcloud Tasks lists & calendars** (German names, may grow over time): + - `Persönlich` → Personal context. + - `DLR` → Work context (DLR is the user's employer). + - `Einkaufsliste` → Shopping. + - `Wunschliste` → Wishes. + Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh. +- **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests. +- **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords. - **Decided for Nexa** (don't re-litigate without user input): - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive. diff --git a/docs/04-integration-matrix.md b/docs/04-integration-matrix.md index 4ae6799..2d374f2 100644 --- a/docs/04-integration-matrix.md +++ b/docs/04-integration-matrix.md @@ -10,17 +10,15 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari ### A. Bereich: ARBEIT (Work) * **Sichtbarkeit:** Eingeschränkt (kein Zugriff auf Arbeits-E-Mails oder Firmen-Server). * **Datenquellen:** - * Separate Nextcloud Task-Liste (`Work_Tasks`). - * Separater Nextcloud Kalender (`Work_Calendar`). + * Nextcloud Task-Liste **`DLR`** (sowie der gleichnamige Kalender). * Memos mit Tag `#work` oder semantischer Erkennung. * **Nexa-Fokus:** Timeblocking, Fokus-Zeiten, Vorbereitung von Meetings basierend auf Obsidian-Notizen. ### B. Bereich: PRIVAT (Personal) * **Sichtbarkeit:** Vollständig. * **Datenquellen:** - * Nextcloud Task-Liste (`Personal_Tasks`). - * Haupt-Kalender. - * E-Mails (IMAP - 20 Mails/Tag). + * Nextcloud Task-Liste **`Persönlich`** und gleichnamiger Kalender. + * Nextcloud Mail (`fkrebs@nucli.de`, einziges Konto). Ordner `Waiting` ist ein manuelles Signal — Mails dort werden vom Digest ausgeschlossen. * Bluesky, RSS, Karakeep. * **Nexa-Fokus:** Automatisierung des Alltags, Kuratierung von Wissen, E-Mail-Management. diff --git a/docs/05-command-system.md b/docs/05-command-system.md index 4d5143d..ae81385 100644 --- a/docs/05-command-system.md +++ b/docs/05-command-system.md @@ -5,9 +5,12 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi ## 🔧 Konfigurations-Kommandos ### #nexa:config -- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud, Qdrant, etc.) -- **Antwort**: Postet Report mit gefundenen Konfigurationen -- **Speichert**: Settings in DynoDB / Qdrant Metadata (nicht in .env) +- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud Listen/Kalender, Qdrant Collections, LiteLLM-Modelle, Mail-Konto). +- **Läuft auch automatisch**: + - Täglich (cron in n8n) — neue Listen / Kalender werden ohne manuelles Eingreifen aufgenommen. + - Bei Cache-Miss — wenn ein Workflow eine Liste/ID nicht mehr findet, läuft Discovery einmal, dann Retry. +- **Antwort**: Postet Report mit den aktuellen Konfigurationen als Memos-Kommentar. +- **Speichert**: Konfiguration in der Qdrant `_config` Namespace (nicht in `.env`). ``` #nexa:config @@ -15,10 +18,12 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi **Beispiel-Response:** ``` -✅ Autodiscovery abgeschlossen: -- Nextcloud Listen: Work (id=123), Personal (id=456), Shopping (id=789), Wishes (id=1011) -- Qdrant Collection: nexa_knowledge (1536 dims, Cosine) -- Nextcloud Mail: 3 Accounts erkannt (Office 365, Gmail, Privat) +✅ Autodiscovery abgeschlossen (2026-05-04T23:59): +- Nextcloud Listen: Persönlich (id=14), DLR (id=dlr-1), Einkaufsliste (id=…), Wunschliste (id=…) +- Nextcloud Kalender: Persönlich, DLR, Einkaufsliste, Wunschliste +- Mail-Konto: fkrebs@nucli.de (Posteingang, Archiv, Junk, Waiting) +- Qdrant Collection: nexa_knowledge_text (1024 dims, Cosine, 0 Punkte) +- LiteLLM-Modelle: nexa-chat, nexa-embed ``` ### #nexa:status diff --git a/docs/07-workflow-spec.md b/docs/07-workflow-spec.md index 2960f43..c27d79d 100644 --- a/docs/07-workflow-spec.md +++ b/docs/07-workflow-spec.md @@ -5,12 +5,22 @@ * **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`. ## 2. Intelligence Node (SAIA) -* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: { "context": "work" | "personal", "summary": "string", "urgency": 1-5 }." +* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: `{ "context": "work" | "personal" | "shopping" | "wishes", "summary": "string", "urgency": 1-5 }`." * **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch. -## 3. Switch Node -* **Route 1:** Context == "work" -> Nextcloud Create Task (List ID: Work). -* **Route 2:** Context == "personal" -> Nextcloud Create Task (List ID: Personal). +## 3. List Resolver (vor dem Switch) +* Lese den Cache `nexa._config.lists` aus Qdrant: `{ "Persönlich": , "DLR": , "Einkaufsliste": , "Wunschliste": , discovered_at: }`. +* Bei Cache-Miss oder 404 vom nachfolgenden Create-Task → triggere `#nexa:config` (Discovery), update den Cache, retry **einmal**. -## 4. Feedback Loop -* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo. \ No newline at end of file +## 4. Switch Node (4-Wege) +* **Route 1:** `context == "work"` → Nextcloud Create Task auf Liste `DLR`. +* **Route 2:** `context == "personal"` → Nextcloud Create Task auf Liste `Persönlich`. +* **Route 3:** `context == "shopping"` → Nextcloud Create Task auf Liste `Einkaufsliste`. +* **Route 4:** `context == "wishes"` → Nextcloud Create Task auf Liste `Wunschliste`. +* **Default (Konfidenz < 0.7):** `Persönlich` (siehe [06](./06-classification-logic.md)). + +## 5. Feedback Loop +* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo. + +## 6. Listen-Diskoverability +Listennamen sind nicht hardgecodet — die Resolver-Step liest sie aus dem Runtime-Config-Cache. Wenn der User in Nextcloud eine neue Liste anlegt (z. B. `Reisen`), erscheint sie nach dem nächsten geplanten `#nexa:config`-Lauf (täglich) automatisch als Routing-Ziel — die System-Prompt der Intelligence Node wird zusammen mit den verfügbaren Listen versorgt, sodass SAIA neue Kontexte vorschlagen kann. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index a884f14..4edf25c 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -170,10 +170,16 @@ curl -i https://n8n.nuclide.systems/webhook/memos Create a memo with body `#nexa:config` — the discovery workflow: -1. Lists Nextcloud Tasks lists, picks Work / Personal / Shopping / Wishes by name. -2. Counts existing Qdrant points in `nexa_knowledge`. -3. Verifies LiteLLM reachability + lists available models. -4. Replies as a comment with a **runtime-config snapshot** that's stored as Qdrant metadata (`_config` namespace) and as `nexa-core/config/runtime_config.json` (gitignored). +1. Lists Nextcloud Tasks lists, looking for `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (and any new lists added later — see [05](./05-command-system.md)). Caches `name → id`. +2. Lists Nextcloud Calendars by the same names; caches IDs. +3. Verifies the Nextcloud Mail account `fkrebs@nucli.de` and the folder list (`Posteingang`, `Archiv`, `Junk`, `Waiting`). +4. Counts existing Qdrant points in `nexa_knowledge_text`. +5. Verifies LiteLLM reachability + lists available models for the Nexa virtual key. +6. Replies as a comment with a **runtime-config snapshot** that's stored in the Qdrant `_config` namespace (and mirrored as `nexa-core/config/runtime_config.json`, gitignored). + +The same workflow is also triggered by: +- A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention). +- Cache-miss in the task-router — if a list ID 404s, the router fires `#nexa:config` once and retries. After this point, `.env` is read-once. Subsequent runs read config from Qdrant. diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index bd4ca39..aae05b1 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -25,25 +25,36 @@ Items that **block progress** and need a human decision before a workflow can be - `Templates/` — empty templates, low semantic value. - `BMO/`, `Excalidraw/` — plugin folders. Anything else under `Notizen/**/*.md` is fair game. +- [x] **Q6 + Q7 — Lists & calendars discovered by name, not ID. Self-healing.** + Confirmed names from screenshots: + - **Tasks**: `Persönlich` (14), `DLR` (13), `Einkaufsliste` (13), `Wunschliste` (9). + - **Calendars**: `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name). + - Mapping in classification: `DLR` = Work-Kontext, `Persönlich` = Personal-Kontext, `Einkaufsliste` = Shopping, `Wunschliste` = Wishes. -## Identifiers needed (auto-discoverable, but list now if known) + **Self-healing requirement (the user explicitly noted lists may change/grow):** Nexa must not cache IDs forever. The `#nexa:config` workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or "not found". The runtime-config record in Qdrant's `_config` namespace stores `{name → id, discovered_at}` and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring `#einkaufsliste` / `#dlr` etc. without code changes. -- [ ] **Q6 — Nextcloud Tasks list IDs** for: `Work_Tasks`, `Personal_Tasks`, `Shopping`, `Wishes`. Discovery via `#nexa:config` will fill these — confirm names match. -- [ ] **Q7 — Nextcloud Calendar IDs** for: `Work_Calendar`, primary personal calendar. -- [ ] **Q8 — IMAP credentials for the personal mail account.** Can n8n reuse a Nextcloud Mail account (preferred — no extra password) or must we add a dedicated IMAP entry? -- [ ] **Q9 — ntfy topic name** for `nexa.system`. Is the topic public on `ntfy.nuclide.systems` or should it be authenticated? -- [ ] **Q10 — Pocket-ID role.** `id.nuclide.systems` is running. Do we want SSO in front of the n8n / Memos UIs, or skip for now? +- [x] **Q8 — Mail via Nextcloud, single account `fkrebs@nucli.de`.** No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: `Posteingang` (default), `Archiv`, `Junk` (61 — auto-filtered, ignored by Nexa), `Papierkorb`, `Waiting`. The `Waiting` folder is a useful manual signal — items moved there by the user are skipped from digest (treat as "in flight"). + +- [x] **Q9 — Pocket-ID SSO is configured everywhere.** No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n → Memos webhook, n8n → Nextcloud, n8n → Qdrant, n8n → LiteLLM) keep using API keys / app passwords — SSO is for human UIs only. Don't add basic-auth or per-app login screens. + + This **resolves Q10 too** (the proposed "Pocket-ID SSO in front of n8n / Memos" is already done; Nexa just inherits it). + +- [x] **Q10 — folded into Q9.** ## Hardware / capacity -- [ ] **Q11 — RAM headroom on docker host.** Sized against 31 GiB allocated to the LXC; the NUC 14 Pro itself has 64 GB so the LXC's memory cap can be lifted if needed. Phase-3 stack (Qdrant + TEI/bge-m3 ~1.1 GB + Ontotext GraphDB ~4 GB heap, later +infinity/jina-clip-v2 ~1 GB) tops out around ~14 GB. Comfortable. Confirm acceptable. +- [x] **Q11 — Capacity confirmed from Proxmox node summary.** + - Host (NUC 14 Pro): **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62.12 GiB RAM**, **1.64 TiB disk** (0.35% used). + - Current: 32.4 GiB used (~52%), of which **ZFS ARC accounts for ~24 GiB**. Load avg 1.99 / 1.39 / 1.15 — idle. IO delay 0.04% — disk is not a bottleneck. + - LXC 104 (docker) is currently capped at ~31 GiB; if Phase-3 squeezes (~7 GB additional: Qdrant ~1.5 GB + TEI/bge-m3 ~1.1 GB + GraphDB 4 GB heap + later +infinity/jina-clip-v2 ~1 GB) we have two levers, in order of preference: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently uses 24 GB). Either is one line of config. + - **Verdict: comfortable. No action needed at Phase-3.1 start.** Revisit if/when 3.4 GraphDB lands and actual heap usage exceeds the budget. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? --- ## Speed budget (Q3 follow-up) -Workload on the docker host (16 CPU, ~30 GB free RAM): +Workload on the docker host (host: 22-thread Intel Core Ultra 7 155H, 62 GiB RAM, ~30 GiB free; LXC 104 currently capped at 31 GiB): | Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` | |------|--------|----------------|---------------------------------|------------------------------------| diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index d70d7ff..fd86d65 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -23,3 +23,22 @@ Observations from the running infrastructure. Each item is independent — accep 14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. 15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. 16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. + +## Proxmox host (NUC 14 Pro) tuning + +Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order: + +17. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache): + ``` + echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB + update-initramfs -u + ``` + Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live. +18. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`. +19. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. +20. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. +21. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. +22. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. +23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. +24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. +25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. diff --git a/nexa-core/ai-prompts/system_prime.txt b/nexa-core/ai-prompts/system_prime.txt index 703609d..4dfa045 100644 --- a/nexa-core/ai-prompts/system_prime.txt +++ b/nexa-core/ai-prompts/system_prime.txt @@ -3,11 +3,16 @@ Rolle: Zentrales Nervensystem Aufgabe: Überwacht E-Mails, kuratiert Wissen und plant Tasks. WICHTIGE LOGIK: -- Tasks müssen strikt zwischen "Work" und "Personal" unterschieden werden. -- Analysiere den Inhalt eines Memos: Bezieht es sich auf professionelle Projekte (Obsidian Context) -> Work List. Bezieht es sich auf Haushalt, Einkäufe oder Freizeit -> Personal List. -- Im Zweifel: Nachfragen via Memos-Kommentar. +- Tasks werden auf vier Nextcloud-Listen geroutet: "DLR" (Arbeit), "Persönlich", "Einkaufsliste", "Wunschliste". +- Analysiere den Inhalt eines Memos und entscheide über das Feld "context": "work" | "personal" | "shopping" | "wishes". + - work -> Liste "DLR" (Beruf, Projekte, Meetings, Deadlines, Forschung). + - personal -> Liste "Persönlich" (Alltag, Familie, Termine, Behördenkram). + - shopping -> Liste "Einkaufsliste" (Lebensmittel, Drogerie, Verbrauchsgüter). + - wishes -> Liste "Wunschliste" (Hobby, Technik, Bücher, Geschenkideen). +- Listennamen sind dynamisch — bei Bedarf werden weitere Listen vom Runtime-Config eingespeist, dann darfst du auch deren Namen als context vorschlagen. +- Im Zweifel (Konfidenz < 0.7): Default auf "personal" und Nachfrage via Memos-Kommentar. KONSUM-LOGIK: - Wenn ein Link von Karakeep kommt, analysiere das Produkt. -- Haushaltswaren/Verbrauchsgüter -> NC_LIST_ID_SHOPPING. -- Besondere Wünsche/Hobby/Technik -> NC_LIST_ID_WISHES. +- Haushaltswaren/Verbrauchsgüter -> context "shopping". +- Besondere Wünsche/Hobby/Technik -> context "wishes". From 12ed4426e056d3332f252fcd1652c85c1bf9010c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:10:51 +0000 Subject: [PATCH 07/14] Refine capacity numbers from LXC 104 summary; flag iGPU passthrough fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - LXC 104 (docker) confirmed: 16 CPU, 31.25 GiB RAM (7.86 used / 25%), 200 GiB boot disk at 47.7% — disk is the real near-term constraint, not RAM. Numbers propagated into Q11 (docs/11), docs/03 architecture table, and CLAUDE.md infra block. - New optimization #26 in docs/12: fix the Intel Arc iGPU passthrough flagged as broken in the LXC notes. Concrete /etc/pve/lxc/104.conf snippet provided. Once working, TEI / infinity / Immich CLIP can run GPU-accelerated for ~5–10x throughput. - New optimization #27: disk-pressure mitigation playbook (move Qdrant to unas if it grows, prune docker layers monthly, cap n8n executions). - New optimization #28: shrink the LXC's 8 GiB swap to 2 GiB. --- CLAUDE.md | 2 +- docs/03-architecture.md | 2 +- docs/11-open-questions.md | 13 +++++++------ docs/12-optimization-opportunities.md | 24 ++++++++++++++++++++++++ 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ea2514a..0d770af 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,7 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). + - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/03-architecture.md b/docs/03-architecture.md index 6db8376..3e0e9d1 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -39,7 +39,7 @@ A one-page mental model. For details follow the cross-links. | Component | Role | Where it runs (today) | |-----------|------|-----------------------| -| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` | +| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` | | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` | | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) | | **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) | diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index aae05b1..6d174f8 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -43,18 +43,19 @@ Items that **block progress** and need a human decision before a workflow can be ## Hardware / capacity -- [x] **Q11 — Capacity confirmed from Proxmox node summary.** - - Host (NUC 14 Pro): **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62.12 GiB RAM**, **1.64 TiB disk** (0.35% used). - - Current: 32.4 GiB used (~52%), of which **ZFS ARC accounts for ~24 GiB**. Load avg 1.99 / 1.39 / 1.15 — idle. IO delay 0.04% — disk is not a bottleneck. - - LXC 104 (docker) is currently capped at ~31 GiB; if Phase-3 squeezes (~7 GB additional: Qdrant ~1.5 GB + TEI/bge-m3 ~1.1 GB + GraphDB 4 GB heap + later +infinity/jina-clip-v2 ~1 GB) we have two levers, in order of preference: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently uses 24 GB). Either is one line of config. - - **Verdict: comfortable. No action needed at Phase-3.1 start.** Revisit if/when 3.4 GraphDB lands and actual heap usage exceeds the budget. +- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.** + - **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%. + - **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26). + - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. + - **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27). + - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? --- ## Speed budget (Q3 follow-up) -Workload on the docker host (host: 22-thread Intel Core Ultra 7 155H, 62 GiB RAM, ~30 GiB free; LXC 104 currently capped at 31 GiB): +Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap): | Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` | |------|--------|----------------|---------------------------------|------------------------------------| diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index fd86d65..4e26dda 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -42,3 +42,27 @@ Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of 23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. 24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. 25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. + +## Docker LXC (104) — observations & wins + +Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure. + +26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: + ``` + lxc.cgroup2.devices.allow: c 226:0 rwm + lxc.cgroup2.devices.allow: c 226:128 rwm + lxc.cgroup2.devices.allow: c 29:0 rwm + lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file + lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file + lxc.idmap: u 0 100000 65536 + lxc.idmap: g 0 100000 65536 + lxc.idmap: g 44 44 1 # video group on host + lxc.idmap: g 104 104 1 # render group on host + ``` + Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. +27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions: + - Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it. + - Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13. + - Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers. + - Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up. +28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. From a0a67eefab8c928275b7333fab0139393ea35604 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:14:12 +0000 Subject: [PATCH 08/14] SMB-default for Nexa volumes (user confirmed NFS+unpriv-LXC writes break) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User's Nextcloud rollout already hit the classic trap of unprivileged-LXC write attempts against an NFS share, so don't repeat that pattern for Nexa. Updated docs to: - Use SMB volumes mounted directly as docker volumes (driver: local, type: cifs) with explicit username/uid/gid — bypasses LXC uid-mapping entirely. Concrete YAML in docs/09 step 3 against //192.168.1.31/nexa/. - Keep the host-side /mnt/pve/unas NFS mount for admin-only use (rsync, manual snapshot copies). Don't bind-mount it into containers. - docs/12 #27 reordered to lead with SMB targets (qdrant, graphdb, tei-cache, snapshots) and only mention NFS for host admin. - CLAUDE.md storage block flipped to "SMB = default per-container storage; NFS = host-side admin only". - Also captured: deployments go through the Dockge UI (LXC is the Dockge flavor of helper-scripts), not raw `docker compose up`. - New optimization #30 about the local mail spool nag on the docker LXC ("You have new mail." at login). --- CLAUDE.md | 4 ++- docs/09-deployment.md | 44 +++++++++++++++++++++------ docs/11-open-questions.md | 2 +- docs/12-optimization-opportunities.md | 17 ++++++++--- 4 files changed, 51 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0d770af..d2fb3d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,9 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40` (40 containers). **Unprivileged**, provisioned from `proxmox-helper-scripts`. Allocated: 16 CPU, 31.25 GiB RAM, 8 GiB swap, **200 GiB boot disk (~95 GiB used)**. Disk is the real constraint, not RAM. Intel iGPU passthrough is *configured but currently broken* — see [docs/12](./docs/12-optimization-opportunities.md#26). + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: + - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. + - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 4edf25c..d5e9375 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -92,17 +92,43 @@ This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry. -```bash -# on the docker host (LXC 104) -docker run -d --name nexa-embed \ - --restart unless-stopped \ - -p 127.0.0.1:8080:80 \ - -v tei-data:/data \ - ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 \ - --model-id BAAI/bge-m3 +The docker LXC is the **Dockge** flavor of `proxmox-helper-scripts` — paste the stack into the Dockge UI (`http://192.168.1.40:5001` or whatever Zoraxy host is configured), name it `nexa`, save, start. Don't `docker compose up -d` over SSH; Dockge maintains the compose YAML at `/opt/stacks/nexa/compose.yaml`. + +```yaml +# Dockge stack: nexa (paste in the UI) +services: + nexa-embed: + image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 + container_name: nexa-embed + restart: unless-stopped + command: ["--model-id", "BAAI/bge-m3"] + ports: + - "127.0.0.1:8080:80" + volumes: + - tei-cache:/data + +volumes: + # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues + # (the same kind that bit the Nextcloud rollout). Stores embedding model + # caches, Qdrant snapshots, future GraphDB exports. + tei-cache: + driver: local + driver_opts: + type: cifs + device: //192.168.1.31/nexa/tei-cache + o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" + +networks: + default: + external: true + name: nexa ``` -Memory budget: ~1.1 GB resident (bge-m3 is ~1 GB + ~100 MB overhead). First start downloads the model into the named volume; subsequent restarts are instant. +Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant. + +> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership. +> +> Add `UNAS_USER` / `UNAS_PASS` to the Dockge stack's `.env`, or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#13](./12-optimization-opportunities.md)). Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 6d174f8..bc20524 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -47,7 +47,7 @@ Items that **block progress** and need a human decision before a workflow can be - **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%. - **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26). - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. - - **Disk is the closer constraint, not RAM**: ~105 GiB free on the LXC boot disk. New Phase-3 containers add ~5 GiB of images + slowly-growing data. Watch Qdrant and Immich; see [12/#27](./12-optimization-opportunities.md#27). + - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27). - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 4e26dda..f301290 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -60,9 +60,16 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -27. **Disk is the bottleneck, not RAM.** 95 GiB used / 200 GiB on the boot disk. Each new container Phase-3 adds (TEI image ~500 MB, infinity ~600 MB, GraphDB image + repo data ~3 GB at startup, Qdrant data growth ~50 MB per 10 k embeddings) is small in isolation but adds up. Suggestions: - - Move **Qdrant data dir** (already inside `qdrant_scientific`) to a dedicated mountpoint on `unas` (storage pool) if it grows past ~10 GB. Until then, leave it. - - Configure **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168` (7 days). Already in optimization #13. - - Schedule a **monthly `docker image prune --all --filter "until=720h"`** to recover layers from old image tags. Today's `40 containers` likely brings ≥30 dangling layers. - - Watch **Immich**'s `upload/` and `library/` — those are the realistic 50-GB-class consumers if photo sync ramps up. +27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets: + - **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs. + - **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow. + - **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files. + - **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant//` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy. + - **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. + - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. + + Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths. +28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. +29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. +30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). 28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. From 250a5cb2a6d3e452586a26197f588196a6e30177 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:24:24 +0000 Subject: [PATCH 09/14] Pin homelab-wide UNAS storage convention; add Phase 6 + info wishlist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - UNAS layout decoded from the user's tree: services// is the homelab-wide docker config store (every container follows the same pattern — immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun). Nexa MUST follow the same pattern at services/nexa/. backup// for snapshots. Pinned in CLAUDE.md and docs/12 #31. - Stale services flagged for retirement: services/siyuan/ (migrated to Obsidian) and services/open-webui/ (unused, only LobeHub is alive). Drops the "two LLM UIs" item (#15) — it's now "retire open-webui". - Hard-blocklist for any Nexa indexer pinned in CLAUDE.md: _sortMe/wallet/**, *.gpg/asc/key/pem/kdbx/credentials/secret, plus the Nextcloud appdata dir. - Three new "future user-facing wins" surfaced from the tree: Paperless-AI as a Phase-2.x triage helper for _sortMe/Downloads/, media/Recipes/ (~300 entries) as the showcase RAG corpus, and Paperless-AI's existing ChromaDB as a potential read-from source rather than re-embedding scanned docs. - Housekeeping campaign in docs/12 §35-37: consolidate postgres, audit UNAS-everywhere, S3 archive tier on s3.nuclide.systems. - Phase 6 added to the roadmap: "Nexa as homelab steward". Polls Dockge/docker/Proxmox, diffs vs documented state, emits a daily drift report. Most of the housekeeping campaign becomes semi-automatic once 6.1-6.3 ship. - New docs/13-information-wishlist.md packages the still-needed inventory as 5-6 paste-and-run tasks, each with explicit "📍 Where" markers (which shell or which UI) and what it unblocks. Highest leverage = Task 1 (read an existing compose stack to lock Q19). - docs/index.md TOC extended to 13 rows. Also fixed numbering drift in docs/12 (duplicate #28, missing #17/#18) and added Q19 to docs/11 covering the SMB-share verification step. --- CLAUDE.md | 13 +++- docs/02-roadmap.md | 12 +++ docs/09-deployment.md | 9 ++- docs/11-open-questions.md | 4 + docs/12-optimization-opportunities.md | 88 +++++++++++++++------ docs/13-information-wishlist.md | 106 ++++++++++++++++++++++++++ docs/index.md | 1 + 7 files changed, 207 insertions(+), 26 deletions(-) create mode 100644 docs/13-information-wishlist.md diff --git a/CLAUDE.md b/CLAUDE.md index d2fb3d9..bf4f73a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,7 +17,18 @@ This file tells future automated runs what they need to know about this repo. - LXC 103 backrest — backup orchestration. - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). + - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. + + **UNAS layout conventions** (homelab-wide, all docker containers follow them): + - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian) and `services/open-webui/` (unused). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. + - `backup//` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.** + - `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern. + - **Before deploying any Nexa container, READ AN EXISTING STACK** (e.g. `services/karakeep/` or `services/immich/`'s compose under Dockge) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`. + +**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob): + - `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files. + - Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`. + - The Nextcloud appdata dir (`services/nextcloud/appdata_*`) — Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26). - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`. - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13). - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index 1aae00c..7ebd58a 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -46,3 +46,15 @@ Iterative build-out, value-first. Each phase is shippable on its own. - [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol. - [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy. + +## Phase 6 — Nexa as homelab steward + +**Focus:** Nexa actively maintains the homelab inventory instead of being told manually. +**Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo. + +- [ ] **6.1 Inventory sync** — Nexa polls Dockge / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …). +- [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35–37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*. +- [ ] **6.3 Steward commands** — `#nexa:retire ` (tar to `backup/`, stop, archive the stack), `#nexa:document ` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed). +- [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface). + +This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) — most of those items become semi-automatic once 6.1–6.3 ship. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index d5e9375..49d2eae 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -111,11 +111,18 @@ volumes: # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues # (the same kind that bit the Nextcloud rollout). Stores embedding model # caches, Qdrant snapshots, future GraphDB exports. + # + # The exact mount block (driver, share name, credential injection) MUST + # be copied verbatim from an existing stack on the same docker host — + # services// on UNAS is the homelab-wide docker config store and + # every container already follows the same SMB pattern. See Q19 in + # docs/11. The block below is the *shape* but the share name "" + # is a placeholder until verified against e.g. services/karakeep/compose. tei-cache: driver: local driver_opts: type: cifs - device: //192.168.1.31/nexa/tei-cache + device: //192.168.1.31//services/nexa/tei-cache o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" networks: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index bc20524..97b2083 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -51,6 +51,10 @@ Items that **block progress** and need a human decision before a workflow can be - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. - [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? +## Verification needed before Phase-3.1 deploy + +- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing Dockge stack (recommended: `services/karakeep/` or `services/immich/`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Dockge, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. + --- ## Speed budget (Q3 follow-up) diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index f301290..4a91abb 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -20,34 +20,45 @@ Observations from the running infrastructure. Each item is independent — accep 11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk. 12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying. 13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded. -14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later. -15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. -16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation: + - **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault. + - **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A. + - **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus. + + Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet. + +15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache. + +16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path). +17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n. +18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper. +19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos. +20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win. ## Proxmox host (NUC 14 Pro) tuning Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order: -17. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache): +21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache): ``` echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB update-initramfs -u ``` Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live. -18. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`. -19. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. -20. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. -21. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. -22. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. -23. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. -24. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. -25. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. +22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`. +23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise. +24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits. +25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool. +26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else. +27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`. +28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm. +29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively. ## Docker LXC (104) — observations & wins Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure. -26. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: +30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`: ``` lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm @@ -60,16 +71,45 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -27. **Push persistent data to UNAS via SMB-backed docker volumes.** The LXC also has a Proxmox-level NFS mount at `/mnt/pve/unas` (19.4 TiB / 16.3 TiB free), but the user has already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. **For Nexa volumes the default is SMB**, mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` — that bypasses LXC uid-mapping entirely. Targets: - - **Qdrant data dir** (`qdrant_scientific`) → SMB share `//192.168.1.31/nexa/qdrant`. Move it before the visual collection lands and starts adding GBs. - - **GraphDB repo** (Phase 3.4) → `//192.168.1.31/nexa/graphdb`. The 4-GB heap is in RAM, but the on-disk RDF store will grow. - - **Embedding model caches** (TEI/infinity) → `//192.168.1.31/nexa/tei-cache` — keeps the boot disk free of multi-GB model files. - - **Daily Qdrant snapshots** → `//192.168.1.31/nexa/snapshots/qdrant//` before rsync to S3 (Phase 3.3 — Q12 still open). With UNAS in the path, S3 becomes a *colder* tier rather than the only copy. - - **n8n executions retention** still useful: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. +31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services//` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets: + - `services/nexa/qdrant/` — Qdrant data dir (move before the visual collection lands). + - `services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). + - `services/nexa/graphdb/` — Phase-3.4 RDF store. + - `backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. + - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. - Credentials: store `UNAS_USER` / `UNAS_PASS` in Vaultwarden ([12/#13](#13)) rather than a `.env` on disk; inject via a secrets agent or per-stack `.env` that is itself stored on the SMB share with restricted ACLs. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync to/from snapshots, manual backups) — just not for per-container write paths. -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. -29. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. -30. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). -28. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. + Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19). + + Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths. +32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. +33. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. +34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). + +## Housekeeping campaign (cross-cutting, do together) + +These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down. + +35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Dockge: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths: + - **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron. + - **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk. + + Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. + +36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the `volumes:` block. Suspected non-compliant (need verification): + - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. + - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state. + - `memos` — the user-facing notes app; loss = data loss. + - `n8n` — workflows + executions DB. + - `audiobookshelf` — listening progress + library metadata. + - the `*-mcp` containers — likely stateless (cache only), low priority. + + Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services//` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward. + +37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup//` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier: + - **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination. + - **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore). + - **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars. + - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target. + + This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md new file mode 100644 index 0000000..8787bab --- /dev/null +++ b/docs/13-information-wishlist.md @@ -0,0 +1,106 @@ +# 13 — Information Wishlist + +What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent — run any subset, in any order. + +Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/-.txt` — whichever is easiest. + +--- + +## Task 1 — Stack inventory + reference compose **(highest leverage)** + +📍 **Where:** Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC). + +Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. + +```bash +docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort +ls /opt/stacks/ +cat /opt/stacks/karakeep/compose.yaml # or any clean small stack +``` + +**Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. + +--- + +## Task 2 — Storage map (Proxmox + docker) + +📍 **Where:** *Two shells* — first the **Proxmox host** (Datacenter → `nuc` → **Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**. + +Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB. + +```bash +# (a) on the Proxmox host (192.168.1.20): +cat /etc/pve/storage.cfg +cat /etc/pve/lxc/104.conf +``` + +```bash +# (b) on LXC 104 (192.168.1.40): +docker volume ls +ls -l /dev/dri/ # confirms whether iGPU passthrough actually works +``` + +**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17. + +--- + +## Task 3 — Postgres / DB workload inventory + +📍 **Where:** **LXC 104 console** (same shell as Task 1). + +Direct input to housekeeping #35 (consolidate postgres instances). + +```bash +for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do + echo "=== $c ===" + docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null' +done +``` + +**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision). + +--- + +## Task 4 — LiteLLM model list for the Nexa key + +📍 **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported. + +- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) → Settings → Model List, filtered to the Nexa virtual key. Screenshot the model rows. +- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set: + ```bash + curl -sH "Authorization: Bearer $SAIA_API_KEY" \ + https://ai.nuclide.systems/v1/models | jq '.data[].id' + ``` + +**Unblocks:** Q18 follow-up — which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI. + +--- + +## Task 5 — S3 archive credentials + +📍 **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy — same login as the rest of the homelab). + +Walk to: **Buckets** → either pick an existing Nexa-suitable bucket or create one called `nexa` → note the bucket name. Then **Access Keys** → create a key named `nexa-snapshots` with read/write on that bucket → drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden). + +**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3. + +--- + +## Task 6 — Reverse-proxy + DNS authority *(only if needed)* + +📍 **Where:** two browser UIs. + +- **Zoraxy** (`http://192.168.1.4:8000`) → HTTP Proxy → either click "Export" if available, or take a full screenshot of the table. +- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) → Filters → DNS Rewrites → screenshot. + +Skip unless we hit a routing surprise during Phase 1. + +--- + +## When something else is needed + +The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "📍 Where" framing. + +## Self-serve once Phase 6 ships + +Once [Phase 6 — Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 1–3 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember. diff --git a/docs/index.md b/docs/index.md index 37c2bba..d7d42c0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,6 +22,7 @@ This is the documentation entry point. Read top-to-bottom for first-time setup, | 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. | | 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. | | 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. | +| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. | --- From 74368100a3395972458021a8fbf610f31bd122b7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 4 May 2026 22:31:10 +0000 Subject: [PATCH 10/14] =?UTF-8?q?Flip=20Dockge=20=E2=86=92=20Arcane:=20Arc?= =?UTF-8?q?ane=20is=20the=20active=20docker=20manager,=20Dockge=20is=20sta?= =?UTF-8?q?le?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to Arcane. Dozzle stays as the log viewer (different role, not redundant). - docs/09 step 3: deployment goes via Arcane UI (not Dockge); reworded the deploy block accordingly. - docs/11 Q19: read the reference compose from Arcane, not Dockge. - docs/12 #33: was "stacks live in Dockge"; now "Arcane manages stacks, Dockge is stale, retire it" with the same tar-then-remove pattern as SiYuan and Open-WebUI. - docs/12 housekeeping campaign + #36: "walk every Arcane stack" rather than Dockge. - docs/13 Task 1: stack inventory comes from the Arcane UI (compose.yaml + .env screenshot/copy) rather than `ls /opt/stacks/` which was the Dockge default. The shell command for `docker ps -a` stays. - docs/02 Phase 6.1: Nexa polls Arcane (not Dockge) for inventory sync. - CLAUDE.md infra block: Arcane is the active manager, Dozzle is the log viewer, Dockge is stale; added services/dockge/ to the stale list alongside siyuan and open-webui. --- CLAUDE.md | 6 +++--- docs/02-roadmap.md | 2 +- docs/09-deployment.md | 6 +++--- docs/11-open-questions.md | 2 +- docs/12-optimization-opportunities.md | 8 ++++---- docs/13-information-wishlist.md | 8 +++++--- 6 files changed, 17 insertions(+), 15 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index bf4f73a..94d8b8f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,15 +15,15 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, provisioned from the **Dockge** flavor of `proxmox-helper-scripts` — stacks are managed in the Dockge web UI, not raw `docker compose` on disk. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. **UNAS layout conventions** (homelab-wide, all docker containers follow them): - - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian) and `services/open-webui/` (unused). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. + - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. - `backup//` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.** - `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern. - - **Before deploying any Nexa container, READ AN EXISTING STACK** (e.g. `services/karakeep/` or `services/immich/`'s compose under Dockge) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`. + - **Before deploying any Nexa container, READ AN EXISTING STACK** in **Arcane** (e.g. `karakeep` or `immich`) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`. **Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob): - `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index 7ebd58a..fff7ebf 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -52,7 +52,7 @@ Iterative build-out, value-first. Each phase is shippable on its own. **Focus:** Nexa actively maintains the homelab inventory instead of being told manually. **Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo. -- [ ] **6.1 Inventory sync** — Nexa polls Dockge / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …). +- [ ] **6.1 Inventory sync** — Nexa polls Arcane / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …). - [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35–37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*. - [ ] **6.3 Steward commands** — `#nexa:retire ` (tar to `backup/`, stop, archive the stack), `#nexa:document ` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed). - [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface). diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 49d2eae..cbebddd 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -92,10 +92,10 @@ This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry. -The docker LXC is the **Dockge** flavor of `proxmox-helper-scripts` — paste the stack into the Dockge UI (`http://192.168.1.40:5001` or whatever Zoraxy host is configured), name it `nexa`, save, start. Don't `docker compose up -d` over SSH; Dockge maintains the compose YAML at `/opt/stacks/nexa/compose.yaml`. +The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle. ```yaml -# Dockge stack: nexa (paste in the UI) +# Nexa stack — paste into Arcane services: nexa-embed: image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 @@ -135,7 +135,7 @@ Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the > **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership. > -> Add `UNAS_USER` / `UNAS_PASS` to the Dockge stack's `.env`, or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#13](./12-optimization-opportunities.md)). +> Add `UNAS_USER` / `UNAS_PASS` to the stack's `.env` (Arcane provides an editor for this), or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#11](./12-optimization-opportunities.md#11)). Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 97b2083..b210c27 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -53,7 +53,7 @@ Items that **block progress** and need a human decision before a workflow can be ## Verification needed before Phase-3.1 deploy -- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing Dockge stack (recommended: `services/karakeep/` or `services/immich/`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Dockge, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. +- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. --- diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 4a91abb..e22ef3c 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -83,20 +83,20 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths. 32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. -33. **Stacks live in Dockge, not raw compose.** The LXC is the Dockge flavor of helper-scripts. Deployment writes a stack named `nexa` in the Dockge UI; the compose YAML is stored under `/opt/stacks/nexa/` (Dockge default) and edited from the web UI. This means [docs/09](./09-deployment.md) "deploy" snippets translate to "paste this into Dockge → save → start", not `docker compose up -d` over SSH. +33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane. 34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). ## Housekeeping campaign (cross-cutting, do together) -These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Dockge, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down. +These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down. -35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Dockge: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths: +35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Arcane: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths: - **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron. - **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk. Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. -36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Dockge stack and check the `volumes:` block. Suspected non-compliant (need verification): +36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Arcane stack and check the `volumes:` block. Suspected non-compliant (need verification): - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state. - `memos` — the user-facing notes app; loss = data loss. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md index 8787bab..dc34d92 100644 --- a/docs/13-information-wishlist.md +++ b/docs/13-information-wishlist.md @@ -8,14 +8,16 @@ Each task header tells you **exactly where to run it** (which shell or which UI) ## Task 1 — Stack inventory + reference compose **(highest leverage)** -📍 **Where:** Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC). +📍 **Two halves — the canonical compose lives in the Arcane UI; the running-container view comes from a shell.** Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. +**(a) Reference compose** — open **Arcane** in your browser → pick a clean small stack (Karakeep is a good choice) → screenshot or copy the **compose.yaml** *and* the **`.env`** (or whatever credential file is referenced). Just the `volumes:` block + the volume credential pattern is enough; everything else is bonus. + +**(b) Running-container view** — Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC): + ```bash docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort -ls /opt/stacks/ -cat /opt/stacks/karakeep/compose.yaml # or any clean small stack ``` **Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. From e8f0d9a2c234dd0562328658f7ae1517b1f540e9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 5 May 2026 04:45:29 +0000 Subject: [PATCH 11/14] Memory retention design + open-question re-asker + UNAS Pro specifics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two design questions answered: 1. Web/news/Karakeep INTO long-term memory? Yes, but with per-source TTL. docs/08 gains a "Memory sources & retention" section that pins TTLs: memo/obsidian/karakeep = permanent, mail = 365d, mail_digest = 90d, rss = 30d, web_search = 90d, system = 30d. Every Qdrant point carries payload.expires_at; a daily prune workflow honours it. New commands in docs/05: #nexa:learn --permanent, #nexa:forget, #nexa:retain, #nexa:ask --web (SearXNG → crawl4ai-mcp → markitdown-mcp → embed). Phase 2.4 added to roadmap. 2. Re-ask unanswered open questions. Backoff schedule (3d → 7d → 21d → 60d) tracked in GraphDB per question. Surface ONE question per day in the morning digest, but only when the digest is otherwise short (capacity guard < 800 chars). User reply parsed → question auto- resolved → docs/11 diff proposed (Phase 6.4 hook). New commands in docs/05: #nexa:digest, #nexa:remind, #nexa:answered. Phase 5.3 added. UNAS Pro details from the UniFi Drive dashboard: - It's a Ubiquiti UNAS Pro (UniFi Drive 4.1.16 on UniFi OS 5.0.17), SFP+, RAID 6, 19.96 TiB raw, 2.05 TiB used. Recorded in CLAUDE.md. - SMB native paths: smb://192.168.1.31/ (mac) / \\192.168.1.31\ (win). UniFi recommends SMB as the modern path — matches our decision to default Nexa volumes to SMB. - ⚠️ Storage-pool snapshots NOT configured ("Click to Setup"). Added as optimization #38: highest-leverage data-protection change in the homelab right now. Daily + weekly UniFi Drive snapshot, native, no agent. RAID 6 doesn't protect against rm -rf or accidental mass- delete; snapshots do. - #39: Nexa workflows that mutate large state can use the same native snapshots for fast rollback (pre-snapshot → operate → verify). --- CLAUDE.md | 5 ++-- docs/02-roadmap.md | 4 ++- docs/05-command-system.md | 42 ++++++++++++++++++++++++++- docs/08-graphrag-architecture.md | 32 ++++++++++++++++++++ docs/12-optimization-opportunities.md | 4 +++ 5 files changed, 83 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 94d8b8f..d52f0f7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,9 +15,10 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways: + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). Reachable two ways: - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. + - **SMB** — **default for Nexa per-container volumes** (Ubiquiti also recommends SMB as the modern path; the URL is `smb://192.168.1.31/` macOS / `\\192.168.1.31\` Windows). The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. + - **Storage-layer snapshots are NOT configured** on the UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, but not "the whole pool, last Tuesday". See [docs/12 #38](./docs/12-optimization-opportunities.md#38). **UNAS layout conventions** (homelab-wide, all docker containers follow them): - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index fff7ebf..b4995c2 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -17,8 +17,9 @@ Iterative build-out, value-first. Each phase is shippable on its own. **Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags. - [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails. -- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. +- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)). - [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets. +- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch. ## Phase 3 — Memory (Qdrant & RAG) @@ -46,6 +47,7 @@ Iterative build-out, value-first. Each phase is shippable on its own. - [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol. - [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy. +- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md). ## Phase 6 — Nexa as homelab steward diff --git a/docs/05-command-system.md b/docs/05-command-system.md index ae81385..12701cc 100644 --- a/docs/05-command-system.md +++ b/docs/05-command-system.md @@ -54,12 +54,22 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi ### #nexa:learn [topic] - **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags) -- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` +- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL) ``` #nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture ``` +### #nexa:forget [query-or-iri] +- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort. +- **Beispiel**: `#nexa:forget --source=rss --older-than=14d` + +### #nexa:retain [source_type] [days] +- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)). + +### #nexa:ask [question] [--web] +- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben). + ## 🎯 Workflow-Kommandos ### #nexa:route-test [text] @@ -84,6 +94,36 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi #nexa:email-digest ``` +### #nexa:digest +- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt: + 1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`). + 2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency). + 3. Drift-Report aus Phase 6 (sobald aktiv). + 4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist. + +#### Re-ask der offenen Fragen (`docs/11-open-questions.md`) +Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema: + +| Mal | Wartezeit bis zur nächsten Frage | +|-----|----------------------------------| +| 1 → 2 | 3 Tage | +| 2 → 3 | 7 Tage | +| 3 → 4 | 21 Tage | +| ≥ 5 | 60 Tage | + +Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn: +- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt). +- `nextAskAt <= today`. +- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten). + +Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review. + +### #nexa:remind +- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag. + +### #nexa:answered +- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor. + ## 🔐 Admin-Kommandos ### #nexa:reset-config diff --git a/docs/08-graphrag-architecture.md b/docs/08-graphrag-architecture.md index 110b24c..2c7d624 100644 --- a/docs/08-graphrag-architecture.md +++ b/docs/08-graphrag-architecture.md @@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH --permanent` overrides the default TTL. +- `#nexa:forget ` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`. +- `#nexa:retain ` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run). diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index e22ef3c..2cee0a7 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -113,3 +113,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target. This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. + +38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything. + +39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3. From dfd9aa7d2a058162d37527b59af0abf966fc7a4b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 5 May 2026 04:50:43 +0000 Subject: [PATCH 12/14] 3-2-1 backup tiering with off-site cold (Jottacloud candidate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - s3.nuclide.systems is on-site only — recasting it as the WARM tier, not the disaster-recovery copy. Off-site cold becomes a separate decision tracked as Q20. - docs/12 #37 expanded with full provider comparison: Jottacloud Unlimited (~€9.50/mo, EU/Norway, soft-cap ~5 TB) is the user's stated candidate and a reasonable fit for the current 2 TB scale with ~10 years of headroom. Alternatives priced: Hetzner Storage Box, Backblaze B2, rsync.net, Storj. Always restic/rclone-crypt before upload. - Off-site priority list pinned: Immich originals + documents + Nextcloud user data + Vaultwarden DB go off-site; movies / music / ROMs / derived caches don't. - docs/10 backup section rewritten as the 3-2-1 pipeline (source → UNAS native snapshot → warm → encrypted cold). Nexa-specific snapshot rows updated for the new collection names. - docs/11 Q20 added with the decision criteria. - CLAUDE.md: backup model pinned so future agents don't propose alternative paths. --- CLAUDE.md | 1 + docs/10-operations.md | 14 ++++++--- docs/11-open-questions.md | 2 ++ docs/12-optimization-opportunities.md | 42 +++++++++++++++++++++++---- 4 files changed, 49 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d52f0f7..4fc9215 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,6 +45,7 @@ This file tells future automated runs what they need to know about this repo. - `Wunschliste` → Wishes. Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh. - **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests. +- **Backup model is 3-2-1**: UNAS native snapshots → `s3.nuclide.systems` (warm, on-site) → encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate — see [Q20](./docs/11-open-questions.md#q20)). Always `restic`/`rclone-crypt` before upload — third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first. - **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords. - **Decided for Nexa** (don't re-litigate without user input): - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). diff --git a/docs/10-operations.md b/docs/10-operations.md index c5fe9da..a58dcc5 100644 --- a/docs/10-operations.md +++ b/docs/10-operations.md @@ -4,15 +4,21 @@ Day-2 concerns. Backups, monitoring, troubleshooting. ## Backups -Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items: +**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)): +1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable). +2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103). +3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext. + +Nexa-specific items inside this pipeline: | What | How | Frequency | |------|-----|-----------| | n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container | -| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) | +| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections//snapshots` → write to `backup/nexa/snapshots/qdrant//` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) | +| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/.ttl.gz` | daily | | Memos DB | Backrest snapshot of Memos data dir | already covered | -| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered | -| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | +| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered | +| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | ## Monitoring diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index b210c27..e3364fd 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -55,6 +55,8 @@ Items that **block progress** and need a human decision before a workflow can be - [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. +- [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end. + --- ## Speed budget (Q3 follow-up) diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 2cee0a7..56b6e41 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -106,13 +106,43 @@ These three are inter-related — picking them up as one campaign is cheaper tha Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services//` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward. -37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup//` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier: - - **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination. - - **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore). - - **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars. - - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target. +37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.** + `s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers: - This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. + **Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):** + - **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination. + - **Qdrant + GraphDB snapshots** flow UNAS → S3 daily. + - Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set. + + **Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):** + - **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`. + - **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative. + - **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores. + - **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication. + - **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem. + + **Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy. + + **Tiering & cadence:** + ``` + [Source] UNAS RAID-6 + native snapshots (#38) + │ daily restic/borg + ▼ + [Warm] s3.nuclide.systems (on-site, fast restore) + │ weekly rclone copy + crypt + ▼ + [Cold] Off-site (Jottacloud / B2 / Hetzner) ← satisfies the "1" in 3-2-1 + ``` + + **What goes off-site (priority order):** + 1. Immich photo originals — irreplaceable. + 2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable. + 3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo. + 4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include. + + **What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate. + + Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it. 38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything. From a8446f83717917025f70f910666c50dce464196e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 5 May 2026 05:01:48 +0000 Subject: [PATCH 13/14] Q19 resolved: storage convention is plain host bind-mount, not SMB volume MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Karakeep compose user shared confirms every container in LXC 104 uses host bind-mounts of /mnt/pve/unas/services// directly. No driver_opts, no CIFS, no per-volume credentials. The earlier SMB-as-docker-volume proposal in this repo was over-fitting to the Nextcloud-specific NFS issue and is withdrawn. The Nextcloud failure was Nextcloud-specific: its setup tooling chowns the data dir to www-data, which fails against root_squash exports. Normal containers don't trigger that path, so plain bind-mounts work fine. Changes: - docs/09 §Step 3: compose block dropped from ~30 lines (volume, driver_opts, CIFS device, credentials env) to a one-line bind-mount + env_file. Added a pre-deploy `mkdir -p` for the UNAS service + backup paths. Explanatory note retained for future agents. - docs/12 #27 marked withdrawn pointing to #31. #31 rewritten with the bind-mount pattern (was SMB-default). #36 (UNAS audit) now looks for `- /var/lib/docker/...` or anonymous volumes as the non-compliant signal. - CLAUDE.md storage block: bind-mount is the standard, SMB is documented escape-hatch only. - docs/11: Q19 moved into Resolved with the new framing. Q12 + Q20 grouped under a new "## Backups" section header (the now-empty "Verification needed" section removed). - docs/13 Task 1 struck through with done-marker pointing to docs/09. Net: Phase-3.1 deployment YAML is now actually deployable as written. --- CLAUDE.md | 9 ++--- docs/09-deployment.md | 48 +++++++++++---------------- docs/11-open-questions.md | 8 ++--- docs/12-optimization-opportunities.md | 16 ++++----- docs/13-information-wishlist.md | 16 ++------- 5 files changed, 38 insertions(+), 59 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4fc9215..036eb22 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,10 +15,11 @@ This file tells future automated runs what they need to know about this repo. - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%. - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`. - LXC 103 backrest — backup orchestration. - - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). Reachable two ways: - - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts. - - **SMB** — **default for Nexa per-container volumes** (Ubiquiti also recommends SMB as the modern path; the URL is `smb://192.168.1.31/` macOS / `\\192.168.1.31\` Windows). The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely. - - **Storage-layer snapshots are NOT configured** on the UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, but not "the whole pool, last Tuesday". See [docs/12 #38](./docs/12-optimization-opportunities.md#38). + - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at `/var/nfs/shared/storage`, mounted by Proxmox at `/mnt/pve/unas`, also exposed via SMB at `smb://192.168.1.31/` (Mac) / `\\192.168.1.31\` (Win). + + **Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of `/mnt/pve/unas/services//` from inside LXC 104.** No `driver_opts`, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an **escape hatch only** ([docs/12 #27](./docs/12-optimization-opportunities.md#27)) for services that hit Nextcloud-style NFS issues — Nexa doesn't, so we don't use it. + + **Storage-layer snapshots are NOT configured** on UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, not "the whole pool last Tuesday". Highest-leverage fix in the homelab right now ([docs/12 #38](./docs/12-optimization-opportunities.md#38)). **UNAS layout conventions** (homelab-wide, all docker containers follow them): - **`services//` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout. diff --git a/docs/09-deployment.md b/docs/09-deployment.md index cbebddd..16c865d 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -95,7 +95,11 @@ Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB im The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle. ```yaml -# Nexa stack — paste into Arcane +# Nexa stack — paste into Arcane. +# Storage convention matches the rest of the homelab (verified against +# the running Karakeep stack, Q19): host bind-mount of +# /mnt/pve/unas/services//. No volume-driver, no CIFS, no +# credentials in the compose — the LXC's NFS mount is already there. services: nexa-embed: image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 @@ -105,37 +109,25 @@ services: ports: - "127.0.0.1:8080:80" volumes: - - tei-cache:/data + - /mnt/pve/unas/services/nexa/tei-cache:/data + env_file: + - .env -volumes: - # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues - # (the same kind that bit the Nextcloud rollout). Stores embedding model - # caches, Qdrant snapshots, future GraphDB exports. - # - # The exact mount block (driver, share name, credential injection) MUST - # be copied verbatim from an existing stack on the same docker host — - # services// on UNAS is the homelab-wide docker config store and - # every container already follows the same SMB pattern. See Q19 in - # docs/11. The block below is the *shape* but the share name "" - # is a placeholder until verified against e.g. services/karakeep/compose. - tei-cache: - driver: local - driver_opts: - type: cifs - device: //192.168.1.31//services/nexa/tei-cache - o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8" - -networks: - default: - external: true - name: nexa +networks: {} ``` -Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant. +Pre-deploy step on the docker LXC (one-time): -> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership. -> -> Add `UNAS_USER` / `UNAS_PASS` to the stack's `.env` (Arcane provides an editor for this), or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#11](./12-optimization-opportunities.md#11)). +```bash +mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb} +mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb} +``` + +Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into `/mnt/pve/unas/services/nexa/tei-cache/`; subsequent restarts are instant. + +Secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, `QDRANT_API_KEY`, `NC_APP_PASSWORD`) go in the stack's `.env` next to the compose — same pattern Karakeep uses (`env_file: .env`). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term ([12/#11](./12-optimization-opportunities.md#11)) but isn't required for the first cut. + +> **Why bind-mount and not SMB?** Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's "had it with Nextcloud" experience. The Karakeep stack confirms the actual convention is the simpler one: **host bind-mount of the LXC's existing `/mnt/pve/unas` NFS mount**. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to `www-data`, which fails against `root_squash` exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in [12/#27](./12-optimization-opportunities.md#27) only as an escape hatch *if* a future service hits Nextcloud-style issues — Nexa doesn't, so we don't use it. Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter: diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index e3364fd..593d868 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -17,6 +17,7 @@ Items that **block progress** and need a human decision before a workflow can be - [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme. - [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich. - [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant). +- [x] **Q19 — Storage convention is plain host bind-mount of `/mnt/pve/unas/services//`.** Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via `env_file: .env` next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn ([12/#27](./12-optimization-opportunities.md#27)) — it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to `www-data` and fails on `root_squash`-style exports; that doesn't apply to normal containers). Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). - [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files//Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index): - `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache. - `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB). @@ -49,12 +50,9 @@ Items that **block progress** and need a human decision before a workflow can be - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27). - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. -- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports? - -## Verification needed before Phase-3.1 deploy - -- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. +## Backups +- [ ] **Q12 — S3 archive bucket** (warm tier). `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots, GraphDB exports, and as the rsync source for the cold-tier copy. - [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end. --- diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 56b6e41..816f444 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -71,17 +71,17 @@ Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (i lxc.idmap: g 104 104 1 # render group on host ``` Then inside the container: `usermod -aG video,render ` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path. -31. **Match the homelab-wide UNAS storage convention.** Every existing docker stack on LXC 104 already binds its persistent data into `services//` on UNAS via SMB (immich, karakeep, nextcloud, ntfy, open-webui, paperless-ai, pocketid, shelfmark, siyuan, stremio, traccar, vaultwarden, gluetun all follow this). Nexa **must** follow the same pattern at `services/nexa/`. Do not invent a parallel layout. Targets: - - `services/nexa/qdrant/` — Qdrant data dir (move before the visual collection lands). - - `services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). - - `services/nexa/graphdb/` — Phase-3.4 RDF store. - - `backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. +31. **Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services//`.** Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services//...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets: + - `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir. + - `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk). + - `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store. + - `/mnt/pve/unas/backup/nexa/snapshots/qdrant//` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup//` pattern used by Home Assistant / Immich. - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`. - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers. - Why SMB and not the host's NFS mount: the user already lived through the unprivileged-LXC + NFS + per-container-write trap with Nextcloud. SMB mounted directly as a docker volume (driver `local`, type `cifs`) with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely. **Pre-deploy step:** read the volume block from an existing stack (Karakeep is small and clean) and copy it verbatim — this guarantees Nexa uses the same SMB share name, credential pattern, uid/gid as the rest of the homelab. Tracked as [Q19](./11-open-questions.md#q19). + Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/:/`. Concrete example in [09-deployment §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). - Credentials (`UNAS_USER` / `UNAS_PASS`) go in Vaultwarden ([12/#11](#11)), injected at deploy. The host-side `/mnt/pve/unas` NFS mount stays useful for admin tasks (rsync, manual backups) — not for per-container write paths. + Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets ([12/#11](#11)) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is **withdrawn** ([12/#27](#27)) — it was an over-fitting to the Nextcloud-specific NFS issue. 32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. 33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane. 34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`). @@ -96,7 +96,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running. -36. **UNAS-integration audit.** `services//` on UNAS is the homelab convention but not every container follows it yet. Walk every Arcane stack and check the `volumes:` block. Suspected non-compliant (need verification): +36. **UNAS-integration audit.** `/mnt/pve/unas/services//` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification): - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection. - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state. - `memos` — the user-facing notes app; loss = data loss. diff --git a/docs/13-information-wishlist.md b/docs/13-information-wishlist.md index dc34d92..7999bba 100644 --- a/docs/13-information-wishlist.md +++ b/docs/13-information-wishlist.md @@ -6,21 +6,9 @@ Each task header tells you **exactly where to run it** (which shell or which UI) --- -## Task 1 — Stack inventory + reference compose **(highest leverage)** +## ~~Task 1 — Stack inventory + reference compose~~ ✅ DONE -📍 **Two halves — the canonical compose lives in the Arcane UI; the running-container view comes from a shell.** - -Locks **Q19** (the SMB share name + driver options Nexa must mirror) and replaces screenshot-derived guesses about what's running. - -**(a) Reference compose** — open **Arcane** in your browser → pick a clean small stack (Karakeep is a good choice) → screenshot or copy the **compose.yaml** *and* the **`.env`** (or whatever credential file is referenced). Just the `volumes:` block + the volume credential pattern is enough; everything else is bonus. - -**(b) Running-container view** — Proxmox UI → Datacenter → `nuc` → LXC `104 (docker)` → **Console** (or `ssh root@192.168.1.40` from your PC): - -```bash -docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort -``` - -**Unblocks:** Q19, the housekeeping migration template (#36), Phase-3.1 deployment YAML. +User pasted the Karakeep compose. Q19 resolved → host bind-mount of `/mnt/pve/unas/services//` is the convention. Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). The `docker ps -a` half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) — but that's not blocking now. --- From e0061ee173d603dd8a70eac6a4d36d5c63e6e20e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 5 May 2026 05:18:58 +0000 Subject: [PATCH 14/14] Defer Q12 and Q20: backup-tier decisions parked until Phase 3.3 is up next --- docs/02-roadmap.md | 2 +- docs/11-open-questions.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md index b4995c2..b46cdf7 100644 --- a/docs/02-roadmap.md +++ b/docs/02-roadmap.md @@ -28,7 +28,7 @@ Iterative build-out, value-first. Each phase is shippable on its own. - [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet. - [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32). -- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. +- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. **Deferred** until [Q12 + Q20](./11-open-questions.md#backups-deferred) are decided. Until then, snapshots stay on UNAS at `backup/nexa/snapshots/qdrant//`; UNAS RAID 6 + native pool snapshots ([12/#38](./12-optimization-opportunities.md#38)) cover the data-loss scenarios that matter most for Phase 3.1. - [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md). ## Phase 4 — Motor (organization & action) diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index 593d868..96a092d 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -50,10 +50,12 @@ Items that **block progress** and need a human decision before a workflow can be - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config. - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27). - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands. -## Backups +## Backups (deferred) -- [ ] **Q12 — S3 archive bucket** (warm tier). `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots, GraphDB exports, and as the rsync source for the cold-tier copy. -- [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end. +Backup-tier decisions are intentionally parked while Phase-3.1/3.2/3.4 are built. UNAS RAID 6 + Backrest already cover file-level recovery; pool-snapshot configuration ([12/#38](./12-optimization-opportunities.md#38)) remains the single highest-leverage data-protection action and doesn't depend on either question below. Re-open both when [Phase 3.3](./02-roadmap.md#phase-3--memory-qdrant--rag) becomes the next-up item. + +- [ ] **Q12 — *deferred.*** S3 archive bucket on `s3.nuclide.systems` (warm tier). Pick when Phase 3.3 is up next. +- [ ] **Q20 — *deferred.*** Off-site cold-tier provider (Jottacloud is the leading candidate). Pick when Phase 3.3 is up next. ---