Merge pull request #1 from noheton/claude/organize-docs-deployment-7N4v2

Reorganize documentation and consolidate deployment guides

CLAUDE.md

@@ -10,10 +10,27 @@ This file tells future automated runs what they need to know about this repo.
 
 ## Real infrastructure (verified from screenshots, May 2026)
 
-- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9).
+- **Proxmox host** `nuc` at `192.168.1.20:8006` (PVE 9.1.9, kernel 6.17.13-4-pve, EFI).
+  - Hardware: **22 threads** (Intel Core Ultra 7 155H, 1 socket), **62 GiB RAM**, **1.64 TiB disk**.
+  - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
   - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
   - LXC 103 backrest — backup orchestration.
-  - LXC 104 docker — main docker host at `192.168.1.40` (40 containers).
+  - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at `/var/nfs/shared/storage`, mounted by Proxmox at `/mnt/pve/unas`, also exposed via SMB at `smb://192.168.1.31/<share>` (Mac) / `\\192.168.1.31\<share>` (Win).
+
+  **Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` from inside LXC 104.** No `driver_opts`, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an **escape hatch only** ([docs/12 #27](./docs/12-optimization-opportunities.md#27)) for services that hit Nextcloud-style NFS issues — Nexa doesn't, so we don't use it.
+
+  **Storage-layer snapshots are NOT configured** on UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, not "the whole pool last Tuesday". Highest-leverage fix in the homelab right now ([docs/12 #38](./docs/12-optimization-opportunities.md#38)).
+
+  **UNAS layout conventions** (homelab-wide, all docker containers follow them):
+  - **`services/<svc>/` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout.
+  - `backup/<svc>/` — per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). **Nexa snapshots → `backup/nexa/`.**
+  - `media/`, `code/`, `_sortMe/`, `dump/`, `test_perm` — user data, not Nexa's concern.
+  - **Before deploying any Nexa container, READ AN EXISTING STACK** in **Arcane** (e.g. `karakeep` or `immich`) to confirm the exact mount syntax in use — driver name, share path, credential injection pattern. Match it. The actual NFS export root is `/var/nfs/shared/storage`; the SMB share name is **still TBD — see Q19** in `docs/11`.
+
+**Hard-blocklist for any Nexa indexer / agent** (never read these paths or matching glob):
+  - `_sortMe/wallet/**` — contains PGP keys + bitcoin wallet files.
+  - Any path matching `*.gpg`, `*.asc`, `*.key`, `*.pem`, `id_rsa*`, `*wallet*`, `*.kdbx`, `*credentials*`, `*secret*`.
+  - The Nextcloud appdata dir (`services/nextcloud/appdata_*`) — Nextcloud-internal, not user content. Intel iGPU passthrough is *configured but currently broken* — see [docs/12 #26](./docs/12-optimization-opportunities.md#26).
   - LXC 105 nextcloud — Nextcloud at `nc.nuclide.systems`.
   - LXC 106 octoprint — currently *Exited*; flagged in [docs/11](./docs/11-open-questions.md#q13).
   - LXC 108 zoraxy — reverse proxy at `192.168.1.4:8000`, TLS for `*.nuclide.systems`.
@@ -22,6 +39,15 @@ This file tells future automated runs what they need to know about this repo.
   - Memos `:5230`, n8n `:5678`, LiteLLM `:4000` (UI LobeHub `:3210`), Qdrant (`qdrant_scientific`), ntfy `:7998`, Karakeep (legacy alias `hoarder.nuclide.systems`), Vaultwarden `:11001`, Pocket-ID `:1411`, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (`crawl4ai-mcp`, `markitdown-mcp`, `papersearch-mcp`).
 - **Octoprint (LXC 106)** is *intentionally* powered down most of the time. Phase-5 monitoring must skip names matching `octoprint*` rather than alert on its Exited state.
 - **Obsidian vault lives inside Nextcloud** at `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV — read-only, no filesystem mount. Ignore list: `.copilot/`, `.copilot-index/`, `.smart-env/`, `.caldav-sync/`, `assets/` (visual queue, Phase 3.2), `Templates/`, `BMO/`, `Excalidraw/`. Index target: `Notizen/**/*.md`.
+- **Nextcloud Tasks lists & calendars** (German names, may grow over time):
+  - `Persönlich` → Personal context.
+  - `DLR` → Work context (DLR is the user's employer).
+  - `Einkaufsliste` → Shopping.
+  - `Wunschliste` → Wishes.
+  Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh.
+- **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests.
+- **Backup model is 3-2-1**: UNAS native snapshots → `s3.nuclide.systems` (warm, on-site) → encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate — see [Q20](./docs/11-open-questions.md#q20)). Always `restic`/`rclone-crypt` before upload — third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first.
+- **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords.
 - **Decided for Nexa** (don't re-litigate without user input):
   - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`).
   - Embeddings staged: **Phase 3.1** TEI + `BAAI/bge-m3` (text-only, 1024-dim). **Phase 3.2** swap to `infinity` and add `jinaai/jina-clip-v2` (768-dim, joint text+image space). All forward-compat fields (`modality`, `media_uri`, `graph_iri`, `nexa:pendingVisualIndex`) exist from 3.1 — adding the visual collection is additive.

docs/02-roadmap.md

@@ -17,8 +17,9 @@ Iterative build-out, value-first. Each phase is shippable on its own.
 **Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
 
 - [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
-- [ ] **2.2 RSS morning digest** — curated daily summary in Memos.
+- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
 - [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
+- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
 
 ## Phase 3 — Memory (Qdrant & RAG)
 
@@ -27,7 +28,7 @@ Iterative build-out, value-first. Each phase is shippable on its own.
 
 - [ ] **3.1 Text RAG (Path A)** — TEI + `bge-m3`, single Qdrant collection `nexa_knowledge_text`. `#nexa:ask` reads it before answering. Image attachments are *recorded* as `nexa:Note` with `pendingVisualIndex` but not embedded yet.
 - [ ] **3.2 Visual collection (Path C)** — swap TEI → infinity, add `jina-clip-v2`, create `nexa_knowledge_visual`, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See [09 § "Phase add-on: visual collection"](./09-deployment.md#phase-add-on-visual-collection-phase-32).
-- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server.
+- [ ] **3.3 S3 long-term archive** — Qdrant snapshots (both collections) to the S3 server. **Deferred** until [Q12 + Q20](./11-open-questions.md#backups-deferred) are decided. Until then, snapshots stay on UNAS at `backup/nexa/snapshots/qdrant/<date>/`; UNAS RAID 6 + native pool snapshots ([12/#38](./12-optimization-opportunities.md#38)) cover the data-loss scenarios that matter most for Phase 3.1.
 - [ ] **3.4 Ontotext GraphDB** — SPARQL pillar; structural queries combined with vector hits in `#nexa:ask`. See [08](./08-graphrag-architecture.md).
 
 ## Phase 4 — Motor (organization & action)
@@ -46,3 +47,16 @@ Iterative build-out, value-first. Each phase is shippable on its own.
 
 - [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
 - [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
+- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
+
+## Phase 6 — Nexa as homelab steward
+
+**Focus:** Nexa actively maintains the homelab inventory instead of being told manually.
+**Milestone:** Nexa runs a daily *drift report* — what changed, what's stale, what's missing — and turns it into actionable comments under a pinned `[STEWARD]` memo.
+
+- [ ] **6.1 Inventory sync** — Nexa polls Arcane / `docker ps` / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (`nexa:Service`, `nexa:LastSeen`, `nexa:DataPath`, …).
+- [ ] **6.2 Drift detection** — diff against the documented desired state in this repo (`docs/12`, `CLAUDE.md`, the housekeeping campaign at #35–37). Emit findings as structured Memos comments: *"`open-webui` last used 38 d ago — retire?"*, *"Container `foo` has no `services/foo/` bind — schedule UNAS migration?"*.
+- [ ] **6.3 Steward commands** — `#nexa:retire <svc>` (tar to `backup/`, stop, archive the stack), `#nexa:document <svc>` (template a CLAUDE.md entry), `#nexa:wishlist-status` (which entries in [docs/13](./13-information-wishlist.md) are still needed).
+- [ ] **6.4 Self-update of docs** — when the user resolves an open question via Memos, Nexa proposes the corresponding edit to `docs/11-open-questions.md` as a PR-ready diff (the doc itself becomes a Nexa-managed surface).
+
+This phase is what turns Nexa from "an assistant that answers questions" into "an assistant that takes care of its own runtime", and it's the natural home for the cross-cutting [housekeeping campaign](./12-optimization-opportunities.md#housekeeping-campaign-cross-cutting-do-together) — most of those items become semi-automatic once 6.1–6.3 ship.

docs/03-architecture.md

@@ -39,7 +39,7 @@ A one-page mental model. For details follow the cross-links.
 
 | Component | Role | Where it runs (today) |
 |-----------|------|-----------------------|
-| **Memos** | Interface, voice input, webhook source | docker host LXC 104 → `memos.nuclide.systems` |
+| **Memos** | Interface, voice input, webhook source | docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) → `memos.nuclide.systems` |
 | **n8n** | Workflow / logic engine | docker host LXC 104 → `n8n.nuclide.systems` |
 | **SAIA / LiteLLM** | Model gateway, embeddings, classification | docker host LXC 104 → `ai.nuclide.systems` (LiteLLM internal :4000) |
 | **Qdrant** | Vector memory (semantic recall) | docker host LXC 104 — reuse `qdrant_scientific`. Collections: `nexa_knowledge_text` (Phase 3.1, 1024-dim) and `nexa_knowledge_visual` (Phase 3.2, 768-dim) |

docs/04-integration-matrix.md

@@ -10,17 +10,15 @@ Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen vari
 ### A. Bereich: ARBEIT (Work)
 * **Sichtbarkeit:** Eingeschränkt (kein Zugriff auf Arbeits-E-Mails oder Firmen-Server).
 * **Datenquellen:**
-    * Separate Nextcloud Task-Liste (`Work_Tasks`).
+    * Nextcloud Task-Liste **`DLR`** (sowie der gleichnamige Kalender).
-    * Separater Nextcloud Kalender (`Work_Calendar`).
     * Memos mit Tag `#work` oder semantischer Erkennung.
 * **Nexa-Fokus:** Timeblocking, Fokus-Zeiten, Vorbereitung von Meetings basierend auf Obsidian-Notizen.
 
 ### B. Bereich: PRIVAT (Personal)
 * **Sichtbarkeit:** Vollständig.
 * **Datenquellen:**
-    * Nextcloud Task-Liste (`Personal_Tasks`).
+    * Nextcloud Task-Liste **`Persönlich`** und gleichnamiger Kalender.
-    * Haupt-Kalender.
+    * Nextcloud Mail (`fkrebs@nucli.de`, einziges Konto). Ordner `Waiting` ist ein manuelles Signal — Mails dort werden vom Digest ausgeschlossen.
-    * E-Mails (IMAP - 20 Mails/Tag).
     * Bluesky, RSS, Karakeep.
 * **Nexa-Fokus:** Automatisierung des Alltags, Kuratierung von Wissen, E-Mail-Management.
 

docs/05-command-system.md

@@ -5,9 +5,12 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 ## 🔧 Konfigurations-Kommandos
 
 ### #nexa:config
-- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud, Qdrant, etc.)
+- **Beschreibung**: Triggert Autodiscovery aller Services (Nextcloud Listen/Kalender, Qdrant Collections, LiteLLM-Modelle, Mail-Konto).
-- **Antwort**: Postet Report mit gefundenen Konfigurationen
+- **Läuft auch automatisch**:
-- **Speichert**: Settings in DynoDB / Qdrant Metadata (nicht in .env)
+  - Täglich (cron in n8n) — neue Listen / Kalender werden ohne manuelles Eingreifen aufgenommen.
+  - Bei Cache-Miss — wenn ein Workflow eine Liste/ID nicht mehr findet, läuft Discovery einmal, dann Retry.
+- **Antwort**: Postet Report mit den aktuellen Konfigurationen als Memos-Kommentar.
+- **Speichert**: Konfiguration in der Qdrant `_config` Namespace (nicht in `.env`).
 
 ```
 #nexa:config
@@ -15,10 +18,12 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 
 **Beispiel-Response:**
 ```
-✅ Autodiscovery abgeschlossen:
+✅ Autodiscovery abgeschlossen (2026-05-04T23:59):
-- Nextcloud Listen: Work (id=123), Personal (id=456), Shopping (id=789), Wishes (id=1011)
+- Nextcloud Listen: Persönlich (id=14), DLR (id=dlr-1), Einkaufsliste (id=…), Wunschliste (id=…)
-- Qdrant Collection: nexa_knowledge (1536 dims, Cosine)
+- Nextcloud Kalender: Persönlich, DLR, Einkaufsliste, Wunschliste
-- Nextcloud Mail: 3 Accounts erkannt (Office 365, Gmail, Privat)
+- Mail-Konto: fkrebs@nucli.de (Posteingang, Archiv, Junk, Waiting)
+- Qdrant Collection: nexa_knowledge_text (1024 dims, Cosine, 0 Punkte)
+- LiteLLM-Modelle: nexa-chat, nexa-embed
 ```
 
 ### #nexa:status
@@ -49,12 +54,22 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 
 ### #nexa:learn [topic]
 - **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
-- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
+- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
 
 ```
 #nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
 ```
 
+### #nexa:forget [query-or-iri]
+- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
+- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
+
+### #nexa:retain [source_type] [days]
+- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
+
+### #nexa:ask [question] [--web]
+- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
+
 ## 🎯 Workflow-Kommandos
 
 ### #nexa:route-test [text]
@@ -79,6 +94,36 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 #nexa:email-digest
 ```
 
+### #nexa:digest
+- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
+  1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
+  2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
+  3. Drift-Report aus Phase 6 (sobald aktiv).
+  4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
+
+#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
+Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
+
+| Mal | Wartezeit bis zur nächsten Frage |
+|-----|----------------------------------|
+| 1 → 2 | 3 Tage |
+| 2 → 3 | 7 Tage |
+| 3 → 4 | 21 Tage |
+| ≥ 5  | 60 Tage |
+
+Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
+- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
+- `nextAskAt <= today`.
+- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
+
+Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
+
+### #nexa:remind <question-id>
+- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
+
+### #nexa:answered <question-id> <text>
+- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
+
 ## 🔐 Admin-Kommandos
 
 ### #nexa:reset-config

docs/07-workflow-spec.md

@@ -5,12 +5,22 @@
 * **Filter:** `payload.content` enthält `- [ ]` ODER `#todo`.
 
 ## 2. Intelligence Node (SAIA)
-* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: { "context": "work" | "personal", "summary": "string", "urgency": 1-5 }."
+* **Prompt:** "Analysiere folgende Notiz. Gib als JSON zurück: `{ "context": "work" | "personal" | "shopping" | "wishes", "summary": "string", "urgency": 1-5 }`."
 * **Kontext-Zufuhr:** Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.
 
-## 3. Switch Node
+## 3. List Resolver (vor dem Switch)
-* **Route 1:** Context == "work" -> Nextcloud Create Task (List ID: Work).
+* Lese den Cache `nexa._config.lists` aus Qdrant: `{ "Persönlich": <id>, "DLR": <id>, "Einkaufsliste": <id>, "Wunschliste": <id>, discovered_at: <ts> }`.
-* **Route 2:** Context == "personal" -> Nextcloud Create Task (List ID: Personal).
+* Bei Cache-Miss oder 404 vom nachfolgenden Create-Task → triggere `#nexa:config` (Discovery), update den Cache, retry **einmal**.
 
-## 4. Feedback Loop
+## 4. Switch Node (4-Wege)
-* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
+* **Route 1:** `context == "work"` → Nextcloud Create Task auf Liste `DLR`.
+* **Route 2:** `context == "personal"` → Nextcloud Create Task auf Liste `Persönlich`.
+* **Route 3:** `context == "shopping"` → Nextcloud Create Task auf Liste `Einkaufsliste`.
+* **Route 4:** `context == "wishes"` → Nextcloud Create Task auf Liste `Wunschliste`.
+* **Default (Konfidenz < 0.7):** `Persönlich` (siehe [06](./06-classification-logic.md)).
+
+## 5. Feedback Loop
+* Poste die Task-ID und den gewählten Kontext als Kommentar unter das Original-Memo.
+
+## 6. Listen-Diskoverability
+Listennamen sind nicht hardgecodet — die Resolver-Step liest sie aus dem Runtime-Config-Cache. Wenn der User in Nextcloud eine neue Liste anlegt (z. B. `Reisen`), erscheint sie nach dem nächsten geplanten `#nexa:config`-Lauf (täglich) automatisch als Routing-Ziel — die System-Prompt der Intelligence Node wird zusammen mit den verfügbaren Listen versorgt, sodass SAIA neue Kontexte vorschlagen kann.

docs/08-graphrag-architecture.md

@@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https
 | "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
 
 Combined: **complete understanding** rather than a search index *or* a structure index.
+
+---
+
+## Memory sources & retention
+
+Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
+
+| `source_type` | Where it comes from | Default TTL | Rationale |
+|---------------|--------------------|-------------|-----------|
+| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
+| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
+| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
+| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
+| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
+| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
+| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
+| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
+| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
+
+**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
+
+### Web search loop
+
+`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
+
+This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
+
+### Manual overrides
+
+- `#nexa:learn <text> --permanent` overrides the default TTL.
+- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
+- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).

docs/09-deployment.md

@@ -92,17 +92,42 @@ This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB
 
 Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.
 
-```bash
+The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle.
-# on the docker host (LXC 104)
+
-docker run -d --name nexa-embed \
+```yaml
-  --restart unless-stopped \
+# Nexa stack — paste into Arcane.
-  -p 127.0.0.1:8080:80 \
+# Storage convention matches the rest of the homelab (verified against
-  -v tei-data:/data \
+# the running Karakeep stack, Q19): host bind-mount of
-  ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 \
+# /mnt/pve/unas/services/<svc>/<vol>. No volume-driver, no CIFS, no
-  --model-id BAAI/bge-m3
+# credentials in the compose — the LXC's NFS mount is already there.
+services:
+  nexa-embed:
+    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
+    container_name: nexa-embed
+    restart: unless-stopped
+    command: ["--model-id", "BAAI/bge-m3"]
+    ports:
+      - "127.0.0.1:8080:80"
+    volumes:
+      - /mnt/pve/unas/services/nexa/tei-cache:/data
+    env_file:
+      - .env
+
+networks: {}
 ```
 
-Memory budget: ~1.1 GB resident (bge-m3 is ~1 GB + ~100 MB overhead). First start downloads the model into the named volume; subsequent restarts are instant.
+Pre-deploy step on the docker LXC (one-time):
+
+```bash
+mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb}
+mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}
+```
+
+Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into `/mnt/pve/unas/services/nexa/tei-cache/`; subsequent restarts are instant.
+
+Secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, `QDRANT_API_KEY`, `NC_APP_PASSWORD`) go in the stack's `.env` next to the compose — same pattern Karakeep uses (`env_file: .env`). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term ([12/#11](./12-optimization-opportunities.md#11)) but isn't required for the first cut.
+
+> **Why bind-mount and not SMB?** Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's "had it with Nextcloud" experience. The Karakeep stack confirms the actual convention is the simpler one: **host bind-mount of the LXC's existing `/mnt/pve/unas` NFS mount**. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to `www-data`, which fails against `root_squash` exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in [12/#27](./12-optimization-opportunities.md#27) only as an escape hatch *if* a future service hits Nextcloud-style issues — Nexa doesn't, so we don't use it.
 
 Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter:
 
@@ -170,10 +195,16 @@ curl -i https://n8n.nuclide.systems/webhook/memos
 
 Create a memo with body `#nexa:config` — the discovery workflow:
 
-1. Lists Nextcloud Tasks lists, picks Work / Personal / Shopping / Wishes by name.
+1. Lists Nextcloud Tasks lists, looking for `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (and any new lists added later — see [05](./05-command-system.md)). Caches `name → id`.
-2. Counts existing Qdrant points in `nexa_knowledge`.
+2. Lists Nextcloud Calendars by the same names; caches IDs.
-3. Verifies LiteLLM reachability + lists available models.
+3. Verifies the Nextcloud Mail account `fkrebs@nucli.de` and the folder list (`Posteingang`, `Archiv`, `Junk`, `Waiting`).
-4. Replies as a comment with a **runtime-config snapshot** that's stored as Qdrant metadata (`_config` namespace) and as `nexa-core/config/runtime_config.json` (gitignored).
+4. Counts existing Qdrant points in `nexa_knowledge_text`.
+5. Verifies LiteLLM reachability + lists available models for the Nexa virtual key.
+6. Replies as a comment with a **runtime-config snapshot** that's stored in the Qdrant `_config` namespace (and mirrored as `nexa-core/config/runtime_config.json`, gitignored).
+
+The same workflow is also triggered by:
+- A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention).
+- Cache-miss in the task-router — if a list ID 404s, the router fires `#nexa:config` once and retries.
 
 After this point, `.env` is read-once. Subsequent runs read config from Qdrant.
 

docs/10-operations.md

@@ -4,15 +4,21 @@ Day-2 concerns. Backups, monitoring, troubleshooting.
 
 ## Backups
 
-Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items:
+**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)):
+1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable).
+2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103).
+3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext.
+
+Nexa-specific items inside this pipeline:
 
 | What | How | Frequency |
 |------|-----|-----------|
 | n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
-| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) |
+| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) |
+| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily |
 | Memos DB | Backrest snapshot of Memos data dir | already covered |
-| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered |
+| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered |
-| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
+| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
 
 ## Monitoring
 

docs/11-open-questions.md

@@ -17,6 +17,7 @@ Items that **block progress** and need a human decision before a workflow can be
 - [x] **Q16 — Image-attachment queue ergonomics → leave bytes at source, reference by `media_uri`.** Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the `Notizen` folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme.
 - [x] **Q17 — Immich out-of-band.** Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich.
 - [x] **Q18 — SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest.** A 2 k-note Obsidian backfill would take ~3.3 h; real-time `#nexa:ask` would queue for tens of seconds during a writing burst. **Decision: deploy TEI as planned.** SAIA embeddings remain available as a manual fallback (e.g. for one-off `#nexa:learn` calls where rate is irrelevant).
+- [x] **Q19 — Storage convention is plain host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via `env_file: .env` next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn ([12/#27](./12-optimization-opportunities.md#27)) — it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to `www-data` and fails on `root_squash`-style exports; that doesn't apply to normal containers). Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
 - [x] **Q4 — Obsidian sync via Nextcloud WebDAV.** Confirmed the vault is `nc.nuclide.systems/Notizen/` (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (`/remote.php/dav/files/<user>/Notizen/`) reusing the existing `NC_APP_PASSWORD` — no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's `notify_push` for sub-second updates is captured as optimization #15. Ignore list (don't index):
   - `.copilot/`, `.copilot-index/` — Obsidian Copilot's own embeddings cache.
   - `.smart-env/` — Smart Connections / Smart Composer plugin data (~13 MB).
@@ -25,25 +26,42 @@ Items that **block progress** and need a human decision before a workflow can be
   - `Templates/` — empty templates, low semantic value.
   - `BMO/`, `Excalidraw/` — plugin folders.
   Anything else under `Notizen/**/*.md` is fair game.
+- [x] **Q6 + Q7 — Lists & calendars discovered by name, not ID. Self-healing.**
+  Confirmed names from screenshots:
+  - **Tasks**: `Persönlich` (14), `DLR` (13), `Einkaufsliste` (13), `Wunschliste` (9).
+  - **Calendars**: `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name).
+  - Mapping in classification: `DLR` = Work-Kontext, `Persönlich` = Personal-Kontext, `Einkaufsliste` = Shopping, `Wunschliste` = Wishes.
 
-## Identifiers needed (auto-discoverable, but list now if known)
+  **Self-healing requirement (the user explicitly noted lists may change/grow):** Nexa must not cache IDs forever. The `#nexa:config` workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or "not found". The runtime-config record in Qdrant's `_config` namespace stores `{name → id, discovered_at}` and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring `#einkaufsliste` / `#dlr` etc. without code changes.
 
-- [ ] **Q6 — Nextcloud Tasks list IDs** for: `Work_Tasks`, `Personal_Tasks`, `Shopping`, `Wishes`. Discovery via `#nexa:config` will fill these — confirm names match.
+- [x] **Q8 — Mail via Nextcloud, single account `fkrebs@nucli.de`.** No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: `Posteingang` (default), `Archiv`, `Junk` (61 — auto-filtered, ignored by Nexa), `Papierkorb`, `Waiting`. The `Waiting` folder is a useful manual signal — items moved there by the user are skipped from digest (treat as "in flight").
-- [ ] **Q7 — Nextcloud Calendar IDs** for: `Work_Calendar`, primary personal calendar.
+
-- [ ] **Q8 — IMAP credentials for the personal mail account.** Can n8n reuse a Nextcloud Mail account (preferred — no extra password) or must we add a dedicated IMAP entry?
+- [x] **Q9 — Pocket-ID SSO is configured everywhere.** No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n → Memos webhook, n8n → Nextcloud, n8n → Qdrant, n8n → LiteLLM) keep using API keys / app passwords — SSO is for human UIs only. Don't add basic-auth or per-app login screens.
-- [ ] **Q9 — ntfy topic name** for `nexa.system`. Is the topic public on `ntfy.nuclide.systems` or should it be authenticated?
+
-- [ ] **Q10 — Pocket-ID role.** `id.nuclide.systems` is running. Do we want SSO in front of the n8n / Memos UIs, or skip for now?
+  This **resolves Q10 too** (the proposed "Pocket-ID SSO in front of n8n / Memos" is already done; Nexa just inherits it).
+
+- [x] **Q10 — folded into Q9.**
 
 ## Hardware / capacity
 
-- [ ] **Q11 — RAM headroom on docker host.** Sized against 31 GiB allocated to the LXC; the NUC 14 Pro itself has 64 GB so the LXC's memory cap can be lifted if needed. Phase-3 stack (Qdrant + TEI/bge-m3 ~1.1 GB + Ontotext GraphDB ~4 GB heap, later +infinity/jina-clip-v2 ~1 GB) tops out around ~14 GB. Comfortable. Confirm acceptable.
+- [x] **Q11 — Capacity confirmed from Proxmox node + LXC summaries.**
-- [ ] **Q12 — S3 archive bucket.** `s3.nuclide.systems` is up. Bucket name + access key for Qdrant snapshots and GraphDB exports?
+  - **Host (NUC 14 Pro)**: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (≈24 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.
+  - **LXC 104 (docker, unprivileged)**: 16 CPU, **31.25 GiB RAM cap (7.86 GiB used = 25%)**, 8 GiB swap (idle), **200 GiB boot disk at 47.7% used (~95 GiB)**. Tag `proxmox-helper-scripts`. Intel iGPU passthrough configured but currently failing — see [12/#26](./12-optimization-opportunities.md#26).
+  - **RAM headroom is comfortable**: Phase-3 budget (~7 GiB additional — Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap `zfs_arc_max` lower (ARC currently 24 GiB). Either is one line of config.
+  - **Disk is fine** — the LXC also NFS-mounts the UNAS at `/mnt/pve/unas` (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See [12/#27](./12-optimization-opportunities.md#27).
+  - **Verdict: no blockers for Phase-3.1.** Revisit once 3.4 GraphDB lands.
+## Backups (deferred)
+
+Backup-tier decisions are intentionally parked while Phase-3.1/3.2/3.4 are built. UNAS RAID 6 + Backrest already cover file-level recovery; pool-snapshot configuration ([12/#38](./12-optimization-opportunities.md#38)) remains the single highest-leverage data-protection action and doesn't depend on either question below. Re-open both when [Phase 3.3](./02-roadmap.md#phase-3--memory-qdrant--rag) becomes the next-up item.
+
+- [ ] **Q12 — *deferred.*** S3 archive bucket on `s3.nuclide.systems` (warm tier). Pick when Phase 3.3 is up next.
+- [ ] **Q20 — *deferred.*** Off-site cold-tier provider (Jottacloud is the leading candidate). Pick when Phase 3.3 is up next.
 
 ---
 
 ## Speed budget (Q3 follow-up)
 
-Workload on the docker host (16 CPU, ~30 GB free RAM):
+Workload measured against the **LXC 104 cap** (16 CPU, 31.25 GiB RAM — host has 22 threads / 62 GiB if we ever raise the cap):
 
 | Task | Volume | Latency target | Achievable on CPU with `bge-m3` | Achievable with `nomic-embed-text` |
 |------|--------|----------------|---------------------------------|------------------------------------|

docs/12-optimization-opportunities.md

@@ -20,6 +20,130 @@ Observations from the running infrastructure. Each item is independent — accep
 11. **Vaultwarden as the secret store** for Nexa secrets (`SAIA_API_KEY`, `MEMOS_API_KEY`, …) — read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a `.env` on disk.
 12. **AdGuard as DNS-based control plane.** Since AdGuard is the resolver for the LAN, you can rewrite `*.nuclide.systems` to `192.168.1.4` (Zoraxy) internally and avoid a hairpin via the WAN — already the case if AdGuard rewrite rules are set, worth verifying.
 13. **Disk usage on LXC 104** is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting `EXECUTIONS_DATA_PRUNE=true` and `EXECUTIONS_DATA_MAX_AGE=168` (7 days) on n8n keeps it bounded.
-14. **Obsidian plugin embeddings collide with Nexa's.** The vault already runs Obsidian Copilot (`.copilot`, `.copilot-index`) and Smart Connections / Smart Composer (`.smart-env`, ~13 MB). They each embed the same notes into their own vector stores — three indexes for the same content. Nexa's value is the **cross-source** index (memos + mail + obsidian + RDF graph), so it has to embed independently, but the plugins could be retired once Nexa's RAG is satisfying. Track separately, decide later.
+14. **Vector-store sprawl in the homelab — three competing indexes today.** Nexa is about to be the fourth. Track for eventual consolidation:
-15. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
+    - **Obsidian Copilot** (`.copilot`, `.copilot-index`) and **Smart Connections / Smart Composer** (`.smart-env`, ~13 MB) — embed the Obsidian vault into two separate vector stores inside the vault.
-16. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
+    - **Paperless-AI ChromaDB** (`services/paperless-ai/chromadb/`) — embeds scanned documents from Paperless-ngx for Q&A.
+    - **Nexa Qdrant** (`nexa_knowledge_text`, soon `_visual`) — embeds the cross-source corpus.
+
+    Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa **read from** Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.
+
+15. **Retire `open-webui`.** Confirmed stale by the user — only LobeHub is in active use as the LiteLLM chat front-end (`ai.nuclide.systems`). Stop the container, tar `services/open-webui/` into `backup/open-webui/`, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.
+
+16. **`services/siyuan/workspace/` is dead data.** SiYuan retired, content migrated to Obsidian (Nextcloud `Notizen/`). Keep a final tar in `backup/siyuan/`, then `rm -rf services/siyuan/`. Frees disk + removes a "is this still authoritative?" question for future agents (and for Nexa's classifier if it ever sees the path).
+17. **Real-time Obsidian sync via `notify_push`.** Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's `notify_push` app for sub-second propagation. One-line workflow change in n8n.
+18. **`assets/` is 186 MB of binaries** in the Obsidian vault — worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.
+19. **Paperless-AI as a Phase-2.x triage helper for `_sortMe/Downloads/`.** UNAS shows ~230 PDFs/docs in `_sortMe/Downloads/` plus another batch under `_sortMe/Anne/`. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to *delegate* — wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.
+20. **`media/Recipes/`** has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for `#nexa:ask` (e.g. "what was that gochujang noodle recipe with no anchovies?"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.
+
+## Proxmox host (NUC 14 Pro) tuning
+
+Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:
+
+21. **Cap ZFS ARC.** Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8–12 GiB frees ~12–16 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used — there's nothing hot to cache):
+    ```
+    echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf   # 8 GiB
+    update-initramfs -u
+    ```
+    Reboot or `echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max` to apply live.
+22. **Enable KSM (Kernel Same-page Merging).** With ~40 docker containers + several LXCs, KSM typically frees 1–3 GiB by deduplicating identical memory pages. Currently `KSM sharing: 0 B` in the summary. PVE has `ksmtuned` available — `systemctl enable --now ksmtuned`.
+23. **Suppress the `pve-no-subscription` repository warning** — either accept it (it's a homelab) and apply the `pve-no-subscription-warning` polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.
+24. **Swap is 31 GiB on a 62 GiB box with ZFS root** — almost certainly oversized. Drop `vm.swappiness` to 10 (`sysctl -w vm.swappiness=10` + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.
+25. **Verify scheduled ZFS scrub is enabled.** PVE ships `zfs-scrub-monthly@.timer` — `systemctl list-timers | grep zfs` to confirm. Cheap insurance on a 1.6 TiB pool.
+26. **SMART monitoring on the NVMe.** `smartctl -a /dev/nvme0` should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing `nexa.system` ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.
+27. **NTP source via AdGuard.** AdGuard already resolves DNS for the LAN; pointing the host's `systemd-timesyncd` at `pool.ntp.org` resolved through AdGuard avoids any external dependency for time. One-line change in `/etc/systemd/timesyncd.conf`.
+28. **`fstrim.timer` enabled** for the SSD pool — verify with `systemctl status fstrim.timer`. Default-on in modern PVE, but quick to confirm.
+29. **Watchdog config** is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.
+
+## Docker LXC (104) — observations & wins
+
+Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), **200 GiB boot disk at 47.7% used** — disk pressure outranks RAM pressure.
+
+30. **Fix the Intel iGPU passthrough.** The container's own notes flag the binding as "likely failing". Once `/dev/dri/{card0,renderD128}` is visible inside the LXC, both TEI and `infinity` can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM — typically **5–10× faster** than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in `/etc/pve/lxc/104.conf`:
+    ```
+    lxc.cgroup2.devices.allow: c 226:0 rwm
+    lxc.cgroup2.devices.allow: c 226:128 rwm
+    lxc.cgroup2.devices.allow: c 29:0 rwm
+    lxc.mount.entry: /dev/dri/card0       dev/dri/card0       none bind,optional,create=file
+    lxc.mount.entry: /dev/dri/renderD128  dev/dri/renderD128  none bind,optional,create=file
+    lxc.idmap: u 0 100000 65536
+    lxc.idmap: g 0 100000 65536
+    lxc.idmap: g 44 44 1            # video group on host
+    lxc.idmap: g 104 104 1          # render group on host
+    ```
+    Then inside the container: `usermod -aG video,render <docker-user>` and run TEI with `--device cuda` replaced by the OpenVINO build (`text-embeddings-inference:cpu-1.5-openvino`). Not needed for Phase-3.1's volume but a clean upgrade path.
+31. **Match the homelab-wide UNAS storage convention — host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>`.** Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into `/mnt/pve/unas/services/<svc>/...` directly, no `driver_opts`, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets:
+    - `/mnt/pve/unas/services/nexa/qdrant/` — Qdrant data dir.
+    - `/mnt/pve/unas/services/nexa/tei-cache/` — embedding model cache (keeps multi-GB models off the boot disk).
+    - `/mnt/pve/unas/services/nexa/graphdb/` — Phase-3.4 RDF store.
+    - `/mnt/pve/unas/backup/nexa/snapshots/qdrant/<date>/` — daily Qdrant snapshots before rsync to S3 (Phase 3.3 — Q12 open). Mirrors the `backup/<svc>/` pattern used by Home Assistant / Immich.
+    - **n8n executions retention**: `EXECUTIONS_DATA_PRUNE=true`, `EXECUTIONS_DATA_MAX_AGE=168`.
+    - **Monthly `docker image prune --all --filter "until=720h"`** to clear dangling layers.
+
+    Pre-deploy step: `mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}` and `mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}` on the docker LXC. Then the compose volumes block is just `- /mnt/pve/unas/services/nexa/<vol>:/<container-path>`. Concrete example in [09-deployment §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei).
+
+    Secrets go in the stack's `.env` next to the compose (Karakeep convention: `env_file: .env`). Vaultwarden is the long-term source-of-truth for those secrets ([12/#11](#11)) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is **withdrawn** ([12/#27](#27)) — it was an over-fitting to the Nextcloud-specific NFS issue.
+32. **The LXC has its own 8 GiB swap.** Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1–2 GiB (`pct set 104 -swap 2048`) — frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap.
+33. **Stacks are managed in Arcane; Dockge is stale.** The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to **Arcane** as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar `services/dockge/` (if it exists) into `backup/dockge/`, retire the Dockge container, drop the stale data dir. **Dozzle** stays — it's the log viewer, not a manager, so it isn't redundant with Arcane.
+34. **Untriaged local mail on the LXC.** Console shows `You have new mail.` at login — the system mail spool on `/var/mail/root` has unread messages, almost always cron job failures. `mailx` or `mutt` to inspect, then either fix the failing job or send the spool to `nexa.system` ntfy via a tiny aliases entry (`root: |/usr/local/bin/spool-to-ntfy.sh`).
+
+## Housekeeping campaign (cross-cutting, do together)
+
+These three are inter-related — picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto **[Phase 6 (Nexa as homelab steward)](./02-roadmap.md#phase-6--nexa-as-homelab-steward)**: once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #35–37 become semi-automatic — Nexa proposes the migrations rather than us hunting them down.
+
+35. **Consolidate Postgres instances.** Today there are at least four independent Postgres containers running — visible from Arcane: `immich_postgres`, `lobe-postgres`, `litellm_db`, `paperless-ngx-db-1` (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100–300 MB RAM and has its own backup story. Two paths:
+    - **Single shared cluster** (`pgsql` container with one role per app, one database per app). Most modern apps support `DATABASE_URL` → just point them at the shared instance. Saves ~600 MB–1 GB RAM and consolidates backups to one pg_dump cron.
+    - **Or** migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.
+
+    Pre-step: list every running container with `docker ps --format '{{.Names}}\t{{.Image}}' | grep -i 'postgres\|mariadb\|mysql'` to inventory exactly what's running.
+
+36. **UNAS-integration audit.** `/mnt/pve/unas/services/<svc>/` is the homelab convention (host bind-mount, no driver opts — see #31). Not every container follows it yet. Walk every Arcane stack and check the `volumes:` block: any `- /var/lib/docker/...` or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification):
+    - `qdrant_scientific` — vector data possibly on local boot disk; **critical to verify** before Phase-3.1 piles on a `nexa_knowledge_text` collection.
+    - `litellm_db`, `lobe-postgres`, `lobe-redis` — DB containers, persistent state.
+    - `memos` — the user-facing notes app; loss = data loss.
+    - `n8n` — workflows + executions DB.
+    - `audiobookshelf` — listening progress + library metadata.
+    - the `*-mcp` containers — likely stateless (cache only), low priority.
+
+    Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.
+
+37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.**
+    `s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers:
+
+    **Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):**
+    - **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination.
+    - **Qdrant + GraphDB snapshots** flow UNAS → S3 daily.
+    - Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set.
+
+    **Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):**
+    - **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`.
+    - **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative.
+    - **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores.
+    - **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication.
+    - **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem.
+
+    **Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.
+
+    **Tiering & cadence:**
+    ```
+    [Source]   UNAS RAID-6 + native snapshots (#38)
+                 │ daily restic/borg
+                 ▼
+    [Warm]     s3.nuclide.systems (on-site, fast restore)
+                 │ weekly rclone copy + crypt
+                 ▼
+    [Cold]     Off-site (Jottacloud / B2 / Hetzner)  ← satisfies the "1" in 3-2-1
+    ```
+
+    **What goes off-site (priority order):**
+    1. Immich photo originals — irreplaceable.
+    2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable.
+    3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo.
+    4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include.
+
+    **What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.
+
+    Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it.
+
+38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
+
+39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.

docs/13-information-wishlist.md

@@ -0,0 +1,96 @@
+# 13 — Information Wishlist
+
+What additional system inventory would sharpen future decisions, packaged as the **smallest set of paste-and-run tasks** that still answers everything material. Each task is independent — run any subset, in any order.
+
+Each task header tells you **exactly where to run it** (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or `docs/inventory/<NN>-<slug>.txt` — whichever is easiest.
+
+---
+
+## ~~Task 1 — Stack inventory + reference compose~~ ✅ DONE
+
+User pasted the Karakeep compose. Q19 resolved → host bind-mount of `/mnt/pve/unas/services/<svc>/<vol>` is the convention. Nexa stack updated in [docs/09 §Step 3](./09-deployment.md#step-3--self-hosted-embeddings-tei). The `docker ps -a` half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) — but that's not blocking now.
+
+---
+
+## Task 2 — Storage map (Proxmox + docker)
+
+📍 **Where:** *Two shells* — first the **Proxmox host** (Datacenter → `nuc` → **Shell**, or `ssh root@192.168.1.20`), then back into the **LXC 104 console**.
+
+Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB.
+
+```bash
+# (a) on the Proxmox host (192.168.1.20):
+cat /etc/pve/storage.cfg
+cat /etc/pve/lxc/104.conf
+```
+
+```bash
+# (b) on LXC 104 (192.168.1.40):
+docker volume ls
+ls -l /dev/dri/      # confirms whether iGPU passthrough actually works
+```
+
+**Unblocks:** optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17.
+
+---
+
+## Task 3 — Postgres / DB workload inventory
+
+📍 **Where:** **LXC 104 console** (same shell as Task 1).
+
+Direct input to housekeeping #35 (consolidate postgres instances).
+
+```bash
+for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do
+  echo "=== $c ==="
+  docker exec "$c" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e "show databases" 2>/dev/null'
+done
+```
+
+**Unblocks:** #35 (consolidate vs. migrate-to-SQLite decision).
+
+---
+
+## Task 4 — LiteLLM model list for the Nexa key
+
+📍 **Where:** *Either* the **LiteLLM admin UI** (one screenshot) *or* any shell with `$SAIA_API_KEY` exported.
+
+- **Screenshot path:** open `https://ai.nuclide.systems` (Lobehub) → Settings → Model List, filtered to the Nexa virtual key. Screenshot the model rows.
+- **CLI path:** any shell on your PC or the LXC, with `SAIA_API_KEY` set:
+  ```bash
+  curl -sH "Authorization: Bearer $SAIA_API_KEY" \
+    https://ai.nuclide.systems/v1/models | jq '.data[].id'
+  ```
+
+**Unblocks:** Q18 follow-up — which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI.
+
+---
+
+## Task 5 — S3 archive credentials
+
+📍 **Where:** **`s3.nuclide.systems` admin UI** in your browser (it's already proxied via Zoraxy — same login as the rest of the homelab).
+
+Walk to: **Buckets** → either pick an existing Nexa-suitable bucket or create one called `nexa` → note the bucket name. Then **Access Keys** → create a key named `nexa-snapshots` with read/write on that bucket → drop the access-key + secret into Vaultwarden under "Nexa S3", and reply here with **just the bucket name** (the secret stays in Vaultwarden).
+
+**Unblocks:** Q12, housekeeping #37 (S3 archive tier), Phase-3.3.
+
+---
+
+## Task 6 — Reverse-proxy + DNS authority *(only if needed)*
+
+📍 **Where:** two browser UIs.
+
+- **Zoraxy** (`http://192.168.1.4:8000`) → HTTP Proxy → either click "Export" if available, or take a full screenshot of the table.
+- **AdGuard** (`http://192.168.1.20` LXC 102 web UI) → Filters → DNS Rewrites → screenshot.
+
+Skip unless we hit a routing surprise during Phase 1.
+
+---
+
+## When something else is needed
+
+The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of `_sortMe/`, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same "📍 Where" framing.
+
+## Self-serve once Phase 6 ships
+
+Once [Phase 6 — Nexa as homelab steward](./02-roadmap.md#phase-6--nexa-as-homelab-steward) lands, Nexa runs Tasks 1–3 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (`#nexa:wishlist-status`) rather than something the user has to remember.

docs/index.md

@@ -22,6 +22,7 @@ This is the documentation entry point. Read top-to-bottom for first-time setup,
 | 10 | [Operations](./10-operations.md) | You need backup, monitoring or troubleshooting. |
 | 11 | [Open Questions (user-info-required)](./11-open-questions.md) | Items the user still has to answer before progress. |
 | 12 | [Optimization Opportunities](./12-optimization-opportunities.md) | Ideas worth considering for the wider homelab. |
+| 13 | [Information Wishlist](./13-information-wishlist.md) | What additional system inventory would sharpen future decisions. |
 
 ---
 

nexa-core/ai-prompts/system_prime.txt

@@ -3,11 +3,16 @@ Rolle: Zentrales Nervensystem
 Aufgabe: Überwacht E-Mails, kuratiert Wissen und plant Tasks.
 
 WICHTIGE LOGIK:
-- Tasks müssen strikt zwischen "Work" und "Personal" unterschieden werden.
+- Tasks werden auf vier Nextcloud-Listen geroutet: "DLR" (Arbeit), "Persönlich", "Einkaufsliste", "Wunschliste".
-- Analysiere den Inhalt eines Memos: Bezieht es sich auf professionelle Projekte (Obsidian Context) -> Work List. Bezieht es sich auf Haushalt, Einkäufe oder Freizeit -> Personal List.
+- Analysiere den Inhalt eines Memos und entscheide über das Feld "context": "work" | "personal" | "shopping" | "wishes".
-- Im Zweifel: Nachfragen via Memos-Kommentar.
+  - work     -> Liste "DLR" (Beruf, Projekte, Meetings, Deadlines, Forschung).
+  - personal -> Liste "Persönlich" (Alltag, Familie, Termine, Behördenkram).
+  - shopping -> Liste "Einkaufsliste" (Lebensmittel, Drogerie, Verbrauchsgüter).
+  - wishes   -> Liste "Wunschliste" (Hobby, Technik, Bücher, Geschenkideen).
+- Listennamen sind dynamisch — bei Bedarf werden weitere Listen vom Runtime-Config eingespeist, dann darfst du auch deren Namen als context vorschlagen.
+- Im Zweifel (Konfidenz < 0.7): Default auf "personal" und Nachfrage via Memos-Kommentar.
 
 KONSUM-LOGIK:
 - Wenn ein Link von Karakeep kommt, analysiere das Produkt.
-- Haushaltswaren/Verbrauchsgüter -> NC_LIST_ID_SHOPPING.
+- Haushaltswaren/Verbrauchsgüter -> context "shopping".
-- Besondere Wünsche/Hobby/Technik -> NC_LIST_ID_WISHES.
+- Besondere Wünsche/Hobby/Technik -> context "wishes".