From e8f0d9a2c234dd0562328658f7ae1517b1f540e9 Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Tue, 5 May 2026 04:45:29 +0000
Subject: [PATCH] Memory retention design + open-question re-asker + UNAS Pro
 specifics
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two design questions answered:

1. Web/news/Karakeep INTO long-term memory? Yes, but with per-source TTL.
   docs/08 gains a "Memory sources & retention" section that pins TTLs:
   memo/obsidian/karakeep = permanent, mail = 365d, mail_digest = 90d,
   rss = 30d, web_search = 90d, system = 30d. Every Qdrant point carries
   payload.expires_at; a daily prune workflow honours it. New commands
   in docs/05: #nexa:learn --permanent, #nexa:forget, #nexa:retain,
   #nexa:ask --web (SearXNG → crawl4ai-mcp → markitdown-mcp → embed).
   Phase 2.4 added to roadmap.

2. Re-ask unanswered open questions. Backoff schedule (3d → 7d → 21d →
   60d) tracked in GraphDB per question. Surface ONE question per day
   in the morning digest, but only when the digest is otherwise short
   (capacity guard < 800 chars). User reply parsed → question auto-
   resolved → docs/11 diff proposed (Phase 6.4 hook). New commands in
   docs/05: #nexa:digest, #nexa:remind, #nexa:answered. Phase 5.3 added.

UNAS Pro details from the UniFi Drive dashboard:
- It's a Ubiquiti UNAS Pro (UniFi Drive 4.1.16 on UniFi OS 5.0.17),
  SFP+, RAID 6, 19.96 TiB raw, 2.05 TiB used. Recorded in CLAUDE.md.
- SMB native paths: smb://192.168.1.31/<share> (mac) / \\192.168.1.31\
  <share> (win). UniFi recommends SMB as the modern path — matches our
  decision to default Nexa volumes to SMB.
- ⚠️ Storage-pool snapshots NOT configured ("Click to Setup"). Added as
  optimization #38: highest-leverage data-protection change in the
  homelab right now. Daily + weekly UniFi Drive snapshot, native, no
  agent. RAID 6 doesn't protect against rm -rf or accidental mass-
  delete; snapshots do.
- #39: Nexa workflows that mutate large state can use the same native
  snapshots for fast rollback (pre-snapshot → operate → verify).
---
 CLAUDE.md                             |  5 ++--
 docs/02-roadmap.md                    |  4 ++-
 docs/05-command-system.md             | 42 ++++++++++++++++++++++++++-
 docs/08-graphrag-architecture.md      | 32 ++++++++++++++++++++
 docs/12-optimization-opportunities.md |  4 +++
 5 files changed, 83 insertions(+), 4 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 94d8b8f..d52f0f7 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -15,9 +15,10 @@ This file tells future automated runs what they need to know about this repo.
   - Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
   - LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
   - LXC 103 backrest — backup orchestration.
-  - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways:
+  - LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). Reachable two ways:
   - **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts.
-  - **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely.
+  - **SMB** — **default for Nexa per-container volumes** (Ubiquiti also recommends SMB as the modern path; the URL is `smb://192.168.1.31/<share>` macOS / `\\192.168.1.31\<share>` Windows). The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely.
+  - **Storage-layer snapshots are NOT configured** on the UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, but not "the whole pool, last Tuesday". See [docs/12 #38](./docs/12-optimization-opportunities.md#38).
 
   **UNAS layout conventions** (homelab-wide, all docker containers follow them):
   - **`services/<svc>/` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout.
diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md
index fff7ebf..b4995c2 100644
--- a/docs/02-roadmap.md
+++ b/docs/02-roadmap.md
@@ -17,8 +17,9 @@ Iterative build-out, value-first. Each phase is shippable on its own.
 **Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
 
 - [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
-- [ ] **2.2 RSS morning digest** — curated daily summary in Memos.
+- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
 - [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
+- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
 
 ## Phase 3 — Memory (Qdrant & RAG)
 
@@ -46,6 +47,7 @@ Iterative build-out, value-first. Each phase is shippable on its own.
 
 - [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
 - [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
+- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
 
 ## Phase 6 — Nexa as homelab steward
 
diff --git a/docs/05-command-system.md b/docs/05-command-system.md
index ae81385..12701cc 100644
--- a/docs/05-command-system.md
+++ b/docs/05-command-system.md
@@ -54,12 +54,22 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 
 ### #nexa:learn [topic]
 - **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
-- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
+- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
 
 ```
 #nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
 ```
 
+### #nexa:forget [query-or-iri]
+- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
+- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
+
+### #nexa:retain [source_type] [days]
+- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
+
+### #nexa:ask [question] [--web]
+- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
+
 ## 🎯 Workflow-Kommandos
 
 ### #nexa:route-test [text]
@@ -84,6 +94,36 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
 #nexa:email-digest
 ```
 
+### #nexa:digest
+- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
+  1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
+  2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
+  3. Drift-Report aus Phase 6 (sobald aktiv).
+  4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
+
+#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
+Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
+
+| Mal | Wartezeit bis zur nächsten Frage |
+|-----|----------------------------------|
+| 1 → 2 | 3 Tage |
+| 2 → 3 | 7 Tage |
+| 3 → 4 | 21 Tage |
+| ≥ 5  | 60 Tage |
+
+Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
+- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
+- `nextAskAt <= today`.
+- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
+
+Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
+
+### #nexa:remind <question-id>
+- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
+
+### #nexa:answered <question-id> <text>
+- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
+
 ## 🔐 Admin-Kommandos
 
 ### #nexa:reset-config
diff --git a/docs/08-graphrag-architecture.md b/docs/08-graphrag-architecture.md
index 110b24c..2c7d624 100644
--- a/docs/08-graphrag-architecture.md
+++ b/docs/08-graphrag-architecture.md
@@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https
 | "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
 
 Combined: **complete understanding** rather than a search index *or* a structure index.
+
+---
+
+## Memory sources & retention
+
+Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
+
+| `source_type` | Where it comes from | Default TTL | Rationale |
+|---------------|--------------------|-------------|-----------|
+| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
+| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
+| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
+| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
+| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
+| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
+| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
+| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
+| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
+
+**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
+
+### Web search loop
+
+`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
+
+This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
+
+### Manual overrides
+
+- `#nexa:learn <text> --permanent` overrides the default TTL.
+- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
+- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).
diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md
index e22ef3c..2cee0a7 100644
--- a/docs/12-optimization-opportunities.md
+++ b/docs/12-optimization-opportunities.md
@@ -113,3 +113,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha
     - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target.
 
     This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs.
+
+38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
+
+39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.