Memory retention design + open-question re-asker + UNAS Pro specifics

Two design questions answered:

1. Web/news/Karakeep INTO long-term memory? Yes, but with per-source TTL.
   docs/08 gains a "Memory sources & retention" section that pins TTLs:
   memo/obsidian/karakeep = permanent, mail = 365d, mail_digest = 90d,
   rss = 30d, web_search = 90d, system = 30d. Every Qdrant point carries
   payload.expires_at; a daily prune workflow honours it. New commands
   in docs/05: #nexa:learn --permanent, #nexa:forget, #nexa:retain,
   #nexa:ask --web (SearXNG → crawl4ai-mcp → markitdown-mcp → embed).
   Phase 2.4 added to roadmap.

2. Re-ask unanswered open questions. Backoff schedule (3d → 7d → 21d →
   60d) tracked in GraphDB per question. Surface ONE question per day
   in the morning digest, but only when the digest is otherwise short
   (capacity guard < 800 chars). User reply parsed → question auto-
   resolved → docs/11 diff proposed (Phase 6.4 hook). New commands in
   docs/05: #nexa:digest, #nexa:remind, #nexa:answered. Phase 5.3 added.

UNAS Pro details from the UniFi Drive dashboard:
- It's a Ubiquiti UNAS Pro (UniFi Drive 4.1.16 on UniFi OS 5.0.17),
  SFP+, RAID 6, 19.96 TiB raw, 2.05 TiB used. Recorded in CLAUDE.md.
- SMB native paths: smb://192.168.1.31/<share> (mac) / \\192.168.1.31\
  <share> (win). UniFi recommends SMB as the modern path — matches our
  decision to default Nexa volumes to SMB.
- ⚠️ Storage-pool snapshots NOT configured ("Click to Setup"). Added as
  optimization #38: highest-leverage data-protection change in the
  homelab right now. Daily + weekly UniFi Drive snapshot, native, no
  agent. RAID 6 doesn't protect against rm -rf or accidental mass-
  delete; snapshots do.
- #39: Nexa workflows that mutate large state can use the same native
  snapshots for fast rollback (pre-snapshot → operate → verify).
This commit is contained in:
Claude
2026-05-05 04:45:29 +00:00
parent 74368100a3
commit e8f0d9a2c2
5 changed files with 83 additions and 4 deletions
+32
View File
@@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
Combined: **complete understanding** rather than a search index *or* a structure index.
---
## Memory sources & retention
Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
| `source_type` | Where it comes from | Default TTL | Rationale |
|---------------|--------------------|-------------|-----------|
| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
### Web search loop
`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
### Manual overrides
- `#nexa:learn <text> --permanent` overrides the default TTL.
- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).