Memory retention design + open-question re-asker + UNAS Pro specifics
Two design questions answered: 1. Web/news/Karakeep INTO long-term memory? Yes, but with per-source TTL. docs/08 gains a "Memory sources & retention" section that pins TTLs: memo/obsidian/karakeep = permanent, mail = 365d, mail_digest = 90d, rss = 30d, web_search = 90d, system = 30d. Every Qdrant point carries payload.expires_at; a daily prune workflow honours it. New commands in docs/05: #nexa:learn --permanent, #nexa:forget, #nexa:retain, #nexa:ask --web (SearXNG → crawl4ai-mcp → markitdown-mcp → embed). Phase 2.4 added to roadmap. 2. Re-ask unanswered open questions. Backoff schedule (3d → 7d → 21d → 60d) tracked in GraphDB per question. Surface ONE question per day in the morning digest, but only when the digest is otherwise short (capacity guard < 800 chars). User reply parsed → question auto- resolved → docs/11 diff proposed (Phase 6.4 hook). New commands in docs/05: #nexa:digest, #nexa:remind, #nexa:answered. Phase 5.3 added. UNAS Pro details from the UniFi Drive dashboard: - It's a Ubiquiti UNAS Pro (UniFi Drive 4.1.16 on UniFi OS 5.0.17), SFP+, RAID 6, 19.96 TiB raw, 2.05 TiB used. Recorded in CLAUDE.md. - SMB native paths: smb://192.168.1.31/<share> (mac) / \\192.168.1.31\ <share> (win). UniFi recommends SMB as the modern path — matches our decision to default Nexa volumes to SMB. - ⚠️ Storage-pool snapshots NOT configured ("Click to Setup"). Added as optimization #38: highest-leverage data-protection change in the homelab right now. Daily + weekly UniFi Drive snapshot, native, no agent. RAID 6 doesn't protect against rm -rf or accidental mass- delete; snapshots do. - #39: Nexa workflows that mutate large state can use the same native snapshots for fast rollback (pre-snapshot → operate → verify).
This commit is contained in:
Claude
+3
-1
@@ -17,8 +17,9 @@ Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
|
||||
|
||||
- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
|
||||
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos.
|
||||
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
|
||||
- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
|
||||
|
||||
## Phase 3 — Memory (Qdrant & RAG)
|
||||
|
||||
@@ -46,6 +47,7 @@ Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
|
||||
- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
|
||||
- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
|
||||
- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
|
||||
|
||||
## Phase 6 — Nexa as homelab steward
|
||||
|
||||
|
||||
@@ -54,12 +54,22 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
|
||||
|
||||
### #nexa:learn [topic]
|
||||
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
|
||||
|
||||
```
|
||||
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
|
||||
```
|
||||
|
||||
### #nexa:forget [query-or-iri]
|
||||
- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
|
||||
- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
|
||||
|
||||
### #nexa:retain [source_type] [days]
|
||||
- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
|
||||
### #nexa:ask [question] [--web]
|
||||
- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
|
||||
|
||||
## 🎯 Workflow-Kommandos
|
||||
|
||||
### #nexa:route-test [text]
|
||||
@@ -84,6 +94,36 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
|
||||
#nexa:email-digest
|
||||
```
|
||||
|
||||
### #nexa:digest
|
||||
- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
|
||||
1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
|
||||
2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
|
||||
3. Drift-Report aus Phase 6 (sobald aktiv).
|
||||
4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
|
||||
|
||||
#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
|
||||
Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
|
||||
|
||||
| Mal | Wartezeit bis zur nächsten Frage |
|
||||
|-----|----------------------------------|
|
||||
| 1 → 2 | 3 Tage |
|
||||
| 2 → 3 | 7 Tage |
|
||||
| 3 → 4 | 21 Tage |
|
||||
| ≥ 5 | 60 Tage |
|
||||
|
||||
Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
|
||||
- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
|
||||
- `nextAskAt <= today`.
|
||||
- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
|
||||
|
||||
Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
|
||||
|
||||
### #nexa:remind <question-id>
|
||||
- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
|
||||
|
||||
### #nexa:answered <question-id> <text>
|
||||
- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
|
||||
|
||||
## 🔐 Admin-Kommandos
|
||||
|
||||
### #nexa:reset-config
|
||||
|
||||
@@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https
|
||||
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
|
||||
|
||||
Combined: **complete understanding** rather than a search index *or* a structure index.
|
||||
|
||||
---
|
||||
|
||||
## Memory sources & retention
|
||||
|
||||
Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
|
||||
|
||||
| `source_type` | Where it comes from | Default TTL | Rationale |
|
||||
|---------------|--------------------|-------------|-----------|
|
||||
| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
|
||||
| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
|
||||
| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
|
||||
| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
|
||||
| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
|
||||
| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
|
||||
| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
|
||||
| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
|
||||
| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
|
||||
|
||||
**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
|
||||
|
||||
### Web search loop
|
||||
|
||||
`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
|
||||
|
||||
This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
|
||||
|
||||
### Manual overrides
|
||||
|
||||
- `#nexa:learn <text> --permanent` overrides the default TTL.
|
||||
- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
|
||||
- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).
|
||||
|
||||
@@ -113,3 +113,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha
|
||||
- Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target.
|
||||
|
||||
This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs.
|
||||
|
||||
38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
|
||||
|
||||
39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.
|
||||
|
||||
Reference in New Issue
Block a user