Memory retention design + open-question re-asker + UNAS Pro specifics
Two design questions answered: 1. Web/news/Karakeep INTO long-term memory? Yes, but with per-source TTL. docs/08 gains a "Memory sources & retention" section that pins TTLs: memo/obsidian/karakeep = permanent, mail = 365d, mail_digest = 90d, rss = 30d, web_search = 90d, system = 30d. Every Qdrant point carries payload.expires_at; a daily prune workflow honours it. New commands in docs/05: #nexa:learn --permanent, #nexa:forget, #nexa:retain, #nexa:ask --web (SearXNG → crawl4ai-mcp → markitdown-mcp → embed). Phase 2.4 added to roadmap. 2. Re-ask unanswered open questions. Backoff schedule (3d → 7d → 21d → 60d) tracked in GraphDB per question. Surface ONE question per day in the morning digest, but only when the digest is otherwise short (capacity guard < 800 chars). User reply parsed → question auto- resolved → docs/11 diff proposed (Phase 6.4 hook). New commands in docs/05: #nexa:digest, #nexa:remind, #nexa:answered. Phase 5.3 added. UNAS Pro details from the UniFi Drive dashboard: - It's a Ubiquiti UNAS Pro (UniFi Drive 4.1.16 on UniFi OS 5.0.17), SFP+, RAID 6, 19.96 TiB raw, 2.05 TiB used. Recorded in CLAUDE.md. - SMB native paths: smb://192.168.1.31/<share> (mac) / \\192.168.1.31\ <share> (win). UniFi recommends SMB as the modern path — matches our decision to default Nexa volumes to SMB. - ⚠️ Storage-pool snapshots NOT configured ("Click to Setup"). Added as optimization #38: highest-leverage data-protection change in the homelab right now. Daily + weekly UniFi Drive snapshot, native, no agent. RAID 6 doesn't protect against rm -rf or accidental mass- delete; snapshots do. - #39: Nexa workflows that mutate large state can use the same native snapshots for fast rollback (pre-snapshot → operate → verify).
This commit is contained in:
Claude
@@ -15,9 +15,10 @@ This file tells future automated runs what they need to know about this repo.
|
||||
- Steady state: ~32 GiB used (≈24 GiB of which is ZFS ARC, tunable via `zfs_arc_max`), CPU load <2.0, IO delay <0.05%.
|
||||
- LXC 102 dns (AdGuard) — internal DNS, rewrites for `*.nuclide.systems`.
|
||||
- LXC 103 backrest — backup orchestration.
|
||||
- LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage on the UNAS (`192.168.1.31`)** is reachable two ways:
|
||||
- LXC 104 docker — main docker host at `192.168.1.40`, hostname `docker`, OS Debian 13. **Unprivileged**, originally provisioned from the Dockge helper-script template, but **Arcane is the active docker manager** today (Dockge is stale, slated for retirement — see [docs/12 #33](./docs/12-optimization-opportunities.md#33)). **Dozzle** is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). **Storage = Ubiquiti UNAS Pro at `192.168.1.31`** (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). Reachable two ways:
|
||||
- **NFS** (Proxmox host-mounted at `/mnt/pve/unas`, 19.4 TiB / 16.3 TiB free) — fine for host-side admin tasks and read-mostly mounts.
|
||||
- **SMB** — **default for Nexa per-container volumes**. The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely.
|
||||
- **SMB** — **default for Nexa per-container volumes** (Ubiquiti also recommends SMB as the modern path; the URL is `smb://192.168.1.31/<share>` macOS / `\\192.168.1.31\<share>` Windows). The user already hit NFS+unprivileged-LXC write-permission issues with Nextcloud; don't repeat that. Mount SMB shares directly as docker volumes inside the stack with explicit `username=`/`uid=`/`gid=` — sidesteps uid-mapping entirely.
|
||||
- **Storage-layer snapshots are NOT configured** on the UNAS Pool 1 ("Click to Setup" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer — Backrest covers files, but not "the whole pool, last Tuesday". See [docs/12 #38](./docs/12-optimization-opportunities.md#38).
|
||||
|
||||
**UNAS layout conventions** (homelab-wide, all docker containers follow them):
|
||||
- **`services/<svc>/` is the general docker config store** — every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. **Stale (retire, do not consume):** `services/siyuan/` (migrated to Obsidian), `services/open-webui/` (unused — LobeHub is the active LLM UI), and `services/dockge/` if it exists (retired in favor of Arcane). **Nexa MUST follow the same pattern: `services/nexa/{qdrant,tei-cache,graphdb,...}`.** Don't invent a parallel layout.
|
||||
|
||||
+3
-1
@@ -17,8 +17,9 @@ Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
**Milestone:** Nexa distinguishes work tasks from personal tasks without manual tags.
|
||||
|
||||
- [ ] **2.1 Email butler** — filter & summarize the ~20 daily personal mails.
|
||||
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos.
|
||||
- [ ] **2.2 RSS morning digest** — curated daily summary in Memos. Items also enter `nexa_knowledge_text` with `source_type=rss, ttl=30d` (see [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
- [ ] **2.3 Bluesky antenna** — store liked posts as knowledge snippets.
|
||||
- [ ] **2.4 On-demand web fetch** — `#nexa:ask --web` runs a SearXNG query through `redis-searxng`, fetches top hits via `crawl4ai-mcp` + `markitdown-mcp`, embeds and answers. Pages stay in memory with `ttl=90d` so the next related question doesn't re-fetch.
|
||||
|
||||
## Phase 3 — Memory (Qdrant & RAG)
|
||||
|
||||
@@ -46,6 +47,7 @@ Iterative build-out, value-first. Each phase is shippable on its own.
|
||||
|
||||
- [ ] **5.1 HA Voice** — Wake-word integration, Wyoming protocol.
|
||||
- [ ] **5.2 System monitoring** — Proxmox / Backrest alerts into the Memos system feed via ntfy.
|
||||
- [ ] **5.3 Morning digest with self-asking re-loop** — daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from [docs/11](./11-open-questions.md) using a backoff schedule (3 d → 7 d → 21 d → 60 d). Answers parsed from the user's reply auto-resolve the question and emit a `docs/11` diff for review (Phase 6.4 hook). See [docs/05 § #nexa:digest](./05-command-system.md).
|
||||
|
||||
## Phase 6 — Nexa as homelab steward
|
||||
|
||||
|
||||
@@ -54,12 +54,22 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
|
||||
|
||||
### #nexa:learn [topic]
|
||||
- **Beschreibung**: Explizit neue Information in Qdrant speichern (mit Tags)
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md`
|
||||
- **Optional**: `--tag=architecture` `--source=obsidian/notes/arch.md` `--permanent` (überschreibt Default-TTL)
|
||||
|
||||
```
|
||||
#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture
|
||||
```
|
||||
|
||||
### #nexa:forget [query-or-iri]
|
||||
- **Beschreibung**: Löscht passende Einträge aus Qdrant + GraphDB sofort.
|
||||
- **Beispiel**: `#nexa:forget --source=rss --older-than=14d`
|
||||
|
||||
### #nexa:retain [source_type] [days]
|
||||
- **Beschreibung**: Setzt die Default-Retention für einen Source-Type (Default-Werte siehe [08 § Memory sources & retention](./08-graphrag-architecture.md#memory-sources--retention)).
|
||||
|
||||
### #nexa:ask [question] [--web]
|
||||
- **Beschreibung**: Semantische Suche in Qdrant. Mit `--web` erweitert Nexa die Suche um eine SearXNG-Abfrage (`redis-searxng`), holt die Top-3 Treffer via `crawl4ai-mcp` + `markitdown-mcp`, embedded sie und nutzt das gemeinsame Material für die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).
|
||||
|
||||
## 🎯 Workflow-Kommandos
|
||||
|
||||
### #nexa:route-test [text]
|
||||
@@ -84,6 +94,36 @@ Nexa "hört" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mi
|
||||
#nexa:email-digest
|
||||
```
|
||||
|
||||
### #nexa:digest
|
||||
- **Beschreibung**: Tägliches Morgen-Digest. Läuft automatisch (07:00). Inhalt:
|
||||
1. Kurze Mail-Zusammenfassung (siehe `#nexa:email-digest`).
|
||||
2. Anstehende Tasks heute (DLR + Persönlich, nach Urgency).
|
||||
3. Drift-Report aus Phase 6 (sobald aktiv).
|
||||
4. **Re-ask Block** — siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.
|
||||
|
||||
#### Re-ask der offenen Fragen (`docs/11-open-questions.md`)
|
||||
Nexa führt im GraphDB pro offener Frage `nexa:askedCount`, `nexa:lastAskedAt`, `nexa:nextAskAt`. Backoff-Schema:
|
||||
|
||||
| Mal | Wartezeit bis zur nächsten Frage |
|
||||
|-----|----------------------------------|
|
||||
| 1 → 2 | 3 Tage |
|
||||
| 2 → 3 | 7 Tage |
|
||||
| 3 → 4 | 21 Tage |
|
||||
| ≥ 5 | 60 Tage |
|
||||
|
||||
Im Morgen-Digest wird **eine** offene Frage angehängt — und zwar nur wenn:
|
||||
- Der Digest sonst < 800 Zeichen lang wäre (Capacity-Guard, damit es nicht nervt).
|
||||
- `nextAskAt <= today`.
|
||||
- Bevorzugt die Frage mit dem **kleinsten** `askedCount` (zuerst neue Fragen, alte selten).
|
||||
|
||||
Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in `docs/11` als resolved (Phase 6.4 — Self-update of docs) und committet einen Diff zur Review.
|
||||
|
||||
### #nexa:remind <question-id>
|
||||
- Manuelles Triggern einer Re-Ask, ignoriert Backoff. Nützlich für Rückfragen mitten im Tag.
|
||||
|
||||
### #nexa:answered <question-id> <text>
|
||||
- Beantwortet eine offene Frage explizit. Setzt `resolved` im GraphDB und schlägt einen `docs/11`-Diff vor.
|
||||
|
||||
## 🔐 Admin-Kommandos
|
||||
|
||||
### #nexa:reset-config
|
||||
|
||||
@@ -252,3 +252,35 @@ Clears the named graph and replays Obsidian + Memos. SPARQL: `CLEAR GRAPH <https
|
||||
| "All JWT-related open work" | ✅ semantic | ✅ crisp | both |
|
||||
|
||||
Combined: **complete understanding** rather than a search index *or* a structure index.
|
||||
|
||||
---
|
||||
|
||||
## Memory sources & retention
|
||||
|
||||
Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries `payload.source_type` and `payload.expires_at` (epoch seconds, or `null` for permanent). A daily prune workflow runs `DELETE WHERE expires_at < NOW()` on each collection and mirrors the deletion in GraphDB.
|
||||
|
||||
| `source_type` | Where it comes from | Default TTL | Rationale |
|
||||
|---------------|--------------------|-------------|-----------|
|
||||
| `memo` | Memos webhook | **permanent** | User-authored, low volume, high signal. |
|
||||
| `obsidian` | Nextcloud `Notizen/` via WebDAV | **permanent** | User-authored knowledge base. |
|
||||
| `mail` | Nextcloud Mail (single account) | 365 d | Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. |
|
||||
| `mail_digest` | Daily digest output | 90 d | Summarised content; the source mails persist longer. |
|
||||
| `karakeep` | Karakeep saved links | **permanent** | User explicitly bookmarked. |
|
||||
| `rss` | Phase-2.2 morning digest feed items | 30 d | News signal decays fast; keep recent for "what was that article last week?". |
|
||||
| `web_search` | On-demand fetch via crawl4ai-mcp / markitdown-mcp during `#nexa:ask` | 90 d | Useful for "what did we look at last quarter?" but not eternal. |
|
||||
| `system` | Backrest / Proxmox / n8n alerts via `nexa.system` ntfy topic | 30 d | Operational telemetry; old alerts have little RAG value. |
|
||||
| `task` | Nextcloud Tasks ↔ GraphDB sync | until task deleted | Mirrors source-of-truth. |
|
||||
|
||||
**External sources go through the same pipeline as memos** — fetch → markitdown-mcp → embed via TEI → upsert into `nexa_knowledge_text` with the appropriate `source_type` + `expires_at`. The graph node carries `nexa:source`, `nexa:fetchedAt`, `nexa:sourceUri`, and `nexa:contentHash` for de-dup (so the same article fetched twice doesn't create two points).
|
||||
|
||||
### Web search loop
|
||||
|
||||
`#nexa:ask` first searches existing memory. If the merged confidence is below a threshold (or the user adds `--web` to the command), Nexa runs a SearXNG query through `redis-searxng`, picks the top 3 results, fetches them through `crawl4ai-mcp` + `markitdown-mcp`, embeds the cleaned markdown, and **answers from the augmented context**. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.
|
||||
|
||||
This means the homelab's existing `*-mcp` containers are part of Nexa's data plane, not just decoration — see [docs/12 #8](./12-optimization-opportunities.md#8).
|
||||
|
||||
### Manual overrides
|
||||
|
||||
- `#nexa:learn <text> --permanent` overrides the default TTL.
|
||||
- `#nexa:forget <iri-or-search>` triggers an immediate Qdrant delete + GraphDB `DELETE WHERE { ?n nexa:vectorId "..." . }`.
|
||||
- `#nexa:retain <source_type> <days>` rewrites the default for that source type (stored in the `_config` namespace, picked up by the next prune run).
|
||||
|
||||
@@ -113,3 +113,7 @@ These three are inter-related — picking them up as one campaign is cheaper tha
|
||||
- Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target.
|
||||
|
||||
This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs.
|
||||
|
||||
38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
|
||||
|
||||
39. **Use UNAS native snapshots for fast Nexa rollback** once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can `pre-snapshot → operate → verify → release` against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.
|
||||
|
||||
Reference in New Issue
Block a user