Reorganize /docs with numbered TOC; deployment rewrite for actual homelab

- /docs/ now contains 12 numbered guides + index.md (navigator). - Old duplicates removed: nexa-core/doc/, DEPLOYMENT.md, PLAN.md, config/{classification_logic,workflows_spec}.md. - 09-deployment.md targets the running infra (LXC 104 docker host, Zoraxy at 192.168.1.4, existing Memos/n8n/LiteLLM/Nextcloud/Qdrant) rather than spinning a parallel stack; minimal-config approach. - 11-open-questions.md tracks user-info-required blockers. - 12-optimization-opportunities.md captures homelab tweaks. - CLAUDE.md added so future agent runs know repo conventions and infra.
2026-05-04 21:18:28 +00:00
parent 18d2360ad9
commit 9b18e2710d
19 changed files with 618 additions and 754 deletions
@@ -0,0 +1,95 @@
+# 10 — Operations
+
+Day-2 concerns. Backups, monitoring, troubleshooting.
+
+## Backups
+
+Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items:
+
+| What | How | Frequency |
+|------|-----|-----------|
+| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
+| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) |
+| Memos DB | Backrest snapshot of Memos data dir | already covered |
+| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered |
+| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
+
+## Monitoring
+
+| Signal | Source | Sink |
+|--------|--------|------|
+| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks | Memos system feed via ntfy |
+| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic |
+| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest |
+| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest |
+| Backrest run status | Backrest webhook | ntfy |
+
+A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo.
+
+## Troubleshooting
+
+### Memos webhook not firing
+
+```bash
+# is the Memos webhook config still pointing at n8n?
+curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \
+  https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'
+
+# does n8n still expose the path?
+curl -i https://n8n.nuclide.systems/webhook/memos     # expect 200/405, never 404
+```
+
+If 404 → workflow is inactive in n8n. Activate.
+
+### LiteLLM 401 / 429
+
+- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential.
+- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.
+
+### Qdrant `nexa_knowledge` empty
+
+```bash
+# is the indexing workflow active?
+docker logs nexa-n8n 2>&1 | grep memos_bridge | tail
+
+# manual index test
+curl -X POST https://memos.nuclide.systems/api/v1/memos \
+  -H "Authorization: Bearer $MEMOS_API_KEY" \
+  -d '{"content":"manual probe #nexa"}'
+
+# point count
+curl -s -H "api-key: $QDRANT_API_KEY" \
+  $QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'
+```
+
+### Wrong list routing (Work vs Personal)
+
+1. Run `#nexa:route-test <text>` and check the confidence value.
+2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)).
+3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change.
+
+### "Where did Nexa store this?"
+
+```bash
+# state dump
+curl -X POST https://memos.nuclide.systems/api/v1/memos \
+  -d '{"content":"#nexa:export-state"}'
+# returns runtime_config + counters as a JSON memo
+```
+
+## Log locations
+
+| Component | Where |
+|-----------|-------|
+| Memos | `docker logs nexa-memos` (LXC 104) |
+| n8n | `docker logs nexa-n8n` |
+| LiteLLM | `docker logs litellm` |
+| Qdrant | `docker logs qdrant_scientific` |
+| Zoraxy | LXC 108 web UI → Statistical Analysis |
+| Backrest | LXC 103 web UI |
+
+For a one-shot dump:
+
+```bash
+ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log
+```