Claude
dfd9aa7d2a
- s3.nuclide.systems is on-site only — recasting it as the WARM tier, not the disaster-recovery copy. Off-site cold becomes a separate decision tracked as Q20. - docs/12 #37 expanded with full provider comparison: Jottacloud Unlimited (~€9.50/mo, EU/Norway, soft-cap ~5 TB) is the user's stated candidate and a reasonable fit for the current 2 TB scale with ~10 years of headroom. Alternatives priced: Hetzner Storage Box, Backblaze B2, rsync.net, Storj. Always restic/rclone-crypt before upload. - Off-site priority list pinned: Immich originals + documents + Nextcloud user data + Vaultwarden DB go off-site; movies / music / ROMs / derived caches don't. - docs/10 backup section rewritten as the 3-2-1 pipeline (source → UNAS native snapshot → warm → encrypted cold). Nexa-specific snapshot rows updated for the new collection names. - docs/11 Q20 added with the decision criteria. - CLAUDE.md: backup model pinned so future agents don't propose alternative paths.
102 lines
4.1 KiB
Markdown
102 lines
4.1 KiB
Markdown
# 10 — Operations
|
|
|
|
Day-2 concerns. Backups, monitoring, troubleshooting.
|
|
|
|
## Backups
|
|
|
|
**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)):
|
|
1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable).
|
|
2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103).
|
|
3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext.
|
|
|
|
Nexa-specific items inside this pipeline:
|
|
|
|
| What | How | Frequency |
|
|
|------|-----|-----------|
|
|
| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
|
|
| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) |
|
|
| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily |
|
|
| Memos DB | Backrest snapshot of Memos data dir | already covered |
|
|
| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered |
|
|
| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
|
|
|
|
## Monitoring
|
|
|
|
| Signal | Source | Sink |
|
|
|--------|--------|------|
|
|
| Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. **Skip names matching `octoprint*`** — that container is intentionally powered down most of the time. | Memos system feed via ntfy |
|
|
| n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic |
|
|
| Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest |
|
|
| LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest |
|
|
| Backrest run status | Backrest webhook | ntfy |
|
|
|
|
A **single ntfy topic `nexa.system`** is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo.
|
|
|
|
## Troubleshooting
|
|
|
|
### Memos webhook not firing
|
|
|
|
```bash
|
|
# is the Memos webhook config still pointing at n8n?
|
|
curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \
|
|
https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'
|
|
|
|
# does n8n still expose the path?
|
|
curl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404
|
|
```
|
|
|
|
If 404 → workflow is inactive in n8n. Activate.
|
|
|
|
### LiteLLM 401 / 429
|
|
|
|
- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential.
|
|
- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.
|
|
|
|
### Qdrant `nexa_knowledge` empty
|
|
|
|
```bash
|
|
# is the indexing workflow active?
|
|
docker logs nexa-n8n 2>&1 | grep memos_bridge | tail
|
|
|
|
# manual index test
|
|
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
|
-H "Authorization: Bearer $MEMOS_API_KEY" \
|
|
-d '{"content":"manual probe #nexa"}'
|
|
|
|
# point count
|
|
curl -s -H "api-key: $QDRANT_API_KEY" \
|
|
$QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'
|
|
```
|
|
|
|
### Wrong list routing (Work vs Personal)
|
|
|
|
1. Run `#nexa:route-test <text>` and check the confidence value.
|
|
2. If <0.7 the router defaults to Personal (by design — see [06](./06-classification-logic.md)).
|
|
3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change.
|
|
|
|
### "Where did Nexa store this?"
|
|
|
|
```bash
|
|
# state dump
|
|
curl -X POST https://memos.nuclide.systems/api/v1/memos \
|
|
-d '{"content":"#nexa:export-state"}'
|
|
# returns runtime_config + counters as a JSON memo
|
|
```
|
|
|
|
## Log locations
|
|
|
|
| Component | Where |
|
|
|-----------|-------|
|
|
| Memos | `docker logs nexa-memos` (LXC 104) |
|
|
| n8n | `docker logs nexa-n8n` |
|
|
| LiteLLM | `docker logs litellm` |
|
|
| Qdrant | `docker logs qdrant_scientific` |
|
|
| Zoraxy | LXC 108 web UI → Statistical Analysis |
|
|
| Backrest | LXC 103 web UI |
|
|
|
|
For a one-shot dump:
|
|
|
|
```bash
|
|
ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log
|
|
```
|