Skip to content

# 10 — Operations

Day-2 concerns. Backups, monitoring, troubleshooting.

## Backups

Tiering (3-2-1, full design in 12 #37): 1. Source — UNAS RAID 6 + native UniFi Drive snapshots (12 #38, still to enable). 2. Warm tier — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103). 3. Cold tier off-site — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in Q20). Always encrypt before upload — provider sees only ciphertext.

Nexa-specific items inside this pipeline:

What | How | Frequency
--- | --- | ---
n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container
Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier)
GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily
Memos DB | Backrest snapshot of Memos data dir | already covered
Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered
`runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a

## Monitoring

Signal | Source | Sink
--- | --- | ---
Container up/down | Dozzle (`192.168.1.40:3553`) + docker healthchecks. Skip names matching `octoprint*` — that container is intentionally powered down most of the time. | Memos system feed via ntfy
n8n workflow failures | n8n built-in failure-webhook | ntfy → `nexa.system` topic
Proxmox / disk / memory alerts | Proxmox notification target → ntfy | ntfy → Memos digest
LiteLLM rate-limit | LiteLLM logs / cost-tracking | Memos morning digest
Backrest run status | Backrest webhook | ntfy

A single ntfy topic `nexa.system` is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned `[SYSTEM]` memo.

## Troubleshooting

### Memos webhook not firing

    # is the Memos webhook config still pointing at n8n?
    curl -s -H "Authorization: Bearer $MEMOS_API_KEY" \
      https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'

    # does n8n still expose the path?
    curl -i https://n8n.nuclide.systems/webhook/memos     # expect 200/405, never 404

If 404 → workflow is inactive in n8n. Activate.

### LiteLLM 401 / 429

- 401 → the Nexa virtual key was rotated. Reissue, update `.env`, re-create the n8n credential.
- 429 → set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.

### Qdrant `nexa_knowledge` empty

    # is the indexing workflow active?
    docker logs nexa-n8n 2>&1 | grep memos_bridge | tail

    # manual index test
    curl -X POST https://memos.nuclide.systems/api/v1/memos \
      -H "Authorization: Bearer $MEMOS_API_KEY" \
      -d '{"content":"manual probe #nexa"}'

    # point count
    curl -s -H "api-key: $QDRANT_API_KEY" \
      $QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'

### Wrong list routing (Work vs Personal)

1. Run `#nexa:route-test <text>` and check the confidence value.
2. If <0.7 the router defaults to Personal (by design — see 06).
3. Tune the system prompt in `nexa-core/ai-prompts/system_prime.txt`; commit the change.

### "Where did Nexa store this?"

    # state dump
    curl -X POST https://memos.nuclide.systems/api/v1/memos \
      -d '{"content":"#nexa:export-state"}'
    # returns runtime_config + counters as a JSON memo

## Log locations

Component | Where
--- | ---
Memos | `docker logs nexa-memos` (LXC 104)
n8n | `docker logs nexa-n8n`
LiteLLM | `docker logs litellm`
Qdrant | `docker logs qdrant_scientific`
Zoraxy | LXC 108 web UI → Statistical Analysis
Backrest | LXC 103 web UI

For a one-shot dump:

    ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log