diff --git a/CLAUDE.md b/CLAUDE.md index d52f0f7..4fc9215 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,6 +45,7 @@ This file tells future automated runs what they need to know about this repo. - `Wunschliste` → Wishes. Lists are discovered **by name** at runtime (Qdrant `_config` namespace caches `name → id`). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh. - **Mail = Nextcloud Mail, single account `fkrebs@nucli.de`.** No separate IMAP entry. The `Waiting` folder is a manual user signal — items there are skipped from digests. +- **Backup model is 3-2-1**: UNAS native snapshots → `s3.nuclide.systems` (warm, on-site) → encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate — see [Q20](./docs/11-open-questions.md#q20)). Always `restic`/`rclone-crypt` before upload — third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first. - **Auth = Pocket-ID SSO is global** at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords. - **Decided for Nexa** (don't re-litigate without user input): - Vector store: **reuse `qdrant_scientific`** with collections suffixed by modality (`nexa_knowledge_text`, `nexa_knowledge_visual`). diff --git a/docs/10-operations.md b/docs/10-operations.md index c5fe9da..a58dcc5 100644 --- a/docs/10-operations.md +++ b/docs/10-operations.md @@ -4,15 +4,21 @@ Day-2 concerns. Backups, monitoring, troubleshooting. ## Backups -Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items: +**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)): +1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable). +2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103). +3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext. + +Nexa-specific items inside this pipeline: | What | How | Frequency | |------|-----|-----------| | n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container | -| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) | +| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections//snapshots` → write to `backup/nexa/snapshots/qdrant//` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) | +| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/.ttl.gz` | daily | | Memos DB | Backrest snapshot of Memos data dir | already covered | -| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered | -| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | +| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered | +| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a | ## Monitoring diff --git a/docs/11-open-questions.md b/docs/11-open-questions.md index b210c27..e3364fd 100644 --- a/docs/11-open-questions.md +++ b/docs/11-open-questions.md @@ -55,6 +55,8 @@ Items that **block progress** and need a human decision before a workflow can be - [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa. +- [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end. + --- ## Speed budget (Q3 follow-up) diff --git a/docs/12-optimization-opportunities.md b/docs/12-optimization-opportunities.md index 2cee0a7..56b6e41 100644 --- a/docs/12-optimization-opportunities.md +++ b/docs/12-optimization-opportunities.md @@ -106,13 +106,43 @@ These three are inter-related — picking them up as one campaign is cheaper tha Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services//` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward. -37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup//` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier: - - **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination. - - **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore). - - **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars. - - Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target. +37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.** + `s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers: - This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs. + **Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):** + - **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination. + - **Qdrant + GraphDB snapshots** flow UNAS → S3 daily. + - Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set. + + **Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):** + - **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`. + - **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative. + - **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores. + - **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication. + - **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem. + + **Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy. + + **Tiering & cadence:** + ``` + [Source] UNAS RAID-6 + native snapshots (#38) + │ daily restic/borg + ▼ + [Warm] s3.nuclide.systems (on-site, fast restore) + │ weekly rclone copy + crypt + ▼ + [Cold] Off-site (Jottacloud / B2 / Hetzner) ← satisfies the "1" in 3-2-1 + ``` + + **What goes off-site (priority order):** + 1. Immich photo originals — irreplaceable. + 2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable. + 3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo. + 4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include. + + **What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate. + + Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it. 38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.