3-2-1 backup tiering with off-site cold (Jottacloud candidate)
- s3.nuclide.systems is on-site only — recasting it as the WARM tier, not the disaster-recovery copy. Off-site cold becomes a separate decision tracked as Q20. - docs/12 #37 expanded with full provider comparison: Jottacloud Unlimited (~€9.50/mo, EU/Norway, soft-cap ~5 TB) is the user's stated candidate and a reasonable fit for the current 2 TB scale with ~10 years of headroom. Alternatives priced: Hetzner Storage Box, Backblaze B2, rsync.net, Storj. Always restic/rclone-crypt before upload. - Off-site priority list pinned: Immich originals + documents + Nextcloud user data + Vaultwarden DB go off-site; movies / music / ROMs / derived caches don't. - docs/10 backup section rewritten as the 3-2-1 pipeline (source → UNAS native snapshot → warm → encrypted cold). Nexa-specific snapshot rows updated for the new collection names. - docs/11 Q20 added with the decision criteria. - CLAUDE.md: backup model pinned so future agents don't propose alternative paths.
This commit is contained in:
Claude
+10
-4
@@ -4,15 +4,21 @@ Day-2 concerns. Backups, monitoring, troubleshooting.
|
||||
|
||||
## Backups
|
||||
|
||||
Backrest (LXC 103) already handles file-level snapshots. Add Nexa-specific items:
|
||||
**Tiering** (3-2-1, full design in [12 #37](./12-optimization-opportunities.md#37)):
|
||||
1. **Source** — UNAS RAID 6 + native UniFi Drive snapshots ([12 #38](./12-optimization-opportunities.md#38), still to enable).
|
||||
2. **Warm tier** — `s3.nuclide.systems` (on-site), restic/borg via Backrest (LXC 103).
|
||||
3. **Cold tier off-site** — encrypted `rclone` copy to a third-party (Jottacloud is the user's candidate; decision tracked in [Q20](./11-open-questions.md#q20)). Always encrypt before upload — provider sees only ciphertext.
|
||||
|
||||
Nexa-specific items inside this pipeline:
|
||||
|
||||
| What | How | Frequency |
|
||||
|------|-----|-----------|
|
||||
| n8n workflow JSON | `nexa-core/scripts/backup_workflows.sh` → git push | hourly cron in n8n container |
|
||||
| Qdrant `nexa_knowledge` | `POST /collections/nexa_knowledge/snapshots` → rsync to `s3.nuclide.systems` | daily (Backrest pre-hook) |
|
||||
| Qdrant `nexa_knowledge_text` (and `_visual` once Phase 3.2) | `POST /collections/<name>/snapshots` → write to `backup/nexa/snapshots/qdrant/<date>/` on UNAS, then Backrest picks it up for warm + cold | daily (snapshot), weekly (cold tier) |
|
||||
| GraphDB `nexa_knowledge` (Phase 3.4+) | scheduled SPARQL `CONSTRUCT` export → `backup/nexa/snapshots/graphdb/<date>.ttl.gz` | daily |
|
||||
| Memos DB | Backrest snapshot of Memos data dir | already covered |
|
||||
| Nextcloud | Nextcloud's own backup app + Backrest of `/var/www/nextcloud/data` | already covered |
|
||||
| `runtime_config.json` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
|
||||
| Nextcloud | Nextcloud's own backup app + Backrest of `services/nextcloud/` | already covered |
|
||||
| `runtime_config` (Qdrant `_config` namespace) | Qdrant snapshot covers it | n/a |
|
||||
|
||||
## Monitoring
|
||||
|
||||
|
||||
@@ -55,6 +55,8 @@ Items that **block progress** and need a human decision before a workflow can be
|
||||
|
||||
- [ ] **Q19 — Confirm the SMB share name + mount syntax** by reading an existing stack in **Arcane** (recommended: `karakeep` or `immich`). The SMB share name on UNAS that maps to the NFS root `/var/nfs/shared/storage` is currently a placeholder `<unas-share>` in [docs/09](./09-deployment.md). Required action: open the running Karakeep stack in Arcane, copy its `volumes:` block verbatim, swap `karakeep` → `nexa` in the path, drop into the Nexa stack. This locks Nexa to the **same convention every other container in the homelab uses** — same driver, same credentials secret, same uid/gid — and avoids reinventing storage just for Nexa.
|
||||
|
||||
- [ ] **Q20 — Off-site cold-tier backup provider.** `s3.nuclide.systems` is on-site only, so today the homelab has no real disaster-recovery copy (fire / theft / ransomware kills it). Comparison + recommendation in [docs/12 #37](./12-optimization-opportunities.md#37). User's stated candidate is **Jottacloud Unlimited** (~€9.50/mo, EU/Norway, soft-cap ~5 TB — comfortable for the next ~10 years given current 2 TB and slow growth). Alternatives: Hetzner Storage Box BX21 (€13/mo, predictable quota), Backblaze B2 (~$12/mo, US, S3-compatible). **Always encrypt before upload** with `restic` or `rclone crypt` regardless of provider. Decision needed: pick one, create the account, share account email here (no password). Once chosen, optimization #37 can be implemented end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Speed budget (Q3 follow-up)
|
||||
|
||||
@@ -106,13 +106,43 @@ These three are inter-related — picking them up as one campaign is cheaper tha
|
||||
|
||||
Output: a one-page table `service | persistent? | currently bound to | should be bound to`. Then migrate the non-compliant ones one-by-one (stop → rsync data to `services/<svc>/` on UNAS → re-create stack with the new bind → verify → keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.
|
||||
|
||||
37. **S3 archive tier (`s3.nuclide.systems`).** Currently UNAS holds backups in `backup/<svc>/` and there's no cold offsite. Make `s3.nuclide.systems` the canonical *cold* tier:
|
||||
- **Restic / borg** repos targeting S3 for: `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add an S3 destination.
|
||||
- **Qdrant + GraphDB snapshots** flow UNAS → S3 weekly (daily on UNAS, weekly to S3 = cheap, keeps the most recent locally for fast restore).
|
||||
- **Immich photo originals** to S3 if the photo library grows past UNAS comfort. Immich has native S3 support via its `IMMICH_*` env vars.
|
||||
- Resolves [Q12](./11-open-questions.md#q12) — once the bucket name + access key are set, every other archive just adds a target.
|
||||
37. **3-2-1 backup tiering: warm on-site (S3) + cold off-site.**
|
||||
`s3.nuclide.systems` is already up but it's **on-site** — same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a *warm* tier, not a disaster-recovery copy. The full design is two tiers:
|
||||
|
||||
This is the right time to do it because we're about to add Nexa snapshots; designing the path for *all* services rather than just Nexa avoids one-offs.
|
||||
**Warm tier — `s3.nuclide.systems` (already exists, just needs a bucket):**
|
||||
- **Restic / borg** repos for `backup/home-assistant/`, `backup/immich/`, `backup/nextcloud/`, future `backup/nexa/`. Backrest already orchestrates Borg — just add the S3 destination.
|
||||
- **Qdrant + GraphDB snapshots** flow UNAS → S3 daily.
|
||||
- Resolves [Q12](./11-open-questions.md#q12) once the bucket name + access key are set.
|
||||
|
||||
**Cold tier — off-site provider (decision pending — see [Q20](./11-open-questions.md#q20)):**
|
||||
- **Jottacloud "Unlimited"** (~€9.50/mo, EU/Norway, soft-cap ~5 TB) — user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use `rclone jottacloud:` or native `jotta-cli`.
|
||||
- **Hetzner Storage Box BX21** (€13/mo, 5 TB EU/DE) — predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative.
|
||||
- **Backblaze B2** (~$12/mo for 2 TB, S3-compatible, US) — cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during *full* restores.
|
||||
- **rsync.net** (~$30/mo) — ZFS send/recv natively; overkill unless you want pool replication.
|
||||
- **Storj DCS** ($4/TB/mo, decentralized, S3-compatible) — newer ecosystem.
|
||||
|
||||
**Always encrypt before upload** regardless of provider — `restic` or `rclone crypt` over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.
|
||||
|
||||
**Tiering & cadence:**
|
||||
```
|
||||
[Source] UNAS RAID-6 + native snapshots (#38)
|
||||
│ daily restic/borg
|
||||
▼
|
||||
[Warm] s3.nuclide.systems (on-site, fast restore)
|
||||
│ weekly rclone copy + crypt
|
||||
▼
|
||||
[Cold] Off-site (Jottacloud / B2 / Hetzner) ← satisfies the "1" in 3-2-1
|
||||
```
|
||||
|
||||
**What goes off-site (priority order):**
|
||||
1. Immich photo originals — irreplaceable.
|
||||
2. `media/documents/`, `_sortMe/Downloads/`, Nextcloud user data — irreplaceable.
|
||||
3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots — replicable from sources but expensive to redo.
|
||||
4. Vaultwarden DB — *cryptographically* sensitive but small; explicitly include.
|
||||
|
||||
**What does not need off-site:** movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.
|
||||
|
||||
Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a `restic init` + `restic backup` of `backup/nexa/` first as a smoke test before pointing the heavyweight repos at it.
|
||||
|
||||
38. **Configure UNAS Pool snapshots.** The UniFi Drive dashboard shows the storage-pool snapshot schedule as "Click to Setup" — i.e. **not configured**. RAID 6 protects against drive failure, not against an `rm -rf` from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. **Highest-leverage data-protection change in the homelab right now** — costs nothing, recovers everything.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user