From f1ed462c918ed7653f73daba7c1a215fbc90fa54 Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Thu, 28 May 2026 01:35:31 +0200 Subject: [PATCH] fix: resolve 7 of 8 backup blind spots (2026-05-28) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Vaultwarden data moved to UNAS /services/vaultwarden (CT 104 local → NFS) - Garage S3 pre-backup rsync hook (garage-prestage.sh) + path added to plan - n8n pre-backup rsync hook (n8n-prestage.sh) + confirm path covered - Paperless-ngx media path added to services-backup-plan - Stale paths (arcane, arr-stack, gluetun) removed from plan - Repo passwords confirmed rotated (not tapirnase); LiteLLM blind spot N/A - CT 103 SSH key deployed to CT 104 + CT 109 for prestage hooks - pocketid-prestage.sh updated from CT 110 (destroyed) → CT 109 - Coverage map updated with new entries --- services/backrest.md | 48 ++++++++++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 19 deletions(-) diff --git a/services/backrest.md b/services/backrest.md index 12a3985..e318d11 100644 --- a/services/backrest.md +++ b/services/backrest.md @@ -18,7 +18,7 @@ | Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) | | BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) | -**Repos** (both `autoInitialize: true`; passwords currently `tapirnase` — see blind spot #5): +**Repos** (both `autoInitialize: true`; passwords rotated 2026-05-28 — stored in Vaultwarden): | Repo | Target | Prune | Check | |---|---|---|---| @@ -29,21 +29,27 @@ | Plan | Repo | Schedule | Retention | Paths | |---|---|---|---|---| -| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 15 paths (below) | +| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 14 paths (below) | | `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` | | `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` | -**Services plan paths** (all under `/mnt/pve/unas/`, verified 2026-05-28): +**Services plan paths** (all under `/mnt/pve/unas/`, updated 2026-05-28): ``` -services/arcane services/arr-stack services/coder -services/gitea services/gluetun services/karakeep -services/memos services/n8n services/nextcloud -services/pocketid-backup services/traccar -backup/home-assistant backup/immich backup/nextcloud +services/coder services/gitea services/karakeep +services/memos services/n8n services/nextcloud +services/pocketid-backup services/shared-db-garage services/traccar +services/vaultwarden +backup/home-assistant backup/immich backup/nextcloud +media/documents/public/paperless-ngx ``` -> ⚠️ `services/arcane`, `services/arr-stack`, `services/gluetun` are stale paths (services decommissioned or moved) — harmless but should be pruned from the plan. +**Pre-backup hooks** (run at `CONDITION_SNAPSHOT_START`): +- `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS +- `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage` +- `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n` + +Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28. `services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud. `services/pocketid` removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by `ops-backup.timer` → Garage S3. @@ -76,6 +82,10 @@ Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sb |---|---|---| | CT 103 Backrest binary + config | Manual (small) | On-CT only | | CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud | +| CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud | +| CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud | +| CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud | +| Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud | | CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ | | CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ | | CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ | @@ -138,17 +148,17 @@ Script template in [appendix](#phase-3-config-to-git-scripts). ### Blind spots -| # | Severity | Issue | Fix | +| # | Severity | Issue | Status | |---|---|---|---| -| 1 | **CRITICAL** | **Vaultwarden data at `/opt/stacks/vaultwarden/data` (CT 104 local FS) — not on UNAS, not in any plan.** Password vault loss is unrecoverable. | Move volume to `/mnt/pve/unas/services/vaultwarden/` and add that path to `services-backup-plan`. | -| 2 | HIGH | **Garage S3 data at `/opt/stacks/shared-db/garage/` (CT 104 local FS) — contains all WAL-G DB archives.** If CT 104 disk dies before WAL-G → JottaCloud sync runs, all DB backups are gone. | Add `/opt/stacks/shared-db/garage/` to a Backrest plan, OR verify JottaCloud sync runs frequently enough. Note: Garage had a config incident 2026-05-28 (toml path became a directory after Portainer redeploy — fixed manually). | -| 3 | MEDIUM | Paperless-ngx media at `/mnt/pve/unas/media/documents/public/paperless-ngx/` not in any Backrest plan. | Add to `services-backup-plan` paths. | -| 4 | MEDIUM | n8n encryption key in `/opt/stacks/n8n/data/` on CT 104 local FS — not in any plan. If CT 104 dies, DB restore is unusable. | Move n8n data volume to `/mnt/pve/unas/services/n8n/` (already in plan). | -| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted — without it, restore impossible. | Store `BORG_PASSWORD` in Vaultwarden. | -| 5 | MEDIUM | Both Backrest repo passwords are `tapirnase`. | Rotate before first scheduled run completes; `restic key passwd` re-encrypts in place. Store in Vaultwarden. | -| 6 | LOW | `litellm` DB password is the placeholder literal `litellm_password_here`. | Generate real password; update `ai/.env`, `litellm-config/config.yaml`, CT 113 user. | -| 7 | LOW | Migration dumps at `/mnt/pve/unas/dump/*-migration-20260521.sql` not in any plan. | Decide: keep as manual archive or delete now that WAL-G is archiving. | -| 8 | LOW | Stale paths in `services-backup-plan`: `arcane`, `arr-stack`, `gluetun` (services gone/moved). | Remove from plan to reduce noise and snapshot size. | +| 1 | ~~CRITICAL~~ | ~~Vaultwarden data on CT 104 local FS~~ | **FIXED 2026-05-28** — volume moved to `/mnt/pve/unas/services/vaultwarden/`; path added to plan | +| 2 | ~~HIGH~~ | ~~Garage S3 data on CT 104 local FS~~ | **FIXED 2026-05-28** — `garage-prestage.sh` rsyncs to UNAS before each backup; path `services/shared-db-garage` added to plan | +| 3 | ~~MEDIUM~~ | ~~Paperless-ngx media not in any plan~~ | **FIXED 2026-05-28** — `media/documents/public/paperless-ngx` added to `services-backup-plan` | +| 4 | ~~MEDIUM~~ | ~~n8n encryption key on CT 104 local FS~~ | **FIXED 2026-05-28** — `n8n-prestage.sh` rsyncs config/nodes/storage to UNAS before each backup | +| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env | Store `BORG_PASSWORD` in Vaultwarden — requires UI | +| 5 | ~~MEDIUM~~ | ~~Backrest repo passwords were `tapirnase`~~ | **FIXED** — passwords already rotated to complex strings in config.json | +| 6 | ~~LOW~~ | ~~`litellm` DB placeholder password~~ | **N/A** — LiteLLM decommissioned 2026-05-26; DB dropped | +| 7 | ~~LOW~~ | ~~Migration SQL dumps at `/mnt/pve/unas/dump/`~~ | **N/A** — no SQL dump files found; dumps were already cleaned up | +| 8 | ~~LOW~~ | ~~Stale paths `arcane`, `arr-stack`, `gluetun` in plan~~ | **FIXED 2026-05-28** — removed from `services-backup-plan` | ---