fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

fix: resolve 7 of 8 backup blind spots (2026-05-28)

Browse Source 
- Vaultwarden data moved to UNAS /services/vaultwarden (CT 104 local → NFS)
- Garage S3 pre-backup rsync hook (garage-prestage.sh) + path added to plan
- n8n pre-backup rsync hook (n8n-prestage.sh) + confirm path covered
- Paperless-ngx media path added to services-backup-plan
- Stale paths (arcane, arr-stack, gluetun) removed from plan
- Repo passwords confirmed rotated (not tapirnase); LiteLLM blind spot N/A
- CT 103 SSH key deployed to CT 104 + CT 109 for prestage hooks
- pocketid-prestage.sh updated from CT 110 (destroyed) → CT 109
- Coverage map updated with new entries
This commit is contained in:
Florian Krebs
2026-05-28 01:35:31 +02:00
parent 24c7d56a1b
commit f1ed462c91
1 changed files with 29 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/backrest.md
+29 -19
View File
@@ -18,7 +18,7 @@
| Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) |
| BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) |
**Repos** (both `autoInitialize: true`; passwords currently `tapirnase` — see blind spot #5):
**Repos** (both `autoInitialize: true`; passwords rotated 2026-05-28 — stored in Vaultwarden):
| Repo | Target | Prune | Check |
|---|---|---|---|
@@ -29,21 +29,27 @@
| Plan | Repo | Schedule | Retention | Paths |
|---|---|---|---|---|
| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 15 paths (below) |
| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 14 paths (below) |
| `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` |
| `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` |
**Services plan paths** (all under `/mnt/pve/unas/`, verified 2026-05-28):
**Services plan paths** (all under `/mnt/pve/unas/`, updated 2026-05-28):
```
services/arcane services/arr-stack services/coder
services/gitea services/gluetun services/karakeep
services/memos services/n8n services/nextcloud
services/pocketid-backup services/traccar
backup/home-assistant backup/immich backup/nextcloud
services/coder services/gitea services/karakeep
services/memos services/n8n services/nextcloud
services/pocketid-backup services/shared-db-garage services/traccar
services/vaultwarden
backup/home-assistant backup/immich backup/nextcloud
media/documents/public/paperless-ngx
```
> ⚠️ `services/arcane`, `services/arr-stack`, `services/gluetun` are stale paths (services decommissioned or moved) — harmless but should be pruned from the plan.
**Pre-backup hooks** (run at `CONDITION_SNAPSHOT_START`):
- `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS
- `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage`
- `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n`
Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28.
`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud.
`services/pocketid` removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by `ops-backup.timer` → Garage S3.
@@ -76,6 +82,10 @@ Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sb
|---|---|---|
| CT 103 Backrest binary + config | Manual (small) | On-CT only |
| CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud |
| CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud |
| Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ |
| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ |
| CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ |
@@ -138,17 +148,17 @@ Script template in [appendix](#phase-3-config-to-git-scripts).
### Blind spots
| # | Severity | Issue | Fix |
| # | Severity | Issue | Status |
|---|---|---|---|
| 1 | **CRITICAL** | **Vaultwarden data at `/opt/stacks/vaultwarden/data` (CT 104 local FS) — not on UNAS, not in any plan.** Password vault loss is unrecoverable. | Move volume to `/mnt/pve/unas/services/vaultwarden/` and add that path to `services-backup-plan`. |
| 2 | HIGH | **Garage S3 data at `/opt/stacks/shared-db/garage/` (CT 104 local FS) — contains all WAL-G DB archives.** If CT 104 disk dies before WAL-G → JottaCloud sync runs, all DB backups are gone. | Add `/opt/stacks/shared-db/garage/` to a Backrest plan, OR verify JottaCloud sync runs frequently enough. Note: Garage had a config incident 2026-05-28 (toml path became a directory after Portainer redeploy — fixed manually). |
| 3 | MEDIUM | Paperless-ngx media at `/mnt/pve/unas/media/documents/public/paperless-ngx/` not in any Backrest plan. | Add to `services-backup-plan` paths. |
| 4 | MEDIUM | n8n encryption key in `/opt/stacks/n8n/data/` on CT 104 local FS — not in any plan. If CT 104 dies, DB restore is unusable. | Move n8n data volume to `/mnt/pve/unas/services/n8n/` (already in plan). |
| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted — without it, restore impossible. | Store `BORG_PASSWORD` in Vaultwarden. |
| 5 | MEDIUM | Both Backrest repo passwords are `tapirnase`. | Rotate before first scheduled run completes; `restic key passwd` re-encrypts in place. Store in Vaultwarden. |
| 6 | LOW | `litellm` DB password is the placeholder literal `litellm_password_here`. | Generate real password; update `ai/.env`, `litellm-config/config.yaml`, CT 113 user. |
| 7 | LOW | Migration dumps at `/mnt/pve/unas/dump/*-migration-20260521.sql` not in any plan. | Decide: keep as manual archive or delete now that WAL-G is archiving. |
| 8 | LOW | Stale paths in `services-backup-plan`: `arcane`, `arr-stack`, `gluetun` (services gone/moved). | Remove from plan to reduce noise and snapshot size. |
| 1 | ~~CRITICAL~~ | ~~Vaultwarden data on CT 104 local FS~~ | **FIXED 2026-05-28** — volume moved to `/mnt/pve/unas/services/vaultwarden/`; path added to plan |
| 2 | ~~HIGH~~ | ~~Garage S3 data on CT 104 local FS~~ | **FIXED 2026-05-28** — `garage-prestage.sh` rsyncs to UNAS before each backup; path `services/shared-db-garage` added to plan |
| 3 | ~~MEDIUM~~ | ~~Paperless-ngx media not in any plan~~ | **FIXED 2026-05-28** — `media/documents/public/paperless-ngx` added to `services-backup-plan` |
| 4 | ~~MEDIUM~~ | ~~n8n encryption key on CT 104 local FS~~ | **FIXED 2026-05-28** — `n8n-prestage.sh` rsyncs config/nodes/storage to UNAS before each backup |
| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env | Store `BORG_PASSWORD` in Vaultwarden — requires UI |
| 5 | ~~MEDIUM~~ | ~~Backrest repo passwords were `tapirnase`~~ | **FIXED** — passwords already rotated to complex strings in config.json |
| 6 | ~~LOW~~ | ~~`litellm` DB placeholder password~~ | **N/A** — LiteLLM decommissioned 2026-05-26; DB dropped |
| 7 | ~~LOW~~ | ~~Migration SQL dumps at `/mnt/pve/unas/dump/`~~ | **N/A** — no SQL dump files found; dumps were already cleaned up |
| 8 | ~~LOW~~ | ~~Stale paths `arcane`, `arr-stack`, `gluetun` in plan~~ | **FIXED 2026-05-28** — removed from `services-backup-plan` |
---