docs/services/backrest.md

# Backup Strategy

> Single Backrest instance on **CT 103** (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at `/mnt/pve/unas`, so it can read all service data without SSH-ing other hosts.

**Goal**: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.

---

## Current state

### Backrest (CT 103) — Phase 1b live since 2026-05-21

| Item | Value |
|---|---|
| UI | `http://192.168.1.3:9898` (LAN-only, auth disabled) |
| Config | `/opt/backrest/config/config.json` (timestamped `.bak` files on every edit) |
| rclone remote | `jottacloud:` Archive section (default device/mountpoint); token at `/root/.config/rclone/rclone.conf` |
| Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) |
| BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) |

**Repos** (both `autoInitialize: true`; passwords rotated 2026-05-28 — stored in Vaultwarden):

| Repo | Target | Prune | Check |
|---|---|---|---|
| `services-repo` | `rclone:jottacloud:services` | weekly Sun 05:00, ≤10 % unused | monthly, 10 % subset |
| `media-repo` | `rclone:jottacloud:media` | monthly, ≤10 % unused | monthly, 10 % subset |

**Plans**:

| Plan | Repo | Schedule | Retention | Paths |
|---|---|---|---|---|
| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 14 paths (below) |
| `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` |
| `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` |

**Services plan paths** (all under `/mnt/pve/unas/`, updated 2026-05-28):

```
services/coder                services/gitea               services/karakeep
services/memos                services/n8n                 services/nextcloud
services/pocketid-backup      services/shared-db-garage    services/traccar
services/vaultwarden
backup/home-assistant         backup/immich                backup/nextcloud
media/documents/public/paperless-ngx
```

**Pre-backup hooks** (run at `CONDITION_SNAPSHOT_START`):
- `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS
- `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage`
- `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n`

Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28.

`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud.
`services/pocketid` removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by `ops-backup.timer` → Garage S3.

> **JottaCloud web UI tip**: rclone writes to the **Archive** section. The default landing page shows only Sync + Backup. Browse to `https://www.jottacloud.com/web/archive` to see the restic repos.

### Postgres (CT 113) — Phase 2 live since 2026-05-21

Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmin on `:5050`. WAL-G archives continuously to Garage S3 (`ct113-pg-backup` on CT 104):

- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
- Daily `wal-g backup-push` cron at 02:00
- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder`

Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`).

**Exceptions** (stay on original hosts):

| DB | Host | Why |
|---|---|---|
| `lobe-postgres` (paradedb pg17) | CT 104 | Uses `pg_search` `USING bm25` indexes; stock postgres 17 can't host. WAL-G → `lobe-pg-backup` + Backrest secondary on data dir. |
| Nextcloud AIO postgres | CT 105 | AIO manages it; Borg archives the whole stack to UNAS. |
| Immich postgres | CT 104 | Version-pinned by Immich. WAL-G → `immich-pg-backup`. |

---

## Coverage map

| Host / data | Method | Offsite |
|---|---|---|
| CT 103 Backrest binary + config | Manual (small) | On-CT only |
| CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud |
| CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud |
| Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud |
| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ |
| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ |
| CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ |
| CT 101 Shepard | Source in Gitea (gitea path covers it) | ✓ |
| CT 102 AdGuard | Phase 3 git push → `fkrebs/adguard-conf` | ✓ daily cron |
| CT 104 Gitea + Coder | `services/{gitea,coder}` (UNAS; moved from CT 111 2026-05-26) | ✓ |
| CT 105 Nextcloud user files | `services/nextcloud` | ✓ |
| CT 105 Nextcloud AIO volumes | AIO Borg → `/mnt/pve/unas/backup/nextcloud/` → Backrest | ✓ |
| ~~CT 108 Zoraxy~~ | **DESTROYED 2026-05-28** — replaced by Traefik on CT 109 | — |
| CT 109 Traefik config + acme.json | Not backed up (LE can reissue; config in `/opt/stacks/traefik/`) | ✗ low risk |
| CT 109 Portainer config | Daily tar → Garage S3 `ct109-portainer-backup` → JottaCloud sync | ✓ |
| CT 109 Pocket-ID | Daily tar → Garage S3 `ct109-portainer-backup` (key `ops-*.tar.gz`) → JottaCloud sync via `ops-backup.timer` | ✓ |
| CT 109 Infisical | pg_dump in `ops-*.tar.gz` (same as above) | ✓ |
| ~~CT 110 Pocket-ID~~ | **CT 110 destroyed 2026-05-26 — see CT 109 row above** | ✓ |
| ~~CT 111 Gitea + Coder~~ | **CT 111 destroyed 2026-05-26 — see CT 104 row above** | ✓ |
| ~~CT 112 Infisical~~ | **CT 112 destroyed 2026-05-26 — see CT 109 row above** | ✓ |
| ~~CT 108 Zoraxy~~ | **CT 108 destroyed 2026-05-28 — Traefik on CT 109 is replacement** | — |
| VM 100 HAOS config | Phase 3 git addon → `fkrebs/home-assistant-config` | ✗ Git Pull addon needs UI install |
| VM 100 HAOS daily tar | HA → UNAS → Backrest | ✓ |
| PVE host `/etc/pve/` | Phase 3 git push → `fkrebs/pve-conf` | ✓ daily cron |
| UNAS `media/images/library` | Phase 1a `rclone sync` → `jottacloud:Photos/` | ✓ Saturdays 01:00 |
| UNAS personal data (`documents`, `_sortMe`, `video-projects`, …) | Phase 1a `rclone sync` → `jottacloud:UNAS/` | ✓ Saturdays 01:00 |

### Intentionally not backed up

Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable.

### Deferred — include if needed

| Host | Data | Notes |
|---|---|---|
| CT 109 Prometheus TSDB, Grafana dashboards | Low priority — metrics are ephemeral; dashboards re-exportable from Grafana; add to services plan if needed |
| CT 109 Portainer data | WAL-G–style daily tar → `ct109-portainer-backup` Garage S3 → JottaCloud offsite sync | ✓ (via walg-offsite-sync.sh) |

---

## Open work

### Phase 1a — rclone sync for UNAS personal data ✅ DEPLOYED 2026-05-22

Script `/usr/local/sbin/unas-sync.sh` on CT 103, cron `0 1 * * 6` (Saturdays 01:00). Last completed run: 2026-05-27 13h32m for 160486 transfers. Log: `/var/log/rclone-unas-sync.log`.

- **Job 1**: `media/images/library/` → `jottacloud:Photos/` (Immich originals, visible at `jottacloud.com/photo`).
- **Job 2**: `/mnt/pve/unas/` → `jottacloud:UNAS/` (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, `services/`/`backup/` already in Backrest, `dump/`).

BW throttle: `--bwlimit "05:30,7.5M 01:00,20M"` (same as Backrest).

Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.

### Phase 3 — Config-to-git for infrastructure ✅ DEPLOYED

Daily cron on each host (~03:00) pushes config to private Gitea repos. Active deployments:

| Host | Cron | Script | Target repo |
|---|---|---|---|
| CT 102 (AdGuard) | daily | `/usr/local/sbin/adguard-conf-backup.sh` | `fkrebs/adguard-conf` |
| CT 103 (Backrest) | `0 3 * * *` | `/usr/local/sbin/ct103-conf-backup.sh` | `fkrebs/ct103-backrest` |
| PVE host (`nuc`) | `15 3 * * *` (central) | `/usr/local/bin/sync-config-repos` | `fkrebs/{ct103-backrest, ct113-db, ct104-stacks, ct109-stacks}` |
| PVE host (`nuc`) | daily | `/usr/local/sbin/pve-conf-backup.sh` | `fkrebs/pve-conf` |

`sync-config-repos` rebuilt 2026-05-28: removed dead CT 110/111/112 references, list of CT 104 stacks now derived from `ls /opt/stacks` (auto-discovers new stacks), added CT 109 `ct109-stacks` (auto-creates repo on first run).

**Open**: VM 100 (HAOS) → `fkrebs/home-assistant-config` — requires installing the native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) via HAOS UI (cannot be automated remotely).

### Blind spots

| # | Severity | Issue | Status |
|---|---|---|---|
| 1 | ~~CRITICAL~~ | ~~Vaultwarden data on CT 104 local FS~~ | **FIXED 2026-05-28** — volume moved to `/mnt/pve/unas/services/vaultwarden/`; path added to plan |
| 2 | ~~HIGH~~ | ~~Garage S3 data on CT 104 local FS~~ | **FIXED 2026-05-28** — `garage-prestage.sh` rsyncs to UNAS before each backup; path `services/shared-db-garage` added to plan |
| 3 | ~~MEDIUM~~ | ~~Paperless-ngx media not in any plan~~ | **FIXED 2026-05-28** — `media/documents/public/paperless-ngx` added to `services-backup-plan` |
| 4 | ~~MEDIUM~~ | ~~n8n encryption key on CT 104 local FS~~ | **FIXED 2026-05-28** — `n8n-prestage.sh` rsyncs config/nodes/storage to UNAS before each backup |
| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env | Store `BORG_PASSWORD` in Vaultwarden — requires UI |
| 5 | ~~MEDIUM~~ | ~~Backrest repo passwords were `tapirnase`~~ | **FIXED** — passwords already rotated to complex strings in config.json |
| 6 | ~~LOW~~ | ~~`litellm` DB placeholder password~~ | **N/A** — LiteLLM decommissioned 2026-05-26; DB dropped |
| 7 | ~~LOW~~ | ~~Migration SQL dumps at `/mnt/pve/unas/dump/`~~ | **N/A** — no SQL dump files found; dumps were already cleaned up |
| 8 | ~~LOW~~ | ~~Stale paths `arcane`, `arr-stack`, `gluetun` in plan~~ | **FIXED 2026-05-28** — removed from `services-backup-plan` |

---

## Operations

### Restore

UI: **Repos → snapshots → Browse → file → Restore**.

CLI on CT 103:

```bash
restic -r rclone:jottacloud:services snapshots
restic -r rclone:jottacloud:services restore latest \
  --target /restore \
  --include /mnt/pve/unas/services/vaultwarden
```

### Monitoring ✅ DEPLOYED 2026-05-28

Gotify hooks on all 3 plans (services / video-projects / media). Gotify app "Backrest" token `AtIMGOYcZsb5HfO`.

- `CONDITION_SNAPSHOT_SUCCESS` → Gotify priority 3 (✓ snapshot summary)
- `CONDITION_SNAPSHOT_ERROR` + `CONDITION_SNAPSHOT_WARNING` → Gotify priority 8 (✗ FAILED)

Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell required, templating via Go text/template with `{{ .Plan.Id }}`, `{{ .SnapshotStats.FilesNew }}`, `{{ formatSizeBytes .SnapshotStats.DataAdded }}`, `{{ .Duration }}`, `{{ .Error }}`.

**WAL-G staleness monitor** (`/usr/local/sbin/walg-staleness-check.sh` on CT 103, hourly timer): SSH-then-`docker exec psql` to each Postgres host and queries `pg_stat_archiver` (last_archived_time, last_failed_time, failed_count, failed_wal). Alerts if archive >26h old OR a recent failure is more recent than the last success. Gotify token `Ag570SdxcXcpp51`. Hosts: CT 113 `postgres`, CT 104 `immich_postgres`. Closes `fkrebs/n8n-flows#6`.

> **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running.

**Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`.

### Applying a new config

```bash
# on CT 103
systemctl stop backrest
cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)
# paste new config.json
systemctl start backrest
```

---

## History

### Provider comparison

Researched 2026-05-21. Sized for ~3 TB/mo.

| Provider | ~3 TB/mo | Backend | EU DC | Egress | Verdict |
|---|---|---|---|---|---|
| **JottaCloud Unlimited** | €9.91 flat (unlimited) | rclone native | Norway (EEA) | Free | ✅ Chosen |
| Hetzner BX31 | €20.80 flat (10 TB) | SFTP/WebDAV | DE, FI | Free | 2.5× the price, capped |
| Backblaze B2 | ~$18 pay-per-GB | S3 | Frankfurt | Free (≤3× stored) | US CLOUD Act risk |
| Cloudflare R2 | ~$45 pay-per-GB | S3 | EU auto | Zero | Expensive at scale; no EU residency lock |
| Wasabi | ~$21–24 | S3 | FRA/AMS | Free (≤ stored) | ❌ 90-day min billing per object → Restic prune disaster |
| Storj DCS | ~$30 | S3 | EU-geofenced | 1× free | Complex, $5 minimum |
| pCloud | €399 one-time / 2 TB | WebDAV | Luxembourg | Free | WebDAV too slow |
| Infomaniak kDrive | ~€36+ / 3 TB | WebDAV | Switzerland | Free | WebDAV only; non-EU |
| Proton Drive | — | rclone beta | Switzerland | Free | rclone backend broken since late 2025 |

### Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21)

**Decision**: provision a dedicated `db` LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse `shared-postgres` on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113).

Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, `mp0=/mnt/pve/unas`. Stack in Gitea `fkrebs/stacks-db`. pgAdmin pre-registers postgres via `pgadmin-servers.json`.

**Migration order**: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB:

```bash
docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql
psql -U postgres -c "CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;"
psql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql
# Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume
```

Stale DBs dropped from `shared-postgres` post-migration: `daytona`, `lobechat` (duplicate), `paradedb` (duplicate).

WAL-G enabled 2026-05-21: `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` via `ALTER SYSTEM`; first base backup verified (`base_000000010000000000000012`). Daily `wal-g backup-push` cron at 02:00. Garage → JottaCloud offsite sync added at 02:30.

### Phase 1a rclone sync script

```bash
#!/bin/bash
set -e
LOG=/var/log/rclone-unas-sync.log

# Job 1: Immich originals → JottaCloud gallery
rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \
  --transfers=4 --checkers=8 \
  --log-file=$LOG --log-level INFO

# Job 2: all other irreplaceable personal data
rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \
  --transfers=4 --checkers=8 \
  --exclude "media/images/library/**" \
  --exclude "media/images/upload/**" \
  --exclude "media/images/thumbs/**" \
  --exclude "media/images/encoded-video/**" \
  --exclude "media/images/profile/**" \
  --exclude "media/images/backups/**" \
  --exclude "media/movies/**" \
  --exclude "media/emulation/**" \
  --exclude "media/Torrents/**" \
  --exclude "media/music/**" \
  --exclude "media/podcasts/**" \
  --exclude "services/**" \
  --exclude "backup/**" \
  --exclude "backup-staging/**" \
  --exclude "test_perm" \
  --log-file=$LOG --log-level INFO
```

Cron (Saturday 01:00): `0 1 * * 6  /usr/local/sbin/unas-sync.sh`

### Phase 3 config-to-git scripts

Same pattern on each host. Replace `GITEA_TOKEN` with the value from `/opt/stacks/ai/.env` or a dedicated scoped Gitea token.

**CT 108 — Zoraxy** (`/usr/local/sbin/zoraxy-conf-backup.sh`):

```bash
#!/bin/bash
set -e
REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git"
WORK="/opt/zoraxy/conf"
git -C "$WORK" init -b main -q 2>/dev/null || true
git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \
  || git -C "$WORK" remote add origin "$REPO_URL"
git -C "$WORK" add -A
git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true
git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true
```

**CT 102 — AdGuard** — same template, copy `/opt/AdGuardHome/AdGuardHome.yaml` into `/tmp/adguard-conf-work` checkout first.

**PVE host** — same template, `rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/` then commit.

**VM 100 — HA** — native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) pushing `/config/` (exclude `secrets.yaml`, `.storage/`) to `fkrebs/ha-config`. HA daily tars already covered by `services-backup-plan` via `/mnt/pve/unas/backup/home-assistant/`.

Cron on each host: `0 3 * * * /usr/local/sbin/<host>-conf-backup.sh`

### Why Wasabi was rejected

Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every `restic forget --prune` generates surprise costs. Well-documented Restic-on-Wasabi trap.

### Why lobe-postgres stays on CT 104

Migration `0093_add_bm25_indexes_with_icu.sql` creates `USING bm25` indexes on 7 tables (`agents`, `topics`, `files`, `knowledge_bases`, `user_memories`, `chat_groups`, `user_memories_contexts`). `bm25` is paradedb-only (`pg_search` extension); stock postgres 17 has no such index access method and the migration fails. Container stays `paradedb/paradedb:latest-pg17`. WAL-G archives to `lobe-pg-backup`; Backrest secondary copies the data dir.

### UNAS data inventory (basis for Phase 1a sizing)

| Path | Size | Notes |
|---|---|---|
| `media/documents/` | 9.5 G | Personal documents |
| `media/video-projects/` | 425 G | Creative work, irreplaceable |
| `media/musical-sheets/` | 23 G | |
| `media/audiobooks/` | 23 G | |
| `media/ebooks/` | 6.8 G | |
| `media/3d-prints/` | 65 M | |
| `media/Recipes/` | 113 M | |
| `_sortMe/` | 335 G | `images/` 171 G, `work Flo/` 147 G, `Anne/` 17 G |
| `code/` | 6.7 M | |
| Excluded `media/movies/` | 118 G | re-streamable |
| Excluded `media/emulation/` | 57 G | re-downloadable |
| Excluded `media/Torrents/` | 19 G | temporary |
| Excluded `media/music/`, `media/podcasts/` | — | re-streamable |