Skip to content

# Backup Strategy

> Single Backrest instance on CT 103 (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at `/mnt/pve/unas`, so it can read all service data without SSH-ing other hosts.

Goal: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.

## Current state

### Backrest (CT 103) — Phase 1b live since 2026-05-21

Item | Value
--- | ---
UI | `http://192.168.1.3:9898` (LAN-only, auth disabled)
Config | `/opt/backrest/config/config.json` (timestamped `.bak` files on every edit)
rclone remote | `jottacloud:` Archive section (default device/mountpoint); token at `/root/.config/rclone/rclone.conf`
Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see provider comparison
BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime)

Repos (both `autoInitialize: true`; passwords currently `tapirnase` — see blind spot #5):

Repo | Target | Prune | Check
--- | --- | --- | ---
`services-repo` | `rclone:jottacloud:services` | weekly Sun 05:00, ≤10 % unused | monthly, 10 % subset
`media-repo` | `rclone:jottacloud:media` | monthly, ≤10 % unused | monthly, 10 % subset

Plans:

Plan | Repo | Schedule | Retention | Paths
--- | --- | --- | --- | ---
`services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 15 paths (below)
`media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library`
`video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects`

Services plan paths (all under `/mnt/pve/unas/`):

    services/vaultwarden    services/n8n         services/memos
    services/karakeep       services/traccar     services/gitea
    services/coder          services/nextcloud
    services/arr-stack      services/gluetun
    backup/home-assistant   backup/immich        backup/nextcloud

`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud. `services/arcane` removed 2026-05-26 — Arcane decommissioned, replaced by Portainer. `services/pocketid` removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by `ops-backup.timer` → Garage S3.

> JottaCloud web UI tip: rclone writes to the Archive section. The default landing page shows only Sync + Backup. Browse to `https://www.jottacloud.com/web/archive` to see the restic repos.

### Postgres (CT 113) — Phase 2 live since 2026-05-21

Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmin on `:5050`. WAL-G archives continuously to Garage S3 (`ct113-pg-backup` on CT 104):

- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
- Daily `wal-g backup-push` cron at 02:00
- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder`

Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`).

Exceptions (stay on original hosts):

DB | Host | Why
--- | --- | ---
`lobe-postgres` (paradedb pg17) | CT 104 | Uses `pg_search` `USING bm25` indexes; stock postgres 17 can't host. WAL-G → `lobe-pg-backup` + Backrest secondary on data dir.
Nextcloud AIO postgres | CT 105 | AIO manages it; Borg archives the whole stack to UNAS.
Immich postgres | CT 104 | Version-pinned by Immich. WAL-G → `immich-pg-backup`.

## Coverage map

Host / data | Method | Offsite
--- | --- | ---
CT 103 Backrest binary + config | Manual (small) | On-CT only
CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud
CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓
CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓
CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓
CT 101 Shepard | Source in Gitea (gitea path covers it) | ✓
CT 102 AdGuard | Phase 3 git push → `fkrebs/adguard-conf` | ✗ not deployed
CT 104 Gitea + Coder | `services/{gitea,coder}` (UNAS; moved from CT 111 2026-05-26) | ✓
CT 105 Nextcloud user files | `services/nextcloud` | ✓
CT 105 Nextcloud AIO volumes | AIO Borg → `/mnt/pve/unas/backup/nextcloud/` → Backrest | ✓
CT 108 Zoraxy | Phase 3 git push → `fkrebs/zoraxy-conf` | ✗ not deployed
CT 109 Portainer config | Daily tar → Garage S3 `ct109-portainer-backup` → JottaCloud sync | ✓
CT 109 Pocket-ID | Daily tar → Garage S3 `ct109-portainer-backup` (key `ops-*.tar.gz`) → JottaCloud sync via `ops-backup.timer` | ✓
CT 109 Infisical | pg_dump in `ops-*.tar.gz` (same as above) | ✓
~~CT 110 Pocket-ID~~ | CT 110 destroyed 2026-05-26 — see CT 109 row above | ✓
~~CT 111 Gitea + Coder~~ | CT 111 destroyed 2026-05-26 — see CT 104 row above | ✓
~~CT 112 Infisical~~ | CT 112 destroyed 2026-05-26 — see CT 109 row above | ✓
VM 100 HAOS config | Phase 3 git addon → `fkrebs/ha-config` | ✗ not deployed
VM 100 HAOS daily tar | HA → UNAS → Backrest | ✓
PVE host `/etc/pve/` | Phase 3 git push → `fkrebs/pve-conf` | ✗ not deployed
UNAS `media/images/library` | Phase 1a `rclone sync` → `jottacloud:Photos/` | ✗ not deployed
UNAS personal data (`documents`, `_sortMe`, `video-projects`, …) | Phase 1a `rclone sync` → `jottacloud:UNAS/` | ✗ not deployed

### Intentionally not backed up

Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable.

### Deferred — include if needed

Host | Data | Notes
--- | --- | ---
CT 109 Prometheus TSDB, Grafana dashboards | Low priority — metrics are ephemeral; dashboards re-exportable from Grafana; add to services plan if needed |
CT 109 Portainer data | WAL-G–style daily tar → `ct109-portainer-backup` Garage S3 → JottaCloud offsite sync | ✓ (via walg-offsite-sync.sh)

## Open work

### Phase 1a — rclone sync for UNAS personal data

Two sync jobs on CT 103, Saturday 01:00:

- Job 1: `media/images/library/` → `jottacloud:Photos/` (Immich originals, visible at `jottacloud.com/photo`).
- Job 2: `/mnt/pve/unas/` → `jottacloud:UNAS/` (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, `services/` already in Backrest).

Initial upload ≈ 820 GB personal + 822 GB Immich (days). Script target `/usr/local/sbin/unas-sync.sh`. Full script in appendix.

Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.

### Phase 3 — Config-to-git for infrastructure

Daily 03:00 cron on each host pushes config to a private Gitea repo. Repos already exist:

- `fkrebs/zoraxy-conf` (CT 108 — `/opt/zoraxy/conf/`)
- `fkrebs/adguard-conf` (CT 102 — `AdGuardHome.yaml`)
- `fkrebs/pve-conf` (PVE — `/etc/pve/`, excludes `priv/`, `*.key`, `authkey.pub*`)
- `fkrebs/ha-config` (VM 100 — `/config/`, excludes `secrets.yaml`, `.storage/`)

Script template in appendix.

### Blind spots

# | Severity | Issue | Fix
--- | --- | --- | ---
4 | MEDIUM | n8n encryption key in `/opt/stacks/n8n/data/` on CT 104 local FS — not in any plan. If CT 104 dies, DB restore is unusable. | Move n8n data volume to `/mnt/pve/unas/services/n8n/` (already in plan).
4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted — without it, restore impossible. | Store `BORG_PASSWORD` in Vaultwarden.
5 | MEDIUM | Both Backrest repo passwords are `tapirnase`. | Rotate before first scheduled run completes; `restic key passwd` re-encrypts in place. Store in Vaultwarden.
6 | LOW | `litellm` DB password is the placeholder literal `litellm_password_here`. | Generate real password; update `ai/.env`, `litellm-config/config.yaml`, CT 113 user.
7 | LOW | Migration dumps at `/mnt/pve/unas/dump/*-migration-20260521.sql` not in any plan. | Decide: keep as manual archive or delete now that WAL-G is archiving.

## Operations

### Restore

UI: Repos → snapshots → Browse → file → Restore.

CLI on CT 103:

    restic -r rclone:jottacloud:services snapshots
    restic -r rclone:jottacloud:services restore latest \
      --target /restore \
      --include /mnt/pve/unas/services/vaultwarden

### Monitoring (planned)

In Backrest UI → each repo → Hooks:

- `CONDITION_BACKUP_ERROR` / `CONDITION_CHECK_ERROR` → POST Gotify priority 8
- `CONDITION_BACKUP_SUCCESS` → POST Gotify priority 3

    curl -s -X POST 'http://gotify:80/message?token=TOKEN' \
      -H 'Content-Type: application/json' \
      -d '{"title":"Backrest: {{.Plan}}","message":"{{.Summary}}","priority":3}'

### Applying a new config

    # on CT 103
    systemctl stop backrest
    cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)
    # paste new config.json
    systemctl start backrest

## History

### Provider comparison

Researched 2026-05-21. Sized for ~3 TB/mo.

Provider | ~3 TB/mo | Backend | EU DC | Egress | Verdict
--- | --- | --- | --- | --- | ---
JottaCloud Unlimited | €9.91 flat (unlimited) | rclone native | Norway (EEA) | Free | ✅ Chosen
Hetzner BX31 | €20.80 flat (10 TB) | SFTP/WebDAV | DE, FI | Free | 2.5× the price, capped
Backblaze B2 | ~$18 pay-per-GB | S3 | Frankfurt | Free (≤3× stored) | US CLOUD Act risk
Cloudflare R2 | ~$45 pay-per-GB | S3 | EU auto | Zero | Expensive at scale; no EU residency lock
Wasabi | ~$21–24 | S3 | FRA/AMS | Free (≤ stored) | ❌ 90-day min billing per object → Restic prune disaster
Storj DCS | ~$30 | S3 | EU-geofenced | 1× free | Complex, $5 minimum
pCloud | €399 one-time / 2 TB | WebDAV | Luxembourg | Free | WebDAV too slow
Infomaniak kDrive | ~€36+ / 3 TB | WebDAV | Switzerland | Free | WebDAV only; non-EU
Proton Drive | — | rclone beta | Switzerland | Free | rclone backend broken since late 2025

### Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21)

Decision: provision a dedicated `db` LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse `shared-postgres` on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113).

Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, `mp0=/mnt/pve/unas`. Stack in Gitea `fkrebs/stacks-db`. pgAdmin pre-registers postgres via `pgadmin-servers.json`.

Migration order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB:

    docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql
    psql -U postgres -c "CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;"
    psql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql
    # Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume

Stale DBs dropped from `shared-postgres` post-migration: `daytona`, `lobechat` (duplicate), `paradedb` (duplicate).

WAL-G enabled 2026-05-21: `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` via `ALTER SYSTEM`; first base backup verified (`base_000000010000000000000012`). Daily `wal-g backup-push` cron at 02:00. Garage → JottaCloud offsite sync added at 02:30.

### Phase 1a rclone sync script

    #!/bin/bash
    set -e
    LOG=/var/log/rclone-unas-sync.log

    # Job 1: Immich originals → JottaCloud gallery
    rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \
      --transfers=4 --checkers=8 \
      --log-file=$LOG --log-level INFO

    # Job 2: all other irreplaceable personal data
    rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \
      --transfers=4 --checkers=8 \
      --exclude "media/images/library/**" \
      --exclude "media/images/upload/**" \
      --exclude "media/images/thumbs/**" \
      --exclude "media/images/encoded-video/**" \
      --exclude "media/images/profile/**" \
      --exclude "media/images/backups/**" \
      --exclude "media/movies/**" \
      --exclude "media/emulation/**" \
      --exclude "media/Torrents/**" \
      --exclude "media/music/**" \
      --exclude "media/podcasts/**" \
      --exclude "services/**" \
      --exclude "backup/**" \
      --exclude "backup-staging/**" \
      --exclude "test_perm" \
      --log-file=$LOG --log-level INFO

Cron (Saturday 01:00): `0 1 * * 6 /usr/local/sbin/unas-sync.sh`

### Phase 3 config-to-git scripts

Same pattern on each host. Replace `GITEA_TOKEN` with the value from `/opt/stacks/ai/.env` or a dedicated scoped Gitea token.

CT 108 — Zoraxy (`/usr/local/sbin/zoraxy-conf-backup.sh`):

    #!/bin/bash
    set -e
    REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git"
    WORK="/opt/zoraxy/conf"
    git -C "$WORK" init -b main -q 2>/dev/null || true
    git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \
      || git -C "$WORK" remote add origin "$REPO_URL"
    git -C "$WORK" add -A
    git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true
    git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true

CT 102 — AdGuard — same template, copy `/opt/AdGuardHome/AdGuardHome.yaml` into `/tmp/adguard-conf-work` checkout first.

PVE host — same template, `rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/` then commit.

VM 100 — HA — native Git Pull addon pushing `/config/` (exclude `secrets.yaml`, `.storage/`) to `fkrebs/ha-config`. HA daily tars already covered by `services-backup-plan` via `/mnt/pve/unas/backup/home-assistant/`.

Cron on each host: `0 3 * * * /usr/local/sbin/<host>-conf-backup.sh`

### Why Wasabi was rejected

Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every `restic forget --prune` generates surprise costs. Well-documented Restic-on-Wasabi trap.

### Why lobe-postgres stays on CT 104

Migration `0093_add_bm25_indexes_with_icu.sql` creates `USING bm25` indexes on 7 tables (`agents`, `topics`, `files`, `knowledge_bases`, `user_memories`, `chat_groups`, `user_memories_contexts`). `bm25` is paradedb-only (`pg_search` extension); stock postgres 17 has no such index access method and the migration fails. Container stays `paradedb/paradedb:latest-pg17`. WAL-G archives to `lobe-pg-backup`; Backrest secondary copies the data dir.

### UNAS data inventory (basis for Phase 1a sizing)

Path | Size | Notes
--- | --- | ---
`media/documents/` | 9.5 G | Personal documents
`media/video-projects/` | 425 G | Creative work, irreplaceable
`media/musical-sheets/` | 23 G |
`media/audiobooks/` | 23 G |
`media/ebooks/` | 6.8 G |
`media/3d-prints/` | 65 M |
`media/Recipes/` | 113 M |
`_sortMe/` | 335 G | `images/` 171 G, `work Flo/` 147 G, `Anne/` 17 G
`code/` | 6.7 M |
Excluded `media/movies/` | 118 G | re-streamable
Excluded `media/emulation/` | 57 G | re-downloadable
Excluded `media/Torrents/` | 19 G | temporary
Excluded `media/music/`, `media/podcasts/` | — | re-streamable