docs/services/backrest.md

Backup Strategy

Single Backrest instance on CT 103 (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at /mnt/pve/unas, so it can read all service data without SSH-ing other hosts.

Goal: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.

---

Current state

Backrest (CT 103) — Phase 1b live since 2026-05-21

Item	Value
UI	http://192.168.1.3:9898 (LAN-only, auth disabled)
Config	/opt/backrest/config/config.json (timestamped .bak files on every edit)
rclone remote	jottacloud: Archive section (default device/mountpoint); token at /root/.config/rclone/rclone.conf
Offsite plan	JottaCloud Unlimited €9.91/mo (Norway, EEA) — see provider comparison
BW throttle	RCLONE_BWLIMIT=05:30,7.5M 01:00,20M (20 MB/s overnight, 7.5 MB/s daytime)

Repos (both autoInitialize: true; passwords rotated 2026-05-28 — stored in Vaultwarden):

Repo	Target	Prune	Check
services-repo	rclone:jottacloud:services	weekly Sun 05:00, ≤10 % unused	monthly, 10 % subset
media-repo	rclone:jottacloud:media	monthly, ≤10 % unused	monthly, 10 % subset

Plans:

Plan	Repo	Schedule	Retention	Paths
services-backup-plan	services-repo	0 1 * * 1-5 weekdays 01:00	7d · 4w · 6m · 1y	14 paths (below)
media-backup-plan	media-repo	30 1 * * 1-5 weekdays 01:30	4w · 6m · 1y	/mnt/pve/unas/media/images/library
video-projects-plan	media-repo	nightly 02:00	per config	/mnt/pve/unas/media/video-projects

Services plan paths (all under /mnt/pve/unas/, updated 2026-05-28):

services/coder                services/gitea               services/karakeep
services/memos                services/n8n                 services/nextcloud
services/pocketid-backup      services/shared-db-garage    services/traccar
services/vaultwarden
backup/home-assistant         backup/immich                backup/nextcloud
media/documents/public/paperless-ngx

Pre-backup hooks (run at CONDITION_SNAPSHOT_START):

• pocketid-prestage.sh — exports Pocket-ID zip + keys from CT 109 → UNAS
• garage-prestage.sh — rsyncs CT 104 Garage S3 data/meta → services/shared-db-garage
• n8n-prestage.sh — rsyncs CT 104 n8n config/nodes/storage → services/n8n

Hook scripts at /opt/backrest/scripts/ on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28.

services/shared-db was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud. services/pocketid removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by ops-backup.timer → Garage S3.

JottaCloud web UI tip: rclone writes to the Archive section. The default landing page shows only Sync + Backup. Browse to https://www.jottacloud.com/web/archive to see the restic repos.

Postgres (CT 113) — Phase 2 live since 2026-05-21

Dedicated db LXC at 192.168.1.6:5432 running postgres:17 in Docker, pgAdmin on :5050. WAL-G archives continuously to Garage S3 (ct113-pg-backup on CT 104):

• archive_mode = on, archive_command = 'wal-g wal-push %p'
• Daily wal-g backup-push cron at 02:00
• DBs hosted: vaultwarden, paperless, litellm, memos, n8n, gitea, coder

Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via /usr/local/sbin/walg-offsite-sync.sh (read-only Garage key GKef577420aadd26d667f2ca4f; mirrors ct113-pg-backup, lobe-pg-backup, immich-pg-backup → jottacloud:WAL-G/).

Exceptions (stay on original hosts):

DB	Host	Why
lobe-postgres (paradedb pg17)	CT 104	Uses pg_search USING bm25 indexes; stock postgres 17 can't host. WAL-G → lobe-pg-backup + Backrest secondary on data dir.
Nextcloud AIO postgres	CT 105	AIO manages it; Borg archives the whole stack to UNAS.
Immich postgres	CT 104	Version-pinned by Immich. WAL-G → immich-pg-backup.

---

Coverage map

Host / data	Method	Offsite
CT 103 Backrest binary + config	Manual (small)	On-CT only
CT 104 Docker app state on UNAS	services-backup-plan	✓ JottaCloud
CT 104 Vaultwarden data	moved to UNAS /services/vaultwarden → services-backup-plan	✓ JottaCloud
CT 104 Garage S3 data	garage-prestage.sh rsync → UNAS /services/shared-db-garage → services-backup-plan	✓ JottaCloud
CT 104 n8n config/encryption key	n8n-prestage.sh rsync → UNAS /services/n8n → services-backup-plan	✓ JottaCloud
Paperless-ngx media	UNAS /media/documents/public/paperless-ngx → services-backup-plan	✓ JottaCloud
CT 104 lobe-postgres (paradedb)	WAL-G → Garage → JottaCloud sync	✓
CT 104 Immich postgres	WAL-G → immich-pg-backup → sync	✓
CT 113 postgres (7 DBs)	WAL-G → ct113-pg-backup → sync	✓
CT 101 Shepard	Source in Gitea (gitea path covers it)	✓
CT 102 AdGuard	Phase 3 git push → fkrebs/adguard-conf	✓ daily cron
CT 104 Gitea + Coder	services/{gitea,coder} (UNAS; moved from CT 111 2026-05-26)	✓
CT 105 Nextcloud user files	services/nextcloud	✓
CT 105 Nextcloud AIO volumes	AIO Borg → /mnt/pve/unas/backup/nextcloud/ → Backrest	✓
CT 108 Zoraxy	DESTROYED 2026-05-28 — replaced by Traefik on CT 109	—
CT 109 Traefik config + acme.json	Not backed up (LE can reissue; config in /opt/stacks/traefik/)	✗ low risk
CT 109 Portainer config	Daily tar → Garage S3 ct109-portainer-backup → JottaCloud sync	✓
CT 109 Pocket-ID	Daily tar → Garage S3 ct109-portainer-backup (key ops-*.tar.gz) → JottaCloud sync via ops-backup.timer	✓
CT 109 Infisical	pg_dump in ops-*.tar.gz (same as above)	✓
CT 110 Pocket-ID	CT 110 destroyed 2026-05-26 — see CT 109 row above	✓
CT 111 Gitea + Coder	CT 111 destroyed 2026-05-26 — see CT 104 row above	✓
CT 112 Infisical	CT 112 destroyed 2026-05-26 — see CT 109 row above	✓
CT 108 Zoraxy	CT 108 destroyed 2026-05-28 — Traefik on CT 109 is replacement	—
VM 100 HAOS config	Phase 3 git addon → fkrebs/home-assistant-config	✗ Git Pull addon needs UI install
VM 100 HAOS daily tar	HA → UNAS → Backrest	✓
PVE host /etc/pve/	Phase 3 git push → fkrebs/pve-conf	✓ daily cron
UNAS media/images/library	Phase 1a rclone sync → jottacloud:Photos/	✓ Saturdays 01:00
UNAS personal data (documents, _sortMe, video-projects, …)	Phase 1a rclone sync → jottacloud:UNAS/	✓ Saturdays 01:00

Intentionally not backed up

Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable.

Deferred — include if needed

Host	Data	Notes
CT 109 Prometheus TSDB, Grafana dashboards	Low priority — metrics are ephemeral; dashboards re-exportable from Grafana; add to services plan if needed
CT 109 Portainer data	WAL-G–style daily tar → ct109-portainer-backup Garage S3 → JottaCloud offsite sync	✓ (via walg-offsite-sync.sh)

---

Open work

Phase 1a — rclone sync for UNAS personal data ✅ DEPLOYED 2026-05-22

Script /usr/local/sbin/unas-sync.sh on CT 103, cron 0 1 * * 6 (Saturdays 01:00). Last completed run: 2026-05-27 13h32m for 160486 transfers. Log: /var/log/rclone-unas-sync.log.

• Job 1: media/images/library/ → jottacloud:Photos/ (Immich originals, visible at jottacloud.com/photo).
• Job 2: /mnt/pve/unas/ → jottacloud:UNAS/ (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, services//backup/ already in Backrest, dump/).

BW throttle: --bwlimit "05:30,7.5M 01:00,20M" (same as Backrest).

Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.

Phase 3 — Config-to-git for infrastructure ✅ DEPLOYED

Daily cron on each host (~03:00) pushes config to private Gitea repos. Active deployments:

Host	Cron	Script	Target repo
CT 102 (AdGuard)	daily	/usr/local/sbin/adguard-conf-backup.sh	fkrebs/adguard-conf
CT 103 (Backrest)	0 3 * * *	/usr/local/sbin/ct103-conf-backup.sh	fkrebs/ct103-backrest
PVE host (nuc)	15 3 * * * (central)	/usr/local/bin/sync-config-repos	fkrebs/{ct103-backrest, ct113-db, ct104-stacks, ct109-stacks}
PVE host (nuc)	daily	/usr/local/sbin/pve-conf-backup.sh	fkrebs/pve-conf

sync-config-repos rebuilt 2026-05-28: removed dead CT 110/111/112 references, list of CT 104 stacks now derived from ls /opt/stacks (auto-discovers new stacks), added CT 109 ct109-stacks (auto-creates repo on first run).

Open: VM 100 (HAOS) → fkrebs/home-assistant-config — requires installing the native Git Pull addon via HAOS UI (cannot be automated remotely).

Blind spots

#	Severity	Issue	Status
1	CRITICAL	Vaultwarden data on CT 104 local FS	FIXED 2026-05-28 — volume moved to /mnt/pve/unas/services/vaultwarden/; path added to plan
2	HIGH	Garage S3 data on CT 104 local FS	FIXED 2026-05-28 — garage-prestage.sh rsyncs to UNAS before each backup; path services/shared-db-garage added to plan
3	MEDIUM	Paperless-ngx media not in any plan	FIXED 2026-05-28 — media/documents/public/paperless-ngx added to services-backup-plan
4	MEDIUM	n8n encryption key on CT 104 local FS	FIXED 2026-05-28 — n8n-prestage.sh rsyncs config/nodes/storage to UNAS before each backup
4b	MEDIUM	Nextcloud AIO Borg passphrase only in container env	Store BORG_PASSWORD in Vaultwarden — requires UI
5	MEDIUM	Backrest repo passwords were tapirnase	FIXED — passwords already rotated to complex strings in config.json
6	LOW	litellm DB placeholder password	N/A — LiteLLM decommissioned 2026-05-26; DB dropped
7	LOW	Migration SQL dumps at /mnt/pve/unas/dump/	N/A — no SQL dump files found; dumps were already cleaned up
8	LOW	Stale paths arcane, arr-stack, gluetun in plan	FIXED 2026-05-28 — removed from services-backup-plan

---

Operations

Restore

UI: Repos → snapshots → Browse → file → Restore.

CLI on CT 103:

restic -r rclone:jottacloud:services snapshots
restic -r rclone:jottacloud:services restore latest \
  --target /restore \
  --include /mnt/pve/unas/services/vaultwarden

Monitoring ✅ DEPLOYED 2026-05-28

Gotify hooks on all 3 plans (services / video-projects / media). Gotify app "Backrest" token AtIMGOYcZsb5HfO.

• CONDITION_SNAPSHOT_SUCCESS → Gotify priority 3 (✓ snapshot summary)
• CONDITION_SNAPSHOT_ERROR + CONDITION_SNAPSHOT_WARNING → Gotify priority 8 (✗ FAILED)

Hooks use Backrest's native actionGotify (not actionCommand) — no shell required, templating via Go text/template with {{ .Plan.Id }}, {{ .SnapshotStats.FilesNew }}, {{ formatSizeBytes .SnapshotStats.DataAdded }}, {{ .Duration }}, {{ .Error }}.

WAL-G staleness monitor (/usr/local/sbin/walg-staleness-check.sh on CT 103, hourly timer): SSH-then-docker exec psql to each Postgres host and queries pg_stat_archiver (last_archived_time, last_failed_time, failed_count, failed_wal). Alerts if archive >26h old OR a recent failure is more recent than the last success. Gotify token Ag570SdxcXcpp51. Hosts: CT 113 postgres, CT 104 immich_postgres. Closes fkrebs/n8n-flows#6.

Incident found by this monitor (2026-05-28): Immich wal-g wal-push had been failing with exit 124 (timeout) for 42 hours. Root cause: garage container was recreated on Docker bridge network instead of shared_backend during the 2026-05-28 Garage incident, making garage:3900 unresolvable from immich_postgres. Fix: docker network connect shared_backend garage; compose image pinned to v0.9.4 to match running.

Gitea daily digest (/usr/local/sbin/gitea-digest.sh + gitea-digest.timer on PVE host nuc, 08:00 daily): walks all fkrebs/* repos, finds yesterday's commits, posts summary to Gotify priority 3 (token A.qpozt3ZQGgEMG). Closes fkrebs/n8n-flows#5.

Applying a new config

# on CT 103
systemctl stop backrest
cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)
# paste new config.json
systemctl start backrest

---

History

Provider comparison

Researched 2026-05-21. Sized for ~3 TB/mo.

Provider	~3 TB/mo	Backend	EU DC	Egress	Verdict
JottaCloud Unlimited	€9.91 flat (unlimited)	rclone native	Norway (EEA)	Free	✅ Chosen
Hetzner BX31	€20.80 flat (10 TB)	SFTP/WebDAV	DE, FI	Free	2.5× the price, capped
Backblaze B2	~$18 pay-per-GB	S3	Frankfurt	Free (≤3× stored)	US CLOUD Act risk
Cloudflare R2	~$45 pay-per-GB	S3	EU auto	Zero	Expensive at scale; no EU residency lock
Wasabi	~$21–24	S3	FRA/AMS	Free (≤ stored)	❌ 90-day min billing per object → Restic prune disaster
Storj DCS	~$30	S3	EU-geofenced	1× free	Complex, $5 minimum
pCloud	€399 one-time / 2 TB	WebDAV	Luxembourg	Free	WebDAV too slow
Infomaniak kDrive	~€36+ / 3 TB	WebDAV	Switzerland	Free	WebDAV only; non-EU
Proton Drive	—	rclone beta	Switzerland	Free	rclone backend broken since late 2025

Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21)

Decision: provision a dedicated db LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse shared-postgres on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113).

Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, mp0=/mnt/pve/unas. Stack in Gitea fkrebs/stacks-db. pgAdmin pre-registers postgres via pgadmin-servers.json.

Migration order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB:

docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql
psql -U postgres -c "CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;"
psql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql
# Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume

Stale DBs dropped from shared-postgres post-migration: daytona, lobechat (duplicate), paradedb (duplicate).

WAL-G enabled 2026-05-21: archive_mode = on, archive_command = 'wal-g wal-push %p' via ALTER SYSTEM; first base backup verified (base_000000010000000000000012). Daily wal-g backup-push cron at 02:00. Garage → JottaCloud offsite sync added at 02:30.

Phase 1a rclone sync script

#!/bin/bash
set -e
LOG=/var/log/rclone-unas-sync.log

# Job 1: Immich originals → JottaCloud gallery
rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \
  --transfers=4 --checkers=8 \
  --log-file=$LOG --log-level INFO

# Job 2: all other irreplaceable personal data
rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \
  --transfers=4 --checkers=8 \
  --exclude "media/images/library/**" \
  --exclude "media/images/upload/**" \
  --exclude "media/images/thumbs/**" \
  --exclude "media/images/encoded-video/**" \
  --exclude "media/images/profile/**" \
  --exclude "media/images/backups/**" \
  --exclude "media/movies/**" \
  --exclude "media/emulation/**" \
  --exclude "media/Torrents/**" \
  --exclude "media/music/**" \
  --exclude "media/podcasts/**" \
  --exclude "services/**" \
  --exclude "backup/**" \
  --exclude "backup-staging/**" \
  --exclude "test_perm" \
  --log-file=$LOG --log-level INFO

Cron (Saturday 01:00): 0 1 * * 6 /usr/local/sbin/unas-sync.sh

Phase 3 config-to-git scripts

Same pattern on each host. Replace GITEA_TOKEN with the value from /opt/stacks/ai/.env or a dedicated scoped Gitea token.

CT 108 — Zoraxy (/usr/local/sbin/zoraxy-conf-backup.sh):

#!/bin/bash
set -e
REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git"
WORK="/opt/zoraxy/conf"
git -C "$WORK" init -b main -q 2>/dev/null || true
git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \
  || git -C "$WORK" remote add origin "$REPO_URL"
git -C "$WORK" add -A
git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true
git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true

CT 102 — AdGuard — same template, copy /opt/AdGuardHome/AdGuardHome.yaml into /tmp/adguard-conf-work checkout first.

PVE host — same template, rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/ then commit.

VM 100 — HA — native Git Pull addon pushing /config/ (exclude secrets.yaml, .storage/) to fkrebs/ha-config. HA daily tars already covered by services-backup-plan via /mnt/pve/unas/backup/home-assistant/.

Cron on each host: 0 3 * * * /usr/local/sbin/<host>-conf-backup.sh

Why Wasabi was rejected

Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every restic forget --prune generates surprise costs. Well-documented Restic-on-Wasabi trap.

Why lobe-postgres stays on CT 104

Migration 0093_add_bm25_indexes_with_icu.sql creates USING bm25 indexes on 7 tables (agents, topics, files, knowledge_bases, user_memories, chat_groups, user_memories_contexts). bm25 is paradedb-only (pg_search extension); stock postgres 17 has no such index access method and the migration fails. Container stays paradedb/paradedb:latest-pg17. WAL-G archives to lobe-pg-backup; Backrest secondary copies the data dir.

UNAS data inventory (basis for Phase 1a sizing)

Path	Size	Notes
media/documents/	9.5 G	Personal documents
media/video-projects/	425 G	Creative work, irreplaceable
media/musical-sheets/	23 G
media/audiobooks/	23 G
media/ebooks/	6.8 G
media/3d-prints/	65 M
media/Recipes/	113 M
_sortMe/	335 G	images/ 171 G, work Flo/ 147 G, Anne/ 17 G
code/	6.7 M
Excluded media/movies/	118 G	re-streamable
Excluded media/emulation/	57 G	re-downloadable
Excluded media/Torrents/	19 G	temporary
Excluded media/music/, media/podcasts/	—	re-streamable