fkrebs
ae20382cbe
- LiteLLM sunset 2026-05-28: container removed, config dir renamed .DECOMMISSIONED-2026-05-28, prometheus scrape job deleted, CT 113 litellm DB + role dropped. Bifrost now connects directly to upstream LLM providers (verified: openai → SAIA chat-ai.academiccloud.de). - Open WebUI data (CT 104 uploads + vector_db, 192K) now staged to UNAS /services/open-webui via openwebui-prestage.sh and included in Backrest services-backup-plan. Open WebUI chats/users live in CT 113 Postgres owui DB (already in WAL-G). - Miniflux confirmed DB-only — already covered by WAL-G. - backrest.md and homelab-architecture.md updated; services-overview.md marks LiteLLM row strikethrough.
341 lines
20 KiB
Markdown
341 lines
20 KiB
Markdown
# Backup Strategy
|
||
|
||
> Single Backrest instance on **CT 103** (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at `/mnt/pve/unas`, so it can read all service data without SSH-ing other hosts.
|
||
|
||
**Goal**: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.
|
||
|
||
---
|
||
|
||
## Current state
|
||
|
||
### Backrest (CT 103) — Phase 1b live since 2026-05-21
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| UI | `http://192.168.1.3:9898` (LAN-only, auth disabled) |
|
||
| Config | `/opt/backrest/config/config.json` (timestamped `.bak` files on every edit) |
|
||
| rclone remote | `jottacloud:` Archive section (default device/mountpoint); token at `/root/.config/rclone/rclone.conf` |
|
||
| Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) |
|
||
| BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) |
|
||
|
||
**Repos** (both `autoInitialize: true`; passwords rotated 2026-05-28 — stored in Vaultwarden):
|
||
|
||
| Repo | Target | Prune | Check |
|
||
|---|---|---|---|
|
||
| `services-repo` | `rclone:jottacloud:services` | weekly Sun 05:00, ≤10 % unused | monthly, 10 % subset |
|
||
| `media-repo` | `rclone:jottacloud:media` | monthly, ≤10 % unused | monthly, 10 % subset |
|
||
|
||
**Plans**:
|
||
|
||
| Plan | Repo | Schedule | Retention | Paths |
|
||
|---|---|---|---|---|
|
||
| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 14 paths (below) |
|
||
| `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` |
|
||
| `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` |
|
||
|
||
**Services plan paths** (all under `/mnt/pve/unas/`, updated 2026-05-28):
|
||
|
||
```
|
||
services/coder services/gitea services/karakeep
|
||
services/memos services/n8n services/nextcloud
|
||
services/pocketid-backup services/shared-db-garage services/traccar
|
||
services/vaultwarden
|
||
backup/home-assistant backup/immich backup/nextcloud
|
||
media/documents/public/paperless-ngx
|
||
```
|
||
|
||
**Pre-backup hooks** (run at `CONDITION_SNAPSHOT_START`):
|
||
- `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS
|
||
- `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage`
|
||
- `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n`
|
||
- `openwebui-prestage.sh` — rsyncs CT 104 Open WebUI uploads + vector_db → `services/open-webui` (cache/ excluded)
|
||
|
||
Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28.
|
||
|
||
`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud.
|
||
`services/pocketid` removed 2026-05-26 — Pocket-ID moved to CT 109 local FS (not UNAS); now covered by `ops-backup.timer` → Garage S3.
|
||
|
||
> **JottaCloud web UI tip**: rclone writes to the **Archive** section. The default landing page shows only Sync + Backup. Browse to `https://www.jottacloud.com/web/archive` to see the restic repos.
|
||
|
||
### Postgres (CT 113) — Phase 2 live since 2026-05-21
|
||
|
||
Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmin on `:5050`. WAL-G archives continuously to Garage S3 (`ct113-pg-backup` on CT 104):
|
||
|
||
- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
|
||
- Daily `wal-g backup-push` cron at 02:00
|
||
- DBs hosted: `vaultwarden`, `paperless`, `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux` (`litellm` DB + role dropped 2026-05-28 when LiteLLM was sunset)
|
||
- `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy
|
||
- WAL-G monitor (CT 103) checks this via `pg_stat_archiver`
|
||
|
||
Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`).
|
||
|
||
**Exceptions** (stay on original hosts):
|
||
|
||
| DB | Host | Why |
|
||
|---|---|---|
|
||
| `lobe-postgres` (paradedb pg17) | CT 104 | Uses `pg_search` `USING bm25` indexes; stock postgres 17 can't host. WAL-G → `lobe-pg-backup` + Backrest secondary on data dir. |
|
||
| Nextcloud AIO postgres | CT 105 | AIO manages it; Borg archives the whole stack to UNAS. |
|
||
| Immich postgres | CT 104 | Version-pinned by Immich. WAL-G → `immich-pg-backup`. |
|
||
|
||
---
|
||
|
||
## Coverage map
|
||
|
||
| Host / data | Method | Offsite |
|
||
|---|---|---|
|
||
| CT 103 Backrest binary + config | Manual (small) | On-CT only |
|
||
| CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud |
|
||
| CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud |
|
||
| CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud |
|
||
| CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud |
|
||
| CT 104 Open WebUI uploads + vector_db | `openwebui-prestage.sh` rsync → UNAS `/services/open-webui` → `services-backup-plan` | ✓ JottaCloud |
|
||
| Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud |
|
||
| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ |
|
||
| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ |
|
||
| CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ |
|
||
| CT 101 Shepard | Source in Gitea (gitea path covers it) | ✓ |
|
||
| CT 102 AdGuard | Phase 3 git push → `fkrebs/adguard-conf` | ✓ daily cron |
|
||
| CT 104 Gitea + Coder | `services/{gitea,coder}` (UNAS; moved from CT 111 2026-05-26) | ✓ |
|
||
| CT 105 Nextcloud user files | `services/nextcloud` | ✓ |
|
||
| CT 105 Nextcloud AIO volumes | AIO Borg → `/mnt/pve/unas/backup/nextcloud/` → Backrest | ✓ |
|
||
| ~~CT 108 Zoraxy~~ | **DESTROYED 2026-05-28** — replaced by Traefik on CT 109 | — |
|
||
| CT 109 Traefik config + acme.json | Not backed up (LE can reissue; config in `/opt/stacks/traefik/`) | ✗ low risk |
|
||
| CT 109 Portainer config | Daily tar → Garage S3 `ct109-portainer-backup` → JottaCloud sync | ✓ |
|
||
| CT 109 Pocket-ID | Daily tar → Garage S3 `ct109-portainer-backup` (key `ops-*.tar.gz`) → JottaCloud sync via `ops-backup.timer` | ✓ |
|
||
| CT 109 Infisical | pg_dump in `ops-*.tar.gz` (same as above) | ✓ |
|
||
| ~~CT 110 Pocket-ID~~ | **CT 110 destroyed 2026-05-26 — see CT 109 row above** | ✓ |
|
||
| ~~CT 111 Gitea + Coder~~ | **CT 111 destroyed 2026-05-26 — see CT 104 row above** | ✓ |
|
||
| ~~CT 112 Infisical~~ | **CT 112 destroyed 2026-05-26 — see CT 109 row above** | ✓ |
|
||
| ~~CT 108 Zoraxy~~ | **CT 108 destroyed 2026-05-28 — Traefik on CT 109 is replacement** | — |
|
||
| VM 100 HAOS config | Phase 3 git addon → `fkrebs/home-assistant-config` | ✗ Git Pull addon needs UI install |
|
||
| VM 100 HAOS daily tar | HA → UNAS → Backrest | ✓ |
|
||
| PVE host `/etc/pve/` | Phase 3 git push → `fkrebs/pve-conf` | ✓ daily cron |
|
||
| UNAS `media/images/library` | Phase 1a `rclone sync` → `jottacloud:Photos/` | ✓ Saturdays 01:00 |
|
||
| UNAS personal data (`documents`, `_sortMe`, `video-projects`, …) | Phase 1a `rclone sync` → `jottacloud:UNAS/` | ✓ Saturdays 01:00 |
|
||
|
||
### Intentionally not backed up
|
||
|
||
Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable.
|
||
|
||
### Deferred — include if needed
|
||
|
||
| Host | Data | Notes |
|
||
|---|---|---|
|
||
| CT 109 Prometheus TSDB, Grafana dashboards | Low priority — metrics are ephemeral; dashboards re-exportable from Grafana; add to services plan if needed |
|
||
| CT 109 Portainer data | WAL-G–style daily tar → `ct109-portainer-backup` Garage S3 → JottaCloud offsite sync | ✓ (via walg-offsite-sync.sh) |
|
||
|
||
---
|
||
|
||
## Open work
|
||
|
||
### Phase 1a — rclone sync for UNAS personal data ✅ DEPLOYED 2026-05-22
|
||
|
||
Script `/usr/local/sbin/unas-sync.sh` on CT 103, cron `0 1 * * 6` (Saturdays 01:00). Last completed run: 2026-05-27 13h32m for 160486 transfers. Log: `/var/log/rclone-unas-sync.log`.
|
||
|
||
- **Job 1**: `media/images/library/` → `jottacloud:Photos/` (Immich originals, visible at `jottacloud.com/photo`).
|
||
- **Job 2**: `/mnt/pve/unas/` → `jottacloud:UNAS/` (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, `services/`/`backup/` already in Backrest, `dump/`).
|
||
|
||
BW throttle: `--bwlimit "05:30,7.5M 01:00,20M"` (same as Backrest).
|
||
|
||
Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.
|
||
|
||
### Phase 3 — Config-to-git for infrastructure ✅ DEPLOYED
|
||
|
||
Daily cron on each host (~03:00) pushes config to private Gitea repos. Active deployments:
|
||
|
||
| Host | Cron | Script | Target repo |
|
||
|---|---|---|---|
|
||
| CT 102 (AdGuard) | daily | `/usr/local/sbin/adguard-conf-backup.sh` | `fkrebs/adguard-conf` |
|
||
| CT 103 (Backrest) | `0 3 * * *` | `/usr/local/sbin/ct103-conf-backup.sh` | `fkrebs/ct103-backrest` |
|
||
| PVE host (`nuc`) | `15 3 * * *` (central) | `/usr/local/bin/sync-config-repos` | `fkrebs/{ct103-backrest, ct113-db, ct104-stacks, ct109-stacks}` |
|
||
| PVE host (`nuc`) | daily | `/usr/local/sbin/pve-conf-backup.sh` | `fkrebs/pve-conf` |
|
||
|
||
`sync-config-repos` rebuilt 2026-05-28: removed dead CT 110/111/112 references, list of CT 104 stacks now derived from `ls /opt/stacks` (auto-discovers new stacks), added CT 109 `ct109-stacks` (auto-creates repo on first run).
|
||
|
||
**Open**: VM 100 (HAOS) → `fkrebs/home-assistant-config` — requires installing the native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) via HAOS UI (cannot be automated remotely).
|
||
|
||
### Blind spots
|
||
|
||
| # | Severity | Issue | Status |
|
||
|---|---|---|---|
|
||
| 1 | ~~CRITICAL~~ | ~~Vaultwarden data on CT 104 local FS~~ | **FIXED 2026-05-28** — volume moved to `/mnt/pve/unas/services/vaultwarden/`; path added to plan |
|
||
| 2 | ~~HIGH~~ | ~~Garage S3 data on CT 104 local FS~~ | **FIXED 2026-05-28** — `garage-prestage.sh` rsyncs to UNAS before each backup; path `services/shared-db-garage` added to plan |
|
||
| 3 | ~~MEDIUM~~ | ~~Paperless-ngx media not in any plan~~ | **FIXED 2026-05-28** — `media/documents/public/paperless-ngx` added to `services-backup-plan` |
|
||
| 4 | ~~MEDIUM~~ | ~~n8n encryption key on CT 104 local FS~~ | **FIXED 2026-05-28** — `n8n-prestage.sh` rsyncs config/nodes/storage to UNAS before each backup |
|
||
| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env | Store `BORG_PASSWORD` in Vaultwarden — requires UI |
|
||
| 5 | ~~MEDIUM~~ | ~~Backrest repo passwords were `tapirnase`~~ | **FIXED** — passwords already rotated to complex strings in config.json |
|
||
| 6 | ~~LOW~~ | ~~`litellm` DB placeholder password~~ | **N/A** — LiteLLM decommissioned 2026-05-26; DB dropped |
|
||
| 7 | ~~LOW~~ | ~~Migration SQL dumps at `/mnt/pve/unas/dump/`~~ | **N/A** — no SQL dump files found; dumps were already cleaned up |
|
||
| 8 | ~~LOW~~ | ~~Stale paths `arcane`, `arr-stack`, `gluetun` in plan~~ | **FIXED 2026-05-28** — removed from `services-backup-plan` |
|
||
|
||
---
|
||
|
||
## Operations
|
||
|
||
### Restore
|
||
|
||
UI: **Repos → snapshots → Browse → file → Restore**.
|
||
|
||
CLI on CT 103:
|
||
|
||
```bash
|
||
restic -r rclone:jottacloud:services snapshots
|
||
restic -r rclone:jottacloud:services restore latest \
|
||
--target /restore \
|
||
--include /mnt/pve/unas/services/vaultwarden
|
||
```
|
||
|
||
### Monitoring ✅ DEPLOYED 2026-05-28
|
||
|
||
Gotify hooks on all 3 plans (services / video-projects / media). Gotify app "Backrest" token `AtIMGOYcZsb5HfO`.
|
||
|
||
- `CONDITION_SNAPSHOT_SUCCESS` → Gotify priority 3 (✓ snapshot summary)
|
||
- `CONDITION_SNAPSHOT_ERROR` + `CONDITION_SNAPSHOT_WARNING` → Gotify priority 8 (✗ FAILED)
|
||
|
||
Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell required, templating via Go text/template with `{{ .Plan.Id }}`, `{{ .SnapshotStats.FilesNew }}`, `{{ formatSizeBytes .SnapshotStats.DataAdded }}`, `{{ .Duration }}`, `{{ .Error }}`.
|
||
|
||
**WAL-G staleness monitor** (`/usr/local/sbin/walg-staleness-check.sh` on CT 103, hourly timer): SSH-then-`docker exec psql` to each Postgres host and queries `pg_stat_archiver` (last_archived_time, last_failed_time, failed_count, failed_wal). Alerts if archive >26h old OR a recent failure is more recent than the last success. Gotify token `Ag570SdxcXcpp51`. Hosts: CT 113 `postgres`, CT 104 `immich_postgres`. Closes `fkrebs/n8n-flows#6`.
|
||
|
||
> **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running.
|
||
|
||
> **Initial false-positive (2026-05-28 16:00)**: First scheduled run alerted "CT 113 query failed". Cause: CT 103 had no SSH access to CT 113 (host key + authorized_keys never deployed); `2>/dev/null` in the script hid the underlying `Host key verification failed` error. Fix: deployed CT 103 pub key to CT 113 + `ssh-keyscan`; added retry loop (2 attempts, 5s gap, 10s ConnectTimeout) for transient blips.
|
||
|
||
**Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`.
|
||
|
||
### Applying a new config
|
||
|
||
```bash
|
||
# on CT 103
|
||
systemctl stop backrest
|
||
cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)
|
||
# paste new config.json
|
||
systemctl start backrest
|
||
```
|
||
|
||
---
|
||
|
||
## History
|
||
|
||
### Provider comparison
|
||
|
||
Researched 2026-05-21. Sized for ~3 TB/mo.
|
||
|
||
| Provider | ~3 TB/mo | Backend | EU DC | Egress | Verdict |
|
||
|---|---|---|---|---|---|
|
||
| **JottaCloud Unlimited** | €9.91 flat (unlimited) | rclone native | Norway (EEA) | Free | ✅ Chosen |
|
||
| Hetzner BX31 | €20.80 flat (10 TB) | SFTP/WebDAV | DE, FI | Free | 2.5× the price, capped |
|
||
| Backblaze B2 | ~$18 pay-per-GB | S3 | Frankfurt | Free (≤3× stored) | US CLOUD Act risk |
|
||
| Cloudflare R2 | ~$45 pay-per-GB | S3 | EU auto | Zero | Expensive at scale; no EU residency lock |
|
||
| Wasabi | ~$21–24 | S3 | FRA/AMS | Free (≤ stored) | ❌ 90-day min billing per object → Restic prune disaster |
|
||
| Storj DCS | ~$30 | S3 | EU-geofenced | 1× free | Complex, $5 minimum |
|
||
| pCloud | €399 one-time / 2 TB | WebDAV | Luxembourg | Free | WebDAV too slow |
|
||
| Infomaniak kDrive | ~€36+ / 3 TB | WebDAV | Switzerland | Free | WebDAV only; non-EU |
|
||
| Proton Drive | — | rclone beta | Switzerland | Free | rclone backend broken since late 2025 |
|
||
|
||
### Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21)
|
||
|
||
**Decision**: provision a dedicated `db` LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse `shared-postgres` on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113).
|
||
|
||
Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, `mp0=/mnt/pve/unas`. Stack in Gitea `fkrebs/stacks-db`. pgAdmin pre-registers postgres via `pgadmin-servers.json`.
|
||
|
||
**Migration order**: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB:
|
||
|
||
```bash
|
||
docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql
|
||
psql -U postgres -c "CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;"
|
||
psql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql
|
||
# Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume
|
||
```
|
||
|
||
Stale DBs dropped from `shared-postgres` post-migration: `daytona`, `lobechat` (duplicate), `paradedb` (duplicate).
|
||
|
||
WAL-G enabled 2026-05-21: `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` via `ALTER SYSTEM`; first base backup verified (`base_000000010000000000000012`). Daily `wal-g backup-push` cron at 02:00. Garage → JottaCloud offsite sync added at 02:30.
|
||
|
||
### Phase 1a rclone sync script
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -e
|
||
LOG=/var/log/rclone-unas-sync.log
|
||
|
||
# Job 1: Immich originals → JottaCloud gallery
|
||
rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \
|
||
--transfers=4 --checkers=8 \
|
||
--log-file=$LOG --log-level INFO
|
||
|
||
# Job 2: all other irreplaceable personal data
|
||
rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \
|
||
--transfers=4 --checkers=8 \
|
||
--exclude "media/images/library/**" \
|
||
--exclude "media/images/upload/**" \
|
||
--exclude "media/images/thumbs/**" \
|
||
--exclude "media/images/encoded-video/**" \
|
||
--exclude "media/images/profile/**" \
|
||
--exclude "media/images/backups/**" \
|
||
--exclude "media/movies/**" \
|
||
--exclude "media/emulation/**" \
|
||
--exclude "media/Torrents/**" \
|
||
--exclude "media/music/**" \
|
||
--exclude "media/podcasts/**" \
|
||
--exclude "services/**" \
|
||
--exclude "backup/**" \
|
||
--exclude "backup-staging/**" \
|
||
--exclude "test_perm" \
|
||
--log-file=$LOG --log-level INFO
|
||
```
|
||
|
||
Cron (Saturday 01:00): `0 1 * * 6 /usr/local/sbin/unas-sync.sh`
|
||
|
||
### Phase 3 config-to-git scripts
|
||
|
||
Same pattern on each host. Replace `GITEA_TOKEN` with the value from `/opt/stacks/ai/.env` or a dedicated scoped Gitea token.
|
||
|
||
**CT 108 — Zoraxy** (`/usr/local/sbin/zoraxy-conf-backup.sh`):
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -e
|
||
REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git"
|
||
WORK="/opt/zoraxy/conf"
|
||
git -C "$WORK" init -b main -q 2>/dev/null || true
|
||
git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \
|
||
|| git -C "$WORK" remote add origin "$REPO_URL"
|
||
git -C "$WORK" add -A
|
||
git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true
|
||
git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true
|
||
```
|
||
|
||
**CT 102 — AdGuard** — same template, copy `/opt/AdGuardHome/AdGuardHome.yaml` into `/tmp/adguard-conf-work` checkout first.
|
||
|
||
**PVE host** — same template, `rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/` then commit.
|
||
|
||
**VM 100 — HA** — native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) pushing `/config/` (exclude `secrets.yaml`, `.storage/`) to `fkrebs/ha-config`. HA daily tars already covered by `services-backup-plan` via `/mnt/pve/unas/backup/home-assistant/`.
|
||
|
||
Cron on each host: `0 3 * * * /usr/local/sbin/<host>-conf-backup.sh`
|
||
|
||
### Why Wasabi was rejected
|
||
|
||
Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every `restic forget --prune` generates surprise costs. Well-documented Restic-on-Wasabi trap.
|
||
|
||
### Why lobe-postgres stays on CT 104
|
||
|
||
Migration `0093_add_bm25_indexes_with_icu.sql` creates `USING bm25` indexes on 7 tables (`agents`, `topics`, `files`, `knowledge_bases`, `user_memories`, `chat_groups`, `user_memories_contexts`). `bm25` is paradedb-only (`pg_search` extension); stock postgres 17 has no such index access method and the migration fails. Container stays `paradedb/paradedb:latest-pg17`. WAL-G archives to `lobe-pg-backup`; Backrest secondary copies the data dir.
|
||
|
||
### UNAS data inventory (basis for Phase 1a sizing)
|
||
|
||
| Path | Size | Notes |
|
||
|---|---|---|
|
||
| `media/documents/` | 9.5 G | Personal documents |
|
||
| `media/video-projects/` | 425 G | Creative work, irreplaceable |
|
||
| `media/musical-sheets/` | 23 G | |
|
||
| `media/audiobooks/` | 23 G | |
|
||
| `media/ebooks/` | 6.8 G | |
|
||
| `media/3d-prints/` | 65 M | |
|
||
| `media/Recipes/` | 113 M | |
|
||
| `_sortMe/` | 335 G | `images/` 171 G, `work Flo/` 147 G, `Anne/` 17 G |
|
||
| `code/` | 6.7 M | |
|
||
| Excluded `media/movies/` | 118 G | re-streamable |
|
||
| Excluded `media/emulation/` | 57 G | re-downloadable |
|
||
| Excluded `media/Torrents/` | 19 G | temporary |
|
||
| Excluded `media/music/`, `media/podcasts/` | — | re-streamable |
|