docs/services/databases.md

Databases

All application databases live on CT 113 (192.168.1.6:5432) after Phase 2 migration. The LXC runs postgres:17 in Docker at /opt/stacks/db/, tracked in Gitea fkrebs/stacks-db. WAL-G archives to Garage S3 bucket ct113-pg-backup on CT 104 (http://192.168.1.40:10004).

---

Database inventory

DB	Owner user	Size (pre-migration)	Service	Stack location
vaultwarden	vaultwarden	11 MB	Vaultwarden	CT 104 /opt/stacks/vaultwarden/
paperless	paperless	20 MB	Paperless-ngx	CT 104 /opt/stacks/apps/paperless-ngx/
litellm	litellm	212 MB	LiteLLM	CT 104 /opt/stacks/ai/
memos	memos	9 MB	Memos	CT 104 /opt/stacks/memos/
n8n	n8n	12 MB	n8n	CT 104 /opt/stacks/n8n/
gitea	gitea	15 MB	Gitea	CT 111 /opt/stacks/gitea/
coder	coder	17 MB	Coder	CT 111 /opt/stacks/coder/

Not on CT 113:

DB	Container	Reason
lobechat	lobe-postgres (paradedb) on CT 104	Requires pg_search extension for USING bm25 indexes
immich	immich_postgres on CT 104	Version-pinned by Immich AIO
nextcloud	Nextcloud AIO on CT 105	AIO manages its own postgres

---

Connection strings (post-migration target)

Service	Connection string
Vaultwarden	postgresql://vaultwarden:<pw>@192.168.1.6:5432/vaultwarden
Paperless	PAPERLESS_DBHOST: 192.168.1.6
LiteLLM	postgresql://litellm:<pw>@192.168.1.6:5432/litellm (in ai/.env and litellm-config/config.yaml)
Memos	postgresql://memos:<pw>@192.168.1.6:5432/memos?sslmode=disable
n8n	DB_POSTGRESDB_HOST=192.168.1.6
Gitea	GITEA__database__HOST: 192.168.1.6:5432
Coder	postgresql://coder:<pw>@192.168.1.6:5432/coder?sslmode=disable

Passwords are in each service's .env file (never committed to git). See init script at /opt/stacks/shared-db/init/01-create-users-dbs.sql on CT 104 for the original credential set.

---

Migration procedure (one DB at a time)

Order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder

# 1. Stop the service
#    docker compose -f <compose> stop <service>

# 2. Dump from source (via PVE host)
#    For CT 104 services:
pct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres <db> \
  > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql

#    For CT 111 services:
pct exec 111 -- docker exec -i <container> pg_dump -U <user> <db> \
  > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql

# 3. Create user + DB on CT 113
pct exec 113 -- docker exec -i postgres psql -U postgres <<EOF
CREATE USER <user> WITH PASSWORD '<pw>';
CREATE DATABASE <db> OWNER <user>;
EOF

# 4. Restore on CT 113
pct exec 113 -- bash -c "docker exec -i postgres psql -U postgres -d <db>" \
  < /mnt/pve/unas/dump/<db>-migration-*.sql

# 5. Update service connection string (shared-postgres → 192.168.1.6)
#    Edit compose or .env

# 6. Start service; verify logs and function

# 7. Verify, then old DB/container can be removed

---

Post-migration cleanup

After all 7 DBs are migrated and verified:

1. Drop stale DBs from shared-postgres: daytona, lobechat (duplicate — real one is in lobe-postgres), paradedb
2. Stop and remove shared-postgres container + named volume shared-pgdata
3. Stop and remove gitea-db and coder-db containers + volumes on CT 111
4. Update Backrest services plan: remove shared-db path, update to CT 113 WAL-G output
5. Enable PVE protection on CT 113 (prevents accidental delete)

---

pgAdmin

pgAdmin on CT 113 at http://192.168.1.6:5050 — pre-registered server: CT 113 postgres. Credentials in /opt/stacks/db/.env (admin@nucli.de).

---

WAL-G monitoring

WAL-G runs as a root crontab on CT 113 (0 2 * * * docker exec -u postgres postgres wal-g backup-push ...) and logs to /var/log/walg-backup.log. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.

Textfile collector (/usr/local/bin/walg-metrics.sh) runs every 10 minutes via walg-metrics.timer and writes /var/lib/prometheus/node-exporter/walg.prom. prometheus-node-exporter (native systemd, port 9100) picks up the file via --collector.textfile.directory=/var/lib/prometheus/node-exporter.

Metrics emitted:

• walg_last_success_timestamp_seconds{db="postgres"} — unix timestamp of last "Wrote backup" in log
• walg_archive_status{db="postgres"} — 1 if backup within 25h, 0 if older or log missing

Prometheus (CT 109) scrapes CT 113 as job node-ct113. Alert rules at /opt/stacks/monitoring/prometheus/rules/walg.yml:

• WalgArchiveStale (warning): backup age > 2h, for 5m
• WalgArchiveFailed (critical): archive_status == 0, for 5m