Skip to content

# Databases

All application databases live on CT 113 (`192.168.1.6:5432`) after Phase 2 migration. The LXC runs `postgres:17` in Docker at `/opt/stacks/db/`, tracked in Gitea `fkrebs/stacks-db`. WAL-G archives to Garage S3 bucket `ct113-pg-backup` on CT 104 (`http://192.168.1.40:10004`).

## Database inventory

DB | Owner user | Size (pre-migration) | Service | Stack location
--- | --- | --- | --- | ---
`vaultwarden` | vaultwarden | 11 MB | Vaultwarden | CT 104 `/opt/stacks/vaultwarden/`
`paperless` | paperless | 20 MB | Paperless-ngx | CT 104 `/opt/stacks/apps/paperless-ngx/`
`litellm` | litellm | 212 MB | LiteLLM | CT 104 `/opt/stacks/ai/`
`memos` | memos | 9 MB | Memos | CT 104 `/opt/stacks/memos/`
`n8n` | n8n | 12 MB | n8n | CT 104 `/opt/stacks/n8n/`
`gitea` | gitea | 15 MB | Gitea | CT 111 `/opt/stacks/gitea/`
`coder` | coder | 17 MB | Coder | CT 111 `/opt/stacks/coder/`

Not on CT 113:

DB | Container | Reason
--- | --- | ---
`lobechat` | `lobe-postgres` (paradedb) on CT 104 | LobeChat decommissioned 2026-05-26 — DB retained pending cleanup; no active service
`immich` | `immich_postgres` on CT 104 | Version-pinned by Immich AIO
`nextcloud` | Nextcloud AIO on CT 105 | AIO manages its own postgres

## Connection strings (post-migration target)

Service | Connection string
--- | ---
Vaultwarden | `postgresql://vaultwarden:<pw>@192.168.1.6:5432/vaultwarden`
Paperless | `PAPERLESS_DBHOST: 192.168.1.6`
LiteLLM | `postgresql://litellm:<pw>@192.168.1.6:5432/litellm` (in `ai/.env` and `litellm-config/config.yaml`)
Memos | `postgresql://memos:<pw>@192.168.1.6:5432/memos?sslmode=disable`
n8n | `DB_POSTGRESDB_HOST=192.168.1.6`
Gitea | `GITEA__database__HOST: 192.168.1.6:5432`
Coder | `postgresql://coder:<pw>@192.168.1.6:5432/coder?sslmode=disable`

Passwords are in each service's `.env` file (never committed to git). See init script at `/opt/stacks/shared-db/init/01-create-users-dbs.sql` on CT 104 for the original credential set.

## Migration procedure (one DB at a time)

Order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder

    # 1. Stop the service
    #    docker compose -f <compose> stop <service>

    # 2. Dump from source (via PVE host)
    #    For CT 104 services:
    pct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres <db> \
      > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql

    #    For CT 111 services:
    pct exec 111 -- docker exec -i <container> pg_dump -U <user> <db> \
      > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql

    # 3. Create user + DB on CT 113
    pct exec 113 -- docker exec -i postgres psql -U postgres <<EOF
    CREATE USER <user> WITH PASSWORD '<pw>';
    CREATE DATABASE <db> OWNER <user>;
    EOF

    # 4. Restore on CT 113
    pct exec 113 -- bash -c "docker exec -i postgres psql -U postgres -d <db>" \
      < /mnt/pve/unas/dump/<db>-migration-*.sql

    # 5. Update service connection string (shared-postgres → 192.168.1.6)
    #    Edit compose or .env

    # 6. Start service; verify logs and function

    # 7. Verify, then old DB/container can be removed

## Post-migration cleanup

After all 7 DBs are migrated and verified:

1. Drop stale DBs from `shared-postgres`: `daytona`, `lobechat` (duplicate — real one is in `lobe-postgres`), `paradedb`
2. Stop and remove `shared-postgres` container + named volume `shared-pgdata`
3. Stop and remove `gitea-db` and `coder-db` containers + volumes on CT 111
4. Update Backrest services plan: remove `shared-db` path, update to CT 113 WAL-G output
5. Enable PVE protection on CT 113 (prevents accidental delete)

## pgAdmin

pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres. Credentials in `/opt/stacks/db/.env` (admin@nucli.de).

## WAL-G monitoring

WAL-G runs as a root crontab on CT 113 (`0 2 * * * docker exec -u postgres postgres wal-g backup-push ...`) and logs to `/var/log/walg-backup.log`. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.

Textfile collector (`/usr/local/bin/walg-metrics.sh`) runs every 10 minutes via `walg-metrics.timer` and writes `/var/lib/prometheus/node-exporter/walg.prom`. `prometheus-node-exporter` (native systemd, port 9100) picks up the file via `--collector.textfile.directory=/var/lib/prometheus/node-exporter`.

Metrics emitted: - `walg_last_success_timestamp_seconds{db="postgres"}` — unix timestamp of last "Wrote backup" in log - `walg_archive_status{db="postgres"}` — 1 if backup within 25h, 0 if older or log missing

Prometheus (CT 109) scrapes CT 113 as job `node-ct113`. Alert rules at `/opt/stacks/monitoring/prometheus/rules/walg.yml`: - `WalgArchiveStale` (warning): backup age > 2h, for 5m - `WalgArchiveFailed` (critical): archive_status == 0, for 5m