fkrebs
10e138b677
Add WAL-G monitoring section to databases.md covering the textfile collector script, walg-metrics.timer, node_exporter config, and Prometheus alert rules added to prevent silent backup stalls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
114 lines
4.8 KiB
Markdown
114 lines
4.8 KiB
Markdown
# Databases
|
|
|
|
All application databases live on **CT 113** (`192.168.1.6:5432`) after Phase 2 migration. The LXC runs `postgres:17` in Docker at `/opt/stacks/db/`, tracked in Gitea `fkrebs/stacks-db`. WAL-G archives to Garage S3 bucket `ct113-pg-backup` on CT 104 (`http://192.168.1.40:10004`).
|
|
|
|
---
|
|
|
|
## Database inventory
|
|
|
|
| DB | Owner user | Size (pre-migration) | Service | Stack location |
|
|
|----|-----------|---------------------|---------|---------------|
|
|
| `vaultwarden` | vaultwarden | 11 MB | Vaultwarden | CT 104 `/opt/stacks/vaultwarden/` |
|
|
| `paperless` | paperless | 20 MB | Paperless-ngx | CT 104 `/opt/stacks/apps/paperless-ngx/` |
|
|
| `litellm` | litellm | 212 MB | LiteLLM | CT 104 `/opt/stacks/ai/` |
|
|
| `memos` | memos | 9 MB | Memos | CT 104 `/opt/stacks/memos/` |
|
|
| `n8n` | n8n | 12 MB | n8n | CT 104 `/opt/stacks/n8n/` |
|
|
| `gitea` | gitea | 15 MB | Gitea | CT 111 `/opt/stacks/gitea/` |
|
|
| `coder` | coder | 17 MB | Coder | CT 111 `/opt/stacks/coder/` |
|
|
|
|
**Not on CT 113:**
|
|
|
|
| DB | Container | Reason |
|
|
|----|-----------|--------|
|
|
| `lobechat` | `lobe-postgres` (paradedb) on CT 104 | Requires `pg_search` extension for `USING bm25` indexes |
|
|
| `immich` | `immich_postgres` on CT 104 | Version-pinned by Immich AIO |
|
|
| `nextcloud` | Nextcloud AIO on CT 105 | AIO manages its own postgres |
|
|
|
|
---
|
|
|
|
## Connection strings (post-migration target)
|
|
|
|
| Service | Connection string |
|
|
|---------|------------------|
|
|
| Vaultwarden | `postgresql://vaultwarden:<pw>@192.168.1.6:5432/vaultwarden` |
|
|
| Paperless | `PAPERLESS_DBHOST: 192.168.1.6` |
|
|
| LiteLLM | `postgresql://litellm:<pw>@192.168.1.6:5432/litellm` (in `ai/.env` and `litellm-config/config.yaml`) |
|
|
| Memos | `postgresql://memos:<pw>@192.168.1.6:5432/memos?sslmode=disable` |
|
|
| n8n | `DB_POSTGRESDB_HOST=192.168.1.6` |
|
|
| Gitea | `GITEA__database__HOST: 192.168.1.6:5432` |
|
|
| Coder | `postgresql://coder:<pw>@192.168.1.6:5432/coder?sslmode=disable` |
|
|
|
|
Passwords are in each service's `.env` file (never committed to git). See init script at `/opt/stacks/shared-db/init/01-create-users-dbs.sql` on CT 104 for the original credential set.
|
|
|
|
---
|
|
|
|
## Migration procedure (one DB at a time)
|
|
|
|
Order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder
|
|
|
|
```bash
|
|
# 1. Stop the service
|
|
# docker compose -f <compose> stop <service>
|
|
|
|
# 2. Dump from source (via PVE host)
|
|
# For CT 104 services:
|
|
pct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres <db> \
|
|
> /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql
|
|
|
|
# For CT 111 services:
|
|
pct exec 111 -- docker exec -i <container> pg_dump -U <user> <db> \
|
|
> /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql
|
|
|
|
# 3. Create user + DB on CT 113
|
|
pct exec 113 -- docker exec -i postgres psql -U postgres <<EOF
|
|
CREATE USER <user> WITH PASSWORD '<pw>';
|
|
CREATE DATABASE <db> OWNER <user>;
|
|
EOF
|
|
|
|
# 4. Restore on CT 113
|
|
pct exec 113 -- bash -c "docker exec -i postgres psql -U postgres -d <db>" \
|
|
< /mnt/pve/unas/dump/<db>-migration-*.sql
|
|
|
|
# 5. Update service connection string (shared-postgres → 192.168.1.6)
|
|
# Edit compose or .env
|
|
|
|
# 6. Start service; verify logs and function
|
|
|
|
# 7. Verify, then old DB/container can be removed
|
|
```
|
|
|
|
---
|
|
|
|
## Post-migration cleanup
|
|
|
|
After all 7 DBs are migrated and verified:
|
|
|
|
1. Drop stale DBs from `shared-postgres`: `daytona`, `lobechat` (duplicate — real one is in `lobe-postgres`), `paradedb`
|
|
2. Stop and remove `shared-postgres` container + named volume `shared-pgdata`
|
|
3. Stop and remove `gitea-db` and `coder-db` containers + volumes on CT 111
|
|
4. Update Backrest services plan: remove `shared-db` path, update to CT 113 WAL-G output
|
|
5. Enable **PVE protection** on CT 113 (prevents accidental delete)
|
|
|
|
---
|
|
|
|
## pgAdmin
|
|
|
|
pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres.
|
|
Credentials in `/opt/stacks/db/.env` (admin@nucli.de).
|
|
|
|
---
|
|
|
|
## WAL-G monitoring
|
|
|
|
WAL-G runs as a root crontab on CT 113 (`0 2 * * * docker exec -u postgres postgres wal-g backup-push ...`) and logs to `/var/log/walg-backup.log`. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.
|
|
|
|
**Textfile collector** (`/usr/local/bin/walg-metrics.sh`) runs every 10 minutes via `walg-metrics.timer` and writes `/var/lib/prometheus/node-exporter/walg.prom`. `prometheus-node-exporter` (native systemd, port 9100) picks up the file via `--collector.textfile.directory=/var/lib/prometheus/node-exporter`.
|
|
|
|
Metrics emitted:
|
|
- `walg_last_success_timestamp_seconds{db="postgres"}` — unix timestamp of last "Wrote backup" in log
|
|
- `walg_archive_status{db="postgres"}` — 1 if backup within 25h, 0 if older or log missing
|
|
|
|
**Prometheus** (CT 109) scrapes CT 113 as job `node-ct113`. Alert rules at `/opt/stacks/monitoring/prometheus/rules/walg.yml`:
|
|
- `WalgArchiveStale` (warning): backup age > 2h, for 5m
|
|
- `WalgArchiveFailed` (critical): archive_status == 0, for 5m
|