diff --git a/services/databases.md b/services/databases.md index 6bc8ab9..56d55d8 100644 --- a/services/databases.md +++ b/services/databases.md @@ -95,3 +95,19 @@ After all 7 DBs are migrated and verified: pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres. Credentials in `/opt/stacks/db/.env` (admin@nucli.de). + +--- + +## WAL-G monitoring + +WAL-G runs as a root crontab on CT 113 (`0 2 * * * docker exec -u postgres postgres wal-g backup-push ...`) and logs to `/var/log/walg-backup.log`. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this. + +**Textfile collector** (`/usr/local/bin/walg-metrics.sh`) runs every 10 minutes via `walg-metrics.timer` and writes `/var/lib/prometheus/node-exporter/walg.prom`. `prometheus-node-exporter` (native systemd, port 9100) picks up the file via `--collector.textfile.directory=/var/lib/prometheus/node-exporter`. + +Metrics emitted: +- `walg_last_success_timestamp_seconds{db="postgres"}` — unix timestamp of last "Wrote backup" in log +- `walg_archive_status{db="postgres"}` — 1 if backup within 25h, 0 if older or log missing + +**Prometheus** (CT 109) scrapes CT 113 as job `node-ct113`. Alert rules at `/opt/stacks/monitoring/prometheus/rules/walg.yml`: +- `WalgArchiveStale` (warning): backup age > 2h, for 5m +- `WalgArchiveFailed` (critical): archive_status == 0, for 5m