docs: document WAL-G monitoring setup on CT113/CT109

Add WAL-G monitoring section to databases.md covering the textfile
collector script, walg-metrics.timer, node_exporter config, and
Prometheus alert rules added to prevent silent backup stalls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-23 14:50:39 +02:00
parent f8d8ee3ac0
commit 10e138b677
+16
View File
@@ -95,3 +95,19 @@ After all 7 DBs are migrated and verified:
pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres.
Credentials in `/opt/stacks/db/.env` (admin@nucli.de).
---
## WAL-G monitoring
WAL-G runs as a root crontab on CT 113 (`0 2 * * * docker exec -u postgres postgres wal-g backup-push ...`) and logs to `/var/log/walg-backup.log`. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.
**Textfile collector** (`/usr/local/bin/walg-metrics.sh`) runs every 10 minutes via `walg-metrics.timer` and writes `/var/lib/prometheus/node-exporter/walg.prom`. `prometheus-node-exporter` (native systemd, port 9100) picks up the file via `--collector.textfile.directory=/var/lib/prometheus/node-exporter`.
Metrics emitted:
- `walg_last_success_timestamp_seconds{db="postgres"}` — unix timestamp of last "Wrote backup" in log
- `walg_archive_status{db="postgres"}` — 1 if backup within 25h, 0 if older or log missing
**Prometheus** (CT 109) scrapes CT 113 as job `node-ct113`. Alert rules at `/opt/stacks/monitoring/prometheus/rules/walg.yml`:
- `WalgArchiveStale` (warning): backup age > 2h, for 5m
- `WalgArchiveFailed` (critical): archive_status == 0, for 5m