docs: document WAL-G monitoring setup on CT113/CT109
Add WAL-G monitoring section to databases.md covering the textfile collector script, walg-metrics.timer, node_exporter config, and Prometheus alert rules added to prevent silent backup stalls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -95,3 +95,19 @@ After all 7 DBs are migrated and verified:
|
||||
|
||||
pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres.
|
||||
Credentials in `/opt/stacks/db/.env` (admin@nucli.de).
|
||||
|
||||
---
|
||||
|
||||
## WAL-G monitoring
|
||||
|
||||
WAL-G runs as a root crontab on CT 113 (`0 2 * * * docker exec -u postgres postgres wal-g backup-push ...`) and logs to `/var/log/walg-backup.log`. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.
|
||||
|
||||
**Textfile collector** (`/usr/local/bin/walg-metrics.sh`) runs every 10 minutes via `walg-metrics.timer` and writes `/var/lib/prometheus/node-exporter/walg.prom`. `prometheus-node-exporter` (native systemd, port 9100) picks up the file via `--collector.textfile.directory=/var/lib/prometheus/node-exporter`.
|
||||
|
||||
Metrics emitted:
|
||||
- `walg_last_success_timestamp_seconds{db="postgres"}` — unix timestamp of last "Wrote backup" in log
|
||||
- `walg_archive_status{db="postgres"}` — 1 if backup within 25h, 0 if older or log missing
|
||||
|
||||
**Prometheus** (CT 109) scrapes CT 113 as job `node-ct113`. Alert rules at `/opt/stacks/monitoring/prometheus/rules/walg.yml`:
|
||||
- `WalgArchiveStale` (warning): backup age > 2h, for 5m
|
||||
- `WalgArchiveFailed` (critical): archive_status == 0, for 5m
|
||||
|
||||
Reference in New Issue
Block a user