fkrebs/n8n-flows
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

WAL-G backup staleness monitor #6

New Issue
Closed
opened 2026-05-23 07:42:06 +00:00 by fkrebs · 1 comment
fkrebs commented 2026-05-23 07:42:06 +00:00
Owner
Copy Link
Copy Source

Every hour, check latest WAL-G backup timestamps per DB (immich, lobe) in Garage S3.
If last backup >26h old → Gotify priority 8 alert.

Context: silent WAL-G stall went undetected for 13h in 2026-05-22 incident.

Nodes: Cron → Execute Command (mc ls garage S3 paths) → IF age >26h → Gotify.

Every hour, check latest WAL-G backup timestamps per DB (immich, lobe) in Garage S3. If last backup >26h old → Gotify priority 8 alert. Context: silent WAL-G stall went undetected for 13h in 2026-05-22 incident. Nodes: Cron → Execute Command (mc ls garage S3 paths) → IF age >26h → Gotify.
fkrebs commented 2026-05-28 13:18:44 +00:00
Author
Owner
Copy Link
Copy Source

Closed by /usr/local/sbin/walg-staleness-check.sh + walg-staleness.timer on CT 103 (hourly). Queries pg_stat_archiver on each Postgres host; alerts on stale archive OR recent failures. Found Immich wal-g had been broken for 42h (garage hostname unresolvable after Garage container was manually recreated on bridge network not shared_backend) — fixed by reconnecting garage to shared_backend.

Closed by /usr/local/sbin/walg-staleness-check.sh + walg-staleness.timer on CT 103 (hourly). Queries pg_stat_archiver on each Postgres host; alerts on stale archive OR recent failures. Found Immich wal-g had been broken for 42h (garage hostname unresolvable after Garage container was manually recreated on bridge network not shared_backend) — fixed by reconnecting garage to shared_backend.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
fkrebs (Florian Krebs)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fkrebs/n8n-flows#6
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?