ops: Loki/Alloy fleet enabled; Backrest+WAL-G+Gitea Gotify hooks; Immich WAL fix

- Alloy now active on CT 102, 103, 104, 109, 101, 105, 113, nuc (mix of
  Docker container + systemd binary); all hosts shipping journal/docker
  logs to Loki on CT 109. CT 103 alloy newly enabled.
- Backrest plans: actionGotify hooks on SUCCESS (prio 3) + ERROR/WARNING
  (prio 8). Gotify app 'Backrest', token AtIMGOYcZsb5HfO.
- WAL-G staleness monitor on CT 103 (hourly): pg_stat_archiver SSH/exec
  against CT 113 + CT 104 immich_postgres. Found Immich WAL push failing
  for 42h — root cause: garage container on bridge network not
  shared_backend (Garage 2026-05-28 incident aftermath). Reconnected to
  shared_backend; image pinned v0.9.4 in compose to match running.
- Gitea daily digest at 08:00 on PVE host nuc — yesterday's commits.
- Closes fkrebs/n8n-flows #2, #5, #6.
This commit is contained in:
2026-05-28 15:19:40 +02:00
parent c05f38d008
commit b26796e30c
2 changed files with 12 additions and 9 deletions
+11 -9
View File
@@ -181,18 +181,20 @@ restic -r rclone:jottacloud:services restore latest \
--include /mnt/pve/unas/services/vaultwarden
```
### Monitoring (planned)
### Monitoring ✅ DEPLOYED 2026-05-28
In Backrest UI → each repo → Hooks:
Gotify hooks on all 3 plans (services / video-projects / media). Gotify app "Backrest" token `AtIMGOYcZsb5HfO`.
- `CONDITION_BACKUP_ERROR` / `CONDITION_CHECK_ERROR` → POST Gotify priority 8
- `CONDITION_BACKUP_SUCCESS` → POST Gotify priority 3
- `CONDITION_SNAPSHOT_SUCCESS` → Gotify priority 3 (✓ snapshot summary)
- `CONDITION_SNAPSHOT_ERROR` + `CONDITION_SNAPSHOT_WARNING` Gotify priority 8 (✗ FAILED)
```
curl -s -X POST 'http://gotify:80/message?token=TOKEN' \
-H 'Content-Type: application/json' \
-d '{"title":"Backrest: {{.Plan}}","message":"{{.Summary}}","priority":3}'
```
Hooks use Backrest's native `actionGotify` (not `actionCommand`) — no shell required, templating via Go text/template with `{{ .Plan.Id }}`, `{{ .SnapshotStats.FilesNew }}`, `{{ formatSizeBytes .SnapshotStats.DataAdded }}`, `{{ .Duration }}`, `{{ .Error }}`.
**WAL-G staleness monitor** (`/usr/local/sbin/walg-staleness-check.sh` on CT 103, hourly timer): SSH-then-`docker exec psql` to each Postgres host and queries `pg_stat_archiver` (last_archived_time, last_failed_time, failed_count, failed_wal). Alerts if archive >26h old OR a recent failure is more recent than the last success. Gotify token `Ag570SdxcXcpp51`. Hosts: CT 113 `postgres`, CT 104 `immich_postgres`. Closes `fkrebs/n8n-flows#6`.
> **Incident found by this monitor (2026-05-28)**: Immich `wal-g wal-push` had been failing with exit 124 (timeout) for 42 hours. Root cause: `garage` container was recreated on Docker `bridge` network instead of `shared_backend` during the 2026-05-28 Garage incident, making `garage:3900` unresolvable from `immich_postgres`. Fix: `docker network connect shared_backend garage`; compose image pinned to v0.9.4 to match running.
**Gitea daily digest** (`/usr/local/sbin/gitea-digest.sh` + `gitea-digest.timer` on PVE host nuc, 08:00 daily): walks all `fkrebs/*` repos, finds yesterday's commits, posts summary to Gotify priority 3 (token `A.qpozt3ZQGgEMG`). Closes `fkrebs/n8n-flows#5`.
### Applying a new config