diff --git a/CHANGELOG.md b/CHANGELOG.md index b48b529..d087953 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ All notable infrastructure / service / doc changes. Newest first. +## 2026-07-15 + +### Immich outage — CT 104 root filesystem full (200 G / 200 G, 0 avail) + +Cascading failure: Immich 502 from Traefik. Root cause CT 104 root FS at 100% — Postgres crash-looping ("Disk quota exceeded" on pg_stat_tmp), Valkey blocking writes ("MISCONF stop-writes-on-bgsave-error"), immich_server not serving on :12000. + +- **Freed 89 GB** via `docker system prune -af --volumes` on CT 104. Post-prune: 183 G / 200 G (92%). +- Docker network state got wedged (orphan endpoint after prune blocked recreate); resolved with `systemctl restart docker` on CT 104 → all 79 containers auto-restored via `restart: unless-stopped`. +- Immich stack recreated via `docker compose up -d --force-recreate`; postgres data preserved on bind mount `/opt/stacks/immich/postgres/`. All 4 containers healthy; external route returns HTTP/2 200. + +**Top disk consumers identified (post-cleanup):** +- `mcp-crawl4ai` container writable layer: **40.7 GB** — crawler cache not persisted to a volume. Needs remediation (bind mount or periodic reset). +- `/opt/stacks/shared-db/garage`: 82 GB (Garage S3 data on root FS). +- `/opt/stacks/arr-stack/media`: 27 GB (dormant per CLAUDE.md — safe to delete). +- Docker daemon logs are already rotated (10 MB × 3 per container in `/etc/docker/daemon.json`). + +**Alerting added on CT 109:** +- Prometheus rules `filesystem.yml` (Warning >85% / Critical >95% / Full >99%) at `/opt/stacks/monitoring/prometheus/rules/`. +- Grafana-managed alert rule `Filesystem >85%` (uid `cfs5tiefwl3b4a`, folder `Alerts`), routes to existing Gotify contact point via root notification policy. First delivery expected in ~10 min while CT 104 sits at 92%. + ## 2026-07-10 ### Karakeep + Memos SUNSET (full wipe)