fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

incident: CT 104 root full → Immich down + follow-ups

Browse Source 
CT 104 root FS hit 200/200 GB. Postgres + Valkey couldn't write,
Immich down. Recovered with docker prune (89 GB freed) + daemon
restart + compose recreate. Added filesystem alerts on CT 109
(Prometheus rules + Grafana alert routing to existing Gotify contact
point). Top remaining disk consumers documented: mcp-crawl4ai
writable layer 40.7 GB, Garage 82 GB, arr-stack 27 GB.
This commit is contained in:
Florian Krebs
2026-07-15 09:08:38 +02:00
parent 449e549a39
commit ac55f25201
1 changed files with 20 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+20
View File
@@ -2,6 +2,26 @@
All notable infrastructure / service / doc changes. Newest first.
## 2026-07-15
### Immich outage — CT 104 root filesystem full (200 G / 200 G, 0 avail)
Cascading failure: Immich 502 from Traefik. Root cause CT 104 root FS at 100% — Postgres crash-looping ("Disk quota exceeded" on pg_stat_tmp), Valkey blocking writes ("MISCONF stop-writes-on-bgsave-error"), immich_server not serving on :12000.
- **Freed 89 GB** via `docker system prune -af --volumes` on CT 104. Post-prune: 183 G / 200 G (92%).
- Docker network state got wedged (orphan endpoint after prune blocked recreate); resolved with `systemctl restart docker` on CT 104 → all 79 containers auto-restored via `restart: unless-stopped`.
- Immich stack recreated via `docker compose up -d --force-recreate`; postgres data preserved on bind mount `/opt/stacks/immich/postgres/`. All 4 containers healthy; external route returns HTTP/2 200.
**Top disk consumers identified (post-cleanup):**
- `mcp-crawl4ai` container writable layer: **40.7 GB** — crawler cache not persisted to a volume. Needs remediation (bind mount or periodic reset).
- `/opt/stacks/shared-db/garage`: 82 GB (Garage S3 data on root FS).
- `/opt/stacks/arr-stack/media`: 27 GB (dormant per CLAUDE.md — safe to delete).
- Docker daemon logs are already rotated (10 MB × 3 per container in `/etc/docker/daemon.json`).
**Alerting added on CT 109:**
- Prometheus rules `filesystem.yml` (Warning >85% / Critical >95% / Full >99%) at `/opt/stacks/monitoring/prometheus/rules/`.
- Grafana-managed alert rule `Filesystem >85%` (uid `cfs5tiefwl3b4a`, folder `Alerts`), routes to existing Gotify contact point via root notification policy. First delivery expected in ~10 min while CT 104 sits at 92%.
## 2026-07-10
### Karakeep + Memos SUNSET (full wipe)