From 40a02f33ef616f5fd1a0cab494e6479fb3654389 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 15 Jul 2026 11:15:53 +0200 Subject: [PATCH] docs: reconcile storage.md + volumes.md with live 2026-07-15 audit Fixes contradictions found during the CT 104 disk-full incident: - Arcane: was DECOMMISSIONED 2026-05-26 + local, not "migrated to UNAS" (volumes.md and storage.md disagreed). - arr-stack: fully on UNAS incl. SQLite configs on services/arr-stack/; media/downloads on media/ share. Radarr/Sonarr not deployed. Removed stale "plan to migrate" rows pointing at local /opt/stacks/arr-stack. - n8n: local (DB on CT 113), not UNAS. - Garage: metadata is SQLite (not LMDB); keep off NFS; refreshed bucket table with real sizes (82 GB, mostly WAL-G PG backups). - Marked sunsets: karakeep, memos, litellm, daytona. - crawl4ai: correct name mcp-crawl4ai, no mounts, cache-growth TODO. - Added ZFS snapshot note: refquota hides snapshot bloat from container df; pre-tier1-backups snapshot held 97.9 GB. --- infra/storage.md | 42 +++++++++++----- infra/volumes.md | 122 ++++++++++++++++------------------------------- 2 files changed, 70 insertions(+), 94 deletions(-) diff --git a/infra/storage.md b/infra/storage.md index c4e5d2b..d4cd1ad 100644 --- a/infra/storage.md +++ b/infra/storage.md @@ -1,7 +1,7 @@ # Storage layout -> Verified state as of 2026-05-21. Source of truth for which service lives on which storage class. -> Earlier revisions of this doc were stale on multiple items (Garage, n8n, Arcane, Karakeep, Pocket-ID, arr-stack, Nextcloud's storage protocol, missing Coder/Gitea). Reconciled via live audit. +> Verified state as of 2026-07-15 (live audit during the CT 104 disk-full incident). Source of truth for which service lives on which storage class. +> Earlier revisions were stale on Garage, n8n, Arcane, Karakeep/Memos, Pocket-ID, arr-stack, Nextcloud's storage protocol, and missing Coder/Gitea. `volumes.md` was reconciled in the same pass. ## Storage classes @@ -24,8 +24,8 @@ Bulk + media + non-latency-sensitive app data. | Paperless-ngx (documents) | 104 | `media/documents/public/paperless-ngx/{consume,export,library}` | none | | | Paperless-AI | 104 | `services/paperless-ai` → `/app/data` | none | | | Traccar (logs, config) | 104 | `services/traccar/{logs,traccar.xml}` | none | data dir reverted to local `/opt/stacks/traccar/data` | -| Memos | 104 | `services/memos` → `/var/opt/memos` | none | | -| Arr-stack (Audiobookshelf, Prowlarr, RDTClient, Shelfarr) + media | 104 | `services/arr-stack/*` + `media/{audiobooks,ebooks,podcasts,Torrents}` | none | already migrated (older doc claimed "still local") | +| ~~Memos~~ | ~~104~~ | ~~`services/memos`~~ | — | **SUNSET 2026-07-10** — service + data + UNAS dir wiped. See [[CHANGELOG]] | +| Arr-stack (Prowlarr, RDTClient, Shelfarr, Audiobookshelf) | 104 | configs on `services/arr-stack/{prowlarr,rdtclient/config,shelfarr/storage,audiobookshelf}`; library + downloads on `media/{audiobooks,ebooks,podcasts,Torrents}` | none | **fully on UNAS incl. SQLite configs** (verified running 2026-07-15). Radarr/Sonarr not deployed. Stale local `/opt/stacks/arr-stack/media` (27 GB pre-migration copy) deleted 2026-07-15. | | Gluetun (VPN) | 104 | `services/gluetun/data` → `/gluetun` | none | | | Gitea | 111 | `services/gitea` → `/data` | none | new 2026-05-20 | | Coder workspace home dirs | 111 | `services/coder//` → `/home/` | none | new 2026-05-20 | @@ -38,27 +38,36 @@ Tier-1 state and anything that should NOT be NFS-backed. | Service | CT | Local path → in container | Backed up? | Notes | |---|---|---|---|---| | Pocket-ID | **110** | `/opt/stacks/pocketid/data` → `/app/data` | none | CT 110 has no NFS mount at all. Earlier doc said UNAS — incorrect. | -| Garage (S3 meta + data) | 104 | `/opt/stacks/shared-db/garage/{data,meta}` → `/var/lib/garage/*` | none | **moved off NFS** 2026-05-19 after WAL-G outage. Stale copy may still exist on UNAS. | +| Garage (S3 meta + data) | 104 | `/opt/stacks/shared-db/garage/{data,meta}` → `/var/lib/garage/*` | none | **moved off NFS** 2026-05-19 after WAL-G outage. Metadata is **SQLite** (`db_engine = sqlite`) — do NOT put on NFS. Data 82 GB on disk (2026-07-15), almost all WAL-G PG backups — see bucket table below. | | Shared Postgres (multi-stack) | 104 | docker named volume `shared-db_shared-pgdata` | **WAL-G → local Garage** (configured; outage 2026-05-19 prompted move) | Garage itself is single-disk local — no off-host copy. | | n8n | 104 | `/opt/stacks/n8n/data` → `/home/node/.n8n` | none | **back to local** after PG migration attempt failed. Stale 408 MB `database.sqlite` left on UNAS (May 15) — clean up. n8n now uses `shared-postgres` as its actual DB. | -| Arcane | 104 | `/opt/stacks/arcane/data` → `/app/data` | none | reverted from UNAS to local 2026-05-19 (undocumented before now) | -| Karakeep + Meilisearch | 104 | `/opt/stacks/karakeep/localdata/{data,meili}` | none | reverted from UNAS to local | +| ~~Arcane~~ | ~~104~~ | ~~`/opt/stacks/arcane/data`~~ | — | **DECOMMISSIONED 2026-05-26** (replaced by Portainer). Compose renamed `.DECOMMISSIONED-2026-05-26`; data was local, not UNAS (volumes.md previously said "migrated" — wrong). | +| ~~Karakeep + Meilisearch~~ | ~~104~~ | ~~`/opt/stacks/karakeep/localdata/{data,meili}`~~ | — | **SUNSET 2026-07-10** — service + local + UNAS data wiped. See [[CHANGELOG]] | | Immich Postgres (pgvector) | 104 | `/opt/stacks/immich/postgres` → `/var/lib/postgresql/data` | Immich pg_dump → UNAS | tier-1; PG demands local disk | -| Homepage | 104 | `/opt/stacks/homepage/{config,icons}` | none | | +| Homepage | 104 | `/opt/stacks/homepage/{config,icons}` | none | decommissioned — replaced by Homarr on CT 109 | | Dozzle | 104 | `/opt/stacks/dozzle/dozzle_data` | none | | -| LiteLLM config | 104 | `/opt/stacks/ai/litellm-config` | none | | +| ~~LiteLLM config~~ | ~~104~~ | ~~`/opt/stacks/ai/litellm-config`~~ | — | **SUNSET 2026-05-26** — replaced by Bifrost | | SearXNG config | 104 | `/opt/stacks/ai/searxng` | none | | | Flaresolverr | 104 | `/var/lib/flaresolver` | none | | | AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup | | Vaultwarden (attachments + key material) | 104 | `/opt/stacks/vaultwarden/data` → `/data` | none | **moved off NFS 2026-05-22**; DB is on CT 113 postgres; stale `db.sqlite3` deleted | | Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | app data on NFS — see below | -### Garage S3 buckets (on local zfs, CT 104) +### Garage S3 buckets (on local zfs, CT 104) — verified 2026-07-15 -| Bucket | Access key | Use | Public URL | +82 GB on disk, dominated by WAL-G Postgres backups. Trim via WAL-G retention on the source hosts, not by moving Garage to NFS. + +| Bucket | Logical size | Objects | Use | |---|---|---|---| -| `lobe-files` | GK55210… (from `.env`) | LobeHub file uploads + WAL-G PG backups | internal only | -| `chat-artifacts` | GK50bfc… | AI chat output artefacts (images, reports, SVG) | `https://chat-artifacts.s3.nuclide.systems/` | +| `ct113-pg-backup` | 66.7 GiB | 1556 | WAL-G for CT 113 shared Postgres fleet | +| `immich-pg-backup` | 36.3 GiB | 4388 | WAL-G / WAL archive for Immich Postgres (+60 MiB unfinished multipart to clean) | +| `ct109-portainer-backup` | 1.9 GiB | 49 | Portainer backups | +| `shared-pg-backup` | 638 MiB | 4986 | shared-postgres WAL-G | +| `chat-artifacts` | — | — | AI chat output artefacts — `https://chat-artifacts.s3.nuclide.systems/` | +| `owui-files` | 0 B | 0 | Open WebUI file storage | +| ~~`lobe-pg-backup`~~ | 151 MiB | 511 | **stale** — LobeChat decommissioned 2026-05-26; safe to delete | +| ~~`lobe-files`~~ | 0 B | 0 | **stale** — LobeChat | +| ~~`memos`~~ | 3.5 MiB | 4 | **orphan** — Memos sunset 2026-07-10; bucket delete blocked by quorum decode error, key revoked, needs Garage repair pass | ### Docker named volumes @@ -93,6 +102,13 @@ This is a known gap. Plans: 1. Extend Backrest plans to snapshot tier-1 paths (Pocket-ID sqlite, Vaultwarden data, Gitea repos, Coder workspace homes) to jottacloud nightly. 2. Once the second NVMe lands (see `infra/proxmox-state.md` §6), mirror `rpool` so a single disk death doesn't take everything. +## ZFS snapshots (CT 104) + +`rpool/data/subvol-104-disk-0` uses `refquota=200G` (not `quota`) — so **snapshots do not count against the container's 200 G limit, but they DO consume pool space**. A stale snapshot can silently exhaust the pool while the container's own `df` looks fine. + +- Watch: `zfs list -t snapshot -o name,used rpool/data/subvol-104-disk-0` +- As of the 2026-07-15 incident, `@pre-tier1-backups-20260521-003708` held **97.9 GB** (pinning everything deleted since 2026-05-21, incl. the Karakeep/Memos sunset and docker prunes). Pool free was down to 57 GB. Destroying stale snapshots is the fastest pool-level reclaim. + ## Drift cleanup TODO - [x] ~~Remove stale `services/n8n/database.sqlite` from UNAS~~ — gone (verified 2026-05-22) diff --git a/infra/volumes.md b/infra/volumes.md index f4b93ac..4a3799f 100644 --- a/infra/volumes.md +++ b/infra/volumes.md @@ -1,7 +1,7 @@ # Volume Mounts — NUC 14 Docker Stacks > Every bind mount and named volume across all running containers. -> Last updated: May 16, 2026 +> Last updated: 2026-07-15 (live audit during CT 104 disk-full incident; reconciled with `storage.md`). ## Legend @@ -16,14 +16,9 @@ ## Infrastructure -### Arcane — `arcane` +### ~~Arcane~~ — DECOMMISSIONED 2026-05-26 -| Type | Source | Container | UNAS | -|---|---|---|---| -| bind | `/var/run/docker.sock` | `/var/run/docker.sock` | — | -| bind | `/opt/stacks` | `/app/data/projects` | — | -| **bind** | **`/mnt/pve/unas/services/arcane`** | **`/app/data`** | ✅ **Migrated** | -| ~~volume~~ | ~~`arcane_arcane-data`~~ | ~~`/app/data`~~ | 🗑️ removed | +> Replaced by Portainer. Compose renamed `/opt/stacks/arcane/docker-compose.yml.DECOMMISSIONED-2026-05-26`; container not running. Its data was **local** (`/opt/stacks/arcane/data`), never UNAS — the earlier "✅ Migrated to `services/arcane`" row was wrong. ### Dozzle — `dozzle` @@ -92,38 +87,40 @@ ## Media — Downloads / Arr Stack -All behind gluetun VPN. +> **Fully on UNAS** — verified running 2026-07-15. SQLite configs live on `services/arr-stack/` and work fine over NFSv3 (low write rate). Library + downloads on the `media/` share so imports hardlink within one export. Radarr/Sonarr are **not** deployed. All behind gluetun VPN. The old local `/opt/stacks/arr-stack/media` (27 GB) was a pre-migration copy — deleted 2026-07-15. ### RDTClient — `rdtclient` | Type | Source | Container | UNAS | |---|---|---|---| -| bind | `/opt/stacks/arr-stack/rdtclient/config` | `/data/db` | ⏳ Plan: `arr-stack/rdtclient` | -| bind | `/opt/stacks/arr-stack/media/Torrents` | `/data/downloads` | ⏳ Plan: `arr-stack/torrents` | +| bind | `/mnt/pve/unas/services/arr-stack/rdtclient/config` | `/data/db` | ✅ **On UNAS** (SQLite) | +| bind | `/mnt/pve/unas/media/Torrents` | `/data/downloads` | ✅ **On UNAS** | ### Prowlarr — `prowlarr` | Type | Source | Container | UNAS | |---|---|---|---| -| bind | `/opt/stacks/arr-stack/prowlarr` | `/config` | ⏳ Plan: `arr-stack/prowlarr` | +| bind | `/mnt/pve/unas/services/arr-stack/prowlarr` | `/config` | ✅ **On UNAS** (SQLite config) | ### Audiobookshelf — `audiobookshelf` +> In compose; config on `services/arr-stack/audiobookshelf`, media on the `media/` share. + | Type | Source | Container | UNAS | |---|---|---|---| -| bind | `/opt/stacks/arr-stack/audiobookshelf` | `/config` | ⏳ Plan: `arr-stack/audiobookshelf` | -| bind | `/opt/stacks/arr-stack/media/audiobooks` | `/audiobooks` | ⏳ Plan: `arr-stack/audiobooks` | -| bind | `/opt/stacks/arr-stack/media/ebooks` | `/ebooks` | ⏳ Plan: `arr-stack/ebooks` | -| bind | `/opt/stacks/arr-stack/media/podcasts` | `/podcasts` | ⏳ Plan: `arr-stack/podcasts` | +| bind | `/mnt/pve/unas/services/arr-stack/audiobookshelf` | `/config` | ✅ **On UNAS** | +| bind | `/mnt/pve/unas/media/audiobooks` | `/audiobooks` | ✅ **On UNAS** | +| bind | `/mnt/pve/unas/media/ebooks` | `/ebooks` | ✅ **On UNAS** | +| bind | `/mnt/pve/unas/media/podcasts` | `/podcasts` | ✅ **On UNAS** | ### ShelfArr — `shelfarr` | Type | Source | Container | UNAS | |---|---|---|---| -| bind | `/opt/stacks/arr-stack/shelfarr/storage` | `/rails/storage` | ⏳ Plan: `arr-stack/shelfarr` | -| bind | `/opt/stacks/arr-stack/media/audiobooks` | `/audiobooks` | (shared) | -| bind | `/opt/stacks/arr-stack/media/ebooks` | `/ebooks` | (shared) | -| bind | `/opt/stacks/arr-stack/media/Torrents` | `/downloads` | (shared) | +| bind | `/mnt/pve/unas/services/arr-stack/shelfarr/storage` | `/rails/storage` | ✅ **On UNAS** (SQLite) | +| bind | `/mnt/pve/unas/media/audiobooks` | `/audiobooks` | ✅ (shared) | +| bind | `/mnt/pve/unas/media/ebooks` | `/ebooks` | ✅ (shared) | +| bind | `/mnt/pve/unas/media/Torrents` | `/downloads` | ✅ (shared) | ### Flaresolverr — `flaresolverr` @@ -135,17 +132,9 @@ All behind gluetun VPN. ## AI Stack -### LiteLLM — `litellm` +### ~~LiteLLM~~ — SUNSET 2026-05-26 -| Type | Source | Container | UNAS | -|---|---|---|---| -| bind | `/opt/stacks/ai/litellm-config` | `/app/config` | (keep in stack dir) | - -### LiteLLM DB — `nuc-ai-core-litellm-db-1` - -| Type | Source | Container | UNAS | -|---|---|---|---| -| **bind** | **`/opt/stacks/ai/postgres_data`** | **`/var/lib/postgresql/data`** | ⏳ Plan: `ai/litellm-db` | +> Replaced by Bifrost (`ai.nuclide.systems`). Service block commented out in `/opt/stacks/ai/docker-compose.yml`; `litellm-config` retained as history only. LiteLLM DB container removed. ### ~~LobeHub~~ — DECOMMISSIONED 2026-05-26 @@ -186,11 +175,9 @@ LobeChat compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Volumes below |---|---|---|---| | bind | `/opt/stacks/ai/saia-image-proxy` | `/app` | (keep in stack dir) | -### Crawl4AI — `crawl4ai-mcp` +### Crawl4AI — `mcp-crawl4ai` -| Type | Source | Container | UNAS | -|---|---|---|---| -| tmpfs | `/dev/shm` | `/dev/shm` | — | +> `docker run` MCP client (Bifrost) on `ai-internal`, **no persistent mounts**. Cache lives in the container writable layer (`/home/appuser/.crawl4ai`, SQLite) and grew to ~40 GB before the 2026-07-15 reset. Recreated with `--shm-size 1g`. **TODO:** bind-mount `.crawl4ai` to a capped local dir or purge periodically — it refills the disk otherwise. --- @@ -227,24 +214,9 @@ LobeChat compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Volumes below ## Productivity & Bookmarks -### Memos — `memos` +### ~~Memos~~ + ~~Karakeep~~ — SUNSET 2026-07-10 -| Type | Source | Container | UNAS | -|---|---|---|---| -| **bind** | **`/mnt/pve/unas/services/memos`** | **`/var/opt/memos`** | ✅ **Migrated** | -| ~~bind~~ | ~~`/opt/stacks/memos/data`~~ | ~~`/var/opt/memos`~~ | 🗑️ replaced | - -### Karakeep — `karakeep` - -| Type | Source | Container | UNAS | -|---|---|---|---| -| **bind** | **`/mnt/pve/unas/services/karakeep/data`** | **`/data`** | ✅ **Already on UNAS** | - -### Karakeep Meilisearch — `karakeep_meilisearch` - -| Type | Source | Container | UNAS | -|---|---|---|---| -| **bind** | **`/mnt/pve/unas/services/karakeep/meilisearch`** | **`/meili_data`** | ✅ **Already on UNAS** | +> Both fully wiped (service, containers, local + UNAS data, Postgres DB, Traefik routes, OIDC clients). See [[CHANGELOG]] and `sunset-karakeep-memos`. Do not restart. One residual: orphan Garage `memos` bucket needs a repair pass. --- @@ -252,32 +224,20 @@ LobeChat compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Volumes below ### n8n — `n8n` -| Type | Source | Container | UNAS | +> Data is **local**, not UNAS — reverted after a failed PG-migration attempt. The actual DB is `shared-postgres` (CT 113); `.n8n` holds config/creds only. A stale UNAS copy at `services/n8n` may linger. + +| Type | Source | Container | Notes | |---|---|---|---| -| **bind** | **`/mnt/pve/unas/services/n8n`** | **`/home/node/.n8n`** | ✅ **Migrated** | +| bind | `/opt/stacks/n8n/data` | `/home/node/.n8n` | 🗄️ Local NVMe | | bind | `/opt/stacks/n8n/hooks.js` | `/home/node/hooks.js` | (keep in stack dir) | -| ~~volume~~ | ~~`n8n_n8n_storage`~~ | ~~`/home/node/.n8n`~~ | 🗑️ removed | --- ## DevOps -### Daytona API — `daytona-minimal-api-1` +### ~~Daytona~~ — DECOMMISSIONED 2026-05-23 -No DB-specific mounts needed (connects via env vars). - -### Daytona DB — `daytona-minimal-db-1` - -| Type | Source | Container | UNAS | -|---|---|---|---| -| volume | `daytona-minimal_db_data` | `/var/lib/postgresql/data` | ⏳ Plan: `daytona/db` | - -### Daytona Runner — `daytona-minimal-runner-1` - -| Type | Source | Container | UNAS | -|---|---|---|---| -| volume | `9416da14...` (anonymous) | `/var/lib/docker` | (runner state, stay) | -| bind | `/var/run/docker.sock` | `/var/run/docker.sock` | — | +> Never fully deployed; compose renamed `.DECOMMISSIONED-2026-05-23`. Orphaned `daytona-minimal_db_data` volume was cleaned up (verified 2026-05-22). --- @@ -313,12 +273,12 @@ No DB-specific mounts needed (connects via env vars). ### Garage S3 — `garage` -> Moved off NFS to local zfs on 2026-05-19 after a WAL-G outage. Stale copy at `services/shared-db/garage/` on UNAS may still exist — clean up. +> Moved off NFS to local zfs on 2026-05-19 after a WAL-G outage. **Keep local** — metadata is SQLite (corrupts over NFS) and it now stores WAL-G backups for the whole PG fleet. See `storage.md` for bucket breakdown. | Type | Source | Container | Notes | |---|---|---|---| -| bind | `/opt/stacks/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data — local NVMe | -| bind | `/opt/stacks/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (LMDB) — local NVMe | +| bind | `/opt/stacks/shared-db/garage/data` | `/var/lib/garage/data` | S3 object data — local NVMe (82 GB, mostly WAL-G) | +| bind | `/opt/stacks/shared-db/garage/meta` | `/var/lib/garage/meta` | S3 metadata (**SQLite**, `db_engine = sqlite`) — local NVMe | --- @@ -338,16 +298,16 @@ No DB-specific mounts needed (connects via env vars). --- -## Summary: Migration Status +## Summary: Migration Status (reconciled 2026-07-15) | Status | Count | Services | |---|---|---| -| ✅ Already on UNAS | 9 | gluetun, immich*(4), karakeep*(2), ntfy*(2), paperless-ai, stremio, traccar*(3), vaultwarden, paperless-docs*(3) | -| ✅ Migrated (Phase 1) | 3 | memos, arcane, n8n | -| 🔷 Shared infrastructure (local) | 2 | shared-postgres (local volume), garage (local NVMe — moved off UNAS 2026-05-19) | +| ✅ On UNAS | 8 | gluetun, immich*(4 media dirs), paperless-ai, stremio, traccar*(3), vaultwarden, paperless-docs*(3), coder | +| ✅ On UNAS incl. SQLite configs | 1 | arr-stack (prowlarr, rdtclient, shelfarr, audiobookshelf + media/downloads) | +| 🔷 Shared infrastructure (local) | 2 | shared-postgres (local volume), garage (local NVMe, SQLite meta — keep local) | +| 🗄️ Reverted to local | 2 | n8n (DB on CT 113), traccar data-dir | | 📋 Own CT, local only | 1 | pocketid (CT 110, no UNAS) | -| ⏳ Phase 2 planned (PG consolidation) | 4 | immich → shared-postgres, paperless → shared-postgres, daytona → shared-postgres, litellm → shared-postgres | -| 🗑️ Decommissioned 2026-05-26 | 3 | lobehub-db, lobe-redis, lobe-rustfs (LobeChat removed) | -| ⏳ Phase 3 planned | 1 | arr-stack*(8 mounts) | -| 📋 Keep local | ~5 | homepage, dozzle, litellm-config, searxng-config, saia-image-proxy | -| 🧠 Cache (stay) | ~5 | immich_model-cache, paperless-ngx redis, lobe-redis, searxng-redis, flaresolverr, daytona-runner | +| ⏳ Phase 2 planned (PG consolidation) | 2 | immich → shared-postgres, paperless → shared-postgres | +| 🗑️ Sunset / decommissioned | many | karakeep + memos (2026-07-10); litellm, mcp-gateway, arcane, lobehub*(3), homepage (2026-05-2x); daytona (2026-05-23) | +| 📋 Keep local | ~4 | dozzle, searxng-config, saia-image-proxy, flaresolverr | +| 🧠 Cache (stay) | ~4 | immich_model-cache, paperless-ngx redis, searxng-redis, mcp-crawl4ai writable layer |