docs/infra/storage.md

Storage layout

Verified state as of 2026-07-15 (live audit during the CT 104 disk-full incident). Source of truth for which service lives on which storage class. Earlier revisions were stale on Garage, n8n, Arcane, Karakeep/Memos, Pocket-ID, arr-stack, Nextcloud's storage protocol, and missing Coder/Gitea. volumes.md was reconciled in the same pass.

Storage classes

Class	Substrate	Where	Typical use
Local zfs (rootfs)	NVMe in nuc	/opt/stacks/<stack>/ on each LXC	Tier-1 hot state: databases, SQLite, secret stores, anything latency-sensitive
Local zfs (subvol)	NVMe in nuc	rpool/data/subvol-<NNN>-disk-0 per LXC	Container rootfs, image cache
Docker named volume	NVMe in nuc	/var/lib/docker/volumes/... on each Docker CT	Per-container persistent state managed by Docker
UNAS over NFSv3	192.168.1.31	/mnt/pve/unas on CTs 101, 103, 104, 105, 111	Bulk media, workspace home dirs, document storage

What lives where (verified 2026-05-21)

UNAS NFS — /mnt/pve/unas/services/...

Bulk + media + non-latency-sensitive app data.

Service	CT	Path on UNAS → in container	Backed up?	Notes
Immich (uploads, thumbs, derived)	104	services/immich/{encoded-video,profile,thumbs} + media/images + backup/immich	Immich own pg_dump → media/images/db-dumps; no off-host copy
Paperless-ngx (documents)	104	media/documents/public/paperless-ngx/{consume,export,library}	none
Paperless-AI	104	services/paperless-ai → /app/data	none
Traccar (logs, config)	104	services/traccar/{logs,traccar.xml}	none	data dir reverted to local /opt/stacks/traccar/data
Memos	104	services/memos	—	SUNSET 2026-07-10 — service + data + UNAS dir wiped. See CHANGELOG
Arr-stack (Prowlarr, RDTClient, Shelfarr, Audiobookshelf)	104	configs on services/arr-stack/{prowlarr,rdtclient/config,shelfarr/storage,audiobookshelf}; library + downloads on media/{audiobooks,ebooks,podcasts,Torrents}	none	fully on UNAS incl. SQLite configs (verified running 2026-07-15). Radarr/Sonarr not deployed. Stale local /opt/stacks/arr-stack/media (27 GB pre-migration copy) deleted 2026-07-15.
Gluetun (VPN)	104	services/gluetun/data → /gluetun	none
Gitea	111	services/gitea → /data	none	new 2026-05-20
Coder workspace home dirs	111	services/coder/<user>/<workspace> → /home/<user>	none	new 2026-05-20
Backrest's jottacloud-mirrored data	103	media/data-dir only	rclone → jottacloud (Backrest)	only path with off-host backup

Local zfs — /opt/stacks/...

Tier-1 state and anything that should NOT be NFS-backed.

Service	CT	Local path → in container	Backed up?	Notes
Pocket-ID	110	/opt/stacks/pocketid/data → /app/data	none	CT 110 has no NFS mount at all. Earlier doc said UNAS — incorrect.
Garage (S3 meta + data)	104	/opt/stacks/shared-db/garage/{data,meta} → /var/lib/garage/*	none	moved off NFS 2026-05-19 after WAL-G outage. Metadata is SQLite (db_engine = sqlite) — do NOT put on NFS. Data 82 GB on disk (2026-07-15), almost all WAL-G PG backups — see bucket table below.
Shared Postgres (multi-stack)	104	docker named volume shared-db_shared-pgdata	WAL-G → local Garage (configured; outage 2026-05-19 prompted move)	Garage itself is single-disk local — no off-host copy.
n8n	104	/opt/stacks/n8n/data → /home/node/.n8n	none	back to local after PG migration attempt failed. Stale 408 MB database.sqlite left on UNAS (May 15) — clean up. n8n now uses shared-postgres as its actual DB.
Arcane	104	/opt/stacks/arcane/data	—	DECOMMISSIONED 2026-05-26 (replaced by Portainer). Compose renamed .DECOMMISSIONED-2026-05-26; data was local, not UNAS (volumes.md previously said "migrated" — wrong).
Karakeep + Meilisearch	104	/opt/stacks/karakeep/localdata/{data,meili}	—	SUNSET 2026-07-10 — service + local + UNAS data wiped. See CHANGELOG
Immich Postgres (pgvector)	104	/opt/stacks/immich/postgres → /var/lib/postgresql/data	Immich pg_dump → UNAS	tier-1; PG demands local disk
Homepage	104	/opt/stacks/homepage/{config,icons}	none	decommissioned — replaced by Homarr on CT 109
Dozzle	104	/opt/stacks/dozzle/dozzle_data	none
LiteLLM config	104	/opt/stacks/ai/litellm-config	—	SUNSET 2026-05-26 — replaced by Bifrost
SearXNG config	104	/opt/stacks/ai/searxng	none
Flaresolverr	104	/var/lib/flaresolver	none
AdGuard / Zoraxy / DNS / Shepard / Backrest binaries	102/108/103/101	local zfs only	none	Backrest itself has no self-backup
Vaultwarden (attachments + key material)	104	/opt/stacks/vaultwarden/data → /data	none	moved off NFS 2026-05-22; DB is on CT 113 postgres; stale db.sqlite3 deleted
Nextcloud config + sidecars	105	local zfs (CT rootfs)	none	app data on NFS — see below

Garage S3 buckets (on local zfs, CT 104) — verified 2026-07-15

82 GB on disk, dominated by WAL-G Postgres backups. Trim via WAL-G retention on the source hosts, not by moving Garage to NFS.

Bucket	Logical size	Objects	Use
ct113-pg-backup	66.7 GiB	1556	WAL-G for CT 113 shared Postgres fleet
immich-pg-backup	36.3 GiB	4388	WAL-G / WAL archive for Immich Postgres (+60 MiB unfinished multipart to clean)
ct109-portainer-backup	1.9 GiB	49	Portainer backups
shared-pg-backup	638 MiB	4986	shared-postgres WAL-G
chat-artifacts	—	—	AI chat output artefacts — https://chat-artifacts.s3.nuclide.systems/<key>
owui-files	0 B	0	Open WebUI file storage
lobe-pg-backup	151 MiB	511	stale — LobeChat decommissioned 2026-05-26; safe to delete
lobe-files	0 B	0	stale — LobeChat
memos	3.5 MiB	4	orphan — Memos sunset 2026-07-10; bucket delete blocked by quorum decode error, key revoked, needs Garage repair pass

Docker named volumes

Local on /var/lib/docker/volumes/. Mostly databases and caches.

Volume	Service / CT	Backed up?
shared-db_shared-pgdata	shared-postgres / 104	WAL-G → local Garage
paperless-ngx_pgdata, _redisdata, _data	paperless-ngx / 104	none
lobe-postgres + lobe-redis	lobehub / 104	none
nuc-ai-core_rustfs-data	lobehub stack / 104	none
pgadmin-data	pgadmin / 104	none (regenerable)
immich_model-cache	immich-ml / 104	regenerable
coder-db + gitea-db (docker volume coder-db)	/ 111	none
(orphaned) daytona-minimal_db_data + runner anon vol	/ 104	none — clean up post-decommission

Nextcloud (CT 105) — data on NFS

Nextcloud AIO mounts /mnt/pve/unas/services/nextcloud via NFSv3 (same share as all other CTs). Migrated from CIFS on 2026-05-22 after the CIFS mount caused a crash-loop. The Postgres + Redis sidecars stay on local docker volumes.

Backup reality

There is effectively no off-host backup of service data. The only configured Backrest plan covers /mnt/pve/unas/media/data-dir → jottacloud — i.e., Backrest backs up one specific UNAS path, not the services that write to UNAS.

Coverage:

• Immich pg_dump → UNAS (same filer; survives container loss, not filer loss)
• shared-postgres WAL-G → Garage (same host; survives container loss, not disk loss)
• Everything else: zero

This is a known gap. Plans:

1. Extend Backrest plans to snapshot tier-1 paths (Pocket-ID sqlite, Vaultwarden data, Gitea repos, Coder workspace homes) to jottacloud nightly.
2. Once the second NVMe lands (see infra/proxmox-state.md §6), mirror rpool so a single disk death doesn't take everything.

ZFS snapshots (CT 104)

rpool/data/subvol-104-disk-0 uses refquota=200G (not quota) — so snapshots do not count against the container's 200 G limit, but they DO consume pool space. A stale snapshot can silently exhaust the pool while the container's own df looks fine.

• Watch: zfs list -t snapshot -o name,used rpool/data/subvol-104-disk-0
• As of the 2026-07-15 incident, @pre-tier1-backups-20260521-003708 held 97.9 GB (pinning everything deleted since 2026-05-21, incl. the Karakeep/Memos sunset and docker prunes). Pool free was down to 57 GB. Destroying stale snapshots is the fastest pool-level reclaim.

Drift cleanup TODO

• Remove stale services/n8n/database.sqlite from UNAS — gone (verified 2026-05-22)
• Remove stale services/shared-db/garage/ copy from UNAS — gone (verified 2026-05-22)
• Remove orphaned daytona-minimal_db_data Docker volume on CT 104 — gone (verified 2026-05-22)
• Vaultwarden data/ on UNAS — migrated to local zfs 2026-05-22. Stale NFS copy at services/vaultwarden/ can be cleaned up.
• Add a Backrest plan for Pocket-ID DB — done 2026-05-22. Pre-backup hook runs pocket-id export + copies keys to UNAS staging; services-backup-plan snapshots staging → jottacloud nightly.