Files
docs/stacks/storage.md
T

410 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Storage Layout — NUC 14 Docker Stacks
## Data Location Architecture
All persistent service data is being consolidated to `/mnt/pve/unas/services/{service}/`
on the UNAS NFS share, with media content in `/mnt/pve/unas/media/`.
**PostgreSQL databases stay on local NVMe** — NFS ownership checks (`stat(2)` UID match) prevent
PostgreSQL from running on network filesystems. PostgreSQL is backed up via `pg_dump` to UNAS instead.
| Mount | Source | Content |
|---|---|---|
| `/mnt/pve/unas/services/` | UNAS NFS | SQLite DBs, configs, files, S3 data |
| `/mnt/pve/unas/media/` | UNAS NFS | User media (photos, docs, books, audio) |
| `/mnt/pve/unas/backup/` | UNAS NFS | Service backups |
| `/var/lib/docker/volumes/` | Local NVMe | PostgreSQL data, Redis data |
| `/opt/stacks/` | Local NVMe | Compose files, small configs |
## Migration Status
### ✅ Already on UNAS — `/mnt/pve/unas/services/`
| Service | UNAS Path(s) | Container Mount | Data Type |
|---|---|---|---|
| gluetun | `gluetun/data/` | `/gluetun` | VPN config |
| immich | `immich/{encoded-video,profile,thumbs}/` | `/usr/src/app/upload/{encoded-video,profile,thumbs}` | App derived data |
| karakeep | `karakeep/data/`, `karakeep/meilisearch/` | `/data`, `/meili_data` | Bookmarks DB + Meilisearch index |
| ntfy | `ntfy/{config,cache}/` | `/etc/ntfy`, `/var/lib/ntfy` | Notification config + cache |
| paperless-ai | `paperless-ai/` | `/app/data` | AI processing data |
| pocketid | `pocketid/` | `/app/data` | Auth data (SQLite) |
| stremio | `stremio/` | `/root/.stremio-server` | Streaming server data |
| traccar | `traccar/{data,logs,traccar.xml}` | `/opt/traccar/{data,logs,conf/traccar.xml}` | GPS tracking data |
| vaultwarden | `vaultwarden/` | `/data` | Password vault (SQLite) |
| garage | `shared-db/garage/{data,meta}/` | `/var/lib/garage/{data,meta}` | S3 object storage (Garage) |
### 🆕 Shared Infrastructure — `/opt/stacks/shared-db/`
| Service | Hostname | Type | Data Location |
|---|---|---|---|
| shared-postgres | `shared-postgres` | PostgreSQL 16 (local volume) | `shared-db_shared-pgdata` volume |
| garage | `garage` | S3-compatible object storage (on UNAS) | `/mnt/pve/unas/services/shared-db/garage/` |
### ✅ Migrated (Phase 1) — SQLite to UNAS
| Service | Old Path | New Path | Status |
|---|---|---|---|
| memos | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | ✅ Healthy |
| arcane | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | ✅ Running |
| n8n | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | ✅ Healthy |
### ⏳ Still Local — `/opt/stacks/` or Docker Volumes
| Service | Current Location | Data Type | Estimated Size | UNAS Target |
|---|---|---|---|---|
| arr-stack | `/opt/stacks/arr-stack/` | Configs + media | ~55 GB | `arr-stack/{config,audiobooks,ebooks,podcasts,torrents}` |
| qdrant | `/opt/stacks/qdrant/qdrant_storage` | Vector DB | ~7 MB | `qdrant/` |
| dozzle | `/opt/stacks/dozzle/dozzle_data` | Log viewer state | tiny | *(ephemeral, optional)* |
| homepage | `/opt/stacks/homepage/{config,icons}` | Dashboard config | ~164 KB | *(keep in stack dir)* |
### 🗄️ Databases Requiring Migration
| Service | Engine | Current Path | Target | Migration Method | Status |
|---|---|---|---|---|---|---|
| immich | PostgreSQL | `/opt/stacks/immich/postgres` | → shared-postgres | `pg_dump``pg_restore` | ⏳ Pending |
| litellm | PostgreSQL | `/opt/stacks/ai/postgres_data` | → shared-postgres | `pg_dump``pg_restore` | ⏳ Pending |
| lobehub | PostgreSQL | `/opt/stacks/ai/lobehub/data` | → shared-postgres | `pg_dump``pg_restore` | ⏳ Pending |
| daytona | PostgreSQL | volume `daytona-minimal_db_data` | → shared-postgres | `pg_dump``pg_restore` | ⏳ Pending |
| paperless-ngx | PostgreSQL | volume `paperless-ngx_pgdata` | → shared-postgres | `pg_dump``pg_restore` | ⏳ Pending |
| memos | SQLite | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | `cp` + `chown` | ✅ Done |
| arcane | SQLite | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | `cp` + `chown` | ✅ Done |
| n8n | SQLite | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | `cp` + `chown` | ✅ Done |
| pocketid | SQLite | `pocketid/` | *(already on UNAS)* | — | ✅ Done |
| vaultwarden | SQLite | `vaultwarden/` | *(already on UNAS)* | — | ✅ Done |
| karakeep | SQLite + Meilisearch | `karakeep/{data,meilisearch}` | *(already on UNAS)* | — | ✅ Done |
### 🧠 ML Models & Caches (Regenerable — Can Stay Local)
| Resource | Path | Decision |
|---|---|---|
| Immich ML models | volume `immich_model-cache` | Cache — stay |
| Paperless Redis | volume `paperless-ngx_redisdata` | Ephemeral — stay |
| AI stack Redis | volume `nuc-ai-core_redis_data` | Ephemeral — stay |
| SearXNG search cache | volume (hash) at `/var/lib/docker/volumes/bb5bb...` | Ephemeral — stay |
| Daytona runner Docker | volume `daytona-minimal` runner data | Can stay |
## Media Content (Already on UNAS)
| Mount | Content |
|---|---|
| `/mnt/pve/unas/media/images/` | Immich photo library |
| `/mnt/pve/unas/media/documents/public/paperless-ngx/` | Paperless documents (consume, export, library) |
| `/mnt/pve/unas/media/audiobooks/` | Audiobookshelf media |
| `/mnt/pve/unas/media/ebooks/` | Audiobookshelf ebooks |
| `/mnt/pve/unas/media/podcasts/` | Audiobookshelf podcasts |
| `/mnt/pve/unas/backup/immich/` | Immich DB backups |
## Prospective UNAS Layout
```
/mnt/pve/unas/
├── services/
│ ├── gluetun/
│ │ └── data/ ← VPN config
│ ├── immich/
│ │ ├── db/ ← PostgreSQL (to migrate)
│ │ ├── encoded-video/ ← Already migrated
│ │ ├── profile/ ← Already migrated
│ │ └── thumbs/ ← Already migrated
│ ├── karakeep/
│ │ ├── data/ ← Already migrated
│ │ └── meilisearch/ ← Already migrated
│ ├── ntfy/
│ │ ├── cache/ ← Already migrated
│ │ └── config/ ← Already migrated
│ ├── paperless-ai/ ← Already migrated
│ ├── paperless-ngx/
│ │ └── pgdata/ ← PostgreSQL (to migrate)
│ ├── pocketid/ ← Already migrated
│ ├── stremio/ ← Already migrated
│ ├── traccar/
│ │ ├── data/ ← Already migrated
│ │ ├── logs/ ← Already migrated
│ │ └── traccar.xml ← Already migrated
│ ├── vaultwarden/ ← Already migrated
│ ├── memos/ ← SQLite ✅ Done
│ ├── qdrant/ ← Vector DB (to migrate)
│ ├── arcane/ ← SQLite ✅ Done
│ ├── n8n/ ← SQLite ✅ Done
│ ├── shared-db/
│ │ ├── postgres/ ← ⚠️ Local volume, NOT on UNAS
│ │ └── garage/
│ │ ├── data/ ← S3 data (on UNAS)
│ │ └── meta/ ← S3 metadata (on UNAS)
│ ├── arr-stack/
│ │ ├── config/ ← Bind mounts (to migrate)
│ │ ├── audiobooks/ ← Media (to migrate)
│ │ ├── ebooks/ ← Media (to migrate)
│ │ ├── podcasts/ ← Media (to migrate)
│ │ └── torrents/ ← Downloads (to migrate)
├── media/
│ ├── images/ ← Immich photo library
│ └── documents/
│ └── public/
│ └── paperless-ngx/
│ ├── consume/
│ ├── export/
│ └── library/
├── backups/
│ └── immich/ ← Immich DB backups
└── (potential future: torrents/, cache/)
```
## Garage S3 — `/opt/stacks/shared-db/`
S3-compatible object storage using [Garage](https://garagehq.deuxfleurs.fr/) v1.0.0.
Replaces RustFS from the LobeHub AI stack. Single-node, replication_factor=1.
| Detail | Value |
|---|---|
| Container | `garage` (dxflrs/garage:v1.0.0) |
| Data | `/mnt/pve/unas/services/shared-db/garage/{data,meta}` |
| S3 API | `garage:3900` (internal, shared_backend) |
| Admin API | `127.0.0.1:3901` (localhost only) |
| Regional | No auth (internal network) |
| Bucket | `lobe-files` |
| Key ID | `GK55210bd6b56d92891db26fac` |
| Secret Key | `d0af9e4eb69d7773ffa17950feee27925dbc70eb3efd4ee23a3e1324672c869a` |
## Shared PostgreSQL — `/opt/stacks/shared-db/`
Centralized PostgreSQL 16 instance for all non-immich services.
| Detail | Value |
|---|---|
| Container | `shared-postgres` (postgres:16) |
| Data | `shared-db_shared-pgdata` (local Docker volume, NOT NFS) |
| Networks | `shared_backend` |
| Auth | `postgres` superuser with password from `.env` |
### Databases & Users (pre-created on init)
| Database | Owner | Password | Used By |
|---|---|---|---|
| `paperless` | `paperless` | `paperless` | paperless-ngx |
| `daytona` | `user` | `pass` | daytona |
| `litellm` | `litellm` | `litellm_password_here` | LiteLLM |
| `lobechat` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub |
| `paradedb` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub |
## Key Architectural Decision: PostgreSQL Stays Local
**Problem:** PostgreSQL checks data directory ownership via `stat(2)` and refuses to start
if the process UID doesn't match the file owner UID. NFS with `all_squash` maps all UIDs
to `65534` (nobody), breaking this check. `chown` also fails on NFS.
**Solution:** All PostgreSQL instances use local Docker volumes on NVMe:
```
Local NVMe (/var/lib/docker/volumes/) UNAS NFS (/mnt/pve/unas/)
├── shared-postgres data ├── pg_dump backups ← scheduled
├── immich_postgres data ├── SQLite DBs (memos, arcane, n8n)
├── paperless-ngx pgdata ├── Garage S3 data
├── daytona-minimal db_data ├── Media files (immich, paperless)
├── litellm-db / ai postgres_data ├── Configs / service data
├── lobehub postgres data
└── all Redis / cache volumes
```
**Backup strategy:** `pg_dump` each database to UNAS on a schedule (e.g. daily cron).
## Migration Priority
| Priority | Service | Effort | Risk | Status | Reason |
|---|---|---|---|---|---|
| P0 | memos | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
| P0 | n8n | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
| P0 | arcane | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
| P0 | qdrant | Minimal (file copy) | Low | ✅ Done | Fresh start on UNAS |
| P1 | daytona → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running |
| P1 | litellm → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running |
| P1 | lobehub → shared-postgres | Medium (pg_dump) | Low | ⛔ Skipped | Uses ParadeDB (pg17+extensions), incompatible with shared-postgres (pg16) |
| P1 | paperless-ngx → shared-postgres | Medium (pg_dump) | Medium | ✅ Done | pg_dump → restore → running |
| P1 | immich → shared-postgres | Medium (pg_dump) | Medium | ⏳ | Custom pgvector image |
| P2 | arr-stack | Large (55 GB copy) | Low | ⏳ | Simple copy, time-consuming |
## Mail Archive (Potential Savings)
Unattached anonymous volumes from old/stopped containers:
| Volume | Size | Likely From | Action |
|---|---|---|---|
| `0b6d07ff...` | ? | old daytona runner | prune |
| (100+ hex-named volumes) | ? | ephemeral build cache | prune |
Recommend: `docker system df` then `docker volume prune` to clean.
---
## Database Consolidation Plan
### Current: 10 Database Instances
| # | Container | Type | Image | Used By | Networks |
|---|---|---|---|---|---|
| 1 | `immich_postgres` | PostgreSQL | `immich-app/postgres:14-vectorchord` (pgvector) | immich | `immich_default` |
| 2 | `paperless-ngx-db-1` | PostgreSQL | `postgres:16` | paperless-ngx | `paperless-ngx_default` |
| 3 | `daytona-minimal-db-1` | PostgreSQL | `postgres:16-alpine` | daytona | `daytona-minimal_daytona-network` |
| 4 | `nuc-ai-core-litellm-db-1` | PostgreSQL | `postgres:16-alpine` | litellm | `ai-internal` |
| 5 | `lobe-postgres` | PostgreSQL | `postgres:16-alpine` | lobehub (lobechat, paradedb) | `ai-internal` |
| 6 | `paperless-ngx-broker-1` | Redis 7 | `redis:7` | paperless broker | `paperless-ngx_default` |
| 7 | `immich_redis` | Valkey 9 | `valkey/valkey:9` | immich cache | `immich_default` |
| 8 | `daytona-minimal-redis-1` | Redis Alpine | `redis:alpine` | daytona | `daytona-minimal_daytona-network` |
| 9 | `lobe-redis` | Redis 7 | `redis:7-alpine` | lobehub | `ai-internal` |
| 10 | `redis-searxng` | Redis 7 | `redis:7-alpine` | searxng | `ai-internal` |
### Target: 6 Database Instances (remove 4)
| # | Container | Type | DBs Hosted | Saved |
|---|---|---|---|---|
| 1 | `immich_postgres` | PostgreSQL immich-app (pgvector) | immich | 🔒 Must stay separate |
| 2 | **`shared-postgres`** 🆕 | **PostgreSQL 16** | **paperless, daytona, litellm, lobechat, paradedb** | **3 containers** |
| 3 | `paperless-ngx-broker-1` | Redis 7 | paperless broker | 🔒 Must stay separate |
| 4 | `immich_redis` | Valkey 9 | immich cache | 🔒 Must stay separate |
| 5 | `daytona-minimal-redis-1` | Redis Alpine | daytona | 🔒 Stay separate |
| 6 | `lobe-redis` | Redis 7 | lobehub | 🔒 Stay separate |
| 7 | `redis-searxng` | Redis 7 | searxng | 🔒 Stay separate |
| 7 | ~~`nuc-ai-core-litellm-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
| 8 | ~~`lobe-postgres`~~ | 🗑️ removed | → shared-postgres | ✅ |
| 9 | ~~`paperless-ngx-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
| 10 | ~~`daytona-minimal-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
**Result: 10 → 6 instances, removing 4 containers, saving ~160MB RAM.**
### Why Redis Stays Separate
| Reason | Detail |
|---|---|
| **Valkey ≠ Redis** | Immich uses Valkey 9 which has diverged from Redis 7 |
| **`SELECT` collisions** | Apps may use specific DB numbers that clash |
| **Password mismatch** | Each app has its own `requirepass` / `REDIS_PASSWORD` |
| **Ephemeral = low value** | Caches/brokers rebuild on restart, savings don't justify risk |
| **`FLUSHALL` risk** | One misconfigured app could wipe another's cache |
### Why PG Consolidation Works
| Reason | Detail |
|---|---|
| **Same major version** | All target DBs use PostgreSQL 16 |
| **Native compatibility** | No conflicting extensions (only immich has pgvector) |
| **Isolation via databases** | Each service gets its own database + user within shared PG |
| **Simple restore** | `pg_dump` + `pg_restore` into shared instance |
| **Single backup** | One `pg_dumpall` covers all 5 databases |
### Implementation: Shared PostgreSQL
Create `/opt/stacks/shared-db/docker-compose.yml`:
```yaml
services:
postgres:
image: postgres:16
container_name: shared-postgres
restart: unless-stopped
volumes:
- /mnt/pve/unas/services/shared-db:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: ${SHARED_DB_PASSWORD}
ports:
- "127.0.0.1:5432:5432"
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
reservations:
cpus: "1.0"
memory: 512M
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 10s
timeout: 5s
retries: 5
networks:
default:
external: true
name: shared_backend
```
### Migration Order (Phase 2)
```
Step 0: Create shared-postgres container
Step 1: Dump ALL 4 databases from their current instances
Step 2: Create users + databases on shared-postgres
Step 3: Restore each dump into shared-postgres
Step 4: One by one, update each app's compose to point to shared-postgres
```
#### Detailed: Paperless-ngx
| Change | Before | After |
|---|---|---|
| DB service | `db` (in paperless compose) | remove entirely |
| `PAPERLESS_DBHOST` | `db` | `shared-postgres` |
| DB credentials | defaults (paperless/paperless) | keep same |
| Network | `paperless-ngx_default` | `shared_backend` |
#### Detailed: Daytona
| Change | Before | After |
|---|---|---|
| DB service | `db` (in daytona compose) | remove entirely |
| `DB_HOST` | `db` | `shared-postgres` |
| `DB_USERNAME` | `user` | keep same |
| `DB_DATABASE` | `daytona` | keep same |
| Network | `daytona-minimal_daytona-network` | `shared_backend` |
#### Detailed: LiteLLM
| Change | Before | After |
|---|---|---|
| DB service | `litellm-db` (in ai compose) | remove entirely |
| `DATABASE_URL` | `postgresql://litellm:...@litellm-db:5432/litellm` | `postgresql://litellm:...@shared-postgres:5432/litellm` |
| Network | `ai-internal` | `shared_backend` |
#### Detailed: LobeHub
| Change | Before | After |
|---|---|---|
| DB service | `postgresql` (in lobehub.yml) | remove entirely |
| `DATABASE_URL` | `postgresql://postgres:...@postgresql:5432/lobechat` | `postgresql://postgres:...@shared-postgres:5432/lobechat` |
| Lobe also needs `paradedb` DB | created manually on initial setup | created via SQL script |
| Network | `ai-internal` | `shared_backend` |
### Total Impact
| Metric | Before | After |
|---|---|---|
| PostgreSQL instances | 5 | 2 |
| Redis instances | 5 | 5 (unchanged) |
| Total container count | 10 DBs | 7 DBs (3) |
| RAM saved (estimated) | — | ~160 MB |
| Complexity | Fragmented | Centralized |
| Backup burden | 5 pg_dumps | 2 pg_dumps |
### Timing Estimates
| Service | Downtime | Cumulative |
|---|---|---|
| Setup shared-postgres + dump all | 0 (live dump) | 0 |
| daytona → shared | ~2 min | 2 min |
| litellm → shared | ~2 min | 4 min |
| lobehub → shared | ~3 min | 7 min |
| paperless-ngx → shared | ~5 min | 12 min |
| immich DB → UNAS bind mount | ~10 min | 22 min |
| **Total** | **~22 min** | |
### Notes
- All compose files are version-controlled in `/opt/stacks/{service}/docker-compose.yml`
- Always test with `docker compose config` before `up -d`
- PostgreSQL stays on local NVMe — backed up via `pg_dump` to UNAS
- Garage S3 replaces RustFS from LobeHub stack (RustFS will be removed when LobeHub is switched)
- LobeHub's `paradedb` extension requires `CREATE EXTENSION IF NOT EXISTS paradedb;` on the shared instance
- All PostgreSQL dbs consolidated into shared-postgres except immich (pgvector requirement)