baseline: docs as of pre-reorg audit (2026-05-20)

2026-05-20 23:23:31 +02:00
commit 04d54ac9b0
40 changed files with 7074 additions and 0 deletions
@@ -0,0 +1,409 @@
+# Storage Layout — NUC 14 Docker Stacks
+
+## Data Location Architecture
+
+All persistent service data is being consolidated to `/mnt/pve/unas/services/{service}/`
+on the UNAS NFS share, with media content in `/mnt/pve/unas/media/`.
+
+**PostgreSQL databases stay on local NVMe** — NFS ownership checks (`stat(2)` UID match) prevent
+PostgreSQL from running on network filesystems. PostgreSQL is backed up via `pg_dump` to UNAS instead.
+
+| Mount | Source | Content |
+|---|---|---|
+| `/mnt/pve/unas/services/` | UNAS NFS | SQLite DBs, configs, files, S3 data |
+| `/mnt/pve/unas/media/` | UNAS NFS | User media (photos, docs, books, audio) |
+| `/mnt/pve/unas/backup/` | UNAS NFS | Service backups |
+| `/var/lib/docker/volumes/` | Local NVMe | PostgreSQL data, Redis data |
+| `/opt/stacks/` | Local NVMe | Compose files, small configs |
+
+## Migration Status
+
+### ✅ Already on UNAS — `/mnt/pve/unas/services/`
+
+| Service | UNAS Path(s) | Container Mount | Data Type |
+|---|---|---|---|
+| gluetun | `gluetun/data/` | `/gluetun` | VPN config |
+| immich | `immich/{encoded-video,profile,thumbs}/` | `/usr/src/app/upload/{encoded-video,profile,thumbs}` | App derived data |
+| karakeep | `karakeep/data/`, `karakeep/meilisearch/` | `/data`, `/meili_data` | Bookmarks DB + Meilisearch index |
+| ntfy | `ntfy/{config,cache}/` | `/etc/ntfy`, `/var/lib/ntfy` | Notification config + cache |
+| paperless-ai | `paperless-ai/` | `/app/data` | AI processing data |
+| pocketid | `pocketid/` | `/app/data` | Auth data (SQLite) |
+| stremio | `stremio/` | `/root/.stremio-server` | Streaming server data |
+| traccar | `traccar/{data,logs,traccar.xml}` | `/opt/traccar/{data,logs,conf/traccar.xml}` | GPS tracking data |
+| vaultwarden | `vaultwarden/` | `/data` | Password vault (SQLite) |
+| garage | `shared-db/garage/{data,meta}/` | `/var/lib/garage/{data,meta}` | S3 object storage (Garage) |
+
+### 🆕 Shared Infrastructure — `/opt/stacks/shared-db/`
+
+| Service | Hostname | Type | Data Location |
+|---|---|---|---|
+| shared-postgres | `shared-postgres` | PostgreSQL 16 (local volume) | `shared-db_shared-pgdata` volume |
+| garage | `garage` | S3-compatible object storage (on UNAS) | `/mnt/pve/unas/services/shared-db/garage/` |
+
+### ✅ Migrated (Phase 1) — SQLite to UNAS
+
+| Service | Old Path | New Path | Status |
+|---|---|---|---|
+| memos | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | ✅ Healthy |
+| arcane | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | ✅ Running |
+| n8n | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | ✅ Healthy |
+
+### ⏳ Still Local — `/opt/stacks/` or Docker Volumes
+
+| Service | Current Location | Data Type | Estimated Size | UNAS Target |
+|---|---|---|---|---|
+| arr-stack | `/opt/stacks/arr-stack/` | Configs + media | ~55 GB | `arr-stack/{config,audiobooks,ebooks,podcasts,torrents}` |
+| qdrant | `/opt/stacks/qdrant/qdrant_storage` | Vector DB | ~7 MB | `qdrant/` |
+| dozzle | `/opt/stacks/dozzle/dozzle_data` | Log viewer state | tiny | *(ephemeral, optional)* |
+| homepage | `/opt/stacks/homepage/{config,icons}` | Dashboard config | ~164 KB | *(keep in stack dir)* |
+
+### 🗄️ Databases Requiring Migration
+
+| Service | Engine | Current Path | Target | Migration Method | Status |
+|---|---|---|---|---|---|---|
+| immich | PostgreSQL | `/opt/stacks/immich/postgres` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending |
+| litellm | PostgreSQL | `/opt/stacks/ai/postgres_data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending |
+| lobehub | PostgreSQL | `/opt/stacks/ai/lobehub/data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending |
+| daytona | PostgreSQL | volume `daytona-minimal_db_data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending |
+| paperless-ngx | PostgreSQL | volume `paperless-ngx_pgdata` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending |
+| memos | SQLite | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | `cp` + `chown` | ✅ Done |
+| arcane | SQLite | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | `cp` + `chown` | ✅ Done |
+| n8n | SQLite | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | `cp` + `chown` | ✅ Done |
+| pocketid | SQLite | `pocketid/` | *(already on UNAS)* | — | ✅ Done |
+| vaultwarden | SQLite | `vaultwarden/` | *(already on UNAS)* | — | ✅ Done |
+| karakeep | SQLite + Meilisearch | `karakeep/{data,meilisearch}` | *(already on UNAS)* | — | ✅ Done |
+
+### 🧠 ML Models & Caches (Regenerable — Can Stay Local)
+
+| Resource | Path | Decision |
+|---|---|---|
+| Immich ML models | volume `immich_model-cache` | Cache — stay |
+| Paperless Redis | volume `paperless-ngx_redisdata` | Ephemeral — stay |
+| AI stack Redis | volume `nuc-ai-core_redis_data` | Ephemeral — stay |
+| SearXNG search cache | volume (hash) at `/var/lib/docker/volumes/bb5bb...` | Ephemeral — stay |
+| Daytona runner Docker | volume `daytona-minimal` runner data | Can stay |
+
+## Media Content (Already on UNAS)
+
+| Mount | Content |
+|---|---|
+| `/mnt/pve/unas/media/images/` | Immich photo library |
+| `/mnt/pve/unas/media/documents/public/paperless-ngx/` | Paperless documents (consume, export, library) |
+| `/mnt/pve/unas/media/audiobooks/` | Audiobookshelf media |
+| `/mnt/pve/unas/media/ebooks/` | Audiobookshelf ebooks |
+| `/mnt/pve/unas/media/podcasts/` | Audiobookshelf podcasts |
+| `/mnt/pve/unas/backup/immich/` | Immich DB backups |
+
+## Prospective UNAS Layout
+
+```
+/mnt/pve/unas/
+├── services/
+│   ├── gluetun/
+│   │   └── data/              ← VPN config
+│   ├── immich/
+│   │   ├── db/                ← PostgreSQL (to migrate)
+│   │   ├── encoded-video/     ← Already migrated
+│   │   ├── profile/           ← Already migrated
+│   │   └── thumbs/            ← Already migrated
+│   ├── karakeep/
+│   │   ├── data/              ← Already migrated
+│   │   └── meilisearch/       ← Already migrated
+│   ├── ntfy/
+│   │   ├── cache/             ← Already migrated
+│   │   └── config/            ← Already migrated
+│   ├── paperless-ai/          ← Already migrated
+│   ├── paperless-ngx/
+│   │   └── pgdata/            ← PostgreSQL (to migrate)
+│   ├── pocketid/              ← Already migrated
+│   ├── stremio/               ← Already migrated
+│   ├── traccar/
+│   │   ├── data/              ← Already migrated
+│   │   ├── logs/              ← Already migrated
+│   │   └── traccar.xml        ← Already migrated
+│   ├── vaultwarden/           ← Already migrated
+│   ├── memos/                 ← SQLite ✅ Done
+│   ├── qdrant/                ← Vector DB (to migrate)
+│   ├── arcane/                ← SQLite ✅ Done
+│   ├── n8n/                   ← SQLite ✅ Done
+│   ├── shared-db/
+│   │   ├── postgres/          ← ⚠️ Local volume, NOT on UNAS
+│   │   └── garage/
+│   │       ├── data/          ← S3 data (on UNAS)
+│   │       └── meta/          ← S3 metadata (on UNAS)
+│   ├── arr-stack/
+│   │   ├── config/            ← Bind mounts (to migrate)
+│   │   ├── audiobooks/        ← Media (to migrate)
+│   │   ├── ebooks/            ← Media (to migrate)
+│   │   ├── podcasts/          ← Media (to migrate)
+│   │   └── torrents/          ← Downloads (to migrate)
+│
+├── media/
+│   ├── images/                ← Immich photo library
+│   └── documents/
+│       └── public/
+│           └── paperless-ngx/
+│               ├── consume/
+│               ├── export/
+│               └── library/
+│
+├── backups/
+│   └── immich/                ← Immich DB backups
+│
+└── (potential future: torrents/, cache/)
+```
+
+## Garage S3 — `/opt/stacks/shared-db/`
+
+S3-compatible object storage using [Garage](https://garagehq.deuxfleurs.fr/) v1.0.0.
+Replaces RustFS from the LobeHub AI stack. Single-node, replication_factor=1.
+
+| Detail | Value |
+|---|---|
+| Container | `garage` (dxflrs/garage:v1.0.0) |
+| Data | `/mnt/pve/unas/services/shared-db/garage/{data,meta}` |
+| S3 API | `garage:3900` (internal, shared_backend) |
+| Admin API | `127.0.0.1:3901` (localhost only) |
+| Regional | No auth (internal network) |
+| Bucket | `lobe-files` |
+| Key ID | `GK55210bd6b56d92891db26fac` |
+| Secret Key | `d0af9e4eb69d7773ffa17950feee27925dbc70eb3efd4ee23a3e1324672c869a` |
+
+## Shared PostgreSQL — `/opt/stacks/shared-db/`
+
+Centralized PostgreSQL 16 instance for all non-immich services.
+
+| Detail | Value |
+|---|---|
+| Container | `shared-postgres` (postgres:16) |
+| Data | `shared-db_shared-pgdata` (local Docker volume, NOT NFS) |
+| Networks | `shared_backend` |
+| Auth | `postgres` superuser with password from `.env` |
+
+### Databases & Users (pre-created on init)
+
+| Database | Owner | Password | Used By |
+|---|---|---|---|
+| `paperless` | `paperless` | `paperless` | paperless-ngx |
+| `daytona` | `user` | `pass` | daytona |
+| `litellm` | `litellm` | `litellm_password_here` | LiteLLM |
+| `lobechat` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub |
+| `paradedb` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub |
+
+## Key Architectural Decision: PostgreSQL Stays Local
+
+**Problem:** PostgreSQL checks data directory ownership via `stat(2)` and refuses to start
+if the process UID doesn't match the file owner UID. NFS with `all_squash` maps all UIDs
+to `65534` (nobody), breaking this check. `chown` also fails on NFS.
+
+**Solution:** All PostgreSQL instances use local Docker volumes on NVMe:
+
+```
+Local NVMe (/var/lib/docker/volumes/)     UNAS NFS (/mnt/pve/unas/)
+├── shared-postgres data                   ├── pg_dump backups ← scheduled
+├── immich_postgres data                   ├── SQLite DBs (memos, arcane, n8n)
+├── paperless-ngx pgdata                    ├── Garage S3 data
+├── daytona-minimal db_data                ├── Media files (immich, paperless)
+├── litellm-db / ai postgres_data          ├── Configs / service data
+├── lobehub postgres data
+└── all Redis / cache volumes
+```
+
+**Backup strategy:** `pg_dump` each database to UNAS on a schedule (e.g. daily cron).
+
+## Migration Priority
+
+| Priority | Service | Effort | Risk | Status | Reason |
+|---|---|---|---|---|---|
+| P0 | memos | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
+| P0 | n8n | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
+| P0 | arcane | Minimal (SQLite copy) | Low | ✅ Done | Simple file move |
+| P0 | qdrant | Minimal (file copy) | Low | ✅ Done | Fresh start on UNAS |
+| P1 | daytona → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running |
+| P1 | litellm → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running |
+| P1 | lobehub → shared-postgres | Medium (pg_dump) | Low | ⛔ Skipped | Uses ParadeDB (pg17+extensions), incompatible with shared-postgres (pg16) |
+| P1 | paperless-ngx → shared-postgres | Medium (pg_dump) | Medium | ✅ Done | pg_dump → restore → running |
+| P1 | immich → shared-postgres | Medium (pg_dump) | Medium | ⏳ | Custom pgvector image |
+| P2 | arr-stack | Large (55 GB copy) | Low | ⏳ | Simple copy, time-consuming |
+
+## Mail Archive (Potential Savings)
+
+Unattached anonymous volumes from old/stopped containers:
+
+| Volume | Size | Likely From | Action |
+|---|---|---|---|
+| `0b6d07ff...` | ? | old daytona runner | prune |
+| (100+ hex-named volumes) | ? | ephemeral build cache | prune |
+
+Recommend: `docker system df` then `docker volume prune` to clean.
+
+---
+
+## Database Consolidation Plan
+
+### Current: 10 Database Instances
+
+| # | Container | Type | Image | Used By | Networks |
+|---|---|---|---|---|---|
+| 1 | `immich_postgres` | PostgreSQL | `immich-app/postgres:14-vectorchord` (pgvector) | immich | `immich_default` |
+| 2 | `paperless-ngx-db-1` | PostgreSQL | `postgres:16` | paperless-ngx | `paperless-ngx_default` |
+| 3 | `daytona-minimal-db-1` | PostgreSQL | `postgres:16-alpine` | daytona | `daytona-minimal_daytona-network` |
+| 4 | `nuc-ai-core-litellm-db-1` | PostgreSQL | `postgres:16-alpine` | litellm | `ai-internal` |
+| 5 | `lobe-postgres` | PostgreSQL | `postgres:16-alpine` | lobehub (lobechat, paradedb) | `ai-internal` |
+| 6 | `paperless-ngx-broker-1` | Redis 7 | `redis:7` | paperless broker | `paperless-ngx_default` |
+| 7 | `immich_redis` | Valkey 9 | `valkey/valkey:9` | immich cache | `immich_default` |
+| 8 | `daytona-minimal-redis-1` | Redis Alpine | `redis:alpine` | daytona | `daytona-minimal_daytona-network` |
+| 9 | `lobe-redis` | Redis 7 | `redis:7-alpine` | lobehub | `ai-internal` |
+| 10 | `redis-searxng` | Redis 7 | `redis:7-alpine` | searxng | `ai-internal` |
+
+### Target: 6 Database Instances (remove 4)
+
+| # | Container | Type | DBs Hosted | Saved |
+|---|---|---|---|---|
+| 1 | `immich_postgres` | PostgreSQL immich-app (pgvector) | immich | 🔒 Must stay separate |
+| 2 | **`shared-postgres`** 🆕 | **PostgreSQL 16** | **paperless, daytona, litellm, lobechat, paradedb** | **−3 containers** |
+| 3 | `paperless-ngx-broker-1` | Redis 7 | paperless broker | 🔒 Must stay separate |
+| 4 | `immich_redis` | Valkey 9 | immich cache | 🔒 Must stay separate |
+| 5 | `daytona-minimal-redis-1` | Redis Alpine | daytona | 🔒 Stay separate |
+| 6 | `lobe-redis` | Redis 7 | lobehub | 🔒 Stay separate |
+| 7 | `redis-searxng` | Redis 7 | searxng | 🔒 Stay separate |
+| 7 | ~~`nuc-ai-core-litellm-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
+| 8 | ~~`lobe-postgres`~~ | 🗑️ removed | → shared-postgres | ✅ |
+| 9 | ~~`paperless-ngx-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
+| 10 | ~~`daytona-minimal-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ |
+
+**Result: 10 → 6 instances, removing 4 containers, saving ~160MB RAM.**
+
+### Why Redis Stays Separate
+
+| Reason | Detail |
+|---|---|
+| **Valkey ≠ Redis** | Immich uses Valkey 9 which has diverged from Redis 7 |
+| **`SELECT` collisions** | Apps may use specific DB numbers that clash |
+| **Password mismatch** | Each app has its own `requirepass` / `REDIS_PASSWORD` |
+| **Ephemeral = low value** | Caches/brokers rebuild on restart, savings don't justify risk |
+| **`FLUSHALL` risk** | One misconfigured app could wipe another's cache |
+
+### Why PG Consolidation Works
+
+| Reason | Detail |
+|---|---|
+| **Same major version** | All target DBs use PostgreSQL 16 |
+| **Native compatibility** | No conflicting extensions (only immich has pgvector) |
+| **Isolation via databases** | Each service gets its own database + user within shared PG |
+| **Simple restore** | `pg_dump` + `pg_restore` into shared instance |
+| **Single backup** | One `pg_dumpall` covers all 5 databases |
+
+### Implementation: Shared PostgreSQL
+
+Create `/opt/stacks/shared-db/docker-compose.yml`:
+
+```yaml
+services:
+  postgres:
+    image: postgres:16
+    container_name: shared-postgres
+    restart: unless-stopped
+    volumes:
+      - /mnt/pve/unas/services/shared-db:/var/lib/postgresql/data
+    environment:
+      POSTGRES_PASSWORD: ${SHARED_DB_PASSWORD}
+    ports:
+      - "127.0.0.1:5432:5432"
+    deploy:
+      resources:
+        limits:
+          cpus: "2.0"
+          memory: 1G
+        reservations:
+          cpus: "1.0"
+          memory: 512M
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+networks:
+  default:
+    external: true
+    name: shared_backend
+```
+
+### Migration Order (Phase 2)
+
+```
+Step 0: Create shared-postgres container
+Step 1: Dump ALL 4 databases from their current instances
+Step 2: Create users + databases on shared-postgres
+Step 3: Restore each dump into shared-postgres
+Step 4: One by one, update each app's compose to point to shared-postgres
+```
+
+#### Detailed: Paperless-ngx
+
+| Change | Before | After |
+|---|---|---|
+| DB service | `db` (in paperless compose) | remove entirely |
+| `PAPERLESS_DBHOST` | `db` | `shared-postgres` |
+| DB credentials | defaults (paperless/paperless) | keep same |
+| Network | `paperless-ngx_default` | `shared_backend` |
+
+#### Detailed: Daytona
+
+| Change | Before | After |
+|---|---|---|
+| DB service | `db` (in daytona compose) | remove entirely |
+| `DB_HOST` | `db` | `shared-postgres` |
+| `DB_USERNAME` | `user` | keep same |
+| `DB_DATABASE` | `daytona` | keep same |
+| Network | `daytona-minimal_daytona-network` | `shared_backend` |
+
+#### Detailed: LiteLLM
+
+| Change | Before | After |
+|---|---|---|
+| DB service | `litellm-db` (in ai compose) | remove entirely |
+| `DATABASE_URL` | `postgresql://litellm:...@litellm-db:5432/litellm` | `postgresql://litellm:...@shared-postgres:5432/litellm` |
+| Network | `ai-internal` | `shared_backend` |
+
+#### Detailed: LobeHub
+
+| Change | Before | After |
+|---|---|---|
+| DB service | `postgresql` (in lobehub.yml) | remove entirely |
+| `DATABASE_URL` | `postgresql://postgres:...@postgresql:5432/lobechat` | `postgresql://postgres:...@shared-postgres:5432/lobechat` |
+| Lobe also needs `paradedb` DB | created manually on initial setup | created via SQL script |
+| Network | `ai-internal` | `shared_backend` |
+
+### Total Impact
+
+| Metric | Before | After |
+|---|---|---|
+| PostgreSQL instances | 5 | 2 |
+| Redis instances | 5 | 5 (unchanged) |
+| Total container count | 10 DBs | 7 DBs (−3) |
+| RAM saved (estimated) | — | ~160 MB |
+| Complexity | Fragmented | Centralized |
+| Backup burden | 5 pg_dumps | 2 pg_dumps |
+
+### Timing Estimates
+
+| Service | Downtime | Cumulative |
+|---|---|---|
+| Setup shared-postgres + dump all | 0 (live dump) | 0 |
+| daytona → shared | ~2 min | 2 min |
+| litellm → shared | ~2 min | 4 min |
+| lobehub → shared | ~3 min | 7 min |
+| paperless-ngx → shared | ~5 min | 12 min |
+| immich DB → UNAS bind mount | ~10 min | 22 min |
+| **Total** | **~22 min** | |
+
+### Notes
+
+- All compose files are version-controlled in `/opt/stacks/{service}/docker-compose.yml`
+- Always test with `docker compose config` before `up -d`
+- PostgreSQL stays on local NVMe — backed up via `pg_dump` to UNAS
+- Garage S3 replaces RustFS from LobeHub stack (RustFS will be removed when LobeHub is switched)
+- LobeHub's `paradedb` extension requires `CREATE EXTENSION IF NOT EXISTS paradedb;` on the shared instance
+- All PostgreSQL dbs consolidated into shared-postgres except immich (pgvector requirement)