From 061f09b2d2270d0be8c84ac0177cabcf9545db1f Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Thu, 21 May 2026 00:20:18 +0200 Subject: [PATCH] docs: top-10 fixes (PORTMAP, storage truth, banners, AdGuard scrub, zoraxy UCI removed, CT 111 section, Daytona purge) --- history/case-study.md | 4 + history/mcp-gateway-requirements.md | 4 + history/scrubbing-list-2026-05-17.md | 4 + history/traefik-migration-docker-labels.md | 4 + history/traefik-migration.md | 4 + services/adguard-dns.md | 11 +- services/homelab-architecture.md | 29 +- services/zoraxy.md | 65 +-- stacks/CLAUDE.md | 4 +- stacks/PORTMAP.md | 47 ++- stacks/nexa/CLAUDE.md | 4 + stacks/storage.md | 453 ++++----------------- 12 files changed, 160 insertions(+), 473 deletions(-) diff --git a/history/case-study.md b/history/case-study.md index 60d7162..7342f9c 100644 --- a/history/case-study.md +++ b/history/case-study.md @@ -1,3 +1,7 @@ +> **STATUS: NARRATIVE — historical writeup. Not authoritative for current state.** + +--- + # Case Study — The Nuclide Homelab, built with Claude ## Origin story diff --git a/history/mcp-gateway-requirements.md b/history/mcp-gateway-requirements.md index b298430..d32a914 100644 --- a/history/mcp-gateway-requirements.md +++ b/history/mcp-gateway-requirements.md @@ -1,3 +1,7 @@ +> **STATUS: SUPERSEDED 2026-05-17 — current implementation lives in services/mcp-gateway.md. Kept for design-rationale history.** + +--- + # MCP Gateway — Reconstructed Design Spec (in-progress, "Phase 1") > Reconstructed 2026-05-16 from code/configs/git history. The gateway is a diff --git a/history/scrubbing-list-2026-05-17.md b/history/scrubbing-list-2026-05-17.md index 1e4b5c1..ef95307 100644 --- a/history/scrubbing-list-2026-05-17.md +++ b/history/scrubbing-list-2026-05-17.md @@ -1,3 +1,7 @@ +> **STATUS: SNAPSHOT — frozen inventory from 2026-05-17. Reality has moved on; consult CHANGELOG.md + services/* for current state.** + +--- + # Scrubbing List — /opt/stacks (2026-05-17) Read-only audit of unused/stale data. **Nothing here has been deleted** — review diff --git a/history/traefik-migration-docker-labels.md b/history/traefik-migration-docker-labels.md index e04b31e..44f0c80 100644 --- a/history/traefik-migration-docker-labels.md +++ b/history/traefik-migration-docker-labels.md @@ -1,3 +1,7 @@ +> **STATUS: ABANDONED 2026-05-16 — Zoraxy is the production reverse proxy. Kept for design-decision history.** + +--- + # Traefik Migration Guide Using Docker Labels ## Overview diff --git a/history/traefik-migration.md b/history/traefik-migration.md index b33b878..70c1234 100644 --- a/history/traefik-migration.md +++ b/history/traefik-migration.md @@ -1,3 +1,7 @@ +> **STATUS: ABANDONED 2026-05-16 — Zoraxy is the production reverse proxy. Kept for design-decision history. See services/zoraxy.md for current setup.** + +--- + # Recommended: Traefik Proxy Replacement ## Why Traefik over Zoraxy? diff --git a/services/adguard-dns.md b/services/adguard-dns.md index 0f736ea..10bb520 100644 --- a/services/adguard-dns.md +++ b/services/adguard-dns.md @@ -5,7 +5,7 @@ AdGuard DNS at `192.168.1.2` can handle internal domain resolution, eliminating ## Current Setup - **AdGuard URL**: `http://192.168.1.2` -- **Status**: Running on Proxmox VM 102 +- **Status**: Running on Proxmox **LXC 102** (`dns`, 192.168.1.2) - **Homepage widget**: Configured ## AI Service DNS Entries @@ -48,7 +48,8 @@ Add these static DNS entries in AdGuard to resolve AI services locally: # Get session token first curl -c /tmp/cookies.txt -k -X POST http://192.168.1.2:3000/ \ -H "Content-Type: application/json" \ - -d '{"username": "root", "password": "tapirnase"}' + -d "{\"username\": \"root\", \"password\": \"${ADGUARD_PASSWORD}\"}" +# Export ADGUARD_PASSWORD from your shell env or a `.env` file — never hardcode. # Add static DNS entry curl -s -k -c /tmp/cookies.txt -b /tmp/cookies.txt \ @@ -115,8 +116,8 @@ echo "✅ Done! Add these in AdGuard UI manually." - `pp-api.nuclide.systems` → `192.168.1.40:8000` (Paperless API) ### Home Assistant -- `ha.nuclide.systems` → `192.168.1.10` (Home Assistant) -- `homeassistant.local` → `192.168.1.10` (mDNS fallback) +- `ha.nuclide.systems` → `192.168.1.60` (Home Assistant OS VM 100) +- `homeassistant.local` → `192.168.1.60` (mDNS fallback) ## Internal DNS Server Setup @@ -200,7 +201,7 @@ S3_ENDPOINT=https://s3.nuclide.systems ## Notes -- AdGuard runs on VM 102 at `192.168.1.2` +- AdGuard runs on **LXC 102** at `192.168.1.2` - All AI services run on `192.168.1.40` - Using same IP for all domains is intentional (single endpoint) - SSL certs are issued by Zoraxy (public domain validation) diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index eb0c6b3..fe1e5a6 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -10,7 +10,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, `rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`. - **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`** (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via - `root@pam!mcp` token (now `PVEAuditor`, read-only). **7 guests:** + `root@pam!mcp` token (now `PVEAuditor`, read-only). **9 guests (+ 2 planned)** — canonical roster in [`ct-inventory.md`](../ct-inventory.md): - `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60` - `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49` - `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking, @@ -22,6 +22,10 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40` - `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41` - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4` + - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 + - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 + - `109` lxc **observe** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md` + - `113` lxc **db** *(planned)* — shared Postgres + pgbouncer once second NVMe lands - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA @@ -133,12 +137,12 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, - **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the **universal SSO/IdP for the entire ecosystem** — effectively every service authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud, - Daytona via the Keycloak→PocketID OIDC adapter, etc.). Single sign-on - everywhere; one identity source to secure/audit. + Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity + source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20. ## Docker networks `ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) · -plus per-stack: `arr-stack_default`, `daytona-minimal_daytona-network`, +plus per-stack: `arr-stack_default`, `immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`. The MCP gateway bridges `ai-internal` + `shared_backend`. @@ -176,12 +180,15 @@ The MCP gateway bridges `ai-internal` + `shared_backend`. `mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents). - **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to `.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions), - daytona, fetch, sequential-thinking, ntfy, memos, docling, markitdown, - crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky (brianellin, - rate-limit pending). Groups incl. `shepard-mcp`. -- **Daytona** (`daytona.nuclide.systems`, daytona-minimal stack) — secure - code sandboxes (Agent Operator deep-dives, shepard). Registry fixed to - internal+host `:5000` (was misconfigured at 6000). Snapshot `sciviz-py`. + **coder** (replaced daytona 2026-05-20), fetch, sequential-thinking, ntfy, memos, + docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui, + bluesky (brianellin, rate-limit pending). Groups incl. `shepard-mcp`. + Gaps to close (community MCPs exist): gitea, paperless, karakeep, vaultwarden, + proxmox-ve, audiobookshelf. See [`services/mcp-gateway.md`](mcp-gateway.md). +- **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox / + dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough, + baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack + pre-baked). See [`services/dev-environment.md`](dev-environment.md). - **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins + agents in `lobechat` DB (lobe-postgres). Agents scriptable via `scripts/lobe_agent.py`. Native QStash scheduler NOT functional here → @@ -199,7 +206,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`. ## Data layer - **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the **primary/standard DB for all deployments**. Tenants: LiteLLM, paperless, - daytona, **memos** (migrated 2026-05-19). Superuser + **memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser `postgres` via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's `.env`). Tuned 2026-05-19 (3G limit, shared_buffers 768M, max_conn 200). diff --git a/services/zoraxy.md b/services/zoraxy.md index d331e86..d8b6788 100644 --- a/services/zoraxy.md +++ b/services/zoraxy.md @@ -1,5 +1,8 @@ # Zoraxy Reverse Proxy Setup Guide +> **Zoraxy is a Go-based reverse proxy + ACME daemon, running as a systemd service in LXC 108. It is NOT OpenWRT. Earlier docs referenced \`uci\` commands — those are removed; configuration is via the admin UI or by editing JSON config files in \`/opt/zoraxy/conf/proxy/\` and restarting \`systemctl restart zoraxy\`.** + + ## Overview Configure reverse proxies for AI service endpoints with automatic SSL certificates. @@ -12,7 +15,7 @@ Configure reverse proxies for AI service endpoints with automatic SSL certificat | MCP Gateway | `mcp.nuclide.systems` | `192.168.1.40` | `8080` | | Garage S3 | `s3.nuclide.systems` | `192.168.1.40` | `10004` | -## Option A: Web UI (Easiest - No SSH Required) +## Method A — Zoraxy admin UI (recommended) ### Step 1: Access Zoraxy Panel ``` @@ -74,65 +77,7 @@ Click **"Save"** at the bottom of the page. - Wait 1-2 minutes for certificates to be issued - Visual indicator: Green padlock icon in the UI -## Option B: SSH/UCI (Automated - Requires Root Access) - -### Prerequisites -- SSH access to Zoraxy with root credentials -- Command-line access to Zoraxy's UCI config system - -### Step 1: SSH into Zoraxy -```bash -ssh root@192.168.1.4 -``` - -### Step 2: Configure via UCI -Run this command directly on Zoraxy: - -```bash -# Apply all rewrite rules -uci set network.rewrite_litellm.enable='1' -uci set network.rewrite_litellm.domain='litellm.nuclide.systems' -uci set network.rewrite_litellm.target_type='0' -uci set network.rewrite_litellm.target='192.168.1.40:14000' -uci set network.rewrite_litellm.ssl='1' -uci set network.rewrite_litellm.hsts='1' - -uci set network.rewrite_lobehub.enable='1' -uci set network.rewrite_lobehub.domain='chat.nuclide.systems' -uci set network.rewrite_lobehub.target_type='0' -uci set network.rewrite_lobehub.target='192.168.1.40:14001' -uci set network.rewrite_lobehub.ssl='1' -uci set network.rewrite_lobehub.hsts='1' - -uci set network.rewrite_mcp.enable='1' -uci set network.rewrite_mcp.domain='mcp.nuclide.systems' -uci set network.rewrite_mcp.target_type='0' -uci set network.rewrite_mcp.target='192.168.1.40:8080' -uci set network.rewrite_mcp.ssl='1' -uci set network.rewrite_mcp.hsts='1' - -uci set network.rewrite_garage.enable='1' -uci set network.rewrite_garage.domain='s3.nuclide.systems' -uci set network.rewrite_garage.target_type='0' -uci set network.rewrite_garage.target='192.168.1.40:10004' -uci set network.rewrite_garage.ssl='1' -uci set network.rewrite_garage.hsts='1' - -# Commit and restart -uci commit network -/etc/init.d/network restart -``` - -### Step 3: Verify -```bash -# Check if rewrite rules are loaded -uci show network.rewrite_* - -# Test connectivity -curl -I http://litellm.nuclide.systems -``` - -## Option C: Automated Script (Local) +## Method B — automation via scripts/zoraxy_sync.py Run from the control node: diff --git a/stacks/CLAUDE.md b/stacks/CLAUDE.md index f0536da..ca77a28 100644 --- a/stacks/CLAUDE.md +++ b/stacks/CLAUDE.md @@ -31,14 +31,14 @@ Do not start these without checking with the operator first. ## MCP gateway leaves, MCP child containers stay (planned) -When CT 109 "observe" lands, **`/opt/stacks/ai/mcp-gateway/` moves to CT 109** (it's a control plane: OIDC, agent scheduling, token store, usage stats). The ~20 MCP child server containers in this CT (`mcp-searxng`, `mcp-gotify`, `mcp-immich`, `mcp-fetch`, `mcp-time`, `comfyui-mcp`, `daytona-mcp`, `kroki-mcp`, etc.) **stay here on CT 104**. +When CT 109 "observe" lands, **`/opt/stacks/ai/mcp-gateway/` moves to CT 109** (it's a control plane: OIDC, agent scheduling, token store, usage stats). The ~20 MCP child server containers in this CT (`mcp-searxng`, `mcp-gotify`, `mcp-immich`, `mcp-fetch`, `mcp-time`, `comfyui-mcp`, `coder-mcp`, `kroki-mcp`, etc.) **stay here on CT 104**. The migrated gateway will reach this CT's docker daemon over the planned docker-socket-proxy (see `/docs/proxmox-optimizations.md` §16). Until CT 109 exists, the gateway runs locally and uses the bind-mounted socket. Do not piecemeal-move the gateway; it's tied to the CT 109 build. ## What is safe to do here - Anything to do with the actively-running ~65 containers (Immich, n8n, Karakeep, - ComfyUI, Daytona, Vaultwarden, Paperless, LobeChat, LiteLLM, MCP servers, etc.). + ComfyUI, Vaultwarden, Paperless, LobeChat, LiteLLM, MCP servers, etc.). Daytona was decommissioned 2026-05-20 — compose files renamed `.DECOMMISSIONED-2026-05-20`; do not restart. - Image pulls + rebuilds for the active stacks (mind the n8n version pin — see its `docker-compose.yaml` comment). - Rotate exposed secrets in `.env` files (compose YAMLs themselves are git-tracked). diff --git a/stacks/PORTMAP.md b/stacks/PORTMAP.md index 769ef70..a2a2bb8 100644 --- a/stacks/PORTMAP.md +++ b/stacks/PORTMAP.md @@ -40,8 +40,9 @@ | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| [11000](http://192.168.1.40:11000) | Pocket ID | `pocket-id` | [id.nuclide.systems](https://id.nuclide.systems) | OIDC provider | -| [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager | +| [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager · **no Pocket-ID client yet — gap** | + +> Pocket-ID **migrated from this CT to LXC 110** on 2026-05-20. See the [External Services](#external-services-not-on-nuc-docker) table below. --- @@ -115,8 +116,8 @@ All arr-stack services run behind `vpn_gluetun` container network. | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| [18000](http://192.168.1.40:18000) | Daytona API | `daytona-minimal-api-1` | [daytona.nuclide.systems](https://daytona.nuclide.systems) | Dev sandbox | -| [18001](http://192.168.1.40:18001) | Daytona SSH | `daytona-minimal-ssh-gateway-1` | — | SSH gateway | + + --- @@ -134,17 +135,38 @@ _Traccar moved to 15000–15001 (Documents range). 19000–19001 now free._ --- +## LXC 111 — Dev (`192.168.1.42`) + +CT 111 hosts the self-hosted dev platform. Same Pocket-ID SSO as the rest. + +| Port | Service | Container | Public URL | Notes | +|---|---|---|---|---| +| [7080](http://192.168.1.42:7080) | Coder | `coder` | [dev.nuclide.systems](https://dev.nuclide.systems) | Workspace orchestrator; OIDC via Pocket-ID; password auth disabled | +| [3000](http://192.168.1.42:3000) | Gitea | `gitea` | [git.nuclide.systems](https://git.nuclide.systems) | Self-hosted Git; OIDC; password form disabled | +| [222](http://192.168.1.42:222) | Gitea SSH | `gitea` | — | `ssh -p 222 git@git.nuclide.systems` | +| 5432 (internal) | coder-db | `coder-db` | — | Postgres 16 for Coder | +| 5432 (internal) | gitea-db | `gitea-db` | — | Postgres 16 for Gitea | +| [13080](http://192.168.1.42:13080) | docs site | `docs-server` | — (LAN-only) | mkdocs Material; auto-rebuilds from `fkrebs/docs` every 5 min | +| (no port) | coder-fkrebs-dev | (workspace) | (Coder app proxy) | Active Coder workspace | +| (no port) | act-runner | `act-runner` | — | Gitea Actions runner (`ct111-runner`) | + +--- + ## External Services (Not on NUC Docker) Zoraxy routes to these external backends: | Domain | Target | Host | Service | |---|---|---|---| -| [ha.nuclide.systems](https://ha.nuclide.systems) | `192.168.1.60:8123` | Home Assistant VM | Home automation | -| [nc.nuclide.systems](https://nc.nuclide.systems) | `192.168.1.41:11000` | Nextcloud LXC | Cloud storage | -| [ocpp.nuclide.systems](https://ocpp.nuclide.systems) | `192.168.1.60:8887` | Home Assistant VM | EV charger OCPP | -| [shepard.nuclide.systems](https://shepard.nuclide.systems) | `192.168.1.49:80` | Docker host | Shepard | -| [shepard-api.nuclide.systems](https://shepard-api.nuclide.systems) | `192.168.1.49:8080` | Docker host | Shepard API | +| [ha.nuclide.systems](https://ha.nuclide.systems) | `192.168.1.60:8123` | Home Assistant VM 100 | Home automation | +| [nc.nuclide.systems](https://nc.nuclide.systems) | `192.168.1.41:11000` | Nextcloud LXC 105 | Cloud storage | +| [ocpp.nuclide.systems](https://ocpp.nuclide.systems) | `192.168.1.60:8887` | Home Assistant VM 100 | EV charger OCPP | +| [shepard.nuclide.systems](https://shepard.nuclide.systems) | `192.168.1.49:80` | Shepard LXC 101 | Shepard | +| [shepard-api.nuclide.systems](https://shepard-api.nuclide.systems) | `192.168.1.49:8080` | Shepard LXC 101 | Shepard API | +| [id.nuclide.systems](https://id.nuclide.systems) | `192.168.1.5:11000` | Pocket-ID LXC 110 | OIDC IdP (migrated 2026-05-20) | +| [git.nuclide.systems](https://git.nuclide.systems) | `192.168.1.42:3000` | Dev LXC 111 | Gitea (self-hosted Git) | +| [dev.nuclide.systems](https://dev.nuclide.systems) | `192.168.1.42:7080` | Dev LXC 111 | Coder (workspace orchestrator) | +| Gitea SSH | `192.168.1.42:222` | Dev LXC 111 | `ssh -p 222 git@git.nuclide.systems` | --- @@ -157,7 +179,6 @@ Zoraxy routes to these external backends: | [mcp.nuclide.systems](https://mcp.nuclide.systems) | mcp-gateway | 8080 | | [arcane.nuclide.systems](https://arcane.nuclide.systems) | arcane | 10002 | | [gotify.nuclide.systems](https://gotify.nuclide.systems) | gotify | 10003 | -| [id.nuclide.systems](https://id.nuclide.systems) | pocket-id | 11000 | | [vault.nuclide.systems](https://vault.nuclide.systems) | vaultwarden | 11001 | | [immich.nuclide.systems](https://immich.nuclide.systems) | immich_server | 12000 | | [immich-tools.nuclide.systems](https://immich-tools.nuclide.systems) | immich_power_tools | (container-internal) | @@ -165,7 +186,6 @@ Zoraxy routes to these external backends: | [n8n.nuclide.systems](https://n8n.nuclide.systems) | n8n | 16000 | | [memos.nuclide.systems](https://memos.nuclide.systems) | memos | 17000 | | [hoarder.nuclide.systems](https://hoarder.nuclide.systems) | karakeep | 17001 | -| [daytona.nuclide.systems](https://daytona.nuclide.systems) | daytona-api | 18000 | | [traccar.nuclide.systems](https://traccar.nuclide.systems) | traccar | 15000 | | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | dozzle | 10001 | @@ -192,7 +212,10 @@ Zoraxy routes to these external backends: | `62bf4e0d` | memos | `https://memos.nuclide.systems/auth/callback` | | `d92f82b0` | karakeep | `https://hoarder.nuclide.systems/api/auth/callback/custom` | | `a14b8076` | nextcloud | `https://nc.nuclide.systems/apps/user_oidc/code` | -| `798a367f` | daytona | `https://daytona.nuclide.systems` | +| `0aee4280` | Coder | `https://dev.nuclide.systems/api/v2/users/oidc/callback` | +| `9444609e` | Gitea | `https://git.nuclide.systems/user/oauth2/pocket-id/callback` | +| `38469e7e` | Proxmox VE | `https://192.168.1.20:8006` | +| `798a367f` | ~~daytona~~ | **ORPHAN — Daytona decommissioned 2026-05-20, remove** | | `cbbf20d5` | Audiobookshelf | `https://abs.nuclide.systems/…` + `*` | | `d8733fcc` | shelfarr | `*` | | `78c78998` | Claude MCP | `https://claude.ai/api/mcp/auth_callback` + `https://mcp.nuclide.systems/mcp/auth/callback` | diff --git a/stacks/nexa/CLAUDE.md b/stacks/nexa/CLAUDE.md index 036eb22..201bc73 100644 --- a/stacks/nexa/CLAUDE.md +++ b/stacks/nexa/CLAUDE.md @@ -1,3 +1,7 @@ +> **STATUS: STALE — many claims (PVE version, Pocket-ID port, Dockge as docker manager) no longer accurate. Source of truth is /CLAUDE.md and /docs/services/. This file kept for the original nexa-stack design notes only.** + +--- + # Instructions for Claude (and other agents) This file tells future automated runs what they need to know about this repo. diff --git a/stacks/storage.md b/stacks/storage.md index af25025..2689691 100644 --- a/stacks/storage.md +++ b/stacks/storage.md @@ -1,409 +1,96 @@ -# Storage Layout — NUC 14 Docker Stacks +# Storage layout -## Data Location Architecture +> Verified state as of 2026-05-21. Source of truth for which service lives on which storage class. +> Earlier revisions of this doc were stale on multiple items (Garage, n8n, Arcane, Karakeep, Pocket-ID, arr-stack, Nextcloud's storage protocol, missing Coder/Gitea). Reconciled via live audit. -All persistent service data is being consolidated to `/mnt/pve/unas/services/{service}/` -on the UNAS NFS share, with media content in `/mnt/pve/unas/media/`. +## Storage classes -**PostgreSQL databases stay on local NVMe** — NFS ownership checks (`stat(2)` UID match) prevent -PostgreSQL from running on network filesystems. PostgreSQL is backed up via `pg_dump` to UNAS instead. - -| Mount | Source | Content | -|---|---|---| -| `/mnt/pve/unas/services/` | UNAS NFS | SQLite DBs, configs, files, S3 data | -| `/mnt/pve/unas/media/` | UNAS NFS | User media (photos, docs, books, audio) | -| `/mnt/pve/unas/backup/` | UNAS NFS | Service backups | -| `/var/lib/docker/volumes/` | Local NVMe | PostgreSQL data, Redis data | -| `/opt/stacks/` | Local NVMe | Compose files, small configs | - -## Migration Status - -### ✅ Already on UNAS — `/mnt/pve/unas/services/` - -| Service | UNAS Path(s) | Container Mount | Data Type | +| Class | Substrate | Where | Typical use | |---|---|---|---| -| gluetun | `gluetun/data/` | `/gluetun` | VPN config | -| immich | `immich/{encoded-video,profile,thumbs}/` | `/usr/src/app/upload/{encoded-video,profile,thumbs}` | App derived data | -| karakeep | `karakeep/data/`, `karakeep/meilisearch/` | `/data`, `/meili_data` | Bookmarks DB + Meilisearch index | -| ntfy | `ntfy/{config,cache}/` | `/etc/ntfy`, `/var/lib/ntfy` | Notification config + cache | -| paperless-ai | `paperless-ai/` | `/app/data` | AI processing data | -| pocketid | `pocketid/` | `/app/data` | Auth data (SQLite) | -| stremio | `stremio/` | `/root/.stremio-server` | Streaming server data | -| traccar | `traccar/{data,logs,traccar.xml}` | `/opt/traccar/{data,logs,conf/traccar.xml}` | GPS tracking data | -| vaultwarden | `vaultwarden/` | `/data` | Password vault (SQLite) | -| garage | `shared-db/garage/{data,meta}/` | `/var/lib/garage/{data,meta}` | S3 object storage (Garage) | +| Local zfs (rootfs) | NVMe in nuc | `/opt/stacks//` on each LXC | Tier-1 hot state: databases, SQLite, secret stores, anything latency-sensitive | +| Local zfs (subvol) | NVMe in nuc | `rpool/data/subvol--disk-0` per LXC | Container rootfs, image cache | +| Docker named volume | NVMe in nuc | `/var/lib/docker/volumes/...` on each Docker CT | Per-container persistent state managed by Docker | +| UNAS over **NFSv3** | `192.168.1.31` | `/mnt/pve/unas` on CTs 101, 103, 104, 111 | Bulk media, workspace home dirs, document storage | +| UNAS over **CIFS/SMB 3.1.1** | `192.168.1.31` | `/mnt/pve/unas` on CT 105 (Nextcloud) | Nextcloud-only path — earlier docs mis-labelled this as NFS | -### 🆕 Shared Infrastructure — `/opt/stacks/shared-db/` +## What lives where (verified 2026-05-21) -| Service | Hostname | Type | Data Location | -|---|---|---|---| -| shared-postgres | `shared-postgres` | PostgreSQL 16 (local volume) | `shared-db_shared-pgdata` volume | -| garage | `garage` | S3-compatible object storage (on UNAS) | `/mnt/pve/unas/services/shared-db/garage/` | +### UNAS NFS — `/mnt/pve/unas/services/...` -### ✅ Migrated (Phase 1) — SQLite to UNAS +Bulk + media + non-latency-sensitive app data. -| Service | Old Path | New Path | Status | -|---|---|---|---| -| memos | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | ✅ Healthy | -| arcane | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | ✅ Running | -| n8n | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | ✅ Healthy | - -### ⏳ Still Local — `/opt/stacks/` or Docker Volumes - -| Service | Current Location | Data Type | Estimated Size | UNAS Target | +| Service | CT | Path on UNAS → in container | Backed up? | Notes | |---|---|---|---|---| -| arr-stack | `/opt/stacks/arr-stack/` | Configs + media | ~55 GB | `arr-stack/{config,audiobooks,ebooks,podcasts,torrents}` | -| qdrant | `/opt/stacks/qdrant/qdrant_storage` | Vector DB | ~7 MB | `qdrant/` | -| dozzle | `/opt/stacks/dozzle/dozzle_data` | Log viewer state | tiny | *(ephemeral, optional)* | -| homepage | `/opt/stacks/homepage/{config,icons}` | Dashboard config | ~164 KB | *(keep in stack dir)* | +| Immich (uploads, thumbs, derived) | 104 | `services/immich/{encoded-video,profile,thumbs}` + `media/images` + `backup/immich` | Immich own pg_dump → `media/images/db-dumps`; no off-host copy | | +| Paperless-ngx (documents) | 104 | `media/documents/public/paperless-ngx/{consume,export,library}` | none | | +| Paperless-AI | 104 | `services/paperless-ai` → `/app/data` | none | | +| Vaultwarden (attachments + SQLite DB) | 104 | `services/vaultwarden` → `/data` | none | tier-1 DB on NFS — risky, plan to move local | +| Traccar (logs, config) | 104 | `services/traccar/{logs,traccar.xml}` | none | data dir reverted to local `/opt/stacks/traccar/data` | +| Memos | 104 | `services/memos` → `/var/opt/memos` | none | | +| Arr-stack (Audiobookshelf, Prowlarr, RDTClient, Shelfarr) + media | 104 | `services/arr-stack/*` + `media/{audiobooks,ebooks,podcasts,Torrents}` | none | already migrated (older doc claimed "still local") | +| Gluetun (VPN) | 104 | `services/gluetun/data` → `/gluetun` | none | | +| Gitea | 111 | `services/gitea` → `/data` | none | new 2026-05-20 | +| Coder workspace home dirs | 111 | `services/coder//` → `/home/` | none | new 2026-05-20 | +| Backrest's jottacloud-mirrored data | 103 | `media/data-dir` only | **rclone → jottacloud (Backrest)** | only path with off-host backup | -### 🗄️ Databases Requiring Migration +### Local zfs — `/opt/stacks/...` -| Service | Engine | Current Path | Target | Migration Method | Status | -|---|---|---|---|---|---|---| -| immich | PostgreSQL | `/opt/stacks/immich/postgres` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending | -| litellm | PostgreSQL | `/opt/stacks/ai/postgres_data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending | -| lobehub | PostgreSQL | `/opt/stacks/ai/lobehub/data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending | -| daytona | PostgreSQL | volume `daytona-minimal_db_data` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending | -| paperless-ngx | PostgreSQL | volume `paperless-ngx_pgdata` | → shared-postgres | `pg_dump` → `pg_restore` | ⏳ Pending | -| memos | SQLite | `/opt/stacks/memos/data` | `/mnt/pve/unas/services/memos` | `cp` + `chown` | ✅ Done | -| arcane | SQLite | volume `arcane_arcane-data` | `/mnt/pve/unas/services/arcane` | `cp` + `chown` | ✅ Done | -| n8n | SQLite | volume `n8n_n8n_storage` | `/mnt/pve/unas/services/n8n` | `cp` + `chown` | ✅ Done | -| pocketid | SQLite | `pocketid/` | *(already on UNAS)* | — | ✅ Done | -| vaultwarden | SQLite | `vaultwarden/` | *(already on UNAS)* | — | ✅ Done | -| karakeep | SQLite + Meilisearch | `karakeep/{data,meilisearch}` | *(already on UNAS)* | — | ✅ Done | +Tier-1 state and anything that should NOT be NFS-backed. -### 🧠 ML Models & Caches (Regenerable — Can Stay Local) - -| Resource | Path | Decision | -|---|---|---| -| Immich ML models | volume `immich_model-cache` | Cache — stay | -| Paperless Redis | volume `paperless-ngx_redisdata` | Ephemeral — stay | -| AI stack Redis | volume `nuc-ai-core_redis_data` | Ephemeral — stay | -| SearXNG search cache | volume (hash) at `/var/lib/docker/volumes/bb5bb...` | Ephemeral — stay | -| Daytona runner Docker | volume `daytona-minimal` runner data | Can stay | - -## Media Content (Already on UNAS) - -| Mount | Content | -|---|---| -| `/mnt/pve/unas/media/images/` | Immich photo library | -| `/mnt/pve/unas/media/documents/public/paperless-ngx/` | Paperless documents (consume, export, library) | -| `/mnt/pve/unas/media/audiobooks/` | Audiobookshelf media | -| `/mnt/pve/unas/media/ebooks/` | Audiobookshelf ebooks | -| `/mnt/pve/unas/media/podcasts/` | Audiobookshelf podcasts | -| `/mnt/pve/unas/backup/immich/` | Immich DB backups | - -## Prospective UNAS Layout - -``` -/mnt/pve/unas/ -├── services/ -│ ├── gluetun/ -│ │ └── data/ ← VPN config -│ ├── immich/ -│ │ ├── db/ ← PostgreSQL (to migrate) -│ │ ├── encoded-video/ ← Already migrated -│ │ ├── profile/ ← Already migrated -│ │ └── thumbs/ ← Already migrated -│ ├── karakeep/ -│ │ ├── data/ ← Already migrated -│ │ └── meilisearch/ ← Already migrated -│ ├── ntfy/ -│ │ ├── cache/ ← Already migrated -│ │ └── config/ ← Already migrated -│ ├── paperless-ai/ ← Already migrated -│ ├── paperless-ngx/ -│ │ └── pgdata/ ← PostgreSQL (to migrate) -│ ├── pocketid/ ← Already migrated -│ ├── stremio/ ← Already migrated -│ ├── traccar/ -│ │ ├── data/ ← Already migrated -│ │ ├── logs/ ← Already migrated -│ │ └── traccar.xml ← Already migrated -│ ├── vaultwarden/ ← Already migrated -│ ├── memos/ ← SQLite ✅ Done -│ ├── qdrant/ ← Vector DB (to migrate) -│ ├── arcane/ ← SQLite ✅ Done -│ ├── n8n/ ← SQLite ✅ Done -│ ├── shared-db/ -│ │ ├── postgres/ ← ⚠️ Local volume, NOT on UNAS -│ │ └── garage/ -│ │ ├── data/ ← S3 data (on UNAS) -│ │ └── meta/ ← S3 metadata (on UNAS) -│ ├── arr-stack/ -│ │ ├── config/ ← Bind mounts (to migrate) -│ │ ├── audiobooks/ ← Media (to migrate) -│ │ ├── ebooks/ ← Media (to migrate) -│ │ ├── podcasts/ ← Media (to migrate) -│ │ └── torrents/ ← Downloads (to migrate) -│ -├── media/ -│ ├── images/ ← Immich photo library -│ └── documents/ -│ └── public/ -│ └── paperless-ngx/ -│ ├── consume/ -│ ├── export/ -│ └── library/ -│ -├── backups/ -│ └── immich/ ← Immich DB backups -│ -└── (potential future: torrents/, cache/) -``` - -## Garage S3 — `/opt/stacks/shared-db/` - -S3-compatible object storage using [Garage](https://garagehq.deuxfleurs.fr/) v1.0.0. -Replaces RustFS from the LobeHub AI stack. Single-node, replication_factor=1. - -| Detail | Value | -|---|---| -| Container | `garage` (dxflrs/garage:v1.0.0) | -| Data | `/mnt/pve/unas/services/shared-db/garage/{data,meta}` | -| S3 API | `garage:3900` (internal, shared_backend) | -| Admin API | `127.0.0.1:3901` (localhost only) | -| Regional | No auth (internal network) | -| Bucket | `lobe-files` | -| Key ID | `GK55210bd6b56d92891db26fac` | -| Secret Key | `d0af9e4eb69d7773ffa17950feee27925dbc70eb3efd4ee23a3e1324672c869a` | - -## Shared PostgreSQL — `/opt/stacks/shared-db/` - -Centralized PostgreSQL 16 instance for all non-immich services. - -| Detail | Value | -|---|---| -| Container | `shared-postgres` (postgres:16) | -| Data | `shared-db_shared-pgdata` (local Docker volume, NOT NFS) | -| Networks | `shared_backend` | -| Auth | `postgres` superuser with password from `.env` | - -### Databases & Users (pre-created on init) - -| Database | Owner | Password | Used By | -|---|---|---|---| -| `paperless` | `paperless` | `paperless` | paperless-ngx | -| `daytona` | `user` | `pass` | daytona | -| `litellm` | `litellm` | `litellm_password_here` | LiteLLM | -| `lobechat` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub | -| `paradedb` | `lobe` | `uWNZugjBqixf8dxC` | LobeHub | - -## Key Architectural Decision: PostgreSQL Stays Local - -**Problem:** PostgreSQL checks data directory ownership via `stat(2)` and refuses to start -if the process UID doesn't match the file owner UID. NFS with `all_squash` maps all UIDs -to `65534` (nobody), breaking this check. `chown` also fails on NFS. - -**Solution:** All PostgreSQL instances use local Docker volumes on NVMe: - -``` -Local NVMe (/var/lib/docker/volumes/) UNAS NFS (/mnt/pve/unas/) -├── shared-postgres data ├── pg_dump backups ← scheduled -├── immich_postgres data ├── SQLite DBs (memos, arcane, n8n) -├── paperless-ngx pgdata ├── Garage S3 data -├── daytona-minimal db_data ├── Media files (immich, paperless) -├── litellm-db / ai postgres_data ├── Configs / service data -├── lobehub postgres data -└── all Redis / cache volumes -``` - -**Backup strategy:** `pg_dump` each database to UNAS on a schedule (e.g. daily cron). - -## Migration Priority - -| Priority | Service | Effort | Risk | Status | Reason | -|---|---|---|---|---|---| -| P0 | memos | Minimal (SQLite copy) | Low | ✅ Done | Simple file move | -| P0 | n8n | Minimal (SQLite copy) | Low | ✅ Done | Simple file move | -| P0 | arcane | Minimal (SQLite copy) | Low | ✅ Done | Simple file move | -| P0 | qdrant | Minimal (file copy) | Low | ✅ Done | Fresh start on UNAS | -| P1 | daytona → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running | -| P1 | litellm → shared-postgres | Medium (pg_dump) | Low | ✅ Done | pg_dump → restore → running | -| P1 | lobehub → shared-postgres | Medium (pg_dump) | Low | ⛔ Skipped | Uses ParadeDB (pg17+extensions), incompatible with shared-postgres (pg16) | -| P1 | paperless-ngx → shared-postgres | Medium (pg_dump) | Medium | ✅ Done | pg_dump → restore → running | -| P1 | immich → shared-postgres | Medium (pg_dump) | Medium | ⏳ | Custom pgvector image | -| P2 | arr-stack | Large (55 GB copy) | Low | ⏳ | Simple copy, time-consuming | - -## Mail Archive (Potential Savings) - -Unattached anonymous volumes from old/stopped containers: - -| Volume | Size | Likely From | Action | -|---|---|---|---| -| `0b6d07ff...` | ? | old daytona runner | prune | -| (100+ hex-named volumes) | ? | ephemeral build cache | prune | - -Recommend: `docker system df` then `docker volume prune` to clean. - ---- - -## Database Consolidation Plan - -### Current: 10 Database Instances - -| # | Container | Type | Image | Used By | Networks | -|---|---|---|---|---|---| -| 1 | `immich_postgres` | PostgreSQL | `immich-app/postgres:14-vectorchord` (pgvector) | immich | `immich_default` | -| 2 | `paperless-ngx-db-1` | PostgreSQL | `postgres:16` | paperless-ngx | `paperless-ngx_default` | -| 3 | `daytona-minimal-db-1` | PostgreSQL | `postgres:16-alpine` | daytona | `daytona-minimal_daytona-network` | -| 4 | `nuc-ai-core-litellm-db-1` | PostgreSQL | `postgres:16-alpine` | litellm | `ai-internal` | -| 5 | `lobe-postgres` | PostgreSQL | `postgres:16-alpine` | lobehub (lobechat, paradedb) | `ai-internal` | -| 6 | `paperless-ngx-broker-1` | Redis 7 | `redis:7` | paperless broker | `paperless-ngx_default` | -| 7 | `immich_redis` | Valkey 9 | `valkey/valkey:9` | immich cache | `immich_default` | -| 8 | `daytona-minimal-redis-1` | Redis Alpine | `redis:alpine` | daytona | `daytona-minimal_daytona-network` | -| 9 | `lobe-redis` | Redis 7 | `redis:7-alpine` | lobehub | `ai-internal` | -| 10 | `redis-searxng` | Redis 7 | `redis:7-alpine` | searxng | `ai-internal` | - -### Target: 6 Database Instances (remove 4) - -| # | Container | Type | DBs Hosted | Saved | +| Service | CT | Local path → in container | Backed up? | Notes | |---|---|---|---|---| -| 1 | `immich_postgres` | PostgreSQL immich-app (pgvector) | immich | 🔒 Must stay separate | -| 2 | **`shared-postgres`** 🆕 | **PostgreSQL 16** | **paperless, daytona, litellm, lobechat, paradedb** | **−3 containers** | -| 3 | `paperless-ngx-broker-1` | Redis 7 | paperless broker | 🔒 Must stay separate | -| 4 | `immich_redis` | Valkey 9 | immich cache | 🔒 Must stay separate | -| 5 | `daytona-minimal-redis-1` | Redis Alpine | daytona | 🔒 Stay separate | -| 6 | `lobe-redis` | Redis 7 | lobehub | 🔒 Stay separate | -| 7 | `redis-searxng` | Redis 7 | searxng | 🔒 Stay separate | -| 7 | ~~`nuc-ai-core-litellm-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ | -| 8 | ~~`lobe-postgres`~~ | 🗑️ removed | → shared-postgres | ✅ | -| 9 | ~~`paperless-ngx-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ | -| 10 | ~~`daytona-minimal-db-1`~~ | 🗑️ removed | → shared-postgres | ✅ | +| Pocket-ID | **110** | `/opt/stacks/pocketid/data` → `/app/data` | none | CT 110 has no NFS mount at all. Earlier doc said UNAS — incorrect. | +| Garage (S3 meta + data) | 104 | `/opt/stacks/shared-db/garage/{data,meta}` → `/var/lib/garage/*` | none | **moved off NFS** 2026-05-19 after WAL-G outage. Stale copy may still exist on UNAS. | +| Shared Postgres (multi-stack) | 104 | docker named volume `shared-db_shared-pgdata` | **WAL-G → local Garage** (configured; outage 2026-05-19 prompted move) | Garage itself is single-disk local — no off-host copy. | +| n8n | 104 | `/opt/stacks/n8n/data` → `/home/node/.n8n` | none | **back to local** after PG migration attempt failed. Stale 408 MB `database.sqlite` left on UNAS (May 15) — clean up. n8n now uses `shared-postgres` as its actual DB. | +| Arcane | 104 | `/opt/stacks/arcane/data` → `/app/data` | none | reverted from UNAS to local 2026-05-19 (undocumented before now) | +| Karakeep + Meilisearch | 104 | `/opt/stacks/karakeep/localdata/{data,meili}` | none | reverted from UNAS to local | +| Immich Postgres (pgvector) | 104 | `/opt/stacks/immich/postgres` → `/var/lib/postgresql/data` | Immich pg_dump → UNAS | tier-1; PG demands local disk | +| Homepage | 104 | `/opt/stacks/homepage/{config,icons}` | none | | +| Dozzle | 104 | `/opt/stacks/dozzle/dozzle_data` | none | | +| LiteLLM config | 104 | `/opt/stacks/ai/litellm-config` | none | | +| SearXNG config | 104 | `/opt/stacks/ai/searxng` | none | | +| Flaresolverr | 104 | `/var/lib/flaresolver` | none | | +| AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup | +| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below | -**Result: 10 → 6 instances, removing 4 containers, saving ~160MB RAM.** +### Docker named volumes -### Why Redis Stays Separate +Local on `/var/lib/docker/volumes/`. Mostly databases and caches. -| Reason | Detail | -|---|---| -| **Valkey ≠ Redis** | Immich uses Valkey 9 which has diverged from Redis 7 | -| **`SELECT` collisions** | Apps may use specific DB numbers that clash | -| **Password mismatch** | Each app has its own `requirepass` / `REDIS_PASSWORD` | -| **Ephemeral = low value** | Caches/brokers rebuild on restart, savings don't justify risk | -| **`FLUSHALL` risk** | One misconfigured app could wipe another's cache | - -### Why PG Consolidation Works - -| Reason | Detail | -|---|---| -| **Same major version** | All target DBs use PostgreSQL 16 | -| **Native compatibility** | No conflicting extensions (only immich has pgvector) | -| **Isolation via databases** | Each service gets its own database + user within shared PG | -| **Simple restore** | `pg_dump` + `pg_restore` into shared instance | -| **Single backup** | One `pg_dumpall` covers all 5 databases | - -### Implementation: Shared PostgreSQL - -Create `/opt/stacks/shared-db/docker-compose.yml`: - -```yaml -services: - postgres: - image: postgres:16 - container_name: shared-postgres - restart: unless-stopped - volumes: - - /mnt/pve/unas/services/shared-db:/var/lib/postgresql/data - environment: - POSTGRES_PASSWORD: ${SHARED_DB_PASSWORD} - ports: - - "127.0.0.1:5432:5432" - deploy: - resources: - limits: - cpus: "2.0" - memory: 1G - reservations: - cpus: "1.0" - memory: 512M - healthcheck: - test: ["CMD-SHELL", "pg_isready -U postgres"] - interval: 10s - timeout: 5s - retries: 5 - -networks: - default: - external: true - name: shared_backend -``` - -### Migration Order (Phase 2) - -``` -Step 0: Create shared-postgres container -Step 1: Dump ALL 4 databases from their current instances -Step 2: Create users + databases on shared-postgres -Step 3: Restore each dump into shared-postgres -Step 4: One by one, update each app's compose to point to shared-postgres -``` - -#### Detailed: Paperless-ngx - -| Change | Before | After | +| Volume | Service / CT | Backed up? | |---|---|---| -| DB service | `db` (in paperless compose) | remove entirely | -| `PAPERLESS_DBHOST` | `db` | `shared-postgres` | -| DB credentials | defaults (paperless/paperless) | keep same | -| Network | `paperless-ngx_default` | `shared_backend` | +| `shared-db_shared-pgdata` | shared-postgres / 104 | WAL-G → local Garage | +| `paperless-ngx_pgdata`, `_redisdata`, `_data` | paperless-ngx / 104 | none | +| `lobe-postgres` + `lobe-redis` | lobehub / 104 | none | +| `nuc-ai-core_rustfs-data` | lobehub stack / 104 | none | +| `pgadmin-data` | pgadmin / 104 | none (regenerable) | +| `immich_model-cache` | immich-ml / 104 | regenerable | +| `coder-db` + `gitea-db` (docker volume `coder-db`) | / 111 | none | +| (orphaned) `daytona-minimal_db_data` + runner anon vol | / 104 | none — clean up post-decommission | -#### Detailed: Daytona +### Nextcloud (CT 105) — the CIFS outlier -| Change | Before | After | -|---|---|---| -| DB service | `db` (in daytona compose) | remove entirely | -| `DB_HOST` | `db` | `shared-postgres` | -| `DB_USERNAME` | `user` | keep same | -| `DB_DATABASE` | `daytona` | keep same | -| Network | `daytona-minimal_daytona-network` | `shared_backend` | +Nextcloud AIO mounts `//192.168.1.31/storage` over **CIFS/SMB 3.1.1**, not NFS like every other CT. Application data path is under that mount. The Postgres + Redis sidecars stay on local docker volumes. -#### Detailed: LiteLLM +## Backup reality -| Change | Before | After | -|---|---|---| -| DB service | `litellm-db` (in ai compose) | remove entirely | -| `DATABASE_URL` | `postgresql://litellm:...@litellm-db:5432/litellm` | `postgresql://litellm:...@shared-postgres:5432/litellm` | -| Network | `ai-internal` | `shared_backend` | +**There is effectively no off-host backup of service data.** The only configured Backrest plan covers `/mnt/pve/unas/media/data-dir → jottacloud` — i.e., Backrest backs up *one specific UNAS path*, not the services that write to UNAS. -#### Detailed: LobeHub +Coverage: -| Change | Before | After | -|---|---|---| -| DB service | `postgresql` (in lobehub.yml) | remove entirely | -| `DATABASE_URL` | `postgresql://postgres:...@postgresql:5432/lobechat` | `postgresql://postgres:...@shared-postgres:5432/lobechat` | -| Lobe also needs `paradedb` DB | created manually on initial setup | created via SQL script | -| Network | `ai-internal` | `shared_backend` | +- Immich pg_dump → UNAS (same filer; survives container loss, not filer loss) +- shared-postgres WAL-G → Garage (same host; survives container loss, not disk loss) +- Everything else: zero -### Total Impact +This is a known gap. Plans: +1. Extend Backrest plans to snapshot tier-1 paths (Pocket-ID sqlite, Vaultwarden data, Gitea repos, Coder workspace homes) to jottacloud nightly. +2. Once the second NVMe lands (see `infra/proxmox-state.md` §6), mirror `rpool` so a single disk death doesn't take everything. -| Metric | Before | After | -|---|---|---| -| PostgreSQL instances | 5 | 2 | -| Redis instances | 5 | 5 (unchanged) | -| Total container count | 10 DBs | 7 DBs (−3) | -| RAM saved (estimated) | — | ~160 MB | -| Complexity | Fragmented | Centralized | -| Backup burden | 5 pg_dumps | 2 pg_dumps | +## Drift cleanup TODO -### Timing Estimates - -| Service | Downtime | Cumulative | -|---|---|---| -| Setup shared-postgres + dump all | 0 (live dump) | 0 | -| daytona → shared | ~2 min | 2 min | -| litellm → shared | ~2 min | 4 min | -| lobehub → shared | ~3 min | 7 min | -| paperless-ngx → shared | ~5 min | 12 min | -| immich DB → UNAS bind mount | ~10 min | 22 min | -| **Total** | **~22 min** | | - -### Notes - -- All compose files are version-controlled in `/opt/stacks/{service}/docker-compose.yml` -- Always test with `docker compose config` before `up -d` -- PostgreSQL stays on local NVMe — backed up via `pg_dump` to UNAS -- Garage S3 replaces RustFS from LobeHub stack (RustFS will be removed when LobeHub is switched) -- LobeHub's `paradedb` extension requires `CREATE EXTENSION IF NOT EXISTS paradedb;` on the shared instance -- All PostgreSQL dbs consolidated into shared-postgres except immich (pgvector requirement) +- [ ] Remove stale `services/n8n/database.sqlite` (408 MB, last write 2026-05-15) from UNAS. +- [ ] Remove stale `services/shared-db/garage/` copy from UNAS if present. +- [ ] Remove orphaned `daytona-minimal_db_data` Docker volume on CT 104. +- [ ] Vaultwarden `data/` on UNAS — plan to migrate to local zfs (latency + tier-1). +- [ ] Add a Backrest plan for Pocket-ID DB and Gitea data → jottacloud.