# Nuclide Ecosystem — Homelab Architecture Living architecture reference for the `/opt/stacks` homelab. ~66 containers across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, `*.nuclide.systems` via one reverse proxy. ## Topology - **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`. Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS `rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`. - **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`** (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via `root@pam!mcp` token (now `PVEAuditor`, read-only). **9 guests (+ 2 planned)** — canonical roster in [`ct-inventory.md`](../ct-inventory.md): - `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60` - `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49` - `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking, 2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard** (confirmed) — so ad/tracker blocking is network-wide despite DHCP not handing out AdGuard's IP directly. - `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G) - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/48G/200G) → `.40` - `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41` - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4` - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 - `109` lxc **ops** `192.168.1.8` — Prometheus + Grafana + Loki + Alloy + pve-exporter (2026-05-23); future: Arcane, Dozzle, Homarr, Tinyauth - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21 - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server` (197 tools, Network App API key); Site Manager/cloud tools need `UNIFI_SITE_MANAGER_ENABLED`, all local API tools work. - **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool `.100–.250` (24 h lease), domain `localdomain`. Static infra lives `.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary" (WAN2, disabled). 47 clients (17 wired, 30 wireless). - **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands 2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat network**: servers, IoT, consoles, phones all share one L2 segment. - **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot), almost entirely system-predefined rules. Port forwards: **80/443 → 192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch protocol — already at UDM level, no Zoraxy stream proxy needed). - ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage): - **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw **6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/ port-clients don't appear in UniFi topology. HTTP-only mgmt (`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19 (read-only):** - **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**: servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default gateway was *fine* — an active static default route `0.0.0.0/0 → .1` exists; the empty `Default_Route_Gateway=[]` is just the unused *interface-level* gateway field — misleading.) Clock had been stuck at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP source standard** (see below), all by IP, DNS-free. **Verified synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2). - **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear. *Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link). - **SNMP agent enabled** on the flat LAN (community strings unverified). *Pending.* - **Trusted-Host mgmt allowlist disabled** — admin UI reachable from any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`. - ⚠️ The switch is a **transparent L2 switch** — every device on it **is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP); only the *wired port-topology through it* is invisible (needs UniFi adoption, impossible here). This is unlike the RE700X repeater. - 🔴 **The D-Link is the network CORE / single point of failure.** FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**, the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16** (their wireless clients appear via those ports). Nearly all wired *and* wireless traffic transits this unmanaged switch → a D-Link failure = total LAN+WAN outage. Amplifies the mgmt-hardening, firmware-currency, monitoring and segmentation items below. - **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited 2026-05-19, read-only): - DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_ Status=0` → switch inserts **no Option 82**; UDM DHCP server gets clean untagged requests. (`Option82_State=1` is moot — only applies when relay/local-relay is active; both are off.) DHCP Server Screening has no entries → doesn't block the UDM (also = no rogue-DHCP protection: security note, not interop). - LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_ Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP frames *are* exchanged & parsed with UniFi gear (no TLV incompat). "0 neighbors" earlier was just a point-in-time aged-out snapshot. UniFi still won't *draw* topology for an unadopted switch — cosmetic only. - RSTP: both run 802.1w (standard, interoperable). Tuning opportunity (not a fault): set a deliberate root-bridge priority — currently all default 32768 so root = lowest MAC (arbitrary). Make the D-Link (physical core) or UDM root. - Flow control: D-Link global FC `=2` (off/auto) — matches UniFi's default (802.3x disabled). Compatible. - **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) — bridges/NATs devices behind it. **The Klipper 3D printer at `.189` sits behind the RE700X**, so UniFi only sees the repeater, never `.189`. - **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the gateway `.1` as resolver, not AdGuard directly) — see DNS note below. **Infrastructure IPs known from UniFi:** | IP | Name / Hostname | Notes | |---|---|---| | .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 | | .4 | Zoraxy LXC 108 | Reverse proxy | | .20 | Proxmox node `nuc` | PVE 9.1.11 | | .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI | | .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI | | .49 | Shepard Docker host | Secondary Docker host | | .50 | U7-Pro-Wall AP | Hallway/entry | | .51 | U7 In-Wall AP | In-wall, UAPA6A5 | | .52 | U7 Mesh (Schlafzimmer) | Bedroom | | .53 | U7 Mesh (Esszimmer) | Dining room | | .60 | Home Assistant OS VM | haos CTID 100 | | .62 | Siemens oven | BSH Hausgeräte, WiFi | | .66 | Tibber Pulse | Energy monitor (Espressif) | | .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) | | .124 | L0018 | Wired, unknown device | | .143 | — | Sony Interactive Ent. (PlayStation) | | .164 | C100_7614A4 | Tapo C100 camera (TP-Link) | | .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) | | .187 | RE700X | TP-Link WiFi extender — NATs devices behind it | | .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) | | .192 | REDMI-Note-15-Pro-5G | Xiaomi phone | | .241 | VS9-EU-MNA3478A | Dyson purifier/fan | - **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems` cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent `scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels. - **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage` (mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs); several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here (see n8n note under Operational rules). - **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the **universal SSO/IdP for the entire ecosystem** — effectively every service authenticates via PocketID OIDC (LiteLLM, Nextcloud, Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20. Open WebUI OIDC not yet wired (as of 2026-05-26). ## Docker networks `ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) · plus per-stack: `arr-stack_default`, `immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`. The MCP gateway bridges `ai-internal` + `shared_backend`. ## AI / Agent platform (the core, `/opt/stacks/ai`) - **Bifrost** (`ai.nuclide.systems`, :14003) — unified LLM gateway + MCP host. Stack `ai/bifrost/`. Auth via `sk-bf-` VKs. LLM inference at `/v1`; MCP at `/mcp` (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to LiteLLM internally. - **LiteLLM** (:14000, **internal only** — no Zoraxy route) — model gateway, backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal **failovers** + `num_retries`, Redis completion cache, Gemini €10/30d `provider_budget_config`. Config: `ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`). Reachable as `http://litellm:4000` inside Docker. - **Open WebUI** (`chat.nuclide.systems`, :14002) — chat UI. Stack `ai/open-webui.yml`. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not yet wired as of 2026-05-26). - **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) — **model syncer/optimizer ONLY** (curation/allowlist → DB, model-health → ntfy). One-shot container. - ~~**mcp-gateway**~~ — **DECOMMISSIONED 2026-05-26.** Was FastAPI/DinD on `:8080` at `mcp.nuclide.systems`. Compose renamed `.DECOMMISSIONED-2026-05-26`. MCP is now served by Bifrost at `https://ai.nuclide.systems/mcp`. - **MCP servers** (Bifrost-managed, formerly gateway-managed): time, home-assistant, kroki, coder, fetch, sequential-thinking, gotify, memos, docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky, obsidian, immich, n8n, nextcloud, paperless, unifi, upload-artifact, gitea, proxmox, shepard, git, gitlab, youtube-transcript, searxng (30 servers). 29 clients, ~760 tools. See [`services/mcp-servers.md`](mcp-servers.md). - **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox / dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough, baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack pre-baked). See [`services/dev-environment.md`](dev-environment.md). - ~~**LobeChat**~~ — **DECOMMISSIONED 2026-05-26.** Was at `chat.nuclide.systems` (:14001). Compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Replaced by Open WebUI. - **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted meta-search). ## Data layer - **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the **primary/standard DB for all deployments**. Tenants: LiteLLM, paperless, **memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser `postgres` via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's `.env`). Tuned 2026-05-19 (3G limit, shared_buffers 768M, max_conn 200). ⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only covers workflows+credentials, **not users/settings/SSO**, so the PG cutover lost the owner account + OIDC. **Reverted to local-disk SQLite** (tier-1 still satisfied, like PocketID). Compounding: unpinned `n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC `hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped. Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11 pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled 2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID (client `33135ad4`, correct redirect/scope/state/nonce). **Lessons: (1) only a FULL pgloader migration of *all* tables is complete — partial `export/import` is not; (2) pin critical images — `latest` drift is a real outage cause.** Migration note: services without a native full SQLite→PG export use **pgloader data-only of ALL tables** into the app-built schema (exclude only the app's own migration-tracking table), app role temp-SUPERUSER for the load (FK/trigger disable) then reverted. - **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). **lobe-redis** — LiteLLM cache (LobeChat decommissioned). **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` / `:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector, on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal). - **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres, lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`). **Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18 14:51 (`failed_count=0` = hung not failing), zero base backups, stalled WAL recycled (that window unrecoverable). **Resolved:** Garage moved to local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p` (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`. Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule as SQLite. ### Database design / standards - **Default = Postgres on `shared-postgres`.** Any app needing persistence that supports Postgres gets a dedicated `role`+`database` there. SQLite is allowed **only** when the app has no Postgres support, and then **only on local disk** — never the UNAS NFS share (broken POSIX/SMB file locking; caused the n8n outage + PocketID latent risk). - **Provisioning pattern:** `CREATE ROLE LOGIN PASSWORD …; CREATE DATABASE OWNER ;` → set the app's `DB_*` env, password lives in that stack's `.env` (vault migration is the long-term plan). - **Why standardize:** unified, durable **WAL-G PITR backup** — the single biggest reason. One backup story instead of N un-backed-up SQLite files. - **Migration reality:** few apps have a native SQLite→PG data port. n8n does (`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not — switching `DB_PROVIDER` starts a *fresh* DB; data port needs `pgloader`/app-specific tooling. **Memos** done via pgloader (data-only, app-built schema, role temp-SUPERUSER for the load). When no safe port exists, fall back to local-disk SQLite (PocketID) until a port is built. ### NFS / storage strategy (corrected 2026-05-19) - The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do in-container NFS mounts** (kernel denies; tested — `operation not permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts `/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at v4.1" (the old tier-3 idea) is **infeasible** here without `pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down). - **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning). - **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but `mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e. no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1 volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth doing for it. Real export path is `/volume//.srv/.unifi-drive/ storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage` alias, works on v3 — leave it). - **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/ sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4` for throughput — not required. - **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved — ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres** (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem); traccar data → local disk. PocketID stays on local SQLite (no native PG migration path, deferred). ## Services Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg), **Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep** (+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities), **ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001), **arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**, **Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups), **Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext). ## Operational rules / conventions - **Separation**: `syncstack` = model curation; `bifrost` = MCP + LLM gateway (mcp-gateway decommissioned 2026-05-26). - **Self-host first**: external/cloud only when no self-hosted form exists. - **Pinned NTP source standard** — all network gear/infra should point at the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure): 1. `192.53.103.108` — PTB ptbtime1 (DE national time) 2. `192.53.103.104` — PTB ptbtime2 (DE national time) 3. `162.159.200.123` — Cloudflare NTP anycast (fallback) Applied to the D-Link switch 2026-05-19. UDM + other infra should converge on the same set (a self-hosted LAN NTP server is a roadmap option, but the standard stays "same pinned sources" everywhere). - Run Python with `uv run` (not python3) in `/opt/stacks`. - Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`. - Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked into `homepage/config/logs/homepage.log` — rotate + scrub when hardening. ⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw — single sniff/leak has broad blast radius. Rotate per-service when hardening (noted 2026-05-19, rotation deferred per user). - **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti network gear requires explicit confirmation + a stated risk assessment first; read-only queries are fine. High blast radius (foundation layer). - **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres` or local disk, never the NFS share. **n8n** hit this (`Database connection timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS; data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite` kept as rollback). Audit other SQLite stacks for NFS-backed data dirs. - Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26. ## Open / roadmap mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration + expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers (in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`, 197 tools, Network App API key, `http` transport; local API fully working). **Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata. Note: Traccar watch protocol port 15001 is already forwarded at the UDM level — no Zoraxy stream proxy needed. Plus: env → secret vault. Wire Open WebUI OIDC via Pocket-ID once Bifrost auth is settled. **Management-plane TLS (planned):** issue/trust proper certs for admin-UI auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed). Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest. **Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`, no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so untrusted appliances can't reach the server/Proxmox subnet. Complications to design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and **TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't honour UniFi VLAN tags natively — segmentation needs a plan for the wired trunk through the D-Link and the repeater's bridge mode. ### Observability LXC (planned — scoped from this session's incidents) **Why it's now a priority:** the WAL-G archiver was hung **silently for ~13 h** (`failed_count=0`, zero base backups) and would never have been noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring must target *exactly these silent-failure classes*. - **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104 hosts everything, so the monitor must survive/alert when 104 is down. - **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager → **ntfy `homelab-ai`** (already the alert channel). - **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter ×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor token), and a **custom WAL-G/archiver textfile collector**: `pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog, and `wal-g backup-list` newest-base age — per PG instance. - **Alerts (priority order, derived from real incidents):** 1. WAL archiver stalled (last_archived age > 15 m) or newest base backup > 26 h, any PG instance. 2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency. 3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under `/mnt/pve/unas` (catches a regression of the tier-1 rule). 4. Container unhealthy/restart-looping > 5 m (the gluetun pattern). 5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d. ## Topology — at a glance ### Physical / CT layout ```mermaid flowchart TB subgraph Host["Proxmox host · 192.168.1.20 · Intel Core Ultra 7 155H · 64 GiB"] direction TB haos["VM 100 · haos
(.60)
Home Assistant"] shepard["CT 101 · shepard
(.49)
Shepard product stack"] dns["CT 102 · dns
(.2)
AdGuard Home"] backrest["CT 103 · backrest
(.3)
Backrest / restic"] docker104["CT 104 · docker
(.40) · 16c/48G/200G
~65 containers · Intel Arc passthrough"] nc["CT 105 · nextcloud
(.41)
Nextcloud AIO (NFS)"] zoraxy["CT 108 · zoraxy
(.4)
reverse proxy + ACME"] obs["CT 109 · ops
(.8)
Prometheus · Grafana · Loki · Alloy · pve-exporter"] id["CT 110 · id
(.5)
Pocket-ID (moved here 2026-05-20)"] dev["CT 111 · dev
(.42) · 12c/32G/60G
Coder + Gitea + workspaces · Intel Arc"] db["CT 113 · db
(.6) · provisioned 2026-05-21
shared Postgres + pgAdmin"] end UNAS[("UNAS
192.168.1.31
NFSv3")] UDM[["UDM-SE · 192.168.1.1
UniFi gateway · DNS → AdGuard"]] Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams]) UDM <--> Host UDM <--> Inet UNAS <--> shepard UNAS <--> backrest UNAS <--> docker104 UNAS <--> nc UNAS <--> dev classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa class db planned ``` ### Auth plane — Pocket-ID is the universal IdP Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones. ```mermaid flowchart LR user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy
CT 108] zx --> coder[Coder · CT 111] zx --> gitea[Gitea · CT 111] zx --> nc2[Nextcloud · CT 105] zx --> immich[Immich · CT 104] zx --> owui[Open WebUI · CT 104] zx --> n8n[n8n · CT 104] zx --> bifrost[Bifrost · CT 104] zx --> vw[Vaultwarden · CT 104] coder --> pid[(Pocket-ID
CT 110)] gitea --> pid nc2 --> pid immich --> pid owui -.->|OIDC not yet wired| pid n8n --> pid bifrost -.->|VK auth, not OIDC| pid vw -.->|via Tinyauth
when CT 109 lands| pid classDef pending stroke-dasharray:4 4,color:#888 class vw pending ``` ### MCP plane — Bifrost host, child workers Bifrost on CT 104 is the MCP host; ~30 child MCP servers run on the same docker network (`ai-internal`). The old mcp-gateway FastAPI/DinD container was decommissioned 2026-05-26. ```mermaid flowchart LR client[["Claude Code / Cursor / Open WebUI"]] gw["Bifrost
(CT 104)
ai.nuclide.systems/mcp"] client -- Bearer VK --> gw subgraph "CT 104 · ai-internal docker net" direction TB coderm[coder-mcp] immm[mcp-immich] n8nm[mcp-n8n] fetch[mcp-fetch] time[mcp-time] cw[mcp-crawl4ai] seq[mcp-sequential-thinking] ham[home-assistant-mcp] kr[kroki-mcp] others[...18 more] end gw --> coderm gw --> immm gw --> n8nm gw --> fetch gw --> time gw --> cw gw --> seq gw --> ham gw --> kr gw --> others coderm -.spawns/controls.-> CoderWS[(Coder workspaces
CT 111)] ham -.bridges to.-> HAOS[(Home Assistant
VM 100)] ``` ### Data plane — what lives where ```mermaid flowchart TB subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"] pid_d[Pocket-ID sqlite · CT 110] immich_pg[Immich Postgres · CT 104] shared_pg[shared-postgres · CT 104] coder_pg[coder-db · CT 111] gitea_pg[gitea-db · CT 111] garage["Garage S3 · CT 104
(moved off NFS 2026-05-19)"] n8n_local["n8n data · CT 104
(reverted from NFS 2026-05-19)"] end subgraph Bulk["Bulk · UNAS NFSv3"] media["Immich media, Paperless docs,
arr-stack media, audiobooks"] coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder] gitea_data[Gitea repos · /mnt/pve/unas/services/gitea] vw_data["Vaultwarden data
(tier-1 leak — plan to move local)"] end subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"] nc_data[Nextcloud user files] end subgraph Backup["Off-host backup"] jottacloud[(jottacloud
via Backrest)] end Tier1 -. WAL-G .-> garage Bulk -. only media/data-dir .-> jottacloud classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc class vw_data,jottacloud gap ``` The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See `stacks/storage.md` for the verified state and the cleanup TODO list.