docs/services/homelab-architecture.md

Nuclide Ecosystem — Homelab Architecture

Living architecture reference for the /opt/stacks homelab. ~66 containers across ~23 compose stacks. Principle: self-host everything, OIDC SSO, *.nuclide.systems via one reverse proxy.

Topology

• NUC 14 Pro (192.168.1.40) — primary Docker host, all /opt/stacks/*. Itself a Proxmox LXC (CTID 104) on node nuc; local data on ZFS rpool/data/subvol-104-disk-0. Mounts the UNAS NFS at /mnt/pve/unas.

• Proxmox VE — host 192.168.1.20:8006, single node nuc (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via root@pam!mcp token (now PVEAuditor, read-only). 9 guests (+ 2 planned) — canonical roster in ct-inventory.md:

  • 100 qemu haos — Home Assistant OS VM (4c/16G) → .60
  • 101 lxc shepard — secondary Docker host (12c/16G/107G) → .49
  • 102 lxc dns — AdGuard Home (network DNS + ad/tracker blocking, 2c/0.5G). UniFi DHCP has dhcpd_dns_enabled: false; clients resolve via the gateway .1, and the UDM forwards DNS upstream to AdGuard (confirmed) — so ad/tracker blocking is network-wide despite DHCP not handing out AdGuard's IP directly.
  • 103 lxc backrest — Restic/Backrest backups (1c/0.5G)
  • 104 lxc docker — the primary /opt/stacks host (16c/48G/200G) → .40
  • 105 lxc nextcloud — Nextcloud (4c/8G/107G) → .41
  • 108 lxc zoraxy — reverse proxy (2c/2G) → .4
  • 110 lxc id — Pocket-ID OIDC IdP (1c/1G/4G) → .5; migrated off 104 on 2026-05-20
  • 111 lxc dev — Coder + Gitea (12c/32G/60G) → .42; new 2026-05-20
  • 109 lxc ops 192.168.1.8 — Prometheus + Grafana + node-exporter (2026-05-23); future: Loki, Arcane, Dozzle, Homarr, Tinyauth
  • 113 lxc db — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → .6; provisioned 2026-05-21
  • Storages: local (dir), local-zfs (zfspool, ~1.9T), unas (nfs, ~20T).

• Ubiquiti UniFi — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series APs. Controller at https://192.168.1.1 (UniFi OS 5.0.16, SSO + MFA enforced). Single site "Default". MCP access: ghcr.io/enuno/unifi-mcp-server (197 tools, Network App API key); Site Manager/cloud tools need UNIFI_SITE_MANAGER_ENABLED, all local API tools work.

  • LAN "Default" — 192.168.1.1/24, corporate, no VLAN. DHCP pool .100–.250 (24 h lease), domain localdomain. Static infra lives .1–.99 (outside the pool). WANs: "Internet 1" (primary), "Secondary" (WAN2, disabled). 47 clients (17 wired, 30 wireless).
  • WiFi — single SSID "nuclide", WPA2 with WPA3-transition, bands 2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. Flat network: servers, IoT, consoles, phones all share one L2 segment.
  • Firewall — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot), almost entirely system-predefined rules. Port forwards: 80/443 → 192.168.1.4 (Zoraxy); 15001 TCP/UDP → 192.168.1.40 (Traccar watch protocol — already at UDM level, no Zoraxy stream proxy needed).
  • ⚠️ Non-UniFi gear = topology blind spots (UniFi can't see/manage):
    • D-Link DGS-1210-28P 28-port PoE switch at .10 (HW F5, fw 6.32.008, S/N TM0I533010201) — unmanaged by UniFi; its ports/ port-clients don't appear in UniFi topology. HTTP-only mgmt (admin/shared tapirnase pw, RSA-login). Audit 2026-05-19 (read-only):
      • SNTP was broken → FIXED 2026-05-19. Real root cause was DNS: servers were 0/1/2.de.pool.ntp.org but the switch has no DNS resolver (DNS_STATE=2) so it couldn't resolve them. (The default gateway was fine — an active static default route 0.0.0.0/0 → .1 exists; the empty Default_Route_Gateway=[] is just the unused interface-level gateway field — misleading.) Clock had been stuck at 31/08/2025 (~8.5 mo). Fix: repointed to the pinned NTP source standard (see below), all by IP, DNS-free. Verified synced 2026-05-19 — clock corrected to live time (CEST/UTC+2).
      • HTTPS/SSL disabled — admin creds cross the flat LAN in clear. Pending (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
      • SNMP agent enabled on the flat LAN (community strings unverified). Pending.
      • Trusted-Host mgmt allowlist disabled — admin UI reachable from any LAN device incl. IoT. Pending: enable allowing 192.168.1.0/24.
      • ⚠️ The switch is a transparent L2 switch — every device on it is fully visible to UniFi as a client (MAC/IP via UDM DHCP/ARP); only the wired port-topology through it is invisible (needs UniFi adoption, impossible here). This is unlike the RE700X repeater.
      • 🔴 The D-Link is the network CORE / single point of failure. FDB-table analysis (2026-05-19): the UDM uplinks on port 26, the entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all VMs/LXCs) is on port 10, and the U7 APs uplink on ports 3 & 16 (their wireless clients appear via those ports). Nearly all wired and wireless traffic transits this unmanaged switch → a D-Link failure = total LAN+WAN outage. Amplifies the mgmt-hardening, firmware-currency, monitoring and segmentation items below.
      • UniFi↔D-Link DHCP/LLDP interop = fully compatible (audited 2026-05-19, read-only):
        • DHCP: BOOTP_Relay_State=0 and DHCP_BOOTP_Local_Relay_ Status=0 → switch inserts no Option 82; UDM DHCP server gets clean untagged requests. (Option82_State=1 is moot — only applies when relay/local-relay is active; both are off.) DHCP Server Screening has no entries → doesn't block the UDM (also = no rogue-DHCP protection: security note, not interop).
        • LLDP: D-Link LLDP enabled, all 28 ports TX+RX (Port_Basic_ Setting mode 3); stats show 25 inserts / 16 ageouts → LLDP frames are exchanged & parsed with UniFi gear (no TLV incompat). "0 neighbors" earlier was just a point-in-time aged-out snapshot. UniFi still won't draw topology for an unadopted switch — cosmetic only.
        • RSTP: both run 802.1w (standard, interoperable). Tuning opportunity (not a fault): set a deliberate root-bridge priority — currently all default 32768 so root = lowest MAC (arbitrary). Make the D-Link (physical core) or UDM root.
        • Flow control: D-Link global FC =2 (off/auto) — matches UniFi's default (802.3x disabled). Compatible.
    • TP-Link RE700X WiFi extender at .187 (MACs …1a:8a:0f/10/11) — bridges/NATs devices behind it. The Klipper 3D printer at .189 sits behind the RE700X, so UniFi only sees the repeater, never .189.
    • DNS: UniFi DHCP has dhcpd_dns_enabled: false (clients get the gateway .1 as resolver, not AdGuard directly) — see DNS note below.

  Infrastructure IPs known from UniFi:

  IP	Name / Hostname	Notes
  .1	UDM Home (UDMA6A8)	Gateway, UCG Fiber, fw 5.0.16
  .4	Zoraxy LXC 108	Reverse proxy
  .20	Proxmox node nuc	PVE 9.1.11
  .40	NUC 14 Pro / Docker LXC 104	Main host, Proxmox OUI
  .41	Nextcloud LXC 105	Named "NextCloud", Proxmox OUI
  .49	Shepard Docker host	Secondary Docker host
  .50	U7-Pro-Wall AP	Hallway/entry
  .51	U7 In-Wall AP	In-wall, UAPA6A5
  .52	U7 Mesh (Schlafzimmer)	Bedroom
  .53	U7 Mesh (Esszimmer)	Dining room
  .60	Home Assistant OS VM	haos CTID 100
  .62	Siemens oven	BSH Hausgeräte, WiFi
  .66	Tibber Pulse	Energy monitor (Espressif)
  .10	DGS-1210-28P	D-Link 28-port PoE switch (NOT UniFi-managed)
  .124	L0018	Wired, unknown device
  .143	—	Sony Interactive Ent. (PlayStation)
  .164	C100_7614A4	Tapo C100 camera (TP-Link)
  .169	awtrix_fcf0bc	AWTRIX LED clock (Espressif)
  .187	RE700X	TP-Link WiFi extender — NATs devices behind it
  .189	—	Klipper 3D printer (behind RE700X, invisible to UniFi)
  .192	REDMI-Note-15-Pro-5G	Xiaomi phone
  .241	VS9-EU-MNA3478A	Dyson purifier/fan

• Zoraxy reverse proxy at 192.168.1.4:8000 — wildcard *.nuclide.systems cert + rules. Source of truth: proxy/zoraxy/routes.json (+ idempotent scripts/zoraxy_sync.py --apply). Cross-host, so no Docker labels.

• Ubiquiti UNAS — NFS server 192.168.1.31:/var/nfs/shared/storage (mounted /mnt/pve/unas, ~19T). Bulk/storage (arr media, qdrant, configs); several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here (see n8n note under Operational rules).

• Auth — PocketID (id.nuclide.systems, OIDC, SQLite) is the universal SSO/IdP for the entire ecosystem — effectively every service authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud, Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.

Docker networks

ai-internal (AI/MCP plane) · shared_backend (cross-stack DB/S3) · plus per-stack: arr-stack_default, immich_default, karakeep_default, homepage_default, vpn_default. The MCP gateway bridges ai-internal + shared_backend.

AI / Agent platform (the core, /opt/stacks/ai)

• LiteLLM (ai.nuclide.systems, :14000) — model gateway. ~28 curated models (provider allowlist), SAIA-terminal failovers + num_retries, Redis completion cache, Gemini €10/30d provider_budget_config. Config: ai/litellm-config/config.yaml (+ DB overlay LiteLLM_Config).
• syncstack (ai/syncstack.py, cron /etc/cron.d/syncstack 15 min) — model syncer/optimizer ONLY (curation/allowlist → DB, OPENAI_MODEL_LIST → .env, recreate lobe, model-health → ntfy). One-shot container.
• mcp-gateway / Agent Operator (mcp.nuclide.systems, :8080) — all things MCP. DinD-spawns MCP servers on ai-internal, OAuth-gated streaming reverse proxy /<server>/mcp + /group/<g>/mcp, multi-group, health-watcher, per-server tools/list cache. Deep health-check (2026-05-19): beyond shallow liveness, the watcher drives a real MCP session (initialize → tools/list → optional health_probe_tool no-side-effect call) against every server incl. spawned ones, recording ok|degraded|down in /api/servers. Usage-aware: a server used within 5 min isn't probed (traffic = health); idle servers probed every 15 min; probe self-excludes via X-Health-Probe. Deep failures are status-only (no restart loops); shallow-liveness auto-restart stays static-only. First run caught the invisible class: memos degraded (search_memo → DNS Errno -2, the ai-internal isolation bug), plus context7/crawl4ai/markitdown down and nextcloud probe false-positive (probe lacks per-user NC creds — candidate for health_check:false). Semantic tool retrieval: groups with >25 aggregated tools expose just find_tools/call_tool meta-tools (LiteLLM embedding index, cosine top-k, two-phase schema promotion) instead of flooding clients with hundreds of schemas. Group members are proxied via the gateway's own path (_mcp_gw) so exact_upstream servers (home-assistant, n8n) aggregate correctly. MCP→LobeChat registration (migrated here from syncstack), and the Agent Operator: cron+event agents (/api/agents/{name}/run webhook), LLM↔MCP executor, Agent Creator (AI-drafts agent specs; overridable meta-prompt), prompt export. Config: mcp-gateway/config.json (servers), agents.json + prompts/ (agents).
• MCP servers (gateway-managed): time, home-assistant (exact_upstream to .60 ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions), coder (replaced daytona 2026-05-20), fetch, sequential-thinking, ntfy, memos, docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky (brianellin, rate-limit pending). Groups incl. shepard-mcp. Gaps to close (community MCPs exist): gitea, paperless, karakeep, vaultwarden, proxmox-ve, audiobookshelf. See services/mcp-gateway.md.
• Coder (dev.nuclide.systems, CT 111) — replaced Daytona as the sandbox / dev-environment runtime. Templates: python-uv (persistent, GPU passthrough, baked LiteLLM env + Claude Code), mcp-sandbox (ephemeral, sci stack pre-baked). See services/dev-environment.md.
• LobeChat (chat.nuclide.systems, :14001) — chat UI; MCP plugins + agents in lobechat DB (lobe-postgres). Agents scriptable via scripts/lobe_agent.py. Native QStash scheduler NOT functional here → scheduling lives in the gateway Agent Operator. Auth = Better Auth (not NextAuth) SSO via PocketID (generic-oidc, client 26f3c26b, PKCE S256). The state_not_found callback bug was fixed by setting AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems; verified 2026-05-19 (sign-in yields a correct PocketID authorize URL; callback now validates with oAuth_code_missing, not state_not_found). Definitive proof = one real browser login. NB: /oidc/auth is LobeChat-as-OIDC-provider (desktop/mobile), distinct from this SSO-client path.
• ComfyUI (img gen, Intel Arc iGPU), Kroki, searxng (self-hosted meta-search).

Data layer

• shared-postgres (pg16, shared_backend, data on local ZFS) — the primary/standard DB for all deployments. Tenants: LiteLLM, paperless, memos (migrated 2026-05-19), n8n (added 2026-05-19). Superuser postgres via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's .env). Tuned 2026-05-19 (3G limit, shared_buffers 768M, max_conn 200). ⚠️ n8n was NOT successfully migrated — n8n's export/import CLI only covers workflows+credentials, not users/settings/SSO, so the PG cutover lost the owner account + OIDC. Reverted to local-disk SQLite (tier-1 still satisfied, like PocketID). Compounding: unpinned n8nio/n8n:latest had drifted 2.7→2.20.11, breaking the custom OIDC hooks.js (hardcoded old /usr/local/lib module paths) → n8n crash-looped. Fixed: image pinned to 2.20.11, hooks.js paths patched for 2.20.11 pnpm layout (/usr/lib/node_modules/n8n/...), OIDC hook re-enabled 2026-05-19 — verified: /auth/oidc/login → 302 to PocketID (client 33135ad4, correct redirect/scope/state/nonce). Lessons: (1) only a FULL pgloader migration of all tables is complete — partial export/import is not; (2) pin critical images — latest drift is a real outage cause. Migration note: services without a native full SQLite→PG export use pgloader data-only of ALL tables into the app-built schema (exclude only the app's own migration-tracking table), app role temp-SUPERUSER for the load (FK/trigger disable) then reverted.
• lobe-postgres (pg17/ParadeDB) — LobeChat. lobe-redis — LiteLLM cache
  • LobeChat. Garage S3 (garage:3900, ext s3.nuclide.systems / :10004) — lobe-files, memos, WAL-G PG backups. Qdrant (vector, on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
• Backups: WAL-G v3 → Garage for all 3 PG instances (shared-postgres, lobe-postgres, immich_postgres); daily 3am cron (/etc/cron.d/pg-backup). Incident + fix (2026-05-19): Garage stored meta+data on the UNAS NFS → NFS stalls hung wal-g wal-push → archivers hung since 2026-05-18 14:51 (failed_count=0 = hung not failing), zero base backups, stalled WAL recycled (that window unrecoverable). Resolved: Garage moved to local disk (/opt/stacks/shared-db/garage/{meta,data}); all 3 archivers drained; archive_command hardened to timeout 60 wal-g wal-push %p (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra safety net: local logical dumps in /opt/stacks/backups/shared-pg/. Lesson: Garage metadata is fsync/lock-heavy — never on NFS, same rule as SQLite.

Database design / standards

• Default = Postgres on shared-postgres. Any app needing persistence that supports Postgres gets a dedicated role+database there. SQLite is allowed only when the app has no Postgres support, and then only on local disk — never the UNAS NFS share (broken POSIX/SMB file locking; caused the n8n outage + PocketID latent risk).
• Provisioning pattern: CREATE ROLE <app> LOGIN PASSWORD …; CREATE DATABASE <app> OWNER <app>; → set the app's DB_* env, password lives in that stack's .env (vault migration is the long-term plan).
• Why standardize: unified, durable WAL-G PITR backup — the single biggest reason. One backup story instead of N un-backed-up SQLite files.
• Migration reality: few apps have a native SQLite→PG data port. n8n does (export/import CLI). Most (PocketID, Vaultwarden, traccar, …) do not — switching DB_PROVIDER starts a fresh DB; data port needs pgloader/app-specific tooling. Memos done via pgloader (data-only, app-built schema, role temp-SUPERUSER for the load). When no safe port exists, fall back to local-disk SQLite (PocketID) until a port is built.

NFS / storage strategy (corrected 2026-05-19)

• The Docker host is an unprivileged Proxmox LXC (104) → it cannot do in-container NFS mounts (kernel denies; tested — operation not permitted). That's why Proxmox NFS-mounts at the host and bind-mounts /mnt/pve/unas in. Consequence: per-stack "Docker NFS volumes at v4.1" (the old tier-3 idea) is infeasible here without pct set 104 --features mount=nfs + a full LXC reboot (all stacks down).
• Tier 1 is the rule and does the real work: DBs/metadata/lock- or fsync-heavy stores → local ZFS or shared-postgres, never NFS (any version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
• NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only (current firmware; verified 2026-05-19 on node nuc: showmount -e works but mount :/ -o vers=4.1 fails server-side No such file or directory, i.e. no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1 volumes) are both dead ends — no v4 to upgrade to, no reboot worth doing for it. Real export path is /volume/<uuid>/.srv/.unifi-drive/ storage/.data (Proxmox unas storage uses the /var/nfs/shared/storage alias, works on v3 — leave it).
• Therefore tier-1 is the whole strategy. NFSv3 stays for bulk/ sequential data (fine for that). Optional marginal v3 tuning: nconnect=4 for throughput — not required.
• Tier-1 complete (2026-05-19): all SQLite-on-NFS backlog resolved — ntfy, Karakeep, arcane → local disk; Vaultwarden → shared-postgres (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem); traccar data → local disk. PocketID stays on local SQLite (no native PG migration path, deferred).

Services

Nextcloud (LXC .41), Paperless-ngx (+paperless-ai, tika/gotenberg), Immich (server/ML/redis/postgres/power-tools), Memos, Karakeep (+chrome), Vaultwarden, n8n, Home Assistant (.60, ~2492 entities), ntfy (homelab-ai topic — model-health + agent alerts), traccar (GPS, HTTP :15000 / watch :15001), arr-stack (prowlarr/shelfarr/flaresolverr behind gluetun VPN), streamio, Arcane (Docker mgmt), Dozzle (logs), Homepage (dashboard, 6 groups), Daytona OIDC adapter (Keycloak→PocketID PKCE proxy for the VS Code ext).

Operational rules / conventions

• Separation: syncstack = models; mcp-gateway = all MCP/agents.
• Self-host first: external/cloud only when no self-hosted form exists.
• Pinned NTP source standard — all network gear/infra should point at the same fixed-IP, DNS-free NTP sources (devices like the D-Link can't resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
  1. 192.53.103.108 — PTB ptbtime1 (DE national time)
  2. 192.53.103.104 — PTB ptbtime2 (DE national time)
  3. 162.159.200.123 — Cloudflare NTP anycast (fallback) Applied to the D-Link switch 2026-05-19. UDM + other infra should converge on the same set (a self-hosted LAN NTP server is a roadmap option, but the standard stays "same pinned sources" everywhere).
• Run Python with uv run (not python3) in /opt/stacks.
• Zoraxy self-service: edit routes.json → scripts/zoraxy_sync.py --apply.
• Secrets in ai/.env (plaintext — env-audit/secret-vault is an open hardening task; Vaultwarden available). Note: a UniFi MFA JWT has leaked into homepage/config/logs/homepage.log — rotate + scrub when hardening. ⚠️ Shared-password reuse: tapirnase is reused as the WiFi PSK, the LiteLLM master key root (sk-tapirnase), and the D-Link switch admin pw — single sniff/leak has broad blast radius. Rotate per-service when hardening (noted 2026-05-19, rotation deferred per user).
• Proxmox/UniFi safety: any write/change to the Proxmox host or Ubiquiti network gear requires explicit confirmation + a stated risk assessment first; read-only queries are fine. High blast radius (foundation layer).
• SQLite-on-NFS is unsafe here: the UNAS NFS share has unreliable file locking (true for SMB/CIFS too). Any SQLite app must use shared-postgres or local disk, never the NFS share. n8n hit this (Database connection timed out) and was migrated to shared-postgres (DB on local ZFS; data dir moved to local /opt/stacks/n8n/data; old NFS database.sqlite kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
• Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename once the agent-platform identity firms up.

Open / roadmap

mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration + expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers (in progress); UniFi MCP — COMPLETE 2026-05-19 (ghcr.io/enuno/unifi-mcp-server, 197 tools, Network App API key, http transport; local API fully working). Tier-1 SQLite-off-NFS: COMPLETE — all services off NFS for DB/metadata. Note: Traccar watch protocol port 15001 is already forwarded at the UDM level — no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep plugins!); env → secret vault.

Management-plane TLS (planned): issue/trust proper certs for admin-UI auth on the Proxmox host and the D-Link DGS-1210 (currently HTTP-only on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed). Brings switch/hypervisor mgmt onto the *.nuclide.systems PKI like the rest.

Network segmentation (planned): the LAN is flat — servers, IoT (Siemens oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (192.168.1.0/24, no VLAN, no L2 isolation). Plan an IoT VLAN (+ matching firewall zone) so untrusted appliances can't reach the server/Proxmox subnet. Complications to design around: the non-UniFi D-Link DGS-1210-28P switch (.10) and TP-Link RE700X extender (.187, NATs the Klipper printer .189) won't honour UniFi VLAN tags natively — segmentation needs a plan for the wired trunk through the D-Link and the repeater's bridge mode.

Observability LXC (planned — scoped from this session's incidents)

Why it's now a priority: the WAL-G archiver was hung silently for ~13 h (failed_count=0, zero base backups) and would never have been noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring must target exactly these silent-failure classes.

• Placement: its own LXC on node nuc, NOT inside LXC 104 — 104 hosts everything, so the monitor must survive/alert when 104 is down.
• Stack: VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager → ntfy homelab-ai (already the alert channel).
• Exporters/probes: node_exporter (per host + key LXCs), postgres_exporter ×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for Zoraxy wildcard), pve-exporter (use the root@pam!mcp PVEAuditor token), and a custom WAL-G/archiver textfile collector: pg_stat_archiver (last_archived age, failed_count), .ready backlog, and wal-g backup-list newest-base age — per PG instance.
• Alerts (priority order, derived from real incidents):
  1. WAL archiver stalled (last_archived age > 15 m) or newest base backup

     26 h, any PG instance.

  2. NFS mount on /mnt/pve/unas unresponsive / high op latency.
  3. Config-drift guard: any *.db/*.sqlite* appears under /mnt/pve/unas (catches a regression of the tier-1 rule).
  4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
  5. local-zfs rpool or NFS pool > 85 %; 6. TLS cert < 14 d.

Topology — at a glance

Physical / CT layout

flowchart TB
  subgraph Host["Proxmox host  ·  192.168.1.20  ·  Intel Core Ultra 7 155H · 64 GiB"]
    direction TB
    haos["VM 100 · haos<br/>(.60)<br/>Home Assistant"]
    shepard["CT 101 · shepard<br/>(.49)<br/>Shepard product stack"]
    dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
    backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
    docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
    nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
    zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
    obs["CT 109 · ops<br/>(.8)<br/>Prometheus · Grafana"]
    id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
    dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
    db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
  end
  UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3")]
  UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
  Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])

  UDM <--> Host
  UDM <--> Inet
  UNAS <--> shepard
  UNAS <--> backrest
  UNAS <--> docker104
  UNAS <--> nc
  UNAS <--> dev

  classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
  class db planned

Auth plane — Pocket-ID is the universal IdP

Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.

flowchart LR
  user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy<br/>CT 108]
  zx --> coder[Coder · CT 111]
  zx --> gitea[Gitea · CT 111]
  zx --> nc2[Nextcloud · CT 105]
  zx --> immich[Immich · CT 104]
  zx --> lobe[LobeChat · CT 104]
  zx --> n8n[n8n · CT 104]
  zx --> mcp[MCP gateway · CT 104]
  zx --> vw[Vaultwarden · CT 104]
  coder --> pid[(Pocket-ID<br/>CT 110)]
  gitea --> pid
  nc2 --> pid
  immich --> pid
  lobe --> pid
  n8n --> pid
  mcp --> pid
  vw -.->|via Tinyauth<br/>when CT 109 lands| pid
  classDef pending stroke-dasharray:4 4,color:#888
  class vw pending

MCP plane — gateway control, child workers

The MCP gateway on CT 104 is the OIDC-gated front; ~23 child MCP servers run on the same docker network (ai-internal). When CT 109 "ops" is built, the gateway moves to CT 109 and reaches CT 104's docker daemon via a socket-proxy; the children stay put.

flowchart LR
  client[["Claude Code / Cursor / LobeChat"]]
  gw["MCP gateway<br/>(CT 104 → CT 109 ops when 109 lands)<br/>mcp.nuclide.systems"]
  client -- Bearer / OIDC --> gw
  subgraph "CT 104 · ai-internal docker net"
    direction TB
    coderm[coder-mcp]
    immm[mcp-immich]
    n8nm[mcp-n8n]
    fetch[mcp-fetch]
    time[mcp-time]
    cw[mcp-crawl4ai]
    seq[mcp-sequential-thinking]
    ham[home-assistant-mcp]
    kr[kroki-mcp]
    others[...18 more]
  end
  gw --> coderm
  gw --> immm
  gw --> n8nm
  gw --> fetch
  gw --> time
  gw --> cw
  gw --> seq
  gw --> ham
  gw --> kr
  gw --> others
  coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]
  ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]

Data plane — what lives where

flowchart TB
  subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"]
    pid_d[Pocket-ID sqlite · CT 110]
    immich_pg[Immich Postgres · CT 104]
    shared_pg[shared-postgres · CT 104]
    coder_pg[coder-db · CT 111]
    gitea_pg[gitea-db · CT 111]
    garage["Garage S3 · CT 104<br/>(moved off NFS 2026-05-19)"]
    n8n_local["n8n data · CT 104<br/>(reverted from NFS 2026-05-19)"]
  end
  subgraph Bulk["Bulk · UNAS NFSv3"]
    media["Immich media, Paperless docs,<br/>arr-stack media, audiobooks"]
    coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder]
    gitea_data[Gitea repos · /mnt/pve/unas/services/gitea]
    vw_data["Vaultwarden data<br/>(tier-1 leak — plan to move local)"]
  end
  subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"]
    nc_data[Nextcloud user files]
  end
  subgraph Backup["Off-host backup"]
    jottacloud[(jottacloud<br/>via Backrest)]
  end
  Tier1 -. WAL-G .-> garage
  Bulk -. only media/data-dir .-> jottacloud
  classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc
  class vw_data,jottacloud gap

The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See stacks/storage.md for the verified state and the cleanup TODO list.