docs/services/homelab-architecture.md

Nuclide Ecosystem — Homelab Architecture

Living architecture reference for the /opt/stacks homelab. ~66 containers across ~23 compose stacks. Principle: self-host everything, OIDC SSO, *.nuclide.systems via one reverse proxy.

Topology

• NUC 14 Pro (192.168.1.40) — primary Docker host, all /opt/stacks/*. Itself a Proxmox LXC (CTID 104) on node nuc; local data on ZFS rpool/data/subvol-104-disk-0. Mounts the UNAS NFS at /mnt/pve/unas.

• Proxmox VE — host 192.168.1.20:8006, single node nuc (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via root@pam!mcp token (now PVEAuditor, read-only). 9 guests (+ 2 planned) — canonical roster in ct-inventory.md:

  • 100 qemu haos — Home Assistant OS VM (4c/16G) → .60
  • 101 lxc shepard — secondary Docker host (12c/16G/107G) → .49
  • 102 lxc dns — AdGuard Home (network DNS + ad/tracker blocking, 2c/0.5G). UniFi DHCP has dhcpd_dns_enabled: false; clients resolve via the gateway .1, and the UDM forwards DNS upstream to AdGuard (confirmed) — so ad/tracker blocking is network-wide despite DHCP not handing out AdGuard's IP directly.
  • 103 lxc backrest — Restic/Backrest backups (1c/0.5G)
  • 104 lxc docker — the primary /opt/stacks host (16c/48G/200G) → .40
  • 105 lxc nextcloud — Nextcloud (4c/8G/107G) → .41
  • 108 lxc zoraxy — reverse proxy (2c/2G) → .4
  • 110 lxc id — Pocket-ID OIDC IdP (1c/1G/4G) → .5; migrated off 104 on 2026-05-20
  • 111 lxc dev — Coder + Gitea (12c/32G/60G) → .42; new 2026-05-20
  • 109 lxc ops 192.168.1.8 — Prometheus + Grafana + Loki + Alloy + pve-exporter (2026-05-23); future: Arcane, Dozzle, Homarr, Tinyauth
  • 113 lxc db — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → .6; provisioned 2026-05-21
  • Storages: local (dir), local-zfs (zfspool, ~1.9T), unas (nfs, ~20T).

• Ubiquiti UniFi — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series APs. Controller at https://192.168.1.1 (UniFi OS 5.0.16, SSO + MFA enforced). Single site "Default". MCP access: ghcr.io/enuno/unifi-mcp-server (197 tools, Network App API key); Site Manager/cloud tools need UNIFI_SITE_MANAGER_ENABLED, all local API tools work.

  • LAN "Default" — 192.168.1.1/24, corporate, no VLAN. DHCP pool .100–.250 (24 h lease), domain localdomain. Static infra lives .1–.99 (outside the pool). WANs: "Internet 1" (primary), "Secondary" (WAN2, disabled). 47 clients (17 wired, 30 wireless).
  • WiFi — single SSID "nuclide", WPA2 with WPA3-transition, bands 2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. Flat network: servers, IoT, consoles, phones all share one L2 segment.
  • Firewall — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot), almost entirely system-predefined rules. Port forwards: 80/443 → 192.168.1.4 (Zoraxy); 15001 TCP/UDP → 192.168.1.40 (Traccar watch protocol — already at UDM level, no Zoraxy stream proxy needed).
  • ⚠️ Non-UniFi gear = topology blind spots (UniFi can't see/manage):
    • D-Link DGS-1210-28P 28-port PoE switch at .10 (HW F5, fw 6.32.008, S/N TM0I533010201) — unmanaged by UniFi; its ports/ port-clients don't appear in UniFi topology. HTTP-only mgmt (admin/shared tapirnase pw, RSA-login). Audit 2026-05-19 (read-only):
      • SNTP was broken → FIXED 2026-05-19. Real root cause was DNS: servers were 0/1/2.de.pool.ntp.org but the switch has no DNS resolver (DNS_STATE=2) so it couldn't resolve them. (The default gateway was fine — an active static default route 0.0.0.0/0 → .1 exists; the empty Default_Route_Gateway=[] is just the unused interface-level gateway field — misleading.) Clock had been stuck at 31/08/2025 (~8.5 mo). Fix: repointed to the pinned NTP source standard (see below), all by IP, DNS-free. Verified synced 2026-05-19 — clock corrected to live time (CEST/UTC+2).
      • HTTPS/SSL disabled — admin creds cross the flat LAN in clear. Pending (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
      • SNMP agent enabled on the flat LAN (community strings unverified). Pending.
      • Trusted-Host mgmt allowlist disabled — admin UI reachable from any LAN device incl. IoT. Pending: enable allowing 192.168.1.0/24.
      • ⚠️ The switch is a transparent L2 switch — every device on it is fully visible to UniFi as a client (MAC/IP via UDM DHCP/ARP); only the wired port-topology through it is invisible (needs UniFi adoption, impossible here). This is unlike the RE700X repeater.
      • 🔴 The D-Link is the network CORE / single point of failure. FDB-table analysis (2026-05-19): the UDM uplinks on port 26, the entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all VMs/LXCs) is on port 10, and the U7 APs uplink on ports 3 & 16 (their wireless clients appear via those ports). Nearly all wired and wireless traffic transits this unmanaged switch → a D-Link failure = total LAN+WAN outage. Amplifies the mgmt-hardening, firmware-currency, monitoring and segmentation items below.
      • UniFi↔D-Link DHCP/LLDP interop = fully compatible (audited 2026-05-19, read-only):
        • DHCP: BOOTP_Relay_State=0 and DHCP_BOOTP_Local_Relay_ Status=0 → switch inserts no Option 82; UDM DHCP server gets clean untagged requests. (Option82_State=1 is moot — only applies when relay/local-relay is active; both are off.) DHCP Server Screening has no entries → doesn't block the UDM (also = no rogue-DHCP protection: security note, not interop).
        • LLDP: D-Link LLDP enabled, all 28 ports TX+RX (Port_Basic_ Setting mode 3); stats show 25 inserts / 16 ageouts → LLDP frames are exchanged & parsed with UniFi gear (no TLV incompat). "0 neighbors" earlier was just a point-in-time aged-out snapshot. UniFi still won't draw topology for an unadopted switch — cosmetic only.
        • RSTP: both run 802.1w (standard, interoperable). Tuning opportunity (not a fault): set a deliberate root-bridge priority — currently all default 32768 so root = lowest MAC (arbitrary). Make the D-Link (physical core) or UDM root.
        • Flow control: D-Link global FC =2 (off/auto) — matches UniFi's default (802.3x disabled). Compatible.
    • TP-Link RE700X WiFi extender at .187 (MACs …1a:8a:0f/10/11) — bridges/NATs devices behind it. The Klipper 3D printer at .189 sits behind the RE700X, so UniFi only sees the repeater, never .189.
    • DNS: UniFi DHCP has dhcpd_dns_enabled: false (clients get the gateway .1 as resolver, not AdGuard directly) — see DNS note below.

  Infrastructure IPs known from UniFi:

  IP	Name / Hostname	Notes
  .1	UDM Home (UDMA6A8)	Gateway, UCG Fiber, fw 5.0.16
  .4	Zoraxy LXC 108	Reverse proxy
  .20	Proxmox node nuc	PVE 9.1.11
  .40	NUC 14 Pro / Docker LXC 104	Main host, Proxmox OUI
  .41	Nextcloud LXC 105	Named "NextCloud", Proxmox OUI
  .49	Shepard Docker host	Secondary Docker host
  .50	U7-Pro-Wall AP	Hallway/entry
  .51	U7 In-Wall AP	In-wall, UAPA6A5
  .52	U7 Mesh (Schlafzimmer)	Bedroom
  .53	U7 Mesh (Esszimmer)	Dining room
  .60	Home Assistant OS VM	haos CTID 100
  .62	Siemens oven	BSH Hausgeräte, WiFi
  .66	Tibber Pulse	Energy monitor (Espressif)
  .10	DGS-1210-28P	D-Link 28-port PoE switch (NOT UniFi-managed)
  .124	L0018	Wired, unknown device
  .143	—	Sony Interactive Ent. (PlayStation)
  .164	C100_7614A4	Tapo C100 camera (TP-Link)
  .169	awtrix_fcf0bc	AWTRIX LED clock (Espressif)
  .187	RE700X	TP-Link WiFi extender — NATs devices behind it
  .189	—	Klipper 3D printer (behind RE700X, invisible to UniFi)
  .192	REDMI-Note-15-Pro-5G	Xiaomi phone
  .241	VS9-EU-MNA3478A	Dyson purifier/fan

• Zoraxy reverse proxy at 192.168.1.4:8000 — wildcard *.nuclide.systems cert + rules. Source of truth: proxy/zoraxy/routes.json (+ idempotent scripts/zoraxy_sync.py --apply). Cross-host, so no Docker labels.

• Ubiquiti UNAS — NFS server 192.168.1.31:/var/nfs/shared/storage (mounted /mnt/pve/unas, ~19T). Bulk/storage (arr media, qdrant, configs); several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here (see n8n note under Operational rules).

• Auth — PocketID (id.nuclide.systems, OIDC, SQLite) is the universal SSO/IdP for the entire ecosystem — effectively every service authenticates via PocketID OIDC (LiteLLM, Nextcloud, Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20. Open WebUI OIDC not yet wired (as of 2026-05-26).

Docker networks

ai-internal (AI/MCP plane) · shared_backend (cross-stack DB/S3) · plus per-stack: arr-stack_default, immich_default, karakeep_default, homepage_default, vpn_default. The MCP gateway bridges ai-internal + shared_backend.

AI / Agent platform (the core, /opt/stacks/ai)

• Bifrost (ai.nuclide.systems, :14003) — unified LLM gateway + MCP host. Stack ai/bifrost/. Auth via sk-bf- VKs. LLM inference at /v1; MCP at /mcp (29 clients, ~760 tools as of 2026-05-26). Connects directly to upstream LLM providers; no LiteLLM hop in the chain anymore.
• LiteLLM — SUNSET 2026-05-28. Container removed, DB + role dropped on CT 113, config dir renamed litellm-config.DECOMMISSIONED-2026-05-28, Prometheus scrape job deleted. Bifrost now handles all LLM gateway responsibilities.
• Open WebUI (chat.nuclide.systems, :14002) — chat UI. Stack ai/open-webui.yml. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not yet wired as of 2026-05-26).
• syncstack (ai/syncstack.py, cron /etc/cron.d/syncstack 15 min) — model syncer/optimizer ONLY (curation/allowlist → DB, model-health → ntfy). One-shot container.
• mcp-gateway — DECOMMISSIONED 2026-05-26. Was FastAPI/DinD on :8080 at mcp.nuclide.systems. Compose renamed .DECOMMISSIONED-2026-05-26. MCP is now served by Bifrost at https://ai.nuclide.systems/mcp.
• MCP servers (Bifrost-managed, formerly gateway-managed): time, home-assistant, kroki, coder, fetch, sequential-thinking, gotify, memos, docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky, obsidian, immich, n8n, nextcloud, paperless, unifi, upload-artifact, gitea, proxmox, shepard, git, gitlab, youtube-transcript, searxng (30 servers). 29 clients, ~760 tools. See services/mcp-servers.md.
• Coder (dev.nuclide.systems, CT 111) — replaced Daytona as the sandbox / dev-environment runtime. Templates: python-uv (persistent, GPU passthrough, baked LiteLLM env + Claude Code), mcp-sandbox (ephemeral, sci stack pre-baked). See services/dev-environment.md.
• LobeChat — DECOMMISSIONED 2026-05-26. Was at chat.nuclide.systems (:14001). Compose renamed .DECOMMISSIONED-lobehub-2026-05-26.yml. Replaced by Open WebUI.
• ComfyUI (img gen, Intel Arc iGPU), Kroki, searxng (self-hosted meta-search).

Data layer

• shared-postgres (pg16, shared_backend, data on local ZFS) — the primary/standard DB for all deployments. Tenants: paperless, memos (migrated 2026-05-19), n8n (added 2026-05-19). Superuser postgres via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's .env). Tuned 2026-05-19 (3G limit, shared_buffers 768M, max_conn 200). ⚠️ n8n was NOT successfully migrated — n8n's export/import CLI only covers workflows+credentials, not users/settings/SSO, so the PG cutover lost the owner account + OIDC. Reverted to local-disk SQLite (tier-1 still satisfied, like PocketID). Compounding: unpinned n8nio/n8n:latest had drifted 2.7→2.20.11, breaking the custom OIDC hooks.js (hardcoded old /usr/local/lib module paths) → n8n crash-looped. Fixed: image pinned to 2.20.11, hooks.js paths patched for 2.20.11 pnpm layout (/usr/lib/node_modules/n8n/...), OIDC hook re-enabled 2026-05-19 — verified: /auth/oidc/login → 302 to PocketID (client 33135ad4, correct redirect/scope/state/nonce). Lessons: (1) only a FULL pgloader migration of all tables is complete — partial export/import is not; (2) pin critical images — latest drift is a real outage cause. Migration note: services without a native full SQLite→PG export use pgloader data-only of ALL tables into the app-built schema (exclude only the app's own migration-tracking table), app role temp-SUPERUSER for the load (FK/trigger disable) then reverted.
• lobe-postgres (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). lobe-redis — was LiteLLM cache; both LobeChat and LiteLLM decommissioned. Garage S3 (garage:3900, ext s3.nuclide.systems / :10004) — lobe-files, memos, WAL-G PG backups. Qdrant (vector, on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
• Backups: WAL-G v3 → Garage for all 3 PG instances (shared-postgres, lobe-postgres, immich_postgres); daily 3am cron (/etc/cron.d/pg-backup). Incident + fix (2026-05-19): Garage stored meta+data on the UNAS NFS → NFS stalls hung wal-g wal-push → archivers hung since 2026-05-18 14:51 (failed_count=0 = hung not failing), zero base backups, stalled WAL recycled (that window unrecoverable). Resolved: Garage moved to local disk (/opt/stacks/shared-db/garage/{meta,data}); all 3 archivers drained; archive_command hardened to timeout 60 wal-g wal-push %p (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra safety net: local logical dumps in /opt/stacks/backups/shared-pg/. Lesson: Garage metadata is fsync/lock-heavy — never on NFS, same rule as SQLite.

Database design / standards

• Default = Postgres on shared-postgres. Any app needing persistence that supports Postgres gets a dedicated role+database there. SQLite is allowed only when the app has no Postgres support, and then only on local disk — never the UNAS NFS share (broken POSIX/SMB file locking; caused the n8n outage + PocketID latent risk).
• Provisioning pattern: CREATE ROLE <app> LOGIN PASSWORD …; CREATE DATABASE <app> OWNER <app>; → set the app's DB_* env, password lives in that stack's .env (vault migration is the long-term plan).
• Why standardize: unified, durable WAL-G PITR backup — the single biggest reason. One backup story instead of N un-backed-up SQLite files.
• Migration reality: few apps have a native SQLite→PG data port. n8n does (export/import CLI). Most (PocketID, Vaultwarden, traccar, …) do not — switching DB_PROVIDER starts a fresh DB; data port needs pgloader/app-specific tooling. Memos done via pgloader (data-only, app-built schema, role temp-SUPERUSER for the load). When no safe port exists, fall back to local-disk SQLite (PocketID) until a port is built.

NFS / storage strategy (corrected 2026-05-19)

• The Docker host is an unprivileged Proxmox LXC (104) → it cannot do in-container NFS mounts (kernel denies; tested — operation not permitted). That's why Proxmox NFS-mounts at the host and bind-mounts /mnt/pve/unas in. Consequence: per-stack "Docker NFS volumes at v4.1" (the old tier-3 idea) is infeasible here without pct set 104 --features mount=nfs + a full LXC reboot (all stacks down).
• Tier 1 is the rule and does the real work: DBs/metadata/lock- or fsync-heavy stores → local ZFS or shared-postgres, never NFS (any version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
• NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only (current firmware; verified 2026-05-19 on node nuc: showmount -e works but mount :/ -o vers=4.1 fails server-side No such file or directory, i.e. no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1 volumes) are both dead ends — no v4 to upgrade to, no reboot worth doing for it. Real export path is /volume/<uuid>/.srv/.unifi-drive/ storage/.data (Proxmox unas storage uses the /var/nfs/shared/storage alias, works on v3 — leave it).
• Therefore tier-1 is the whole strategy. NFSv3 stays for bulk/ sequential data (fine for that). Optional marginal v3 tuning: nconnect=4 for throughput — not required.
• Tier-1 complete (2026-05-19): all SQLite-on-NFS backlog resolved — ntfy, Karakeep, arcane → local disk; Vaultwarden → shared-postgres (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem); traccar data → local disk. PocketID stays on local SQLite (no native PG migration path, deferred).

Services

Nextcloud (LXC .41), Paperless-ngx (+paperless-ai, tika/gotenberg), Immich (server/ML/redis/postgres/power-tools), Memos, Karakeep (+chrome), Vaultwarden, n8n, Home Assistant (.60, ~2492 entities), ntfy (homelab-ai topic — model-health + agent alerts), traccar (GPS, HTTP :15000 / watch :15001), arr-stack (prowlarr/shelfarr/flaresolverr behind gluetun VPN), streamio, Arcane (Docker mgmt), Dozzle (logs), Homepage (dashboard, 6 groups), Daytona OIDC adapter (Keycloak→PocketID PKCE proxy for the VS Code ext).

Operational rules / conventions

• Separation: syncstack = model curation; bifrost = MCP + LLM gateway (mcp-gateway decommissioned 2026-05-26).
• Self-host first: external/cloud only when no self-hosted form exists.
• Pinned NTP source standard — all network gear/infra should point at the same fixed-IP, DNS-free NTP sources (devices like the D-Link can't resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
  1. 192.53.103.108 — PTB ptbtime1 (DE national time)
  2. 192.53.103.104 — PTB ptbtime2 (DE national time)
  3. 162.159.200.123 — Cloudflare NTP anycast (fallback) Applied to the D-Link switch 2026-05-19. UDM + other infra should converge on the same set (a self-hosted LAN NTP server is a roadmap option, but the standard stays "same pinned sources" everywhere).
• Run Python with uv run (not python3) in /opt/stacks.
• Zoraxy self-service: edit routes.json → scripts/zoraxy_sync.py --apply.
• Secrets in ai/.env (plaintext — env-audit/secret-vault is an open hardening task; Vaultwarden available). Note: a UniFi MFA JWT has leaked into homepage/config/logs/homepage.log — rotate + scrub when hardening. ⚠️ Shared-password reuse: tapirnase is reused as the WiFi PSK, the LiteLLM master key root (sk-tapirnase), and the D-Link switch admin pw — single sniff/leak has broad blast radius. Rotate per-service when hardening (noted 2026-05-19, rotation deferred per user).
• Proxmox/UniFi safety: any write/change to the Proxmox host or Ubiquiti network gear requires explicit confirmation + a stated risk assessment first; read-only queries are fine. High blast radius (foundation layer).
• SQLite-on-NFS is unsafe here: the UNAS NFS share has unreliable file locking (true for SMB/CIFS too). Any SQLite app must use shared-postgres or local disk, never the NFS share. n8n hit this (Database connection timed out) and was migrated to shared-postgres (DB on local ZFS; data dir moved to local /opt/stacks/n8n/data; old NFS database.sqlite kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
• Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.

Ops coverage matrix (verified 2026-05-28)

Capability	Coverage
Log aggregation (Loki @ CT 109)	All 8 hosts shipping via Alloy: CT 101, 102, 103 (newly enabled today), 104, 105, 109, 113, nuc. Last 5m: 3.6k lines from CT 104, 2.1k from CT 113, 100s on quieter hosts.
Metrics (Prometheus @ CT 109)	Scrapes node-exporter on CT 104/109, pve-exporter, traefik, home-assistant prom-endpoint, WAL-G textfile collector
Backups (Backrest @ CT 103 → JottaCloud)	14 services paths + HA tars + Immich tars + Nextcloud Borg + paperless media + Garage S3 + n8n local data. All 16 plan paths verified present. 3 pre-stage hooks (pocketid, garage, n8n). Success/error hooks → Gotify on all 3 plans.
Postgres archiving (WAL-G → Garage S3 → JottaCloud)	CT 113 main pg17 (9 DBs incl. owui/miniflux) ✓, CT 104 immich_postgres ✓ (fixed today). Hourly staleness monitor on CT 103 with SSH+psql probe.
Image-update notifications (Diun @ CT 109)	Watches CT 104 docker (67 images), Mondays 08:00 → Gotify. CT 109/113/101 own containers not watched (future work).
Config-to-git (Phase 3)	Per-host scripts: CT 102 AdGuard, CT 103 Backrest. Central sync-config-repos on PVE 03:15 covers CT 103/104/109/113 stacks (auto-discovers CT 104+109 stacks). HAOS gap: Git Pull addon needs UI install.
Offsite personal data (Phase 1a rclone sync)	Saturdays 01:00 on CT 103. Immich originals → JottaCloud Photos; UNAS personal → JottaCloud UNAS. Last run 2026-05-27 13h32m / 160k files.
Gotify apps	Diun (#1), Backrest (#2), WAL-G monitor (#3), Gitea digest (#4)
systemd timers active	CT 103: walg-staleness (hourly). nuc: gitea-digest (08:00), vm100-watchdog (1m), unas-nfs-watchdog (15m), protect-haos-kvm (5m), homelab-health (1h).
Cron jobs active	nuc 03:15 sync-config-repos. CT 103 Sat 01:00 unas-sync, 02:30 walg-offsite-sync, 03:00 ct103-conf-backup. CT 102 daily adguard-conf-backup.

Recent ops debt cleared (2026-05-28)

• Vaultwarden data migrated CT 104 local → UNAS (was unrecoverable on disk loss)
• Backrest false-positive: CT 103 → CT 113 SSH keys never deployed; fixed + retry added to monitor
• Immich WAL-G push broken for 42h (garage container on wrong Docker network post-incident) — fixed
• Diun, Backrest hooks, WAL-G monitor, Gitea digest deployed
• Alloy enabled on CT 103
• Stale CT 110/111/112 references removed from sync-config-repos

Still open

• HAOS Git Pull addon (UI-only install)
• Vaultwarden Borg passphrase storage (UI-only — Vaultwarden web UI)
• n8n-flows issues #1, #3, #4, #7, #8, #9, #10
• Diun second instance for CT 109/113/101 own containers
• DAVx5 → HonorOS org.tasks battery-whitelist (user phone, not server)
• Shared-password rotation (tapirnase reuse — deferred)

Open / roadmap

mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration + expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers (in progress); UniFi MCP — COMPLETE 2026-05-19 (ghcr.io/enuno/unifi-mcp-server, 197 tools, Network App API key, http transport; local API fully working). Tier-1 SQLite-off-NFS: COMPLETE — all services off NFS for DB/metadata. Note: Traccar watch protocol port 15001 is already forwarded at the UDM level — no Zoraxy stream proxy needed. Plus: env → secret vault. Wire Open WebUI OIDC via Pocket-ID once Bifrost auth is settled.

Management-plane TLS (planned): issue/trust proper certs for admin-UI auth on the Proxmox host and the D-Link DGS-1210 (currently HTTP-only on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed). Brings switch/hypervisor mgmt onto the *.nuclide.systems PKI like the rest.

Network segmentation (planned): the LAN is flat — servers, IoT (Siemens oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (192.168.1.0/24, no VLAN, no L2 isolation). Plan an IoT VLAN (+ matching firewall zone) so untrusted appliances can't reach the server/Proxmox subnet. Complications to design around: the non-UniFi D-Link DGS-1210-28P switch (.10) and TP-Link RE700X extender (.187, NATs the Klipper printer .189) won't honour UniFi VLAN tags natively — segmentation needs a plan for the wired trunk through the D-Link and the repeater's bridge mode.

Observability LXC (planned — scoped from this session's incidents)

Why it's now a priority: the WAL-G archiver was hung silently for ~13 h (failed_count=0, zero base backups) and would never have been noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring must target exactly these silent-failure classes.

• Placement: its own LXC on node nuc, NOT inside LXC 104 — 104 hosts everything, so the monitor must survive/alert when 104 is down.
• Stack: VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager → ntfy homelab-ai (already the alert channel).
• Exporters/probes: node_exporter (per host + key LXCs), postgres_exporter ×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for Zoraxy wildcard), pve-exporter (use the root@pam!mcp PVEAuditor token), and a custom WAL-G/archiver textfile collector: pg_stat_archiver (last_archived age, failed_count), .ready backlog, and wal-g backup-list newest-base age — per PG instance.
• Alerts (priority order, derived from real incidents):
  1. WAL archiver stalled (last_archived age > 15 m) or newest base backup

     26 h, any PG instance.

  2. NFS mount on /mnt/pve/unas unresponsive / high op latency.
  3. Config-drift guard: any *.db/*.sqlite* appears under /mnt/pve/unas (catches a regression of the tier-1 rule).
  4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
  5. local-zfs rpool or NFS pool > 85 %; 6. TLS cert < 14 d.

Topology — at a glance

Physical / CT layout

flowchart TB
  subgraph Host["Proxmox host  ·  192.168.1.20  ·  Intel Core Ultra 7 155H · 64 GiB"]
    direction TB
    haos["VM 100 · haos<br/>(.60)<br/>Home Assistant"]
    shepard["CT 101 · shepard<br/>(.49)<br/>Shepard product stack"]
    dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
    backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
    docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
    nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
    zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
    obs["CT 109 · ops<br/>(.8)<br/>Prometheus · Grafana · Loki · Alloy · pve-exporter"]
    id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
    dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
    db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
  end
  UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3")]
  UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
  Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])

  UDM <--> Host
  UDM <--> Inet
  UNAS <--> shepard
  UNAS <--> backrest
  UNAS <--> docker104
  UNAS <--> nc
  UNAS <--> dev

  classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
  class db planned

Auth plane — Pocket-ID is the universal IdP

Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.

flowchart LR
  user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy<br/>CT 108]
  zx --> coder[Coder · CT 111]
  zx --> gitea[Gitea · CT 111]
  zx --> nc2[Nextcloud · CT 105]
  zx --> immich[Immich · CT 104]
  zx --> owui[Open WebUI · CT 104]
  zx --> n8n[n8n · CT 104]
  zx --> bifrost[Bifrost · CT 104]
  zx --> vw[Vaultwarden · CT 104]
  coder --> pid[(Pocket-ID<br/>CT 110)]
  gitea --> pid
  nc2 --> pid
  immich --> pid
  owui -.->|OIDC not yet wired| pid
  n8n --> pid
  bifrost -.->|VK auth, not OIDC| pid
  vw -.->|via Tinyauth<br/>when CT 109 lands| pid
  classDef pending stroke-dasharray:4 4,color:#888
  class vw pending

MCP plane — Bifrost host, child workers

Bifrost on CT 104 is the MCP host; ~30 child MCP servers run on the same docker network (ai-internal). The old mcp-gateway FastAPI/DinD container was decommissioned 2026-05-26.

flowchart LR
  client[["Claude Code / Cursor / Open WebUI"]]
  gw["Bifrost<br/>(CT 104)<br/>ai.nuclide.systems/mcp"]
  client -- Bearer VK --> gw
  subgraph "CT 104 · ai-internal docker net"
    direction TB
    coderm[coder-mcp]
    immm[mcp-immich]
    n8nm[mcp-n8n]
    fetch[mcp-fetch]
    time[mcp-time]
    cw[mcp-crawl4ai]
    seq[mcp-sequential-thinking]
    ham[home-assistant-mcp]
    kr[kroki-mcp]
    others[...18 more]
  end
  gw --> coderm
  gw --> immm
  gw --> n8nm
  gw --> fetch
  gw --> time
  gw --> cw
  gw --> seq
  gw --> ham
  gw --> kr
  gw --> others
  coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]
  ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]

Data plane — what lives where

flowchart TB
  subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"]
    pid_d[Pocket-ID sqlite · CT 110]
    immich_pg[Immich Postgres · CT 104]
    shared_pg[shared-postgres · CT 104]
    coder_pg[coder-db · CT 111]
    gitea_pg[gitea-db · CT 111]
    garage["Garage S3 · CT 104<br/>(moved off NFS 2026-05-19)"]
    n8n_local["n8n data · CT 104<br/>(reverted from NFS 2026-05-19)"]
  end
  subgraph Bulk["Bulk · UNAS NFSv3"]
    media["Immich media, Paperless docs,<br/>arr-stack media, audiobooks"]
    coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder]
    gitea_data[Gitea repos · /mnt/pve/unas/services/gitea]
    vw_data["Vaultwarden data<br/>(tier-1 leak — plan to move local)"]
  end
  subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"]
    nc_data[Nextcloud user files]
  end
  subgraph Backup["Off-host backup"]
    jottacloud[(jottacloud<br/>via Backrest)]
  end
  Tier1 -. WAL-G .-> garage
  Bulk -. only media/data-dir .-> jottacloud
  classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc
  class vw_data,jottacloud gap

The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See stacks/storage.md for the verified state and the cleanup TODO list.