docs/services/homelab-architecture.md

# Nuclide Ecosystem — Homelab Architecture

Living architecture reference for the `/opt/stacks` homelab. ~66 containers
across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
`*.nuclide.systems` via one reverse proxy.

## Topology
- **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`.
  Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS
  `rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`.
- **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`**
  (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via
  `root@pam!mcp` token (now `PVEAuditor`, read-only). **9 guests (+ 2 planned)** — canonical roster in [`ct-inventory.md`](../ct-inventory.md):
  - `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60`
  - `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49`
  - `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking,
    2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via
    the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard**
    (confirmed) — so ad/tracker blocking is network-wide despite DHCP not
    handing out AdGuard's IP directly.
  - `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
  - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40`
  - `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
  - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
  - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
  - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20
  - `109` lxc **ops** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md`
  - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21
  - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
  APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
  enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server`
  (197 tools, Network App API key); Site Manager/cloud tools need
  `UNIFI_SITE_MANAGER_ENABLED`, all local API tools work.
  - **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool
    `.100–.250` (24 h lease), domain `localdomain`. Static infra lives
    `.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary"
    (WAN2, disabled). 47 clients (17 wired, 30 wireless).
  - **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands
    2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat
    network**: servers, IoT, consoles, phones all share one L2 segment.
  - **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot),
    almost entirely system-predefined rules. Port forwards: **80/443 →
    192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch
    protocol — already at UDM level, no Zoraxy stream proxy needed).
  - ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage):
    - **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw
      **6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/
      port-clients don't appear in UniFi topology. HTTP-only mgmt
      (`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19
      (read-only):**
      - **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**:
        servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS
        resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default
        gateway was *fine* — an active static default route `0.0.0.0/0 →
        .1` exists; the empty `Default_Route_Gateway=[]` is just the unused
        *interface-level* gateway field — misleading.) Clock had been stuck
        at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP
        source standard** (see below), all by IP, DNS-free. **Verified
        synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2).
      - **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear.
        *Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
      - **SNMP agent enabled** on the flat LAN (community strings unverified).
        *Pending.*
      - **Trusted-Host mgmt allowlist disabled** — admin UI reachable from
        any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`.
      - ⚠️ The switch is a **transparent L2 switch** — every device on it
        **is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP);
        only the *wired port-topology through it* is invisible (needs UniFi
        adoption, impossible here). This is unlike the RE700X repeater.
      - 🔴 **The D-Link is the network CORE / single point of failure.**
        FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**,
        the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all
        VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16**
        (their wireless clients appear via those ports). Nearly all wired
        *and* wireless traffic transits this unmanaged switch → a D-Link
        failure = total LAN+WAN outage. Amplifies the mgmt-hardening,
        firmware-currency, monitoring and segmentation items below.
      - **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited
        2026-05-19, read-only):
        - DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_
          Status=0` → switch inserts **no Option 82**; UDM DHCP server
          gets clean untagged requests. (`Option82_State=1` is moot —
          only applies when relay/local-relay is active; both are off.)
          DHCP Server Screening has no entries → doesn't block the UDM
          (also = no rogue-DHCP protection: security note, not interop).
        - LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_
          Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP
          frames *are* exchanged & parsed with UniFi gear (no TLV
          incompat). "0 neighbors" earlier was just a point-in-time
          aged-out snapshot. UniFi still won't *draw* topology for an
          unadopted switch — cosmetic only.
        - RSTP: both run 802.1w (standard, interoperable). Tuning
          opportunity (not a fault): set a deliberate root-bridge
          priority — currently all default 32768 so root = lowest MAC
          (arbitrary). Make the D-Link (physical core) or UDM root.
        - Flow control: D-Link global FC `=2` (off/auto) — matches
          UniFi's default (802.3x disabled). Compatible.
    - **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) —
      bridges/NATs devices behind it. **The Klipper 3D printer at `.189`
      sits behind the RE700X**, so UniFi only sees the repeater, never `.189`.
    - **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the
      gateway `.1` as resolver, not AdGuard directly) — see DNS note below.

  **Infrastructure IPs known from UniFi:**
  | IP | Name / Hostname | Notes |
  |---|---|---|
  | .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 |
  | .4 | Zoraxy LXC 108 | Reverse proxy |
  | .20 | Proxmox node `nuc` | PVE 9.1.11 |
  | .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI |
  | .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI |
  | .49 | Shepard Docker host | Secondary Docker host |
  | .50 | U7-Pro-Wall AP | Hallway/entry |
  | .51 | U7 In-Wall AP | In-wall, UAPA6A5 |
  | .52 | U7 Mesh (Schlafzimmer) | Bedroom |
  | .53 | U7 Mesh (Esszimmer) | Dining room |
  | .60 | Home Assistant OS VM | haos CTID 100 |
  | .62 | Siemens oven | BSH Hausgeräte, WiFi |
  | .66 | Tibber Pulse | Energy monitor (Espressif) |
  | .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) |
  | .124 | L0018 | Wired, unknown device |
  | .143 | — | Sony Interactive Ent. (PlayStation) |
  | .164 | C100_7614A4 | Tapo C100 camera (TP-Link) |
  | .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) |
  | .187 | RE700X | TP-Link WiFi extender — NATs devices behind it |
  | .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) |
  | .192 | REDMI-Note-15-Pro-5G | Xiaomi phone |
  | .241 | VS9-EU-MNA3478A | Dyson purifier/fan |
- **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems`
  cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent
  `scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels.
- **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage`
  (mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs);
  several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here
  (see n8n note under Operational rules).
- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
  **universal SSO/IdP for the entire ecosystem** — effectively every service
  authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud,
  Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity
  source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.

## Docker networks
`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
plus per-stack: `arr-stack_default`,
`immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`.
The MCP gateway bridges `ai-internal` + `shared_backend`.

## AI / Agent platform  (the core, `/opt/stacks/ai`)
- **LiteLLM** (`ai.nuclide.systems`, :14000) — model gateway. ~28 curated
  models (provider allowlist), SAIA-terminal **failovers** + `num_retries`,
  Redis completion cache, Gemini €10/30d `provider_budget_config`. Config:
  `ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`).
- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
  **model syncer/optimizer ONLY** (curation/allowlist → DB, `OPENAI_MODEL_LIST`
  → `.env`, recreate lobe, model-health → ntfy). One-shot container.
- **mcp-gateway / Agent Operator** (`mcp.nuclide.systems`, :8080) — **all
  things MCP**. DinD-spawns MCP servers on `ai-internal`, OAuth-gated streaming
  reverse proxy `/<server>/mcp` + `/group/<g>/mcp`, multi-group, health-watcher,
  per-server tools/list cache. **Deep health-check (2026-05-19):** beyond
  shallow liveness, the watcher drives a real MCP session (`initialize` →
  `tools/list` → optional `health_probe_tool` no-side-effect call) against
  **every** server incl. spawned ones, recording `ok|degraded|down` in
  `/api/servers`. Usage-aware: a server used within 5 min isn't probed
  (traffic = health); idle servers probed every 15 min; probe self-excludes
  via `X-Health-Probe`. Deep failures are status-only (no restart loops);
  shallow-liveness auto-restart stays static-only. First run caught the
  invisible class: `memos` degraded (search_memo → DNS `Errno -2`, the
  ai-internal isolation bug), plus `context7`/`crawl4ai`/`markitdown` down
  and `nextcloud` probe false-positive (probe lacks per-user NC creds —
  candidate for `health_check:false`). **Semantic tool retrieval**: groups with >25
  aggregated tools expose just `find_tools`/`call_tool` meta-tools (LiteLLM
  embedding index, cosine top-k, two-phase schema promotion) instead of
  flooding clients with hundreds of schemas. Group members are proxied via the
  gateway's own path (`_mcp_gw`) so `exact_upstream` servers (home-assistant,
  n8n) aggregate correctly. **MCP→LobeChat registration** (migrated here
  from syncstack), and the **Agent Operator**: cron+event agents
  (`/api/agents/{name}/run` webhook), LLM↔MCP executor, **Agent Creator**
  (AI-drafts agent specs; overridable meta-prompt), prompt export. Config:
  `mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents).
- **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to
  `.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions),
  **coder** (replaced daytona 2026-05-20), fetch, sequential-thinking, ntfy, memos,
  docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui,
  bluesky (brianellin, rate-limit pending). Groups incl. `shepard-mcp`.
  Gaps to close (community MCPs exist): gitea, paperless, karakeep, vaultwarden,
  proxmox-ve, audiobookshelf. See [`services/mcp-gateway.md`](mcp-gateway.md).
- **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox /
  dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough,
  baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack
  pre-baked). See [`services/dev-environment.md`](dev-environment.md).
- **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins +
  agents in `lobechat` DB (lobe-postgres). Agents scriptable via
  `scripts/lobe_agent.py`. Native QStash scheduler NOT functional here →
  scheduling lives in the gateway Agent Operator. **Auth = Better Auth**
  (not NextAuth) SSO via PocketID (`generic-oidc`, client `26f3c26b`, PKCE
  S256). The `state_not_found` callback bug was fixed by setting
  `AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems`; verified 2026-05-19
  (sign-in yields a correct PocketID authorize URL; callback now validates
  with `oAuth_code_missing`, not `state_not_found`). Definitive proof =
  one real browser login. NB: `/oidc/auth` is LobeChat-*as*-OIDC-provider
  (desktop/mobile), distinct from this SSO-client path.
- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
  meta-search).

## Data layer
- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
  **primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
  **memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser
  `postgres` via unix socket; per-app dedicated role+db (role owns its db,
  password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
  shared_buffers 768M, max_conn 200).
  ⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only
  covers workflows+credentials, **not users/settings/SSO**, so the PG
  cutover lost the owner account + OIDC. **Reverted to local-disk SQLite**
  (tier-1 still satisfied, like PocketID). Compounding: unpinned
  `n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC
  `hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped.
  Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11
  pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled
  2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID
  (client `33135ad4`, correct redirect/scope/state/nonce). **Lessons:
  (1) only a FULL pgloader migration of *all*
  tables is complete — partial `export/import` is not; (2) pin critical
  images — `latest` drift is a real outage cause.**
  Migration note: services without a native full SQLite→PG export use
  **pgloader data-only of ALL tables** into the app-built schema (exclude
  only the app's own migration-tracking table), app role temp-SUPERUSER for
  the load (FK/trigger disable) then reverted.
- **lobe-postgres** (pg17/ParadeDB) — LobeChat. **lobe-redis** — LiteLLM cache
  + LobeChat. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
  `:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
  on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
  lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`).
  **Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS
  NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18
  14:51 (`failed_count=0` = hung not failing), zero base backups, stalled
  WAL recycled (that window unrecoverable). **Resolved:** Garage moved to
  local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers
  drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p`
  (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra
  safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`.
  Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule
  as SQLite.

### Database design / standards
- **Default = Postgres on `shared-postgres`.** Any app needing persistence
  that supports Postgres gets a dedicated `role`+`database` there. SQLite is
  allowed **only** when the app has no Postgres support, and then **only on
  local disk** — never the UNAS NFS share (broken POSIX/SMB file locking;
  caused the n8n outage + PocketID latent risk).
- **Provisioning pattern:** `CREATE ROLE <app> LOGIN PASSWORD …; CREATE
  DATABASE <app> OWNER <app>;` → set the app's `DB_*` env, password lives in
  that stack's `.env` (vault migration is the long-term plan).
- **Why standardize:** unified, durable **WAL-G PITR backup** — the single
  biggest reason. One backup story instead of N un-backed-up SQLite files.
- **Migration reality:** few apps have a native SQLite→PG data port. n8n does
  (`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not —
  switching `DB_PROVIDER` starts a *fresh* DB; data port needs
  `pgloader`/app-specific tooling. **Memos** done via pgloader (data-only,
  app-built schema, role temp-SUPERUSER for the load). When no safe port
  exists, fall back to local-disk SQLite (PocketID) until a port is built.

### NFS / storage strategy (corrected 2026-05-19)
- The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do
  in-container NFS mounts** (kernel denies; tested — `operation not
  permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts
  `/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at
  v4.1" (the old tier-3 idea) is **infeasible** here without
  `pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down).
- **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or
  fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any
  version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
- **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current
  firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but
  `mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e.
  no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1
  volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth
  doing for it. Real export path is `/volume/<uuid>/.srv/.unifi-drive/
  storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage`
  alias, works on v3 — leave it).
- **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/
  sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4`
  for throughput — not required.
- **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved —
  ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres**
  (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem);
  traccar data → local disk. PocketID stays on local SQLite (no native
  PG migration path, deferred).

## Services
Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
**Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep**
(+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities),
**ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001),
**arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**,
**Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups),
**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).

## Operational rules / conventions
- **Separation**: `syncstack` = models; `mcp-gateway` = all MCP/agents.
- **Self-host first**: external/cloud only when no self-hosted form exists.
- **Pinned NTP source standard** — all network gear/infra should point at
  the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
  resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
  1. `192.53.103.108` — PTB ptbtime1 (DE national time)
  2. `192.53.103.104` — PTB ptbtime2 (DE national time)
  3. `162.159.200.123` — Cloudflare NTP anycast (fallback)
  Applied to the D-Link switch 2026-05-19. UDM + other infra should converge
  on the same set (a self-hosted LAN NTP server is a roadmap option, but the
  standard stays "same pinned sources" everywhere).
- Run Python with `uv run` (not python3) in `/opt/stacks`.
- Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`.
- Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open
  hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked
  into `homepage/config/logs/homepage.log` — rotate + scrub when hardening.
  ⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the
  LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw —
  single sniff/leak has broad blast radius. Rotate per-service when hardening
  (noted 2026-05-19, rotation deferred per user).
- **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti
  network gear requires explicit confirmation + a stated risk assessment
  first; read-only queries are fine. High blast radius (foundation layer).
- **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file
  locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres`
  or local disk, never the NFS share. **n8n** hit this (`Database connection
  timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
  data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
  kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
- Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename
  once the agent-platform identity firms up.

## Open / roadmap
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
(in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`,
197 tools, Network App API key, `http` transport; local API fully working).
**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep
`plugins`!); env → secret vault.

**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed).
Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest.

**Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens
oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`,
no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so
untrusted appliances can't reach the server/Proxmox subnet. Complications to
design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and
**TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't
honour UniFi VLAN tags natively — segmentation needs a plan for the wired
trunk through the D-Link and the repeater's bridge mode.

### Observability LXC (planned — scoped from this session's incidents)
**Why it's now a priority:** the WAL-G archiver was hung **silently for
~13 h** (`failed_count=0`, zero base backups) and would never have been
noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring
must target *exactly these silent-failure classes*.
- **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104
  hosts everything, so the monitor must survive/alert when 104 is down.
- **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager
  → **ntfy `homelab-ai`** (already the alert channel).
- **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter
  ×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for
  Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor
  token), and a **custom WAL-G/archiver textfile collector**:
  `pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog,
  and `wal-g backup-list` newest-base age — per PG instance.
- **Alerts (priority order, derived from real incidents):**
  1. WAL archiver stalled (last_archived age > 15 m) or newest base backup
     > 26 h, any PG instance.
  2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency.
  3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under
     `/mnt/pve/unas` (catches a regression of the tier-1 rule).
  4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
  5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d.

## Topology — at a glance

### Physical / CT layout

```mermaid
flowchart TB
  subgraph Host["Proxmox host  ·  192.168.1.20  ·  Intel Core Ultra 7 155H · 64 GiB"]
    direction TB
    haos["VM 100 · haos<br/>(.60)<br/>Home Assistant"]
    shepard["CT 101 · shepard<br/>(.49)<br/>Shepard product stack"]
    dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
    backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
    docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
    nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (CIFS to UNAS)"]
    zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
    obs(["CT 109 · ops<br/>(.8) · <i>planned</i><br/>Prom / Graf / Arcane / Homarr / Tinyauth"])
    id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
    dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
    db(["CT 113 · db<br/>(.?) · <i>planned</i><br/>shared Postgres + pgbouncer"])
  end
  UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3 + CIFS")]
  UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
  Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])

  UDM <--> Host
  UDM <--> Inet
  UNAS <--> shepard
  UNAS <--> backrest
  UNAS <--> docker104
  UNAS <--> nc
  UNAS <--> dev

  classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
  class obs,db planned
```

### Auth plane — Pocket-ID is the universal IdP

Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.

```mermaid
flowchart LR
  user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy<br/>CT 108]
  zx --> coder[Coder · CT 111]
  zx --> gitea[Gitea · CT 111]
  zx --> nc2[Nextcloud · CT 105]
  zx --> immich[Immich · CT 104]
  zx --> lobe[LobeChat · CT 104]
  zx --> n8n[n8n · CT 104]
  zx --> mcp[MCP gateway · CT 104]
  zx --> vw[Vaultwarden · CT 104]
  coder --> pid[(Pocket-ID<br/>CT 110)]
  gitea --> pid
  nc2 --> pid
  immich --> pid
  lobe --> pid
  n8n --> pid
  mcp --> pid
  vw -.->|via Tinyauth<br/>when CT 109 lands| pid
  classDef pending stroke-dasharray:4 4,color:#888
  class vw pending
```

### MCP plane — gateway control, child workers

The MCP gateway on CT 104 is the OIDC-gated front; ~23 child MCP servers run on the same docker network (`ai-internal`). When CT 109 "ops" is built, the gateway moves to CT 109 and reaches CT 104's docker daemon via a socket-proxy; the children stay put.

```mermaid
flowchart LR
  client[["Claude Code / Cursor / LobeChat"]]
  gw["MCP gateway<br/>(CT 104 → CT 109 ops when 109 lands)<br/>mcp.nuclide.systems"]
  client -- Bearer / OIDC --> gw
  subgraph "CT 104 · ai-internal docker net"
    direction TB
    coderm[coder-mcp]
    immm[mcp-immich]
    n8nm[mcp-n8n]
    fetch[mcp-fetch]
    time[mcp-time]
    cw[mcp-crawl4ai]
    seq[mcp-sequential-thinking]
    ham[home-assistant-mcp]
    kr[kroki-mcp]
    others[...18 more]
  end
  gw --> coderm
  gw --> immm
  gw --> n8nm
  gw --> fetch
  gw --> time
  gw --> cw
  gw --> seq
  gw --> ham
  gw --> kr
  gw --> others
  coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]
  ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]
```

### Data plane — what lives where

```mermaid
flowchart TB
  subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"]
    pid_d[Pocket-ID sqlite · CT 110]
    immich_pg[Immich Postgres · CT 104]
    shared_pg[shared-postgres · CT 104]
    coder_pg[coder-db · CT 111]
    gitea_pg[gitea-db · CT 111]
    garage["Garage S3 · CT 104<br/>(moved off NFS 2026-05-19)"]
    n8n_local["n8n data · CT 104<br/>(reverted from NFS 2026-05-19)"]
  end
  subgraph Bulk["Bulk · UNAS NFSv3"]
    media["Immich media, Paperless docs,<br/>arr-stack media, audiobooks"]
    coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder]
    gitea_data[Gitea repos · /mnt/pve/unas/services/gitea]
    vw_data["Vaultwarden data<br/>(tier-1 leak — plan to move local)"]
  end
  subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"]
    nc_data[Nextcloud user files]
  end
  subgraph Backup["Off-host backup"]
    jottacloud[(jottacloud<br/>via Backrest)]
  end
  Tier1 -. WAL-G .-> garage
  Bulk -. only media/data-dir .-> jottacloud
  classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc
  class vw_data,jottacloud gap
```

The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See `stacks/storage.md` for the verified state and the cleanup TODO list.