reorg: split into infra/ services/ history/ ideas/

services/homelab-architecture.md

@@ -0,0 +1,373 @@
+# Nuclide Ecosystem — Homelab Architecture
+
+Living architecture reference for the `/opt/stacks` homelab. ~66 containers
+across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
+`*.nuclide.systems` via one reverse proxy.
+
+## Topology
+- **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`.
+  Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS
+  `rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`.
+- **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`**
+  (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via
+  `root@pam!mcp` token (now `PVEAuditor`, read-only). **7 guests:**
+  - `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60`
+  - `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49`
+  - `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking,
+    2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via
+    the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard**
+    (confirmed) — so ad/tracker blocking is network-wide despite DHCP not
+    handing out AdGuard's IP directly.
+  - `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
+  - `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40`
+  - `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
+  - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
+  - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
+- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
+  APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
+  enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server`
+  (197 tools, Network App API key); Site Manager/cloud tools need
+  `UNIFI_SITE_MANAGER_ENABLED`, all local API tools work.
+  - **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool
+    `.100–.250` (24 h lease), domain `localdomain`. Static infra lives
+    `.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary"
+    (WAN2, disabled). 47 clients (17 wired, 30 wireless).
+  - **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands
+    2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat
+    network**: servers, IoT, consoles, phones all share one L2 segment.
+  - **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot),
+    almost entirely system-predefined rules. Port forwards: **80/443 →
+    192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch
+    protocol — already at UDM level, no Zoraxy stream proxy needed).
+  - ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage):
+    - **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw
+      **6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/
+      port-clients don't appear in UniFi topology. HTTP-only mgmt
+      (`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19
+      (read-only):**
+      - **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**:
+        servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS
+        resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default
+        gateway was *fine* — an active static default route `0.0.0.0/0 →
+        .1` exists; the empty `Default_Route_Gateway=[]` is just the unused
+        *interface-level* gateway field — misleading.) Clock had been stuck
+        at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP
+        source standard** (see below), all by IP, DNS-free. **Verified
+        synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2).
+      - **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear.
+        *Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
+      - **SNMP agent enabled** on the flat LAN (community strings unverified).
+        *Pending.*
+      - **Trusted-Host mgmt allowlist disabled** — admin UI reachable from
+        any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`.
+      - ⚠️ The switch is a **transparent L2 switch** — every device on it
+        **is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP);
+        only the *wired port-topology through it* is invisible (needs UniFi
+        adoption, impossible here). This is unlike the RE700X repeater.
+      - 🔴 **The D-Link is the network CORE / single point of failure.**
+        FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**,
+        the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all
+        VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16**
+        (their wireless clients appear via those ports). Nearly all wired
+        *and* wireless traffic transits this unmanaged switch → a D-Link
+        failure = total LAN+WAN outage. Amplifies the mgmt-hardening,
+        firmware-currency, monitoring and segmentation items below.
+      - **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited
+        2026-05-19, read-only):
+        - DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_
+          Status=0` → switch inserts **no Option 82**; UDM DHCP server
+          gets clean untagged requests. (`Option82_State=1` is moot —
+          only applies when relay/local-relay is active; both are off.)
+          DHCP Server Screening has no entries → doesn't block the UDM
+          (also = no rogue-DHCP protection: security note, not interop).
+        - LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_
+          Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP
+          frames *are* exchanged & parsed with UniFi gear (no TLV
+          incompat). "0 neighbors" earlier was just a point-in-time
+          aged-out snapshot. UniFi still won't *draw* topology for an
+          unadopted switch — cosmetic only.
+        - RSTP: both run 802.1w (standard, interoperable). Tuning
+          opportunity (not a fault): set a deliberate root-bridge
+          priority — currently all default 32768 so root = lowest MAC
+          (arbitrary). Make the D-Link (physical core) or UDM root.
+        - Flow control: D-Link global FC `=2` (off/auto) — matches
+          UniFi's default (802.3x disabled). Compatible.
+    - **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) —
+      bridges/NATs devices behind it. **The Klipper 3D printer at `.189`
+      sits behind the RE700X**, so UniFi only sees the repeater, never `.189`.
+    - **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the
+      gateway `.1` as resolver, not AdGuard directly) — see DNS note below.
+
+  **Infrastructure IPs known from UniFi:**
+  | IP | Name / Hostname | Notes |
+  |---|---|---|
+  | .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 |
+  | .4 | Zoraxy LXC 108 | Reverse proxy |
+  | .20 | Proxmox node `nuc` | PVE 9.1.11 |
+  | .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI |
+  | .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI |
+  | .49 | Shepard Docker host | Secondary Docker host |
+  | .50 | U7-Pro-Wall AP | Hallway/entry |
+  | .51 | U7 In-Wall AP | In-wall, UAPA6A5 |
+  | .52 | U7 Mesh (Schlafzimmer) | Bedroom |
+  | .53 | U7 Mesh (Esszimmer) | Dining room |
+  | .60 | Home Assistant OS VM | haos CTID 100 |
+  | .62 | Siemens oven | BSH Hausgeräte, WiFi |
+  | .66 | Tibber Pulse | Energy monitor (Espressif) |
+  | .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) |
+  | .124 | L0018 | Wired, unknown device |
+  | .143 | — | Sony Interactive Ent. (PlayStation) |
+  | .164 | C100_7614A4 | Tapo C100 camera (TP-Link) |
+  | .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) |
+  | .187 | RE700X | TP-Link WiFi extender — NATs devices behind it |
+  | .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) |
+  | .192 | REDMI-Note-15-Pro-5G | Xiaomi phone |
+  | .241 | VS9-EU-MNA3478A | Dyson purifier/fan |
+- **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems`
+  cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent
+  `scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels.
+- **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage`
+  (mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs);
+  several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here
+  (see n8n note under Operational rules).
+- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
+  **universal SSO/IdP for the entire ecosystem** — effectively every service
+  authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud,
+  Daytona via the Keycloak→PocketID OIDC adapter, etc.). Single sign-on
+  everywhere; one identity source to secure/audit.
+
+## Docker networks
+`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
+plus per-stack: `arr-stack_default`, `daytona-minimal_daytona-network`,
+`immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`.
+The MCP gateway bridges `ai-internal` + `shared_backend`.
+
+## AI / Agent platform  (the core, `/opt/stacks/ai`)
+- **LiteLLM** (`ai.nuclide.systems`, :14000) — model gateway. ~28 curated
+  models (provider allowlist), SAIA-terminal **failovers** + `num_retries`,
+  Redis completion cache, Gemini €10/30d `provider_budget_config`. Config:
+  `ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`).
+- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
+  **model syncer/optimizer ONLY** (curation/allowlist → DB, `OPENAI_MODEL_LIST`
+  → `.env`, recreate lobe, model-health → ntfy). One-shot container.
+- **mcp-gateway / Agent Operator** (`mcp.nuclide.systems`, :8080) — **all
+  things MCP**. DinD-spawns MCP servers on `ai-internal`, OAuth-gated streaming
+  reverse proxy `/<server>/mcp` + `/group/<g>/mcp`, multi-group, health-watcher,
+  per-server tools/list cache. **Deep health-check (2026-05-19):** beyond
+  shallow liveness, the watcher drives a real MCP session (`initialize` →
+  `tools/list` → optional `health_probe_tool` no-side-effect call) against
+  **every** server incl. spawned ones, recording `ok|degraded|down` in
+  `/api/servers`. Usage-aware: a server used within 5 min isn't probed
+  (traffic = health); idle servers probed every 15 min; probe self-excludes
+  via `X-Health-Probe`. Deep failures are status-only (no restart loops);
+  shallow-liveness auto-restart stays static-only. First run caught the
+  invisible class: `memos` degraded (search_memo → DNS `Errno -2`, the
+  ai-internal isolation bug), plus `context7`/`crawl4ai`/`markitdown` down
+  and `nextcloud` probe false-positive (probe lacks per-user NC creds —
+  candidate for `health_check:false`). **Semantic tool retrieval**: groups with >25
+  aggregated tools expose just `find_tools`/`call_tool` meta-tools (LiteLLM
+  embedding index, cosine top-k, two-phase schema promotion) instead of
+  flooding clients with hundreds of schemas. Group members are proxied via the
+  gateway's own path (`_mcp_gw`) so `exact_upstream` servers (home-assistant,
+  n8n) aggregate correctly. **MCP→LobeChat registration** (migrated here
+  from syncstack), and the **Agent Operator**: cron+event agents
+  (`/api/agents/{name}/run` webhook), LLM↔MCP executor, **Agent Creator**
+  (AI-drafts agent specs; overridable meta-prompt), prompt export. Config:
+  `mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents).
+- **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to
+  `.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions),
+  daytona, fetch, sequential-thinking, ntfy, memos, docling, markitdown,
+  crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky (brianellin,
+  rate-limit pending). Groups incl. `shepard-mcp`.
+- **Daytona** (`daytona.nuclide.systems`, daytona-minimal stack) — secure
+  code sandboxes (Agent Operator deep-dives, shepard). Registry fixed to
+  internal+host `:5000` (was misconfigured at 6000). Snapshot `sciviz-py`.
+- **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins +
+  agents in `lobechat` DB (lobe-postgres). Agents scriptable via
+  `scripts/lobe_agent.py`. Native QStash scheduler NOT functional here →
+  scheduling lives in the gateway Agent Operator. **Auth = Better Auth**
+  (not NextAuth) SSO via PocketID (`generic-oidc`, client `26f3c26b`, PKCE
+  S256). The `state_not_found` callback bug was fixed by setting
+  `AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems`; verified 2026-05-19
+  (sign-in yields a correct PocketID authorize URL; callback now validates
+  with `oAuth_code_missing`, not `state_not_found`). Definitive proof =
+  one real browser login. NB: `/oidc/auth` is LobeChat-*as*-OIDC-provider
+  (desktop/mobile), distinct from this SSO-client path.
+- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
+  meta-search).
+
+## Data layer
+- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
+  **primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
+  daytona, **memos** (migrated 2026-05-19). Superuser
+  `postgres` via unix socket; per-app dedicated role+db (role owns its db,
+  password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
+  shared_buffers 768M, max_conn 200).
+  ⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only
+  covers workflows+credentials, **not users/settings/SSO**, so the PG
+  cutover lost the owner account + OIDC. **Reverted to local-disk SQLite**
+  (tier-1 still satisfied, like PocketID). Compounding: unpinned
+  `n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC
+  `hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped.
+  Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11
+  pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled
+  2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID
+  (client `33135ad4`, correct redirect/scope/state/nonce). **Lessons:
+  (1) only a FULL pgloader migration of *all*
+  tables is complete — partial `export/import` is not; (2) pin critical
+  images — `latest` drift is a real outage cause.**
+  Migration note: services without a native full SQLite→PG export use
+  **pgloader data-only of ALL tables** into the app-built schema (exclude
+  only the app's own migration-tracking table), app role temp-SUPERUSER for
+  the load (FK/trigger disable) then reverted.
+- **lobe-postgres** (pg17/ParadeDB) — LobeChat. **lobe-redis** — LiteLLM cache
+  + LobeChat. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
+  `:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
+  on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
+- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
+  lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`).
+  **Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS
+  NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18
+  14:51 (`failed_count=0` = hung not failing), zero base backups, stalled
+  WAL recycled (that window unrecoverable). **Resolved:** Garage moved to
+  local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers
+  drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p`
+  (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra
+  safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`.
+  Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule
+  as SQLite.
+
+### Database design / standards
+- **Default = Postgres on `shared-postgres`.** Any app needing persistence
+  that supports Postgres gets a dedicated `role`+`database` there. SQLite is
+  allowed **only** when the app has no Postgres support, and then **only on
+  local disk** — never the UNAS NFS share (broken POSIX/SMB file locking;
+  caused the n8n outage + PocketID latent risk).
+- **Provisioning pattern:** `CREATE ROLE <app> LOGIN PASSWORD …; CREATE
+  DATABASE <app> OWNER <app>;` → set the app's `DB_*` env, password lives in
+  that stack's `.env` (vault migration is the long-term plan).
+- **Why standardize:** unified, durable **WAL-G PITR backup** — the single
+  biggest reason. One backup story instead of N un-backed-up SQLite files.
+- **Migration reality:** few apps have a native SQLite→PG data port. n8n does
+  (`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not —
+  switching `DB_PROVIDER` starts a *fresh* DB; data port needs
+  `pgloader`/app-specific tooling. **Memos** done via pgloader (data-only,
+  app-built schema, role temp-SUPERUSER for the load). When no safe port
+  exists, fall back to local-disk SQLite (PocketID) until a port is built.
+
+### NFS / storage strategy (corrected 2026-05-19)
+- The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do
+  in-container NFS mounts** (kernel denies; tested — `operation not
+  permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts
+  `/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at
+  v4.1" (the old tier-3 idea) is **infeasible** here without
+  `pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down).
+- **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or
+  fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any
+  version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
+- **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current
+  firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but
+  `mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e.
+  no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1
+  volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth
+  doing for it. Real export path is `/volume/<uuid>/.srv/.unifi-drive/
+  storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage`
+  alias, works on v3 — leave it).
+- **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/
+  sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4`
+  for throughput — not required.
+- **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved —
+  ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres**
+  (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem);
+  traccar data → local disk. PocketID stays on local SQLite (no native
+  PG migration path, deferred).
+
+## Services
+Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
+**Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep**
+(+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities),
+**ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001),
+**arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**,
+**Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups),
+**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).
+
+## Operational rules / conventions
+- **Separation**: `syncstack` = models; `mcp-gateway` = all MCP/agents.
+- **Self-host first**: external/cloud only when no self-hosted form exists.
+- **Pinned NTP source standard** — all network gear/infra should point at
+  the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
+  resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
+  1. `192.53.103.108` — PTB ptbtime1 (DE national time)
+  2. `192.53.103.104` — PTB ptbtime2 (DE national time)
+  3. `162.159.200.123` — Cloudflare NTP anycast (fallback)
+  Applied to the D-Link switch 2026-05-19. UDM + other infra should converge
+  on the same set (a self-hosted LAN NTP server is a roadmap option, but the
+  standard stays "same pinned sources" everywhere).
+- Run Python with `uv run` (not python3) in `/opt/stacks`.
+- Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`.
+- Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open
+  hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked
+  into `homepage/config/logs/homepage.log` — rotate + scrub when hardening.
+  ⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the
+  LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw —
+  single sniff/leak has broad blast radius. Rotate per-service when hardening
+  (noted 2026-05-19, rotation deferred per user).
+- **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti
+  network gear requires explicit confirmation + a stated risk assessment
+  first; read-only queries are fine. High blast radius (foundation layer).
+- **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file
+  locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres`
+  or local disk, never the NFS share. **n8n** hit this (`Database connection
+  timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
+  data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
+  kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
+- Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename
+  once the agent-platform identity firms up.
+
+## Open / roadmap
+mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
+expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
+(in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`,
+197 tools, Network App API key, `http` transport; local API fully working).
+**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
+Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
+no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep
+`plugins`!); env → secret vault.
+
+**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
+auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
+on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed).
+Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest.
+
+**Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens
+oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`,
+no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so
+untrusted appliances can't reach the server/Proxmox subnet. Complications to
+design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and
+**TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't
+honour UniFi VLAN tags natively — segmentation needs a plan for the wired
+trunk through the D-Link and the repeater's bridge mode.
+
+### Observability LXC (planned — scoped from this session's incidents)
+**Why it's now a priority:** the WAL-G archiver was hung **silently for
+~13 h** (`failed_count=0`, zero base backups) and would never have been
+noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring
+must target *exactly these silent-failure classes*.
+- **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104
+  hosts everything, so the monitor must survive/alert when 104 is down.
+- **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager
+  → **ntfy `homelab-ai`** (already the alert channel).
+- **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter
+  ×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for
+  Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor
+  token), and a **custom WAL-G/archiver textfile collector**:
+  `pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog,
+  and `wal-g backup-list` newest-base age — per PG instance.
+- **Alerts (priority order, derived from real incidents):**
+  1. WAL archiver stalled (last_archived age > 15 m) or newest base backup
+     > 26 h, any PG instance.
+  2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency.
+  3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under
+     `/mnt/pve/unas` (catches a regression of the tier-1 rule).
+  4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
+  5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d.