fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f8d8ee3ac053fe680384b6166fd26c83d8940b99
docs/services/homelab-architecture.md
T
fkrebs b2cb123414 monitoring: document Loki/Alloy/pve-exporter, dashboards and alerts 
CT 109 ops stack now includes Loki (log agg), Grafana Alloy (log agent on
all 12 hosts), prometheus-pve-exporter. 5 Grafana dashboards imported;
3 alert rules wired to Gotify.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 07:03:51 +02:00

513 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Nuclide Ecosystem — Homelab Architecture
Living architecture reference for the `/opt/stacks` homelab. ~66 containers
across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
`*.nuclide.systems` via one reverse proxy.
## Topology
- **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`.
Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS
`rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`.
- **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`**
(Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via
`root@pam!mcp` token (now `PVEAuditor`, read-only). **9 guests (+ 2 planned)** — canonical roster in [`ct-inventory.md`](../ct-inventory.md):
- `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60`
- `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49`
- `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking,
2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via
the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard**
(confirmed) — so ad/tracker blocking is network-wide despite DHCP not
handing out AdGuard's IP directly.
- `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
- `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/48G/200G) → `.40`
- `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
- `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
- `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
- `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20
- `109` lxc **ops** `192.168.1.8` — Prometheus + Grafana + Loki + Alloy + pve-exporter (2026-05-23); future: Arcane, Dozzle, Homarr, Tinyauth
- `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21
- Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server`
(197 tools, Network App API key); Site Manager/cloud tools need
`UNIFI_SITE_MANAGER_ENABLED`, all local API tools work.
- **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool
`.100–.250` (24 h lease), domain `localdomain`. Static infra lives
`.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary"
(WAN2, disabled). 47 clients (17 wired, 30 wireless).
- **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands
2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat
network**: servers, IoT, consoles, phones all share one L2 segment.
- **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot),
almost entirely system-predefined rules. Port forwards: **80/443 →
192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch
protocol — already at UDM level, no Zoraxy stream proxy needed).
- ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage):
- **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw
**6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/
port-clients don't appear in UniFi topology. HTTP-only mgmt
(`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19
(read-only):**
- **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**:
servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS
resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default
gateway was *fine* — an active static default route `0.0.0.0/0 →
.1` exists; the empty `Default_Route_Gateway=[]` is just the unused
*interface-level* gateway field — misleading.) Clock had been stuck
at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP
source standard** (see below), all by IP, DNS-free. **Verified
synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2).
- **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear.
*Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
- **SNMP agent enabled** on the flat LAN (community strings unverified).
*Pending.*
- **Trusted-Host mgmt allowlist disabled** — admin UI reachable from
any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`.
- ⚠️ The switch is a **transparent L2 switch** — every device on it
**is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP);
only the *wired port-topology through it* is invisible (needs UniFi
adoption, impossible here). This is unlike the RE700X repeater.
- 🔴 **The D-Link is the network CORE / single point of failure.**
FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**,
the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all
VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16**
(their wireless clients appear via those ports). Nearly all wired
*and* wireless traffic transits this unmanaged switch → a D-Link
failure = total LAN+WAN outage. Amplifies the mgmt-hardening,
firmware-currency, monitoring and segmentation items below.
- **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited
2026-05-19, read-only):
- DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_
Status=0` → switch inserts **no Option 82**; UDM DHCP server
gets clean untagged requests. (`Option82_State=1` is moot —
only applies when relay/local-relay is active; both are off.)
DHCP Server Screening has no entries → doesn't block the UDM
(also = no rogue-DHCP protection: security note, not interop).
- LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_
Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP
frames *are* exchanged & parsed with UniFi gear (no TLV
incompat). "0 neighbors" earlier was just a point-in-time
aged-out snapshot. UniFi still won't *draw* topology for an
unadopted switch — cosmetic only.
- RSTP: both run 802.1w (standard, interoperable). Tuning
opportunity (not a fault): set a deliberate root-bridge
priority — currently all default 32768 so root = lowest MAC
(arbitrary). Make the D-Link (physical core) or UDM root.
- Flow control: D-Link global FC `=2` (off/auto) — matches
UniFi's default (802.3x disabled). Compatible.
- **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) —
bridges/NATs devices behind it. **The Klipper 3D printer at `.189`
sits behind the RE700X**, so UniFi only sees the repeater, never `.189`.
- **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the
gateway `.1` as resolver, not AdGuard directly) — see DNS note below.
**Infrastructure IPs known from UniFi:**
| IP | Name / Hostname | Notes |
|---|---|---|
| .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 |
| .4 | Zoraxy LXC 108 | Reverse proxy |
| .20 | Proxmox node `nuc` | PVE 9.1.11 |
| .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI |
| .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI |
| .49 | Shepard Docker host | Secondary Docker host |
| .50 | U7-Pro-Wall AP | Hallway/entry |
| .51 | U7 In-Wall AP | In-wall, UAPA6A5 |
| .52 | U7 Mesh (Schlafzimmer) | Bedroom |
| .53 | U7 Mesh (Esszimmer) | Dining room |
| .60 | Home Assistant OS VM | haos CTID 100 |
| .62 | Siemens oven | BSH Hausgeräte, WiFi |
| .66 | Tibber Pulse | Energy monitor (Espressif) |
| .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) |
| .124 | L0018 | Wired, unknown device |
| .143 | — | Sony Interactive Ent. (PlayStation) |
| .164 | C100_7614A4 | Tapo C100 camera (TP-Link) |
| .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) |
| .187 | RE700X | TP-Link WiFi extender — NATs devices behind it |
| .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) |
| .192 | REDMI-Note-15-Pro-5G | Xiaomi phone |
| .241 | VS9-EU-MNA3478A | Dyson purifier/fan |
- **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems`
cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent
`scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels.
- **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage`
(mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs);
several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here
(see n8n note under Operational rules).
- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
**universal SSO/IdP for the entire ecosystem** — effectively every service
authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud,
Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity
source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.
## Docker networks
`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
plus per-stack: `arr-stack_default`,
`immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`.
The MCP gateway bridges `ai-internal` + `shared_backend`.
## AI / Agent platform (the core, `/opt/stacks/ai`)
- **LiteLLM** (`ai.nuclide.systems`, :14000) — model gateway. ~28 curated
models (provider allowlist), SAIA-terminal **failovers** + `num_retries`,
Redis completion cache, Gemini €10/30d `provider_budget_config`. Config:
`ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`).
- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
**model syncer/optimizer ONLY** (curation/allowlist → DB, `OPENAI_MODEL_LIST`
→ `.env`, recreate lobe, model-health → ntfy). One-shot container.
- **mcp-gateway / Agent Operator** (`mcp.nuclide.systems`, :8080) — **all
things MCP**. DinD-spawns MCP servers on `ai-internal`, OAuth-gated streaming
reverse proxy `/<server>/mcp` + `/group/<g>/mcp`, multi-group, health-watcher,
per-server tools/list cache. **Deep health-check (2026-05-19):** beyond
shallow liveness, the watcher drives a real MCP session (`initialize` →
`tools/list` → optional `health_probe_tool` no-side-effect call) against
**every** server incl. spawned ones, recording `ok|degraded|down` in
`/api/servers`. Usage-aware: a server used within 5 min isn't probed
(traffic = health); idle servers probed every 15 min; probe self-excludes
via `X-Health-Probe`. Deep failures are status-only (no restart loops);
shallow-liveness auto-restart stays static-only. First run caught the
invisible class: `memos` degraded (search_memo → DNS `Errno -2`, the
ai-internal isolation bug), plus `context7`/`crawl4ai`/`markitdown` down
and `nextcloud` probe false-positive (probe lacks per-user NC creds —
candidate for `health_check:false`). **Semantic tool retrieval**: groups with >25
aggregated tools expose just `find_tools`/`call_tool` meta-tools (LiteLLM
embedding index, cosine top-k, two-phase schema promotion) instead of
flooding clients with hundreds of schemas. Group members are proxied via the
gateway's own path (`_mcp_gw`) so `exact_upstream` servers (home-assistant,
n8n) aggregate correctly. **MCP→LobeChat registration** (migrated here
from syncstack), and the **Agent Operator**: cron+event agents
(`/api/agents/{name}/run` webhook), LLM↔MCP executor, **Agent Creator**
(AI-drafts agent specs; overridable meta-prompt), prompt export. Config:
`mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents).
- **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to
`.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions),
**coder** (replaced daytona 2026-05-20), fetch, sequential-thinking, ntfy, memos,
docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui,
bluesky (brianellin, rate-limit pending). Groups incl. `shepard-mcp`.
Gaps to close (community MCPs exist): gitea, paperless, karakeep, vaultwarden,
proxmox-ve, audiobookshelf. See [`services/mcp-gateway.md`](mcp-gateway.md).
- **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox /
dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough,
baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack
pre-baked). See [`services/dev-environment.md`](dev-environment.md).
- **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins +
agents in `lobechat` DB (lobe-postgres). Agents scriptable via
`scripts/lobe_agent.py`. Native QStash scheduler NOT functional here →
scheduling lives in the gateway Agent Operator. **Auth = Better Auth**
(not NextAuth) SSO via PocketID (`generic-oidc`, client `26f3c26b`, PKCE
S256). The `state_not_found` callback bug was fixed by setting
`AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems`; verified 2026-05-19
(sign-in yields a correct PocketID authorize URL; callback now validates
with `oAuth_code_missing`, not `state_not_found`). Definitive proof =
one real browser login. NB: `/oidc/auth` is LobeChat-*as*-OIDC-provider
(desktop/mobile), distinct from this SSO-client path.
- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
meta-search).
## Data layer
- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
**primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
**memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser
`postgres` via unix socket; per-app dedicated role+db (role owns its db,
password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
shared_buffers 768M, max_conn 200).
⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only
covers workflows+credentials, **not users/settings/SSO**, so the PG
cutover lost the owner account + OIDC. **Reverted to local-disk SQLite**
(tier-1 still satisfied, like PocketID). Compounding: unpinned
`n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC
`hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped.
Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11
pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled
2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID
(client `33135ad4`, correct redirect/scope/state/nonce). **Lessons:
(1) only a FULL pgloader migration of *all*
tables is complete — partial `export/import` is not; (2) pin critical
images — `latest` drift is a real outage cause.**
Migration note: services without a native full SQLite→PG export use
**pgloader data-only of ALL tables** into the app-built schema (exclude
only the app's own migration-tracking table), app role temp-SUPERUSER for
the load (FK/trigger disable) then reverted.
- **lobe-postgres** (pg17/ParadeDB) — LobeChat. **lobe-redis** — LiteLLM cache
+ LobeChat. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
`:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`).
**Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS
NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18
14:51 (`failed_count=0` = hung not failing), zero base backups, stalled
WAL recycled (that window unrecoverable). **Resolved:** Garage moved to
local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers
drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p`
(fail-fast vs infinite hang); fresh base backups taken for all 3. Extra
safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`.
Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule
as SQLite.
### Database design / standards
- **Default = Postgres on `shared-postgres`.** Any app needing persistence
that supports Postgres gets a dedicated `role`+`database` there. SQLite is
allowed **only** when the app has no Postgres support, and then **only on
local disk** — never the UNAS NFS share (broken POSIX/SMB file locking;
caused the n8n outage + PocketID latent risk).
- **Provisioning pattern:** `CREATE ROLE <app> LOGIN PASSWORD …; CREATE
DATABASE <app> OWNER <app>;` → set the app's `DB_*` env, password lives in
that stack's `.env` (vault migration is the long-term plan).
- **Why standardize:** unified, durable **WAL-G PITR backup** — the single
biggest reason. One backup story instead of N un-backed-up SQLite files.
- **Migration reality:** few apps have a native SQLite→PG data port. n8n does
(`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not —
switching `DB_PROVIDER` starts a *fresh* DB; data port needs
`pgloader`/app-specific tooling. **Memos** done via pgloader (data-only,
app-built schema, role temp-SUPERUSER for the load). When no safe port
exists, fall back to local-disk SQLite (PocketID) until a port is built.
### NFS / storage strategy (corrected 2026-05-19)
- The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do
in-container NFS mounts** (kernel denies; tested — `operation not
permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts
`/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at
v4.1" (the old tier-3 idea) is **infeasible** here without
`pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down).
- **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or
fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any
version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
- **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current
firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but
`mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e.
no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1
volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth
doing for it. Real export path is `/volume/<uuid>/.srv/.unifi-drive/
storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage`
alias, works on v3 — leave it).
- **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/
sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4`
for throughput — not required.
- **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved —
ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres**
(pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem);
traccar data → local disk. PocketID stays on local SQLite (no native
PG migration path, deferred).
## Services
Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
**Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep**
(+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities),
**ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001),
**arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**,
**Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups),
**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).
## Operational rules / conventions
- **Separation**: `syncstack` = models; `mcp-gateway` = all MCP/agents.
- **Self-host first**: external/cloud only when no self-hosted form exists.
- **Pinned NTP source standard** — all network gear/infra should point at
the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
1. `192.53.103.108` — PTB ptbtime1 (DE national time)
2. `192.53.103.104` — PTB ptbtime2 (DE national time)
3. `162.159.200.123` — Cloudflare NTP anycast (fallback)
Applied to the D-Link switch 2026-05-19. UDM + other infra should converge
on the same set (a self-hosted LAN NTP server is a roadmap option, but the
standard stays "same pinned sources" everywhere).
- Run Python with `uv run` (not python3) in `/opt/stacks`.
- Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`.
- Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open
hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked
into `homepage/config/logs/homepage.log` — rotate + scrub when hardening.
⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the
LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw —
single sniff/leak has broad blast radius. Rotate per-service when hardening
(noted 2026-05-19, rotation deferred per user).
- **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti
network gear requires explicit confirmation + a stated risk assessment
first; read-only queries are fine. High blast radius (foundation layer).
- **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file
locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres`
or local disk, never the NFS share. **n8n** hit this (`Database connection
timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
- Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename
once the agent-platform identity firms up.
## Open / roadmap
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
(in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`,
197 tools, Network App API key, `http` transport; local API fully working).
**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep
`plugins`!); env → secret vault.
**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed).
Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest.
**Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens
oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`,
no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so
untrusted appliances can't reach the server/Proxmox subnet. Complications to
design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and
**TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't
honour UniFi VLAN tags natively — segmentation needs a plan for the wired
trunk through the D-Link and the repeater's bridge mode.
### Observability LXC (planned — scoped from this session's incidents)
**Why it's now a priority:** the WAL-G archiver was hung **silently for
~13 h** (`failed_count=0`, zero base backups) and would never have been
noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring
must target *exactly these silent-failure classes*.
- **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104
hosts everything, so the monitor must survive/alert when 104 is down.
- **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager
→ **ntfy `homelab-ai`** (already the alert channel).
- **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter
×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for
Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor
token), and a **custom WAL-G/archiver textfile collector**:
`pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog,
and `wal-g backup-list` newest-base age — per PG instance.
- **Alerts (priority order, derived from real incidents):**
1. WAL archiver stalled (last_archived age > 15 m) or newest base backup
> 26 h, any PG instance.
2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency.
3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under
`/mnt/pve/unas` (catches a regression of the tier-1 rule).
4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d.
## Topology — at a glance
### Physical / CT layout
```mermaid
flowchart TB
subgraph Host["Proxmox host · 192.168.1.20 · Intel Core Ultra 7 155H · 64 GiB"]
direction TB
haos["VM 100 · haos<br/>(.60)<br/>Home Assistant"]
shepard["CT 101 · shepard<br/>(.49)<br/>Shepard product stack"]
dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
obs["CT 109 · ops<br/>(.8)<br/>Prometheus · Grafana · Loki · Alloy · pve-exporter"]
id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
end
UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3")]
UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])
UDM <--> Host
UDM <--> Inet
UNAS <--> shepard
UNAS <--> backrest
UNAS <--> docker104
UNAS <--> nc
UNAS <--> dev
classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
class db planned
```
### Auth plane — Pocket-ID is the universal IdP
Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.
```mermaid
flowchart LR
user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy<br/>CT 108]
zx --> coder[Coder · CT 111]
zx --> gitea[Gitea · CT 111]
zx --> nc2[Nextcloud · CT 105]
zx --> immich[Immich · CT 104]
zx --> lobe[LobeChat · CT 104]
zx --> n8n[n8n · CT 104]
zx --> mcp[MCP gateway · CT 104]
zx --> vw[Vaultwarden · CT 104]
coder --> pid[(Pocket-ID<br/>CT 110)]
gitea --> pid
nc2 --> pid
immich --> pid
lobe --> pid
n8n --> pid
mcp --> pid
vw -.->|via Tinyauth<br/>when CT 109 lands| pid
classDef pending stroke-dasharray:4 4,color:#888
class vw pending
```
### MCP plane — gateway control, child workers
The MCP gateway on CT 104 is the OIDC-gated front; ~23 child MCP servers run on the same docker network (`ai-internal`). When CT 109 "ops" is built, the gateway moves to CT 109 and reaches CT 104's docker daemon via a socket-proxy; the children stay put.
```mermaid
flowchart LR
client[["Claude Code / Cursor / LobeChat"]]
gw["MCP gateway<br/>(CT 104 → CT 109 ops when 109 lands)<br/>mcp.nuclide.systems"]
client -- Bearer / OIDC --> gw
subgraph "CT 104 · ai-internal docker net"
direction TB
coderm[coder-mcp]
immm[mcp-immich]
n8nm[mcp-n8n]
fetch[mcp-fetch]
time[mcp-time]
cw[mcp-crawl4ai]
seq[mcp-sequential-thinking]
ham[home-assistant-mcp]
kr[kroki-mcp]
others[...18 more]
end
gw --> coderm
gw --> immm
gw --> n8nm
gw --> fetch
gw --> time
gw --> cw
gw --> seq
gw --> ham
gw --> kr
gw --> others
coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]
ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]
```
### Data plane — what lives where
```mermaid
flowchart TB
subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"]
pid_d[Pocket-ID sqlite · CT 110]
immich_pg[Immich Postgres · CT 104]
shared_pg[shared-postgres · CT 104]
coder_pg[coder-db · CT 111]
gitea_pg[gitea-db · CT 111]
garage["Garage S3 · CT 104<br/>(moved off NFS 2026-05-19)"]
n8n_local["n8n data · CT 104<br/>(reverted from NFS 2026-05-19)"]
end
subgraph Bulk["Bulk · UNAS NFSv3"]
media["Immich media, Paperless docs,<br/>arr-stack media, audiobooks"]
coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder]
gitea_data[Gitea repos · /mnt/pve/unas/services/gitea]
vw_data["Vaultwarden data<br/>(tier-1 leak — plan to move local)"]
end
subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"]
nc_data[Nextcloud user files]
end
subgraph Backup["Off-host backup"]
jottacloud[(jottacloud<br/>via Backrest)]
end
Tier1 -. WAL-G .-> garage
Bulk -. only media/data-dir .-> jottacloud
classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc
class vw_data,jottacloud gap
```
The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See `stacks/storage.md` for the verified state and the cleanup TODO list.
Reference in New Issue View Git Blame Copy Permalink