fkrebs
9cbd02e3ff
- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts), Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3 plans have hooks), Diun, config-to-git, offsite sync, Gotify apps, timers and crons across the fleet. - backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access to CT 113; 2>/dev/null hid the host-key error). Documented retry loop + SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux; noted litellm decommissioned but DB retained).
521 lines
30 KiB
Markdown
521 lines
30 KiB
Markdown
# Nuclide Ecosystem — Homelab Architecture
|
||
|
||
Living architecture reference for the `/opt/stacks` homelab. ~66 containers
|
||
across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
|
||
`*.nuclide.systems` via one reverse proxy.
|
||
|
||
## Topology
|
||
- **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`.
|
||
Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS
|
||
`rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`.
|
||
- **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`**
|
||
(Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via
|
||
`root@pam!mcp` token (now `PVEAuditor`, read-only). **9 guests (+ 2 planned)** — canonical roster in [`ct-inventory.md`](../ct-inventory.md):
|
||
- `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60`
|
||
- `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49`
|
||
- `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking,
|
||
2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via
|
||
the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard**
|
||
(confirmed) — so ad/tracker blocking is network-wide despite DHCP not
|
||
handing out AdGuard's IP directly.
|
||
- `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
|
||
- `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/48G/200G) → `.40`
|
||
- `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
|
||
- `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
|
||
- `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
|
||
- `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20
|
||
- `109` lxc **ops** `192.168.1.8` — Prometheus + Grafana + Loki + Alloy + pve-exporter (2026-05-23); future: Arcane, Dozzle, Homarr, Tinyauth
|
||
- `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21
|
||
- Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
|
||
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
|
||
APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
|
||
enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server`
|
||
(197 tools, Network App API key); Site Manager/cloud tools need
|
||
`UNIFI_SITE_MANAGER_ENABLED`, all local API tools work.
|
||
- **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool
|
||
`.100–.250` (24 h lease), domain `localdomain`. Static infra lives
|
||
`.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary"
|
||
(WAN2, disabled). 47 clients (17 wired, 30 wireless).
|
||
- **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands
|
||
2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat
|
||
network**: servers, IoT, consoles, phones all share one L2 segment.
|
||
- **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot),
|
||
almost entirely system-predefined rules. Port forwards: **80/443 →
|
||
192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch
|
||
protocol — already at UDM level, no Zoraxy stream proxy needed).
|
||
- ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage):
|
||
- **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw
|
||
**6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/
|
||
port-clients don't appear in UniFi topology. HTTP-only mgmt
|
||
(`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19
|
||
(read-only):**
|
||
- **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**:
|
||
servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS
|
||
resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default
|
||
gateway was *fine* — an active static default route `0.0.0.0/0 →
|
||
.1` exists; the empty `Default_Route_Gateway=[]` is just the unused
|
||
*interface-level* gateway field — misleading.) Clock had been stuck
|
||
at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP
|
||
source standard** (see below), all by IP, DNS-free. **Verified
|
||
synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2).
|
||
- **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear.
|
||
*Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
|
||
- **SNMP agent enabled** on the flat LAN (community strings unverified).
|
||
*Pending.*
|
||
- **Trusted-Host mgmt allowlist disabled** — admin UI reachable from
|
||
any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`.
|
||
- ⚠️ The switch is a **transparent L2 switch** — every device on it
|
||
**is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP);
|
||
only the *wired port-topology through it* is invisible (needs UniFi
|
||
adoption, impossible here). This is unlike the RE700X repeater.
|
||
- 🔴 **The D-Link is the network CORE / single point of failure.**
|
||
FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**,
|
||
the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all
|
||
VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16**
|
||
(their wireless clients appear via those ports). Nearly all wired
|
||
*and* wireless traffic transits this unmanaged switch → a D-Link
|
||
failure = total LAN+WAN outage. Amplifies the mgmt-hardening,
|
||
firmware-currency, monitoring and segmentation items below.
|
||
- **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited
|
||
2026-05-19, read-only):
|
||
- DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_
|
||
Status=0` → switch inserts **no Option 82**; UDM DHCP server
|
||
gets clean untagged requests. (`Option82_State=1` is moot —
|
||
only applies when relay/local-relay is active; both are off.)
|
||
DHCP Server Screening has no entries → doesn't block the UDM
|
||
(also = no rogue-DHCP protection: security note, not interop).
|
||
- LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_
|
||
Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP
|
||
frames *are* exchanged & parsed with UniFi gear (no TLV
|
||
incompat). "0 neighbors" earlier was just a point-in-time
|
||
aged-out snapshot. UniFi still won't *draw* topology for an
|
||
unadopted switch — cosmetic only.
|
||
- RSTP: both run 802.1w (standard, interoperable). Tuning
|
||
opportunity (not a fault): set a deliberate root-bridge
|
||
priority — currently all default 32768 so root = lowest MAC
|
||
(arbitrary). Make the D-Link (physical core) or UDM root.
|
||
- Flow control: D-Link global FC `=2` (off/auto) — matches
|
||
UniFi's default (802.3x disabled). Compatible.
|
||
- **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) —
|
||
bridges/NATs devices behind it. **The Klipper 3D printer at `.189`
|
||
sits behind the RE700X**, so UniFi only sees the repeater, never `.189`.
|
||
- **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the
|
||
gateway `.1` as resolver, not AdGuard directly) — see DNS note below.
|
||
|
||
**Infrastructure IPs known from UniFi:**
|
||
| IP | Name / Hostname | Notes |
|
||
|---|---|---|
|
||
| .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 |
|
||
| .4 | Zoraxy LXC 108 | Reverse proxy |
|
||
| .20 | Proxmox node `nuc` | PVE 9.1.11 |
|
||
| .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI |
|
||
| .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI |
|
||
| .49 | Shepard Docker host | Secondary Docker host |
|
||
| .50 | U7-Pro-Wall AP | Hallway/entry |
|
||
| .51 | U7 In-Wall AP | In-wall, UAPA6A5 |
|
||
| .52 | U7 Mesh (Schlafzimmer) | Bedroom |
|
||
| .53 | U7 Mesh (Esszimmer) | Dining room |
|
||
| .60 | Home Assistant OS VM | haos CTID 100 |
|
||
| .62 | Siemens oven | BSH Hausgeräte, WiFi |
|
||
| .66 | Tibber Pulse | Energy monitor (Espressif) |
|
||
| .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) |
|
||
| .124 | L0018 | Wired, unknown device |
|
||
| .143 | — | Sony Interactive Ent. (PlayStation) |
|
||
| .164 | C100_7614A4 | Tapo C100 camera (TP-Link) |
|
||
| .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) |
|
||
| .187 | RE700X | TP-Link WiFi extender — NATs devices behind it |
|
||
| .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) |
|
||
| .192 | REDMI-Note-15-Pro-5G | Xiaomi phone |
|
||
| .241 | VS9-EU-MNA3478A | Dyson purifier/fan |
|
||
- **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems`
|
||
cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent
|
||
`scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels.
|
||
- **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage`
|
||
(mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs);
|
||
several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here
|
||
(see n8n note under Operational rules).
|
||
- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
|
||
**universal SSO/IdP for the entire ecosystem** — effectively every service
|
||
authenticates via PocketID OIDC (LiteLLM, Nextcloud, Coder, Gitea,
|
||
Vaultwarden, etc.). Single sign-on everywhere; one identity source to
|
||
secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.
|
||
Open WebUI OIDC not yet wired (as of 2026-05-26).
|
||
|
||
## Docker networks
|
||
`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
|
||
plus per-stack: `arr-stack_default`,
|
||
`immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`.
|
||
The MCP gateway bridges `ai-internal` + `shared_backend`.
|
||
|
||
## AI / Agent platform (the core, `/opt/stacks/ai`)
|
||
- **Bifrost** (`ai.nuclide.systems`, :14003) — unified LLM gateway + MCP host.
|
||
Stack `ai/bifrost/`. Auth via `sk-bf-` VKs. LLM inference at `/v1`; MCP at
|
||
`/mcp` (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to
|
||
LiteLLM internally.
|
||
- **LiteLLM** (:14000, **internal only** — no Zoraxy route) — model gateway,
|
||
backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal
|
||
**failovers** + `num_retries`, Redis completion cache, Gemini €10/30d
|
||
`provider_budget_config`. Config: `ai/litellm-config/config.yaml` (+ DB
|
||
overlay `LiteLLM_Config`). Reachable as `http://litellm:4000` inside Docker.
|
||
- **Open WebUI** (`chat.nuclide.systems`, :14002) — chat UI. Stack
|
||
`ai/open-webui.yml`. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not
|
||
yet wired as of 2026-05-26).
|
||
- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
|
||
**model syncer/optimizer ONLY** (curation/allowlist → DB, model-health → ntfy).
|
||
One-shot container.
|
||
- ~~**mcp-gateway**~~ — **DECOMMISSIONED 2026-05-26.** Was FastAPI/DinD on
|
||
`:8080` at `mcp.nuclide.systems`. Compose renamed `.DECOMMISSIONED-2026-05-26`.
|
||
MCP is now served by Bifrost at `https://ai.nuclide.systems/mcp`.
|
||
- **MCP servers** (Bifrost-managed, formerly gateway-managed): time, home-assistant,
|
||
kroki, coder, fetch, sequential-thinking, gotify, memos, docling, markitdown,
|
||
crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky, obsidian,
|
||
immich, n8n, nextcloud, paperless, unifi, upload-artifact, gitea, proxmox,
|
||
shepard, git, gitlab, youtube-transcript, searxng (30 servers). 29 clients,
|
||
~760 tools. See [`services/mcp-servers.md`](mcp-servers.md).
|
||
- **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox /
|
||
dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough,
|
||
baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack
|
||
pre-baked). See [`services/dev-environment.md`](dev-environment.md).
|
||
- ~~**LobeChat**~~ — **DECOMMISSIONED 2026-05-26.** Was at `chat.nuclide.systems`
|
||
(:14001). Compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Replaced
|
||
by Open WebUI.
|
||
- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
|
||
meta-search).
|
||
|
||
## Data layer
|
||
- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
|
||
**primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
|
||
**memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser
|
||
`postgres` via unix socket; per-app dedicated role+db (role owns its db,
|
||
password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
|
||
shared_buffers 768M, max_conn 200).
|
||
⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only
|
||
covers workflows+credentials, **not users/settings/SSO**, so the PG
|
||
cutover lost the owner account + OIDC. **Reverted to local-disk SQLite**
|
||
(tier-1 still satisfied, like PocketID). Compounding: unpinned
|
||
`n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC
|
||
`hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped.
|
||
Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11
|
||
pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled
|
||
2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID
|
||
(client `33135ad4`, correct redirect/scope/state/nonce). **Lessons:
|
||
(1) only a FULL pgloader migration of *all*
|
||
tables is complete — partial `export/import` is not; (2) pin critical
|
||
images — `latest` drift is a real outage cause.**
|
||
Migration note: services without a native full SQLite→PG export use
|
||
**pgloader data-only of ALL tables** into the app-built schema (exclude
|
||
only the app's own migration-tracking table), app role temp-SUPERUSER for
|
||
the load (FK/trigger disable) then reverted.
|
||
- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). **lobe-redis** — LiteLLM cache (LobeChat decommissioned). **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
|
||
`:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
|
||
on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
|
||
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
|
||
lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`).
|
||
**Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS
|
||
NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18
|
||
14:51 (`failed_count=0` = hung not failing), zero base backups, stalled
|
||
WAL recycled (that window unrecoverable). **Resolved:** Garage moved to
|
||
local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers
|
||
drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p`
|
||
(fail-fast vs infinite hang); fresh base backups taken for all 3. Extra
|
||
safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`.
|
||
Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule
|
||
as SQLite.
|
||
|
||
### Database design / standards
|
||
- **Default = Postgres on `shared-postgres`.** Any app needing persistence
|
||
that supports Postgres gets a dedicated `role`+`database` there. SQLite is
|
||
allowed **only** when the app has no Postgres support, and then **only on
|
||
local disk** — never the UNAS NFS share (broken POSIX/SMB file locking;
|
||
caused the n8n outage + PocketID latent risk).
|
||
- **Provisioning pattern:** `CREATE ROLE <app> LOGIN PASSWORD …; CREATE
|
||
DATABASE <app> OWNER <app>;` → set the app's `DB_*` env, password lives in
|
||
that stack's `.env` (vault migration is the long-term plan).
|
||
- **Why standardize:** unified, durable **WAL-G PITR backup** — the single
|
||
biggest reason. One backup story instead of N un-backed-up SQLite files.
|
||
- **Migration reality:** few apps have a native SQLite→PG data port. n8n does
|
||
(`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not —
|
||
switching `DB_PROVIDER` starts a *fresh* DB; data port needs
|
||
`pgloader`/app-specific tooling. **Memos** done via pgloader (data-only,
|
||
app-built schema, role temp-SUPERUSER for the load). When no safe port
|
||
exists, fall back to local-disk SQLite (PocketID) until a port is built.
|
||
|
||
### NFS / storage strategy (corrected 2026-05-19)
|
||
- The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do
|
||
in-container NFS mounts** (kernel denies; tested — `operation not
|
||
permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts
|
||
`/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at
|
||
v4.1" (the old tier-3 idea) is **infeasible** here without
|
||
`pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down).
|
||
- **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or
|
||
fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any
|
||
version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
|
||
- **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current
|
||
firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but
|
||
`mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e.
|
||
no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1
|
||
volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth
|
||
doing for it. Real export path is `/volume/<uuid>/.srv/.unifi-drive/
|
||
storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage`
|
||
alias, works on v3 — leave it).
|
||
- **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/
|
||
sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4`
|
||
for throughput — not required.
|
||
- **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved —
|
||
ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres**
|
||
(pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem);
|
||
traccar data → local disk. PocketID stays on local SQLite (no native
|
||
PG migration path, deferred).
|
||
|
||
## Services
|
||
Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
|
||
**Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep**
|
||
(+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities),
|
||
**ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001),
|
||
**arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**,
|
||
**Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups),
|
||
**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).
|
||
|
||
## Operational rules / conventions
|
||
- **Separation**: `syncstack` = model curation; `bifrost` = MCP + LLM gateway (mcp-gateway decommissioned 2026-05-26).
|
||
- **Self-host first**: external/cloud only when no self-hosted form exists.
|
||
- **Pinned NTP source standard** — all network gear/infra should point at
|
||
the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
|
||
resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
|
||
1. `192.53.103.108` — PTB ptbtime1 (DE national time)
|
||
2. `192.53.103.104` — PTB ptbtime2 (DE national time)
|
||
3. `162.159.200.123` — Cloudflare NTP anycast (fallback)
|
||
Applied to the D-Link switch 2026-05-19. UDM + other infra should converge
|
||
on the same set (a self-hosted LAN NTP server is a roadmap option, but the
|
||
standard stays "same pinned sources" everywhere).
|
||
- Run Python with `uv run` (not python3) in `/opt/stacks`.
|
||
- Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`.
|
||
- Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open
|
||
hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked
|
||
into `homepage/config/logs/homepage.log` — rotate + scrub when hardening.
|
||
⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the
|
||
LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw —
|
||
single sniff/leak has broad blast radius. Rotate per-service when hardening
|
||
(noted 2026-05-19, rotation deferred per user).
|
||
- **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti
|
||
network gear requires explicit confirmation + a stated risk assessment
|
||
first; read-only queries are fine. High blast radius (foundation layer).
|
||
- **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file
|
||
locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres`
|
||
or local disk, never the NFS share. **n8n** hit this (`Database connection
|
||
timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
|
||
data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
|
||
kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
|
||
- Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.
|
||
|
||
## Ops coverage matrix (verified 2026-05-28)
|
||
|
||
| Capability | Coverage |
|
||
|---|---|
|
||
| **Log aggregation** (Loki @ CT 109) | All 8 hosts shipping via Alloy: CT 101, 102, 103 (newly enabled today), 104, 105, 109, 113, nuc. Last 5m: 3.6k lines from CT 104, 2.1k from CT 113, 100s on quieter hosts. |
|
||
| **Metrics** (Prometheus @ CT 109) | Scrapes node-exporter on CT 104/109, pve-exporter, traefik, home-assistant prom-endpoint, WAL-G textfile collector |
|
||
| **Backups** (Backrest @ CT 103 → JottaCloud) | 14 services paths + HA tars + Immich tars + Nextcloud Borg + paperless media + Garage S3 + n8n local data. All 16 plan paths verified present. 3 pre-stage hooks (pocketid, garage, n8n). Success/error hooks → Gotify on all 3 plans. |
|
||
| **Postgres archiving** (WAL-G → Garage S3 → JottaCloud) | CT 113 main pg17 (9 DBs incl. owui/miniflux) ✓, CT 104 immich_postgres ✓ (fixed today). Hourly staleness monitor on CT 103 with SSH+psql probe. |
|
||
| **Image-update notifications** (Diun @ CT 109) | Watches CT 104 docker (67 images), Mondays 08:00 → Gotify. CT 109/113/101 own containers not watched (future work). |
|
||
| **Config-to-git** (Phase 3) | Per-host scripts: CT 102 AdGuard, CT 103 Backrest. Central `sync-config-repos` on PVE 03:15 covers CT 103/104/109/113 stacks (auto-discovers CT 104+109 stacks). HAOS gap: Git Pull addon needs UI install. |
|
||
| **Offsite personal data** (Phase 1a rclone sync) | Saturdays 01:00 on CT 103. Immich originals → JottaCloud Photos; UNAS personal → JottaCloud UNAS. Last run 2026-05-27 13h32m / 160k files. |
|
||
| **Gotify apps** | Diun (#1), Backrest (#2), WAL-G monitor (#3), Gitea digest (#4) |
|
||
| **systemd timers active** | CT 103: walg-staleness (hourly). nuc: gitea-digest (08:00), vm100-watchdog (1m), unas-nfs-watchdog (15m), protect-haos-kvm (5m), homelab-health (1h). |
|
||
| **Cron jobs active** | nuc 03:15 sync-config-repos. CT 103 Sat 01:00 unas-sync, 02:30 walg-offsite-sync, 03:00 ct103-conf-backup. CT 102 daily adguard-conf-backup. |
|
||
|
||
### Recent ops debt cleared (2026-05-28)
|
||
- Vaultwarden data migrated CT 104 local → UNAS (was unrecoverable on disk loss)
|
||
- Backrest false-positive: CT 103 → CT 113 SSH keys never deployed; fixed + retry added to monitor
|
||
- Immich WAL-G push broken for 42h (garage container on wrong Docker network post-incident) — fixed
|
||
- Diun, Backrest hooks, WAL-G monitor, Gitea digest deployed
|
||
- Alloy enabled on CT 103
|
||
- Stale CT 110/111/112 references removed from `sync-config-repos`
|
||
|
||
### Still open
|
||
- HAOS Git Pull addon (UI-only install)
|
||
- Vaultwarden Borg passphrase storage (UI-only — Vaultwarden web UI)
|
||
- n8n-flows issues #1, #3, #4, #7, #8, #9, #10
|
||
- Diun second instance for CT 109/113/101 own containers
|
||
- DAVx5 → HonorOS `org.tasks` battery-whitelist (user phone, not server)
|
||
- Shared-password rotation (`tapirnase` reuse — deferred)
|
||
|
||
## Open / roadmap
|
||
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
|
||
expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
|
||
(in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`,
|
||
197 tools, Network App API key, `http` transport; local API fully working).
|
||
**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
|
||
Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
|
||
no Zoraxy stream proxy needed. Plus: env → secret vault. Wire Open WebUI OIDC
|
||
via Pocket-ID once Bifrost auth is settled.
|
||
|
||
**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
|
||
auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
|
||
on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed).
|
||
Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest.
|
||
|
||
**Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens
|
||
oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`,
|
||
no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so
|
||
untrusted appliances can't reach the server/Proxmox subnet. Complications to
|
||
design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and
|
||
**TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't
|
||
honour UniFi VLAN tags natively — segmentation needs a plan for the wired
|
||
trunk through the D-Link and the repeater's bridge mode.
|
||
|
||
### Observability LXC (planned — scoped from this session's incidents)
|
||
**Why it's now a priority:** the WAL-G archiver was hung **silently for
|
||
~13 h** (`failed_count=0`, zero base backups) and would never have been
|
||
noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring
|
||
must target *exactly these silent-failure classes*.
|
||
- **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104
|
||
hosts everything, so the monitor must survive/alert when 104 is down.
|
||
- **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager
|
||
→ **ntfy `homelab-ai`** (already the alert channel).
|
||
- **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter
|
||
×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for
|
||
Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor
|
||
token), and a **custom WAL-G/archiver textfile collector**:
|
||
`pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog,
|
||
and `wal-g backup-list` newest-base age — per PG instance.
|
||
- **Alerts (priority order, derived from real incidents):**
|
||
1. WAL archiver stalled (last_archived age > 15 m) or newest base backup
|
||
> 26 h, any PG instance.
|
||
2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency.
|
||
3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under
|
||
`/mnt/pve/unas` (catches a regression of the tier-1 rule).
|
||
4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
|
||
5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d.
|
||
|
||
## Topology — at a glance
|
||
|
||
### Physical / CT layout
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Host["Proxmox host · 192.168.1.20 · Intel Core Ultra 7 155H · 64 GiB"]
|
||
direction TB
|
||
haos["VM 100 · haos<br/>(.60)<br/>Home Assistant"]
|
||
shepard["CT 101 · shepard<br/>(.49)<br/>Shepard product stack"]
|
||
dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
|
||
backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
|
||
docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
|
||
nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
|
||
zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
|
||
obs["CT 109 · ops<br/>(.8)<br/>Prometheus · Grafana · Loki · Alloy · pve-exporter"]
|
||
id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
|
||
dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
|
||
db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
|
||
end
|
||
UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3")]
|
||
UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
|
||
Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])
|
||
|
||
UDM <--> Host
|
||
UDM <--> Inet
|
||
UNAS <--> shepard
|
||
UNAS <--> backrest
|
||
UNAS <--> docker104
|
||
UNAS <--> nc
|
||
UNAS <--> dev
|
||
|
||
classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
|
||
class db planned
|
||
```
|
||
|
||
### Auth plane — Pocket-ID is the universal IdP
|
||
|
||
Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
user(["fkrebs · browser / VS Code / Claude"]) --> zx[Zoraxy<br/>CT 108]
|
||
zx --> coder[Coder · CT 111]
|
||
zx --> gitea[Gitea · CT 111]
|
||
zx --> nc2[Nextcloud · CT 105]
|
||
zx --> immich[Immich · CT 104]
|
||
zx --> owui[Open WebUI · CT 104]
|
||
zx --> n8n[n8n · CT 104]
|
||
zx --> bifrost[Bifrost · CT 104]
|
||
zx --> vw[Vaultwarden · CT 104]
|
||
coder --> pid[(Pocket-ID<br/>CT 110)]
|
||
gitea --> pid
|
||
nc2 --> pid
|
||
immich --> pid
|
||
owui -.->|OIDC not yet wired| pid
|
||
n8n --> pid
|
||
bifrost -.->|VK auth, not OIDC| pid
|
||
vw -.->|via Tinyauth<br/>when CT 109 lands| pid
|
||
classDef pending stroke-dasharray:4 4,color:#888
|
||
class vw pending
|
||
```
|
||
|
||
### MCP plane — Bifrost host, child workers
|
||
|
||
Bifrost on CT 104 is the MCP host; ~30 child MCP servers run on the same docker network (`ai-internal`). The old mcp-gateway FastAPI/DinD container was decommissioned 2026-05-26.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
client[["Claude Code / Cursor / Open WebUI"]]
|
||
gw["Bifrost<br/>(CT 104)<br/>ai.nuclide.systems/mcp"]
|
||
client -- Bearer VK --> gw
|
||
subgraph "CT 104 · ai-internal docker net"
|
||
direction TB
|
||
coderm[coder-mcp]
|
||
immm[mcp-immich]
|
||
n8nm[mcp-n8n]
|
||
fetch[mcp-fetch]
|
||
time[mcp-time]
|
||
cw[mcp-crawl4ai]
|
||
seq[mcp-sequential-thinking]
|
||
ham[home-assistant-mcp]
|
||
kr[kroki-mcp]
|
||
others[...18 more]
|
||
end
|
||
gw --> coderm
|
||
gw --> immm
|
||
gw --> n8nm
|
||
gw --> fetch
|
||
gw --> time
|
||
gw --> cw
|
||
gw --> seq
|
||
gw --> ham
|
||
gw --> kr
|
||
gw --> others
|
||
coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]
|
||
ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]
|
||
```
|
||
|
||
### Data plane — what lives where
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Tier1["Tier-1 / latency-sensitive · local NVMe"]
|
||
pid_d[Pocket-ID sqlite · CT 110]
|
||
immich_pg[Immich Postgres · CT 104]
|
||
shared_pg[shared-postgres · CT 104]
|
||
coder_pg[coder-db · CT 111]
|
||
gitea_pg[gitea-db · CT 111]
|
||
garage["Garage S3 · CT 104<br/>(moved off NFS 2026-05-19)"]
|
||
n8n_local["n8n data · CT 104<br/>(reverted from NFS 2026-05-19)"]
|
||
end
|
||
subgraph Bulk["Bulk · UNAS NFSv3"]
|
||
media["Immich media, Paperless docs,<br/>arr-stack media, audiobooks"]
|
||
coder_homes[Coder workspace homes · /mnt/pve/unas/services/coder]
|
||
gitea_data[Gitea repos · /mnt/pve/unas/services/gitea]
|
||
vw_data["Vaultwarden data<br/>(tier-1 leak — plan to move local)"]
|
||
end
|
||
subgraph BulkCIFS["Bulk · UNAS CIFS (Nextcloud only)"]
|
||
nc_data[Nextcloud user files]
|
||
end
|
||
subgraph Backup["Off-host backup"]
|
||
jottacloud[(jottacloud<br/>via Backrest)]
|
||
end
|
||
Tier1 -. WAL-G .-> garage
|
||
Bulk -. only media/data-dir .-> jottacloud
|
||
classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc
|
||
class vw_data,jottacloud gap
|
||
```
|
||
|
||
The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See `stacks/storage.md` for the verified state and the cleanup TODO list.
|