docs: reconcile sunset of LiteLLM external, mcp-gateway, and LobeChat

- Bifrost replaces LiteLLM at ai.nuclide.systems (CT 104:14003); LiteLLM
  now internal-only backend at :14000
- mcp-gateway (FastAPI/DinD, mcp.nuclide.systems) decommissioned 2026-05-26;
  MCP now served by Bifrost at ai.nuclide.systems/mcp (29 clients, ~760 tools)
- LobeChat decommissioned 2026-05-26; chat.nuclide.systems now Open WebUI
  (CT 104:14002, stack ai/open-webui.yml)
- Update portmap, volumes, services-overview, homelab-architecture, zoraxy,
  mcp-servers, databases, README accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-26 13:35:29 +02:00
parent 550b54085c
commit 6c401f2376
8 changed files with 119 additions and 158 deletions
+44 -67
View File
@@ -136,9 +136,10 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
(see n8n note under Operational rules).
- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
**universal SSO/IdP for the entire ecosystem** — effectively every service
authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud,
Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity
source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.
authenticates via PocketID OIDC (LiteLLM, Nextcloud, Coder, Gitea,
Vaultwarden, etc.). Single sign-on everywhere; one identity source to
secure/audit. Vaultwarden remains the lone gap as of 2026-05-20.
Open WebUI OIDC not yet wired (as of 2026-05-26).
## Docker networks
`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
@@ -147,59 +148,37 @@ plus per-stack: `arr-stack_default`,
The MCP gateway bridges `ai-internal` + `shared_backend`.
## AI / Agent platform (the core, `/opt/stacks/ai`)
- **LiteLLM** (`ai.nuclide.systems`, :14000) — model gateway. ~28 curated
models (provider allowlist), SAIA-terminal **failovers** + `num_retries`,
Redis completion cache, Gemini €10/30d `provider_budget_config`. Config:
`ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`).
- **Bifrost** (`ai.nuclide.systems`, :14003) — unified LLM gateway + MCP host.
Stack `ai/bifrost/`. Auth via `sk-bf-` VKs. LLM inference at `/v1`; MCP at
`/mcp` (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to
LiteLLM internally.
- **LiteLLM** (:14000, **internal only** — no Zoraxy route) — model gateway,
backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal
**failovers** + `num_retries`, Redis completion cache, Gemini €10/30d
`provider_budget_config`. Config: `ai/litellm-config/config.yaml` (+ DB
overlay `LiteLLM_Config`). Reachable as `http://litellm:4000` inside Docker.
- **Open WebUI** (`chat.nuclide.systems`, :14002) — chat UI. Stack
`ai/open-webui.yml`. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not
yet wired as of 2026-05-26).
- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
**model syncer/optimizer ONLY** (curation/allowlist → DB, `OPENAI_MODEL_LIST`
→ `.env`, recreate lobe, model-health → ntfy). One-shot container.
- **mcp-gateway / Agent Operator** (`mcp.nuclide.systems`, :8080) — **all
things MCP**. DinD-spawns MCP servers on `ai-internal`, OAuth-gated streaming
reverse proxy `/<server>/mcp` + `/group/<g>/mcp`, multi-group, health-watcher,
per-server tools/list cache. **Deep health-check (2026-05-19):** beyond
shallow liveness, the watcher drives a real MCP session (`initialize` →
`tools/list` → optional `health_probe_tool` no-side-effect call) against
**every** server incl. spawned ones, recording `ok|degraded|down` in
`/api/servers`. Usage-aware: a server used within 5 min isn't probed
(traffic = health); idle servers probed every 15 min; probe self-excludes
via `X-Health-Probe`. Deep failures are status-only (no restart loops);
shallow-liveness auto-restart stays static-only. First run caught the
invisible class: `memos` degraded (search_memo → DNS `Errno -2`, the
ai-internal isolation bug), plus `context7`/`crawl4ai`/`markitdown` down
and `nextcloud` probe false-positive (probe lacks per-user NC creds —
candidate for `health_check:false`). **Semantic tool retrieval**: groups with >25
aggregated tools expose just `find_tools`/`call_tool` meta-tools (LiteLLM
embedding index, cosine top-k, two-phase schema promotion) instead of
flooding clients with hundreds of schemas. Group members are proxied via the
gateway's own path (`_mcp_gw`) so `exact_upstream` servers (home-assistant,
n8n) aggregate correctly. **MCP→LobeChat registration** (migrated here
from syncstack), and the **Agent Operator**: cron+event agents
(`/api/agents/{name}/run` webhook), LLM↔MCP executor, **Agent Creator**
(AI-drafts agent specs; overridable meta-prompt), prompt export. Config:
`mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents).
- **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to
`.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions),
**coder** (replaced daytona 2026-05-20), fetch, sequential-thinking, ntfy, memos,
docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui,
bluesky (brianellin, rate-limit pending). Groups incl. `shepard-mcp`.
Gaps to close (community MCPs exist): gitea, paperless, karakeep, vaultwarden,
proxmox-ve, audiobookshelf. See [`services/mcp-gateway.md`](mcp-gateway.md).
**model syncer/optimizer ONLY** (curation/allowlist → DB, model-health → ntfy).
One-shot container.
- ~~**mcp-gateway**~~ — **DECOMMISSIONED 2026-05-26.** Was FastAPI/DinD on
`:8080` at `mcp.nuclide.systems`. Compose renamed `.DECOMMISSIONED-2026-05-26`.
MCP is now served by Bifrost at `https://ai.nuclide.systems/mcp`.
- **MCP servers** (Bifrost-managed, formerly gateway-managed): time, home-assistant,
kroki, coder, fetch, sequential-thinking, gotify, memos, docling, markitdown,
crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky, obsidian,
immich, n8n, nextcloud, paperless, unifi, upload-artifact, gitea, proxmox,
shepard, git, gitlab, youtube-transcript, searxng (30 servers). 29 clients,
~760 tools. See [`services/mcp-servers.md`](mcp-servers.md).
- **Coder** (`dev.nuclide.systems`, CT 111) — replaced Daytona as the sandbox /
dev-environment runtime. Templates: `python-uv` (persistent, GPU passthrough,
baked LiteLLM env + Claude Code), `mcp-sandbox` (ephemeral, sci stack
pre-baked). See [`services/dev-environment.md`](dev-environment.md).
- **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins +
agents in `lobechat` DB (lobe-postgres). Agents scriptable via
`scripts/lobe_agent.py`. Native QStash scheduler NOT functional here →
scheduling lives in the gateway Agent Operator. **Auth = Better Auth**
(not NextAuth) SSO via PocketID (`generic-oidc`, client `26f3c26b`, PKCE
S256). The `state_not_found` callback bug was fixed by setting
`AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems`; verified 2026-05-19
(sign-in yields a correct PocketID authorize URL; callback now validates
with `oAuth_code_missing`, not `state_not_found`). Definitive proof =
one real browser login. NB: `/oidc/auth` is LobeChat-*as*-OIDC-provider
(desktop/mobile), distinct from this SSO-client path.
- ~~**LobeChat**~~ — **DECOMMISSIONED 2026-05-26.** Was at `chat.nuclide.systems`
(:14001). Compose renamed `.DECOMMISSIONED-lobehub-2026-05-26.yml`. Replaced
by Open WebUI.
- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
meta-search).
@@ -227,8 +206,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`.
**pgloader data-only of ALL tables** into the app-built schema (exclude
only the app's own migration-tracking table), app role temp-SUPERUSER for
the load (FK/trigger disable) then reverted.
- **lobe-postgres** (pg17/ParadeDB) — LobeChat. **lobe-redis** — LiteLLM cache
+ LobeChat. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). **lobe-redis** — LiteLLM cache (LobeChat decommissioned). **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
`:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
@@ -299,7 +277,7 @@ Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).
## Operational rules / conventions
- **Separation**: `syncstack` = models; `mcp-gateway` = all MCP/agents.
- **Separation**: `syncstack` = model curation; `bifrost` = MCP + LLM gateway (mcp-gateway decommissioned 2026-05-26).
- **Self-host first**: external/cloud only when no self-hosted form exists.
- **Pinned NTP source standard** — all network gear/infra should point at
the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
@@ -328,8 +306,7 @@ Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
- Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename
once the agent-platform identity firms up.
- Naming debt: "syncstack" has outgrown its name — rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.
## Open / roadmap
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
@@ -338,8 +315,8 @@ expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
197 tools, Network App API key, `http` transport; local API fully working).
**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep
`plugins`!); env → secret vault.
no Zoraxy stream proxy needed. Plus: env → secret vault. Wire Open WebUI OIDC
via Pocket-ID once Bifrost auth is settled.
**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
@@ -426,31 +403,31 @@ flowchart LR
zx --> gitea[Gitea · CT 111]
zx --> nc2[Nextcloud · CT 105]
zx --> immich[Immich · CT 104]
zx --> lobe[LobeChat · CT 104]
zx --> owui[Open WebUI · CT 104]
zx --> n8n[n8n · CT 104]
zx --> mcp[MCP gateway · CT 104]
zx --> bifrost[Bifrost · CT 104]
zx --> vw[Vaultwarden · CT 104]
coder --> pid[(Pocket-ID<br/>CT 110)]
gitea --> pid
nc2 --> pid
immich --> pid
lobe --> pid
owui -.->|OIDC not yet wired| pid
n8n --> pid
mcp --> pid
bifrost -.->|VK auth, not OIDC| pid
vw -.->|via Tinyauth<br/>when CT 109 lands| pid
classDef pending stroke-dasharray:4 4,color:#888
class vw pending
```
### MCP plane — gateway control, child workers
### MCP plane — Bifrost host, child workers
The MCP gateway on CT 104 is the OIDC-gated front; ~23 child MCP servers run on the same docker network (`ai-internal`). When CT 109 "ops" is built, the gateway moves to CT 109 and reaches CT 104's docker daemon via a socket-proxy; the children stay put.
Bifrost on CT 104 is the MCP host; ~30 child MCP servers run on the same docker network (`ai-internal`). The old mcp-gateway FastAPI/DinD container was decommissioned 2026-05-26.
```mermaid
flowchart LR
client[["Claude Code / Cursor / LobeChat"]]
gw["MCP gateway<br/>(CT 104 → CT 109 ops when 109 lands)<br/>mcp.nuclide.systems"]
client -- Bearer / OIDC --> gw
client[["Claude Code / Cursor / Open WebUI"]]
gw["Bifrost<br/>(CT 104)<br/>ai.nuclide.systems/mcp"]
client -- Bearer VK --> gw
subgraph "CT 104 · ai-internal docker net"
direction TB
coderm[coder-mcp]