reorg: split into infra/ services/ history/ ideas/

2026-05-20 23:25:20 +02:00
parent 04d54ac9b0
commit 99f93b585f
14 changed files with 0 additions and 0 deletions
@@ -0,0 +1,91 @@
+# MCP Gateway v2 — Operational
+
+Finished implementation of the DinD MCP gateway (Option A). Supersedes the
+"Phase 1" stub described in `mcp-gateway-requirements.md`.
+
+## What it is
+
+`ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`, public via Zoraxy at
+`https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). Single OAuth-gated
+entrypoint fronting several MCP servers.
+
+## Architecture (as built)
+
+- **Spawn model**: Docker-in-Docker. Each spawnable server runs as
+  `mcp-<name>` on the **`ai-internal`** network with **no host ports**
+  (fixes the old `8000:8000` collision — servers talk over internal DNS).
+- **Transport bridge**: generic streaming reverse proxy
+  `https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`.
+  Transport-agnostic — works for streamable-HTTP (`/mcp`) and legacy SSE
+  (`/sse`+`/messages`). Registered LAST so specific routes win.
+- **Auth**: Pocket ID. Two paths off the same OIDC app:
+  - **MCP clients / API / proxy** — `Bearer` token, validated via OIDC
+    userinfo (300s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728
+    `/.well-known/oauth-protected-resource` for discovery.
+  - **Browser Web UI** — full Authorization-Code flow with an httpOnly
+    session cookie: unauthenticated `/ui` → `302 /login` → Pocket ID
+    `authorize` (CSRF `state` cookie) → `/sso/callback` (state-checked
+    code→token exchange) sets the `mcp_session` cookie and shows the page +
+    a copyable Bearer token for MCP clients. `/logout` clears it. The UI and
+    `/api/*` accept **either** the cookie or a Bearer header; the `/{srv}/`
+    proxy stays Bearer-only. `/health` + the well-known metadata are open.
+- **Secrets**: all per-server OAuth client secrets come from `ai/.env`
+  (`env_file: ../.env`), read via `os.environ` — none hardcoded in source.
+
+## Server registry (`SERVERS` in server.py)
+
+| name | kind | upstream | notes |
+|---|---|---|---|
+| comfyui | static | `http://comfyui-mcp:8000` | runs as its own stack (`ai/comfyui-mcp.yml`); gateway only proxies |
+| nextcloud | spawn | `mcp-nextcloud:8000` | `ghcr.io/cbcoutinho/nextcloud-mcp-server` streamable-http |
+| crawl4ai | spawn | `mcp-crawl4ai:11235` | `unclecode/crawl4ai` |
+| mermaid | spawn | `mcp-mermaid:8000` | `node:20-slim` + npx mcp-mermaid (validated working) |
+| markitdown | spawn | `mcp-markitdown:8000` | `ghcr.io/astral-sh/uv` + uvx markitdown-mcp |
+| papersearch | spawn | `mcp-papersearch:8000` | python + uv + mcp-proxy → paper_search_mcp |
+
+## Verified
+
+- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth).
+- `/mcp.json`, `/<server>/mcp` → 401 without token; 401 with bad token
+  (userinfo path); correct `WWW-Authenticate`.
+- Upstream path proven: gateway → `comfyui-mcp` `/mcp` initialize → 200;
+  gateway → spawned `mcp-mermaid` `/mcp` → 200 (spawn pattern + proxy).
+- Route precedence correct (specific routes beat the catch-all proxy).
+
+## Not yet exercised (needs a real token / operational bring-up)
+
+- **Token-authenticated end-to-end** requires an interactive Pocket ID login.
+  Get a token: open `https://mcp.nuclide.systems/sso/callback` via the OAuth
+  authorize flow (or any Pocket ID token with scopes `openid,mcp`); the
+  callback page prints the access token.
+- **Bring up the 4 remaining spawnable servers** (nextcloud/crawl4ai/
+  markitdown/papersearch): `POST /api/servers/<name>/start` (bearer) or
+  `POST /api/servers/bulk/start`. Each may need a transport/flag tweak on
+  first run — mermaid is the proven reference.
+
+## Client registration
+
+- **LobeChat**: Settings → Skills → Skill Store → Custom → Import JSON, e.g.
+  ```json
+  { "mcpServers": { "comfyui": { "type": "http",
+    "url": "https://mcp.nuclide.systems/comfyui/mcp",
+    "auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
+  ```
+  (Or, for ComfyUI specifically, connect LobeChat straight to
+  `http://comfyui-mcp:8000/mcp` internally — see `comfyui-mcp.md` — no token.)
+- **Claude.ai**: add `https://mcp.nuclide.systems/<server>/mcp` as a custom
+  connector; it will discover auth via the protected-resource metadata.
+
+## Ops
+
+- `cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build`
+- Logs: `docker logs mcp-gateway`; UI: `https://mcp.nuclide.systems/ui`
+  (Pocket-ID-gated; `/login` to sign in).
+- Config overlay persisted in `ai/mcp-gateway/config.json`.
+- **UI check (Playwright)** — the live UI is auth-gated, so the check
+  serves the local template + mocks `/api/servers`, screenshots, and
+  asserts cards render:
+  ```
+  uv run --with playwright -- python -m playwright install chromium   # once
+  uv run --with playwright -- python ai/mcp-gateway/ui_check.py        # -> /tmp/mcp_ui.png
+  ```