fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: MCP gateway git/gitlab/paper-search; CT 109 plan; new backrest/db/arcane docs

Browse Source 
- mcp-gateway.md: 26 servers (added git, gitlab, paper-search catalog bridge);
  document --pass-environment requirement for env var passthrough in DinD bridges
- ct-inventory.md + homelab-architecture.md: CT 109 IP conflict, CT 113 live
- proxmox-state.md §16: Loki, sshwifty, Alloy, IP conflict note, sidecar table
  expanded to CT 103/111/113; build order updated
- infra/portmap.md: CT 113 db section; QNAP TS-251D (Klipper/Mainsail)
- services/backrest.md: full backup strategy doc (new)
- services/databases.md: CT 113 postgres/WAL-G/pgAdmin doc (new)
- services/arcane.md: Docker management IDE doc (new)
- CHANGELOG.md: 2026-05-22 entry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Florian Krebs
2026-05-22 09:35:07 +02:00
parent 0def5c0be9
commit 0d83f83bc0
9 changed files with 693 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/mcp-gateway.md
+102 -94
View File
@@ -1,120 +1,128 @@
# MCP Gateway v2 — Operational
Finished implementation of the DinD MCP gateway (Option A). Supersedes the
"Phase 1" stub described in `mcp-gateway-requirements.md`.
DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP servers (mix of static, spawned, and Docker-catalog-bridged). Public via Zoraxy at `https://mcp.nuclide.systems` (→ `192.168.1.40:8080`).
## What it is
## Architecture
`ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`, public via Zoraxy at
`https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). Single OAuth-gated
entrypoint fronting several MCP servers.
## Architecture (as built)
- **Spawn model**: Docker-in-Docker. Each spawnable server runs as
`mcp-<name>` on the **`ai-internal`** network with **no host ports**
(fixes the old `8000:8000` collision — servers talk over internal DNS).
- **Transport bridge**: generic streaming reverse proxy
`https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`.
Transport-agnostic — works for streamable-HTTP (`/mcp`) and legacy SSE
(`/sse`+`/messages`). Registered LAST so specific routes win.
- **Stack**: `ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`.
- **Spawn model**: Docker-in-Docker. Each spawnable server runs as `mcp-<name>` on the `ai-internal` network with no host ports — servers talk over internal DNS.
- **Transport bridge**: generic streaming reverse proxy `https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`. Transport-agnostic (streamable-HTTP `/mcp` and legacy SSE `/sse` + `/messages`). Registered last so specific routes win.
- **Auth**: Pocket ID. Two paths off the same OIDC app:
- **MCP clients / API / proxy** — `Bearer` token, validated via OIDC
userinfo (300s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728
`/.well-known/oauth-protected-resource` for discovery.
- **Browser Web UI** — full Authorization-Code flow with an httpOnly
session cookie: unauthenticated `/ui` → `302 /login` → Pocket ID
`authorize` (CSRF `state` cookie) → `/sso/callback` (state-checked
code→token exchange) sets the `mcp_session` cookie and shows the page +
a copyable Bearer token for MCP clients. `/logout` clears it. The UI and
`/api/*` accept **either** the cookie or a Bearer header; the `/{srv}/`
proxy stays Bearer-only. `/health` + the well-known metadata are open.
- **Secrets**: all per-server OAuth client secrets come from `ai/.env`
(`env_file: ../.env`), read via `os.environ` — none hardcoded in source.
- **MCP clients / API / proxy** — `Bearer` token, validated via OIDC userinfo (300 s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728 `/.well-known/oauth-protected-resource` for discovery.
- **Browser Web UI** — Authorization-Code flow with httpOnly session cookie. `/ui` and `/api/*` accept cookie or Bearer; `/{srv}/` stays Bearer-only.
- **Secrets**: per-server OAuth client secrets live in `ai/.env` (read via `os.environ`), never hardcoded.
## Server registry (`SERVERS` in server.py)
## Server inventory (as of 2026-05-22)
| name | kind | upstream | notes |
26 servers configured in `mcp-gateway/config.json`. **Status: 25 healthy, 1 disabled.**
| Group | Server | Kind | Notes |
|---|---|---|---|
| comfyui | static | `http://comfyui-mcp:8000` | runs as its own stack (`ai/comfyui-mcp.yml`); gateway only proxies |
| nextcloud | spawn | `mcp-nextcloud:8000` | `ghcr.io/cbcoutinho/nextcloud-mcp-server` streamable-http |
| crawl4ai | spawn | `mcp-crawl4ai:11235` | `unclecode/crawl4ai` |
| mermaid | spawn | `mcp-mermaid:8000` | `node:20-slim` + npx mcp-mermaid (validated working) |
| markitdown | spawn | `mcp-markitdown:8000` | `ghcr.io/astral-sh/uv` + uvx markitdown-mcp |
| papersearch | spawn | `mcp-papersearch:8000` | python + uv + mcp-proxy → paper_search_mcp |
| **dev** | context7 | catalog | `mcp/context7` |
| | fetch | spawn | `mcp-server-fetch` via uvx |
| | git | catalog | `mcp/git` — Docker MCP Catalog; `mcp-git` bridge + DinD child |
| | gitlab | catalog | `mcp/gitlab` — `GITLAB_API_URL=https://gitlab.dlr.de/api/v4`; requires `--pass-environment` in bridge cmd |
| | kroki | static | `kroki-mcp:8000` (own stack) |
| | markitdown | catalog | `mcp/markitdown` |
| | sequential-thinking | catalog | `mcp/sequentialthinking` |
| | time | spawn | `mcp-server-time` |
| | docling | static | `docling-mcp:8000` |
| | coder | static | `coder-mcp:8000` |
| | shepard | static | `https://shepard.nuclide.systems/v2/mcp` |
| **research** | crawl4ai | spawn | `unclecode/crawl4ai` |
| | paper-search | catalog | `mcp/paper-search` — Docker MCP Catalog; replaces old `python:3.12-slim` inline approach |
| | searxng | spawn | `isokoliuk/mcp-searxng` |
| | wikipedia-mcp | catalog | |
| | youtube-transcript | catalog | `mcp/youtube-transcript` |
| **personal** | bluesky | static | `ariel-mcp:8000` |
| | gotify | spawn | |
| | home-assistant | static | `http://192.168.1.60:9583/private_...` (HA add-on) |
| | immich | spawn | |
| | memos | spawn | uses `MEMOS_TOKEN` from `.env` |
| | n8n | static | `https://n8n.nuclide.systems/mcp-server/http` |
| | nextcloud | spawn | ❌ **disabled** — SSL CERTIFICATE_VERIFY_FAILED contacting `https://nc.nuclide.systems` (self-signed). `"enabled": false` set 2026-05-21. |
| | unifi | spawn | |
| **image** | comfyui | static | `comfyui-mcp:8000` (own stack, see [comfyui.md](comfyui.md)) |
| **storage** | upload-artifact | static | `upload-artifact-mcp:8000` |
## Verified
> **Catalog bridge env var passthrough**: `mcp-proxy`'s `stdio_client` uses a sanitized env (`get_default_environment()`), stripping custom vars. For catalog servers that need env vars (e.g. `gitlab`), the bridge command **must** include `--pass-environment` so Docker CLI can resolve `-e KEY` passthrough from the bridge container's env.
- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth).
- `/mcp.json`, `/<server>/mcp` → 401 without token; 401 with bad token
(userinfo path); correct `WWW-Authenticate`.
- Upstream path proven: gateway → `comfyui-mcp` `/mcp` initialize → 200;
gateway → spawned `mcp-mermaid` `/mcp` → 200 (spawn pattern + proxy).
- Route precedence correct (specific routes beat the catch-all proxy).
## Not yet exercised (needs a real token / operational bring-up)
- **Token-authenticated end-to-end** requires an interactive Pocket ID login.
Get a token: open `https://mcp.nuclide.systems/sso/callback` via the OAuth
authorize flow (or any Pocket ID token with scopes `openid,mcp`); the
callback page prints the access token.
- **Bring up the 4 remaining spawnable servers** (nextcloud/crawl4ai/
markitdown/papersearch): `POST /api/servers/<name>/start` (bearer) or
`POST /api/servers/bulk/start`. Each may need a transport/flag tweak on
first run — mermaid is the proven reference.
> **Re-enabling nextcloud**: needs either (a) a CA bundle in the container that trusts the Zoraxy/internal cert chain, or (b) skip-verify env var passed to the upstream client, or (c) point `NEXTCLOUD_HOST` at the internal Nextcloud LAN URL on `ai-internal`. Tracked as an open issue.
## Client registration
- **LobeChat**: Settings → Skills → Skill Store → Custom → Import JSON, e.g.
```json
{ "mcpServers": { "comfyui": { "type": "http",
"url": "https://mcp.nuclide.systems/comfyui/mcp",
"auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
```
(Or, for ComfyUI specifically, connect LobeChat straight to
`http://comfyui-mcp:8000/mcp` internally — see `comfyui-mcp.md` — no token.)
- **Claude.ai**: add `https://mcp.nuclide.systems/<server>/mcp` as a custom
connector; it will discover auth via the protected-resource metadata.
### Easy path — `/mcp-config` download
## Ops
- `cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build`
- Logs: `docker logs mcp-gateway`; UI: `https://mcp.nuclide.systems/ui`
(Pocket-ID-gated; `/login` to sign in).
- Config overlay persisted in `ai/mcp-gateway/config.json`.
- **UI check (Playwright)** — the live UI is auth-gated, so the check
serves the local template + mocks `/api/servers`, screenshots, and
asserts cards render:
```
uv run --with playwright -- python -m playwright install chromium # once
uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png
```
## Client config download — `/mcp-config` (added 2026-05-21)
Instead of asking the user to paste a bearer token into per-server entries
in `~/.claude/settings.json`, the gateway now hands back a ready-to-use
config blob for the signed-in user:
The gateway hands back a ready-to-use config blob for the signed-in user:
```
GET https://mcp.nuclide.systems/mcp-config?format=claude # default
GET https://mcp.nuclide.systems/mcp-config?format=cursor
GET https://mcp.nuclide.systems/mcp-config?format=raw # token + server URLs only
GET https://mcp.nuclide.systems/mcp-config?format=raw # token + URLs
```
- Requires the user's Pocket-ID session cookie OR an existing Bearer token (303 → `/login` if neither).
- Mints a fresh long-lived gateway token (`mcp_<urlsafe>`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:<format>` so it's distinguishable from OAuth-issued tokens (and revocable).
- Returns a JSON file with `Content-Disposition: attachment` so the browser saves it.
- Requires Pocket-ID session cookie or existing Bearer (303 → `/login` otherwise).
- Mints a fresh long-lived gateway token (`mcp_<urlsafe>`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:<format>` so it's distinguishable from OAuth-issued tokens and revocable.
- Returns JSON with `Content-Disposition: attachment`.
### Claude Code workflow
**Claude Code workflow:**
1. Sign in at `https://mcp.nuclide.systems`.
2. Click **Claude config** in the top nav. Browser downloads `nuclide-mcp.claude.json`.
2. Click **Claude config** in the top nav → browser downloads `nuclide-mcp.claude.json`.
3. Merge the `mcpServers` block into `~/.claude/settings.json`. Restart Claude Code.
The downloaded file has 23 entries, one per spawnable MCP server, with `type: http` + `url: https://mcp.nuclide.systems/<server>/mcp` + an `Authorization: Bearer ...` header. No further per-server configuration needed.
The downloaded file has one entry per enabled MCP server, with `type: http` + `url: https://mcp.nuclide.systems/<server>/mcp` + `Authorization: Bearer ...` header. No per-server config needed.
### Rotation
Each download mints a NEW token. Old tokens remain valid until you remove them from `gateway_tokens.json` (manual prune; future: admin UI button).
### Manual
### Server list reflected in the config
The gateway populates from `mcp-gateway/config.json` `mcpServers` keys — exactly the set running at the time of download. New servers added later require re-downloading the config.
**LobeChat** — Settings → Skills → Skill Store → Custom → Import JSON:
```json
{ "mcpServers": { "comfyui": { "type": "http",
"url": "https://mcp.nuclide.systems/comfyui/mcp",
"auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
```
(For ComfyUI specifically, LobeChat can also connect straight to `http://comfyui-mcp:8000/mcp` internally — see [comfyui.md](comfyui.md) — no token.)
**Claude.ai** — add `https://mcp.nuclide.systems/<server>/mcp` as a custom connector; auth is discovered via the protected-resource metadata.
### Token rotation
Each `/mcp-config` download mints a NEW token. Old tokens stay valid until removed from `gateway_tokens.json` (manual prune; future: admin UI button).
## Ops
```bash
cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build
docker logs mcp-gateway
```
UI: `https://mcp.nuclide.systems/ui` (Pocket-ID-gated; `/login` to sign in).
Config: `mcp-gateway/config.json` (live config, persisted). Timestamped backups created on every edit (`config.json.bak.YYYYMMDD-HHMMSS`).
### Add or enable a server
1. Edit `mcp-gateway/config.json` — add an `mcpServers.<name>` entry (or set `enabled: true`).
2. Restart the gateway: `docker restart mcp-gateway`.
3. Spawn servers come up on demand via `POST /api/servers/<name>/start` (bearer); or `POST /api/servers/bulk/start` for all.
### Disable a server
Set `mcpServers.<name>.enabled = false` and add `disabled_reason`. Stop+rm the container (`docker stop mcp-<name> && docker rm mcp-<name>`); the gateway will not re-spawn it.
### UI check (Playwright)
Live UI is auth-gated, so the local check serves the template + mocks `/api/servers`:
```bash
uv run --with playwright -- python -m playwright install chromium # once
uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png
```
## Verified end-to-end
- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth).
- `/mcp.json`, `/<server>/mcp` → 401 without token; correct `WWW-Authenticate`.
- Static upstreams (`bluesky`, `coder`, `comfyui`, `docling`, `kroki`, `upload-artifact`) — endpoints reachable from inside `ai-internal` (406/405 to plain GET, which is the correct MCP behavior).
- All spawn/catalog containers up (`docker ps mcp-*`): 16/17 (nextcloud disabled; git/gitlab/paper-search added 2026-05-22).