76 lines
3.4 KiB
Markdown
76 lines
3.4 KiB
Markdown
# ComfyUI → LobeChat via MCP
|
|
|
|
Image generation (FLUX.1-schnell GGUF on the Intel Arc iGPU) exposed to LobeChat
|
|
as an MCP tool. Chosen over LobeChat's native ComfyUI provider because that
|
|
provider is hardcoded to non-GGUF nodes and is not configurable without forking
|
|
LobeChat (see `docs/mcp-gateway-requirements.md` research notes).
|
|
|
|
## Components
|
|
|
|
- **`ai/mcp-servers/comfyui/`** — purpose-built MCP server (async queue edition)
|
|
- `server.py` — FastMCP, streamable-HTTP on `:8000` at `/mcp`. Six tools:
|
|
- `generate_image(prompt, width, height, steps, seed)` — txt2img; returns `job_id` immediately
|
|
- `img2img(prompt, images, strength, steps, seed)` — img2img for one or more images; one job per image; images can be URLs, base64 data URIs, or `job:<id>` references
|
|
- `get_job_status(job_ids)` — returns status text + inline PNG for completed jobs
|
|
- `list_queue(limit)` — lists all jobs in the in-process registry
|
|
- `cancel_job(job_id)` — cancels queued/running job (user must confirm first)
|
|
- `list_recent_images(n)` — shows n most recent completed images for reference/chaining
|
|
- Background polling thread updates job state every 3 s via ComfyUI `/history`.
|
|
- `flux-gguf-api.json` — txt2img workflow; nodes 4=prompt, 6=size, 8=steps/seed.
|
|
- `flux-img2img-api.json` — img2img workflow; node 11=LoadImage, 12=VAEEncode, 8=KSampler with denoise.
|
|
- `Dockerfile`, `requirements.txt` (`mcp[cli]`, `httpx`).
|
|
- **`ai/comfyui-mcp.yml`** — compose; container `comfyui-mcp` on
|
|
`shared_backend` (reaches `comfyui:8188`; reachable by `lobehub`, which is on
|
|
`shared_backend`). Host port `18003:8000` for testing/Zoraxy.
|
|
|
|
## Async workflow
|
|
|
|
```
|
|
generate_image("a red apple") → "Job submitted: txt-1a8bbeda" (< 1 s)
|
|
[ComfyUI rendering... ~90-300 s]
|
|
get_job_status(["txt-1a8bbeda"]) → status text + inline PNG image
|
|
|
|
# Chain: use previous output as img2img input (no bytes through LLM)
|
|
img2img("add a blue bowl", images=["job:txt-1a8bbeda"], strength=0.6)
|
|
→ "img-2b9ccefa"
|
|
```
|
|
|
|
## Verified
|
|
|
|
- All 6 tools discovered via MCP `tools/list`.
|
|
- `generate_image` returns in < 0.1 s (non-blocking); job shows in `list_queue` as "running".
|
|
- Previous end-to-end: `generate_image` → `get_job_status` → inline PNG ~94 s bare, ~160 s via MCP.
|
|
- LobeChat v2.1.58 renders MCP `image` blocks (uploads to Garage S3 → inline ``).
|
|
|
|
## Known characteristics / limitations
|
|
|
|
- **In-memory registry**: job state is lost on container restart. Resubmit if needed.
|
|
- **No auth** on the MCP server (internal `shared_backend` only). Host port
|
|
18003 is LAN-exposed and unauthenticated — fine for a trusted homelab LAN.
|
|
- ~90-300 s/image latency; use `get_job_status()` to poll — never block waiting.
|
|
|
|
## Final hookup (manual — LobeChat UI/DB, no server-mode config path)
|
|
|
|
LobeChat → **Settings → Skills (Tools) → Skill Store → Custom → Import JSON**:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"comfyui-flux": {
|
|
"type": "http",
|
|
"url": "http://comfyui-mcp:8000/mcp"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Then enable the `comfyui-flux` skill in an agent/chat and ask the model to
|
|
"generate an image of …". Images appear inline in the conversation.
|
|
|
|
## Ops
|
|
|
|
- Build/deploy: `cd /opt/stacks/ai && docker compose -f comfyui-mcp.yml up -d --build`
|
|
- Logs: `docker logs comfyui-mcp`
|
|
- The workflow is the single source of truth in `flux-gguf-api.json`; keep it in
|
|
sync with `ai/comfyui/workflows/flux-schnell-api.json` if the graph changes.
|