Skip to content

# ComfyUI → LobeChat via MCP

Image generation (FLUX.1-schnell GGUF on the Intel Arc iGPU) exposed to LobeChat as an MCP tool. Chosen over LobeChat's native ComfyUI provider because that provider is hardcoded to non-GGUF nodes and is not configurable without forking LobeChat (see `docs/mcp-gateway-requirements.md` research notes).

## Components

- `ai/mcp-servers/comfyui/` — purpose-built MCP server (async queue edition)
- `server.py` — FastMCP, streamable-HTTP on `:8000` at `/mcp`. Six tools:
  - `generate_image(prompt, width, height, steps, seed)` — txt2img; returns `job_id` immediately
  - `img2img(prompt, images, strength, steps, seed)` — img2img for one or more images; one job per image; images can be URLs, base64 data URIs, or `job:<id>` references
  - `get_job_status(job_ids)` — returns status text + inline PNG for completed jobs
  - `list_queue(limit)` — lists all jobs in the in-process registry
  - `cancel_job(job_id)` — cancels queued/running job (user must confirm first)
  - `list_recent_images(n)` — shows n most recent completed images for reference/chaining
- Background polling thread updates job state every 3 s via ComfyUI `/history`.
- `flux-gguf-api.json` — txt2img workflow; nodes 4=prompt, 6=size, 8=steps/seed.
- `flux-img2img-api.json` — img2img workflow; node 11=LoadImage, 12=VAEEncode, 8=KSampler with denoise.
- `Dockerfile`, `requirements.txt` (`mcp[cli]`, `httpx`).
- `ai/comfyui-mcp.yml` — compose; container `comfyui-mcp` on `shared_backend` (reaches `comfyui:8188`; reachable by `lobehub`, which is on `shared_backend`). Host port `18003:8000` for testing/Zoraxy.

## Async workflow

    generate_image("a red apple")  →  "Job submitted: txt-1a8bbeda"   (< 1 s)
                                       [ComfyUI rendering... ~90-300 s]
    get_job_status(["txt-1a8bbeda"])  →  status text + inline PNG image

    # Chain: use previous output as img2img input (no bytes through LLM)
    img2img("add a blue bowl", images=["job:txt-1a8bbeda"], strength=0.6)
      →  "img-2b9ccefa"

## Verified

- All 6 tools discovered via MCP `tools/list`.
- `generate_image` returns in < 0.1 s (non-blocking); job shows in `list_queue` as "running".
- Previous end-to-end: `generate_image` → `get_job_status` → inline PNG ~94 s bare, ~160 s via MCP.
- LobeChat v2.1.58 renders MCP `image` blocks (uploads to Garage S3 → inline `![](url)`).

## Known characteristics / limitations

- In-memory registry: job state is lost on container restart. Resubmit if needed.
- No auth on the MCP server (internal `shared_backend` only). Host port 18003 is LAN-exposed and unauthenticated — fine for a trusted homelab LAN.
- ~90-300 s/image latency; use `get_job_status()` to poll — never block waiting.

## Final hookup (manual — LobeChat UI/DB, no server-mode config path)

LobeChat → Settings → Skills (Tools) → Skill Store → Custom → Import JSON:

    {
      "mcpServers": {
        "comfyui-flux": {
          "type": "http",
          "url": "http://comfyui-mcp:8000/mcp"
        }
      }
    }

Then enable the `comfyui-flux` skill in an agent/chat and ask the model to "generate an image of …". Images appear inline in the conversation.

## Ops

- Build/deploy: `cd /opt/stacks/ai && docker compose -f comfyui-mcp.yml up -d --build`
- Logs: `docker logs comfyui-mcp`
- The workflow is the single source of truth in `flux-gguf-api.json`; keep it in sync with `ai/comfyui/workflows/flux-schnell-api.json` if the graph changes.