565 lines
16 KiB
Markdown
565 lines
16 KiB
Markdown
# Cloud GPU Extension — Scaleway L40S
|
||
|
||
Seamless on-demand GPU (FLUX.1-dev, video, multi-user) via a Scaleway L40S instance.
|
||
The instance auto-starts on first request and shuts down after 45 min idle.
|
||
Everything personal stays on the NUC.
|
||
|
||
---
|
||
|
||
## Pricing (current as of May 2026)
|
||
|
||
| Resource | Rate | Notes |
|
||
|---|---|---|
|
||
| L40S-1-48G compute | **€1.40/hour** | PAR-2, billed per minute |
|
||
| Block volume 200 GB | **€16/month** | SSD, keeps models across restarts |
|
||
| Flexible IP | €0.004/hour | Static IP for WireGuard endpoint |
|
||
| Snapshots | €0.000044/GB/h | Only needed for image backups |
|
||
|
||
### Realistic monthly cost
|
||
|
||
| Usage pattern | Compute | Storage | Total |
|
||
|---|---|---|---|
|
||
| Weekend sessions (8 h/week) | €45 | €16 | **~€61/month** |
|
||
| Daily 1–2 h | €63–126 | €16 | **~€79–142/month** |
|
||
| Heavy (4 h/day) | €168 | €16 | **~€184/month** |
|
||
| Always-on (don't) | €1,008 | €16 | €1,024/month |
|
||
|
||
The on-demand proxy below makes "daily 1–2 h" the natural default — you just click generate, it starts automatically.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
NUC (home, always-on) Scaleway PAR-2 (on demand)
|
||
───────────────────────────── ───────────────────────────────
|
||
LobeChat ──▶ gpu-proxy:8190 ─── wg1 ──▶ ComfyUI :8188
|
||
comfyui-mcp ──▶ (Docker) Ollama :11434 (optional)
|
||
│
|
||
├─ start/stop via Scaleway API
|
||
└─ idle watchdog (45 min → stop)
|
||
```
|
||
|
||
`gpu-proxy` is a small Docker service that:
|
||
1. Forwards requests to the GPU instance
|
||
2. Auto-starts the Scaleway instance if it's stopped (cold-start ~90 s)
|
||
3. Shuts it down after 45 min with no traffic
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
```bash
|
||
# Scaleway CLI
|
||
curl -s https://raw.githubusercontent.com/scaleway/scaleway-cli/master/scripts/get.sh | sh
|
||
scw init # enter API key + project ID
|
||
```
|
||
|
||
---
|
||
|
||
## Step 1 — Persistent block volume (models live here)
|
||
|
||
```bash
|
||
# Create 200 GB SSD volume in PAR-2
|
||
scw block volume create name=nuclide-gpu-models size=200GB zone=fr-par-2
|
||
# Note the volume ID: vol-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||
```
|
||
|
||
Models are kept on this volume. The compute instance can be deleted and recreated freely.
|
||
|
||
---
|
||
|
||
## Step 2 — Create the L40S instance with cloud-init
|
||
|
||
Save as `/opt/stacks/scripts/gpu-cloud-init.yaml`:
|
||
|
||
```yaml
|
||
#cloud-config
|
||
package_update: true
|
||
packages:
|
||
- wireguard-tools
|
||
- docker.io
|
||
- docker-compose-v2
|
||
- nvidia-driver-545
|
||
- nvidia-container-toolkit
|
||
|
||
write_files:
|
||
- path: /etc/wireguard/wg0.conf
|
||
permissions: '0600'
|
||
content: |
|
||
[Interface]
|
||
Address = 10.200.0.1/24
|
||
ListenPort = 51820
|
||
PrivateKey = SCALEWAY_WG_PRIVKEY
|
||
|
||
[Peer]
|
||
# NUC
|
||
PublicKey = NUC_WG_PUBKEY
|
||
AllowedIPs = 10.200.0.2/32
|
||
PersistentKeepalive = 25
|
||
|
||
- path: /opt/gpu/docker-compose.yml
|
||
content: |
|
||
services:
|
||
comfyui:
|
||
image: yanwk/comfyui-boot:cu124
|
||
container_name: comfyui
|
||
ports:
|
||
- "10.200.0.1:8188:8188"
|
||
volumes:
|
||
- /mnt/models:/root/ComfyUI/models
|
||
- comfyui_config:/root/ComfyUI
|
||
environment:
|
||
- CLI_ARGS=--listen 0.0.0.0
|
||
deploy:
|
||
resources:
|
||
reservations:
|
||
devices:
|
||
- driver: nvidia
|
||
count: all
|
||
capabilities: [gpu]
|
||
restart: unless-stopped
|
||
|
||
ollama:
|
||
image: ollama/ollama:latest
|
||
container_name: ollama
|
||
ports:
|
||
- "10.200.0.1:11434:11434"
|
||
volumes:
|
||
- /mnt/models/ollama:/root/.ollama
|
||
deploy:
|
||
resources:
|
||
reservations:
|
||
devices:
|
||
- driver: nvidia
|
||
count: all
|
||
capabilities: [gpu]
|
||
restart: unless-stopped
|
||
|
||
volumes:
|
||
comfyui_config:
|
||
|
||
runcmd:
|
||
# Mount the models volume (will be /dev/sdb or similar)
|
||
- mkdir -p /mnt/models
|
||
- |
|
||
DISK=$(lsblk -ndo NAME,SIZE | awk '$2=="200G"{print "/dev/"$1}' | head -1)
|
||
if [ -n "$DISK" ]; then
|
||
blkid "$DISK" || mkfs.ext4 "$DISK"
|
||
echo "$DISK /mnt/models ext4 defaults 0 2" >> /etc/fstab
|
||
mount "$DISK" /mnt/models
|
||
fi
|
||
- mkdir -p /mnt/models/ollama
|
||
# Enable WireGuard
|
||
- systemctl enable --now wg-quick@wg0
|
||
# Configure nvidia-container-toolkit
|
||
- nvidia-ctk runtime configure --runtime=docker
|
||
- systemctl restart docker
|
||
# Start GPU services
|
||
- cd /opt/gpu && docker compose up -d
|
||
```
|
||
|
||
**Before using this file**, replace:
|
||
- `SCALEWAY_WG_PRIVKEY` → output of `wg genkey` (run on Scaleway instance side first)
|
||
- `NUC_WG_PUBKEY` → output of `cat /etc/wireguard/nuc_wg_pub` (see Step 3)
|
||
|
||
Launch the instance:
|
||
|
||
```bash
|
||
# Generate WireGuard keys first (do this on the NUC)
|
||
wg genkey | tee /etc/wireguard/scaleway_wg_priv | wg pubkey > /etc/wireguard/scaleway_wg_pub
|
||
wg genkey | tee /etc/wireguard/nuc_wg_priv | wg pubkey > /etc/wireguard/nuc_wg_pub
|
||
|
||
# Fill in cloud-init.yaml with the keys, then:
|
||
INSTANCE_ID=$(scw instance server create \
|
||
type=L40S-1-48G \
|
||
image=ubuntu_jammy_gpu \
|
||
zone=fr-par-2 \
|
||
name=nuclide-gpu \
|
||
cloud-init=@/opt/stacks/scripts/gpu-cloud-init.yaml \
|
||
output=json | jq -r '.id')
|
||
|
||
echo "Instance ID: $INSTANCE_ID"
|
||
|
||
# Attach the models volume
|
||
scw instance server attach-volume \
|
||
server-id=$INSTANCE_ID \
|
||
volume-id=vol-xxxxxxxx \
|
||
zone=fr-par-2
|
||
|
||
# Allocate a Flexible IP (static IP that survives instance restarts)
|
||
FLEXIP_ID=$(scw instance ip create zone=fr-par-2 output=json | jq -r '.id')
|
||
scw instance server attach-flexible-ip \
|
||
server-id=$INSTANCE_ID \
|
||
ip-id=$FLEXIP_ID \
|
||
zone=fr-par-2
|
||
|
||
FLEXIP=$(scw instance ip get $FLEXIP_ID zone=fr-par-2 output=json | jq -r '.address')
|
||
echo "Scaleway public IP: $FLEXIP"
|
||
```
|
||
|
||
---
|
||
|
||
## Step 3 — WireGuard on the NUC
|
||
|
||
```bash
|
||
# /etc/wireguard/wg1.conf (separate from any existing VPN tunnel)
|
||
cat > /etc/wireguard/wg1.conf << EOF
|
||
[Interface]
|
||
Address = 10.200.0.2/24
|
||
PrivateKey = $(cat /etc/wireguard/nuc_wg_priv)
|
||
|
||
[Peer]
|
||
# Scaleway GPU
|
||
PublicKey = $(cat /etc/wireguard/scaleway_wg_pub)
|
||
Endpoint = ${FLEXIP}:51820
|
||
AllowedIPs = 10.200.0.1/32
|
||
PersistentKeepalive = 25
|
||
EOF
|
||
|
||
chmod 600 /etc/wireguard/wg1.conf
|
||
systemctl enable --now wg-quick@wg1
|
||
```
|
||
|
||
Test when the instance is running:
|
||
```bash
|
||
ping 10.200.0.1 # WireGuard tunnel
|
||
curl http://10.200.0.1:8188 # ComfyUI
|
||
```
|
||
|
||
---
|
||
|
||
## Step 4 — gpu-proxy (seamless on-demand startup)
|
||
|
||
This Docker service runs on the NUC. It proxies to the GPU instance and auto-starts/stops it.
|
||
|
||
### `ai/gpu-proxy/docker-compose.yml`
|
||
|
||
```yaml
|
||
services:
|
||
gpu-proxy:
|
||
build: .
|
||
container_name: gpu-proxy
|
||
environment:
|
||
- SCW_SECRET_KEY=${SCW_SECRET_KEY}
|
||
- SCW_PROJECT_ID=${SCW_PROJECT_ID}
|
||
- SCW_INSTANCE_ID=${SCW_INSTANCE_ID}
|
||
- SCW_ZONE=fr-par-2
|
||
- GPU_HOST=10.200.0.1
|
||
- COMFYUI_PORT=8188
|
||
- OLLAMA_PORT=11434
|
||
- IDLE_TIMEOUT=2700 # 45 min
|
||
ports:
|
||
- "8190:8190" # ComfyUI proxy
|
||
- "8191:8191" # Ollama proxy
|
||
restart: unless-stopped
|
||
networks:
|
||
- shared_backend
|
||
|
||
networks:
|
||
shared_backend:
|
||
external: true
|
||
```
|
||
|
||
### `ai/gpu-proxy/proxy.py`
|
||
|
||
```python
|
||
"""
|
||
On-demand GPU proxy. Auto-starts the Scaleway instance on first request,
|
||
shuts it down after IDLE_TIMEOUT seconds of inactivity.
|
||
"""
|
||
import asyncio, os, time, httpx, subprocess
|
||
from fastapi import FastAPI, Request
|
||
from fastapi.responses import StreamingResponse, JSONResponse
|
||
|
||
app = FastAPI()
|
||
|
||
SCW_KEY = os.environ["SCW_SECRET_KEY"]
|
||
SCW_PROJECT = os.environ["SCW_PROJECT_ID"]
|
||
INSTANCE_ID = os.environ["SCW_INSTANCE_ID"]
|
||
ZONE = os.environ.get("SCW_ZONE", "fr-par-2")
|
||
GPU_HOST = os.environ.get("GPU_HOST", "10.200.0.1")
|
||
COMFYUI_PORT = int(os.environ.get("COMFYUI_PORT", 8188))
|
||
OLLAMA_PORT = int(os.environ.get("OLLAMA_PORT", 11434))
|
||
IDLE_TIMEOUT = int(os.environ.get("IDLE_TIMEOUT", 2700))
|
||
|
||
SCW_API = f"https://api.scaleway.com/instance/v1/zones/{ZONE}"
|
||
HEADERS = {"X-Auth-Token": SCW_KEY, "Content-Type": "application/json"}
|
||
|
||
_state = {"last_activity": 0.0, "starting": False, "up": False}
|
||
_lock = asyncio.Lock()
|
||
|
||
|
||
async def _scw(method: str, path: str, **kwargs):
|
||
async with httpx.AsyncClient() as c:
|
||
r = await c.request(method, f"{SCW_API}{path}", headers=HEADERS, **kwargs)
|
||
r.raise_for_status()
|
||
return r.json()
|
||
|
||
|
||
async def _instance_state() -> str:
|
||
d = await _scw("GET", f"/servers/{INSTANCE_ID}")
|
||
return d["server"]["state"] # running | stopped | stopping | starting
|
||
|
||
|
||
async def _start_instance():
|
||
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweron"})
|
||
|
||
|
||
async def _stop_instance():
|
||
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweroff"})
|
||
|
||
|
||
async def _wait_ready(host: str, port: int, timeout=180) -> bool:
|
||
deadline = time.time() + timeout
|
||
while time.time() < deadline:
|
||
try:
|
||
async with httpx.AsyncClient(timeout=3) as c:
|
||
await c.get(f"http://{host}:{port}/")
|
||
return True
|
||
except Exception:
|
||
await asyncio.sleep(5)
|
||
return False
|
||
|
||
|
||
async def ensure_up(port: int) -> bool:
|
||
async with _lock:
|
||
if _state["up"]:
|
||
_state["last_activity"] = time.time()
|
||
return True
|
||
if _state["starting"]:
|
||
return False # caller will retry
|
||
_state["starting"] = True
|
||
|
||
try:
|
||
state = await _instance_state()
|
||
if state != "running":
|
||
print(f"[gpu-proxy] instance {state} → starting")
|
||
await _start_instance()
|
||
# wait for API to report running
|
||
for _ in range(60):
|
||
await asyncio.sleep(5)
|
||
if await _instance_state() == "running":
|
||
break
|
||
# wait for service to respond
|
||
if await _wait_ready(GPU_HOST, port):
|
||
async with _lock:
|
||
_state["up"] = True
|
||
_state["last_activity"] = time.time()
|
||
print("[gpu-proxy] GPU instance ready")
|
||
return True
|
||
return False
|
||
finally:
|
||
async with _lock:
|
||
_state["starting"] = False
|
||
|
||
|
||
async def idle_watchdog():
|
||
while True:
|
||
await asyncio.sleep(60)
|
||
async with _lock:
|
||
if not _state["up"]:
|
||
continue
|
||
idle = time.time() - _state["last_activity"]
|
||
if idle > IDLE_TIMEOUT:
|
||
print(f"[gpu-proxy] idle {idle:.0f}s → stopping instance")
|
||
try:
|
||
await _stop_instance()
|
||
async with _lock:
|
||
_state["up"] = False
|
||
_state["last_activity"] = 0.0
|
||
except Exception as e:
|
||
print(f"[gpu-proxy] stop error: {e}")
|
||
|
||
|
||
@app.on_event("startup")
|
||
async def startup():
|
||
asyncio.create_task(idle_watchdog())
|
||
# Probe: is the instance already running from a previous session?
|
||
try:
|
||
if await _instance_state() == "running":
|
||
if await _wait_ready(GPU_HOST, COMFYUI_PORT, timeout=10):
|
||
async with _lock:
|
||
_state["up"] = True
|
||
_state["last_activity"] = time.time()
|
||
print("[gpu-proxy] GPU already up on startup")
|
||
except Exception:
|
||
pass
|
||
|
||
|
||
async def _proxy(request: Request, host: str, port: int):
|
||
_state["last_activity"] = time.time()
|
||
ready = await ensure_up(port)
|
||
if not ready:
|
||
# Starting up — keep trying for up to 3 min
|
||
for _ in range(36):
|
||
await asyncio.sleep(5)
|
||
if _state["up"]:
|
||
break
|
||
else:
|
||
return JSONResponse({"error": "GPU instance failed to start"}, 503)
|
||
|
||
url = f"http://{host}:{port}{request.url.path}"
|
||
if request.url.query:
|
||
url += f"?{request.url.query}"
|
||
body = await request.body()
|
||
client = httpx.AsyncClient(timeout=httpx.Timeout(None, connect=10))
|
||
req = client.build_request(request.method, url,
|
||
headers={k: v for k, v in request.headers.items()
|
||
if k.lower() not in {"host", "content-length"}},
|
||
content=body)
|
||
try:
|
||
resp = await client.send(req, stream=True)
|
||
except httpx.ConnectError:
|
||
await client.aclose()
|
||
return JSONResponse({"error": "GPU not reachable"}, 502)
|
||
|
||
async def stream():
|
||
async for chunk in resp.aiter_raw():
|
||
yield chunk
|
||
await resp.aclose()
|
||
await client.aclose()
|
||
|
||
return StreamingResponse(stream(), status_code=resp.status_code,
|
||
headers=dict(resp.headers),
|
||
media_type=resp.headers.get("content-type"))
|
||
|
||
|
||
@app.api_route("/status", methods=["GET"])
|
||
async def status():
|
||
try:
|
||
scw_state = await _instance_state()
|
||
except Exception as e:
|
||
scw_state = f"error: {e}"
|
||
return {"instance": scw_state, "proxy_up": _state["up"],
|
||
"idle_s": int(time.time() - _state["last_activity"]) if _state["up"] else None}
|
||
|
||
|
||
@app.api_route("/{path:path}", methods=["GET","POST","PUT","DELETE","OPTIONS","PATCH"])
|
||
async def comfyui_proxy(request: Request, path: str):
|
||
return await _proxy(request, GPU_HOST, COMFYUI_PORT)
|
||
```
|
||
|
||
### `ai/gpu-proxy/Dockerfile`
|
||
|
||
```dockerfile
|
||
FROM python:3.12-slim
|
||
RUN pip install fastapi uvicorn httpx
|
||
COPY proxy.py /app/proxy.py
|
||
WORKDIR /app
|
||
EXPOSE 8190 8191
|
||
CMD ["uvicorn", "proxy:app", "--host", "0.0.0.0", "--port", "8190"]
|
||
```
|
||
|
||
---
|
||
|
||
## Step 5 — Wire up NUC services
|
||
|
||
Add to `ai/.env`:
|
||
```bash
|
||
SCW_SECRET_KEY=<your-scaleway-api-key>
|
||
SCW_PROJECT_ID=<your-project-id>
|
||
SCW_INSTANCE_ID=<instance-id-from-step-2>
|
||
```
|
||
|
||
In `ai/lobehub.yml`:
|
||
```yaml
|
||
- 'COMFYUI_BASE_URL=http://gpu-proxy:8190'
|
||
```
|
||
|
||
In `ai/mcp-gateway/server.py` (SERVERS dict):
|
||
```python
|
||
"comfyui": {
|
||
"static": True,
|
||
"upstream": "http://gpu-proxy:8190",
|
||
"group": "image",
|
||
},
|
||
```
|
||
|
||
For Immich ML (optional), add to `immich/docker-compose.yml`:
|
||
```yaml
|
||
environment:
|
||
- IMMICH_MACHINE_LEARNING_URL=http://gpu-proxy:8192 # add a third port for ML
|
||
```
|
||
|
||
---
|
||
|
||
## Step 6 — Deploy
|
||
|
||
```bash
|
||
# Deploy the proxy
|
||
cd /opt/stacks/ai/gpu-proxy
|
||
docker compose up -d
|
||
|
||
# Restart LobeChat and MCP gateway to pick up new env
|
||
cd /opt/stacks/ai
|
||
docker compose -f lobehub.yml up -d --force-recreate lobe
|
||
cd mcp-gateway && docker compose up -d --force-recreate mcp-gateway
|
||
```
|
||
|
||
### What the user experience looks like
|
||
|
||
1. Open LobeChat, generate an image → first request triggers auto-start
|
||
2. "Starting GPU…" — ComfyUI returns a brief wait (up to 90 s cold start)
|
||
3. Images generate normally; the proxy keeps the instance alive
|
||
4. After 45 min with no requests → instance stops automatically
|
||
5. `/status` endpoint on port 8190: `{"instance":"stopped","proxy_up":false}`
|
||
|
||
---
|
||
|
||
## Quick-start / stop scripts
|
||
|
||
```bash
|
||
# /usr/local/bin/nuclide-gpu (NUC helper)
|
||
#!/bin/bash
|
||
ZONE=fr-par-2
|
||
ID=$(docker exec gpu-proxy env | grep SCW_INSTANCE_ID | cut -d= -f2)
|
||
case "$1" in
|
||
start) scw instance server action action=poweron server-id=$ID zone=$ZONE ;;
|
||
stop) scw instance server action action=poweroff server-id=$ID zone=$ZONE ;;
|
||
status) curl -s http://localhost:8190/status | python3 -m json.tool ;;
|
||
*) echo "Usage: nuclide-gpu start|stop|status" ;;
|
||
esac
|
||
```
|
||
|
||
---
|
||
|
||
## Scaleway firewall (security groups)
|
||
|
||
On the Scaleway instance, only expose WireGuard. All services bind to `10.200.0.1` (WireGuard IP):
|
||
|
||
```bash
|
||
ufw default deny incoming
|
||
ufw allow 51820/udp # WireGuard from anywhere
|
||
ufw allow from 10.200.0.0/24 # full access over tunnel
|
||
ufw enable
|
||
```
|
||
|
||
---
|
||
|
||
## Model storage layout (on the 200 GB volume)
|
||
|
||
```
|
||
/mnt/models/
|
||
├── checkpoints/ # FLUX.1-dev, SDXL, etc.
|
||
├── vae/
|
||
├── loras/
|
||
├── controlnet/
|
||
├── ollama/ # Ollama model blobs
|
||
└── upscale_models/
|
||
```
|
||
|
||
FLUX.1-dev (FP16): ~24 GB
|
||
FLUX.1-schnell (NF4): ~8 GB
|
||
Ollama llama3:70b (Q4): ~40 GB — fits alongside FLUX on 200 GB with room for LoRAs.
|
||
|
||
To pre-download models after first boot:
|
||
```bash
|
||
ssh root@10.200.0.1 # via WireGuard
|
||
docker exec comfyui python3 -c "
|
||
from huggingface_hub import hf_hub_download
|
||
hf_hub_download('black-forest-labs/FLUX.1-dev', 'flux1-dev.safetensors',
|
||
local_dir='/root/ComfyUI/models/checkpoints')
|
||
"
|
||
```
|