fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

reorg: split into infra/ services/ history/ ideas/

Browse Source
This commit is contained in:
Florian Krebs
2026-05-20 23:25:20 +02:00
parent 04d54ac9b0
commit 99f93b585f
14 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/cloud-gpu.md
+564
View File
@@ -0,0 +1,564 @@
# Cloud GPU Extension — Scaleway L40S
Seamless on-demand GPU (FLUX.1-dev, video, multi-user) via a Scaleway L40S instance.
The instance auto-starts on first request and shuts down after 45 min idle.
Everything personal stays on the NUC.
---
## Pricing (current as of May 2026)
| Resource | Rate | Notes |
|---|---|---|
| L40S-1-48G compute | **€1.40/hour** | PAR-2, billed per minute |
| Block volume 200 GB | **€16/month** | SSD, keeps models across restarts |
| Flexible IP | €0.004/hour | Static IP for WireGuard endpoint |
| Snapshots | €0.000044/GB/h | Only needed for image backups |
### Realistic monthly cost
| Usage pattern | Compute | Storage | Total |
|---|---|---|---|
| Weekend sessions (8 h/week) | €45 | €16 | **~€61/month** |
| Daily 1–2 h | €63–126 | €16 | **~€79–142/month** |
| Heavy (4 h/day) | €168 | €16 | **~€184/month** |
| Always-on (don't) | €1,008 | €16 | €1,024/month |
The on-demand proxy below makes "daily 1–2 h" the natural default — you just click generate, it starts automatically.
---
## Architecture
```
NUC (home, always-on) Scaleway PAR-2 (on demand)
───────────────────────────── ───────────────────────────────
LobeChat ──▶ gpu-proxy:8190 ─── wg1 ──▶ ComfyUI :8188
comfyui-mcp ──▶ (Docker) Ollama :11434 (optional)
│
├─ start/stop via Scaleway API
└─ idle watchdog (45 min → stop)
```
`gpu-proxy` is a small Docker service that:
1. Forwards requests to the GPU instance
2. Auto-starts the Scaleway instance if it's stopped (cold-start ~90 s)
3. Shuts it down after 45 min with no traffic
---
## Prerequisites
```bash
# Scaleway CLI
curl -s https://raw.githubusercontent.com/scaleway/scaleway-cli/master/scripts/get.sh | sh
scw init # enter API key + project ID
```
---
## Step 1 — Persistent block volume (models live here)
```bash
# Create 200 GB SSD volume in PAR-2
scw block volume create name=nuclide-gpu-models size=200GB zone=fr-par-2
# Note the volume ID: vol-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```
Models are kept on this volume. The compute instance can be deleted and recreated freely.
---
## Step 2 — Create the L40S instance with cloud-init
Save as `/opt/stacks/scripts/gpu-cloud-init.yaml`:
```yaml
#cloud-config
package_update: true
packages:
- wireguard-tools
- docker.io
- docker-compose-v2
- nvidia-driver-545
- nvidia-container-toolkit
write_files:
- path: /etc/wireguard/wg0.conf
permissions: '0600'
content: |
[Interface]
Address = 10.200.0.1/24
ListenPort = 51820
PrivateKey = SCALEWAY_WG_PRIVKEY
[Peer]
# NUC
PublicKey = NUC_WG_PUBKEY
AllowedIPs = 10.200.0.2/32
PersistentKeepalive = 25
- path: /opt/gpu/docker-compose.yml
content: |
services:
comfyui:
image: yanwk/comfyui-boot:cu124
container_name: comfyui
ports:
- "10.200.0.1:8188:8188"
volumes:
- /mnt/models:/root/ComfyUI/models
- comfyui_config:/root/ComfyUI
environment:
- CLI_ARGS=--listen 0.0.0.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- "10.200.0.1:11434:11434"
volumes:
- /mnt/models/ollama:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped
volumes:
comfyui_config:
runcmd:
# Mount the models volume (will be /dev/sdb or similar)
- mkdir -p /mnt/models
- |
DISK=$(lsblk -ndo NAME,SIZE | awk '$2=="200G"{print "/dev/"$1}' | head -1)
if [ -n "$DISK" ]; then
blkid "$DISK" || mkfs.ext4 "$DISK"
echo "$DISK /mnt/models ext4 defaults 0 2" >> /etc/fstab
mount "$DISK" /mnt/models
fi
- mkdir -p /mnt/models/ollama
# Enable WireGuard
- systemctl enable --now wg-quick@wg0
# Configure nvidia-container-toolkit
- nvidia-ctk runtime configure --runtime=docker
- systemctl restart docker
# Start GPU services
- cd /opt/gpu && docker compose up -d
```
**Before using this file**, replace:
- `SCALEWAY_WG_PRIVKEY` → output of `wg genkey` (run on Scaleway instance side first)
- `NUC_WG_PUBKEY` → output of `cat /etc/wireguard/nuc_wg_pub` (see Step 3)
Launch the instance:
```bash
# Generate WireGuard keys first (do this on the NUC)
wg genkey | tee /etc/wireguard/scaleway_wg_priv | wg pubkey > /etc/wireguard/scaleway_wg_pub
wg genkey | tee /etc/wireguard/nuc_wg_priv | wg pubkey > /etc/wireguard/nuc_wg_pub
# Fill in cloud-init.yaml with the keys, then:
INSTANCE_ID=$(scw instance server create \
type=L40S-1-48G \
image=ubuntu_jammy_gpu \
zone=fr-par-2 \
name=nuclide-gpu \
cloud-init=@/opt/stacks/scripts/gpu-cloud-init.yaml \
output=json | jq -r '.id')
echo "Instance ID: $INSTANCE_ID"
# Attach the models volume
scw instance server attach-volume \
server-id=$INSTANCE_ID \
volume-id=vol-xxxxxxxx \
zone=fr-par-2
# Allocate a Flexible IP (static IP that survives instance restarts)
FLEXIP_ID=$(scw instance ip create zone=fr-par-2 output=json | jq -r '.id')
scw instance server attach-flexible-ip \
server-id=$INSTANCE_ID \
ip-id=$FLEXIP_ID \
zone=fr-par-2
FLEXIP=$(scw instance ip get $FLEXIP_ID zone=fr-par-2 output=json | jq -r '.address')
echo "Scaleway public IP: $FLEXIP"
```
---
## Step 3 — WireGuard on the NUC
```bash
# /etc/wireguard/wg1.conf (separate from any existing VPN tunnel)
cat > /etc/wireguard/wg1.conf << EOF
[Interface]
Address = 10.200.0.2/24
PrivateKey = $(cat /etc/wireguard/nuc_wg_priv)
[Peer]
# Scaleway GPU
PublicKey = $(cat /etc/wireguard/scaleway_wg_pub)
Endpoint = ${FLEXIP}:51820
AllowedIPs = 10.200.0.1/32
PersistentKeepalive = 25
EOF
chmod 600 /etc/wireguard/wg1.conf
systemctl enable --now wg-quick@wg1
```
Test when the instance is running:
```bash
ping 10.200.0.1 # WireGuard tunnel
curl http://10.200.0.1:8188 # ComfyUI
```
---
## Step 4 — gpu-proxy (seamless on-demand startup)
This Docker service runs on the NUC. It proxies to the GPU instance and auto-starts/stops it.
### `ai/gpu-proxy/docker-compose.yml`
```yaml
services:
gpu-proxy:
build: .
container_name: gpu-proxy
environment:
- SCW_SECRET_KEY=${SCW_SECRET_KEY}
- SCW_PROJECT_ID=${SCW_PROJECT_ID}
- SCW_INSTANCE_ID=${SCW_INSTANCE_ID}
- SCW_ZONE=fr-par-2
- GPU_HOST=10.200.0.1
- COMFYUI_PORT=8188
- OLLAMA_PORT=11434
- IDLE_TIMEOUT=2700 # 45 min
ports:
- "8190:8190" # ComfyUI proxy
- "8191:8191" # Ollama proxy
restart: unless-stopped
networks:
- shared_backend
networks:
shared_backend:
external: true
```
### `ai/gpu-proxy/proxy.py`
```python
"""
On-demand GPU proxy. Auto-starts the Scaleway instance on first request,
shuts it down after IDLE_TIMEOUT seconds of inactivity.
"""
import asyncio, os, time, httpx, subprocess
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, JSONResponse
app = FastAPI()
SCW_KEY = os.environ["SCW_SECRET_KEY"]
SCW_PROJECT = os.environ["SCW_PROJECT_ID"]
INSTANCE_ID = os.environ["SCW_INSTANCE_ID"]
ZONE = os.environ.get("SCW_ZONE", "fr-par-2")
GPU_HOST = os.environ.get("GPU_HOST", "10.200.0.1")
COMFYUI_PORT = int(os.environ.get("COMFYUI_PORT", 8188))
OLLAMA_PORT = int(os.environ.get("OLLAMA_PORT", 11434))
IDLE_TIMEOUT = int(os.environ.get("IDLE_TIMEOUT", 2700))
SCW_API = f"https://api.scaleway.com/instance/v1/zones/{ZONE}"
HEADERS = {"X-Auth-Token": SCW_KEY, "Content-Type": "application/json"}
_state = {"last_activity": 0.0, "starting": False, "up": False}
_lock = asyncio.Lock()
async def _scw(method: str, path: str, **kwargs):
async with httpx.AsyncClient() as c:
r = await c.request(method, f"{SCW_API}{path}", headers=HEADERS, **kwargs)
r.raise_for_status()
return r.json()
async def _instance_state() -> str:
d = await _scw("GET", f"/servers/{INSTANCE_ID}")
return d["server"]["state"] # running | stopped | stopping | starting
async def _start_instance():
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweron"})
async def _stop_instance():
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweroff"})
async def _wait_ready(host: str, port: int, timeout=180) -> bool:
deadline = time.time() + timeout
while time.time() < deadline:
try:
async with httpx.AsyncClient(timeout=3) as c:
await c.get(f"http://{host}:{port}/")
return True
except Exception:
await asyncio.sleep(5)
return False
async def ensure_up(port: int) -> bool:
async with _lock:
if _state["up"]:
_state["last_activity"] = time.time()
return True
if _state["starting"]:
return False # caller will retry
_state["starting"] = True
try:
state = await _instance_state()
if state != "running":
print(f"[gpu-proxy] instance {state} → starting")
await _start_instance()
# wait for API to report running
for _ in range(60):
await asyncio.sleep(5)
if await _instance_state() == "running":
break
# wait for service to respond
if await _wait_ready(GPU_HOST, port):
async with _lock:
_state["up"] = True
_state["last_activity"] = time.time()
print("[gpu-proxy] GPU instance ready")
return True
return False
finally:
async with _lock:
_state["starting"] = False
async def idle_watchdog():
while True:
await asyncio.sleep(60)
async with _lock:
if not _state["up"]:
continue
idle = time.time() - _state["last_activity"]
if idle > IDLE_TIMEOUT:
print(f"[gpu-proxy] idle {idle:.0f}s → stopping instance")
try:
await _stop_instance()
async with _lock:
_state["up"] = False
_state["last_activity"] = 0.0
except Exception as e:
print(f"[gpu-proxy] stop error: {e}")
@app.on_event("startup")
async def startup():
asyncio.create_task(idle_watchdog())
# Probe: is the instance already running from a previous session?
try:
if await _instance_state() == "running":
if await _wait_ready(GPU_HOST, COMFYUI_PORT, timeout=10):
async with _lock:
_state["up"] = True
_state["last_activity"] = time.time()
print("[gpu-proxy] GPU already up on startup")
except Exception:
pass
async def _proxy(request: Request, host: str, port: int):
_state["last_activity"] = time.time()
ready = await ensure_up(port)
if not ready:
# Starting up — keep trying for up to 3 min
for _ in range(36):
await asyncio.sleep(5)
if _state["up"]:
break
else:
return JSONResponse({"error": "GPU instance failed to start"}, 503)
url = f"http://{host}:{port}{request.url.path}"
if request.url.query:
url += f"?{request.url.query}"
body = await request.body()
client = httpx.AsyncClient(timeout=httpx.Timeout(None, connect=10))
req = client.build_request(request.method, url,
headers={k: v for k, v in request.headers.items()
if k.lower() not in {"host", "content-length"}},
content=body)
try:
resp = await client.send(req, stream=True)
except httpx.ConnectError:
await client.aclose()
return JSONResponse({"error": "GPU not reachable"}, 502)
async def stream():
async for chunk in resp.aiter_raw():
yield chunk
await resp.aclose()
await client.aclose()
return StreamingResponse(stream(), status_code=resp.status_code,
headers=dict(resp.headers),
media_type=resp.headers.get("content-type"))
@app.api_route("/status", methods=["GET"])
async def status():
try:
scw_state = await _instance_state()
except Exception as e:
scw_state = f"error: {e}"
return {"instance": scw_state, "proxy_up": _state["up"],
"idle_s": int(time.time() - _state["last_activity"]) if _state["up"] else None}
@app.api_route("/{path:path}", methods=["GET","POST","PUT","DELETE","OPTIONS","PATCH"])
async def comfyui_proxy(request: Request, path: str):
return await _proxy(request, GPU_HOST, COMFYUI_PORT)
```
### `ai/gpu-proxy/Dockerfile`
```dockerfile
FROM python:3.12-slim
RUN pip install fastapi uvicorn httpx
COPY proxy.py /app/proxy.py
WORKDIR /app
EXPOSE 8190 8191
CMD ["uvicorn", "proxy:app", "--host", "0.0.0.0", "--port", "8190"]
```
---
## Step 5 — Wire up NUC services
Add to `ai/.env`:
```bash
SCW_SECRET_KEY=<your-scaleway-api-key>
SCW_PROJECT_ID=<your-project-id>
SCW_INSTANCE_ID=<instance-id-from-step-2>
```
In `ai/lobehub.yml`:
```yaml
- 'COMFYUI_BASE_URL=http://gpu-proxy:8190'
```
In `ai/mcp-gateway/server.py` (SERVERS dict):
```python
"comfyui": {
"static": True,
"upstream": "http://gpu-proxy:8190",
"group": "image",
},
```
For Immich ML (optional), add to `immich/docker-compose.yml`:
```yaml
environment:
- IMMICH_MACHINE_LEARNING_URL=http://gpu-proxy:8192 # add a third port for ML
```
---
## Step 6 — Deploy
```bash
# Deploy the proxy
cd /opt/stacks/ai/gpu-proxy
docker compose up -d
# Restart LobeChat and MCP gateway to pick up new env
cd /opt/stacks/ai
docker compose -f lobehub.yml up -d --force-recreate lobe
cd mcp-gateway && docker compose up -d --force-recreate mcp-gateway
```
### What the user experience looks like
1. Open LobeChat, generate an image → first request triggers auto-start
2. "Starting GPU…" — ComfyUI returns a brief wait (up to 90 s cold start)
3. Images generate normally; the proxy keeps the instance alive
4. After 45 min with no requests → instance stops automatically
5. `/status` endpoint on port 8190: `{"instance":"stopped","proxy_up":false}`
---
## Quick-start / stop scripts
```bash
# /usr/local/bin/nuclide-gpu (NUC helper)
#!/bin/bash
ZONE=fr-par-2
ID=$(docker exec gpu-proxy env | grep SCW_INSTANCE_ID | cut -d= -f2)
case "$1" in
start) scw instance server action action=poweron server-id=$ID zone=$ZONE ;;
stop) scw instance server action action=poweroff server-id=$ID zone=$ZONE ;;
status) curl -s http://localhost:8190/status | python3 -m json.tool ;;
*) echo "Usage: nuclide-gpu start|stop|status" ;;
esac
```
---
## Scaleway firewall (security groups)
On the Scaleway instance, only expose WireGuard. All services bind to `10.200.0.1` (WireGuard IP):
```bash
ufw default deny incoming
ufw allow 51820/udp # WireGuard from anywhere
ufw allow from 10.200.0.0/24 # full access over tunnel
ufw enable
```
---
## Model storage layout (on the 200 GB volume)
```
/mnt/models/
├── checkpoints/ # FLUX.1-dev, SDXL, etc.
├── vae/
├── loras/
├── controlnet/
├── ollama/ # Ollama model blobs
└── upscale_models/
```
FLUX.1-dev (FP16): ~24 GB
FLUX.1-schnell (NF4): ~8 GB
Ollama llama3:70b (Q4): ~40 GB — fits alongside FLUX on 200 GB with room for LoRAs.
To pre-download models after first boot:
```bash
ssh root@10.200.0.1 # via WireGuard
docker exec comfyui python3 -c "
from huggingface_hub import hf_hub_download
hf_hub_download('black-forest-labs/FLUX.1-dev', 'flux1-dev.safetensors',
local_dir='/root/ComfyUI/models/checkpoints')
"
```