reorg: split into infra/ services/ history/ ideas/
This commit is contained in:
@@ -1,207 +0,0 @@
|
||||
# AdGuard DNS Rewrite Opportunities
|
||||
|
||||
## Overview
|
||||
AdGuard DNS at `192.168.1.2` can handle internal domain resolution, eliminating need for external DNS or hosts file entries.
|
||||
|
||||
## Current Setup
|
||||
- **AdGuard URL**: `http://192.168.1.2`
|
||||
- **Status**: Running on Proxmox VM 102
|
||||
- **Homepage widget**: Configured
|
||||
|
||||
## AI Service DNS Entries
|
||||
|
||||
### Recommended DNS Rewrites
|
||||
|
||||
Add these static DNS entries in AdGuard to resolve AI services locally:
|
||||
|
||||
| Domain | IP Address | TTL | Purpose |
|
||||
|--------|------------|-----|---------|
|
||||
| `ai.nuclide.systems` | `192.168.1.40` | 300 | LiteLLM gateway |
|
||||
| `chat.nuclide.systems` | `192.168.1.40` | 300 | LobeHub chat |
|
||||
| `mcp.nuclide.systems` | `192.168.1.40` | 300 | MCP servers |
|
||||
| `litellm.nuclide.systems` | `192.168.1.40` | 300 | LiteLLM API |
|
||||
| `s3.nuclide.systems` | `192.168.1.40` | 300 | Garage S3 (Zoraxy proxy) |
|
||||
|
||||
### Benefits
|
||||
1. **No external DNS needed** - All AI services resolve internally
|
||||
2. **Failover protection** - Works even if external DNS is unreachable
|
||||
3. **Faster resolution** - Local DNS vs external lookup
|
||||
4. **Simplified client config** - Services can use domain names directly
|
||||
|
||||
## How to Add
|
||||
|
||||
### Method 1: Web UI (Recommended)
|
||||
|
||||
1. Open AdGuard: `http://192.168.1.2` (or via Zone: `http://192.168.1.2:3000`)
|
||||
2. Navigate to **DNS Settings** → **Static DNS Entries**
|
||||
3. Click **Add** for each service:
|
||||
```
|
||||
Domain: ai.nuclide.systems
|
||||
IP Address: 192.168.1.40
|
||||
TTL: 300
|
||||
```
|
||||
4. Click **Save**
|
||||
|
||||
### Method 2: API
|
||||
|
||||
```bash
|
||||
# Get session token first
|
||||
curl -c /tmp/cookies.txt -k -X POST http://192.168.1.2:3000/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username": "root", "password": "tapirnase"}'
|
||||
|
||||
# Add static DNS entry
|
||||
curl -s -k -c /tmp/cookies.txt -b /tmp/cookies.txt \
|
||||
-X POST http://192.168.1.2:3000/admin/api/staticDNS \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"domain": "ai.nuclide.systems",
|
||||
"ip": "192.168.1.40",
|
||||
"ttl": 300
|
||||
}'
|
||||
```
|
||||
|
||||
### Method 3: Auto-configure (Script)
|
||||
|
||||
Create `/opt/stacks/scripts/configure_adguard_dns.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Configure AdGuard DNS for AI stack
|
||||
|
||||
ADGUARD_HOST="192.168.1.2"
|
||||
ADGUARD_PORT="3000"
|
||||
TARGET_IP="192.168.1.40"
|
||||
|
||||
# AI service domains to add
|
||||
DOMAINS=(
|
||||
"ai.nuclide.systems"
|
||||
"chat.nuclide.systems"
|
||||
"mcp.nuclide.systems"
|
||||
"litellm.nuclide.systems"
|
||||
"s3.nuclide.systems"
|
||||
)
|
||||
|
||||
echo "🔧 Adding DNS entries to AdGuard..."
|
||||
|
||||
for domain in "${DOMAINS[@]}"; do
|
||||
echo " → $domain → $TARGET_IP"
|
||||
# Note: This is a placeholder - actual API call needed
|
||||
done
|
||||
|
||||
echo "✅ Done! Add these in AdGuard UI manually."
|
||||
```
|
||||
|
||||
## Alternative Useful DNS Entries
|
||||
|
||||
### NextCloud & Sync
|
||||
- `nc.nuclide.systems` → `192.168.1.40` (NextCloud web)
|
||||
- `sync.nuclide.systems` → Internal IP of Sync client
|
||||
|
||||
### Garage S3
|
||||
- `storage.nuclide.systems` → `192.168.1.40` (Garage internal, port 3900)
|
||||
- `s3-local.nuclide.systems` → `192.168.1.40` (S3 API for local only)
|
||||
|
||||
### Service Discovery
|
||||
- `api.nuclide.systems` → `192.168.1.40` (future API gateway)
|
||||
- `web.nuclide.systems` → `192.168.1.40` (general web services)
|
||||
|
||||
### Immich
|
||||
- `immich.nuclide.systems` → `192.168.1.40` (Immich web & API)
|
||||
- `immich-api.nuclide.systems` → `192.168.1.40:2283` (Immich API only)
|
||||
|
||||
### Paperless-ngx
|
||||
- `pp.nuclide.systems` → `192.168.1.40` (Paperless web)
|
||||
- `pp-api.nuclide.systems` → `192.168.1.40:8000` (Paperless API)
|
||||
|
||||
### Home Assistant
|
||||
- `ha.nuclide.systems` → `192.168.1.10` (Home Assistant)
|
||||
- `homeassistant.local` → `192.168.1.10` (mDNS fallback)
|
||||
|
||||
## Internal DNS Server Setup
|
||||
|
||||
### Option A: Use AdGuard as Forwarding DNS
|
||||
|
||||
Configure clients to use `192.168.1.2` as their DNS server:
|
||||
|
||||
1. Proxmox Host: Edit `/etc/resolv.conf`
|
||||
2. Docker containers: Add DNS in docker-compose.yml
|
||||
3. VMs/PFs: Configure network settings
|
||||
|
||||
### Option B: Create Custom Zone in AdGuard
|
||||
|
||||
1. AdGuard → **DNS Settings** → **Zone Management**
|
||||
2. Add zone: `nuclide.systems`
|
||||
3. Add A records for all subdomains with proper IPs
|
||||
4. Zone file format:
|
||||
```
|
||||
@ IN SOA ns1.nuclide.systems. admin.nuclide.systems. (
|
||||
1 ; Serial
|
||||
3600 ; Refresh
|
||||
1800 ; Retry
|
||||
604800 ; Expire
|
||||
86400 ) ; Minimum TTL
|
||||
|
||||
@ IN NS ns1.nuclide.systems.
|
||||
@ IN A 192.168.1.40
|
||||
ai IN A 192.168.1.40
|
||||
chat IN A 192.168.1.40
|
||||
mcp IN A 192.168.1.40
|
||||
l IN A 192.168.1.40 ; LiteLLM
|
||||
s3 IN A 192.168.1.40
|
||||
nc IN A 192.168.1.40 ; NextCloud
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
After adding DNS entries:
|
||||
|
||||
```bash
|
||||
# Test from any machine on network
|
||||
dig ai.nuclide.systems @192.168.1.2
|
||||
dig chat.nuclide.systems @192.168.1.2
|
||||
|
||||
# Should return: 192.168.1.40
|
||||
```
|
||||
|
||||
## Integration with MCP/Gateway Config
|
||||
|
||||
Update service configs to use domain names:
|
||||
|
||||
### In LiteLLM Config (`/opt/stacks/ai/litellm-config/config.yaml`)
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
proxy_base_url: https://ai.nuclide.systems
|
||||
control_plane_url: https://ai.nuclide.systems
|
||||
```
|
||||
|
||||
### In Karakeep (`.env`)
|
||||
|
||||
```bash
|
||||
OPENAI_BASE_URL=https://ai.nuclide.systems/v1
|
||||
```
|
||||
|
||||
### In LobeHub (`.env`)
|
||||
|
||||
```bash
|
||||
APP_URL=https://chat.nuclide.systems
|
||||
S3_ENDPOINT=https://s3.nuclide.systems
|
||||
```
|
||||
|
||||
## Migration Checklist
|
||||
|
||||
- [ ] Add DNS entries to AdGuard
|
||||
- [ ] Verify resolution: `dig ai.nuclide.systems`
|
||||
- [ ] Update service configs to use domains
|
||||
- [ ] Test HTTPS connectivity
|
||||
- [ ] Remove old hosts file entries (if any)
|
||||
- [ ] Document for team
|
||||
|
||||
## Notes
|
||||
|
||||
- AdGuard runs on VM 102 at `192.168.1.2`
|
||||
- All AI services run on `192.168.1.40`
|
||||
- Using same IP for all domains is intentional (single endpoint)
|
||||
- SSL certs are issued by Zoraxy (public domain validation)
|
||||
- DNS-only setup doesn't require HTTP proxy (Zoraxy still handles HTTPS)
|
||||
@@ -1,114 +0,0 @@
|
||||
# Case Study — The Nuclide Homelab, built with Claude
|
||||
|
||||
## Origin story
|
||||
|
||||
One Saturday, the owner's wife left him home alone. He got bored, subscribed to
|
||||
Claude, and started tinkering with a home server. That afternoon of boredom
|
||||
turned into the `/opt/stacks` ecosystem documented here — a ~66-container,
|
||||
~23-stack self-hosted platform with SSO, an MCP/agent gateway, GPU offload,
|
||||
and a fully audited network. This is that story, kept as a record of what a
|
||||
curiosity-driven collaboration produced.
|
||||
|
||||
> This is a personal passion project, not a work deliverable. The tone and
|
||||
> scope reflect that: depth and exploration over minimum-viable.
|
||||
|
||||
## What was built (high level)
|
||||
|
||||
- **AI/agent core** — LiteLLM model gateway (~28 curated models), an
|
||||
MCP gateway that DinD-spawns and OAuth-gates MCP servers, an Agent Operator
|
||||
(cron/event agents), semantic tool retrieval, MCP→LobeChat registration.
|
||||
- **Identity** — PocketID as the universal OIDC IdP; every service behind SSO
|
||||
(LiteLLM, LobeChat, n8n, Nextcloud, Daytona via a Keycloak→PocketID adapter).
|
||||
- **Data layer** — shared-postgres standard; a full **tier-1 "no SQLite on
|
||||
NFS"** migration (ntfy, Karakeep, Arcane, traccar → local disk; Memos +
|
||||
Vaultwarden → Postgres via pgloader); WAL-G PITR backups re-established.
|
||||
- **Network** — full read-only UniFi audit + the unmanaged D-Link DGS-1210
|
||||
core (SNTP fixed, topology/FDB mapped, DHCP/LLDP interop verified, SPOF
|
||||
identified); pinned-NTP-source standard adopted.
|
||||
- **Reliability** — gateway **deep health-check** (real MCP tool-call probe,
|
||||
usage-aware backoff) that caught silently-broken servers.
|
||||
- **Cloud burst** — on-demand Scaleway L40S GPU offload via WireGuard.
|
||||
|
||||
See `homelab-architecture.md` for the living technical reference and
|
||||
`PORTMAP.md` for the authoritative port/route map.
|
||||
|
||||
## Activity signal
|
||||
|
||||
- **~79 commits in the trailing 14 days** (`git log --since="14 days ago"`),
|
||||
spanning gateway OAuth/health, OIDC bolt-ons, DB migrations, network audit,
|
||||
GPU integration, and docs.
|
||||
- Multi-session, incident-driven: several entries trace to real failures
|
||||
caught and fixed (WAL-G archiver hung silently ~13 h; n8n `latest`-drift
|
||||
outage; SQLite-on-NFS corruption risk; an 8.5-month-stale switch clock).
|
||||
|
||||
## Productivity estimate (honest framing)
|
||||
|
||||
These are **rough order-of-magnitude estimates**, not measurements. Assumptions
|
||||
are stated so they can be challenged.
|
||||
|
||||
- Scope delivered ≈ a small platform: identity, AI gateway, agent runtime,
|
||||
~23 service stacks, a DB-migration program, a full network audit, backups.
|
||||
- A solo engineer doing this unaided, part-time, learning the unfamiliar
|
||||
pieces (OIDC internals, pgloader, MCP, UniFi/D-Link internals): a
|
||||
conservative bound is **several hundred focused hours** (≈ 8–16 part-time
|
||||
weeks). Assumes the owner is competent but not a specialist in every domain
|
||||
touched (identity, Postgres ops, embedded-switch web UIs, MCP).
|
||||
- With the assistant: compressed into a small number of intensive sessions
|
||||
over ~2 weeks. The leverage is largest where the work is *research-heavy
|
||||
but low-novelty* — reverse-engineering a D-Link form POST, deriving an
|
||||
OIDC redirect, mapping an FDB table — i.e. tasks that are tedious solo but
|
||||
not conceptually hard. The leverage is smallest on genuine judgment calls
|
||||
(what to prioritise, what risk is acceptable), which stayed with the owner.
|
||||
- **Caveat:** estimate excludes the owner's own steering/review time, which
|
||||
was substantial and is the reason the output is coherent rather than just
|
||||
voluminous.
|
||||
|
||||
## CO2 estimate (honest framing)
|
||||
|
||||
Also order-of-magnitude, assumptions explicit.
|
||||
|
||||
- **LLM inference:** a heavy multi-session collaboration of this kind is on
|
||||
the order of a few million tokens. Public estimates put frontier-model
|
||||
inference at roughly 1–5 Wh per ~1k output tokens equivalent (wide error
|
||||
bars). Taking ~3 M tokens × ~2 Wh/1k ≈ **~6 kWh** → at a ~0.35 kgCO2e/kWh
|
||||
grid ≈ **~2 kgCO2e**. Plausible range **1–5 kgCO2e**. Datacenter PUE and
|
||||
exact model size dominate the uncertainty.
|
||||
- **Homelab runtime** (the larger ongoing footprint): the NUC 14 Pro draws
|
||||
~15–45 W under mixed load. At ~30 W average → ~0.72 kWh/day → ~260 kWh/yr
|
||||
→ **~90 kgCO2e/yr** at the same grid factor. The German grid is cleaner
|
||||
than that average in many hours, so treat as an upper-ish bound. The
|
||||
on-demand Scaleway L40S burst is deliberately *on-demand* precisely to
|
||||
avoid a 24/7 GPU's footprint.
|
||||
- **Takeaway:** the assistant-collaboration carbon is a rounding error next
|
||||
to a year of always-on homelab power. Efficiency wins (CPU-only default,
|
||||
on-demand GPU, idle-aware health probing) matter more than the chat cost.
|
||||
|
||||
## Handover / current state
|
||||
|
||||
**Healthy & verified**
|
||||
- Tier-1 SQLite-off-NFS: complete.
|
||||
- OIDC: n8n (302→PocketID, client `33135ad4`) and LobeChat (`AUTH_TRUSTED_
|
||||
ORIGINS` fix, sign-in→PocketID) — both verified; LobeChat wants one real
|
||||
browser login as final proof.
|
||||
- D-Link SNTP: fixed (pinned PTB+Cloudflare IPs, clock corrected & synced).
|
||||
- Gateway deep health-check: live, usage-aware, surfaced in `/api/servers`.
|
||||
|
||||
**Open / pending** (see `homelab-architecture.md` roadmap for detail)
|
||||
- **Broken MCP servers** surfaced by the new health-check: `memos`
|
||||
(degraded — `mcp-memos` can't resolve `memos` host; Docker-network
|
||||
isolation), `context7`/`crawl4ai`/`markitdown` (down), `nextcloud`
|
||||
(probe false-positive — needs `health_check:false` or per-user creds).
|
||||
- **D-Link mgmt hardening** (bundle, confirm-first): HTTPS, SNMP review,
|
||||
Trusted-Host allowlist `192.168.1.0/24`. Shared `tapirnase` password
|
||||
reuse (WiFi/LiteLLM/switch) — rotation deferred, noted.
|
||||
- **Network**: IoT-VLAN segmentation; D-Link is the unmanaged core/SPOF;
|
||||
mgmt-TLS certs for Proxmox + D-Link.
|
||||
- **Platform**: env→secret vault; LobeChat external-feature disable;
|
||||
observability LXC; agent-platform evolution (memory/teams/MCP-exposed).
|
||||
- `nexa` analysis blocked — private repo; deploy key pending authorization.
|
||||
|
||||
**Operating rules to preserve**
|
||||
- Confirm + risk-assess before any Proxmox / Ubiquiti / network-gear write.
|
||||
- Never put DB/SQLite on the UNAS NFS share.
|
||||
- Only a full pgloader of *all* tables is a complete DB migration.
|
||||
- Prefer self-hosted; pin critical container images (no `latest` drift).
|
||||
@@ -1,564 +0,0 @@
|
||||
# Cloud GPU Extension — Scaleway L40S
|
||||
|
||||
Seamless on-demand GPU (FLUX.1-dev, video, multi-user) via a Scaleway L40S instance.
|
||||
The instance auto-starts on first request and shuts down after 45 min idle.
|
||||
Everything personal stays on the NUC.
|
||||
|
||||
---
|
||||
|
||||
## Pricing (current as of May 2026)
|
||||
|
||||
| Resource | Rate | Notes |
|
||||
|---|---|---|
|
||||
| L40S-1-48G compute | **€1.40/hour** | PAR-2, billed per minute |
|
||||
| Block volume 200 GB | **€16/month** | SSD, keeps models across restarts |
|
||||
| Flexible IP | €0.004/hour | Static IP for WireGuard endpoint |
|
||||
| Snapshots | €0.000044/GB/h | Only needed for image backups |
|
||||
|
||||
### Realistic monthly cost
|
||||
|
||||
| Usage pattern | Compute | Storage | Total |
|
||||
|---|---|---|---|
|
||||
| Weekend sessions (8 h/week) | €45 | €16 | **~€61/month** |
|
||||
| Daily 1–2 h | €63–126 | €16 | **~€79–142/month** |
|
||||
| Heavy (4 h/day) | €168 | €16 | **~€184/month** |
|
||||
| Always-on (don't) | €1,008 | €16 | €1,024/month |
|
||||
|
||||
The on-demand proxy below makes "daily 1–2 h" the natural default — you just click generate, it starts automatically.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
NUC (home, always-on) Scaleway PAR-2 (on demand)
|
||||
───────────────────────────── ───────────────────────────────
|
||||
LobeChat ──▶ gpu-proxy:8190 ─── wg1 ──▶ ComfyUI :8188
|
||||
comfyui-mcp ──▶ (Docker) Ollama :11434 (optional)
|
||||
│
|
||||
├─ start/stop via Scaleway API
|
||||
└─ idle watchdog (45 min → stop)
|
||||
```
|
||||
|
||||
`gpu-proxy` is a small Docker service that:
|
||||
1. Forwards requests to the GPU instance
|
||||
2. Auto-starts the Scaleway instance if it's stopped (cold-start ~90 s)
|
||||
3. Shuts it down after 45 min with no traffic
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# Scaleway CLI
|
||||
curl -s https://raw.githubusercontent.com/scaleway/scaleway-cli/master/scripts/get.sh | sh
|
||||
scw init # enter API key + project ID
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Persistent block volume (models live here)
|
||||
|
||||
```bash
|
||||
# Create 200 GB SSD volume in PAR-2
|
||||
scw block volume create name=nuclide-gpu-models size=200GB zone=fr-par-2
|
||||
# Note the volume ID: vol-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
```
|
||||
|
||||
Models are kept on this volume. The compute instance can be deleted and recreated freely.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Create the L40S instance with cloud-init
|
||||
|
||||
Save as `/opt/stacks/scripts/gpu-cloud-init.yaml`:
|
||||
|
||||
```yaml
|
||||
#cloud-config
|
||||
package_update: true
|
||||
packages:
|
||||
- wireguard-tools
|
||||
- docker.io
|
||||
- docker-compose-v2
|
||||
- nvidia-driver-545
|
||||
- nvidia-container-toolkit
|
||||
|
||||
write_files:
|
||||
- path: /etc/wireguard/wg0.conf
|
||||
permissions: '0600'
|
||||
content: |
|
||||
[Interface]
|
||||
Address = 10.200.0.1/24
|
||||
ListenPort = 51820
|
||||
PrivateKey = SCALEWAY_WG_PRIVKEY
|
||||
|
||||
[Peer]
|
||||
# NUC
|
||||
PublicKey = NUC_WG_PUBKEY
|
||||
AllowedIPs = 10.200.0.2/32
|
||||
PersistentKeepalive = 25
|
||||
|
||||
- path: /opt/gpu/docker-compose.yml
|
||||
content: |
|
||||
services:
|
||||
comfyui:
|
||||
image: yanwk/comfyui-boot:cu124
|
||||
container_name: comfyui
|
||||
ports:
|
||||
- "10.200.0.1:8188:8188"
|
||||
volumes:
|
||||
- /mnt/models:/root/ComfyUI/models
|
||||
- comfyui_config:/root/ComfyUI
|
||||
environment:
|
||||
- CLI_ARGS=--listen 0.0.0.0
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
restart: unless-stopped
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
container_name: ollama
|
||||
ports:
|
||||
- "10.200.0.1:11434:11434"
|
||||
volumes:
|
||||
- /mnt/models/ollama:/root/.ollama
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
comfyui_config:
|
||||
|
||||
runcmd:
|
||||
# Mount the models volume (will be /dev/sdb or similar)
|
||||
- mkdir -p /mnt/models
|
||||
- |
|
||||
DISK=$(lsblk -ndo NAME,SIZE | awk '$2=="200G"{print "/dev/"$1}' | head -1)
|
||||
if [ -n "$DISK" ]; then
|
||||
blkid "$DISK" || mkfs.ext4 "$DISK"
|
||||
echo "$DISK /mnt/models ext4 defaults 0 2" >> /etc/fstab
|
||||
mount "$DISK" /mnt/models
|
||||
fi
|
||||
- mkdir -p /mnt/models/ollama
|
||||
# Enable WireGuard
|
||||
- systemctl enable --now wg-quick@wg0
|
||||
# Configure nvidia-container-toolkit
|
||||
- nvidia-ctk runtime configure --runtime=docker
|
||||
- systemctl restart docker
|
||||
# Start GPU services
|
||||
- cd /opt/gpu && docker compose up -d
|
||||
```
|
||||
|
||||
**Before using this file**, replace:
|
||||
- `SCALEWAY_WG_PRIVKEY` → output of `wg genkey` (run on Scaleway instance side first)
|
||||
- `NUC_WG_PUBKEY` → output of `cat /etc/wireguard/nuc_wg_pub` (see Step 3)
|
||||
|
||||
Launch the instance:
|
||||
|
||||
```bash
|
||||
# Generate WireGuard keys first (do this on the NUC)
|
||||
wg genkey | tee /etc/wireguard/scaleway_wg_priv | wg pubkey > /etc/wireguard/scaleway_wg_pub
|
||||
wg genkey | tee /etc/wireguard/nuc_wg_priv | wg pubkey > /etc/wireguard/nuc_wg_pub
|
||||
|
||||
# Fill in cloud-init.yaml with the keys, then:
|
||||
INSTANCE_ID=$(scw instance server create \
|
||||
type=L40S-1-48G \
|
||||
image=ubuntu_jammy_gpu \
|
||||
zone=fr-par-2 \
|
||||
name=nuclide-gpu \
|
||||
cloud-init=@/opt/stacks/scripts/gpu-cloud-init.yaml \
|
||||
output=json | jq -r '.id')
|
||||
|
||||
echo "Instance ID: $INSTANCE_ID"
|
||||
|
||||
# Attach the models volume
|
||||
scw instance server attach-volume \
|
||||
server-id=$INSTANCE_ID \
|
||||
volume-id=vol-xxxxxxxx \
|
||||
zone=fr-par-2
|
||||
|
||||
# Allocate a Flexible IP (static IP that survives instance restarts)
|
||||
FLEXIP_ID=$(scw instance ip create zone=fr-par-2 output=json | jq -r '.id')
|
||||
scw instance server attach-flexible-ip \
|
||||
server-id=$INSTANCE_ID \
|
||||
ip-id=$FLEXIP_ID \
|
||||
zone=fr-par-2
|
||||
|
||||
FLEXIP=$(scw instance ip get $FLEXIP_ID zone=fr-par-2 output=json | jq -r '.address')
|
||||
echo "Scaleway public IP: $FLEXIP"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — WireGuard on the NUC
|
||||
|
||||
```bash
|
||||
# /etc/wireguard/wg1.conf (separate from any existing VPN tunnel)
|
||||
cat > /etc/wireguard/wg1.conf << EOF
|
||||
[Interface]
|
||||
Address = 10.200.0.2/24
|
||||
PrivateKey = $(cat /etc/wireguard/nuc_wg_priv)
|
||||
|
||||
[Peer]
|
||||
# Scaleway GPU
|
||||
PublicKey = $(cat /etc/wireguard/scaleway_wg_pub)
|
||||
Endpoint = ${FLEXIP}:51820
|
||||
AllowedIPs = 10.200.0.1/32
|
||||
PersistentKeepalive = 25
|
||||
EOF
|
||||
|
||||
chmod 600 /etc/wireguard/wg1.conf
|
||||
systemctl enable --now wg-quick@wg1
|
||||
```
|
||||
|
||||
Test when the instance is running:
|
||||
```bash
|
||||
ping 10.200.0.1 # WireGuard tunnel
|
||||
curl http://10.200.0.1:8188 # ComfyUI
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — gpu-proxy (seamless on-demand startup)
|
||||
|
||||
This Docker service runs on the NUC. It proxies to the GPU instance and auto-starts/stops it.
|
||||
|
||||
### `ai/gpu-proxy/docker-compose.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
gpu-proxy:
|
||||
build: .
|
||||
container_name: gpu-proxy
|
||||
environment:
|
||||
- SCW_SECRET_KEY=${SCW_SECRET_KEY}
|
||||
- SCW_PROJECT_ID=${SCW_PROJECT_ID}
|
||||
- SCW_INSTANCE_ID=${SCW_INSTANCE_ID}
|
||||
- SCW_ZONE=fr-par-2
|
||||
- GPU_HOST=10.200.0.1
|
||||
- COMFYUI_PORT=8188
|
||||
- OLLAMA_PORT=11434
|
||||
- IDLE_TIMEOUT=2700 # 45 min
|
||||
ports:
|
||||
- "8190:8190" # ComfyUI proxy
|
||||
- "8191:8191" # Ollama proxy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- shared_backend
|
||||
|
||||
networks:
|
||||
shared_backend:
|
||||
external: true
|
||||
```
|
||||
|
||||
### `ai/gpu-proxy/proxy.py`
|
||||
|
||||
```python
|
||||
"""
|
||||
On-demand GPU proxy. Auto-starts the Scaleway instance on first request,
|
||||
shuts it down after IDLE_TIMEOUT seconds of inactivity.
|
||||
"""
|
||||
import asyncio, os, time, httpx, subprocess
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import StreamingResponse, JSONResponse
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
SCW_KEY = os.environ["SCW_SECRET_KEY"]
|
||||
SCW_PROJECT = os.environ["SCW_PROJECT_ID"]
|
||||
INSTANCE_ID = os.environ["SCW_INSTANCE_ID"]
|
||||
ZONE = os.environ.get("SCW_ZONE", "fr-par-2")
|
||||
GPU_HOST = os.environ.get("GPU_HOST", "10.200.0.1")
|
||||
COMFYUI_PORT = int(os.environ.get("COMFYUI_PORT", 8188))
|
||||
OLLAMA_PORT = int(os.environ.get("OLLAMA_PORT", 11434))
|
||||
IDLE_TIMEOUT = int(os.environ.get("IDLE_TIMEOUT", 2700))
|
||||
|
||||
SCW_API = f"https://api.scaleway.com/instance/v1/zones/{ZONE}"
|
||||
HEADERS = {"X-Auth-Token": SCW_KEY, "Content-Type": "application/json"}
|
||||
|
||||
_state = {"last_activity": 0.0, "starting": False, "up": False}
|
||||
_lock = asyncio.Lock()
|
||||
|
||||
|
||||
async def _scw(method: str, path: str, **kwargs):
|
||||
async with httpx.AsyncClient() as c:
|
||||
r = await c.request(method, f"{SCW_API}{path}", headers=HEADERS, **kwargs)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
async def _instance_state() -> str:
|
||||
d = await _scw("GET", f"/servers/{INSTANCE_ID}")
|
||||
return d["server"]["state"] # running | stopped | stopping | starting
|
||||
|
||||
|
||||
async def _start_instance():
|
||||
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweron"})
|
||||
|
||||
|
||||
async def _stop_instance():
|
||||
await _scw("POST", f"/servers/{INSTANCE_ID}/action", json={"action": "poweroff"})
|
||||
|
||||
|
||||
async def _wait_ready(host: str, port: int, timeout=180) -> bool:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3) as c:
|
||||
await c.get(f"http://{host}:{port}/")
|
||||
return True
|
||||
except Exception:
|
||||
await asyncio.sleep(5)
|
||||
return False
|
||||
|
||||
|
||||
async def ensure_up(port: int) -> bool:
|
||||
async with _lock:
|
||||
if _state["up"]:
|
||||
_state["last_activity"] = time.time()
|
||||
return True
|
||||
if _state["starting"]:
|
||||
return False # caller will retry
|
||||
_state["starting"] = True
|
||||
|
||||
try:
|
||||
state = await _instance_state()
|
||||
if state != "running":
|
||||
print(f"[gpu-proxy] instance {state} → starting")
|
||||
await _start_instance()
|
||||
# wait for API to report running
|
||||
for _ in range(60):
|
||||
await asyncio.sleep(5)
|
||||
if await _instance_state() == "running":
|
||||
break
|
||||
# wait for service to respond
|
||||
if await _wait_ready(GPU_HOST, port):
|
||||
async with _lock:
|
||||
_state["up"] = True
|
||||
_state["last_activity"] = time.time()
|
||||
print("[gpu-proxy] GPU instance ready")
|
||||
return True
|
||||
return False
|
||||
finally:
|
||||
async with _lock:
|
||||
_state["starting"] = False
|
||||
|
||||
|
||||
async def idle_watchdog():
|
||||
while True:
|
||||
await asyncio.sleep(60)
|
||||
async with _lock:
|
||||
if not _state["up"]:
|
||||
continue
|
||||
idle = time.time() - _state["last_activity"]
|
||||
if idle > IDLE_TIMEOUT:
|
||||
print(f"[gpu-proxy] idle {idle:.0f}s → stopping instance")
|
||||
try:
|
||||
await _stop_instance()
|
||||
async with _lock:
|
||||
_state["up"] = False
|
||||
_state["last_activity"] = 0.0
|
||||
except Exception as e:
|
||||
print(f"[gpu-proxy] stop error: {e}")
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
asyncio.create_task(idle_watchdog())
|
||||
# Probe: is the instance already running from a previous session?
|
||||
try:
|
||||
if await _instance_state() == "running":
|
||||
if await _wait_ready(GPU_HOST, COMFYUI_PORT, timeout=10):
|
||||
async with _lock:
|
||||
_state["up"] = True
|
||||
_state["last_activity"] = time.time()
|
||||
print("[gpu-proxy] GPU already up on startup")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _proxy(request: Request, host: str, port: int):
|
||||
_state["last_activity"] = time.time()
|
||||
ready = await ensure_up(port)
|
||||
if not ready:
|
||||
# Starting up — keep trying for up to 3 min
|
||||
for _ in range(36):
|
||||
await asyncio.sleep(5)
|
||||
if _state["up"]:
|
||||
break
|
||||
else:
|
||||
return JSONResponse({"error": "GPU instance failed to start"}, 503)
|
||||
|
||||
url = f"http://{host}:{port}{request.url.path}"
|
||||
if request.url.query:
|
||||
url += f"?{request.url.query}"
|
||||
body = await request.body()
|
||||
client = httpx.AsyncClient(timeout=httpx.Timeout(None, connect=10))
|
||||
req = client.build_request(request.method, url,
|
||||
headers={k: v for k, v in request.headers.items()
|
||||
if k.lower() not in {"host", "content-length"}},
|
||||
content=body)
|
||||
try:
|
||||
resp = await client.send(req, stream=True)
|
||||
except httpx.ConnectError:
|
||||
await client.aclose()
|
||||
return JSONResponse({"error": "GPU not reachable"}, 502)
|
||||
|
||||
async def stream():
|
||||
async for chunk in resp.aiter_raw():
|
||||
yield chunk
|
||||
await resp.aclose()
|
||||
await client.aclose()
|
||||
|
||||
return StreamingResponse(stream(), status_code=resp.status_code,
|
||||
headers=dict(resp.headers),
|
||||
media_type=resp.headers.get("content-type"))
|
||||
|
||||
|
||||
@app.api_route("/status", methods=["GET"])
|
||||
async def status():
|
||||
try:
|
||||
scw_state = await _instance_state()
|
||||
except Exception as e:
|
||||
scw_state = f"error: {e}"
|
||||
return {"instance": scw_state, "proxy_up": _state["up"],
|
||||
"idle_s": int(time.time() - _state["last_activity"]) if _state["up"] else None}
|
||||
|
||||
|
||||
@app.api_route("/{path:path}", methods=["GET","POST","PUT","DELETE","OPTIONS","PATCH"])
|
||||
async def comfyui_proxy(request: Request, path: str):
|
||||
return await _proxy(request, GPU_HOST, COMFYUI_PORT)
|
||||
```
|
||||
|
||||
### `ai/gpu-proxy/Dockerfile`
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
RUN pip install fastapi uvicorn httpx
|
||||
COPY proxy.py /app/proxy.py
|
||||
WORKDIR /app
|
||||
EXPOSE 8190 8191
|
||||
CMD ["uvicorn", "proxy:app", "--host", "0.0.0.0", "--port", "8190"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Wire up NUC services
|
||||
|
||||
Add to `ai/.env`:
|
||||
```bash
|
||||
SCW_SECRET_KEY=<your-scaleway-api-key>
|
||||
SCW_PROJECT_ID=<your-project-id>
|
||||
SCW_INSTANCE_ID=<instance-id-from-step-2>
|
||||
```
|
||||
|
||||
In `ai/lobehub.yml`:
|
||||
```yaml
|
||||
- 'COMFYUI_BASE_URL=http://gpu-proxy:8190'
|
||||
```
|
||||
|
||||
In `ai/mcp-gateway/server.py` (SERVERS dict):
|
||||
```python
|
||||
"comfyui": {
|
||||
"static": True,
|
||||
"upstream": "http://gpu-proxy:8190",
|
||||
"group": "image",
|
||||
},
|
||||
```
|
||||
|
||||
For Immich ML (optional), add to `immich/docker-compose.yml`:
|
||||
```yaml
|
||||
environment:
|
||||
- IMMICH_MACHINE_LEARNING_URL=http://gpu-proxy:8192 # add a third port for ML
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Deploy
|
||||
|
||||
```bash
|
||||
# Deploy the proxy
|
||||
cd /opt/stacks/ai/gpu-proxy
|
||||
docker compose up -d
|
||||
|
||||
# Restart LobeChat and MCP gateway to pick up new env
|
||||
cd /opt/stacks/ai
|
||||
docker compose -f lobehub.yml up -d --force-recreate lobe
|
||||
cd mcp-gateway && docker compose up -d --force-recreate mcp-gateway
|
||||
```
|
||||
|
||||
### What the user experience looks like
|
||||
|
||||
1. Open LobeChat, generate an image → first request triggers auto-start
|
||||
2. "Starting GPU…" — ComfyUI returns a brief wait (up to 90 s cold start)
|
||||
3. Images generate normally; the proxy keeps the instance alive
|
||||
4. After 45 min with no requests → instance stops automatically
|
||||
5. `/status` endpoint on port 8190: `{"instance":"stopped","proxy_up":false}`
|
||||
|
||||
---
|
||||
|
||||
## Quick-start / stop scripts
|
||||
|
||||
```bash
|
||||
# /usr/local/bin/nuclide-gpu (NUC helper)
|
||||
#!/bin/bash
|
||||
ZONE=fr-par-2
|
||||
ID=$(docker exec gpu-proxy env | grep SCW_INSTANCE_ID | cut -d= -f2)
|
||||
case "$1" in
|
||||
start) scw instance server action action=poweron server-id=$ID zone=$ZONE ;;
|
||||
stop) scw instance server action action=poweroff server-id=$ID zone=$ZONE ;;
|
||||
status) curl -s http://localhost:8190/status | python3 -m json.tool ;;
|
||||
*) echo "Usage: nuclide-gpu start|stop|status" ;;
|
||||
esac
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scaleway firewall (security groups)
|
||||
|
||||
On the Scaleway instance, only expose WireGuard. All services bind to `10.200.0.1` (WireGuard IP):
|
||||
|
||||
```bash
|
||||
ufw default deny incoming
|
||||
ufw allow 51820/udp # WireGuard from anywhere
|
||||
ufw allow from 10.200.0.0/24 # full access over tunnel
|
||||
ufw enable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Model storage layout (on the 200 GB volume)
|
||||
|
||||
```
|
||||
/mnt/models/
|
||||
├── checkpoints/ # FLUX.1-dev, SDXL, etc.
|
||||
├── vae/
|
||||
├── loras/
|
||||
├── controlnet/
|
||||
├── ollama/ # Ollama model blobs
|
||||
└── upscale_models/
|
||||
```
|
||||
|
||||
FLUX.1-dev (FP16): ~24 GB
|
||||
FLUX.1-schnell (NF4): ~8 GB
|
||||
Ollama llama3:70b (Q4): ~40 GB — fits alongside FLUX on 200 GB with room for LoRAs.
|
||||
|
||||
To pre-download models after first boot:
|
||||
```bash
|
||||
ssh root@10.200.0.1 # via WireGuard
|
||||
docker exec comfyui python3 -c "
|
||||
from huggingface_hub import hf_hub_download
|
||||
hf_hub_download('black-forest-labs/FLUX.1-dev', 'flux1-dev.safetensors',
|
||||
local_dir='/root/ComfyUI/models/checkpoints')
|
||||
"
|
||||
```
|
||||
@@ -1,75 +0,0 @@
|
||||
# ComfyUI → LobeChat via MCP
|
||||
|
||||
Image generation (FLUX.1-schnell GGUF on the Intel Arc iGPU) exposed to LobeChat
|
||||
as an MCP tool. Chosen over LobeChat's native ComfyUI provider because that
|
||||
provider is hardcoded to non-GGUF nodes and is not configurable without forking
|
||||
LobeChat (see `docs/mcp-gateway-requirements.md` research notes).
|
||||
|
||||
## Components
|
||||
|
||||
- **`ai/mcp-servers/comfyui/`** — purpose-built MCP server (async queue edition)
|
||||
- `server.py` — FastMCP, streamable-HTTP on `:8000` at `/mcp`. Six tools:
|
||||
- `generate_image(prompt, width, height, steps, seed)` — txt2img; returns `job_id` immediately
|
||||
- `img2img(prompt, images, strength, steps, seed)` — img2img for one or more images; one job per image; images can be URLs, base64 data URIs, or `job:<id>` references
|
||||
- `get_job_status(job_ids)` — returns status text + inline PNG for completed jobs
|
||||
- `list_queue(limit)` — lists all jobs in the in-process registry
|
||||
- `cancel_job(job_id)` — cancels queued/running job (user must confirm first)
|
||||
- `list_recent_images(n)` — shows n most recent completed images for reference/chaining
|
||||
- Background polling thread updates job state every 3 s via ComfyUI `/history`.
|
||||
- `flux-gguf-api.json` — txt2img workflow; nodes 4=prompt, 6=size, 8=steps/seed.
|
||||
- `flux-img2img-api.json` — img2img workflow; node 11=LoadImage, 12=VAEEncode, 8=KSampler with denoise.
|
||||
- `Dockerfile`, `requirements.txt` (`mcp[cli]`, `httpx`).
|
||||
- **`ai/comfyui-mcp.yml`** — compose; container `comfyui-mcp` on
|
||||
`shared_backend` (reaches `comfyui:8188`; reachable by `lobehub`, which is on
|
||||
`shared_backend`). Host port `18003:8000` for testing/Zoraxy.
|
||||
|
||||
## Async workflow
|
||||
|
||||
```
|
||||
generate_image("a red apple") → "Job submitted: txt-1a8bbeda" (< 1 s)
|
||||
[ComfyUI rendering... ~90-300 s]
|
||||
get_job_status(["txt-1a8bbeda"]) → status text + inline PNG image
|
||||
|
||||
# Chain: use previous output as img2img input (no bytes through LLM)
|
||||
img2img("add a blue bowl", images=["job:txt-1a8bbeda"], strength=0.6)
|
||||
→ "img-2b9ccefa"
|
||||
```
|
||||
|
||||
## Verified
|
||||
|
||||
- All 6 tools discovered via MCP `tools/list`.
|
||||
- `generate_image` returns in < 0.1 s (non-blocking); job shows in `list_queue` as "running".
|
||||
- Previous end-to-end: `generate_image` → `get_job_status` → inline PNG ~94 s bare, ~160 s via MCP.
|
||||
- LobeChat v2.1.58 renders MCP `image` blocks (uploads to Garage S3 → inline ``).
|
||||
|
||||
## Known characteristics / limitations
|
||||
|
||||
- **In-memory registry**: job state is lost on container restart. Resubmit if needed.
|
||||
- **No auth** on the MCP server (internal `shared_backend` only). Host port
|
||||
18003 is LAN-exposed and unauthenticated — fine for a trusted homelab LAN.
|
||||
- ~90-300 s/image latency; use `get_job_status()` to poll — never block waiting.
|
||||
|
||||
## Final hookup (manual — LobeChat UI/DB, no server-mode config path)
|
||||
|
||||
LobeChat → **Settings → Skills (Tools) → Skill Store → Custom → Import JSON**:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"comfyui-flux": {
|
||||
"type": "http",
|
||||
"url": "http://comfyui-mcp:8000/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then enable the `comfyui-flux` skill in an agent/chat and ask the model to
|
||||
"generate an image of …". Images appear inline in the conversation.
|
||||
|
||||
## Ops
|
||||
|
||||
- Build/deploy: `cd /opt/stacks/ai && docker compose -f comfyui-mcp.yml up -d --build`
|
||||
- Logs: `docker logs comfyui-mcp`
|
||||
- The workflow is the single source of truth in `flux-gguf-api.json`; keep it in
|
||||
sync with `ai/comfyui/workflows/flux-schnell-api.json` if the graph changes.
|
||||
@@ -1,262 +0,0 @@
|
||||
# Dev Environment — CT 111 (dev.nuclide.systems)
|
||||
|
||||
CT 111 hosts the self-hosted development platform: **Coder** (dev workspaces) + **Gitea** (internal repos).
|
||||
|
||||
## Services
|
||||
|
||||
| Service | URL | Port |
|
||||
|---|---|---|
|
||||
| Coder | https://dev.nuclide.systems | 7080 |
|
||||
| Gitea | https://git.nuclide.systems | 3000 |
|
||||
|
||||
Both use **Pocket-ID OIDC** (`https://id.nuclide.systems`) for SSO.
|
||||
|
||||
---
|
||||
|
||||
## Coder — Dev Workspaces
|
||||
|
||||
### What it is
|
||||
Coder provisions isolated Docker-based dev environments (workspaces) on CT 111. Each workspace has:
|
||||
- A full Linux environment with your tools
|
||||
- Persistent home dir on UNAS (`/mnt/pve/unas/services/coder/`)
|
||||
- Intel Arc GPU renderD128 available
|
||||
- VS Code Server (browser or desktop SSH tunnel)
|
||||
|
||||
### First-time setup
|
||||
1. Go to `https://dev.nuclide.systems` → log in via Pocket-ID
|
||||
2. Create a workspace from a template (admin must create templates first)
|
||||
3. Connect via VS Code: install **Coder extension** → sign in → open workspace
|
||||
|
||||
### VS Code connection
|
||||
```bash
|
||||
# Install Coder CLI on your local machine
|
||||
curl -fsSL https://coder.com/install.sh | sh
|
||||
|
||||
# Authenticate
|
||||
coder login https://dev.nuclide.systems
|
||||
|
||||
# Open workspace in VS Code
|
||||
coder open <workspace-name>
|
||||
```
|
||||
Or use the Coder VS Code extension directly from the marketplace (`coder.coder-remote`).
|
||||
|
||||
### Claude Code inside a workspace
|
||||
```bash
|
||||
# Inside the workspace terminal
|
||||
npm install -g @anthropic/claude-code
|
||||
claude
|
||||
```
|
||||
Claude Code runs inside the workspace container — same environment, same files, same GPU.
|
||||
|
||||
### Coder MCP (AI agent sandbox execution)
|
||||
Add to Claude Code's MCP config (`~/.claude/claude_desktop_config.json` or via `/mcp add`):
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"coder": {
|
||||
"command": "coder",
|
||||
"args": ["mcp", "server"],
|
||||
"env": {
|
||||
"CODER_URL": "https://dev.nuclide.systems",
|
||||
"CODER_TOKEN": "<your-api-token>"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Claude can then create workspaces, execute code, and read output via MCP tools:
|
||||
- `coder_list_workspaces`
|
||||
- `coder_create_workspace`
|
||||
- `coder_execute_command` ← sandbox code execution
|
||||
- `coder_start_workspace` / `coder_stop_workspace`
|
||||
|
||||
Get your API token: `coder tokens create`
|
||||
|
||||
### Creating workspace templates
|
||||
Templates are Terraform configs stored in Gitea. Basic Docker template:
|
||||
```bash
|
||||
coder templates push <template-name> --directory ./template/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gitea — Internal Repos
|
||||
|
||||
### What it is
|
||||
Self-hosted Git for internal infrastructure: compose files, CT configs, dotfiles, Coder templates.
|
||||
**Not** the primary remote for Claude Code collaboration — use GitHub for that.
|
||||
|
||||
### First-time setup (admin)
|
||||
1. Go to `https://git.nuclide.systems` → complete installation wizard
|
||||
2. Set admin account, confirm DB settings (pre-filled from env)
|
||||
3. Add OIDC provider: Admin → Site Administration → Authentication Sources
|
||||
- Auth type: OAuth2
|
||||
- Provider: OpenID Connect
|
||||
- Discovery URL: `https://id.nuclide.systems/.well-known/openid-configuration`
|
||||
- Client ID/Secret: create a new client in Pocket-ID for Gitea
|
||||
|
||||
### SSH access
|
||||
```bash
|
||||
# Gitea SSH runs on port 222
|
||||
git clone ssh://git@git.nuclide.systems:222/<user>/<repo>.git
|
||||
|
||||
# Or add to ~/.ssh/config:
|
||||
Host git.nuclide.systems
|
||||
Port 222
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
### Recommended repos to create
|
||||
| Repo | Contents |
|
||||
|---|---|
|
||||
| `infra/proxmox` | `/etc/pve/` snapshots, CT configs |
|
||||
| `infra/stacks` | Compose files from CT 104/101/111 |
|
||||
| `infra/docs` | Mirror of `/docs/` on Proxmox host |
|
||||
| `dev/templates` | Coder workspace Terraform templates |
|
||||
|
||||
---
|
||||
|
||||
## Storage layout (UNAS)
|
||||
|
||||
```
|
||||
/mnt/pve/unas/services/
|
||||
├── coder/ # Coder workspace home dirs (persistent)
|
||||
└── gitea/ # Gitea repos + data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CT 111 specs
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| IP | 192.168.1.42 |
|
||||
| Cores | 12 |
|
||||
| RAM | 32GB |
|
||||
| Rootfs | 60GB local-zfs (Docker image cache) |
|
||||
| UNAS | /mnt/pve/unas (mp0) |
|
||||
| GPU | renderD128 (Intel Arc Xe, by-path) |
|
||||
|
||||
---
|
||||
|
||||
## OIDC clients in Pocket-ID
|
||||
|
||||
| Client | Callback URL |
|
||||
|---|---|
|
||||
| Coder | `https://dev.nuclide.systems/api/v2/users/oidc/callback` |
|
||||
| Gitea | Add via Gitea admin UI (see above) |
|
||||
|
||||
To create new OIDC clients programmatically, see `/docs/proxmox-optimizations.md` § OIDC client creation via SQLite.
|
||||
|
||||
---
|
||||
|
||||
## Auth lockdown (SSO-only)
|
||||
|
||||
Both Gitea and Coder are locked to **Pocket-ID OIDC only**. Local password and GitHub login are disabled. Passkey/WebAuthn login *to Gitea* remains available because it's tied to OIDC accounts, not to a separate password.
|
||||
|
||||
### Compose env flags
|
||||
|
||||
**Coder** (`/opt/stacks/coder/compose.yaml`):
|
||||
```yaml
|
||||
CODER_OIDC_ISSUER_URL: "https://id.nuclide.systems"
|
||||
CODER_OIDC_CLIENT_ID: "${CODER_OIDC_CLIENT_ID}"
|
||||
CODER_OIDC_CLIENT_SECRET: "${CODER_OIDC_CLIENT_SECRET}"
|
||||
CODER_OIDC_ALLOW_SIGNUPS: "true"
|
||||
CODER_OIDC_EMAIL_DOMAIN: "nucli.de"
|
||||
CODER_DISABLE_PASSWORD_AUTH: "true"
|
||||
CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE: "false"
|
||||
```
|
||||
Note: the bundled "GitHub external auth provider" log line is for *workspaces* cloning from GitHub, not for login — leaving it on is fine.
|
||||
|
||||
**Gitea** (`/opt/stacks/gitea/compose.yaml`):
|
||||
```yaml
|
||||
GITEA__oauth2__ENABLED: "true"
|
||||
GITEA__openid__ENABLE_OPENID_SIGNIN: "false" # legacy OpenID 2.0 button off
|
||||
GITEA__openid__ENABLE_OPENID_SIGNUP: "false"
|
||||
GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM: "false" # local username/password form off
|
||||
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "true"
|
||||
GITEA__oauth2_client__ACCOUNT_LINKING: "auto"
|
||||
GITEA__oauth2_client__USERNAME: "preferred_username"
|
||||
GITEA__oauth2_client__UPDATE_AVATAR: "true"
|
||||
```
|
||||
Note: `ENABLE_OPENID_SIGNIN` lives in `[openid]`, *not* `[service]`. Putting it under `GITEA__service__` is a no-op and leaves the legacy button visible.
|
||||
|
||||
Gitea login source (DB row in `login_source`):
|
||||
- `name = "pocket-id"` (case-sensitive — becomes part of the callback URL)
|
||||
- `Scopes = ["openid","profile","email"]`
|
||||
- `two_factor_policy = "skip"` (Pocket-ID passkey already enforces 2FA)
|
||||
|
||||
### Pocket-ID client secret pitfall (important)
|
||||
|
||||
Pocket-ID 2.7.0 verifies client secrets with **bcrypt** (`bcrypt.CompareHashAndPassword`). The stored `oidc_clients.secret` column must be a 60-char bcrypt hash like `$2a$10$...` or `$2b$10$...`.
|
||||
|
||||
A raw 64-char hex SHA-256 in that column **silently fails every token exchange** with `invalid client secret`. The Gitea and Coder clients on this host were initially provisioned that way and had to be regenerated.
|
||||
|
||||
To programmatically add or rotate an OIDC client secret in Pocket-ID:
|
||||
|
||||
```bash
|
||||
# On Proxmox host (has python3-bcrypt installed)
|
||||
python3 <<'EOF'
|
||||
import bcrypt, secrets, string
|
||||
alphabet = string.ascii_letters + string.digits
|
||||
plain = "".join(secrets.choice(alphabet) for _ in range(40))
|
||||
hashed = bcrypt.hashpw(plain.encode(), bcrypt.gensalt(rounds=10)).decode()
|
||||
print("PLAINTEXT (give to client app):", plain)
|
||||
print("HASH (store in oidc_clients.secret):", hashed)
|
||||
EOF
|
||||
```
|
||||
|
||||
Always push the hash to Pocket-ID via a tmp file (`pct push 110 ...`) — never inline the bcrypt hash in a shell command, the `$` chars get expanded.
|
||||
|
||||
To verify a client's secret is the right format:
|
||||
```bash
|
||||
pct exec 110 -- sqlite3 /opt/stacks/pocketid/data/pocket-id.db \
|
||||
"SELECT name, length(secret), substr(secret,1,7) FROM oidc_clients;"
|
||||
# Working clients: length=60, prefix "$2a$10$" or "$2b$10$"
|
||||
# Broken clients : length=64, prefix is hex (e.g. "1180664")
|
||||
```
|
||||
|
||||
### Break-glass recovery (if SSO is broken)
|
||||
|
||||
You have host SSH access, so you're never locked out — but the UI will be unusable until you re-enable a local login path. From the Proxmox host:
|
||||
|
||||
**Gitea — re-enable local login + reset password:**
|
||||
```bash
|
||||
# 1. Temporarily put the form back
|
||||
ssh nuc
|
||||
python3 -c "
|
||||
p='/rpool/data/subvol-111-disk-0/opt/stacks/gitea/compose.yaml'
|
||||
s=open(p).read().replace('ENABLE_PASSWORD_SIGNIN_FORM: \"false\"','ENABLE_PASSWORD_SIGNIN_FORM: \"true\"')
|
||||
open(p,'w').write(s)"
|
||||
pct exec 111 -- bash -c 'cd /opt/stacks/gitea && docker compose up -d --force-recreate gitea'
|
||||
|
||||
# 2. Reset admin password
|
||||
pct exec 111 -- docker exec -u git gitea gitea -c /data/gitea/conf/app.ini admin user change-password -u fkrebs -p 'temp-strong-pass'
|
||||
```
|
||||
|
||||
**Coder — flip user back to password login + re-enable:**
|
||||
```bash
|
||||
ssh nuc
|
||||
# 1. Flip login_type back
|
||||
pct exec 111 -- docker exec coder-db psql -U coder -d coder -c \
|
||||
"UPDATE users SET login_type='password' WHERE username='fkrebs';"
|
||||
|
||||
# 2. Re-enable password auth in compose
|
||||
python3 -c "
|
||||
p='/rpool/data/subvol-111-disk-0/opt/stacks/coder/compose.yaml'
|
||||
s=open(p).read().replace('CODER_DISABLE_PASSWORD_AUTH: \"true\"','CODER_DISABLE_PASSWORD_AUTH: \"false\"')
|
||||
open(p,'w').write(s)"
|
||||
pct exec 111 -- bash -c 'cd /opt/stacks/coder && docker compose up -d --force-recreate coder'
|
||||
|
||||
# 3. Reset password (interactive)
|
||||
pct exec 111 -- docker exec -it coder coder reset-password fkrebs
|
||||
```
|
||||
|
||||
**Pocket-ID — bootstrap a one-time access token if you're locked out of *Pocket-ID itself*:**
|
||||
```bash
|
||||
pct exec 110 -- docker exec pocket-id pocket-id-cli one-time-access-token \
|
||||
--email fkrebs@nucli.de --duration 1h
|
||||
# Open the printed URL in a browser to log in once and register a new passkey.
|
||||
```
|
||||
|
||||
After SSO is fixed: undo each of the above steps (re-disable password auth, flip `login_type` back to `oidc`).
|
||||
@@ -1,373 +0,0 @@
|
||||
# Nuclide Ecosystem — Homelab Architecture
|
||||
|
||||
Living architecture reference for the `/opt/stacks` homelab. ~66 containers
|
||||
across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
|
||||
`*.nuclide.systems` via one reverse proxy.
|
||||
|
||||
## Topology
|
||||
- **NUC 14 Pro** (`192.168.1.40`) — primary Docker host, all `/opt/stacks/*`.
|
||||
Itself a **Proxmox LXC (CTID 104)** on node `nuc`; local data on ZFS
|
||||
`rpool/data/subvol-104-disk-0`. Mounts the UNAS NFS at `/mnt/pve/unas`.
|
||||
- **Proxmox VE** — host **`192.168.1.20:8006`**, single node **`nuc`**
|
||||
(Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via
|
||||
`root@pam!mcp` token (now `PVEAuditor`, read-only). **7 guests:**
|
||||
- `100` qemu **haos** — Home Assistant OS VM (4c/16G) → `.60`
|
||||
- `101` lxc **shepard** — secondary Docker host (12c/16G/107G) → `.49`
|
||||
- `102` lxc **dns** — **AdGuard Home** (network DNS + ad/tracker blocking,
|
||||
2c/0.5G). UniFi DHCP has `dhcpd_dns_enabled: false`; clients resolve via
|
||||
the gateway `.1`, **and the UDM forwards DNS upstream to AdGuard**
|
||||
(confirmed) — so ad/tracker blocking is network-wide despite DHCP not
|
||||
handing out AdGuard's IP directly.
|
||||
- `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
|
||||
- `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40`
|
||||
- `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
|
||||
- `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
|
||||
- Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
|
||||
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
|
||||
APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
|
||||
enforced). Single site "Default". MCP access: `ghcr.io/enuno/unifi-mcp-server`
|
||||
(197 tools, Network App API key); Site Manager/cloud tools need
|
||||
`UNIFI_SITE_MANAGER_ENABLED`, all local API tools work.
|
||||
- **LAN** "Default" — `192.168.1.1/24`, corporate, no VLAN. DHCP pool
|
||||
`.100–.250` (24 h lease), domain `localdomain`. Static infra lives
|
||||
`.1–.99` (outside the pool). WANs: "Internet 1" (primary), "Secondary"
|
||||
(WAN2, disabled). 47 clients (17 wired, 30 wireless).
|
||||
- **WiFi** — single SSID **"nuclide"**, WPA2 with WPA3-transition, bands
|
||||
2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. **Flat
|
||||
network**: servers, IoT, consoles, phones all share one L2 segment.
|
||||
- **Firewall** — zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot),
|
||||
almost entirely system-predefined rules. Port forwards: **80/443 →
|
||||
192.168.1.4** (Zoraxy); **15001 TCP/UDP → 192.168.1.40** (Traccar watch
|
||||
protocol — already at UDM level, no Zoraxy stream proxy needed).
|
||||
- ⚠️ **Non-UniFi gear = topology blind spots** (UniFi can't see/manage):
|
||||
- **D-Link DGS-1210-28P** 28-port PoE switch at `.10` (HW F5, fw
|
||||
**6.32.008**, S/N TM0I533010201) — unmanaged by UniFi; its ports/
|
||||
port-clients don't appear in UniFi topology. HTTP-only mgmt
|
||||
(`admin`/shared `tapirnase` pw, RSA-login). **Audit 2026-05-19
|
||||
(read-only):**
|
||||
- **SNTP was broken → FIXED 2026-05-19.** Real root cause was **DNS**:
|
||||
servers were `0/1/2.de.pool.ntp.org` but the switch has no DNS
|
||||
resolver (`DNS_STATE=2`) so it couldn't resolve them. (The default
|
||||
gateway was *fine* — an active static default route `0.0.0.0/0 →
|
||||
.1` exists; the empty `Default_Route_Gateway=[]` is just the unused
|
||||
*interface-level* gateway field — misleading.) Clock had been stuck
|
||||
at **31/08/2025** (~8.5 mo). Fix: repointed to the **pinned NTP
|
||||
source standard** (see below), all by IP, DNS-free. **Verified
|
||||
synced** 2026-05-19 — clock corrected to live time (CEST/UTC+2).
|
||||
- **HTTPS/SSL disabled** — admin creds cross the flat LAN in clear.
|
||||
*Pending* (see roadmap: mgmt-TLS certs for Proxmox + D-Link).
|
||||
- **SNMP agent enabled** on the flat LAN (community strings unverified).
|
||||
*Pending.*
|
||||
- **Trusted-Host mgmt allowlist disabled** — admin UI reachable from
|
||||
any LAN device incl. IoT. *Pending:* enable allowing `192.168.1.0/24`.
|
||||
- ⚠️ The switch is a **transparent L2 switch** — every device on it
|
||||
**is fully visible to UniFi as a client** (MAC/IP via UDM DHCP/ARP);
|
||||
only the *wired port-topology through it* is invisible (needs UniFi
|
||||
adoption, impossible here). This is unlike the RE700X repeater.
|
||||
- 🔴 **The D-Link is the network CORE / single point of failure.**
|
||||
FDB-table analysis (2026-05-19): the **UDM uplinks on port 26**,
|
||||
the **entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all
|
||||
VMs/LXCs) is on port 10**, and the **U7 APs uplink on ports 3 & 16**
|
||||
(their wireless clients appear via those ports). Nearly all wired
|
||||
*and* wireless traffic transits this unmanaged switch → a D-Link
|
||||
failure = total LAN+WAN outage. Amplifies the mgmt-hardening,
|
||||
firmware-currency, monitoring and segmentation items below.
|
||||
- **UniFi↔D-Link DHCP/LLDP interop = fully compatible** (audited
|
||||
2026-05-19, read-only):
|
||||
- DHCP: `BOOTP_Relay_State=0` **and** `DHCP_BOOTP_Local_Relay_
|
||||
Status=0` → switch inserts **no Option 82**; UDM DHCP server
|
||||
gets clean untagged requests. (`Option82_State=1` is moot —
|
||||
only applies when relay/local-relay is active; both are off.)
|
||||
DHCP Server Screening has no entries → doesn't block the UDM
|
||||
(also = no rogue-DHCP protection: security note, not interop).
|
||||
- LLDP: D-Link LLDP enabled, **all 28 ports TX+RX** (`Port_Basic_
|
||||
Setting` mode 3); stats show 25 inserts / 16 ageouts → LLDP
|
||||
frames *are* exchanged & parsed with UniFi gear (no TLV
|
||||
incompat). "0 neighbors" earlier was just a point-in-time
|
||||
aged-out snapshot. UniFi still won't *draw* topology for an
|
||||
unadopted switch — cosmetic only.
|
||||
- RSTP: both run 802.1w (standard, interoperable). Tuning
|
||||
opportunity (not a fault): set a deliberate root-bridge
|
||||
priority — currently all default 32768 so root = lowest MAC
|
||||
(arbitrary). Make the D-Link (physical core) or UDM root.
|
||||
- Flow control: D-Link global FC `=2` (off/auto) — matches
|
||||
UniFi's default (802.3x disabled). Compatible.
|
||||
- **TP-Link RE700X** WiFi extender at `.187` (MACs `…1a:8a:0f/10/11`) —
|
||||
bridges/NATs devices behind it. **The Klipper 3D printer at `.189`
|
||||
sits behind the RE700X**, so UniFi only sees the repeater, never `.189`.
|
||||
- **DNS:** UniFi DHCP has `dhcpd_dns_enabled: false` (clients get the
|
||||
gateway `.1` as resolver, not AdGuard directly) — see DNS note below.
|
||||
|
||||
**Infrastructure IPs known from UniFi:**
|
||||
| IP | Name / Hostname | Notes |
|
||||
|---|---|---|
|
||||
| .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 |
|
||||
| .4 | Zoraxy LXC 108 | Reverse proxy |
|
||||
| .20 | Proxmox node `nuc` | PVE 9.1.11 |
|
||||
| .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI |
|
||||
| .41 | Nextcloud LXC 105 | Named "NextCloud", Proxmox OUI |
|
||||
| .49 | Shepard Docker host | Secondary Docker host |
|
||||
| .50 | U7-Pro-Wall AP | Hallway/entry |
|
||||
| .51 | U7 In-Wall AP | In-wall, UAPA6A5 |
|
||||
| .52 | U7 Mesh (Schlafzimmer) | Bedroom |
|
||||
| .53 | U7 Mesh (Esszimmer) | Dining room |
|
||||
| .60 | Home Assistant OS VM | haos CTID 100 |
|
||||
| .62 | Siemens oven | BSH Hausgeräte, WiFi |
|
||||
| .66 | Tibber Pulse | Energy monitor (Espressif) |
|
||||
| .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) |
|
||||
| .124 | L0018 | Wired, unknown device |
|
||||
| .143 | — | Sony Interactive Ent. (PlayStation) |
|
||||
| .164 | C100_7614A4 | Tapo C100 camera (TP-Link) |
|
||||
| .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) |
|
||||
| .187 | RE700X | TP-Link WiFi extender — NATs devices behind it |
|
||||
| .189 | — | Klipper 3D printer (behind RE700X, invisible to UniFi) |
|
||||
| .192 | REDMI-Note-15-Pro-5G | Xiaomi phone |
|
||||
| .241 | VS9-EU-MNA3478A | Dyson purifier/fan |
|
||||
- **Zoraxy** reverse proxy at `192.168.1.4:8000` — wildcard `*.nuclide.systems`
|
||||
cert + rules. Source of truth: `proxy/zoraxy/routes.json` (+ idempotent
|
||||
`scripts/zoraxy_sync.py --apply`). Cross-host, so no Docker labels.
|
||||
- **Ubiquiti UNAS** — NFS server `192.168.1.31:/var/nfs/shared/storage`
|
||||
(mounted `/mnt/pve/unas`, ~19T). Bulk/storage (arr media, qdrant, configs);
|
||||
several stacks bind data dirs here. ⚠️ SQLite-on-NFS is fragile here
|
||||
(see n8n note under Operational rules).
|
||||
- **Auth** — **PocketID** (`id.nuclide.systems`, OIDC, SQLite) is the
|
||||
**universal SSO/IdP for the entire ecosystem** — effectively every service
|
||||
authenticates via PocketID OIDC (LiteLLM, LobeChat, MCP gateway, Nextcloud,
|
||||
Daytona via the Keycloak→PocketID OIDC adapter, etc.). Single sign-on
|
||||
everywhere; one identity source to secure/audit.
|
||||
|
||||
## Docker networks
|
||||
`ai-internal` (AI/MCP plane) · `shared_backend` (cross-stack DB/S3) ·
|
||||
plus per-stack: `arr-stack_default`, `daytona-minimal_daytona-network`,
|
||||
`immich_default`, `karakeep_default`, `homepage_default`, `vpn_default`.
|
||||
The MCP gateway bridges `ai-internal` + `shared_backend`.
|
||||
|
||||
## AI / Agent platform (the core, `/opt/stacks/ai`)
|
||||
- **LiteLLM** (`ai.nuclide.systems`, :14000) — model gateway. ~28 curated
|
||||
models (provider allowlist), SAIA-terminal **failovers** + `num_retries`,
|
||||
Redis completion cache, Gemini €10/30d `provider_budget_config`. Config:
|
||||
`ai/litellm-config/config.yaml` (+ DB overlay `LiteLLM_Config`).
|
||||
- **syncstack** (`ai/syncstack.py`, cron `/etc/cron.d/syncstack` 15 min) —
|
||||
**model syncer/optimizer ONLY** (curation/allowlist → DB, `OPENAI_MODEL_LIST`
|
||||
→ `.env`, recreate lobe, model-health → ntfy). One-shot container.
|
||||
- **mcp-gateway / Agent Operator** (`mcp.nuclide.systems`, :8080) — **all
|
||||
things MCP**. DinD-spawns MCP servers on `ai-internal`, OAuth-gated streaming
|
||||
reverse proxy `/<server>/mcp` + `/group/<g>/mcp`, multi-group, health-watcher,
|
||||
per-server tools/list cache. **Deep health-check (2026-05-19):** beyond
|
||||
shallow liveness, the watcher drives a real MCP session (`initialize` →
|
||||
`tools/list` → optional `health_probe_tool` no-side-effect call) against
|
||||
**every** server incl. spawned ones, recording `ok|degraded|down` in
|
||||
`/api/servers`. Usage-aware: a server used within 5 min isn't probed
|
||||
(traffic = health); idle servers probed every 15 min; probe self-excludes
|
||||
via `X-Health-Probe`. Deep failures are status-only (no restart loops);
|
||||
shallow-liveness auto-restart stays static-only. First run caught the
|
||||
invisible class: `memos` degraded (search_memo → DNS `Errno -2`, the
|
||||
ai-internal isolation bug), plus `context7`/`crawl4ai`/`markitdown` down
|
||||
and `nextcloud` probe false-positive (probe lacks per-user NC creds —
|
||||
candidate for `health_check:false`). **Semantic tool retrieval**: groups with >25
|
||||
aggregated tools expose just `find_tools`/`call_tool` meta-tools (LiteLLM
|
||||
embedding index, cosine top-k, two-phase schema promotion) instead of
|
||||
flooding clients with hundreds of schemas. Group members are proxied via the
|
||||
gateway's own path (`_mcp_gw`) so `exact_upstream` servers (home-assistant,
|
||||
n8n) aggregate correctly. **MCP→LobeChat registration** (migrated here
|
||||
from syncstack), and the **Agent Operator**: cron+event agents
|
||||
(`/api/agents/{name}/run` webhook), LLM↔MCP executor, **Agent Creator**
|
||||
(AI-drafts agent specs; overridable meta-prompt), prompt export. Config:
|
||||
`mcp-gateway/config.json` (servers), `agents.json` + `prompts/` (agents).
|
||||
- **MCP servers** (gateway-managed): time, home-assistant (exact_upstream to
|
||||
`.60` ha-mcp), kroki (self-hosted: kroki+mermaid+excalidraw companions),
|
||||
daytona, fetch, sequential-thinking, ntfy, memos, docling, markitdown,
|
||||
crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky (brianellin,
|
||||
rate-limit pending). Groups incl. `shepard-mcp`.
|
||||
- **Daytona** (`daytona.nuclide.systems`, daytona-minimal stack) — secure
|
||||
code sandboxes (Agent Operator deep-dives, shepard). Registry fixed to
|
||||
internal+host `:5000` (was misconfigured at 6000). Snapshot `sciviz-py`.
|
||||
- **LobeChat** (`chat.nuclide.systems`, :14001) — chat UI; MCP plugins +
|
||||
agents in `lobechat` DB (lobe-postgres). Agents scriptable via
|
||||
`scripts/lobe_agent.py`. Native QStash scheduler NOT functional here →
|
||||
scheduling lives in the gateway Agent Operator. **Auth = Better Auth**
|
||||
(not NextAuth) SSO via PocketID (`generic-oidc`, client `26f3c26b`, PKCE
|
||||
S256). The `state_not_found` callback bug was fixed by setting
|
||||
`AUTH_TRUSTED_ORIGINS=https://chat.nuclide.systems`; verified 2026-05-19
|
||||
(sign-in yields a correct PocketID authorize URL; callback now validates
|
||||
with `oAuth_code_missing`, not `state_not_found`). Definitive proof =
|
||||
one real browser login. NB: `/oidc/auth` is LobeChat-*as*-OIDC-provider
|
||||
(desktop/mobile), distinct from this SSO-client path.
|
||||
- **ComfyUI** (img gen, Intel Arc iGPU), **Kroki**, **searxng** (self-hosted
|
||||
meta-search).
|
||||
|
||||
## Data layer
|
||||
- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
|
||||
**primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
|
||||
daytona, **memos** (migrated 2026-05-19). Superuser
|
||||
`postgres` via unix socket; per-app dedicated role+db (role owns its db,
|
||||
password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
|
||||
shared_buffers 768M, max_conn 200).
|
||||
⚠️ **n8n was NOT successfully migrated** — n8n's `export/import` CLI only
|
||||
covers workflows+credentials, **not users/settings/SSO**, so the PG
|
||||
cutover lost the owner account + OIDC. **Reverted to local-disk SQLite**
|
||||
(tier-1 still satisfied, like PocketID). Compounding: unpinned
|
||||
`n8nio/n8n:latest` had drifted 2.7→**2.20.11**, breaking the custom OIDC
|
||||
`hooks.js` (hardcoded old `/usr/local/lib` module paths) → n8n crash-looped.
|
||||
Fixed: image pinned to `2.20.11`, `hooks.js` paths patched for 2.20.11
|
||||
pnpm layout (`/usr/lib/node_modules/n8n/...`), **OIDC hook re-enabled
|
||||
2026-05-19** — **verified**: `/auth/oidc/login` → 302 to PocketID
|
||||
(client `33135ad4`, correct redirect/scope/state/nonce). **Lessons:
|
||||
(1) only a FULL pgloader migration of *all*
|
||||
tables is complete — partial `export/import` is not; (2) pin critical
|
||||
images — `latest` drift is a real outage cause.**
|
||||
Migration note: services without a native full SQLite→PG export use
|
||||
**pgloader data-only of ALL tables** into the app-built schema (exclude
|
||||
only the app's own migration-tracking table), app role temp-SUPERUSER for
|
||||
the load (FK/trigger disable) then reverted.
|
||||
- **lobe-postgres** (pg17/ParadeDB) — LobeChat. **lobe-redis** — LiteLLM cache
|
||||
+ LobeChat. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
|
||||
`:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
|
||||
on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
|
||||
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
|
||||
lobe-postgres, immich_postgres); daily 3am cron (`/etc/cron.d/pg-backup`).
|
||||
**Incident + fix (2026-05-19):** Garage stored `meta`+`data` on the UNAS
|
||||
NFS → NFS stalls hung `wal-g wal-push` → archivers hung since 2026-05-18
|
||||
14:51 (`failed_count=0` = hung not failing), zero base backups, stalled
|
||||
WAL recycled (that window unrecoverable). **Resolved:** Garage moved to
|
||||
local disk (`/opt/stacks/shared-db/garage/{meta,data}`); all 3 archivers
|
||||
drained; `archive_command` hardened to `timeout 60 wal-g wal-push %p`
|
||||
(fail-fast vs infinite hang); fresh base backups taken for all 3. Extra
|
||||
safety net: local logical dumps in `/opt/stacks/backups/shared-pg/`.
|
||||
Lesson: Garage metadata is fsync/lock-heavy — **never** on NFS, same rule
|
||||
as SQLite.
|
||||
|
||||
### Database design / standards
|
||||
- **Default = Postgres on `shared-postgres`.** Any app needing persistence
|
||||
that supports Postgres gets a dedicated `role`+`database` there. SQLite is
|
||||
allowed **only** when the app has no Postgres support, and then **only on
|
||||
local disk** — never the UNAS NFS share (broken POSIX/SMB file locking;
|
||||
caused the n8n outage + PocketID latent risk).
|
||||
- **Provisioning pattern:** `CREATE ROLE <app> LOGIN PASSWORD …; CREATE
|
||||
DATABASE <app> OWNER <app>;` → set the app's `DB_*` env, password lives in
|
||||
that stack's `.env` (vault migration is the long-term plan).
|
||||
- **Why standardize:** unified, durable **WAL-G PITR backup** — the single
|
||||
biggest reason. One backup story instead of N un-backed-up SQLite files.
|
||||
- **Migration reality:** few apps have a native SQLite→PG data port. n8n does
|
||||
(`export/import` CLI). Most (PocketID, Vaultwarden, traccar, …) do not —
|
||||
switching `DB_PROVIDER` starts a *fresh* DB; data port needs
|
||||
`pgloader`/app-specific tooling. **Memos** done via pgloader (data-only,
|
||||
app-built schema, role temp-SUPERUSER for the load). When no safe port
|
||||
exists, fall back to local-disk SQLite (PocketID) until a port is built.
|
||||
|
||||
### NFS / storage strategy (corrected 2026-05-19)
|
||||
- The Docker host is an **unprivileged Proxmox LXC (104)** → it **cannot do
|
||||
in-container NFS mounts** (kernel denies; tested — `operation not
|
||||
permitted`). That's why Proxmox NFS-mounts at the host and bind-mounts
|
||||
`/mnt/pve/unas` in. **Consequence:** per-stack "Docker NFS volumes at
|
||||
v4.1" (the old tier-3 idea) is **infeasible** here without
|
||||
`pct set 104 --features mount=nfs` + a full LXC reboot (all stacks down).
|
||||
- **Tier 1 is the rule and does the real work:** DBs/metadata/lock- or
|
||||
fsync-heavy stores → local ZFS or shared-postgres, **never NFS** (any
|
||||
version — SQLite/LMDB/sled are unsafe on NFS regardless of tuning).
|
||||
- **NFSv4 is NOT available — the Ubiquiti UNAS Pro is NFSv3-only** (current
|
||||
firmware; verified 2026-05-19 on node `nuc`: `showmount -e` works but
|
||||
`mount :/ -o vers=4.1` fails server-side `No such file or directory`, i.e.
|
||||
no v4 pseudo-root). So tier-2 (host v3→v4.1) and tier-3 (Docker NFS v4.1
|
||||
volumes) are **both dead ends** — no v4 to upgrade to, no reboot worth
|
||||
doing for it. Real export path is `/volume/<uuid>/.srv/.unifi-drive/
|
||||
storage/.data` (Proxmox `unas` storage uses the `/var/nfs/shared/storage`
|
||||
alias, works on v3 — leave it).
|
||||
- **Therefore tier-1 is the whole strategy.** NFSv3 stays for bulk/
|
||||
sequential data (fine for that). Optional marginal v3 tuning: `nconnect=4`
|
||||
for throughput — not required.
|
||||
- **Tier-1 complete (2026-05-19):** all SQLite-on-NFS backlog resolved —
|
||||
ntfy, Karakeep, arcane → local disk; **Vaultwarden → shared-postgres**
|
||||
(pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem);
|
||||
traccar data → local disk. PocketID stays on local SQLite (no native
|
||||
PG migration path, deferred).
|
||||
|
||||
## Services
|
||||
Nextcloud (LXC .41), **Paperless-ngx** (+paperless-ai, tika/gotenberg),
|
||||
**Immich** (server/ML/redis/postgres/power-tools), **Memos**, **Karakeep**
|
||||
(+chrome), **Vaultwarden**, **n8n**, **Home Assistant** (.60, ~2492 entities),
|
||||
**ntfy** (`homelab-ai` topic — model-health + agent alerts), **traccar** (GPS, HTTP :15000 / watch :15001),
|
||||
**arr-stack** (prowlarr/shelfarr/flaresolverr behind gluetun VPN), **streamio**,
|
||||
**Arcane** (Docker mgmt), **Dozzle** (logs), **Homepage** (dashboard, 6 groups),
|
||||
**Daytona OIDC adapter** (Keycloak→PocketID PKCE proxy for the VS Code ext).
|
||||
|
||||
## Operational rules / conventions
|
||||
- **Separation**: `syncstack` = models; `mcp-gateway` = all MCP/agents.
|
||||
- **Self-host first**: external/cloud only when no self-hosted form exists.
|
||||
- **Pinned NTP source standard** — all network gear/infra should point at
|
||||
the *same fixed-IP, DNS-free* NTP sources (devices like the D-Link can't
|
||||
resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):
|
||||
1. `192.53.103.108` — PTB ptbtime1 (DE national time)
|
||||
2. `192.53.103.104` — PTB ptbtime2 (DE national time)
|
||||
3. `162.159.200.123` — Cloudflare NTP anycast (fallback)
|
||||
Applied to the D-Link switch 2026-05-19. UDM + other infra should converge
|
||||
on the same set (a self-hosted LAN NTP server is a roadmap option, but the
|
||||
standard stays "same pinned sources" everywhere).
|
||||
- Run Python with `uv run` (not python3) in `/opt/stacks`.
|
||||
- Zoraxy self-service: edit `routes.json` → `scripts/zoraxy_sync.py --apply`.
|
||||
- Secrets in `ai/.env` (plaintext — **env-audit/secret-vault is an open
|
||||
hardening task**; Vaultwarden available). Note: a UniFi MFA JWT has leaked
|
||||
into `homepage/config/logs/homepage.log` — rotate + scrub when hardening.
|
||||
⚠️ **Shared-password reuse:** `tapirnase` is reused as the WiFi PSK, the
|
||||
LiteLLM master key root (`sk-tapirnase`), and the D-Link switch admin pw —
|
||||
single sniff/leak has broad blast radius. Rotate per-service when hardening
|
||||
(noted 2026-05-19, rotation deferred per user).
|
||||
- **Proxmox/UniFi safety**: any write/change to the Proxmox host or Ubiquiti
|
||||
network gear requires explicit confirmation + a stated risk assessment
|
||||
first; read-only queries are fine. High blast radius (foundation layer).
|
||||
- **SQLite-on-NFS is unsafe here**: the UNAS NFS share has unreliable file
|
||||
locking (true for SMB/CIFS too). Any SQLite app must use `shared-postgres`
|
||||
or local disk, never the NFS share. **n8n** hit this (`Database connection
|
||||
timed out`) and was **migrated to `shared-postgres`** (DB on local ZFS;
|
||||
data dir moved to local `/opt/stacks/n8n/data`; old NFS `database.sqlite`
|
||||
kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.
|
||||
- Naming debt: "mcp-gateway"/"syncstack" have outgrown their names — rename
|
||||
once the agent-platform identity firms up.
|
||||
|
||||
## Open / roadmap
|
||||
mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration +
|
||||
expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers
|
||||
(in progress); **UniFi MCP** — COMPLETE 2026-05-19 (`ghcr.io/enuno/unifi-mcp-server`,
|
||||
197 tools, Network App API key, `http` transport; local API fully working).
|
||||
**Tier-1 SQLite-off-NFS: COMPLETE** — all services off NFS for DB/metadata.
|
||||
Note: Traccar watch protocol port 15001 is already forwarded at the UDM level —
|
||||
no Zoraxy stream proxy needed. Plus: disable LobeChat external features (keep
|
||||
`plugins`!); env → secret vault.
|
||||
|
||||
**Management-plane TLS (planned):** issue/trust proper certs for admin-UI
|
||||
auth on the **Proxmox** host and the **D-Link DGS-1210** (currently HTTP-only
|
||||
on the D-Link → admin creds in clear on the flat LAN; Proxmox self-signed).
|
||||
Brings switch/hypervisor mgmt onto the `*.nuclide.systems` PKI like the rest.
|
||||
|
||||
**Network segmentation (planned):** the LAN is flat — servers, IoT (Siemens
|
||||
oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (`192.168.1.0/24`,
|
||||
no VLAN, no L2 isolation). Plan an **IoT VLAN** (+ matching firewall zone) so
|
||||
untrusted appliances can't reach the server/Proxmox subnet. Complications to
|
||||
design around: the non-UniFi **D-Link DGS-1210-28P** switch (`.10`) and
|
||||
**TP-Link RE700X** extender (`.187`, NATs the Klipper printer `.189`) won't
|
||||
honour UniFi VLAN tags natively — segmentation needs a plan for the wired
|
||||
trunk through the D-Link and the repeater's bridge mode.
|
||||
|
||||
### Observability LXC (planned — scoped from this session's incidents)
|
||||
**Why it's now a priority:** the WAL-G archiver was hung **silently for
|
||||
~13 h** (`failed_count=0`, zero base backups) and would never have been
|
||||
noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring
|
||||
must target *exactly these silent-failure classes*.
|
||||
- **Placement:** its **own LXC on node `nuc`**, NOT inside LXC 104 — 104
|
||||
hosts everything, so the monitor must survive/alert when 104 is down.
|
||||
- **Stack:** VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager
|
||||
→ **ntfy `homelab-ai`** (already the alert channel).
|
||||
- **Exporters/probes:** node_exporter (per host + key LXCs), postgres_exporter
|
||||
×3 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for
|
||||
Zoraxy wildcard), **pve-exporter** (use the `root@pam!mcp` PVEAuditor
|
||||
token), and a **custom WAL-G/archiver textfile collector**:
|
||||
`pg_stat_archiver` (last_archived age, `failed_count`), `.ready` backlog,
|
||||
and `wal-g backup-list` newest-base age — per PG instance.
|
||||
- **Alerts (priority order, derived from real incidents):**
|
||||
1. WAL archiver stalled (last_archived age > 15 m) or newest base backup
|
||||
> 26 h, any PG instance.
|
||||
2. NFS mount on `/mnt/pve/unas` unresponsive / high op latency.
|
||||
3. **Config-drift guard:** any `*.db`/`*.sqlite*` appears under
|
||||
`/mnt/pve/unas` (catches a regression of the tier-1 rule).
|
||||
4. Container unhealthy/restart-looping > 5 m (the gluetun pattern).
|
||||
5. local-zfs `rpool` or NFS pool > 85 %; 6. TLS cert < 14 d.
|
||||
@@ -1,127 +0,0 @@
|
||||
# MCP Gateway — Reconstructed Design Spec (in-progress, "Phase 1")
|
||||
|
||||
> Reconstructed 2026-05-16 from code/configs/git history. The gateway is a
|
||||
> single-squash-commit first draft (`0cad389 "Phase 1: Create MCP Gateway with
|
||||
> Docker-in-Docker support"`, preceded by `726bd10 "WIP: MCP gateway prep"`).
|
||||
> Nothing has a second iteration in git — everything below is first-draft intent.
|
||||
|
||||
## 1. Goal / Intent
|
||||
|
||||
A single **OAuth-protected HTTP entrypoint at `https://mcp.nuclide.systems`** that
|
||||
exposes a curated set of MCP servers to AI clients on the homelab. Primary
|
||||
consumer: **Claude.ai** as a remote connector (SSE at `/`, every README's
|
||||
"Usage in Claude.ai"). Secondary: **LobeChat** (`chat.nuclide.systems`) and
|
||||
**LiteLLM** (`ai.nuclide.systems`), sharing the same Pocket ID OAuth app. It is
|
||||
meant to replace the "cumbersome" static-compose approach (`mcp-tools.yaml`)
|
||||
with a dynamic, UI-managed, self-hosting model — answering the open `todo.md`
|
||||
question "MCP deployment seems cumbersome — can litellm host directly? how to
|
||||
integrate npx, uvx, docker-based containers?". Unifying idea: normalize
|
||||
npx / uvx / docker MCP servers behind one Dockerized gateway.
|
||||
|
||||
## 2. Architecture (three competing models in-repo; A chosen, B orphaned, C aspirational)
|
||||
|
||||
**A. FastAPI gateway + Docker-in-Docker (chosen)** — `ai/mcp-gateway/`
|
||||
- FastAPI + `uvicorn` on `0.0.0.0:8080`, container `mcp-gateway`.
|
||||
- DinD via bind-mounted `/var/run/docker.sock`; `docker.from_env()`.
|
||||
- Per-server containers spawned `mcp-<name>`, hardcoded onto `ai-internal`.
|
||||
- Config `config.json` (RW bind, currently EMPTY → falls back to `DEFAULT_SERVERS`).
|
||||
- Gateway joins `ai-internal` + `shared_backend` (both `external: true`).
|
||||
|
||||
**B. Static compose `mcp-tools.yaml`** — orphaned; `ai/docker-compose.yml:6`
|
||||
include is **commented out**. Internally malformed (see §4).
|
||||
|
||||
**C. LiteLLM-hosted** — `litellm-config/config.yaml` `mcp_servers: {}` empty.
|
||||
Confirms MCP hosting was intended for the gateway, not LiteLLM (the
|
||||
`todo.md` "can litellm host directly?" question remains open).
|
||||
|
||||
**Transports** (normalized to HTTP-on-:8000): streamable-http (nextcloud,
|
||||
mermaid), `mcp-proxy --stateless` stdio→HTTP (papersearch), native HTTP
|
||||
(markitdown, crawl4ai :11235), and the gateway's own **SSE `/` endpoint —
|
||||
a STUB** (fake `initialize` + 60s pings, no routing to backends).
|
||||
|
||||
**Reverse proxy:** Zoraxy `mcp.nuclide.systems → 192.168.1.40:8080`.
|
||||
`mcp-auth.nuclide.systems` is an abandoned auth-sidecar idea (not exposed).
|
||||
|
||||
**OAuth (Pocket ID @ `id.nuclide.systems`):** `OAuth2AuthorizationCodeBearer`,
|
||||
scopes `{openid, mcp}`, token validation via userinfo. Shared gateway client
|
||||
(`GENERIC_CLIENT_ID`, same as LiteLLM/LobeChat). Per-server OAuth for
|
||||
papersearch & nextcloud against the same Pocket ID.
|
||||
|
||||
## 3. MCP Server Inventory (reconciled — `server.py` MCP_SERVERS is authoritative)
|
||||
|
||||
| Server | Image / build | Transport | Port | Auth | Status |
|
||||
|---|---|---|---|---|---|
|
||||
| papersearch | `python:3.12-slim` + runtime `uv tool install mcp-proxy` → `paper_search_mcp.server` | mcp-proxy stdio→http | 8000 | Pocket ID `PAPERSEARCH_MCP_OAUTH_*` | plausible, runtime-install fragile |
|
||||
| nextcloud | `ghcr.io/cbcoutinho/nextcloud-mcp-server:latest` | streamable-http | 8000 | Pocket ID `NEXTCLOUD_MCP_OAUTH_*` | likely workable (real image) |
|
||||
| markitdown | `python:3.12-slim` + `uvx markitdown-mcp --http` | http | 8000 | none | **broken as written** (`uvx` not in base image) |
|
||||
| comfyui | `ghcr.io/richardi-ai/comfyui-mcp-server:latest` (`type:"npm"` mismatch) | unspecified | 8000 | none | **image not pullable**; backend ComfyUI was crash-looping |
|
||||
| crawl4ai | `unclecode/crawl4ai:latest` | http | 11235 | none | likely workable; resource limits lost in rewrite |
|
||||
| mermaid | `node:20-slim` + runtime `npx -y mcp-mermaid` | streamable-http | 8000 | none | plausible, slow first start |
|
||||
|
||||
## 4. Implemented vs Unfinished vs Broken
|
||||
|
||||
**Implemented:** FastAPI app + OAuth scheme + userinfo token validation;
|
||||
container lifecycle CRUD + persistence; Web UI SPA (`templates/ui.html` @ `/ui`);
|
||||
gateway compose/Dockerfile + Zoraxy route.
|
||||
|
||||
**Unfinished / stub:**
|
||||
- **SSE `/` is fake** — no MCP transport bridging Claude.ai → spawned servers. *Core gap.*
|
||||
- **No routing to per-server containers**; all five servers bind the same `:8000`
|
||||
and `spawn_container` host-publishes `8000:8000` → **two servers can't run at once**.
|
||||
- **OAuth callback non-functional** — token-exchange URL built via
|
||||
`OAUTH_REDIRECT_URI.replace("/sso/callback","/token")` (→ wrong host, not the
|
||||
Pocket ID token endpoint); token never stored/used.
|
||||
- `config.json` empty → always defaults; secrets hardcoded plaintext in `server.py`.
|
||||
|
||||
**Broken / contradictory:**
|
||||
- `ai/docker-compose.yml:6` mcp-tools include commented out; gateway compose is a
|
||||
*separate project* not referenced by the stack either — wired in only via Zoraxy.
|
||||
- `mcp-tools.yaml`: duplicate `markitdown-mcp` key; `comfyui-mcp` env missing `=`
|
||||
(`- COMFYUI_URL http://comfyui:8188`); missing images/ports.
|
||||
- `comfyui` server `type:"npm"` vs Docker-image mismatch; upstream image/npm
|
||||
package existence unverified (image confirmed **not pullable**).
|
||||
- `.env` has `CRAWL4AI_MCP_OAUTH_*`, `COMFYUI_MCP_OAUTH_*` that `server.py`
|
||||
never consumes; code hardcodes secrets instead of `${ENV}` substitution.
|
||||
|
||||
## 5. Relevant `.env` keys (names only)
|
||||
|
||||
Gateway: `GENERIC_CLIENT_ID/_SECRET/_REDIRECT_URI`,
|
||||
`GENERIC_{AUTHORIZATION,TOKEN,USERINFO}_ENDPOINT`, `GENERIC_CLIENT_USE_PKCE`,
|
||||
`OAUTH_SCOPES`, `OAUTH_TOKEN_URL`.
|
||||
Per-server: `NEXTCLOUD_MCP_OAUTH_CLIENT_ID/_SECRET`,
|
||||
`PAPERSEARCH_MCP_OAUTH_CLIENT_ID/_SECRET`,
|
||||
`MARKITDOWN_MCP_OAUTH_CLIENT_ID/_SECRET` (declared, unused),
|
||||
`CRAWL4AI_/COMFYUI_MCP_OAUTH_*` (orphaned).
|
||||
papersearch data sources: `UNPAYWALL_EMAIL`, `CORE_API_KEY`,
|
||||
`SEMANTIC_SCHOLAR_API_KEY`, `ZENODO_ACCESS_TOKEN`, `GOOGLE_SCHOLAR_PROXY_URL`,
|
||||
`DOAJ_API_KEY`. comfyui: `COMFYUI_URL`, `COMFYUI_WS_URL`.
|
||||
|
||||
## 6. Open Design Decisions (must resolve)
|
||||
|
||||
1. **Hosting model**: DinD gateway vs static `mcp-tools.yaml` vs LiteLLM-hosted.
|
||||
2. **How Claude.ai/LobeChat reach a tool**: the MCP transport bridge doesn't exist.
|
||||
3. **Port allocation**: all servers hardcode `:8000` — need internal DNS, no host publish.
|
||||
4. **Uniform npx/uvx/docker run model**: prebuilt images vs runtime install.
|
||||
5. **Auth model**: gateway-terminated vs per-MCP vs pass-through (callback is broken).
|
||||
6. **Secret handling**: hardcoded → `${ENV}` from `ai/.env`.
|
||||
7. **comfyui server**: image vs npm; keep only once ComfyUI itself is stable.
|
||||
8. **Discovery for LobeChat/LiteLLM**: `/mcp.json` is OAuth-gated, lists config not endpoints.
|
||||
|
||||
## 7. Recommended Path (ordered, lowest-risk first)
|
||||
|
||||
1. **Pick the static-compose path, not DinD** — lowest risk on a single NUC;
|
||||
DinD adds socket-exposure risk + a broken SSE bridge for little gain.
|
||||
Fix and re-enable `mcp-tools.yaml` (uncomment `ai/docker-compose.yml:6`).
|
||||
2. **Fix `mcp-tools.yaml`**: dedupe `markitdown-mcp`, fix `comfyui-mcp` env `=`,
|
||||
unique service names → `ai-internal` DNS, pin images, drop comfyui for now.
|
||||
3. **One streamable-http reverse proxy keyed by path** (`mcp.nuclide.systems/<server>`)
|
||||
via Zoraxy or a small `httpx` proxy — replace the fake SSE stub. Backends stay
|
||||
internal on `ai-internal:8000`, never host-published.
|
||||
4. **Move secrets to `${ENV}`** from `ai/.env` (keys already exist).
|
||||
5. **Fix OAuth callback**: exchange code against `GENERIC_TOKEN_ENDPOINT`.
|
||||
6. **Verify each upstream image/tool exists** before marking a server "working".
|
||||
7. **Defer the DinD gateway + Web UI** to phase-2 (read-only status over the
|
||||
running compose, not spawning).
|
||||
8. **Answer the litellm question**: once stable `https://mcp.nuclide.systems/<server>`
|
||||
URLs exist, populate `litellm-config/config.yaml` `mcp_servers:` so
|
||||
LiteLLM/LobeChat discover them — no bespoke discovery path needed.
|
||||
@@ -1,91 +0,0 @@
|
||||
# MCP Gateway v2 — Operational
|
||||
|
||||
Finished implementation of the DinD MCP gateway (Option A). Supersedes the
|
||||
"Phase 1" stub described in `mcp-gateway-requirements.md`.
|
||||
|
||||
## What it is
|
||||
|
||||
`ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`, public via Zoraxy at
|
||||
`https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). Single OAuth-gated
|
||||
entrypoint fronting several MCP servers.
|
||||
|
||||
## Architecture (as built)
|
||||
|
||||
- **Spawn model**: Docker-in-Docker. Each spawnable server runs as
|
||||
`mcp-<name>` on the **`ai-internal`** network with **no host ports**
|
||||
(fixes the old `8000:8000` collision — servers talk over internal DNS).
|
||||
- **Transport bridge**: generic streaming reverse proxy
|
||||
`https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`.
|
||||
Transport-agnostic — works for streamable-HTTP (`/mcp`) and legacy SSE
|
||||
(`/sse`+`/messages`). Registered LAST so specific routes win.
|
||||
- **Auth**: Pocket ID. Two paths off the same OIDC app:
|
||||
- **MCP clients / API / proxy** — `Bearer` token, validated via OIDC
|
||||
userinfo (300s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728
|
||||
`/.well-known/oauth-protected-resource` for discovery.
|
||||
- **Browser Web UI** — full Authorization-Code flow with an httpOnly
|
||||
session cookie: unauthenticated `/ui` → `302 /login` → Pocket ID
|
||||
`authorize` (CSRF `state` cookie) → `/sso/callback` (state-checked
|
||||
code→token exchange) sets the `mcp_session` cookie and shows the page +
|
||||
a copyable Bearer token for MCP clients. `/logout` clears it. The UI and
|
||||
`/api/*` accept **either** the cookie or a Bearer header; the `/{srv}/`
|
||||
proxy stays Bearer-only. `/health` + the well-known metadata are open.
|
||||
- **Secrets**: all per-server OAuth client secrets come from `ai/.env`
|
||||
(`env_file: ../.env`), read via `os.environ` — none hardcoded in source.
|
||||
|
||||
## Server registry (`SERVERS` in server.py)
|
||||
|
||||
| name | kind | upstream | notes |
|
||||
|---|---|---|---|
|
||||
| comfyui | static | `http://comfyui-mcp:8000` | runs as its own stack (`ai/comfyui-mcp.yml`); gateway only proxies |
|
||||
| nextcloud | spawn | `mcp-nextcloud:8000` | `ghcr.io/cbcoutinho/nextcloud-mcp-server` streamable-http |
|
||||
| crawl4ai | spawn | `mcp-crawl4ai:11235` | `unclecode/crawl4ai` |
|
||||
| mermaid | spawn | `mcp-mermaid:8000` | `node:20-slim` + npx mcp-mermaid (validated working) |
|
||||
| markitdown | spawn | `mcp-markitdown:8000` | `ghcr.io/astral-sh/uv` + uvx markitdown-mcp |
|
||||
| papersearch | spawn | `mcp-papersearch:8000` | python + uv + mcp-proxy → paper_search_mcp |
|
||||
|
||||
## Verified
|
||||
|
||||
- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth).
|
||||
- `/mcp.json`, `/<server>/mcp` → 401 without token; 401 with bad token
|
||||
(userinfo path); correct `WWW-Authenticate`.
|
||||
- Upstream path proven: gateway → `comfyui-mcp` `/mcp` initialize → 200;
|
||||
gateway → spawned `mcp-mermaid` `/mcp` → 200 (spawn pattern + proxy).
|
||||
- Route precedence correct (specific routes beat the catch-all proxy).
|
||||
|
||||
## Not yet exercised (needs a real token / operational bring-up)
|
||||
|
||||
- **Token-authenticated end-to-end** requires an interactive Pocket ID login.
|
||||
Get a token: open `https://mcp.nuclide.systems/sso/callback` via the OAuth
|
||||
authorize flow (or any Pocket ID token with scopes `openid,mcp`); the
|
||||
callback page prints the access token.
|
||||
- **Bring up the 4 remaining spawnable servers** (nextcloud/crawl4ai/
|
||||
markitdown/papersearch): `POST /api/servers/<name>/start` (bearer) or
|
||||
`POST /api/servers/bulk/start`. Each may need a transport/flag tweak on
|
||||
first run — mermaid is the proven reference.
|
||||
|
||||
## Client registration
|
||||
|
||||
- **LobeChat**: Settings → Skills → Skill Store → Custom → Import JSON, e.g.
|
||||
```json
|
||||
{ "mcpServers": { "comfyui": { "type": "http",
|
||||
"url": "https://mcp.nuclide.systems/comfyui/mcp",
|
||||
"auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
|
||||
```
|
||||
(Or, for ComfyUI specifically, connect LobeChat straight to
|
||||
`http://comfyui-mcp:8000/mcp` internally — see `comfyui-mcp.md` — no token.)
|
||||
- **Claude.ai**: add `https://mcp.nuclide.systems/<server>/mcp` as a custom
|
||||
connector; it will discover auth via the protected-resource metadata.
|
||||
|
||||
## Ops
|
||||
|
||||
- `cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build`
|
||||
- Logs: `docker logs mcp-gateway`; UI: `https://mcp.nuclide.systems/ui`
|
||||
(Pocket-ID-gated; `/login` to sign in).
|
||||
- Config overlay persisted in `ai/mcp-gateway/config.json`.
|
||||
- **UI check (Playwright)** — the live UI is auth-gated, so the check
|
||||
serves the local template + mocks `/api/servers`, screenshots, and
|
||||
asserts cards render:
|
||||
```
|
||||
uv run --with playwright -- python -m playwright install chromium # once
|
||||
uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png
|
||||
```
|
||||
@@ -1,53 +0,0 @@
|
||||
# Scrubbing List — /opt/stacks (2026-05-17)
|
||||
|
||||
Read-only audit of unused/stale data. **Nothing here has been deleted** — review
|
||||
the labels and run the commands yourself. Root FS was **165G/200G used (83%)**;
|
||||
`/var/lib/docker` is 86G of `/opt/stacks`'s 99G.
|
||||
|
||||
No stopped/exited/`*_old` containers; no unused custom networks (already clean).
|
||||
|
||||
## 1. Docker reclaimable
|
||||
|
||||
| Target | Size | Label | Command |
|
||||
|---|---|---|---|
|
||||
| Build cache (268 entries, old comfyui CPU→Arc rebuilds, 0 in use) | **~20.85 GB** | **SAFE** | `docker builder prune -af` |
|
||||
| 4× dangling `<none>` mcp-gateway rebuild images (1.59 GB ea) | **~6.36 GB** | **SAFE** | `docker image prune` |
|
||||
| 2 old dangling images (756 MB + 113 MB) | **~0.87 GB** | **SAFE** | (same `docker image prune`) |
|
||||
| ~265 anon volumes; one `54bcf7…` = **8.69 GB** unidentified, rest ~0B | ~9 GB | **REVIEW** | inspect `54bcf7…` then `docker volume prune` |
|
||||
| Named dangling vols: `n8n_n8n_storage` 128M, `ai_mcpo-data` 84M, `paperless-ngx_pgdata` 26M, `metamcp_postgres_data` 18M, `daytona*_db_data` 15M×2, `librechat_pgdata2` 13M, `arcane_arcane-data` 12M, `ai_redis_data` 7.6M | ~0.3 GB | **REVIEW** | `docker volume rm <name>` per-item after confirming the stack is retired |
|
||||
|
||||
In-use, **DO NOT REMOVE**: `comfyui-comfyui` (6.45G), `clusterzx/paperless-ai` (8.59G).
|
||||
|
||||
## 2. Migrated/abandoned local data dirs (compose now points to UNAS)
|
||||
|
||||
| Path | Size | Label | Note |
|
||||
|---|---|---|---|
|
||||
| `arr-stack/media` | **55 G** | **REVIEW** | No container mounts it; arr → `/mnt/pve/unas/media`. Audiobooks/ebooks have recent mtimes (rsync residue) — **parity-check vs UNAS before `rm -rf`** |
|
||||
| `ai/data` (old postgres) | 195 M | **SAFE** | Not mounted, not referenced |
|
||||
| `ai/postgres_data` (incl 38M pg_wal) | 120 M | **REVIEW** | Not mounted/referenced but recent mtime |
|
||||
| `ai/meili_data_v1.35.1` | 19 M | **SAFE** | Old Meili, not mounted |
|
||||
| `qdrant/qdrant_storage` | 7 M | **SAFE** | qdrant migrated to UNAS (fresh start) |
|
||||
| `daytona/db_data` | 14 M | **REVIEW** | No mount; daytona uses named volumes |
|
||||
| `n8n/data` | empty | **SAFE** | `rmdir` |
|
||||
|
||||
Active local, **KEEP**: `immich/postgres` (814M), `ai/lobehub/data` (25M),
|
||||
`shared-db/wal-g` (27M, RO mount), arr-stack configs, `ai/litellm-config`,
|
||||
`ai/searxng`.
|
||||
|
||||
## 3. Migration / scratch artifacts
|
||||
|
||||
| Path | Label | Note |
|
||||
|---|---|---|
|
||||
| `ai/docker` (0 B), `ai/bucket.config.json` (empty dir) | **SAFE** | junk |
|
||||
| `scripts/traefik-*.sh` | **SAFE** | Traefik abandoned for Zoraxy |
|
||||
| `scripts/{migrate_*,test_adguard_api,zoraxy_csrf,zoraxy_test,configure_zoraxy_*}.py` | **REVIEW** | one-shot done; confirm no rerun need |
|
||||
| `scripts/zoraxy_sync.py` | **KEEP** | ongoing proxy tooling |
|
||||
| `scripts/.venv` (29M), `scripts/.kilo` (30M) | **REVIEW** | regenerable caches |
|
||||
| `/tmp/{flux_*,pw_ui,add_*,fix_*}.* , /tmp/*.png , /tmp/*.log` | **SAFE** | ~1.8M scratch (this session) |
|
||||
|
||||
## Bottom line
|
||||
|
||||
- **Safe-only reclaim: ~29–30 GB** (build cache + dangling images dominate).
|
||||
- **With review: +~64 GB** (55G stale `arr-stack/media` after UNAS parity check, ~9G anon vols).
|
||||
- **Highest-value, zero-risk action:** `docker builder prune -af` → ~20.85 GB now.
|
||||
- Biggest overall prize: verify+remove `arr-stack/media` (55 G) — parity-check vs `/mnt/pve/unas/media` first.
|
||||
@@ -1,248 +0,0 @@
|
||||
# Stack Ideas & Improvements
|
||||
|
||||
Hardware baseline: **NUC 14 Pro** — Intel Core Ultra (Meteor Lake), Intel Arc iGPU
|
||||
(currently used for ComfyUI / XPU image gen), no NVIDIA, UNAS for media/bulk storage.
|
||||
|
||||
---
|
||||
|
||||
## 1. Document ingestion → LobeChat knowledge base + MCP
|
||||
|
||||
**Goal**: ingest PDFs, Word, PPT, XLS → searchable in LobeChat chat + accessible via MCP.
|
||||
Qdrant is a nice-to-have, not a requirement (no live pipeline uses it yet).
|
||||
|
||||
**LobeChat already has a built-in knowledge base** (`knowledge_base_files`, `chunks`,
|
||||
`embeddings` tables in Postgres). The missing piece is an ingest pipeline that feeds it.
|
||||
|
||||
**Recommended architecture**:
|
||||
```
|
||||
Nextcloud folder / Paperless webhook
|
||||
→ n8n trigger (already running)
|
||||
→ Docling (PDF/DOCX/PPTX/XLSX → Markdown + structure)
|
||||
→ LiteLLM /v1/embeddings (Mistral-embed / codestral-embed)
|
||||
→ LobeChat knowledge base API ←— available in chat natively
|
||||
→ Qdrant sidecar (optional, if multi-app search needed)
|
||||
```
|
||||
|
||||
**Document conversion options**:
|
||||
|
||||
| Tool | Docker image | Formats | NUC 14 CPU fit |
|
||||
|------|-------------|---------|----------------|
|
||||
| **Docling** (IBM, 2024) | `ghcr.io/ds4sd/docling` | PDF, DOCX, PPTX, XLSX, HTML, Markdown | ✅ CPU-only, 10-60s/doc |
|
||||
| **MinerU** (OpenDataLab) | `opendatalab/mineru` | PDF (layout-aware, OCR) | ✅ CPU mode; GPU optional for speed |
|
||||
| **Unstructured** | `quay.io/unstructured-io/unstructured-api` | Very broad (25+ formats) | ✅ lighter than Docling |
|
||||
| **Markitdown** (already in MCP gateway) | — | Office + PDFs → Markdown | ✅ ad-hoc only, not batch |
|
||||
|
||||
**Docling vs MinerU**: Docling is better for structured Office/PDF with tables and
|
||||
figures. MinerU (OpenDataLab) is better for pure PDF optical layout analysis (e.g.,
|
||||
academic papers, scanned documents). Both run on NUC 14 CPU. Start with Docling — single
|
||||
container, REST API, well-documented.
|
||||
|
||||
**MCP access**: add a `knowledge-search` MCP tool to the gateway that calls LobeChat's
|
||||
knowledge base search API. Zero new infra — LobeChat is already on `shared_backend`.
|
||||
|
||||
**IBM Granite embedding** (granite-embedding-30m-english) — good quality, tiny (30M params),
|
||||
runs CPU-only. Alternative to Mistral-embed if you want fully local embeddings. Not on
|
||||
LiteLLM yet but can add as a custom provider pointing at an Ollama/IPEX-LLM instance.
|
||||
|
||||
---
|
||||
|
||||
## 2. Document conversion pipeline
|
||||
|
||||
**Problem**: No systematic ingestion path from raw docs (PDF, DOCX, HTML) into
|
||||
structured text for chunking/embedding.
|
||||
|
||||
**Tools to consider**:
|
||||
|
||||
| Tool | Docker image | Best for |
|
||||
|------|-------------|----------|
|
||||
| **Docling** (IBM, 2024) | `ghcr.io/ds4sd/docling` | PDFs with complex layout (tables, columns, figures); outputs Markdown |
|
||||
| **Unstructured** | `quay.io/unstructured-io/unstructured-api` | Wide format support (HTML, DOCX, PPTX, images with OCR); REST API |
|
||||
| **Gotenberg** | `gotenberg/gotenberg` | HTML/Office → PDF pre-processing stage; not a text extractor |
|
||||
| **Apache Tika** | `apache/tika` | Broad format support; outputs plain text; lower quality than Docling for PDFs |
|
||||
| **Markitdown MCP** | already in gateway | Ad-hoc on-demand conversion; not suitable for batch pipelines |
|
||||
|
||||
**Recommended pipeline (n8n-orchestrated)**:
|
||||
```
|
||||
Nextcloud / Paperless webhook
|
||||
→ Gotenberg (Office → PDF)
|
||||
→ Docling (PDF → Markdown chunks)
|
||||
→ LiteLLM /v1/embeddings (Mistral-embed)
|
||||
→ Qdrant upsert
|
||||
```
|
||||
|
||||
Docling runs CPU-only comfortably on the NUC. Unstructured is heavier but has a
|
||||
managed API if the self-hosted version is too slow.
|
||||
|
||||
Paperless-ngx already OCRs documents — its full-text content is accessible via the
|
||||
REST API. An n8n workflow polling `/api/documents/?added__gt=<last_run>` can extract
|
||||
and re-embed directly, without re-running OCR.
|
||||
|
||||
---
|
||||
|
||||
## 3. On-device LLM (Arc iGPU)
|
||||
|
||||
The Arc iGPU is currently ComfyUI-only. Small language models can also run on it:
|
||||
|
||||
- **IPEX-LLM** (Intel) + **Ollama** — Intel provides a patched Ollama build that uses
|
||||
IPEX-LLM for Arc acceleration. Supports Phi-3.5-mini, Gemma-2B, TinyLlama.
|
||||
Useful as a fast/cheap fallback when Groq/Mistral rate limits hit.
|
||||
- **llama.cpp with Vulkan** — alternative to IPEX-LLM; no Intel-specific driver,
|
||||
uses Vulkan compute. Less optimised for Arc but simpler to deploy.
|
||||
- **Practical constraint**: Arc iGPU shares system RAM; sustained LLM inference
|
||||
competes with other XPU workloads (e.g., ComfyUI). A model serving config that
|
||||
pauses one workload while the other runs is needed, or deploy them on separate
|
||||
CPU/GPU budgets.
|
||||
|
||||
If a second NUC or small GPU box becomes available, this becomes the primary use case.
|
||||
|
||||
---
|
||||
|
||||
## 4. Arcane agents on NUC (central instance on Proxmox LXC)
|
||||
|
||||
Arcane's central server moves to a **Proxmox LXC** (lightweight, stays responsive even
|
||||
if the Docker stack has issues). The NUC and any other Docker host runs the **Arcane
|
||||
agent** (headless worker), which connects back to the central instance.
|
||||
|
||||
**Deployment**:
|
||||
- LXC: central Arcane server + observability stack (see §8)
|
||||
- NUC (`.40`) and `.49`: `arcane-agent` (headless) in compose, connects to LXC
|
||||
- Arcane agents reach Docker via either:
|
||||
- **TCP Docker socket + TLS certs** — most secure; requires cert generation per host
|
||||
- **SSH Docker contexts** — simpler; gateway mounts socket into agent via SSH tunnel
|
||||
|
||||
**Observability co-located on the LXC** (see §8 — lightweight, OTEL-aware stack that
|
||||
survives Docker restarts).
|
||||
|
||||
---
|
||||
|
||||
## 5. AI memory / personalization
|
||||
|
||||
**Mem0** (`mem0ai/mem0`) — persistent user memory layer that sits in front of LLM calls.
|
||||
Stores facts extracted from conversations into a vector DB (Qdrant backend supported).
|
||||
Can integrate with LobeChat or as an MCP tool. Lets the AI remember preferences,
|
||||
past context, and user-specific facts across sessions.
|
||||
|
||||
Alternative: **Letta** (formerly MemGPT) — stateful agent framework with persistent
|
||||
memory; more opinionated.
|
||||
|
||||
---
|
||||
|
||||
## 6. Workflow / automation upgrades
|
||||
|
||||
- **n8n → AI agent nodes**: n8n 1.x has native AI agent nodes (LangChain under the hood).
|
||||
Can build RAG, email triage, document processing workflows visually.
|
||||
- **Activepieces** — lighter-weight n8n alternative; good for simple integrations.
|
||||
Probably not worth adding since n8n is already running.
|
||||
- **Temporal** — durable workflow engine for long-running or retry-heavy pipelines
|
||||
(e.g., large document batch processing). Overkill unless pipelines become complex.
|
||||
|
||||
---
|
||||
|
||||
## 7. External services worth considering
|
||||
|
||||
| Service | Purpose | Notes |
|
||||
|---------|---------|-------|
|
||||
| **Backblaze B2** | Off-site backup (already in todos) | rclone sync from Garage; restic for PG dumps |
|
||||
| **Cloudflare R2** | S3-compatible CDN-backed storage | Free egress; good for Lobe file serving |
|
||||
| **Resend** | Transactional email | 3K/mo free; better deliverability than self-hosted Postal |
|
||||
| **ntfy.sh (cloud)** | Push notifications fallback | Already running self-hosted ntfy; cloud as relay |
|
||||
| **Cloudflare Turnstile** | Bot protection for public endpoints | Free; no JS challenge |
|
||||
| **Novu** | Notification orchestration | Multi-channel (email, push, Slack); self-hostable |
|
||||
|
||||
---
|
||||
|
||||
## 8. Observability (on Proxmox LXC, alongside Arcane)
|
||||
|
||||
**Runs on the LXC, not the NUC** — stays alive if the Docker stack misbehaves. Lightweight
|
||||
enough for a 2 vCPU / 4GB LXC.
|
||||
|
||||
**Recommended stack** (all OTEL-aware, compose-based):
|
||||
|
||||
| Component | Image | Role |
|
||||
|-----------|-------|------|
|
||||
| **OpenTelemetry Collector** | `otel/opentelemetry-collector-contrib` | Receives traces/metrics/logs from all services (OTLP gRPC+HTTP); fans out to backends |
|
||||
| **VictoriaMetrics** | `victoriametrics/victoria-metrics` | Prometheus-compatible TSDB; scrapes NUC exporters + receives from OTEL collector; lighter than Prometheus |
|
||||
| **Grafana** | `grafana/grafana` | Dashboards; datasource = VictoriaMetrics + Loki |
|
||||
| **Loki** | `grafana/loki` | Log aggregation; receives from OTEL collector |
|
||||
| **Uptime Kuma** | `louislam/uptime-kuma` | HTTP/TCP uptime checks for all public endpoints; alerts via ntfy |
|
||||
|
||||
**OTEL receivers** from the NUC Docker stack:
|
||||
- **LiteLLM**: native OTEL export — traces every LLM call with token counts, model, latency
|
||||
- **n8n**: Prometheus `/metrics` endpoint
|
||||
- **Garage**: Prometheus `/metrics`
|
||||
- **mcp-gateway**: add `opentelemetry-sdk` instrumentation to `server.py`
|
||||
- **Docker host**: `node_exporter` + `cadvisor` on the NUC, scraped by VictoriaMetrics
|
||||
|
||||
**NUC → LXC connectivity**: Both are on the same LAN. OTEL collector listens on the LXC's
|
||||
LAN IP (e.g., `192.168.1.X:4317` gRPC). Services push OTEL directly to it; Prometheus
|
||||
pull-scraping from VictoriaMetrics goes to NUC exporters over LAN.
|
||||
|
||||
---
|
||||
|
||||
## 9. Storage / S3 improvements
|
||||
|
||||
- **Garage external S3** is currently non-responsive (tracked in todos). Fix or
|
||||
replace with **MinIO** (single-node, much better tooling/UI, easy migration).
|
||||
- **Lobe file serving via Cloudflare R2**: Lobe uploads to Garage → sync to R2 →
|
||||
serve from CDN. Reduces NUC egress for image-heavy sessions.
|
||||
- **Qdrant data on UNAS**: Mount `qdrant_data` from NAS for persistence across
|
||||
container re-creates (same pattern as arr-stack).
|
||||
|
||||
---
|
||||
|
||||
## 9. ComfyUI MCP — async queue + image-to-image + reference
|
||||
|
||||
**Current problem**: MCP tool blocks until the image is done (~90-290s). LLMs time out at ~60s.
|
||||
The fix is a job-queue pattern:
|
||||
|
||||
```
|
||||
generate_image(prompt) → returns {job_id, status: "queued"} immediately
|
||||
get_image_status(job_id) → returns {status, progress, image_url_when_done}
|
||||
list_queue() → shows all pending/running jobs
|
||||
cancel_job(job_id) → cancels a queued job (confirms with user first)
|
||||
```
|
||||
|
||||
ComfyUI's own API is already async (`POST /prompt` → poll `/history/{id}`). The MCP server
|
||||
just needs to expose this model instead of blocking.
|
||||
|
||||
**Image-to-image**: new workflow `flux-schnell-img2img-api.json`. Takes an init image URL +
|
||||
denoise strength. ComfyUI `LoadImageFromURL` node (or upload + `LoadImage`).
|
||||
|
||||
**Reference tool**: `list_previous_images(n=5)` — queries ComfyUI `/history` API,
|
||||
returns recent job thumbnails + prompts. User can pick one to reference or iterate from.
|
||||
|
||||
**Delivery to S3/Garage**: on completion, upload output PNG to Garage `comfyui-outputs` bucket
|
||||
→ return a permanent URL. Avoids ComfyUI's ephemeral `/view` endpoint.
|
||||
|
||||
---
|
||||
|
||||
## 10. LiteLLM custom provider / LobeChat provider extension
|
||||
|
||||
**Question**: build an adapter in LiteLLM to use models via "quasi API" (non-standard
|
||||
endpoints, auth, or routing)?
|
||||
|
||||
**LiteLLM supports custom providers** via `custom_llm_provider` in config.yaml. You write
|
||||
a Python class that implements `completion()` and `async_completion()`. This is the right
|
||||
path for wrapping non-standard APIs (local models, proprietary endpoints, protocol bridges).
|
||||
|
||||
Example use cases:
|
||||
- Wrap a Claude Max subscription via Anthropic's API (different billing model)
|
||||
- Add a local model served by IPEX-LLM on the Arc iGPU
|
||||
- Bridge a custom inference server that speaks a different protocol
|
||||
|
||||
**LobeChat provider extension**: LobeChat's provider list is compiled into the app.
|
||||
Adding a new provider requires rebuilding LobeChat from source (fork + add to
|
||||
`src/config/aiModels/`). High effort; only worth it for a permanent/long-term provider.
|
||||
For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the existing
|
||||
`ai.nuclide.systems` OpenAI-compatible endpoint.
|
||||
|
||||
---
|
||||
|
||||
## Priority order (rough)
|
||||
|
||||
1. **ComfyUI MCP async queue** — fixes timeout; unblocks img2img + reference features
|
||||
2. **Docling container + n8n ingest pipeline** — Nextcloud/Paperless → LobeChat knowledge base
|
||||
3. **Arcane LXC + agents on NUC** — when ready to migrate
|
||||
4. **Observability LXC** — OTEL collector + VictoriaMetrics + Grafana + Uptime Kuma, co-located
|
||||
5. **Backblaze B2 off-site backup** — already in todos
|
||||
6. **On-device LLM (Arc)** — depends on IPEX-LLM + Ollama Intel build stability
|
||||
@@ -1,341 +0,0 @@
|
||||
# Traefik Migration Guide Using Docker Labels
|
||||
|
||||
## Overview
|
||||
Migrate from Zoraxy reverse proxy to Traefik using Docker labels for zero-touch service discovery.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Install Traefik
|
||||
|
||||
### Step 1: Create Directory Structure
|
||||
```bash
|
||||
mkdir -p /opt/stacks/proxy/traefik/{config,dynamic,letsencrypt}
|
||||
```
|
||||
|
||||
### Step 2: Create docker-compose.yml
|
||||
```yaml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:v3.2
|
||||
container_name: traefik
|
||||
restart: always
|
||||
network_mode: host
|
||||
security_opt:
|
||||
- no-new-privileges=true
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /opt/stacks/proxy/traefik/config:/etc/traefik
|
||||
- /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic
|
||||
- /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt
|
||||
command:
|
||||
- "--api.insecure=true"
|
||||
- "--providers.docker=true"
|
||||
- "--providers.docker.exposedbydefault=false"
|
||||
- "--providers.docker.network=ai-internal"
|
||||
- "--providers.docker.network=shared_backend"
|
||||
- "--providers.docker.defaultRule=Host(`{{ .Name }}.nuclide.systems`)"
|
||||
- "--entrypoints.web.address=:80"
|
||||
- "--entrypoints.websecure.address=:443"
|
||||
- "--certificatesresolvers.letsencrypt.acme.httpChallenge=true"
|
||||
- "--certificatesresolvers.letsencrypt.acme.email=admin@nuclide.systems"
|
||||
- "--certificatesresolvers.letsencrypt.acme.storage=/etc/letsencrypt/acme.json"
|
||||
```
|
||||
|
||||
### Step 3: Start Traefik
|
||||
```bash
|
||||
cd /opt/stacks/proxy/traefik
|
||||
docker compose up -d
|
||||
|
||||
# Verify
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Migrate AI Services
|
||||
|
||||
### AI Service Labels (Add to litellm, chat, mcp-compose.yml)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
litellm:
|
||||
image: litellm
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)"
|
||||
- "traefik.http.routers.litellm.entrypoints=websecure"
|
||||
- "traefik.http.routers.litellm.tls=true"
|
||||
- "traefik.http.routers.litellm.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.litellm.priority=10"
|
||||
- "traefik.http.services.litellm.loadbalancer.server.port=14000"
|
||||
|
||||
chat:
|
||||
image: lobehub
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)"
|
||||
- "traefik.http.routers.chat.entrypoints=websecure"
|
||||
- "traefik.http.routers.chat.tls=true"
|
||||
- "traefik.http.routers.chat.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.chat.priority=10"
|
||||
- "traefik.http.services.chat.loadbalancer.server.port=14001"
|
||||
|
||||
mcp:
|
||||
image: mcp-gateway
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)"
|
||||
- "traefik.http.routers.mcp.entrypoints=websecure"
|
||||
- "traefik.http.routers.mcp.tls=true"
|
||||
- "traefik.http.routers.mcp.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.mcp.priority=10"
|
||||
- "traefik.http.services.mcp.loadbalancer.server.port=8080"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Migrate Garage S3
|
||||
|
||||
### Option A: Use Traefik Proxy
|
||||
```yaml
|
||||
services:
|
||||
garage-proxy:
|
||||
image: nginx:alpine
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.s3.rule=Host(`s3.nuclide.systems`)"
|
||||
- "traefik.http.routers.s3.entrypoints=websecure"
|
||||
- "traefik.http.routers.s3.tls=true"
|
||||
- "traefik.http.routers.s3.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.s3.loadbalancer.server.port=10004"
|
||||
volumes:
|
||||
- garage-data:/data
|
||||
networks:
|
||||
- shared_backend
|
||||
```
|
||||
|
||||
### Option B: Keep Internal Garage Access
|
||||
```yaml
|
||||
services:
|
||||
# No proxy needed - access Garage via internal IP:10004
|
||||
garage:
|
||||
image: garageio/garage
|
||||
ports:
|
||||
- "3900:3900" # Internal only
|
||||
- "10004:10004" # Public via Traefik
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Create Helper Scripts
|
||||
|
||||
### script 1: Add Service to Traefik
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /opt/stacks/scripts/add-traefik-service.sh
|
||||
|
||||
NAME=$1
|
||||
DOMAIN=$2
|
||||
PORT=$3
|
||||
|
||||
cat > /opt/stacks/proxy/traefik/dynamic/${NAME}.yml << EOF
|
||||
http:
|
||||
routers:
|
||||
${NAME}-router:
|
||||
rule: "Host(\`${DOMAIN}\`)"
|
||||
service: ${NAME}-service
|
||||
entrypoints:
|
||||
- websecure
|
||||
tls:
|
||||
certresolver: letsencrypt
|
||||
|
||||
services:
|
||||
${NAME}-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://192.168.1.40:${PORT}"
|
||||
EOF
|
||||
|
||||
# Reload Traefik docker automatically (no manual step needed)
|
||||
echo "✅ Service ${NAME} added via labels"
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
/opt/stacks/scripts/add-traefik-service.sh myservice myservice.nuclide.systems 8000
|
||||
```
|
||||
|
||||
### Script 2: Generate Labels for Existing Services
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /opt/stacks/scripts/traefik-labels-gen.sh
|
||||
|
||||
cat > /opt/stacks/proxy/traefik/labels.yaml << 'EOF'
|
||||
# Add these labels to service docker-compose.yml files
|
||||
|
||||
# LiteLLM
|
||||
services:
|
||||
litellm:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)"
|
||||
- "traefik.http.routers.litellm.entrypoints=websecure"
|
||||
- "traefik.http.routers.litellm.tls=true"
|
||||
- "traefik.http.routers.litellm.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.litellm.loadbalancer.server.port=14000"
|
||||
|
||||
# LobeHub Chat
|
||||
services:
|
||||
chat:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)"
|
||||
- "traefik.http.routers.chat.entrypoints=websecure"
|
||||
- "traefik.http.routers.chat.tls=true"
|
||||
- "traefik.http.routers.chat.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.chat.loadbalancer.server.port=14001"
|
||||
|
||||
# MCP Gateway
|
||||
services:
|
||||
mcp-gateway:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)"
|
||||
- "traefik.http.routers.mcp.entrypoints=websecure"
|
||||
- "traefik.http.routers.mcp.tls=true"
|
||||
- "traefik.http.routers.mcp.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.mcp.loadbalancer.server.port=8080"
|
||||
EOF
|
||||
|
||||
echo "✅ Labels saved to /opt/stacks/proxy/traefik/labels.yaml"
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
/opt/stacks/scripts/traefik-labels-gen.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Update Service Configs
|
||||
|
||||
### Update `/opt/stacks/ai/.env`
|
||||
```bash
|
||||
# OLD (Zoraxy):
|
||||
LITELLM_BASE_URL=https://litellm.nuclide.systems
|
||||
PROXY_BASE_URL=https://mcp.nuclide.systems
|
||||
|
||||
# NEW (Traefik) - same URLs, different backend:
|
||||
LITELLM_BASE_URL=https://litellm.nuclide.systems
|
||||
PROXY_BASE_URL=https://mcp.nuclide.systems
|
||||
CHATAI_BASE_URL=https://chat.nuclide.systems
|
||||
```
|
||||
|
||||
### Update `/opt/stacks/ai/litellm-config/config.yaml`
|
||||
```yaml
|
||||
general_settings:
|
||||
proxy_base_url: https://litellm.nuclide.systems
|
||||
control_plane_url: https://litellm.nuclide.systems
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Verify SSL
|
||||
|
||||
### Step 1: Generate Let's Encrypt Certificates
|
||||
```bash
|
||||
# Verify Traefik is running
|
||||
docker compose ps traefik
|
||||
|
||||
# Create ACME cert file
|
||||
touch /opt/stacks/proxy/traefik/letsencrypt/acme.json
|
||||
chmod 600 /opt/stacks/proxy/traefik/letsencrypt/acme.json
|
||||
|
||||
# Trigger certificate generation (will happen automatically)
|
||||
# Check status:
|
||||
curl -s https://acme-v02.api.letsencrypt.org/directory | head
|
||||
```
|
||||
|
||||
### Step 2: Test HTTPS
|
||||
```bash
|
||||
# Test endpoints
|
||||
curl -k https://litellm.nuclide.systems/health
|
||||
curl -k https://chat.nuclide.systems/health
|
||||
curl -k https://mcp.nuclide.systems/health
|
||||
|
||||
# Verify cert
|
||||
curl -v https://litellm.nuclide.systems 2>&1 | grep -A 5 "SSL certificate"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Remove Zoraxy
|
||||
|
||||
### Backup first
|
||||
```bash
|
||||
# Backup Zoraxy configs
|
||||
docker cp zoraxy:/data/configs /backup/zoraxy-backup/
|
||||
|
||||
# Optional: Stop Zoraxy
|
||||
docker stop zoraxy
|
||||
docker rm zoraxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Migration Checklist
|
||||
|
||||
- [ ] Install Traefik (Phase 1)
|
||||
- [ ] Add labels to AI services (Phase 2)
|
||||
- [ ] Add Garage S3 proxy (Phase 3)
|
||||
- [ ] Run label generation script (Phase 4)
|
||||
- [ ] Update `.env` and config files (Phase 5)
|
||||
- [ ] Wait for SSL certificates (auto 24-48h)
|
||||
- [ ] Test all HTTPS endpoints
|
||||
- [ ] Remove Zoraxy (Phase 7)
|
||||
- [ ] Update AdGuard DNS if needed (optional)
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Troubleshooting
|
||||
|
||||
### Check Traefik Dashboard
|
||||
```bash
|
||||
# Access web UI (unsecured - use only on trusted network)
|
||||
open http://192.168.1.40:8080/dashboard
|
||||
|
||||
# View all routers
|
||||
curl http://localhost:8080/api/http/routers | jq '.[] | {name: .rule, status: .entryPoints}'
|
||||
|
||||
# View all services
|
||||
curl http://localhost:8080/api/http/services | jq '.[] | {name: .name, servers: .servers}'
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| 404 errors | Check router labels match domain exactly |
|
||||
| SSL expired | Wait for auto-renew or trigger manually |
|
||||
| Port mismatch | Verify `loadbalancer.server.port` matches service |
|
||||
| No SSL cert | Check email in `acme.json` config |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan (If Needed)
|
||||
|
||||
```bash
|
||||
# Stop Traefik
|
||||
docker compose -f /opt/stacks/proxy/traefik/docker-compose.yml down
|
||||
|
||||
# Restore Zoraxy configs
|
||||
docker cp /backup/zoraxy-backup/ configs/
|
||||
|
||||
# Restart Zoraxy (if you kept backup)
|
||||
docker start zoraxy || true
|
||||
```
|
||||
@@ -1,193 +0,0 @@
|
||||
# Recommended: Traefik Proxy Replacement
|
||||
|
||||
## Why Traefik over Zoraxy?
|
||||
|
||||
| Feature | Zoraxy | Traefik |
|
||||
|---------|--------|---------|
|
||||
| **API** | ❌ No public API | ✅ Full REST API |
|
||||
| **SSL** | 💬 Manual (Zoraxy Web UI) | 🔥 Auto-Let's Encrypt |
|
||||
| **Dynamic** | ⚠️ Manual config reload | ✅ Hot-reload configs |
|
||||
| **File watching** | ❌ | ✅ Auto-detect changes |
|
||||
| **Docker integration** | ⚠️ Manual | ✅ Native labels |
|
||||
| **API endpoints** | 403 Forbidden | ✅ JSON API everywhere |
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Current setup:
|
||||
```
|
||||
Zoraxy (192.168.1.4:8000) → Reverse Proxy Rules (Manual Web UI)
|
||||
- litellm.nuclide.systems → 192.168.1.40:14000
|
||||
- chat.nuclide.systems → 192.168.1.40:14001
|
||||
- mcp.nuclide.systems → 192.168.1.40:8080
|
||||
- s3.nuclide.systems → Garage:10004
|
||||
```
|
||||
|
||||
### New setup with Traefik:
|
||||
```
|
||||
Traefik (public SSL) → Dynamic Router (labels/consul)
|
||||
- All services auto-discovered via Docker labels
|
||||
- SSL certificates auto-provisioned
|
||||
- No manual Zoraxy configuration needed
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### Step 1: Install Traefik
|
||||
|
||||
```bash
|
||||
# Create Traefik directory
|
||||
mkdir -p /opt/stacks/proxy/traefik/{conf,dynamic}
|
||||
|
||||
# Create docker-compose.yml
|
||||
cat > /opt/stacks/proxy/traefik/docker-compose.yml << 'EOF'
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:v3.2
|
||||
container_name: traefik
|
||||
restart: always
|
||||
security_opt:
|
||||
- no-new-privileges=true
|
||||
network_mode: host
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /opt/stacks/proxy/traefik/conf:/etc/traefik
|
||||
- /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic
|
||||
- /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt
|
||||
command:
|
||||
- "--api.insecure=true"
|
||||
- "--providers.docker=true"
|
||||
- "--providers.docker.exposedbydefault=false"
|
||||
- "--entrypoints.web.address=:80"
|
||||
- "--entrypoints.websecure.address=:443"
|
||||
- "--certificatesresletsencryptemail=admin@nuclide.systems"
|
||||
- "--certificatesresletsencryptstorage=/etc/letsencrypt/acme.json"
|
||||
EOF
|
||||
|
||||
# Start Traefik
|
||||
cd /opt/stacks/proxy/traefik && docker compose up -d
|
||||
```
|
||||
|
||||
### Step 2: Create Dynamic Configuration
|
||||
|
||||
```bash
|
||||
# Create router rules
|
||||
cat > /opt/stacks/proxy/traefik/dynamic/router.yml << 'EOF'
|
||||
http:
|
||||
routers:
|
||||
litellm-router:
|
||||
rule: "Host(`litellm.nuclide.systems`)"
|
||||
service: litellm-service
|
||||
entrypoints:
|
||||
- websecure
|
||||
tls:
|
||||
certresolver: letsencrypt
|
||||
|
||||
chat-router:
|
||||
rule: "Host(`chat.nuclide.systems`)"
|
||||
service: chat-service
|
||||
entrypoints:
|
||||
- websecure
|
||||
tls:
|
||||
certresolver: letsencrypt
|
||||
|
||||
mcp-router:
|
||||
rule: "Host(`mcp.nuclide.systems`)"
|
||||
service: mcp-service
|
||||
entrypoints:
|
||||
- websecure
|
||||
tls:
|
||||
certresolver: letsencrypt
|
||||
|
||||
s3-router:
|
||||
rule: "Host(`s3.nuclide.systems`)"
|
||||
service: garage-service
|
||||
entrypoints:
|
||||
- websecure
|
||||
tls:
|
||||
certresolver: letsencrypt
|
||||
|
||||
services:
|
||||
litellm-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://192.168.1.40:14000"
|
||||
|
||||
chat-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://192.168.1.40:14001"
|
||||
|
||||
mcp-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://192.168.1.40:8080"
|
||||
|
||||
garage-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://garage:10004"
|
||||
EOF
|
||||
```
|
||||
|
||||
### Step 3: Add Docker Labels to Services
|
||||
|
||||
For any Docker service you want to proxy:
|
||||
|
||||
```yaml
|
||||
# Example: Add to your service docker-compose.yml
|
||||
services:
|
||||
ai-service:
|
||||
image: your-service
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.your-service.rule=Host(`your-service.nuclide.systems`)"
|
||||
- "traefik.http.routers.your-service.entrypoints=websecure"
|
||||
- "traefik.http.routers.your-service.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.your-service.loadBalancer.server.port=8000"
|
||||
```
|
||||
|
||||
### Step 4: Delete Zoraxy (Optional)
|
||||
|
||||
```bash
|
||||
# Backup Zoraxy configs first
|
||||
tar -czf /backup/zoraxy-backup.tar.gz /path/to/zoraxy/configs
|
||||
|
||||
# Stop and remove Zoraxy
|
||||
docker rm -f zoraxy || true
|
||||
```
|
||||
|
||||
## API Example (Traefik)
|
||||
|
||||
```bash
|
||||
# Get list of services
|
||||
curl -u traefik:YOUR_TRAEFIK_API_PASSWORD http://localhost:8080/api/http/routers
|
||||
|
||||
# Get Traefik metrics
|
||||
curl http://localhost:8080/metrics
|
||||
|
||||
# Reload configuration (live)
|
||||
curl -X POST http://localhost:8080/api/http/routers -H "Content-Type: application/json" -d '{...}'
|
||||
```
|
||||
|
||||
## Migration Checklist
|
||||
|
||||
- [ ] Deploy Traefik in host network mode
|
||||
- [ ] Create Let's Encrypt certificate for your domain
|
||||
- [ ] Migrate all reverse proxy rules to Traefik labels or dynamic config
|
||||
- [ ] Test SSL certificates work: `curl -k https://your-domain.nuclide.systems`
|
||||
- [ ] Remove Zoraxy Docker container
|
||||
- [ ] Update DNS if needed
|
||||
- [ ] Verify all endpoints: health checks at new URLs
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Zero maintenance SSL** - Let's Encrypt auto-renews
|
||||
2. **API-driven** - No manual Web UI needed
|
||||
3. **Hot reloading** - Changes apply immediately
|
||||
4. **Docker-native** - Watches container labels automatically
|
||||
5. **Enterprise-grade** - Used by major cloud providers
|
||||
@@ -1,182 +0,0 @@
|
||||
# Zoraxy Reverse Proxy Setup Guide
|
||||
|
||||
## Overview
|
||||
Configure reverse proxies for AI service endpoints with automatic SSL certificates.
|
||||
|
||||
## Target Services
|
||||
|
||||
| Service | Domain | Target | Port |
|
||||
|---------|--------|--------|------|
|
||||
| LiteLLM | `litellm.nuclide.systems` | `192.168.1.40` | `14000` |
|
||||
| LobeHub | `chat.nuclide.systems` | `192.168.1.40` | `14001` |
|
||||
| MCP Gateway | `mcp.nuclide.systems` | `192.168.1.40` | `8080` |
|
||||
| Garage S3 | `s3.nuclide.systems` | `192.168.1.40` | `10004` |
|
||||
|
||||
## Option A: Web UI (Easiest - No SSH Required)
|
||||
|
||||
### Step 1: Access Zoraxy Panel
|
||||
```
|
||||
http://192.168.1.4:8000
|
||||
```
|
||||
|
||||
### Step 2: Navigate to Rewrite Rules
|
||||
1. Go to **"Reverse Proxy"** in the left sidebar
|
||||
2. Click **"Rewrite Rules"**
|
||||
|
||||
### Step 3: Add Each Service
|
||||
Click **"Add New Rule"** and fill in:
|
||||
|
||||
**For LiteLLM (`litellm.nuclide.systems`):**
|
||||
- **Domain:** `litellm.nuclide.systems`
|
||||
- **SSL:** `ON`
|
||||
- **HTTPS Redirect:** `ON`
|
||||
- **HSTS:** `ON`
|
||||
- **Target Type:** `Server`
|
||||
- **Target:** `192.168.1.40:14000`
|
||||
- **Timeout:** `300` seconds
|
||||
- **Proxy Time:** `300` seconds
|
||||
|
||||
**For LobeHub (`chat.nuclide.systems`):**
|
||||
- **Domain:** `chat.nuclide.systems`
|
||||
- **SSL:** `ON`
|
||||
- **HTTPS Redirect:** `ON`
|
||||
- **HSTS:** `ON`
|
||||
- **Target Type:** `Server`
|
||||
- **Target:** `192.168.1.40:14001`
|
||||
- **Timeout:** `300` seconds
|
||||
- **Proxy Time:** `300` seconds
|
||||
|
||||
**For MCP Gateway (`mcp.nuclide.systems`):**
|
||||
- **Domain:** `mcp.nuclide.systems`
|
||||
- **SSL:** `ON`
|
||||
- **HTTPS Redirect:** `ON`
|
||||
- **HSTS:** `ON`
|
||||
- **Target Type:** `Server`
|
||||
- **Target:** `192.168.1.40:8080`
|
||||
- **Timeout:** `300` seconds
|
||||
- **Proxy Time:** `300` seconds
|
||||
|
||||
**For Garage S3 (`s3.nuclide.systems`):**
|
||||
- **Domain:** `s3.nuclide.systems`
|
||||
- **SSL:** `ON`
|
||||
- **HTTPS Redirect:** `ON`
|
||||
- **HSTS:** `ON`
|
||||
- **Target Type:** `Server`
|
||||
- **Target:** `192.168.1.40:10004`
|
||||
- **Timeout:** `300` seconds
|
||||
- **Proxy Time:** `300` seconds
|
||||
|
||||
### Step 4: Save All Rules
|
||||
Click **"Save"** at the bottom of the page.
|
||||
|
||||
### Step 5: SSL Certificate Generation
|
||||
- Zoraxy automatically initiates Let's Encrypt certificate generation
|
||||
- Wait 1-2 minutes for certificates to be issued
|
||||
- Visual indicator: Green padlock icon in the UI
|
||||
|
||||
## Option B: SSH/UCI (Automated - Requires Root Access)
|
||||
|
||||
### Prerequisites
|
||||
- SSH access to Zoraxy with root credentials
|
||||
- Command-line access to Zoraxy's UCI config system
|
||||
|
||||
### Step 1: SSH into Zoraxy
|
||||
```bash
|
||||
ssh root@192.168.1.4
|
||||
```
|
||||
|
||||
### Step 2: Configure via UCI
|
||||
Run this command directly on Zoraxy:
|
||||
|
||||
```bash
|
||||
# Apply all rewrite rules
|
||||
uci set network.rewrite_litellm.enable='1'
|
||||
uci set network.rewrite_litellm.domain='litellm.nuclide.systems'
|
||||
uci set network.rewrite_litellm.target_type='0'
|
||||
uci set network.rewrite_litellm.target='192.168.1.40:14000'
|
||||
uci set network.rewrite_litellm.ssl='1'
|
||||
uci set network.rewrite_litellm.hsts='1'
|
||||
|
||||
uci set network.rewrite_lobehub.enable='1'
|
||||
uci set network.rewrite_lobehub.domain='chat.nuclide.systems'
|
||||
uci set network.rewrite_lobehub.target_type='0'
|
||||
uci set network.rewrite_lobehub.target='192.168.1.40:14001'
|
||||
uci set network.rewrite_lobehub.ssl='1'
|
||||
uci set network.rewrite_lobehub.hsts='1'
|
||||
|
||||
uci set network.rewrite_mcp.enable='1'
|
||||
uci set network.rewrite_mcp.domain='mcp.nuclide.systems'
|
||||
uci set network.rewrite_mcp.target_type='0'
|
||||
uci set network.rewrite_mcp.target='192.168.1.40:8080'
|
||||
uci set network.rewrite_mcp.ssl='1'
|
||||
uci set network.rewrite_mcp.hsts='1'
|
||||
|
||||
uci set network.rewrite_garage.enable='1'
|
||||
uci set network.rewrite_garage.domain='s3.nuclide.systems'
|
||||
uci set network.rewrite_garage.target_type='0'
|
||||
uci set network.rewrite_garage.target='192.168.1.40:10004'
|
||||
uci set network.rewrite_garage.ssl='1'
|
||||
uci set network.rewrite_garage.hsts='1'
|
||||
|
||||
# Commit and restart
|
||||
uci commit network
|
||||
/etc/init.d/network restart
|
||||
```
|
||||
|
||||
### Step 3: Verify
|
||||
```bash
|
||||
# Check if rewrite rules are loaded
|
||||
uci show network.rewrite_*
|
||||
|
||||
# Test connectivity
|
||||
curl -I http://litellm.nuclide.systems
|
||||
```
|
||||
|
||||
## Option C: Automated Script (Local)
|
||||
|
||||
Run from the control node:
|
||||
|
||||
```bash
|
||||
# Generate UCI config
|
||||
/opt/stacks/scripts/configure_zoraxy_helpers.sh --export-uci
|
||||
|
||||
# Or deploy via SSH (requires root@192.168.1.4 SSH access)
|
||||
/opt/stacks/scripts/configure_zoraxy_helpers.sh --deploy
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
After setup, verify all endpoints:
|
||||
|
||||
```bash
|
||||
# Test each service
|
||||
curl -I https://litellm.nuclide.systems/health
|
||||
curl -I https://chat.nuclide.systems/api/health
|
||||
curl -I https://mcp.nuclide.systems/health
|
||||
curl -I https://s3.nuclide.systems/
|
||||
```
|
||||
|
||||
Expected: HTTP 200 with SSL certificate.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SSL Certificate Issues
|
||||
- Wait 2-3 minutes for Let's Encrypt to respond
|
||||
- Check domain DNS resolution: `dig litellm.nuclide.systems`
|
||||
- Verify firewall allows port 80 validation
|
||||
|
||||
### Proxy Connection Failed
|
||||
- Check backend service is running: `htop` or `docker ps`
|
||||
- Verify internal IP: `192.168.1.40` is reachable from Zoraxy
|
||||
- Check target port is correct
|
||||
|
||||
### Configuration Not Saving
|
||||
- Check Web UI is logged in (session expiry)
|
||||
- Try clearing browser cache
|
||||
- Verify UCI config syntax
|
||||
|
||||
## Notes
|
||||
- SSL certificates are **automatic** via Let's Encrypt
|
||||
- Zoraxy auto-renews certificates before expiry
|
||||
- HSTS is recommended for production use
|
||||
- All services use the same target IP (`192.168.1.40`)
|
||||
Reference in New Issue
Block a user