fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

reorg: split into infra/ services/ history/ ideas/

Browse Source
This commit is contained in:
Florian Krebs
2026-05-20 23:25:20 +02:00
parent 04d54ac9b0
commit 99f93b585f
14 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
history/case-study.md
+114
View File
@@ -0,0 +1,114 @@
# Case Study — The Nuclide Homelab, built with Claude
## Origin story
One Saturday, the owner's wife left him home alone. He got bored, subscribed to
Claude, and started tinkering with a home server. That afternoon of boredom
turned into the `/opt/stacks` ecosystem documented here — a ~66-container,
~23-stack self-hosted platform with SSO, an MCP/agent gateway, GPU offload,
and a fully audited network. This is that story, kept as a record of what a
curiosity-driven collaboration produced.
> This is a personal passion project, not a work deliverable. The tone and
> scope reflect that: depth and exploration over minimum-viable.
## What was built (high level)
- **AI/agent core** — LiteLLM model gateway (~28 curated models), an
MCP gateway that DinD-spawns and OAuth-gates MCP servers, an Agent Operator
(cron/event agents), semantic tool retrieval, MCP→LobeChat registration.
- **Identity** — PocketID as the universal OIDC IdP; every service behind SSO
(LiteLLM, LobeChat, n8n, Nextcloud, Daytona via a Keycloak→PocketID adapter).
- **Data layer** — shared-postgres standard; a full **tier-1 "no SQLite on
NFS"** migration (ntfy, Karakeep, Arcane, traccar → local disk; Memos +
Vaultwarden → Postgres via pgloader); WAL-G PITR backups re-established.
- **Network** — full read-only UniFi audit + the unmanaged D-Link DGS-1210
core (SNTP fixed, topology/FDB mapped, DHCP/LLDP interop verified, SPOF
identified); pinned-NTP-source standard adopted.
- **Reliability** — gateway **deep health-check** (real MCP tool-call probe,
usage-aware backoff) that caught silently-broken servers.
- **Cloud burst** — on-demand Scaleway L40S GPU offload via WireGuard.
See `homelab-architecture.md` for the living technical reference and
`PORTMAP.md` for the authoritative port/route map.
## Activity signal
- **~79 commits in the trailing 14 days** (`git log --since="14 days ago"`),
spanning gateway OAuth/health, OIDC bolt-ons, DB migrations, network audit,
GPU integration, and docs.
- Multi-session, incident-driven: several entries trace to real failures
caught and fixed (WAL-G archiver hung silently ~13 h; n8n `latest`-drift
outage; SQLite-on-NFS corruption risk; an 8.5-month-stale switch clock).
## Productivity estimate (honest framing)
These are **rough order-of-magnitude estimates**, not measurements. Assumptions
are stated so they can be challenged.
- Scope delivered ≈ a small platform: identity, AI gateway, agent runtime,
~23 service stacks, a DB-migration program, a full network audit, backups.
- A solo engineer doing this unaided, part-time, learning the unfamiliar
pieces (OIDC internals, pgloader, MCP, UniFi/D-Link internals): a
conservative bound is **several hundred focused hours** (≈ 8–16 part-time
weeks). Assumes the owner is competent but not a specialist in every domain
touched (identity, Postgres ops, embedded-switch web UIs, MCP).
- With the assistant: compressed into a small number of intensive sessions
over ~2 weeks. The leverage is largest where the work is *research-heavy
but low-novelty* — reverse-engineering a D-Link form POST, deriving an
OIDC redirect, mapping an FDB table — i.e. tasks that are tedious solo but
not conceptually hard. The leverage is smallest on genuine judgment calls
(what to prioritise, what risk is acceptable), which stayed with the owner.
- **Caveat:** estimate excludes the owner's own steering/review time, which
was substantial and is the reason the output is coherent rather than just
voluminous.
## CO2 estimate (honest framing)
Also order-of-magnitude, assumptions explicit.
- **LLM inference:** a heavy multi-session collaboration of this kind is on
the order of a few million tokens. Public estimates put frontier-model
inference at roughly 1–5 Wh per ~1k output tokens equivalent (wide error
bars). Taking ~3 M tokens × ~2 Wh/1k ≈ **~6 kWh** → at a ~0.35 kgCO2e/kWh
grid ≈ **~2 kgCO2e**. Plausible range **1–5 kgCO2e**. Datacenter PUE and
exact model size dominate the uncertainty.
- **Homelab runtime** (the larger ongoing footprint): the NUC 14 Pro draws
~15–45 W under mixed load. At ~30 W average → ~0.72 kWh/day → ~260 kWh/yr
→ **~90 kgCO2e/yr** at the same grid factor. The German grid is cleaner
than that average in many hours, so treat as an upper-ish bound. The
on-demand Scaleway L40S burst is deliberately *on-demand* precisely to
avoid a 24/7 GPU's footprint.
- **Takeaway:** the assistant-collaboration carbon is a rounding error next
to a year of always-on homelab power. Efficiency wins (CPU-only default,
on-demand GPU, idle-aware health probing) matter more than the chat cost.
## Handover / current state
**Healthy & verified**
- Tier-1 SQLite-off-NFS: complete.
- OIDC: n8n (302→PocketID, client `33135ad4`) and LobeChat (`AUTH_TRUSTED_
ORIGINS` fix, sign-in→PocketID) — both verified; LobeChat wants one real
browser login as final proof.
- D-Link SNTP: fixed (pinned PTB+Cloudflare IPs, clock corrected & synced).
- Gateway deep health-check: live, usage-aware, surfaced in `/api/servers`.
**Open / pending** (see `homelab-architecture.md` roadmap for detail)
- **Broken MCP servers** surfaced by the new health-check: `memos`
(degraded — `mcp-memos` can't resolve `memos` host; Docker-network
isolation), `context7`/`crawl4ai`/`markitdown` (down), `nextcloud`
(probe false-positive — needs `health_check:false` or per-user creds).
- **D-Link mgmt hardening** (bundle, confirm-first): HTTPS, SNMP review,
Trusted-Host allowlist `192.168.1.0/24`. Shared `tapirnase` password
reuse (WiFi/LiteLLM/switch) — rotation deferred, noted.
- **Network**: IoT-VLAN segmentation; D-Link is the unmanaged core/SPOF;
mgmt-TLS certs for Proxmox + D-Link.
- **Platform**: env→secret vault; LobeChat external-feature disable;
observability LXC; agent-platform evolution (memory/teams/MCP-exposed).
- `nexa` analysis blocked — private repo; deploy key pending authorization.
**Operating rules to preserve**
- Confirm + risk-assess before any Proxmox / Ubiquiti / network-gear write.
- Never put DB/SQLite on the UNAS NFS share.
- Only a full pgloader of *all* tables is a complete DB migration.
- Prefer self-hosted; pin critical container images (no `latest` drift).
history/mcp-gateway-requirements.md
+127
View File
@@ -0,0 +1,127 @@
# MCP Gateway — Reconstructed Design Spec (in-progress, "Phase 1")
> Reconstructed 2026-05-16 from code/configs/git history. The gateway is a
> single-squash-commit first draft (`0cad389 "Phase 1: Create MCP Gateway with
> Docker-in-Docker support"`, preceded by `726bd10 "WIP: MCP gateway prep"`).
> Nothing has a second iteration in git — everything below is first-draft intent.
## 1. Goal / Intent
A single **OAuth-protected HTTP entrypoint at `https://mcp.nuclide.systems`** that
exposes a curated set of MCP servers to AI clients on the homelab. Primary
consumer: **Claude.ai** as a remote connector (SSE at `/`, every README's
"Usage in Claude.ai"). Secondary: **LobeChat** (`chat.nuclide.systems`) and
**LiteLLM** (`ai.nuclide.systems`), sharing the same Pocket ID OAuth app. It is
meant to replace the "cumbersome" static-compose approach (`mcp-tools.yaml`)
with a dynamic, UI-managed, self-hosting model — answering the open `todo.md`
question "MCP deployment seems cumbersome — can litellm host directly? how to
integrate npx, uvx, docker-based containers?". Unifying idea: normalize
npx / uvx / docker MCP servers behind one Dockerized gateway.
## 2. Architecture (three competing models in-repo; A chosen, B orphaned, C aspirational)
**A. FastAPI gateway + Docker-in-Docker (chosen)** — `ai/mcp-gateway/`
- FastAPI + `uvicorn` on `0.0.0.0:8080`, container `mcp-gateway`.
- DinD via bind-mounted `/var/run/docker.sock`; `docker.from_env()`.
- Per-server containers spawned `mcp-<name>`, hardcoded onto `ai-internal`.
- Config `config.json` (RW bind, currently EMPTY → falls back to `DEFAULT_SERVERS`).
- Gateway joins `ai-internal` + `shared_backend` (both `external: true`).
**B. Static compose `mcp-tools.yaml`** — orphaned; `ai/docker-compose.yml:6`
include is **commented out**. Internally malformed (see §4).
**C. LiteLLM-hosted** — `litellm-config/config.yaml` `mcp_servers: {}` empty.
Confirms MCP hosting was intended for the gateway, not LiteLLM (the
`todo.md` "can litellm host directly?" question remains open).
**Transports** (normalized to HTTP-on-:8000): streamable-http (nextcloud,
mermaid), `mcp-proxy --stateless` stdio→HTTP (papersearch), native HTTP
(markitdown, crawl4ai :11235), and the gateway's own **SSE `/` endpoint —
a STUB** (fake `initialize` + 60s pings, no routing to backends).
**Reverse proxy:** Zoraxy `mcp.nuclide.systems → 192.168.1.40:8080`.
`mcp-auth.nuclide.systems` is an abandoned auth-sidecar idea (not exposed).
**OAuth (Pocket ID @ `id.nuclide.systems`):** `OAuth2AuthorizationCodeBearer`,
scopes `{openid, mcp}`, token validation via userinfo. Shared gateway client
(`GENERIC_CLIENT_ID`, same as LiteLLM/LobeChat). Per-server OAuth for
papersearch & nextcloud against the same Pocket ID.
## 3. MCP Server Inventory (reconciled — `server.py` MCP_SERVERS is authoritative)
| Server | Image / build | Transport | Port | Auth | Status |
|---|---|---|---|---|---|
| papersearch | `python:3.12-slim` + runtime `uv tool install mcp-proxy` → `paper_search_mcp.server` | mcp-proxy stdio→http | 8000 | Pocket ID `PAPERSEARCH_MCP_OAUTH_*` | plausible, runtime-install fragile |
| nextcloud | `ghcr.io/cbcoutinho/nextcloud-mcp-server:latest` | streamable-http | 8000 | Pocket ID `NEXTCLOUD_MCP_OAUTH_*` | likely workable (real image) |
| markitdown | `python:3.12-slim` + `uvx markitdown-mcp --http` | http | 8000 | none | **broken as written** (`uvx` not in base image) |
| comfyui | `ghcr.io/richardi-ai/comfyui-mcp-server:latest` (`type:"npm"` mismatch) | unspecified | 8000 | none | **image not pullable**; backend ComfyUI was crash-looping |
| crawl4ai | `unclecode/crawl4ai:latest` | http | 11235 | none | likely workable; resource limits lost in rewrite |
| mermaid | `node:20-slim` + runtime `npx -y mcp-mermaid` | streamable-http | 8000 | none | plausible, slow first start |
## 4. Implemented vs Unfinished vs Broken
**Implemented:** FastAPI app + OAuth scheme + userinfo token validation;
container lifecycle CRUD + persistence; Web UI SPA (`templates/ui.html` @ `/ui`);
gateway compose/Dockerfile + Zoraxy route.
**Unfinished / stub:**
- **SSE `/` is fake** — no MCP transport bridging Claude.ai → spawned servers. *Core gap.*
- **No routing to per-server containers**; all five servers bind the same `:8000`
and `spawn_container` host-publishes `8000:8000` → **two servers can't run at once**.
- **OAuth callback non-functional** — token-exchange URL built via
`OAUTH_REDIRECT_URI.replace("/sso/callback","/token")` (→ wrong host, not the
Pocket ID token endpoint); token never stored/used.
- `config.json` empty → always defaults; secrets hardcoded plaintext in `server.py`.
**Broken / contradictory:**
- `ai/docker-compose.yml:6` mcp-tools include commented out; gateway compose is a
*separate project* not referenced by the stack either — wired in only via Zoraxy.
- `mcp-tools.yaml`: duplicate `markitdown-mcp` key; `comfyui-mcp` env missing `=`
(`- COMFYUI_URL http://comfyui:8188`); missing images/ports.
- `comfyui` server `type:"npm"` vs Docker-image mismatch; upstream image/npm
package existence unverified (image confirmed **not pullable**).
- `.env` has `CRAWL4AI_MCP_OAUTH_*`, `COMFYUI_MCP_OAUTH_*` that `server.py`
never consumes; code hardcodes secrets instead of `${ENV}` substitution.
## 5. Relevant `.env` keys (names only)
Gateway: `GENERIC_CLIENT_ID/_SECRET/_REDIRECT_URI`,
`GENERIC_{AUTHORIZATION,TOKEN,USERINFO}_ENDPOINT`, `GENERIC_CLIENT_USE_PKCE`,
`OAUTH_SCOPES`, `OAUTH_TOKEN_URL`.
Per-server: `NEXTCLOUD_MCP_OAUTH_CLIENT_ID/_SECRET`,
`PAPERSEARCH_MCP_OAUTH_CLIENT_ID/_SECRET`,
`MARKITDOWN_MCP_OAUTH_CLIENT_ID/_SECRET` (declared, unused),
`CRAWL4AI_/COMFYUI_MCP_OAUTH_*` (orphaned).
papersearch data sources: `UNPAYWALL_EMAIL`, `CORE_API_KEY`,
`SEMANTIC_SCHOLAR_API_KEY`, `ZENODO_ACCESS_TOKEN`, `GOOGLE_SCHOLAR_PROXY_URL`,
`DOAJ_API_KEY`. comfyui: `COMFYUI_URL`, `COMFYUI_WS_URL`.
## 6. Open Design Decisions (must resolve)
1. **Hosting model**: DinD gateway vs static `mcp-tools.yaml` vs LiteLLM-hosted.
2. **How Claude.ai/LobeChat reach a tool**: the MCP transport bridge doesn't exist.
3. **Port allocation**: all servers hardcode `:8000` — need internal DNS, no host publish.
4. **Uniform npx/uvx/docker run model**: prebuilt images vs runtime install.
5. **Auth model**: gateway-terminated vs per-MCP vs pass-through (callback is broken).
6. **Secret handling**: hardcoded → `${ENV}` from `ai/.env`.
7. **comfyui server**: image vs npm; keep only once ComfyUI itself is stable.
8. **Discovery for LobeChat/LiteLLM**: `/mcp.json` is OAuth-gated, lists config not endpoints.
## 7. Recommended Path (ordered, lowest-risk first)
1. **Pick the static-compose path, not DinD** — lowest risk on a single NUC;
DinD adds socket-exposure risk + a broken SSE bridge for little gain.
Fix and re-enable `mcp-tools.yaml` (uncomment `ai/docker-compose.yml:6`).
2. **Fix `mcp-tools.yaml`**: dedupe `markitdown-mcp`, fix `comfyui-mcp` env `=`,
unique service names → `ai-internal` DNS, pin images, drop comfyui for now.
3. **One streamable-http reverse proxy keyed by path** (`mcp.nuclide.systems/<server>`)
via Zoraxy or a small `httpx` proxy — replace the fake SSE stub. Backends stay
internal on `ai-internal:8000`, never host-published.
4. **Move secrets to `${ENV}`** from `ai/.env` (keys already exist).
5. **Fix OAuth callback**: exchange code against `GENERIC_TOKEN_ENDPOINT`.
6. **Verify each upstream image/tool exists** before marking a server "working".
7. **Defer the DinD gateway + Web UI** to phase-2 (read-only status over the
running compose, not spawning).
8. **Answer the litellm question**: once stable `https://mcp.nuclide.systems/<server>`
URLs exist, populate `litellm-config/config.yaml` `mcp_servers:` so
LiteLLM/LobeChat discover them — no bespoke discovery path needed.
history/scrubbing-list-2026-05-17.md
+53
View File
@@ -0,0 +1,53 @@
# Scrubbing List — /opt/stacks (2026-05-17)
Read-only audit of unused/stale data. **Nothing here has been deleted** — review
the labels and run the commands yourself. Root FS was **165G/200G used (83%)**;
`/var/lib/docker` is 86G of `/opt/stacks`'s 99G.
No stopped/exited/`*_old` containers; no unused custom networks (already clean).
## 1. Docker reclaimable
| Target | Size | Label | Command |
|---|---|---|---|
| Build cache (268 entries, old comfyui CPU→Arc rebuilds, 0 in use) | **~20.85 GB** | **SAFE** | `docker builder prune -af` |
| 4× dangling `<none>` mcp-gateway rebuild images (1.59 GB ea) | **~6.36 GB** | **SAFE** | `docker image prune` |
| 2 old dangling images (756 MB + 113 MB) | **~0.87 GB** | **SAFE** | (same `docker image prune`) |
| ~265 anon volumes; one `54bcf7…` = **8.69 GB** unidentified, rest ~0B | ~9 GB | **REVIEW** | inspect `54bcf7…` then `docker volume prune` |
| Named dangling vols: `n8n_n8n_storage` 128M, `ai_mcpo-data` 84M, `paperless-ngx_pgdata` 26M, `metamcp_postgres_data` 18M, `daytona*_db_data` 15M×2, `librechat_pgdata2` 13M, `arcane_arcane-data` 12M, `ai_redis_data` 7.6M | ~0.3 GB | **REVIEW** | `docker volume rm <name>` per-item after confirming the stack is retired |
In-use, **DO NOT REMOVE**: `comfyui-comfyui` (6.45G), `clusterzx/paperless-ai` (8.59G).
## 2. Migrated/abandoned local data dirs (compose now points to UNAS)
| Path | Size | Label | Note |
|---|---|---|---|
| `arr-stack/media` | **55 G** | **REVIEW** | No container mounts it; arr → `/mnt/pve/unas/media`. Audiobooks/ebooks have recent mtimes (rsync residue) — **parity-check vs UNAS before `rm -rf`** |
| `ai/data` (old postgres) | 195 M | **SAFE** | Not mounted, not referenced |
| `ai/postgres_data` (incl 38M pg_wal) | 120 M | **REVIEW** | Not mounted/referenced but recent mtime |
| `ai/meili_data_v1.35.1` | 19 M | **SAFE** | Old Meili, not mounted |
| `qdrant/qdrant_storage` | 7 M | **SAFE** | qdrant migrated to UNAS (fresh start) |
| `daytona/db_data` | 14 M | **REVIEW** | No mount; daytona uses named volumes |
| `n8n/data` | empty | **SAFE** | `rmdir` |
Active local, **KEEP**: `immich/postgres` (814M), `ai/lobehub/data` (25M),
`shared-db/wal-g` (27M, RO mount), arr-stack configs, `ai/litellm-config`,
`ai/searxng`.
## 3. Migration / scratch artifacts
| Path | Label | Note |
|---|---|---|
| `ai/docker` (0 B), `ai/bucket.config.json` (empty dir) | **SAFE** | junk |
| `scripts/traefik-*.sh` | **SAFE** | Traefik abandoned for Zoraxy |
| `scripts/{migrate_*,test_adguard_api,zoraxy_csrf,zoraxy_test,configure_zoraxy_*}.py` | **REVIEW** | one-shot done; confirm no rerun need |
| `scripts/zoraxy_sync.py` | **KEEP** | ongoing proxy tooling |
| `scripts/.venv` (29M), `scripts/.kilo` (30M) | **REVIEW** | regenerable caches |
| `/tmp/{flux_*,pw_ui,add_*,fix_*}.* , /tmp/*.png , /tmp/*.log` | **SAFE** | ~1.8M scratch (this session) |
## Bottom line
- **Safe-only reclaim: ~29–30 GB** (build cache + dangling images dominate).
- **With review: +~64 GB** (55G stale `arr-stack/media` after UNAS parity check, ~9G anon vols).
- **Highest-value, zero-risk action:** `docker builder prune -af` → ~20.85 GB now.
- Biggest overall prize: verify+remove `arr-stack/media` (55 G) — parity-check vs `/mnt/pve/unas/media` first.
history/traefik-migration-docker-labels.md
+341
View File
@@ -0,0 +1,341 @@
# Traefik Migration Guide Using Docker Labels
## Overview
Migrate from Zoraxy reverse proxy to Traefik using Docker labels for zero-touch service discovery.
---
## Phase 1: Install Traefik
### Step 1: Create Directory Structure
```bash
mkdir -p /opt/stacks/proxy/traefik/{config,dynamic,letsencrypt}
```
### Step 2: Create docker-compose.yml
```yaml
version: "3.8"
services:
traefik:
image: traefik:v3.2
container_name: traefik
restart: always
network_mode: host
security_opt:
- no-new-privileges=true
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /opt/stacks/proxy/traefik/config:/etc/traefik
- /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic
- /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=ai-internal"
- "--providers.docker.network=shared_backend"
- "--providers.docker.defaultRule=Host(`{{ .Name }}.nuclide.systems`)"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.letsencrypt.acme.httpChallenge=true"
- "--certificatesresolvers.letsencrypt.acme.email=admin@nuclide.systems"
- "--certificatesresolvers.letsencrypt.acme.storage=/etc/letsencrypt/acme.json"
```
### Step 3: Start Traefik
```bash
cd /opt/stacks/proxy/traefik
docker compose up -d
# Verify
docker compose ps
```
---
## Phase 2: Migrate AI Services
### AI Service Labels (Add to litellm, chat, mcp-compose.yml)
```yaml
services:
litellm:
image: litellm
labels:
- "traefik.enable=true"
- "traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)"
- "traefik.http.routers.litellm.entrypoints=websecure"
- "traefik.http.routers.litellm.tls=true"
- "traefik.http.routers.litellm.tls.certresolver=letsencrypt"
- "traefik.http.routers.litellm.priority=10"
- "traefik.http.services.litellm.loadbalancer.server.port=14000"
chat:
image: lobehub
labels:
- "traefik.enable=true"
- "traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)"
- "traefik.http.routers.chat.entrypoints=websecure"
- "traefik.http.routers.chat.tls=true"
- "traefik.http.routers.chat.tls.certresolver=letsencrypt"
- "traefik.http.routers.chat.priority=10"
- "traefik.http.services.chat.loadbalancer.server.port=14001"
mcp:
image: mcp-gateway
labels:
- "traefik.enable=true"
- "traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)"
- "traefik.http.routers.mcp.entrypoints=websecure"
- "traefik.http.routers.mcp.tls=true"
- "traefik.http.routers.mcp.tls.certresolver=letsencrypt"
- "traefik.http.routers.mcp.priority=10"
- "traefik.http.services.mcp.loadbalancer.server.port=8080"
```
---
## Phase 3: Migrate Garage S3
### Option A: Use Traefik Proxy
```yaml
services:
garage-proxy:
image: nginx:alpine
labels:
- "traefik.enable=true"
- "traefik.http.routers.s3.rule=Host(`s3.nuclide.systems`)"
- "traefik.http.routers.s3.entrypoints=websecure"
- "traefik.http.routers.s3.tls=true"
- "traefik.http.routers.s3.tls.certresolver=letsencrypt"
- "traefik.http.services.s3.loadbalancer.server.port=10004"
volumes:
- garage-data:/data
networks:
- shared_backend
```
### Option B: Keep Internal Garage Access
```yaml
services:
# No proxy needed - access Garage via internal IP:10004
garage:
image: garageio/garage
ports:
- "3900:3900" # Internal only
- "10004:10004" # Public via Traefik
```
---
## Phase 4: Create Helper Scripts
### script 1: Add Service to Traefik
```bash
#!/bin/bash
# /opt/stacks/scripts/add-traefik-service.sh
NAME=$1
DOMAIN=$2
PORT=$3
cat > /opt/stacks/proxy/traefik/dynamic/${NAME}.yml << EOF
http:
routers:
${NAME}-router:
rule: "Host(\`${DOMAIN}\`)"
service: ${NAME}-service
entrypoints:
- websecure
tls:
certresolver: letsencrypt
services:
${NAME}-service:
loadBalancer:
servers:
- url: "http://192.168.1.40:${PORT}"
EOF
# Reload Traefik docker automatically (no manual step needed)
echo "✅ Service ${NAME} added via labels"
```
Usage:
```bash
/opt/stacks/scripts/add-traefik-service.sh myservice myservice.nuclide.systems 8000
```
### Script 2: Generate Labels for Existing Services
```bash
#!/bin/bash
# /opt/stacks/scripts/traefik-labels-gen.sh
cat > /opt/stacks/proxy/traefik/labels.yaml << 'EOF'
# Add these labels to service docker-compose.yml files
# LiteLLM
services:
litellm:
labels:
- "traefik.enable=true"
- "traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)"
- "traefik.http.routers.litellm.entrypoints=websecure"
- "traefik.http.routers.litellm.tls=true"
- "traefik.http.routers.litellm.tls.certresolver=letsencrypt"
- "traefik.http.services.litellm.loadbalancer.server.port=14000"
# LobeHub Chat
services:
chat:
labels:
- "traefik.enable=true"
- "traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)"
- "traefik.http.routers.chat.entrypoints=websecure"
- "traefik.http.routers.chat.tls=true"
- "traefik.http.routers.chat.tls.certresolver=letsencrypt"
- "traefik.http.services.chat.loadbalancer.server.port=14001"
# MCP Gateway
services:
mcp-gateway:
labels:
- "traefik.enable=true"
- "traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)"
- "traefik.http.routers.mcp.entrypoints=websecure"
- "traefik.http.routers.mcp.tls=true"
- "traefik.http.routers.mcp.tls.certresolver=letsencrypt"
- "traefik.http.services.mcp.loadbalancer.server.port=8080"
EOF
echo "✅ Labels saved to /opt/stacks/proxy/traefik/labels.yaml"
```
Usage:
```bash
/opt/stacks/scripts/traefik-labels-gen.sh
```
---
## Phase 5: Update Service Configs
### Update `/opt/stacks/ai/.env`
```bash
# OLD (Zoraxy):
LITELLM_BASE_URL=https://litellm.nuclide.systems
PROXY_BASE_URL=https://mcp.nuclide.systems
# NEW (Traefik) - same URLs, different backend:
LITELLM_BASE_URL=https://litellm.nuclide.systems
PROXY_BASE_URL=https://mcp.nuclide.systems
CHATAI_BASE_URL=https://chat.nuclide.systems
```
### Update `/opt/stacks/ai/litellm-config/config.yaml`
```yaml
general_settings:
proxy_base_url: https://litellm.nuclide.systems
control_plane_url: https://litellm.nuclide.systems
```
---
## Phase 6: Verify SSL
### Step 1: Generate Let's Encrypt Certificates
```bash
# Verify Traefik is running
docker compose ps traefik
# Create ACME cert file
touch /opt/stacks/proxy/traefik/letsencrypt/acme.json
chmod 600 /opt/stacks/proxy/traefik/letsencrypt/acme.json
# Trigger certificate generation (will happen automatically)
# Check status:
curl -s https://acme-v02.api.letsencrypt.org/directory | head
```
### Step 2: Test HTTPS
```bash
# Test endpoints
curl -k https://litellm.nuclide.systems/health
curl -k https://chat.nuclide.systems/health
curl -k https://mcp.nuclide.systems/health
# Verify cert
curl -v https://litellm.nuclide.systems 2>&1 | grep -A 5 "SSL certificate"
```
---
## Phase 7: Remove Zoraxy
### Backup first
```bash
# Backup Zoraxy configs
docker cp zoraxy:/data/configs /backup/zoraxy-backup/
# Optional: Stop Zoraxy
docker stop zoraxy
docker rm zoraxy
```
---
## Quick Migration Checklist
- [ ] Install Traefik (Phase 1)
- [ ] Add labels to AI services (Phase 2)
- [ ] Add Garage S3 proxy (Phase 3)
- [ ] Run label generation script (Phase 4)
- [ ] Update `.env` and config files (Phase 5)
- [ ] Wait for SSL certificates (auto 24-48h)
- [ ] Test all HTTPS endpoints
- [ ] Remove Zoraxy (Phase 7)
- [ ] Update AdGuard DNS if needed (optional)
---
## Monitoring & Troubleshooting
### Check Traefik Dashboard
```bash
# Access web UI (unsecured - use only on trusted network)
open http://192.168.1.40:8080/dashboard
# View all routers
curl http://localhost:8080/api/http/routers | jq '.[] | {name: .rule, status: .entryPoints}'
# View all services
curl http://localhost:8080/api/http/services | jq '.[] | {name: .name, servers: .servers}'
```
### Common Issues
| Issue | Solution |
|-------|----------|
| 404 errors | Check router labels match domain exactly |
| SSL expired | Wait for auto-renew or trigger manually |
| Port mismatch | Verify `loadbalancer.server.port` matches service |
| No SSL cert | Check email in `acme.json` config |
---
## Rollback Plan (If Needed)
```bash
# Stop Traefik
docker compose -f /opt/stacks/proxy/traefik/docker-compose.yml down
# Restore Zoraxy configs
docker cp /backup/zoraxy-backup/ configs/
# Restart Zoraxy (if you kept backup)
docker start zoraxy || true
```
history/traefik-migration.md
+193
View File
@@ -0,0 +1,193 @@
# Recommended: Traefik Proxy Replacement
## Why Traefik over Zoraxy?
| Feature | Zoraxy | Traefik |
|---------|--------|---------|
| **API** | ❌ No public API | ✅ Full REST API |
| **SSL** | 💬 Manual (Zoraxy Web UI) | 🔥 Auto-Let's Encrypt |
| **Dynamic** | ⚠️ Manual config reload | ✅ Hot-reload configs |
| **File watching** | ❌ | ✅ Auto-detect changes |
| **Docker integration** | ⚠️ Manual | ✅ Native labels |
| **API endpoints** | 403 Forbidden | ✅ JSON API everywhere |
## Migration Path
### Current setup:
```
Zoraxy (192.168.1.4:8000) → Reverse Proxy Rules (Manual Web UI)
- litellm.nuclide.systems → 192.168.1.40:14000
- chat.nuclide.systems → 192.168.1.40:14001
- mcp.nuclide.systems → 192.168.1.40:8080
- s3.nuclide.systems → Garage:10004
```
### New setup with Traefik:
```
Traefik (public SSL) → Dynamic Router (labels/consul)
- All services auto-discovered via Docker labels
- SSL certificates auto-provisioned
- No manual Zoraxy configuration needed
```
## Installation
### Step 1: Install Traefik
```bash
# Create Traefik directory
mkdir -p /opt/stacks/proxy/traefik/{conf,dynamic}
# Create docker-compose.yml
cat > /opt/stacks/proxy/traefik/docker-compose.yml << 'EOF'
version: "3.8"
services:
traefik:
image: traefik:v3.2
container_name: traefik
restart: always
security_opt:
- no-new-privileges=true
network_mode: host
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /opt/stacks/proxy/traefik/conf:/etc/traefik
- /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic
- /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresletsencryptemail=admin@nuclide.systems"
- "--certificatesresletsencryptstorage=/etc/letsencrypt/acme.json"
EOF
# Start Traefik
cd /opt/stacks/proxy/traefik && docker compose up -d
```
### Step 2: Create Dynamic Configuration
```bash
# Create router rules
cat > /opt/stacks/proxy/traefik/dynamic/router.yml << 'EOF'
http:
routers:
litellm-router:
rule: "Host(`litellm.nuclide.systems`)"
service: litellm-service
entrypoints:
- websecure
tls:
certresolver: letsencrypt
chat-router:
rule: "Host(`chat.nuclide.systems`)"
service: chat-service
entrypoints:
- websecure
tls:
certresolver: letsencrypt
mcp-router:
rule: "Host(`mcp.nuclide.systems`)"
service: mcp-service
entrypoints:
- websecure
tls:
certresolver: letsencrypt
s3-router:
rule: "Host(`s3.nuclide.systems`)"
service: garage-service
entrypoints:
- websecure
tls:
certresolver: letsencrypt
services:
litellm-service:
loadBalancer:
servers:
- url: "http://192.168.1.40:14000"
chat-service:
loadBalancer:
servers:
- url: "http://192.168.1.40:14001"
mcp-service:
loadBalancer:
servers:
- url: "http://192.168.1.40:8080"
garage-service:
loadBalancer:
servers:
- url: "http://garage:10004"
EOF
```
### Step 3: Add Docker Labels to Services
For any Docker service you want to proxy:
```yaml
# Example: Add to your service docker-compose.yml
services:
ai-service:
image: your-service
labels:
- "traefik.enable=true"
- "traefik.http.routers.your-service.rule=Host(`your-service.nuclide.systems`)"
- "traefik.http.routers.your-service.entrypoints=websecure"
- "traefik.http.routers.your-service.tls.certresolver=letsencrypt"
- "traefik.http.services.your-service.loadBalancer.server.port=8000"
```
### Step 4: Delete Zoraxy (Optional)
```bash
# Backup Zoraxy configs first
tar -czf /backup/zoraxy-backup.tar.gz /path/to/zoraxy/configs
# Stop and remove Zoraxy
docker rm -f zoraxy || true
```
## API Example (Traefik)
```bash
# Get list of services
curl -u traefik:YOUR_TRAEFIK_API_PASSWORD http://localhost:8080/api/http/routers
# Get Traefik metrics
curl http://localhost:8080/metrics
# Reload configuration (live)
curl -X POST http://localhost:8080/api/http/routers -H "Content-Type: application/json" -d '{...}'
```
## Migration Checklist
- [ ] Deploy Traefik in host network mode
- [ ] Create Let's Encrypt certificate for your domain
- [ ] Migrate all reverse proxy rules to Traefik labels or dynamic config
- [ ] Test SSL certificates work: `curl -k https://your-domain.nuclide.systems`
- [ ] Remove Zoraxy Docker container
- [ ] Update DNS if needed
- [ ] Verify all endpoints: health checks at new URLs
## Benefits
1. **Zero maintenance SSL** - Let's Encrypt auto-renews
2. **API-driven** - No manual Web UI needed
3. **Hot reloading** - Changes apply immediately
4. **Docker-native** - Watches container labels automatically
5. **Enterprise-grade** - Used by major cloud providers