fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: MCP gateway git/gitlab/paper-search; CT 109 plan; new backrest/db/arcane docs

Browse Source 
- mcp-gateway.md: 26 servers (added git, gitlab, paper-search catalog bridge);
  document --pass-environment requirement for env var passthrough in DinD bridges
- ct-inventory.md + homelab-architecture.md: CT 109 IP conflict, CT 113 live
- proxmox-state.md §16: Loki, sshwifty, Alloy, IP conflict note, sidecar table
  expanded to CT 103/111/113; build order updated
- infra/portmap.md: CT 113 db section; QNAP TS-251D (Klipper/Mainsail)
- services/backrest.md: full backup strategy doc (new)
- services/databases.md: CT 113 postgres/WAL-G/pgAdmin doc (new)
- services/arcane.md: Docker management IDE doc (new)
- CHANGELOG.md: 2026-05-22 entry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Florian Krebs
2026-05-22 09:35:07 +02:00
parent 0def5c0be9
commit 0d83f83bc0
9 changed files with 693 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -2,6 +2,13 @@
All notable infrastructure / service / doc changes. Newest first. All notable infrastructure / service / doc changes. Newest first.
## 2026-05-22
- **MCP gateway**: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`.
- **CT 109 plan updated**: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.
- **Portmap**: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added.
- **New docs**: `services/backrest.md`, `services/databases.md`, `services/arcane.md`.
## 2026-05-21 ## 2026-05-21
### Docs & docs infrastructure ### Docs & docs infrastructure
ct-inventory.md
+1 -1
View File
@@ -11,7 +11,7 @@ Verified 2026-05-20 via `pct list`, `pct config <id>`, `qm config 100` on `nuc`.
| 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running |
| 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 **CIFS** (`//192.168.1.31/storage`) | Intel iGPU (render only) | running | | 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 **CIFS** (`//192.168.1.31/storage`) | Intel iGPU (render only) | running |
| 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running |
| 109 | observe | LXC | 192.168.1.6/24 | 4 | 6144 | 30 GiB | **Planned** — Prometheus, Grafana, Arcane Manager, Dozzle, Homarr, Diun, Tinyauth, socket-proxy | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 109 | observe | LXC | ⚠️ TBD (`.6` taken by CT 113) | 4 | 8192 | 50 GiB | **Planned** — Prometheus, Loki, Grafana, Arcane Manager, Dozzle, Homarr, Diun, sshwifty, Tinyauth, socket-proxy; Alloy agents on all hosts | — | no | **planned** (see `infra/proxmox-state.md` §16) |
| 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running |
| 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running |
| 112 | secrets | LXC unpriv | TBD | 2 | 2048 | 8 GiB | **Planned** — Infisical secrets manager (replaces scattered .env files) | — | no | **planned** | | 112 | secrets | LXC unpriv | TBD | 2 | 2048 | 8 GiB | **Planned** — Infisical secrets manager (replaces scattered .env files) | — | no | **planned** |
infra/portmap.md
+26 -2
View File
@@ -83,7 +83,7 @@ All arr-stack services run behind `vpn_gluetun` container network.
| [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway | | [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway |
| [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway | | [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway |
| [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only | | [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only |
| ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — Shepard will provide a native MCP server | | ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — replaced by native `https://shepard.nuclide.systems/v2/mcp` (streamable HTTP; gateway entry: `shepard`, auth via `${SHEPARD_API_KEY}`) |
| [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway | | [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway |
| — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub |
@@ -136,7 +136,18 @@ _Traccar moved to 15000–15001 (Documents range). 19000–19001 now free._
| Port | Service | Container | Public URL | Notes | | Port | Service | Container | Public URL | Notes |
|---|---|---|---|---| |---|---|---|---|---|
| 20010 (127.0.0.1 only) | pgAdmin | `pgadmin` | — | localhost only | | 20010 (127.0.0.1 only) | pgAdmin | `pgadmin` | — | shared-db pgAdmin, localhost only (CT 104) |
---
## LXC 113 — db (`192.168.1.6`)
CT 113 is the dedicated postgres LXC. No public proxy routes — LAN access only.
| Port | Service | Container | Notes |
|---|---|---|---|
| 5432 | Postgres 17 | `postgres` | LAN: `192.168.1.6:5432` — accepts app connections from all CTs |
| 5050 | pgAdmin 4 | `pgadmin` | LAN only: `http://192.168.1.6:5050` — no Zoraxy route |
--- ---
@@ -157,6 +168,19 @@ CT 111 hosts the self-hosted dev platform. Same Pocket-ID SSO as the rest.
--- ---
## QNAP TS-251D (`192.168.1.189`)
Celeron J4025, 2-core. Hosts Klipper natively (not Docker).
| Port | Service | Notes |
|---|---|---|
| 80 | Mainsail | 3D printer web UI |
| 7125 | Moonraker | Klipper API |
Config backed up daily to `git.nuclide.systems/fkrebs/klipper-config` via cron at 03:00 → covered offsite by Backrest `services/gitea` path.
---
## External Services (Not on NUC Docker) ## External Services (Not on NUC Docker)
Zoraxy routes to these external backends: Zoraxy routes to these external backends:
infra/proxmox-state.md
+30 -18
View File
@@ -712,30 +712,41 @@ Recovery latency improved from 5 min → 1 min; logging structured in `journalct
## 16. CT 109 "observe" — planned observability + ops LXC ## 16. CT 109 "observe" — planned observability + ops LXC
Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores / 6 GiB RAM / 30 GiB rootfs**, unprivileged, nesting=1, IP `192.168.1.6` (next free in the infra range after CT 110). Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores / 8 GiB RAM / 50 GiB rootfs**, unprivileged, nesting=1. RAM bumped from 6 → 8 GiB to accommodate Loki. Disk bumped from 30 → 50 GiB for Loki log retention (30d) alongside Prometheus TSDB.
⚠️ **IP conflict:** inventory originally assigned `192.168.1.6` but CT 113 (db) is already live at `.6`. CT 109 needs the next free infra IP — likely `.7` or `.8` (verify against UniFi DHCP table before provisioning).
**Access model (initial):** LAN-only. No Zoraxy routes until Tinyauth is deployed. Services reachable directly by IP.
### Stack to deploy on CT 109 ### Stack to deploy on CT 109
| Service | Purpose | | Service | Purpose |
|---|---| |---|---|
| Arcane Manager | central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out) | | Prometheus | metrics TSDB, 30d retention |
| Dozzle UI | aggregated container logs; Dozzle agents on CT 101 + CT 104 + CT 110 | | Loki | log aggregation backend — receives from Alloy agents on all hosts |
| Prometheus | metrics TSDB, 30 d retention | | Grafana | dashboards over Prometheus + Loki (unified metrics + log search) |
| Grafana | dashboards over Prometheus |
| Alertmanager + alertmanager-gotify-bridge | alert routing → Gotify | | Alertmanager + alertmanager-gotify-bridge | alert routing → Gotify |
| **Homarr** (selected over Homepage) | unified dashboard, native OIDC with Pocket-ID, native Prometheus widget + Grafana iframe support | | Arcane Manager | central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out) |
| Dozzle UI | live log tail (quick debugging); agents on CT 101 + CT 104 + CT 110. Complements Loki — Dozzle for live, Loki for historical/search |
| **Homarr** | unified dashboard, native Pocket-ID OIDC, Prometheus widget + Grafana iframe support |
| Diun | docker image update notifier → Gotify | | Diun | docker image update notifier → Gotify |
| Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana). Speaks OIDC client to Pocket-ID, returns Basic-Auth/Remote-User header to upstream | | Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID |
| docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr discovery | | docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr/Arcane discovery |
| **sshwifty** | web SSH client, multi-tab — multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred. |
### Sidecars deployed on each Docker host ### Sidecars deployed on each host
Two agents per host — **node-exporter** (metrics) and **Alloy** (logs). Kept separate: node-exporter metric names are assumed by every Prometheus dashboard/alert; Alloy emitting compatible metrics adds validation risk for no gain.
| Host | Sidecars | | Host | Sidecars |
|---|---| |---|---|
| Host (PVE) | node-exporter, smartctl-exporter, pve-exporter (host metrics into Prometheus) | | PVE host | node-exporter, smartctl-exporter, pve-exporter, **Alloy** (journald → Loki: pve-manager, pveproxy, pvedaemon, LXC/VM lifecycle) |
| CT 101 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy | | CT 101 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, **Alloy** (Docker logs + journald → Loki) |
| CT 104 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy | | CT 103 | node-exporter, **Alloy** (backrest.service journal + `/var/log/rclone-*.log` → Loki) |
| CT 110 | node-exporter, Dozzle agent, Arcane edge agent (test target — small/clean) | | CT 104 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, intel_gpu_exporter, **Alloy** (Docker logs + journald → Loki) |
| CT 110 | node-exporter, Dozzle agent, Arcane edge agent, **Alloy** (journald → Loki) |
| CT 111 | node-exporter, intel_gpu_exporter, **Alloy** (journald + Coder/Gitea logs → Loki) |
| CT 113 | node-exporter, postgres_exporter, **Alloy** (journald + postgres logs → Loki) |
### Services that stay where they are (NOT on CT 109) ### Services that stay where they are (NOT on CT 109)
@@ -774,11 +785,12 @@ Build-order slot: after Arcane + socket-proxy land in §16's checklist, before T
4. Issue agent tokens, deploy Edge agents on CT 101/104/110 4. Issue agent tokens, deploy Edge agents on CT 101/104/110
5. Deploy Dozzle Manager + agents 5. Deploy Dozzle Manager + agents
6. Deploy Diun 6. Deploy Diun
7. Deploy Tinyauth (configure Pocket-ID client first) 7. Deploy sshwifty (configure host list + SSH keys for PVE, CT 103, CT 104, CT 111; LAN-only port 8182)
8. Deploy Homarr 8. Deploy Tinyauth (configure Pocket-ID client first)
9. Front the lot via Zoraxy: `arcane.`, `dozzle.`, `grafana.`, `prom.`, `home.` `.nuclide.systems` 9. Deploy Homarr
10. Verify each through the Tinyauth gate where applicable 10. Front the lot via Zoraxy: `arcane.`, `dozzle.`, `grafana.`, `prom.`, `home.`, `shell.` `.nuclide.systems`
11. Add Diun watchlist + Alertmanager routing → Gotify 11. Verify each through the Tinyauth gate where applicable
12. Add Diun watchlist + Alertmanager routing → Gotify
--- ---
services/arcane.md
+126
View File
@@ -0,0 +1,126 @@
# Arcane
Web-based Docker management IDE. Main instance on **CT 104** (`192.168.1.40:10002`, `arcane.nuclide.systems`). Manages containers on the local Docker host plus remote CTs via **edge agents**.
Stack: `/opt/stacks/arcane/docker-compose.yml` on CT 104.
Image: `ghcr.io/getarcaneapp/arcane:latest`
Auth: OIDC via Pocket ID, admin: `fkrebs@nucli.de`
---
## Edge agents
An edge agent (`ghcr.io/getarcaneapp/arcane-headless:latest`) runs inside each remote CT and connects **outbound** to the main Arcane server via gRPC poll. The main server then manages that CT's Docker.
### Currently deployed agents
| CT | Hostname | Compose path | Status |
|----|----------|--------------|--------|
| CT 104 | docker | *(built-in NUC environment)* | online |
| CT 113 | db | `/opt/stacks/db/docker-compose.yml` | online (2026-05-21) |
### Adding an edge agent to a new CT
**Step 1 — Create the environment in Arcane** (requires Arcane API or UI access)
With the admin CLI API key (stored in Arcane DB, regenerate if needed):
```bash
# Get or create an admin API key — see "Admin API key" section below
API_KEY="arc_..."
curl -s -X POST -H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \
"http://192.168.1.40:10002/api/environments" \
-d '{"name":"<ctname>","apiUrl":"edge://<ctname>","isEdge":true}'
# Note the environment id from the response
```
**Step 2 — Generate and wire the AGENT_TOKEN**
Arcane stores the raw AGENT_TOKEN in `environments.access_token`. Insert it via:
```python
# On PVE host, run: python3 /tmp/arcane-bootstrap.py
import argon2, secrets, sqlite3, uuid
from datetime import datetime, timezone
db_path = "/rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db"
env_id = "<id from step 1>" # paste the environment UUID here
conn = sqlite3.connect(db_path)
raw_token = "arc_" + secrets.token_hex(32)
key_prefix = "arc_" + raw_token[4:12]
ph = argon2.PasswordHasher(memory_cost=65536, time_cost=3, parallelism=2)
key_hash = ph.hash(raw_token)
key_id = str(uuid.uuid4())
now = datetime.now(timezone.utc).isoformat()
conn.execute(
"INSERT INTO api_keys (id, name, description, key_hash, key_prefix, environment_id, managed_by, created_at, updated_at)"
" VALUES (?,?,?,?,?,?,?,?,?)",
(key_id, f"Environment Bootstrap Key - {env_id[:8]}",
"Auto-generated key for environment pairing",
key_hash, key_prefix, env_id, "system", now, now))
conn.execute("UPDATE environments SET access_token=? WHERE id=?", (raw_token, env_id))
conn.commit()
conn.close()
print("AGENT_TOKEN:", raw_token)
```
**Step 3 — Add the agent to the CT's compose**
```yaml
arcane-agent:
image: ghcr.io/getarcaneapp/arcane-headless:latest
container_name: arcane-agent
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./arcane-agent:/app/data
environment:
EDGE_AGENT: "true"
EDGE_TRANSPORT: poll
AGENT_TOKEN: ${ARCANE_AGENT_TOKEN}
MANAGER_API_URL: http://192.168.1.40:10002
deploy:
resources:
limits:
cpus: "0.5"
memory: 256M
```
Add `ARCANE_AGENT_TOKEN=<raw_token>` to the CT's `.env`.
**Step 4 — Start and verify**
```bash
docker compose up -d arcane-agent
docker logs arcane-agent 2>&1 | grep "Edge gRPC tunnel"
# Expected: Edge gRPC tunnel connected to manager environment_id=<uuid>
```
The environment should flip to `online` in `environments.status` within ~5 seconds.
---
## When Arcane moves to CT 109
When Arcane migrates from CT 104 to CT 109 (the planned "observe" LXC):
1. Update `MANAGER_API_URL` in every edge agent's `.env` to the new CT 109 LAN IP
2. Update the Arcane compose `APP_URL` and network bindings
3. Update Zoraxy route for `arcane.nuclide.systems` to point to CT 109
4. Migrate the SQLite DB: `cp -a /opt/stacks/arcane/data/ <ct109-path>/`
5. Regenerate the admin API key (stored in memory, not config) since the DB port changes
6. Restart all edge agents with the new `MANAGER_API_URL`
Arcane environments (NUC, db, shepard) will reconnect automatically once `MANAGER_API_URL` is updated.
---
## Admin API key (non-OIDC access)
Arcane doesn't store plaintext API keys — use the DB-insert method above when a new admin key is needed. The `python3-argon2` package must be installed on the PVE host (`apt install python3-argon2`).
The key inserted for CLI use during CT 113 provisioning (`arc_a9695182...`) is linked to `fkrebs` user in the api_keys table. Rotate it after provisioning work is complete by deleting the row:
```sql
sqlite3 /rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db \
"DELETE FROM api_keys WHERE name='admin-cli';"
```
services/backrest.md
+303
View File
@@ -0,0 +1,303 @@
# Backup Strategy
> Single Backrest instance on **CT 103** (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at `/mnt/pve/unas`, so it can read all service data without SSH-ing other hosts.
**Goal**: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.
---
## Current state
### Backrest (CT 103) — Phase 1b live since 2026-05-21
| Item | Value |
|---|---|
| UI | `http://192.168.1.3:9898` (LAN-only, auth disabled) |
| Config | `/opt/backrest/config/config.json` (timestamped `.bak` files on every edit) |
| rclone remote | `jottacloud:` Archive section (default device/mountpoint); token at `/root/.config/rclone/rclone.conf` |
| Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) |
| BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) |
**Repos** (both `autoInitialize: true`; passwords currently `tapirnase` — see blind spot #5):
| Repo | Target | Prune | Check |
|---|---|---|---|
| `services-repo` | `rclone:jottacloud:services` | weekly Sun 05:00, ≤10 % unused | monthly, 10 % subset |
| `media-repo` | `rclone:jottacloud:media` | monthly, ≤10 % unused | monthly, 10 % subset |
**Plans**:
| Plan | Repo | Schedule | Retention | Paths |
|---|---|---|---|---|
| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 15 paths (below) |
| `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` |
| `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` |
**Services plan paths** (all under `/mnt/pve/unas/`):
```
services/vaultwarden services/n8n services/memos
services/karakeep services/traccar services/gitea
services/coder services/nextcloud services/pocketid
services/arcane services/arr-stack services/gluetun
backup/home-assistant backup/immich backup/nextcloud
```
`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud.
> **JottaCloud web UI tip**: rclone writes to the **Archive** section. The default landing page shows only Sync + Backup. Browse to `https://www.jottacloud.com/web/archive` to see the restic repos.
### Postgres (CT 113) — Phase 2 live since 2026-05-21
Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmin on `:5050`. WAL-G archives continuously to Garage S3 (`ct113-pg-backup` on CT 104):
- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
- Daily `wal-g backup-push` cron at 02:00
- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder`
Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`).
**Exceptions** (stay on original hosts):
| DB | Host | Why |
|---|---|---|
| `lobe-postgres` (paradedb pg17) | CT 104 | Uses `pg_search` `USING bm25` indexes; stock postgres 17 can't host. WAL-G → `lobe-pg-backup` + Backrest secondary on data dir. |
| Nextcloud AIO postgres | CT 105 | AIO manages it; Borg archives the whole stack to UNAS. |
| Immich postgres | CT 104 | Version-pinned by Immich. WAL-G → `immich-pg-backup`. |
---
## Coverage map
| Host / data | Method | Offsite |
|---|---|---|
| CT 103 Backrest binary + config | Manual (small) | On-CT only |
| CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud |
| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ |
| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ |
| CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ |
| CT 101 Shepard | Source in Gitea (gitea path covers it) | ✓ |
| CT 102 AdGuard | Phase 3 git push → `fkrebs/adguard-conf` | ✗ not deployed |
| CT 105 Nextcloud user files | `services/nextcloud` | ✓ |
| CT 105 Nextcloud AIO volumes | AIO Borg → `/mnt/pve/unas/backup/nextcloud/` → Backrest | ✓ |
| CT 108 Zoraxy | Phase 3 git push → `fkrebs/zoraxy-conf` | ✗ not deployed |
| CT 110 Pocket-ID | `services/pocketid` | ✓ |
| CT 111 Gitea + Coder | `services/{gitea,coder}` | ✓ |
| VM 100 HAOS config | Phase 3 git addon → `fkrebs/ha-config` | ✗ not deployed |
| VM 100 HAOS daily tar | HA → UNAS → Backrest | ✓ |
| PVE host `/etc/pve/` | Phase 3 git push → `fkrebs/pve-conf` | ✗ not deployed |
| UNAS `media/images/library` | Phase 1a `rclone sync` → `jottacloud:Photos/` | ✗ not deployed |
| UNAS personal data (`documents`, `_sortMe`, `video-projects`, …) | Phase 1a `rclone sync` → `jottacloud:UNAS/` | ✗ not deployed |
### Intentionally not backed up
Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable.
### Deferred — include if needed
| Host | Data | Notes |
|---|---|---|
| CT 109 observe | Prometheus TSDB, Grafana dashboards, Arcane SQLite, sshwifty config | Low priority — metrics are ephemeral; dashboards can be re-exported from Grafana; add to services plan if state becomes hard to recreate |
---
## Open work
### Phase 1a — rclone sync for UNAS personal data
Two sync jobs on CT 103, Saturday 01:00:
- **Job 1**: `media/images/library/` → `jottacloud:Photos/` (Immich originals, visible at `jottacloud.com/photo`).
- **Job 2**: `/mnt/pve/unas/` → `jottacloud:UNAS/` (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, `services/` already in Backrest).
Initial upload ≈ 820 GB personal + 822 GB Immich (days). Script target `/usr/local/sbin/unas-sync.sh`. Full script in [appendix](#phase-1a-rclone-sync-script).
Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.
### Phase 3 — Config-to-git for infrastructure
Daily 03:00 cron on each host pushes config to a private Gitea repo. Repos already exist:
- `fkrebs/zoraxy-conf` (CT 108 — `/opt/zoraxy/conf/`)
- `fkrebs/adguard-conf` (CT 102 — `AdGuardHome.yaml`)
- `fkrebs/pve-conf` (PVE — `/etc/pve/`, excludes `priv/`, `*.key`, `authkey.pub*`)
- `fkrebs/ha-config` (VM 100 — `/config/`, excludes `secrets.yaml`, `.storage/`)
Script template in [appendix](#phase-3-config-to-git-scripts).
### Blind spots
| # | Severity | Issue | Fix |
|---|---|---|---|
| 4 | MEDIUM | n8n encryption key in `/opt/stacks/n8n/data/` on CT 104 local FS — not in any plan. If CT 104 dies, DB restore is unusable. | Move n8n data volume to `/mnt/pve/unas/services/n8n/` (already in plan). |
| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted — without it, restore impossible. | Store `BORG_PASSWORD` in Vaultwarden. |
| 5 | MEDIUM | Both Backrest repo passwords are `tapirnase`. | Rotate before first scheduled run completes; `restic key passwd` re-encrypts in place. Store in Vaultwarden. |
| 6 | LOW | `litellm` DB password is the placeholder literal `litellm_password_here`. | Generate real password; update `ai/.env`, `litellm-config/config.yaml`, CT 113 user. |
| 7 | LOW | Migration dumps at `/mnt/pve/unas/dump/*-migration-20260521.sql` not in any plan. | Decide: keep as manual archive or delete now that WAL-G is archiving. |
---
## Operations
### Restore
UI: **Repos → snapshots → Browse → file → Restore**.
CLI on CT 103:
```bash
restic -r rclone:jottacloud:services snapshots
restic -r rclone:jottacloud:services restore latest \
--target /restore \
--include /mnt/pve/unas/services/vaultwarden
```
### Monitoring (planned)
In Backrest UI → each repo → Hooks:
- `CONDITION_BACKUP_ERROR` / `CONDITION_CHECK_ERROR` → POST Gotify priority 8
- `CONDITION_BACKUP_SUCCESS` → POST Gotify priority 3
```
curl -s -X POST 'http://gotify:80/message?token=TOKEN' \
-H 'Content-Type: application/json' \
-d '{"title":"Backrest: {{.Plan}}","message":"{{.Summary}}","priority":3}'
```
### Applying a new config
```bash
# on CT 103
systemctl stop backrest
cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)
# paste new config.json
systemctl start backrest
```
---
## History
### Provider comparison
Researched 2026-05-21. Sized for ~3 TB/mo.
| Provider | ~3 TB/mo | Backend | EU DC | Egress | Verdict |
|---|---|---|---|---|---|
| **JottaCloud Unlimited** | €9.91 flat (unlimited) | rclone native | Norway (EEA) | Free | ✅ Chosen |
| Hetzner BX31 | €20.80 flat (10 TB) | SFTP/WebDAV | DE, FI | Free | 2.5× the price, capped |
| Backblaze B2 | ~$18 pay-per-GB | S3 | Frankfurt | Free (≤3× stored) | US CLOUD Act risk |
| Cloudflare R2 | ~$45 pay-per-GB | S3 | EU auto | Zero | Expensive at scale; no EU residency lock |
| Wasabi | ~$21–24 | S3 | FRA/AMS | Free (≤ stored) | ❌ 90-day min billing per object → Restic prune disaster |
| Storj DCS | ~$30 | S3 | EU-geofenced | 1× free | Complex, $5 minimum |
| pCloud | €399 one-time / 2 TB | WebDAV | Luxembourg | Free | WebDAV too slow |
| Infomaniak kDrive | ~€36+ / 3 TB | WebDAV | Switzerland | Free | WebDAV only; non-EU |
| Proton Drive | — | rclone beta | Switzerland | Free | rclone backend broken since late 2025 |
### Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21)
**Decision**: provision a dedicated `db` LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse `shared-postgres` on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113).
Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, `mp0=/mnt/pve/unas`. Stack in Gitea `fkrebs/stacks-db`. pgAdmin pre-registers postgres via `pgadmin-servers.json`.
**Migration order**: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB:
```bash
docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql
psql -U postgres -c "CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;"
psql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql
# Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume
```
Stale DBs dropped from `shared-postgres` post-migration: `daytona`, `lobechat` (duplicate), `paradedb` (duplicate).
WAL-G enabled 2026-05-21: `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` via `ALTER SYSTEM`; first base backup verified (`base_000000010000000000000012`). Daily `wal-g backup-push` cron at 02:00. Garage → JottaCloud offsite sync added at 02:30.
### Phase 1a rclone sync script
```bash
#!/bin/bash
set -e
LOG=/var/log/rclone-unas-sync.log
# Job 1: Immich originals → JottaCloud gallery
rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \
--transfers=4 --checkers=8 \
--log-file=$LOG --log-level INFO
# Job 2: all other irreplaceable personal data
rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \
--transfers=4 --checkers=8 \
--exclude "media/images/library/**" \
--exclude "media/images/upload/**" \
--exclude "media/images/thumbs/**" \
--exclude "media/images/encoded-video/**" \
--exclude "media/images/profile/**" \
--exclude "media/images/backups/**" \
--exclude "media/movies/**" \
--exclude "media/emulation/**" \
--exclude "media/Torrents/**" \
--exclude "media/music/**" \
--exclude "media/podcasts/**" \
--exclude "services/**" \
--exclude "backup/**" \
--exclude "backup-staging/**" \
--exclude "test_perm" \
--log-file=$LOG --log-level INFO
```
Cron (Saturday 01:00): `0 1 * * 6 /usr/local/sbin/unas-sync.sh`
### Phase 3 config-to-git scripts
Same pattern on each host. Replace `GITEA_TOKEN` with the value from `/opt/stacks/ai/.env` or a dedicated scoped Gitea token.
**CT 108 — Zoraxy** (`/usr/local/sbin/zoraxy-conf-backup.sh`):
```bash
#!/bin/bash
set -e
REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git"
WORK="/opt/zoraxy/conf"
git -C "$WORK" init -b main -q 2>/dev/null || true
git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \
|| git -C "$WORK" remote add origin "$REPO_URL"
git -C "$WORK" add -A
git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true
git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true
```
**CT 102 — AdGuard** — same template, copy `/opt/AdGuardHome/AdGuardHome.yaml` into `/tmp/adguard-conf-work` checkout first.
**PVE host** — same template, `rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/` then commit.
**VM 100 — HA** — native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) pushing `/config/` (exclude `secrets.yaml`, `.storage/`) to `fkrebs/ha-config`. HA daily tars already covered by `services-backup-plan` via `/mnt/pve/unas/backup/home-assistant/`.
Cron on each host: `0 3 * * * /usr/local/sbin/<host>-conf-backup.sh`
### Why Wasabi was rejected
Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every `restic forget --prune` generates surprise costs. Well-documented Restic-on-Wasabi trap.
### Why lobe-postgres stays on CT 104
Migration `0093_add_bm25_indexes_with_icu.sql` creates `USING bm25` indexes on 7 tables (`agents`, `topics`, `files`, `knowledge_bases`, `user_memories`, `chat_groups`, `user_memories_contexts`). `bm25` is paradedb-only (`pg_search` extension); stock postgres 17 has no such index access method and the migration fails. Container stays `paradedb/paradedb:latest-pg17`. WAL-G archives to `lobe-pg-backup`; Backrest secondary copies the data dir.
### UNAS data inventory (basis for Phase 1a sizing)
| Path | Size | Notes |
|---|---|---|
| `media/documents/` | 9.5 G | Personal documents |
| `media/video-projects/` | 425 G | Creative work, irreplaceable |
| `media/musical-sheets/` | 23 G | |
| `media/audiobooks/` | 23 G | |
| `media/ebooks/` | 6.8 G | |
| `media/3d-prints/` | 65 M | |
| `media/Recipes/` | 113 M | |
| `_sortMe/` | 335 G | `images/` 171 G, `work Flo/` 147 G, `Anne/` 17 G |
| `code/` | 6.7 M | |
| Excluded `media/movies/` | 118 G | re-streamable |
| Excluded `media/emulation/` | 57 G | re-downloadable |
| Excluded `media/Torrents/` | 19 G | temporary |
| Excluded `media/music/`, `media/podcasts/` | — | re-streamable |
services/databases.md
+97
View File
@@ -0,0 +1,97 @@
# Databases
All application databases live on **CT 113** (`192.168.1.6:5432`) after Phase 2 migration. The LXC runs `postgres:17` in Docker at `/opt/stacks/db/`, tracked in Gitea `fkrebs/stacks-db`. WAL-G archives to Garage S3 bucket `ct113-pg-backup` on CT 104 (`http://192.168.1.40:10004`).
---
## Database inventory
| DB | Owner user | Size (pre-migration) | Service | Stack location |
|----|-----------|---------------------|---------|---------------|
| `vaultwarden` | vaultwarden | 11 MB | Vaultwarden | CT 104 `/opt/stacks/vaultwarden/` |
| `paperless` | paperless | 20 MB | Paperless-ngx | CT 104 `/opt/stacks/apps/paperless-ngx/` |
| `litellm` | litellm | 212 MB | LiteLLM | CT 104 `/opt/stacks/ai/` |
| `memos` | memos | 9 MB | Memos | CT 104 `/opt/stacks/memos/` |
| `n8n` | n8n | 12 MB | n8n | CT 104 `/opt/stacks/n8n/` |
| `gitea` | gitea | 15 MB | Gitea | CT 111 `/opt/stacks/gitea/` |
| `coder` | coder | 17 MB | Coder | CT 111 `/opt/stacks/coder/` |
**Not on CT 113:**
| DB | Container | Reason |
|----|-----------|--------|
| `lobechat` | `lobe-postgres` (paradedb) on CT 104 | Requires `pg_search` extension for `USING bm25` indexes |
| `immich` | `immich_postgres` on CT 104 | Version-pinned by Immich AIO |
| `nextcloud` | Nextcloud AIO on CT 105 | AIO manages its own postgres |
---
## Connection strings (post-migration target)
| Service | Connection string |
|---------|------------------|
| Vaultwarden | `postgresql://vaultwarden:<pw>@192.168.1.6:5432/vaultwarden` |
| Paperless | `PAPERLESS_DBHOST: 192.168.1.6` |
| LiteLLM | `postgresql://litellm:<pw>@192.168.1.6:5432/litellm` (in `ai/.env` and `litellm-config/config.yaml`) |
| Memos | `postgresql://memos:<pw>@192.168.1.6:5432/memos?sslmode=disable` |
| n8n | `DB_POSTGRESDB_HOST=192.168.1.6` |
| Gitea | `GITEA__database__HOST: 192.168.1.6:5432` |
| Coder | `postgresql://coder:<pw>@192.168.1.6:5432/coder?sslmode=disable` |
Passwords are in each service's `.env` file (never committed to git). See init script at `/opt/stacks/shared-db/init/01-create-users-dbs.sql` on CT 104 for the original credential set.
---
## Migration procedure (one DB at a time)
Order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder
```bash
# 1. Stop the service
# docker compose -f <compose> stop <service>
# 2. Dump from source (via PVE host)
# For CT 104 services:
pct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres <db> \
> /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql
# For CT 111 services:
pct exec 111 -- docker exec -i <container> pg_dump -U <user> <db> \
> /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql
# 3. Create user + DB on CT 113
pct exec 113 -- docker exec -i postgres psql -U postgres <<EOF
CREATE USER <user> WITH PASSWORD '<pw>';
CREATE DATABASE <db> OWNER <user>;
EOF
# 4. Restore on CT 113
pct exec 113 -- bash -c "docker exec -i postgres psql -U postgres -d <db>" \
< /mnt/pve/unas/dump/<db>-migration-*.sql
# 5. Update service connection string (shared-postgres → 192.168.1.6)
# Edit compose or .env
# 6. Start service; verify logs and function
# 7. Verify, then old DB/container can be removed
```
---
## Post-migration cleanup
After all 7 DBs are migrated and verified:
1. Drop stale DBs from `shared-postgres`: `daytona`, `lobechat` (duplicate — real one is in `lobe-postgres`), `paradedb`
2. Stop and remove `shared-postgres` container + named volume `shared-pgdata`
3. Stop and remove `gitea-db` and `coder-db` containers + volumes on CT 111
4. Update Backrest services plan: remove `shared-db` path, update to CT 113 WAL-G output
5. Enable **PVE protection** on CT 113 (prevents accidental delete)
---
## pgAdmin
pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres.
Credentials in `/opt/stacks/db/.env` (admin@nucli.de).
services/homelab-architecture.md
+1 -1
View File
@@ -25,7 +25,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
- `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
- `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20
- `109` lxc **observe** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md` - `109` lxc **observe** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md`
- `113` lxc **db** *(planned)* — shared Postgres + pgbouncer once second NVMe lands - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21
- Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA
services/mcp-gateway.md
+102 -94
View File
@@ -1,120 +1,128 @@
# MCP Gateway v2 — Operational # MCP Gateway v2 — Operational
Finished implementation of the DinD MCP gateway (Option A). Supersedes the DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP servers (mix of static, spawned, and Docker-catalog-bridged). Public via Zoraxy at `https://mcp.nuclide.systems` (→ `192.168.1.40:8080`).
"Phase 1" stub described in `mcp-gateway-requirements.md`.
## What it is ## Architecture
`ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`, public via Zoraxy at - **Stack**: `ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`.
`https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). Single OAuth-gated - **Spawn model**: Docker-in-Docker. Each spawnable server runs as `mcp-<name>` on the `ai-internal` network with no host ports — servers talk over internal DNS.
entrypoint fronting several MCP servers. - **Transport bridge**: generic streaming reverse proxy `https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`. Transport-agnostic (streamable-HTTP `/mcp` and legacy SSE `/sse` + `/messages`). Registered last so specific routes win.
## Architecture (as built)
- **Spawn model**: Docker-in-Docker. Each spawnable server runs as
`mcp-<name>` on the **`ai-internal`** network with **no host ports**
(fixes the old `8000:8000` collision — servers talk over internal DNS).
- **Transport bridge**: generic streaming reverse proxy
`https://mcp.nuclide.systems/<server>/<path>` → `http://<upstream>/<path>`.
Transport-agnostic — works for streamable-HTTP (`/mcp`) and legacy SSE
(`/sse`+`/messages`). Registered LAST so specific routes win.
- **Auth**: Pocket ID. Two paths off the same OIDC app: - **Auth**: Pocket ID. Two paths off the same OIDC app:
- **MCP clients / API / proxy** — `Bearer` token, validated via OIDC - **MCP clients / API / proxy** — `Bearer` token, validated via OIDC userinfo (300 s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728 `/.well-known/oauth-protected-resource` for discovery.
userinfo (300s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728 - **Browser Web UI** — Authorization-Code flow with httpOnly session cookie. `/ui` and `/api/*` accept cookie or Bearer; `/{srv}/` stays Bearer-only.
`/.well-known/oauth-protected-resource` for discovery. - **Secrets**: per-server OAuth client secrets live in `ai/.env` (read via `os.environ`), never hardcoded.
- **Browser Web UI** — full Authorization-Code flow with an httpOnly
session cookie: unauthenticated `/ui` → `302 /login` → Pocket ID
`authorize` (CSRF `state` cookie) → `/sso/callback` (state-checked
code→token exchange) sets the `mcp_session` cookie and shows the page +
a copyable Bearer token for MCP clients. `/logout` clears it. The UI and
`/api/*` accept **either** the cookie or a Bearer header; the `/{srv}/`
proxy stays Bearer-only. `/health` + the well-known metadata are open.
- **Secrets**: all per-server OAuth client secrets come from `ai/.env`
(`env_file: ../.env`), read via `os.environ` — none hardcoded in source.
## Server registry (`SERVERS` in server.py) ## Server inventory (as of 2026-05-22)
| name | kind | upstream | notes | 26 servers configured in `mcp-gateway/config.json`. **Status: 25 healthy, 1 disabled.**
| Group | Server | Kind | Notes |
|---|---|---|---| |---|---|---|---|
| comfyui | static | `http://comfyui-mcp:8000` | runs as its own stack (`ai/comfyui-mcp.yml`); gateway only proxies | | **dev** | context7 | catalog | `mcp/context7` |
| nextcloud | spawn | `mcp-nextcloud:8000` | `ghcr.io/cbcoutinho/nextcloud-mcp-server` streamable-http | | | fetch | spawn | `mcp-server-fetch` via uvx |
| crawl4ai | spawn | `mcp-crawl4ai:11235` | `unclecode/crawl4ai` | | | git | catalog | `mcp/git` — Docker MCP Catalog; `mcp-git` bridge + DinD child |
| mermaid | spawn | `mcp-mermaid:8000` | `node:20-slim` + npx mcp-mermaid (validated working) | | | gitlab | catalog | `mcp/gitlab` — `GITLAB_API_URL=https://gitlab.dlr.de/api/v4`; requires `--pass-environment` in bridge cmd |
| markitdown | spawn | `mcp-markitdown:8000` | `ghcr.io/astral-sh/uv` + uvx markitdown-mcp | | | kroki | static | `kroki-mcp:8000` (own stack) |
| papersearch | spawn | `mcp-papersearch:8000` | python + uv + mcp-proxy → paper_search_mcp | | | markitdown | catalog | `mcp/markitdown` |
| | sequential-thinking | catalog | `mcp/sequentialthinking` |
| | time | spawn | `mcp-server-time` |
| | docling | static | `docling-mcp:8000` |
| | coder | static | `coder-mcp:8000` |
| | shepard | static | `https://shepard.nuclide.systems/v2/mcp` |
| **research** | crawl4ai | spawn | `unclecode/crawl4ai` |
| | paper-search | catalog | `mcp/paper-search` — Docker MCP Catalog; replaces old `python:3.12-slim` inline approach |
| | searxng | spawn | `isokoliuk/mcp-searxng` |
| | wikipedia-mcp | catalog | |
| | youtube-transcript | catalog | `mcp/youtube-transcript` |
| **personal** | bluesky | static | `ariel-mcp:8000` |
| | gotify | spawn | |
| | home-assistant | static | `http://192.168.1.60:9583/private_...` (HA add-on) |
| | immich | spawn | |
| | memos | spawn | uses `MEMOS_TOKEN` from `.env` |
| | n8n | static | `https://n8n.nuclide.systems/mcp-server/http` |
| | nextcloud | spawn | ❌ **disabled** — SSL CERTIFICATE_VERIFY_FAILED contacting `https://nc.nuclide.systems` (self-signed). `"enabled": false` set 2026-05-21. |
| | unifi | spawn | |
| **image** | comfyui | static | `comfyui-mcp:8000` (own stack, see [comfyui.md](comfyui.md)) |
| **storage** | upload-artifact | static | `upload-artifact-mcp:8000` |
## Verified > **Catalog bridge env var passthrough**: `mcp-proxy`'s `stdio_client` uses a sanitized env (`get_default_environment()`), stripping custom vars. For catalog servers that need env vars (e.g. `gitlab`), the bridge command **must** include `--pass-environment` so Docker CLI can resolve `-e KEY` passthrough from the bridge container's env.
- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth). > **Re-enabling nextcloud**: needs either (a) a CA bundle in the container that trusts the Zoraxy/internal cert chain, or (b) skip-verify env var passed to the upstream client, or (c) point `NEXTCLOUD_HOST` at the internal Nextcloud LAN URL on `ai-internal`. Tracked as an open issue.
- `/mcp.json`, `/<server>/mcp` → 401 without token; 401 with bad token
(userinfo path); correct `WWW-Authenticate`.
- Upstream path proven: gateway → `comfyui-mcp` `/mcp` initialize → 200;
gateway → spawned `mcp-mermaid` `/mcp` → 200 (spawn pattern + proxy).
- Route precedence correct (specific routes beat the catch-all proxy).
## Not yet exercised (needs a real token / operational bring-up)
- **Token-authenticated end-to-end** requires an interactive Pocket ID login.
Get a token: open `https://mcp.nuclide.systems/sso/callback` via the OAuth
authorize flow (or any Pocket ID token with scopes `openid,mcp`); the
callback page prints the access token.
- **Bring up the 4 remaining spawnable servers** (nextcloud/crawl4ai/
markitdown/papersearch): `POST /api/servers/<name>/start` (bearer) or
`POST /api/servers/bulk/start`. Each may need a transport/flag tweak on
first run — mermaid is the proven reference.
## Client registration ## Client registration
- **LobeChat**: Settings → Skills → Skill Store → Custom → Import JSON, e.g. ### Easy path — `/mcp-config` download
```json
{ "mcpServers": { "comfyui": { "type": "http",
"url": "https://mcp.nuclide.systems/comfyui/mcp",
"auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
```
(Or, for ComfyUI specifically, connect LobeChat straight to
`http://comfyui-mcp:8000/mcp` internally — see `comfyui-mcp.md` — no token.)
- **Claude.ai**: add `https://mcp.nuclide.systems/<server>/mcp` as a custom
connector; it will discover auth via the protected-resource metadata.
## Ops The gateway hands back a ready-to-use config blob for the signed-in user:
- `cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build`
- Logs: `docker logs mcp-gateway`; UI: `https://mcp.nuclide.systems/ui`
(Pocket-ID-gated; `/login` to sign in).
- Config overlay persisted in `ai/mcp-gateway/config.json`.
- **UI check (Playwright)** — the live UI is auth-gated, so the check
serves the local template + mocks `/api/servers`, screenshots, and
asserts cards render:
```
uv run --with playwright -- python -m playwright install chromium # once
uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png
```
## Client config download — `/mcp-config` (added 2026-05-21)
Instead of asking the user to paste a bearer token into per-server entries
in `~/.claude/settings.json`, the gateway now hands back a ready-to-use
config blob for the signed-in user:
``` ```
GET https://mcp.nuclide.systems/mcp-config?format=claude # default GET https://mcp.nuclide.systems/mcp-config?format=claude # default
GET https://mcp.nuclide.systems/mcp-config?format=cursor GET https://mcp.nuclide.systems/mcp-config?format=cursor
GET https://mcp.nuclide.systems/mcp-config?format=raw # token + server URLs only GET https://mcp.nuclide.systems/mcp-config?format=raw # token + URLs
``` ```
- Requires the user's Pocket-ID session cookie OR an existing Bearer token (303 → `/login` if neither). - Requires Pocket-ID session cookie or existing Bearer (303 → `/login` otherwise).
- Mints a fresh long-lived gateway token (`mcp_<urlsafe>`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:<format>` so it's distinguishable from OAuth-issued tokens (and revocable). - Mints a fresh long-lived gateway token (`mcp_<urlsafe>`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:<format>` so it's distinguishable from OAuth-issued tokens and revocable.
- Returns a JSON file with `Content-Disposition: attachment` so the browser saves it. - Returns JSON with `Content-Disposition: attachment`.
### Claude Code workflow **Claude Code workflow:**
1. Sign in at `https://mcp.nuclide.systems`. 1. Sign in at `https://mcp.nuclide.systems`.
2. Click **Claude config** in the top nav. Browser downloads `nuclide-mcp.claude.json`. 2. Click **Claude config** in the top nav → browser downloads `nuclide-mcp.claude.json`.
3. Merge the `mcpServers` block into `~/.claude/settings.json`. Restart Claude Code. 3. Merge the `mcpServers` block into `~/.claude/settings.json`. Restart Claude Code.
The downloaded file has 23 entries, one per spawnable MCP server, with `type: http` + `url: https://mcp.nuclide.systems/<server>/mcp` + an `Authorization: Bearer ...` header. No further per-server configuration needed. The downloaded file has one entry per enabled MCP server, with `type: http` + `url: https://mcp.nuclide.systems/<server>/mcp` + `Authorization: Bearer ...` header. No per-server config needed.
### Rotation ### Manual
Each download mints a NEW token. Old tokens remain valid until you remove them from `gateway_tokens.json` (manual prune; future: admin UI button).
### Server list reflected in the config **LobeChat** — Settings → Skills → Skill Store → Custom → Import JSON:
The gateway populates from `mcp-gateway/config.json` `mcpServers` keys — exactly the set running at the time of download. New servers added later require re-downloading the config.
```json
{ "mcpServers": { "comfyui": { "type": "http",
"url": "https://mcp.nuclide.systems/comfyui/mcp",
"auth": { "type": "bearer", "accessToken": "<pocket-id token>" } } } }
```
(For ComfyUI specifically, LobeChat can also connect straight to `http://comfyui-mcp:8000/mcp` internally — see [comfyui.md](comfyui.md) — no token.)
**Claude.ai** — add `https://mcp.nuclide.systems/<server>/mcp` as a custom connector; auth is discovered via the protected-resource metadata.
### Token rotation
Each `/mcp-config` download mints a NEW token. Old tokens stay valid until removed from `gateway_tokens.json` (manual prune; future: admin UI button).
## Ops
```bash
cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build
docker logs mcp-gateway
```
UI: `https://mcp.nuclide.systems/ui` (Pocket-ID-gated; `/login` to sign in).
Config: `mcp-gateway/config.json` (live config, persisted). Timestamped backups created on every edit (`config.json.bak.YYYYMMDD-HHMMSS`).
### Add or enable a server
1. Edit `mcp-gateway/config.json` — add an `mcpServers.<name>` entry (or set `enabled: true`).
2. Restart the gateway: `docker restart mcp-gateway`.
3. Spawn servers come up on demand via `POST /api/servers/<name>/start` (bearer); or `POST /api/servers/bulk/start` for all.
### Disable a server
Set `mcpServers.<name>.enabled = false` and add `disabled_reason`. Stop+rm the container (`docker stop mcp-<name> && docker rm mcp-<name>`); the gateway will not re-spawn it.
### UI check (Playwright)
Live UI is auth-gated, so the local check serves the template + mocks `/api/servers`:
```bash
uv run --with playwright -- python -m playwright install chromium # once
uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png
```
## Verified end-to-end
- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth).
- `/mcp.json`, `/<server>/mcp` → 401 without token; correct `WWW-Authenticate`.
- Static upstreams (`bluesky`, `coder`, `comfyui`, `docling`, `kroki`, `upload-artifact`) — endpoints reachable from inside `ai-internal` (406/405 to plain GET, which is the correct MCP behavior).
- All spawn/catalog containers up (`docker ps mcp-*`): 16/17 (nextcloud disabled; git/gitlab/paper-search added 2026-05-22).