diff --git a/CHANGELOG.md b/CHANGELOG.md index 259f18f..a3355ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ All notable infrastructure / service / doc changes. Newest first. +## 2026-05-22 + +- **MCP gateway**: added `git`, `gitlab` (DLR, `--pass-environment` fix for mcp-proxy env passthrough bug), `paper-search` (Docker catalog image replaces inline `python:3.12-slim`); `UNPAYWALL_EMAIL` wired into paper-search. Removed duplicate `papersearch` container. Updated `services/mcp-gateway.md`. +- **CT 109 plan updated**: IP conflict noted (`.6` taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113. +- **Portmap**: CT 113 (`db`, `.6`) section added; QNAP TS-251D (Klipper/Mainsail) section added. +- **New docs**: `services/backrest.md`, `services/databases.md`, `services/arcane.md`. + ## 2026-05-21 ### Docs & docs infrastructure diff --git a/ct-inventory.md b/ct-inventory.md index fd6142c..9f53cd7 100644 --- a/ct-inventory.md +++ b/ct-inventory.md @@ -11,7 +11,7 @@ Verified 2026-05-20 via `pct list`, `pct config `, `qm config 100` on `nuc`. | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | | 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 **CIFS** (`//192.168.1.31/storage`) | Intel iGPU (render only) | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | -| 109 | observe | LXC | 192.168.1.6/24 | 4 | 6144 | 30 GiB | **Planned** — Prometheus, Grafana, Arcane Manager, Dozzle, Homarr, Diun, Tinyauth, socket-proxy | — | no | **planned** (see `infra/proxmox-state.md` §16) | +| 109 | observe | LXC | ⚠️ TBD (`.6` taken by CT 113) | 4 | 8192 | 50 GiB | **Planned** — Prometheus, Loki, Grafana, Arcane Manager, Dozzle, Homarr, Diun, sshwifty, Tinyauth, socket-proxy; Alloy agents on all hosts | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 112 | secrets | LXC unpriv | TBD | 2 | 2048 | 8 GiB | **Planned** — Infisical secrets manager (replaces scattered .env files) | — | no | **planned** | diff --git a/infra/portmap.md b/infra/portmap.md index e2e39b3..c8ebf82 100644 --- a/infra/portmap.md +++ b/infra/portmap.md @@ -83,7 +83,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway | | [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway | | [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only | -| ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — Shepard will provide a native MCP server | +| ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — replaced by native `https://shepard.nuclide.systems/v2/mcp` (streamable HTTP; gateway entry: `shepard`, auth via `${SHEPARD_API_KEY}`) | | [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | @@ -136,7 +136,18 @@ _Traccar moved to 15000–15001 (Documents range). 19000–19001 now free._ | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| 20010 (127.0.0.1 only) | pgAdmin | `pgadmin` | — | localhost only | +| 20010 (127.0.0.1 only) | pgAdmin | `pgadmin` | — | shared-db pgAdmin, localhost only (CT 104) | + +--- + +## LXC 113 — db (`192.168.1.6`) + +CT 113 is the dedicated postgres LXC. No public proxy routes — LAN access only. + +| Port | Service | Container | Notes | +|---|---|---|---| +| 5432 | Postgres 17 | `postgres` | LAN: `192.168.1.6:5432` — accepts app connections from all CTs | +| 5050 | pgAdmin 4 | `pgadmin` | LAN only: `http://192.168.1.6:5050` — no Zoraxy route | --- @@ -157,6 +168,19 @@ CT 111 hosts the self-hosted dev platform. Same Pocket-ID SSO as the rest. --- +## QNAP TS-251D (`192.168.1.189`) + +Celeron J4025, 2-core. Hosts Klipper natively (not Docker). + +| Port | Service | Notes | +|---|---|---| +| 80 | Mainsail | 3D printer web UI | +| 7125 | Moonraker | Klipper API | + +Config backed up daily to `git.nuclide.systems/fkrebs/klipper-config` via cron at 03:00 → covered offsite by Backrest `services/gitea` path. + +--- + ## External Services (Not on NUC Docker) Zoraxy routes to these external backends: diff --git a/infra/proxmox-state.md b/infra/proxmox-state.md index 1802c6d..aeb8d9f 100644 --- a/infra/proxmox-state.md +++ b/infra/proxmox-state.md @@ -712,30 +712,41 @@ Recovery latency improved from 5 min → 1 min; logging structured in `journalct ## 16. CT 109 "observe" — planned observability + ops LXC -Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores / 6 GiB RAM / 30 GiB rootfs**, unprivileged, nesting=1, IP `192.168.1.6` (next free in the infra range after CT 110). +Single LXC holding everything monitoring/ops-shaped. Sizing target: **4 cores / 8 GiB RAM / 50 GiB rootfs**, unprivileged, nesting=1. RAM bumped from 6 → 8 GiB to accommodate Loki. Disk bumped from 30 → 50 GiB for Loki log retention (30d) alongside Prometheus TSDB. + +⚠️ **IP conflict:** inventory originally assigned `192.168.1.6` but CT 113 (db) is already live at `.6`. CT 109 needs the next free infra IP — likely `.7` or `.8` (verify against UniFi DHCP table before provisioning). + +**Access model (initial):** LAN-only. No Zoraxy routes until Tinyauth is deployed. Services reachable directly by IP. ### Stack to deploy on CT 109 | Service | Purpose | |---|---| -| Arcane Manager | central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out) | -| Dozzle UI | aggregated container logs; Dozzle agents on CT 101 + CT 104 + CT 110 | -| Prometheus | metrics TSDB, 30 d retention | -| Grafana | dashboards over Prometheus | +| Prometheus | metrics TSDB, 30d retention | +| Loki | log aggregation backend — receives from Alloy agents on all hosts | +| Grafana | dashboards over Prometheus + Loki (unified metrics + log search) | | Alertmanager + alertmanager-gotify-bridge | alert routing → Gotify | -| **Homarr** (selected over Homepage) | unified dashboard, native OIDC with Pocket-ID, native Prometheus widget + Grafana iframe support | +| Arcane Manager | central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out) | +| Dozzle UI | live log tail (quick debugging); agents on CT 101 + CT 104 + CT 110. Complements Loki — Dozzle for live, Loki for historical/search | +| **Homarr** | unified dashboard, native Pocket-ID OIDC, Prometheus widget + Grafana iframe support | | Diun | docker image update notifier → Gotify | -| Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana). Speaks OIDC client to Pocket-ID, returns Basic-Auth/Remote-User header to upstream | -| docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr discovery | +| Tinyauth | forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID | +| docker-socket-proxy | local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr/Arcane discovery | +| **sshwifty** | web SSH client, multi-tab — multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred. | -### Sidecars deployed on each Docker host +### Sidecars deployed on each host + +Two agents per host — **node-exporter** (metrics) and **Alloy** (logs). Kept separate: node-exporter metric names are assumed by every Prometheus dashboard/alert; Alloy emitting compatible metrics adds validation risk for no gain. | Host | Sidecars | |---|---| -| Host (PVE) | node-exporter, smartctl-exporter, pve-exporter (host metrics into Prometheus) | -| CT 101 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy | -| CT 104 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy | -| CT 110 | node-exporter, Dozzle agent, Arcane edge agent (test target — small/clean) | +| PVE host | node-exporter, smartctl-exporter, pve-exporter, **Alloy** (journald → Loki: pve-manager, pveproxy, pvedaemon, LXC/VM lifecycle) | +| CT 101 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, **Alloy** (Docker logs + journald → Loki) | +| CT 103 | node-exporter, **Alloy** (backrest.service journal + `/var/log/rclone-*.log` → Loki) | +| CT 104 | node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, intel_gpu_exporter, **Alloy** (Docker logs + journald → Loki) | +| CT 110 | node-exporter, Dozzle agent, Arcane edge agent, **Alloy** (journald → Loki) | +| CT 111 | node-exporter, intel_gpu_exporter, **Alloy** (journald + Coder/Gitea logs → Loki) | +| CT 113 | node-exporter, postgres_exporter, **Alloy** (journald + postgres logs → Loki) | ### Services that stay where they are (NOT on CT 109) @@ -774,11 +785,12 @@ Build-order slot: after Arcane + socket-proxy land in §16's checklist, before T 4. Issue agent tokens, deploy Edge agents on CT 101/104/110 5. Deploy Dozzle Manager + agents 6. Deploy Diun -7. Deploy Tinyauth (configure Pocket-ID client first) -8. Deploy Homarr -9. Front the lot via Zoraxy: `arcane.`, `dozzle.`, `grafana.`, `prom.`, `home.` `.nuclide.systems` -10. Verify each through the Tinyauth gate where applicable -11. Add Diun watchlist + Alertmanager routing → Gotify +7. Deploy sshwifty (configure host list + SSH keys for PVE, CT 103, CT 104, CT 111; LAN-only port 8182) +8. Deploy Tinyauth (configure Pocket-ID client first) +9. Deploy Homarr +10. Front the lot via Zoraxy: `arcane.`, `dozzle.`, `grafana.`, `prom.`, `home.`, `shell.` `.nuclide.systems` +11. Verify each through the Tinyauth gate where applicable +12. Add Diun watchlist + Alertmanager routing → Gotify --- diff --git a/services/arcane.md b/services/arcane.md new file mode 100644 index 0000000..d1e4d25 --- /dev/null +++ b/services/arcane.md @@ -0,0 +1,126 @@ +# Arcane + +Web-based Docker management IDE. Main instance on **CT 104** (`192.168.1.40:10002`, `arcane.nuclide.systems`). Manages containers on the local Docker host plus remote CTs via **edge agents**. + +Stack: `/opt/stacks/arcane/docker-compose.yml` on CT 104. +Image: `ghcr.io/getarcaneapp/arcane:latest` +Auth: OIDC via Pocket ID, admin: `fkrebs@nucli.de` + +--- + +## Edge agents + +An edge agent (`ghcr.io/getarcaneapp/arcane-headless:latest`) runs inside each remote CT and connects **outbound** to the main Arcane server via gRPC poll. The main server then manages that CT's Docker. + +### Currently deployed agents + +| CT | Hostname | Compose path | Status | +|----|----------|--------------|--------| +| CT 104 | docker | *(built-in NUC environment)* | online | +| CT 113 | db | `/opt/stacks/db/docker-compose.yml` | online (2026-05-21) | + +### Adding an edge agent to a new CT + +**Step 1 — Create the environment in Arcane** (requires Arcane API or UI access) + +With the admin CLI API key (stored in Arcane DB, regenerate if needed): +```bash +# Get or create an admin API key — see "Admin API key" section below +API_KEY="arc_..." +curl -s -X POST -H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \ + "http://192.168.1.40:10002/api/environments" \ + -d '{"name":"","apiUrl":"edge://","isEdge":true}' +# Note the environment id from the response +``` + +**Step 2 — Generate and wire the AGENT_TOKEN** + +Arcane stores the raw AGENT_TOKEN in `environments.access_token`. Insert it via: +```python +# On PVE host, run: python3 /tmp/arcane-bootstrap.py +import argon2, secrets, sqlite3, uuid +from datetime import datetime, timezone + +db_path = "/rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db" +env_id = "" # paste the environment UUID here + +conn = sqlite3.connect(db_path) +raw_token = "arc_" + secrets.token_hex(32) +key_prefix = "arc_" + raw_token[4:12] +ph = argon2.PasswordHasher(memory_cost=65536, time_cost=3, parallelism=2) +key_hash = ph.hash(raw_token) +key_id = str(uuid.uuid4()) +now = datetime.now(timezone.utc).isoformat() + +conn.execute( + "INSERT INTO api_keys (id, name, description, key_hash, key_prefix, environment_id, managed_by, created_at, updated_at)" + " VALUES (?,?,?,?,?,?,?,?,?)", + (key_id, f"Environment Bootstrap Key - {env_id[:8]}", + "Auto-generated key for environment pairing", + key_hash, key_prefix, env_id, "system", now, now)) +conn.execute("UPDATE environments SET access_token=? WHERE id=?", (raw_token, env_id)) +conn.commit() +conn.close() +print("AGENT_TOKEN:", raw_token) +``` + +**Step 3 — Add the agent to the CT's compose** + +```yaml + arcane-agent: + image: ghcr.io/getarcaneapp/arcane-headless:latest + container_name: arcane-agent + restart: unless-stopped + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./arcane-agent:/app/data + environment: + EDGE_AGENT: "true" + EDGE_TRANSPORT: poll + AGENT_TOKEN: ${ARCANE_AGENT_TOKEN} + MANAGER_API_URL: http://192.168.1.40:10002 + deploy: + resources: + limits: + cpus: "0.5" + memory: 256M +``` + +Add `ARCANE_AGENT_TOKEN=` to the CT's `.env`. + +**Step 4 — Start and verify** + +```bash +docker compose up -d arcane-agent +docker logs arcane-agent 2>&1 | grep "Edge gRPC tunnel" +# Expected: Edge gRPC tunnel connected to manager environment_id= +``` + +The environment should flip to `online` in `environments.status` within ~5 seconds. + +--- + +## When Arcane moves to CT 109 + +When Arcane migrates from CT 104 to CT 109 (the planned "observe" LXC): + +1. Update `MANAGER_API_URL` in every edge agent's `.env` to the new CT 109 LAN IP +2. Update the Arcane compose `APP_URL` and network bindings +3. Update Zoraxy route for `arcane.nuclide.systems` to point to CT 109 +4. Migrate the SQLite DB: `cp -a /opt/stacks/arcane/data/ /` +5. Regenerate the admin API key (stored in memory, not config) since the DB port changes +6. Restart all edge agents with the new `MANAGER_API_URL` + +Arcane environments (NUC, db, shepard) will reconnect automatically once `MANAGER_API_URL` is updated. + +--- + +## Admin API key (non-OIDC access) + +Arcane doesn't store plaintext API keys — use the DB-insert method above when a new admin key is needed. The `python3-argon2` package must be installed on the PVE host (`apt install python3-argon2`). + +The key inserted for CLI use during CT 113 provisioning (`arc_a9695182...`) is linked to `fkrebs` user in the api_keys table. Rotate it after provisioning work is complete by deleting the row: +```sql +sqlite3 /rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db \ + "DELETE FROM api_keys WHERE name='admin-cli';" +``` diff --git a/services/backrest.md b/services/backrest.md new file mode 100644 index 0000000..3bf0c72 --- /dev/null +++ b/services/backrest.md @@ -0,0 +1,303 @@ +# Backup Strategy + +> Single Backrest instance on **CT 103** (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at `/mnt/pve/unas`, so it can read all service data without SSH-ing other hosts. + +**Goal**: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud. + +--- + +## Current state + +### Backrest (CT 103) — Phase 1b live since 2026-05-21 + +| Item | Value | +|---|---| +| UI | `http://192.168.1.3:9898` (LAN-only, auth disabled) | +| Config | `/opt/backrest/config/config.json` (timestamped `.bak` files on every edit) | +| rclone remote | `jottacloud:` Archive section (default device/mountpoint); token at `/root/.config/rclone/rclone.conf` | +| Offsite plan | JottaCloud Unlimited €9.91/mo (Norway, EEA) — see [provider comparison](#provider-comparison) | +| BW throttle | `RCLONE_BWLIMIT=05:30,7.5M 01:00,20M` (20 MB/s overnight, 7.5 MB/s daytime) | + +**Repos** (both `autoInitialize: true`; passwords currently `tapirnase` — see blind spot #5): + +| Repo | Target | Prune | Check | +|---|---|---|---| +| `services-repo` | `rclone:jottacloud:services` | weekly Sun 05:00, ≤10 % unused | monthly, 10 % subset | +| `media-repo` | `rclone:jottacloud:media` | monthly, ≤10 % unused | monthly, 10 % subset | + +**Plans**: + +| Plan | Repo | Schedule | Retention | Paths | +|---|---|---|---|---| +| `services-backup-plan` | `services-repo` | `0 1 * * 1-5` weekdays 01:00 | 7d · 4w · 6m · 1y | 15 paths (below) | +| `media-backup-plan` | `media-repo` | `30 1 * * 1-5` weekdays 01:30 | 4w · 6m · 1y | `/mnt/pve/unas/media/images/library` | +| `video-projects-plan` | `media-repo` | nightly 02:00 | per config | `/mnt/pve/unas/media/video-projects` | + +**Services plan paths** (all under `/mnt/pve/unas/`): + +``` +services/vaultwarden services/n8n services/memos +services/karakeep services/traccar services/gitea +services/coder services/nextcloud services/pocketid +services/arcane services/arr-stack services/gluetun +backup/home-assistant backup/immich backup/nextcloud +``` + +`services/shared-db` was removed when CT 113 came online — postgres is now WAL-G → Garage S3 → JottaCloud. + +> **JottaCloud web UI tip**: rclone writes to the **Archive** section. The default landing page shows only Sync + Backup. Browse to `https://www.jottacloud.com/web/archive` to see the restic repos. + +### Postgres (CT 113) — Phase 2 live since 2026-05-21 + +Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmin on `:5050`. WAL-G archives continuously to Garage S3 (`ct113-pg-backup` on CT 104): + +- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` +- Daily `wal-g backup-push` cron at 02:00 +- DBs hosted: `vaultwarden`, `paperless`, `litellm`, `memos`, `n8n`, `gitea`, `coder` + +Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sbin/walg-offsite-sync.sh` (read-only Garage key `GKef577420aadd26d667f2ca4f`; mirrors `ct113-pg-backup`, `lobe-pg-backup`, `immich-pg-backup` → `jottacloud:WAL-G/`). + +**Exceptions** (stay on original hosts): + +| DB | Host | Why | +|---|---|---| +| `lobe-postgres` (paradedb pg17) | CT 104 | Uses `pg_search` `USING bm25` indexes; stock postgres 17 can't host. WAL-G → `lobe-pg-backup` + Backrest secondary on data dir. | +| Nextcloud AIO postgres | CT 105 | AIO manages it; Borg archives the whole stack to UNAS. | +| Immich postgres | CT 104 | Version-pinned by Immich. WAL-G → `immich-pg-backup`. | + +--- + +## Coverage map + +| Host / data | Method | Offsite | +|---|---|---| +| CT 103 Backrest binary + config | Manual (small) | On-CT only | +| CT 104 Docker app state on UNAS | `services-backup-plan` | ✓ JottaCloud | +| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ | +| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ | +| CT 113 postgres (7 DBs) | WAL-G → `ct113-pg-backup` → sync | ✓ | +| CT 101 Shepard | Source in Gitea (gitea path covers it) | ✓ | +| CT 102 AdGuard | Phase 3 git push → `fkrebs/adguard-conf` | ✗ not deployed | +| CT 105 Nextcloud user files | `services/nextcloud` | ✓ | +| CT 105 Nextcloud AIO volumes | AIO Borg → `/mnt/pve/unas/backup/nextcloud/` → Backrest | ✓ | +| CT 108 Zoraxy | Phase 3 git push → `fkrebs/zoraxy-conf` | ✗ not deployed | +| CT 110 Pocket-ID | `services/pocketid` | ✓ | +| CT 111 Gitea + Coder | `services/{gitea,coder}` | ✓ | +| VM 100 HAOS config | Phase 3 git addon → `fkrebs/ha-config` | ✗ not deployed | +| VM 100 HAOS daily tar | HA → UNAS → Backrest | ✓ | +| PVE host `/etc/pve/` | Phase 3 git push → `fkrebs/pve-conf` | ✗ not deployed | +| UNAS `media/images/library` | Phase 1a `rclone sync` → `jottacloud:Photos/` | ✗ not deployed | +| UNAS personal data (`documents`, `_sortMe`, `video-projects`, …) | Phase 1a `rclone sync` → `jottacloud:UNAS/` | ✗ not deployed | + +### Intentionally not backed up + +Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes — all regenerable or re-downloadable. + +### Deferred — include if needed + +| Host | Data | Notes | +|---|---|---| +| CT 109 observe | Prometheus TSDB, Grafana dashboards, Arcane SQLite, sshwifty config | Low priority — metrics are ephemeral; dashboards can be re-exported from Grafana; add to services plan if state becomes hard to recreate | + +--- + +## Open work + +### Phase 1a — rclone sync for UNAS personal data + +Two sync jobs on CT 103, Saturday 01:00: + +- **Job 1**: `media/images/library/` → `jottacloud:Photos/` (Immich originals, visible at `jottacloud.com/photo`). +- **Job 2**: `/mnt/pve/unas/` → `jottacloud:UNAS/` (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, `services/` already in Backrest). + +Initial upload ≈ 820 GB personal + 822 GB Immich (days). Script target `/usr/local/sbin/unas-sync.sh`. Full script in [appendix](#phase-1a-rclone-sync-script). + +Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media. + +### Phase 3 — Config-to-git for infrastructure + +Daily 03:00 cron on each host pushes config to a private Gitea repo. Repos already exist: + +- `fkrebs/zoraxy-conf` (CT 108 — `/opt/zoraxy/conf/`) +- `fkrebs/adguard-conf` (CT 102 — `AdGuardHome.yaml`) +- `fkrebs/pve-conf` (PVE — `/etc/pve/`, excludes `priv/`, `*.key`, `authkey.pub*`) +- `fkrebs/ha-config` (VM 100 — `/config/`, excludes `secrets.yaml`, `.storage/`) + +Script template in [appendix](#phase-3-config-to-git-scripts). + +### Blind spots + +| # | Severity | Issue | Fix | +|---|---|---|---| +| 4 | MEDIUM | n8n encryption key in `/opt/stacks/n8n/data/` on CT 104 local FS — not in any plan. If CT 104 dies, DB restore is unusable. | Move n8n data volume to `/mnt/pve/unas/services/n8n/` (already in plan). | +| 4b | MEDIUM | Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted — without it, restore impossible. | Store `BORG_PASSWORD` in Vaultwarden. | +| 5 | MEDIUM | Both Backrest repo passwords are `tapirnase`. | Rotate before first scheduled run completes; `restic key passwd` re-encrypts in place. Store in Vaultwarden. | +| 6 | LOW | `litellm` DB password is the placeholder literal `litellm_password_here`. | Generate real password; update `ai/.env`, `litellm-config/config.yaml`, CT 113 user. | +| 7 | LOW | Migration dumps at `/mnt/pve/unas/dump/*-migration-20260521.sql` not in any plan. | Decide: keep as manual archive or delete now that WAL-G is archiving. | + +--- + +## Operations + +### Restore + +UI: **Repos → snapshots → Browse → file → Restore**. + +CLI on CT 103: + +```bash +restic -r rclone:jottacloud:services snapshots +restic -r rclone:jottacloud:services restore latest \ + --target /restore \ + --include /mnt/pve/unas/services/vaultwarden +``` + +### Monitoring (planned) + +In Backrest UI → each repo → Hooks: + +- `CONDITION_BACKUP_ERROR` / `CONDITION_CHECK_ERROR` → POST Gotify priority 8 +- `CONDITION_BACKUP_SUCCESS` → POST Gotify priority 3 + +``` +curl -s -X POST 'http://gotify:80/message?token=TOKEN' \ + -H 'Content-Type: application/json' \ + -d '{"title":"Backrest: {{.Plan}}","message":"{{.Summary}}","priority":3}' +``` + +### Applying a new config + +```bash +# on CT 103 +systemctl stop backrest +cp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S) +# paste new config.json +systemctl start backrest +``` + +--- + +## History + +### Provider comparison + +Researched 2026-05-21. Sized for ~3 TB/mo. + +| Provider | ~3 TB/mo | Backend | EU DC | Egress | Verdict | +|---|---|---|---|---|---| +| **JottaCloud Unlimited** | €9.91 flat (unlimited) | rclone native | Norway (EEA) | Free | ✅ Chosen | +| Hetzner BX31 | €20.80 flat (10 TB) | SFTP/WebDAV | DE, FI | Free | 2.5× the price, capped | +| Backblaze B2 | ~$18 pay-per-GB | S3 | Frankfurt | Free (≤3× stored) | US CLOUD Act risk | +| Cloudflare R2 | ~$45 pay-per-GB | S3 | EU auto | Zero | Expensive at scale; no EU residency lock | +| Wasabi | ~$21–24 | S3 | FRA/AMS | Free (≤ stored) | ❌ 90-day min billing per object → Restic prune disaster | +| Storj DCS | ~$30 | S3 | EU-geofenced | 1× free | Complex, $5 minimum | +| pCloud | €399 one-time / 2 TB | WebDAV | Luxembourg | Free | WebDAV too slow | +| Infomaniak kDrive | ~€36+ / 3 TB | WebDAV | Switzerland | Free | WebDAV only; non-EU | +| Proton Drive | — | rclone beta | Switzerland | Free | rclone backend broken since late 2025 | + +### Phase 2 — postgres consolidation onto CT 113 (✅ done 2026-05-21) + +**Decision**: provision a dedicated `db` LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse `shared-postgres` on CT 104. Direct-to-target avoided migrating twice (CT 111 → CT 104 → CT 113). + +Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, `mp0=/mnt/pve/unas`. Stack in Gitea `fkrebs/stacks-db`. pgAdmin pre-registers postgres via `pgadmin-servers.json`. + +**Migration order**: vaultwarden → paperless → litellm → memos → n8n → gitea → coder. Procedure per DB: + +```bash +docker exec pg_dump -U > /mnt/pve/unas/dump/-migration.sql +psql -U postgres -c "CREATE USER WITH PASSWORD '...'; CREATE DATABASE OWNER ;" +psql -U < /mnt/pve/unas/dump/-migration.sql +# Update connection strings → 192.168.1.6:5432, restart service, verify, remove old PG container + volume +``` + +Stale DBs dropped from `shared-postgres` post-migration: `daytona`, `lobechat` (duplicate), `paradedb` (duplicate). + +WAL-G enabled 2026-05-21: `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` via `ALTER SYSTEM`; first base backup verified (`base_000000010000000000000012`). Daily `wal-g backup-push` cron at 02:00. Garage → JottaCloud offsite sync added at 02:30. + +### Phase 1a rclone sync script + +```bash +#!/bin/bash +set -e +LOG=/var/log/rclone-unas-sync.log + +# Job 1: Immich originals → JottaCloud gallery +rclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \ + --transfers=4 --checkers=8 \ + --log-file=$LOG --log-level INFO + +# Job 2: all other irreplaceable personal data +rclone sync /mnt/pve/unas/ jottacloud:UNAS/ \ + --transfers=4 --checkers=8 \ + --exclude "media/images/library/**" \ + --exclude "media/images/upload/**" \ + --exclude "media/images/thumbs/**" \ + --exclude "media/images/encoded-video/**" \ + --exclude "media/images/profile/**" \ + --exclude "media/images/backups/**" \ + --exclude "media/movies/**" \ + --exclude "media/emulation/**" \ + --exclude "media/Torrents/**" \ + --exclude "media/music/**" \ + --exclude "media/podcasts/**" \ + --exclude "services/**" \ + --exclude "backup/**" \ + --exclude "backup-staging/**" \ + --exclude "test_perm" \ + --log-file=$LOG --log-level INFO +``` + +Cron (Saturday 01:00): `0 1 * * 6 /usr/local/sbin/unas-sync.sh` + +### Phase 3 config-to-git scripts + +Same pattern on each host. Replace `GITEA_TOKEN` with the value from `/opt/stacks/ai/.env` or a dedicated scoped Gitea token. + +**CT 108 — Zoraxy** (`/usr/local/sbin/zoraxy-conf-backup.sh`): + +```bash +#!/bin/bash +set -e +REPO_URL="https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git" +WORK="/opt/zoraxy/conf" +git -C "$WORK" init -b main -q 2>/dev/null || true +git -C "$WORK" remote set-url origin "$REPO_URL" 2>/dev/null \ + || git -C "$WORK" remote add origin "$REPO_URL" +git -C "$WORK" add -A +git -C "$WORK" commit -q -m "auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null || true +git -C "$WORK" push -q origin main 2>&1 | grep -v "Everything up-to-date" || true +``` + +**CT 102 — AdGuard** — same template, copy `/opt/AdGuardHome/AdGuardHome.yaml` into `/tmp/adguard-conf-work` checkout first. + +**PVE host** — same template, `rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/` then commit. + +**VM 100 — HA** — native [Git Pull addon](https://github.com/home-assistant/addons/tree/master/git_pull) pushing `/config/` (exclude `secrets.yaml`, `.storage/`) to `fkrebs/ha-config`. HA daily tars already covered by `services-backup-plan` via `/mnt/pve/unas/backup/home-assistant/`. + +Cron on each host: `0 3 * * * /usr/local/sbin/-conf-backup.sh` + +### Why Wasabi was rejected + +Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion — every `restic forget --prune` generates surprise costs. Well-documented Restic-on-Wasabi trap. + +### Why lobe-postgres stays on CT 104 + +Migration `0093_add_bm25_indexes_with_icu.sql` creates `USING bm25` indexes on 7 tables (`agents`, `topics`, `files`, `knowledge_bases`, `user_memories`, `chat_groups`, `user_memories_contexts`). `bm25` is paradedb-only (`pg_search` extension); stock postgres 17 has no such index access method and the migration fails. Container stays `paradedb/paradedb:latest-pg17`. WAL-G archives to `lobe-pg-backup`; Backrest secondary copies the data dir. + +### UNAS data inventory (basis for Phase 1a sizing) + +| Path | Size | Notes | +|---|---|---| +| `media/documents/` | 9.5 G | Personal documents | +| `media/video-projects/` | 425 G | Creative work, irreplaceable | +| `media/musical-sheets/` | 23 G | | +| `media/audiobooks/` | 23 G | | +| `media/ebooks/` | 6.8 G | | +| `media/3d-prints/` | 65 M | | +| `media/Recipes/` | 113 M | | +| `_sortMe/` | 335 G | `images/` 171 G, `work Flo/` 147 G, `Anne/` 17 G | +| `code/` | 6.7 M | | +| Excluded `media/movies/` | 118 G | re-streamable | +| Excluded `media/emulation/` | 57 G | re-downloadable | +| Excluded `media/Torrents/` | 19 G | temporary | +| Excluded `media/music/`, `media/podcasts/` | — | re-streamable | diff --git a/services/databases.md b/services/databases.md new file mode 100644 index 0000000..6bc8ab9 --- /dev/null +++ b/services/databases.md @@ -0,0 +1,97 @@ +# Databases + +All application databases live on **CT 113** (`192.168.1.6:5432`) after Phase 2 migration. The LXC runs `postgres:17` in Docker at `/opt/stacks/db/`, tracked in Gitea `fkrebs/stacks-db`. WAL-G archives to Garage S3 bucket `ct113-pg-backup` on CT 104 (`http://192.168.1.40:10004`). + +--- + +## Database inventory + +| DB | Owner user | Size (pre-migration) | Service | Stack location | +|----|-----------|---------------------|---------|---------------| +| `vaultwarden` | vaultwarden | 11 MB | Vaultwarden | CT 104 `/opt/stacks/vaultwarden/` | +| `paperless` | paperless | 20 MB | Paperless-ngx | CT 104 `/opt/stacks/apps/paperless-ngx/` | +| `litellm` | litellm | 212 MB | LiteLLM | CT 104 `/opt/stacks/ai/` | +| `memos` | memos | 9 MB | Memos | CT 104 `/opt/stacks/memos/` | +| `n8n` | n8n | 12 MB | n8n | CT 104 `/opt/stacks/n8n/` | +| `gitea` | gitea | 15 MB | Gitea | CT 111 `/opt/stacks/gitea/` | +| `coder` | coder | 17 MB | Coder | CT 111 `/opt/stacks/coder/` | + +**Not on CT 113:** + +| DB | Container | Reason | +|----|-----------|--------| +| `lobechat` | `lobe-postgres` (paradedb) on CT 104 | Requires `pg_search` extension for `USING bm25` indexes | +| `immich` | `immich_postgres` on CT 104 | Version-pinned by Immich AIO | +| `nextcloud` | Nextcloud AIO on CT 105 | AIO manages its own postgres | + +--- + +## Connection strings (post-migration target) + +| Service | Connection string | +|---------|------------------| +| Vaultwarden | `postgresql://vaultwarden:@192.168.1.6:5432/vaultwarden` | +| Paperless | `PAPERLESS_DBHOST: 192.168.1.6` | +| LiteLLM | `postgresql://litellm:@192.168.1.6:5432/litellm` (in `ai/.env` and `litellm-config/config.yaml`) | +| Memos | `postgresql://memos:@192.168.1.6:5432/memos?sslmode=disable` | +| n8n | `DB_POSTGRESDB_HOST=192.168.1.6` | +| Gitea | `GITEA__database__HOST: 192.168.1.6:5432` | +| Coder | `postgresql://coder:@192.168.1.6:5432/coder?sslmode=disable` | + +Passwords are in each service's `.env` file (never committed to git). See init script at `/opt/stacks/shared-db/init/01-create-users-dbs.sql` on CT 104 for the original credential set. + +--- + +## Migration procedure (one DB at a time) + +Order: vaultwarden → paperless → litellm → memos → n8n → gitea → coder + +```bash +# 1. Stop the service +# docker compose -f stop + +# 2. Dump from source (via PVE host) +# For CT 104 services: +pct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres \ + > /mnt/pve/unas/dump/-migration-$(date +%Y%m%d).sql + +# For CT 111 services: +pct exec 111 -- docker exec -i pg_dump -U \ + > /mnt/pve/unas/dump/-migration-$(date +%Y%m%d).sql + +# 3. Create user + DB on CT 113 +pct exec 113 -- docker exec -i postgres psql -U postgres < WITH PASSWORD ''; +CREATE DATABASE OWNER ; +EOF + +# 4. Restore on CT 113 +pct exec 113 -- bash -c "docker exec -i postgres psql -U postgres -d " \ + < /mnt/pve/unas/dump/-migration-*.sql + +# 5. Update service connection string (shared-postgres → 192.168.1.6) +# Edit compose or .env + +# 6. Start service; verify logs and function + +# 7. Verify, then old DB/container can be removed +``` + +--- + +## Post-migration cleanup + +After all 7 DBs are migrated and verified: + +1. Drop stale DBs from `shared-postgres`: `daytona`, `lobechat` (duplicate — real one is in `lobe-postgres`), `paradedb` +2. Stop and remove `shared-postgres` container + named volume `shared-pgdata` +3. Stop and remove `gitea-db` and `coder-db` containers + volumes on CT 111 +4. Update Backrest services plan: remove `shared-db` path, update to CT 113 WAL-G output +5. Enable **PVE protection** on CT 113 (prevents accidental delete) + +--- + +## pgAdmin + +pgAdmin on CT 113 at `http://192.168.1.6:5050` — pre-registered server: CT 113 postgres. +Credentials in `/opt/stacks/db/.env` (admin@nucli.de). diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index ea08932..b13de6e 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -25,7 +25,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 - `109` lxc **observe** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md` - - `113` lxc **db** *(planned)* — shared Postgres + pgbouncer once second NVMe lands + - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21 - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series APs. Controller at **`https://192.168.1.1`** (UniFi OS 5.0.16, SSO + MFA diff --git a/services/mcp-gateway.md b/services/mcp-gateway.md index e8e1d99..05acada 100644 --- a/services/mcp-gateway.md +++ b/services/mcp-gateway.md @@ -1,120 +1,128 @@ # MCP Gateway v2 — Operational -Finished implementation of the DinD MCP gateway (Option A). Supersedes the -"Phase 1" stub described in `mcp-gateway-requirements.md`. +DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP servers (mix of static, spawned, and Docker-catalog-bridged). Public via Zoraxy at `https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). -## What it is +## Architecture -`ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`, public via Zoraxy at -`https://mcp.nuclide.systems` (→ `192.168.1.40:8080`). Single OAuth-gated -entrypoint fronting several MCP servers. - -## Architecture (as built) - -- **Spawn model**: Docker-in-Docker. Each spawnable server runs as - `mcp-` on the **`ai-internal`** network with **no host ports** - (fixes the old `8000:8000` collision — servers talk over internal DNS). -- **Transport bridge**: generic streaming reverse proxy - `https://mcp.nuclide.systems//` → `http:///`. - Transport-agnostic — works for streamable-HTTP (`/mcp`) and legacy SSE - (`/sse`+`/messages`). Registered LAST so specific routes win. +- **Stack**: `ai/mcp-gateway/` — FastAPI app, container `mcp-gateway`. +- **Spawn model**: Docker-in-Docker. Each spawnable server runs as `mcp-` on the `ai-internal` network with no host ports — servers talk over internal DNS. +- **Transport bridge**: generic streaming reverse proxy `https://mcp.nuclide.systems//` → `http:///`. Transport-agnostic (streamable-HTTP `/mcp` and legacy SSE `/sse` + `/messages`). Registered last so specific routes win. - **Auth**: Pocket ID. Two paths off the same OIDC app: - - **MCP clients / API / proxy** — `Bearer` token, validated via OIDC - userinfo (300s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728 - `/.well-known/oauth-protected-resource` for discovery. - - **Browser Web UI** — full Authorization-Code flow with an httpOnly - session cookie: unauthenticated `/ui` → `302 /login` → Pocket ID - `authorize` (CSRF `state` cookie) → `/sso/callback` (state-checked - code→token exchange) sets the `mcp_session` cookie and shows the page + - a copyable Bearer token for MCP clients. `/logout` clears it. The UI and - `/api/*` accept **either** the cookie or a Bearer header; the `/{srv}/` - proxy stays Bearer-only. `/health` + the well-known metadata are open. -- **Secrets**: all per-server OAuth client secrets come from `ai/.env` - (`env_file: ../.env`), read via `os.environ` — none hardcoded in source. + - **MCP clients / API / proxy** — `Bearer` token, validated via OIDC userinfo (300 s TTL cache). `401` carries `WWW-Authenticate` + RFC 9728 `/.well-known/oauth-protected-resource` for discovery. + - **Browser Web UI** — Authorization-Code flow with httpOnly session cookie. `/ui` and `/api/*` accept cookie or Bearer; `/{srv}/` stays Bearer-only. +- **Secrets**: per-server OAuth client secrets live in `ai/.env` (read via `os.environ`), never hardcoded. -## Server registry (`SERVERS` in server.py) +## Server inventory (as of 2026-05-22) -| name | kind | upstream | notes | +26 servers configured in `mcp-gateway/config.json`. **Status: 25 healthy, 1 disabled.** + +| Group | Server | Kind | Notes | |---|---|---|---| -| comfyui | static | `http://comfyui-mcp:8000` | runs as its own stack (`ai/comfyui-mcp.yml`); gateway only proxies | -| nextcloud | spawn | `mcp-nextcloud:8000` | `ghcr.io/cbcoutinho/nextcloud-mcp-server` streamable-http | -| crawl4ai | spawn | `mcp-crawl4ai:11235` | `unclecode/crawl4ai` | -| mermaid | spawn | `mcp-mermaid:8000` | `node:20-slim` + npx mcp-mermaid (validated working) | -| markitdown | spawn | `mcp-markitdown:8000` | `ghcr.io/astral-sh/uv` + uvx markitdown-mcp | -| papersearch | spawn | `mcp-papersearch:8000` | python + uv + mcp-proxy → paper_search_mcp | +| **dev** | context7 | catalog | `mcp/context7` | +| | fetch | spawn | `mcp-server-fetch` via uvx | +| | git | catalog | `mcp/git` — Docker MCP Catalog; `mcp-git` bridge + DinD child | +| | gitlab | catalog | `mcp/gitlab` — `GITLAB_API_URL=https://gitlab.dlr.de/api/v4`; requires `--pass-environment` in bridge cmd | +| | kroki | static | `kroki-mcp:8000` (own stack) | +| | markitdown | catalog | `mcp/markitdown` | +| | sequential-thinking | catalog | `mcp/sequentialthinking` | +| | time | spawn | `mcp-server-time` | +| | docling | static | `docling-mcp:8000` | +| | coder | static | `coder-mcp:8000` | +| | shepard | static | `https://shepard.nuclide.systems/v2/mcp` | +| **research** | crawl4ai | spawn | `unclecode/crawl4ai` | +| | paper-search | catalog | `mcp/paper-search` — Docker MCP Catalog; replaces old `python:3.12-slim` inline approach | +| | searxng | spawn | `isokoliuk/mcp-searxng` | +| | wikipedia-mcp | catalog | | +| | youtube-transcript | catalog | `mcp/youtube-transcript` | +| **personal** | bluesky | static | `ariel-mcp:8000` | +| | gotify | spawn | | +| | home-assistant | static | `http://192.168.1.60:9583/private_...` (HA add-on) | +| | immich | spawn | | +| | memos | spawn | uses `MEMOS_TOKEN` from `.env` | +| | n8n | static | `https://n8n.nuclide.systems/mcp-server/http` | +| | nextcloud | spawn | ❌ **disabled** — SSL CERTIFICATE_VERIFY_FAILED contacting `https://nc.nuclide.systems` (self-signed). `"enabled": false` set 2026-05-21. | +| | unifi | spawn | | +| **image** | comfyui | static | `comfyui-mcp:8000` (own stack, see [comfyui.md](comfyui.md)) | +| **storage** | upload-artifact | static | `upload-artifact-mcp:8000` | -## Verified +> **Catalog bridge env var passthrough**: `mcp-proxy`'s `stdio_client` uses a sanitized env (`get_default_environment()`), stripping custom vars. For catalog servers that need env vars (e.g. `gitlab`), the bridge command **must** include `--pass-environment` so Docker CLI can resolve `-e KEY` passthrough from the bridge container's env. -- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth). -- `/mcp.json`, `//mcp` → 401 without token; 401 with bad token - (userinfo path); correct `WWW-Authenticate`. -- Upstream path proven: gateway → `comfyui-mcp` `/mcp` initialize → 200; - gateway → spawned `mcp-mermaid` `/mcp` → 200 (spawn pattern + proxy). -- Route precedence correct (specific routes beat the catch-all proxy). - -## Not yet exercised (needs a real token / operational bring-up) - -- **Token-authenticated end-to-end** requires an interactive Pocket ID login. - Get a token: open `https://mcp.nuclide.systems/sso/callback` via the OAuth - authorize flow (or any Pocket ID token with scopes `openid,mcp`); the - callback page prints the access token. -- **Bring up the 4 remaining spawnable servers** (nextcloud/crawl4ai/ - markitdown/papersearch): `POST /api/servers//start` (bearer) or - `POST /api/servers/bulk/start`. Each may need a transport/flag tweak on - first run — mermaid is the proven reference. +> **Re-enabling nextcloud**: needs either (a) a CA bundle in the container that trusts the Zoraxy/internal cert chain, or (b) skip-verify env var passed to the upstream client, or (c) point `NEXTCLOUD_HOST` at the internal Nextcloud LAN URL on `ai-internal`. Tracked as an open issue. ## Client registration -- **LobeChat**: Settings → Skills → Skill Store → Custom → Import JSON, e.g. - ```json - { "mcpServers": { "comfyui": { "type": "http", - "url": "https://mcp.nuclide.systems/comfyui/mcp", - "auth": { "type": "bearer", "accessToken": "" } } } } - ``` - (Or, for ComfyUI specifically, connect LobeChat straight to - `http://comfyui-mcp:8000/mcp` internally — see `comfyui-mcp.md` — no token.) -- **Claude.ai**: add `https://mcp.nuclide.systems//mcp` as a custom - connector; it will discover auth via the protected-resource metadata. +### Easy path — `/mcp-config` download -## Ops - -- `cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build` -- Logs: `docker logs mcp-gateway`; UI: `https://mcp.nuclide.systems/ui` - (Pocket-ID-gated; `/login` to sign in). -- Config overlay persisted in `ai/mcp-gateway/config.json`. -- **UI check (Playwright)** — the live UI is auth-gated, so the check - serves the local template + mocks `/api/servers`, screenshots, and - asserts cards render: - ``` - uv run --with playwright -- python -m playwright install chromium # once - uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png - ``` - -## Client config download — `/mcp-config` (added 2026-05-21) - -Instead of asking the user to paste a bearer token into per-server entries -in `~/.claude/settings.json`, the gateway now hands back a ready-to-use -config blob for the signed-in user: +The gateway hands back a ready-to-use config blob for the signed-in user: ``` GET https://mcp.nuclide.systems/mcp-config?format=claude # default GET https://mcp.nuclide.systems/mcp-config?format=cursor -GET https://mcp.nuclide.systems/mcp-config?format=raw # token + server URLs only +GET https://mcp.nuclide.systems/mcp-config?format=raw # token + URLs ``` -- Requires the user's Pocket-ID session cookie OR an existing Bearer token (303 → `/login` if neither). -- Mints a fresh long-lived gateway token (`mcp_`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:` so it's distinguishable from OAuth-issued tokens (and revocable). -- Returns a JSON file with `Content-Disposition: attachment` so the browser saves it. +- Requires Pocket-ID session cookie or existing Bearer (303 → `/login` otherwise). +- Mints a fresh long-lived gateway token (`mcp_`) bound to the user; persisted in `gateway_tokens.json` with `purpose=client-config:` so it's distinguishable from OAuth-issued tokens and revocable. +- Returns JSON with `Content-Disposition: attachment`. -### Claude Code workflow +**Claude Code workflow:** 1. Sign in at `https://mcp.nuclide.systems`. -2. Click **Claude config** in the top nav. Browser downloads `nuclide-mcp.claude.json`. +2. Click **Claude config** in the top nav → browser downloads `nuclide-mcp.claude.json`. 3. Merge the `mcpServers` block into `~/.claude/settings.json`. Restart Claude Code. -The downloaded file has 23 entries, one per spawnable MCP server, with `type: http` + `url: https://mcp.nuclide.systems//mcp` + an `Authorization: Bearer ...` header. No further per-server configuration needed. +The downloaded file has one entry per enabled MCP server, with `type: http` + `url: https://mcp.nuclide.systems//mcp` + `Authorization: Bearer ...` header. No per-server config needed. -### Rotation -Each download mints a NEW token. Old tokens remain valid until you remove them from `gateway_tokens.json` (manual prune; future: admin UI button). +### Manual -### Server list reflected in the config -The gateway populates from `mcp-gateway/config.json` `mcpServers` keys — exactly the set running at the time of download. New servers added later require re-downloading the config. +**LobeChat** — Settings → Skills → Skill Store → Custom → Import JSON: + +```json +{ "mcpServers": { "comfyui": { "type": "http", + "url": "https://mcp.nuclide.systems/comfyui/mcp", + "auth": { "type": "bearer", "accessToken": "" } } } } +``` + +(For ComfyUI specifically, LobeChat can also connect straight to `http://comfyui-mcp:8000/mcp` internally — see [comfyui.md](comfyui.md) — no token.) + +**Claude.ai** — add `https://mcp.nuclide.systems//mcp` as a custom connector; auth is discovered via the protected-resource metadata. + +### Token rotation + +Each `/mcp-config` download mints a NEW token. Old tokens stay valid until removed from `gateway_tokens.json` (manual prune; future: admin UI button). + +## Ops + +```bash +cd /opt/stacks/ai/mcp-gateway && docker compose up -d --build +docker logs mcp-gateway +``` + +UI: `https://mcp.nuclide.systems/ui` (Pocket-ID-gated; `/login` to sign in). + +Config: `mcp-gateway/config.json` (live config, persisted). Timestamped backups created on every edit (`config.json.bak.YYYYMMDD-HHMMSS`). + +### Add or enable a server + +1. Edit `mcp-gateway/config.json` — add an `mcpServers.` entry (or set `enabled: true`). +2. Restart the gateway: `docker restart mcp-gateway`. +3. Spawn servers come up on demand via `POST /api/servers//start` (bearer); or `POST /api/servers/bulk/start` for all. + +### Disable a server + +Set `mcpServers..enabled = false` and add `disabled_reason`. Stop+rm the container (`docker stop mcp- && docker rm mcp-`); the gateway will not re-spawn it. + +### UI check (Playwright) + +Live UI is auth-gated, so the local check serves the template + mocks `/api/servers`: + +```bash +uv run --with playwright -- python -m playwright install chromium # once +uv run --with playwright -- python ai/mcp-gateway/ui_check.py # -> /tmp/mcp_ui.png +``` + +## Verified end-to-end + +- `/health` 200 (no auth); `/.well-known/oauth-protected-resource` JSON (no auth). +- `/mcp.json`, `//mcp` → 401 without token; correct `WWW-Authenticate`. +- Static upstreams (`bluesky`, `coder`, `comfyui`, `docling`, `kroki`, `upload-artifact`) — endpoints reachable from inside `ai-internal` (406/405 to plain GET, which is the correct MCP behavior). +- All spawn/catalog containers up (`docker ps mcp-*`): 16/17 (nextcloud disabled; git/gitlab/paper-search added 2026-05-22).