diff --git a/RESUME.md b/RESUME.md index 0fe3e79..9250eb6 100644 --- a/RESUME.md +++ b/RESUME.md @@ -1,6 +1,6 @@ # Session Resume -Last updated: 2026-05-23 (session continued ×3). +Last updated: 2026-05-23 (session continued ×4). ## Open items (urgency order) @@ -13,12 +13,12 @@ Last updated: 2026-05-23 (session continued ×3). - [x] ~~**Arcane OIDC secret**~~ — fixed 2026-05-23. New secret generated, Pocket-ID bcrypt updated, Arcane compose updated, force-recreated. Login confirmed working. - [ ] **Backrest media-repo** — `video-projects-plan` has NEVER completed a successful snapshot. `media-backup-plan` last OK 2026-05-21; failed 2026-05-22 01:30; orphaned restic run in progress since 19:10 (Backrest restarted 21:23, orphaned it). Both plans next scheduled 2026-05-25 02:00/01:30. Monitor for completion; if they fail again, investigate rclone:jottacloud:media auth. - [ ] **AdGuard split-horizon DNS** — internal clients still resolve to public IPs. Rewrite rules audit. See `services/adguard-dns.md`. -- [ ] **WAL-G monitoring** — silent stall went undetected for 13 h. Add textfile collector + alert for hung archiver. See `services/homelab-architecture.md` roadmap. -- [ ] **Zoraxy audit** — CT 108 routes: WS headers, ACME coverage, decommissioned routes, auth consistency. See `services/zoraxy.md`. +- [x] ~~**WAL-G monitoring**~~ — done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`. +- [x] ~~**Zoraxy audit**~~ — done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`. ### 🟡 Medium — incomplete migrations / cleanup debt - [ ] **Docker disk reclaim** — ~17 GB reclaimable on CT 104 (10.3 GB unused images, 6.9 GB build cache). `docker system prune -a` after confirming no needed images. See `infra/proxmox-state.md` §10c. -- [ ] **Dormant stacks audit** — ~10 stacks defined but not running (homepage, dozzle, arr-stack, qdrant, etc.). Document: parked vs. abandoned. See `infra/proxmox-state.md` §10c. +- [ ] **Dormant stacks audit** — 4 decommissioned 2026-05-23 (daytona, homepage, proxy, streamio — composes renamed `.DECOMMISSIONED`). Remaining ~6 still to audit: dozzle (CT 104 copy), arr-stack, qdrant, etc. See `infra/proxmox-state.md` §10c. - [x] ~~**Config-to-git**~~ — deployed 2026-05-23. Scripts + daily 03:00 crons on CT 108, CT 102, PVE. Pushes to `fkrebs/zoraxy-conf`, `fkrebs/adguard-conf`, `fkrebs/pve-conf` (force-push). HA (`fkrebs/ha-config`) still pending — needs Git Pull addon on HAOS. - [ ] **D-Link hardening** — admin UI open to full LAN, SNMP unverified, HTTP-only. Enable trusted-host allowlist, harden SNMP, add TLS. See `services/homelab-architecture.md`. - [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22. @@ -27,7 +27,7 @@ Last updated: 2026-05-23 (session continued ×3). - [ ] **Infisical OIDC config** — Pocket-ID client created (`b2069075-ede2-4251-ad1f-9a62e6a188b3`). Manual step needed: enter OIDC settings in Infisical admin at `http://192.168.1.7:8200/admin` → Settings → OIDC. Issuer: `https://id.nuclide.systems`, ClientID: `b2069075`, Secret: `qsANw95zsza0_tzJR9SzSp4Z6iEIhidgJPHpDih1ffE`. - [ ] **Infisical secrets migration** — Phase 2: import CT104 `.env` keys. Use `infisical import`. See `services/secrets-manager.md` Phase 2. - [ ] **Arcane auto-update notifications** — ntfy.nuclide.systems provider configured; auto-update daily 3am set. Gotify Apprise incompatible (payload format). Workaround: configure via Arcane UI or add shoutrrr plugin. Gotify app token: `AQnDNB2Y6JlOewc` (app: arcane-updates). -- [ ] **Loki dashboard broken in Grafana** — Loki is healthy (labels returning). Likely dashboard query issue. Needs investigation. +- [x] ~~**Loki dashboard broken in Grafana**~~ — fixed 2026-05-23. K8s labels (namespace/pod) replaced with Docker labels (host/container/job) in 14 places across template variables and panel LogQL expressions. Provisioning file updated + pushed via Grafana API. - [ ] **Grafana HAOS dashboard** — HA recorder reconfiguration planned (exclude Prometheus-covered metrics). Reduce recorder include list. - [ ] **HAOS SSH key** — manual step: paste nuc public key in HA terminal. @@ -42,10 +42,22 @@ Last updated: 2026-05-23 (session continued ×3). - [ ] **ZFS ARC tuning** — raise `zfs_arc_max` from 6.2 GiB → 16 GiB once CT 101 memory right-sized. See `infra/proxmox-state.md` §3. - [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`. - [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`. -- [ ] **Additional MCP servers** — community MCPs exist for Gitea, Paperless, Karakeep, Vaultwarden, Proxmox-VE, Audiobookshelf. Evaluate + wire in. +- [ ] **Additional MCP servers** — Gitea ✓, Paperless ✓, Proxmox-VE ✓ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf. - [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts with unique tokens per environment. 8 environments total: NUC, db, shepard, docker, nextcloud, id, dev, secrets. Auto-update daily 3am. ntfy notifications configured. - [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers. +## Recently completed (2026-05-23, session continued ×4) + +- **Docs consistency pass**: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state. +- **Wetty deployed** on CT 109 (:4090, LAN-only). `jump-menu.sh` on nuc → SSH jump menu to all 10 hosts. +- **MCP servers added**: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers. +- **Loki dashboard fixed**: K8s labels replaced with Docker labels in 14 places. +- **WAL-G monitoring**: textfile collector on CT 113 + Prometheus alert rules on CT 109. +- **CT 103 RAM**: 2 GB → 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup). +- **Zoraxy WS**: `SkipWebSocketOriginCheck` enabled on 13 routes. Arcane upstream corrected to CT 109. +- **CT 104 cleanup**: daytona, homepage, proxy, streamio decommissioned. +- **Paperless-ngx**: USERMAP_UID=997 + file permissions fixed, container stable. + ## Recently completed (2026-05-23, session continued ×3) - **Homarr board deployed**: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: `x_offset`/`y_offset` on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board `nuclide` set as home board. @@ -100,7 +112,8 @@ Last updated: 2026-05-23 (session continued ×3). | Host | IP | Role | |---|---|---| -| nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs | +| nuc (PVE) | 192.168.1.20 | Proxmox host — SSH gateway to all CTs; jump-menu.sh | +| CT 109 ops | 192.168.1.8 | Monitoring — Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty | | CT 104 docker | 192.168.1.40 | Main Docker host — AI/ML, media, ~65 containers | | CT 113 db | 192.168.1.6 | Shared Postgres 17 + WAL-G → Garage S3 (cron 02:00) | | CT 110 id | 192.168.1.5 | Pocket-ID OIDC IdP | diff --git a/ct-inventory.md b/ct-inventory.md index 521587c..76012b2 100644 --- a/ct-inventory.md +++ b/ct-inventory.md @@ -7,7 +7,7 @@ Verified 2026-05-20 via `pct list`, `pct config `, `qm config 100` on `nuc`. | 100 | haos | VM (q35/OVMF) | DHCP via vmbr0 (`.60`) | 4 | 16384 (balloon 4096) | 32 GiB local-zfs | Home Assistant OS; USB Zigbee dongle (10c4:ea60) passed through → **Zigbee2MQTT** add-on + Mosquitto broker add-on; OCPP (EV charger); ~2492 entities | — | no | running | | 101 | shepard | LXC unpriv | 192.168.1.49/24 | 12 | 32768 | 500 GiB | Shepard product stack (Caddy, frontend, backend, Keycloak, Mongo, Neo4j, TimescaleDB) | mp0 NFS | Intel iGPU (card+render) | running | | 102 | dns | LXC unpriv | 192.168.1.2/24 | 2 | 1024 | 4 GiB | **AdGuard Home** — LAN DNS resolver + filter | — | no | running | -| 103 | backrest | LXC unpriv | 192.168.1.3/24 | 1 | 512 | 8 GiB | **Backrest** (restic) backup scheduler | mp0 NFS | no | running | +| 103 | backrest | LXC unpriv | 192.168.1.3/24 | 1 | 4096 + 1024 swap | 8 GiB | **Backrest** (restic) backup scheduler | mp0 NFS | no | running | | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | | 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 NFS | Intel iGPU (render only) | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | @@ -49,4 +49,4 @@ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuVqBW3VXg... root@nuc CT 113 ("db") provisioned 2026-05-21; see `services/databases.md`. Future role: consolidate per-stack Postgres instances once second NVMe lands. -CT 109 ("ops") provisioned 2026-05-23. Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring`: Prometheus (:9090), Grafana (:3000 — LAN only, admin/tapirnase), node-exporter (:9100 host-net). Scrapes: `litellm` on CT 104 `:14000/metrics/`, `node-ct104` `:9100`, `node-ct109` `:9100`. CT 104 runs a standalone node-exporter (`/opt/stacks/monitoring/`) for CT 109 to scrape. +CT 109 ("ops") provisioned 2026-05-23. Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring`: Prometheus (:9090), Grafana (:3000 — LAN only, admin/tapirnase), Loki (:3100), node-exporter (:9100 host-net), pve-exporter (:9221), Alloy (:12345), Homarr (:7575), Arcane server (:10002), Dozzle server (:10001), docs-server (:13080), Wetty web-SSH (:4090 LAN-only). Scrapes: `litellm` CT 104 `:14000/metrics/`, `node-ct104` `:9100`, `node-ct109` `:9100`, `walg` CT 113 `:9100/textfile`. CT 104 runs a standalone node-exporter (`/opt/stacks/monitoring/`) for CT 109 to scrape. diff --git a/infra/portmap.md b/infra/portmap.md index 08cf7b9..293d5d2 100644 --- a/infra/portmap.md +++ b/infra/portmap.md @@ -37,14 +37,15 @@ | [3100](http://192.168.1.8:3100) | CT 109 | Loki | LAN only; 30d retention; log aggregation | | [9221](http://192.168.1.8:9221) | CT 109 | pve-exporter | Proxmox VE metrics; auth: `monitor@pve!prometheus` | | [12345](http://192.168.1.8:12345) | CT 109 | Alloy (self) | agent UI; also runs on all other hosts at :12345 | +| [4090](http://192.168.1.8:4090) | CT 109 | Wetty | LAN only (127.0.0.1); web SSH → jump-menu.sh on nuc | | [7575](http://192.168.1.8:7575) | CT 109 | Homarr | LAN only; service dashboard | -| [10001](http://192.168.1.8:10001) | CT 109 | Dozzle | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | Live log viewer; agents on all 7 Docker hosts | +| [10001](http://192.168.1.8:10001) | CT 109 | Dozzle | LAN only; no Zoraxy route — `dozzle.nuclide.systems` not configured | Live log viewer; agents on all 7 Docker hosts | | [10002](http://192.168.1.8:10002) | CT 109 | Arcane | [arcane.nuclide.systems](https://arcane.nuclide.systems) | Docker mgmt; OIDC; headless agents on all 7 Docker hosts | | [13080](http://192.168.1.8:13080) | CT 109 | docs-server | LAN only; mkdocs Material; auto-rebuilds from `fkrebs/docs` every 5 min — migrated from CT 111 2026-05-23 | | 9100 | CT 109 | node-exporter | host-network, self-scrape | | 9100 | CT 104 | node-exporter | standalone stack `/opt/stacks/monitoring/`; scraped by CT 109 | -Scrape targets (CT 109 Prometheus): `litellm` CT104:14000/metrics/ (bearer sk-tapirnase), `node-ct104` :9100, `node-ct109` :9100, `home-assistant` 192.168.1.60:8123/api/prometheus (HA token), `prometheus` self. +Scrape targets (CT 109 Prometheus): `litellm` CT104:14000/metrics/ (bearer sk-tapirnase), `node-ct104` :9100, `node-ct109` :9100, `home-assistant` 192.168.1.60:8123/api/prometheus (HA token), `prometheus` self, `walg` CT113:9100/textfile (WAL-G backup freshness, added 2026-05-23). LiteLLM metrics: `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-config/config.yaml`. **Alloy (log agent)**: deployed on all 12 hosts → ships to Loki at CT 109:3100. @@ -58,7 +59,7 @@ LiteLLM metrics: `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-con | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| [10000](http://192.168.1.40:10000) | Homepage | `homepage` | — | Dashboard, LAN only | +| [10000](http://192.168.1.40:10000) | Homepage | `homepage` | — | **DECOMMISSIONED 2026-05-23** — compose renamed `.DECOMMISSIONED` | | [10001](http://192.168.1.40:10001) | Dozzle | `dozzle` | — | **DECOMMISSIONED** — moved to CT 109 2026-05-23 | | [10002](http://192.168.1.40:10002) | Arcane | `arcane` | — | **DECOMMISSIONED** — moved to CT 109 2026-05-23 | | [10003](http://192.168.1.40:10003) | Gotify | `gotify` | [gotify.nuclide.systems](https://gotify.nuclide.systems) | Push notifications | @@ -250,7 +251,7 @@ Zoraxy routes to these external backends: | [ai.nuclide.systems](https://ai.nuclide.systems) | litellm | 14000 | | [chat.nuclide.systems](https://chat.nuclide.systems) | lobehub | 14001 | | [mcp.nuclide.systems](https://mcp.nuclide.systems) | mcp-gateway | 8080 | -| [arcane.nuclide.systems](https://arcane.nuclide.systems) | arcane | 10002 | +| [arcane.nuclide.systems](https://arcane.nuclide.systems) | arcane on CT 109 | **192.168.1.8**:10002 | | [gotify.nuclide.systems](https://gotify.nuclide.systems) | gotify | 10003 | | [vault.nuclide.systems](https://vault.nuclide.systems) | vaultwarden | 11001 | | [immich.nuclide.systems](https://immich.nuclide.systems) | immich_server | 12000 | diff --git a/infra/proxmox-memory-audit.md b/infra/proxmox-memory-audit.md index 8926cdb..cca313f 100644 --- a/infra/proxmox-memory-audit.md +++ b/infra/proxmox-memory-audit.md @@ -15,14 +15,16 @@ Last audited: 2026-05-22 |----|------|----------------|------------|-------------|-------| | 101 | shepard | 32,768 | 8,192 | ~4 GB | Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak | | 102 | dns | 1,024 | 512 | ~100 MB | AdGuard Home | -| 103 | backrest | 2,048 | 512 | ~200 MB | Restic scheduler | +| 103 | backrest | **4,096** | **1,024** | ~200 MB | Restic scheduler — bumped 2026-05-23 for 822 GB initial backup OOM fix | | 104 | docker | **49,152** | 32,000 | **6–22 GB** | Main Docker host; FLUX spikes to ~22 GB | | 105 | nextcloud | 8,196 | 8,196 | ~2 GB | Nextcloud AIO | | 108 | zoraxy | 2,048 | 512 | ~300 MB | Reverse proxy | +| 109 | ops | 4,096 | 0 | ~1.5 GB | Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr | | 110 | id | 1,024 | 512 | ~200 MB | Pocket-ID | | 111 | dev | 32,768 | 8,192 | ~3 GB | Coder + Gitea workspaces | | 112 | secrets | 4,096 | 512 | ~600 MB | Infisical | -| **Sum** | | **133,124 MiB (130 GiB)** | | | **2× overprovisioned vs physical RAM** | +| 113 | db | 4,096 | 0 | ~800 MB | Postgres 17 + WAL-G | +| **Sum** | | **141,316 MiB (138 GiB)** | | | **2.2× overprovisioned vs physical RAM** | ## Key findings @@ -66,7 +68,7 @@ Peak host usage at 24 G container limit during FLUX generation: `24 + 7 (other c 2. **CT 104 LXC allocation (49 GiB)**: appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact. 3. **CT 111 (dev, 32 GiB)**: Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template. 4. **Watch list**: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB → host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104. -5. **Long-term**: when CT 109 (ops) is built, run Prometheus `node_exporter` on the PVE host and alert when host available RAM drops below 8 GiB. +5. ~~**Long-term**: when CT 109 (ops) is built, run Prometheus `node_exporter` on the PVE host and alert when host available RAM drops below 8 GiB.~~ Done 2026-05-23 — CT 109 live, node_exporter scraping PVE host via pve-exporter. ## ComfyUI memory optimisation log diff --git a/services/arcane.md b/services/arcane.md index d1e4d25..6993357 100644 --- a/services/arcane.md +++ b/services/arcane.md @@ -1,8 +1,8 @@ # Arcane -Web-based Docker management IDE. Main instance on **CT 104** (`192.168.1.40:10002`, `arcane.nuclide.systems`). Manages containers on the local Docker host plus remote CTs via **edge agents**. +Web-based Docker management IDE. Main instance on **CT 109** (`192.168.1.8:10002`, `arcane.nuclide.systems`). Manages containers on all Docker hosts via **edge agents**. Migrated from CT 104 → CT 109 on 2026-05-23. -Stack: `/opt/stacks/arcane/docker-compose.yml` on CT 104. +Stack: `/opt/stacks/arcane/docker-compose.yml` on CT 109. Image: `ghcr.io/getarcaneapp/arcane:latest` Auth: OIDC via Pocket ID, admin: `fkrebs@nucli.de` @@ -14,10 +14,16 @@ An edge agent (`ghcr.io/getarcaneapp/arcane-headless:latest`) runs inside each r ### Currently deployed agents -| CT | Hostname | Compose path | Status | -|----|----------|--------------|--------| -| CT 104 | docker | *(built-in NUC environment)* | online | -| CT 113 | db | `/opt/stacks/db/docker-compose.yml` | online (2026-05-21) | +| CT | Hostname | Environment | Compose path | Status | +|----|----------|------------|--------------|--------| +| CT 101 | shepard | shepard | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 104 | docker | docker | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 105 | nextcloud | nextcloud | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 110 | id | id | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 111 | dev | dev | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 112 | secrets | secrets | `/opt/stacks/ops-agents/docker-compose.yml` | online | +| CT 113 | db | db | `/opt/stacks/db/docker-compose.yml` | online | +| nuc | nuc | NUC (built-in) | *(built-in environment)* | online | ### Adding an edge agent to a new CT @@ -41,7 +47,7 @@ Arcane stores the raw AGENT_TOKEN in `environments.access_token`. Insert it via: import argon2, secrets, sqlite3, uuid from datetime import datetime, timezone -db_path = "/rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db" +db_path = "/rpool/data/subvol-109-disk-0/opt/stacks/arcane/data/arcane.db" env_id = "" # paste the environment UUID here conn = sqlite3.connect(db_path) @@ -78,7 +84,7 @@ print("AGENT_TOKEN:", raw_token) EDGE_AGENT: "true" EDGE_TRANSPORT: poll AGENT_TOKEN: ${ARCANE_AGENT_TOKEN} - MANAGER_API_URL: http://192.168.1.40:10002 + MANAGER_API_URL: http://192.168.1.8:10002 deploy: resources: limits: @@ -100,18 +106,9 @@ The environment should flip to `online` in `environments.status` within ~5 secon --- -## When Arcane moves to CT 109 +## Migration to CT 109 — completed 2026-05-23 -When Arcane migrates from CT 104 to CT 109 (the planned "observe" LXC): - -1. Update `MANAGER_API_URL` in every edge agent's `.env` to the new CT 109 LAN IP -2. Update the Arcane compose `APP_URL` and network bindings -3. Update Zoraxy route for `arcane.nuclide.systems` to point to CT 109 -4. Migrate the SQLite DB: `cp -a /opt/stacks/arcane/data/ /` -5. Regenerate the admin API key (stored in memory, not config) since the DB port changes -6. Restart all edge agents with the new `MANAGER_API_URL` - -Arcane environments (NUC, db, shepard) will reconnect automatically once `MANAGER_API_URL` is updated. +Arcane migrated from CT 104 → CT 109. All edge agents updated with `MANAGER_API_URL: http://192.168.1.8:10002`. Zoraxy upstream updated to `192.168.1.8:10002`. SQLite DB at `/opt/stacks/arcane/data/` on CT 109. --- @@ -121,6 +118,6 @@ Arcane doesn't store plaintext API keys — use the DB-insert method above when The key inserted for CLI use during CT 113 provisioning (`arc_a9695182...`) is linked to `fkrebs` user in the api_keys table. Rotate it after provisioning work is complete by deleting the row: ```sql -sqlite3 /rpool/data/subvol-104-disk-0/opt/stacks/arcane/data/arcane.db \ +sqlite3 /rpool/data/subvol-109-disk-0/opt/stacks/arcane/data/arcane.db \ "DELETE FROM api_keys WHERE name='admin-cli';" ``` diff --git a/services/mcp-gateway.md b/services/mcp-gateway.md index a85ea55..1008124 100644 --- a/services/mcp-gateway.md +++ b/services/mcp-gateway.md @@ -12,9 +12,9 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve - **Browser Web UI** — Authorization-Code flow with httpOnly session cookie. `/ui` and `/api/*` accept cookie or Bearer; `/{srv}/` stays Bearer-only. - **Secrets**: per-server OAuth client secrets live in `ai/.env` (read via `os.environ`), never hardcoded. -## Server inventory (as of 2026-05-22) +## Server inventory (as of 2026-05-23) -26 servers configured in `mcp-gateway/config.json`. **Status: 27 healthy.** +29 servers configured in `mcp-gateway/config.json`. **Status: 30 healthy** (includes gitea, paperless, proxmox added 2026-05-23). | Group | Server | Kind | Notes | |---|---|---|---| @@ -45,10 +45,13 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve | | unifi | spawn | | | **image** | comfyui | static | `comfyui-mcp:8000` (own stack, see [comfyui.md](comfyui.md)) | | **storage** | upload-artifact | static | `upload-artifact-mcp:8000` | +| **infra** | gitea | static | `gitea-mcp:8000` — Gitea API MCP server | +| | paperless | static | `paperless-mcp:8000` — Paperless-ngx MCP (`@nloui/paperless-mcp` via mcp-proxy) | +| | proxmox | static | `proxmox-mcp:8000` — read-only PVE API (`PVEAuditor` role, token `root@pam!mcp`) | > **Catalog bridge env var passthrough**: `mcp-proxy`'s `stdio_client` uses a sanitized env (`get_default_environment()`), stripping custom vars. For catalog servers that need env vars (e.g. `gitlab`), the bridge command **must** include `--pass-environment` so Docker CLI can resolve `-e KEY` passthrough from the bridge container's env. -> **Nextcloud server count**: 27 total (26 enabled + nextcloud now re-enabled 2026-05-22). Gateway updated to persist `enabled` flag across restarts. +> **Server count**: 30 total (27 from 2026-05-22 + gitea, paperless, proxmox added 2026-05-23). Gateway updated to persist `enabled` flag across restarts. ## Client registration diff --git a/services/zoraxy.md b/services/zoraxy.md index a2723bf..0ec61ff 100644 --- a/services/zoraxy.md +++ b/services/zoraxy.md @@ -4,35 +4,39 @@ ## Current route table -Audited and fully configured 2026-05-21. All 22 public routes have: +Audited 2026-05-23. All 22 public routes have: - `EnableAutoHTTPS: true` — Zoraxy requests LE certs for `*.nuclide.systems` - `EnableWebsocketCustomHeaders: true` — preserves WS upgrade headers through the proxy - `DisableHopByHopHeaderRemoval: true` — keeps hop-by-hop headers intact for upstream -| Domain | Upstream | Notes | -|--------|----------|-------| -| abs.nuclide.systems | 192.168.1.40:13003 | Absolut (abs) | -| ai.nuclide.systems | 192.168.1.40:14000 | LiteLLM gateway | -| arcane.nuclide.systems | 192.168.1.40:10002 | Arcane Manager | -| chat.nuclide.systems | 192.168.1.40:14001 | LobeHub | -| dev.nuclide.systems | 192.168.1.42:7080 | Coder (CT 111) | -| git.nuclide.systems | 192.168.1.42:3000 | Gitea (CT 111) | -| gotify.nuclide.systems | 192.168.1.40:10003 | Gotify push notifications | -| ha.nuclide.systems | 192.168.1.60:8123 | Home Assistant (VM 100) | -| hoarder.nuclide.systems | 192.168.1.40:17001 | Karakeep bookmarks | -| id.nuclide.systems | 192.168.1.5:11000 | Pocket-ID OIDC (CT 110) | -| immich.nuclide.systems | 192.168.1.40:12000 | Immich photos | -| mcp.nuclide.systems | 192.168.1.40:8080 | MCP gateway | -| memos.nuclide.systems | 192.168.1.40:17000 | Memos notes | -| n8n.nuclide.systems | 192.168.1.40:16000 | n8n workflows | -| nc.nuclide.systems | 192.168.1.41:11000 | Nextcloud AIO (CT 105) | -| ocpp.nuclide.systems | 192.168.1.60:8887 | OCPP charger endpoint (HAOS) | -| s3.nuclide.systems | 192.168.1.40:10004 | Garage S3 (web endpoint) | -| shepard.nuclide.systems | 192.168.1.49:80 | Shepard frontend (CT 101) | -| shepard-api.nuclide.systems | 192.168.1.49:8080 | Shepard API (CT 101) | -| shepard-auth.nuclide.systems | 192.168.1.49:8082 | Shepard auth (CT 101) | -| traccar.nuclide.systems | 192.168.1.40:15000 | Traccar GPS | -| vault.nuclide.systems | 192.168.1.40:11001 | Vaultwarden | +`SkipWebSocketOriginCheck` is enabled on routes that use WebSocket heavily (n8n, Coder, Gotify, Immich, Arcane, abs, ai, chat, ha, mcp, nc, ocpp, s3, shepard*). Disabled on git, hoarder, id, memos (not needed). + +| Domain | Upstream | SkipWSOrigin | Notes | +|--------|----------|:---:|-------| +| abs.nuclide.systems | 192.168.1.40:13003 | ✓ | Audiobookshelf | +| ai.nuclide.systems | 192.168.1.40:14000 | ✓ | LiteLLM gateway | +| arcane.nuclide.systems | **192.168.1.8:10002** | ✓ | Arcane (moved to CT 109 2026-05-23) | +| chat.nuclide.systems | 192.168.1.40:14001 | ✓ | LobeHub | +| dev.nuclide.systems | 192.168.1.42:7080 | ✓ | Coder (CT 111) | +| git.nuclide.systems | 192.168.1.42:3000 | — | Gitea (CT 111) | +| gotify.nuclide.systems | 192.168.1.40:10003 | ✓ | Gotify push notifications | +| ha.nuclide.systems | 192.168.1.60:8123 | ✓ | Home Assistant (VM 100) | +| hoarder.nuclide.systems | 192.168.1.40:17001 | — | Karakeep bookmarks | +| id.nuclide.systems | 192.168.1.5:11000 | — | Pocket-ID OIDC (CT 110) | +| immich.nuclide.systems | 192.168.1.40:12000 | ✓ | Immich photos | +| mcp.nuclide.systems | 192.168.1.40:8080 | ✓ | MCP gateway | +| memos.nuclide.systems | 192.168.1.40:17000 | — | Memos notes | +| n8n.nuclide.systems | 192.168.1.40:16000 | ✓ | n8n workflows | +| nc.nuclide.systems | 192.168.1.41:11000 | ✓ | Nextcloud AIO (CT 105) | +| ocpp.nuclide.systems | 192.168.1.60:8887 | ✓ | OCPP charger endpoint (HAOS) | +| s3.nuclide.systems | 192.168.1.40:10004 | ✓ | Garage S3 (web endpoint) | +| shepard.nuclide.systems | 192.168.1.49:80 | ✓ | Shepard frontend (CT 101) | +| shepard-api.nuclide.systems | 192.168.1.49:8080 | ✓ | Shepard API (CT 101) | +| shepard-auth.nuclide.systems | 192.168.1.49:8082 | ✓ | Shepard auth (CT 101) | +| traccar.nuclide.systems | 192.168.1.40:15000 | ✓ | Traccar GPS | +| vault.nuclide.systems | 192.168.1.40:11001 | ✓ | Vaultwarden | + +**Missing routes** (services listed in portmap but no Zoraxy config exists): `dozzle.nuclide.systems`, `immich-tools.nuclide.systems`, `paperless.nuclide.systems`, `paperless-ai.nuclide.systems`. These are either LAN-only or the routes were never created. `root.config` — ProxyType=0 internal dashboard fallback (127.0.0.1:5487), no ACME.