fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: dissolve Stacks chapter → infra/; reconcile portmap, storage, changelog (2026-05-21)

Browse Source 
- Move PORTMAP, storage, volumes, docker-networks from stacks/ → infra/
- Remove stacks/todo.md (content migrated to CHANGELOG + ideas)
- portmap: add new MCP servers (18005/07/09/10/11), fix s3.nuclide.systems entry,
  update Daytona OIDC row to DECOMMISSIONED, add Vaultwarden OIDC placeholder,
  add DevOps table row for MCP server Gitea repos
- storage: add Garage S3 buckets subsection (lobe-files + chat-artifacts)
- docker-networks: remove daytona-minimal_daytona-network row + daytona from
  "Stacks Keeping Own Networks" list
- ct-inventory: add CT 112 (secrets/Infisical, planned)
- CHANGELOG: prepend 2026-05-21 entry (docs, S3, MCP, dev env, Gitea repos, planned)
- ideas/stack-ideas: add §11 Renovate Bot, §12 egress firewall, §13 local LLM
- mkdocs.yml: replace nav — dissolve Stacks chapter, add infra/ entries
This commit is contained in:
Florian Krebs
2026-05-21 07:54:21 +02:00
parent 630c5e0ae5
commit ad0fdd2beb
8 changed files with 91 additions and 179 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+31
View File
@@ -2,6 +2,37 @@
All notable infrastructure / service / doc changes. Newest first. All notable infrastructure / service / doc changes. Newest first.
## 2026-05-21
### Docs & docs infrastructure
- **Mermaid diagrams now rendering** in mkdocs Material site — was missing `custom_fences` config in `pymdownx.superfences`. Fixed in `docs-server/mkdocs.yml` and rebuilt container.
- **`site_url` corrected** to `http://192.168.1.111:13080` (docs are LAN-only; no Zoraxy route exists).
- **Docs restructured**: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to `infra/` section. TODO items migrated into CHANGELOG + ideas.
- **Secrets manager design doc** added at `services/secrets-manager.md` — recommends Infisical on a new CT 112.
### Storage / S3
- **`s3.nuclide.systems` Zoraxy route fixed** — `EnableWebsocketCustomHeaders`, `DisableHopByHopHeaderRemoval`, `EnableAutoHTTPS` all set. Garage S3 API now reachable externally.
- **`chat-artifacts` Garage S3 bucket created** — public-read via `https://chat-artifacts.s3.nuclide.systems/<key>`. Credentials at `/root/garage-chat-artifacts.creds` (not git-tracked).
### MCP surface
- **`upload-artifact` MCP server** deployed (port 18011, CT 104). Tools: `upload_text`, `upload_base64`, `list_artifacts`, `get_url`, `delete_artifact`. Registered in gateway (now 24 servers). Source: `git.nuclide.systems/fkrebs/mcp-upload-artifact`.
- **ComfyUI MCP** now auto-uploads completed images to `chat-artifacts` S3 and includes the public URL alongside the inline base64 image.
- **DAYTONA_API_KEY** removed from CT 104 `ai/.env` (Daytona decommissioned 2026-05-20).
### Dev environment
- **`savefig` dotfiles helper** added to `fkrebs/dotfiles` — uploads any file to `chat-artifacts` using `uv run --with boto3`; reads credentials from env vars.
- **Coder `python-uv` template** updated with `CHAT_ARTIFACTS_*` env vars baked in; template pushed to Coder server.
### Gitea repos
- New repos created: `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact`, `home-assistant-config` — all at `git.nuclide.systems/fkrebs/`.
- **HAOS config** (`192.168.1.60`, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see `services/secrets-manager.md` for SSH access notes).
### Planned
- CT 112 "secrets" — Infisical deployment (see `services/secrets-manager.md`)
- Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt)
- Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)
- LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred)
## 2026-05-20 ## 2026-05-20
### Auth / Identity (single sign-on lockdown) ### Auth / Identity (single sign-on lockdown)
ct-inventory.md
+3
View File
@@ -14,7 +14,10 @@ Verified 2026-05-20 via `pct list`, `pct config <id>`, `qm config 100` on `nuc`.
| 109 | observe | LXC | 192.168.1.6/24 | 4 | 6144 | 30 GiB | **Planned** — Prometheus, Grafana, Arcane Manager, Dozzle, Homarr, Diun, Tinyauth, socket-proxy | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 109 | observe | LXC | 192.168.1.6/24 | 4 | 6144 | 30 GiB | **Planned** — Prometheus, Grafana, Arcane Manager, Dozzle, Homarr, Diun, Tinyauth, socket-proxy | — | no | **planned** (see `infra/proxmox-state.md` §16) |
| 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running |
| 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running |
| 112 | secrets | LXC unpriv | TBD | 2 | 2048 | 8 GiB | **Planned** — Infisical secrets manager (replaces scattered .env files) | — | no | **planned** |
UNAS NFS = `192.168.1.31:/var/nfs/shared/storage` over NFSv3. CT 105 (Nextcloud) is the lone outlier — it mounts the same UNAS share over **CIFS/SMB 3.1.1**, not NFS. UNAS NFS = `192.168.1.31:/var/nfs/shared/storage` over NFSv3. CT 105 (Nextcloud) is the lone outlier — it mounts the same UNAS share over **CIFS/SMB 3.1.1**, not NFS.
CT 112 ("secrets") — Infisical deployment planned; see `services/secrets-manager.md`.
CT 113 ("db") is mentioned in memory notes (`postgres-consolidation`) as a future consolidation target. Not yet planned in any source-of-truth doc. CT 113 ("db") is mentioned in memory notes (`postgres-consolidation`) as a future consolidation target. Not yet planned in any source-of-truth doc.
ideas/stack-ideas.md
+38
View File
@@ -238,6 +238,42 @@ For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the ex
--- ---
## 11. Dependency updates via Renovate Bot (Gitea Actions)
Run **Renovate Bot** as a Gitea Actions workflow to automatically open PRs for outdated
dependencies in MCP server repos (`mcp-comfyui`, `mcp-docling`, `mcp-shepard`,
`mcp-upload-artifact`). Targets: `Dockerfile` base image tags + `requirements.txt` /
`pyproject.toml` Python deps.
Renovate supports Gitea natively via `platform: gitea` in `renovate.json`. The Gitea
Actions runner (`ct111-runner`) already exists; add a scheduled workflow calling
`renovate/renovate` Docker image once daily.
---
## 12. Egress firewall on UDM (UniFi)
Enforce outbound allow-list on the UDM / UniFi gateway — block all non-approved egress
by default. Goals:
- Prevent exfiltration from compromised containers
- Audit unexpected outbound connections (model providers, analytics, telemetry)
- Approved: LiteLLM model provider endpoints, Jottacloud, UNAS internal, NTP, DNS
Implementation: UDM firewall rules (WAN_OUT) + Threat Management IDS in monitor mode first.
---
## 13. Local LLM on Arc GPU via vllm / ollama for sensitive workloads
Run a privacy-sensitive LLM locally on the Arc iGPU using **vllm** (with XPU/IPEX backend)
or the Intel-patched **Ollama** build. Use cases: document classification in Paperless
workflows, offline coding assistant, fallback when cloud rate limits hit.
See §3 (On-device LLM) for implementation notes. This item tracks the specific motivation
of *sensitive workload isolation* — i.e., running prompts that should not leave the LAN.
---
## Priority order (rough) ## Priority order (rough)
1. **ComfyUI MCP async queue** — fixes timeout; unblocks img2img + reference features 1. **ComfyUI MCP async queue** — fixes timeout; unblocks img2img + reference features
@@ -246,3 +282,5 @@ For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the ex
4. **Observability LXC** — OTEL collector + VictoriaMetrics + Grafana + Uptime Kuma, co-located 4. **Observability LXC** — OTEL collector + VictoriaMetrics + Grafana + Uptime Kuma, co-located
5. **Backblaze B2 off-site backup** — already in todos 5. **Backblaze B2 off-site backup** — already in todos
6. **On-device LLM (Arc)** — depends on IPEX-LLM + Ollama Intel build stability 6. **On-device LLM (Arc)** — depends on IPEX-LLM + Ollama Intel build stability
7. **Renovate Bot** — low-effort automation win for MCP repos
8. **Egress firewall** — security hygiene; plan before adding more external-facing services
stacks/docker-networks.md → infra/docker-networks.md
+1 -3
View File
@@ -14,7 +14,6 @@ has no explicit `networks:` declaration. This has exhausted Docker's built-in
| `ai-internal` | 172.31.0.0/16 | 8 | | `ai-internal` | 172.31.0.0/16 | 8 |
| `arcane_default` | 172.22.0.0/16 | 1 | | `arcane_default` | 172.22.0.0/16 | 1 |
| `arr-stack_default` | 172.21.0.0/16 | 4 | | `arr-stack_default` | 172.21.0.0/16 | 4 |
| `daytona-minimal_daytona-network` | 172.23.0.0/16 | 4 |
| `dozzle_default` | 192.168.16.0/20 | 1 | | `dozzle_default` | 192.168.16.0/20 | 1 |
| `homepage_default` | 172.19.0.0/16 | 1 | | `homepage_default` | 172.19.0.0/16 | 1 |
| `immich_default` | 172.18.0.0/16 | 5 | | `immich_default` | 172.18.0.0/16 | 5 |
@@ -30,7 +29,7 @@ has no explicit `networks:` declaration. This has exhausted Docker's built-in
| `vaultwarden_default` | 192.168.64.0/20 | 1 | | `vaultwarden_default` | 192.168.64.0/20 | 1 |
| `vpn_default` | 192.168.80.0/20 | 1 | | `vpn_default` | 192.168.80.0/20 | 1 |
Total: 19 user-defined bridge networks. Total: 18 user-defined bridge networks (Daytona decommissioned 2026-05-20).
### Problem ### Problem
@@ -90,7 +89,6 @@ These stacks have complex internal networking and should keep their own:
- **immich** — 5 services with inter-dependencies - **immich** — 5 services with inter-dependencies
- **paperless-ngx** — 5 services (webserver, broker, db, gotenberg, tika) - **paperless-ngx** — 5 services (webserver, broker, db, gotenberg, tika)
- **arr-stack** — 5 services (rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr) - **arr-stack** — 5 services (rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr)
- **daytona** — explicit `daytona-minimal_daytona-network`
- **karakeep** — 3 services with chrome dependency - **karakeep** — 3 services with chrome dependency
- **homepage** — single service, can stay or migrate - **homepage** — single service, can stay or migrate
- **streamio** — already removed, stremio uses VPN container directly - **streamio** — already removed, stremio uses VPN container directly
stacks/PORTMAP.md → infra/portmap.md
+11 -5
View File
@@ -31,7 +31,7 @@
| [10001](http://192.168.1.40:10001) | Dozzle | `dozzle` | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | Log viewer | | [10001](http://192.168.1.40:10001) | Dozzle | `dozzle` | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | Log viewer |
| [10002](http://192.168.1.40:10002) | Arcane | `arcane` | [arcane.nuclide.systems](https://arcane.nuclide.systems) | Web IDE | | [10002](http://192.168.1.40:10002) | Arcane | `arcane` | [arcane.nuclide.systems](https://arcane.nuclide.systems) | Web IDE |
| [10003](http://192.168.1.40:10003) | Gotify | `gotify` | [gotify.nuclide.systems](https://gotify.nuclide.systems) | Push notifications | | [10003](http://192.168.1.40:10003) | Gotify | `gotify` | [gotify.nuclide.systems](https://gotify.nuclide.systems) | Push notifications |
| [10004](http://192.168.1.40:10004) | Garage S3 API | `garage` | s3.nuclide.systems (non-responsive, see notes) | Internal: `http://garage:3900` | | [10004](http://192.168.1.40:10004) | Garage S3 API | `garage` | [s3.nuclide.systems](https://s3.nuclide.systems) | FIXED 2026-05-21: proxied by Zoraxy with ACME TLS. Garage API accessible at `https://s3.nuclide.systems`. Internal: `http://garage:3900` |
| 10005 (127.0.0.1 only) | Garage Admin | `garage` | — | localhost only | | 10005 (127.0.0.1 only) | Garage Admin | `garage` | — | localhost only |
--- ---
@@ -40,7 +40,7 @@
| Port | Service | Container | Public URL | Notes | | Port | Service | Container | Public URL | Notes |
|---|---|---|---|---| |---|---|---|---|---|
| [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager · **no Pocket-ID client yet — gap** | | [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager · Pocket-ID OIDC client created 2026-05-21; auth flow not yet configured |
> Pocket-ID **migrated from this CT to LXC 110** on 2026-05-20. See the [External Services](#external-services-not-on-nuc-docker) table below. > Pocket-ID **migrated from this CT to LXC 110** on 2026-05-20. See the [External Services](#external-services-not-on-nuc-docker) table below.
@@ -80,6 +80,11 @@ All arr-stack services run behind `vpn_gluetun` container network.
| [8080](http://192.168.1.40:8080) | MCP Gateway | `mcp-gateway` | [mcp.nuclide.systems](https://mcp.nuclide.systems) | Pocket ID gated; spawns MCP servers on `ai-internal` | | [8080](http://192.168.1.40:8080) | MCP Gateway | `mcp-gateway` | [mcp.nuclide.systems](https://mcp.nuclide.systems) | Pocket ID gated; spawns MCP servers on `ai-internal` |
| [18002](http://192.168.1.40:18002) | ComfyUI | `comfyui` | — | LAN only (Intel Arc iGPU, FLUX.1-schnell GGUF) | | [18002](http://192.168.1.40:18002) | ComfyUI | `comfyui` | — | LAN only (Intel Arc iGPU, FLUX.1-schnell GGUF) |
| [18003](http://192.168.1.40:18003) | ComfyUI MCP | `comfyui-mcp` | — | FastMCP; reachable via gateway at `mcp.nuclide.systems/comfyui/mcp` | | [18003](http://192.168.1.40:18003) | ComfyUI MCP | `comfyui-mcp` | — | FastMCP; reachable via gateway at `mcp.nuclide.systems/comfyui/mcp` |
| [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway |
| [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway |
| [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only |
| [18010](http://192.168.1.40:18010) | Shepard MCP | `shepard-mcp` | — | Shepard research platform; reachable via gateway |
| [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway |
| — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub |
--- ---
@@ -117,7 +122,7 @@ All arr-stack services run behind `vpn_gluetun` container network.
| Port | Service | Container | Public URL | Notes | | Port | Service | Container | Public URL | Notes |
|---|---|---|---|---| |---|---|---|---|---|
<!-- Daytona DECOMMISSIONED 2026-05-20; replaced by Coder on CT 111. Compose renamed *.DECOMMISSIONED-2026-05-20. --> <!-- Daytona DECOMMISSIONED 2026-05-20; replaced by Coder on CT 111. Compose renamed *.DECOMMISSIONED-2026-05-20. -->
| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` |
--- ---
@@ -188,6 +193,7 @@ Zoraxy routes to these external backends:
| [hoarder.nuclide.systems](https://hoarder.nuclide.systems) | karakeep | 17001 | | [hoarder.nuclide.systems](https://hoarder.nuclide.systems) | karakeep | 17001 |
| [traccar.nuclide.systems](https://traccar.nuclide.systems) | traccar | 15000 | | [traccar.nuclide.systems](https://traccar.nuclide.systems) | traccar | 15000 |
| [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | dozzle | 10001 | | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | dozzle | 10001 |
| [s3.nuclide.systems](https://s3.nuclide.systems) | garage | 10004 |
**Not publicly proxied** (LAN / localhost only): pgadmin, paperless, paperless-ai, comfyui, comfyui-mcp, rdtclient, prowlarr, shelfarr, qdrant. **Not publicly proxied** (LAN / localhost only): pgadmin, paperless, paperless-ai, comfyui, comfyui-mcp, rdtclient, prowlarr, shelfarr, qdrant.
@@ -195,7 +201,6 @@ Zoraxy routes to these external backends:
## Known Issues ## Known Issues
- **`s3.nuclide.systems`** (Garage external S3): non-responsive. Internal S3 `http://garage:3900` works fine for wal-g, memos, lobe. External binding risks quorum loss — deferred.
- **`lobe-postgres` / `lobe-redis`**: host-port exposed (`0.0.0.0:5432/6379`) — firewall blocks external access but ideally restricted to localhost. - **`lobe-postgres` / `lobe-redis`**: host-port exposed (`0.0.0.0:5432/6379`) — firewall blocks external access but ideally restricted to localhost.
--- ---
@@ -215,13 +220,14 @@ Zoraxy routes to these external backends:
| `0aee4280` | Coder | `https://dev.nuclide.systems/api/v2/users/oidc/callback` | | `0aee4280` | Coder | `https://dev.nuclide.systems/api/v2/users/oidc/callback` |
| `9444609e` | Gitea | `https://git.nuclide.systems/user/oauth2/pocket-id/callback` | | `9444609e` | Gitea | `https://git.nuclide.systems/user/oauth2/pocket-id/callback` |
| `38469e7e` | Proxmox VE | `https://192.168.1.20:8006` | | `38469e7e` | Proxmox VE | `https://192.168.1.20:8006` |
| `798a367f` | ~~daytona~~ | **ORPHAN — Daytona decommissioned 2026-05-20, remove** | | ~~798a367f~~ | ~~daytona~~ | **DECOMMISSIONED — remove from Pocket-ID** |
| `cbbf20d5` | Audiobookshelf | `https://abs.nuclide.systems/…` + `*` | | `cbbf20d5` | Audiobookshelf | `https://abs.nuclide.systems/…` + `*` |
| `d8733fcc` | shelfarr | `*` | | `d8733fcc` | shelfarr | `*` |
| `78c78998` | Claude MCP | `https://claude.ai/api/mcp/auth_callback` + `https://mcp.nuclide.systems/mcp/auth/callback` | | `78c78998` | Claude MCP | `https://claude.ai/api/mcp/auth_callback` + `https://mcp.nuclide.systems/mcp/auth/callback` |
| `82ca2d53` | nuc-ai | (empty) — used by spawned MCP servers | | `82ca2d53` | nuc-ai | (empty) — used by spawned MCP servers |
| `7fe1a14b` | zoraxy | (empty) | | `7fe1a14b` | zoraxy | (empty) |
| `af2f837b` | mcp-auth | (empty) — legacy, unused | | `af2f837b` | mcp-auth | (empty) — legacy, unused |
| (new) | vaultwarden | `https://vault.nuclide.systems/auth/callback` | Created 2026-05-21; SSO wiring deferred |
**OIDC Endpoints** (corrected 2026-05-17 — previously used wrong `/api/v1/oauth2/` path): **OIDC Endpoints** (corrected 2026-05-17 — previously used wrong `/api/v1/oauth2/` path):
- Authorization: `https://id.nuclide.systems/authorize` - Authorization: `https://id.nuclide.systems/authorize`
stacks/storage.md → infra/storage.md
+7
View File
@@ -54,6 +54,13 @@ Tier-1 state and anything that should NOT be NFS-backed.
| AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup | | AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup |
| Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below | | Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below |
### Garage S3 buckets (on local zfs, CT 104)
| Bucket | Access key | Use | Public URL |
|---|---|---|---|
| `lobe-files` | GK55210… (from `.env`) | LobeHub file uploads + WAL-G PG backups | internal only |
| `chat-artifacts` | GK50bfc… | AI chat output artefacts (images, reports, SVG) | `https://chat-artifacts.s3.nuclide.systems/<key>` |
### Docker named volumes ### Docker named volumes
Local on `/var/lib/docker/volumes/`. Mostly databases and caches. Local on `/var/lib/docker/volumes/`. Mostly databases and caches.
stacks/volumes.md → infra/volumes.md
View File
stacks/todo.md
-171
View File
@@ -1,171 +0,0 @@
# Todo
## Goal
Consolidate databases, migrate storage to UNAS, integrate all services with Homepage.
---
# Ideas (user)
- Claude Max sub integration with litellm (how to handle auth, rate limits, etc.?)
- MCP deployment process ideas? Current setup seems cumbersome — can litellm host directly?
(how to integrate npx, uvx, and maybe docker-based containers?)
- Select offsite backup hosting provider, set up backup process (maybe restic to Backblaze B2
or similar), then set up rclone for off-site sync of Garage buckets to remote S3
(Backblaze B2, Jottacloud, or Hetzner)
- Research the LiteLLM MCP `source_url` column problem across GitHub issues, PRs, and docs
to understand the full picture
- Add Google Gemini (vision + 1M ctx + free text-embedding-004; needs GEMINI_API_KEY)
- Add fal.ai MCP server (`mcp/fal`) for fast cloud image gen < 5 s (vs ~150 s local ComfyUI)
- Add Cerebras (Llama 3 at ~1M tok/s, generous free tier; CEREBRAS_API_KEY)
# Next Steps / Todos
- [x] **ComfyUI added & fixed (2026-05-17)** — root-caused the self-referential
bind-mount, repaired mounts/healthcheck, migrated CPU→**Intel Arc iGPU**
(torch 2.7.1+xpu, no IPEX), pinned v0.21.1. FLUX.1-schnell GGUF, legible
text, ~94 s/image. Canonical workflow in `ai/comfyui/workflows/`.
- [x] **ComfyUI→LobeChat via MCP (2026-05-17)** — purpose-built `comfyui-mcp`
server, proven end-to-end. See `docs/comfyui-mcp.md`.
- [x] **MCP gateway finished (2026-05-17)** — DinD, real streaming MCP
reverse-proxy, Pocket ID bearer + RFC9728 discovery, OAuth callback fixed,
secrets from `.env`, no port collisions. See `docs/mcp-gateway.md`.
- [x] **LobeChat S3 images fixed (2026-05-17)** — added `S3_PUBLIC_DOMAIN=http://192.168.1.40:10004`
to `lobehub.yml`; pre-signed URLs now use the LAN-accessible Garage port instead of
the internal `garage:3900` hostname.
- [x] **MCP gateway groups + combined endpoints (2026-05-17)** — servers now grouped
(research/dev/personal/image); `/group/{name}/mcp` aggregates all member tools with
`server__tool` namespacing; UI shows grouped cards with copy-URL buttons; `/` → `/ui`
redirect added.
- [x] **LiteLLM Redis caching (2026-05-17)** — `cache: true` with Redis backend
(`lobe-redis`, already on `ai-internal`); 10-min TTL on completion + embedding calls.
- [x] **Cohere added to LiteLLM (2026-05-17)** — `command-a-03-2025` (20 RPM),
`cohere-embed-english/multilingual/light-v3` (2000 RPM); `COHERE_API_KEY` in `.env`.
Note: command-r / command-r-plus retired 2025-09-15.
- [x] **LobeChat MCP import done** — `comfyui-flux` server imported via UI.
- [x] **ComfyUI MCP async queue (2026-05-17)** — full rewrite: `generate_image`
returns job_id in <0.1 s; background polling thread updates state; `img2img`
accepts many images (URL/base64/`job:<id>` chaining); `get_job_status`,
`list_queue`, `cancel_job`, `list_recent_images`. New `flux-img2img-api.json`
workflow (VAEEncode path). See `docs/comfyui-mcp.md`.
- [ ] Nextcloud MCP: bearer token enabled in NC admin (user_oidc provider settings).
Gateway now provisions NC app-password at SSO login time (PocketID token live).
**Needs end-to-end test**: log in via mcp.nuclide.systems, then call /nextcloud/mcp.
- [ ] Observability LXC: deploy OTEL collector + VictoriaMetrics + Grafana + Loki
+ Uptime Kuma on a Proxmox LXC; see `docs/stack-ideas.md §8`.
- [ ] Arcane: migrate central server to Proxmox LXC; deploy arcane-agent (headless)
on NUC + .49; connect via TCP Docker socket or SSH Docker context.
- [ ] Docling + n8n ingest pipeline: Nextcloud/Paperless → Docling → LiteLLM embed
→ LobeChat knowledge base; see `docs/stack-ideas.md §1`.
- [x] **Gateway spawn servers (2026-05-17)** — papersearch, crawl4ai, markitdown
all running on `ai-internal`. Fixed papersearch command (`uvx --with` not
`uv tool run --from`). nextcloud deferred: Nextcloud OIDC discovery not
enabled; needs `NEXTCLOUD_USERNAME`/`NEXTCLOUD_PASSWORD` in `ai/.env` for
BasicAuth mode (server.py already updated).
- [x] **ComfyUI inpaint smoke-tested (2026-05-17)** — Kodak #23 (768×512) + center
alpha-mask, submitted via API, output in 290s. Workflow confirmed working end-to-end.
- [x] **garage external S3 working** — `s3.nuclide.systems` returns 403 (AccessDenied
on root, expected); Zoraxy routes correctly to `10004→3900`. Was already fixed by
the earlier port binding change (`0.0.0.0:10004`); todo was stale.
- [x] **syncstack gone** — container no longer present; one-shot job completed/pruned.
- [x] **immich_machine_learning healthcheck fixed** — already on :3003/ping.
Container shows healthy.
- [x] **LiteLLM healthcheck fixed** — was hitting host port 14000 inside container;
corrected to container port 4000. Now healthy.
- [x] **immich_power_tools / karakeep / karakeep_chrome** — all healthy (slow warmup
self-resolved).
## In Progress
- None
## Blocked
- None
---
# Reference / History
## Done
- Created `/opt/stacks/shared-db/` with shared-postgres (pg16), Garage S3, pgAdmin
- Consolidated daytona, litellm, paperless-ngx → shared-postgres
- LobeHub kept on own pg17 (ParadeDB)
- Switched LobeHub from RustFS → Garage S3 (`lobe-files` bucket)
- Migrated arr-stack configs + media to UNAS via rsync
- Migrated qdrant to UNAS (fresh start)
- Added 29 selfh.st icons, updated all homepage labels to `sh-` prefix
- Homepage: instance hrefs (192.168.1.40:PORT) + public hrefs (*.nuclide.systems)
- Homepage widgets for 11 services
- Groups consolidated from 14 → 6 (infrastructure, security, media, ai, productivity, diagnostic)
- Configured memos with Garage S3
- Fixed Garage port binding: `127.0.0.1:10004` → `0.0.0.0:10004` (Zoraxy couldn't reach it)
- Fixed memos S3 endpoint: `https://s3.nuclide.systems` → `http://garage:3900` (TLS cert mismatch)
- Cleaned up old local data remnant at `/opt/stacks/memos/data/`
- traccar GPS port fixed: 15001 → 19001 (paperless-ai port conflict)
- Updated PORTMAP.md, created storage.md and volumes.md
- WAL-G v3.0.8 setup for all 3 PG instances (shared-postgres, immich, lobe)
- Garage S3 buckets + per-instance keys for PG backups
- WAL archiving enabled and flowing to Garage (`wal-g wal-push`)
- Daily full backup cron at 3am (`/etc/cron.d/pg-backup`)
- Home Assistant added to Homepage (infrastructure, widget token)
- ShelfArr added to Homepage (diagnostic, MDI icon)
## Proxy migration → Zoraxy + clean-state recreate — audit 2026-05-16
### Done
- Reverse-proxy plan moved Traefik → **Zoraxy** v3.3.2 (`192.168.1.4:8000`, noauth, internal).
Traefik abandoned: it runs on a different host than the containers, so Docker labels
can't work cross-host; Zoraxy already had the wildcard `*.nuclide.systems` cert + rules.
- Source of truth: `proxy/zoraxy/routes.json` + idempotent CSRF-aware
`scripts/zoraxy_sync.py` (`--dry-run` default / `--apply` / `--prune` / `--list`).
Self-service: edit the JSON, run `--apply`.
- Zoraxy reconciled: fixed `ai`→14000 (litellm), `mcp`→8080 (mcp-gateway),
added `chat`→14001 (lobehub). Pruned (not exposed): dozzle, immich-tools,
mcp-auth. Internal-only: pgadmin, paperless, paperless-ai.
- Fixed AI URL configs (`litellm.nuclide.systems` had no route):
`ai/litellm-config/config.yaml`, `ai/.env`, `karakeep/.env` → `ai.nuclide.systems`;
`paperless_ai/.env` CUSTOM_BASE_URL `mcp`→`ai`.
- paperless-ngx host port `15000:8000` published (internal LAN); paperless-ai 15001 already in compose.
- Stripped **all** `traefik.*` labels from every compose file; fixed memos YAML escaping.
- Cleaned `proxy/` (removed dead Traefik artifacts) + added `proxy/zoraxy/README.md`.
- `shared-db/docker-compose.yml` restored from HEAD (WIP had deleted shared-postgres
+ `GARBAGE_` typos) and force-recreated cleanly.
- Removed stale `*_old` containers; clean-state force-recreate of all 19 stacks.
### Verified working (Zoraxy end-to-end, https via .4)
- ai, chat, mcp, arcane, ntfy, id, vault, abs, n8n, memos, hoarder, daytona, traccar — all serving.
- Static/external: ha, nc, ocpp, shepard, shepard-api — reachable.
- shared-postgres healthy; litellm/lobehub/mcp-gateway healthy & proxied; garage S3 working
internally (active wal-g WAL backups).
### Issues found → fix outcomes (2026-05-16)
- [x] **FIXED — immich_server crash-loop** (`getaddrinfo ENOTFOUND database`):
the `database` service was on `immich_immich_default`+`shared_backend` while
immich-server/redis/ml were on the implicit `immich_default` → no shared net.
Fix: `immich/docker-compose.yml` top-level network is now
`default: {name: immich_default}` and `database` joins `[default, shared_backend]`.
Verified: immich_server **healthy**, `immich.nuclide.systems` → **200** via Zoraxy.
- [x] **FIXED — vpn_gluetun "unhealthy"**: removed the bogus custom healthcheck
(`wget :8000`, 404s) so gluetun's built-in healthcheck (127.0.0.1:9999) runs.
Verified: `vpn_gluetun` **healthy**; arr-stack recreated (it uses
`network_mode: container:vpn_gluetun`) — prowlarr/rdtclient/shelfarr/flaresolverr healthy.
- [ ] **NOT fixed — garage external S3 (`s3.nuclide.systems`)**: tried
`api_bind_addr`/`rpc_bind_addr` `[::]`→`0.0.0.0`, but applying it needs a
force-recreate, and garage's single-node cluster is fragile to restarts — the
recreate caused "could not reach quorum" (RPC). **Reverted `garage.toml` to HEAD
and recreated**; garage is back to the known-good state (HEALTHY node, layout v1,
buckets OK, internal S3 / PG WAL backups working). External `s3.nuclide.systems`
remains non-responsive **as it was before** — tracked in Todos above.
- [ ] `nuc-ai-core-syncstack-1` left `Created` — tracked in Todos above.
### Other (pre-existing, NOT in the fix-1/2/3 scope, surfaced by clean recreate)
- `immich_machine_learning` shows "unhealthy" but **works** — its healthcheck tests
`localhost:3000/health` while ML serves `:3003` (immich_server logs confirm
"Machine learning server became healthy"). Cosmetic healthcheck-port mismatch in
the immich compose.
- `immich_power_tools`, `karakeep`, `karakeep_chrome` showing "unhealthy" after the
clean recreate — slow/long healthcheck warmup; revisit if still unhealthy later.
### Notes
- Zoraxy self-service: `python3 scripts/zoraxy_sync.py [--apply|--prune|--list]`.
- The shared-db WIP (garage/postgres rework) was reverted to HEAD — redo it intentionally
if that work was wanted.