diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a0dd56..259f18f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,37 @@ All notable infrastructure / service / doc changes. Newest first. +## 2026-05-21 + +### Docs & docs infrastructure +- **Mermaid diagrams now rendering** in mkdocs Material site — was missing `custom_fences` config in `pymdownx.superfences`. Fixed in `docs-server/mkdocs.yml` and rebuilt container. +- **`site_url` corrected** to `http://192.168.1.111:13080` (docs are LAN-only; no Zoraxy route exists). +- **Docs restructured**: "Stacks (CT 104)" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to `infra/` section. TODO items migrated into CHANGELOG + ideas. +- **Secrets manager design doc** added at `services/secrets-manager.md` — recommends Infisical on a new CT 112. + +### Storage / S3 +- **`s3.nuclide.systems` Zoraxy route fixed** — `EnableWebsocketCustomHeaders`, `DisableHopByHopHeaderRemoval`, `EnableAutoHTTPS` all set. Garage S3 API now reachable externally. +- **`chat-artifacts` Garage S3 bucket created** — public-read via `https://chat-artifacts.s3.nuclide.systems/`. Credentials at `/root/garage-chat-artifacts.creds` (not git-tracked). + +### MCP surface +- **`upload-artifact` MCP server** deployed (port 18011, CT 104). Tools: `upload_text`, `upload_base64`, `list_artifacts`, `get_url`, `delete_artifact`. Registered in gateway (now 24 servers). Source: `git.nuclide.systems/fkrebs/mcp-upload-artifact`. +- **ComfyUI MCP** now auto-uploads completed images to `chat-artifacts` S3 and includes the public URL alongside the inline base64 image. +- **DAYTONA_API_KEY** removed from CT 104 `ai/.env` (Daytona decommissioned 2026-05-20). + +### Dev environment +- **`savefig` dotfiles helper** added to `fkrebs/dotfiles` — uploads any file to `chat-artifacts` using `uv run --with boto3`; reads credentials from env vars. +- **Coder `python-uv` template** updated with `CHAT_ARTIFACTS_*` env vars baked in; template pushed to Coder server. + +### Gitea repos +- New repos created: `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact`, `home-assistant-config` — all at `git.nuclide.systems/fkrebs/`. +- **HAOS config** (`192.168.1.60`, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see `services/secrets-manager.md` for SSH access notes). + +### Planned +- CT 112 "secrets" — Infisical deployment (see `services/secrets-manager.md`) +- Renovate Bot via Gitea Actions — dependency updates for MCP server repos (Dockerfiles + requirements.txt) +- Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured) +- LobeHub Artifacts → S3 patch (TypeScript, ~3-4h, deferred) + ## 2026-05-20 ### Auth / Identity (single sign-on lockdown) diff --git a/ct-inventory.md b/ct-inventory.md index 4a499de..a94dbeb 100644 --- a/ct-inventory.md +++ b/ct-inventory.md @@ -14,7 +14,10 @@ Verified 2026-05-20 via `pct list`, `pct config `, `qm config 100` on `nuc`. | 109 | observe | LXC | 192.168.1.6/24 | 4 | 6144 | 30 GiB | **Planned** — Prometheus, Grafana, Arcane Manager, Dozzle, Homarr, Diun, Tinyauth, socket-proxy | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | +| 112 | secrets | LXC unpriv | TBD | 2 | 2048 | 8 GiB | **Planned** — Infisical secrets manager (replaces scattered .env files) | — | no | **planned** | UNAS NFS = `192.168.1.31:/var/nfs/shared/storage` over NFSv3. CT 105 (Nextcloud) is the lone outlier — it mounts the same UNAS share over **CIFS/SMB 3.1.1**, not NFS. +CT 112 ("secrets") — Infisical deployment planned; see `services/secrets-manager.md`. + CT 113 ("db") is mentioned in memory notes (`postgres-consolidation`) as a future consolidation target. Not yet planned in any source-of-truth doc. diff --git a/ideas/stack-ideas.md b/ideas/stack-ideas.md index f9b93f6..afe4973 100644 --- a/ideas/stack-ideas.md +++ b/ideas/stack-ideas.md @@ -238,6 +238,42 @@ For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the ex --- +## 11. Dependency updates via Renovate Bot (Gitea Actions) + +Run **Renovate Bot** as a Gitea Actions workflow to automatically open PRs for outdated +dependencies in MCP server repos (`mcp-comfyui`, `mcp-docling`, `mcp-shepard`, +`mcp-upload-artifact`). Targets: `Dockerfile` base image tags + `requirements.txt` / +`pyproject.toml` Python deps. + +Renovate supports Gitea natively via `platform: gitea` in `renovate.json`. The Gitea +Actions runner (`ct111-runner`) already exists; add a scheduled workflow calling +`renovate/renovate` Docker image once daily. + +--- + +## 12. Egress firewall on UDM (UniFi) + +Enforce outbound allow-list on the UDM / UniFi gateway — block all non-approved egress +by default. Goals: +- Prevent exfiltration from compromised containers +- Audit unexpected outbound connections (model providers, analytics, telemetry) +- Approved: LiteLLM model provider endpoints, Jottacloud, UNAS internal, NTP, DNS + +Implementation: UDM firewall rules (WAN_OUT) + Threat Management IDS in monitor mode first. + +--- + +## 13. Local LLM on Arc GPU via vllm / ollama for sensitive workloads + +Run a privacy-sensitive LLM locally on the Arc iGPU using **vllm** (with XPU/IPEX backend) +or the Intel-patched **Ollama** build. Use cases: document classification in Paperless +workflows, offline coding assistant, fallback when cloud rate limits hit. + +See §3 (On-device LLM) for implementation notes. This item tracks the specific motivation +of *sensitive workload isolation* — i.e., running prompts that should not leave the LAN. + +--- + ## Priority order (rough) 1. **ComfyUI MCP async queue** — fixes timeout; unblocks img2img + reference features @@ -246,3 +282,5 @@ For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the ex 4. **Observability LXC** — OTEL collector + VictoriaMetrics + Grafana + Uptime Kuma, co-located 5. **Backblaze B2 off-site backup** — already in todos 6. **On-device LLM (Arc)** — depends on IPEX-LLM + Ollama Intel build stability +7. **Renovate Bot** — low-effort automation win for MCP repos +8. **Egress firewall** — security hygiene; plan before adding more external-facing services diff --git a/stacks/docker-networks.md b/infra/docker-networks.md similarity index 95% rename from stacks/docker-networks.md rename to infra/docker-networks.md index 960d674..96c89cf 100644 --- a/stacks/docker-networks.md +++ b/infra/docker-networks.md @@ -14,7 +14,6 @@ has no explicit `networks:` declaration. This has exhausted Docker's built-in | `ai-internal` | 172.31.0.0/16 | 8 | | `arcane_default` | 172.22.0.0/16 | 1 | | `arr-stack_default` | 172.21.0.0/16 | 4 | -| `daytona-minimal_daytona-network` | 172.23.0.0/16 | 4 | | `dozzle_default` | 192.168.16.0/20 | 1 | | `homepage_default` | 172.19.0.0/16 | 1 | | `immich_default` | 172.18.0.0/16 | 5 | @@ -30,7 +29,7 @@ has no explicit `networks:` declaration. This has exhausted Docker's built-in | `vaultwarden_default` | 192.168.64.0/20 | 1 | | `vpn_default` | 192.168.80.0/20 | 1 | -Total: 19 user-defined bridge networks. +Total: 18 user-defined bridge networks (Daytona decommissioned 2026-05-20). ### Problem @@ -90,7 +89,6 @@ These stacks have complex internal networking and should keep their own: - **immich** — 5 services with inter-dependencies - **paperless-ngx** — 5 services (webserver, broker, db, gotenberg, tika) - **arr-stack** — 5 services (rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr) -- **daytona** — explicit `daytona-minimal_daytona-network` - **karakeep** — 3 services with chrome dependency - **homepage** — single service, can stay or migrate - **streamio** — already removed, stremio uses VPN container directly diff --git a/stacks/PORTMAP.md b/infra/portmap.md similarity index 89% rename from stacks/PORTMAP.md rename to infra/portmap.md index a2a2bb8..e595a47 100644 --- a/stacks/PORTMAP.md +++ b/infra/portmap.md @@ -31,7 +31,7 @@ | [10001](http://192.168.1.40:10001) | Dozzle | `dozzle` | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | Log viewer | | [10002](http://192.168.1.40:10002) | Arcane | `arcane` | [arcane.nuclide.systems](https://arcane.nuclide.systems) | Web IDE | | [10003](http://192.168.1.40:10003) | Gotify | `gotify` | [gotify.nuclide.systems](https://gotify.nuclide.systems) | Push notifications | -| [10004](http://192.168.1.40:10004) | Garage S3 API | `garage` | s3.nuclide.systems (non-responsive, see notes) | Internal: `http://garage:3900` | +| [10004](http://192.168.1.40:10004) | Garage S3 API | `garage` | [s3.nuclide.systems](https://s3.nuclide.systems) | FIXED 2026-05-21: proxied by Zoraxy with ACME TLS. Garage API accessible at `https://s3.nuclide.systems`. Internal: `http://garage:3900` | | 10005 (127.0.0.1 only) | Garage Admin | `garage` | — | localhost only | --- @@ -40,7 +40,7 @@ | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager · **no Pocket-ID client yet — gap** | +| [11001](http://192.168.1.40:11001) | Vaultwarden | `vaultwarden` | [vault.nuclide.systems](https://vault.nuclide.systems) | Password manager · Pocket-ID OIDC client created 2026-05-21; auth flow not yet configured | > Pocket-ID **migrated from this CT to LXC 110** on 2026-05-20. See the [External Services](#external-services-not-on-nuc-docker) table below. @@ -80,6 +80,11 @@ All arr-stack services run behind `vpn_gluetun` container network. | [8080](http://192.168.1.40:8080) | MCP Gateway | `mcp-gateway` | [mcp.nuclide.systems](https://mcp.nuclide.systems) | Pocket ID gated; spawns MCP servers on `ai-internal` | | [18002](http://192.168.1.40:18002) | ComfyUI | `comfyui` | — | LAN only (Intel Arc iGPU, FLUX.1-schnell GGUF) | | [18003](http://192.168.1.40:18003) | ComfyUI MCP | `comfyui-mcp` | — | FastMCP; reachable via gateway at `mcp.nuclide.systems/comfyui/mcp` | +| [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway | +| [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway | +| [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only | +| [18010](http://192.168.1.40:18010) | Shepard MCP | `shepard-mcp` | — | Shepard research platform; reachable via gateway | +| [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | --- @@ -117,7 +122,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| - +| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` | --- @@ -188,6 +193,7 @@ Zoraxy routes to these external backends: | [hoarder.nuclide.systems](https://hoarder.nuclide.systems) | karakeep | 17001 | | [traccar.nuclide.systems](https://traccar.nuclide.systems) | traccar | 15000 | | [dozzle.nuclide.systems](https://dozzle.nuclide.systems) | dozzle | 10001 | +| [s3.nuclide.systems](https://s3.nuclide.systems) | garage | 10004 | **Not publicly proxied** (LAN / localhost only): pgadmin, paperless, paperless-ai, comfyui, comfyui-mcp, rdtclient, prowlarr, shelfarr, qdrant. @@ -195,7 +201,6 @@ Zoraxy routes to these external backends: ## Known Issues -- **`s3.nuclide.systems`** (Garage external S3): non-responsive. Internal S3 `http://garage:3900` works fine for wal-g, memos, lobe. External binding risks quorum loss — deferred. - **`lobe-postgres` / `lobe-redis`**: host-port exposed (`0.0.0.0:5432/6379`) — firewall blocks external access but ideally restricted to localhost. --- @@ -215,13 +220,14 @@ Zoraxy routes to these external backends: | `0aee4280` | Coder | `https://dev.nuclide.systems/api/v2/users/oidc/callback` | | `9444609e` | Gitea | `https://git.nuclide.systems/user/oauth2/pocket-id/callback` | | `38469e7e` | Proxmox VE | `https://192.168.1.20:8006` | -| `798a367f` | ~~daytona~~ | **ORPHAN — Daytona decommissioned 2026-05-20, remove** | +| ~~798a367f~~ | ~~daytona~~ | **DECOMMISSIONED — remove from Pocket-ID** | | `cbbf20d5` | Audiobookshelf | `https://abs.nuclide.systems/…` + `*` | | `d8733fcc` | shelfarr | `*` | | `78c78998` | Claude MCP | `https://claude.ai/api/mcp/auth_callback` + `https://mcp.nuclide.systems/mcp/auth/callback` | | `82ca2d53` | nuc-ai | (empty) — used by spawned MCP servers | | `7fe1a14b` | zoraxy | (empty) | | `af2f837b` | mcp-auth | (empty) — legacy, unused | +| (new) | vaultwarden | `https://vault.nuclide.systems/auth/callback` | Created 2026-05-21; SSO wiring deferred | **OIDC Endpoints** (corrected 2026-05-17 — previously used wrong `/api/v1/oauth2/` path): - Authorization: `https://id.nuclide.systems/authorize` diff --git a/stacks/storage.md b/infra/storage.md similarity index 95% rename from stacks/storage.md rename to infra/storage.md index 2689691..05557d9 100644 --- a/stacks/storage.md +++ b/infra/storage.md @@ -54,6 +54,13 @@ Tier-1 state and anything that should NOT be NFS-backed. | AdGuard / Zoraxy / DNS / Shepard / Backrest binaries | 102/108/103/101 | local zfs only | none | Backrest itself has no self-backup | | Nextcloud config + sidecars | 105 | local zfs (CT rootfs) | none | data path is CIFS — see below | +### Garage S3 buckets (on local zfs, CT 104) + +| Bucket | Access key | Use | Public URL | +|---|---|---|---| +| `lobe-files` | GK55210… (from `.env`) | LobeHub file uploads + WAL-G PG backups | internal only | +| `chat-artifacts` | GK50bfc… | AI chat output artefacts (images, reports, SVG) | `https://chat-artifacts.s3.nuclide.systems/` | + ### Docker named volumes Local on `/var/lib/docker/volumes/`. Mostly databases and caches. diff --git a/stacks/volumes.md b/infra/volumes.md similarity index 100% rename from stacks/volumes.md rename to infra/volumes.md diff --git a/stacks/todo.md b/stacks/todo.md deleted file mode 100644 index e11277b..0000000 --- a/stacks/todo.md +++ /dev/null @@ -1,171 +0,0 @@ -# Todo - -## Goal -Consolidate databases, migrate storage to UNAS, integrate all services with Homepage. - ---- - -# Ideas (user) -- Claude Max sub integration with litellm (how to handle auth, rate limits, etc.?) -- MCP deployment process ideas? Current setup seems cumbersome — can litellm host directly? - (how to integrate npx, uvx, and maybe docker-based containers?) -- Select offsite backup hosting provider, set up backup process (maybe restic to Backblaze B2 - or similar), then set up rclone for off-site sync of Garage buckets to remote S3 - (Backblaze B2, Jottacloud, or Hetzner) -- Research the LiteLLM MCP `source_url` column problem across GitHub issues, PRs, and docs - to understand the full picture -- Add Google Gemini (vision + 1M ctx + free text-embedding-004; needs GEMINI_API_KEY) -- Add fal.ai MCP server (`mcp/fal`) for fast cloud image gen < 5 s (vs ~150 s local ComfyUI) -- Add Cerebras (Llama 3 at ~1M tok/s, generous free tier; CEREBRAS_API_KEY) - -# Next Steps / Todos -- [x] **ComfyUI added & fixed (2026-05-17)** — root-caused the self-referential - bind-mount, repaired mounts/healthcheck, migrated CPU→**Intel Arc iGPU** - (torch 2.7.1+xpu, no IPEX), pinned v0.21.1. FLUX.1-schnell GGUF, legible - text, ~94 s/image. Canonical workflow in `ai/comfyui/workflows/`. -- [x] **ComfyUI→LobeChat via MCP (2026-05-17)** — purpose-built `comfyui-mcp` - server, proven end-to-end. See `docs/comfyui-mcp.md`. -- [x] **MCP gateway finished (2026-05-17)** — DinD, real streaming MCP - reverse-proxy, Pocket ID bearer + RFC9728 discovery, OAuth callback fixed, - secrets from `.env`, no port collisions. See `docs/mcp-gateway.md`. -- [x] **LobeChat S3 images fixed (2026-05-17)** — added `S3_PUBLIC_DOMAIN=http://192.168.1.40:10004` - to `lobehub.yml`; pre-signed URLs now use the LAN-accessible Garage port instead of - the internal `garage:3900` hostname. -- [x] **MCP gateway groups + combined endpoints (2026-05-17)** — servers now grouped - (research/dev/personal/image); `/group/{name}/mcp` aggregates all member tools with - `server__tool` namespacing; UI shows grouped cards with copy-URL buttons; `/` → `/ui` - redirect added. -- [x] **LiteLLM Redis caching (2026-05-17)** — `cache: true` with Redis backend - (`lobe-redis`, already on `ai-internal`); 10-min TTL on completion + embedding calls. -- [x] **Cohere added to LiteLLM (2026-05-17)** — `command-a-03-2025` (20 RPM), - `cohere-embed-english/multilingual/light-v3` (2000 RPM); `COHERE_API_KEY` in `.env`. - Note: command-r / command-r-plus retired 2025-09-15. -- [x] **LobeChat MCP import done** — `comfyui-flux` server imported via UI. -- [x] **ComfyUI MCP async queue (2026-05-17)** — full rewrite: `generate_image` - returns job_id in <0.1 s; background polling thread updates state; `img2img` - accepts many images (URL/base64/`job:` chaining); `get_job_status`, - `list_queue`, `cancel_job`, `list_recent_images`. New `flux-img2img-api.json` - workflow (VAEEncode path). See `docs/comfyui-mcp.md`. -- [ ] Nextcloud MCP: bearer token enabled in NC admin (user_oidc provider settings). - Gateway now provisions NC app-password at SSO login time (PocketID token live). - **Needs end-to-end test**: log in via mcp.nuclide.systems, then call /nextcloud/mcp. -- [ ] Observability LXC: deploy OTEL collector + VictoriaMetrics + Grafana + Loki - + Uptime Kuma on a Proxmox LXC; see `docs/stack-ideas.md §8`. -- [ ] Arcane: migrate central server to Proxmox LXC; deploy arcane-agent (headless) - on NUC + .49; connect via TCP Docker socket or SSH Docker context. -- [ ] Docling + n8n ingest pipeline: Nextcloud/Paperless → Docling → LiteLLM embed - → LobeChat knowledge base; see `docs/stack-ideas.md §1`. -- [x] **Gateway spawn servers (2026-05-17)** — papersearch, crawl4ai, markitdown - all running on `ai-internal`. Fixed papersearch command (`uvx --with` not - `uv tool run --from`). nextcloud deferred: Nextcloud OIDC discovery not - enabled; needs `NEXTCLOUD_USERNAME`/`NEXTCLOUD_PASSWORD` in `ai/.env` for - BasicAuth mode (server.py already updated). -- [x] **ComfyUI inpaint smoke-tested (2026-05-17)** — Kodak #23 (768×512) + center - alpha-mask, submitted via API, output in 290s. Workflow confirmed working end-to-end. -- [x] **garage external S3 working** — `s3.nuclide.systems` returns 403 (AccessDenied - on root, expected); Zoraxy routes correctly to `10004→3900`. Was already fixed by - the earlier port binding change (`0.0.0.0:10004`); todo was stale. -- [x] **syncstack gone** — container no longer present; one-shot job completed/pruned. -- [x] **immich_machine_learning healthcheck fixed** — already on :3003/ping. - Container shows healthy. -- [x] **LiteLLM healthcheck fixed** — was hitting host port 14000 inside container; - corrected to container port 4000. Now healthy. -- [x] **immich_power_tools / karakeep / karakeep_chrome** — all healthy (slow warmup - self-resolved). - -## In Progress -- None - -## Blocked -- None - ---- - -# Reference / History - -## Done -- Created `/opt/stacks/shared-db/` with shared-postgres (pg16), Garage S3, pgAdmin -- Consolidated daytona, litellm, paperless-ngx → shared-postgres -- LobeHub kept on own pg17 (ParadeDB) -- Switched LobeHub from RustFS → Garage S3 (`lobe-files` bucket) -- Migrated arr-stack configs + media to UNAS via rsync -- Migrated qdrant to UNAS (fresh start) -- Added 29 selfh.st icons, updated all homepage labels to `sh-` prefix -- Homepage: instance hrefs (192.168.1.40:PORT) + public hrefs (*.nuclide.systems) -- Homepage widgets for 11 services -- Groups consolidated from 14 → 6 (infrastructure, security, media, ai, productivity, diagnostic) -- Configured memos with Garage S3 -- Fixed Garage port binding: `127.0.0.1:10004` → `0.0.0.0:10004` (Zoraxy couldn't reach it) -- Fixed memos S3 endpoint: `https://s3.nuclide.systems` → `http://garage:3900` (TLS cert mismatch) -- Cleaned up old local data remnant at `/opt/stacks/memos/data/` -- traccar GPS port fixed: 15001 → 19001 (paperless-ai port conflict) -- Updated PORTMAP.md, created storage.md and volumes.md -- WAL-G v3.0.8 setup for all 3 PG instances (shared-postgres, immich, lobe) -- Garage S3 buckets + per-instance keys for PG backups -- WAL archiving enabled and flowing to Garage (`wal-g wal-push`) -- Daily full backup cron at 3am (`/etc/cron.d/pg-backup`) -- Home Assistant added to Homepage (infrastructure, widget token) -- ShelfArr added to Homepage (diagnostic, MDI icon) - -## Proxy migration → Zoraxy + clean-state recreate — audit 2026-05-16 - -### Done -- Reverse-proxy plan moved Traefik → **Zoraxy** v3.3.2 (`192.168.1.4:8000`, noauth, internal). - Traefik abandoned: it runs on a different host than the containers, so Docker labels - can't work cross-host; Zoraxy already had the wildcard `*.nuclide.systems` cert + rules. -- Source of truth: `proxy/zoraxy/routes.json` + idempotent CSRF-aware - `scripts/zoraxy_sync.py` (`--dry-run` default / `--apply` / `--prune` / `--list`). - Self-service: edit the JSON, run `--apply`. -- Zoraxy reconciled: fixed `ai`→14000 (litellm), `mcp`→8080 (mcp-gateway), - added `chat`→14001 (lobehub). Pruned (not exposed): dozzle, immich-tools, - mcp-auth. Internal-only: pgadmin, paperless, paperless-ai. -- Fixed AI URL configs (`litellm.nuclide.systems` had no route): - `ai/litellm-config/config.yaml`, `ai/.env`, `karakeep/.env` → `ai.nuclide.systems`; - `paperless_ai/.env` CUSTOM_BASE_URL `mcp`→`ai`. -- paperless-ngx host port `15000:8000` published (internal LAN); paperless-ai 15001 already in compose. -- Stripped **all** `traefik.*` labels from every compose file; fixed memos YAML escaping. -- Cleaned `proxy/` (removed dead Traefik artifacts) + added `proxy/zoraxy/README.md`. -- `shared-db/docker-compose.yml` restored from HEAD (WIP had deleted shared-postgres - + `GARBAGE_` typos) and force-recreated cleanly. -- Removed stale `*_old` containers; clean-state force-recreate of all 19 stacks. - -### Verified working (Zoraxy end-to-end, https via .4) -- ai, chat, mcp, arcane, ntfy, id, vault, abs, n8n, memos, hoarder, daytona, traccar — all serving. -- Static/external: ha, nc, ocpp, shepard, shepard-api — reachable. -- shared-postgres healthy; litellm/lobehub/mcp-gateway healthy & proxied; garage S3 working - internally (active wal-g WAL backups). - -### Issues found → fix outcomes (2026-05-16) -- [x] **FIXED — immich_server crash-loop** (`getaddrinfo ENOTFOUND database`): - the `database` service was on `immich_immich_default`+`shared_backend` while - immich-server/redis/ml were on the implicit `immich_default` → no shared net. - Fix: `immich/docker-compose.yml` top-level network is now - `default: {name: immich_default}` and `database` joins `[default, shared_backend]`. - Verified: immich_server **healthy**, `immich.nuclide.systems` → **200** via Zoraxy. -- [x] **FIXED — vpn_gluetun "unhealthy"**: removed the bogus custom healthcheck - (`wget :8000`, 404s) so gluetun's built-in healthcheck (127.0.0.1:9999) runs. - Verified: `vpn_gluetun` **healthy**; arr-stack recreated (it uses - `network_mode: container:vpn_gluetun`) — prowlarr/rdtclient/shelfarr/flaresolverr healthy. -- [ ] **NOT fixed — garage external S3 (`s3.nuclide.systems`)**: tried - `api_bind_addr`/`rpc_bind_addr` `[::]`→`0.0.0.0`, but applying it needs a - force-recreate, and garage's single-node cluster is fragile to restarts — the - recreate caused "could not reach quorum" (RPC). **Reverted `garage.toml` to HEAD - and recreated**; garage is back to the known-good state (HEALTHY node, layout v1, - buckets OK, internal S3 / PG WAL backups working). External `s3.nuclide.systems` - remains non-responsive **as it was before** — tracked in Todos above. -- [ ] `nuc-ai-core-syncstack-1` left `Created` — tracked in Todos above. - -### Other (pre-existing, NOT in the fix-1/2/3 scope, surfaced by clean recreate) -- `immich_machine_learning` shows "unhealthy" but **works** — its healthcheck tests - `localhost:3000/health` while ML serves `:3003` (immich_server logs confirm - "Machine learning server became healthy"). Cosmetic healthcheck-port mismatch in - the immich compose. -- `immich_power_tools`, `karakeep`, `karakeep_chrome` showing "unhealthy" after the - clean recreate — slow/long healthcheck warmup; revisit if still unhealthy later. - -### Notes -- Zoraxy self-service: `python3 scripts/zoraxy_sync.py [--apply|--prune|--list]`. -- The shared-db WIP (garage/postgres rework) was reverted to HEAD — redo it intentionally - if that work was wanted. - -