# Todo

## Goal
Consolidate databases, migrate storage to UNAS, integrate all services with Homepage.

---

# Ideas (user)
- Claude Max sub integration with litellm (how to handle auth, rate limits, etc.?)
- MCP deployment process ideas? Current setup seems cumbersome — can litellm host directly?
  (how to integrate npx, uvx, and maybe docker-based containers?)
- Select offsite backup hosting provider, set up backup process (maybe restic to Backblaze B2
  or similar), then set up rclone for off-site sync of Garage buckets to remote S3
  (Backblaze B2, Jottacloud, or Hetzner)
- Research the LiteLLM MCP `source_url` column problem across GitHub issues, PRs, and docs
  to understand the full picture
- Add Google Gemini (vision + 1M ctx + free text-embedding-004; needs GEMINI_API_KEY)
- Add fal.ai MCP server (`mcp/fal`) for fast cloud image gen < 5 s (vs ~150 s local ComfyUI)
- Add Cerebras (Llama 3 at ~1M tok/s, generous free tier; CEREBRAS_API_KEY)

# Next Steps / Todos
- [x] **ComfyUI added & fixed (2026-05-17)** — root-caused the self-referential
  bind-mount, repaired mounts/healthcheck, migrated CPU→**Intel Arc iGPU**
  (torch 2.7.1+xpu, no IPEX), pinned v0.21.1. FLUX.1-schnell GGUF, legible
  text, ~94 s/image. Canonical workflow in `ai/comfyui/workflows/`.
- [x] **ComfyUI→LobeChat via MCP (2026-05-17)** — purpose-built `comfyui-mcp`
  server, proven end-to-end. See `docs/comfyui-mcp.md`.
- [x] **MCP gateway finished (2026-05-17)** — DinD, real streaming MCP
  reverse-proxy, Pocket ID bearer + RFC9728 discovery, OAuth callback fixed,
  secrets from `.env`, no port collisions. See `docs/mcp-gateway.md`.
- [x] **LobeChat S3 images fixed (2026-05-17)** — added `S3_PUBLIC_DOMAIN=http://192.168.1.40:10004`
  to `lobehub.yml`; pre-signed URLs now use the LAN-accessible Garage port instead of
  the internal `garage:3900` hostname.
- [x] **MCP gateway groups + combined endpoints (2026-05-17)** — servers now grouped
  (research/dev/personal/image); `/group/{name}/mcp` aggregates all member tools with
  `server__tool` namespacing; UI shows grouped cards with copy-URL buttons; `/` → `/ui`
  redirect added.
- [x] **LiteLLM Redis caching (2026-05-17)** — `cache: true` with Redis backend
  (`lobe-redis`, already on `ai-internal`); 10-min TTL on completion + embedding calls.
- [x] **Cohere added to LiteLLM (2026-05-17)** — `command-a-03-2025` (20 RPM),
  `cohere-embed-english/multilingual/light-v3` (2000 RPM); `COHERE_API_KEY` in `.env`.
  Note: command-r / command-r-plus retired 2025-09-15.
- [x] **LobeChat MCP import done** — `comfyui-flux` server imported via UI.
- [x] **ComfyUI MCP async queue (2026-05-17)** — full rewrite: `generate_image`
  returns job_id in <0.1 s; background polling thread updates state; `img2img`
  accepts many images (URL/base64/`job:<id>` chaining); `get_job_status`,
  `list_queue`, `cancel_job`, `list_recent_images`. New `flux-img2img-api.json`
  workflow (VAEEncode path). See `docs/comfyui-mcp.md`.
- [ ] Nextcloud MCP: bearer token enabled in NC admin (user_oidc provider settings).
  Gateway now provisions NC app-password at SSO login time (PocketID token live).
  **Needs end-to-end test**: log in via mcp.nuclide.systems, then call /nextcloud/mcp.
- [ ] Observability LXC: deploy OTEL collector + VictoriaMetrics + Grafana + Loki
  + Uptime Kuma on a Proxmox LXC; see `docs/stack-ideas.md §8`.
- [ ] Arcane: migrate central server to Proxmox LXC; deploy arcane-agent (headless)
  on NUC + .49; connect via TCP Docker socket or SSH Docker context.
- [ ] Docling + n8n ingest pipeline: Nextcloud/Paperless → Docling → LiteLLM embed
  → LobeChat knowledge base; see `docs/stack-ideas.md §1`.
- [x] **Gateway spawn servers (2026-05-17)** — papersearch, crawl4ai, markitdown
  all running on `ai-internal`. Fixed papersearch command (`uvx --with` not
  `uv tool run --from`). nextcloud deferred: Nextcloud OIDC discovery not
  enabled; needs `NEXTCLOUD_USERNAME`/`NEXTCLOUD_PASSWORD` in `ai/.env` for
  BasicAuth mode (server.py already updated).
- [x] **ComfyUI inpaint smoke-tested (2026-05-17)** — Kodak #23 (768×512) + center
  alpha-mask, submitted via API, output in 290s. Workflow confirmed working end-to-end.
- [x] **garage external S3 working** — `s3.nuclide.systems` returns 403 (AccessDenied
  on root, expected); Zoraxy routes correctly to `10004→3900`. Was already fixed by
  the earlier port binding change (`0.0.0.0:10004`); todo was stale.
- [x] **syncstack gone** — container no longer present; one-shot job completed/pruned.
- [x] **immich_machine_learning healthcheck fixed** — already on :3003/ping.
  Container shows healthy.
- [x] **LiteLLM healthcheck fixed** — was hitting host port 14000 inside container;
  corrected to container port 4000. Now healthy.
- [x] **immich_power_tools / karakeep / karakeep_chrome** — all healthy (slow warmup
  self-resolved).

## In Progress
- None

## Blocked
- None

---

# Reference / History

## Done
- Created `/opt/stacks/shared-db/` with shared-postgres (pg16), Garage S3, pgAdmin
- Consolidated daytona, litellm, paperless-ngx → shared-postgres
- LobeHub kept on own pg17 (ParadeDB)
- Switched LobeHub from RustFS → Garage S3 (`lobe-files` bucket)
- Migrated arr-stack configs + media to UNAS via rsync
- Migrated qdrant to UNAS (fresh start)
- Added 29 selfh.st icons, updated all homepage labels to `sh-` prefix
- Homepage: instance hrefs (192.168.1.40:PORT) + public hrefs (*.nuclide.systems)
- Homepage widgets for 11 services
- Groups consolidated from 14 → 6 (infrastructure, security, media, ai, productivity, diagnostic)
- Configured memos with Garage S3
- Fixed Garage port binding: `127.0.0.1:10004` → `0.0.0.0:10004` (Zoraxy couldn't reach it)
- Fixed memos S3 endpoint: `https://s3.nuclide.systems` → `http://garage:3900` (TLS cert mismatch)
- Cleaned up old local data remnant at `/opt/stacks/memos/data/`
- traccar GPS port fixed: 15001 → 19001 (paperless-ai port conflict)
- Updated PORTMAP.md, created storage.md and volumes.md
- WAL-G v3.0.8 setup for all 3 PG instances (shared-postgres, immich, lobe)
- Garage S3 buckets + per-instance keys for PG backups
- WAL archiving enabled and flowing to Garage (`wal-g wal-push`)
- Daily full backup cron at 3am (`/etc/cron.d/pg-backup`)
- Home Assistant added to Homepage (infrastructure, widget token)
- ShelfArr added to Homepage (diagnostic, MDI icon)

## Proxy migration → Zoraxy + clean-state recreate — audit 2026-05-16

### Done
- Reverse-proxy plan moved Traefik → **Zoraxy** v3.3.2 (`192.168.1.4:8000`, noauth, internal).
  Traefik abandoned: it runs on a different host than the containers, so Docker labels
  can't work cross-host; Zoraxy already had the wildcard `*.nuclide.systems` cert + rules.
- Source of truth: `proxy/zoraxy/routes.json` + idempotent CSRF-aware
  `scripts/zoraxy_sync.py` (`--dry-run` default / `--apply` / `--prune` / `--list`).
  Self-service: edit the JSON, run `--apply`.
- Zoraxy reconciled: fixed `ai`→14000 (litellm), `mcp`→8080 (mcp-gateway),
  added `chat`→14001 (lobehub). Pruned (not exposed): dozzle, immich-tools,
  mcp-auth. Internal-only: pgadmin, paperless, paperless-ai.
- Fixed AI URL configs (`litellm.nuclide.systems` had no route):
  `ai/litellm-config/config.yaml`, `ai/.env`, `karakeep/.env` → `ai.nuclide.systems`;
  `paperless_ai/.env` CUSTOM_BASE_URL `mcp`→`ai`.
- paperless-ngx host port `15000:8000` published (internal LAN); paperless-ai 15001 already in compose.
- Stripped **all** `traefik.*` labels from every compose file; fixed memos YAML escaping.
- Cleaned `proxy/` (removed dead Traefik artifacts) + added `proxy/zoraxy/README.md`.
- `shared-db/docker-compose.yml` restored from HEAD (WIP had deleted shared-postgres
  + `GARBAGE_` typos) and force-recreated cleanly.
- Removed stale `*_old` containers; clean-state force-recreate of all 19 stacks.

### Verified working (Zoraxy end-to-end, https via .4)
- ai, chat, mcp, arcane, ntfy, id, vault, abs, n8n, memos, hoarder, daytona, traccar — all serving.
- Static/external: ha, nc, ocpp, shepard, shepard-api — reachable.
- shared-postgres healthy; litellm/lobehub/mcp-gateway healthy & proxied; garage S3 working
  internally (active wal-g WAL backups).

### Issues found → fix outcomes (2026-05-16)
- [x] **FIXED — immich_server crash-loop** (`getaddrinfo ENOTFOUND database`):
  the `database` service was on `immich_immich_default`+`shared_backend` while
  immich-server/redis/ml were on the implicit `immich_default` → no shared net.
  Fix: `immich/docker-compose.yml` top-level network is now
  `default: {name: immich_default}` and `database` joins `[default, shared_backend]`.
  Verified: immich_server **healthy**, `immich.nuclide.systems` → **200** via Zoraxy.
- [x] **FIXED — vpn_gluetun "unhealthy"**: removed the bogus custom healthcheck
  (`wget :8000`, 404s) so gluetun's built-in healthcheck (127.0.0.1:9999) runs.
  Verified: `vpn_gluetun` **healthy**; arr-stack recreated (it uses
  `network_mode: container:vpn_gluetun`) — prowlarr/rdtclient/shelfarr/flaresolverr healthy.
- [ ] **NOT fixed — garage external S3 (`s3.nuclide.systems`)**: tried
  `api_bind_addr`/`rpc_bind_addr` `[::]`→`0.0.0.0`, but applying it needs a
  force-recreate, and garage's single-node cluster is fragile to restarts — the
  recreate caused "could not reach quorum" (RPC). **Reverted `garage.toml` to HEAD
  and recreated**; garage is back to the known-good state (HEALTHY node, layout v1,
  buckets OK, internal S3 / PG WAL backups working). External `s3.nuclide.systems`
  remains non-responsive **as it was before** — tracked in Todos above.
- [ ] `nuc-ai-core-syncstack-1` left `Created` — tracked in Todos above.

### Other (pre-existing, NOT in the fix-1/2/3 scope, surfaced by clean recreate)
- `immich_machine_learning` shows "unhealthy" but **works** — its healthcheck tests
  `localhost:3000/health` while ML serves `:3003` (immich_server logs confirm
  "Machine learning server became healthy"). Cosmetic healthcheck-port mismatch in
  the immich compose.
- `immich_power_tools`, `karakeep`, `karakeep_chrome` showing "unhealthy" after the
  clean recreate — slow/long healthcheck warmup; revisit if still unhealthy later.

### Notes
- Zoraxy self-service: `python3 scripts/zoraxy_sync.py [--apply|--prune|--list]`.
- The shared-db WIP (garage/postgres rework) was reverted to HEAD — redo it intentionally
  if that work was wanted.