From ae20382cbe4c3351cd899f4f7964d2dbd8a1af67 Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Thu, 28 May 2026 21:47:50 +0200 Subject: [PATCH] ops: sunset LiteLLM, add owui to Backrest, drop stale DB+role MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - LiteLLM sunset 2026-05-28: container removed, config dir renamed .DECOMMISSIONED-2026-05-28, prometheus scrape job deleted, CT 113 litellm DB + role dropped. Bifrost now connects directly to upstream LLM providers (verified: openai → SAIA chat-ai.academiccloud.de). - Open WebUI data (CT 104 uploads + vector_db, 192K) now staged to UNAS /services/open-webui via openwebui-prestage.sh and included in Backrest services-backup-plan. Open WebUI chats/users live in CT 113 Postgres owui DB (already in WAL-G). - Miniflux confirmed DB-only — already covered by WAL-G. - backrest.md and homelab-architecture.md updated; services-overview.md marks LiteLLM row strikethrough. --- services-overview.md | 2 +- services/backrest.md | 4 +++- services/homelab-architecture.md | 16 +++++++--------- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/services-overview.md b/services-overview.md index 7f5daf9..5f91865 100644 --- a/services-overview.md +++ b/services-overview.md @@ -10,7 +10,7 @@ Every service running in the homelab, with its access URL(s) and host. **Externa |---|---|---|---|---| | Bifrost (LLM/MCP gateway) | | | CT 104 | — | | Open WebUI | | | CT 104 | — | -| LiteLLM | — | (internal only) | CT 104 | — | +| ~~LiteLLM~~ | — | — | ~~CT 104~~ | SUNSET 2026-05-28 | | ComfyUI | — | | CT 104 | [[comfyui]] | | Nexa | — | — | CT 104 | [[nexa]] | | n8n (automation) | — | | CT 104 | — | diff --git a/services/backrest.md b/services/backrest.md index 57f8ec6..5b2ea8f 100644 --- a/services/backrest.md +++ b/services/backrest.md @@ -48,6 +48,7 @@ media/documents/public/paperless-ngx - `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS - `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage` - `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n` +- `openwebui-prestage.sh` — rsyncs CT 104 Open WebUI uploads + vector_db → `services/open-webui` (cache/ excluded) Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28. @@ -62,7 +63,7 @@ Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmi - `archive_mode = on`, `archive_command = 'wal-g wal-push %p'` - Daily `wal-g backup-push` cron at 02:00 -- DBs hosted: `vaultwarden`, `paperless`, `litellm` (decommissioned but DB retained), `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux` +- DBs hosted: `vaultwarden`, `paperless`, `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux` (`litellm` DB + role dropped 2026-05-28 when LiteLLM was sunset) - `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy - WAL-G monitor (CT 103) checks this via `pg_stat_archiver` @@ -87,6 +88,7 @@ Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sb | CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud | | CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud | | CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud | +| CT 104 Open WebUI uploads + vector_db | `openwebui-prestage.sh` rsync → UNAS `/services/open-webui` → `services-backup-plan` | ✓ JottaCloud | | Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud | | CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ | | CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ | diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index 36aa801..cfdfc87 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -150,13 +150,11 @@ The MCP gateway bridges `ai-internal` + `shared_backend`. ## AI / Agent platform (the core, `/opt/stacks/ai`) - **Bifrost** (`ai.nuclide.systems`, :14003) — unified LLM gateway + MCP host. Stack `ai/bifrost/`. Auth via `sk-bf-` VKs. LLM inference at `/v1`; MCP at - `/mcp` (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to - LiteLLM internally. -- **LiteLLM** (:14000, **internal only** — no Zoraxy route) — model gateway, - backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal - **failovers** + `num_retries`, Redis completion cache, Gemini €10/30d - `provider_budget_config`. Config: `ai/litellm-config/config.yaml` (+ DB - overlay `LiteLLM_Config`). Reachable as `http://litellm:4000` inside Docker. + `/mcp` (29 clients, ~760 tools as of 2026-05-26). Connects directly to + upstream LLM providers; no LiteLLM hop in the chain anymore. +- ~~**LiteLLM**~~ — **SUNSET 2026-05-28**. Container removed, DB + role dropped on CT 113, + config dir renamed `litellm-config.DECOMMISSIONED-2026-05-28`, Prometheus + scrape job deleted. Bifrost now handles all LLM gateway responsibilities. - **Open WebUI** (`chat.nuclide.systems`, :14002) — chat UI. Stack `ai/open-webui.yml`. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not yet wired as of 2026-05-26). @@ -184,7 +182,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`. ## Data layer - **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the - **primary/standard DB for all deployments**. Tenants: LiteLLM, paperless, + **primary/standard DB for all deployments**. Tenants: paperless, **memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser `postgres` via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's `.env`). Tuned 2026-05-19 (3G limit, @@ -206,7 +204,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`. **pgloader data-only of ALL tables** into the app-built schema (exclude only the app's own migration-tracking table), app role temp-SUPERUSER for the load (FK/trigger disable) then reverted. -- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). **lobe-redis** — LiteLLM cache (LobeChat decommissioned). **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` / +- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). ~~**lobe-redis**~~ — was LiteLLM cache; both LobeChat and LiteLLM decommissioned. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` / `:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector, on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal). - **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,