ops: sunset LiteLLM, add owui to Backrest, drop stale DB+role
- LiteLLM sunset 2026-05-28: container removed, config dir renamed .DECOMMISSIONED-2026-05-28, prometheus scrape job deleted, CT 113 litellm DB + role dropped. Bifrost now connects directly to upstream LLM providers (verified: openai → SAIA chat-ai.academiccloud.de). - Open WebUI data (CT 104 uploads + vector_db, 192K) now staged to UNAS /services/open-webui via openwebui-prestage.sh and included in Backrest services-backup-plan. Open WebUI chats/users live in CT 113 Postgres owui DB (already in WAL-G). - Miniflux confirmed DB-only — already covered by WAL-G. - backrest.md and homelab-architecture.md updated; services-overview.md marks LiteLLM row strikethrough.
This commit is contained in:
@@ -10,7 +10,7 @@ Every service running in the homelab, with its access URL(s) and host. **Externa
|
||||
|---|---|---|---|---|
|
||||
| Bifrost (LLM/MCP gateway) | <https://ai.nuclide.systems> | <http://192.168.1.40:14003> | CT 104 | — |
|
||||
| Open WebUI | <https://chat.nuclide.systems> | <http://192.168.1.40:14002> | CT 104 | — |
|
||||
| LiteLLM | — | <http://192.168.1.40:14000> (internal only) | CT 104 | — |
|
||||
| ~~LiteLLM~~ | — | — | ~~CT 104~~ | SUNSET 2026-05-28 |
|
||||
| ComfyUI | — | <http://192.168.1.40:18188> | CT 104 | [[comfyui]] |
|
||||
| Nexa | — | — | CT 104 | [[nexa]] |
|
||||
| n8n (automation) | — | <http://192.168.1.40:15678> | CT 104 | — |
|
||||
|
||||
@@ -48,6 +48,7 @@ media/documents/public/paperless-ngx
|
||||
- `pocketid-prestage.sh` — exports Pocket-ID zip + keys from CT 109 → UNAS
|
||||
- `garage-prestage.sh` — rsyncs CT 104 Garage S3 data/meta → `services/shared-db-garage`
|
||||
- `n8n-prestage.sh` — rsyncs CT 104 n8n config/nodes/storage → `services/n8n`
|
||||
- `openwebui-prestage.sh` — rsyncs CT 104 Open WebUI uploads + vector_db → `services/open-webui` (cache/ excluded)
|
||||
|
||||
Hook scripts at `/opt/backrest/scripts/` on CT 103. CT 103 SSH key deployed to CT 104 + CT 109 2026-05-28.
|
||||
|
||||
@@ -62,7 +63,7 @@ Dedicated `db` LXC at `192.168.1.6:5432` running `postgres:17` in Docker, pgAdmi
|
||||
|
||||
- `archive_mode = on`, `archive_command = 'wal-g wal-push %p'`
|
||||
- Daily `wal-g backup-push` cron at 02:00
|
||||
- DBs hosted: `vaultwarden`, `paperless`, `litellm` (decommissioned but DB retained), `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux`
|
||||
- DBs hosted: `vaultwarden`, `paperless`, `memos`, `n8n`, `gitea`, `coder`, `owui`, `miniflux` (`litellm` DB + role dropped 2026-05-28 when LiteLLM was sunset)
|
||||
- `failed_count=96` historical (pre-2026-05-21 setup); current archive is healthy
|
||||
- WAL-G monitor (CT 103) checks this via `pg_stat_archiver`
|
||||
|
||||
@@ -87,6 +88,7 @@ Garage → JottaCloud offsite sync runs daily 02:30 on CT 103 via `/usr/local/sb
|
||||
| CT 104 Vaultwarden data | moved to UNAS `/services/vaultwarden` → `services-backup-plan` | ✓ JottaCloud |
|
||||
| CT 104 Garage S3 data | `garage-prestage.sh` rsync → UNAS `/services/shared-db-garage` → `services-backup-plan` | ✓ JottaCloud |
|
||||
| CT 104 n8n config/encryption key | `n8n-prestage.sh` rsync → UNAS `/services/n8n` → `services-backup-plan` | ✓ JottaCloud |
|
||||
| CT 104 Open WebUI uploads + vector_db | `openwebui-prestage.sh` rsync → UNAS `/services/open-webui` → `services-backup-plan` | ✓ JottaCloud |
|
||||
| Paperless-ngx media | UNAS `/media/documents/public/paperless-ngx` → `services-backup-plan` | ✓ JottaCloud |
|
||||
| CT 104 `lobe-postgres` (paradedb) | WAL-G → Garage → JottaCloud sync | ✓ |
|
||||
| CT 104 Immich postgres | WAL-G → `immich-pg-backup` → sync | ✓ |
|
||||
|
||||
@@ -150,13 +150,11 @@ The MCP gateway bridges `ai-internal` + `shared_backend`.
|
||||
## AI / Agent platform (the core, `/opt/stacks/ai`)
|
||||
- **Bifrost** (`ai.nuclide.systems`, :14003) — unified LLM gateway + MCP host.
|
||||
Stack `ai/bifrost/`. Auth via `sk-bf-` VKs. LLM inference at `/v1`; MCP at
|
||||
`/mcp` (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to
|
||||
LiteLLM internally.
|
||||
- **LiteLLM** (:14000, **internal only** — no Zoraxy route) — model gateway,
|
||||
backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal
|
||||
**failovers** + `num_retries`, Redis completion cache, Gemini €10/30d
|
||||
`provider_budget_config`. Config: `ai/litellm-config/config.yaml` (+ DB
|
||||
overlay `LiteLLM_Config`). Reachable as `http://litellm:4000` inside Docker.
|
||||
`/mcp` (29 clients, ~760 tools as of 2026-05-26). Connects directly to
|
||||
upstream LLM providers; no LiteLLM hop in the chain anymore.
|
||||
- ~~**LiteLLM**~~ — **SUNSET 2026-05-28**. Container removed, DB + role dropped on CT 113,
|
||||
config dir renamed `litellm-config.DECOMMISSIONED-2026-05-28`, Prometheus
|
||||
scrape job deleted. Bifrost now handles all LLM gateway responsibilities.
|
||||
- **Open WebUI** (`chat.nuclide.systems`, :14002) — chat UI. Stack
|
||||
`ai/open-webui.yml`. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not
|
||||
yet wired as of 2026-05-26).
|
||||
@@ -184,7 +182,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`.
|
||||
|
||||
## Data layer
|
||||
- **shared-postgres** (pg16, `shared_backend`, data on local ZFS) — the
|
||||
**primary/standard DB for all deployments**. Tenants: LiteLLM, paperless,
|
||||
**primary/standard DB for all deployments**. Tenants: paperless,
|
||||
**memos** (migrated 2026-05-19), n8n (added 2026-05-19). Superuser
|
||||
`postgres` via unix socket; per-app dedicated role+db (role owns its db,
|
||||
password in the app stack's `.env`). Tuned 2026-05-19 (3G limit,
|
||||
@@ -206,7 +204,7 @@ The MCP gateway bridges `ai-internal` + `shared_backend`.
|
||||
**pgloader data-only of ALL tables** into the app-built schema (exclude
|
||||
only the app's own migration-tracking table), app role temp-SUPERUSER for
|
||||
the load (FK/trigger disable) then reverted.
|
||||
- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). **lobe-redis** — LiteLLM cache (LobeChat decommissioned). **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
|
||||
- **lobe-postgres** (pg17/ParadeDB) — LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). ~~**lobe-redis**~~ — was LiteLLM cache; both LobeChat and LiteLLM decommissioned. **Garage S3** (`garage:3900`, ext `s3.nuclide.systems` /
|
||||
`:10004`) — lobe-files, memos, **WAL-G PG backups**. **Qdrant** (vector,
|
||||
on UNAS) — unused yet (future RAG/mem0). pgAdmin (internal).
|
||||
- **Backups**: WAL-G v3 → Garage for all 3 PG instances (shared-postgres,
|
||||
|
||||
Reference in New Issue
Block a user