fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate LiteLLM → Bifrost (LLM gateway) #6

New Issue
Closed
opened 2026-05-26 04:21:00 +00:00 by fkrebs · 1 comment
fkrebs commented 2026-05-26 04:21:00 +00:00
Owner
Copy Link
Copy Source

Replace LiteLLM with Bifrost (Go binary, ~5.2k stars, v1.3+ has Responses API + chaining). Adds: native semantic cache against Qdrant, first-class MCP routing with per-VK tool filtering + Code Mode (~50% token spend cut at 3+ servers), stronger RBAC.

Risks / gaps:

  • Generic OIDC (Pocket-ID) not clearly documented — TEST FIRST
  • Fewer providers than LiteLLM (still covers Anthropic/OpenAI/Mistral/Gemini/Ollama)
  • Pass-through to claude-max-bridge — Bifrost equivalent uncertain
  • No automated config converter — manual model_list rewrite

Plan phases:

  1. Stand up Bifrost on free port alongside LiteLLM
  2. Verify Pocket-ID OIDC
  3. Test /v1/responses + chaining
  4. Re-point consumers one at a time (LobeChat → Nextcloud → Karakeep → Paperless → Coder)
  5. Decommission LiteLLM

Depends-on: LobeChat→OWUI issue, MCP-gateway simplification issue.

Replace LiteLLM with [Bifrost](https://github.com/maximhq/bifrost) (Go binary, ~5.2k stars, v1.3+ has Responses API + chaining). Adds: native semantic cache against Qdrant, first-class MCP routing with per-VK tool filtering + Code Mode (~50% token spend cut at 3+ servers), stronger RBAC. **Risks / gaps:** - Generic OIDC (Pocket-ID) not clearly documented — TEST FIRST - Fewer providers than LiteLLM (still covers Anthropic/OpenAI/Mistral/Gemini/Ollama) - Pass-through to claude-max-bridge — Bifrost equivalent uncertain - No automated config converter — manual model_list rewrite **Plan phases:** 1. Stand up Bifrost on free port alongside LiteLLM 2. Verify Pocket-ID OIDC 3. Test `/v1/responses` + chaining 4. Re-point consumers one at a time (LobeChat → Nextcloud → Karakeep → Paperless → Coder) 5. Decommission LiteLLM Depends-on: LobeChat→OWUI issue, MCP-gateway simplification issue.
fkrebs commented 2026-05-26 05:57:15 +00:00
Author
Owner
Copy Link
Copy Source

Completed 2026-05-26.

Completed 2026-05-26.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
fkrebs (Florian Krebs)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fkrebs/docs#6
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?