fkrebs
143343decf
- Add RESUME.md for cross-session continuity (open items, key state, constraints) - claude-max-bridge: implement /v1/responses (OpenAI Responses API) with previous_response_id chaining via server-side history injection into system prompt. Root cause of "session already in use": CLI leaves JSONL in un-resumable "dequeued" state after each --print run; fix avoids session reuse entirely. Also fixed: assistant content must be array-of-blocks not plain string (silent JS crash otherwise). - LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge - Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs, Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS) - Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
191 lines
8.1 KiB
Markdown
191 lines
8.1 KiB
Markdown
# Design: Claude Max Subscription → LiteLLM Gateway Bridge
|
||
|
||
**Status**: Proposal — not implemented
|
||
**Context**: LiteLLM at `https://ai.nuclide.systems` currently has no Anthropic models. Claude Max subscription (claude.ai) provides high-rate access to Sonnet 4.5/4.6 and Opus 4.7 but is decoupled from Anthropic API billing. This doc explores bridging the two.
|
||
|
||
## Problem
|
||
|
||
Claude Max and the Anthropic API are separate products with separate billing:
|
||
|
||
| | Claude Max | Anthropic API |
|
||
|---|---|---|
|
||
| Auth | OAuth / browser session | API key |
|
||
| Billing | Flat monthly subscription | Per-token |
|
||
| Rate limits | 5h rolling windows, model-specific | Tier-based RPM/TPM |
|
||
| Access | claude.ai web + Claude Code CLI | Any HTTP client |
|
||
|
||
The goal is to surface Max-subscription capacity through LiteLLM so that LobeHub, n8n, Coder workspaces, and other internal tools can call `claude-sonnet-4-6` at zero marginal cost and fall back to paid providers only when Max limits are hit.
|
||
|
||
## Approaches
|
||
|
||
### A — Session Cookie Reverse-Engineering (not recommended)
|
||
|
||
Several community projects (e.g. `claude-unofficial-api`) scrape the claude.ai WebSocket/HTTP protocol and expose an OpenAI-compatible endpoint. LiteLLM would point at this as a custom `openai/` provider.
|
||
|
||
**Pros**: Exposes the full web model lineup; streaming works.
|
||
**Cons**: Violates Anthropic ToS; breaks on any claude.ai front-end change; auth flow requires persisting browser cookies; no multimodal or tool-use parity guarantees.
|
||
|
||
**Verdict**: Avoid. Fragile and non-compliant.
|
||
|
||
---
|
||
|
||
### B — Claude Code CLI Bridge (recommended)
|
||
|
||
Claude Code CLI (`claude`) is already installed on CT 104 and authenticated with the Max subscription via `~/.claude/`. It ships a `--print` / `--output-format stream-json` mode designed for non-interactive use, and Anthropic explicitly supports programmatic use of the CLI.
|
||
|
||
A small **`claude-max-bridge`** service wraps this CLI as an OpenAI-compatible HTTP endpoint. LiteLLM registers it as a custom `openai/` base URL. No ToS issues — this is the supported surface.
|
||
|
||
```
|
||
LobeHub / n8n / Coder / Claude Code
|
||
│
|
||
▼
|
||
LiteLLM Gateway (ai.nuclide.systems)
|
||
│ model: claude-sonnet-4-6 → openai/claude-sonnet-4-6
|
||
│ api_base: http://claude-max-bridge:8000
|
||
▼
|
||
claude-max-bridge (new container, CT 104 ai-internal)
|
||
│ subprocess: claude --model ... --print --output-format stream-json
|
||
▼
|
||
~/.claude/ (Max subscription session)
|
||
│
|
||
▼
|
||
Anthropic (claude.ai)
|
||
```
|
||
|
||
**Pros**:
|
||
- Uses the officially supported programmatic interface
|
||
- Auth is already set up; no cookie management
|
||
- `claude` CLI handles retries, token limits, context window management
|
||
- Subprocess overhead is ~200–400 ms cold; warm invocations faster
|
||
|
||
**Cons**:
|
||
- One subprocess per request — cannot multiplex a single session (unlike streaming HTTP)
|
||
- CLI is tied to the single authenticated user; no multi-user isolation
|
||
- Max rate limits apply per-account, same pool as interactive use
|
||
- Claude Code SDK (TypeScript) is cleaner but adds Node dependency
|
||
|
||
---
|
||
|
||
### C — Official API + Budget Cap (stop-gap)
|
||
|
||
Add Anthropic API key to LiteLLM with a hard budget cap (e.g. $20/month). Use it for Claude-specific features (tool use, long context) and let the existing free SAIA/Gemini fallback chain absorb general-purpose traffic.
|
||
|
||
**Pros**: Zero implementation work; full API feature parity.
|
||
**Cons**: Still costs money; no benefit from Max subscription.
|
||
|
||
**Verdict**: Valid fallback if B proves too complex, or as a complement for tool-heavy workloads that need the official API surface.
|
||
|
||
---
|
||
|
||
## Recommended Architecture (Option B)
|
||
|
||
### claude-max-bridge service
|
||
|
||
```
|
||
/opt/stacks/ai/claude-max-bridge/
|
||
Dockerfile
|
||
server.py # FastAPI, ~150 lines
|
||
docker-compose.yml (or entry in ai/docker-compose.yml)
|
||
```
|
||
|
||
**Dockerfile** — reuse the existing `claude` CLI install:
|
||
```dockerfile
|
||
FROM python:3.12-slim
|
||
RUN pip install fastapi uvicorn
|
||
# Mount ~/.claude from host; claude binary from host PATH or copied in
|
||
COPY server.py /app/server.py
|
||
CMD ["uvicorn", "app.server:app", "--host", "0.0.0.0", "--port", "8000"]
|
||
```
|
||
|
||
**server.py** — OpenAI `/v1/chat/completions` shim:
|
||
```python
|
||
# POST /v1/chat/completions
|
||
# Translates messages[] → claude --print --model ... --output-format stream-json
|
||
# Streams NDJSON lines back as SSE (text/event-stream)
|
||
# Maps finish_reason, usage tokens from CLI output headers
|
||
```
|
||
|
||
Key translation notes:
|
||
- `messages` → write to a temp file, pass via `--input-file` (avoids shell quoting issues)
|
||
- `stream: true` → parse `stream-json` lines, emit `data: {...}` SSE chunks
|
||
- `stream: false` → buffer all chunks, return single response object
|
||
- `max_tokens`, `temperature`, `system` → map to `--max-tokens`, `--temperature`, `--system`
|
||
- Tool use: **not supported** in first iteration; return 501 for requests with `tools`
|
||
- Model name passthrough: `claude-sonnet-4-6` → `--model claude-sonnet-4-6`
|
||
|
||
### LiteLLM config additions
|
||
|
||
```yaml
|
||
model_list:
|
||
- model_name: claude-sonnet-4-6
|
||
litellm_params:
|
||
model: openai/claude-sonnet-4-6
|
||
api_base: http://claude-max-bridge:8000
|
||
api_key: "dummy" # bridge ignores it; LiteLLM requires a value
|
||
stream_timeout: 120
|
||
timeout: 120
|
||
|
||
- model_name: claude-opus-4-7
|
||
litellm_params:
|
||
model: openai/claude-opus-4-7
|
||
api_base: http://claude-max-bridge:8000
|
||
api_key: "dummy"
|
||
stream_timeout: 180
|
||
timeout: 180
|
||
|
||
- model_name: claude-haiku-4-5
|
||
litellm_params:
|
||
model: openai/claude-haiku-4-5-20251001
|
||
api_base: http://claude-max-bridge:8000
|
||
api_key: "dummy"
|
||
stream_timeout: 60
|
||
timeout: 60
|
||
```
|
||
|
||
Add to `router_settings.fallbacks` — Max hits rate limit → fall to SAIA:
|
||
```yaml
|
||
fallbacks:
|
||
- claude-sonnet-4-6: [qwen3.5-397b-a17b]
|
||
- claude-opus-4-7: [qwen3.5-397b-a17b, mistral-large-latest]
|
||
- claude-haiku-4-5: [qwen3.5-122b-a10b, cerebras-llama-3.1-8b]
|
||
```
|
||
|
||
### Rate limit handling
|
||
|
||
The bridge should detect the CLI's rate-limit exit code / stderr message and return HTTP 429. LiteLLM's router will then trigger the fallback chain. No custom logic needed in LiteLLM itself.
|
||
|
||
Max limits as of 2026 (approximate, vary by plan tier):
|
||
- Sonnet 4.6: ~50 messages / 5-hour window at Pro; higher at Max
|
||
- Opus 4.7: ~10–20 messages / 5-hour window
|
||
- Haiku 4.5: effectively unlimited under Max
|
||
|
||
## Volume estimate
|
||
|
||
Typical homelab workloads (LobeHub chat, n8n automations, Coder coding assist) generate maybe 200–500 completions/day. Max's 5-hour windows reset 4–5× daily, so the practical limit is not usually hit unless Opus is used heavily.
|
||
|
||
## Risks
|
||
|
||
| Risk | Mitigation |
|
||
|---|---|
|
||
| CLI API changes between Claude Code releases | Pin the `claude` binary version; watch for breaking changes in release notes |
|
||
| Max rate limit shared with interactive use | Monitor via `claude usage`; bridge adds `X-Max-Usage` header from CLI output |
|
||
| Auth session expiry | Bridge returns 401 on auth failure; detect and alert via Gotify |
|
||
| Subprocess latency (cold start) | Keep a warm subprocess pool (1–2 persistent processes) using `--interactive` + message passing, or accept the 200–400 ms overhead |
|
||
| No tool use support | Fallback chain routes tool-use requests to API providers (Gemini, Mistral) |
|
||
|
||
## Implementation plan
|
||
|
||
1. Write `server.py` shim (~150 lines) + Dockerfile
|
||
2. Build image on CT 104, add to `ai/docker-compose.yml`
|
||
3. Bind-mount `~/.claude` read-only into the container
|
||
4. Add model entries to `litellm-config/config.yaml`
|
||
5. Test streaming via `curl` against the bridge directly
|
||
6. Test end-to-end via LiteLLM → LobeHub
|
||
7. Watch `docker logs claude-max-bridge` for rate-limit / auth errors for 48h
|
||
|
||
## Open questions
|
||
|
||
- Does `claude --print` support all message roles (`system`, `user`, `assistant` turns)? Needs verification with multi-turn conversation format.
|
||
- Should the bridge be on CT 104 (access to `~/.claude`) or CT 109 ops (remote docker socket to CT 104)? CT 104 is simpler for v1.
|
||
- Is there an official Claude Code SDK for Python? (TypeScript SDK exists at `@anthropic-ai/claude-code`; Python wrapper may be cleaner than shelling out.)
|