fkrebs
143343decf
- Add RESUME.md for cross-session continuity (open items, key state, constraints) - claude-max-bridge: implement /v1/responses (OpenAI Responses API) with previous_response_id chaining via server-side history injection into system prompt. Root cause of "session already in use": CLI leaves JSONL in un-resumable "dequeued" state after each --print run; fix avoids session reuse entirely. Also fixed: assistant content must be array-of-blocks not plain string (silent JS crash otherwise). - LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge - Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs, Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS) - Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
90 lines
4.0 KiB
Markdown
90 lines
4.0 KiB
Markdown
# LLM Benchmark
|
||
|
||
Streaming performance benchmark for all LiteLLM-routed models. Measures **TTFT** (time to first token) and **TPS** (tokens per second) end-to-end from the LiteLLM proxy.
|
||
|
||
## Script
|
||
|
||
`/opt/stacks/ai/benchmark/bench.py` on CT 104.
|
||
|
||
## How to run
|
||
|
||
Must run inside a container on `ai-internal` to reach both LiteLLM and the internal Kroki instance:
|
||
|
||
```bash
|
||
docker run --rm --network ai-internal \
|
||
-v /opt/stacks/ai/benchmark/bench.py:/bench.py \
|
||
-v /opt/stacks/ai/benchmark:/charts \
|
||
-e LITELLM_API_KEY=sk-tapirnase \
|
||
-e KROKI_URL=http://kroki:8000 \
|
||
python:3.12-slim \
|
||
bash -c 'pip install httpx -q && python /bench.py --charts /charts'
|
||
```
|
||
|
||
Charts (`ttft.svg`, `tps.svg`) land in `/opt/stacks/ai/benchmark/`.
|
||
|
||
### Options
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--models a,b` | all 11 | Comma-separated subset to test |
|
||
| `--prompt` | `medium` | `ttft_short`, `medium`, or `code_medium` |
|
||
| `--runs N` | `2` | Runs per model (mean reported) |
|
||
| `--out PATH` | `/tmp/bench_results.json` | Raw JSON output |
|
||
| `--charts DIR` | `/tmp/bench_charts` | SVG chart output directory |
|
||
|
||
### Environment
|
||
|
||
| Var | Default | Notes |
|
||
|-----|---------|-------|
|
||
| `LITELLM_API_KEY` | — | Required (`sk-tapirnase`) |
|
||
| `LITELLM_BASE_URL` | `https://ai.nuclide.systems/v1` | Override for in-network: `http://litellm:4000` |
|
||
| `KROKI_URL` | `http://kroki:8000` | Internal Kroki for chart rendering |
|
||
|
||
## Models tested
|
||
|
||
| Model ID | Provider | Notes |
|
||
|----------|----------|-------|
|
||
| `claude-haiku-4-5` | claude-max-bridge | Via `claude` CLI subprocess |
|
||
| `claude-sonnet-4-6` | claude-max-bridge | |
|
||
| `claude-opus-4-7` | claude-max-bridge | |
|
||
| `gemini-2.5-flash-lite` | Google | |
|
||
| `gemini-2.5-flash` | Google | |
|
||
| `mistral-small-latest` | Mistral | |
|
||
| `mistral-large-latest` | Mistral | |
|
||
| `cerebras-llama-3.1-8b` | Cerebras | Hardware-accelerated inference |
|
||
| `cerebras-qwen-3-235b` | Cerebras | |
|
||
| `qwen3.5-122b-a10b` | — | Returns empty responses — likely extended-thinking mode with no visible content; excluded from charts |
|
||
| `deepseek-r1-distill-llama-70b` | DeepSeek | Reasoning model; total latency high due to thinking tokens |
|
||
|
||
## Methodology
|
||
|
||
- Each run sends a single streaming `POST /v1/chat/completions` request with `max_tokens: 512`
|
||
- **TTFT**: `perf_counter()` delta from request start to first SSE chunk containing `content`
|
||
- **TPS**: completion chars ÷ total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by `usage.completion_tokens` if the provider returns it.
|
||
- Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary.
|
||
- Models that return no `content` deltas (e.g. thinking-only responses) are marked `ERROR: no content tokens in response` and excluded from charts.
|
||
|
||
## Results — 2026-05-22
|
||
|
||
Prompt: `"Explain what TCP/IP is in 3 sentences."` · 2 runs each · warm LiteLLM cache
|
||
|
||
| Model | TTFT (ms) | TPS (chars/s) | Notes |
|
||
|-------|----------:|--------------|-------|
|
||
| **mistral-small-latest** | **43** | 10,111 | Fastest TTFT |
|
||
| **mistral-large-latest** | **58** | 11,042 | Fastest TPS |
|
||
| claude-opus-4-7 | 106 | 4,184 | |
|
||
| cerebras-qwen-3-235b | 118 | 3,777 | |
|
||
| claude-haiku-4-5 | 125 | 3,454 | |
|
||
| gemini-2.5-flash | 139 | 3,519 | |
|
||
| claude-sonnet-4-6 | 130 | 3,856 | |
|
||
| gemini-2.5-flash-lite | 202 | 2,396 | |
|
||
| deepseek-r1-distill-llama-70b | 297 | 10,561 | Total 5.4 s (thinking tokens) |
|
||
| cerebras-llama-3.1-8b | 155 | 2,947 | |
|
||
| qwen3.5-122b-a10b | ERROR | — | Empty response |
|
||
|
||
**Key observations:**
|
||
- Mistral leads on both TTFT and raw throughput — likely a combination of low-latency EU endpoints and LiteLLM response caching
|
||
- Claude models via the subprocess bridge perform well: 101–143 ms TTFT is competitive with direct cloud APIs
|
||
- DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer
|
||
- TPS figures are char/s estimates; actual token/s ≈ TPS ÷ 3.5
|