# LLM Benchmark

Streaming performance benchmark for all LiteLLM-routed models. Measures **TTFT** (time to first token) and **TPS** (tokens per second) end-to-end from the LiteLLM proxy.

## Script

`/opt/stacks/ai/benchmark/bench.py` on CT 104.

## How to run

Must run inside a container on `ai-internal` to reach both LiteLLM and the internal Kroki instance:

```bash
docker run --rm --network ai-internal \
  -v /opt/stacks/ai/benchmark/bench.py:/bench.py \
  -v /opt/stacks/ai/benchmark:/charts \
  -e LITELLM_API_KEY=sk-tapirnase \
  -e KROKI_URL=http://kroki:8000 \
  python:3.12-slim \
  bash -c 'pip install httpx -q && python /bench.py --charts /charts'
```

Charts (`ttft.svg`, `tps.svg`) land in `/opt/stacks/ai/benchmark/`.

### Options

| Flag | Default | Description |
|------|---------|-------------|
| `--models a,b` | all 11 | Comma-separated subset to test |
| `--prompt` | `medium` | `ttft_short`, `medium`, or `code_medium` |
| `--runs N` | `2` | Runs per model (mean reported) |
| `--out PATH` | `/tmp/bench_results.json` | Raw JSON output |
| `--charts DIR` | `/tmp/bench_charts` | SVG chart output directory |

### Environment

| Var | Default | Notes |
|-----|---------|-------|
| `LITELLM_API_KEY` | — | Required (`sk-tapirnase`) |
| `LITELLM_BASE_URL` | `https://ai.nuclide.systems/v1` | Override for in-network: `http://litellm:4000` |
| `KROKI_URL` | `http://kroki:8000` | Internal Kroki for chart rendering |

## Models tested

| Model ID | Provider | Notes |
|----------|----------|-------|
| `claude-haiku-4-5` | claude-max-bridge | Via `claude` CLI subprocess |
| `claude-sonnet-4-6` | claude-max-bridge | |
| `claude-opus-4-7` | claude-max-bridge | |
| `gemini-2.5-flash-lite` | Google | |
| `gemini-2.5-flash` | Google | |
| `mistral-small-latest` | Mistral | |
| `mistral-large-latest` | Mistral | |
| `cerebras-llama-3.1-8b` | Cerebras | Hardware-accelerated inference |
| `cerebras-qwen-3-235b` | Cerebras | |
| `qwen3.5-122b-a10b` | — | Returns empty responses — likely extended-thinking mode with no visible content; excluded from charts |
| `deepseek-r1-distill-llama-70b` | DeepSeek | Reasoning model; total latency high due to thinking tokens |

## Methodology

- Each run sends a single streaming `POST /v1/chat/completions` request with `max_tokens: 512`
- **TTFT**: `perf_counter()` delta from request start to first SSE chunk containing `content`
- **TPS**: completion chars ÷ total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by `usage.completion_tokens` if the provider returns it.
- Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary.
- Models that return no `content` deltas (e.g. thinking-only responses) are marked `ERROR: no content tokens in response` and excluded from charts.

## Results — 2026-05-22

Prompt: `"Explain what TCP/IP is in 3 sentences."` · 2 runs each · warm LiteLLM cache

| Model | TTFT (ms) | TPS (chars/s) | Notes |
|-------|----------:|--------------|-------|
| **mistral-small-latest** | **43** | 10,111 | Fastest TTFT |
| **mistral-large-latest** | **58** | 11,042 | Fastest TPS |
| claude-opus-4-7 | 106 | 4,184 | |
| cerebras-qwen-3-235b | 118 | 3,777 | |
| claude-haiku-4-5 | 125 | 3,454 | |
| gemini-2.5-flash | 139 | 3,519 | |
| claude-sonnet-4-6 | 130 | 3,856 | |
| gemini-2.5-flash-lite | 202 | 2,396 | |
| deepseek-r1-distill-llama-70b | 297 | 10,561 | Total 5.4 s (thinking tokens) |
| cerebras-llama-3.1-8b | 155 | 2,947 | |
| qwen3.5-122b-a10b | ERROR | — | Empty response |

**Key observations:**
- Mistral leads on both TTFT and raw throughput — likely a combination of low-latency EU endpoints and LiteLLM response caching
- Claude models via the subprocess bridge perform well: 101–143 ms TTFT is competitive with direct cloud APIs
- DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer
- TPS figures are char/s estimates; actual token/s ≈ TPS ÷ 3.5