Files

T

fkrebs 143343decf docs: session continuity file + claude-max-bridge /v1/responses fix

- Add RESUME.md for cross-session continuity (open items, key state, constraints)
- claude-max-bridge: implement /v1/responses (OpenAI Responses API) with
  previous_response_id chaining via server-side history injection into system prompt.
  Root cause of "session already in use": CLI leaves JSONL in un-resumable
  "dequeued" state after each --print run; fix avoids session reuse entirely.
  Also fixed: assistant content must be array-of-blocks not plain string (silent
  JS crash otherwise).
- LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge
- Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs,
  Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS)
- Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-22 22:42:17 +02:00

4.0 KiB

Raw Blame History

LLM Benchmark

Streaming performance benchmark for all LiteLLM-routed models. Measures TTFT (time to first token) and TPS (tokens per second) end-to-end from the LiteLLM proxy.

Script

/opt/stacks/ai/benchmark/bench.py on CT 104.

How to run

Must run inside a container on ai-internal to reach both LiteLLM and the internal Kroki instance:

docker run --rm --network ai-internal \
  -v /opt/stacks/ai/benchmark/bench.py:/bench.py \
  -v /opt/stacks/ai/benchmark:/charts \
  -e LITELLM_API_KEY=sk-tapirnase \
  -e KROKI_URL=http://kroki:8000 \
  python:3.12-slim \
  bash -c 'pip install httpx -q && python /bench.py --charts /charts'

Charts (ttft.svg, tps.svg) land in /opt/stacks/ai/benchmark/.

Options

Flag	Default	Description
`--models a,b`	all 11	Comma-separated subset to test
`--prompt`	`medium`	`ttft_short`, `medium`, or `code_medium`
`--runs N`	`2`	Runs per model (mean reported)
`--out PATH`	`/tmp/bench_results.json`	Raw JSON output
`--charts DIR`	`/tmp/bench_charts`	SVG chart output directory

Environment

Var	Default	Notes
`LITELLM_API_KEY`	—	Required (`sk-tapirnase`)
`LITELLM_BASE_URL`	`https://ai.nuclide.systems/v1`	Override for in-network: `http://litellm:4000`
`KROKI_URL`	`http://kroki:8000`	Internal Kroki for chart rendering

Models tested

Model ID	Provider	Notes
`claude-haiku-4-5`	claude-max-bridge	Via `claude` CLI subprocess
`claude-sonnet-4-6`	claude-max-bridge
`claude-opus-4-7`	claude-max-bridge
`gemini-2.5-flash-lite`	Google
`gemini-2.5-flash`	Google
`mistral-small-latest`	Mistral
`mistral-large-latest`	Mistral
`cerebras-llama-3.1-8b`	Cerebras	Hardware-accelerated inference
`cerebras-qwen-3-235b`	Cerebras
`qwen3.5-122b-a10b`	—	Returns empty responses — likely extended-thinking mode with no visible content; excluded from charts
`deepseek-r1-distill-llama-70b`	DeepSeek	Reasoning model; total latency high due to thinking tokens

Methodology

Each run sends a single streaming POST /v1/chat/completions request with max_tokens: 512
TTFT: perf_counter() delta from request start to first SSE chunk containing content
TPS: completion chars ÷ total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by usage.completion_tokens if the provider returns it.
Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary.
Models that return no content deltas (e.g. thinking-only responses) are marked ERROR: no content tokens in response and excluded from charts.

Results — 2026-05-22

Prompt: "Explain what TCP/IP is in 3 sentences." · 2 runs each · warm LiteLLM cache

Model	TTFT (ms)	TPS (chars/s)	Notes
mistral-small-latest	43	10,111	Fastest TTFT
mistral-large-latest	58	11,042	Fastest TPS
claude-opus-4-7	106	4,184
cerebras-qwen-3-235b	118	3,777
claude-haiku-4-5	125	3,454
gemini-2.5-flash	139	3,519
claude-sonnet-4-6	130	3,856
gemini-2.5-flash-lite	202	2,396
deepseek-r1-distill-llama-70b	297	10,561	Total 5.4 s (thinking tokens)
cerebras-llama-3.1-8b	155	2,947
qwen3.5-122b-a10b	ERROR	—	Empty response

Key observations:

Mistral leads on both TTFT and raw throughput — likely a combination of low-latency EU endpoints and LiteLLM response caching
Claude models via the subprocess bridge perform well: 101–143 ms TTFT is competitive with direct cloud APIs
DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer
TPS figures are char/s estimates; actual token/s ≈ TPS ÷ 3.5

4.0 KiB Raw Blame History Unescape Escape