diff --git a/services/llm-benchmark.md b/services/llm-benchmark.md index 232e961..965d3dc 100644 --- a/services/llm-benchmark.md +++ b/services/llm-benchmark.md @@ -1,89 +1,395 @@ -# LLM Benchmark +# LLM Model Benchmark & Service Catalogue -Streaming performance benchmark for all LiteLLM-routed models. Measures **TTFT** (time to first token) and **TPS** (tokens per second) end-to-end from the LiteLLM proxy. +_Last updated: 2026-05-23 | Models: 60 | Source: LiteLLM /model/info + live benchmarks_ -## Script +## Overview -`/opt/stacks/ai/benchmark/bench.py` on CT 104. +This catalogue covers all 60 models registered in the homelab LiteLLM proxy (`http://192.168.1.40:14000`). +Live latency figures are TTFT proxies measured from this host via a single `max_tokens=5` completion request. +Speed tiers are based on live measurements and published inference benchmarks. -## How to run +**Providers at a glance:** -Must run inside a container on `ai-internal` to reach both LiteLLM and the internal Kroki instance: +| Provider | Models | Notes | +|----------|--------|-------| +| Claude (Anthropic via openai-compat) | 8 | claude-max subscription; aliases included | +| Mistral API | 10 | voxtral voice family + codestral + OCR | +| Gemini API | 12 | Flash/Pro/embedding families | +| SAIA (self-hosted GPU cluster) | 22 | OpenAI-compatible; local GPU inference | +| Groq | 2 | Ultra-fast cloud inference | +| Cerebras | 2 | Ultra-fast wafer-scale inference | +| Cohere | 2 | Embeddings only | -```bash -docker run --rm --network ai-internal \ - -v /opt/stacks/ai/benchmark/bench.py:/bench.py \ - -v /opt/stacks/ai/benchmark:/charts \ - -e LITELLM_API_KEY=sk-tapirnase \ - -e KROKI_URL=http://kroki:8000 \ - python:3.12-slim \ - bash -c 'pip install httpx -q && python /bench.py --charts /charts' +--- + +## Performance Tiers + +| Tier | Symbol | Typical TTFT | Profile | +|------|--------|-------------|---------| +| **Ultra-fast** | ๐Ÿš€ | < 200 ms | Groq, Cerebras, cached SAIA small models | +| **Fast** | โšก | 200โ€“600 ms | Mistral API, Gemini Flash, SAIA mid-size | +| **Standard** | ๐Ÿ”ต | 600โ€“2 000 ms | Claude, Gemini Pro, large API models | +| **Self-hosted** | ๐Ÿ  | varies | SAIA cluster; latency depends on GPU load & model size | + +> Note: SAIA models with very low latency (< 50 ms) on the trivial benchmark likely hit a cached/KV-prefilled response; real-world TTFT for longer prompts will be higher. Treat SAIA figures as best-case. + +--- + +## Model Catalogue + +### Chat & Reasoning Models + +| Model ID | Provider | Backend | Context | Vision | Tools | Cost In $/1M | Cost Out $/1M | Live Latency | Speed Tier | Notes | +|----------|----------|---------|---------|--------|-------|-------------|--------------|-------------|------------|-------| +| claude-sonnet-4-6 | Anthropic | openai-compat | 200K | โœ“ | โœ“ | $3.00 | $15.00 | 2 083 ms | ๐Ÿ”ต | Flagship balance model | +| claude-opus-4-7 | Anthropic | openai-compat | 200K | โœ“ | โœ“ | $15.00 | $75.00 | 3 218 ms | ๐Ÿ”ต | Highest capability Claude | +| claude-haiku-4-5 | Anthropic | openai-compat | 200K | โœ“ | โœ“ | $0.80 | $4.00 | 1 453 ms | ๐Ÿ”ต | Fast + cheap Claude tier | +| claude-sonnet-4-6-high | Anthropic | openai-compat | 200K | โœ“ | โœ“ | $3.00 | $15.00 | 1 746 ms | ๐Ÿ”ต | Extended thinking variant | +| claude-opus-4-7-high | Anthropic | openai-compat | 200K | โœ“ | โœ“ | $15.00 | $75.00 | 2 285 ms | ๐Ÿ”ต | Extended thinking variant | +| voxtral-small-latest | Mistral | Mistral API | 256K | โœ“ | โœ“ | โ€” | โ€” | 160 ms | ๐Ÿš€ | Voice+text multimodal | +| mistral-small-latest | Mistral | Mistral API | 131K | โœ“ | โœ“ | $0.06 | $0.18 | 222 ms | ๐Ÿš€ | Cheapest Mistral chat | +| voxtral-mini-latest | Mistral | Mistral API | 100K | โœ“ | โœ“ | โ€” | โ€” | ERROR | โŒ | Proxy config error (saia-image-proxy unreachable) | +| codestral-latest | Mistral | Mistral API | 16K | โœ“ | โœ“ | $1.00 | $3.00 | 275 ms | โšก | Coding-specialised Mistral | +| mistral-large-latest | Mistral | Mistral API | 262K | โœ“ | โœ“ | $0.50 | $1.50 | 309 ms | โšก | Flagship Mistral chat | +| pixtral-large-latest | Mistral | Mistral API | 128K | โœ“ | โœ“ | $2.00 | $6.00 | 27 ms | ๐Ÿš€ | Vision flagship; very low latency (likely cached) | +| voxtral-mini-realtime-latest | Mistral | Mistral API | 4K | โœ— | โœ“ | โ€” | โ€” | ERROR | โŒ | Invalid model per API; realtime/ws endpoint only | +| gemini-2.5-flash | Google | Gemini API | 65K | โœ“ | โœ“ | $0.30 | $2.50 | 501 ms | โšก | Best-value Gemini; fast + smart | +| gemini-2.5-flash-lite | Google | Gemini API | 65K | โœ“ | โœ“ | $0.10 | $0.40 | 518 ms | โšก | Lightest + cheapest Gemini | +| gemini-2.5-pro | Google | Gemini API | 65K | โœ“ | โœ“ | $1.25 | $10.00 | 888 ms | ๐Ÿ”ต | Top Gemini reasoning | +| gemini-3.1-pro-preview | Google | Gemini API | 65K | โœ“ | โœ“ | $2.00 | $12.00 | 1 215 ms | ๐Ÿ”ต | Next-gen Gemini Pro preview | +| gemini-3-pro-preview | Google | Gemini API | 65K | โœ“ | โœ“ | $2.00 | $12.00 | 1 258 ms | ๐Ÿ”ต | Gemini 3 Pro preview | +| gemini-3.1-flash-lite | Google | Gemini API | โ€” | โœ“ | โœ“ | โ€” | โ€” | 482 ms | โšก | Gemini 3.1 flash lite preview | +| devstral-2-123b-instruct-2512 | Mistral | SAIA | 4K | โœ— | โœ“ | โ€” | โ€” | 157 ms | ๐Ÿ  | 123B coding model via SAIA GPU | +| qwen3-coder-30b-a3b-instruct | Alibaba | SAIA | 32K | โœ— | โœ“ | โ€” | โ€” | 5 ms | ๐Ÿ  | MoE coding model; 5 ms = cached | +| openai-gpt-oss-120b | OpenAI | SAIA | 131K | โœ— | โœ— | โ€” | โ€” | 118 ms | ๐Ÿ  | OpenAI open-weight 120B via SAIA | +| qwen3-omni-30b-a3b-instruct | Alibaba | SAIA | 16K | โœ“ | โœ“ | โ€” | โ€” | 178 ms | ๐Ÿ  | Multimodal MoE; voice+vision | +| llama-3.3-70b-instruct | Meta | SAIA | 131K | โœ— | โœ“ | โ€” | โ€” | 168 ms | ๐Ÿ  | Reliable general-purpose 70B | +| deepseek-r1-distill-llama-70b | DeepSeek | SAIA | 131K | โœ— | โœ— | โ€” | โ€” | 325 ms | ๐Ÿ  | R1 reasoning distill; outputs `` tokens | +| qwen3.5-35b-a3b | Alibaba | SAIA | 65K | โœ“ | โœ“ | โ€” | โ€” | 250 ms | ๐Ÿ  | MoE 35B A3B | +| qwen3.5-27b | Alibaba | SAIA | 131K | โœ— | โœ“ | โ€” | โ€” | 266 ms | ๐Ÿ  | Dense 27B | +| qwen3.6-35b-a3b | Alibaba | SAIA | 65K | โœ“ | โœ“ | โ€” | โ€” | 204 ms | ๐Ÿ  | MoE 35B A3B v3.6 | +| apertus-70b-instruct-2509 | Apertus | SAIA | 4K | โœ— | โœ— | โ€” | โ€” | 143 ms | ๐Ÿ  | Small context; general chat | +| glm-4.7 | Zhipu | SAIA | 128K | โœ— | โœ“ | โ€” | โ€” | 210 ms | ๐Ÿ  | GLM-4 series | +| qwen3-30b-a3b-instruct-2507 | Alibaba | SAIA | 262K | โœ— | โœ— | โ€” | โ€” | 127 ms | ๐Ÿ  | MoE 30B, very large context | +| gemma-3-27b-it | Google | SAIA | 131K | โœ— | โœ“ | โ€” | โ€” | 284 ms | ๐Ÿ  | Gemma 3 27B instruct | +| internvl3.5-30b-a3b | InternLM | SAIA | 16K | โœ“ | โœ“ | โ€” | โ€” | 116 ms | ๐Ÿ  | Vision+tools MoE 30B | +| qwen3.5-122b-a10b | Alibaba | SAIA | 65K | โœ“ | โœ“ | โ€” | โ€” | 372 ms | ๐Ÿ  | MoE 122B A10B; larger/slower | +| qwen3.5-397b-a17b | Alibaba | SAIA | 65K | โœ“ | โœ“ | โ€” | โ€” | 248 ms | ๐Ÿ  | Largest SAIA MoE model | +| gemma-4-31b-it | Google | SAIA | 8K | โœ“ | โœ“ | โ€” | โ€” | 177 ms | ๐Ÿ  | Gemma 4 multimodal | +| meta-llama/llama-4-scout-17b-16e-instruct | Meta | Groq | 8K | โœ“ | โœ“ | $0.11 | $0.34 | 442 ms | ๐Ÿš€ | Groq-accelerated; 442 ms incl. queue | +| cerebras-llama-3.1-8b | Meta | Cerebras | 128K | โœ— | โœ“ | $0.10 | $0.10 | 258 ms | ๐Ÿš€ | Wafer-scale ~2 000 TPS | +| cerebras-qwen-3-235b | Alibaba | Cerebras | โ€” | โœ— | โœ— | โ€” | โ€” | 199 ms | ๐Ÿš€ | 235B at wafer-scale speed | + +### Embedding Models + +| Model ID | Provider | Backend | Context | Dimensions | Cost $/1M | Notes | +|----------|----------|---------|---------|------------|-----------|-------| +| gemini-embedding-2 | Google | Gemini API | 8K | โ€” | $0.20 | Primary Gemini embedding | +| gemini-embedding-001 | Google | Gemini API | 2K | โ€” | $0.15 | Legacy Gemini embedding | +| text-embedding-ada-002 | Google | Gemini API | 8K | โ€” | $0.20 | Alias โ†’ gemini-embedding-2 | +| text-embedding-3-small | Google | Gemini API | 8K | โ€” | $0.20 | Alias โ†’ gemini-embedding-2 | +| text-embedding-3-large | Google | Gemini API | 8K | โ€” | $0.20 | Alias โ†’ gemini-embedding-2 | +| multilingual-e5-large-instruct | Microsoft | SAIA | 8K | 1 024 | โ€” | Self-hosted multilingual; strong for DE/EN RAG | +| cohere-embed-multilingual-v3 | Cohere | Cohere API | 1K | 1 024 | $0.10 | 100+ languages | +| cohere-embed-english-v3 | Cohere | Cohere API | 1K | 1 024 | $0.10 | English-only; higher EN accuracy | + +### Audio Models (TTS / ASR) + +| Model ID | Provider | Backend | Type | Language | Notes | +|----------|----------|---------|------|----------|-------| +| tts-1-de | SAIA | Piper | TTS | German | Self-hosted German TTS | +| tts-1 | SAIA | Kokoro-82M | TTS | EN + others | Self-hosted multilingual TTS | +| voxtral-mini-tts-latest | Mistral | Mistral API | TTS | Multilingual | Mistral voice synthesis | +| saia-whisper | SAIA | whisper-large-v2 | ASR | Multilingual | Self-hosted transcription | +| whisper-1 | SAIA | faster-whisper-large-v3 | ASR | Multilingual | Faster self-hosted transcription | +| voxtral-mini-transcribe-2507 | Mistral | Mistral API | ASR | Multilingual | Mistral audio transcription; ctx 16K | +| whisper-large-v3-turbo | Meta | Groq | ASR | Multilingual | Groq-accelerated; fastest transcription | + +### Image Models + +| Model ID | Provider | Backend | Type | Notes | +|----------|----------|---------|------|-------| +| saia-flux | SAIA | FLUX | Image gen | Self-hosted FLUX; note: garbles text labels | +| gemini-2.5-flash-image | Google | Gemini API | Image gen | ctx 32K; multimodal image generation | +| saia-image-edit | SAIA | Qwen-Image-Edit | Image edit | Image editing/inpainting | +| mistral-ocr-latest | Mistral | Mistral API | OCR | Document OCR; not a chat model | + +--- + +## Service Recommendations + +### Nextcloud Assistant +_Smart file/email/calendar assistant, summaries, writing help โ€” multilingual DE/EN_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `mistral-small-latest` | Cheapest API model with tool use, vision, 131K context, and 222 ms TTFT. Handles German natively. | +| **Fallback** | `claude-haiku-4-5` | If higher quality needed; still cost-effective at $0.80/$4 and 1 453 ms TTFT. | +| **Alt (local)** | `qwen3.5-27b` | Free if staying fully on SAIA GPU cluster; 131K context + tools. | + +``` +model: mistral-small-latest ``` -Charts (`ttft.svg`, `tps.svg`) land in `/opt/stacks/ai/benchmark/`. +--- -### Options +### Karakeep (Bookmarks / Reading) +_Summarise articles, extract key points, tag/categorise โ€” no vision required_ -| Flag | Default | Description | -|------|---------|-------------| -| `--models a,b` | all 11 | Comma-separated subset to test | -| `--prompt` | `medium` | `ttft_short`, `medium`, or `code_medium` | -| `--runs N` | `2` | Runs per model (mean reported) | -| `--out PATH` | `/tmp/bench_results.json` | Raw JSON output | -| `--charts DIR` | `/tmp/bench_charts` | SVG chart output directory | +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `gemini-2.5-flash-lite` | Cheapest API model at $0.10/$0.40, 518 ms TTFT, strong comprehension. | +| **Fallback** | `mistral-small-latest` | Slightly pricier but faster at 222 ms. | -### Environment +``` +model: gemini-2.5-flash-lite +``` -| Var | Default | Notes | -|-----|---------|-------| -| `LITELLM_API_KEY` | โ€” | Required (`sk-tapirnase`) | -| `LITELLM_BASE_URL` | `https://ai.nuclide.systems/v1` | Override for in-network: `http://litellm:4000` | -| `KROKI_URL` | `http://kroki:8000` | Internal Kroki for chart rendering | +--- -## Models tested +### Home Assistant +_Intent recognition, automation triggers, voice pipeline โ€” ultra-low latency critical_ -| Model ID | Provider | Notes | -|----------|----------|-------| -| `claude-haiku-4-5` | claude-max-bridge | Via `claude` CLI subprocess | -| `claude-sonnet-4-6` | claude-max-bridge | | -| `claude-opus-4-7` | claude-max-bridge | | -| `gemini-2.5-flash-lite` | Google | | -| `gemini-2.5-flash` | Google | | -| `mistral-small-latest` | Mistral | | -| `mistral-large-latest` | Mistral | | -| `cerebras-llama-3.1-8b` | Cerebras | Hardware-accelerated inference | -| `cerebras-qwen-3-235b` | Cerebras | | -| `qwen3.5-122b-a10b` | โ€” | Returns empty responses โ€” likely extended-thinking mode with no visible content; excluded from charts | -| `deepseek-r1-distill-llama-70b` | DeepSeek | Reasoning model; total latency high due to thinking tokens | +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `cerebras-llama-3.1-8b` | 258 ms measured TTFT, ~2 000 TPS on Cerebras wafer silicon; best latency for real-time voice. 128K context, tools. | +| **Fallback** | `mistral-small-latest` | 222 ms TTFT, API-based, reliable tool calling. | +| **Local alt** | `internvl3.5-30b-a3b` | 116 ms on SAIA; avoids API cost for high-frequency automations. | -## Methodology +``` +model: cerebras-llama-3.1-8b +``` -- Each run sends a single streaming `POST /v1/chat/completions` request with `max_tokens: 512` -- **TTFT**: `perf_counter()` delta from request start to first SSE chunk containing `content` -- **TPS**: completion chars รท total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by `usage.completion_tokens` if the provider returns it. -- Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary. -- Models that return no `content` deltas (e.g. thinking-only responses) are marked `ERROR: no content tokens in response` and excluded from charts. +--- -## Results โ€” 2026-05-22 +### LobeChat Default +_General chat assistant for daily use โ€” balanced quality / speed / cost, vision nice_ -Prompt: `"Explain what TCP/IP is in 3 sentences."` ยท 2 runs each ยท warm LiteLLM cache +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `gemini-2.5-flash` | 501 ms, vision, tools, large context, excellent reasoning at $0.30/$2.50. Best all-rounder. | +| **Fallback** | `claude-sonnet-4-6` | Higher quality ceiling; use when depth matters over cost. | +| **Free alt** | `qwen3.5-397b-a17b` | Largest self-hosted model; free on SAIA with vision + tools at 248 ms. | -| Model | TTFT (ms) | TPS (chars/s) | Notes | -|-------|----------:|--------------|-------| -| **mistral-small-latest** | **43** | 10,111 | Fastest TTFT | -| **mistral-large-latest** | **58** | 11,042 | Fastest TPS | -| claude-opus-4-7 | 106 | 4,184 | | -| cerebras-qwen-3-235b | 118 | 3,777 | | -| claude-haiku-4-5 | 125 | 3,454 | | -| gemini-2.5-flash | 139 | 3,519 | | -| claude-sonnet-4-6 | 130 | 3,856 | | -| gemini-2.5-flash-lite | 202 | 2,396 | | -| deepseek-r1-distill-llama-70b | 297 | 10,561 | Total 5.4 s (thinking tokens) | -| cerebras-llama-3.1-8b | 155 | 2,947 | | -| qwen3.5-122b-a10b | ERROR | โ€” | Empty response | +``` +model: gemini-2.5-flash +``` -**Key observations:** -- Mistral leads on both TTFT and raw throughput โ€” likely a combination of low-latency EU endpoints and LiteLLM response caching -- Claude models via the subprocess bridge perform well: 101โ€“143 ms TTFT is competitive with direct cloud APIs -- DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer -- TPS figures are char/s estimates; actual token/s โ‰ˆ TPS รท 3.5 +--- + +### Code Assistant (Coder / IDE) +_Code completion, review, debugging โ€” strong code ability, large context, tools_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `claude-sonnet-4-6` | Best overall coding + reasoning; 200K context, tool use, reliable output. | +| **Fast/cheap** | `codestral-latest` | Coding-specialist Mistral at 275 ms with 16K context; good for completion. | +| **Local coding** | `devstral-2-123b-instruct-2512` | 123B SAIA coding model at 157 ms; free inference. | +| **MoE coding** | `qwen3-coder-30b-a3b-instruct` | 32K context, tools, extremely fast (5 ms cached); best SAIA coding model. | + +``` +model: claude-sonnet-4-6 # IDE / review +model: qwen3-coder-30b-a3b-instruct # local completion +``` + +--- + +### Document OCR / Ingestion +_Paperless โ†’ Docling โ†’ extract text โ€” vision + OCR capable, large context_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `mistral-ocr-latest` | Dedicated OCR endpoint; purpose-built for document text extraction. | +| **Fallback** | `gemini-2.5-pro` | 65K context, vision, strong at structured extraction from images. | +| **Alt vision** | `pixtral-large-latest` | Mistral vision flagship at 27 ms (cached); good document parsing. | + +``` +model: mistral-ocr-latest # OCR pipeline +model: gemini-2.5-pro # fallback / complex layouts +``` + +--- + +### Embeddings (Karakeep / LobeChat KB) +_Semantic search, RAG, knowledge base โ€” multilingual, high dimensions_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary (API)** | `gemini-embedding-2` | 8K context, $0.20/1M, strong multilingual. | +| **Primary (local)** | `multilingual-e5-large-instruct` | Self-hosted on SAIA, 1 024-dim, excellent DE/EN RAG, zero API cost. | +| **Multilingual API** | `cohere-embed-multilingual-v3` | 100+ languages, 1K context, $0.10/1M. | + +``` +model: multilingual-e5-large-instruct # local RAG +model: gemini-embedding-2 # API fallback +``` + +> **Note:** `text-embedding-ada-002`, `text-embedding-3-small`, and `text-embedding-3-large` are all aliases for `gemini-embedding-2` โ€” use the canonical ID to avoid confusion. + +--- + +### Image Generation +_ComfyUI complement, quick drafts_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `saia-flux` | Self-hosted FLUX on SAIA GPU; no API cost. Note: avoid text in generated images (garbles). | +| **API alt** | `gemini-2.5-flash-image` | Gemini multimodal image gen for quick API-based drafts. | +| **Editing** | `saia-image-edit` | Qwen image editing for inpainting / modifications. | + +``` +model: saia-flux +``` + +--- + +### TTS (Voice Interfaces) +_Read content aloud, voice responses_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **German** | `tts-1-de` | Self-hosted Piper; native German pronunciation. | +| **Multilingual** | `tts-1` | Self-hosted Kokoro-82M; covers EN + others, zero cost. | +| **API quality** | `voxtral-mini-tts-latest` | Mistral neural TTS for higher-quality voice synthesis. | + +``` +model: tts-1-de # German HA / Nextcloud voice +model: tts-1 # English / multilingual +``` + +--- + +### Transcription (Meetings / Voice) +_Speech to text_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `whisper-large-v3-turbo` | Groq-accelerated; fastest available transcription. | +| **Local** | `whisper-1` | faster-whisper-large-v3 on SAIA; fully self-hosted, no API cost. | +| **Fallback** | `saia-whisper` | whisper-large-v2 on SAIA; slightly older model. | + +``` +model: whisper-large-v3-turbo # real-time meetings +model: whisper-1 # offline / batch +``` + +--- + +### Reasoning / Analysis +_Complex problem solving, research, multi-step tasks_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `claude-opus-4-7-high` | Extended thinking mode; highest reasoning capability available. | +| **Cheaper** | `gemini-2.5-pro` | Strong reasoning at 888 ms, $1.25/$10.00; good for research tasks. | +| **Local reasoning** | `deepseek-r1-distill-llama-70b` | R1 chain-of-thought via SAIA at 325 ms; outputs `` tokens. | +| **Fast reasoning** | `cerebras-qwen-3-235b` | 235B model at 199 ms on Cerebras wafer silicon. | + +``` +model: claude-opus-4-7-high # deep analysis +model: deepseek-r1-distill-llama-70b # local reasoning +``` + +--- + +### Batch / Offline Processing +_Non-real-time document processing โ€” cost-optimised, high throughput_ + +| Role | Model | Reasoning | +|------|-------|-----------| +| **Primary** | `gemini-2.5-flash-lite` | $0.10/$0.40; cheapest API model with tools + vision. | +| **Free** | `llama-3.3-70b-instruct` | SAIA self-hosted 70B at 168 ms; 131K context, no API cost. | +| **Alt** | `qwen3-30b-a3b-instruct-2507` | 262K context MoE; good for long-document batch on SAIA. | + +``` +model: gemini-2.5-flash-lite # cost-sensitive API batch +model: llama-3.3-70b-instruct # free local batch +``` + +--- + +## Aliases & Duplicates + +The following model IDs are aliases that route to the same backend model. Use the canonical ID in production to avoid ambiguity: + +| Alias ID | Canonical Model | Notes | +|----------|----------------|-------| +| `sonnet` | `claude-sonnet-4-6` | Short alias | +| `opus` | `claude-opus-4-7` | Short alias | +| `haiku` | `claude-haiku-4-5` | Short alias | +| `text-embedding-ada-002` | `gemini-embedding-2` | OpenAI compat alias | +| `text-embedding-3-small` | `gemini-embedding-2` | OpenAI compat alias | +| `text-embedding-3-large` | `gemini-embedding-2` | OpenAI compat alias | + +--- + +## Experimental / Not Yet Validated + +The following models returned errors or have unresolved issues in live testing: + +| Model ID | Status | Error Detail | Action | +|----------|--------|-------------|--------| +| `voxtral-mini-latest` | โŒ HTTP 500 | `Cannot connect to host saia-image-proxy:5999` โ€” LiteLLM proxy routing misconfiguration | Fix proxy route; model itself is valid on Mistral API | +| `voxtral-mini-realtime-latest` | โŒ HTTP 400 | `Invalid model: voxtral-mini-realtime-latest` โ€” requires WebSocket realtime endpoint, not REST completions | Use Mistral realtime WS API directly; not suitable for standard chat completions | +| `mistral-ocr-latest` | โš ๏ธ Not benchmarked | OCR-mode model; requires document input, not chat completions | Use via dedicated OCR pipeline only | +| `voxtral-mini-transcribe-2507` | โš ๏ธ Not benchmarked | Audio transcription; not a chat completions model | Use via audio transcription endpoint | +| `gemini-2.5-flash-image` | โš ๏ธ Not benchmarked | Image generation; not a chat completions model | Use via images endpoint | +| `saia-flux` | โš ๏ธ Not benchmarked | FLUX image generation | Use via images endpoint | +| `saia-image-edit` | โš ๏ธ Not benchmarked | Image editing | Use via image edit endpoint | +| `tts-1-de` | โš ๏ธ Not benchmarked | Piper TTS audio output | Use via audio/speech endpoint | +| `tts-1` | โš ๏ธ Not benchmarked | Kokoro-82M TTS | Use via audio/speech endpoint | +| `voxtral-mini-tts-latest` | โš ๏ธ Not benchmarked | Mistral TTS | Use via audio/speech endpoint | +| `saia-whisper` | โš ๏ธ Not benchmarked | Whisper ASR | Use via audio/transcriptions endpoint | +| `whisper-1` | โš ๏ธ Not benchmarked | faster-whisper ASR | Use via audio/transcriptions endpoint | +| `whisper-large-v3-turbo` | โš ๏ธ Not benchmarked | Groq Whisper ASR | Use via audio/transcriptions endpoint | +| `gemini-3.1-flash-lite` | โš ๏ธ Context unknown | Preview model; ctx window not documented | Monitor Gemini API release notes | +| `cerebras-qwen-3-235b` | โš ๏ธ Context unknown | Context window not documented in LiteLLM config | Check Cerebras API docs | + +--- + +## Raw Benchmark Data + +All measurements from `2026-05-23`. Single `max_tokens=5` completion, prompt: _"Reply with exactly: ok"_. + +| Model ID | HTTP Status | Latency (ms) | Prompt Tokens | Completion Tokens | +|----------|-------------|-------------|--------------|------------------| +| claude-sonnet-4-6 | 200 | 2 083 | 3 | 4 | +| claude-opus-4-7 | 200 | 3 218 | 6 | 6 | +| claude-haiku-4-5 | 200 | 1 453 | 10 | 41 | +| claude-sonnet-4-6-high | 200 | 1 746 | 3 | 4 | +| claude-opus-4-7-high | 200 | 2 285 | 6 | 6 | +| voxtral-small-latest | 200 | 160 | 8 | 2 | +| mistral-small-latest | 200 | 222 | 20 | 2 | +| voxtral-mini-latest | 500 | 48 | โ€” | โ€” | +| codestral-latest | 200 | 275 | 13 | 2 | +| devstral-2-123b-instruct-2512 | 200 | 157 | 8 | 2 | +| qwen3-coder-30b-a3b-instruct | 200 | 5 | 13 | 2 | +| openai-gpt-oss-120b | 200 | 118 | 74 | 5 | +| gemini-2.5-flash | 200 | 501 | 6 | 1 | +| cerebras-llama-3.1-8b | 200 | 258 | 40 | 2 | +| meta-llama/llama-4-scout-17b-16e-instruct | 200 | 442 | 15 | 2 | +| mistral-large-latest | 200 | 309 | 8 | 2 | +| gemini-2.5-flash-lite | 200 | 518 | 6 | 1 | +| qwen3-omni-30b-a3b-instruct | 200 | 178 | 13 | 2 | +| llama-3.3-70b-instruct | 200 | 168 | 102 | 2 | +| deepseek-r1-distill-llama-70b | 200 | 325 | 8 | 5 | +| gemini-2.5-pro | 200 | 888 | 6 | 2 | +| pixtral-large-latest | 200 | 27 | 13 | 2 | +| voxtral-mini-realtime-latest | 400 | 140 | โ€” | โ€” | +| qwen3.5-35b-a3b | 200 | 250 | 15 | 5 | +| qwen3.5-27b | 200 | 266 | 15 | 5 | +| qwen3.6-35b-a3b | 200 | 204 | 15 | 5 | +| apertus-70b-instruct-2509 | 200 | 143 | 66 | 2 | +| glm-4.7 | 200 | 210 | 9 | 2 | +| gemini-3.1-pro-preview | 200 | 1 215 | 6 | 2 | +| gemini-3-pro-preview | 200 | 1 258 | 6 | 2 | +| qwen3-30b-a3b-instruct-2507 | 200 | 127 | 13 | 2 | +| gemini-3.1-flash-lite | 200 | 482 | 6 | 1 | +| gemma-3-27b-it | 200 | 284 | 14 | 3 | +| internvl3.5-30b-a3b | 200 | 116 | 13 | 2 | +| qwen3.5-122b-a10b | 200 | 372 | 15 | 5 | +| qwen3.5-397b-a17b | 200 | 248 | 15 | 5 | +| gemma-4-31b-it | 200 | 177 | 18 | 2 | +| cerebras-qwen-3-235b | 200 | 199 | 13 | 2 | + +> **Benchmark note on SAIA models with < 50 ms latency** (qwen3-coder: 5 ms, pixtral: 27 ms): these figures reflect a KV-cache or pre-warmed response for the trivial prompt. Real-world TTFT for cold prompts will be 100โ€“400 ms depending on model size and GPU availability.