docs: session continuity file + claude-max-bridge /v1/responses fix
- Add RESUME.md for cross-session continuity (open items, key state, constraints) - claude-max-bridge: implement /v1/responses (OpenAI Responses API) with previous_response_id chaining via server-side history injection into system prompt. Root cause of "session already in use": CLI leaves JSONL in un-resumable "dequeued" state after each --print run; fix avoids session reuse entirely. Also fixed: assistant content must be array-of-blocks not plain string (silent JS crash otherwise). - LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge - Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs, Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS) - Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -19,7 +19,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
|
||||
(confirmed) — so ad/tracker blocking is network-wide despite DHCP not
|
||||
handing out AdGuard's IP directly.
|
||||
- `103` lxc **backrest** — Restic/Backrest backups (1c/0.5G)
|
||||
- `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/32G/215G) → `.40`
|
||||
- `104` lxc **docker** — **the primary `/opt/stacks` host** (16c/48G/200G) → `.40`
|
||||
- `105` lxc **nextcloud** — Nextcloud (4c/8G/107G) → `.41`
|
||||
- `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
|
||||
- `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
|
||||
@@ -392,14 +392,14 @@ flowchart TB
|
||||
dns["CT 102 · dns<br/>(.2)<br/>AdGuard Home"]
|
||||
backrest["CT 103 · backrest<br/>(.3)<br/>Backrest / restic"]
|
||||
docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
|
||||
nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (CIFS to UNAS)"]
|
||||
nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
|
||||
zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
|
||||
obs(["CT 109 · ops<br/>(.8) · <i>planned</i><br/>Prom / Graf / Arcane / Homarr / Tinyauth"])
|
||||
id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
|
||||
dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
|
||||
db(["CT 113 · db<br/>(.?) · <i>planned</i><br/>shared Postgres + pgbouncer"])
|
||||
db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
|
||||
end
|
||||
UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3 + CIFS")]
|
||||
UNAS[("UNAS<br/>192.168.1.31<br/>NFSv3")]
|
||||
UDM[["UDM-SE · 192.168.1.1<br/>UniFi gateway · DNS → AdGuard"]]
|
||||
Inet([Internet · ACME challenges · jottacloud · LiteLLM upstreams])
|
||||
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# LLM Benchmark
|
||||
|
||||
Streaming performance benchmark for all LiteLLM-routed models. Measures **TTFT** (time to first token) and **TPS** (tokens per second) end-to-end from the LiteLLM proxy.
|
||||
|
||||
## Script
|
||||
|
||||
`/opt/stacks/ai/benchmark/bench.py` on CT 104.
|
||||
|
||||
## How to run
|
||||
|
||||
Must run inside a container on `ai-internal` to reach both LiteLLM and the internal Kroki instance:
|
||||
|
||||
```bash
|
||||
docker run --rm --network ai-internal \
|
||||
-v /opt/stacks/ai/benchmark/bench.py:/bench.py \
|
||||
-v /opt/stacks/ai/benchmark:/charts \
|
||||
-e LITELLM_API_KEY=sk-tapirnase \
|
||||
-e KROKI_URL=http://kroki:8000 \
|
||||
python:3.12-slim \
|
||||
bash -c 'pip install httpx -q && python /bench.py --charts /charts'
|
||||
```
|
||||
|
||||
Charts (`ttft.svg`, `tps.svg`) land in `/opt/stacks/ai/benchmark/`.
|
||||
|
||||
### Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--models a,b` | all 11 | Comma-separated subset to test |
|
||||
| `--prompt` | `medium` | `ttft_short`, `medium`, or `code_medium` |
|
||||
| `--runs N` | `2` | Runs per model (mean reported) |
|
||||
| `--out PATH` | `/tmp/bench_results.json` | Raw JSON output |
|
||||
| `--charts DIR` | `/tmp/bench_charts` | SVG chart output directory |
|
||||
|
||||
### Environment
|
||||
|
||||
| Var | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `LITELLM_API_KEY` | — | Required (`sk-tapirnase`) |
|
||||
| `LITELLM_BASE_URL` | `https://ai.nuclide.systems/v1` | Override for in-network: `http://litellm:4000` |
|
||||
| `KROKI_URL` | `http://kroki:8000` | Internal Kroki for chart rendering |
|
||||
|
||||
## Models tested
|
||||
|
||||
| Model ID | Provider | Notes |
|
||||
|----------|----------|-------|
|
||||
| `claude-haiku-4-5` | claude-max-bridge | Via `claude` CLI subprocess |
|
||||
| `claude-sonnet-4-6` | claude-max-bridge | |
|
||||
| `claude-opus-4-7` | claude-max-bridge | |
|
||||
| `gemini-2.5-flash-lite` | Google | |
|
||||
| `gemini-2.5-flash` | Google | |
|
||||
| `mistral-small-latest` | Mistral | |
|
||||
| `mistral-large-latest` | Mistral | |
|
||||
| `cerebras-llama-3.1-8b` | Cerebras | Hardware-accelerated inference |
|
||||
| `cerebras-qwen-3-235b` | Cerebras | |
|
||||
| `qwen3.5-122b-a10b` | — | Returns empty responses — likely extended-thinking mode with no visible content; excluded from charts |
|
||||
| `deepseek-r1-distill-llama-70b` | DeepSeek | Reasoning model; total latency high due to thinking tokens |
|
||||
|
||||
## Methodology
|
||||
|
||||
- Each run sends a single streaming `POST /v1/chat/completions` request with `max_tokens: 512`
|
||||
- **TTFT**: `perf_counter()` delta from request start to first SSE chunk containing `content`
|
||||
- **TPS**: completion chars ÷ total elapsed seconds (char-based; divide by ~3.5 for true token TPS). Overridden by `usage.completion_tokens` if the provider returns it.
|
||||
- Run 1 of each model includes cold-start overhead (subprocess fork for Claude bridge, connection setup for cloud APIs). The mean across all runs is reported in the summary.
|
||||
- Models that return no `content` deltas (e.g. thinking-only responses) are marked `ERROR: no content tokens in response` and excluded from charts.
|
||||
|
||||
## Results — 2026-05-22
|
||||
|
||||
Prompt: `"Explain what TCP/IP is in 3 sentences."` · 2 runs each · warm LiteLLM cache
|
||||
|
||||
| Model | TTFT (ms) | TPS (chars/s) | Notes |
|
||||
|-------|----------:|--------------|-------|
|
||||
| **mistral-small-latest** | **43** | 10,111 | Fastest TTFT |
|
||||
| **mistral-large-latest** | **58** | 11,042 | Fastest TPS |
|
||||
| claude-opus-4-7 | 106 | 4,184 | |
|
||||
| cerebras-qwen-3-235b | 118 | 3,777 | |
|
||||
| claude-haiku-4-5 | 125 | 3,454 | |
|
||||
| gemini-2.5-flash | 139 | 3,519 | |
|
||||
| claude-sonnet-4-6 | 130 | 3,856 | |
|
||||
| gemini-2.5-flash-lite | 202 | 2,396 | |
|
||||
| deepseek-r1-distill-llama-70b | 297 | 10,561 | Total 5.4 s (thinking tokens) |
|
||||
| cerebras-llama-3.1-8b | 155 | 2,947 | |
|
||||
| qwen3.5-122b-a10b | ERROR | — | Empty response |
|
||||
|
||||
**Key observations:**
|
||||
- Mistral leads on both TTFT and raw throughput — likely a combination of low-latency EU endpoints and LiteLLM response caching
|
||||
- Claude models via the subprocess bridge perform well: 101–143 ms TTFT is competitive with direct cloud APIs
|
||||
- DeepSeek R1 has a fast TTFT (297 ms) but 5+ s total due to streaming reasoning tokens before the final answer
|
||||
- TPS figures are char/s estimates; actual token/s ≈ TPS ÷ 3.5
|
||||
@@ -14,7 +14,7 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve
|
||||
|
||||
## Server inventory (as of 2026-05-22)
|
||||
|
||||
26 servers configured in `mcp-gateway/config.json`. **Status: 26 healthy, 1 disabled (nextcloud re-enabled 2026-05-22).**
|
||||
26 servers configured in `mcp-gateway/config.json`. **Status: 27 healthy.**
|
||||
|
||||
| Group | Server | Kind | Notes |
|
||||
|---|---|---|---|
|
||||
@@ -35,6 +35,7 @@ DinD MCP gateway on CT 104. Single OAuth-gated entry point fronting 24 MCP serve
|
||||
| | wikipedia-mcp | catalog | |
|
||||
| | youtube-transcript | catalog | `mcp/youtube-transcript` |
|
||||
| **personal** | bluesky | static | `ariel-mcp:8000` |
|
||||
| | obsidian | spawn | `mcp-obsidian-bridge` + mcpvault; vault at Nextcloud `fkrebs@nucli.de/files/Notizen` (UNAS mount) |
|
||||
| | gotify | spawn | |
|
||||
| | home-assistant | static | `http://192.168.1.60:9583/private_...` (HA add-on) |
|
||||
| | immich | spawn | |
|
||||
|
||||
Reference in New Issue
Block a user