Skip to content

Data-leak audit comparison — TR-004 sessions

Two LobeHub conversations on the same task (LUMEN TR-004 failure analysis), 24 hours apart, with different models — compared for what leaked, where, and why.

Headline verdict. Session B (2026-05-21) was clean — all data stayed within the homelab + approved-partner perimeter (SAIA / GWDG, DLR-vetted, IdP-federated). Session A (2026-05-20) was a breach: 9 lobe-cloud-sandbox calls sent proprietary aerospace code+identifiers to commercial third parties (api.lobehub.com + codesandbox.io). The single variable that changed the outcome was the model choice.

Sessions at a glance

2026-05-20 2026-05-21
Title Lumen TR-004 Test Run Analysis Analyzing LUMEN TR004 Test Data
Model qwen3.5-397b-a17b qwen3-coder-30b-a3b-instruct + llama-3.3-70b-instruct
Messages 60 (2 user · 21 asst · 37 tool) 41 (3 user · 21 asst · 17 tool)
Shepard MCP calls (LAN) 27 9
lobe-cloud-sandbox calls (off-host code exec) 9 ❌ 0 ✅
Artifacts builtin (in-container) 0 5 (broken render)
lobe-agent-documents (in-container) 1 3
LLM inference destination SAIA / GWDG SAIA / GWDG
Picture rendering worked? yes (PNGs returned via exportFile) no (Artifacts plugin empty results, links broken)

Channel-by-channel comparison

flowchart TB
  subgraph s1 ["2026-05-20 · qwen3.5-397b"]
    direction LR
    u1(["You"]) --> l1[LobeHub]
    l1 -->|on-LAN, 27 calls| sh1[Shepard CT 101]
    l1 -. inference, 21 calls .-> ll1[LiteLLM] -.-> sa1[(SAIA)]
    l1 -. 9 code-exec calls .-> sb1[(LobeHub Cloud Sandbox<br/>+ codesandbox.io)]
  end
  subgraph s2 ["2026-05-21 · qwen3-coder-30b"]
    direction LR
    u2(["You"]) --> l2[LobeHub]
    l2 -->|on-LAN, 9 calls| sh2[Shepard CT 101]
    l2 -. inference, 21 calls .-> ll2[LiteLLM] -.-> sa2[(SAIA)]
    l2 -- 5 calls --> af2[Artifacts plugin<br/>in-container, broken]
  end
  classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc
  class sa1,sa2,sb1 leak
  classDef ok fill:#234c2a,stroke:#4c8,color:#cfc
  class sh1,sh2,l1,l2,ll1,ll2,af2 ok

The second session closes the worst channel (lobe-cloud-sandbox) entirely, at the cost of broken pictures. The fundamental "all LLM context goes to SAIA" leak is identical across both.

Risk reduction between the two

Risk 2026-05-20 2026-05-21 Δ
Proprietary code → commercial cloud HIGH (9 calls, ~1500 lines of LUMEN/P3-Lampoldshausen Python sent to LobeHub Cloud + codesandbox.io) NONE (0 calls) ✅ −100%
Proprietary text/data → academic provider HIGH (21 inference calls) HIGH (21 inference calls) ↔ no change
Picture rendering works ✅ ❌ regression — needs S3 hub
Egress destinations 3 (SAIA + lobehub + codesandbox) 1 (SAIA) ✅ −67%

What drove the difference: model behavior

Same user prompt template both days. The model choice changed the tool selection: - qwen3.5-397b-a17b → reached for lobe-cloud-sandbox (full Python interpreter) — because it can generate complex matplotlib pipelines and execute them. - qwen3-coder-30b-a3b-instruct → reached for Artifacts (SVG + HTML generators) — code-focused model, prefers structured output over runtime execution.

Operational takeaway: model selection is a privacy control. A LobeHub policy that defaults sensitive chats to qwen3-coder-30b (or any model that doesn't reach for cloud sandbox) materially reduces the worst-case leak — even before you disable the sandbox plugin entirely.

Common ground (both sessions)

  • All Shepard MCP fetches stayed on LAN. Good.
  • All assistant turns hit SAIA / GWDG via LiteLLM. SAIA is operated by GWDG, vetted by DLR, and integrated with the DLR IdP federation — approved partner for DLR-context work like LUMEN. SAIA is registered as a LiteLLM backend in this homelab and serves as the terminal endpoint of the fallback chains for the free-tier Qwen / Llama models (see services/litellm.md).
  • Cerebras / Gemini / Mistral were NOT reached in either session — they're configured as fallbacks ahead of SAIA, but SAIA is preferred and was used directly.
  • The DAYTONA_API_KEY env var is still in the LobeHub container after Daytona's decommissioning on 2026-05-20. Trivial cleanup.

What this means for policy

Permanent fixes

  1. Disable lobe-cloud-sandbox — remove a whole class of leak; nothing was gained by having it that the local Coder mcp-sandbox template can't replicate.
  2. Pin LobeHub egress to LiteLLM only — single chokepoint; easier to audit, swap, and route.
  3. Default sensitive chats to a model with no cloud-sandbox affinity — operationally enforce via system-prompt prefixes or LobeHub agent presets.

Capability gaps to close

  1. S3-backed artifact store — fixes the broken picture rendering AND gives every AI surface (LobeHub, Coder, n8n agents, MCP gateway) a uniform "show me a thing in a browser" channel. See data-leak-audit-2026-05-21-tr004-artifacts.md § "Fix: S3 as a data-exchange hub" for design.

Defense in depth

  1. UDM egress firewall blocking api.cerebras.ai, api.lobehub.com, *.codesandbox.io unless explicitly whitelisted per request.
  2. Local LLM (e.g. qwen3-coder-30b on Arc via ollama/vllm) so the truly sensitive subset doesn't leave at all.

Action backlog (cumulative, deduped)

[no prereqs — start any]
├── A. Disable lobe-cloud-sandbox in LobeHub                 (CT 104 env, ~10 min)
├── B. Pin LobeHub to LiteLLM only                           (CT 104 env, ~15 min)
├── C. Local LLM behind LiteLLM (ollama / vllm + Arc)        (~1-2 h)
├── D. Default sensitive chats to qwen3-coder (LobeHub preset) (~30 min)
├── E. Egress firewall block on UDM                          (operator OK, ~30 min)
├── F. Remove stale DAYTONA_API_KEY                          (CT 104 env, trivial)
└── G. Fix s3.nuclide.systems route                          (Zoraxy — needs operator OK)
       └── H. chat-artifacts bucket + ACL + lifecycle        (Garage)
              ├── I. upload_artifact MCP server               (CT 104 gateway)
              ├── J. LobeHub Artifacts → S3                   (TypeScript patch)
              └── K. Coder savefig helper in dotfiles

Method

Both audits used the same workflow: 1. Parse the LobeHub-exported JSON (messages[]). 2. Bucket by role and plugin.identifier / plugin.apiName. 3. Regex-extract all referenced URLs; categorize hosts as LAN vs off-host. 4. Cross-check live infrastructure: - docker exec lobehub env — provider keys SET - docker exec litellm grep of proxy_server_config.yaml — model routing - ss -tnp state established from CT 104 — live outbound (snapshot only) 5. Score per channel: (sensitivity × likelihood) → criticality. 6. Recommend mitigations with effort estimates and dependency tree.

Audit artifacts at /root/.claude/uploads/... (transcripts), reports under /docs/security/.