Skip to content

# Data-leak audit comparison — TR-004 sessions

Two LobeHub conversations on the same task (LUMEN TR-004 failure analysis), 24 hours apart, with different models — compared for what leaked, where, and why.

> Headline verdict. Session B (2026-05-21) was clean — all data stayed within the homelab + approved-partner perimeter (SAIA / GWDG, DLR-vetted, IdP-federated). Session A (2026-05-20) was a breach: 9 `lobe-cloud-sandbox` calls sent proprietary aerospace code+identifiers to commercial third parties (`api.lobehub.com` + codesandbox.io). The single variable that changed the outcome was the model choice.

## Sessions at a glance

| 2026-05-20 | 2026-05-21
--- | --- | ---
Title | `Lumen TR-004 Test Run Analysis` | `Analyzing LUMEN TR004 Test Data`
Model | `qwen3.5-397b-a17b` | `qwen3-coder-30b-a3b-instruct` + `llama-3.3-70b-instruct`
Messages | 60 (2 user · 21 asst · 37 tool) | 41 (3 user · 21 asst · 17 tool)
Shepard MCP calls (LAN) | 27 | 9
`lobe-cloud-sandbox` calls (off-host code exec) | 9 ❌ | 0 ✅
`Artifacts` builtin (in-container) | 0 | 5 (broken render)
`lobe-agent-documents` (in-container) | 1 | 3
LLM inference destination | SAIA / GWDG | SAIA / GWDG
Picture rendering worked? | yes (PNGs returned via `exportFile`) | no (Artifacts plugin empty results, links broken)

## Channel-by-channel comparison

    flowchart TB
      subgraph s1 ["2026-05-20 · qwen3.5-397b"]
        direction LR
        u1(["You"]) --> l1[LobeHub]
        l1 -->|on-LAN, 27 calls| sh1[Shepard CT 101]
        l1 -. inference, 21 calls .-> ll1[LiteLLM] -.-> sa1[(SAIA)]
        l1 -. 9 code-exec calls .-> sb1[(LobeHub Cloud Sandbox<br/>+ codesandbox.io)]
      end
      subgraph s2 ["2026-05-21 · qwen3-coder-30b"]
        direction LR
        u2(["You"]) --> l2[LobeHub]
        l2 -->|on-LAN, 9 calls| sh2[Shepard CT 101]
        l2 -. inference, 21 calls .-> ll2[LiteLLM] -.-> sa2[(SAIA)]
        l2 -- 5 calls --> af2[Artifacts plugin<br/>in-container, broken]
      end
      classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc
      class sa1,sa2,sb1 leak
      classDef ok fill:#234c2a,stroke:#4c8,color:#cfc
      class sh1,sh2,l1,l2,ll1,ll2,af2 ok

The second session closes the worst channel (lobe-cloud-sandbox) entirely, at the cost of broken pictures. The fundamental "all LLM context goes to SAIA" leak is identical across both.

## Risk reduction between the two

Risk | 2026-05-20 | 2026-05-21 | Δ
--- | --- | --- | ---
Proprietary code → commercial cloud | HIGH (9 calls, ~1500 lines of LUMEN/P3-Lampoldshausen Python sent to LobeHub Cloud + codesandbox.io) | NONE (0 calls) | ✅ −100%
Proprietary text/data → academic provider | HIGH (21 inference calls) | HIGH (21 inference calls) | ↔ no change
Picture rendering works | ✅ | ❌ | regression — needs S3 hub
Egress destinations | 3 (SAIA + lobehub + codesandbox) | 1 (SAIA) | ✅ −67%

## What drove the difference: model behavior

Same user prompt template both days. The model choice changed the tool selection: - `qwen3.5-397b-a17b` → reached for `lobe-cloud-sandbox` (full Python interpreter) — because it can generate complex matplotlib pipelines and execute them. - `qwen3-coder-30b-a3b-instruct` → reached for `Artifacts` (SVG + HTML generators) — code-focused model, prefers structured output over runtime execution.

Operational takeaway: model selection is a privacy control. A LobeHub policy that defaults sensitive chats to `qwen3-coder-30b` (or any model that doesn't reach for cloud sandbox) materially reduces the worst-case leak — even before you disable the sandbox plugin entirely.

## Common ground (both sessions)

- All Shepard MCP fetches stayed on LAN. Good.
- All assistant turns hit SAIA / GWDG via LiteLLM. SAIA is operated by GWDG, vetted by DLR, and integrated with the DLR IdP federation — approved partner for DLR-context work like LUMEN. SAIA is registered as a LiteLLM backend in this homelab and serves as the terminal endpoint of the fallback chains for the free-tier Qwen / Llama models (see `services/litellm.md`).
- Cerebras / Gemini / Mistral were NOT reached in either session — they're configured as fallbacks ahead of SAIA, but SAIA is preferred and was used directly.
- The `DAYTONA_API_KEY` env var is still in the LobeHub container after Daytona's decommissioning on 2026-05-20. Trivial cleanup.

## What this means for policy

### Permanent fixes

1. Disable `lobe-cloud-sandbox` — remove a whole class of leak; nothing was gained by having it that the local Coder `mcp-sandbox` template can't replicate.
2. Pin LobeHub egress to LiteLLM only — single chokepoint; easier to audit, swap, and route.
3. Default sensitive chats to a model with no cloud-sandbox affinity — operationally enforce via system-prompt prefixes or LobeHub agent presets.

### Capability gaps to close

1. S3-backed artifact store — fixes the broken picture rendering AND gives every AI surface (LobeHub, Coder, n8n agents, MCP gateway) a uniform "show me a thing in a browser" channel. See `data-leak-audit-2026-05-21-tr004-artifacts.md` § "Fix: S3 as a data-exchange hub" for design.

### Defense in depth

1. UDM egress firewall blocking `api.cerebras.ai`, `api.lobehub.com`, `*.codesandbox.io` unless explicitly whitelisted per request.
2. Local LLM (e.g. `qwen3-coder-30b` on Arc via ollama/vllm) so the truly sensitive subset doesn't leave at all.

## Action backlog (cumulative, deduped)

    [no prereqs — start any]
    ├── A. Disable lobe-cloud-sandbox in LobeHub                 (CT 104 env, ~10 min)
    ├── B. Pin LobeHub to LiteLLM only                           (CT 104 env, ~15 min)
    ├── C. Local LLM behind LiteLLM (ollama / vllm + Arc)        (~1-2 h)
    ├── D. Default sensitive chats to qwen3-coder (LobeHub preset) (~30 min)
    ├── E. Egress firewall block on UDM                          (operator OK, ~30 min)
    ├── F. Remove stale DAYTONA_API_KEY                          (CT 104 env, trivial)
    └── G. Fix s3.nuclide.systems route                          (Zoraxy — needs operator OK)
           └── H. chat-artifacts bucket + ACL + lifecycle        (Garage)
                  ├── I. upload_artifact MCP server               (CT 104 gateway)
                  ├── J. LobeHub Artifacts → S3                   (TypeScript patch)
                  └── K. Coder savefig helper in dotfiles

## Method

Both audits used the same workflow: 1. Parse the LobeHub-exported JSON (`messages[]`). 2. Bucket by `role` and `plugin.identifier` / `plugin.apiName`. 3. Regex-extract all referenced URLs; categorize hosts as LAN vs off-host. 4. Cross-check live infrastructure: - `docker exec lobehub env` — provider keys SET - `docker exec litellm grep` of `proxy_server_config.yaml` — model routing - `ss -tnp state established` from CT 104 — live outbound (snapshot only) 5. Score per channel: (sensitivity × likelihood) → criticality. 6. Recommend mitigations with effort estimates and dependency tree.

Audit artifacts at `/root/.claude/uploads/...` (transcripts), reports under `/docs/security/`.