116 lines
7.2 KiB
Markdown
116 lines
7.2 KiB
Markdown
# Data-leak audit comparison — TR-004 sessions
|
||
|
||
Two LobeHub conversations on the same task (LUMEN TR-004 failure analysis), 24 hours apart, with different models — compared for what leaked, where, and why.
|
||
|
||
> **Headline verdict.** Session B (2026-05-21) was clean — all data stayed within the homelab + approved-partner perimeter (SAIA / GWDG, [DLR-vetted](https://docs.hpc.gwdg.de/services/ai-services/saia/index.html), IdP-federated). Session A (2026-05-20) **was a breach**: 9 `lobe-cloud-sandbox` calls sent proprietary aerospace code+identifiers to commercial third parties (`api.lobehub.com` + codesandbox.io). **The single variable that changed the outcome was the model choice.**
|
||
|
||
## Sessions at a glance
|
||
|
||
| | **2026-05-20** | **2026-05-21** |
|
||
|---|---|---|
|
||
| Title | `Lumen TR-004 Test Run Analysis` | `Analyzing LUMEN TR004 Test Data` |
|
||
| Model | `qwen3.5-397b-a17b` | `qwen3-coder-30b-a3b-instruct` + `llama-3.3-70b-instruct` |
|
||
| Messages | 60 (2 user · 21 asst · 37 tool) | 41 (3 user · 21 asst · 17 tool) |
|
||
| Shepard MCP calls (LAN) | 27 | 9 |
|
||
| **`lobe-cloud-sandbox` calls (off-host code exec)** | **9** ❌ | **0** ✅ |
|
||
| `Artifacts` builtin (in-container) | 0 | 5 (broken render) |
|
||
| `lobe-agent-documents` (in-container) | 1 | 3 |
|
||
| LLM inference destination | SAIA / GWDG | SAIA / GWDG |
|
||
| Picture rendering worked? | yes (PNGs returned via `exportFile`) | **no** (Artifacts plugin empty results, links broken) |
|
||
|
||
## Channel-by-channel comparison
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph s1 ["2026-05-20 · qwen3.5-397b"]
|
||
direction LR
|
||
u1(["You"]) --> l1[LobeHub]
|
||
l1 -->|on-LAN, 27 calls| sh1[Shepard CT 101]
|
||
l1 -. inference, 21 calls .-> ll1[LiteLLM] -.-> sa1[(SAIA)]
|
||
l1 -. 9 code-exec calls .-> sb1[(LobeHub Cloud Sandbox<br/>+ codesandbox.io)]
|
||
end
|
||
subgraph s2 ["2026-05-21 · qwen3-coder-30b"]
|
||
direction LR
|
||
u2(["You"]) --> l2[LobeHub]
|
||
l2 -->|on-LAN, 9 calls| sh2[Shepard CT 101]
|
||
l2 -. inference, 21 calls .-> ll2[LiteLLM] -.-> sa2[(SAIA)]
|
||
l2 -- 5 calls --> af2[Artifacts plugin<br/>in-container, broken]
|
||
end
|
||
classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc
|
||
class sa1,sa2,sb1 leak
|
||
classDef ok fill:#234c2a,stroke:#4c8,color:#cfc
|
||
class sh1,sh2,l1,l2,ll1,ll2,af2 ok
|
||
```
|
||
|
||
The second session **closes the worst channel** (lobe-cloud-sandbox) entirely, at the cost of broken pictures. The fundamental "all LLM context goes to SAIA" leak is identical across both.
|
||
|
||
## Risk reduction between the two
|
||
|
||
| Risk | 2026-05-20 | 2026-05-21 | Δ |
|
||
|---|---|---|---|
|
||
| Proprietary code → commercial cloud | **HIGH** (9 calls, ~1500 lines of LUMEN/P3-Lampoldshausen Python sent to LobeHub Cloud + codesandbox.io) | **NONE** (0 calls) | ✅ −100% |
|
||
| Proprietary text/data → academic provider | HIGH (21 inference calls) | HIGH (21 inference calls) | ↔ no change |
|
||
| Picture rendering works | ✅ | ❌ | regression — needs S3 hub |
|
||
| Egress destinations | 3 (SAIA + lobehub + codesandbox) | 1 (SAIA) | ✅ −67% |
|
||
|
||
## What drove the difference: model behavior
|
||
|
||
Same user prompt template both days. The model choice changed the tool selection:
|
||
- `qwen3.5-397b-a17b` → reached for `lobe-cloud-sandbox` (full Python interpreter) — because it can generate complex matplotlib pipelines and execute them.
|
||
- `qwen3-coder-30b-a3b-instruct` → reached for `Artifacts` (SVG + HTML generators) — code-focused model, prefers structured output over runtime execution.
|
||
|
||
**Operational takeaway:** **model selection is a privacy control**. A LobeHub policy that defaults sensitive chats to `qwen3-coder-30b` (or any model that doesn't reach for cloud sandbox) materially reduces the worst-case leak — even before you disable the sandbox plugin entirely.
|
||
|
||
## Common ground (both sessions)
|
||
|
||
- **All Shepard MCP fetches stayed on LAN.** Good.
|
||
- **All assistant turns hit SAIA / GWDG via LiteLLM.** SAIA is operated by GWDG, [vetted by DLR](https://docs.hpc.gwdg.de/services/ai-services/saia/index.html), and integrated with the DLR IdP federation — **approved partner for DLR-context work** like LUMEN. SAIA is registered as a LiteLLM backend in this homelab and serves as the terminal endpoint of the fallback chains for the free-tier Qwen / Llama models (see `services/litellm.md`).
|
||
- **Cerebras / Gemini / Mistral** were NOT reached in either session — they're configured as fallbacks ahead of SAIA, but SAIA is preferred and was used directly.
|
||
- **The `DAYTONA_API_KEY` env var is still in the LobeHub container** after Daytona's decommissioning on 2026-05-20. Trivial cleanup.
|
||
|
||
## What this means for policy
|
||
|
||
### Permanent fixes
|
||
1. **Disable `lobe-cloud-sandbox`** — remove a whole class of leak; nothing was gained by having it that the local Coder `mcp-sandbox` template can't replicate.
|
||
2. **Pin LobeHub egress to LiteLLM only** — single chokepoint; easier to audit, swap, and route.
|
||
3. **Default sensitive chats to a model with no cloud-sandbox affinity** — operationally enforce via system-prompt prefixes or LobeHub agent presets.
|
||
|
||
### Capability gaps to close
|
||
4. **S3-backed artifact store** — fixes the broken picture rendering AND gives every AI surface (LobeHub, Coder, n8n agents, MCP gateway) a uniform "show me a thing in a browser" channel. See `data-leak-audit-2026-05-21-tr004-artifacts.md` § "Fix: S3 as a data-exchange hub" for design.
|
||
|
||
### Defense in depth
|
||
5. **UDM egress firewall** blocking `api.cerebras.ai`, `api.lobehub.com`, `*.codesandbox.io` unless explicitly whitelisted per request.
|
||
6. **Local LLM** (e.g. `qwen3-coder-30b` on Arc via ollama/vllm) so the truly sensitive subset doesn't leave at all.
|
||
|
||
## Action backlog (cumulative, deduped)
|
||
|
||
```
|
||
[no prereqs — start any]
|
||
├── A. Disable lobe-cloud-sandbox in LobeHub (CT 104 env, ~10 min)
|
||
├── B. Pin LobeHub to LiteLLM only (CT 104 env, ~15 min)
|
||
├── C. Local LLM behind LiteLLM (ollama / vllm + Arc) (~1-2 h)
|
||
├── D. Default sensitive chats to qwen3-coder (LobeHub preset) (~30 min)
|
||
├── E. Egress firewall block on UDM (operator OK, ~30 min)
|
||
├── F. Remove stale DAYTONA_API_KEY (CT 104 env, trivial)
|
||
└── G. Fix s3.nuclide.systems route (Zoraxy — needs operator OK)
|
||
└── H. chat-artifacts bucket + ACL + lifecycle (Garage)
|
||
├── I. upload_artifact MCP server (CT 104 gateway)
|
||
├── J. LobeHub Artifacts → S3 (TypeScript patch)
|
||
└── K. Coder savefig helper in dotfiles
|
||
```
|
||
|
||
## Method
|
||
|
||
Both audits used the same workflow:
|
||
1. Parse the LobeHub-exported JSON (`messages[]`).
|
||
2. Bucket by `role` and `plugin.identifier` / `plugin.apiName`.
|
||
3. Regex-extract all referenced URLs; categorize hosts as LAN vs off-host.
|
||
4. Cross-check live infrastructure:
|
||
- `docker exec lobehub env` — provider keys SET
|
||
- `docker exec litellm grep` of `proxy_server_config.yaml` — model routing
|
||
- `ss -tnp state established` from CT 104 — live outbound (snapshot only)
|
||
5. Score per channel: (sensitivity × likelihood) → criticality.
|
||
6. Recommend mitigations with effort estimates and dependency tree.
|
||
|
||
Audit artifacts at `/root/.claude/uploads/...` (transcripts), reports under `/docs/security/`.
|