auto: doc-ingestion: Qdrant+TEI deployed; OWUI RAG config; Bifrost embed models
This commit is contained in:
+64
-21
@@ -1,27 +1,61 @@
|
||||
# Document Ingestion Pipeline (planned)
|
||||
# Document Ingestion Pipeline
|
||||
|
||||
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in
|
||||
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into Open WebUI's
|
||||
knowledge base and make them searchable via MCP.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Nextcloud folder / Paperless webhook
|
||||
→ n8n trigger (already running on CT 104)
|
||||
→ n8n trigger (CT 104)
|
||||
→ Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
|
||||
→ LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large
|
||||
→ LobeChat knowledge base API ← natively searchable in chat
|
||||
→ (optional) Qdrant sidecar — if multi-app vector search needed
|
||||
→ TEI /v1/embeddings — multilingual-e5-base (local, 768d)
|
||||
→ Qdrant (shared vector store, http://qdrant:6333)
|
||||
→ Open WebUI knowledge base API ← searchable in chat
|
||||
```
|
||||
|
||||
LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its
|
||||
Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing
|
||||
piece is the n8n orchestration workflow.
|
||||
|
||||
**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
|
||||
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
|
||||
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.
|
||||
|
||||
## Deployed components (as of 2026-05-26)
|
||||
|
||||
| Component | Location | Endpoint |
|
||||
|-----------|----------|---------|
|
||||
| **Qdrant** | CT 104, `ai-internal` net | `http://qdrant:6333` |
|
||||
| **TEI** (Text Embeddings Inference) | CT 104, `ai-internal` net | `http://tei:80` |
|
||||
| **Open WebUI** | CT 104, port 14002 | Uses Qdrant + TEI natively |
|
||||
| **Docling MCP** | CT 104, port 18005 | MCP server in gateway |
|
||||
|
||||
### Embedding stack
|
||||
|
||||
- **Model**: `intfloat/multilingual-e5-base` (768d, multilingual DE+EN, ~278 MB ONNX)
|
||||
- **Server**: HuggingFace TEI `cpu-1.6` — OpenAI-compatible at `http://tei:80/v1/embeddings`
|
||||
- **Auth**: none (internal network only)
|
||||
- **Upgrade path**: switch to `mistral/mistral-embed` via Bifrost (1024d, better quality) — requires re-index (drop + recreate Qdrant collections)
|
||||
|
||||
### OWUI RAG config (env-driven)
|
||||
|
||||
```
|
||||
VECTOR_DB=qdrant
|
||||
QDRANT_URI=http://qdrant:6333
|
||||
RAG_EMBEDDING_ENGINE=openai
|
||||
RAG_OPENAI_API_BASE_URL=http://tei:80
|
||||
RAG_OPENAI_API_KEY=none
|
||||
RAG_EMBEDDING_MODEL=intfloat/multilingual-e5-base
|
||||
CONTENT_EXTRACTION_ENGINE=docling
|
||||
CHUNK_SIZE=1200
|
||||
CHUNK_OVERLAP=150
|
||||
ENABLE_RAG_HYBRID_SEARCH=true
|
||||
```
|
||||
|
||||
### Bifrost embedding models (available for external services / upgrade)
|
||||
|
||||
Three embedding models tested and working via `http://bifrost:8080/v1/embeddings`:
|
||||
- `mistral/mistral-embed` (1024d) — ✓ production-ready
|
||||
- `mistral/codestral-embed` (1024d) — ✓
|
||||
- `gemini/gemini-embedding-001` (768d/1536d) — ✓
|
||||
|
||||
## Converter comparison
|
||||
|
||||
| Tool | Image | Formats | Notes |
|
||||
@@ -32,19 +66,28 @@ directly from Paperless's REST API without re-running Docling for already-OCR'd
|
||||
|
||||
Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.
|
||||
|
||||
## MCP knowledge search
|
||||
## Sharing Qdrant with other services
|
||||
|
||||
Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base
|
||||
search API. No new infra — LobeChat is already on `shared_backend`.
|
||||
Qdrant is on `ai-internal` network — any service on that network can use it:
|
||||
|
||||
## Embedding options
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
client = QdrantClient(url="http://qdrant:6333")
|
||||
```
|
||||
|
||||
- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces
|
||||
- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom
|
||||
provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
|
||||
- **Mistral-embed / codestral-embed** — already available via LiteLLM
|
||||
n8n, MCP tools, and custom pipelines should use `http://tei:80/v1/embeddings` for
|
||||
consistent 768d vectors. Mixing models/dimensions in the same collection will fail.
|
||||
|
||||
## Status
|
||||
## Intel Arc GPU passthrough (planned)
|
||||
|
||||
Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API
|
||||
integration are the remaining work.
|
||||
CT 104 has Intel Core Ultra 7 155H iGPU but no `/dev/dri/render*` device is
|
||||
passed through. When passthrough is enabled, switch TEI to the Intel image:
|
||||
```
|
||||
image: ghcr.io/huggingface/text-embeddings-inference:intel-1.6
|
||||
```
|
||||
This uses IPEX and runs ~5–10× faster for embedding batches.
|
||||
|
||||
## Pending
|
||||
|
||||
- n8n workflow: Nextcloud/Paperless → Docling → TEI → Qdrant
|
||||
- MCP knowledge-search tool pointing at Qdrant (replaces LobeChat search intent)
|
||||
|
||||
Reference in New Issue
Block a user