123 lines
4.9 KiB
Markdown
123 lines
4.9 KiB
Markdown
# Document Ingestion Pipeline
|
|
|
|
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into Open WebUI's
|
|
knowledge base and make them searchable via MCP.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Nextcloud folder / Paperless webhook
|
|
→ n8n trigger (CT 104)
|
|
→ Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
|
|
→ TEI /v1/embeddings — multilingual-e5-base (local, 768d)
|
|
→ Qdrant (shared vector store, http://qdrant:6333)
|
|
→ Open WebUI knowledge base API ← searchable in chat
|
|
```
|
|
|
|
**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
|
|
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
|
|
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.
|
|
|
|
## Deployed components (as of 2026-05-26)
|
|
|
|
| Component | Location | Endpoint |
|
|
|-----------|----------|---------|
|
|
| **Qdrant** | CT 104, `ai-internal` net | `http://qdrant:6333` (REST), `qdrant:6334` (gRPC) |
|
|
| **nomic** | CT 104, `ai-internal` net | `http://nomic:80` — text+vision 768d |
|
|
| **TEI** (Text Embeddings Inference) | CT 104, `ai-internal` net | `http://tei:80` — text-only 768d (standby) |
|
|
| **Open WebUI** | CT 104, port 14002 | Uses Qdrant + nomic natively |
|
|
| **Bifrost** | CT 104, port 14003 | Semantic cache → Qdrant gRPC, mistral-embed 1024d |
|
|
| **Docling MCP** | CT 104, port 18005 | MCP server in gateway |
|
|
|
|
### Embedding stack
|
|
|
|
| Service | Model | Dimensions | Use |
|
|
|---------|-------|-----------|-----|
|
|
| **nomic** | `nomic-ai/nomic-embed-text-v1.5` + `nomic-embed-vision-v1.5` | 768d | OWUI RAG, ingest pipeline |
|
|
| **TEI** | `intfloat/multilingual-e5-base` | 768d | Standby; same vector space as nomic text |
|
|
| **Bifrost cache** | `mistral/mistral-embed` via Bifrost | 1024d | Semantic cache only (separate Qdrant collection) |
|
|
|
|
Both nomic models share a 768d embedding space — text and image queries work on the same `documents` Qdrant collection.
|
|
|
|
### OWUI RAG config (env-driven)
|
|
|
|
```
|
|
VECTOR_DB=qdrant
|
|
QDRANT_URI=http://qdrant:6333
|
|
RAG_EMBEDDING_ENGINE=openai
|
|
RAG_OPENAI_API_BASE_URL=http://nomic:80
|
|
RAG_OPENAI_API_KEY=none
|
|
RAG_EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5
|
|
CONTENT_EXTRACTION_ENGINE=docling
|
|
CHUNK_SIZE=1200
|
|
CHUNK_OVERLAP=150
|
|
ENABLE_RAG_HYBRID_SEARCH=true
|
|
```
|
|
|
|
To switch to Bifrost embeddings (1024d, better quality — requires re-indexing `documents` collection):
|
|
```
|
|
RAG_OPENAI_API_BASE_URL=http://bifrost:8080/v1
|
|
RAG_EMBEDDING_MODEL=mistral/mistral-embed
|
|
```
|
|
|
|
### Bifrost embedding models (available for external services / upgrade)
|
|
|
|
Three embedding models tested and working via `http://bifrost:8080/v1/embeddings`:
|
|
- `mistral/mistral-embed` (1024d) — ✓ production-ready
|
|
- `mistral/codestral-embed` (1024d) — ✓
|
|
- `gemini/gemini-embedding-001` (768d/1536d) — ✓
|
|
|
|
## Converter comparison
|
|
|
|
| Tool | Image | Formats | Notes |
|
|
|------|-------|---------|-------|
|
|
| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables |
|
|
| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs |
|
|
| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch |
|
|
|
|
Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.
|
|
|
|
## Sharing Qdrant with other services
|
|
|
|
Qdrant is on `ai-internal` network — any service on that network can use it:
|
|
|
|
```python
|
|
from qdrant_client import QdrantClient
|
|
client = QdrantClient(url="http://qdrant:6333")
|
|
```
|
|
|
|
n8n, MCP tools, and custom pipelines should use `http://nomic:80/v1/embeddings` for
|
|
consistent 768d vectors. Mixing models/dimensions in the same collection will fail.
|
|
|
|
## Bifrost semantic caching
|
|
|
|
Bifrost uses Qdrant (gRPC port 6334) as a semantic cache backend. Config lives in
|
|
`/opt/stacks/ai/bifrost/data/config.json`:
|
|
|
|
```json
|
|
{
|
|
"$schema": "https://www.getbifrost.ai/schema",
|
|
"vector_store": {"enabled": true, "type": "qdrant", "config": {"host": "qdrant", "port": 6334}},
|
|
"plugins": [{
|
|
"enabled": true, "name": "semantic_cache",
|
|
"config": {
|
|
"provider": "mistral", "embedding_model": "mistral-embed", "dimension": 1024,
|
|
"ttl": "10m", "threshold": 0.85, "conversation_history_threshold": 3, "exclude_system_prompt": true
|
|
}
|
|
}]
|
|
}
|
|
```
|
|
|
|
The semantic cache uses a separate Qdrant collection (auto-created) at 1024d — no collision with the `documents` collection at 768d. TTL: 10 min, similarity threshold: 0.85.
|
|
|
|
## Intel Arc GPU passthrough (enabled)
|
|
|
|
CT 104 Intel Core Ultra 7 155H iGPU is passed through via PVE `dev3`/`dev4` entries
|
|
(`/dev/dri/renderD128` and `/dev/dri/card1`). No TEI Intel image exists currently — passthrough is available for future inference acceleration.
|
|
|
|
## Pending
|
|
|
|
- n8n workflow: Nextcloud/Paperless → Docling → nomic → Qdrant
|
|
- MCP knowledge-search tool pointing at Qdrant (replaces LobeChat search intent)
|
|
- `ingest.py` filesystem scan script (see [ingest-pipeline.md](ingest-pipeline.md))
|