# Document Ingestion Pipeline (planned)

Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in
knowledge base and make them searchable via MCP.

## Architecture

```
Nextcloud folder / Paperless webhook
  → n8n trigger (already running on CT 104)
  → Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
  → LiteLLM /v1/embeddings  — Mistral-embed or text-embedding-3-large
  → LobeChat knowledge base API  ← natively searchable in chat
  → (optional) Qdrant sidecar — if multi-app vector search needed
```

LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its
Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing
piece is the n8n orchestration workflow.

**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.

## Converter comparison

| Tool | Image | Formats | Notes |
|------|-------|---------|-------|
| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables |
| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs |
| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch |

Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.

## MCP knowledge search

Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base
search API. No new infra — LobeChat is already on `shared_backend`.

## Embedding options

- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces
- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom
  provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
- **Mistral-embed / codestral-embed** — already available via LiteLLM

## Status

Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API
integration are the remaining work.