fkrebs
52db8ce9ed
- portmap: mark port 18010 (shepard-mcp) as DECOMMISSIONED 2026-05-21; remove mcp-shepard from Gitea repos note - services/doc-ingestion.md: new — n8n → Docling → LiteLLM embeddings → LobeChat knowledge base pipeline architecture + converter comparison table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
51 lines
2.2 KiB
Markdown
51 lines
2.2 KiB
Markdown
# Document Ingestion Pipeline (planned)
|
|
|
|
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in
|
|
knowledge base and make them searchable via MCP.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Nextcloud folder / Paperless webhook
|
|
→ n8n trigger (already running on CT 104)
|
|
→ Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
|
|
→ LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large
|
|
→ LobeChat knowledge base API ← natively searchable in chat
|
|
→ (optional) Qdrant sidecar — if multi-app vector search needed
|
|
```
|
|
|
|
LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its
|
|
Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing
|
|
piece is the n8n orchestration workflow.
|
|
|
|
**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
|
|
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
|
|
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.
|
|
|
|
## Converter comparison
|
|
|
|
| Tool | Image | Formats | Notes |
|
|
|------|-------|---------|-------|
|
|
| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables |
|
|
| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs |
|
|
| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch |
|
|
|
|
Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.
|
|
|
|
## MCP knowledge search
|
|
|
|
Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base
|
|
search API. No new infra — LobeChat is already on `shared_backend`.
|
|
|
|
## Embedding options
|
|
|
|
- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces
|
|
- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom
|
|
provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
|
|
- **Mistral-embed / codestral-embed** — already available via LiteLLM
|
|
|
|
## Status
|
|
|
|
Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API
|
|
integration are the remaining work.
|