docs: portmap shepard-mcp decommissioned; add doc-ingestion pipeline design
- portmap: mark port 18010 (shepard-mcp) as DECOMMISSIONED 2026-05-21; remove mcp-shepard from Gitea repos note - services/doc-ingestion.md: new — n8n → Docling → LiteLLM embeddings → LobeChat knowledge base pipeline architecture + converter comparison table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# Document Ingestion Pipeline (planned)
|
||||
|
||||
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in
|
||||
knowledge base and make them searchable via MCP.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Nextcloud folder / Paperless webhook
|
||||
→ n8n trigger (already running on CT 104)
|
||||
→ Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
|
||||
→ LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large
|
||||
→ LobeChat knowledge base API ← natively searchable in chat
|
||||
→ (optional) Qdrant sidecar — if multi-app vector search needed
|
||||
```
|
||||
|
||||
LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its
|
||||
Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing
|
||||
piece is the n8n orchestration workflow.
|
||||
|
||||
**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
|
||||
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
|
||||
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.
|
||||
|
||||
## Converter comparison
|
||||
|
||||
| Tool | Image | Formats | Notes |
|
||||
|------|-------|---------|-------|
|
||||
| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables |
|
||||
| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs |
|
||||
| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch |
|
||||
|
||||
Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.
|
||||
|
||||
## MCP knowledge search
|
||||
|
||||
Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base
|
||||
search API. No new infra — LobeChat is already on `shared_backend`.
|
||||
|
||||
## Embedding options
|
||||
|
||||
- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces
|
||||
- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom
|
||||
provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
|
||||
- **Mistral-embed / codestral-embed** — already available via LiteLLM
|
||||
|
||||
## Status
|
||||
|
||||
Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API
|
||||
integration are the remaining work.
|
||||
Reference in New Issue
Block a user