docs: portmap shepard-mcp decommissioned; add doc-ingestion pipeline design

- portmap: mark port 18010 (shepard-mcp) as DECOMMISSIONED 2026-05-21;
  remove mcp-shepard from Gitea repos note
- services/doc-ingestion.md: new — n8n → Docling → LiteLLM embeddings →
  LobeChat knowledge base pipeline architecture + converter comparison table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-21 09:33:35 +02:00
parent 0baf283678
commit 52db8ce9ed
2 changed files with 52 additions and 2 deletions
+50
View File
@@ -0,0 +1,50 @@
# Document Ingestion Pipeline (planned)
Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in
knowledge base and make them searchable via MCP.
## Architecture
```
Nextcloud folder / Paperless webhook
→ n8n trigger (already running on CT 104)
→ Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
→ LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large
→ LobeChat knowledge base API ← natively searchable in chat
→ (optional) Qdrant sidecar — if multi-app vector search needed
```
LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its
Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing
piece is the n8n orchestration workflow.
**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is
available at `/api/documents/?added__gt=<last_run>`. An n8n workflow can re-embed
directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.
## Converter comparison
| Tool | Image | Formats | Notes |
|------|-------|---------|-------|
| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables |
| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs |
| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch |
Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.
## MCP knowledge search
Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base
search API. No new infra — LobeChat is already on `shared_backend`.
## Embedding options
- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces
- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom
provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
- **Mistral-embed / codestral-embed** — already available via LiteLLM
## Status
Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API
integration are the remaining work.