Files

T

fkrebs 52db8ce9ed docs: portmap shepard-mcp decommissioned; add doc-ingestion pipeline design

- portmap: mark port 18010 (shepard-mcp) as DECOMMISSIONED 2026-05-21;
  remove mcp-shepard from Gitea repos note
- services/doc-ingestion.md: new — n8n → Docling → LiteLLM embeddings →
  LobeChat knowledge base pipeline architecture + converter comparison table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-21 09:33:35 +02:00

2.2 KiB

Raw Blame History

Document Ingestion Pipeline (planned)

Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in knowledge base and make them searchable via MCP.

Architecture

Nextcloud folder / Paperless webhook
  → n8n trigger (already running on CT 104)
  → Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure
  → LiteLLM /v1/embeddings  — Mistral-embed or text-embedding-3-large
  → LobeChat knowledge base API  ← natively searchable in chat
  → (optional) Qdrant sidecar — if multi-app vector search needed

LobeChat already has knowledge_base_files, chunks, embeddings tables in its Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing piece is the n8n orchestration workflow.

Paperless shortcut: Paperless-ngx already OCRs documents. Its full-text content is available at /api/documents/?added__gt=<last_run>. An n8n workflow can re-embed directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.

Converter comparison

Tool	Image	Formats	Notes
Docling (deployed)	MCP on 18005	PDF, DOCX, PPTX, XLSX, HTML	Best for structured Office/PDF with tables
MinerU	`opendatalab/mineru`	PDF (layout-aware, OCR)	Better for academic papers / scanned PDFs
Markitdown (in gateway)	—	Office, PDF	Ad-hoc only; not suitable for batch

Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed.

MCP knowledge search

Add a knowledge-search MCP tool to the gateway that calls LobeChat's knowledge base search API. No new infra — LobeChat is already on shared_backend.

Embedding options

text-embedding-3-large — already in LiteLLM, used by Coder workspaces
IBM Granite embedding 30M — tiny, fully local, CPU-only; add to LiteLLM as custom provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired
Mistral-embed / codestral-embed — already available via LiteLLM

Status

Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API integration are the remaining work.

2.2 KiB Raw Blame History