From 52db8ce9edd2916863563a5c5da2a9a6a1901071 Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Thu, 21 May 2026 09:33:35 +0200 Subject: [PATCH] docs: portmap shepard-mcp decommissioned; add doc-ingestion pipeline design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - portmap: mark port 18010 (shepard-mcp) as DECOMMISSIONED 2026-05-21; remove mcp-shepard from Gitea repos note - services/doc-ingestion.md: new — n8n → Docling → LiteLLM embeddings → LobeChat knowledge base pipeline architecture + converter comparison table Co-Authored-By: Claude Sonnet 4.6 --- infra/portmap.md | 4 ++-- services/doc-ingestion.md | 50 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 2 deletions(-) create mode 100644 services/doc-ingestion.md diff --git a/infra/portmap.md b/infra/portmap.md index e595a47..e2e39b3 100644 --- a/infra/portmap.md +++ b/infra/portmap.md @@ -83,7 +83,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway | | [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway | | [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only | -| [18010](http://192.168.1.40:18010) | Shepard MCP | `shepard-mcp` | — | Shepard research platform; reachable via gateway | +| ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — Shepard will provide a native MCP server | | [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | @@ -122,7 +122,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` | +| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` (`mcp-shepard` decommissioned) | --- diff --git a/services/doc-ingestion.md b/services/doc-ingestion.md new file mode 100644 index 0000000..f31f128 --- /dev/null +++ b/services/doc-ingestion.md @@ -0,0 +1,50 @@ +# Document Ingestion Pipeline (planned) + +Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in +knowledge base and make them searchable via MCP. + +## Architecture + +``` +Nextcloud folder / Paperless webhook + → n8n trigger (already running on CT 104) + → Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure + → LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large + → LobeChat knowledge base API ← natively searchable in chat + → (optional) Qdrant sidecar — if multi-app vector search needed +``` + +LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its +Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing +piece is the n8n orchestration workflow. + +**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is +available at `/api/documents/?added__gt=`. An n8n workflow can re-embed +directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs. + +## Converter comparison + +| Tool | Image | Formats | Notes | +|------|-------|---------|-------| +| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables | +| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs | +| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch | + +Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed. + +## MCP knowledge search + +Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base +search API. No new infra — LobeChat is already on `shared_backend`. + +## Embedding options + +- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces +- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom + provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired +- **Mistral-embed / codestral-embed** — already available via LiteLLM + +## Status + +Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API +integration are the remaining work.