diff --git a/infra/portmap.md b/infra/portmap.md index e595a47..e2e39b3 100644 --- a/infra/portmap.md +++ b/infra/portmap.md @@ -83,7 +83,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | [18005](http://192.168.1.40:18005) | Docling MCP | `docling-mcp` | — | SAIA Docling PDF→Markdown; reachable via gateway | | [18007](http://192.168.1.40:18007) | Kroki MCP | `kroki-mcp` | — | Diagram rendering; reachable via gateway | | [18009](http://192.168.1.40:18009) | Speaches | `speaches` | — | TTS/STT; LAN only | -| [18010](http://192.168.1.40:18010) | Shepard MCP | `shepard-mcp` | — | Shepard research platform; reachable via gateway | +| ~~18010~~ | ~~Shepard MCP~~ | ~~`shepard-mcp`~~ | — | **DECOMMISSIONED 2026-05-21** — Shepard will provide a native MCP server | | [18011](http://192.168.1.40:18011) | Upload-artifact MCP | `upload-artifact-mcp` | — | S3 chat-artifacts upload; reachable via gateway | | — | SearXNG | `searxng` | — | Internal, `shared_backend`, used by LobeHub | @@ -122,7 +122,7 @@ All arr-stack services run behind `vpn_gluetun` container network. | Port | Service | Container | Public URL | Notes | |---|---|---|---|---| -| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-shepard`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` | +| — | MCP servers (Gitea repos) | See AI Stack §18000 | — | `mcp-comfyui`, `mcp-docling`, `mcp-upload-artifact` at `git.nuclide.systems/fkrebs/` (`mcp-shepard` decommissioned) | --- diff --git a/services/doc-ingestion.md b/services/doc-ingestion.md new file mode 100644 index 0000000..f31f128 --- /dev/null +++ b/services/doc-ingestion.md @@ -0,0 +1,50 @@ +# Document Ingestion Pipeline (planned) + +Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into LobeChat's built-in +knowledge base and make them searchable via MCP. + +## Architecture + +``` +Nextcloud folder / Paperless webhook + → n8n trigger (already running on CT 104) + → Docling MCP (port 18005) — PDF/DOCX/PPTX/XLSX → Markdown + structure + → LiteLLM /v1/embeddings — Mistral-embed or text-embedding-3-large + → LobeChat knowledge base API ← natively searchable in chat + → (optional) Qdrant sidecar — if multi-app vector search needed +``` + +LobeChat already has `knowledge_base_files`, `chunks`, `embeddings` tables in its +Postgres instance. Docling is already deployed as an MCP server on CT 104 — the missing +piece is the n8n orchestration workflow. + +**Paperless shortcut**: Paperless-ngx already OCRs documents. Its full-text content is +available at `/api/documents/?added__gt=`. An n8n workflow can re-embed +directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs. + +## Converter comparison + +| Tool | Image | Formats | Notes | +|------|-------|---------|-------| +| **Docling** (deployed) | MCP on 18005 | PDF, DOCX, PPTX, XLSX, HTML | Best for structured Office/PDF with tables | +| **MinerU** | `opendatalab/mineru` | PDF (layout-aware, OCR) | Better for academic papers / scanned PDFs | +| **Markitdown** (in gateway) | — | Office, PDF | Ad-hoc only; not suitable for batch | + +Start with Docling — already deployed. Add MinerU if academic paper OCR quality is needed. + +## MCP knowledge search + +Add a `knowledge-search` MCP tool to the gateway that calls LobeChat's knowledge base +search API. No new infra — LobeChat is already on `shared_backend`. + +## Embedding options + +- **text-embedding-3-large** — already in LiteLLM, used by Coder workspaces +- **IBM Granite embedding 30M** — tiny, fully local, CPU-only; add to LiteLLM as custom + provider pointing at an Ollama/IPEX-LLM instance if fully local embeddings are desired +- **Mistral-embed / codestral-embed** — already available via LiteLLM + +## Status + +Planned. Docling MCP is the only deployed component. n8n workflow and LobeChat API +integration are the remaining work.