Skip to content

# Ingest Pipeline — Filesystem Scan

One-shot pipeline that crawls a directory, converts documents to text, embeds them via the nomic service, and stores them in Qdrant. Run it manually against any mounted path.

## Architecture

     ┌─────────────────────────┐
     │  ingest.py --path /...  │  runs on CT 104 (has direct access to ai-internal network)
     └──────────┬──────────────┘
                │ per file
                ▼
     ┌────────────────────────────────────────────────────────────────┐
     │  1. File scanner          glob recursively, filter by ext      │
     │  2. Change detection      sha256(path + mtime) → skip if seen  │
     │  3. Docling converter     POST http://docling:5001/convert      │
     │     (PDF/DOCX/PPTX/XLSX/HTML → Markdown)                       │
     │     Plain text/Markdown   read directly                        │
     │     Images (jpg/png/...)  send to nomic vision endpoint        │
     │  4. Chunker               split Markdown by headers + size     │
     │     chunk_size=1200  overlap=150  (matches OWUI RAG config)    │
     │  5. Nomic embedder        POST http://nomic:80/v1/embeddings   │
     │     text chunks → 768d   images → 768d (same space!)          │
     │  6. Qdrant upsert         collection `documents`, named vectors │
     └────────────────────────────────────────────────────────────────┘
                │
                ▼
     ┌──────────────────────────────────────────────────────────────┐
     │  Qdrant: http://qdrant:6333                                  │
     │  collection: documents                                       │
     │  vector: {size: 768, distance: Cosine}                       │
     │  payload: {path, title, chunk_idx, text, type, mtime, hash} │
     └──────────────────────────────────────────────────────────────┘
                │
                ▼
     ┌────────────────────────────┐
     │  Open WebUI RAG search     │  queries `documents` collection
     │  n8n / MCP tools           │  same collection, same embedding space
     └────────────────────────────┘

## Why not n8n for this

n8n is the right tool for event-driven pipelines (webhook on new file, Paperless webhook, Nextcloud activity, scheduled re-sync). For a bulk one-shot crawl, a Python script is better: - direct filesystem access with `os.walk` - proper progress bar and error recovery - batched Qdrant upserts (n8n does one HTTP call per node) - easy CLI: `python3 ingest.py /mnt/pve/unas/Notizen`

n8n handles the ongoing layer (see Ongoing ingestion).

## Supported file types

Extension | Handler | Notes
--- | --- | ---
`.pdf` | Docling | best quality, preserves tables
`.docx`, `.odt` | Docling |
`.pptx` | Docling | slides → sections
`.xlsx`, `.ods` | Docling | tables → Markdown
`.html`, `.htm` | Docling |
`.md`, `.txt`, `.rst` | direct read | no conversion needed
`.jpg`, `.jpeg`, `.png`, `.webp`, `.gif` | nomic vision | 768d image embedding, no text chunks

## Script: `ingest.py`

Lives at `/opt/stacks/ai/ingest/ingest.py`. Runs directly on CT 104.

### Usage

    # Index everything under a path
    python3 /opt/stacks/ai/ingest/ingest.py --path /mnt/pve/unas/Notizen

    # Different collection, force re-index
    python3 /opt/stacks/ai/ingest/ingest.py \
      --path /mnt/pve/unas/Dokumente \
      --collection work-docs \
      --force

    # Dry run (print files, don't embed)
    python3 /opt/stacks/ai/ingest/ingest.py --path /path/to/docs --dry-run

### State file

`/opt/stacks/ai/ingest/state/<collection>.json` tracks `{path: {hash, indexed_at}}`. Subsequent runs skip unchanged files. Delete the state file to force full re-index.

### Qdrant collection schema

    vectors_config = VectorParams(size=768, distance=Distance.COSINE)
    # payload per point:
    {
        "path":      "/mnt/pve/unas/Notizen/someFile.md",
        "title":     "someFile",        # filename without ext
        "chunk_idx": 0,                 # 0-based chunk index within file
        "total_chunks": 3,
        "text":      "…chunk content…", # empty string for images
        "type":      "markdown",        # markdown | pdf | docx | image | …
        "mtime":     1716700000.0,
        "hash":      "a3f…",            # sha256 of file content
        "source":    "filesystem",
    }

### Chunking strategy

For Markdown output from Docling (and raw .md/.txt): 1. Split on `##` / `###` headers first (keep header as first line of chunk) 2. If chunk > 1200 chars, split further on double-newline (`\n\n`) 3. If still > 1200 chars, hard-split with 150-char overlap

Images: single point per file, no chunking.

### Error handling

- Docling timeout (>60s): skip file, log to `ingest_errors.log`, continue
- Qdrant upsert failure: retry 3×, then log and continue
- Re-run is safe: state file prevents double-indexing

## Building and running

    # On CT 104
    mkdir -p /opt/stacks/ai/ingest
    cd /opt/stacks/ai/ingest

    # Install deps (lightweight — no torch needed, calls services via HTTP)
    pip3 install qdrant-client requests tqdm

    # Run
    python3 ingest.py --path /mnt/pve/unas/Notizen

No container needed for the script itself — it runs on CT 104 bare Python and calls `http://docling:5001`, `http://nomic:80`, `http://qdrant:6333` via `ai-internal` (all on the same Docker network/host).

To run it from outside CT 104 (e.g. the PVE host), wrap it in a container later.

## Open WebUI integration

OWUI's knowledge base already points at Qdrant (`VECTOR_DB=qdrant`). To surface documents from the `documents` collection in chat:

1. Admin → Knowledge → Create Knowledge Base
2. Name: "Local Documents"
3. The collection is populated by `ingest.py` — OWUI will search it on `#`-prefixed RAG queries or when the knowledge base is enabled in a chat.

> Note: OWUI creates its own internal collection names. To share the same `documents` collection between `ingest.py` and OWUI, use the OWUI API to create a knowledge base pointing to the pre-populated collection — or let OWUI manage its own collection and have `ingest.py` add documents via the OWUI knowledge API (`POST /api/v1/knowledge/{id}/file/add`). The OWUI API path is cleaner for OWUI search integration; the direct Qdrant path is better for external tools (n8n, MCP).

## Ongoing ingestion (n8n)

After the one-shot crawl, wire n8n for continuous ingestion:

Trigger | n8n nodes | Notes
--- | --- | ---
Nextcloud webhook (file created/modified) | HTTP → SSH → `ingest.py --path <file>` | Nextcloud admin → Webhooks app
Paperless post-consume webhook | HTTP → Docling → nomic → Qdrant upsert | Paperless has `POST_CONSUME_SCRIPT` hook
Cron re-scan | Schedule → SSH → `ingest.py --path /mnt/pve/unas/Notizen` | weekly full re-sync

The cron re-scan is safe because `ingest.py` skips unchanged files via state hash.

## Paths available on CT 104

Path | Contents
--- | ---
`/mnt/pve/unas/Notizen/` | Obsidian vault (Markdown)
`/mnt/pve/unas/` | full UNAS NFS share
`/opt/stacks/*/` | stack configs (already in Git, lower priority)

Nextcloud files are on CT 105 (`192.168.1.41`). Access via: - WebDAV: `https://nc.nuclide.systems/remote.php/dav/files/fkrebs@nucli.de/` - Or mount the NC data volume — not currently mounted on CT 104

## Status

Not yet implemented. Design only. Next step: write `ingest.py`.