nexa/docs/09-deployment.md

# 09 — Deployment

Pragmatic deployment guide that **assumes the existing homelab** and adds only what's missing.

## What's already running (no action required)

Surveyed from Homepage / Dozzle / Proxmox / Zoraxy:

| Service | Host / port | URL |
|---------|-------------|-----|
| Memos | docker LXC 104 → `:5230` | `https://memos.nuclide.systems` |
| n8n | docker LXC 104 → `:5678` | `https://n8n.nuclide.systems` |
| LiteLLM (SAIA gateway) | docker LXC 104 → `:4000` | `https://ai.nuclide.systems` (proxies LobeHub UI :3210; API on :4000) |
| Nextcloud | LXC 105 | `https://nc.nuclide.systems` |
| ntfy | docker LXC 104 → `:7998` | `https://ntfy.nuclide.systems` |
| Karakeep | docker LXC 104 → `:3090` | `https://hoarder.nuclide.systems` (legacy host alias kept for compatibility) |
| Home Assistant | VM 100 (HAOS) | `https://ha.nuclide.systems` |
| Pocket-ID (OAuth/SSO) | docker LXC 104 → `:1411` | `https://id.nuclide.systems` |
| Vaultwarden | docker LXC 104 → `:11001` | `https://vault.nuclide.systems` |
| Backrest | LXC 103 | (internal) |
| AdGuard DNS | LXC 102 | (internal) |
| Zoraxy reverse proxy | LXC 108 → `192.168.1.4:8000` | TLS for *.nuclide.systems |
| `qdrant_scientific` (existing) | docker LXC 104 | **reused** — Nexa uses `nexa_*` collections in this instance |

The deployment task is **not** "spin up the stack" — most of the stack is already up. It is **wire Nexa across these services + add the small bits that are missing**.

## What's missing for Nexa

1. **Qdrant collection** for Nexa (`nexa_knowledge`) inside the existing `qdrant_scientific` instance — vector dim follows Q15 (1024 for `bge-m3`, 768 for `nomic-embed-text`).
2. **TEI** (HF text-embeddings-inference) on the docker host for self-hosted embeddings (LiteLLM key is **not** authorised for OpenAI embeddings — see [11/Q3+Q15](./11-open-questions.md)). Lighter than Ollama: single Rust binary, ~500 MB image, no LLM runtime.
3. **n8n workflows** (`./nexa-core/n8n-workflows/`) imported into the running n8n.
4. **Nextcloud lists & calendars** for Work / Personal / Shopping / Wishes (auto-discovered via `#nexa:config`).
5. **Memos webhook → n8n** wired through the Memos config.
6. **LiteLLM virtual key** for the `nexa` user with chat-only access (no embeddings — handled by Ollama).
7. **A Zoraxy host entry** is *not* needed — Memos / n8n / LiteLLM are already proxied.
8. **(Phase 3.4) Ontotext GraphDB** for the SPARQL pillar — see add-on at the bottom of this doc.

---

## Step 1 — Secrets

Copy `nexa-core/.env.example` → `nexa-core/.env` and fill **only** the secrets:

```bash
cd nexa-core
cp .env.example .env
$EDITOR .env       # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY
```

The `.env` is **only used at bootstrap time**. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via `#nexa:config` (see [05](./05-command-system.md)). No secrets should ever live in n8n workflow JSON — use n8n credentials instead.

## Step 2 — Qdrant collection (`nexa_knowledge_text`)

Phase 3.1 ships **Path A** (text-only) but the schema and naming already make room for **Path C** (text + visual) so adding a `nexa_knowledge_visual` collection later is a pure additive operation — no rename, no migration, no n8n rewiring.

```bash
# adjust QDRANT_HOST in .env first
source nexa-core/.env

# create the text collection from the schema file
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_text" \
  -H "Content-Type: application/json" \
  -H "api-key: $QDRANT_API_KEY" \
  -d @nexa-core/config/qdrant_schema.json
```

The collection name is **always suffixed with the modality** (`_text`, `_visual`) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields ([source](../nexa-core/config/qdrant_schema.json)):

| Field | Why it's there now |
|-------|--------------------|
| `modality` | Always `"text"` in `_text`, `"image"` in `_visual`. Future-proofs cross-modality filters. |
| `source_type` | `memo` / `mail` / `obsidian` / `screenshot` / `image` — used by classification and digest workflows. |
| `media_uri` | `memos://…`, `nextcloud://…`, `obsidian://…`. Empty for text-only rows; populated when Path C ships. |
| `graph_iri` | IRI of the corresponding `nexa:Note` in GraphDB. The same value is stored on the GraphDB side as `nexa:vectorId` — this is the cross-pillar bridge. |
| `content_hash` | de-dup. |
| `context` | `work` / `personal`. |

> Targets the existing `qdrant_scientific` instance — just an extra collection, no new container.
> The `vectors.size` field follows [Q15](./11-open-questions.md): **1024** for `bge-m3`, **768** for `nomic-embed-text-v1.5`.

### Image attachments today (queue them)

Memos can already attach images. Until Phase 3.2 the indexer **does not** embed them, but it **does** record them so they can be replayed later:

- Memo with an image → text body still goes into `nexa_knowledge_text`.
- The image attachment(s) are written as `nexa:Note` triples in GraphDB with `nexa:modality "image"` and `nexa:vectorId` left empty (`nexa:pendingVisualIndex true`).
- A Phase-3.2 backfill workflow will pick up everything where `?n nexa:pendingVisualIndex true` and embed it through the visual collection.

This means **no data is lost** between 3.1 and 3.2 — the queue is the GraphDB itself.

## Step 3 — Self-hosted embeddings (TEI)

Use HuggingFace **text-embeddings-inference** — single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.

The active docker manager on this LXC is **Arcane** (visible from Homepage as the running container manager — the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see [12/#33](./12-optimization-opportunities.md#33)). Paste the stack into Arcane → name it `nexa` → save → start. Don't `docker compose up -d` over SSH; Arcane manages the compose lifecycle.

```yaml
# Nexa stack — paste into Arcane
services:
  nexa-embed:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
    container_name: nexa-embed
    restart: unless-stopped
    command: ["--model-id", "BAAI/bge-m3"]
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - tei-cache:/data

volumes:
  # SMB volume on UNAS — sidesteps NFS+unprivileged-LXC permission issues
  # (the same kind that bit the Nextcloud rollout). Stores embedding model
  # caches, Qdrant snapshots, future GraphDB exports.
  #
  # The exact mount block (driver, share name, credential injection) MUST
  # be copied verbatim from an existing stack on the same docker host —
  # services/<svc>/ on UNAS is the homelab-wide docker config store and
  # every container already follows the same SMB pattern. See Q19 in
  # docs/11. The block below is the *shape* but the share name "<unas-share>"
  # is a placeholder until verified against e.g. services/karakeep/compose.
  tei-cache:
    driver: local
    driver_opts:
      type: cifs
      device: //192.168.1.31/<unas-share>/services/nexa/tei-cache
      o: "username=${UNAS_USER},password=${UNAS_PASS},uid=1000,gid=1000,vers=3.0,iocharset=utf8"

networks:
  default:
    external: true
    name: nexa
```

Memory budget: ~1.1 GB resident. First start downloads `bge-m3` (~1 GB) into the SMB-backed volume; subsequent restarts are instant.

> **Why SMB and not the Proxmox `/mnt/pve/unas` NFS mount?** The host-side NFS mount works fine for host admin, but **unprivileged LXC + NFS + per-container writes** historically breaks (user confirmed: "had it with Nextcloud"). Mounting SMB **directly as a docker volume** with explicit `username=`/`uid=`/`gid=` bypasses LXC uid-mapping entirely — the docker daemon is root inside the LXC so it can mount, and the container sees a regular volume with the right ownership.
>
> Add `UNAS_USER` / `UNAS_PASS` to the stack's `.env` (Arcane provides an editor for this), or — better — store them in Vaultwarden and inject via a secrets agent (see [12/#11](./12-optimization-opportunities.md#11)).

Register it inside LiteLLM (admin UI → Models) with the OpenAI-compatible adapter:

- model name: `nexa-embed`
- provider: `openai`
- model: `bge-m3`
- api_base: `http://nexa-embed:80/v1`
- api_key: any non-empty string (TEI ignores it)

Now n8n only ever talks to LiteLLM and the model is swappable without touching workflows.

## Step 4 — LiteLLM virtual key

In the LiteLLM admin UI (`ai.nuclide.systems`):

1. Create user `nexa`.
2. Issue a virtual key with access to:
   - one chat model (already-available model from your SAIA gateway).
   - the `nexa-embed` model from Step 3.
3. Paste the key into `SAIA_API_KEY` in `.env`.

## Step 4 — n8n workflows

Import the JSON exports — credentials are filled inside n8n, not in the JSON:

```bash
# n8n personal access token from the n8n UI: Settings → API
N8N_URL=https://n8n.nuclide.systems
N8N_TOKEN=...    # from the n8n UI

for f in nexa-core/n8n-workflows/phase-1/*.json \
         nexa-core/n8n-workflows/phase-2/*.json; do
  curl -X POST "$N8N_URL/api/v1/workflows" \
    -H "X-N8N-API-KEY: $N8N_TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary "@$f"
done
```

Inside n8n, attach credentials to the imported nodes:

- **Memos** → HTTP header `Authorization: Bearer $MEMOS_API_KEY`
- **LiteLLM** → header `Authorization: Bearer $SAIA_API_KEY` (chat + `nexa-embed`)
- **Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under `Notizen/`)** → one app password (`$NC_APP_PASSWORD`), reused across all three node types
- **Qdrant** → header `api-key: $QDRANT_API_KEY`

Activate each workflow individually after smoke-test.

## Step 5 — Memos webhook

In the Memos admin UI, set the webhook URL to the production address of the discovery workflow:

```
https://n8n.nuclide.systems/webhook/memos
```

The same URL is the one the workflow exposes; verify with:

```bash
curl -i https://n8n.nuclide.systems/webhook/memos
# expect 200 / 405, never 404
```

## Step 6 — Bootstrap commands via Memos

Create a memo with body `#nexa:config` — the discovery workflow:

1. Lists Nextcloud Tasks lists, looking for `Persönlich`, `DLR`, `Einkaufsliste`, `Wunschliste` (and any new lists added later — see [05](./05-command-system.md)). Caches `name → id`.
2. Lists Nextcloud Calendars by the same names; caches IDs.
3. Verifies the Nextcloud Mail account `fkrebs@nucli.de` and the folder list (`Posteingang`, `Archiv`, `Junk`, `Waiting`).
4. Counts existing Qdrant points in `nexa_knowledge_text`.
5. Verifies LiteLLM reachability + lists available models for the Nexa virtual key.
6. Replies as a comment with a **runtime-config snapshot** that's stored in the Qdrant `_config` namespace (and mirrored as `nexa-core/config/runtime_config.json`, gitignored).

The same workflow is also triggered by:
- A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention).
- Cache-miss in the task-router — if a list ID 404s, the router fires `#nexa:config` once and retries.

After this point, `.env` is read-once. Subsequent runs read config from Qdrant.

## Step 7 — Smoke tests

```bash
# (1) Memos round-trip — should produce a comment within ~5 s
curl -X POST https://memos.nuclide.systems/api/v1/memos \
  -H "Authorization: Bearer $MEMOS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"content":"- [ ] testing the router #nexa"}'

# (2) Classification dry-run
curl -X POST https://memos.nuclide.systems/api/v1/memos \
  -H "Authorization: Bearer $MEMOS_API_KEY" \
  -d '{"content":"#nexa:route-test buy milk"}'

# (3) RAG test (requires at least one indexed memo/note)
curl -X POST https://memos.nuclide.systems/api/v1/memos \
  -H "Authorization: Bearer $MEMOS_API_KEY" \
  -d '{"content":"#nexa:ask what is the goal of nexa?"}'
```

## Step 8 — Reverse proxy

Already done — Zoraxy at `192.168.1.4:8000` terminates TLS for `*.nuclide.systems` and forwards to docker LXC 104 (`192.168.1.40`). **No new entry is required for Nexa**: every service Nexa talks to already has a host entry.

## Step 9 — Backups

Already covered by Backrest (LXC 103). Add:

- **n8n workflows** → `nexa-core/scripts/backup_workflows.sh` (already present) into a Backrest schedule.
- **Qdrant snapshots** → schedule a daily `POST /collections/nexa_knowledge/snapshots` and rsync to S3 (`s3.nuclide.systems`). Add as a Backrest pre-hook on the docker host.

For deeper detail: [10 — Operations](./10-operations.md).

---

## Phase add-on: Ontotext GraphDB (Phase 3.4)

Defer until 3.1–3.3 ship.

```yaml
# nexa-core/docker-compose.graph.yml
services:
  graphdb:
    image: ontotext/graphdb:10.7.0
    container_name: nexa-graphdb
    ports: ["127.0.0.1:7200:7200"]
    environment:
      GDB_JAVA_OPTS: "-Xmx4g -Xms1g"
    volumes:
      - ./data/graphdb:/opt/graphdb/home
    restart: unless-stopped
```

After first start, create the repository (one-time):

```bash
curl -X POST http://localhost:7200/rest/repositories \
  -H 'Content-Type: application/json' \
  -d '{
    "id": "nexa_knowledge",
    "title": "Nexa Knowledge Graph",
    "type": "graphdb",
    "params": {
      "ruleset":     {"value": "rdfsplus-optimized"},
      "baseURL":     {"value": "https://nuclide.systems/nexa/"}
    }
  }'
```

Optional Zoraxy entry `graph.nuclide.systems` → `192.168.1.40:7200` if you want the SPARQL Workbench in a browser; otherwise n8n talks to it on the docker network at `http://nexa-graphdb:7200`.

For schema and example queries: [08-graphrag-architecture](./08-graphrag-architecture.md).

---

## Phase add-on: visual collection (Phase 3.2)

Adds Path C — image embeddings without disturbing the text path. Schema is already in `nexa-core/config/qdrant_schema_visual.json`.

```bash
# (1) replace TEI with infinity (or run alongside) for CLIP-family support
docker rm -f nexa-embed
docker run -d --name nexa-embed \
  --restart unless-stopped \
  -p 127.0.0.1:8080:80 \
  -v infinity-data:/app/.cache \
  michaelf34/infinity:latest \
  v2 \
  --model-id BAAI/bge-m3 \
  --model-id jinaai/jina-clip-v2 \
  --port 80

# (2) create the visual collection
curl -X PUT "$QDRANT_HOST/collections/nexa_knowledge_visual" \
  -H "Content-Type: application/json" \
  -H "api-key: $QDRANT_API_KEY" \
  -d @nexa-core/config/qdrant_schema_visual.json

# (3) register the second model in LiteLLM as `nexa-embed-visual`
#     (same OpenAI-compatible route, different model id)

# (4) backfill queued images:
#     SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }
#     For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,
#     UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.
```

n8n RAG workflow gains a parallel branch: text-query → both `nexa-embed-text` and `nexa-embed-visual` text encoders → kNN against both collections → merge by score before SAIA prompt.

---

## Step-back / rollback

- **Disable any Nexa workflow** in n8n — deactivates the side effect immediately, Memos webhooks become no-ops.
- **Drop a Qdrant collection** — `curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H "api-key: $QDRANT_API_KEY"`.
- **Re-discover** — `#nexa:reset-config` then `#nexa:config`.