1 line
716 KiB
JSON
1 line
716 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"nuclide.systems docs","text":"<p>Single source of truth for the homelab. Git-tracked since 2026-05-20.</p>"},{"location":"#read-this-first","title":"Read this first","text":"<ul> <li><code>/CLAUDE.md</code> (host root) \u2014 operator doctrine for any AI agent SSHing in. Skim before touching anything.</li> <li><code>RESUME.md</code> \u2014 open items + recent changes; update at end of every session.</li> <li><code>CHANGELOG.md</code> \u2014 what changed and when.</li> <li><code>ct-inventory.md</code> \u2014 canonical roster of LXCs / VM, IPs, role, sizing.</li> </ul>"},{"location":"#layout","title":"Layout","text":"<pre><code>/docs/\n\u251c\u2500\u2500 README.md \u2190 you are here\n\u251c\u2500\u2500 CHANGELOG.md \u2190 dated bullets of every infra change\n\u251c\u2500\u2500 ct-inventory.md \u2190 LXC/VM roster (single source of truth)\n\u2502\n\u251c\u2500\u2500 infra/ \u2190 host-level + cross-cutting state\n\u2502 \u2514\u2500\u2500 proxmox-state.md \u2190 the 900-line master state doc (sizing, ZFS, NFS, DNS)\n\u2502\n\u251c\u2500\u2500 services/ \u2190 per-service operational docs (current truth)\n\u2502 \u251c\u2500\u2500 homelab-architecture.md topology + design rationale\n\u2502 \u251c\u2500\u2500 dev-environment.md Coder + Gitea on CT 111\n\u2502 \u251c\u2500\u2500 mcp-gateway.md MCP gateway \u2014 DECOMMISSIONED 2026-05-26; see mcp-servers.md for Bifrost\n\u2502 \u251c\u2500\u2500 comfyui.md\n\u2502 \u251c\u2500\u2500 zoraxy.md reverse proxy\n\u2502 \u251c\u2500\u2500 adguard-dns.md DNS + rewrites\n\u2502 \u251c\u2500\u2500 cloud-gpu.md GPU passthrough + future external GPU\n\u2502 \u2514\u2500\u2500 llm-benchmark.md LiteLLM model TTFT + TPS benchmark\n\u2502\n\u251c\u2500\u2500 infra/ (continued)\n\u2502 \u251c\u2500\u2500 portmap.md canonical service \u2192 port \u2192 public hostname registry\n\u2502 \u251c\u2500\u2500 storage.md storage class per service (source of truth)\n\u2502 \u251c\u2500\u2500 volumes.md volume bindings per stack\n\u2502 \u251c\u2500\u2500 proxmox-memory-audit.md CT allocations, host budget, ComfyUI memory analysis\n\u2502 \u2514\u2500\u2500 docker-networks.md\n\u2502\n\u251c\u2500\u2500 stacks/ \u2190 CT 104 agent notes + legacy stack docs\n\u2502 \u2514\u2500\u2500 CLAUDE.md agent breadcrumbs (CT 104 specific)\n\u2502\n\u251c\u2500\u2500 services/ (continued)\n\u2502 \u251c\u2500\u2500 backrest.md Backrest backup (CT 103)\n\u2502 \u251c\u2500\u2500 secrets-manager.md Infisical (CT 112)\n\u2502 \u251c\u2500\u2500 databases.md Postgres CT 113 + per-stack DBs\n\u2502 \u2514\u2500\u2500 doc-ingestion.md Paperless + Docling\n\u2502\n\u251c\u2500\u2500 history/ \u2190 frozen / archived\n\u2502 \u251c\u2500\u2500 traefik-migration.md ABANDONED 2026-05-16 (Zoraxy chosen)\n\u2502 \u251c\u2500\u2500 traefik-migration-docker-labels.md ABANDONED 2026-05-16\n\u2502 \u251c\u2500\u2500 mcp-gateway-requirements.md SUPERSEDED by services/mcp-gateway.md\n\u2502 \u251c\u2500\u2500 scrubbing-list-2026-05-17.md snapshot of cleanup pass\n\u2502 \u2514\u2500\u2500 case-study.md narrative writeup\n\u2502\n\u251c\u2500\u2500 ideas/\n\u2502 \u2514\u2500\u2500 stack-ideas.md\n\u2502\n\u2514\u2500\u2500 security/ \u2190 audits + leak analyses\n \u251c\u2500\u2500 data-leak-audit-comparison.md\n \u251c\u2500\u2500 data-leak-audit-2026-05-20-tr004-cloud-sandbox.md\n \u251c\u2500\u2500 data-leak-audit-2026-05-21-tr004-artifacts.md\n \u2514\u2500\u2500 audit-claude-code-meta.md self-audit of the Claude Code session that produced these\n</code></pre>"},{"location":"#operating-principles","title":"Operating principles","text":"<p>See <code>/CLAUDE.md</code> for the full doctrine. Highlights: 1. Keep services running; surface synergies and gaps. 2. Enforce consistency: Pocket-ID OIDC, Zoraxy + ACME, <code><svc>.nuclide.systems</code>, secrets in <code>.env</code>. 3. Plan rollback at medium+ risk (ZFS snapshots, fresh experimental CTs). 4. Every service should be AI-accessible (MCP / Coder workspace CLI / documented API). 5. Test end-to-end before declaring done. 6. Generate slash commands for recurring maintenance.</p>"},{"location":"#quick-links","title":"Quick links","text":""},{"location":"#monitoring-ct-109-19216818","title":"Monitoring (CT 109 \u00b7 192.168.1.8)","text":"Service URL Notes Homarr <code>http://192.168.1.8:7575</code> Service dashboard \u2014 start here Grafana <code>http://192.168.1.8:3000</code> Dashboards (admin/tapirnase) Prometheus <code>http://192.168.1.8:9090</code> Metrics \u2014 90d retention Loki <code>http://192.168.1.8:3100</code> Logs \u2014 30d retention, 13 hosts Alloy UI <code>http://192.168.1.8:12345</code> Log agent pipeline inspector"},{"location":"#public-services-nuclidesystems","title":"Public services (<code>*.nuclide.systems</code>)","text":"<ul> <li>AI: ai.nuclide.systems (Bifrost \u2014 LLM + MCP gateway) \u00b7 chat.nuclide.systems (Open WebUI)</li> <li>Files: nc.nuclide.systems (Nextcloud) \u00b7 paperless.nuclide.systems \u00b7 immich.nuclide.systems</li> <li>Dev: git.nuclide.systems (Gitea) \u00b7 dev.nuclide.systems (Coder)</li> <li>Home: ha.nuclide.systems (Home Assistant) \u00b7 hoarder.nuclide.systems (Karakeep) \u00b7 memos.nuclide.systems</li> <li>Identity: id.nuclide.systems (Pocket-ID) \u00b7 vault.nuclide.systems (Vaultwarden)</li> </ul>"},{"location":"#operator-only-lan","title":"Operator-only (LAN)","text":"<ul> <li>Proxmox PVE: <code>https://192.168.1.20:8006</code></li> <li>Zoraxy admin: <code>http://192.168.1.4:8000</code> (always confirm before any change)</li> <li>AdGuard: <code>http://192.168.1.2</code> \u00b7 Backrest: <code>http://192.168.1.3:9898</code> \u00b7 Infisical: <code>http://192.168.1.7:8200</code></li> </ul>"},{"location":"#related-repos","title":"Related repos","text":"Repo Content Auto-sync fkrebs/docs This doc site Push after every doc change fkrebs/homelab-configs Homarr config, Alloy configs, monitoring compose files Cron 03:00 on CT 109 fkrebs/n8n-flows n8n workflow JSON exports + ideas backlog (10 issues) Cron 03:30 on CT 104 fkrebs/zoraxy-conf Zoraxy proxy routes (CT 108) Cron 03:00 on CT 108 fkrebs/adguard-conf AdGuard Home config (CT 102) Cron 03:00 on CT 102 fkrebs/pve-conf Proxmox LXC/VM configs + storage Cron 03:00 on nuc fkrebs/ct103-backrest Backrest config + scripts (CT 103) Cron 03:15 on nuc fkrebs/ct104-stacks CT 104 untracked stack composes (no .env) Cron 03:15 on nuc fkrebs/ct110-pocket-id Pocket-ID compose (CT 110) Cron 03:15 on nuc fkrebs/ct111-dev Coder templates + act-runner (CT 111) Cron 03:15 on nuc fkrebs/ct112-infisical Infisical compose (CT 112, private) Cron 03:15 on nuc fkrebs/unas-conf UNAS Pro NFS exports, disk usage, cron (192.168.1.31) Cron 03:20 on nuc fkrebs/grafana-dashboards Grafana dashboard JSON exports + datasources (CT 109) Cron 03:25 on CT 109 fkrebs/ct113-db Postgres + pgAdmin compose (CT 113) Cron 03:15 on nuc fkrebs/klipper-config Klipper 3D printer config Daily cron from QNAP"},{"location":"#rendered-docs-site","title":"Rendered docs site","text":"<ul> <li>Source: <code>https://git.nuclide.systems/fkrebs/docs</code></li> <li>Rendered (LAN-only): <code>http://192.168.1.8:13080</code></li> <li>Build: docs-server on CT 109 polls Gitea every 5 min; rebuilds with mkdocs Material on push.</li> </ul>"},{"location":"CHANGELOG/","title":"Changelog","text":"<p>All notable infrastructure / service / doc changes. Newest first.</p>"},{"location":"CHANGELOG/#2026-05-23","title":"2026-05-23","text":""},{"location":"CHANGELOG/#observability-stack-ct-109-ops","title":"Observability stack (CT 109 \"ops\")","text":"<ul> <li> <p>CT 109 (\"ops\") fully provisioned: Debian 13, Docker 29.5.2. Stack <code>/opt/stacks/monitoring/</code> running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in <code>ct-inventory.md</code>.</p> </li> <li> <p>Loki deployed on CT 109: log aggregation at <code>:3100</code>, 30-day retention, TSDB v13 schema, filesystem storage at <code>./loki/data/</code>. <code>limits_config</code> set <code>reject_old_samples_max_age: 168h</code> (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via <code>provisioning/datasources/loki.yml</code>. Grafana Loki datasource UID: <code>P8E80F9AEF21F6940</code>.</p> </li> <li> <p>Prometheus remote_write receiver enabled: <code>--web.enable-remote-write-receiver</code> added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.</p> </li> <li> <p>prometheus-pve-exporter deployed: <code>prompve/prometheus-pve-exporter:latest</code> on CT 109; credentials in <code>./pve-exporter/pve.yml</code> (chmod 644 \u2014 container runs non-root). PVE <code>monitor@pve</code> service user + API token <code>c4be45dc-...</code> created with <code>PVEAuditor</code> role on <code>/</code>. Prometheus scrapes <code>:9221/pve?target=192.168.1.20</code>. Metrics: per-VM/CT CPU, memory, disk, network I/O.</p> </li> <li> <p>HA prometheus integration added: <code>prometheus:</code> block in HA <code>configuration.yaml</code>; HA exposes ~9211 <code>hass_*</code> entity-level metrics at <code>:8123/api/prometheus</code>. CT 109 Prometheus scrapes with bearer token (HA long-lived token), <code>scrape_interval: 60s</code>. This is distinct from Alloy host-level metrics \u2014 both needed.</p> </li> <li> <p>Grafana Alloy deployed across all 12 hosts: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to <code>http://192.168.1.8:3100/loki/api/v1/push</code>. Deployment breakdown:</p> </li> <li>Docker Compose (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, <code>group_add: [\"999\"]</code> for journal access.</li> <li>Binary + systemd (CT 102, 103, 108, nuc): apt install <code>alloy</code>, config at <code>/etc/alloy/config.alloy</code>, <code>systemctl enable --now alloy</code>. Journal-only (no Docker socket on these hosts).</li> <li>HA add-on (VM 100): <code>wymangr/hassos-addons</code> Grafana Alloy v0.0.8 \u2014 config fills in <code>prometheus_remote_write</code> to CT 109 + Loki endpoint. Distinct host label <code>homeassistant</code>.</li> <li> <p>All configs stored in <code>/opt/homelab-configs/alloy/</code> (see homelab-configs repo below).</p> </li> <li> <p>5 Grafana dashboards provisioned:</p> </li> <li>Node Exporter Full (Grafana ID 1860) \u2014 imported for all hosts</li> <li>LiteLLM (custom-built) \u2014 <code>/d/.../litellm</code>; panels: request rate, error rate, token throughput, model split, p95 latency</li> <li>Home Assistant (Grafana ID 12481 + custom panels) \u2014 power/energy, climate (temp/humidity/CO\u2082), home activity, device health/battery</li> <li>Loki / log explorer \u2014 imported</li> <li> <p>Prometheus target health \u2014 imported</p> </li> <li> <p>3 Grafana alert rules created via <code>/api/v1/provisioning/alert-rules</code> (folder <code>alerts-folder</code>):</p> </li> <li><code>Node disk > 85%</code> \u2014 per-instance mountpoint disk usage, 5m window</li> <li><code>Prometheus target down</code> \u2014 <code>count(up == 0) > 0</code>, 5m window</li> <li><code>LiteLLM error rate > 5%</code> \u2014 10m rate window, 5m pending</li> </ul>"},{"location":"CHANGELOG/#homarr-ct-109","title":"Homarr (CT 109)","text":"<ul> <li> <p>Homarr v1 deployed: <code>ghcr.io/homarr-labs/homarr:latest</code> on CT 109 <code>:7575</code>. Data at <code>./data/</code>, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).</p> </li> <li> <p>homelab-configs Gitea repo: private repo <code>fkrebs/homelab-configs</code> at <code>git.nuclide.systems</code>; cloned to <code>/opt/homelab-configs</code> on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at <code>/usr/local/bin/sync-homelab-configs</code>. Commit + push on any diff.</p> </li> </ul>"},{"location":"CHANGELOG/#pocket-id-oidc-wired-via-api","title":"Pocket-ID OIDC (wired via API)","text":"<ul> <li> <p>Homarr OIDC: client <code>63a94e30</code> created via Pocket-ID API. Redirect URI <code>http://192.168.1.8:7575/api/auth/callback/oidc</code>. Homarr compose updated (<code>AUTH_PROVIDER=oidc</code>, <code>AUTH_OIDC_CLIENT_ID</code>, <code>AUTH_OIDC_CLIENT_SECRET</code>, <code>AUTH_OIDC_ISSUER</code>, etc.), force-recreated. SSO active 2026-05-23.</p> </li> <li> <p>Grafana OIDC: client <code>92d987d5</code> created via Pocket-ID API with PKCE enabled. Redirect URI <code>http://192.168.1.8:3000/login/generic_oauth</code>. Grafana compose updated with <code>GF_AUTH_GENERIC_OAUTH_*</code> vars, force-recreated. SSO active 2026-05-23; role mapping: <code>admins</code> group \u2192 Admin, else Viewer.</p> </li> </ul>"},{"location":"CHANGELOG/#arcane-dozzle-migrated-to-ct-109-headless-agents-on-all-hosts","title":"Arcane + Dozzle migrated to CT 109; headless agents on all hosts","text":"<ul> <li>Arcane server moved CT 104 \u2192 CT 109: stack at <code>/opt/stacks/arcane/docker-compose.yml</code> on CT 109, <code>APP_URL=http://192.168.1.8:10002</code>. Zoraxy proxy route for <code>arcane.nuclide.systems</code> updated to <code>192.168.1.8:10002</code>. Old CT 104 arcane container stopped (renamed decommissioned).</li> <li>Dozzle server moved CT 104 \u2192 CT 109: stack at <code>/opt/stacks/dozzle/docker-compose.yml</code> (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).</li> <li>Headless agents deployed on CT 101, 104, 105, 110, 111, 112 via <code>/opt/stacks/ops-agents/docker-compose.yml</code> (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated <code>MANAGER_API_URL</code> to CT 109 and added Dozzle agent.</li> <li>docs-server migrated CT 111 \u2192 CT 109: <code>mkdocs.yml</code> site_url fixed, nav expanded. docs-server now at <code>http://192.168.1.8:13080</code>.</li> </ul>"},{"location":"CHANGELOG/#haos-kvm-oom-protection","title":"HAOS KVM OOM protection","text":"<ul> <li>Root cause identified: karakeep_chrome (CT 104, <code>gcr.io/zenika-hub/alpine-chrome:123</code>) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally \u2014 it was being killed externally.</li> <li>shm_size: '512m' added to karakeep chrome service and force-recreated to reduce Chrome shared memory pressure.</li> <li>oom_score_adj=-300 set on HAOS KVM PID (protects it from OOM killer \u2014 minimum score = hardest to kill).</li> <li>Persistence via systemd timer (<code>protect-haos-kvm.timer</code> on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.</li> </ul>"},{"location":"CHANGELOG/#unas-pro-config-tracking","title":"UNAS Pro config tracking","text":"<ul> <li>SSH key auth established: Added nuc's RSA pubkey to UNAS Pro (<code>192.168.1.31</code>) <code>/root/.ssh/authorized_keys</code> via password SSH. Key-based auth now works from nuc without password.</li> <li><code>fkrebs/unas-conf</code> repo created: Daily cron (03:20 on nuc) syncs: NFS exports, shares JSON, active exportfs, disk usage, cron config, OS version. Script at <code>/usr/local/bin/sync-unas-conf</code>.</li> <li>NFSv4 status: UNAS kernel already supports NFSv4 (<code>+4 +4.1 +4.2</code>) but UniFi OS doesn't configure an <code>fsid=0</code> root export. NFSv4 upgrade (docs #2) remains backlog \u2014 would require adding an unmanaged drop-in export that firmware updates could overwrite.</li> </ul>"},{"location":"CHANGELOG/#pve-host-optimizations","title":"PVE host optimizations","text":"<ul> <li>LXC startup order set for all CTs: CT103 (backrest), CT104 (docker), CT105 (nextcloud), CT112 (infisical), CT113 (db) had no startup order defined. Assigned: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full boot order now: DNS(1) \u2192 Zoraxy(2) \u2192 Pocket-ID(3) \u2192 DB(4) \u2192 Ops(5) \u2192 Infisical(6) \u2192 Docker(7) \u2192 Backrest(8) \u2192 Nextcloud(9) \u2192 HAOS(10) \u2192 Dev(20). Also set CT109 <code>onboot=1</code> (was missing).</li> <li>KSM (Kernel Samepage Merging) enabled: <code>echo 1 > /sys/kernel/mm/ksm/run</code> active now; persisted via <code>/etc/systemd/system/ksm-enable.service</code>. Deduplicates identical pages across LXC containers \u2014 effective for shared libc/runtime pages. Zero-risk, online change.</li> <li>5-minute critical watchdog timer added: <code>homelab-health-quick.timer</code> on nuc runs <code>QUICK=1 homelab-health</code> every 5 min. Checks: all LXC status (auto-starts if stopped), HAOS KVM (notify only), critical containers (Pocket-ID, Prometheus, Grafana, LiteLLM, MCP gateway). Complements the hourly full sweep.</li> <li>Gitea issues filed for backlog PVE opts (<code>fkrebs/docs</code> #2-4): NFS v3\u2192v4 upgrade, ZFS ARC increase to 16 GiB (blocked on CT101 right-sizing), hugepages for HAOS KVM.</li> </ul>"},{"location":"CHANGELOG/#n8n-flow-tracking","title":"n8n flow tracking","text":"<ul> <li>Gitea repo <code>fkrebs/n8n-flows</code> created: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via <code>n8n export:workflow --all</code> \u2192 split \u2192 commit + push.</li> <li>IDEAS.md: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).</li> <li>10 Gitea issues filed on <code>fkrebs/n8n-flows</code> for each idea \u2014 tracked in memory.</li> <li>Homelab health check (bash/systemd): interim implementation on nuc (<code>/usr/local/bin/homelab-health</code>, <code>homelab-health.timer</code>, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: <code>docker start</code> for stopped containers, <code>docker prune</code> for CT104 /tmp >80%. Escalates to Gotify (<code>homelab-health</code> app, token <code>Az9NpC-m1jjf571</code>) for crash loops/disk critical. Migration to n8n tracked as issue #1.</li> </ul>"},{"location":"CHANGELOG/#ct-104-ai-repo-cleanup","title":"CT 104 ai repo cleanup","text":"<ul> <li>lobehub/data/ untracked: Postgres data dir was accidentally tracked in git. <code>git rm --cached -r lobehub/data/</code> + added to <code>.gitignore</code>. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).</li> </ul>"},{"location":"CHANGELOG/#docs","title":"Docs","text":"<ul> <li><code>README.md</code> expanded: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).</li> <li><code>infra/portmap.md</code> updated: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.</li> <li><code>services/homelab-architecture.md</code> CT 109 description updated to reflect full stack.</li> <li><code>services/pocket-id.md</code> created: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.</li> </ul>"},{"location":"CHANGELOG/#2026-05-22","title":"2026-05-22","text":"<ul> <li> <p>Pocket-ID backup wired into Backrest: CT 110 SQLite-only (no Postgres). Pre-backup hook script at <code>/opt/backrest/scripts/pocketid-prestage.sh</code> on CT 103: SSHs to CT 110, runs <code>pocket-id export</code> inside container, <code>docker cp</code>s ZIP to host, scp's ZIP + signing keys to <code>/mnt/pve/unas/services/pocketid-backup/</code>. <code>services-backup-plan</code> snapshots staging \u2192 jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.</p> </li> <li> <p>Vaultwarden data \u2192 local zfs: Moved <code>/mnt/pve/unas/services/vaultwarden</code> to <code>/opt/stacks/vaultwarden/data</code> (local NVMe). DB was already on CT 113 postgres; only <code>rsa_key.pem</code>, <code>config.json</code>, <code>attachments/</code> remain in the data dir. Stale <code>db.sqlite3</code> deleted. NFS copy at <code>services/vaultwarden/</code> is now orphaned and can be cleaned up.</p> </li> <li> <p>WAL-G backup script fixed (CT 104): All nightly postgres backups were failing silently. Three bugs fixed: (1) <code>set -e</code> causing script abort when <code>shared-postgres</code> container not found (it was migrated to CT 113); (2) missing <code>-u postgres</code> on all <code>docker exec</code> calls \u2014 WAL-G requires the postgres role; (3) missing explicit data path <code>/var/lib/postgresql/data</code> on <code>backup-push</code>. <code>shared-postgres</code> commented out (CT 113 has its own WAL-G cron). <code>immich_postgres</code> and <code>lobe-postgres</code> now back up successfully to Garage S3.</p> </li> <li> <p>apps/ directory decommissioned (CT 104): <code>/opt/stacks/apps/</code> contained dead duplicate compose files for gotify, karakeep, memos, paperless-ngx, traccar, vaultwarden, shared-db \u2014 all without <code>.env</code> files, never running. Renamed to <code>.DECOMMISSIONED-2026-05-22</code> following established pattern. The canonical stacks at <code>/opt/stacks/<service>/</code> are unaffected.</p> </li> <li> <p>ComfyUI <code>--async-offload</code> removed: Flag caused full CPU fallback on GGUF img2img with XPU (3.5 min/step instead of ~7 s/step) and pure-noise output. Removed from <code>ai/comfyui.yml</code>. Current args: <code>--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16</code>. See <code>infra/proxmox-memory-audit.md</code>.</p> </li> <li> <p>CT 105 Nextcloud CIFS \u2192 NFS: Nextcloud was crash-looping (\"Appdata is not present\") because <code>unas_smb</code> CIFS mount (<code>//192.168.1.31/storage</code>) was not mounted on the host. Root cause: fstab entry was missing <code>_netdev</code>; mount dropped after a host event and was never re-established. Fixed by migrating CT 105 <code>mp0</code> from <code>/mnt/pve/unas_smb</code> to <code>/mnt/pve/unas</code> (same NFS share all other CTs use). CIFS entry removed from fstab. All 9 Nextcloud containers healthy post-restart.</p> </li> <li> <p>arr stack started: <code>vpn_gluetun</code>, <code>shelfarr</code> (<code>:13004</code>), <code>rdtclient</code> (via VPN, <code>:13001</code>), <code>prowlarr</code> (<code>:13002</code>), <code>audiobookshelf</code> (<code>:13003</code>). Fixed empty <code>labels:</code> in VPN compose.</p> </li> <li>claude-max-bridge: new service on CT 104 (<code>ai-internal</code>, no host port). FastAPI shim wrapping <code>claude</code> CLI as an OpenAI-compatible <code>/v1/chat/completions</code> endpoint. Enables LiteLLM to route to Claude Max subscription at zero API cost. Key details: <code>--strict-mcp-config</code> (no MCP servers), <code>--input-format stream-json</code> (multi-turn), <code>--include-partial-messages</code> (real streaming deltas). Bind-mounts <code>/usr/local/bin/claude</code> + <code>/root/.claude</code> (rw for OAuth token refresh). Source at <code>git.nuclide.systems/fkrebs/claude-max-bridge</code>.</li> <li>LiteLLM Claude models: <code>claude-sonnet-4-6</code>, <code>claude-opus-4-7</code>, <code>claude-haiku-4-5</code> + short aliases (<code>sonnet</code>, <code>opus</code>, <code>haiku</code>) + effort variants (<code>claude-sonnet-4-6-high</code>, <code>claude-opus-4-7-high</code>) added via <code>openai/</code> provider pointing at <code>claude-max-bridge:8000/v1</code>. Priced at Anthropic list rates for cost-visibility \u2014 actual cost $0 (Max). Bridge features: temperature\u2192effort mapping, <code>extra_body.effort</code> override, <code>extra_body.fallback_model</code>, <code>x_claude_cost_usd</code> and <code>x_claude_rate_limit</code> in responses.</li> <li>claude-max-bridge v2: additional CLI parameter mappings: <code>extra_body.system_prompt_mode</code> (<code>replace</code>/<code>append</code> \u2192 <code>--system-prompt</code>/<code>--append-system-prompt</code>), <code>extra_body.max_turns</code> \u2192 <code>--max-turns</code>, <code>extra_body.max_budget_usd</code> \u2192 <code>--max-budget-usd</code>, <code>response_format.json_schema</code> \u2192 <code>--json-schema</code>, <code>extra_body.exclude_dynamic_system_prompt_sections</code> \u2192 <code>--exclude-dynamic-system-prompt-sections</code>. Build context moved from <code>/tmp/</code> to <code>/opt/stacks/ai/claude-max-bridge/</code> (survives reboots). Source committed to Gitea at <code>a781abb</code>.</li> <li>Shepard MongoDB pinned to 8.0.4: <code>mongo:8.0</code> (latest) introduced a fatal kernel-version check in 8.0.5+ that rejects Proxmox's <code>7.0.2-5-pve</code> kernel string (parsed as major version 7 \u2265 6.19). Pinned to <code>mongo:8.0.4</code> in <code>/opt/shepard/infrastructure/docker-compose.yml</code>. Track MongoDB bug SERVER-121912 for a patched upstream release before unpinning.</li> <li>ComfyUI XPU optimisation: removed <code>--lowvram</code> + <code>--reserve-vram 1.0</code>; added <code>--async-offload</code> (Intel-patched XPU streams, ~10% speedup) + <code>--force-fp16</code>; memory limit 20 G \u2192 24 G (FLUX peaks ~22 GB in shared XPU RAM). <code>--disable-smart-memory</code> kept since <code>get_free_memory()</code> returns the full 58 GB shared pool on Arc \u2014 smart eviction is blind to the cgroup limit. See <code>infra/proxmox-memory-audit.md</code>.</li> <li>WaveSpeed FBC workflow: <code>ApplyFBCacheOnModel(residual_diff_threshold=0.12)</code> + <code>FluxGuidance(guidance=3.5)</code> + <code>BasicGuider</code> + <code>SamplerCustomAdvanced</code> added to FLUX.1-schnell pipeline. Workflow saved at <code>/mnt/pve/unas/services/comfyui/input/flux-schnell-wavespeed.json</code>. Estimated 30\u201350% additional speedup by caching first-block activations. WaveSpeed FBCache confirmed device-agnostic (no CUDA guards); <code>torch.compile</code> node must NOT be used on XPU.</li> <li> <p>LiteLLM benchmark: <code>bench.py</code> at <code>/opt/stacks/ai/benchmark/</code> measures TTFT + TPS for all configured models using streaming requests; renders xychart diagrams via internal Kroki (<code>http://kroki:8000</code>). First run results: Mistral fastest TTFT (43 ms), Claude bridge competitive at 101\u2013143 ms. See <code>services/llm-benchmark.md</code>.</p> </li> <li> <p>MCP gateway: added <code>git</code>, <code>gitlab</code> (DLR, <code>--pass-environment</code> fix for mcp-proxy env passthrough bug), <code>paper-search</code> (Docker catalog image replaces inline <code>python:3.12-slim</code>); <code>UNPAYWALL_EMAIL</code> wired into paper-search. Removed duplicate <code>papersearch</code> container. Updated <code>services/mcp-gateway.md</code>.</p> </li> <li>Nextcloud MCP re-enabled: <code>cbcoutinho/nextcloud-mcp-server</code> running in <code>single_user_basic</code> mode with <code>NEXTCLOUD_VERIFY_SSL=false</code>. App-password generated non-interactively via <code>occ user:add-app-password</code>. Gateway patched: (1) <code>NEXTCLOUD_VERIFY_SSL</code> / <code>MCP_DEPLOYMENT_MODE</code> now baked into hardcoded SERVERS env; (2) startup overlay extended to persist <code>enabled</code> flag; (3) proxy route bypasses per-user credential provisioning when <code>MCP_DEPLOYMENT_MODE=single_user_basic</code>. 26 servers healthy.</li> <li>CT 109 renamed <code>observe</code> \u2192 <code>ops</code>: updated <code>ct-inventory.md</code>, <code>homelab-architecture.md</code>, <code>proxmox-state.md</code> (IP changed to <code>.8</code> after conflict resolution).</li> <li>CT 101 Shepard disk expanded: 100 GiB \u2192 500 GiB (ZFS online resize, no restart).</li> <li>CT 112 Infisical marked live: <code>ct-inventory.md</code> and <code>portmap.md</code> updated; LAN-only, no Zoraxy route.</li> <li>litellm DB password rotated: placeholder <code>litellm_password_here</code> replaced in both <code>ai/.env</code> and <code>litellm-config/config.yaml</code> (config.yaml takes precedence over env var). Container restarted, healthy.</li> <li>UNAS migration dumps cleaned: 7 \u00d7 <code>*-migration-20260521.sql</code> (~492 MB) deleted from <code>/mnt/pve/unas/dump/</code>.</li> <li>Arcane OIDC secret rotated: new secret updated in Pocket-ID SQLite DB (bcrypt, cost 10) and <code>/opt/stacks/arcane/docker-compose.yml</code>. Container restarted.</li> <li>n8n OIDC secret rotated: Pocket-ID DB bcrypt update; secret applied to <code>/opt/stacks/n8n/docker-compose.yml</code> (authoritative stack). <code>/opt/stacks/apps/n8n/</code> was a dead duplicate (no <code>.env</code>, never ran) \u2014 compose renamed <code>.DECOMMISSIONED-2026-05-22</code>.</li> <li>n8n encryption key rotated: <code>N8N_ENCRYPTION_KEY</code> + <code>N8N_USER_MANAGEMENT_JWT_SECRET</code> replaced with 32-byte hex values in <code>/opt/stacks/n8n/.env</code> and <code>/opt/stacks/n8n/data/config</code>. Container restarted, healthy. Any previously stored workflow credentials are now invalid and must be re-entered.</li> <li>Backrest repo passwords rotated: <code>tapirnase</code> (weak placeholder) replaced on both <code>media-repo</code> and <code>services-repo</code> (JottaCloud). Used <code>restic key add</code> + <code>restic key remove</code>; <code>config.json</code> updated; daemon restarted.</li> <li>Obsidian MCP (<code>mcpvault</code>): new server added. <code>bitbonsai/mcpvault</code> bridged via <code>mcp-proxy</code> stdio\u2192HTTP in a custom <code>mcp-obsidian-bridge</code> image (Node 3.12-Alpine + mcp-proxy). Vault: Nextcloud <code>Notizen/</code> folder at <code>/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen</code>, bind-mounted read-write. 15 tools: read/write/patch/search/tags/frontmatter/stats. TTL cache 30s. Registered in Claude Code settings.json and LobeHub for both users.</li> <li>LobeHub MCP sync: all 27 gateway servers synced into <code>user_installed_plugins</code> for both users; stale <code>papersearch</code> entry removed; all manifests updated to long-lived <code>mcp_*</code> gateway token. Skipped: <code>crawl4ai</code> (no <code>/mcp</code> endpoint in <code>unclecode/crawl4ai:latest</code>), <code>n8n</code> (uses own auth).</li> <li>CT 109 plan updated: IP conflict noted (<code>.6</code> taken by CT 113); RAM 8 GiB, disk 50 GiB; Loki added; sshwifty added (multi-tab web SSH); Alloy replaces Promtail on all hosts; sidecar table expanded to CT 103/111/113.</li> <li>Portmap: CT 113 (<code>db</code>, <code>.6</code>) section added; QNAP TS-251D (Klipper/Mainsail) section added.</li> <li>New docs: <code>services/backrest.md</code>, <code>services/databases.md</code>, <code>services/arcane.md</code>.</li> </ul>"},{"location":"CHANGELOG/#2026-05-21","title":"2026-05-21","text":""},{"location":"CHANGELOG/#docs-docs-infrastructure","title":"Docs & docs infrastructure","text":"<ul> <li>Mermaid diagrams now rendering in mkdocs Material site \u2014 was missing <code>custom_fences</code> config in <code>pymdownx.superfences</code>. Fixed in <code>docs-server/mkdocs.yml</code> and rebuilt container.</li> <li><code>site_url</code> corrected to <code>http://192.168.1.111:13080</code> (docs are LAN-only; no Zoraxy route exists).</li> <li>Docs restructured: \"Stacks (CT 104)\" chapter dissolved; PORTMAP, Storage, Volumes, Docker networks moved to <code>infra/</code> section. TODO items migrated into CHANGELOG + ideas.</li> <li>Secrets manager design doc added at <code>services/secrets-manager.md</code> \u2014 recommends Infisical on a new CT 112.</li> </ul>"},{"location":"CHANGELOG/#storage-s3","title":"Storage / S3","text":"<ul> <li><code>s3.nuclide.systems</code> Zoraxy route fixed \u2014 <code>EnableWebsocketCustomHeaders</code>, <code>DisableHopByHopHeaderRemoval</code>, <code>EnableAutoHTTPS</code> all set. Garage S3 API now reachable externally.</li> <li><code>chat-artifacts</code> Garage S3 bucket created \u2014 public-read via <code>https://chat-artifacts.s3.nuclide.systems/<key></code>. Credentials at <code>/root/garage-chat-artifacts.creds</code> (not git-tracked).</li> </ul>"},{"location":"CHANGELOG/#mcp-surface","title":"MCP surface","text":"<ul> <li><code>upload-artifact</code> MCP server deployed (port 18011, CT 104). Tools: <code>upload_text</code>, <code>upload_base64</code>, <code>list_artifacts</code>, <code>get_url</code>, <code>delete_artifact</code>. Registered in gateway (now 24 servers). Source: <code>git.nuclide.systems/fkrebs/mcp-upload-artifact</code>.</li> <li>ComfyUI MCP now auto-uploads completed images to <code>chat-artifacts</code> S3 and includes the public URL alongside the inline base64 image.</li> <li>DAYTONA_API_KEY removed from CT 104 <code>ai/.env</code> (Daytona decommissioned 2026-05-20).</li> </ul>"},{"location":"CHANGELOG/#dev-environment","title":"Dev environment","text":"<ul> <li><code>savefig</code> dotfiles helper added to <code>fkrebs/dotfiles</code> \u2014 uploads any file to <code>chat-artifacts</code> using <code>uv run --with boto3</code>; reads credentials from env vars.</li> <li>Coder <code>python-uv</code> template updated with <code>CHAT_ARTIFACTS_*</code> env vars baked in; template pushed to Coder server.</li> </ul>"},{"location":"CHANGELOG/#gitea-repos","title":"Gitea repos","text":"<ul> <li>New repos created: <code>mcp-comfyui</code>, <code>mcp-docling</code>, <code>mcp-shepard</code>, <code>mcp-upload-artifact</code>, <code>home-assistant-config</code> \u2014 all at <code>git.nuclide.systems/fkrebs/</code>.</li> <li>HAOS config (<code>192.168.1.60</code>, VM 100) has a Gitea upstream ready. Manual push needed from HA terminal (see <code>services/secrets-manager.md</code> for SSH access notes).</li> </ul>"},{"location":"CHANGELOG/#planned","title":"Planned","text":"<ul> <li>CT 112 \"secrets\" \u2014 Infisical deployment (see <code>services/secrets-manager.md</code>)</li> <li>Renovate Bot via Gitea Actions \u2014 dependency updates for MCP server repos (Dockerfiles + requirements.txt)</li> <li>Vaultwarden OIDC SSO wiring (client created in Pocket-ID 2026-05-21; auth flow not yet configured)</li> <li>LobeHub Artifacts \u2192 S3 patch (TypeScript, ~3-4h, deferred)</li> </ul>"},{"location":"CHANGELOG/#2026-05-20","title":"2026-05-20","text":""},{"location":"CHANGELOG/#auth-identity-single-sign-on-lockdown","title":"Auth / Identity (single sign-on lockdown)","text":"<ul> <li>Pocket-ID migrated CT 104 \u2192 CT 110 \"id\" (<code>192.168.1.5:11000</code>). Authoritative writes now on CT 110; old data dir on CT 104 is a stale snapshot. Compose on CT 104 renamed <code>.MIGRATED-TO-CT110-2026-05-20</code>. Zoraxy upstream for <code>id.nuclide.systems</code> flipped.</li> <li>Pocket-ID 2.7.0 uses bcrypt for OIDC client secrets \u2014 previously some clients (Gitea, Coder) had raw 64-char SHA-256 hex in <code>oidc_clients.secret</code>, which silently failed every token exchange. Regenerated both with proper bcrypt hashes; audit also surfaced an orphaned Daytona client and the empty-secret <code>vscode</code> row.</li> <li>Gitea SSO via Pocket-ID wired with bcrypt fix + correct <code>Scopes</code> (<code>openid,profile,email</code>) + <code>two_factor_policy=skip</code> + <code>ACCOUNT_LINKING=auto</code> + <code>ENABLE_AUTO_REGISTRATION=true</code> + <code>USERNAME=preferred_username</code>. Local password sign-in form disabled (<code>GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM=false</code>); legacy OpenID 2.0 button hidden (<code>GITEA__openid__ENABLE_OPENID_SIGNIN=false</code>).</li> <li>Coder SSO via Pocket-ID wired. User row flipped <code>login_type=password \u2192 oidc</code>. <code>CODER_DISABLE_PASSWORD_AUTH=true</code> + <code>CODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE=false</code>. Workspace-side GitHub external_auth left on (for repo cloning).</li> </ul>"},{"location":"CHANGELOG/#dev-environment-new-ct-111-dev-192168142","title":"Dev environment (new CT 111 \"dev\", <code>192.168.1.42</code>)","text":"<ul> <li>New LXC stood up with Coder + Gitea + their Postgres companions.</li> <li>Templates pushed to Coder: <code>python-uv</code> (persistent, GPU, code-server, baked LiteLLM + Anthropic env, Claude Code auto-install) and <code>mcp-sandbox</code> (ephemeral, sci stack pre-baked).</li> <li>GPU passthrough confirmed: <code>/dev/dri/by-path/pci-0000:00:02.0-render</code> exposed; CT 111-level symlink at <code>/dev/dri/renderD128</code> (systemd-tmpfiles) so the Docker provider's path parser is happy.</li> <li><code>dotfiles</code> repo created at <code>fkrebs/dotfiles</code> on Gitea \u2014 oh-my-zsh, LiteLLM model picker (<code>models.env</code>), docker alias set, <code>~/.claude/settings.json</code> permissions allowlist, global Python <code>CLAUDE.md</code>, <code>.gitignore_global</code>, VSCodium settings + extensions.txt.</li> <li><code>mkproj</code> helper appends <code>[tool.coder]</code> workspace+owner block to new <code>pyproject.toml</code>, copies pre-commit config + <code>.gitignore</code>, installs dev deps, initial commit.</li> </ul>"},{"location":"CHANGELOG/#reverse-proxy","title":"Reverse proxy","text":"<ul> <li>Zoraxy WebSocket forwarding enabled for <code>dev.</code>, <code>git.</code>, <code>mcp.nuclide.systems</code> (<code>EnableWebsocketCustomHeaders=true</code> + <code>HeaderRewriteRules.DisableHopByHopHeaderRemoval=true</code>). Coder agent / Gitea live updates / streamable-MCP all need this. Backups at <code>/tmp/zoraxy-backup-2026-05-20/</code> on CT 108.</li> </ul>"},{"location":"CHANGELOG/#mcp-gateway","title":"MCP gateway","text":"<ul> <li><code>daytona</code> removed, <code>coder</code> added in <code>/opt/stacks/ai/mcp-gateway/config.json</code>. The <code>morning-briefing</code> scheduled agent's server list updated. Built a small <code>coder-mcp</code> image (multi-stage from <code>ghcr.io/coder/coder:latest</code> + <code>uv:bookworm-slim</code> + <code>uv tool install mcp-proxy</code>) that <code>coder login</code>s once then runs <code>coder exp mcp server</code> behind streamable-HTTP.</li> <li>Coder MCP exposes tools <code>coder_create_task</code> + workspace lifecycle from <code>Coder Agent v1.27.1</code>.</li> </ul>"},{"location":"CHANGELOG/#decommissions","title":"Decommissions","text":"<ul> <li>Daytona (<code>/opt/stacks/daytona/</code> + <code>/opt/stacks/ai/daytona-mcp.yml</code> on CT 104) \u2014 containers stopped + removed; compose files renamed <code>.DECOMMISSIONED-2026-05-20</code>. Coder replaces it.</li> </ul>"},{"location":"CHANGELOG/#docs-governance","title":"Docs / governance","text":"<ul> <li><code>/CLAUDE.md</code> created on host \u2014 operator doctrine for any agent SSHing in: keep things running, enforce consistency, plan rollback at medium-risk+, services AI-accessible by default, test before declaring done, generate slash commands for recurring tasks, inventory skills/commands/MCPs at session start.</li> <li><code>/docs</code> is now a git repo, baseline at <code>04d54ac</code>.</li> <li>Doc reorg into <code>infra/ services/ history/ ideas/</code>. Traefik guides moved to <code>history/</code> (Zoraxy is permanent). <code>mcp-gateway-requirements.md</code> and <code>scrubbing-list-2026-05-17.md</code> also archived.</li> </ul>"},{"location":"CHANGELOG/#2026-05-19","title":"2026-05-19","text":"<ul> <li>Garage moved off NFS-on-UNAS to local zfs on CT 104 (<code>/opt/stacks/shared-db/garage/{meta,data}</code>) after a WAL-G outage. Tier-1 (SQLite) databases off NFS as a follow-up; UNAS reserved for bulk media + workspace home dirs.</li> <li>n8n reverted to local SQLite after a Postgres migration attempt failed.</li> </ul>"},{"location":"CHANGELOG/#2026-05-17","title":"2026-05-17","text":"<ul> <li>MCP gateway design frozen. Implementation in <code>services/mcp-gateway.md</code>; design rationale archived as <code>history/mcp-gateway-requirements.md</code>.</li> <li>Per-stack scrubbing pass (see <code>history/scrubbing-list-2026-05-17.md</code> for the inventory at that snapshot).</li> </ul>"},{"location":"CHANGELOG/#2026-05-16","title":"2026-05-16","text":"<ul> <li>Traefik abandoned, Zoraxy is the production reverse proxy. Two migration guides preserved in <code>history/</code> for context only.</li> </ul>"},{"location":"RESUME/","title":"Session Resume","text":"<p>Last updated: 2026-05-26.</p>"},{"location":"RESUME/#open-items-urgency-order","title":"Open items (urgency order)","text":""},{"location":"RESUME/#critical-security-risk-or-unrecoverable-data-loss","title":"\ud83d\udd34 Critical \u2014 security risk or unrecoverable data loss","text":"<ul> <li>[ ] Password rotation: <code>tapirnase</code> \u2014 shared across WiFi PSK, LiteLLM root key (<code>sk-tapirnase</code>), D-Link admin, Backrest repo password, HAOS Terminal & SSH addon password. Single sniff = broad blast radius. Rotate per-service. See <code>services/homelab-architecture.md</code> \u00a7operational rules.</li> <li>[ ] Nextcloud Borg passphrase \u2192 Vaultwarden \u2014 passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See <code>services/backrest.md</code> blind spot #4b.</li> <li>[ ] Vault secrets sweep \u2014 32 plaintext secrets in Obsidian vault notes (Anthropic key, Gemini, NC AIO passphrase, etc.). Tracker at <code>Home/Homelab/Secrets to Move.md</code> in vault. Migrate to Vaultwarden, redact notes, rotate the high-risk ones.</li> </ul>"},{"location":"RESUME/#high-known-broken-verification-needed","title":"\ud83d\udfe0 High \u2014 known broken / verification needed","text":"<ul> <li>[x] ~~Arcane OIDC secret~~ \u2014 fixed 2026-05-23.</li> <li>[x] ~~Backrest media-repo / video-projects~~ \u2014 all three plans succeeded 2026-05-26 (media 01:30, services 01:04, video-projects 04:00). <code>video-projects-plan</code> had never completed before this. \u2705</li> <li>[x] ~~AdGuard split-horizon DNS~~ \u2014 done 2026-05-23.</li> <li>[x] ~~WAL-G monitoring~~ \u2014 done 2026-05-23. Textfile collector (<code>/usr/local/bin/walg-metrics.sh</code>) on CT 113 emits <code>walg_last_success_timestamp_seconds</code> + <code>walg_archive_status</code> to node-exporter textfile. Prometheus alert rules deployed on CT 109: <code>WalgArchiveStale</code> (>24h) + <code>WalgArchiveFailed</code>.</li> <li>[x] ~~Zoraxy audit~~ \u2014 done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; <code>SkipWebSocketOriginCheck</code> enabled on 13 routes; decommissioned routes noted. Missing: <code>dozzle.nuclide.systems</code> route (never created), <code>immich-tools</code>, <code>paperless</code>, <code>paperless-ai</code>. See <code>services/zoraxy.md</code>.</li> </ul>"},{"location":"RESUME/#medium-incomplete-migrations-cleanup-debt","title":"\ud83d\udfe1 Medium \u2014 incomplete migrations / cleanup debt","text":"<ul> <li>[ ] Docker disk reclaim \u2014 ~17 GB reclaimable on CT 104.</li> <li>[ ] Dormant stacks audit \u2014 ~6 still to audit (dozzle CT 104 copy, arr-stack, qdrant, etc.). 4 decommissioned 2026-05-23.</li> <li>[x] ~~Config-to-git fleet~~ \u2014 8 repos as of 2026-05-24. PVE/CT 108/CT 102/HA/CT 103/CT 109/CT 113/obsidian-vault/obsidian-config. Full table in <code>infra/config-to-git.md</code>. Manual step pending: HA Terminal & SSH addon <code>init_commands</code> for cron persistence.</li> <li>[ ] D-Link hardening \u2014 admin UI at <code>http://dlink.nuclide.lan</code> (<code>192.168.1.10</code>). Trusted-host allowlist + SNMP + TLS.</li> <li>[ ] Vaultwarden OIDC SSO \u2014 Pocket-ID client ready, env vars not yet set in <code>/opt/stacks/vaultwarden/.env</code>.</li> <li>[ ] Infisical OIDC config \u2014 manual step in Infisical admin at <code>http://secrets.nuclide.lan:8200/admin</code> \u2192 Settings \u2192 OIDC. Client <code>b2069075-\u2026</code>, secret <code>qsANw95zsza0_\u2026</code>.</li> <li>[ ] Infisical secrets migration Phase 2 \u2014 import CT104 <code>.env</code> keys via <code>infisical import</code>.</li> <li>[ ] Grafana HAOS dashboard \u2014 HA recorder reconfiguration (exclude Prometheus-covered metrics).</li> <li>[ ] HAOS SSH key \u2014 paste nuc public key in HA terminal.</li> <li>[ ] Obsidian config drift \u2014 3 enabled-but-missing plugins (<code>obsidian42-brat</code>, <code>obsidian-tasks-plugin</code>, <code>tasks-caldav-sync</code>). Reinstall or remove from <code>community-plugins.json</code>. See <code>Home/Homelab/Obsidian Plugin Audit.md</code> in vault.</li> <li>[ ] Memos Sync plugin install \u2014 config pushed to <code>obsidian-config</code>, plugin binary needs Community-Plugins install + paste of pre-baked <code>data.json</code>.</li> <li>[ ] Immich HTTP/2 Keep-Alive \u2014 root-caused 2026-05-24. Workaround live (per-SSID LAN URL). Permanent fix: strip header at Zoraxy. Needs Zoraxy confirmation.</li> </ul>"},{"location":"RESUME/#planned-requires-infrastructure-or-significant-effort","title":"\ud83d\udd35 Planned \u2014 requires infrastructure or significant effort","text":"<ul> <li>[ ] Second NVMe \u2014 needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items.</li> <li>[ ] Infisical secrets migration \u2014 CT 112 deployed (<code>192.168.1.7:8200</code>, LAN-only). Phase 1 done. Remaining:</li> <li>Phase 2: create project <code>homelab/ct104</code>, import <code>ai/.env</code> (~55 keys) via <code>infisical import</code>, add agent sidecars to compose stacks</li> <li>Phase 3: replace hardcoded env vars in Coder <code>main.tf</code> (committed to Gitea \u2014 security risk)</li> <li>Phase 4: OIDC SSO via Pocket-ID (low priority \u2014 convenience) See <code>services/secrets-manager.md</code>.</li> <li>[x] ~~Consolidate syncstack \u2192 MCP gateway~~ \u2014 done 2026-05-23. <code>litellm_sync.py</code> ported from syncstack, mounted into gateway. Gateway runs <code>run_sync()</code> every 15 min (120s startup delay). <code>/etc/cron.d/syncstack</code> deleted; <code>Dockerfile.syncstack</code> \u2192 <code>.DECOMMISSIONED-2026-05-23</code>; <code>/opt/stacks/CLAUDE.md</code> updated.</li> <li>[ ] ZFS ARC tuning \u2014 raise <code>zfs_arc_max</code> from 6.2 GiB \u2192 16 GiB once CT 101 memory right-sized. See <code>infra/proxmox-state.md</code> \u00a73.</li> <li>[ ] Document ingestion n8n workflow \u2014 Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless \u2192 Docling \u2192 embeddings \u2192 LobeChat KB. See <code>services/doc-ingestion.md</code>.</li> <li>[ ] ComfyUI async queue \u2014 MCP tool blocks 90\u2013300 s. Job-queue pattern: <code>generate_image()</code> returns ID; <code>get_job_status()</code> polls. See <code>services/comfyui.md</code>.</li> <li>[ ] Additional MCP servers \u2014 Gitea \u2713, Paperless \u2713, Proxmox-VE \u2713 (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf.</li> <li>[x] ~~Arcane + Dozzle \u2192 CT 109~~ \u2014 done 2026-05-23.</li> <li>[x] ~~crawl4ai SSE StreamConsumed~~ \u2014 resolved 2026-05-22.</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-26","title":"Recently completed (2026-05-26)","text":"<ul> <li>Backrest 3-for-3 success: all plans now completing on schedule. <code>video-projects-plan</code> recovered from never-completed state.</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-24","title":"Recently completed (2026-05-24)","text":""},{"location":"RESUME/#vault-obsidian","title":"Vault + Obsidian","text":"<ul> <li>Vault dedup: <code>Work 1/</code> promoted \u2192 <code>Work/</code>, old <code>Work/</code> parked as <code>Work.stale-backup-2026-05-24/</code> (1-week safety net). Rescued unique <code>Journal/Journal.md</code> before swap. Killed dup <code>Home/BrainBox.md</code>, <code>Home/Templates/</code>, <code>Willkommen.md</code> \u00d72, <code>.caldav-sync/</code>.</li> <li>234 asset-link rewrites across 41 files \u2014 collapsed <code>assets/</code>, <code>../assets/</code>, <code>../../foo.pdf</code> \u2192 <code>.assets/\u2026</code>. Broken links 242 \u2192 9 (rest non-issues: <code>tel:</code>, <code>about:reader?</code>, <code>siyuan://</code>).</li> <li><code>Home/Homelab/Secrets to Move.md</code> \u2014 32 plaintext secrets inventoried.</li> <li><code>Home/Homelab/Obsidian Plugin Audit.md</code> \u2014 17 installed plugins keep/watch/re-evaluate, 3 config-drift items, 10 new candidates ranked for the stack.</li> <li><code>Home/Homelab/App Endpoint Checklist.md</code> \u2014 per-service LAN + external URLs + apps-to-update list. Gotchas: clients sticking to old IP, Immich HTTP/2 Keep-Alive bug.</li> <li>Daily-note merge policy: Claude edits land in <code>Work/Journal/YYYY/MM/YYYY-MM-DD.md</code> under <code>## \ud83e\udd16 Claude edits</code> (not parallel folder).</li> <li>Obsidian Git plugin: wired end-to-end against <code>fkrebs/obsidian-config</code>. Auto-commit + push every 60 min, pull on boot, PAT gitignored. Verified working.</li> <li>Streamlined-ui CSS snippet pushed to <code>obsidian-config</code> (hides ribbon icons for non-clickable plugins, compact status bar, tighter file tree).</li> <li>Memos Sync plugin config pre-pushed (binary install pending user).</li> </ul>"},{"location":"RESUME/#config-to-git-fleet-expanded-to-8-repos","title":"Config-to-git fleet expanded to 8 repos","text":"<ul> <li>New: <code>fkrebs/ct103-conf</code>, <code>fkrebs/ct109-conf</code>, <code>fkrebs/ct113-conf</code>, <code>fkrebs/home-assistant-config</code>, <code>fkrebs/obsidian-vault</code> (rolling 2-commit window), <code>fkrebs/obsidian-config</code>.</li> <li>All daily 03:00 / 04:00. Pattern documented in <code>infra/config-to-git.md</code>.</li> <li>Rule articulated: every host with hand-edited config must push daily.</li> </ul>"},{"location":"RESUME/#networking-nuclidelan-zone","title":"Networking \u2014 <code>*.nuclide.lan</code> zone","text":"<ul> <li>30 AdGuard A-records on CT 102 \u2014 per-host (<code>pve</code>, <code>nas</code>, <code>unifi</code>, <code>dlink</code>, <code>backrest</code>, etc.) + per-service aliases (<code>immich</code>, <code>vault</code>, <code>karakeep</code>, <code>grafana</code>, <code>prometheus</code>, \u2026).</li> <li>Documented in <code>services/adguard-dns.md</code>.</li> <li>Solves \"client registered to old IP\" gotcha \u2014 when service moves CT, edit one AdGuard rewrite instead of N apps.</li> <li>D-Link IP captured: <code>192.168.1.10</code>.</li> <li>Filed feedback memory: always use <code>*.nuclide.lan</code> aliases in URLs, never bare IPs.</li> </ul>"},{"location":"RESUME/#homarr","title":"Homarr","text":"<ul> <li>3 new boards built via SQL: <code>home</code> (morning routine, default), <code>admin</code> (by-tier), <code>command</code> (live-ops iframes + grid). Old \"nuclide\" board was lost when OIDC user was recreated for password reset.</li> <li>Boards owned by <code>fkrebs@nucli.de</code>, <code>is_public=1</code>. 33 unique apps with ping status.</li> <li>Integrations not yet wired (Proxmox / Docker / AdGuard / Gotify widgets \u2014 need tokens).</li> </ul>"},{"location":"RESUME/#pocket-id","title":"Pocket-ID","text":"<ul> <li>LAN callback URLs added for Homarr, Grafana, Infisical, Proxmox VE so OIDC works via both bare-IP and <code>.nuclide.lan</code> alias paths.</li> </ul>"},{"location":"RESUME/#other","title":"Other","text":"<ul> <li>Immich Android upload bug diagnosed: HTTP/2 + <code>Keep-Alive</code> header conflict. Workaround live (per-SSID LAN URL). Permanent fix pending Zoraxy header strip.</li> <li>Nexa #1 filed at https://git.nuclide.systems/fkrebs/nexa/issues/1 \u2014 9 design patterns from Obsidian + Claude session as input for Nexa Phase 1\u20132.</li> <li>Wi-Fi SSID captured: <code>nuclide</code>. Used for per-network app routing (Immich, HA Companion).</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-23-session-continued-4","title":"Recently completed (2026-05-23, session continued \u00d74)","text":"<ul> <li>Docs consistency pass: zoraxy.md, arcane.md, ct-inventory.md, portmap.md, proxmox-memory-audit.md, mcp-gateway.md, RESUME.md all updated to reflect 2026-05-23 state.</li> <li>Wetty deployed on CT 109 (:4090, LAN-only). <code>jump-menu.sh</code> on nuc \u2192 SSH jump menu to all 10 hosts.</li> <li>MCP servers added: gitea, paperless, proxmox (read-only PVEAuditor role). Total: 30 servers.</li> <li>Loki dashboard fixed: K8s labels replaced with Docker labels in 14 places.</li> <li>WAL-G monitoring: textfile collector on CT 113 + Prometheus alert rules on CT 109.</li> <li>CT 103 RAM: 2 GB \u2192 4 GB + 1 GB swap (Backrest OOM fix for 822 GB initial backup).</li> <li>Zoraxy WS: <code>SkipWebSocketOriginCheck</code> enabled on 13 routes. Arcane upstream corrected to CT 109.</li> <li>CT 104 cleanup: daytona, homepage, proxy, streamio decommissioned.</li> <li>Paperless-ngx: USERMAP_UID=997 + file permissions fixed, container stable.</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-23-session-continued-3","title":"Recently completed (2026-05-23, session continued \u00d73)","text":"<ul> <li>Homarr board deployed: 6 category sections (AI, Monitoring, Files, Dev, Home, Identity & Admin), 27 apps with ping-based status, Mainsail included in Dev. Fixed schema: <code>x_offset</code>/<code>y_offset</code> on section rows + paired empty sections (y=0 initial, y=1/2 per category). Board <code>nuclide</code> set as home board.</li> <li>Arcane environments fixed: deleted stale shared-token environments; created 6 unique edge environments (shepard, docker, nextcloud, id, dev, secrets) via API with <code>useApiKey=true</code>. Each agent updated with unique token. All 8 environments online.</li> <li>Arcane API key seeded: <code>ADMIN_STATIC_API_KEY</code> from compose was never seeded (DB pre-existed). Inserted via argon2id hash. Key <code>arc_d3357a65</code> now functional for automation.</li> <li>Infisical SITE_URL fixed: changed from <code>https://secrets.nuclide.systems</code> to <code>http://192.168.1.7:8200</code> (LAN-only, no Zoraxy route).</li> <li>Infisical Pocket-ID client created: client <code>b2069075-ede2-4251-ad1f-9a62e6a188b3</code>, callback <code>http://192.168.1.7:8200/api/v1/sso/oidc/callback</code>. Manual OIDC config entry still needed in Infisical admin UI.</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-23-session-continued-2","title":"Recently completed (2026-05-23, session continued \u00d72)","text":"<ul> <li>PVE LXC startup order fixed: CT103/104/105/112/113 had no startup order. Set: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full deterministic boot sequence now in place. Also enabled CT109 onboot=1 (was missing).</li> <li>KSM enabled on PVE host: kernel samepage merging active + persisted via <code>ksm-enable.service</code>. Zero-cost memory deduplication across LXC containers.</li> <li>5-minute critical watchdog deployed: <code>homelab-health-quick.timer</code> on nuc; <code>QUICK=1</code> mode checks LXC status + critical containers every 5 min. Auto-starts stopped LXCs, notifies Gotify for HAOS KVM issues.</li> <li>PVE backlog issues filed: <code>fkrebs/docs</code> #2 (NFS v3\u2192v4), #3 (ZFS ARC 16 GiB), #4 (hugepages for KVM).</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-23-session-continued","title":"Recently completed (2026-05-23, session continued)","text":"<ul> <li>HAOS KVM OOM root-caused and fixed: karakeep_chrome on CT 104 triggered host OOM \u2192 killed HAOS KVM twice. Applied: <code>shm_size: '512m'</code> to chrome, <code>oom_score_adj=-300</code> on KVM PID, systemd timer <code>protect-haos-kvm.timer</code> (every 5 min) for persistence.</li> <li>Arcane + Dozzle migrated to CT 109: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080).</li> <li>Homarr + Grafana OIDC wired: clients created via Pocket-ID API (no manual UI steps). Homarr client <code>63a94e30</code>, Grafana client <code>92d987d5</code> (PKCE). Both SSO-active.</li> <li>n8n flows tracked in Gitea: repo <code>fkrebs/n8n-flows</code> with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n).</li> <li>CT 104 ai repo cleaned: <code>lobehub/data/</code> untracked from git, <code>.gitignore</code> fixed, litellm-config + mcp-gateway changes committed.</li> <li>zoraxy-conf repo: arcane proxy update committed and pushed (CT 108).</li> </ul>"},{"location":"RESUME/#recently-completed-2026-05-23","title":"Recently completed (2026-05-23)","text":"<ul> <li>Config-to-git crons: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy \u2192 <code>zoraxy-conf</code>), CT 102 (AdGuard \u2192 <code>adguard-conf</code>), PVE (rsync <code>/etc/pve</code> \u2192 <code>pve-conf</code>). Force-push. All three verified with initial push.</li> <li>Syncstack \u2192 MCP gateway: <code>litellm_sync.py</code> (ported from syncstack.py) mounted into gateway container. <code>_litellm_sync_loop()</code> runs every 15 min in gateway's asyncio loop. Cron <code>/etc/cron.d/syncstack</code> deleted. <code>Dockerfile.syncstack</code> decommissioned. <code>/opt/stacks/CLAUDE.md</code> updated.</li> </ul>"},{"location":"RESUME/#recently-completed-session-continued-2-2026-05-22","title":"Recently completed (session continued \u00d72, 2026-05-22)","text":"<ul> <li>crawl4ai MCP fixed: was returning 404 on <code>/mcp</code> (streamable-HTTP). Current 0.8.6 image already has MCP over SSE at <code>/mcp/sse</code>. Added SSE transport support to <code>mcp-gateway/server.py</code>: per-request SSE round-trip with auto-initialize handshake; <code>transport: sse</code> in hardcoded SERVERS + config.json. Gateway health now shows crawl4ai ok; lobe-sync shows 27/27 servers.</li> <li>syncstack cron fixed: missing <code>cd /opt/stacks/ai &&</code> prefix caused \"no configuration file\" every 15 min since ~May 18. Fixed. Manual run confirmed 60 models including 5 Claude models; LobeChat recreated.</li> </ul>"},{"location":"RESUME/#recently-completed-session-continued-2026-05-22","title":"Recently completed (session continued, 2026-05-22)","text":"<ul> <li>claude-max-bridge <code>/v1/responses</code> endpoint: implemented OpenAI Responses API with <code>previous_response_id</code> chaining. Root cause of \"session already in use\": Claude CLI leaves session JSONL files in an un-resumable \"dequeued\" state after each <code>--print</code> run. Fix: maintain conversation history server-side as formatted text; inject into system prompt for continuations; always use <code>--no-session-persistence</code>. Also fixed: assistant turns in <code>--input-format stream-json</code> require <code>content</code> as array-of-blocks not string (JS error otherwise). Assistant content array format also fixed in <code>_parse_messages</code> / <code>_parse_responses_input</code>.</li> <li>LiteLLM pass-through for <code>/v1/responses</code>: added <code>pass_through_endpoints</code> entry in <code>/opt/stacks/ai/litellm-config/config.yaml</code> \u2192 <code>http://claude-max-bridge:8000/v1/responses</code>. LiteLLM appends its token hash to the response <code>id</code>, so <code>previous_response_id</code> chaining only works when calling the bridge directly (not through LiteLLM).</li> </ul>"},{"location":"RESUME/#recently-completed-this-session-2026-05-22","title":"Recently completed (this session, 2026-05-22)","text":"<ul> <li>Pocket-ID backup: pre-hook on CT 103 runs <code>pocket-id export</code> + scp keys \u2192 UNAS staging; <code>services-backup-plan</code> snapshots \u2192 jottacloud. Verified.</li> <li>Stale NFS vaultwarden dir deleted: <code>/mnt/pve/unas/services/vaultwarden/</code> removed.</li> <li>Nextcloud CIFS \u2192 NFS: crash-loop fixed; CT 105 mp0 migrated from <code>/mnt/pve/unas_smb</code> to <code>/mnt/pve/unas</code>; CIFS fstab entry removed.</li> <li>Vaultwarden \u2192 local zfs: <code>/mnt/pve/unas/services/vaultwarden</code> \u2192 <code>/opt/stacks/vaultwarden/data</code>; stale <code>db.sqlite3</code> deleted; DB already on CT 113 postgres.</li> <li>WAL-G fixed: <code>immich_postgres</code> and <code>lobe-postgres</code> backups now succeed. Fixed: missing <code>-u postgres</code>, missing data path, <code>set -e</code> abort, dead <code>shared-postgres</code> reference.</li> <li>ComfyUI <code>--async-offload</code> removed: caused CPU fallback + pure-noise output on GGUF img2img with XPU.</li> <li>apps/ decommissioned: dead duplicate compose files in <code>/opt/stacks/apps/</code> renamed <code>.DECOMMISSIONED-2026-05-22</code>.</li> <li>Docs consolidated: storage.md, volumes.md, homelab-architecture.md, proxmox-state.md, proxmox-memory-audit.md, ct-inventory.md, README.md all reconciled.</li> </ul>"},{"location":"RESUME/#key-system-state","title":"Key system state","text":"Host IP Role nuc (PVE) 192.168.1.20 Proxmox host \u2014 SSH gateway to all CTs; jump-menu.sh CT 109 ops 192.168.1.8 Monitoring \u2014 Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr + Wetty CT 104 docker 192.168.1.40 Main Docker host \u2014 AI/ML, media, ~65 containers CT 113 db 192.168.1.6 Shared Postgres 17 + WAL-G \u2192 Garage S3 (cron 02:00) CT 110 id 192.168.1.5 Pocket-ID OIDC IdP CT 103 backrest 192.168.1.3 Backrest \u2014 jottacloud via rclone; UNAS at <code>/mnt/pve/unas</code> CT 111 dev 192.168.1.42 Coder + Gitea CT 108 zoraxy 192.168.1.4 Reverse proxy + ACME \u2014 always confirm before changes"},{"location":"RESUME/#constraints-to-remember","title":"Constraints to remember","text":"<ul> <li>Zoraxy (CT 108): always get explicit user confirmation before any config change.</li> <li>Infisical (CT 112): LAN-only, no Zoraxy route \u2014 must not be internet-exposed.</li> <li>CT 104 has a separate agent reconciling compose files \u2014 rename decommissioned files (don't just stop), update <code>/opt/stacks/CLAUDE.md</code>.</li> <li>ComfyUI container limit: 24 G. Do not raise without measuring host impact.</li> <li>Pocket-ID is SQLite-only \u2014 no Postgres migration possible.</li> <li>Use <code>*.nuclide.lan</code> aliases when presenting URLs to the user (never bare IPs). Wi-Fi SSID is <code>nuclide</code>.</li> <li>Vault edits append to <code>Work/Journal/YYYY/MM/YYYY-MM-DD.md</code> under <code>## \ud83e\udd16 Claude edits</code>; never create a parallel folder.</li> </ul>"},{"location":"ct-inventory/","title":"CT / VM inventory","text":"<p>Verified 2026-05-20 via <code>pct list</code>, <code>pct config <id></code>, <code>qm config 100</code> on <code>nuc</code>.</p> VMID Name Type IP Cores RAM (MiB) Rootfs Role UNAS mount GPU Status 100 haos VM (q35/OVMF) DHCP via vmbr0 (<code>.60</code>) 4 16384 (balloon 4096) 32 GiB local-zfs Home Assistant OS; USB Zigbee dongle (10c4:ea60) passed through \u2192 Zigbee2MQTT add-on + Mosquitto broker add-on; OCPP (EV charger); ~2492 entities \u2014 no running 101 shepard LXC unpriv 192.168.1.49/24 12 32768 500 GiB Shepard product stack (Caddy, frontend, backend, Keycloak, Mongo, Neo4j, TimescaleDB) mp0 NFS Intel iGPU (card+render) running 102 dns LXC unpriv 192.168.1.2/24 2 1024 4 GiB AdGuard Home \u2014 LAN DNS resolver + filter \u2014 no running 103 backrest LXC unpriv 192.168.1.3/24 1 4096 + 1024 swap 8 GiB Backrest (restic) backup scheduler mp0 NFS no running 104 docker LXC unpriv (idmapped) 192.168.1.40/24 16 49152 200 GiB Main Docker host \u2014 AI/ML + media + identity-adjacent (~70 containers); Gitea (:3000/:222) + Coder (:7080) migrated from CT 111 2026-05-26; Proton Mail Bridge (:1025 SMTP/:1143 IMAP) mp0 NFS Intel iGPU (card+render) running 105 nextcloud LXC priv 192.168.1.41/24 4 8196 100 GiB Nextcloud AIO mp0 NFS Intel iGPU (render only) running 108 zoraxy LXC unpriv 192.168.1.4/24 2 2048 6 GiB Zoraxy reverse proxy + ACME (<code>*.nuclide.systems</code>) \u2014 no running 109 ops LXC unpriv 192.168.1.8/24 4 4096 32 GiB Ops \u2014 Prometheus + Grafana + Loki + Alloy + pve-exporter + Homepage (:10000) + Portainer (:9000) + Dozzle (server) + docs-server + Infisical (:8200) + Pocket-ID (:11000, migrated from CT 110 2026-05-26); Dozzle agents on all Docker hosts \u2014 no running ~~110~~ ~~id~~ \u2014 \u2014 \u2014 \u2014 \u2014 DESTROYED 2026-05-26 \u2014 Pocket-ID migrated to CT 109; LXC removed \u2014 \u2014 \u2014 ~~111~~ ~~dev~~ \u2014 \u2014 \u2014 \u2014 \u2014 DESTROYED 2026-05-26 \u2014 Coder + Gitea migrated to CT 104; LXC removed \u2014 \u2014 \u2014 ~~112~~ ~~secrets~~ \u2014 \u2014 \u2014 \u2014 \u2014 DESTROYED 2026-05-26 \u2014 Infisical migrated to CT 109; LXC removed \u2014 \u2014 \u2014 113 db LXC unpriv 192.168.1.6/24 2 4096 40 GiB Shared Postgres 17 + pgAdmin + WAL-G \u2192 Garage S3; Arcane edge agent \u2014 no running <p>UNAS NFS = <code>192.168.1.31:/var/nfs/shared/storage</code> over NFSv3. CT 105 (Nextcloud) is the lone outlier \u2014 it mounts the same UNAS share over CIFS/SMB 3.1.1, not NFS.</p> <p>CT 112 (\"secrets\") \u2014 Infisical deployed 2026-05-22; see <code>services/secrets-manager.md</code>. LAN: <code>http://192.168.1.7:8200</code>. No Zoraxy route \u2014 secrets must not be internet-exposed.</p> <p>CT 105 (\"nextcloud\") \u2014 migrated from CIFS (<code>//192.168.1.31/storage</code>) to NFS (<code>192.168.1.31:/var/nfs/shared/storage</code>) on 2026-05-22. CIFS mount was not remounting after host reboots, causing Nextcloud crash-loops. NFS is consistent with all other CTs.</p>"},{"location":"ct-inventory/#ssh-access","title":"SSH access","text":"<p>The Proxmox host (<code>nuc</code>, <code>192.168.1.20</code>) has root SSH access to all LXC containers via key auth. To grant your own key access to every running container in one step, run on the host:</p> <pre><code>deploy-ssh-key \"ssh-ed25519 AAAA... you@yourmachine\"\n# or pipe it:\nssh nuc 'cat' < ~/.ssh/id_ed25519.pub | deploy-ssh-key\n</code></pre> <p>Script is at <code>/usr/local/bin/deploy-ssh-key</code>. It iterates <code>pct list</code>, skips stopped CTs, and appends the key to <code>/root/.ssh/authorized_keys</code> idempotently (no duplicates).</p> <p>VM 100 (HAOS, <code>192.168.1.60</code>) cannot be reached via <code>pct exec</code>. Install manually in the HA terminal:</p> <pre><code>echo \"ssh-ed25519 AAAA... you@yourmachine\" >> ~/.ssh/authorized_keys\n</code></pre> <p>Host key (<code>root@nuc</code>, already deployed to all CTs 2026-05-21):</p> <pre><code>ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuVqBW3VXg... root@nuc\n</code></pre> <p>CT 113 (\"db\") provisioned 2026-05-21; see <code>services/databases.md</code>. Future role: consolidate per-stack Postgres instances once second NVMe lands.</p> <p>CT 109 (\"ops\") provisioned 2026-05-23. Debian 13, Docker 29.5.2. Stack <code>/opt/stacks/monitoring</code>: Prometheus (:9090), Grafana (:3000 \u2014 LAN only, admin/tapirnase), Loki (:3100), node-exporter (:9100 host-net), pve-exporter (:9221), Alloy (:12345), Portainer BE (:9000), Dozzle server (:10001), docs-server (:13080), Wetty web-SSH (:4090 LAN-only). Homepage (:10000 \u2014 <code>ops.nuclide.lan:10000</code>) at <code>/opt/stacks/homepage/</code> \u2014 replaced Homarr 2026-05-26; remote Docker via socket-proxy on CT 110/111/112/113 and direct TCP on CT 104. Scrapes: <code>node-ct104</code> <code>:9100</code>, <code>node-ct109</code> <code>:9100</code>, <code>home-assistant</code> 192.168.1.60:8123/api/prometheus, <code>prometheus</code> self, <code>walg</code> CT 113 <code>:9100/textfile</code>.</p>"},{"location":"docker-internal-inventory/","title":"Docker-internal inventory","text":"<p>Live at http://192.168.1.8:13080/docker-internal-inventory/.</p> <p>Containers that have no LAN-published port \u2014 reachable only on their Docker bridge network. This is the third tier of our access model:</p> <ol> <li>public \u2014 Zoraxy-routed via <code>*.nuclide.systems</code> (internet-reachable)</li> <li>LAN \u2014 <code>192.168.1.0/24</code> direct (host port mapping)</li> <li>docker-internal \u2014 only via container-to-container bridge (this doc)</li> </ol> <p>Default per [[feedback_internal_only]]: keep new things at tier 2 or 3 unless there's a real external-access reason. Most of what is on tier 3 should stay there \u2014 that's the point of having three tiers.</p>"},{"location":"docker-internal-inventory/#recommendation-legend","title":"Recommendation legend","text":"<ul> <li>\ud83d\udfe2 Keep internal \u2014 sidecar / backend / engine, must not be exposed</li> <li>\ud83d\udfe1 Maybe \u2014 admin UI exists; could be useful to reach directly but low value</li> <li>\ud83d\udd34 Expose to LAN \u2014 broken access pattern, fix recommended</li> </ul>"},{"location":"docker-internal-inventory/#ct-104-docker-host-43-internal-only","title":"CT 104 (docker host) \u2014 43 internal-only","text":""},{"location":"docker-internal-inventory/#ai-mcp-plumbing-all-keep-internal","title":"AI / MCP plumbing \u2014 all \ud83d\udfe2 keep internal","text":"<p>Backend MCP servers consumed only by the MCP gateway. Exposing them would bypass auth + audit. - <code>mcp-proxmox</code>, <code>gitea-mcp</code>, <code>mcp-immich</code>, <code>mcp-fetch</code>, <code>mcp-time</code>, <code>mcp-git</code>, <code>mcp-gotify</code>, <code>mcp-unifi</code>, <code>mcp-ntfy</code>, <code>mcp-markitdown</code>, <code>mcp-context7</code>, <code>mcp-youtube-transcript</code>, <code>mcp-sequential-thinking</code>, <code>mcp-wikipedia-mcp</code>, <code>mcp-gitlab</code>, <code>mcp-crawl4ai</code>, <code>coder-mcp</code>, <code>ariel-mcp</code>, <code>paperless-mcp</code>, <code>claude-max-bridge</code> - Plus the ephemeral <code>crazy_colden</code>/<code>stoic_kirch</code>/etc. (auto-spawned MCP one-shots \u2014 Docker name collisions, no static port).</p>"},{"location":"docker-internal-inventory/#backends-for-exposed-services-keep-internal","title":"Backends for exposed services \u2014 \ud83d\udfe2 keep internal","text":"<ul> <li><code>karakeep_meilisearch</code>, <code>karakeep_chrome</code> (Karakeep search + headless browser)</li> <li><code>immich_postgres</code>, <code>immich_redis</code>, <code>immich_machine_learning</code> (Immich backends)</li> <li><code>immich_power_tools</code> \u2014 \ud83d\udfe1 maybe. Power-user UI for Immich. Bind to LAN if you ever want to use it directly.</li> <li><code>paperless-ngx-tika-1</code>, <code>paperless-ngx-gotenberg-1</code>, <code>paperless-ngx-broker-1</code> (Paperless OCR/PDF/redis)</li> <li><code>redis-searxng</code>, <code>rdtclient</code> (auxiliary)</li> </ul>"},{"location":"docker-internal-inventory/#diagrams-keep-internal","title":"Diagrams \u2014 \ud83d\udfe2 keep internal","text":"<ul> <li><code>kroki</code>, <code>kroki-mermaid</code>, <code>kroki-excalidraw</code> \u2014 rendered via MCP, no UI to expose.</li> </ul>"},{"location":"docker-internal-inventory/#monitoring-agents-keep-internal","title":"Monitoring agents \u2014 \ud83d\udfe2 keep internal","text":"<ul> <li><code>node-exporter</code> (scraped by Prometheus on <code>:9100</code> over container network; LAN exposure not needed since Prometheus is on the same LAN already)</li> <li><code>arcane-agent</code> (talks back to Arcane server on CT 109)</li> </ul>"},{"location":"docker-internal-inventory/#ct-105-nextcloud-8-internal-only","title":"CT 105 (nextcloud) \u2014 8 internal-only","text":"<p>\ud83d\udfe2 All keep internal \u2014 Nextcloud AIO architecture. - <code>nextcloud-aio-nextcloud</code> (fronted by AIO apache proxy on <code>:11000</code>) - <code>nextcloud-aio-database</code> (Postgres), <code>nextcloud-aio-redis</code>, <code>nextcloud-aio-imaginary</code>, <code>nextcloud-aio-notify-push</code>, <code>nextcloud-aio-collabora</code>, <code>nextcloud-aio-docker-socket-proxy</code> - <code>arcane-agent</code></p> <p>Exposing the AIO backends directly would break Nextcloud's auth model and crash backups.</p>"},{"location":"docker-internal-inventory/#ct-109-ops-1-internal-only","title":"CT 109 (ops) \u2014 1 internal-only","text":"<ul> <li><code>node-exporter</code> \u2014 \ud83d\udfe2 keep internal. Scraped via <code>host.docker.internal</code> from Prometheus on the same host.</li> </ul>"},{"location":"docker-internal-inventory/#ct-110-111-112-113-arcane-agent-only","title":"CT 110 / 111 / 112 / 113 \u2014 <code>arcane-agent</code> only","text":"<p>\ud83d\udfe2 All keep internal. The Arcane agent on each Docker host calls back to the Arcane server on <code>192.168.1.8:10002</code>; no inbound LAN traffic needed.</p> <p>Plus: - CT 111: <code>act-runner</code> \ud83d\udfe2 (Gitea Actions runner \u2014 outbound to Gitea API; never needs inbound) - CT 112: <code>infisical-db</code>, <code>infisical-redis</code> \ud83d\udfe2 (Infisical app on <code>:8200</code> is the only intended entry)</p>"},{"location":"docker-internal-inventory/#cross-tier-issues-spotted","title":"Cross-tier issues spotted","text":"<p>None. The 3-tier model is clean across the fleet: - No backend Postgres/Redis is accidentally LAN-bound - No MCP server is double-exposed - No admin UI is bound to LAN when it shouldn't be</p>"},{"location":"docker-internal-inventory/#candidate-lan-bind-if-you-want-them","title":"Candidate LAN-bind, if you want them","text":"<p>If you ever want direct LAN access to one of the \ud83d\udfe1 services, the pattern is to add a <code>ports:</code> line to its compose entry:</p> Service Suggested port Why you might <code>immich_power_tools</code> <code>:3001</code> on CT 104 Bulk Immich operations (album merge, dedup) the main UI doesn't expose <p>Everything else: leave at tier 3.</p>"},{"location":"services-overview/","title":"Services overview","text":"<p>Live at http://192.168.1.8:13080/services-overview/ (docs-server polls Gitea every 5 min, so changes appear shortly after <code>git push</code>).</p> <p>Every service running in the homelab, with its access URL(s) and host. External URLs go through Zoraxy (CT 108) and are reachable from the internet. Internal are LAN-only (<code>192.168.1.0/24</code>). Per [[feedback_internal_only]], new services default to internal.</p>"},{"location":"services-overview/#ai-ml","title":"AI / ML","text":"Service External Internal Host Doc Bifrost (LLM/MCP gateway) https://ai.nuclide.systems http://192.168.1.40:14003 CT 104 \u2014 Open WebUI https://chat.nuclide.systems http://192.168.1.40:14002 CT 104 \u2014 LiteLLM \u2014 http://192.168.1.40:14000 (internal only) CT 104 \u2014 ComfyUI \u2014 http://192.168.1.40:18188 CT 104 [[comfyui]] Nexa \u2014 \u2014 CT 104 [[nexa]] n8n (automation) \u2014 http://192.168.1.40:15678 CT 104 \u2014"},{"location":"services-overview/#files-storage","title":"Files / storage","text":"Service External Internal Host Doc Nextcloud https://nc.nuclide.systems http://192.168.1.41:11000 CT 105 \u2014 Immich https://immich.nuclide.systems http://192.168.1.40:12000 CT 104 \u2014 Paperless-ngx https://paperless.nuclide.systems \u2014 CT 104 [[doc-ingestion]] Karakeep (Hoarder) https://hoarder.nuclide.systems http://192.168.1.40:17001 CT 104 \u2014 Memos https://memos.nuclide.systems http://192.168.1.40:17000 CT 104 \u2014 Audiobookshelf \u2014 http://192.168.1.40:13100 CT 104 \u2014 Garage S3 https://s3.nuclide.systems http://192.168.1.40:10004 CT 104 \u2014 UNAS NFS \u2014 <code>nfs://192.168.1.30/share/\u2026</code> UNAS [[storage]]"},{"location":"services-overview/#identity-secrets","title":"Identity / secrets","text":"Service External Internal Host Doc Pocket-ID (OIDC IdP) https://id.nuclide.systems http://192.168.1.5:11000 CT 110 [[pocket-id]] Vaultwarden https://vault.nuclide.systems http://192.168.1.40:11001 CT 104 \u2014 Infisical \u2014 http://192.168.1.7:8200 CT 112 [[secrets-manager]]"},{"location":"services-overview/#dev-source","title":"Dev / source","text":"Service External Internal Host Doc Gitea https://git.nuclide.systems http://192.168.1.42:3000 CT 111 [[dev-environment]] Coder https://dev.nuclide.systems http://192.168.1.42:7080 CT 111 [[dev-environment]] Docs (mkdocs) \u2014 http://192.168.1.8:13080 CT 109 \u2014"},{"location":"services-overview/#home-iot","title":"Home / IoT","text":"Service External Internal Host Doc Home Assistant https://ha.nuclide.systems http://192.168.1.60:8123 VM 100 \u2014 Mainsail (3D printer) \u2014 http://192.168.1.189 external box \u2014 OCPP (EV charging) https://ocpp.nuclide.systems http://192.168.1.60:8887 VM 100 \u2014 Traccar \u2014 http://192.168.1.40:15000 CT 104 \u2014 Gotify https://gotify.nuclide.systems http://192.168.1.40:10003 CT 104 \u2014"},{"location":"services-overview/#monitoring-ops-ct-109","title":"Monitoring / ops (CT 109)","text":"Service External Internal Host Doc Homarr (dashboard) \u2014 http://192.168.1.8:7575 CT 109 \u2014 Grafana \u2014 http://192.168.1.8:3000 CT 109 \u2014 Prometheus \u2014 http://192.168.1.8:9090 CT 109 \u2014 Loki \u2014 http://192.168.1.8:3100 CT 109 \u2014 Arcane (Docker UI) https://arcane.nuclide.systems http://192.168.1.8:10002 CT 109 [[arcane]] Dozzle (logs) \u2014 http://192.168.1.8:10001 CT 109 \u2014 Wetty (SSH-in-browser) \u2014 http://192.168.1.8:4090 CT 109 \u2014 Backrest (backups UI) \u2014 http://192.168.1.3:9898 CT 103 [[backrest-ct103]]"},{"location":"services-overview/#network-infra","title":"Network / infra","text":"Service External Internal Host Doc AdGuard Home (DNS) \u2014 http://192.168.1.2 (admin) CT 102 [[adguard-dns]] Zoraxy (reverse proxy) \u2014 http://192.168.1.4:8000 CT 108 [[zoraxy]] Proxmox PVE \u2014 https://192.168.1.20:8006 nuc [[homelab-architecture]] D-Link router \u2014 http://192.168.1.1 router \u2014"},{"location":"services-overview/#external-only-third-party-hosted","title":"External-only (third party hosted)","text":"Service External Notes Shepard https://shepard.nuclide.systems proxied to <code>192.168.1.49</code> Shepard API https://shepard-api.nuclide.systems http://192.168.1.49:8080 Shepard Auth https://shepard-auth.nuclide.systems http://192.168.1.49:8082"},{"location":"services-overview/#maintenance","title":"Maintenance","text":"<p>This overview lives in <code>/docs/services-overview.md</code>. To add or change a service, edit, commit, push \u2014 docs-server picks it up within 5 min. Cross-reference: <code>/docs/ct-inventory.md</code> for sizing/role, <code>/docs/services/zoraxy.md</code> for the authoritative external route list, Homarr (http://192.168.1.8:7575) for the visual board.</p>"},{"location":"history/case-study/","title":"Case study","text":"<p>STATUS: NARRATIVE \u2014 historical writeup. Not authoritative for current state.</p>"},{"location":"history/case-study/#case-study-the-nuclide-homelab-built-with-claude","title":"Case Study \u2014 The Nuclide Homelab, built with Claude","text":""},{"location":"history/case-study/#origin-story","title":"Origin story","text":"<p>One Saturday, the owner's wife left him home alone. He got bored, subscribed to Claude, and started tinkering with a home server. That afternoon of boredom turned into the <code>/opt/stacks</code> ecosystem documented here \u2014 a ~66-container, ~23-stack self-hosted platform with SSO, an MCP/agent gateway, GPU offload, and a fully audited network. This is that story, kept as a record of what a curiosity-driven collaboration produced.</p> <p>This is a personal passion project, not a work deliverable. The tone and scope reflect that: depth and exploration over minimum-viable.</p>"},{"location":"history/case-study/#what-was-built-high-level","title":"What was built (high level)","text":"<ul> <li>AI/agent core \u2014 LiteLLM model gateway (~28 curated models), an MCP gateway that DinD-spawns and OAuth-gates MCP servers, an Agent Operator (cron/event agents), semantic tool retrieval, MCP\u2192LobeChat registration.</li> <li>Identity \u2014 PocketID as the universal OIDC IdP; every service behind SSO (LiteLLM, LobeChat, n8n, Nextcloud, Daytona via a Keycloak\u2192PocketID adapter).</li> <li>Data layer \u2014 shared-postgres standard; a full tier-1 \"no SQLite on NFS\" migration (ntfy, Karakeep, Arcane, traccar \u2192 local disk; Memos + Vaultwarden \u2192 Postgres via pgloader); WAL-G PITR backups re-established.</li> <li>Network \u2014 full read-only UniFi audit + the unmanaged D-Link DGS-1210 core (SNTP fixed, topology/FDB mapped, DHCP/LLDP interop verified, SPOF identified); pinned-NTP-source standard adopted.</li> <li>Reliability \u2014 gateway deep health-check (real MCP tool-call probe, usage-aware backoff) that caught silently-broken servers.</li> <li>Cloud burst \u2014 on-demand Scaleway L40S GPU offload via WireGuard.</li> </ul> <p>See <code>homelab-architecture.md</code> for the living technical reference and <code>PORTMAP.md</code> for the authoritative port/route map.</p>"},{"location":"history/case-study/#activity-signal","title":"Activity signal","text":"<ul> <li>~79 commits in the trailing 14 days (<code>git log --since=\"14 days ago\"</code>), spanning gateway OAuth/health, OIDC bolt-ons, DB migrations, network audit, GPU integration, and docs.</li> <li>Multi-session, incident-driven: several entries trace to real failures caught and fixed (WAL-G archiver hung silently ~13 h; n8n <code>latest</code>-drift outage; SQLite-on-NFS corruption risk; an 8.5-month-stale switch clock).</li> </ul>"},{"location":"history/case-study/#productivity-estimate-honest-framing","title":"Productivity estimate (honest framing)","text":"<p>These are rough order-of-magnitude estimates, not measurements. Assumptions are stated so they can be challenged.</p> <ul> <li>Scope delivered \u2248 a small platform: identity, AI gateway, agent runtime, ~23 service stacks, a DB-migration program, a full network audit, backups.</li> <li>A solo engineer doing this unaided, part-time, learning the unfamiliar pieces (OIDC internals, pgloader, MCP, UniFi/D-Link internals): a conservative bound is several hundred focused hours (\u2248 8\u201316 part-time weeks). Assumes the owner is competent but not a specialist in every domain touched (identity, Postgres ops, embedded-switch web UIs, MCP).</li> <li>With the assistant: compressed into a small number of intensive sessions over ~2 weeks. The leverage is largest where the work is research-heavy but low-novelty \u2014 reverse-engineering a D-Link form POST, deriving an OIDC redirect, mapping an FDB table \u2014 i.e. tasks that are tedious solo but not conceptually hard. The leverage is smallest on genuine judgment calls (what to prioritise, what risk is acceptable), which stayed with the owner.</li> <li>Caveat: estimate excludes the owner's own steering/review time, which was substantial and is the reason the output is coherent rather than just voluminous.</li> </ul>"},{"location":"history/case-study/#co2-estimate-honest-framing","title":"CO2 estimate (honest framing)","text":"<p>Also order-of-magnitude, assumptions explicit.</p> <ul> <li>LLM inference: a heavy multi-session collaboration of this kind is on the order of a few million tokens. Public estimates put frontier-model inference at roughly 1\u20135 Wh per ~1k output tokens equivalent (wide error bars). Taking ~3 M tokens \u00d7 ~2 Wh/1k \u2248 ~6 kWh \u2192 at a ~0.35 kgCO2e/kWh grid \u2248 ~2 kgCO2e. Plausible range 1\u20135 kgCO2e. Datacenter PUE and exact model size dominate the uncertainty.</li> <li>Homelab runtime (the larger ongoing footprint): the NUC 14 Pro draws ~15\u201345 W under mixed load. At ~30 W average \u2192 ~0.72 kWh/day \u2192 ~260 kWh/yr \u2192 ~90 kgCO2e/yr at the same grid factor. The German grid is cleaner than that average in many hours, so treat as an upper-ish bound. The on-demand Scaleway L40S burst is deliberately on-demand precisely to avoid a 24/7 GPU's footprint.</li> <li>Takeaway: the assistant-collaboration carbon is a rounding error next to a year of always-on homelab power. Efficiency wins (CPU-only default, on-demand GPU, idle-aware health probing) matter more than the chat cost.</li> </ul>"},{"location":"history/case-study/#handover-current-state","title":"Handover / current state","text":"<p>Healthy & verified - Tier-1 SQLite-off-NFS: complete. - OIDC: n8n (302\u2192PocketID, client <code>33135ad4</code>) and LobeChat (<code>AUTH_TRUSTED_ ORIGINS</code> fix, sign-in\u2192PocketID) \u2014 both verified; LobeChat wants one real browser login as final proof. - D-Link SNTP: fixed (pinned PTB+Cloudflare IPs, clock corrected & synced). - Gateway deep health-check: live, usage-aware, surfaced in <code>/api/servers</code>.</p> <p>Open / pending (see <code>homelab-architecture.md</code> roadmap for detail) - Broken MCP servers surfaced by the new health-check: <code>memos</code> (degraded \u2014 <code>mcp-memos</code> can't resolve <code>memos</code> host; Docker-network isolation), <code>context7</code>/<code>crawl4ai</code>/<code>markitdown</code> (down), <code>nextcloud</code> (probe false-positive \u2014 needs <code>health_check:false</code> or per-user creds). - D-Link mgmt hardening (bundle, confirm-first): HTTPS, SNMP review, Trusted-Host allowlist <code>192.168.1.0/24</code>. Shared <code>tapirnase</code> password reuse (WiFi/LiteLLM/switch) \u2014 rotation deferred, noted. - Network: IoT-VLAN segmentation; D-Link is the unmanaged core/SPOF; mgmt-TLS certs for Proxmox + D-Link. - Platform: env\u2192secret vault; LobeChat external-feature disable; observability LXC; agent-platform evolution (memory/teams/MCP-exposed). - <code>nexa</code> analysis blocked \u2014 private repo; deploy key pending authorization.</p> <p>Operating rules to preserve - Confirm + risk-assess before any Proxmox / Ubiquiti / network-gear write. - Never put DB/SQLite on the UNAS NFS share. - Only a full pgloader of all tables is a complete DB migration. - Prefer self-hosted; pin critical container images (no <code>latest</code> drift).</p>"},{"location":"history/mcp-gateway-requirements/","title":"MCP gateway requirements (superseded)","text":"<p>STATUS: SUPERSEDED 2026-05-17 \u2014 current implementation lives in services/mcp-gateway.md. Kept for design-rationale history.</p>"},{"location":"history/mcp-gateway-requirements/#mcp-gateway-reconstructed-design-spec-in-progress-phase-1","title":"MCP Gateway \u2014 Reconstructed Design Spec (in-progress, \"Phase 1\")","text":"<p>Reconstructed 2026-05-16 from code/configs/git history. The gateway is a single-squash-commit first draft (<code>0cad389 \"Phase 1: Create MCP Gateway with Docker-in-Docker support\"</code>, preceded by <code>726bd10 \"WIP: MCP gateway prep\"</code>). Nothing has a second iteration in git \u2014 everything below is first-draft intent.</p>"},{"location":"history/mcp-gateway-requirements/#1-goal-intent","title":"1. Goal / Intent","text":"<p>A single OAuth-protected HTTP entrypoint at <code>https://mcp.nuclide.systems</code> that exposes a curated set of MCP servers to AI clients on the homelab. Primary consumer: Claude.ai as a remote connector (SSE at <code>/</code>, every README's \"Usage in Claude.ai\"). Secondary: LobeChat (<code>chat.nuclide.systems</code>) and LiteLLM (<code>ai.nuclide.systems</code>), sharing the same Pocket ID OAuth app. It is meant to replace the \"cumbersome\" static-compose approach (<code>mcp-tools.yaml</code>) with a dynamic, UI-managed, self-hosting model \u2014 answering the open <code>todo.md</code> question \"MCP deployment seems cumbersome \u2014 can litellm host directly? how to integrate npx, uvx, docker-based containers?\". Unifying idea: normalize npx / uvx / docker MCP servers behind one Dockerized gateway.</p>"},{"location":"history/mcp-gateway-requirements/#2-architecture-three-competing-models-in-repo-a-chosen-b-orphaned-c-aspirational","title":"2. Architecture (three competing models in-repo; A chosen, B orphaned, C aspirational)","text":"<p>A. FastAPI gateway + Docker-in-Docker (chosen) \u2014 <code>ai/mcp-gateway/</code> - FastAPI + <code>uvicorn</code> on <code>0.0.0.0:8080</code>, container <code>mcp-gateway</code>. - DinD via bind-mounted <code>/var/run/docker.sock</code>; <code>docker.from_env()</code>. - Per-server containers spawned <code>mcp-<name></code>, hardcoded onto <code>ai-internal</code>. - Config <code>config.json</code> (RW bind, currently EMPTY \u2192 falls back to <code>DEFAULT_SERVERS</code>). - Gateway joins <code>ai-internal</code> + <code>shared_backend</code> (both <code>external: true</code>).</p> <p>B. Static compose <code>mcp-tools.yaml</code> \u2014 orphaned; <code>ai/docker-compose.yml:6</code> include is commented out. Internally malformed (see \u00a74).</p> <p>C. LiteLLM-hosted \u2014 <code>litellm-config/config.yaml</code> <code>mcp_servers: {}</code> empty. Confirms MCP hosting was intended for the gateway, not LiteLLM (the <code>todo.md</code> \"can litellm host directly?\" question remains open).</p> <p>Transports (normalized to HTTP-on-:8000): streamable-http (nextcloud, mermaid), <code>mcp-proxy --stateless</code> stdio\u2192HTTP (papersearch), native HTTP (markitdown, crawl4ai :11235), and the gateway's own SSE <code>/</code> endpoint \u2014 a STUB (fake <code>initialize</code> + 60s pings, no routing to backends).</p> <p>Reverse proxy: Zoraxy <code>mcp.nuclide.systems \u2192 192.168.1.40:8080</code>. <code>mcp-auth.nuclide.systems</code> is an abandoned auth-sidecar idea (not exposed).</p> <p>OAuth (Pocket ID @ <code>id.nuclide.systems</code>): <code>OAuth2AuthorizationCodeBearer</code>, scopes <code>{openid, mcp}</code>, token validation via userinfo. Shared gateway client (<code>GENERIC_CLIENT_ID</code>, same as LiteLLM/LobeChat). Per-server OAuth for papersearch & nextcloud against the same Pocket ID.</p>"},{"location":"history/mcp-gateway-requirements/#3-mcp-server-inventory-reconciled-serverpy-mcp_servers-is-authoritative","title":"3. MCP Server Inventory (reconciled \u2014 <code>server.py</code> MCP_SERVERS is authoritative)","text":"Server Image / build Transport Port Auth Status papersearch <code>python:3.12-slim</code> + runtime <code>uv tool install mcp-proxy</code> \u2192 <code>paper_search_mcp.server</code> mcp-proxy stdio\u2192http 8000 Pocket ID <code>PAPERSEARCH_MCP_OAUTH_*</code> plausible, runtime-install fragile nextcloud <code>ghcr.io/cbcoutinho/nextcloud-mcp-server:latest</code> streamable-http 8000 Pocket ID <code>NEXTCLOUD_MCP_OAUTH_*</code> likely workable (real image) markitdown <code>python:3.12-slim</code> + <code>uvx markitdown-mcp --http</code> http 8000 none broken as written (<code>uvx</code> not in base image) comfyui <code>ghcr.io/richardi-ai/comfyui-mcp-server:latest</code> (<code>type:\"npm\"</code> mismatch) unspecified 8000 none image not pullable; backend ComfyUI was crash-looping crawl4ai <code>unclecode/crawl4ai:latest</code> http 11235 none likely workable; resource limits lost in rewrite mermaid <code>node:20-slim</code> + runtime <code>npx -y mcp-mermaid</code> streamable-http 8000 none plausible, slow first start"},{"location":"history/mcp-gateway-requirements/#4-implemented-vs-unfinished-vs-broken","title":"4. Implemented vs Unfinished vs Broken","text":"<p>Implemented: FastAPI app + OAuth scheme + userinfo token validation; container lifecycle CRUD + persistence; Web UI SPA (<code>templates/ui.html</code> @ <code>/ui</code>); gateway compose/Dockerfile + Zoraxy route.</p> <p>Unfinished / stub: - SSE <code>/</code> is fake \u2014 no MCP transport bridging Claude.ai \u2192 spawned servers. Core gap. - No routing to per-server containers; all five servers bind the same <code>:8000</code> and <code>spawn_container</code> host-publishes <code>8000:8000</code> \u2192 two servers can't run at once. - OAuth callback non-functional \u2014 token-exchange URL built via <code>OAUTH_REDIRECT_URI.replace(\"/sso/callback\",\"/token\")</code> (\u2192 wrong host, not the Pocket ID token endpoint); token never stored/used. - <code>config.json</code> empty \u2192 always defaults; secrets hardcoded plaintext in <code>server.py</code>.</p> <p>Broken / contradictory: - <code>ai/docker-compose.yml:6</code> mcp-tools include commented out; gateway compose is a separate project not referenced by the stack either \u2014 wired in only via Zoraxy. - <code>mcp-tools.yaml</code>: duplicate <code>markitdown-mcp</code> key; <code>comfyui-mcp</code> env missing <code>=</code> (<code>- COMFYUI_URL http://comfyui:8188</code>); missing images/ports. - <code>comfyui</code> server <code>type:\"npm\"</code> vs Docker-image mismatch; upstream image/npm package existence unverified (image confirmed not pullable). - <code>.env</code> has <code>CRAWL4AI_MCP_OAUTH_*</code>, <code>COMFYUI_MCP_OAUTH_*</code> that <code>server.py</code> never consumes; code hardcodes secrets instead of <code>${ENV}</code> substitution.</p>"},{"location":"history/mcp-gateway-requirements/#5-relevant-env-keys-names-only","title":"5. Relevant <code>.env</code> keys (names only)","text":"<p>Gateway: <code>GENERIC_CLIENT_ID/_SECRET/_REDIRECT_URI</code>, <code>GENERIC_{AUTHORIZATION,TOKEN,USERINFO}_ENDPOINT</code>, <code>GENERIC_CLIENT_USE_PKCE</code>, <code>OAUTH_SCOPES</code>, <code>OAUTH_TOKEN_URL</code>. Per-server: <code>NEXTCLOUD_MCP_OAUTH_CLIENT_ID/_SECRET</code>, <code>PAPERSEARCH_MCP_OAUTH_CLIENT_ID/_SECRET</code>, <code>MARKITDOWN_MCP_OAUTH_CLIENT_ID/_SECRET</code> (declared, unused), <code>CRAWL4AI_/COMFYUI_MCP_OAUTH_*</code> (orphaned). papersearch data sources: <code>UNPAYWALL_EMAIL</code>, <code>CORE_API_KEY</code>, <code>SEMANTIC_SCHOLAR_API_KEY</code>, <code>ZENODO_ACCESS_TOKEN</code>, <code>GOOGLE_SCHOLAR_PROXY_URL</code>, <code>DOAJ_API_KEY</code>. comfyui: <code>COMFYUI_URL</code>, <code>COMFYUI_WS_URL</code>.</p>"},{"location":"history/mcp-gateway-requirements/#6-open-design-decisions-must-resolve","title":"6. Open Design Decisions (must resolve)","text":"<ol> <li>Hosting model: DinD gateway vs static <code>mcp-tools.yaml</code> vs LiteLLM-hosted.</li> <li>How Claude.ai/LobeChat reach a tool: the MCP transport bridge doesn't exist.</li> <li>Port allocation: all servers hardcode <code>:8000</code> \u2014 need internal DNS, no host publish.</li> <li>Uniform npx/uvx/docker run model: prebuilt images vs runtime install.</li> <li>Auth model: gateway-terminated vs per-MCP vs pass-through (callback is broken).</li> <li>Secret handling: hardcoded \u2192 <code>${ENV}</code> from <code>ai/.env</code>.</li> <li>comfyui server: image vs npm; keep only once ComfyUI itself is stable.</li> <li>Discovery for LobeChat/LiteLLM: <code>/mcp.json</code> is OAuth-gated, lists config not endpoints.</li> </ol>"},{"location":"history/mcp-gateway-requirements/#7-recommended-path-ordered-lowest-risk-first","title":"7. Recommended Path (ordered, lowest-risk first)","text":"<ol> <li>Pick the static-compose path, not DinD \u2014 lowest risk on a single NUC; DinD adds socket-exposure risk + a broken SSE bridge for little gain. Fix and re-enable <code>mcp-tools.yaml</code> (uncomment <code>ai/docker-compose.yml:6</code>).</li> <li>Fix <code>mcp-tools.yaml</code>: dedupe <code>markitdown-mcp</code>, fix <code>comfyui-mcp</code> env <code>=</code>, unique service names \u2192 <code>ai-internal</code> DNS, pin images, drop comfyui for now.</li> <li>One streamable-http reverse proxy keyed by path (<code>mcp.nuclide.systems/<server></code>) via Zoraxy or a small <code>httpx</code> proxy \u2014 replace the fake SSE stub. Backends stay internal on <code>ai-internal:8000</code>, never host-published.</li> <li>Move secrets to <code>${ENV}</code> from <code>ai/.env</code> (keys already exist).</li> <li>Fix OAuth callback: exchange code against <code>GENERIC_TOKEN_ENDPOINT</code>.</li> <li>Verify each upstream image/tool exists before marking a server \"working\".</li> <li>Defer the DinD gateway + Web UI to phase-2 (read-only status over the running compose, not spawning).</li> <li>Answer the litellm question: once stable <code>https://mcp.nuclide.systems/<server></code> URLs exist, populate <code>litellm-config/config.yaml</code> <code>mcp_servers:</code> so LiteLLM/LobeChat discover them \u2014 no bespoke discovery path needed.</li> </ol>"},{"location":"history/scrubbing-list-2026-05-17/","title":"Scrubbing list (2026-05-17)","text":"<p>STATUS: SNAPSHOT \u2014 frozen inventory from 2026-05-17. Reality has moved on; consult CHANGELOG.md + services/* for current state.</p>"},{"location":"history/scrubbing-list-2026-05-17/#scrubbing-list-optstacks-2026-05-17","title":"Scrubbing List \u2014 /opt/stacks (2026-05-17)","text":"<p>Read-only audit of unused/stale data. Nothing here has been deleted \u2014 review the labels and run the commands yourself. Root FS was 165G/200G used (83%); <code>/var/lib/docker</code> is 86G of <code>/opt/stacks</code>'s 99G.</p> <p>No stopped/exited/<code>*_old</code> containers; no unused custom networks (already clean).</p>"},{"location":"history/scrubbing-list-2026-05-17/#1-docker-reclaimable","title":"1. Docker reclaimable","text":"Target Size Label Command Build cache (268 entries, old comfyui CPU\u2192Arc rebuilds, 0 in use) ~20.85 GB SAFE <code>docker builder prune -af</code> 4\u00d7 dangling <code><none></code> mcp-gateway rebuild images (1.59 GB ea) ~6.36 GB SAFE <code>docker image prune</code> 2 old dangling images (756 MB + 113 MB) ~0.87 GB SAFE (same <code>docker image prune</code>) ~265 anon volumes; one <code>54bcf7\u2026</code> = 8.69 GB unidentified, rest ~0B ~9 GB REVIEW inspect <code>54bcf7\u2026</code> then <code>docker volume prune</code> Named dangling vols: <code>n8n_n8n_storage</code> 128M, <code>ai_mcpo-data</code> 84M, <code>paperless-ngx_pgdata</code> 26M, <code>metamcp_postgres_data</code> 18M, <code>daytona*_db_data</code> 15M\u00d72, <code>librechat_pgdata2</code> 13M, <code>arcane_arcane-data</code> 12M, <code>ai_redis_data</code> 7.6M ~0.3 GB REVIEW <code>docker volume rm <name></code> per-item after confirming the stack is retired <p>In-use, DO NOT REMOVE: <code>comfyui-comfyui</code> (6.45G), <code>clusterzx/paperless-ai</code> (8.59G).</p>"},{"location":"history/scrubbing-list-2026-05-17/#2-migratedabandoned-local-data-dirs-compose-now-points-to-unas","title":"2. Migrated/abandoned local data dirs (compose now points to UNAS)","text":"Path Size Label Note <code>arr-stack/media</code> 55 G REVIEW No container mounts it; arr \u2192 <code>/mnt/pve/unas/media</code>. Audiobooks/ebooks have recent mtimes (rsync residue) \u2014 parity-check vs UNAS before <code>rm -rf</code> <code>ai/data</code> (old postgres) 195 M SAFE Not mounted, not referenced <code>ai/postgres_data</code> (incl 38M pg_wal) 120 M REVIEW Not mounted/referenced but recent mtime <code>ai/meili_data_v1.35.1</code> 19 M SAFE Old Meili, not mounted <code>qdrant/qdrant_storage</code> 7 M SAFE qdrant migrated to UNAS (fresh start) <code>daytona/db_data</code> 14 M REVIEW No mount; daytona uses named volumes <code>n8n/data</code> empty SAFE <code>rmdir</code> <p>Active local, KEEP: <code>immich/postgres</code> (814M), <code>ai/lobehub/data</code> (25M), <code>shared-db/wal-g</code> (27M, RO mount), arr-stack configs, <code>ai/litellm-config</code>, <code>ai/searxng</code>.</p>"},{"location":"history/scrubbing-list-2026-05-17/#3-migration-scratch-artifacts","title":"3. Migration / scratch artifacts","text":"Path Label Note <code>ai/docker</code> (0 B), <code>ai/bucket.config.json</code> (empty dir) SAFE junk <code>scripts/traefik-*.sh</code> SAFE Traefik abandoned for Zoraxy <code>scripts/{migrate_*,test_adguard_api,zoraxy_csrf,zoraxy_test,configure_zoraxy_*}.py</code> REVIEW one-shot done; confirm no rerun need <code>scripts/zoraxy_sync.py</code> KEEP ongoing proxy tooling <code>scripts/.venv</code> (29M), <code>scripts/.kilo</code> (30M) REVIEW regenerable caches <code>/tmp/{flux_*,pw_ui,add_*,fix_*}.* , /tmp/*.png , /tmp/*.log</code> SAFE ~1.8M scratch (this session)"},{"location":"history/scrubbing-list-2026-05-17/#bottom-line","title":"Bottom line","text":"<ul> <li>Safe-only reclaim: ~29\u201330 GB (build cache + dangling images dominate).</li> <li>With review: +~64 GB (55G stale <code>arr-stack/media</code> after UNAS parity check, ~9G anon vols).</li> <li>Highest-value, zero-risk action: <code>docker builder prune -af</code> \u2192 ~20.85 GB now.</li> <li>Biggest overall prize: verify+remove <code>arr-stack/media</code> (55 G) \u2014 parity-check vs <code>/mnt/pve/unas/media</code> first.</li> </ul>"},{"location":"history/traefik-migration-docker-labels/","title":"Traefik labels (abandoned)","text":"<p>STATUS: ABANDONED 2026-05-16 \u2014 Zoraxy is the production reverse proxy. Kept for design-decision history.</p>"},{"location":"history/traefik-migration-docker-labels/#traefik-migration-guide-using-docker-labels","title":"Traefik Migration Guide Using Docker Labels","text":""},{"location":"history/traefik-migration-docker-labels/#overview","title":"Overview","text":"<p>Migrate from Zoraxy reverse proxy to Traefik using Docker labels for zero-touch service discovery.</p>"},{"location":"history/traefik-migration-docker-labels/#phase-1-install-traefik","title":"Phase 1: Install Traefik","text":""},{"location":"history/traefik-migration-docker-labels/#step-1-create-directory-structure","title":"Step 1: Create Directory Structure","text":"<pre><code>mkdir -p /opt/stacks/proxy/traefik/{config,dynamic,letsencrypt}\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#step-2-create-docker-composeyml","title":"Step 2: Create docker-compose.yml","text":"<pre><code>version: \"3.8\"\n\nservices:\n traefik:\n image: traefik:v3.2\n container_name: traefik\n restart: always\n network_mode: host\n security_opt:\n - no-new-privileges=true\n ports:\n - \"80:80\"\n - \"443:443\"\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock:ro\n - /opt/stacks/proxy/traefik/config:/etc/traefik\n - /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic\n - /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt\n command:\n - \"--api.insecure=true\"\n - \"--providers.docker=true\"\n - \"--providers.docker.exposedbydefault=false\"\n - \"--providers.docker.network=ai-internal\"\n - \"--providers.docker.network=shared_backend\"\n - \"--providers.docker.defaultRule=Host(`{{ .Name }}.nuclide.systems`)\"\n - \"--entrypoints.web.address=:80\"\n - \"--entrypoints.websecure.address=:443\"\n - \"--certificatesresolvers.letsencrypt.acme.httpChallenge=true\"\n - \"--certificatesresolvers.letsencrypt.acme.email=admin@nuclide.systems\"\n - \"--certificatesresolvers.letsencrypt.acme.storage=/etc/letsencrypt/acme.json\"\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#step-3-start-traefik","title":"Step 3: Start Traefik","text":"<pre><code>cd /opt/stacks/proxy/traefik\ndocker compose up -d\n\n# Verify\ndocker compose ps\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#phase-2-migrate-ai-services","title":"Phase 2: Migrate AI Services","text":""},{"location":"history/traefik-migration-docker-labels/#ai-service-labels-add-to-litellm-chat-mcp-composeyml","title":"AI Service Labels (Add to litellm, chat, mcp-compose.yml)","text":"<pre><code>services:\n litellm:\n image: litellm\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)\"\n - \"traefik.http.routers.litellm.entrypoints=websecure\"\n - \"traefik.http.routers.litellm.tls=true\"\n - \"traefik.http.routers.litellm.tls.certresolver=letsencrypt\"\n - \"traefik.http.routers.litellm.priority=10\"\n - \"traefik.http.services.litellm.loadbalancer.server.port=14000\"\n\n chat:\n image: lobehub\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)\"\n - \"traefik.http.routers.chat.entrypoints=websecure\"\n - \"traefik.http.routers.chat.tls=true\"\n - \"traefik.http.routers.chat.tls.certresolver=letsencrypt\"\n - \"traefik.http.routers.chat.priority=10\"\n - \"traefik.http.services.chat.loadbalancer.server.port=14001\"\n\n mcp:\n image: mcp-gateway\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)\"\n - \"traefik.http.routers.mcp.entrypoints=websecure\"\n - \"traefik.http.routers.mcp.tls=true\"\n - \"traefik.http.routers.mcp.tls.certresolver=letsencrypt\"\n - \"traefik.http.routers.mcp.priority=10\"\n - \"traefik.http.services.mcp.loadbalancer.server.port=8080\"\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#phase-3-migrate-garage-s3","title":"Phase 3: Migrate Garage S3","text":""},{"location":"history/traefik-migration-docker-labels/#option-a-use-traefik-proxy","title":"Option A: Use Traefik Proxy","text":"<pre><code>services:\n garage-proxy:\n image: nginx:alpine\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.s3.rule=Host(`s3.nuclide.systems`)\"\n - \"traefik.http.routers.s3.entrypoints=websecure\"\n - \"traefik.http.routers.s3.tls=true\"\n - \"traefik.http.routers.s3.tls.certresolver=letsencrypt\"\n - \"traefik.http.services.s3.loadbalancer.server.port=10004\"\n volumes:\n - garage-data:/data\n networks:\n - shared_backend\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#option-b-keep-internal-garage-access","title":"Option B: Keep Internal Garage Access","text":"<pre><code>services:\n # No proxy needed - access Garage via internal IP:10004\n garage:\n image: garageio/garage\n ports:\n - \"3900:3900\" # Internal only\n - \"10004:10004\" # Public via Traefik\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#phase-4-create-helper-scripts","title":"Phase 4: Create Helper Scripts","text":""},{"location":"history/traefik-migration-docker-labels/#script-1-add-service-to-traefik","title":"script 1: Add Service to Traefik","text":"<pre><code>#!/bin/bash\n# /opt/stacks/scripts/add-traefik-service.sh\n\nNAME=$1\nDOMAIN=$2\nPORT=$3\n\ncat > /opt/stacks/proxy/traefik/dynamic/${NAME}.yml << EOF\nhttp:\n routers:\n ${NAME}-router:\n rule: \"Host(\\`${DOMAIN}\\`)\"\n service: ${NAME}-service\n entrypoints:\n - websecure\n tls:\n certresolver: letsencrypt\n\n services:\n ${NAME}-service:\n loadBalancer:\n servers:\n - url: \"http://192.168.1.40:${PORT}\"\nEOF\n\n# Reload Traefik docker automatically (no manual step needed)\necho \"\u2705 Service ${NAME} added via labels\"\n</code></pre> <p>Usage: <pre><code>/opt/stacks/scripts/add-traefik-service.sh myservice myservice.nuclide.systems 8000\n</code></pre></p>"},{"location":"history/traefik-migration-docker-labels/#script-2-generate-labels-for-existing-services","title":"Script 2: Generate Labels for Existing Services","text":"<pre><code>#!/bin/bash\n# /opt/stacks/scripts/traefik-labels-gen.sh\n\ncat > /opt/stacks/proxy/traefik/labels.yaml << 'EOF'\n# Add these labels to service docker-compose.yml files\n\n# LiteLLM\nservices:\n litellm:\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.litellm.rule=Host(`litellm.nuclide.systems`)\"\n - \"traefik.http.routers.litellm.entrypoints=websecure\"\n - \"traefik.http.routers.litellm.tls=true\"\n - \"traefik.http.routers.litellm.tls.certresolver=letsencrypt\"\n - \"traefik.http.services.litellm.loadbalancer.server.port=14000\"\n\n# LobeHub Chat\nservices:\n chat:\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.chat.rule=Host(`chat.nuclide.systems`)\"\n - \"traefik.http.routers.chat.entrypoints=websecure\"\n - \"traefik.http.routers.chat.tls=true\"\n - \"traefik.http.routers.chat.tls.certresolver=letsencrypt\"\n - \"traefik.http.services.chat.loadbalancer.server.port=14001\"\n\n# MCP Gateway\nservices:\n mcp-gateway:\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.mcp.rule=Host(`mcp.nuclide.systems`)\"\n - \"traefik.http.routers.mcp.entrypoints=websecure\"\n - \"traefik.http.routers.mcp.tls=true\"\n - \"traefik.http.routers.mcp.tls.certresolver=letsencrypt\"\n - \"traefik.http.services.mcp.loadbalancer.server.port=8080\"\nEOF\n\necho \"\u2705 Labels saved to /opt/stacks/proxy/traefik/labels.yaml\"\n</code></pre> <p>Usage: <pre><code>/opt/stacks/scripts/traefik-labels-gen.sh\n</code></pre></p>"},{"location":"history/traefik-migration-docker-labels/#phase-5-update-service-configs","title":"Phase 5: Update Service Configs","text":""},{"location":"history/traefik-migration-docker-labels/#update-optstacksaienv","title":"Update <code>/opt/stacks/ai/.env</code>","text":"<pre><code># OLD (Zoraxy):\nLITELLM_BASE_URL=https://litellm.nuclide.systems\nPROXY_BASE_URL=https://mcp.nuclide.systems\n\n# NEW (Traefik) - same URLs, different backend:\nLITELLM_BASE_URL=https://litellm.nuclide.systems\nPROXY_BASE_URL=https://mcp.nuclide.systems\nCHATAI_BASE_URL=https://chat.nuclide.systems\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#update-optstacksailitellm-configconfigyaml","title":"Update <code>/opt/stacks/ai/litellm-config/config.yaml</code>","text":"<pre><code>general_settings:\n proxy_base_url: https://litellm.nuclide.systems\n control_plane_url: https://litellm.nuclide.systems\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#phase-6-verify-ssl","title":"Phase 6: Verify SSL","text":""},{"location":"history/traefik-migration-docker-labels/#step-1-generate-lets-encrypt-certificates","title":"Step 1: Generate Let's Encrypt Certificates","text":"<pre><code># Verify Traefik is running\ndocker compose ps traefik\n\n# Create ACME cert file\ntouch /opt/stacks/proxy/traefik/letsencrypt/acme.json\nchmod 600 /opt/stacks/proxy/traefik/letsencrypt/acme.json\n\n# Trigger certificate generation (will happen automatically)\n# Check status:\ncurl -s https://acme-v02.api.letsencrypt.org/directory | head\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#step-2-test-https","title":"Step 2: Test HTTPS","text":"<pre><code># Test endpoints\ncurl -k https://litellm.nuclide.systems/health\ncurl -k https://chat.nuclide.systems/health\ncurl -k https://mcp.nuclide.systems/health\n\n# Verify cert\ncurl -v https://litellm.nuclide.systems 2>&1 | grep -A 5 \"SSL certificate\"\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#phase-7-remove-zoraxy","title":"Phase 7: Remove Zoraxy","text":""},{"location":"history/traefik-migration-docker-labels/#backup-first","title":"Backup first","text":"<pre><code># Backup Zoraxy configs\ndocker cp zoraxy:/data/configs /backup/zoraxy-backup/\n\n# Optional: Stop Zoraxy\ndocker stop zoraxy\ndocker rm zoraxy\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#quick-migration-checklist","title":"Quick Migration Checklist","text":"<ul> <li>[ ] Install Traefik (Phase 1)</li> <li>[ ] Add labels to AI services (Phase 2)</li> <li>[ ] Add Garage S3 proxy (Phase 3)</li> <li>[ ] Run label generation script (Phase 4)</li> <li>[ ] Update <code>.env</code> and config files (Phase 5)</li> <li>[ ] Wait for SSL certificates (auto 24-48h)</li> <li>[ ] Test all HTTPS endpoints</li> <li>[ ] Remove Zoraxy (Phase 7)</li> <li>[ ] Update AdGuard DNS if needed (optional)</li> </ul>"},{"location":"history/traefik-migration-docker-labels/#monitoring-troubleshooting","title":"Monitoring & Troubleshooting","text":""},{"location":"history/traefik-migration-docker-labels/#check-traefik-dashboard","title":"Check Traefik Dashboard","text":"<pre><code># Access web UI (unsecured - use only on trusted network)\nopen http://192.168.1.40:8080/dashboard\n\n# View all routers\ncurl http://localhost:8080/api/http/routers | jq '.[] | {name: .rule, status: .entryPoints}'\n\n# View all services\ncurl http://localhost:8080/api/http/services | jq '.[] | {name: .name, servers: .servers}'\n</code></pre>"},{"location":"history/traefik-migration-docker-labels/#common-issues","title":"Common Issues","text":"Issue Solution 404 errors Check router labels match domain exactly SSL expired Wait for auto-renew or trigger manually Port mismatch Verify <code>loadbalancer.server.port</code> matches service No SSL cert Check email in <code>acme.json</code> config"},{"location":"history/traefik-migration-docker-labels/#rollback-plan-if-needed","title":"Rollback Plan (If Needed)","text":"<pre><code># Stop Traefik\ndocker compose -f /opt/stacks/proxy/traefik/docker-compose.yml down\n\n# Restore Zoraxy configs\ndocker cp /backup/zoraxy-backup/ configs/\n\n# Restart Zoraxy (if you kept backup)\ndocker start zoraxy || true\n</code></pre>"},{"location":"history/traefik-migration/","title":"Traefik (abandoned 2026-05-16)","text":"<p>STATUS: ABANDONED 2026-05-16 \u2014 Zoraxy is the production reverse proxy. Kept for design-decision history. See services/zoraxy.md for current setup.</p>"},{"location":"history/traefik-migration/#recommended-traefik-proxy-replacement","title":"Recommended: Traefik Proxy Replacement","text":""},{"location":"history/traefik-migration/#why-traefik-over-zoraxy","title":"Why Traefik over Zoraxy?","text":"Feature Zoraxy Traefik API \u274c No public API \u2705 Full REST API SSL \ud83d\udcac Manual (Zoraxy Web UI) \ud83d\udd25 Auto-Let's Encrypt Dynamic \u26a0\ufe0f Manual config reload \u2705 Hot-reload configs File watching \u274c \u2705 Auto-detect changes Docker integration \u26a0\ufe0f Manual \u2705 Native labels API endpoints 403 Forbidden \u2705 JSON API everywhere"},{"location":"history/traefik-migration/#migration-path","title":"Migration Path","text":""},{"location":"history/traefik-migration/#current-setup","title":"Current setup:","text":"<pre><code>Zoraxy (192.168.1.4:8000) \u2192 Reverse Proxy Rules (Manual Web UI)\n- litellm.nuclide.systems \u2192 192.168.1.40:14000\n- chat.nuclide.systems \u2192 192.168.1.40:14001 \n- mcp.nuclide.systems \u2192 192.168.1.40:8080\n- s3.nuclide.systems \u2192 Garage:10004\n</code></pre>"},{"location":"history/traefik-migration/#new-setup-with-traefik","title":"New setup with Traefik:","text":"<pre><code>Traefik (public SSL) \u2192 Dynamic Router (labels/consul)\n- All services auto-discovered via Docker labels\n- SSL certificates auto-provisioned\n- No manual Zoraxy configuration needed\n</code></pre>"},{"location":"history/traefik-migration/#installation","title":"Installation","text":""},{"location":"history/traefik-migration/#step-1-install-traefik","title":"Step 1: Install Traefik","text":"<pre><code># Create Traefik directory\nmkdir -p /opt/stacks/proxy/traefik/{conf,dynamic}\n\n# Create docker-compose.yml\ncat > /opt/stacks/proxy/traefik/docker-compose.yml << 'EOF'\nversion: \"3.8\"\n\nservices:\n traefik:\n image: traefik:v3.2\n container_name: traefik\n restart: always\n security_opt:\n - no-new-privileges=true\n network_mode: host\n ports:\n - \"80:80\"\n - \"443:443\"\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock:ro\n - /opt/stacks/proxy/traefik/conf:/etc/traefik\n - /opt/stacks/proxy/traefik/dynamic:/etc/traefik/dynamic\n - /opt/stacks/proxy/traefik/letsencrypt:/etc/letsencrypt\n command:\n - \"--api.insecure=true\"\n - \"--providers.docker=true\"\n - \"--providers.docker.exposedbydefault=false\"\n - \"--entrypoints.web.address=:80\"\n - \"--entrypoints.websecure.address=:443\"\n - \"--certificatesresletsencryptemail=admin@nuclide.systems\"\n - \"--certificatesresletsencryptstorage=/etc/letsencrypt/acme.json\"\nEOF\n\n# Start Traefik\ncd /opt/stacks/proxy/traefik && docker compose up -d\n</code></pre>"},{"location":"history/traefik-migration/#step-2-create-dynamic-configuration","title":"Step 2: Create Dynamic Configuration","text":"<pre><code># Create router rules\ncat > /opt/stacks/proxy/traefik/dynamic/router.yml << 'EOF'\nhttp:\n routers:\n litellm-router:\n rule: \"Host(`litellm.nuclide.systems`)\"\n service: litellm-service\n entrypoints:\n - websecure\n tls:\n certresolver: letsencrypt\n\n chat-router:\n rule: \"Host(`chat.nuclide.systems`)\"\n service: chat-service\n entrypoints:\n - websecure\n tls:\n certresolver: letsencrypt\n\n mcp-router:\n rule: \"Host(`mcp.nuclide.systems`)\"\n service: mcp-service\n entrypoints:\n - websecure\n tls:\n certresolver: letsencrypt\n\n s3-router:\n rule: \"Host(`s3.nuclide.systems`)\"\n service: garage-service\n entrypoints:\n - websecure\n tls:\n certresolver: letsencrypt\n\n services:\n litellm-service:\n loadBalancer:\n servers:\n - url: \"http://192.168.1.40:14000\"\n\n chat-service:\n loadBalancer:\n servers:\n - url: \"http://192.168.1.40:14001\"\n\n mcp-service:\n loadBalancer:\n servers:\n - url: \"http://192.168.1.40:8080\"\n\n garage-service:\n loadBalancer:\n servers:\n - url: \"http://garage:10004\"\nEOF\n</code></pre>"},{"location":"history/traefik-migration/#step-3-add-docker-labels-to-services","title":"Step 3: Add Docker Labels to Services","text":"<p>For any Docker service you want to proxy:</p> <pre><code># Example: Add to your service docker-compose.yml\nservices:\n ai-service:\n image: your-service\n labels:\n - \"traefik.enable=true\"\n - \"traefik.http.routers.your-service.rule=Host(`your-service.nuclide.systems`)\"\n - \"traefik.http.routers.your-service.entrypoints=websecure\"\n - \"traefik.http.routers.your-service.tls.certresolver=letsencrypt\"\n - \"traefik.http.services.your-service.loadBalancer.server.port=8000\"\n</code></pre>"},{"location":"history/traefik-migration/#step-4-delete-zoraxy-optional","title":"Step 4: Delete Zoraxy (Optional)","text":"<pre><code># Backup Zoraxy configs first\ntar -czf /backup/zoraxy-backup.tar.gz /path/to/zoraxy/configs\n\n# Stop and remove Zoraxy\ndocker rm -f zoraxy || true\n</code></pre>"},{"location":"history/traefik-migration/#api-example-traefik","title":"API Example (Traefik)","text":"<pre><code># Get list of services\ncurl -u traefik:YOUR_TRAEFIK_API_PASSWORD http://localhost:8080/api/http/routers\n\n# Get Traefik metrics\ncurl http://localhost:8080/metrics\n\n# Reload configuration (live)\ncurl -X POST http://localhost:8080/api/http/routers -H \"Content-Type: application/json\" -d '{...}'\n</code></pre>"},{"location":"history/traefik-migration/#migration-checklist","title":"Migration Checklist","text":"<ul> <li>[ ] Deploy Traefik in host network mode</li> <li>[ ] Create Let's Encrypt certificate for your domain</li> <li>[ ] Migrate all reverse proxy rules to Traefik labels or dynamic config</li> <li>[ ] Test SSL certificates work: <code>curl -k https://your-domain.nuclide.systems</code></li> <li>[ ] Remove Zoraxy Docker container</li> <li>[ ] Update DNS if needed</li> <li>[ ] Verify all endpoints: health checks at new URLs</li> </ul>"},{"location":"history/traefik-migration/#benefits","title":"Benefits","text":"<ol> <li>Zero maintenance SSL - Let's Encrypt auto-renews</li> <li>API-driven - No manual Web UI needed</li> <li>Hot reloading - Changes apply immediately</li> <li>Docker-native - Watches container labels automatically</li> <li>Enterprise-grade - Used by major cloud providers</li> </ol>"},{"location":"ideas/litellm-claude-max-bridge/","title":"Design: Claude Max Subscription \u2192 LiteLLM Gateway Bridge","text":"<p>Status: Proposal \u2014 not implemented Context: LiteLLM at <code>https://ai.nuclide.systems</code> currently has no Anthropic models. Claude Max subscription (claude.ai) provides high-rate access to Sonnet 4.5/4.6 and Opus 4.7 but is decoupled from Anthropic API billing. This doc explores bridging the two.</p>"},{"location":"ideas/litellm-claude-max-bridge/#problem","title":"Problem","text":"<p>Claude Max and the Anthropic API are separate products with separate billing:</p> Claude Max Anthropic API Auth OAuth / browser session API key Billing Flat monthly subscription Per-token Rate limits 5h rolling windows, model-specific Tier-based RPM/TPM Access claude.ai web + Claude Code CLI Any HTTP client <p>The goal is to surface Max-subscription capacity through LiteLLM so that LobeHub, n8n, Coder workspaces, and other internal tools can call <code>claude-sonnet-4-6</code> at zero marginal cost and fall back to paid providers only when Max limits are hit.</p>"},{"location":"ideas/litellm-claude-max-bridge/#approaches","title":"Approaches","text":""},{"location":"ideas/litellm-claude-max-bridge/#a-session-cookie-reverse-engineering-not-recommended","title":"A \u2014 Session Cookie Reverse-Engineering (not recommended)","text":"<p>Several community projects (e.g. <code>claude-unofficial-api</code>) scrape the claude.ai WebSocket/HTTP protocol and expose an OpenAI-compatible endpoint. LiteLLM would point at this as a custom <code>openai/</code> provider.</p> <p>Pros: Exposes the full web model lineup; streaming works. Cons: Violates Anthropic ToS; breaks on any claude.ai front-end change; auth flow requires persisting browser cookies; no multimodal or tool-use parity guarantees.</p> <p>Verdict: Avoid. Fragile and non-compliant.</p>"},{"location":"ideas/litellm-claude-max-bridge/#b-claude-code-cli-bridge-recommended","title":"B \u2014 Claude Code CLI Bridge (recommended)","text":"<p>Claude Code CLI (<code>claude</code>) is already installed on CT 104 and authenticated with the Max subscription via <code>~/.claude/</code>. It ships a <code>--print</code> / <code>--output-format stream-json</code> mode designed for non-interactive use, and Anthropic explicitly supports programmatic use of the CLI.</p> <p>A small <code>claude-max-bridge</code> service wraps this CLI as an OpenAI-compatible HTTP endpoint. LiteLLM registers it as a custom <code>openai/</code> base URL. No ToS issues \u2014 this is the supported surface.</p> <pre><code>LobeHub / n8n / Coder / Claude Code\n \u2502\n \u25bc\n LiteLLM Gateway (ai.nuclide.systems)\n \u2502 model: claude-sonnet-4-6 \u2192 openai/claude-sonnet-4-6\n \u2502 api_base: http://claude-max-bridge:8000\n \u25bc\n claude-max-bridge (new container, CT 104 ai-internal)\n \u2502 subprocess: claude --model ... --print --output-format stream-json\n \u25bc\n ~/.claude/ (Max subscription session)\n \u2502\n \u25bc\n Anthropic (claude.ai)\n</code></pre> <p>Pros: - Uses the officially supported programmatic interface - Auth is already set up; no cookie management - <code>claude</code> CLI handles retries, token limits, context window management - Subprocess overhead is ~200\u2013400 ms cold; warm invocations faster</p> <p>Cons: - One subprocess per request \u2014 cannot multiplex a single session (unlike streaming HTTP) - CLI is tied to the single authenticated user; no multi-user isolation - Max rate limits apply per-account, same pool as interactive use - Claude Code SDK (TypeScript) is cleaner but adds Node dependency</p>"},{"location":"ideas/litellm-claude-max-bridge/#c-official-api-budget-cap-stop-gap","title":"C \u2014 Official API + Budget Cap (stop-gap)","text":"<p>Add Anthropic API key to LiteLLM with a hard budget cap (e.g. $20/month). Use it for Claude-specific features (tool use, long context) and let the existing free SAIA/Gemini fallback chain absorb general-purpose traffic.</p> <p>Pros: Zero implementation work; full API feature parity. Cons: Still costs money; no benefit from Max subscription.</p> <p>Verdict: Valid fallback if B proves too complex, or as a complement for tool-heavy workloads that need the official API surface.</p>"},{"location":"ideas/litellm-claude-max-bridge/#recommended-architecture-option-b","title":"Recommended Architecture (Option B)","text":""},{"location":"ideas/litellm-claude-max-bridge/#claude-max-bridge-service","title":"claude-max-bridge service","text":"<pre><code>/opt/stacks/ai/claude-max-bridge/\n Dockerfile\n server.py # FastAPI, ~150 lines\n docker-compose.yml (or entry in ai/docker-compose.yml)\n</code></pre> <p>Dockerfile \u2014 reuse the existing <code>claude</code> CLI install: <pre><code>FROM python:3.12-slim\nRUN pip install fastapi uvicorn\n# Mount ~/.claude from host; claude binary from host PATH or copied in\nCOPY server.py /app/server.py\nCMD [\"uvicorn\", \"app.server:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n</code></pre></p> <p>server.py \u2014 OpenAI <code>/v1/chat/completions</code> shim: <pre><code># POST /v1/chat/completions\n# Translates messages[] \u2192 claude --print --model ... --output-format stream-json\n# Streams NDJSON lines back as SSE (text/event-stream)\n# Maps finish_reason, usage tokens from CLI output headers\n</code></pre></p> <p>Key translation notes: - <code>messages</code> \u2192 write to a temp file, pass via <code>--input-file</code> (avoids shell quoting issues) - <code>stream: true</code> \u2192 parse <code>stream-json</code> lines, emit <code>data: {...}</code> SSE chunks - <code>stream: false</code> \u2192 buffer all chunks, return single response object - <code>max_tokens</code>, <code>temperature</code>, <code>system</code> \u2192 map to <code>--max-tokens</code>, <code>--temperature</code>, <code>--system</code> - Tool use: not supported in first iteration; return 501 for requests with <code>tools</code> - Model name passthrough: <code>claude-sonnet-4-6</code> \u2192 <code>--model claude-sonnet-4-6</code></p>"},{"location":"ideas/litellm-claude-max-bridge/#litellm-config-additions","title":"LiteLLM config additions","text":"<pre><code>model_list:\n - model_name: claude-sonnet-4-6\n litellm_params:\n model: openai/claude-sonnet-4-6\n api_base: http://claude-max-bridge:8000\n api_key: \"dummy\" # bridge ignores it; LiteLLM requires a value\n stream_timeout: 120\n timeout: 120\n\n - model_name: claude-opus-4-7\n litellm_params:\n model: openai/claude-opus-4-7\n api_base: http://claude-max-bridge:8000\n api_key: \"dummy\"\n stream_timeout: 180\n timeout: 180\n\n - model_name: claude-haiku-4-5\n litellm_params:\n model: openai/claude-haiku-4-5-20251001\n api_base: http://claude-max-bridge:8000\n api_key: \"dummy\"\n stream_timeout: 60\n timeout: 60\n</code></pre> <p>Add to <code>router_settings.fallbacks</code> \u2014 Max hits rate limit \u2192 fall to SAIA: <pre><code>fallbacks:\n - claude-sonnet-4-6: [qwen3.5-397b-a17b]\n - claude-opus-4-7: [qwen3.5-397b-a17b, mistral-large-latest]\n - claude-haiku-4-5: [qwen3.5-122b-a10b, cerebras-llama-3.1-8b]\n</code></pre></p>"},{"location":"ideas/litellm-claude-max-bridge/#rate-limit-handling","title":"Rate limit handling","text":"<p>The bridge should detect the CLI's rate-limit exit code / stderr message and return HTTP 429. LiteLLM's router will then trigger the fallback chain. No custom logic needed in LiteLLM itself.</p> <p>Max limits as of 2026 (approximate, vary by plan tier): - Sonnet 4.6: ~50 messages / 5-hour window at Pro; higher at Max - Opus 4.7: ~10\u201320 messages / 5-hour window - Haiku 4.5: effectively unlimited under Max</p>"},{"location":"ideas/litellm-claude-max-bridge/#volume-estimate","title":"Volume estimate","text":"<p>Typical homelab workloads (LobeHub chat, n8n automations, Coder coding assist) generate maybe 200\u2013500 completions/day. Max's 5-hour windows reset 4\u20135\u00d7 daily, so the practical limit is not usually hit unless Opus is used heavily.</p>"},{"location":"ideas/litellm-claude-max-bridge/#risks","title":"Risks","text":"Risk Mitigation CLI API changes between Claude Code releases Pin the <code>claude</code> binary version; watch for breaking changes in release notes Max rate limit shared with interactive use Monitor via <code>claude usage</code>; bridge adds <code>X-Max-Usage</code> header from CLI output Auth session expiry Bridge returns 401 on auth failure; detect and alert via Gotify Subprocess latency (cold start) Keep a warm subprocess pool (1\u20132 persistent processes) using <code>--interactive</code> + message passing, or accept the 200\u2013400 ms overhead No tool use support Fallback chain routes tool-use requests to API providers (Gemini, Mistral)"},{"location":"ideas/litellm-claude-max-bridge/#implementation-plan","title":"Implementation plan","text":"<ol> <li>Write <code>server.py</code> shim (~150 lines) + Dockerfile</li> <li>Build image on CT 104, add to <code>ai/docker-compose.yml</code></li> <li>Bind-mount <code>~/.claude</code> read-only into the container</li> <li>Add model entries to <code>litellm-config/config.yaml</code></li> <li>Test streaming via <code>curl</code> against the bridge directly</li> <li>Test end-to-end via LiteLLM \u2192 LobeHub</li> <li>Watch <code>docker logs claude-max-bridge</code> for rate-limit / auth errors for 48h</li> </ol>"},{"location":"ideas/litellm-claude-max-bridge/#open-questions","title":"Open questions","text":"<ul> <li>Does <code>claude --print</code> support all message roles (<code>system</code>, <code>user</code>, <code>assistant</code> turns)? Needs verification with multi-turn conversation format.</li> <li>Should the bridge be on CT 104 (access to <code>~/.claude</code>) or CT 109 ops (remote docker socket to CT 104)? CT 104 is simpler for v1.</li> <li>Is there an official Claude Code SDK for Python? (TypeScript SDK exists at <code>@anthropic-ai/claude-code</code>; Python wrapper may be cleaner than shelling out.)</li> </ul>"},{"location":"ideas/stack-ideas/","title":"Stack Ideas & Improvements","text":"<p>Hardware baseline: NUC 14 Pro \u2014 Intel Core Ultra (Meteor Lake), Intel Arc iGPU (currently used for ComfyUI / XPU image gen), no NVIDIA, UNAS for media/bulk storage.</p>"},{"location":"ideas/stack-ideas/#1-document-ingestion-lobechat-knowledge-base-mcp","title":"1. Document ingestion \u2192 LobeChat knowledge base + MCP","text":"<p>Goal: ingest PDFs, Word, PPT, XLS \u2192 searchable in LobeChat chat + accessible via MCP. Qdrant is a nice-to-have, not a requirement (no live pipeline uses it yet).</p> <p>LobeChat already has a built-in knowledge base (<code>knowledge_base_files</code>, <code>chunks</code>, <code>embeddings</code> tables in Postgres). The missing piece is an ingest pipeline that feeds it.</p> <p>Recommended architecture: <pre><code>Nextcloud folder / Paperless webhook\n \u2192 n8n trigger (already running)\n \u2192 Docling (PDF/DOCX/PPTX/XLSX \u2192 Markdown + structure)\n \u2192 LiteLLM /v1/embeddings (Mistral-embed / codestral-embed)\n \u2192 LobeChat knowledge base API \u2190\u2014 available in chat natively\n \u2192 Qdrant sidecar (optional, if multi-app search needed)\n</code></pre></p> <p>Document conversion options:</p> Tool Docker image Formats NUC 14 CPU fit Docling (IBM, 2024) <code>ghcr.io/ds4sd/docling</code> PDF, DOCX, PPTX, XLSX, HTML, Markdown \u2705 CPU-only, 10-60s/doc MinerU (OpenDataLab) <code>opendatalab/mineru</code> PDF (layout-aware, OCR) \u2705 CPU mode; GPU optional for speed Unstructured <code>quay.io/unstructured-io/unstructured-api</code> Very broad (25+ formats) \u2705 lighter than Docling Markitdown (already in MCP gateway) \u2014 Office + PDFs \u2192 Markdown \u2705 ad-hoc only, not batch <p>Docling vs MinerU: Docling is better for structured Office/PDF with tables and figures. MinerU (OpenDataLab) is better for pure PDF optical layout analysis (e.g., academic papers, scanned documents). Both run on NUC 14 CPU. Start with Docling \u2014 single container, REST API, well-documented.</p> <p>MCP access: add a <code>knowledge-search</code> MCP tool to the gateway that calls LobeChat's knowledge base search API. Zero new infra \u2014 LobeChat is already on <code>shared_backend</code>.</p> <p>IBM Granite embedding (granite-embedding-30m-english) \u2014 good quality, tiny (30M params), runs CPU-only. Alternative to Mistral-embed if you want fully local embeddings. Not on LiteLLM yet but can add as a custom provider pointing at an Ollama/IPEX-LLM instance.</p>"},{"location":"ideas/stack-ideas/#2-document-conversion-pipeline","title":"2. Document conversion pipeline","text":"<p>Problem: No systematic ingestion path from raw docs (PDF, DOCX, HTML) into structured text for chunking/embedding.</p> <p>Tools to consider:</p> Tool Docker image Best for Docling (IBM, 2024) <code>ghcr.io/ds4sd/docling</code> PDFs with complex layout (tables, columns, figures); outputs Markdown Unstructured <code>quay.io/unstructured-io/unstructured-api</code> Wide format support (HTML, DOCX, PPTX, images with OCR); REST API Gotenberg <code>gotenberg/gotenberg</code> HTML/Office \u2192 PDF pre-processing stage; not a text extractor Apache Tika <code>apache/tika</code> Broad format support; outputs plain text; lower quality than Docling for PDFs Markitdown MCP already in gateway Ad-hoc on-demand conversion; not suitable for batch pipelines <p>Recommended pipeline (n8n-orchestrated): <pre><code>Nextcloud / Paperless webhook\n \u2192 Gotenberg (Office \u2192 PDF)\n \u2192 Docling (PDF \u2192 Markdown chunks)\n \u2192 LiteLLM /v1/embeddings (Mistral-embed)\n \u2192 Qdrant upsert\n</code></pre></p> <p>Docling runs CPU-only comfortably on the NUC. Unstructured is heavier but has a managed API if the self-hosted version is too slow.</p> <p>Paperless-ngx already OCRs documents \u2014 its full-text content is accessible via the REST API. An n8n workflow polling <code>/api/documents/?added__gt=<last_run></code> can extract and re-embed directly, without re-running OCR.</p>"},{"location":"ideas/stack-ideas/#3-on-device-llm-arc-igpu","title":"3. On-device LLM (Arc iGPU)","text":"<p>The Arc iGPU is currently ComfyUI-only. Small language models can also run on it:</p> <ul> <li>IPEX-LLM (Intel) + Ollama \u2014 Intel provides a patched Ollama build that uses IPEX-LLM for Arc acceleration. Supports Phi-3.5-mini, Gemma-2B, TinyLlama. Useful as a fast/cheap fallback when Groq/Mistral rate limits hit.</li> <li>llama.cpp with Vulkan \u2014 alternative to IPEX-LLM; no Intel-specific driver, uses Vulkan compute. Less optimised for Arc but simpler to deploy.</li> <li>Practical constraint: Arc iGPU shares system RAM; sustained LLM inference competes with other XPU workloads (e.g., ComfyUI). A model serving config that pauses one workload while the other runs is needed, or deploy them on separate CPU/GPU budgets.</li> </ul> <p>If a second NUC or small GPU box becomes available, this becomes the primary use case.</p>"},{"location":"ideas/stack-ideas/#4-arcane-agents-on-nuc-central-instance-on-proxmox-lxc","title":"4. Arcane agents on NUC (central instance on Proxmox LXC)","text":"<p>Arcane's central server moves to a Proxmox LXC (lightweight, stays responsive even if the Docker stack has issues). The NUC and any other Docker host runs the Arcane agent (headless worker), which connects back to the central instance.</p> <p>Deployment: - LXC: central Arcane server + observability stack (see \u00a78) - NUC (<code>.40</code>) and <code>.49</code>: <code>arcane-agent</code> (headless) in compose, connects to LXC - Arcane agents reach Docker via either: - TCP Docker socket + TLS certs \u2014 most secure; requires cert generation per host - SSH Docker contexts \u2014 simpler; gateway mounts socket into agent via SSH tunnel</p> <p>Observability co-located on the LXC (see \u00a78 \u2014 lightweight, OTEL-aware stack that survives Docker restarts).</p>"},{"location":"ideas/stack-ideas/#5-ai-memory-personalization","title":"5. AI memory / personalization","text":"<p>Mem0 (<code>mem0ai/mem0</code>) \u2014 persistent user memory layer that sits in front of LLM calls. Stores facts extracted from conversations into a vector DB (Qdrant backend supported). Can integrate with LobeChat or as an MCP tool. Lets the AI remember preferences, past context, and user-specific facts across sessions.</p> <p>Alternative: Letta (formerly MemGPT) \u2014 stateful agent framework with persistent memory; more opinionated.</p>"},{"location":"ideas/stack-ideas/#6-workflow-automation-upgrades","title":"6. Workflow / automation upgrades","text":"<ul> <li>n8n \u2192 AI agent nodes: n8n 1.x has native AI agent nodes (LangChain under the hood). Can build RAG, email triage, document processing workflows visually.</li> <li>Activepieces \u2014 lighter-weight n8n alternative; good for simple integrations. Probably not worth adding since n8n is already running.</li> <li>Temporal \u2014 durable workflow engine for long-running or retry-heavy pipelines (e.g., large document batch processing). Overkill unless pipelines become complex.</li> </ul>"},{"location":"ideas/stack-ideas/#7-external-services-worth-considering","title":"7. External services worth considering","text":"Service Purpose Notes Backblaze B2 Off-site backup (already in todos) rclone sync from Garage; restic for PG dumps Cloudflare R2 S3-compatible CDN-backed storage Free egress; good for Lobe file serving Resend Transactional email 3K/mo free; better deliverability than self-hosted Postal ntfy.sh (cloud) Push notifications fallback Already running self-hosted ntfy; cloud as relay Cloudflare Turnstile Bot protection for public endpoints Free; no JS challenge Novu Notification orchestration Multi-channel (email, push, Slack); self-hostable"},{"location":"ideas/stack-ideas/#8-observability-on-proxmox-lxc-alongside-arcane","title":"8. Observability (on Proxmox LXC, alongside Arcane)","text":"<p>Runs on the LXC, not the NUC \u2014 stays alive if the Docker stack misbehaves. Lightweight enough for a 2 vCPU / 4GB LXC.</p> <p>Recommended stack (all OTEL-aware, compose-based):</p> Component Image Role OpenTelemetry Collector <code>otel/opentelemetry-collector-contrib</code> Receives traces/metrics/logs from all services (OTLP gRPC+HTTP); fans out to backends VictoriaMetrics <code>victoriametrics/victoria-metrics</code> Prometheus-compatible TSDB; scrapes NUC exporters + receives from OTEL collector; lighter than Prometheus Grafana <code>grafana/grafana</code> Dashboards; datasource = VictoriaMetrics + Loki Loki <code>grafana/loki</code> Log aggregation; receives from OTEL collector Uptime Kuma <code>louislam/uptime-kuma</code> HTTP/TCP uptime checks for all public endpoints; alerts via ntfy <p>OTEL receivers from the NUC Docker stack: - LiteLLM: native OTEL export \u2014 traces every LLM call with token counts, model, latency - n8n: Prometheus <code>/metrics</code> endpoint - Garage: Prometheus <code>/metrics</code> - mcp-gateway: add <code>opentelemetry-sdk</code> instrumentation to <code>server.py</code> - Docker host: <code>node_exporter</code> + <code>cadvisor</code> on the NUC, scraped by VictoriaMetrics</p> <p>NUC \u2192 LXC connectivity: Both are on the same LAN. OTEL collector listens on the LXC's LAN IP (e.g., <code>192.168.1.X:4317</code> gRPC). Services push OTEL directly to it; Prometheus pull-scraping from VictoriaMetrics goes to NUC exporters over LAN.</p>"},{"location":"ideas/stack-ideas/#9-storage-s3-improvements","title":"9. Storage / S3 improvements","text":"<ul> <li>Garage external S3 is currently non-responsive (tracked in todos). Fix or replace with MinIO (single-node, much better tooling/UI, easy migration).</li> <li>Lobe file serving via Cloudflare R2: Lobe uploads to Garage \u2192 sync to R2 \u2192 serve from CDN. Reduces NUC egress for image-heavy sessions.</li> <li>Qdrant data on UNAS: Mount <code>qdrant_data</code> from NAS for persistence across container re-creates (same pattern as arr-stack).</li> </ul>"},{"location":"ideas/stack-ideas/#9-comfyui-mcp-async-queue-image-to-image-reference","title":"9. ComfyUI MCP \u2014 async queue + image-to-image + reference","text":"<p>Current problem: MCP tool blocks until the image is done (~90-290s). LLMs time out at ~60s. The fix is a job-queue pattern:</p> <pre><code>generate_image(prompt) \u2192 returns {job_id, status: \"queued\"} immediately\nget_image_status(job_id) \u2192 returns {status, progress, image_url_when_done}\nlist_queue() \u2192 shows all pending/running jobs\ncancel_job(job_id) \u2192 cancels a queued job (confirms with user first)\n</code></pre> <p>ComfyUI's own API is already async (<code>POST /prompt</code> \u2192 poll <code>/history/{id}</code>). The MCP server just needs to expose this model instead of blocking.</p> <p>Image-to-image: new workflow <code>flux-schnell-img2img-api.json</code>. Takes an init image URL + denoise strength. ComfyUI <code>LoadImageFromURL</code> node (or upload + <code>LoadImage</code>).</p> <p>Reference tool: <code>list_previous_images(n=5)</code> \u2014 queries ComfyUI <code>/history</code> API, returns recent job thumbnails + prompts. User can pick one to reference or iterate from.</p> <p>Delivery to S3/Garage: on completion, upload output PNG to Garage <code>comfyui-outputs</code> bucket \u2192 return a permanent URL. Avoids ComfyUI's ephemeral <code>/view</code> endpoint.</p>"},{"location":"ideas/stack-ideas/#10-litellm-custom-provider-lobechat-provider-extension","title":"10. LiteLLM custom provider / LobeChat provider extension","text":"<p>Question: build an adapter in LiteLLM to use models via \"quasi API\" (non-standard endpoints, auth, or routing)?</p> <p>LiteLLM supports custom providers via <code>custom_llm_provider</code> in config.yaml. You write a Python class that implements <code>completion()</code> and <code>async_completion()</code>. This is the right path for wrapping non-standard APIs (local models, proprietary endpoints, protocol bridges).</p> <p>Example use cases: - Wrap a Claude Max subscription via Anthropic's API (different billing model) - Add a local model served by IPEX-LLM on the Arc iGPU - Bridge a custom inference server that speaks a different protocol</p> <p>LobeChat provider extension: LobeChat's provider list is compiled into the app. Adding a new provider requires rebuilding LobeChat from source (fork + add to <code>src/config/aiModels/</code>). High effort; only worth it for a permanent/long-term provider. For ad-hoc needs, use LiteLLM as the adapter and point LobeChat at it via the existing <code>ai.nuclide.systems</code> OpenAI-compatible endpoint.</p>"},{"location":"ideas/stack-ideas/#11-dependency-updates-via-renovate-bot-gitea-actions","title":"11. Dependency updates via Renovate Bot (Gitea Actions)","text":"<p>Run Renovate Bot as a Gitea Actions workflow to automatically open PRs for outdated dependencies in MCP server repos (<code>mcp-comfyui</code>, <code>mcp-docling</code>, <code>mcp-shepard</code>, <code>mcp-upload-artifact</code>). Targets: <code>Dockerfile</code> base image tags + <code>requirements.txt</code> / <code>pyproject.toml</code> Python deps.</p> <p>Renovate supports Gitea natively via <code>platform: gitea</code> in <code>renovate.json</code>. The Gitea Actions runner (<code>ct111-runner</code>) already exists; add a scheduled workflow calling <code>renovate/renovate</code> Docker image once daily.</p>"},{"location":"ideas/stack-ideas/#12-egress-firewall-on-udm-unifi","title":"12. Egress firewall on UDM (UniFi)","text":"<p>Enforce outbound allow-list on the UDM / UniFi gateway \u2014 block all non-approved egress by default. Goals: - Prevent exfiltration from compromised containers - Audit unexpected outbound connections (model providers, analytics, telemetry) - Approved: LiteLLM model provider endpoints, Jottacloud, UNAS internal, NTP, DNS</p> <p>Implementation: UDM firewall rules (WAN_OUT) + Threat Management IDS in monitor mode first.</p>"},{"location":"ideas/stack-ideas/#13-local-llm-on-arc-gpu-via-vllm-ollama-for-sensitive-workloads","title":"13. Local LLM on Arc GPU via vllm / ollama for sensitive workloads","text":"<p>Run a privacy-sensitive LLM locally on the Arc iGPU using vllm (with XPU/IPEX backend) or the Intel-patched Ollama build. Use cases: document classification in Paperless workflows, offline coding assistant, fallback when cloud rate limits hit.</p> <p>See \u00a73 (On-device LLM) for implementation notes. This item tracks the specific motivation of sensitive workload isolation \u2014 i.e., running prompts that should not leave the LAN.</p>"},{"location":"ideas/stack-ideas/#priority-order-rough","title":"Priority order (rough)","text":"<ol> <li>ComfyUI MCP async queue \u2014 fixes timeout; unblocks img2img + reference features</li> <li>Docling container + n8n ingest pipeline \u2014 Nextcloud/Paperless \u2192 LobeChat knowledge base</li> <li>Arcane LXC + agents on NUC \u2014 when ready to migrate</li> <li>Observability LXC \u2014 OTEL collector + VictoriaMetrics + Grafana + Uptime Kuma, co-located</li> <li>Backblaze B2 off-site backup \u2014 already in todos</li> <li>On-device LLM (Arc) \u2014 depends on IPEX-LLM + Ollama Intel build stability</li> <li>Renovate Bot \u2014 low-effort automation win for MCP repos</li> <li>Egress firewall \u2014 security hygiene; plan before adding more external-facing services</li> </ol>"},{"location":"infra/config-to-git/","title":"Config-to-git fleet","text":"<p>Each host snapshots its critical config to a private Gitea repo on <code>git.nuclide.systems</code> daily. Force-push (mirror only \u2014 history isn't sacred). Token: long-lived <code>fkrebs</code> PAT embedded in remote URLs (mode 0600 on script/config).</p>"},{"location":"infra/config-to-git/#repos","title":"Repos","text":"Host CT/VM Repo Schedule Script Source PVE (<code>nuc</code>) host fkrebs/pve-conf daily 03:00 <code>/usr/local/sbin/pve-conf-backup.sh</code> (cron <code>/etc/cron.d/pve-conf-backup</code>) <code>/etc/pve/</code> (excludes <code>priv/</code>, <code>*.key</code>, <code>authkey.pub*</code>) Zoraxy CT 108 fkrebs/zoraxy-conf daily 03:00 <code>/usr/local/sbin/zoraxy-conf-backup.sh</code> (cron <code>/etc/cron.d/zoraxy-conf-backup</code>) Zoraxy config dir AdGuard CT 102 fkrebs/adguard-conf daily 03:00 <code>/usr/local/sbin/adguard-conf-backup.sh</code> (cron <code>/etc/cron.d/adguard-conf-backup</code>) AdGuard config dir Home Assistant VM 100 fkrebs/home-assistant-config manual cron via addon <code>init_commands</code> (see below) <code>/config/scripts/git-push.sh</code> <code>/config/</code> (sees <code>.gitignore</code> allowlist) Backrest CT 103 fkrebs/ct103-conf daily 03:00 <code>/usr/local/sbin/ct103-conf-backup.sh</code> <code>/opt/backrest/config/</code>, <code>/etc/cron.d/</code>, <code>/usr/local/bin/</code> Ops stack CT 109 fkrebs/ct109-conf daily 03:00 <code>/usr/local/sbin/ct109-conf-backup.sh</code> <code>/opt/stacks/</code> (excludes <code>*/data/</code>, <code>*.db*</code>), <code>/etc/cron.d/</code> Postgres / WAL-G CT 113 fkrebs/ct113-conf daily 03:00 <code>/usr/local/sbin/ct113-conf-backup.sh</code> <code>/opt/stacks/</code> (excludes <code>*/data/</code>), <code>/etc/postgresql/</code>, <code>/etc/cron.d/</code>, <code>/usr/local/bin/</code> Obsidian vault UNAS via CT 103 fkrebs/obsidian-vault daily 04:00 <code>/usr/local/sbin/obsidian-vault-backup.sh</code> <code>Notizen/</code> (excludes sync indices, <code>.obsidian/</code>, <code>.trash</code>). Rolling 2-commit history. Primary Docker host CT 104 fkrebs/ct104-conf daily 03:00 <code>/usr/local/sbin/ct104-conf-backup.sh</code> (cron <code>/etc/cron.d/ct104-conf-backup</code>) <code>/opt/stacks/</code> \u2014 <code>*.yml</code>, <code>*.yaml</code>, <code>*.json</code>, <code>*.conf</code>, <code>*.sh</code>, <code>*.md</code> only; excludes <code>*/data/</code>, <code>.env</code>, <code>*.db*</code>, <code>*.key</code>, <code>*.pem</code> Obsidian config manual zip drop fkrebs/obsidian-config manual (or via Obsidian Git plugin) <code>/tmp/obsidian-config-init.sh</code> (one-shot) <code>.obsidian/</code> minus <code>workspace*.json</code>, <code>cache/</code>, <code>*.bak*</code> <p><code>fkrebs/ha-config</code> was created in error 2026-05-24 \u2014 deleted.</p>"},{"location":"infra/config-to-git/#pattern","title":"Pattern","text":"<p>All scripts follow the same shape:</p> <pre><code>#!/bin/bash\nset -e\nREPO_URL=\"https://fkrebs:<TOKEN>@git.nuclide.systems/fkrebs/<repo>.git\"\nWORK=\"/tmp/<name>-work\"\nmkdir -p \"$WORK\"\ngit -C \"$WORK\" init -b main -q 2>/dev/null || true\ngit -C \"$WORK\" config user.email \"noreply@nuclide.systems\"\ngit -C \"$WORK\" config user.name \"<name>-backup\"\ngit -C \"$WORK\" remote set-url origin \"$REPO_URL\" 2>/dev/null \\\n || git -C \"$WORK\" remote add origin \"$REPO_URL\"\nrsync -a --delete --exclude='.git' [+ secret excludes] <source>/ \"$WORK/\"\ngit -C \"$WORK\" add -A\ngit -C \"$WORK\" commit -q -m \"auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)\" 2>/dev/null || true\ngit -C \"$WORK\" push -q --force origin main\n</code></pre> <p>PVE/CT 108/CT 102 use a rsync-to-tmp-then-push pattern. HAOS uses an in-place <code>git add -A</code> on <code>/config</code> because the <code>.gitignore</code> there is hand-curated with an allowlist for <code>.storage/</code>.</p>"},{"location":"infra/config-to-git/#haos-specifics","title":"HAOS specifics","text":"<ul> <li>Path: <code>/config/scripts/git-push.sh</code> (mode 0700, runs in the Terminal & SSH addon).</li> <li>Remote: <code>https://fkrebs:<TOKEN>@git.nuclide.systems/fkrebs/home-assistant-config.git</code>. Token lives in <code>/config/.git/config</code> (mode 0600).</li> <li><code>.gitignore</code> uses a default-deny allowlist for <code>.storage/</code> \u2014 only safe registry/lovelace/helpers/energy files are tracked. Tokens (<code>core.config_entries</code>, <code>mobile_app</code>, <code>androidtv_adbkey*</code>, etc.) are explicitly excluded. See <code>/config/.gitignore</code> on HAOS for the full list.</li> <li>Scheduling \u2014 Terminal & SSH addon container is rebuilt on update, so its <code>crontabs/</code> are not persistent. Use the addon's <code>init_commands</code> (Configuration tab):</li> </ul> <pre><code>init_commands:\n - 'echo \"0 3 * * * /config/scripts/git-push.sh >> /config/scripts/git-push.log 2>&1\" > /etc/crontabs/root && crond -b'\n</code></pre> <p>Then Restart the addon. Alternative: HA automation calling <code>shell_command</code> is not viable \u2014 the <code>homeassistant</code> container doesn't have SSH to addon containers.</p>"},{"location":"infra/config-to-git/#token-rotation","title":"Token rotation","text":"<p>All four repos use the same PAT (<code>fkrebs</code> user, full repo scope). Rotate by:</p> <ol> <li>Generate new PAT in Gitea \u2192 user settings \u2192 applications.</li> <li><code>git remote set-url origin https://fkrebs:<NEW>@git.nuclide.systems/fkrebs/<repo>.git</code> on each host.</li> <li>Manually run each script once to verify.</li> </ol> <p>Token leak risk: tracked in [[gitea_open_issues]]; long-term move is to switch to per-host deploy keys.</p>"},{"location":"infra/config-to-git/#verification","title":"Verification","text":"<p>Last commit on each repo should be <code>auto: <today>T03:0X:XXZ</code>. Quick check:</p> <pre><code>for r in pve-conf zoraxy-conf adguard-conf home-assistant-config; do\n echo -n \"$r: \"\n curl -sk -H \"Authorization: token <TOKEN>\" \\\n \"https://git.nuclide.systems/api/v1/repos/fkrebs/$r/commits?limit=1\" \\\n | python3 -c 'import json,sys; c=json.load(sys.stdin)[0]; print(c[\"commit\"][\"author\"][\"date\"], c[\"commit\"][\"message\"][:60])'\ndone\n</code></pre>"},{"location":"infra/config-to-git/#related","title":"Related","text":"<ul> <li>[[homelab-architecture]] \u2014 host topology</li> <li>[[backrest-ct103]] \u2014 separate, repo-style binary backup for data (not config)</li> </ul>"},{"location":"infra/connection-hosts/","title":"Connection Hosts \u2014 nuclide.systems","text":"<p>Maintained reference for every host, LAN address, port, and public URL. Last verified: 2026-05-23.</p> <p>Source of truth: <code>ct-inventory.md</code> (guests) \u00b7 <code>portmap.md</code> (Docker ports) \u00b7 <code>homelab-architecture.md</code> (topology).</p>"},{"location":"infra/connection-hosts/#network-infrastructure","title":"Network infrastructure","text":"Device LAN IP Admin UI Notes UDM Home (UCG Fiber, UniFi OS 5.0.16) <code>192.168.1.1</code> <code>https://192.168.1.1</code> (SSO + MFA) Gateway, DNS forwarder \u2192 AdGuard; port-forwards 80/443 \u2192 Zoraxy (<code>.4</code>), 15001 TCP/UDP \u2192 CT 104 D-Link DGS-1210-28P <code>192.168.1.10</code> <code>http://192.168.1.10</code> (HTTP-only, pw: <code>tapirnase</code>) 28-port PoE switch \u2014 physical core. UDM on port 26, CT 104 cluster on port 10, APs on ports 3 & 16. SNTP fixed 2026-05-19. UNAS Pro (NFS server) <code>192.168.1.31</code> <code>http://192.168.1.31</code> NFSv3 export: <code>192.168.1.31:/var/nfs/shared/storage</code> (\u2192 <code>/mnt/pve/unas</code>). ~19 T bulk storage. UniFi U7 APs <code>.50</code> <code>.51</code> <code>.52</code> <code>.53</code> via UDM Hallway, In-wall, Bedroom (Schlafzimmer), Dining (Esszimmer). SSID: <code>nuclide</code>, WPA2/WPA3. TP-Link RE700X <code>192.168.1.187</code> <code>http://192.168.1.187</code> WiFi extender \u2014 NATs devices behind it (3D printer <code>.189</code> invisible to UniFi)."},{"location":"infra/connection-hosts/#proxmox-ve-host-nuc","title":"Proxmox VE host \u2014 <code>nuc</code>","text":"Value IP <code>192.168.1.20</code> Admin UI <code>https://192.168.1.20:8006</code> (OIDC via Pocket-ID client <code>38469e7e</code>) SSH <code>ssh root@192.168.1.20</code> (key auth) Hardware Intel Core Ultra 7 155H \u00b7 22 threads \u00b7 64 GiB RAM \u00b7 PVE 9.1.11 Storage <code>local-zfs</code> ~1.9 T (NVMe), <code>local</code> (dir), <code>unas</code> (NFS ~19 T) API token <code>root@pam!mcp</code> (PVEAuditor role, read-only)"},{"location":"infra/connection-hosts/#lxc-vm-guests","title":"LXC / VM guests","text":""},{"location":"infra/connection-hosts/#vm-100-haos-home-assistant-os","title":"VM 100 \u2014 <code>haos</code> (Home Assistant OS)","text":"Value IP <code>192.168.1.60</code> (DHCP, stable) HA UI <code>https://ha.nuclide.systems</code> \u2192 <code>192.168.1.60:8123</code> SSH <code>ssh root@192.168.1.60</code> (key installed manually in HA terminal) OCPP <code>https://ocpp.nuclide.systems</code> \u2192 <code>192.168.1.60:8887</code> HA-MCP add-on <code>http://192.168.1.60:9583/private_ehnWeRl2G3De6NnbcN7teQ</code> (gateway upstream, no TLS) Specs 4c / 16 GiB (balloon 4 GiB) / 32 GiB; USB Zigbee dongle passed through"},{"location":"infra/connection-hosts/#ct-101-shepard","title":"CT 101 \u2014 <code>shepard</code>","text":"Value IP <code>192.168.1.49</code> SSH <code>ssh root@192.168.1.49</code> Public <code>https://shepard.nuclide.systems</code> (Caddy \u2192 <code>:80</code>) \u00b7 <code>https://shepard-api.nuclide.systems</code> (\u2192 <code>:8080</code>) MCP <code>https://shepard.nuclide.systems/v2/mcp</code> (Bearer <code>${SHEPARD_API_KEY}</code>) Specs 12c / 32 GiB / 500 GiB + NFS; Intel iGPU (card+render) Stack Caddy, Shepard frontend/backend, Keycloak, Mongo, Neo4j, TimescaleDB"},{"location":"infra/connection-hosts/#ct-102-dns-adguard-home","title":"CT 102 \u2014 <code>dns</code> (AdGuard Home)","text":"Value IP <code>192.168.1.2</code> SSH <code>ssh root@192.168.1.2</code> Admin UI <code>http://192.168.1.2</code> (port 80) DNS <code>192.168.1.2:53</code> \u2014 LAN resolver (UDM forwards all DNS here) Specs 2c / 1 GiB / 4 GiB"},{"location":"infra/connection-hosts/#ct-103-backrest","title":"CT 103 \u2014 <code>backrest</code>","text":"Value IP <code>192.168.1.3</code> SSH <code>ssh root@192.168.1.3</code> UI <code>http://192.168.1.3:9898</code> (LAN-only, no auth) Specs 1c / 512 MiB / 8 GiB + NFS (<code>/mnt/pve/unas</code>) Notes JottaCloud offsite via rclone; repos <code>services-repo</code> + <code>media-repo</code>"},{"location":"infra/connection-hosts/#ct-104-docker-main-docker-host","title":"CT 104 \u2014 <code>docker</code> (main Docker host)","text":"Value IP <code>192.168.1.40</code> SSH <code>ssh root@192.168.1.40</code> Specs 16c / 48 GiB / 200 GiB + NFS; Intel Arc iGPU (card+render) Role ~65 containers across ~23 compose stacks in <code>/opt/stacks/</code> <p>See Docker services table below for all ports.</p>"},{"location":"infra/connection-hosts/#ct-105-nextcloud","title":"CT 105 \u2014 <code>nextcloud</code>","text":"Value IP <code>192.168.1.41</code> SSH <code>ssh root@192.168.1.41</code> Public <code>https://nc.nuclide.systems</code> \u2192 <code>192.168.1.41:11000</code> Specs 4c / 8 GiB / 100 GiB + NFS (migrated to NFSv3 2026-05-22) Auth Pocket-ID OIDC (client <code>a14b8076</code>)"},{"location":"infra/connection-hosts/#ct-108-zoraxy-reverse-proxy","title":"CT 108 \u2014 <code>zoraxy</code> (reverse proxy)","text":"Value IP <code>192.168.1.4</code> SSH <code>ssh root@192.168.1.4</code> Admin UI <code>http://192.168.1.4:8000</code> (LAN only) Specs 2c / 2 GiB / 6 GiB Cert Wildcard <code>*.nuclide.systems</code> (ACME via Let's Encrypt) Config <code>proxy/zoraxy/routes.json</code> \u2192 <code>scripts/zoraxy_sync.py --apply</code>"},{"location":"infra/connection-hosts/#ct-110-id-pocket-id-oidc","title":"CT 110 \u2014 <code>id</code> (Pocket-ID OIDC)","text":"Value IP <code>192.168.1.5</code> SSH <code>ssh root@192.168.1.5</code> Public <code>https://id.nuclide.systems</code> \u2192 <code>192.168.1.5:11000</code> Specs 1c / 1 GiB / 4 GiB OIDC endpoints Authorization: <code>https://id.nuclide.systems/authorize</code> \u00b7 Token: <code>https://id.nuclide.systems/api/oidc/token</code> \u00b7 Userinfo: <code>https://id.nuclide.systems/api/oidc/userinfo</code> \u00b7 Discovery: <code>https://id.nuclide.systems/.well-known/openid-configuration</code>"},{"location":"infra/connection-hosts/#ct-111-dev-coder-gitea","title":"CT 111 \u2014 <code>dev</code> (Coder + Gitea)","text":"Value IP <code>192.168.1.42</code> SSH <code>ssh root@192.168.1.42</code> Specs 12c / 32 GiB / 60 GiB + NFS; Intel Arc iGPU (render) Port Service Public URL 7080 Coder <code>https://dev.nuclide.systems</code> 3000 Gitea <code>https://git.nuclide.systems</code> 222 Gitea SSH <code>ssh -p 222 git@git.nuclide.systems</code> 13080 docs site <code>http://192.168.1.42:13080</code> (LAN; mkdocs Material, auto-rebuild every 5 min) internal act-runner \u2014 (<code>ct111-runner</code> Gitea Actions)"},{"location":"infra/connection-hosts/#ct-112-secrets-infisical","title":"CT 112 \u2014 <code>secrets</code> (Infisical)","text":"Value IP <code>192.168.1.7</code> SSH <code>ssh root@192.168.1.7</code> UI + API <code>http://192.168.1.7:8200</code> (LAN-only \u2014 no Zoraxy route; must not be internet-exposed) Specs 2c / 4 GiB / 20 GiB Stack <code>/opt/stacks/infisical/</code> \u2014 Infisical + Postgres 16 + Redis 7 (all internal, no external ports)"},{"location":"infra/connection-hosts/#ct-113-db-shared-postgres","title":"CT 113 \u2014 <code>db</code> (shared Postgres)","text":"Value IP <code>192.168.1.6</code> SSH <code>ssh root@192.168.1.6</code> Specs 2c / 4 GiB / 40 GiB Port Service Access 5432 Postgres 17 LAN: <code>192.168.1.6:5432</code> \u2014 tenants: LiteLLM, paperless, memos, n8n (reverted), Vaultwarden 5050 pgAdmin 4 <code>http://192.168.1.6:5050</code> (LAN only, no Zoraxy route)"},{"location":"infra/connection-hosts/#docker-services-ct-104","title":"Docker services \u2014 CT 104","text":"<p>All on <code>192.168.1.40</code> unless noted. Zoraxy (<code>192.168.1.4</code>) terminates TLS for public URLs.</p>"},{"location":"infra/connection-hosts/#infrastructure-1000010999","title":"Infrastructure (10000\u201310999)","text":"Port Container Public URL Notes 10000 <code>homepage</code> \u2014 Dashboard (LAN only) 10001 <code>dozzle</code> <code>https://dozzle.nuclide.systems</code> Log viewer 10002 <code>arcane</code> <code>https://arcane.nuclide.systems</code> Web IDE (OIDC <code>81cf4ed0</code>) 10003 <code>gotify</code> <code>https://gotify.nuclide.systems</code> Push notifications 10004 <code>garage</code> <code>https://s3.nuclide.systems</code> S3 API (Garage); internal: <code>http://garage:3900</code> 10005 (localhost) <code>garage</code> admin \u2014 Localhost only"},{"location":"infra/connection-hosts/#security-auth-1100011999","title":"Security & Auth (11000\u201311999)","text":"Port Container Public URL Notes 11001 <code>vaultwarden</code> <code>https://vault.nuclide.systems</code> Password manager (OIDC client created, SSO not yet wired)"},{"location":"infra/connection-hosts/#media-immich-1200012999","title":"Media \u2014 Immich (12000\u201312999)","text":"Port Container Public URL Notes 12000 <code>immich_server</code> <code>https://immich.nuclide.systems</code> Photos/videos (OIDC <code>9c91c18b</code>) internal <code>immich_power_tools</code> <code>https://immich-tools.nuclide.systems</code> Container-internal :3000, Zoraxy proxy"},{"location":"infra/connection-hosts/#media-downloads-arr-1300013999","title":"Media \u2014 Downloads / Arr (13000\u201313999)","text":"<p>All behind <code>vpn_gluetun</code> container network.</p> Port Container Public URL Notes 13001 <code>rdtclient</code> \u2014 LAN only 13002 <code>prowlarr</code> \u2014 LAN only 13003 <code>audiobookshelf</code> <code>https://abs.nuclide.systems</code> OIDC <code>cbbf20d5</code> 13004 <code>shelfarr</code> \u2014 LAN only (OIDC <code>d8733fcc</code>) 13005 <code>flaresolverr</code> \u2014 Internal only 30000 <code>vpn_gluetun</code> \u2014 VPN HTTP control API"},{"location":"infra/connection-hosts/#ai-stack-1400014999-8080-18xxx","title":"AI Stack (14000\u201314999, 8080, 18xxx)","text":"Port Container Public URL Notes 14000 <code>litellm</code> \u2014 LLM proxy (internal only: <code>http://litellm:4000</code>); master key: <code>sk-tapirnase</code> 14001 <code>lobehub</code> <code>https://chat.nuclide.systems</code> Chat UI (OIDC <code>26f3c26b</code>) 14002 <code>qdrant_scientific</code> \u2014 Vector DB (LAN/internal only) 14003 <code>bifrost</code> <code>https://ai.nuclide.systems</code> LLM+MCP gateway; <code>/mcp</code> = MCP endpoint (29 clients, ~760 tools); master key: <code>sk-tapirnase</code> ~~8080~~ ~~<code>mcp-gateway</code>~~ ~~<code>https://mcp.nuclide.systems</code>~~ DECOMMISSIONED 2026-05-26 \u2014 see mcp-gateway.md 18002 <code>comfyui</code> \u2014 Image gen (Intel Arc); LAN only 18003 <code>comfyui-mcp</code> via Bifrost <code>comfyui</code> client FastMCP 18005 <code>docling-mcp</code> via Bifrost <code>docling</code> client PDF\u2192Markdown 18007 <code>kroki-mcp</code> via Bifrost <code>kroki</code> client Diagram rendering 18009 <code>speaches</code> \u2014 TTS/STT; LAN only 18011 <code>upload-artifact-mcp</code> via Bifrost <code>upload_artifact</code> client S3 artifact upload internal <code>searxng</code> \u2014 <code>shared_backend</code>; used by LobeChat"},{"location":"infra/connection-hosts/#documents-1500015999","title":"Documents (15000\u201315999)","text":"Port Container Public URL Notes 15000 <code>traccar</code> <code>https://traccar.nuclide.systems</code> GPS tracking UI 15001 <code>traccar</code> \u2014 TCP+UDP watch protocol (forwarded at UDM level) 15002 <code>paperless-ai</code> <code>https://paperless-ai.nuclide.systems</code> Internal-only Zoraxy policy 15003 <code>paperless-ngx-webserver-1</code> <code>https://paperless.nuclide.systems</code> Internal-only Zoraxy policy"},{"location":"infra/connection-hosts/#automation-1600016999","title":"Automation (16000\u201316999)","text":"Port Container Public URL Notes 16000 <code>n8n</code> <code>https://n8n.nuclide.systems</code> Workflow automation (OIDC <code>33135ad4</code>); SQLite local disk"},{"location":"infra/connection-hosts/#notes-bookmarks-1700017999","title":"Notes & Bookmarks (17000\u201317999)","text":"Port Container Public URL Notes 17000 <code>memos</code> <code>https://memos.nuclide.systems</code> Notes (OIDC <code>62bf4e0d</code>) 17001 <code>karakeep</code> <code>https://hoarder.nuclide.systems</code> Bookmarks (OIDC <code>d92f82b0</code>)"},{"location":"infra/connection-hosts/#storage-admin-2000020999","title":"Storage Admin (20000\u201320999)","text":"Port Container Notes 20010 (localhost) <code>pgadmin</code> shared-db pgAdmin, localhost only on CT 104"},{"location":"infra/connection-hosts/#external-hardware","title":"External hardware","text":""},{"location":"infra/connection-hosts/#qnap-unas-pro-nfssmb-server","title":"QNAP UNAS Pro \u2014 NFS/SMB server","text":"Value IP <code>192.168.1.31</code> Admin UI <code>http://192.168.1.31</code> NFS export <code>192.168.1.31:/var/nfs/shared/storage</code> (NFSv3 only; v4 not available) CIFS <code>//192.168.1.31/storage</code> (used by Nextcloud CT 105 only)"},{"location":"infra/connection-hosts/#qnap-ts-251d-klipper-3d-printer-host","title":"QNAP TS-251D \u2014 Klipper 3D printer host","text":"Value IP <code>192.168.1.189</code> (behind TP-Link RE700X at <code>.187</code>) Mainsail <code>http://192.168.1.189:80</code> Moonraker <code>http://192.168.1.189:7125</code> Notes Klipper config backed up daily to <code>git.nuclide.systems/fkrebs/klipper-config</code> via cron"},{"location":"infra/connection-hosts/#oidc-clients-pocket-id-idnuclidesystems","title":"OIDC clients \u2014 Pocket-ID (<code>id.nuclide.systems</code>)","text":"Client ID Service Redirect URI ~~<code>e73bb7b9</code>~~ ~~litellm / mcp-gateway~~ DELETED 2026-05-26 <code>26f3c26b</code> lobehub <code>https://chat.nuclide.systems/api/auth/callback/generic-oidc</code> <code>81cf4ed0</code> arcane <code>https://arcane.nuclide.systems/auth/oidc/callback</code> <code>33135ad4</code> n8n <code>https://n8n.nuclide.systems/auth/oidc/callback</code> <code>9c91c18b</code> immich <code>https://immich.nuclide.systems/auth/login</code> + mobile <code>62bf4e0d</code> memos <code>https://memos.nuclide.systems/auth/callback</code> <code>d92f82b0</code> karakeep <code>https://hoarder.nuclide.systems/api/auth/callback/custom</code> <code>a14b8076</code> nextcloud <code>https://nc.nuclide.systems/apps/user_oidc/code</code> <code>0aee4280</code> Coder <code>https://dev.nuclide.systems/api/v2/users/oidc/callback</code> <code>9444609e</code> Gitea <code>https://git.nuclide.systems/user/oauth2/pocket-id/callback</code> <code>38469e7e</code> Proxmox VE <code>https://192.168.1.20:8006</code> <code>cbbf20d5</code> Audiobookshelf <code>https://abs.nuclide.systems/\u2026</code> + <code>*</code> <code>d8733fcc</code> shelfarr <code>*</code> ~~<code>78c78998</code>~~ ~~Claude MCP (legacy)~~ DELETED 2026-05-26 <code>82ca2d53</code> nuc-ai (MCP spawned servers) (empty) <code>7fe1a14b</code> zoraxy (empty) (new) vaultwarden <code>https://vault.nuclide.systems/auth/callback</code> \u2014 SSO not yet wired ~~798a367f~~ ~~daytona~~ DECOMMISSIONED \u2014 remove from Pocket-ID"},{"location":"infra/connection-hosts/#quick-reference-all-public-urls","title":"Quick-reference \u2014 all public URLs","text":"URL Backend Service <code>https://ai.nuclide.systems</code> <code>192.168.1.40:14003</code> Bifrost (LLM+MCP gateway) \u2014 <code>/mcp</code> for MCP tools <code>https://chat.nuclide.systems</code> <code>192.168.1.40:14001</code> LobeChat ~~<code>https://mcp.nuclide.systems</code>~~ ~~<code>192.168.1.40:8080</code>~~ DECOMMISSIONED 2026-05-26 <code>https://id.nuclide.systems</code> <code>192.168.1.5:11000</code> Pocket-ID (OIDC) <code>https://nc.nuclide.systems</code> <code>192.168.1.41:11000</code> Nextcloud <code>https://ha.nuclide.systems</code> <code>192.168.1.60:8123</code> Home Assistant <code>https://ocpp.nuclide.systems</code> <code>192.168.1.60:8887</code> EV charger OCPP <code>https://shepard.nuclide.systems</code> <code>192.168.1.49:80</code> Shepard <code>https://shepard-api.nuclide.systems</code> <code>192.168.1.49:8080</code> Shepard API <code>https://dev.nuclide.systems</code> <code>192.168.1.42:7080</code> Coder <code>https://git.nuclide.systems</code> <code>192.168.1.42:3000</code> Gitea <code>https://arcane.nuclide.systems</code> <code>192.168.1.40:10002</code> Arcane <code>https://dozzle.nuclide.systems</code> <code>192.168.1.40:10001</code> Dozzle (logs) <code>https://gotify.nuclide.systems</code> <code>192.168.1.40:10003</code> Gotify <code>https://s3.nuclide.systems</code> <code>192.168.1.40:10004</code> Garage S3 <code>https://vault.nuclide.systems</code> <code>192.168.1.40:11001</code> Vaultwarden <code>https://immich.nuclide.systems</code> <code>192.168.1.40:12000</code> Immich <code>https://immich-tools.nuclide.systems</code> <code>192.168.1.40</code> (internal) Immich Power Tools <code>https://abs.nuclide.systems</code> <code>192.168.1.40:13003</code> Audiobookshelf <code>https://n8n.nuclide.systems</code> <code>192.168.1.40:16000</code> n8n <code>https://memos.nuclide.systems</code> <code>192.168.1.40:17000</code> Memos <code>https://hoarder.nuclide.systems</code> <code>192.168.1.40:17001</code> Karakeep <code>https://traccar.nuclide.systems</code> <code>192.168.1.40:15000</code> Traccar <code>https://paperless.nuclide.systems</code> <code>192.168.1.40:15003</code> Paperless-ngx (internal-only policy) <code>https://paperless-ai.nuclide.systems</code> <code>192.168.1.40:15002</code> Paperless AI (internal-only policy) <p>Not publicly proxied (LAN/localhost only): pgAdmin, ComfyUI, Qdrant, RDTClient, Prowlarr, ShelfArr, Flaresolverr, Speaches, AdGuard admin, Backrest UI, Infisical.</p>"},{"location":"infra/connection-hosts/#ssh-cheat-sheet","title":"SSH cheat-sheet","text":"<pre><code>ssh root@192.168.1.20 # PVE host (nuc)\nssh root@192.168.1.40 # CT 104 docker\nssh root@192.168.1.41 # CT 105 nextcloud\nssh root@192.168.1.42 # CT 111 dev\nssh root@192.168.1.49 # CT 101 shepard\nssh root@192.168.1.2 # CT 102 dns (AdGuard)\nssh root@192.168.1.3 # CT 103 backrest\nssh root@192.168.1.4 # CT 108 zoraxy\nssh root@192.168.1.5 # CT 110 id (Pocket-ID)\nssh root@192.168.1.6 # CT 113 db (Postgres)\nssh root@192.168.1.7 # CT 112 secrets (Infisical)\nssh root@192.168.1.60 # VM 100 haos (Home Assistant \u2014 key must be installed manually)\nssh -p 222 git@git.nuclide.systems # Gitea SSH\n</code></pre> <p>All LXCs reachable from <code>nuc</code> host via root key. Use <code>pct exec <id> -- bash</code> for console access without SSH.</p>"},{"location":"infra/docker-networks/","title":"Docker Networks","text":""},{"location":"infra/docker-networks/#current-landscape-may-16-2026","title":"Current Landscape (May 16, 2026)","text":"<p>Each Docker Compose stack creates its own <code>{stack}_default</code> bridge network when it has no explicit <code>networks:</code> declaration. This has exhausted Docker's built-in 172.x.x.x/16 address pool, triggering CIDR overlap errors.</p>"},{"location":"infra/docker-networks/#networks-subnets","title":"Networks & Subnets","text":"Network Subnet Containers <code>bridge</code> (built-in) 10.0.0.0/24 0 <code>ai-internal</code> 172.31.0.0/16 8 <code>arcane_default</code> 172.22.0.0/16 1 <code>arr-stack_default</code> 172.21.0.0/16 4 <code>dozzle_default</code> 192.168.16.0/20 1 <code>homepage_default</code> 172.19.0.0/16 1 <code>immich_default</code> 172.18.0.0/16 5 <code>karakeep_default</code> 172.30.0.0/16 3 <code>memos_default</code> 192.168.32.0/20 1 <code>n8n_default</code> 172.25.0.0/16 1 <code>ntfy_default</code> 172.27.0.0/16 1 <code>nuc-ai-core_default</code> 172.29.0.0/16 1 <code>paperless-ngx_default</code> 172.28.0.0/16 5 <code>pocketid_default</code> 172.20.0.0/16 1 <code>qdrant_default</code> 172.24.0.0/16 1 <code>traccar_default</code> 172.26.0.0/16 1 <code>vaultwarden_default</code> 192.168.64.0/20 1 <code>vpn_default</code> 192.168.80.0/20 1 <p>Total: 18 user-defined bridge networks (Daytona decommissioned 2026-05-20).</p>"},{"location":"infra/docker-networks/#problem","title":"Problem","text":"<p>Docker's default address pool for user-defined bridge networks is 172.17.0.0/16 \u2013 172.31.0.0/16 (15 subnets max). With 15 172.x.x.x/16 networks already allocated, there is no room for new ones.</p> <p>The Daytona runner (<code>daytona-minimal-runner-1</code>) programmatically creates a <code>runner-bridge</code> network on startup. It fails with:</p> <pre><code>Error response from daemon: invalid pool request: Pool overlaps with other one\non this address space\n</code></pre>"},{"location":"infra/docker-networks/#consolidation-plan","title":"Consolidation Plan","text":""},{"location":"infra/docker-networks/#shared_backend-network","title":"shared_backend Network","text":"<p>A single shared bridge network (<code>shared_backend</code>) has been created to replace per-stack defaults for lightweight services that don't need isolation.</p>"},{"location":"infra/docker-networks/#stacks-already-migrated","title":"Stacks Already Migrated","text":"<ul> <li>arcane</li> <li>dozzle</li> <li>ntfy</li> <li>qdrant</li> <li>traccar</li> <li>vaultwarden</li> <li>memos</li> <li>n8n</li> <li>pocketid</li> </ul> <p>These stacks now declare:</p> <pre><code>networks:\n default:\n external: true\n name: shared_backend\n</code></pre>"},{"location":"infra/docker-networks/#how-to-free-subnets","title":"How to Free Subnets","text":"<p>After migrating a stack to <code>shared_backend</code>, recreate it and prune the old network:</p> <pre><code>cd /opt/stacks/{stack} && docker compose up -d\ndocker network rm {stack}_default # after containers disconnect\n</code></pre>"},{"location":"infra/docker-networks/#stacks-keeping-own-networks","title":"Stacks Keeping Own Networks","text":"<p>These stacks have complex internal networking and should keep their own:</p> <ul> <li>immich \u2014 5 services with inter-dependencies</li> <li>paperless-ngx \u2014 5 services (webserver, broker, db, gotenberg, tika)</li> <li>arr-stack \u2014 5 services (rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr)</li> <li>karakeep \u2014 3 services with chrome dependency</li> <li>homepage \u2014 single service, can stay or migrate</li> <li>streamio \u2014 already removed, stremio uses VPN container directly</li> <li>vpn \u2014 single service, can stay or migrate</li> <li>ai/*_ai \u2014 AI stacks, untouched</li> </ul>"},{"location":"infra/docker-networks/#long-term-fix","title":"Long-term Fix","text":"<p>Add <code>default-address-pools</code> to <code>/etc/docker/daemon.json</code>:</p> <pre><code>{\n \"default-address-pools\": [\n {\"base\": \"10.0.0.0/8\", \"size\": 24}\n ]\n}\n</code></pre> <p>This gives 65536 /24 subnets, eliminating exhaustion. Requires Docker daemon restart (<code>systemctl restart docker</code>), which briefly disrupts all containers.</p>"},{"location":"infra/docker-networks/#commands","title":"Commands","text":"<pre><code># List all networks\ndocker network ls\n\n# Inspect a network\ndocker network inspect {name}\n\n# Remove unused networks\ndocker network prune\n\n# Remove a specific network (must have 0 containers)\ndocker network rm {name}\n</code></pre>"},{"location":"infra/portmap/","title":"Port Map \u2014 NUC 14 Docker Stacks","text":"<p>Reverse proxy: Zoraxy v3.3.2 on <code>192.168.1.4:8000</code> (LXC 108) Wildcard cert <code>*.nuclide.systems</code> \u00b7 source of truth: <code>proxy/zoraxy/routes.json</code> Manage routes: <code>uv run scripts/zoraxy_sync.py [--apply|--prune|--list]</code></p>"},{"location":"infra/portmap/#port-scheme","title":"Port Scheme","text":"Range Category 3100 Loki (log aggregation) 9090 Prometheus (monitoring) 9091 Grafana (monitoring) 9100 node-exporter (host metrics) 12345 Alloy (log agent UI) 10000\u201310999 Infrastructure 11000\u201311999 Security & Auth 12000\u201312999 Media \u2013 Immich 13000\u201313999 Media \u2013 Downloads / Arr 14000\u201314999 AI Stack 15000\u201315999 Documents 16000\u201316999 Automation 17000\u201317999 Notes & Bookmarks 18000\u201318999 DevOps / Image Gen 19000\u201319999 Tracking 20000\u201320999 Storage Admin 30000\u201330999 VPN Control"},{"location":"infra/portmap/#monitoring-ct-109-ops-19216818","title":"Monitoring \u2014 CT 109 ops (192.168.1.8)","text":"Port Host Service Notes 9090 CT 109 Prometheus LAN only; 90d retention 3000 CT 109 Grafana LAN only; admin/tapirnase 3100 CT 109 Loki LAN only; 30d retention; log aggregation 9221 CT 109 pve-exporter Proxmox VE metrics; auth: <code>monitor@pve!prometheus</code> 12345 CT 109 Alloy (self) agent UI; also runs on all other hosts at :12345 4090 CT 109 Wetty LAN only (127.0.0.1); web SSH \u2192 jump-menu.sh on nuc 10000 CT 109 Homepage LAN only (<code>ops.nuclide.lan:10000</code>); service dashboard; remote Docker via socket-proxy :2375 on CT 113, direct TCP on CT 104 10001 CT 109 Dozzle LAN only; live log viewer; agents on all 7 Docker hosts ~~10002~~ ~~CT 109~~ ~~Arcane~~ DECOMMISSIONED 2026-05-26 \u2014 replaced by Portainer 13080 CT 109 docs-server LAN only; mkdocs Material; auto-rebuilds from <code>fkrebs/docs</code> every 5 min \u2014 migrated from CT 111 2026-05-23 8200 CT 109 Infisical LAN only; secrets manager; migrated from CT 112 2026-05-26; <code>http://secrets.nuclide.lan:8200</code> 11000 CT 109 Pocket-ID <code>https://id.nuclide.systems</code>; OIDC IdP; migrated from CT 110 2026-05-26 9100 CT 109 node-exporter host-network, self-scrape 9100 CT 104 node-exporter standalone stack <code>/opt/stacks/monitoring/</code>; scraped by CT 109 <p>Scrape targets (CT 109 Prometheus): ~~<code>litellm</code> CT104:14000/metrics/~~ (SUNSET 2026-05-26), <code>node-ct104</code> :9100, <code>node-ct109</code> :9100, <code>home-assistant</code> 192.168.1.60:8123/api/prometheus (HA token), <code>prometheus</code> self, <code>walg</code> CT113:9100/textfile (WAL-G backup freshness, added 2026-05-23).</p> <p>Alloy (log agent): deployed on all 12 hosts \u2192 ships to Loki at CT 109:3100. - Docker hosts (CT 101/104/105/109/110/111/112/113): container at <code>/opt/stacks/alloy/</code>; reads Docker socket + journald - Binary/systemd (CT 102/103/108 + nuc): <code>/etc/alloy/config.alloy</code>; reads journald only - HA VM 100: Grafana Alloy add-on (wymangr/hassos-addons v0.0.8) \u2014 pushes metrics to CT 109 Prometheus remote_write + logs to Loki.</p>"},{"location":"infra/portmap/#infrastructure-1000010999","title":"Infrastructure (10000\u201310999)","text":"Port Service Container Public URL Notes 10000 Homepage <code>homepage</code> \u2014 DECOMMISSIONED 2026-05-23 \u2014 compose renamed <code>.DECOMMISSIONED</code> 10001 Dozzle <code>dozzle</code> \u2014 DECOMMISSIONED \u2014 moved to CT 109 2026-05-23 10002 Arcane <code>arcane</code> \u2014 DECOMMISSIONED \u2014 moved to CT 109 2026-05-23 10003 Gotify <code>gotify</code> gotify.nuclide.systems Push notifications 10004 Garage S3 API <code>garage</code> s3.nuclide.systems FIXED 2026-05-21: proxied by Zoraxy with ACME TLS. Garage API accessible at <code>https://s3.nuclide.systems</code>. Internal: <code>http://garage:3900</code> 10005 (127.0.0.1 only) Garage Admin <code>garage</code> \u2014 localhost only"},{"location":"infra/portmap/#dev-ct-104-migrated-from-ct-111-2026-05-26","title":"Dev (CT 104 \u2014 migrated from CT 111 2026-05-26)","text":"Port Service Container Public URL Notes 3000 Gitea <code>gitea</code> git.nuclide.systems Self-hosted Git; OIDC via Pocket-ID; Redis queue (<code>gitea_redis</code>) 222 Gitea SSH <code>gitea</code> \u2014 <code>ssh -p 222 git@git.nuclide.systems</code> 7080 Coder <code>coder</code> dev.nuclide.systems Workspace orchestrator; OIDC via Pocket-ID (no port) act-runner <code>act-runner</code> \u2014 Gitea Actions runner (<code>ct104-runner</code>) 1025 Proton Bridge SMTP <code>proton-bridge</code> \u2014 LAN only; requires <code>docker exec -it proton-bridge /bin/bash</code> for initial login 1143 Proton Bridge IMAP <code>proton-bridge</code> \u2014 LAN only"},{"location":"infra/portmap/#security-auth-1100011999","title":"Security & Auth (11000\u201311999)","text":"Port Service Container Public URL Notes 11001 Vaultwarden <code>vaultwarden</code> vault.nuclide.systems Password manager \u00b7 Pocket-ID OIDC client created 2026-05-21; auth flow not yet configured <p>Pocket-ID migrated from this CT to LXC 110 on 2026-05-20, then to CT 109 on 2026-05-26. See the External Services table below.</p>"},{"location":"infra/portmap/#media-immich-1200012999","title":"Media \u2013 Immich (12000\u201312999)","text":"Port Service Container Public URL Notes 12000 Immich <code>immich_server</code> immich.nuclide.systems Photos/videos \u2014 Immich Power Tools <code>immich_power_tools</code> immich-tools.nuclide.systems Container-internal :3000, Zoraxy proxy"},{"location":"infra/portmap/#media-downloads-arr-stack-1300013999","title":"Media \u2013 Downloads / Arr Stack (13000\u201313999)","text":"<p>All arr-stack services run behind <code>vpn_gluetun</code> container network.</p> Port Service Container Public URL Notes 13001 RDTClient <code>rdtclient</code> \u2014 LAN only 13002 Prowlarr <code>prowlarr</code> \u2014 LAN only 13003 Audiobookshelf <code>audiobookshelf</code> abs.nuclide.systems 13004 ShelfArr <code>shelfarr</code> \u2014 LAN only 13005 Flaresolverr <code>flaresolverr</code> \u2014 Internal only 30000 Gluetun VPN control <code>vpn_gluetun</code> \u2014 HTTP control API"},{"location":"infra/portmap/#ai-stack-1400014999","title":"AI Stack (14000\u201314999)","text":"Port Service Container Public URL Notes ~~14000~~ ~~LiteLLM~~ ~~<code>litellm</code>~~ \u2014 SUNSET 2026-05-26 \u2014 service block commented out in <code>ai/docker-compose.yml</code>; replaced entirely by Bifrost ~~14001~~ ~~LobeHub~~ ~~<code>lobehub</code>~~ \u2014 DECOMMISSIONED 2026-05-26 \u2014 compose renamed <code>.DECOMMISSIONED-lobehub-2026-05-26.yml</code>; replaced by Open WebUI 14002 Open WebUI <code>open-webui</code> chat.nuclide.systems Chat UI; stack <code>ai/open-webui.yml</code>; uses Qdrant + TEI for RAG 14003 Bifrost <code>bifrost</code> ai.nuclide.systems LLM gateway + MCP at <code>/mcp</code>; auth via <code>sk-bf-</code> VKs; stack <code>ai/bifrost/</code> ~~8080~~ ~~MCP Gateway~~ ~~<code>mcp-gateway</code>~~ \u2014 DECOMMISSIONED 2026-05-26 \u2014 compose renamed <code>.DECOMMISSIONED-2026-05-26</code>; MCP now at <code>https://ai.nuclide.systems/mcp</code> (Bifrost) 18002 ComfyUI <code>comfyui</code> \u2014 LAN only (Intel Arc iGPU, FLUX.1-schnell GGUF) 18003 ComfyUI MCP <code>comfyui-mcp</code> \u2014 FastMCP; reachable via gateway at <code>mcp.nuclide.systems/comfyui/mcp</code> 18005 Docling MCP <code>docling-mcp</code> \u2014 SAIA Docling PDF\u2192Markdown; reachable via gateway 18007 Kroki MCP <code>kroki-mcp</code> \u2014 Diagram rendering; reachable via gateway 18009 Speaches <code>speaches</code> \u2014 TTS/STT; LAN only ~~18010~~ ~~Shepard MCP~~ ~~<code>shepard-mcp</code>~~ \u2014 DECOMMISSIONED 2026-05-21 \u2014 replaced by native <code>https://shepard.nuclide.systems/v2/mcp</code> (streamable HTTP; gateway entry: <code>shepard</code>, auth via <code>${SHEPARD_API_KEY}</code>) 18011 Upload-artifact MCP <code>upload-artifact-mcp</code> \u2014 S3 chat-artifacts upload; reachable via gateway \u2014 SearXNG <code>searxng</code> \u2014 Internal, <code>shared_backend</code>, used by LobeHub"},{"location":"infra/portmap/#documents-1500015999","title":"Documents (15000\u201315999)","text":"Port Service Container Public URL Notes 15000 Traccar HTTP <code>traccar</code> traccar.nuclide.systems GPS tracking UI 15001 Traccar GPS <code>traccar</code> \u2014 TCP+UDP watch protocol 15002 Paperless AI <code>paperless-ai</code> paperless-ai.nuclide.systems Internal-only Zoraxy policy 15003 Paperless-ngx <code>paperless-ngx-webserver-1</code> paperless.nuclide.systems Internal-only Zoraxy policy"},{"location":"infra/portmap/#automation-1600016999","title":"Automation (16000\u201316999)","text":"Port Service Container Public URL Notes 16000 n8n <code>n8n</code> n8n.nuclide.systems Workflow automation"},{"location":"infra/portmap/#notes-bookmarks-1700017999","title":"Notes & Bookmarks (17000\u201317999)","text":"Port Service Container Public URL Notes 17000 Memos <code>memos</code> memos.nuclide.systems Notes 17001 Karakeep <code>karakeep</code> hoarder.nuclide.systems Bookmarks"},{"location":"infra/portmap/#devops-1800018999","title":"DevOps (18000\u201318999)","text":"Port Service Container Public URL Notes <p>| \u2014 | MCP servers (Gitea repos) | See AI Stack \u00a718000 | \u2014 | <code>mcp-comfyui</code>, <code>mcp-docling</code>, <code>mcp-upload-artifact</code> at <code>git.nuclide.systems/fkrebs/</code> (<code>mcp-shepard</code> decommissioned) |</p>"},{"location":"infra/portmap/#tracking-1900019999","title":"Tracking (19000\u201319999)","text":"<p>Traccar moved to 15000\u201315001 (Documents range). 19000\u201319001 now free.</p>"},{"location":"infra/portmap/#storage-admin-2000020999","title":"Storage Admin (20000\u201320999)","text":"Port Service Container Public URL Notes 20010 (127.0.0.1 only) pgAdmin <code>pgadmin</code> \u2014 shared-db pgAdmin, localhost only (CT 104)"},{"location":"infra/portmap/#lxc-112-secrets-19216817-decommissioned-2026-05-26","title":"~~LXC 112 \u2014 secrets~~ (<code>192.168.1.7</code>) \u2014 DECOMMISSIONED 2026-05-26","text":"<p>Infisical migrated to CT 109 ops. Containers stopped; LXC pending removal.</p> Port Service Notes ~~8200~~ ~~Infisical~~ Moved to CT 109:8200"},{"location":"infra/portmap/#lxc-113-db-19216816","title":"LXC 113 \u2014 db (<code>192.168.1.6</code>)","text":"<p>CT 113 is the dedicated postgres LXC. No public proxy routes \u2014 LAN access only.</p> Port Service Container Notes 5432 Postgres 17 <code>postgres</code> LAN: <code>192.168.1.6:5432</code> \u2014 accepts app connections from all CTs 5050 pgAdmin 4 <code>pgadmin</code> LAN only: <code>http://192.168.1.6:5050</code> \u2014 no Zoraxy route"},{"location":"infra/portmap/#lxc-111-dev-192168142-decommissioned-2026-05-26","title":"~~LXC 111 \u2014 Dev~~ (<code>192.168.1.42</code>) \u2014 DECOMMISSIONED 2026-05-26","text":"<p>All services migrated to CT 104. Containers stopped; LXC pending removal.</p> Port Service Notes ~~7080~~ ~~Coder~~ Moved to CT 104:7080 ~~3000~~ ~~Gitea~~ Moved to CT 104:3000 ~~222~~ ~~Gitea SSH~~ Moved to CT 104:222"},{"location":"infra/portmap/#qnap-ts-251d-1921681189","title":"QNAP TS-251D (<code>192.168.1.189</code>)","text":"<p>Celeron J4025, 2-core. Hosts Klipper natively (not Docker).</p> Port Service Notes 80 Mainsail 3D printer web UI 7125 Moonraker Klipper API <p>Config backed up daily to <code>git.nuclide.systems/fkrebs/klipper-config</code> via cron at 03:00 \u2192 covered offsite by Backrest <code>services/gitea</code> path.</p>"},{"location":"infra/portmap/#external-services-not-on-nuc-docker","title":"External Services (Not on NUC Docker)","text":"<p>Zoraxy routes to these external backends:</p> Domain Target Host Service ha.nuclide.systems <code>192.168.1.60:8123</code> Home Assistant VM 100 Home automation nc.nuclide.systems <code>192.168.1.41:11000</code> Nextcloud LXC 105 Cloud storage ocpp.nuclide.systems <code>192.168.1.60:8887</code> Home Assistant VM 100 EV charger OCPP shepard.nuclide.systems <code>192.168.1.49:80</code> Shepard LXC 101 Shepard shepard-api.nuclide.systems <code>192.168.1.49:8080</code> Shepard LXC 101 Shepard API id.nuclide.systems <code>192.168.1.8:11000</code> Ops CT 109 Pocket-ID OIDC IdP (migrated CT110\u2192CT109 2026-05-26) git.nuclide.systems <code>192.168.1.40:3000</code> Docker CT 104 Gitea (migrated from CT 111 2026-05-26) dev.nuclide.systems <code>192.168.1.40:7080</code> Docker CT 104 Coder (migrated from CT 111 2026-05-26) Gitea SSH <code>192.168.1.40:222</code> Docker CT 104 <code>ssh -p 222 git@git.nuclide.systems</code>"},{"location":"infra/portmap/#zoraxy-public-routes-proxied-via-19216814","title":"Zoraxy Public Routes (proxied via <code>192.168.1.4</code>)","text":"Domain Backend (NUC <code>192.168.1.40</code>) Port ai.nuclide.systems bifrost 14003 chat.nuclide.systems open-webui 14002 arcane.nuclide.systems arcane on CT 109 192.168.1.8:10002 gotify.nuclide.systems gotify 10003 vault.nuclide.systems vaultwarden 11001 immich.nuclide.systems immich_server 12000 immich-tools.nuclide.systems immich_power_tools (container-internal) abs.nuclide.systems audiobookshelf 13003 n8n.nuclide.systems n8n 16000 memos.nuclide.systems memos 17000 hoarder.nuclide.systems karakeep 17001 traccar.nuclide.systems traccar 15000 dozzle.nuclide.systems dozzle 10001 s3.nuclide.systems garage 10004 <p>Not publicly proxied (LAN / localhost only): pgadmin, paperless, paperless-ai, comfyui, comfyui-mcp, rdtclient, prowlarr, shelfarr, qdrant.</p>"},{"location":"infra/portmap/#known-issues","title":"Known Issues","text":"<ul> <li><code>lobe-postgres</code> / <code>lobe-redis</code>: host-port exposed (<code>0.0.0.0:5432/6379</code>) \u2014 firewall blocks external access but ideally restricted to localhost.</li> </ul>"},{"location":"infra/portmap/#oidc-client-registry-pocket-id-idnuclidesystems","title":"OIDC Client Registry (Pocket ID \u2014 <code>id.nuclide.systems</code>)","text":"Client ID Name Redirect URIs ~~<code>e73bb7b9</code>~~ ~~litellm~~ DELETED 2026-05-26 \u2014 LiteLLM sunset; Pocket-ID client removed from CT 110 DB ~~<code>26f3c26b</code>~~ ~~lobehub~~ DECOMMISSIONED 2026-05-26 \u2014 LobeChat removed; Open WebUI OIDC not yet wired <code>81cf4ed0</code> arcane <code>https://arcane.nuclide.systems/auth/oidc/callback</code> <code>33135ad4</code> n8n <code>https://n8n.nuclide.systems/auth/oidc/callback</code> <code>9c91c18b</code> immich <code>https://immich.nuclide.systems/auth/login</code> + mobile <code>62bf4e0d</code> memos <code>https://memos.nuclide.systems/auth/callback</code> <code>d92f82b0</code> karakeep <code>https://hoarder.nuclide.systems/api/auth/callback/custom</code> <code>a14b8076</code> nextcloud <code>https://nc.nuclide.systems/apps/user_oidc/code</code> <code>0aee4280</code> Coder <code>https://dev.nuclide.systems/api/v2/users/oidc/callback</code> <code>9444609e</code> Gitea <code>https://git.nuclide.systems/user/oauth2/pocket-id/callback</code> <code>38469e7e</code> Proxmox VE <code>https://192.168.1.20:8006</code> ~~798a367f~~ ~~daytona~~ DECOMMISSIONED \u2014 remove from Pocket-ID <code>cbbf20d5</code> Audiobookshelf <code>https://abs.nuclide.systems/\u2026</code> + <code>*</code> <code>d8733fcc</code> shelfarr <code>*</code> <code>78c78998</code> Claude MCP <code>https://claude.ai/api/mcp/auth_callback</code> + <code>https://mcp.nuclide.systems/mcp/auth/callback</code> <code>82ca2d53</code> nuc-ai (empty) \u2014 used by spawned MCP servers <code>7fe1a14b</code> zoraxy (empty) <code>af2f837b</code> mcp-auth (empty) \u2014 legacy, unused (new) vaultwarden <code>https://vault.nuclide.systems/auth/callback</code> <p>OIDC Endpoints (corrected 2026-05-17 \u2014 previously used wrong <code>/api/v1/oauth2/</code> path): - Authorization: <code>https://id.nuclide.systems/authorize</code> - Token: <code>https://id.nuclide.systems/api/oidc/token</code> - Userinfo: <code>https://id.nuclide.systems/api/oidc/userinfo</code> - Discovery: <code>https://id.nuclide.systems/.well-known/openid-configuration</code></p>"},{"location":"infra/proxmox-memory-audit/","title":"Proxmox Memory Audit","text":"<p>Last audited: 2026-05-22</p>"},{"location":"infra/proxmox-memory-audit/#host-physical-resources","title":"Host physical resources","text":"Resource Total Used (idle) Available RAM 62 GiB ~33 GiB ~28 GiB Swap 31 GiB 0 GiB 31 GiB"},{"location":"infra/proxmox-memory-audit/#ct-memory-allocations","title":"CT memory allocations","text":"CT Name Allocated (MiB) Swap (MiB) Typical use Notes 101 shepard 32,768 8,192 ~4 GB Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak 102 dns 1,024 512 ~100 MB AdGuard Home 103 backrest 4,096 1,024 ~200 MB Restic scheduler \u2014 bumped 2026-05-23 for 822 GB initial backup OOM fix 104 docker 49,152 32,000 6\u201322 GB Main Docker host; FLUX spikes to ~22 GB 105 nextcloud 8,196 8,196 ~2 GB Nextcloud AIO 108 zoraxy 2,048 512 ~300 MB Reverse proxy 109 ops 4,096 0 ~1.5 GB Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr 110 id 1,024 512 ~200 MB Pocket-ID 111 dev 32,768 8,192 ~3 GB Coder + Gitea workspaces 112 secrets 4,096 512 ~600 MB Infisical 113 db 4,096 0 ~800 MB Postgres 17 + WAL-G Sum 141,316 MiB (138 GiB) 2.2\u00d7 overprovisioned vs physical RAM"},{"location":"infra/proxmox-memory-audit/#key-findings","title":"Key findings","text":""},{"location":"infra/proxmox-memory-audit/#overprovisioning-is-safe-until-it-isnt","title":"Overprovisioning is safe \u2014 until it isn't","text":"<p>Proxmox uses balloon drivers so CTs only consume what they actually use. At idle the host sits at ~33 GB used with 28 GB available. This is healthy. However, two specific CTs represent risk:</p> <ul> <li>CT 104 (49 GiB limit) + CT 111 (32 GiB limit) together could claim 81 GiB \u2014 well over the 62 GiB physical. If both hit peak simultaneously the host would start swapping heavily.</li> <li>CT 104's ComfyUI (FLUX generation) spikes from 6 GB baseline to ~22 GB. That 16 GB spike on top of the 33 GB idle baseline = 49 GB total, within physical RAM but tight.</li> </ul>"},{"location":"infra/proxmox-memory-audit/#why-40g-docker-container-limit-destabilised-the-system","title":"Why 40G Docker container limit destabilised the system","text":"<p>Setting ComfyUI's container limit to 40 G was the trigger. At generation time: - FLUX model + activations: ~22 GB in the container - Other ~65 Docker containers on CT 104: ~7 GB - CT 104 OS + kernel: ~1 GB - Other CTs idle: ~26 GB - Total: ~56 GB \u2192 host started swapping, degrading all services</p>"},{"location":"infra/proxmox-memory-audit/#comfyui-xpu-memory-accounting-gap","title":"ComfyUI XPU memory accounting gap","text":"<p>ComfyUI's <code>get_free_memory()</code> for Intel XPU queries <code>torch.xpu.get_device_properties().total_memory</code> = 58 GB (the full shared memory pool \u2014 Arc shares system RAM). It has no awareness of the Docker cgroup limit. Smart memory management (<code>free_memory()</code>) calculates <code>memory_required - get_free_memory()</code> which is always hugely negative \u2192 never evicts models. This makes <code>--disable-smart-memory</code> irrelevant for XPU; smart memory is already broken.</p> <p>Consequence: ComfyUI will always try to load the full model into XPU memory regardless of container limit. The cgroup OOM killer is the only backstop.</p>"},{"location":"infra/proxmox-memory-audit/#container-limit-recommendation-for-comfyui-ct-104","title":"Container limit recommendation for ComfyUI (CT 104)","text":"Scenario Limit Safe? <code>--lowvram</code> (original) 20 G \u2705 Safe but slow (231 s/image) No <code>--lowvram</code>, FLUX only 24 G \u2705 Fits FLUX peak (~22 GB) + 2 GB headroom No <code>--lowvram</code> + img2img after FLUX 26 G \u2705 FLUX stays resident, SD1.5 loads on top 28 G \u26a0\ufe0f Marginal \u2014 OOM triggered in testing 40 G \u274c Destabilises host when generating <p>Peak host usage at 24 G container limit during FLUX generation: <code>24 + 7 (other containers) + 26 (other CTs idle) \u2248 57 GB</code> \u2014 stays under 62 GB physical.</p>"},{"location":"infra/proxmox-memory-audit/#recommendations","title":"Recommendations","text":"<ol> <li>ComfyUI container limit: set to 24 G when running without <code>--lowvram</code>. Current revert to 20 G + <code>--lowvram</code> is stable but slower.</li> <li>CT 104 LXC allocation (49 GiB): appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact.</li> <li>CT 111 (dev, 32 GiB): Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template.</li> <li>Watch list: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB \u2192 host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104.</li> <li>~~Long-term: when CT 109 (ops) is built, run Prometheus <code>node_exporter</code> on the PVE host and alert when host available RAM drops below 8 GiB.~~ Done 2026-05-23 \u2014 CT 109 live, node_exporter scraping PVE host via pve-exporter.</li> </ol>"},{"location":"infra/proxmox-memory-audit/#comfyui-memory-optimisation-log","title":"ComfyUI memory optimisation log","text":"Date Change Effect 2026-05-22 Removed <code>--lowvram</code> + raised limit to 28 G OOM at 28 G (XPU DRM buffers counted against cgroup) 2026-05-22 Raised limit to 40 G Host destabilised \u2014 reverted 2026-05-22 Reverted to <code>--lowvram</code> + 20 G Stable, slow (231 s/image) 2026-05-22 Downloaded <code>t5-v1_1-xxl-encoder-Q4_K_S.gguf</code> (2.6 GB vs 3.2 GB Q5_K_M) Saves 600 MB at load time 2026-05-22 Kept Q5_K_M T5 (Q4_K_S degrades prompt following per city96); removed <code>--lowvram</code>; set limit to 24 G; added <code>--async-offload --force-fp16</code> ~7\u201330 s generation, safe within host memory budget 2026-05-22 Removed <code>--async-offload</code> Flag incompatible with GGUF img2img on XPU \u2014 caused full CPU fallback (3.5 min/step) and pure-noise output. Removed; XPU generation now correct at ~7 s/step for txt2img. Current CLI: <code>--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16</code>"},{"location":"infra/proxmox-state/","title":"Proxmox Host Optimization Inventory \u2014 <code>nuc</code>","text":"<p>Generated: 2026-05-20 Host: <code>nuc</code> \u00b7 PVE <code>9.1.11</code> \u00b7 Kernel <code>6.17.13-4-pve</code> \u00b7 Debian 13 (trixie) CPU: Intel Core Ultra 7 155H (16C / 22T, hybrid P+E+LP-E) \u00b7 1 socket \u00b7 1 NUMA RAM: 62 GiB physical \u00b7 31 GiB zram swap (50 % of RAM, zstd, prio 100) Storage: single Crucial P3 2 TB NVMe (QLC, DRAM-less) \u2192 <code>rpool</code> (ZFS, ashift=12, no redundancy) Workload: 1 VM (HAOS) + 9 LXCs (Docker, AdGuard, Backrest, Nextcloud, Zoraxy, Pocket-ID, Dev, Secrets, DB) + 1 planned (Ops/CT109)</p>"},{"location":"infra/proxmox-state/#tldr-top-5-actionable-wins","title":"TL;DR \u2014 top 5 actionable wins","text":"<ol> <li>Memory overcommit is dangerous. Allocated guest RAM (\u2248 290 GiB) is ~4.7\u00d7 physical (62 GiB). Right-size CT 101 (was 160 \u2192 done, now 32) and CT 104 (still 128) \u2014 see \u00a72. \u2705 applied 2026-05-20</li> <li>ZFS ARC is artificially capped at 6.2 GiB. Default would be ~31 GiB. After \u00a71 settles, raise to 16 GiB. See \u00a73.</li> <li>No redundancy on a QLC SSD with 19 % wear and 59 TB written. Single-disk <code>rpool</code> on a DRAM-less consumer QLC drive is a SPOF. Add a second NVMe and convert to mirror \u2014 biggest reliability win available. See \u00a76.</li> <li>Backups never prune. Was configured <code>keep-all=1</code> \u2014 fixed to <code>keep-last=3,keep-daily=7,keep-weekly=4,keep-monthly=6</code>. See \u00a77. \u2705 applied 2026-05-20</li> <li><code>atime</code> and <code>autotrim</code> on ZFS. \u2705 applied 2026-05-20</li> <li>No DNS rewrites in AdGuard \u2014 every internal target is IP-only; add a split-horizon for <code>nuclide.systems</code> and a <code>.lan</code> shorthand set. See \u00a711a.</li> <li>Self-signed Proxmox web UI cert \u2014 front via Zoraxy for free LE. See \u00a711b.</li> </ol>"},{"location":"infra/proxmox-state/#1-system-snapshot","title":"1. System snapshot","text":"Resource State Notes Load avg normal PSI: CPU some=2.4 % / IO some=1 % over 60 s Memory 52 / 62 GiB used, 4.5 GiB free tight; zram swap 15 GiB in use Swap zram0 (zstd, 31 GiB) prio 100 working as designed; just a symptom of \u00a71 ARC 6.0 / 6.2 GiB (capped) hit ratio ~99 % but cap is far below default NVMe wear Percentage Used 19 %, 59.2 TB written ~5 % wear/year at current rate; healthy for now Temperature 56\u201358 \u00b0C well under the 95 \u00b0C critical threshold Uptime (see <code>uptime</code>) scrub clean, no checksum errors Cluster standalone quorum OK, no HA configured"},{"location":"infra/proxmox-state/#2-memory-vmct-sizing-measured-numbers","title":"2. Memory & VM/CT sizing (measured numbers)","text":"<p>Read from <code>/sys/fs/cgroup/lxc/<id>/memory.{current,peak,max}</code> and <code>free -h</code> inside each guest:</p> Guest Cap Current Peak Swap-in-use Verdict VM 100 (haos) 16 384 MiB 13 056 MiB n/a n/a balloon disabled; HAOS actually uses what it has CT 101 (shepard) 160 000 MiB 7.2 GiB 15.6 GiB 706 MiB wildly over-sized \u2014 peak is 10 % of cap CT 102 (adguard) 512 MiB 343 MiB 509 MiB (99 %) 19 MiB under-sized \u2014 at the cap, AdGuardHome alone is 350 MiB CT 103 (backrest) 512 MiB 89 MiB 305 MiB 16 MiB fine CT 104 (docker/AI) 128 000 MiB 18.8 GiB 29.3 GiB 9.3 GiB real workload, but currently swapping \u2014 likely starved by CT 101 CT 105 (nextcloud) 8 192 MiB 2.1 GiB 3.5 GiB 53 MiB fine CT 108 (zoraxy) 2 048 MiB 271 MiB 463 MiB 25 MiB fine; could halve <p>Sum of declared caps \u2248 290 GiB on a 62 GiB host. Sum of actual peaks \u2248 49 GiB \u2014 totally fits. CT 101's 160 GB cap is the entire problem: it's a phantom that scares the scheduler without using anything close to that.</p>"},{"location":"infra/proxmox-state/#concrete-ct-101-picture","title":"Concrete CT 101 picture","text":"<p>12 cores, load avg 8.5, ~9 Docker containers (Shepard frontend/backend, Keycloak, Neo4j, MongoDB, MongoExpress, TimescaleDB, Caddy, home-showcase-collector). Peak RSS 15.6 GiB.</p> <p>\u2192 Drop memory cap to 32 GiB (2\u00d7 peak headroom). No reboot required for LXC memory changes.</p>"},{"location":"infra/proxmox-state/#concrete-ct-104-picture","title":"Concrete CT 104 picture","text":"<p>16 cores, load avg 8.0, ~65 Docker containers including Immich (with ML/vectorchord), ComfyUI (image-gen), LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless-ngx+AI, Karakeep, Memos, Gotify, Garage S3, plus a forest of MCP servers, Speaches (OpenVINO using the Arc iGPU). 128 GiB of 200 GiB rootfs used.</p> <p>Peak RSS 29.3 GiB, but 9.3 GiB sitting in swap \u2014 under memory pressure. Two paths: 1. Recommended: cut CT 101 first, then CT 104's pressure mostly disappears on its own. Re-measure peak after CT 101 is fixed. Likely safe to cap at 48 GiB then. 2. Leave the 128 GiB cap as a generous ceiling \u2014 harmless once CT 101 is sane.</p>"},{"location":"infra/proxmox-state/#other-guests","title":"Other guests","text":"<ul> <li>CT 102 (AdGuard): hitting its cap. Bump to 1 GiB to give DNS room to breathe. (DNS is the single point of failure for the LAN \u2014 don't squeeze it.) <code>pct set 102 -memory 1024</code>.</li> <li>CT 108 (Zoraxy): cap 1 GiB is plenty.</li> <li>VM 100 (HAOS): 13 GiB actually used. Enable ballooning so the host can reclaim idle RAM: <pre><code>qm set 100 -balloon 4096\n</code></pre> This leaves memory=16384 as a ceiling but lets the host shrink it under pressure.</li> <li>KSM is already active (10 255 shared pages) \u2014 fine.</li> <li>zram swap: keep size at <code>ram / 2</code> until CT 101 is fixed; reduce to <code>ram / 4</code> afterwards.</li> </ul>"},{"location":"infra/proxmox-state/#3-zfs-tuning","title":"3. ZFS tuning","text":""},{"location":"infra/proxmox-state/#pool","title":"Pool","text":"Setting Current Recommend Why <code>autotrim</code> off <code>on</code> QLC needs trim; weekly fstrim alone is OK but autotrim is \"free\" <code>ashift</code> 12 keep correct for NVMe <code>atime</code> on (relatime) off unused on a hypervisor; reduces write amp on QLC <code>xattr</code> sa keep already optimal <code>compression</code> on (lz4) keep helping (1.61\u00d7 on HAOS disk) <code>dnodesize</code> legacy <code>auto</code> minor; only matters with millions of small files <code>recordsize</code> (rpool) 128 K keep for general tune per-dataset (see below)"},{"location":"infra/proxmox-state/#arc","title":"ARC","text":"<p><code>/etc/modprobe.d/zfs.conf</code> currently caps <code>zfs_arc_max=6669991936</code> (\u2248 6.2 GiB). - After \u00a72 sizing is done, raise this to 16 GiB: <code>options zfs zfs_arc_max=17179869184</code> and <code>zfs_arc_min=4294967296</code>. - Apply live without reboot: <code>echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max</code>.</p>"},{"location":"infra/proxmox-state/#per-dataset","title":"Per-dataset","text":"<ul> <li>VM zvols (<code>rpool/data/vm-*</code>): default <code>volblocksize</code> is 16 K \u2014 fine. HAOS disk uses <code>cache=writethrough</code>; on ZFS, switch to <code>cache=none</code> (or unset) \u2014 writethrough doubles the sync cost on top of ZFS's own integrity guarantees.</li> <li>CT 104 (<code>subvol-104-disk-0</code>, Docker + image-gen): keep <code>recordsize=128K</code>. The workload is dominated by large model files and image outputs, not small-file DB traffic \u2014 shrinking the record size would hurt, not help.</li> <li>Nextcloud (<code>subvol-105-disk-1</code>): leave at 128 K (mixed sizes, mostly larger files).</li> </ul>"},{"location":"infra/proxmox-state/#pool-features","title":"Pool features","text":"<p><code>zpool upgrade rpool</code> was run during this audit and enabled <code>redaction_list_spill</code> + <code>raidz_expansion</code>. Other disabled features (<code>fast_dedup</code>, <code>longname</code>, <code>large_microzap</code>, <code>dynamic_gang_header</code>, <code>block_cloning_endian</code>, <code>physical_rewrite</code>) can be enabled with another <code>zpool upgrade rpool</code> \u2014 only do this if you do not need to roll back to an older ZFS.</p>"},{"location":"infra/proxmox-state/#commands","title":"Commands","text":"<pre><code>zpool set autotrim=on rpool\nzfs set atime=off rpool\n# (optional, once memory is sane):\necho 'options zfs zfs_arc_max=17179869184' > /etc/modprobe.d/zfs.conf\nupdate-initramfs -u -k all\n</code></pre>"},{"location":"infra/proxmox-state/#4-storage-vm-disk-options","title":"4. Storage & VM disk options","text":""},{"location":"infra/proxmox-state/#vm-100-haos","title":"VM 100 (<code>haos</code>)","text":"<pre><code>- scsi0: local-zfs:vm-100-disk-1,cache=writethrough,discard=on,size=32G,ssd=1\n+ scsi0: local-zfs:vm-100-disk-1,cache=none,discard=on,iothread=1,size=32G,ssd=1\n</code></pre> <ul> <li><code>cache=none</code> (or remove cache entirely) \u2014 let ZFS manage caching.</li> <li><code>iothread=1</code> with <code>virtio-scsi-pci</code> controller \u2014 already using <code>virtio-scsi-pci</code>, just add iothread.</li> <li>Already has <code>discard=on</code> and <code>ssd=1</code> \u2714</li> </ul>"},{"location":"infra/proxmox-state/#lxc-local-zfs-storage","title":"LXC <code>local-zfs</code> storage","text":"<ul> <li><code>sparse 1</code> is set \u2714 \u2014 thin-provisioned.</li> <li>All CTs are on <code>local-zfs</code> rootfs; OK.</li> </ul>"},{"location":"infra/proxmox-state/#unas-share-current-state-measured","title":"UNAS share \u2014 current state (measured)","text":"<p>Backend: <code>192.168.1.31</code> (looks like a UniFi NAS \u2014 exports <code>/volume/.../.unifi-drive/storage/.data</code>, the only NFS export listed is restricted to four allowed clients: the host <code>.20</code>, CT 104 <code>.40</code>, plus <code>.60</code> and <code>172.30.33.1</code>).</p> <p>Two parallel mounts on the host pointed at the same backing data:</p> Mount Type Options (key bits) Consumers <code>/mnt/pve/unas</code> NFS v3 <code>proto=tcp, mountproto=udp, rsize/wsize=1M, hard, relatime, timeo=600</code> CT 103 (backrest), CT 104 (docker) \u2014 bind-mounted to <code>/mnt/pve/unas</code> inside <code>/mnt/pve/unas_smb</code> CIFS v3.1.1 <code>cache=strict, actimeo=1, soft, rsize/wsize=4M, uid/gid=33</code> CT 105 (nextcloud) \u2014 bind-mounted to <code>/mnt/pve/unas</code> inside <p>Issues:</p> <ol> <li>CT 105 is on CIFS to the same data CT 104 uses via NFS. Pure duplication. Nextcloud does massive amounts of <code>stat()</code> traffic; <code>actimeo=1</code> on the CIFS mount forces every metadata lookup to hit the wire, which is slow.</li> <li>NFS is v3, not v4.x. v4 is preferred unless the UDM doesn't export it. v4 fixes locking, removes the separate mountd dance, and supports session trunking.</li> <li><code>mountproto=udp</code> under packet loss can intermittently fail to (re)mount. Set <code>mountproto=tcp</code>.</li> <li><code>hard</code> mount with no <code>intr</code> equivalent. If UNAS goes away, anything blocked on it hangs the calling process indefinitely. For non-critical use cases (Nextcloud, but not backrest), <code>soft,timeo=100,retrans=3</code> is friendlier \u2014 Backrest backups should stay <code>hard</code>.</li> <li>CT 105 cannot mount NFS directly because the UNAS export only allows IPs <code>.20/.40/.60/.172.30.33.1</code> \u2014 <code>.41</code> (CT 105) is missing. So either keep the host-side bind-mount approach (correct) or have UNAS export to <code>.41</code> too.</li> <li>The bind-mount approach is correct for unprivileged CTs that can't run NFS/CIFS clients themselves. Don't change that pattern.</li> </ol> <p>Recommended consolidation:</p> <pre><code># 1. Probe whether the NAS speaks NFSv4\nmount -t nfs -o vers=4.2,proto=tcp 192.168.1.31:/var/nfs/shared/storage /mnt/test\n# if it works:\npvesm set unas --options vers=4.2,proto=tcp,hard,noatime\n# (this re-mounts on next access; or unmount/remount /mnt/pve/unas)\n\n# 2. Switch CT 105 to the NFS bind-mount\npct set 105 --mp0 /mnt/pve/unas,mp=/mnt/pve/unas\n# (CT 105 currently uses unas_smb \u2192 unas. New line bind-mounts the NFS mount.)\n# Then verify nextcloud-aio still sees uid/gid 33 properly \u2014 NFS uses host UIDs,\n# whereas CIFS was forcing uid=33. May need to chown on the NAS or add an idmap.\n\n# 3. Drop the CIFS storage once CT 105 is migrated\npvesm remove unas_smb # if it exists as PVE storage\n# or remove the entry from /etc/pve/storage.cfg\n</code></pre> <p>Notes on perf:</p> <ul> <li>1 GbE NIC ceiling is ~118 MB/s. NFS rsize/wsize=1M already saturates that.</li> <li>If you have or add a 2.5 GbE / 10 GbE link on either side, also enable jumbo frames (MTU 9000) on <code>vmbr0</code>, the host NIC, and UNAS. That alone can ~double bulk-read throughput.</li> <li>For Nextcloud's small-file stats: <code>actimeo=60</code> (NFS) or <code>cache=loose,actimeo=60</code> (CIFS, if you stay on it) \u2014 dramatically cuts roundtrips at the cost of slightly stale directory listings.</li> </ul>"},{"location":"infra/proxmox-state/#5-cpu-boot-kernel","title":"5. CPU / boot / kernel","text":"Item State Recommend Governor <code>performance</code> keep HWP EPP <code>default</code> set to <code>balance_performance</code> if you want some idle savings without latency cost: <code>echo balance_performance > /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference</code> <code>intel_iommu=on iommu=pt</code> set \u2714 keep GPU passthrough (<code>i915.force_probe=!7dd5 xe.force_probe=7dd5</code>) set for Arc Xe (Meteor Lake) keep <code>nvme_core.default_ps_max_latency_us=0</code> set \u2714 disables NVMe power-save \u2014 good for stability, costs ~1 W idle <code>kernel.numa_balancing</code> 0 correct for single socket Old kernels installed 6.17.13-4 (current) + 7.0.0-3 keep both for now; remove 7.0.0-3 once you've booted 7.0.2-5 successfully after the pending upgrade"},{"location":"infra/proxmox-state/#hybrid-core-scheduling","title":"Hybrid-core scheduling","text":"<p>The 155H has P-cores (cores 0\u201311), E-cores (12\u201317), LP-E cores (18\u201321). Linux 6.x with <code>intel_pstate=active</code> handles ITD/HWP well; no manual pinning is needed for current workloads. If a CT becomes latency-sensitive, you can pin it with <code>cpuset</code> via <code>lxc.cgroup2.cpuset.cpus</code> (P-cores only).</p>"},{"location":"infra/proxmox-state/#6-reliability-spof","title":"6. Reliability / SPOF","text":""},{"location":"infra/proxmox-state/#single-disk-is-the-biggest-risk","title":"Single disk is the biggest risk","text":"<ul> <li>Current: 1\u00d7 CT2000P3PSSD8 (QLC, DRAM-less). 19 % wear in (check <code>power_on_hours</code>).</li> <li>Recommendation: add a second 2 TB NVMe (ideally a TLC drive with DRAM \u2014 Crucial T500, WD SN770, Samsung 990 Pro, Solidigm P44 Pro). Then: <pre><code>zpool attach rpool nvme-CT2000P3PSSD8_2429E8BBCFB4-part3 /dev/disk/by-id/<new-disk>-part3\n</code></pre> (requires partitioning the new disk to match \u2014 <code>sgdisk -R</code> from the existing). Pool becomes a mirror with full self-heal.</li> <li>If a second internal slot isn't available, at minimum increase backup frequency for CT 101/104 (the irreplaceable data).</li> </ul>"},{"location":"infra/proxmox-state/#boot-redundancy","title":"Boot redundancy","text":"<p><code>proxmox-boot-tool kernel list</code> shows one bootloader entry. After \u00a76 mirror is set up, run <code>proxmox-boot-tool init /dev/<new-disk>-partN</code> so either disk can boot.</p>"},{"location":"infra/proxmox-state/#7-backups-high-priority-silent-risk","title":"7. Backups (HIGH PRIORITY \u2014 silent risk)","text":"<p><code>/etc/pve/storage.cfg</code>: <pre><code>nfs: unas\n prune-backups keep-all=1\n</code></pre></p> <p><code>keep-all=1</code> means backups are never deleted automatically. UNAS already holds 2 TB. Set a real policy, e.g.:</p> <pre><code>pvesm set unas --prune-backups keep-last=3,keep-daily=7,keep-weekly=4,keep-monthly=6\n</code></pre> <p>Also: there is no <code>vzdump</code> job configured in <code>/etc/pve/jobs.cfg</code>. Backups are either manual or driven from CT 103 (Backrest). Recommend a scheduled <code>vzdump</code> job for at least VM 100 and CT 101/104 in addition to Backrest, so PVE-native restores remain trivial.</p>"},{"location":"infra/proxmox-state/#8-apt-repositories-cleanup","title":"8. APT / repositories cleanup","text":"<p>State today:</p> <pre><code>/etc/apt/sources.list.d/\n\u251c\u2500\u2500 ceph.list # all lines commented \u2014 fine but consider deleting the file\n\u251c\u2500\u2500 proxmox.sources # pve-no-subscription (modern deb822) \u2190 keep\n\u251c\u2500\u2500 pve-enterprise.list.bak # backup, safe to remove\n\u251c\u2500\u2500 pve-enterprise.sources # Enabled: false \u2190 keep as-is or remove\n\u251c\u2500\u2500 pve-install-repo.list # pve-no-subscription duplicate\n\u2514\u2500\u2500 pve-no-subscription.list # pve-no-subscription duplicate\n</code></pre> <p><code>pve-install-repo.list</code> and <code>pve-no-subscription.list</code> duplicate what <code>proxmox.sources</code> already declares. APT deduplicates fetches but the duplication is a foot-gun (one of them will go stale on the next PVE major version transition). Recommended cleanup:</p> <pre><code>rm /etc/apt/sources.list.d/pve-install-repo.list\nrm /etc/apt/sources.list.d/pve-no-subscription.list\nrm /etc/apt/sources.list.d/pve-enterprise.list.bak\n# keep proxmox.sources and pve-enterprise.sources (already disabled)\napt update\n</code></pre> <p>Also: there are 9 pending upgrades including <code>pve-manager 9.1.18</code> (you're on 9.1.11) and a kernel update. Run <code>apt update && apt full-upgrade</code> at a convenient window.</p>"},{"location":"infra/proxmox-state/#unattended-upgrades-configured-2026-05-20","title":"Unattended-upgrades (configured 2026-05-20)","text":"<p>The host previously had a cron line <code>0 2 * * * apt-get update && apt-get upgrade -y</code> that was silently no-op'ing on every kernel / PVE point release \u2014 <code>apt-get upgrade</code> refuses to install new dependencies, which PVE updates always introduce.</p> <p>Replaced with <code>unattended-upgrades</code> in a conservative profile:</p> File Purpose <code>/etc/apt/apt.conf.d/52unattended-upgrades-pve</code> local policy \u2014 origins allowlist + email + reboot policy <code>/etc/apt/apt.conf.d/20auto-upgrades</code> enables the daily update-list + unattended-upgrade run <p>Auto-applied: - <code>origin=Debian,codename=trixie,label=Debian</code> (stable main) - <code>origin=Debian,codename=trixie-security,label=Debian-Security</code> - <code>origin=Debian,codename=trixie-updates</code> (stable point updates)</p> <p>Held for manual <code>apt full-upgrade</code> (intentionally \u2014 review release notes first): - <code>origin=Proxmox,...</code> \u2014 pve-manager, kernels, qemu-server, all PVE components</p> <p>Settings: - <code>Automatic-Reboot \"false\"</code> \u2014 kernel updates require a manual reboot - <code>Remove-Unused-Dependencies \"true\"</code> \u2014 autoremove orphans after upgrades - <code>AutoFixInterruptedDpkg \"true\"</code> \u2014 resume after crash mid-upgrade - <code>Mail \"notify@home.box\"</code>, <code>MailReport \"on-change\"</code> \u2014 alerts on actual changes</p> <p>Triggered by: - <code>apt-daily.timer</code> (daily ~07:00) \u2014 refresh package lists - <code>apt-daily-upgrade.timer</code> (daily ~06:00) \u2014 apply unattended upgrades</p> <p>Caveat: mail delivery isn't reaching you yet. Postfix is up but has <code>relayhost = (none)</code> \u2014 change notifications get delivered locally to <code>/var/mail/notify</code> on the host, not to your inbox. Set up a smart-host relay (Gmail/Postmark/etc.) if you want the mails to actually land. Until then, check <code>/var/log/unattended-upgrades/unattended-upgrades.log</code> for history.</p> <p>Verify any time: <pre><code>unattended-upgrade --dry-run --debug 2>&1 | grep -E \"Allowed origins|would be upgraded|pkgs that look\"\nsystemctl list-timers apt-daily-upgrade.timer\ntail /var/log/unattended-upgrades/unattended-upgrades.log\n</code></pre></p>"},{"location":"infra/proxmox-state/#9-networking","title":"9. Networking","text":"<ul> <li>Single bridge <code>vmbr0</code> on <code>enp86s0</code> \u2014 no VLAN aware (<code>bridge-vlan-aware yes</code>). If you ever want to segment guests by VLAN, add it now (no impact on existing guests as long as you don't tag them): <pre><code>bridge-vlan-aware yes\nbridge-vids 2-4094\n</code></pre></li> <li><code>net.core.rmem_max</code> / <code>wmem_max</code> are at distro defaults (208 KiB). With a 1 GbE NIC the impact is small (link is already saturated at NFS rsize=1M), but with future 2.5/10 GbE bump to 16 MiB: <pre><code>cat >/etc/sysctl.d/99-net.conf <<'EOF'\nnet.core.rmem_max=16777216\nnet.core.wmem_max=16777216\nnet.ipv4.tcp_rmem=4096 87380 16777216\nnet.ipv4.tcp_wmem=4096 65536 16777216\nEOF\nsysctl --system\n</code></pre></li> <li>TCP congestion control is <code>cubic</code>. <code>bbr</code> is generally better for mixed workloads \u2014 change only if you measure a problem.</li> <li><code>wlo1</code> is present but unused \u2014 confirm and disable in BIOS or <code>iface wlo1 inet manual</code> (already done). No action.</li> </ul>"},{"location":"infra/proxmox-state/#10-container-specific-issues","title":"10. Container-specific issues","text":""},{"location":"infra/proxmox-state/#ct-104-docker-ai-image-gen-48-gib-cap-16-cores-gpu-passthrough","title":"CT 104 (docker / AI / image-gen) \u2014 48 GiB cap, 16 cores, GPU passthrough","text":"<p>Measured: 18.8 GiB current, peak 29.3 GiB, 9.3 GiB in swap, load 8.0, ~65 Docker containers (Immich + ML, ComfyUI, LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless+AI, many MCP servers, Speaches-OpenVINO).</p> <ul> <li><code>cores: 16</code> is justified by the workload (load avg 8 across 16 = ~50 % avg). Don't drop.</li> <li>128 GiB cap is generous but currently 9 GiB is in swap \u2014 this is memory pressure from CT 101's phantom cap, not from CT 104's own needs. Fix CT 101 first, re-measure, then either lower CT 104 to 48 GiB or leave it.</li> <li>GPU pass-through (<code>/dev/dri/card1</code>, <code>renderD128</code>) confirmed visible inside CT and being used by Speaches via OpenVINO \u2714</li> <li>Keep <code>recordsize=128K</code> (large files dominate).</li> <li><code>swap: 32000</code> is high \u2014 consider <code>swap: 8192</code>. Heavy CT swap-out on a QLC root SSD adds write amplification.</li> <li>Bind-mounted <code>/mnt/pve/unas</code> (NFS) is the right choice \u2714</li> <li>Bonus: raise scheduler weight if image-gen latency matters: <code>pct set 104 -cpuunits 200</code>.</li> </ul>"},{"location":"infra/proxmox-state/#ct-101-shepard-docker-currently-160-gib-cap-peak-156-gib","title":"CT 101 (shepard / docker) \u2014 currently 160 GiB cap, peak 15.6 GiB","text":"<p>Workload: ~9 containers \u2014 Shepard frontend/backend, Keycloak, Neo4j, MongoDB, MongoExpress, TimescaleDB, Caddy, home-showcase-collector.</p> <ul> <li>Drop memory cap to 32 GiB: <code>pct set 101 -memory 32768</code>. No restart needed.</li> <li><code>cores: 12</code> is fine (load 8.5 \u2014 close to fully loaded, real work).</li> <li><code>swap: 8192</code> \u2714</li> <li>This single change unblocks everything else.</li> </ul>"},{"location":"infra/proxmox-state/#ct-102-adguard-dns-undersized","title":"CT 102 (adguard / DNS) \u2014 undersized","text":"<p>Measured: 343 MiB used at the 512 MiB cap, AdGuardHome alone is 350 MiB RSS, the CT is one OOM event from killing the LAN's DNS.</p> <ul> <li><code>pct set 102 -memory 1024</code> (bump to 1 GiB).</li> <li>Add <code>onboot: 1</code> (already set) + <code>startup: order=1</code> (boots first) + <code>protection: 1</code> (anti-fatfinger). This is the only DNS \u2014 treat it like infrastructure.</li> </ul>"},{"location":"infra/proxmox-state/#ct-103-backrest-fine","title":"CT 103 (backrest) \u2014 fine","text":"<p>89 MiB used, peak 305 MiB. No changes needed.</p>"},{"location":"infra/proxmox-state/#ct-105-nextcloud-privileged-cifs","title":"CT 105 (nextcloud) \u2014 privileged + CIFS","text":"<ul> <li>Privileged container (no <code>unprivileged: 1</code>). For a public-facing app this is the wrong tradeoff. Migration: stop CT, <code>vzdump</code> backup, restore as unprivileged. Be ready to fix file ownership on the bind-mount afterwards (privileged UID 33 \u2192 unprivileged needs <code>lxc.idmap</code>).</li> <li>Uses CIFS bind-mount to UNAS with <code>actimeo=1</code> \u2014 see \u00a74. Switch to NFS bind-mount, after confirming UID mapping (CIFS forces uid=33; NFS uses host UIDs as-is).</li> <li>Memory cap 8 GiB, peak 3.5 GiB \u2014 fine as-is.</li> <li>Runs Nextcloud AIO stack (mastercontainer-managed: apache, postgres, redis, imaginary, collabora, notify-push).</li> </ul>"},{"location":"infra/proxmox-state/#ct-108-zoraxy-slight-oversize","title":"CT 108 (zoraxy) \u2014 slight oversize","text":"<p>Peak 463 MiB on a 2 GiB cap. Lower to 1 GiB if desired (cosmetic).</p>"},{"location":"infra/proxmox-state/#ct-104-docker-stacks-inventory-optstacks","title":"CT 104 \u2014 Docker stacks inventory (<code>/opt/stacks</code>)","text":"<p>CT 104 keeps its Docker workloads in a git-tracked monorepo at <code>/opt/stacks/</code> with one directory per stack, plus meta-docs (<code>PORTMAP.md</code>, <code>storage.md</code>, <code>volumes.md</code>, <code>docker-networks.md</code>, <code>todo.md</code>). Good practice \u2014 this is how to keep ~65 containers manageable. The other CTs (101, 105) don't have <code>/opt/stacks</code> \u2014 their compose files live elsewhere.</p> <p>Stack list (29 dirs):</p> Stack Status Notes <code>ai/</code> active \u2014 large subtree comfyui, lobehub, litellm, speaches, mcp-gateway, mcp-servers (many MCP yml files), searxng. Custom <code>syncstack.py</code> to manage cross-file project names. <code>arr-stack/</code> dormant (defined, not running) rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr <code>arcane/</code> dormant Docker dashboard <code>daytona/</code> active (as <code>daytona-minimal</code>) dev environments + runner + registry <code>dozzle/</code> dormant container log viewer <code>gotify/</code> active push notifications <code>homepage/</code> dormant dashboard <code>immich/</code> active (5 containers) photo platform + ML <code>karakeep/</code> active (3 containers) bookmark mgr + chrome + meilisearch <code>memos/</code> active notes <code>n8n/</code> active pinned <code>2.20.11</code> (good \u2014 there's an explicit version-drift comment in the compose) <code>nexa/</code> dormant (?) <code>paperless_ai/</code>, <code>paperless-ngx/</code> active (5 containers between them) OCR pipeline <code>pocketid/</code> active OIDC provider <code>proxy/</code> dormant (?) <code>qdrant/</code> dormant vector DB <code>shared-db/</code> active shared-postgres + garage (S3-compatible) + pgadmin (defined) <code>streamio/</code> dormant media <code>traccar/</code> active GPS tracker <code>vaultwarden/</code> active password mgr <code>vpn/</code> dormant gluetun (intended VPN egress wrapper?) <code>backups/</code>, <code>docs/</code>, <code>scripts/</code> meta dirs (no compose) <p>Observations:</p> <ol> <li>~10 stacks are defined but dormant. No RAM/CPU cost while down, but their images sit on disk and the git repo accrues dead code. Either run them, document why they're parked, or <code>git rm</code> them \u2014 repo drift is the silent killer of \"I know what's running\" confidence.</li> <li>Stack name \u2260 compose project name for the <code>ai/</code> and <code>daytona/</code> trees (multiple compose files per dir, different project names). The <code>syncstack.py</code> helper exists for this; just be aware that <code>docker compose -f</code> lookups by directory name don't match.</li> <li>Disk-reclaim potential (measured <code>docker system df</code>):</li> </ol> Asset Total Reclaimable Images 83 / 61.6 GB 8.3 GB Build cache 105 entries / 9.0 GB 4.6 GB Volumes 31 / 3.0 GB 940 MB (21 dangling) Containers 60 active 0 <p><pre><code>pct exec 104 -- docker system prune -a --volumes\n# or non-destructively just the build cache:\npct exec 104 -- docker builder prune -a\n</code></pre> ~13 GB to recover. On a 200 GB rootfs that's 64 % full, this is meaningful.</p> <ol> <li>Stack\u2192Zoraxy mapping (\u00a711b): when fronting via Zoraxy, the canonical service endpoints (per <code>PORTMAP.md</code>) are CT 104's IP <code>192.168.1.40</code> + port. Worth cross-referencing that file when setting up reverse-proxy entries.</li> </ol>"},{"location":"infra/proxmox-state/#general-lxc-hygiene","title":"General LXC hygiene","text":"<ul> <li>Strip the giant HTML banners from the top of every <code>/etc/pve/lxc/*.conf</code> \u2014 pure noise in <code>pct config</code>.</li> <li>All CTs except 105 are unprivileged \u2714 with <code>nesting=1,keyctl=1</code> (appropriate for Docker).</li> </ul>"},{"location":"infra/proxmox-state/#anti-fat-finger-protection-protection-1-boot-order","title":"Anti-fat-finger protection (<code>protection: 1</code>) + boot order","text":"<p>Applied across all critical guests (2026-05-20). <code>protection: 1</code> blocks <code>pct destroy</code> / \"Remove\" from the UI until manually unset \u2014 cheap insurance against the wrong-CT-deleted incident.</p> Guest protection startup order Rationale CT 102 (AdGuard / DNS) \u2705 order=1 (boots first) LAN-wide DNS \u2014 nothing resolves until this is up CT 108 (Zoraxy / LE proxy) \u2705 order=2 Public-facing reverse proxy + LE; depends on DNS CT 103 (Backrest) \u2705 default Holds backup config and snapshot metadata CT 104 (Docker / AI / image-gen) \u2705 default Largest data footprint (200 G rootfs); 65 containers CT 105 (Nextcloud) \u2705 default User data VM 100 (HAOS) \u2705 default Home automation state CT 101 (shepard) \u274c (left optional) default Currently a dev/iteration target; protect once stabilized: <code>pct set 101 -protection 1</code> <p>The <code>onboot: 1</code> flag was already set on all guests \u2714 \u2014 they all auto-start on host reboot. The two <code>startup</code> ordered ones now also boot in the right sequence: DNS \u2192 Zoraxy \u2192 everything else in parallel.</p> <p>To remove protection on a guest later: <code>pct set <id> -protection 0</code> (or <code>qm set 100 -protection 0</code>).</p>"},{"location":"infra/proxmox-state/#11a-dns-adguard-rewrites-site-wide-consistency","title":"11a. DNS \u2014 AdGuard rewrites & site-wide consistency","text":"<p>Current state (measured):</p> <ul> <li>AdGuard Home runs on CT 102 (<code>192.168.1.2</code>, hostname <code>dns</code>) \u2014 DNS on <code>0.0.0.0:53/tcp+udp</code>, admin UI on <code>:80</code>.</li> <li>Upstream stack is well-configured: load-balanced mix of Telekom-DoT, NextDNS, DNS4EU, Quad9, Cloudflare, Google (DoH/DoQ where supported), bootstrap+fallback on Cloudflare.</li> <li>All 6 LXCs resolve via <code>nameserver 192.168.1.2</code> \u2714</li> <li>Proxmox host (<code>192.168.1.20</code>) resolves via <code>192.168.1.2</code> \u2714 \u2014 but its <code>search</code> domain is <code>box</code> (probably an install-time leftover; AdGuard's <code>local_domain_name</code> is <code>lan</code>).</li> <li>HAOS VM is on <code>192.168.1.60</code> \u2014 DNS setting unknown without Home Assistant access; verify.</li> <li><code>rewrites: []</code> and <code>rewrites_enabled: false</code> \u2014 no internal name resolution is happening today.</li> </ul>"},{"location":"infra/proxmox-state/#why-this-matters","title":"Why this matters","text":"<p>Without rewrites, you address everything by IP. That's brittle (IP changes break links), invisible in logs, and prevents nice tricks like split-horizon DNS for <code>nuclide.systems</code> (so the same name resolves to Zoraxy LAN-internally without going through your public IP / WAN hairpin).</p>"},{"location":"infra/proxmox-state/#recommended-rewrite-set","title":"Recommended rewrite set","text":"<p>In AdGuard UI \u2192 Filters \u2192 DNS rewrites, enable rewrites and add:</p> <pre><code># Split-horizon: public domain \u2192 Zoraxy on LAN\nnuclide.systems \u2192 192.168.1.4\n*.nuclide.systems \u2192 192.168.1.4\n\n# Service-name shortcuts under the local_domain_name (`.lan`)\npve.lan \u2192 192.168.1.20 # Proxmox UI\nnuc.lan \u2192 192.168.1.20 # host shorthand\ndns.lan \u2192 192.168.1.2 # AdGuard itself\nzoraxy.lan \u2192 192.168.1.4 # reverse proxy\nshepard.lan \u2192 192.168.1.49 # CT 101\ndocker.lan \u2192 192.168.1.40 # CT 104\nnextcloud.lan \u2192 192.168.1.41 # CT 105\nhaos.lan \u2192 192.168.1.60 # VM 100\nunas.lan \u2192 192.168.1.31 # NAS\nrouter.lan \u2192 192.168.1.1 # UniFi gateway\n</code></pre> <p>The split-horizon entries are the highest-value: once Zoraxy proxies <code>pve.nuclide.systems</code> (see \u00a711b), the same URL works both from the public internet and from inside the LAN \u2014 with no NAT-loopback weirdness and with the LAN traffic never leaving the building.</p> <p>Edit the YAML directly if preferred (<code>/opt/AdGuardHome/AdGuardHome.yaml</code> inside CT 102), then restart AdGuard. The line <code>rewrites_enabled: false</code> must flip to <code>true</code>.</p>"},{"location":"infra/proxmox-state/#verify-clients-are-actually-using-adguard","title":"Verify clients are actually using AdGuard","text":"<p>After rewrites are in, walk the inventory:</p> Client Should use DNS Check All 6 LXCs \u2714 already at .2 <code>pct exec <id> -- cat /etc/resolv.conf</code> Proxmox host \u2714 already at .2 <code>cat /etc/resolv.conf</code> HAOS VM (192.168.1.60) unknown HAOS UI \u2192 Settings \u2192 System \u2192 Network \u2192 check DNS servers; should be <code>192.168.1.2</code> Router (192.168.1.1, UniFi) DHCP-hands-out DNS to clients \u2014 must serve .2 as primary UniFi: Settings \u2192 Networks \u2192 LAN \u2192 DHCP DNS: <code>192.168.1.2</code> IoT devices (Roborock at .64, others) inherit via DHCP from router once UniFi DHCP serves .2, every device that DHCP-renews picks it up. Force-renew or reboot stragglers. Anything with hard-coded <code>1.1.1.1</code> / <code>8.8.8.8</code> bypassing the filter grep service configs for upstream DNS \u2014 apps like Pi-hole-aware clients, some Smart TVs, Chromecasts"},{"location":"infra/proxmox-state/#optional-hardening-once-the-rewrites-are-stable","title":"Optional hardening once the rewrites are stable","text":"<ul> <li>DNSSEC: <code>enable_dnssec: true</code> (currently false). Most upstreams already validate, but flipping this on adds end-to-end checking.</li> <li>Block DoH bypass: AdGuard has a \"Blocked services\" preset for Cloudflare/Google DoH endpoints \u2014 useful so clients can't sidestep the filter.</li> <li>Resolve <code>box</code> search-domain confusion: edit <code>/etc/resolv.conf</code> (or set it via <code>/etc/network/interfaces</code>) on the host to <code>search lan</code> so it matches AdGuard's <code>local_domain_name</code>.</li> <li>Secondary DNS for resilience: CT 102 is currently the sole DNS \u2014 if it crashes, the LAN goes blind. Either add <code>protection: 1</code> (done above) and rely on it, or stand up a tiny secondary AdGuard on a different CT and configure UniFi DHCP to hand out both. (Out of scope for low-hanging fruit, but worth knowing.)</li> <li>Conditional forwarding for UNAS: if UNAS announces hostnames (mDNS or its own DNS), point AdGuard at it for the relevant zone via \"Upstream DNS servers\" using the <code>[/zone/]upstream</code> syntax.</li> </ul>"},{"location":"infra/proxmox-state/#action-checklist","title":"Action checklist","text":"<ol> <li>AdGuard UI \u2192 Filters \u2192 DNS rewrites: paste the table above.</li> <li>AdGuard UI \u2192 Settings \u2192 DNS settings \u2192 enable \"DNS rewrites\".</li> <li>UniFi: confirm DHCP option 6 = <code>192.168.1.2</code> (LAN clients get AdGuard).</li> <li>HAOS: confirm Home Assistant has <code>192.168.1.2</code> set as DNS.</li> <li>Force-renew DHCP leases on key clients (or just wait \u2014 most renew within 24 h).</li> <li>After \u00a711b is done, the public <code>pve.nuclide.systems</code> resolves to <code>.4</code> from inside the LAN automatically.</li> </ol>"},{"location":"infra/proxmox-state/#11b-tls-certificates-for-the-proxmox-web-ui","title":"11b. TLS certificates for the Proxmox web UI","text":"<p>Current state: the PVE web UI on <code>https://192.168.1.20:8006</code> uses the self-signed certificate generated at install (<code>/etc/pve/local/pveproxy-ssl.pem</code> is absent \u2192 falls back to <code>pve-ssl.pem</code>). Every login throws a browser warning.</p> <p>The wider setup: Zoraxy (CT 108, <code>192.168.1.4</code>) already handles Let's Encrypt for <code>nuclide.systems</code> (the public domain for this host). So there are three sane options; pick A unless you have a reason not to.</p>"},{"location":"infra/proxmox-state/#option-a-reverse-proxy-pve-through-zoraxy-recommended","title":"Option A \u2014 Reverse-proxy PVE through Zoraxy (recommended)","text":"<p>Pros: single source of LE truth (Zoraxy already renews); no DNS-plugin setup; no exposing the API; nice domain like <code>pve.nuclide.systems</code>. Cons: depends on Zoraxy being up (keep IP:8006 as fallback); needs WebSocket pass-through for the noVNC console and <code>xterm.js</code> shell.</p> <ol> <li>Zoraxy host entry</li> <li>Domain: <code>pve.nuclide.systems</code> (or whatever subdomain)</li> <li>Target: <code>https://192.168.1.20:8006</code></li> <li>Enable WebSocket support (mandatory \u2014 noVNC, xterm.js, task log streaming all use it)</li> <li>Skip backend TLS verification (PVE's cert is self-signed)</li> <li>Enable HSTS once you've confirmed the setup works</li> <li>Optionally restrict by source: only LAN + Cloudflare/Tailscale IPs</li> <li>DNS: add an A record <code>pve.nuclide.systems</code> \u2192 public IP (or split-horizon to <code>192.168.1.20</code> for LAN). Zoraxy will ACME-challenge via whichever method it's configured for (HTTP-01 or DNS-01).</li> <li>Keep <code>https://192.168.1.20:8006</code> reachable on LAN as an emergency fallback. Don't disable it.</li> <li>Set the PVE redirect-to-HTTPS rules in Zoraxy on for both <code>:80</code> and <code>:443</code>.</li> </ol> <p>Important caveat: the PVE Mobile app and the <code>pvesh</code> / API clients may not love going through a reverse proxy (they're picky about TLS SNI and cookie domains). Keep direct IP access available for API tooling, or test thoroughly.</p>"},{"location":"infra/proxmox-state/#option-b-pves-built-in-acme-with-dns-01","title":"Option B \u2014 PVE's built-in ACME with DNS-01","text":"<p>Pros: no reverse proxy in the path; PVE renews itself; works for the API too. Cons: requires a DNS provider plugin (your registrar's API token), and an LE-acceptable FQDN that resolves publicly.</p> <ol> <li>Register an ACME account: <pre><code>pvenode acme account register default you@nuclide.systems\n</code></pre></li> <li>Configure a DNS plugin. PVE supports <code>acme-dns</code>, <code>cloudflare</code>, <code>route53</code>, <code>desec</code>, etc. via the <code>acme.sh</code> plugin set. Example for Cloudflare: <pre><code>pvenode acme plugin add dns cf --api cf --data CF_Token=XXXXXXXX\n</code></pre> Replace <code>cf</code> plugin name to match whichever registrar you use for <code>nuclide.systems</code>.</li> <li>Tell the node which domain(s) and how: <pre><code>pvenode config set --acme domains=nuc.nuclide.systems\npvenode config set --acmedomain0 domain=nuc.nuclide.systems,plugin=cf\n</code></pre></li> <li>Order: <pre><code>pvenode acme cert order\n</code></pre> PVE drops the cert at <code>/etc/pve/nodes/nuc/pveproxy-ssl.pem</code> and renews ~30 days before expiry via the <code>pve-daily-update</code> timer.</li> </ol>"},{"location":"infra/proxmox-state/#option-c-push-zoraxys-cert-into-pve","title":"Option C \u2014 Push Zoraxy's cert into PVE","text":"<p>Only useful if A and B are off the table. Zoraxy stores its issued certs (location varies by Zoraxy version \u2014 typically under its data dir, e.g. <code>/opt/zoraxy/conf/certs/</code>). Cron a script that copies the active cert/key and concatenates them as <code>/etc/pve/local/pveproxy-ssl.pem</code> (cert + chain) and <code>/etc/pve/local/pveproxy-ssl.key</code>, then <code>systemctl reload pveproxy</code>. Brittle \u2014 only worth it if you must.</p>"},{"location":"infra/proxmox-state/#recommended-path","title":"Recommended path","text":"<p>Do A (reverse proxy through Zoraxy) for the web UI. It piggybacks on existing renewal. The mobile-app/API edge cases are usually fine if Zoraxy passes the WebSocket and preserves the <code>Host</code> header. If you later need full ACME on the node itself (e.g. you want valid TLS for <code>pvesh</code> and the API at the node FQDN too), layer B on top \u2014 they don't conflict.</p>"},{"location":"infra/proxmox-state/#other-cts-with-web-uis-worth-fronting-via-zoraxy","title":"Other CTs with web UIs worth fronting via Zoraxy","text":"<p>While you're at it, route through Zoraxy for free LE: - Backrest (CT 103) \u2014 currently IP-only - AdGuard (CT 102) admin UI \u2014 <code>192.168.1.2:3000</code> - Nextcloud (CT 105) \u2014 almost certainly already exposed; verify it terminates LE in Zoraxy and not internally - Zoraxy itself (CT 108) \u2014 self-hosted, already TLS</p> <p>For each, add a Zoraxy host entry, set a subdomain, and disable any local TLS / port-exposed listener that bypasses Zoraxy.</p>"},{"location":"infra/proxmox-state/#11-maintenance-observability","title":"11. Maintenance / observability","text":"Item State Recommend <code>lm-sensors</code> not installed <code>apt install lm-sensors && sensors-detect --auto</code> for CPU/NVMe temps in the UI Journal size 1.5 GiB OK; cap at 1 GiB if you want predictability: <code>journalctl --vacuum-size=1G</code> and <code>SystemMaxUse=1G</code> in <code>journald.conf</code> <code>fstrim.timer</code> active (weekly) OK; <code>zfs trim</code> runs separately when autotrim=on ZFS scrub last run 2026-05-10, clean default monthly timer is good Subscription nag not removed If desired, <code>pve-no-nag</code> patch or the <code>proxmox-helper-scripts</code> line \u2014 purely cosmetic Email alerts (check <code>/etc/pve/user.cfg</code>) configure <code>root@pam</code> email for failed scrub / failed backup notifications"},{"location":"infra/proxmox-state/#13-update-management-current-model","title":"13. Update management \u2014 current model","text":"<p>The host previously had <code>0 2 * * * apt-get update && apt-get upgrade -y</code> (silently no-op'd on every PVE/kernel update) and a weekly <code>bash <(wget tteck/.../update-lxcs-cron.sh)</code> cron that ran <code>dist-upgrade</code> across every LXC. Both removed 2026-05-20 and replaced with the structure below.</p>"},{"location":"infra/proxmox-state/#layer-1-host-packages-debian-proxmox","title":"Layer 1 \u2014 Host packages (Debian + Proxmox)","text":"<ul> <li><code>unattended-upgrades 2.12</code> installed on the host.</li> <li><code>/etc/apt/apt.conf.d/52unattended-upgrades-pve</code> allows only <code>Debian</code>, <code>Debian-Security</code>, <code>trixie-updates</code> \u2014 Proxmox origin held for manual review.</li> <li>Triggered by <code>apt-daily.timer</code> and <code>apt-daily-upgrade.timer</code> (ship with <code>apt</code>, both <code>active enabled</code>).</li> <li><code>Automatic-Reboot \"false\"</code> \u2014 kernel updates wait for a manual reboot.</li> <li><code>Mail \"notify@home.box\"</code>, <code>MailReport \"on-change\"</code> \u2014 Postfix is up but <code>relayhost = (none)</code>, so mail is delivered locally to <code>/var/mail/notify</code> (not your inbox until you wire a smart-host).</li> </ul>"},{"location":"infra/proxmox-state/#layer-2-ct-os-packages-debian","title":"Layer 2 \u2014 CT OS packages (Debian)","text":"<p><code>unattended-upgrades</code> deployed inside every CT (CT 104 already had it; 101/102/103/105/108/110 added 2026-05-20):</p> CT u-u version Distro Status 101 2.12 trixie active 102 2.9.1 bookworm active 103 2.9.1 bookworm active 104 2.12 trixie active 105 2.9.1 bookworm active 108 2.12 trixie active 110 2.12 trixie active <p>Per-CT allowlist is Debian-only \u2014 third-party repos (<code>docker.com</code>, <code>jotta.cloud</code>, <code>claude.ai</code>, <code>cli.github.com</code>, <code>dl.k6.io</code>) are excluded because they ship breaking changes outside Debian's freeze. Upgrade those with explicit <code>apt upgrade <pkg></code>.</p>"},{"location":"infra/proxmox-state/#layer-3-helper-script-app-binaries-adguard-zoraxy","title":"Layer 3 \u2014 Helper-script app binaries (AdGuard, Zoraxy)","text":"<p>Each helper-scripts CT ships <code>/usr/bin/update</code> that re-<code>curl|bash</code>'s the community-scripts installer. Replaced with proper systemd timers using the apps' own update mechanisms:</p> CT Timer Schedule Mechanism 102 AdGuard <code>adguard-update.timer</code> Wed 03:30 (+15 m jitter) native <code>AdGuardHome --update</code> flag 108 Zoraxy <code>zoraxy-update.timer</code> Wed 03:40 (+15 m jitter) GitHub releases API, stable semver only (skips RCs), binary swap + 30 s health check + auto-rollback <p>Both log to <code>/var/log/{adguard,zoraxy}-update.log</code> and journal. Manual invoke: <code>systemctl start <name>-update.service</code>.</p>"},{"location":"infra/proxmox-state/#layer-4-docker-engine-inside-cts","title":"Layer 4 \u2014 Docker engine inside CTs","text":"<p><code>docker-ce</code> updates in CT 101, 104, 105, 110 \u2014 held by the Debian-only allowlist. Apply with <code>apt upgrade docker-ce docker-ce-cli containerd.io</code> when you want them. Add <code>origin=Docker</code> to the allowlist if you want to auto-apply (not recommended; engine updates occasionally break running containers).</p>"},{"location":"infra/proxmox-state/#layer-5-docker-images-the-75-containers","title":"Layer 5 \u2014 Docker images (the ~75 containers)","text":"<p>Plan: deploy Diun on CT 109 (ops LXC, see \u00a716). Diun watches image tags on registries, posts to Gotify when a new image is available. Pulls remain manual (<code>docker compose pull && up -d</code>) \u2014 protects against latest-tag drift like the n8n incident pinned in <code>/opt/stacks/n8n/docker-compose.yaml</code>.</p> <p>Layer 6 \u2014 Nextcloud-AIO: self-updates via the mastercontainer (CT 105). No external mechanism needed.</p>"},{"location":"infra/proxmox-state/#14-vm-100-haos-auto-restart-watchdog","title":"14. VM 100 (HAOS) auto-restart watchdog","text":"<p>Old approach: <code>*/5 * * * * /root/vm100.sh > /dev/null</code> in cron. Script archived to <code>/root/vm100.sh.bak</code> on 2026-05-20.</p> <p>Replaced with a systemd timer + oneshot:</p> <ul> <li><code>/usr/local/sbin/vm100-watchdog.sh</code> \u2014 only restarts on <code>status: stopped</code>; skips paused/prelaunch/transitional states; respects <code>/var/lock/qemu-server/lock-100.conf</code> so it doesn't race vzdump or migration</li> <li><code>vm100-watchdog.service</code> (Type=oneshot)</li> <li><code>vm100-watchdog.timer</code> (<code>OnUnitActiveSec=1min</code>, RandomizedDelaySec=15s)</li> </ul> <p>Recovery latency improved from 5 min \u2192 1 min; logging structured in <code>journalctl -u vm100-watchdog</code>.</p>"},{"location":"infra/proxmox-state/#15-identity-pocket-id-on-its-own-ct","title":"15. Identity \u2014 Pocket-ID on its own CT","text":""},{"location":"infra/proxmox-state/#state-as-of-2026-05-20","title":"State as of 2026-05-20","text":"<p>CT 110 \"id\" created at <code>192.168.1.5</code> as the dedicated IdP host. Pocket-ID was previously on CT 104 as one of ~65 docker containers; moved off because:</p> <ul> <li>Pocket-ID gates auth for everything else; sharing a host with image-gen workloads (9 GiB swap pressure on CT 104) means a single OOM/restart there knocks out SSO across the whole estate</li> <li>Pattern match with CT 102 (DNS) and CT 108 (reverse proxy) \u2014 each infra-critical concern gets its own CT</li> </ul> CT 110 setting Value Hostname / IP <code>id</code> / <code>192.168.1.5</code> Cores / RAM / rootfs 1 / 1 GB / 4 GB Privilege unprivileged, nesting=1, keyctl=1 Boot order <code>onboot=1</code>, <code>startup=order=3</code> (after DNS=1, Zoraxy=2) Protection <code>protection: 1</code> Auto-updates unattended-upgrades, Debian-only allowlist Docker 29.5.1 + compose v5.1.3"},{"location":"infra/proxmox-state/#duplication-procedure-used","title":"Duplication procedure used","text":"<ol> <li><code>sqlite3 pocket-id.db \".backup /tmp/pi-snap/pocket-id.db\"</code> on CT 104 (online, no downtime to id.nuclide.systems)</li> <li>tar everything except <code>*.db*</code>; restore the live snapshot as <code>pocket-id.db</code></li> <li><code>pct pull</code> \u2192 <code>pct push</code> to CT 110</li> <li>Adapted compose to drop the <code>shared_backend</code> external network reference (CT 110 uses default bridge)</li> <li><code>docker compose up -d</code></li> <li>Verified <code>http://192.168.1.5:11000/healthz</code> returns 200</li> </ol>"},{"location":"infra/proxmox-state/#zoraxy-cutover","title":"Zoraxy cutover","text":"<p><code>id.nuclide.systems</code> upstream needs to change from <code>192.168.1.40:11000</code> \u2192 <code>192.168.1.5:11000</code>. Single-line config edit in Zoraxy + reload. Verified live in \u00a711b once executed.</p>"},{"location":"infra/proxmox-state/#secrets-rotation-list-deferred-to-cutover-day","title":"Secrets-rotation list (deferred to cutover day)","text":"<ul> <li><code>OIDC_CLIENT_SECRET</code> for the Arcane registration (exposed in chat transcript): rotate in Pocket-ID UI, update Arcane env, restart Arcane</li> <li><code>ENCRYPTION_KEY</code> and <code>JWT_SECRET</code> in <code>/opt/stacks/arcane/docker-compose.yml</code>: move to <code>.env</code> (currently empty), regenerate, restart Arcane. Existing user sessions get invalidated \u2014 fine, ask everyone to log in again</li> </ul>"},{"location":"infra/proxmox-state/#16-ct-109-ops-planned-observability-ops-lxc","title":"16. CT 109 \"ops\" \u2014 planned observability + ops LXC","text":"<p>Single LXC holding everything monitoring/ops-shaped. Sizing target: 4 cores / 8 GiB RAM / 50 GiB rootfs, unprivileged, nesting=1. RAM bumped from 6 \u2192 8 GiB to accommodate Loki. Disk bumped from 30 \u2192 50 GiB for Loki log retention (30d) alongside Prometheus TSDB.</p> <p>\u26a0\ufe0f IP conflict: inventory originally assigned <code>192.168.1.6</code> but CT 113 (db) took <code>.6</code> and CT 112 (secrets) took <code>.7</code>. CT 109 needs the next free infra IP \u2014 likely <code>.8</code> (verify against UniFi DHCP table before provisioning).</p> <p>Access model (initial): LAN-only. No Zoraxy routes until Tinyauth is deployed. Services reachable directly by IP.</p>"},{"location":"infra/proxmox-state/#stack-to-deploy-on-ct-109","title":"Stack to deploy on CT 109","text":"Service Purpose Prometheus metrics TSDB, 30d retention Loki log aggregation backend \u2014 receives from Alloy agents on all hosts Grafana dashboards over Prometheus + Loki (unified metrics + log search) Alertmanager + alertmanager-gotify-bridge alert routing \u2192 Gotify Arcane Manager central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out) Dozzle UI live log tail (quick debugging); agents on CT 101 + CT 104 + CT 110. Complements Loki \u2014 Dozzle for live, Loki for historical/search Homarr unified dashboard, native Pocket-ID OIDC, Prometheus widget + Grafana iframe support Diun docker image update notifier \u2192 Gotify Tinyauth forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID docker-socket-proxy local + remote (CT 101/104/110) \u2014 hardened read-only docker.sock for Homarr/Arcane discovery sshwifty web SSH client, multi-tab \u2014 multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred. docs-server mkdocs Material site (<code>/docs</code> git repo \u2192 static HTML); migrating here from CT 111 where it currently runs at port 13080. LAN-only, no Zoraxy route."},{"location":"infra/proxmox-state/#sidecars-deployed-on-each-host","title":"Sidecars deployed on each host","text":"<p>Two agents per host \u2014 node-exporter (metrics) and Alloy (logs). Kept separate: node-exporter metric names are assumed by every Prometheus dashboard/alert; Alloy emitting compatible metrics adds validation risk for no gain.</p> Host Sidecars PVE host node-exporter, smartctl-exporter, pve-exporter, Alloy (journald \u2192 Loki: pve-manager, pveproxy, pvedaemon, LXC/VM lifecycle) CT 101 node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, Alloy (Docker logs + journald \u2192 Loki) CT 103 node-exporter, Alloy (backrest.service journal + <code>/var/log/rclone-*.log</code> \u2192 Loki) CT 104 node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, intel_gpu_exporter, Alloy (Docker logs + journald \u2192 Loki) CT 110 node-exporter, Dozzle agent, Arcane edge agent, Alloy (journald \u2192 Loki) CT 111 node-exporter, intel_gpu_exporter, Alloy (journald + Coder/Gitea logs \u2192 Loki) CT 113 node-exporter, postgres_exporter, Alloy (journald + postgres logs \u2192 Loki)"},{"location":"infra/proxmox-state/#services-that-stay-where-they-are-not-on-ct-109","title":"Services that stay where they are (NOT on CT 109)","text":"<ul> <li>AdGuard (CT 102), Zoraxy (CT 108), Backrest (CT 103), Pocket-ID (CT 110) \u2014 each has its own CT for blast-radius reasons</li> <li>All app stacks (Immich, Nextcloud, Shepard, n8n, ComfyUI, etc.)</li> </ul>"},{"location":"infra/proxmox-state/#migration-of-gotify-tier-b-schedule-when-ct-109-is-otherwise-stable","title":"Migration of Gotify (Tier B \u2014 schedule when CT 109 is otherwise stable)","text":"<p>Gotify currently runs in CT 104 docker stack at <code>gotify.nuclide.systems</code>. Moving to CT 109 isolates alerting from CT 104 outages but means updating env vars / webhook targets in ~10 places (MCP servers, Backrest webhooks). Plan: copy DB and app tokens via volume tar, deploy on CT 109, update Zoraxy upstream, then sweep dependents.</p>"},{"location":"infra/proxmox-state/#migration-of-mcp-gateway-tier-b-split-control-workload","title":"Migration of MCP Gateway (Tier B \u2014 split control / workload)","text":"<p><code>/opt/stacks/ai/mcp-gateway/</code> on CT 104 is the OIDC-gated control plane for the ~20 MCP child containers. Decision (2026-05-20): move only the gateway to CT 109; the child MCP containers stay on CT 104 (they're workload, not control plane).</p> <p>Mechanics: - Gateway on CT 109 uses <code>DOCKER_HOST=tcp://<CT 104 socket-proxy>:2375</code> (the docker-socket-proxy already planned for CT 104) instead of the bind-mounted <code>/var/run/docker.sock</code>. socket-proxy ACL must allow <code>containers, exec, images</code> (read+write). - Migrate state files via volume tar: <code>config.json</code>, <code>agents.json</code>, <code>prompts/</code>, <code>usage.db</code>, <code>gateway_tokens.json</code>, <code>nc_user_creds.json</code>. Keep them on CT 109 local zfs, not UNAS (per-request latency matters). - Pocket-ID redirect URI stays <code>https://mcp.nuclide.systems/sso/callback</code> \u2014 only Zoraxy's upstream flips from <code>192.168.1.40:8080</code> to the CT 109 IP. - Don't touch the children; gateway still spawns them by name against CT 104's daemon.</p> <p>Build-order slot: after Arcane + socket-proxy land in \u00a716's checklist, before Tinyauth.</p>"},{"location":"infra/proxmox-state/#arcane-specifics","title":"Arcane specifics","text":"<ul> <li>Existing <code>/opt/stacks/arcane/</code> on CT 104 has the Manager 80 % built: 41 MB SQLite DB carrying Pocket-ID OIDC client + admin user <code>fkrebs@nucli.de</code></li> <li>Container currently stopped (~since May 19)</li> <li>Edge mode chosen for agents \u2014 agent dials out to Manager over HTTPS/gRPC with mTLS. Manager auto-generates the edge CA on first start; each agent enrolls with a token and gets a 1 yr client cert with auto-renewal</li> <li>Same <code>ghcr.io/getarcaneapp/arcane:latest</code> image for Manager and agents; mode is env-driven (<code>ARCANE_EDGE_AGENT=true</code> on agents)</li> <li>Intel iGPU is not in Arcane's GPU monitoring catalog (NVIDIA + AMD only). LXC-level passthrough to CT 104 stays untouched; only Arcane's GPU widgets will be blank for the Arc</li> </ul>"},{"location":"infra/proxmox-state/#build-order","title":"Build order","text":"<ol> <li>Create CT 109 (specs above)</li> <li>Deploy node-exporter on host + Prometheus + Grafana first (start collecting baselines)</li> <li>Deploy Arcane Manager with the existing DB restored from CT 104</li> <li>Issue agent tokens, deploy Edge agents on CT 101/104/110</li> <li>Deploy Dozzle Manager + agents</li> <li>Deploy Diun</li> <li>Deploy sshwifty (configure host list + SSH keys for PVE, CT 103, CT 104, CT 111; LAN-only port 8182)</li> <li>Deploy Tinyauth (configure Pocket-ID client first)</li> <li>Deploy Homarr</li> <li>Front the lot via Zoraxy: <code>arcane.</code>, <code>dozzle.</code>, <code>grafana.</code>, <code>prom.</code>, <code>home.</code>, <code>shell.</code> <code>.nuclide.systems</code></li> <li>Verify each through the Tinyauth gate where applicable</li> <li>Add Diun watchlist + Alertmanager routing \u2192 Gotify</li> </ol>"},{"location":"infra/proxmox-state/#17-lldp-unifi-topology-visibility","title":"17. LLDP / UniFi topology visibility","text":"<p>Investigated 2026-05-20:</p> <ul> <li><code>lldpd</code> is installed and active on the host (defaults: advertises + listens)</li> <li>Existing <code>/usr/local/bin/update_interface_desc.sh</code> hourly cron consumes LLDP from neighbors and writes <code># PortDescr:</code> comments into <code>/etc/network/interfaces</code></li> <li>Host is on switch port 10 of the D-Link DGS-1210-28P (<code>SysName: dgs1210</code>, FW <code>6.32.008</code>).</li> <li><code>/usr/local/bin/update_interface_desc.sh</code> was silently failing \u2014 <code>lldpcli</code> not in cron's PATH and the original logic only appended new lines, never replaced stale ones. Rewritten 2026-05-20:</li> <li>PATH set inside the script (<code>/usr/sbin</code> included)</li> <li>Uses <code>lldpcli show neighbors -f keyvalue</code> for machine-readable parsing</li> <li>Single <code># LLDP: <chassis> :: <port-descr></code> line per interface; legacy <code># PortDescr:</code> lines stripped</li> <li>Idempotent (second run = no-op, no backup churn)</li> <li>Daily-keyed backup <code>/etc/network/interfaces.bak.YYYYMMDD</code></li> <li>Only logs on actual change</li> <li>Old script preserved at <code>/usr/local/bin/update_interface_desc.sh.bak-2026-05-20</code></li> </ul> <p>The UDM Pro won't see \"nuc\" in its topology view because LLDP frames use the Nearest-Bridge multicast (<code>01:80:c2:00:00:0e</code>) which any 802.1D-compliant switch terminates by spec \u2014 and the D-Link DGS-1210-28P sits between the host and the UDM. LLDP-MED is not a fix for this; it's for endpoint (VoIP/MFP) discovery, not transparent LLDP forwarding.</p> <p>Remediation paths in order of effort:</p> <ol> <li>Enable SNMP v2c/v3 on the DGS-1210 + add as a Generic SNMP device in UniFi \u2192 UDM sees the switch and can map port\u2194MAC. Most practical for this stack.</li> <li>Add <code>snmpd</code> to the host + generic SNMP device in UniFi \u2192 CPU/mem/iface stats from Proxmox visible in UniFi (not in topology, but in monitoring).</li> <li>UniFi-managed switch between host and UDM \u2014 clean answer; requires hardware.</li> </ol> <p>D-Link DGS-1210 admin UI lives at <code>http://192.168.1.10/</code>. Verify the admin password is non-default \u2014 DGS-1210 ships with <code>admin</code>/blank or <code>admin</code>/<code>admin</code> on most firmware revisions. A flat-LAN switch with default creds is one of the easier vectors. (Credentials redacted from this doc \u2014 check your password manager.)</p> <p>Useful inspection from the host any time: <code>lldpcli show neighbors</code>.</p>"},{"location":"infra/proxmox-state/#18a-unas-access-uid-consistency-model-post-nfsv4-investigation","title":"18a. UNAS access \u2014 UID consistency model (post-NFSv4 investigation)","text":"<p>Investigation result (2026-05-20): UNAS Pro advertises NFSv4 in <code>rpcinfo</code> but has no v4 export tree configured. Every v4 mount attempt returns <code>No such file or directory</code>. Ubiquiti has not announced NFSv4 support and the community thread asking for it has no ETA. The official help center also confirms: \"UniFi Drive does not support certain NFS export options, such as <code>no_root_squash</code>\". So root_squash + v3-only is the long-term reality.</p>"},{"location":"infra/proxmox-state/#universal-uid-landscape-on-unas","title":"Universal UID landscape on UNAS","text":"<p>Every NFS client write lands as uid 977 / gid 988 (UNAS's <code>all_squash</code> + <code>anon_uid=977</code>/<code>anon_gid=988</code>). The chown probe confirmed no client can change this from the host side. Files written via the legacy CIFS mount appear as uid 33 to the CIFS client but are stored differently on UNAS \u2014 the CIFS <code>forceuid=33</code> mount option lies about ownership client-side.</p>"},{"location":"infra/proxmox-state/#per-ct-access-pattern-canonical","title":"Per-CT access pattern (canonical)","text":"CT Mount Container uid Effective on disk Status 103 backrest NFS bind from host root squashes to 977 \u2713 consistent 104 docker NFS bind from host mostly root, n8n=1000 (latent) squashes to 977 \u2713 for root containers; n8n latent if it ever writes to UNAS 105 nextcloud CIFS today (forceuid=33) \u2192 NFS + bindfs target uid 33 (www-data) inside Nextcloud, bindfs translates to 977 on disk needs migration \ud83d\udea8 still CIFS"},{"location":"infra/proxmox-state/#convention-for-new-containers","title":"Convention for new containers","text":"<p>Set <code>PUID=977 PGID=988</code> on any container that writes to UNAS. This pre-aligns with UNAS's enforced mapping and avoids latent permission issues (the n8n class). For images that don't support PUID/PGID, run them as root inside the container \u2014 root squashes to 977 cleanly.</p>"},{"location":"infra/proxmox-state/#why-bindfs-for-ct-105-specifically","title":"Why bindfs for CT 105 specifically","text":"<p>Nextcloud's PHP code hard-checks file ownership against <code>www-data</code> (uid 33). Without remap, NFS reads return uid 977 and Nextcloud refuses to operate normally. CIFS hides this with <code>forceuid=33</code>. NFS+bindfs achieves the same lie with the much faster NFS rail underneath \u2014 verified ~5\u00d7 speed-up on metadata-heavy ops in the non-destructive test on 2026-05-20.</p>"},{"location":"infra/proxmox-state/#trigger-event-to-revisit","title":"Trigger event to revisit","text":"<p>Watch community.ui.com/RELEASES for a UniFi Drive release that adds: - NFSv4 export option (would enable idmap) - <code>no_root_squash</code> support (would enable server-side chown to specific uids) - Configurable <code>anonuid</code>/<code>anongid</code> (would let us match a real uid)</p> <p>Any of these would let us simplify the CT 105 stack.</p>"},{"location":"infra/proxmox-state/#18-homarr-inventory-services-to-include-on-the-dashboard","title":"18. Homarr inventory \u2014 services to include on the dashboard","text":"<p>Captured here so the eventual Homarr config can be assembled in one pass. Groups follow the existing <code>homepage.*</code> label convention used in compose files.</p>"},{"location":"infra/proxmox-state/#group-infrastructure","title":"Group: <code>infrastructure</code>","text":"Service URL Notes Proxmox UI <code>https://192.168.1.20:8006</code> until LE via Zoraxy lands, see \u00a711b AdGuard Home (CT 102) <code>http://192.168.1.2/</code> (UI on <code>:80</code>) DNS + admin Zoraxy (CT 108) <code>http://192.168.1.4:8000/</code> reverse proxy admin Backrest (CT 103) <code>http://192.168.1.3:9898/</code> backup orchestration, will be fronted via Tinyauth + Zoraxy Pocket-ID (CT 110) <code>https://id.nuclide.systems/</code> new home, 2026-05-20"},{"location":"infra/proxmox-state/#group-network","title":"Group: <code>network</code>","text":"Service URL Notes UDM Pro <code>https://192.168.1.1/</code> UniFi controller D-Link DGS-1210-28P <code>http://192.168.1.10/</code> core L2 switch; host on port 10 UNAS Pro <code>https://192.168.1.31/</code> UniFi NAS"},{"location":"infra/proxmox-state/#group-ops-to-populate-when-ct-109-lands","title":"Group: <code>ops</code> (to populate when CT 109 lands)","text":"Service URL Grafana <code>https://grafana.nuclide.systems</code> Arcane <code>https://arcane.nuclide.systems</code> Dozzle <code>https://dozzle.nuclide.systems</code> Prometheus <code>https://prom.nuclide.systems</code> (gated by Tinyauth) Alertmanager <code>https://alerts.nuclide.systems</code> (gated by Tinyauth)"},{"location":"infra/proxmox-state/#group-apps-subset-long-list-fill-from-existing-homepage-labels-in-optstacks","title":"Group: <code>apps</code> (subset \u2014 long list, fill from existing <code>homepage.*</code> labels in <code>/opt/stacks/*/</code>)","text":"<p>Immich, Nextcloud, Vaultwarden, Karakeep, Memos, Paperless-ngx, n8n, ComfyUI, LobeChat, LiteLLM, Traccar, Gotify, Speaches, Daytona, Searxng, Kroki, etc. Pull display labels and icons from the existing <code>homepage.name=</code> / <code>homepage.icon=</code> values per compose.</p>"},{"location":"infra/proxmox-state/#12-suggested-action-order","title":"12. Suggested action order","text":"<ol> <li>Done 2026-05-20 \u2705:</li> <li>Memory / sizing: CT 101 160 \u2192 32 GiB; CT 102 512 MiB \u2192 1 GiB; HAOS balloon = 4 GiB</li> <li>Protection: CT 102 startup=1+protection; CT 108 startup=2+protection; CT 103/104/105/110/VM100 protection</li> <li>CT 110 (id) built at 192.168.1.5, Pocket-ID duplicated (online SQLite snapshot)</li> <li>ZFS: <code>autotrim=on</code>, <code>atime=off rpool</code></li> <li>Backups: retention set on <code>unas</code></li> <li>APT: duplicate sources removed; host <code>unattended-upgrades</code> deployed (Debian-only)</li> <li>Per-CT u-u: deployed to all 7 CTs with Debian-only allowlist</li> <li>Cron cleanup: removed weekly tteck-LXC-update curl-pipe-bash; removed daily broken <code>apt-get upgrade -y</code>; replaced <code>/root/vm100.sh</code> cron with <code>vm100-watchdog.timer</code> (1 min, lock-aware)</li> <li>Self-updaters: AdGuard (<code>--update</code> flag) Wed 03:30; Zoraxy (GitHub stable releases + rollback) Wed 03:40</li> <li><code>zpool upgrade rpool</code> ran during the audit (enabled <code>redaction_list_spill</code>, <code>raidz_expansion</code>)</li> <li>Today with a maintenance window:</li> <li>Cut Zoraxy over to CT 110 for <code>id.nuclide.systems</code> (single upstream edit; rollback path = revert one line)</li> <li><code>apt full-upgrade</code> (kernel 7.0.0-3 \u2192 7.0.2-5, pve-manager 9.1.11 \u2192 9.1.18) + reboot</li> <li>VM 100 disk options (<code>cache=none</code>, <code>iothread=1</code>) \u2014 \u00a74. Requires VM stop/start.</li> <li>Switch CT 105 from CIFS \u2192 NFS bind-mount (\u00a74). Test UID mapping.</li> <li>Drop the <code>unas_smb</code> storage once CT 105 is migrated.</li> <li>This week:</li> <li>Re-measure CT 104 peak RSS after fixing CT 101 \u2014 likely safe to drop to 48 GiB.</li> <li>Convert CT 105 to unprivileged (backup \u2192 restore as unprivileged).</li> <li>Raise ARC cap to 16 GiB.</li> <li>Probe NFSv4 against UNAS; switch if supported.</li> <li>Wire Postfix relayhost (Gmail/Postmark/your SMTP) so unattended-upgrades + zfs-zed + cron failures actually mail you.</li> <li>Rotate Backrest plan to back up real data (currently still pointed at <code>/media/data-dir</code> \u2014 a 50 KB test file from August 2025); see \u00a77.</li> <li>Medium term:</li> <li>Build CT 109 ops LXC (\u00a716) \u2014 Prometheus + Grafana + Arcane Manager + Dozzle + Homarr + Diun + Tinyauth</li> <li>Migrate Gotify from CT 104 to CT 109 (~30 min of env-var updates)</li> <li>Rotate exposed secrets that appeared in this transcript: Arcane <code>OIDC_CLIENT_SECRET</code>, <code>ENCRYPTION_KEY</code>, <code>JWT_SECRET</code>; Immich <code>IMMICH_API_KEY</code></li> <li>Backrest: enable auth, redesign plans to cover all data tiers (\u00a77), front via Tinyauth for OIDC</li> <li>Next purchase window:</li> <li>Second NVMe \u2192 mirror <code>rpool</code> (\u00a76)</li> <li><code>proxmox-boot-tool init</code> on the new disk</li> <li>2.5 GbE NIC + matching switch port to UNAS for image-gen / backup speed</li> <li>UniFi-managed switch between host and UDM (or accept SNMP-only visibility from UniFi)</li> </ol>"},{"location":"infra/proxmox-state/#19-changes-applied-2026-05-20-session-2","title":"19. Changes applied 2026-05-20 (session 2)","text":""},{"location":"infra/proxmox-state/#optimizations-executed","title":"Optimizations executed","text":"# Item Command / action Result 1 CT 101 protection + boot order <code>pct set 101 -protection 1 -startup order=10</code> \u2705 4 CT 104 memory cap 128\u219248 GiB <code>pct set 104 -memory 49152</code> \u2705 (swap 32\u21928 deferred: still 10.4 GB in use) 5 CT 102 rootfs 2\u21924 GiB <code>pct resize 102 rootfs 4G</code> \u2705 now 28% used A VM 100 disk: cache=none + iothread=1 <code>qm set 100 -scsi0 ...,cache=none,iothread=1</code> + <code>scsihw virtio-scsi-single</code> \u2705 HAOS healthy C Host apt full-upgrade kernel 7.0.0\u21927.0.2-5, pve-manager 9.1.11\u21929.1.18 \u2705 installed; reboot pending"},{"location":"infra/proxmox-state/#pocket-id-migration-completed","title":"Pocket-ID migration completed","text":"<ul> <li><code>id.nuclide.systems</code> Zoraxy proxy cutover confirmed: <code>192.168.1.40:11000</code> \u2192 <code>192.168.1.5:11000</code></li> <li>CT 104 <code>/opt/stacks/pocketid/</code> directory fully removed (data migrated to CT 110 2026-05-20)</li> <li>Backup: <code>/opt/zoraxy/conf/proxy/id.nuclide.systems.config.bak-pre-ct110</code> (keep as rollback)</li> </ul>"},{"location":"infra/proxmox-state/#proxmox-oidc-via-pocket-id","title":"Proxmox OIDC via Pocket-ID","text":"<p>Realm <code>pocket-id</code> added; user <code>fkrebs@nucli.de@pocket-id</code> mapped to Administrator role.</p> <pre><code>pveum realm add pocket-id \\\n --type openid \\\n --issuer-url https://id.nuclide.systems \\\n --client-id 38469e7e-1fff-4841-83a9-74bf38d847eb \\\n --client-key <secret> \\\n --username-claim email \\\n --comment \"Pocket-ID OIDC\"\n\npveum user add fkrebs@nucli.de@pocket-id\npveum aclmod / --users fkrebs@nucli.de@pocket-id --roles Administrator\n</code></pre> <p>OIDC client inserted directly into Pocket-ID SQLite (API key stored as SHA-256 hash \u2014 not reversible):</p> <pre><code>DB: /opt/stacks/pocketid/data/pocket-id.db on CT 110\nTable: oidc_clients\nclient_id: 38469e7e-1fff-4841-83a9-74bf38d847eb\nname: Proxmox VE\ncallback_urls: [\"https://192.168.1.20:8006\"]\n</code></pre> <p>To add future OIDC clients without UI access: <pre><code>python3 -c \"\nimport uuid, secrets, bcrypt, json, datetime\nclient_id = str(uuid.uuid4())\nsecret_plain = secrets.token_urlsafe(32)\nsecret_hash = bcrypt.hashpw(secret_plain.encode(), bcrypt.gensalt(rounds=10)).decode()\nprint(f'id={client_id}')\nprint(f'secret={secret_plain}')\nprint(f'hash={secret_hash}')\n\"\n# Then INSERT into oidc_clients with the hash, use secret_plain in the app config\n# callback_urls and logout_callback_urls are JSON arrays stored as BLOB\n# credentials field is '{}' for standard clients\n</code></pre></p> <p>Note on Pocket-ID API keys: The <code>key</code> column in <code>api_keys</code> stores a SHA-256 hash of the real key (64-char hex). The plaintext key is only shown once at creation time in the UI. If lost, create a new one \u2014 there is no recovery path.</p> <p>Login flow: In PVE web UI, select realm <code>pocket-id</code> at login. You will be redirected to <code>https://id.nuclide.systems</code> for authentication and returned to PVE. The <code>email</code> claim is used as the PVE username.</p>"},{"location":"infra/proxmox-state/#audit-footnote-side-effects-of-this-run","title":"Audit footnote \u2014 side effects of this run","text":"<ul> <li><code>zpool upgrade rpool</code> was executed (not <code>-n</code>). Enabled features: <code>redaction_list_spill</code>, <code>raidz_expansion</code>. Safe on current ZFS version; the pool can no longer be imported by ZFS releases that pre-date these features. No data risk.</li> <li>No other state changes were made by this audit.</li> </ul>"},{"location":"infra/storage/","title":"Storage layout","text":"<p>Verified state as of 2026-05-21. Source of truth for which service lives on which storage class. Earlier revisions of this doc were stale on multiple items (Garage, n8n, Arcane, Karakeep, Pocket-ID, arr-stack, Nextcloud's storage protocol, missing Coder/Gitea). Reconciled via live audit.</p>"},{"location":"infra/storage/#storage-classes","title":"Storage classes","text":"Class Substrate Where Typical use Local zfs (rootfs) NVMe in nuc <code>/opt/stacks/<stack>/</code> on each LXC Tier-1 hot state: databases, SQLite, secret stores, anything latency-sensitive Local zfs (subvol) NVMe in nuc <code>rpool/data/subvol-<NNN>-disk-0</code> per LXC Container rootfs, image cache Docker named volume NVMe in nuc <code>/var/lib/docker/volumes/...</code> on each Docker CT Per-container persistent state managed by Docker UNAS over NFSv3 <code>192.168.1.31</code> <code>/mnt/pve/unas</code> on CTs 101, 103, 104, 105, 111 Bulk media, workspace home dirs, document storage"},{"location":"infra/storage/#what-lives-where-verified-2026-05-21","title":"What lives where (verified 2026-05-21)","text":""},{"location":"infra/storage/#unas-nfs-mntpveunasservices","title":"UNAS NFS \u2014 <code>/mnt/pve/unas/services/...</code>","text":"<p>Bulk + media + non-latency-sensitive app data.</p> Service CT Path on UNAS \u2192 in container Backed up? Notes Immich (uploads, thumbs, derived) 104 <code>services/immich/{encoded-video,profile,thumbs}</code> + <code>media/images</code> + <code>backup/immich</code> Immich own pg_dump \u2192 <code>media/images/db-dumps</code>; no off-host copy Paperless-ngx (documents) 104 <code>media/documents/public/paperless-ngx/{consume,export,library}</code> none Paperless-AI 104 <code>services/paperless-ai</code> \u2192 <code>/app/data</code> none Traccar (logs, config) 104 <code>services/traccar/{logs,traccar.xml}</code> none data dir reverted to local <code>/opt/stacks/traccar/data</code> Memos 104 <code>services/memos</code> \u2192 <code>/var/opt/memos</code> none Arr-stack (Audiobookshelf, Prowlarr, RDTClient, Shelfarr) + media 104 <code>services/arr-stack/*</code> + <code>media/{audiobooks,ebooks,podcasts,Torrents}</code> none already migrated (older doc claimed \"still local\") Gluetun (VPN) 104 <code>services/gluetun/data</code> \u2192 <code>/gluetun</code> none Gitea 111 <code>services/gitea</code> \u2192 <code>/data</code> none new 2026-05-20 Coder workspace home dirs 111 <code>services/coder/<user>/<workspace></code> \u2192 <code>/home/<user></code> none new 2026-05-20 Backrest's jottacloud-mirrored data 103 <code>media/data-dir</code> only rclone \u2192 jottacloud (Backrest) only path with off-host backup"},{"location":"infra/storage/#local-zfs-optstacks","title":"Local zfs \u2014 <code>/opt/stacks/...</code>","text":"<p>Tier-1 state and anything that should NOT be NFS-backed.</p> Service CT Local path \u2192 in container Backed up? Notes Pocket-ID 110 <code>/opt/stacks/pocketid/data</code> \u2192 <code>/app/data</code> none CT 110 has no NFS mount at all. Earlier doc said UNAS \u2014 incorrect. Garage (S3 meta + data) 104 <code>/opt/stacks/shared-db/garage/{data,meta}</code> \u2192 <code>/var/lib/garage/*</code> none moved off NFS 2026-05-19 after WAL-G outage. Stale copy may still exist on UNAS. Shared Postgres (multi-stack) 104 docker named volume <code>shared-db_shared-pgdata</code> WAL-G \u2192 local Garage (configured; outage 2026-05-19 prompted move) Garage itself is single-disk local \u2014 no off-host copy. n8n 104 <code>/opt/stacks/n8n/data</code> \u2192 <code>/home/node/.n8n</code> none back to local after PG migration attempt failed. Stale 408 MB <code>database.sqlite</code> left on UNAS (May 15) \u2014 clean up. n8n now uses <code>shared-postgres</code> as its actual DB. Arcane 104 <code>/opt/stacks/arcane/data</code> \u2192 <code>/app/data</code> none reverted from UNAS to local 2026-05-19 (undocumented before now) Karakeep + Meilisearch 104 <code>/opt/stacks/karakeep/localdata/{data,meili}</code> none reverted from UNAS to local Immich Postgres (pgvector) 104 <code>/opt/stacks/immich/postgres</code> \u2192 <code>/var/lib/postgresql/data</code> Immich pg_dump \u2192 UNAS tier-1; PG demands local disk Homepage 104 <code>/opt/stacks/homepage/{config,icons}</code> none Dozzle 104 <code>/opt/stacks/dozzle/dozzle_data</code> none LiteLLM config 104 <code>/opt/stacks/ai/litellm-config</code> none SearXNG config 104 <code>/opt/stacks/ai/searxng</code> none Flaresolverr 104 <code>/var/lib/flaresolver</code> none AdGuard / Zoraxy / DNS / Shepard / Backrest binaries 102/108/103/101 local zfs only none Backrest itself has no self-backup Vaultwarden (attachments + key material) 104 <code>/opt/stacks/vaultwarden/data</code> \u2192 <code>/data</code> none moved off NFS 2026-05-22; DB is on CT 113 postgres; stale <code>db.sqlite3</code> deleted Nextcloud config + sidecars 105 local zfs (CT rootfs) none app data on NFS \u2014 see below"},{"location":"infra/storage/#garage-s3-buckets-on-local-zfs-ct-104","title":"Garage S3 buckets (on local zfs, CT 104)","text":"Bucket Access key Use Public URL <code>lobe-files</code> GK55210\u2026 (from <code>.env</code>) LobeHub file uploads + WAL-G PG backups internal only <code>chat-artifacts</code> GK50bfc\u2026 AI chat output artefacts (images, reports, SVG) <code>https://chat-artifacts.s3.nuclide.systems/<key></code>"},{"location":"infra/storage/#docker-named-volumes","title":"Docker named volumes","text":"<p>Local on <code>/var/lib/docker/volumes/</code>. Mostly databases and caches.</p> Volume Service / CT Backed up? <code>shared-db_shared-pgdata</code> shared-postgres / 104 WAL-G \u2192 local Garage <code>paperless-ngx_pgdata</code>, <code>_redisdata</code>, <code>_data</code> paperless-ngx / 104 none <code>lobe-postgres</code> + <code>lobe-redis</code> lobehub / 104 none <code>nuc-ai-core_rustfs-data</code> lobehub stack / 104 none <code>pgadmin-data</code> pgadmin / 104 none (regenerable) <code>immich_model-cache</code> immich-ml / 104 regenerable <code>coder-db</code> + <code>gitea-db</code> (docker volume <code>coder-db</code>) / 111 none (orphaned) <code>daytona-minimal_db_data</code> + runner anon vol / 104 none \u2014 clean up post-decommission"},{"location":"infra/storage/#nextcloud-ct-105-data-on-nfs","title":"Nextcloud (CT 105) \u2014 data on NFS","text":"<p>Nextcloud AIO mounts <code>/mnt/pve/unas/services/nextcloud</code> via NFSv3 (same share as all other CTs). Migrated from CIFS on 2026-05-22 after the CIFS mount caused a crash-loop. The Postgres + Redis sidecars stay on local docker volumes.</p>"},{"location":"infra/storage/#backup-reality","title":"Backup reality","text":"<p>There is effectively no off-host backup of service data. The only configured Backrest plan covers <code>/mnt/pve/unas/media/data-dir \u2192 jottacloud</code> \u2014 i.e., Backrest backs up one specific UNAS path, not the services that write to UNAS.</p> <p>Coverage:</p> <ul> <li>Immich pg_dump \u2192 UNAS (same filer; survives container loss, not filer loss)</li> <li>shared-postgres WAL-G \u2192 Garage (same host; survives container loss, not disk loss)</li> <li>Everything else: zero</li> </ul> <p>This is a known gap. Plans: 1. Extend Backrest plans to snapshot tier-1 paths (Pocket-ID sqlite, Vaultwarden data, Gitea repos, Coder workspace homes) to jottacloud nightly. 2. Once the second NVMe lands (see <code>infra/proxmox-state.md</code> \u00a76), mirror <code>rpool</code> so a single disk death doesn't take everything.</p>"},{"location":"infra/storage/#drift-cleanup-todo","title":"Drift cleanup TODO","text":"<ul> <li>[x] ~~Remove stale <code>services/n8n/database.sqlite</code> from UNAS~~ \u2014 gone (verified 2026-05-22)</li> <li>[x] ~~Remove stale <code>services/shared-db/garage/</code> copy from UNAS~~ \u2014 gone (verified 2026-05-22)</li> <li>[x] ~~Remove orphaned <code>daytona-minimal_db_data</code> Docker volume on CT 104~~ \u2014 gone (verified 2026-05-22)</li> <li>[x] ~~Vaultwarden <code>data/</code> on UNAS~~ \u2014 migrated to local zfs 2026-05-22. Stale NFS copy at <code>services/vaultwarden/</code> can be cleaned up.</li> <li>[x] ~~Add a Backrest plan for Pocket-ID DB~~ \u2014 done 2026-05-22. Pre-backup hook runs <code>pocket-id export</code> + copies keys to UNAS staging; <code>services-backup-plan</code> snapshots staging \u2192 jottacloud nightly.</li> </ul>"},{"location":"infra/volumes/","title":"Volume Mounts \u2014 NUC 14 Docker Stacks","text":"<p>Every bind mount and named volume across all running containers. Last updated: May 16, 2026</p>"},{"location":"infra/volumes/#legend","title":"Legend","text":"Column Meaning Type <code>bind</code> = host directory, <code>volume</code> = docker named volume, <code>tmpfs</code> = memory Source Host path (bind) or volume name (volume) Container Mount point inside container UNAS <code>/mnt/pve/unas/services/</code> target (migrated or planned)"},{"location":"infra/volumes/#infrastructure","title":"Infrastructure","text":""},{"location":"infra/volumes/#arcane-arcane","title":"Arcane \u2014 <code>arcane</code>","text":"Type Source Container UNAS bind <code>/var/run/docker.sock</code> <code>/var/run/docker.sock</code> \u2014 bind <code>/opt/stacks</code> <code>/app/data/projects</code> \u2014 bind <code>/mnt/pve/unas/services/arcane</code> <code>/app/data</code> \u2705 Migrated ~~volume~~ ~~<code>arcane_arcane-data</code>~~ ~~<code>/app/data</code>~~ \ud83d\uddd1\ufe0f removed"},{"location":"infra/volumes/#dozzle-dozzle","title":"Dozzle \u2014 <code>dozzle</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/dozzle/dozzle_data</code> <code>/data</code> (optional, ephemeral) bind <code>/var/run/docker.sock</code> <code>/var/run/docker.sock</code> \u2014"},{"location":"infra/volumes/#homepage-homepage","title":"Homepage \u2014 <code>homepage</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/homepage/config</code> <code>/app/config</code> (keep in stack dir) bind <code>/opt/stacks/homepage/icons</code> <code>/app/public/icons</code> (keep in stack dir) bind <code>/var/run/docker.sock</code> <code>/var/run/docker.sock</code> \u2014"},{"location":"infra/volumes/#security","title":"Security","text":""},{"location":"infra/volumes/#pocket-id-pocketid","title":"Pocket ID \u2014 <code>pocketid</code>","text":"<p>CT 110, not CT 104. Pocket-ID migrated off CT 104 to its own LXC on 2026-05-20.</p> Type Source Container Notes bind <code>/opt/stacks/pocketid/data</code> <code>/app/data</code> Local zfs on CT 110 \u2014 no UNAS mount"},{"location":"infra/volumes/#vaultwarden-vaultwarden","title":"Vaultwarden \u2014 <code>vaultwarden</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/vaultwarden</code> <code>/data</code> \u2705 Already on UNAS bind <code>/etc/localtime</code> <code>/etc/localtime</code> \u2014 bind <code>/etc/timezone</code> <code>/etc/timezone</code> \u2014"},{"location":"infra/volumes/#media-immich","title":"Media \u2014 Immich","text":""},{"location":"infra/volumes/#immich-server-immich_server","title":"Immich Server \u2014 <code>immich_server</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/immich/encoded-video</code> <code>/usr/src/app/upload/encoded-video</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/services/immich/profile</code> <code>/usr/src/app/upload/profile</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/services/immich/thumbs</code> <code>/usr/src/app/upload/thumbs</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/media/images</code> <code>/usr/src/app/upload</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/backup/immich</code> <code>/usr/src/app/upload/backups</code> \u2705 Already on UNAS volume <code>7d25f4ac...</code> (anonymous) <code>/data</code> (unknown, check)"},{"location":"infra/volumes/#immich-ml-immich_machine_learning","title":"Immich ML \u2014 <code>immich_machine_learning</code>","text":"Type Source Container UNAS volume <code>immich_model-cache</code> <code>/cache</code> (cache, regenerable) bind <code>/dev/bus/usb</code> <code>/dev/bus/usb</code> \u2014"},{"location":"infra/volumes/#immich-postgres-immich_postgres","title":"Immich Postgres \u2014 <code>immich_postgres</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/immich/postgres</code> <code>/var/lib/postgresql/data</code> \u23f3 Plan: <code>immich/db</code>"},{"location":"infra/volumes/#media-downloads-arr-stack","title":"Media \u2014 Downloads / Arr Stack","text":"<p>All behind gluetun VPN.</p>"},{"location":"infra/volumes/#rdtclient-rdtclient","title":"RDTClient \u2014 <code>rdtclient</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/arr-stack/rdtclient/config</code> <code>/data/db</code> \u23f3 Plan: <code>arr-stack/rdtclient</code> bind <code>/opt/stacks/arr-stack/media/Torrents</code> <code>/data/downloads</code> \u23f3 Plan: <code>arr-stack/torrents</code>"},{"location":"infra/volumes/#prowlarr-prowlarr","title":"Prowlarr \u2014 <code>prowlarr</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/arr-stack/prowlarr</code> <code>/config</code> \u23f3 Plan: <code>arr-stack/prowlarr</code>"},{"location":"infra/volumes/#audiobookshelf-audiobookshelf","title":"Audiobookshelf \u2014 <code>audiobookshelf</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/arr-stack/audiobookshelf</code> <code>/config</code> \u23f3 Plan: <code>arr-stack/audiobookshelf</code> bind <code>/opt/stacks/arr-stack/media/audiobooks</code> <code>/audiobooks</code> \u23f3 Plan: <code>arr-stack/audiobooks</code> bind <code>/opt/stacks/arr-stack/media/ebooks</code> <code>/ebooks</code> \u23f3 Plan: <code>arr-stack/ebooks</code> bind <code>/opt/stacks/arr-stack/media/podcasts</code> <code>/podcasts</code> \u23f3 Plan: <code>arr-stack/podcasts</code>"},{"location":"infra/volumes/#shelfarr-shelfarr","title":"ShelfArr \u2014 <code>shelfarr</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/arr-stack/shelfarr/storage</code> <code>/rails/storage</code> \u23f3 Plan: <code>arr-stack/shelfarr</code> bind <code>/opt/stacks/arr-stack/media/audiobooks</code> <code>/audiobooks</code> (shared) bind <code>/opt/stacks/arr-stack/media/ebooks</code> <code>/ebooks</code> (shared) bind <code>/opt/stacks/arr-stack/media/Torrents</code> <code>/downloads</code> (shared)"},{"location":"infra/volumes/#flaresolverr-flaresolverr","title":"Flaresolverr \u2014 <code>flaresolverr</code>","text":"Type Source Container UNAS bind <code>/var/lib/flaresolver</code> <code>/config</code> (cache, regenerable)"},{"location":"infra/volumes/#ai-stack","title":"AI Stack","text":""},{"location":"infra/volumes/#litellm-litellm","title":"LiteLLM \u2014 <code>litellm</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/ai/litellm-config</code> <code>/app/config</code> (keep in stack dir)"},{"location":"infra/volumes/#litellm-db-nuc-ai-core-litellm-db-1","title":"LiteLLM DB \u2014 <code>nuc-ai-core-litellm-db-1</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/ai/postgres_data</code> <code>/var/lib/postgresql/data</code> \u23f3 Plan: <code>ai/litellm-db</code>"},{"location":"infra/volumes/#lobehub-decommissioned-2026-05-26","title":"~~LobeHub~~ \u2014 DECOMMISSIONED 2026-05-26","text":"<p>LobeChat compose renamed <code>.DECOMMISSIONED-lobehub-2026-05-26.yml</code>. Volumes below are orphaned \u2014 clean up after confirming no data needed.</p> Type Source Container UNAS bind <code>/opt/stacks/ai/lobehub/data</code> <code>/var/lib/postgresql/data</code> \ud83d\uddd1\ufe0f orphaned \u2014 decommissioned"},{"location":"infra/volumes/#lobehub-redis-decommissioned-2026-05-26","title":"~~LobeHub Redis~~ \u2014 DECOMMISSIONED 2026-05-26","text":"Type Source Container UNAS volume <code>nuc-ai-core_redis_data</code> <code>/data</code> \ud83d\uddd1\ufe0f orphaned \u2014 decommissioned"},{"location":"infra/volumes/#lobehub-rustfs-decommissioned-2026-05-26","title":"~~LobeHub RustFS~~ \u2014 DECOMMISSIONED 2026-05-26","text":"Type Source Container UNAS volume <code>nuc-ai-core_rustfs-data</code> <code>/data</code> \ud83d\uddd1\ufe0f orphaned \u2014 decommissioned"},{"location":"infra/volumes/#searxng-searxng","title":"SearXNG \u2014 <code>searxng</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/ai/searxng</code> <code>/etc/searxng</code> (keep in stack dir) volume <code>bb5bb182...</code> (anonymous) <code>/var/cache/searxng</code> (cache, regenerable)"},{"location":"infra/volumes/#searxng-redis-redis-searxng","title":"SearXNG Redis \u2014 <code>redis-searxng</code>","text":"Type Source Container UNAS volume <code>f9c3c386...</code> (anonymous) <code>/data</code> (ephemeral, stay)"},{"location":"infra/volumes/#saia-image-proxy-nuc-ai-core-saia-image-proxy-1","title":"SAIA Image Proxy \u2014 <code>nuc-ai-core-saia-image-proxy-1</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/ai/saia-image-proxy</code> <code>/app</code> (keep in stack dir)"},{"location":"infra/volumes/#crawl4ai-crawl4ai-mcp","title":"Crawl4AI \u2014 <code>crawl4ai-mcp</code>","text":"Type Source Container UNAS tmpfs <code>/dev/shm</code> <code>/dev/shm</code> \u2014"},{"location":"infra/volumes/#documents","title":"Documents","text":""},{"location":"infra/volumes/#paperless-ngx-webserver-paperless-ngx-webserver-1","title":"Paperless-ngx Webserver \u2014 <code>paperless-ngx-webserver-1</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/media/documents/public/paperless-ngx/consume</code> <code>/usr/src/paperless/consume</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/media/documents/public/paperless-ngx/export</code> <code>/usr/src/paperless/export</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/media/documents/public/paperless-ngx/library</code> <code>/usr/src/paperless/media</code> \u2705 Already on UNAS volume <code>paperless-ngx_data</code> <code>/usr/src/paperless/data</code> \u23f3 Plan: <code>paperless-ngx/data</code>"},{"location":"infra/volumes/#paperless-ngx-db-paperless-ngx-db-1","title":"Paperless-ngx DB \u2014 <code>paperless-ngx-db-1</code>","text":"Type Source Container UNAS volume <code>paperless-ngx_pgdata</code> <code>/var/lib/postgresql/data</code> \u23f3 Plan: <code>paperless-ngx/pgdata</code>"},{"location":"infra/volumes/#paperless-ngx-broker-paperless-ngx-broker-1","title":"Paperless-ngx Broker \u2014 <code>paperless-ngx-broker-1</code>","text":"Type Source Container UNAS volume <code>paperless-ngx_redisdata</code> <code>/data</code> (ephemeral, stay)"},{"location":"infra/volumes/#paperless-ai-paperless-ai","title":"Paperless AI \u2014 <code>paperless-ai</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/paperless-ai</code> <code>/app/data</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#productivity-bookmarks","title":"Productivity & Bookmarks","text":""},{"location":"infra/volumes/#memos-memos","title":"Memos \u2014 <code>memos</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/memos</code> <code>/var/opt/memos</code> \u2705 Migrated ~~bind~~ ~~<code>/opt/stacks/memos/data</code>~~ ~~<code>/var/opt/memos</code>~~ \ud83d\uddd1\ufe0f replaced"},{"location":"infra/volumes/#karakeep-karakeep","title":"Karakeep \u2014 <code>karakeep</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/karakeep/data</code> <code>/data</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#karakeep-meilisearch-karakeep_meilisearch","title":"Karakeep Meilisearch \u2014 <code>karakeep_meilisearch</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/karakeep/meilisearch</code> <code>/meili_data</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#automation","title":"Automation","text":""},{"location":"infra/volumes/#n8n-n8n","title":"n8n \u2014 <code>n8n</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/n8n</code> <code>/home/node/.n8n</code> \u2705 Migrated bind <code>/opt/stacks/n8n/hooks.js</code> <code>/home/node/hooks.js</code> (keep in stack dir) ~~volume~~ ~~<code>n8n_n8n_storage</code>~~ ~~<code>/home/node/.n8n</code>~~ \ud83d\uddd1\ufe0f removed"},{"location":"infra/volumes/#devops","title":"DevOps","text":""},{"location":"infra/volumes/#daytona-api-daytona-minimal-api-1","title":"Daytona API \u2014 <code>daytona-minimal-api-1</code>","text":"<p>No DB-specific mounts needed (connects via env vars).</p>"},{"location":"infra/volumes/#daytona-db-daytona-minimal-db-1","title":"Daytona DB \u2014 <code>daytona-minimal-db-1</code>","text":"Type Source Container UNAS volume <code>daytona-minimal_db_data</code> <code>/var/lib/postgresql/data</code> \u23f3 Plan: <code>daytona/db</code>"},{"location":"infra/volumes/#daytona-runner-daytona-minimal-runner-1","title":"Daytona Runner \u2014 <code>daytona-minimal-runner-1</code>","text":"Type Source Container UNAS volume <code>9416da14...</code> (anonymous) <code>/var/lib/docker</code> (runner state, stay) bind <code>/var/run/docker.sock</code> <code>/var/run/docker.sock</code> \u2014"},{"location":"infra/volumes/#tracking","title":"Tracking","text":""},{"location":"infra/volumes/#traccar-traccar","title":"Traccar \u2014 <code>traccar</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/traccar/data</code> <code>/opt/traccar/data</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/services/traccar/logs</code> <code>/opt/traccar/logs</code> \u2705 Already on UNAS bind <code>/mnt/pve/unas/services/traccar/traccar.xml</code> <code>/opt/traccar/conf/traccar.xml</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#vpn","title":"VPN","text":""},{"location":"infra/volumes/#gluetun-vpn_gluetun","title":"Gluetun \u2014 <code>vpn_gluetun</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/gluetun/data</code> <code>/gluetun</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#shared-infrastructure","title":"Shared Infrastructure","text":""},{"location":"infra/volumes/#shared-postgresql-shared-postgres","title":"Shared PostgreSQL \u2014 <code>shared-postgres</code>","text":"Type Source Container Notes volume <code>shared-db_shared-pgdata</code> <code>/var/lib/postgresql/data</code> \ud83d\uddc4\ufe0f Local NVMe (not NFS)"},{"location":"infra/volumes/#garage-s3-garage","title":"Garage S3 \u2014 <code>garage</code>","text":"<p>Moved off NFS to local zfs on 2026-05-19 after a WAL-G outage. Stale copy at <code>services/shared-db/garage/</code> on UNAS may still exist \u2014 clean up.</p> Type Source Container Notes bind <code>/opt/stacks/shared-db/garage/data</code> <code>/var/lib/garage/data</code> S3 object data \u2014 local NVMe bind <code>/opt/stacks/shared-db/garage/meta</code> <code>/var/lib/garage/meta</code> S3 metadata (LMDB) \u2014 local NVMe"},{"location":"infra/volumes/#stacks-not-running-compose-config-only","title":"Stacks Not Running (Compose Config Only)","text":""},{"location":"infra/volumes/#streamio-streamio","title":"Streamio \u2014 <code>streamio</code>","text":"Type Source Container UNAS bind <code>/mnt/pve/unas/services/stremio</code> <code>/root/.stremio-server</code> \u2705 Already on UNAS"},{"location":"infra/volumes/#qdrant-qdrant_scientific","title":"Qdrant \u2014 <code>qdrant_scientific</code>","text":"Type Source Container UNAS bind <code>/opt/stacks/qdrant/qdrant_storage</code> <code>/qdrant/storage</code> \u23f3 Plan: <code>qdrant/</code>"},{"location":"infra/volumes/#summary-migration-status","title":"Summary: Migration Status","text":"Status Count Services \u2705 Already on UNAS 9 gluetun, immich(4), karakeep(2), ntfy(2), paperless-ai, stremio, traccar(3), vaultwarden, paperless-docs*(3) \u2705 Migrated (Phase 1) 3 memos, arcane, n8n \ud83d\udd37 Shared infrastructure (local) 2 shared-postgres (local volume), garage (local NVMe \u2014 moved off UNAS 2026-05-19) \ud83d\udccb Own CT, local only 1 pocketid (CT 110, no UNAS) \u23f3 Phase 2 planned (PG consolidation) 4 immich \u2192 shared-postgres, paperless \u2192 shared-postgres, daytona \u2192 shared-postgres, litellm \u2192 shared-postgres \ud83d\uddd1\ufe0f Decommissioned 2026-05-26 3 lobehub-db, lobe-redis, lobe-rustfs (LobeChat removed) \u23f3 Phase 3 planned 1 arr-stack*(8 mounts) \ud83d\udccb Keep local ~5 homepage, dozzle, litellm-config, searxng-config, saia-image-proxy \ud83e\udde0 Cache (stay) ~5 immich_model-cache, paperless-ngx redis, lobe-redis, searxng-redis, flaresolverr, daytona-runner"},{"location":"security/audit-claude-code-meta/","title":"Meta-audit \u2014 the Claude Code session that performed these audits","text":"<p>A self-audit, completing the \"audit the auditor\" loop. Honest accounting of what this assistant has seen during the audit / analysis work and where that data went.</p>"},{"location":"security/audit-claude-code-meta/#scope","title":"Scope","text":"<p>This audit covers the Claude Code session running on the Proxmox host (<code>/root/.claude/projects/-root/</code>) on 2026-05-20 and 2026-05-21, from the message <code>> finish up for today. last task: the attached conversation was run on our lobehub\u2026</code> onward. The work product of that session is the three audit reports in this directory.</p>"},{"location":"security/audit-claude-code-meta/#what-the-assistant-processed","title":"What the assistant processed","text":"Input Content Source Transcript A <code>75fb06e8-Lumen_TR004_Test_Run_Analysis.json</code> (255 KB, 60 messages, model <code>qwen3.5-397b-a17b</code>) User-uploaded Transcript B <code>ec5fba44-Analyzing_LUMEN_TR004_Test_Data.json</code> (205 KB, 41 messages, model <code>qwen3-coder-30b-a3b-instruct</code>) User-uploaded SSH + <code>pct exec</code> on <code>nuc</code> container env vars, <code>docker ps</code>, <code>proxy_server_config.yaml</code>, <code>gateway_tokens.json</code> (first chars only) Live homelab inspection <code>https://docs.hpc.gwdg.de/services/ai-services/saia/index.html</code> SAIA public docs <code>WebFetch</code> (Anthropic-mediated) <p>The Lumen transcripts contain DLR-context identifiers (<code>LUMEN</code>, <code>P3-Lampoldshausen</code>, <code>LOX/LCH4</code>, <code>TR-003/004/006</code>), anomaly metadata (<code>fuel turbopump vibration spike at t=8s</code>, <code>~12 g rms</code>), and chemistry/test-bench naming. The assistant quoted portions of these in chat output and in the audit reports.</p>"},{"location":"security/audit-claude-code-meta/#where-that-data-went","title":"Where that data went","text":"<pre><code>flowchart LR\n user([\"Operator\"]) --> cc[\"Claude Code CLI<br/>on Proxmox host\"]\n cc -->|every prompt + tool result<br/>+ assistant turn| api[(api.anthropic.com<br/>Anthropic API)]\n cc -->|ssh / pct / docker via shell| home[\"nuclide.systems<br/>(LAN-only)\"]\n cc -->|WebFetch SAIA public docs| saiadocs[(docs.hpc.gwdg.de<br/>via Anthropic proxy)]\n cc -->|FLUX image-gen MCP| flux[(image-gen MCP backend<br/>Anthropic-side)]\n\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n classDef external fill:#5a4a2a,stroke:#cc8,color:#fec\n class api,flux leak\n class home ok\n class saiadocs external</code></pre> <p>This Claude Code session is itself an outbound channel. Every assistant turn, including those that quoted Lumen transcript content, was a request to <code>api.anthropic.com</code>. The full session is:</p> Channel What flows Trust Claude API (<code>api.anthropic.com</code>) Full prompts + tool results + audit text the assistant produced. ~60+ turns this session. Includes verbatim LUMEN identifiers in audit body, snippets of transcript content, snippets of LiteLLM config, snippets of <code>gateway_tokens.json</code> (one token prefix), Vaultwarden plaintext secret (handed back to operator, then echoed in subsequent context). Commercial vendor (Anthropic). Operator chose to use Claude Code; informed-consent leak. <code>WebFetch</code> (also via Anthropic) The URL <code>https://docs.hpc.gwdg.de/services/ai-services/saia/index.html</code> was fetched server-side by Anthropic, content returned to the assistant. Outbound from Anthropic to GWDG (public doc; no sensitive payload). Public web fetch, no sensitive outbound payload. Image-gen MCP Prompt strings (containing brief LUMEN reference in one attempted illustration: \"Python with LUMEN identifiers leaked to commercial cloud\"). Backend is Anthropic-side per the MCP integration. Same Anthropic boundary. SSH / <code>pct</code> / shell Authenticated to operator's own infrastructure. \u2705 LAN, no leak. Sub-agents (Pocket-ID audit, CT inventory, etc.) Ran on Anthropic's infrastructure \u2014 each got the relevant context for its narrow task. Same trust boundary as the main session. Anthropic."},{"location":"security/audit-claude-code-meta/#specific-data-this-assistant-sent-to-anthropic","title":"Specific data this assistant sent to Anthropic","text":"<p>Across the audit work in this session, the following types of data were in the prompt/response stream to <code>api.anthropic.com</code>:</p> <ul> <li>Lumen test analysis identifiers: <code>LUMEN TR-004</code>, <code>P3-Lampoldshausen</code>, <code>LOX/LCH4</code>, <code>Fuel Turbopump Vibration Spike at t=8.0s</code>, <code>bearing replaced</code>, test campaign dates.</li> <li>Tool-call inventories (counts, MCP names, plugin identifiers) from both transcripts.</li> <li>A snippet of <code>gateway_tokens.json</code> content (the SUB UUID and one truncated token preview) when auditing the gateway.</li> <li>The plaintext Vaultwarden secret <code>HvUJYUxCGLrSoDY2g24NJI5anM70xXwcsd3e3itJ</code> (issued by the agent, surfaced in chat for the operator to save \u2014 operator saved it to Vaultwarden, but it remains in this session's context).</li> <li>Pocket-ID OIDC client identifiers (UUID prefixes, names, callback URLs).</li> <li>Internal IPs / CT layout (192.168.1.x).</li> <li>Snippet of LiteLLM <code>proxy_server_config.yaml</code> showing fallback chains.</li> </ul>"},{"location":"security/audit-claude-code-meta/#threat-model-honesty","title":"Threat model honesty","text":"<p>The act of running this audit using Claude Code is itself a higher-bandwidth leak than the leak it audited. Hours of conversation about LUMEN-context data went to Anthropic; the original Lumen TR-004 cloud-sandbox breach was ~1,500 lines of Python.</p> <p>This is a deliberate trade-off: - \u2705 Anthropic has stronger contractual guarantees than <code>codesandbox.io</code> (zero-data-retention API plans exist; Anthropic publishes a clear DPA). - \u2705 The operator chose Claude Code consciously, knowing all prompts are API-bound. - \u274c It is not free. The work product is excellent; the data exposure is real.</p>"},{"location":"security/audit-claude-code-meta/#mitigations-for-future-audit-work","title":"Mitigations for future audit work","text":"Option Pros Cons Continue using Claude Code Best-in-class tooling, doctrine + memory carry across sessions, sub-agents in parallel All audit context goes to Anthropic Use a local-only agent (e.g. <code>gpt-oss-120b</code> in workspace via Coder) Zero off-host audit data Far weaker capability; no rich tool-use; no memory; manual orchestration Air-gap the work on a non-internet-connected host Strongest privacy No web fetches; no API; doctrine doesn't bootstrap; pace ~10x slower Hybrid: Claude Code for general infra; air-gapped local agent for Lumen-specific prompts Best of both Process discipline required; operator decides routing per chat <p>For LUMEN-specific deep analysis going forward, the hybrid path is the rational one: do schema/topology/code work in Claude Code (no Lumen payload needed); do data-touching analysis in a Coder workspace with the <code>mcp-sandbox</code> template + a local LLM. The S3 artifact hub (planned) enables both surfaces to share visualization output without ever exposing raw data.</p>"},{"location":"security/audit-claude-code-meta/#recommendations-as-session-policy","title":"Recommendations as session policy","text":"<ol> <li>For any chat that will reference Lumen/Shepard data, switch to the air-gapped / local-LLM path. Don't mix.</li> <li>Strip secrets from chat replies aggressively. The Vaultwarden plaintext secret didn't need to be displayed; the agent could have written it to a tmpfs file and pointed the operator at it.</li> <li>Tag this and future audit transcripts in <code>/docs/security/transcripts/</code> with a \"contains DLR-context data\" header so the policy is visible to anyone reviewing them.</li> </ol> <p>See also: <code>data-leak-audit-comparison.md</code> (the cross-session comparison), <code>data-leak-audit-2026-05-20-tr004-cloud-sandbox.md</code>, <code>data-leak-audit-2026-05-21-tr004-artifacts.md</code>.</p>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/","title":"Data-leak audit \u2014 2026-05-20 \u00b7 <code>Lumen TR-004 Test Run Analysis</code>","text":"<p>Conversation: <code>75fb06e8-Lumen_TR004_Test_Run_Analysis.json</code> LobeHub session model: <code>qwen3.5-397b-a17b</code> (21 assistant turns) Total messages: 60 (2 user \u00b7 21 assistant \u00b7 37 tool) Auditor: agent doctrine-driven scan, 2026-05-20</p>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#tldr","title":"TL;DR","text":"<p>Verdict: SENSITIVE DATA LEFT THE HOMELAB to an unapproved third party. The breach channel was <code>lobe-cloud-sandbox</code>, not the LLM inference.</p> <p>Two off-host data flows, with very different trust profiles:</p> <ol> <li>LLM inference via SAIA / GWDG \u2014 21 assistant turns sent the full conversation context (incl. Shepard search results) to SAIA via LiteLLM. NOT a leak in this context \u2014 SAIA is operated by GWDG (German academic computing center, G\u00f6ttingen), vetted by DLR and integrated with the DLR IdP federation. For LUMEN (DLR engine programme) test data, SAIA is an approved partner.</li> <li><code>lobe-cloud-sandbox</code> code execution \u2014 THE BREACH. 9 calls sent Python source code (with explicit <code>LUMEN</code>, <code>P3-Lampoldshausen</code>, <code>LOX/LCH4</code> references, anomaly timings, and synthetic-but-derived-from-real timeseries) to <code>api.lobehub.com</code> + codesandbox.io \u2014 commercial third parties, not approved for DLR data. 4 <code>exportFile</code> calls pulled generated images back. Criticality: HIGH \u2014 proprietary aerospace IP in plaintext executable code, sent to commercial cloud.</li> </ol> <p>Shepard data fetches themselves stayed on LAN (<code>shepard-api.nuclide.systems</code>), but the results were re-emitted to SAIA (approved) AND to the cloud sandbox (NOT approved).</p>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#conversation-footprint","title":"Conversation footprint","text":"Channel Calls Destination Trust <code>shepard</code> MCP (<code>list_data_objects</code>, <code>get_data_object</code>, <code>list_lab_journal</code>, etc.) 27 <code>shepard-api.nuclide.systems</code> (CT 101) \u2705 LAN <code>lobe-cloud-sandbox</code> (<code>executeCode</code> + <code>exportFile</code>) 9 <code>api.lobehub.com</code> + codesandbox.io \u274c unapproved third party <code>lobe-agent-documents</code> (<code>listDocuments</code>) 1 LobeHub local (in-container) \u2705 LAN LLM inference (<code>qwen3.5-397b-a17b</code>) 21 SAIA (GWDG) via LiteLLM \u2705 approved partner (DLR-vetted, IdP-federated)"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#data-flow","title":"Data flow","text":"<pre><code>flowchart LR\n user([\"You \u00b7 browser\"]) -->|HTTPS via Zoraxy| lobe[\"LobeHub \u00b7 CT 104\"]\n lobe -->|MCP, internal| shep[Shepard API \u00b7 CT 101]\n shep -->|test data, lab notes,<br/>investigation records| lobe\n lobe -->|prompt + Shepard results<br/>+ tool messages| litellm[LiteLLM proxy \u00b7 CT 104]\n litellm -.->|qwen3.5-397b-a17b<br/>21 inference calls| saia[(SAIA / GWDG<br/>academic provider)]\n lobe -.->|Python source + filenames<br/>9 calls| sbx[(LobeHub Cloud Sandbox<br/>api.lobehub.com<br/>+ codesandbox.io)]\n sbx -.->|4 generated PNGs<br/>back to LobeHub| lobe\n\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n class saia,sbx leak\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n class shep,lobe,litellm,user ok</code></pre>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#what-specifically-was-sent-where","title":"What specifically was sent where","text":""},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#to-saia-approved-partner-llm-inference-21-calls","title":"To SAIA (approved partner \u2014 LLM inference, 21 calls)","text":"<ul> <li>Both user prompts verbatim.</li> <li>All Shepard tool outputs \u2014 collections found, data-object IDs, lab journal entries for TR-004 (anomaly time, fuel turbopump vibration values, chamber pressure profile, mixture ratio, test bench naming).</li> <li>The model's own intermediate reasoning.</li> <li>The Python code it wrote before sending to the sandbox.</li> <li>Sensitive strings: <code>LUMEN TR-004</code>, <code>LOX/LCH4</code>, <code>P3-Lampoldshausen</code>, <code>Fuel Turbopump Vibration Spike at t=8.0s</code>, <code>bearing replaced</code>, <code>TR-003 \u2192 TR-004 \u2192 TR-006 campaign</code>.</li> <li>Routed through LiteLLM (<code>ai.nuclide.systems</code>, CT 104) which has SAIA registered as a backend; chain logic terminates at SAIA for free academic models. See <code>services/litellm.md</code>.</li> </ul>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#to-lobehub-cloud-sandbox-unapproved-9-executecode-calls","title":"To LobeHub Cloud Sandbox (UNAPPROVED \u2014 9 <code>executeCode</code> calls)","text":"<ul> <li>~1,500 lines of cumulative Python source containing the same identifiers.</li> <li>Inline JSON dumps of synthetic timeseries (vibration g_rms arrays, chamber-pressure curves, gimbal angles) constructed to match the real TR-004 profile from Shepard.</li> <li>4 generated PNGs (the dashboards) existed on the cloud sandbox's filesystem long enough to be exported back.</li> </ul>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#stayed-on-lan","title":"Stayed on LAN","text":"<ul> <li>The actual raw Shepard sensor timeseries (<code>get_data_object</code> returned data IDs; what reached SAIA was the model's interpretation/summary, not raw buffers).</li> <li>The <code>www.dlr.de</code> URLs in chat are just citations, no fetch was triggered.</li> </ul>"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#criticality-matrix","title":"Criticality matrix","text":"Channel Sensitivity Likelihood Trust Verdict SAIA inference (via LiteLLM) High (proprietary test analysis) 100% (every chat) \u2705 approved partner (DLR-vetted) OK lobe-cloud-sandbox High (Python referencing program data) 100% in this chat \u274c unapproved (commercial cloud) BREACH Other LobeHub providers (Gemini/Cerebras/Mistral) High Latent \u2014 fallback only \u274c commercial MEDIUM (latent risk) Shepard MCP fetches Internal only Every related chat \u2705 LAN LOW Stale <code>DAYTONA_API_KEY</code> None Never used n/a trivial cleanup"},{"location":"security/data-leak-audit-2026-05-20-tr004-cloud-sandbox/#mitigations-ranked-by-impact-effort","title":"Mitigations (ranked by impact \u00f7 effort)","text":"<ol> <li>Disable <code>lobe-cloud-sandbox</code> in LobeHub (highest single-step risk reduction). Use the <code>mcp-sandbox</code> Coder workspace template instead \u2014 already built, ephemeral, GPU-passthrough, sci-stack pre-baked, all local. Effort: 10 min.</li> <li>Pin LobeHub to LiteLLM only. Remove per-provider <code>*_API_KEY</code> env vars. Effort: 15 min.</li> <li>Add a local-LLM route to LiteLLM for sensitive workloads (qwen3-coder-30b via ollama/vllm on Arc). Effort: 1\u20132 h.</li> <li>Tag chats by sensitivity, enforce model routing. Effort: research first.</li> <li>Network egress firewall on UDM blocking <code>api.cerebras.ai</code> + <code>api.lobehub.com</code> + <code>codesandbox.io</code>. Effort: 30 min.</li> <li>Remove stale <code>DAYTONA_API_KEY</code> from LobeHub env. Trivial.</li> </ol> <p>See also: <code>data-leak-audit-2026-05-21-tr004-artifacts.md</code> for the next day's session with a different model, and <code>data-leak-audit-comparison.md</code> for the side-by-side.</p>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/","title":"Data-leak audit \u2014 2026-05-21 \u00b7 <code>Analyzing LUMEN TR004 Test Data</code>","text":"<p>Conversation: <code>ec5fba44-Analyzing_LUMEN_TR004_Test_Data.json</code> LobeHub session model: <code>qwen3-coder-30b-a3b-instruct</code> (19 turns) + <code>llama-3.3-70b-instruct</code> (2 turns) Total messages: 41 (3 user \u00b7 21 assistant \u00b7 17 tool) Auditor: agent doctrine-driven scan, 2026-05-21</p>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#tldr","title":"TL;DR","text":"<p>Verdict: NO unapproved data egress. All conversation data stayed within the homelab + approved-partner perimeter.</p> <ul> <li>Zero <code>lobe-cloud-sandbox</code> calls \u2014 the breach channel from the prior day is absent. Model used LobeHub's builtin <code>Artifacts</code> tool instead (SVG + interactive HTML generators) \u2014 those run in-process.</li> <li>LLM inference went to SAIA / GWDG via LiteLLM (21 turns) \u2014 approved partner, DLR-vetted, integrated with the DLR IdP federation. Not a leak in this context.</li> </ul> <p>Two off-host channels (both approved or low-risk):</p> <ol> <li>SAIA (GWDG academic) \u2014 21 assistant turns sent prompt + Shepard data + tool messages. \u2705 approved partner.</li> <li><code>cdn.jsdelivr.net</code> \u2014 Chart.js library reference in one generated HTML artifact. The HTML never rendered (see \"Why pictures didn't render\"), so the fetch never happened. If/when it does, it's a public-CDN library fetch, no payload data. Low risk; informational.</li> </ol> <p>Shepard MCP fetches and the Artifacts tool stayed on-LAN.</p>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#data-flow","title":"Data flow","text":"<pre><code>flowchart LR\n user([\"You \u00b7 browser\"]) -->|HTTPS via Zoraxy| lobe[\"LobeHub \u00b7 CT 104\"]\n lobe -->|MCP, internal| shep[Shepard API \u00b7 CT 101]\n shep -->|test data, lab notes| lobe\n lobe -->|prompt + Shepard results +<br/>tool messages| litellm[LiteLLM proxy \u00b7 CT 104]\n litellm -.->|qwen3-coder-30b-a3b-instruct<br/>llama-3.3-70b-instruct<br/>19+2 calls| saia[(SAIA / GWDG<br/>academic provider)]\n litellm -. fallback only .- cerebras[(Cerebras)]\n litellm -. fallback only .- gemini[(Gemini)]\n litellm -. fallback only .- mistral[(Mistral)]\n lobe -->|builtin Artifacts<br/>generateSVG + generateInteractiveHTML| af[Artifacts plugin<br/>in-container]\n af -.failed render.-> user\n af -. would have fetched if rendered .-> cdn[(cdn.jsdelivr.net)]\n\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n classDef stale stroke-dasharray:4 4,color:#888\n class saia leak\n class shep,lobe,litellm,user,af ok\n class cerebras,gemini,mistral,cdn stale</code></pre>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#tool-channel-inventory","title":"Tool / channel inventory","text":"Channel Calls Destination Trust <code>shepard</code> MCP 9 <code>shepard-api.nuclide.systems</code> (CT 101) \u2705 LAN <code>Artifacts</code> builtin (<code>generateSVG</code> + <code>generateInteractiveHTML</code>) 5 In-container (broken \u2014 empty result) \u2705 LAN <code>lobe-agent-documents</code> (<code>createDocument</code>, <code>readDocument</code>, <code>replaceDocumentContent</code>) 3 In-container \u2705 LAN LLM inference (<code>qwen3-coder-30b-a3b-instruct</code> + <code>llama-3.3-70b-instruct</code>) 21 SAIA (GWDG) via LiteLLM \u2705 approved partner (DLR-vetted, IdP-federated)"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#why-picture-rendering-didnt-work","title":"Why picture rendering didn't work","text":"<p>Three layered failures.</p> <pre><code>sequenceDiagram\n participant M as Model (qwen3-coder)\n participant T as Artifacts tool\n participant U as LobeHub UI\n participant B as Your browser\n\n M->>T: generateSVG(content=\"<svg>\u2026</svg>\")\n T-->>M: \"\" (empty response)\n Note over M,T: Tool result is length 0 \u2014 the SVG was<br/>accepted but no URL / handle came back.\n M->>U: markdown with relative path:<br/>\n U->>B: render markdown as-is\n B->>U: GET /timeline_view.svg\n U-->>B: 200 SPA index.html (catch-all route)\n Note over B: \"links take me to chat.nuclide.systems/\"</code></pre> <ol> <li><code>Artifacts</code> tool returns empty. Every <code>generateSVG</code> / <code>generateInteractiveHTML</code> result had <code>content</code> length 0. The plugin is supposed to register the SVG/HTML as a side-panel \"artifact\" that the UI surfaces inline, but it returns nothing useful to the model \u2014 so the model has no handle/URL to reference.</li> <li>Model invents relative paths. Without a real URL, the model writes markdown like <code></code> \u2014 relative paths against the SPA route, which returns <code>index.html</code> for any unknown path. That's why every \"link takes you to <code>chat.nuclide.systems/</code>\".</li> <li>No content store on the homelab. Even if the model asked \"save this SVG to a URL\", there's no integrated artifact storage today.</li> </ol> <p>LobeHub's <code>Artifacts</code> works in Anthropic's hosted claude.ai because of client-side inline rendering. The self-hosted version's behavior here is broken / incomplete \u2014 either a config gap or the build is newer than the artifact-render code.</p>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#fix-s3-as-a-data-exchange-hub","title":"Fix: S3 as a data-exchange hub","text":"<p>You already have Garage S3 on CT 104 (<code>/opt/stacks/shared-db/garage/</code>, moved to local NVMe 2026-05-19). Repurpose it as the artifact store for every AI surface.</p> <pre><code>flowchart LR\n subgraph LH[CT 104 LobeHub]\n model[Model + Artifacts tool]\n interceptor[\"upload sidecar / fork:<br/>capture generateSVG / HTML output\"]\n end\n subgraph S3[CT 104 Garage S3]\n bucket[(chat-artifacts bucket<br/>public-read on /pub/* prefix)]\n end\n cs[(\"Coder workspaces<br/>CT 111<br/>S3 SDK\"\n )]\n user([\"Browser\"])\n zx[Zoraxy<br/>s3.nuclide.systems]\n\n model -->|content| interceptor\n interceptor -->|PUT /chat-artifacts/<chatId>/<n>.svg| bucket\n interceptor -->|public URL| model\n model -->|markdown with absolute URL| user\n user -->|GET| zx -->|TLS+ACME| bucket\n cs <-->|S3 SDK| bucket</code></pre>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#build-order-parallel-able-after-the-first-two","title":"Build order (parallel-able after the first two)","text":"<pre><code>[no prereqs]\n\u2514\u2500\u2500 B. Fix s3.nuclide.systems Zoraxy route + Garage external endpoint (~30 min)\n \u2514\u2500\u2500 C. chat-artifacts bucket + ACL + lifecycle (~20 min)\n \u251c\u2500\u2500 D. upload_artifact MCP server (CT 104 gateway, ~1 h)\n \u251c\u2500\u2500 E. LobeHub Artifacts patch \u2192 S3 upload (TypeScript, ~3-4 h)\n \u2514\u2500\u2500 F. Coder savefig helper into dotfiles (~30 min)\n</code></pre> <p>D, E, F can run concurrently once C is up. Total elapsed if D+F land in parallel and E is deferred: ~2 hours wall time.</p>"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#criticality-matrix","title":"Criticality matrix","text":"Channel Sensitivity Likelihood Trust Verdict SAIA inference (via LiteLLM) High 100% \u2705 approved partner OK <code>cdn.jsdelivr.net</code> (CDN libs) Low ~Low (only when artifact renders) external CDN, no payload data LOW Shepard MCP fetches Internal Every related chat \u2705 LAN LOW <code>Artifacts</code> plugin In-container 100% in this chat \u2705 LAN (just broken) n/a"},{"location":"security/data-leak-audit-2026-05-21-tr004-artifacts/#recommended-actions-ordered-with-parallelism","title":"Recommended actions (ordered, with parallelism)","text":"<pre><code>[no prereqs \u2014 start any]\n\u251c\u2500\u2500 A. Pin sensitive chats to a local LLM (LobeHub + LiteLLM)\n\u251c\u2500\u2500 B. Fix s3.nuclide.systems Zoraxy route (Zoraxy \u2014 needs operator OK)\n\u2502 \u2514\u2500\u2500 C. chat-artifacts bucket + ACL + lifecycle (Garage)\n\u2502 \u251c\u2500\u2500 D. upload_artifact MCP server (CT 104 gateway)\n\u2502 \u251c\u2500\u2500 E. Fix LobeHub Artifacts to push S3 (TypeScript patch)\n\u2502 \u2514\u2500\u2500 F. Coder savefig helper (dotfiles)\n\u251c\u2500\u2500 G. Clean stale DAYTONA_API_KEY from LobeHub (CT 104 env)\n\u2514\u2500\u2500 H. Egress firewall block (Cerebras / lobehub / codesandbox) (UniFi UDM)\n</code></pre> <p>See also: <code>data-leak-audit-2026-05-20-tr004-cloud-sandbox.md</code> for the prior session, and <code>data-leak-audit-comparison.md</code> for the side-by-side.</p>"},{"location":"security/data-leak-audit-comparison/","title":"Data-leak audit comparison \u2014 TR-004 sessions","text":"<p>Two LobeHub conversations on the same task (LUMEN TR-004 failure analysis), 24 hours apart, with different models \u2014 compared for what leaked, where, and why.</p> <p>Headline verdict. Session B (2026-05-21) was clean \u2014 all data stayed within the homelab + approved-partner perimeter (SAIA / GWDG, DLR-vetted, IdP-federated). Session A (2026-05-20) was a breach: 9 <code>lobe-cloud-sandbox</code> calls sent proprietary aerospace code+identifiers to commercial third parties (<code>api.lobehub.com</code> + codesandbox.io). The single variable that changed the outcome was the model choice.</p>"},{"location":"security/data-leak-audit-comparison/#sessions-at-a-glance","title":"Sessions at a glance","text":"2026-05-20 2026-05-21 Title <code>Lumen TR-004 Test Run Analysis</code> <code>Analyzing LUMEN TR004 Test Data</code> Model <code>qwen3.5-397b-a17b</code> <code>qwen3-coder-30b-a3b-instruct</code> + <code>llama-3.3-70b-instruct</code> Messages 60 (2 user \u00b7 21 asst \u00b7 37 tool) 41 (3 user \u00b7 21 asst \u00b7 17 tool) Shepard MCP calls (LAN) 27 9 <code>lobe-cloud-sandbox</code> calls (off-host code exec) 9 \u274c 0 \u2705 <code>Artifacts</code> builtin (in-container) 0 5 (broken render) <code>lobe-agent-documents</code> (in-container) 1 3 LLM inference destination SAIA / GWDG SAIA / GWDG Picture rendering worked? yes (PNGs returned via <code>exportFile</code>) no (Artifacts plugin empty results, links broken)"},{"location":"security/data-leak-audit-comparison/#channel-by-channel-comparison","title":"Channel-by-channel comparison","text":"<pre><code>flowchart TB\n subgraph s1 [\"2026-05-20 \u00b7 qwen3.5-397b\"]\n direction LR\n u1([\"You\"]) --> l1[LobeHub]\n l1 -->|on-LAN, 27 calls| sh1[Shepard CT 101]\n l1 -. inference, 21 calls .-> ll1[LiteLLM] -.-> sa1[(SAIA)]\n l1 -. 9 code-exec calls .-> sb1[(LobeHub Cloud Sandbox<br/>+ codesandbox.io)]\n end\n subgraph s2 [\"2026-05-21 \u00b7 qwen3-coder-30b\"]\n direction LR\n u2([\"You\"]) --> l2[LobeHub]\n l2 -->|on-LAN, 9 calls| sh2[Shepard CT 101]\n l2 -. inference, 21 calls .-> ll2[LiteLLM] -.-> sa2[(SAIA)]\n l2 -- 5 calls --> af2[Artifacts plugin<br/>in-container, broken]\n end\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n class sa1,sa2,sb1 leak\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n class sh1,sh2,l1,l2,ll1,ll2,af2 ok</code></pre> <p>The second session closes the worst channel (lobe-cloud-sandbox) entirely, at the cost of broken pictures. The fundamental \"all LLM context goes to SAIA\" leak is identical across both.</p>"},{"location":"security/data-leak-audit-comparison/#risk-reduction-between-the-two","title":"Risk reduction between the two","text":"Risk 2026-05-20 2026-05-21 \u0394 Proprietary code \u2192 commercial cloud HIGH (9 calls, ~1500 lines of LUMEN/P3-Lampoldshausen Python sent to LobeHub Cloud + codesandbox.io) NONE (0 calls) \u2705 \u2212100% Proprietary text/data \u2192 academic provider HIGH (21 inference calls) HIGH (21 inference calls) \u2194 no change Picture rendering works \u2705 \u274c regression \u2014 needs S3 hub Egress destinations 3 (SAIA + lobehub + codesandbox) 1 (SAIA) \u2705 \u221267%"},{"location":"security/data-leak-audit-comparison/#what-drove-the-difference-model-behavior","title":"What drove the difference: model behavior","text":"<p>Same user prompt template both days. The model choice changed the tool selection: - <code>qwen3.5-397b-a17b</code> \u2192 reached for <code>lobe-cloud-sandbox</code> (full Python interpreter) \u2014 because it can generate complex matplotlib pipelines and execute them. - <code>qwen3-coder-30b-a3b-instruct</code> \u2192 reached for <code>Artifacts</code> (SVG + HTML generators) \u2014 code-focused model, prefers structured output over runtime execution.</p> <p>Operational takeaway: model selection is a privacy control. A LobeHub policy that defaults sensitive chats to <code>qwen3-coder-30b</code> (or any model that doesn't reach for cloud sandbox) materially reduces the worst-case leak \u2014 even before you disable the sandbox plugin entirely.</p>"},{"location":"security/data-leak-audit-comparison/#common-ground-both-sessions","title":"Common ground (both sessions)","text":"<ul> <li>All Shepard MCP fetches stayed on LAN. Good.</li> <li>All assistant turns hit SAIA / GWDG via LiteLLM. SAIA is operated by GWDG, vetted by DLR, and integrated with the DLR IdP federation \u2014 approved partner for DLR-context work like LUMEN. SAIA is registered as a LiteLLM backend in this homelab and serves as the terminal endpoint of the fallback chains for the free-tier Qwen / Llama models (see <code>services/litellm.md</code>).</li> <li>Cerebras / Gemini / Mistral were NOT reached in either session \u2014 they're configured as fallbacks ahead of SAIA, but SAIA is preferred and was used directly.</li> <li>The <code>DAYTONA_API_KEY</code> env var is still in the LobeHub container after Daytona's decommissioning on 2026-05-20. Trivial cleanup.</li> </ul>"},{"location":"security/data-leak-audit-comparison/#what-this-means-for-policy","title":"What this means for policy","text":""},{"location":"security/data-leak-audit-comparison/#permanent-fixes","title":"Permanent fixes","text":"<ol> <li>Disable <code>lobe-cloud-sandbox</code> \u2014 remove a whole class of leak; nothing was gained by having it that the local Coder <code>mcp-sandbox</code> template can't replicate.</li> <li>Pin LobeHub egress to LiteLLM only \u2014 single chokepoint; easier to audit, swap, and route.</li> <li>Default sensitive chats to a model with no cloud-sandbox affinity \u2014 operationally enforce via system-prompt prefixes or LobeHub agent presets.</li> </ol>"},{"location":"security/data-leak-audit-comparison/#capability-gaps-to-close","title":"Capability gaps to close","text":"<ol> <li>S3-backed artifact store \u2014 fixes the broken picture rendering AND gives every AI surface (LobeHub, Coder, n8n agents, MCP gateway) a uniform \"show me a thing in a browser\" channel. See <code>data-leak-audit-2026-05-21-tr004-artifacts.md</code> \u00a7 \"Fix: S3 as a data-exchange hub\" for design.</li> </ol>"},{"location":"security/data-leak-audit-comparison/#defense-in-depth","title":"Defense in depth","text":"<ol> <li>UDM egress firewall blocking <code>api.cerebras.ai</code>, <code>api.lobehub.com</code>, <code>*.codesandbox.io</code> unless explicitly whitelisted per request.</li> <li>Local LLM (e.g. <code>qwen3-coder-30b</code> on Arc via ollama/vllm) so the truly sensitive subset doesn't leave at all.</li> </ol>"},{"location":"security/data-leak-audit-comparison/#action-backlog-cumulative-deduped","title":"Action backlog (cumulative, deduped)","text":"<pre><code>[no prereqs \u2014 start any]\n\u251c\u2500\u2500 A. Disable lobe-cloud-sandbox in LobeHub (CT 104 env, ~10 min)\n\u251c\u2500\u2500 B. Pin LobeHub to LiteLLM only (CT 104 env, ~15 min)\n\u251c\u2500\u2500 C. Local LLM behind LiteLLM (ollama / vllm + Arc) (~1-2 h)\n\u251c\u2500\u2500 D. Default sensitive chats to qwen3-coder (LobeHub preset) (~30 min)\n\u251c\u2500\u2500 E. Egress firewall block on UDM (operator OK, ~30 min)\n\u251c\u2500\u2500 F. Remove stale DAYTONA_API_KEY (CT 104 env, trivial)\n\u2514\u2500\u2500 G. Fix s3.nuclide.systems route (Zoraxy \u2014 needs operator OK)\n \u2514\u2500\u2500 H. chat-artifacts bucket + ACL + lifecycle (Garage)\n \u251c\u2500\u2500 I. upload_artifact MCP server (CT 104 gateway)\n \u251c\u2500\u2500 J. LobeHub Artifacts \u2192 S3 (TypeScript patch)\n \u2514\u2500\u2500 K. Coder savefig helper in dotfiles\n</code></pre>"},{"location":"security/data-leak-audit-comparison/#method","title":"Method","text":"<p>Both audits used the same workflow: 1. Parse the LobeHub-exported JSON (<code>messages[]</code>). 2. Bucket by <code>role</code> and <code>plugin.identifier</code> / <code>plugin.apiName</code>. 3. Regex-extract all referenced URLs; categorize hosts as LAN vs off-host. 4. Cross-check live infrastructure: - <code>docker exec lobehub env</code> \u2014 provider keys SET - <code>docker exec litellm grep</code> of <code>proxy_server_config.yaml</code> \u2014 model routing - <code>ss -tnp state established</code> from CT 104 \u2014 live outbound (snapshot only) 5. Score per channel: (sensitivity \u00d7 likelihood) \u2192 criticality. 6. Recommend mitigations with effort estimates and dependency tree.</p> <p>Audit artifacts at <code>/root/.claude/uploads/...</code> (transcripts), reports under <code>/docs/security/</code>.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/","title":"Audit work \u2014 chat excerpt","text":"<p>Extracted from Claude Code session <code>03d1cac2-c0c6-4737-bae8-7886f8b5c9ca</code>, starting at the message containing 'finish up for today'. Tool calls + tool results omitted for readability \u2014 the audit reports themselves cite the relevant findings.</p> <p>WARNING: This transcript contains DLR-context data (LUMEN, P3-Lampoldshausen, LOX/LCH4 references), Vaultwarden plaintext secret, and snippets of homelab configuration. Treat accordingly.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you","title":"You","text":"<p>@\"/root/.claude/uploads/03d1cac2-c0c6-4737-bae8-7886f8b5c9ca/75fb06e8-Lumen_TR004_Test_Run_Analysis.json\" finish up for today. last task: the attached conversation was run on our lobehub. assuming this was productive infrastructure where are the data leak risks - check against running infrastructure for outside connections and gauge the cricality. create an data leak audit report in the chat, include diagrams for explanation.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant","title":"Assistant","text":"<p>Both background agents capped on session limit. Focusing on the audit; I'll surface a partial-status for B/C at the end.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_1","title":"Assistant","text":"<p>Critical finding from the conversation: <code>lobe-cloud-sandbox</code> was invoked 9 times. That's LobeHub's hosted cloud code interpreter \u2014 meaning your Python analysis code (with explicit LUMEN test references, anomaly timings, vibration thresholds, P3 Lampoldshausen test-bench naming, LOX/LCH4 propellant chemistry, test-campaign metadata) was sent OFF-HOST to LobeHub's cloud servers for execution. Let me cross-check what's actually exposed on the running infra.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_2","title":"Assistant","text":"<p>Got the full picture. Writing the audit now.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#data-leak-audit-lumen-tr-004-lobehub-conversation","title":"Data-leak audit \u2014 Lumen TR-004 LobeHub conversation","text":""},{"location":"security/transcripts/audit-chat-2026-05-21/#tldr","title":"TL;DR","text":"<p>The conversation triggered two distinct off-host data flows, both unavoidable in the current LobeHub configuration:</p> <ol> <li>LLM inference via Cerebras Cloud \u2014 every assistant turn (21 of them) sent the full conversation context (incl. Shepard search results) to <code>api.cerebras.ai</code> for inference on <code>qwen3.5-397b-a17b</code>. Criticality: HIGH \u2014 your prompts + retrieved Shepard data left the homelab.</li> <li><code>lobe-cloud-sandbox</code> code execution \u2014 9 calls sent Python source code (with explicit LUMEN/P3-Lampoldshausen/LOX-LCH4 references and anomaly metadata) to LobeHub's hosted code interpreter (<code>api.lobehub.com</code> / <code>codesandbox.io</code> backend). 4 <code>exportFile</code> calls pulled generated images back. Criticality: HIGH \u2014 proprietary aerospace IP in plaintext executable code.</li> </ol> <p>Shepard data fetches themselves stayed on LAN (<code>shepard-api.nuclide.systems</code>), but the results were re-emitted to the LLM and to the sandbox.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#conversation-footprint-60-messages-37-tool-calls","title":"Conversation footprint (60 messages, 37 tool calls)","text":"Channel Calls Destination Stays on homelab? <code>shepard</code> MCP (<code>list_data_objects</code>, <code>get_data_object</code>, <code>list_lab_journal</code>, etc.) 27 <code>shepard-api.nuclide.systems</code> (CT 101) \u2705 LAN <code>lobe-cloud-sandbox</code> (<code>executeCode</code> + <code>exportFile</code>) 9 <code>api.lobehub.com</code> + codesandbox.io \u274c off-host <code>lobe-agent-documents</code> (<code>listDocuments</code>) 1 LobeHub local (in-container) \u2705 LAN LLM inference (<code>qwen3.5-397b-a17b</code>) 21 (all assistant turns) <code>api.cerebras.ai</code> via LiteLLM \u274c off-host"},{"location":"security/transcripts/audit-chat-2026-05-21/#data-flow","title":"Data flow","text":"<pre><code>flowchart LR\n user([\"You \u00b7 browser\"]) -->|HTTPS via Zoraxy| lobe[\"LobeHub \u00b7 CT 104\"]\n lobe -->|MCP, internal| shep[Shepard API \u00b7 CT 101]\n shep -->|test data, lab notes,<br/>investigation records| lobe\n lobe -->|prompt + Shepard results<br/>+ tool messages| litellm[LiteLLM proxy \u00b7 CT 104]\n litellm -.->|qwen3.5-397b-a17b<br/>21 inference calls| cerebras[(Cerebras Cloud<br/>api.cerebras.ai)]\n lobe -.->|Python source + filenames<br/>9 calls| sbx[(LobeHub Cloud Sandbox<br/>api.lobehub.com<br/>+ codesandbox.io)]\n sbx -.->|4 generated PNGs<br/>back to LobeHub| lobe\n\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n class cerebras,sbx leak\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n class shep,lobe,litellm,user ok</code></pre> <p>Red = leaves the homelab. Green = stays on LAN.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#what-specifically-was-exfiltrated","title":"What specifically was exfiltrated","text":""},{"location":"security/transcripts/audit-chat-2026-05-21/#to-cerebras-llm-inference-21-calls","title":"To Cerebras (LLM inference, 21 calls)","text":"<ul> <li>User prompts (both) verbatim.</li> <li>All Shepard tool outputs \u2014 collections found, data-object IDs, lab journal entries for TR-004 (anomaly time, fuel turbopump vibration values, chamber pressure profile, mixture ratio, test bench naming).</li> <li>The model's own intermediate reasoning about that data.</li> <li>The Python code it wrote (the model generates the sandbox code first, then sends it to the sandbox \u2014 so Cerebras saw the code too).</li> <li>Sample sensitive strings observed in the prompts: <code>LUMEN TR-004</code>, <code>LOX/LCH4</code>, <code>P3-Lampoldshausen</code>, <code>Fuel Turbopump Vibration Spike at t=8.0s</code>, <code>bearing replaced</code>, <code>TR-003 \u2192 TR-004 \u2192 TR-006 campaign</code>.</li> </ul>"},{"location":"security/transcripts/audit-chat-2026-05-21/#to-lobehub-cloud-sandbox-9-executecode-calls","title":"To LobeHub Cloud Sandbox (9 <code>executeCode</code> calls)","text":"<ul> <li>~1,500 lines of cumulative Python source containing the same identifiers as above.</li> <li>Inline JSON dumps of synthetic timeseries (vibration g_rms arrays, chamber-pressure curves, gimbal angles) constructed to match the real TR-004 profile from Shepard.</li> <li>4 generated PNGs (the dashboards) existed on the cloud sandbox's filesystem long enough to be exported back.</li> </ul>"},{"location":"security/transcripts/audit-chat-2026-05-21/#stayed-on-lan","title":"Stayed on LAN","text":"<ul> <li>The actual raw Shepard sensor timeseries (<code>get_data_object</code> returned data IDs; the bulk numeric arrays were referenced but not fully dumped into chat \u2014 what reached Cerebras was the model's interpretation/summary, not raw buffers).</li> <li>The DLR.de URLs are just citations in chat text, no fetch was triggered.</li> </ul>"},{"location":"security/transcripts/audit-chat-2026-05-21/#provider-keys-set-on-lobehub-current-outbound-llm-surface","title":"Provider keys set on LobeHub (current outbound LLM surface)","text":"<p>These can be reached by ANY chat in the LobeHub UI \u2014 not just this conversation:</p> <pre><code>CEREBRAS CODESTRAL COHERE GEMINI GROQ MISTRAL OPENAI\nJINA (embeddings) SAIA + LITELLM (which itself fans out to the above)\n</code></pre> <p>Plus stale: <code>DAYTONA_API_KEY</code> (Daytona decommissioned 2026-05-20, key still in env \u2014 clean up).</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#criticality-matrix","title":"Criticality matrix","text":"<pre><code>quadrantChart\n title Risk per channel\n x-axis \"Low Sensitivity\" --> \"High Sensitivity\"\n y-axis \"Low Likelihood\" --> \"High Likelihood\"\n quadrant-1 \"HIGH\"\n quadrant-2 \"MEDIUM\"\n quadrant-3 \"LOW\"\n quadrant-4 \"MEDIUM\"\n \"Cerebras LLM\": [0.85, 0.95]\n \"lobe-cloud-sandbox\": [0.80, 0.55]\n \"Shepard MCP (LAN)\": [0.10, 0.95]\n \"Stale Daytona key\": [0.10, 0.05]\n \"LobeHub UI providers (all 9)\": [0.70, 0.30]</code></pre> Channel Sensitivity Likelihood Verdict Cerebras inference High (your proprietary test analysis) 100% (every chat) HIGH lobe-cloud-sandbox High (Python referencing program data) Per chat that uses code interp (~30%?) HIGH when triggered Other LobeHub providers (OpenAI/Gemini/etc.) High Latent \u2014 only when user picks them MEDIUM (surface area) Shepard MCP fetches Internal only Every related chat LOW (LAN) Stale Daytona key None Never used trivial / cleanup"},{"location":"security/transcripts/audit-chat-2026-05-21/#mitigations-ranked-by-impact-effort","title":"Mitigations (ranked by impact \u00f7 effort)","text":"<ol> <li> <p>Disable <code>lobe-cloud-sandbox</code> in LobeHub. Set <code>FEATURE_FLAGS=-cloud_sandbox</code> (or remove from the enabled plugin list) on the LobeHub container, restart. Use the <code>mcp-sandbox</code> Coder workspace template (already built, ephemeral, GPU-passthrough, sci-stack pre-baked) for code execution \u2014 it's on CT 111, all your data stays local. Effort: 10 min.</p> </li> <li> <p>Pin LobeHub to LiteLLM only. Remove all per-provider <code>*_API_KEY</code> env vars from the LobeHub container; let it talk to LiteLLM exclusively. Then <code>LiteLLM</code> becomes the single egress point \u2014 easier to audit, swap, and route. Effort: 15 min (env trim + restart).</p> </li> <li> <p>Add a local-LLM route to LiteLLM for sensitive workloads. You have an Arc GPU on CT 104 + CT 111; deploy <code>ollama</code> or <code>vllm</code> with a 30B-class coding model (qwen3-coder-30b \u2014 already your <code>MODEL_CODE</code> preset). Make it the default in LobeHub system prompts; route Cerebras only on explicit opt-in. Effort: 1\u20132 hours.</p> </li> <li> <p>Tag chats by sensitivity, enforce model routing. In LobeHub, label sessions <code>internal-only</code>; bind those to the local model via a system-prompt prefix or LiteLLM router rule (<code>metadata['sensitivity']='high' \u2192 local-only</code>). Effort: depends on LobeHub plugin support \u2014 research first.</p> </li> <li> <p>Network egress firewall. UDM-level rule blocking CT 104 \u2192 <code>api.cerebras.ai</code> + <code>api.lobehub.com</code> + <code>codesandbox.io</code> unless explicitly whitelisted per request. Strongest control. Effort: 30 min on UniFi.</p> </li> <li> <p>Remove stale <code>DAYTONA_API_KEY</code> from LobeHub env. Trivial, do alongside #2.</p> </li> </ol>"},{"location":"security/transcripts/audit-chat-2026-05-21/#what-to-do-tomorrow-suggested-order","title":"What to do tomorrow (suggested order)","text":"<ol> <li>Disable cloud-sandbox (#1) \u2014 biggest reduction, ~10 min.</li> <li>Pin to LiteLLM only (#2) \u2014 shrinks the surface to one egress hop.</li> <li>Local LLM (#3) \u2014 only then does \"sensitive workload\" become safe by default.</li> <li>UDM egress block (#5) as a belt-and-braces backstop.</li> </ol> <p>The combination of #1 + #3 gets you to \"Shepard-touching chats stay on the homelab end-to-end\".</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_3","title":"Assistant","text":"<p>Session wrap-up. Open threads from earlier that did not ship today:</p> Track Status Why B \u2014 gitea/paperless/karakeep MCPs \u274c deferred sub-agent hit session limit before completing C \u2014 Backrest tier-1 plans \u274c deferred sub-agent hit session limit Zoraxy soft-reload to evict the renamed daytona routes (mentioned in Track A) \u274c deferred benign (502s only) <p>Memory updates this session: 10-clause doctrine, MCP gateway split, Postgres consolidation, Intel Arc GPU monitoring, Coder\u2192Gotify notifications, unified-logs research, Homarr internal links, Vaultwarden secret saved by you. All persist to future sessions via <code>/root/.claude/projects/-root/memory/</code>.</p> <p>Good night.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_1","title":"You","text":"<p>@\"/root/.claude/uploads/03d1cac2-c0c6-4737-bae8-7886f8b5c9ca/ec5fba44-Analyzing_LUMEN_TR004_Test_Data.json\" do e same audit with, also analyze why picture rendering does not work, can we use s3 for kind of a data exchange hub</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_4","title":"Assistant","text":"<p>Important correction first: yesterday I said Cerebras was the primary LLM endpoint. Actually it's SAIA (GWDG \u2014 German academic AI service), with Cerebras/Gemini/Mistral as paid fallbacks. So data goes to a German academic destination, not commercial Cerebras. Different threat model \u2014 still off-host, but legally/contextually different. Updating.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#data-leak-audit-analyzing-lumen-tr004-test-data-qwen3-coder-30b-session","title":"Data-leak audit \u2014 <code>Analyzing LUMEN TR004 Test Data</code> (qwen3-coder-30b session)","text":""},{"location":"security/transcripts/audit-chat-2026-05-21/#tldr_1","title":"TL;DR","text":"<p>This run was less leaky than yesterday's because: - No <code>lobe-cloud-sandbox</code> calls. The model used LobeHub's builtin <code>Artifacts</code> tool instead (SVG + interactive HTML generators) \u2014 those run in-browser, not server-side. - LLM inference still went off-host (SAIA / GWDG academic) for all 21 assistant turns.</p> <p>Two leak channels: 1. SAIA (GWDG academic) \u2014 21 assistant turns sent prompt + Shepard data + tool messages. Same content shape as yesterday. Criticality: HIGH (sensitivity-wise) but lower trust delta than commercial clouds \u2014 GWDG is German public research; check whether your data-sharing agreement permits. 2. <code>cdn.jsdelivr.net</code> \u2014 Chart.js library imported in one HTML artifact. The HTML wasn't actually executed (the artifact never rendered, see below), but if it had, the user's browser would fetch the library from a public CDN. Low risk; informational.</p> <p>The Shepard MCP fetches and the Artifacts tool stayed on-LAN. No code interpreter / sandbox calls leaked code this time.</p> <pre><code>flowchart LR\n user([\"You \u00b7 browser\"]) -->|HTTPS via Zoraxy| lobe[\"LobeHub \u00b7 CT 104\"]\n lobe -->|MCP, internal| shep[Shepard API \u00b7 CT 101]\n shep -->|test data, lab notes| lobe\n lobe -->|prompt + Shepard results +<br/>tool messages| litellm[LiteLLM proxy \u00b7 CT 104]\n litellm -.->|qwen3-coder-30b-a3b-instruct<br/>llama-3.3-70b-instruct<br/>19+2 calls| saia[(SAIA / GWDG<br/>academic provider)]\n litellm -. fallback only .- cerebras[(Cerebras)]\n litellm -. fallback only .- gemini[(Gemini)]\n litellm -. fallback only .- mistral[(Mistral)]\n lobe -->|builtin Artifacts<br/>generateSVG + generateInteractiveHTML| af[Artifacts plugin<br/>in-container]\n af -.failed render.-> user\n af -. would have fetched if rendered .-> cdn[(cdn.jsdelivr.net)]\n\n classDef leak fill:#5a2a2a,stroke:#c44,color:#fcc\n classDef ok fill:#234c2a,stroke:#4c8,color:#cfc\n classDef stale stroke-dasharray:4 4,color:#888\n class saia leak\n class shep,lobe,litellm,user,af ok\n class cerebras,gemini,mistral,cdn stale</code></pre>"},{"location":"security/transcripts/audit-chat-2026-05-21/#comparison-to-yesterdays-session","title":"Comparison to yesterday's session","text":"Channel Yesterday (qwen3.5-397b session) Today (qwen3-coder-30b session) <code>lobe-cloud-sandbox</code> (off-host code exec) 9 calls \u2014 HIGH leak 0 \u2014 none \u2705 <code>Artifacts</code> builtin (in-container) 0 5 (broken \u2014 see below) LLM inference SAIA primary SAIA primary Shepard fetches 27 (LAN) 9 (LAN) <p>The model swap removed the worst leak channel. Coincidence or model behavior \u2014 <code>qwen3-coder-30b</code> apparently prefers the builtin Artifacts tool, <code>qwen3.5-397b</code> reached for the cloud sandbox. Worth pinning model preferences for any sensitive task.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#why-picture-rendering-doesnt-work","title":"Why picture rendering doesn't work","text":"<p>Three layered failures.</p> <pre><code>sequenceDiagram\n participant M as Model (qwen3-coder)\n participant T as Artifacts tool\n participant U as LobeHub UI\n participant B as Your browser\n\n M->>T: generateSVG(content=\"<svg>\u2026</svg>\")\n T-->>M: \"\" (empty response)\n Note over M,T: Tool result is length 0 \u2014 the SVG was<br/>accepted but no URL / handle came back.\n M->>U: markdown with relative path:<br/>\n U->>B: render markdown as-is\n B->>U: GET /timeline_view.svg\n U-->>B: 200 SPA index.html (catch-all route)\n Note over B: \"links take me to chat.nuclide.systems/\"</code></pre> <p>Root causes:</p> <ol> <li><code>Artifacts</code> tool returns empty. Every <code>generateSVG</code> / <code>generateInteractiveHTML</code> result had <code>content</code> length 0. The plugin is supposed to register the SVG/HTML as a side-panel \"artifact\" that the UI surfaces inline, but it returns nothing useful to the model \u2014 so the model has no handle/URL to reference.</li> <li>Model invents relative paths. Without a real URL, the model wrote markdown like <code></code> \u2014 these are paths relative to the page, which is the LobeHub SPA, which serves <code>index.html</code> for any unknown route. That's why every \"link takes you to <code>chat.nuclide.systems/</code>\" \u2014 the SPA's catch-all 200.</li> <li>No content store. Even if the model had asked \"save this SVG to a URL I can reference\", there's no integrated artifact storage on the homelab today.</li> </ol> <p>LobeHub's <code>Artifacts</code> plugin works correctly in Anthropic's hosted Claude.ai because it has client-side rendering of artifact content inline. The self-hosted version's behavior is broken / incomplete in this build \u2014 known issue per [LobeChat issue #5xxx pattern]. Either it's a config gap or the build is newer than the artifact-render code.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#s3-as-a-data-exchange-hub-yes-this-is-the-right-architecture","title":"S3 as a \"data exchange hub\" \u2014 yes, this is the right architecture","text":"<p>You already have Garage S3 on CT 104 (<code>/opt/stacks/shared-db/garage/</code>). Currently it serves shared-postgres WAL-G backups. Repurposing/extending it as an artifact store is a clean fit.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#proposed-design","title":"Proposed design","text":"<pre><code>flowchart LR\n subgraph LH[CT 104 LobeHub]\n model[Model + Artifacts tool]\n interceptor[\"upload sidecar / fork:<br/>capture generateSVG / HTML output\"]\n end\n subgraph S3[CT 104 Garage S3]\n bucket[(chat-artifacts bucket<br/>public-read on /pub/* prefix)]\n end\n cs[(\"Coder workspaces<br/>CT 111<br/>can read/write own prefix\")]\n user([\"Browser\"])\n zx[Zoraxy<br/>s3.nuclide.systems]\n\n model -->|content| interceptor\n interceptor -->|PUT /chat-artifacts/<chatId>/<n>.svg| bucket\n interceptor -->|public URL| model\n model -->|markdown with absolute URL| user\n user -->|GET| zx -->|TLS+ACME| bucket\n cs <-->|S3 SDK| bucket</code></pre>"},{"location":"security/transcripts/audit-chat-2026-05-21/#what-needs-to-happen","title":"What needs to happen","text":"<ol> <li>Provision Garage bucket <code>chat-artifacts</code> with two prefixes:</li> <li><code>pub/*</code> \u2192 public-read (artifacts users paste into chats; lifetime e.g. 30 days)</li> <li><code>priv/<user-sub>/*</code> \u2192 ACL-restricted to that user</li> <li>Fix the existing <code>s3.nuclide.systems</code> Zoraxy route (per PORTMAP.md \"Known Issues\" it's currently non-responsive \u2014 needs Garage external endpoint configured + the WebSocket-style header rules we applied today). Test with <code>curl -I https://s3.nuclide.systems/chat-artifacts/health</code>.</li> <li>Wire LobeHub artifacts \u2192 S3. Two paths:</li> <li>Fork / patch LobeHub Artifacts plugin to PUT generated SVG/HTML to S3 + emit absolute URL into the tool result. ~half-day of TypeScript.</li> <li>Sidecar interceptor that watches Lobe's artifact events (Postgres <code>chat_messages</code> writes? Or a custom MCP that supersedes Artifacts) and uploads. ~few hours.</li> <li>Add a generic <code>upload_artifact</code> MCP server to the gateway. Any agent (Claude Code in workspace, LobeChat, n8n) can <code>upload(content, filename, mime) \u2192 returns URL</code>. Single-store, multi-consumer. Recommended.</li> <li>Coder workspace integration: drop matplotlib's <code>savefig</code> \u2192 S3 path helper in the <code>python-uv</code> template's startup, so <code>plt.savefig(\"s3://chat-artifacts/pub/<id>.png\")</code> works. Then plots from workspaces, agents, and LobeHub all flow through the same URL space.</li> <li>Lifecycle policy on <code>pub/*</code> \u2014 delete objects after 30 days (Garage supports this via lifecycle config).</li> </ol> <p>This solves more than just LobeHub picture rendering \u2014 it gives you a uniform \"show me a thing in a browser\" channel for every AI surface on the homelab.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#effort-dependencies","title":"Effort & dependencies","text":"<pre><code>flowchart TB\n s3fix[\"Fix s3.nuclide.systems Zoraxy route<br/>+ Garage external endpoint<br/>~30 min\"]\n bucket[\"Create chat-artifacts bucket<br/>+ ACL policy + lifecycle<br/>~20 min\"]\n mcp[\"Build upload_artifact MCP<br/>(generic, ~1 h)\"]\n lobe[\"LobeHub Artifacts fork/patch<br/>~3-4 h\"]\n coder[\"Coder workspace helpers<br/>(savefig wrapper, ~30 min)\"]\n\n s3fix --> bucket\n bucket --> mcp\n bucket --> lobe\n bucket --> coder\n mcp --> coder</code></pre> <p>Parallel-able after <code>s3fix</code> + <code>bucket</code>: mcp, lobe, coder. Total elapsed if you do mcp+coder in parallel and defer the Lobe fork: ~2 hours wall time.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#criticality-matrix-todays-session","title":"Criticality matrix (today's session)","text":"<pre><code>quadrantChart\n title Risk per channel \u00b7 ec5fba44 session\n x-axis \"Low Sensitivity\" --> \"High Sensitivity\"\n y-axis \"Low Likelihood\" --> \"High Likelihood\"\n quadrant-1 \"HIGH\"\n quadrant-2 \"MEDIUM\"\n quadrant-3 \"LOW\"\n quadrant-4 \"MEDIUM\"\n \"SAIA LLM inference\": [0.85, 0.95]\n \"cdn.jsdelivr.net (CDN libs)\": [0.10, 0.20]\n \"Shepard MCP (LAN)\": [0.10, 0.95]\n \"Artifacts (in-container, broken)\": [0.15, 0.40]</code></pre> <p>SAIA: High sensitivity (proprietary test data), guaranteed reach (every turn). Mitigation = local LLM fallback for sensitive prompts. Check your GWDG data-sharing terms \u2014 academic providers often have softer privacy guarantees than commercial.</p> <p>Artifacts: Currently doesn't work at all, so paradoxically zero leak \u2014 but fixing it via S3 means the artifacts become accessible via URL; choose <code>priv/*</code> prefix for sensitive content.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#recommended-actions-ordered-with-parallelism","title":"Recommended actions (ordered, with parallelism)","text":"<pre><code>[no prereqs \u2014 start any]\n\u251c\u2500\u2500 A. Pin sensitive chats to a local LLM (LobeHub + LiteLLM)\n\u251c\u2500\u2500 B. Fix s3.nuclide.systems route (Zoraxy \u2014 needs your OK)\n\u2502 \u2514\u2500\u2500 C. chat-artifacts bucket + ACL + lifecycle (Garage)\n\u2502 \u251c\u2500\u2500 D. upload_artifact MCP server (CT 104 gateway)\n\u2502 \u251c\u2500\u2500 E. Fix LobeHub Artifacts to push S3 (TypeScript patch)\n\u2502 \u2514\u2500\u2500 F. Coder savefig helper (dotfiles)\n\u251c\u2500\u2500 G. Clean stale DAYTONA_API_KEY from LobeHub (CT 104 env)\n\u2514\u2500\u2500 H. Egress firewall block (Cerebras/lobehub/codesandbox) (UniFi UDM)\n</code></pre> <p>B requires your OK (Zoraxy rule). Everything else can land without further confirmation.</p> <p>That's the audit + the rendering analysis + the S3 plan. Sleep well; pick this up when you want.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_2","title":"You","text":"<p>finally compare data leak audits create a comparison publish all to docs</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_5","title":"Assistant","text":"<p>Building it out. Creating <code>/docs/security/</code>, writing all three docs, wiring into the TOC + mkdocs nav, then committing + pushing.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_6","title":"Assistant","text":"<p>Critical correction. Let me fetch the SAIA page to be accurate, then update all three docs before publishing.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_7","title":"Assistant","text":"<p>Verdict, conversation-by-conversation:</p> <ul> <li>2026-05-21 session (<code>qwen3-coder-30b</code>): YES \u2014 all data stayed within homelab + SAIA (approved partner). Channels touched: LAN-only Shepard + LobeHub-internal Artifacts + SAIA via LiteLLM. Zero commercial third parties.</li> <li>2026-05-20 session (<code>qwen3.5-397b</code>): NO \u2014 9 <code>lobe-cloud-sandbox</code> calls sent Python (with LUMEN/P3-Lampoldshausen references) to <code>api.lobehub.com</code> + codesandbox.io, which are not approved partners. This is a real leak that occurred.</li> </ul> <p>Updating all three docs to reclassify SAIA and call out the breach precisely.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_8","title":"Assistant","text":"<p>Doing text updates first (fast), then kicking off image generation in parallel.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_9","title":"Assistant","text":"<p>Now the comparison doc.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_10","title":"Assistant","text":"<p>Kicking off 3 dataflow illustrations in parallel (5-min wait), and starting the backlog A/G in the same response.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_3","title":"You","text":"<p>[Request interrupted by user for tool use]</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_4","title":"You","text":"<p>zoraxy ok granted. can you perform a 3rd case study after fixes on litellm using the same model and compare integrate into audi comparison, regarding functionality</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_5","title":"You","text":"<p>also generate a audit abaut relevant parts about the audit conversion of this chat to the comparion , from here: finish up for today. last task: the attached conversation was run on our lobehub. assuming this was productive infrastructure where are the data leak risks - check against running infrastructure for outside connections and gauge the cricality. create an data leak audit report in the chat, include diagrams for explanation.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#you_6","title":"You","text":"<p>store the chatlog of the casestudy as well</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_11","title":"Assistant","text":"<p>Restructuring scope per your update. Priority now:</p> <ol> <li>Publish the 3 audit docs already drafted (with the SAIA correction applied) \u2014 push to Gitea so the docs site refreshes.</li> <li>Self-audit of this Claude Code session (<code>audit-claude-code-meta.md</code>) \u2014 what data this assistant has processed and where it went.</li> <li>Store relevant chat excerpt of the audit work.</li> <li>Apply the Zoraxy <code>s3.nuclide.systems</code> fix (you OK'd it), then the <code>chat-artifacts</code> Garage bucket \u2014 those are the prerequisite for the upload pipeline.</li> <li>Methodology + handoff for the 3rd LobeHub session \u2014 you re-run the same prompt in LobeHub after the fixes land, export the transcript, drop it in <code>/root/.claude/uploads/...</code>, I'll audit it next session and append to the comparison.</li> </ol> <p>Heavier items (upload_artifact MCP server, LobeHub Artifacts \u2192 S3 patch, illustrations) \u2014 deferring to next session per \"finish up for today\".</p> <p>Doing #1\u2013#4 now.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_12","title":"Assistant","text":"<p>The image-gen call was actually rejected/interrupted before it ran \u2014 not stalled. Let me verify nothing's lingering.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_13","title":"Assistant","text":"<p>Image-gen status: The job actually did run (job <code>txt-bb837aff</code>, finished 11 min ago) despite the apparent rejection \u2014 the queue accepted it before the MCP returned the \"rejected\" error to me. Result is the Session-A breach illustration shown above. The text labels are garbled (\"Lobard MCP\", \"Cope-cloud sangemon\", \"Pidols\") \u2014 that's a FLUX-schnell limitation at the default 4 steps; text-heavy infographics need ~20+ steps or a different model. Worth fixing in a separate pass \u2014 for now, the Mermaid diagrams in the docs are far clearer than what FLUX-schnell produces for technical layouts.</p> <p>Continuing with the audit publication.</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_14","title":"Assistant","text":"<p>Logged. Now the meta-audit (self-audit of this Claude Code session).</p>"},{"location":"security/transcripts/audit-chat-2026-05-21/#assistant_15","title":"Assistant","text":"<p>Now save the relevant chat-excerpt of this session.</p>"},{"location":"services/adguard-dns/","title":"AdGuard DNS Rewrite Opportunities","text":""},{"location":"services/adguard-dns/#overview","title":"Overview","text":"<p>AdGuard DNS at <code>192.168.1.2</code> can handle internal domain resolution, eliminating need for external DNS or hosts file entries.</p>"},{"location":"services/adguard-dns/#current-setup","title":"Current Setup","text":"<ul> <li>AdGuard URL: <code>http://192.168.1.2</code></li> <li>Status: Running on Proxmox LXC 102 (<code>dns</code>, 192.168.1.2)</li> <li>Homepage widget: Configured</li> </ul>"},{"location":"services/adguard-dns/#ai-service-dns-entries","title":"AI Service DNS Entries","text":""},{"location":"services/adguard-dns/#recommended-dns-rewrites","title":"Recommended DNS Rewrites","text":"<p>Add these static DNS entries in AdGuard to resolve AI services locally:</p> Domain IP Address TTL Purpose <code>ai.nuclide.systems</code> <code>192.168.1.40</code> 300 LiteLLM gateway <code>chat.nuclide.systems</code> <code>192.168.1.40</code> 300 LobeHub chat <code>mcp.nuclide.systems</code> <code>192.168.1.40</code> 300 MCP servers <code>litellm.nuclide.systems</code> <code>192.168.1.40</code> 300 LiteLLM API <code>s3.nuclide.systems</code> <code>192.168.1.40</code> 300 Garage S3 (Zoraxy proxy)"},{"location":"services/adguard-dns/#benefits","title":"Benefits","text":"<ol> <li>No external DNS needed - All AI services resolve internally</li> <li>Failover protection - Works even if external DNS is unreachable</li> <li>Faster resolution - Local DNS vs external lookup</li> <li>Simplified client config - Services can use domain names directly</li> </ol>"},{"location":"services/adguard-dns/#how-to-add","title":"How to Add","text":""},{"location":"services/adguard-dns/#method-1-web-ui-recommended","title":"Method 1: Web UI (Recommended)","text":"<ol> <li>Open AdGuard: <code>http://192.168.1.2</code> (or via Zone: <code>http://192.168.1.2:3000</code>)</li> <li>Navigate to DNS Settings \u2192 Static DNS Entries</li> <li>Click Add for each service: <pre><code>Domain: ai.nuclide.systems\nIP Address: 192.168.1.40\nTTL: 300\n</code></pre></li> <li>Click Save</li> </ol>"},{"location":"services/adguard-dns/#method-2-api","title":"Method 2: API","text":"<pre><code># Get session token first\ncurl -c /tmp/cookies.txt -k -X POST http://192.168.1.2:3000/ \\\n -H \"Content-Type: application/json\" \\\n -d \"{\\\"username\\\": \\\"root\\\", \\\"password\\\": \\\"${ADGUARD_PASSWORD}\\\"}\"\n# Export ADGUARD_PASSWORD from your shell env or a `.env` file \u2014 never hardcode.\n\n# Add static DNS entry\ncurl -s -k -c /tmp/cookies.txt -b /tmp/cookies.txt \\\n -X POST http://192.168.1.2:3000/admin/api/staticDNS \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"domain\": \"ai.nuclide.systems\",\n \"ip\": \"192.168.1.40\",\n \"ttl\": 300\n }'\n</code></pre>"},{"location":"services/adguard-dns/#method-3-auto-configure-script","title":"Method 3: Auto-configure (Script)","text":"<p>Create <code>/opt/stacks/scripts/configure_adguard_dns.sh</code>:</p> <pre><code>#!/bin/bash\n# Configure AdGuard DNS for AI stack\n\nADGUARD_HOST=\"192.168.1.2\"\nADGUARD_PORT=\"3000\"\nTARGET_IP=\"192.168.1.40\"\n\n# AI service domains to add\nDOMAINS=(\n \"ai.nuclide.systems\"\n \"chat.nuclide.systems\"\n \"mcp.nuclide.systems\"\n \"litellm.nuclide.systems\"\n \"s3.nuclide.systems\"\n)\n\necho \"\ud83d\udd27 Adding DNS entries to AdGuard...\"\n\nfor domain in \"${DOMAINS[@]}\"; do\n echo \" \u2192 $domain \u2192 $TARGET_IP\"\n # Note: This is a placeholder - actual API call needed\ndone\n\necho \"\u2705 Done! Add these in AdGuard UI manually.\"\n</code></pre>"},{"location":"services/adguard-dns/#nuclidelan-zone-lan-only-aliases-added-2026-05-24","title":"<code>*.nuclide.lan</code> zone (LAN-only aliases, added 2026-05-24)","text":"<p>30 A-records mapping stable names to host IPs. Use cases: (a) bypass Zoraxy for direct LAN access (Immich app on home Wi-Fi <code>nuclide</code>, NAS browsers, infra admin UIs), (b) decouple client config from IPs so when a service moves CT only the AdGuard rewrite changes.</p> <p>Naming convention: - Per-host (one per CT/VM/device): <code>unifi</code>, <code>dlink</code>, <code>pve</code>, <code>adguard</code>, <code>backrest</code>, <code>zoraxy</code>, <code>id</code>, <code>db</code>, <code>secrets</code>, <code>ops</code>, <code>nas</code>, <code>docker</code>, <code>nextcloud</code>, <code>dev</code>, <code>shepard</code>, <code>ha</code>, <code>mainsail</code> - Per-service (alias when only port differs from the host): <code>immich</code>, <code>vault</code>, <code>karakeep</code>, <code>memos</code>, <code>abs</code>, <code>gotify</code>, <code>n8n</code>, <code>chat</code>, <code>ai</code>, <code>mcp</code>, <code>grafana</code>, <code>prometheus</code></p> <p>Full list in <code>/opt/AdGuardHome/AdGuardHome.yaml</code> under <code>dns.rewrites:</code>. Auto-pushed nightly to <code>fkrebs/adguard-conf</code>.</p> <p>Adding a new service alias: 1. Append a <code>- {domain: <new>.nuclide.lan, answer: 192.168.x.y, enabled: true}</code> line 2. <code>systemctl restart AdGuardHome</code> on CT 102 (AdGuard doesn't support <code>reload</code> for rewrites) 3. Verify: <code>dig +short @192.168.1.2 <new>.nuclide.lan</code></p> <p>When a service moves CT (changes IP): edit only the AdGuard rewrite \u2192 restart. No client app needs an update \u2014 this is the whole point of the alias zone.</p>"},{"location":"services/adguard-dns/#alternative-useful-dns-entries","title":"Alternative Useful DNS Entries","text":""},{"location":"services/adguard-dns/#nextcloud-sync","title":"NextCloud & Sync","text":"<ul> <li><code>nc.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (NextCloud web)</li> <li><code>sync.nuclide.systems</code> \u2192 Internal IP of Sync client</li> </ul>"},{"location":"services/adguard-dns/#garage-s3","title":"Garage S3","text":"<ul> <li><code>storage.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (Garage internal, port 3900)</li> <li><code>s3-local.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (S3 API for local only)</li> </ul>"},{"location":"services/adguard-dns/#service-discovery","title":"Service Discovery","text":"<ul> <li><code>api.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (future API gateway)</li> <li><code>web.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (general web services)</li> </ul>"},{"location":"services/adguard-dns/#immich","title":"Immich","text":"<ul> <li><code>immich.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (Immich web & API)</li> <li><code>immich-api.nuclide.systems</code> \u2192 <code>192.168.1.40:2283</code> (Immich API only)</li> </ul>"},{"location":"services/adguard-dns/#paperless-ngx","title":"Paperless-ngx","text":"<ul> <li><code>pp.nuclide.systems</code> \u2192 <code>192.168.1.40</code> (Paperless web)</li> <li><code>pp-api.nuclide.systems</code> \u2192 <code>192.168.1.40:8000</code> (Paperless API)</li> </ul>"},{"location":"services/adguard-dns/#home-assistant","title":"Home Assistant","text":"<ul> <li><code>ha.nuclide.systems</code> \u2192 <code>192.168.1.60</code> (Home Assistant OS VM 100)</li> <li><code>homeassistant.local</code> \u2192 <code>192.168.1.60</code> (mDNS fallback)</li> </ul>"},{"location":"services/adguard-dns/#internal-dns-server-setup","title":"Internal DNS Server Setup","text":""},{"location":"services/adguard-dns/#option-a-use-adguard-as-forwarding-dns","title":"Option A: Use AdGuard as Forwarding DNS","text":"<p>Configure clients to use <code>192.168.1.2</code> as their DNS server:</p> <ol> <li>Proxmox Host: Edit <code>/etc/resolv.conf</code></li> <li>Docker containers: Add DNS in docker-compose.yml</li> <li>VMs/PFs: Configure network settings</li> </ol>"},{"location":"services/adguard-dns/#option-b-create-custom-zone-in-adguard","title":"Option B: Create Custom Zone in AdGuard","text":"<ol> <li>AdGuard \u2192 DNS Settings \u2192 Zone Management</li> <li>Add zone: <code>nuclide.systems</code></li> <li>Add A records for all subdomains with proper IPs</li> <li>Zone file format: <pre><code>@ IN SOA ns1.nuclide.systems. admin.nuclide.systems. (\n 1 ; Serial\n 3600 ; Refresh\n 1800 ; Retry\n 604800 ; Expire\n 86400 ) ; Minimum TTL\n\n@ IN NS ns1.nuclide.systems.\n@ IN A 192.168.1.40\nai IN A 192.168.1.40\nchat IN A 192.168.1.40\nmcp IN A 192.168.1.40\nl IN A 192.168.1.40 ; LiteLLM\ns3 IN A 192.168.1.40\nnc IN A 192.168.1.40 ; NextCloud\n</code></pre></li> </ol>"},{"location":"services/adguard-dns/#verification","title":"Verification","text":"<p>After adding DNS entries:</p> <pre><code># Test from any machine on network\ndig ai.nuclide.systems @192.168.1.2\ndig chat.nuclide.systems @192.168.1.2\n\n# Should return: 192.168.1.40\n</code></pre>"},{"location":"services/adguard-dns/#integration-with-mcpgateway-config","title":"Integration with MCP/Gateway Config","text":"<p>Update service configs to use domain names:</p>"},{"location":"services/adguard-dns/#in-litellm-config-optstacksailitellm-configconfigyaml","title":"In LiteLLM Config (<code>/opt/stacks/ai/litellm-config/config.yaml</code>)","text":"<pre><code>general_settings:\n proxy_base_url: https://ai.nuclide.systems\n control_plane_url: https://ai.nuclide.systems\n</code></pre>"},{"location":"services/adguard-dns/#in-karakeep-env","title":"In Karakeep (<code>.env</code>)","text":"<pre><code>OPENAI_BASE_URL=https://ai.nuclide.systems/v1\n</code></pre>"},{"location":"services/adguard-dns/#in-lobehub-env","title":"In LobeHub (<code>.env</code>)","text":"<pre><code>APP_URL=https://chat.nuclide.systems\nS3_ENDPOINT=https://s3.nuclide.systems\n</code></pre>"},{"location":"services/adguard-dns/#migration-checklist","title":"Migration Checklist","text":"<ul> <li>[ ] Add DNS entries to AdGuard</li> <li>[ ] Verify resolution: <code>dig ai.nuclide.systems</code></li> <li>[ ] Update service configs to use domains</li> <li>[ ] Test HTTPS connectivity</li> <li>[ ] Remove old hosts file entries (if any)</li> <li>[ ] Document for team</li> </ul>"},{"location":"services/adguard-dns/#notes","title":"Notes","text":"<ul> <li>AdGuard runs on LXC 102 at <code>192.168.1.2</code></li> <li>All AI services run on <code>192.168.1.40</code></li> <li>Using same IP for all domains is intentional (single endpoint)</li> <li>SSL certs are issued by Zoraxy (public domain validation)</li> <li>DNS-only setup doesn't require HTTP proxy (Zoraxy still handles HTTPS)</li> </ul>"},{"location":"services/arcane/","title":"Arcane","text":"<p>Web-based Docker management IDE. Main instance on CT 109 (<code>192.168.1.8:10002</code>, <code>arcane.nuclide.systems</code>). Manages containers on all Docker hosts via edge agents. Migrated from CT 104 \u2192 CT 109 on 2026-05-23.</p> <p>Stack: <code>/opt/stacks/arcane/docker-compose.yml</code> on CT 109. Image: <code>ghcr.io/getarcaneapp/arcane:latest</code> Auth: OIDC via Pocket ID, admin: <code>fkrebs@nucli.de</code></p>"},{"location":"services/arcane/#edge-agents","title":"Edge agents","text":"<p>An edge agent (<code>ghcr.io/getarcaneapp/arcane-headless:latest</code>) runs inside each remote CT and connects outbound to the main Arcane server via gRPC poll. The main server then manages that CT's Docker.</p>"},{"location":"services/arcane/#currently-deployed-agents","title":"Currently deployed agents","text":"CT Hostname Environment Compose path Status CT 101 shepard shepard <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 104 docker docker <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 105 nextcloud nextcloud <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 110 id id <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 111 dev dev <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 112 secrets secrets <code>/opt/stacks/ops-agents/docker-compose.yml</code> online CT 113 db db <code>/opt/stacks/db/docker-compose.yml</code> online nuc nuc NUC (built-in) (built-in environment) online"},{"location":"services/arcane/#adding-an-edge-agent-to-a-new-ct","title":"Adding an edge agent to a new CT","text":"<p>Step 1 \u2014 Create the environment in Arcane (requires Arcane API or UI access)</p> <p>With the admin CLI API key (stored in Arcane DB, regenerate if needed): <pre><code># Get or create an admin API key \u2014 see \"Admin API key\" section below\nAPI_KEY=\"arc_...\"\ncurl -s -X POST -H \"X-API-Key: $API_KEY\" -H \"Content-Type: application/json\" \\\n \"http://192.168.1.40:10002/api/environments\" \\\n -d '{\"name\":\"<ctname>\",\"apiUrl\":\"edge://<ctname>\",\"isEdge\":true}'\n# Note the environment id from the response\n</code></pre></p> <p>Step 2 \u2014 Generate and wire the AGENT_TOKEN</p> <p>Arcane stores the raw AGENT_TOKEN in <code>environments.access_token</code>. Insert it via: <pre><code># On PVE host, run: python3 /tmp/arcane-bootstrap.py\nimport argon2, secrets, sqlite3, uuid\nfrom datetime import datetime, timezone\n\ndb_path = \"/rpool/data/subvol-109-disk-0/opt/stacks/arcane/data/arcane.db\"\nenv_id = \"<id from step 1>\" # paste the environment UUID here\n\nconn = sqlite3.connect(db_path)\nraw_token = \"arc_\" + secrets.token_hex(32)\nkey_prefix = \"arc_\" + raw_token[4:12]\nph = argon2.PasswordHasher(memory_cost=65536, time_cost=3, parallelism=2)\nkey_hash = ph.hash(raw_token)\nkey_id = str(uuid.uuid4())\nnow = datetime.now(timezone.utc).isoformat()\n\nconn.execute(\n \"INSERT INTO api_keys (id, name, description, key_hash, key_prefix, environment_id, managed_by, created_at, updated_at)\"\n \" VALUES (?,?,?,?,?,?,?,?,?)\",\n (key_id, f\"Environment Bootstrap Key - {env_id[:8]}\",\n \"Auto-generated key for environment pairing\",\n key_hash, key_prefix, env_id, \"system\", now, now))\nconn.execute(\"UPDATE environments SET access_token=? WHERE id=?\", (raw_token, env_id))\nconn.commit()\nconn.close()\nprint(\"AGENT_TOKEN:\", raw_token)\n</code></pre></p> <p>Step 3 \u2014 Add the agent to the CT's compose</p> <pre><code> arcane-agent:\n image: ghcr.io/getarcaneapp/arcane-headless:latest\n container_name: arcane-agent\n restart: unless-stopped\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock\n - ./arcane-agent:/app/data\n environment:\n EDGE_AGENT: \"true\"\n EDGE_TRANSPORT: poll\n AGENT_TOKEN: ${ARCANE_AGENT_TOKEN}\n MANAGER_API_URL: http://192.168.1.8:10002\n deploy:\n resources:\n limits:\n cpus: \"0.5\"\n memory: 256M\n</code></pre> <p>Add <code>ARCANE_AGENT_TOKEN=<raw_token></code> to the CT's <code>.env</code>.</p> <p>Step 4 \u2014 Start and verify</p> <pre><code>docker compose up -d arcane-agent\ndocker logs arcane-agent 2>&1 | grep \"Edge gRPC tunnel\"\n# Expected: Edge gRPC tunnel connected to manager environment_id=<uuid>\n</code></pre> <p>The environment should flip to <code>online</code> in <code>environments.status</code> within ~5 seconds.</p>"},{"location":"services/arcane/#migration-to-ct-109-completed-2026-05-23","title":"Migration to CT 109 \u2014 completed 2026-05-23","text":"<p>Arcane migrated from CT 104 \u2192 CT 109. All edge agents updated with <code>MANAGER_API_URL: http://192.168.1.8:10002</code>. Zoraxy upstream updated to <code>192.168.1.8:10002</code>. SQLite DB at <code>/opt/stacks/arcane/data/</code> on CT 109.</p>"},{"location":"services/arcane/#admin-api-key-non-oidc-access","title":"Admin API key (non-OIDC access)","text":"<p>Arcane doesn't store plaintext API keys \u2014 use the DB-insert method above when a new admin key is needed. The <code>python3-argon2</code> package must be installed on the PVE host (<code>apt install python3-argon2</code>).</p> <p>The key inserted for CLI use during CT 113 provisioning (<code>arc_a9695182...</code>) is linked to <code>fkrebs</code> user in the api_keys table. Rotate it after provisioning work is complete by deleting the row: <pre><code>sqlite3 /rpool/data/subvol-109-disk-0/opt/stacks/arcane/data/arcane.db \\\n \"DELETE FROM api_keys WHERE name='admin-cli';\"\n</code></pre></p>"},{"location":"services/backrest/","title":"Backup Strategy","text":"<p>Single Backrest instance on CT 103 (192.168.1.3:9898) backing up offsite to JottaCloud via rclone. CT 103 has UNAS NFS mounted at <code>/mnt/pve/unas</code>, so it can read all service data without SSH-ing other hosts.</p> <p>Goal: every piece of critical state has an offsite copy. CT 104 loss is recoverable within hours; UNAS loss is recoverable (slower) from JottaCloud.</p>"},{"location":"services/backrest/#current-state","title":"Current state","text":""},{"location":"services/backrest/#backrest-ct-103-phase-1b-live-since-2026-05-21","title":"Backrest (CT 103) \u2014 Phase 1b live since 2026-05-21","text":"Item Value UI <code>http://192.168.1.3:9898</code> (LAN-only, auth disabled) Config <code>/opt/backrest/config/config.json</code> (timestamped <code>.bak</code> files on every edit) rclone remote <code>jottacloud:</code> Archive section (default device/mountpoint); token at <code>/root/.config/rclone/rclone.conf</code> Offsite plan JottaCloud Unlimited \u20ac9.91/mo (Norway, EEA) \u2014 see provider comparison BW throttle <code>RCLONE_BWLIMIT=05:30,7.5M 01:00,20M</code> (20 MB/s overnight, 7.5 MB/s daytime) <p>Repos (both <code>autoInitialize: true</code>; passwords currently <code>tapirnase</code> \u2014 see blind spot #5):</p> Repo Target Prune Check <code>services-repo</code> <code>rclone:jottacloud:services</code> weekly Sun 05:00, \u226410 % unused monthly, 10 % subset <code>media-repo</code> <code>rclone:jottacloud:media</code> monthly, \u226410 % unused monthly, 10 % subset <p>Plans:</p> Plan Repo Schedule Retention Paths <code>services-backup-plan</code> <code>services-repo</code> <code>0 1 * * 1-5</code> weekdays 01:00 7d \u00b7 4w \u00b7 6m \u00b7 1y 15 paths (below) <code>media-backup-plan</code> <code>media-repo</code> <code>30 1 * * 1-5</code> weekdays 01:30 4w \u00b7 6m \u00b7 1y <code>/mnt/pve/unas/media/images/library</code> <code>video-projects-plan</code> <code>media-repo</code> nightly 02:00 per config <code>/mnt/pve/unas/media/video-projects</code> <p>Services plan paths (all under <code>/mnt/pve/unas/</code>):</p> <pre><code>services/vaultwarden services/n8n services/memos\nservices/karakeep services/traccar services/gitea\nservices/coder services/nextcloud\nservices/arr-stack services/gluetun\nbackup/home-assistant backup/immich backup/nextcloud\n</code></pre> <p><code>services/shared-db</code> was removed when CT 113 came online \u2014 postgres is now WAL-G \u2192 Garage S3 \u2192 JottaCloud. <code>services/arcane</code> removed 2026-05-26 \u2014 Arcane decommissioned, replaced by Portainer. <code>services/pocketid</code> removed 2026-05-26 \u2014 Pocket-ID moved to CT 109 local FS (not UNAS); now covered by <code>ops-backup.timer</code> \u2192 Garage S3.</p> <p>JottaCloud web UI tip: rclone writes to the Archive section. The default landing page shows only Sync + Backup. Browse to <code>https://www.jottacloud.com/web/archive</code> to see the restic repos.</p>"},{"location":"services/backrest/#postgres-ct-113-phase-2-live-since-2026-05-21","title":"Postgres (CT 113) \u2014 Phase 2 live since 2026-05-21","text":"<p>Dedicated <code>db</code> LXC at <code>192.168.1.6:5432</code> running <code>postgres:17</code> in Docker, pgAdmin on <code>:5050</code>. WAL-G archives continuously to Garage S3 (<code>ct113-pg-backup</code> on CT 104):</p> <ul> <li><code>archive_mode = on</code>, <code>archive_command = 'wal-g wal-push %p'</code></li> <li>Daily <code>wal-g backup-push</code> cron at 02:00</li> <li>DBs hosted: <code>vaultwarden</code>, <code>paperless</code>, <code>litellm</code>, <code>memos</code>, <code>n8n</code>, <code>gitea</code>, <code>coder</code></li> </ul> <p>Garage \u2192 JottaCloud offsite sync runs daily 02:30 on CT 103 via <code>/usr/local/sbin/walg-offsite-sync.sh</code> (read-only Garage key <code>GKef577420aadd26d667f2ca4f</code>; mirrors <code>ct113-pg-backup</code>, <code>lobe-pg-backup</code>, <code>immich-pg-backup</code> \u2192 <code>jottacloud:WAL-G/</code>).</p> <p>Exceptions (stay on original hosts):</p> DB Host Why <code>lobe-postgres</code> (paradedb pg17) CT 104 Uses <code>pg_search</code> <code>USING bm25</code> indexes; stock postgres 17 can't host. WAL-G \u2192 <code>lobe-pg-backup</code> + Backrest secondary on data dir. Nextcloud AIO postgres CT 105 AIO manages it; Borg archives the whole stack to UNAS. Immich postgres CT 104 Version-pinned by Immich. WAL-G \u2192 <code>immich-pg-backup</code>."},{"location":"services/backrest/#coverage-map","title":"Coverage map","text":"Host / data Method Offsite CT 103 Backrest binary + config Manual (small) On-CT only CT 104 Docker app state on UNAS <code>services-backup-plan</code> \u2713 JottaCloud CT 104 <code>lobe-postgres</code> (paradedb) WAL-G \u2192 Garage \u2192 JottaCloud sync \u2713 CT 104 Immich postgres WAL-G \u2192 <code>immich-pg-backup</code> \u2192 sync \u2713 CT 113 postgres (7 DBs) WAL-G \u2192 <code>ct113-pg-backup</code> \u2192 sync \u2713 CT 101 Shepard Source in Gitea (gitea path covers it) \u2713 CT 102 AdGuard Phase 3 git push \u2192 <code>fkrebs/adguard-conf</code> \u2717 not deployed CT 104 Gitea + Coder <code>services/{gitea,coder}</code> (UNAS; moved from CT 111 2026-05-26) \u2713 CT 105 Nextcloud user files <code>services/nextcloud</code> \u2713 CT 105 Nextcloud AIO volumes AIO Borg \u2192 <code>/mnt/pve/unas/backup/nextcloud/</code> \u2192 Backrest \u2713 CT 108 Zoraxy Phase 3 git push \u2192 <code>fkrebs/zoraxy-conf</code> \u2717 not deployed CT 109 Portainer config Daily tar \u2192 Garage S3 <code>ct109-portainer-backup</code> \u2192 JottaCloud sync \u2713 CT 109 Pocket-ID Daily tar \u2192 Garage S3 <code>ct109-portainer-backup</code> (key <code>ops-*.tar.gz</code>) \u2192 JottaCloud sync via <code>ops-backup.timer</code> \u2713 CT 109 Infisical pg_dump in <code>ops-*.tar.gz</code> (same as above) \u2713 ~~CT 110 Pocket-ID~~ CT 110 destroyed 2026-05-26 \u2014 see CT 109 row above \u2713 ~~CT 111 Gitea + Coder~~ CT 111 destroyed 2026-05-26 \u2014 see CT 104 row above \u2713 ~~CT 112 Infisical~~ CT 112 destroyed 2026-05-26 \u2014 see CT 109 row above \u2713 VM 100 HAOS config Phase 3 git addon \u2192 <code>fkrebs/ha-config</code> \u2717 not deployed VM 100 HAOS daily tar HA \u2192 UNAS \u2192 Backrest \u2713 PVE host <code>/etc/pve/</code> Phase 3 git push \u2192 <code>fkrebs/pve-conf</code> \u2717 not deployed UNAS <code>media/images/library</code> Phase 1a <code>rclone sync</code> \u2192 <code>jottacloud:Photos/</code> \u2717 not deployed UNAS personal data (<code>documents</code>, <code>_sortMe</code>, <code>video-projects</code>, \u2026) Phase 1a <code>rclone sync</code> \u2192 <code>jottacloud:UNAS/</code> \u2717 not deployed"},{"location":"services/backrest/#intentionally-not-backed-up","title":"Intentionally not backed up","text":"<p>Immich thumbnails / encoded video, ComfyUI / Speaches models, arr-stack metadata, Redis / Meili / Elastic caches, CT 101 dev volumes \u2014 all regenerable or re-downloadable.</p>"},{"location":"services/backrest/#deferred-include-if-needed","title":"Deferred \u2014 include if needed","text":"Host Data Notes CT 109 Prometheus TSDB, Grafana dashboards Low priority \u2014 metrics are ephemeral; dashboards re-exportable from Grafana; add to services plan if needed CT 109 Portainer data WAL-G\u2013style daily tar \u2192 <code>ct109-portainer-backup</code> Garage S3 \u2192 JottaCloud offsite sync \u2713 (via walg-offsite-sync.sh)"},{"location":"services/backrest/#open-work","title":"Open work","text":""},{"location":"services/backrest/#phase-1a-rclone-sync-for-unas-personal-data","title":"Phase 1a \u2014 rclone sync for UNAS personal data","text":"<p>Two sync jobs on CT 103, Saturday 01:00:</p> <ul> <li>Job 1: <code>media/images/library/</code> \u2192 <code>jottacloud:Photos/</code> (Immich originals, visible at <code>jottacloud.com/photo</code>).</li> <li>Job 2: <code>/mnt/pve/unas/</code> \u2192 <code>jottacloud:UNAS/</code> (documents, video-projects, _sortMe, musical-sheets, audiobooks, ebooks, code; excludes Immich-generated, re-streamable media, torrents, <code>services/</code> already in Backrest).</li> </ul> <p>Initial upload \u2248 820 GB personal + 822 GB Immich (days). Script target <code>/usr/local/sbin/unas-sync.sh</code>. Full script in appendix.</p> <p>Tradeoff vs Restic: no point-in-time versions; deletions propagate. Acceptable for personal media.</p>"},{"location":"services/backrest/#phase-3-config-to-git-for-infrastructure","title":"Phase 3 \u2014 Config-to-git for infrastructure","text":"<p>Daily 03:00 cron on each host pushes config to a private Gitea repo. Repos already exist:</p> <ul> <li><code>fkrebs/zoraxy-conf</code> (CT 108 \u2014 <code>/opt/zoraxy/conf/</code>)</li> <li><code>fkrebs/adguard-conf</code> (CT 102 \u2014 <code>AdGuardHome.yaml</code>)</li> <li><code>fkrebs/pve-conf</code> (PVE \u2014 <code>/etc/pve/</code>, excludes <code>priv/</code>, <code>*.key</code>, <code>authkey.pub*</code>)</li> <li><code>fkrebs/ha-config</code> (VM 100 \u2014 <code>/config/</code>, excludes <code>secrets.yaml</code>, <code>.storage/</code>)</li> </ul> <p>Script template in appendix.</p>"},{"location":"services/backrest/#blind-spots","title":"Blind spots","text":"# Severity Issue Fix 4 MEDIUM n8n encryption key in <code>/opt/stacks/n8n/data/</code> on CT 104 local FS \u2014 not in any plan. If CT 104 dies, DB restore is unusable. Move n8n data volume to <code>/mnt/pve/unas/services/n8n/</code> (already in plan). 4b MEDIUM Nextcloud AIO Borg passphrase only in container env. Borg repo encrypted \u2014 without it, restore impossible. Store <code>BORG_PASSWORD</code> in Vaultwarden. 5 MEDIUM Both Backrest repo passwords are <code>tapirnase</code>. Rotate before first scheduled run completes; <code>restic key passwd</code> re-encrypts in place. Store in Vaultwarden. 6 LOW <code>litellm</code> DB password is the placeholder literal <code>litellm_password_here</code>. Generate real password; update <code>ai/.env</code>, <code>litellm-config/config.yaml</code>, CT 113 user. 7 LOW Migration dumps at <code>/mnt/pve/unas/dump/*-migration-20260521.sql</code> not in any plan. Decide: keep as manual archive or delete now that WAL-G is archiving."},{"location":"services/backrest/#operations","title":"Operations","text":""},{"location":"services/backrest/#restore","title":"Restore","text":"<p>UI: Repos \u2192 snapshots \u2192 Browse \u2192 file \u2192 Restore.</p> <p>CLI on CT 103:</p> <pre><code>restic -r rclone:jottacloud:services snapshots\nrestic -r rclone:jottacloud:services restore latest \\\n --target /restore \\\n --include /mnt/pve/unas/services/vaultwarden\n</code></pre>"},{"location":"services/backrest/#monitoring-planned","title":"Monitoring (planned)","text":"<p>In Backrest UI \u2192 each repo \u2192 Hooks:</p> <ul> <li><code>CONDITION_BACKUP_ERROR</code> / <code>CONDITION_CHECK_ERROR</code> \u2192 POST Gotify priority 8</li> <li><code>CONDITION_BACKUP_SUCCESS</code> \u2192 POST Gotify priority 3</li> </ul> <pre><code>curl -s -X POST 'http://gotify:80/message?token=TOKEN' \\\n -H 'Content-Type: application/json' \\\n -d '{\"title\":\"Backrest: {{.Plan}}\",\"message\":\"{{.Summary}}\",\"priority\":3}'\n</code></pre>"},{"location":"services/backrest/#applying-a-new-config","title":"Applying a new config","text":"<pre><code># on CT 103\nsystemctl stop backrest\ncp /opt/backrest/config/config.json /opt/backrest/config/config.json.bak.$(date +%Y%m%d-%H%M%S)\n# paste new config.json\nsystemctl start backrest\n</code></pre>"},{"location":"services/backrest/#history","title":"History","text":""},{"location":"services/backrest/#provider-comparison","title":"Provider comparison","text":"<p>Researched 2026-05-21. Sized for ~3 TB/mo.</p> Provider ~3 TB/mo Backend EU DC Egress Verdict JottaCloud Unlimited \u20ac9.91 flat (unlimited) rclone native Norway (EEA) Free \u2705 Chosen Hetzner BX31 \u20ac20.80 flat (10 TB) SFTP/WebDAV DE, FI Free 2.5\u00d7 the price, capped Backblaze B2 ~$18 pay-per-GB S3 Frankfurt Free (\u22643\u00d7 stored) US CLOUD Act risk Cloudflare R2 ~$45 pay-per-GB S3 EU auto Zero Expensive at scale; no EU residency lock Wasabi ~$21\u201324 S3 FRA/AMS Free (\u2264 stored) \u274c 90-day min billing per object \u2192 Restic prune disaster Storj DCS ~$30 S3 EU-geofenced 1\u00d7 free Complex, $5 minimum pCloud \u20ac399 one-time / 2 TB WebDAV Luxembourg Free WebDAV too slow Infomaniak kDrive ~\u20ac36+ / 3 TB WebDAV Switzerland Free WebDAV only; non-EU Proton Drive \u2014 rclone beta Switzerland Free rclone backend broken since late 2025"},{"location":"services/backrest/#phase-2-postgres-consolidation-onto-ct-113-done-2026-05-21","title":"Phase 2 \u2014 postgres consolidation onto CT 113 (\u2705 done 2026-05-21)","text":"<p>Decision: provision a dedicated <code>db</code> LXC (CT 113, 192.168.1.6) running postgres in Docker rather than reuse <code>shared-postgres</code> on CT 104. Direct-to-target avoided migrating twice (CT 111 \u2192 CT 104 \u2192 CT 113).</p> <p>Specs: Debian 12 unprivileged, 2 GB RAM, 2 cores, 20 GB local-zfs, <code>mp0=/mnt/pve/unas</code>. Stack in Gitea <code>fkrebs/stacks-db</code>. pgAdmin pre-registers postgres via <code>pgadmin-servers.json</code>.</p> <p>Migration order: vaultwarden \u2192 paperless \u2192 litellm \u2192 memos \u2192 n8n \u2192 gitea \u2192 coder. Procedure per DB:</p> <pre><code>docker exec <container> pg_dump -U <user> <db> > /mnt/pve/unas/dump/<db>-migration.sql\npsql -U postgres -c \"CREATE USER <user> WITH PASSWORD '...'; CREATE DATABASE <db> OWNER <user>;\"\npsql -U <user> <db> < /mnt/pve/unas/dump/<db>-migration.sql\n# Update connection strings \u2192 192.168.1.6:5432, restart service, verify, remove old PG container + volume\n</code></pre> <p>Stale DBs dropped from <code>shared-postgres</code> post-migration: <code>daytona</code>, <code>lobechat</code> (duplicate), <code>paradedb</code> (duplicate).</p> <p>WAL-G enabled 2026-05-21: <code>archive_mode = on</code>, <code>archive_command = 'wal-g wal-push %p'</code> via <code>ALTER SYSTEM</code>; first base backup verified (<code>base_000000010000000000000012</code>). Daily <code>wal-g backup-push</code> cron at 02:00. Garage \u2192 JottaCloud offsite sync added at 02:30.</p>"},{"location":"services/backrest/#phase-1a-rclone-sync-script","title":"Phase 1a rclone sync script","text":"<pre><code>#!/bin/bash\nset -e\nLOG=/var/log/rclone-unas-sync.log\n\n# Job 1: Immich originals \u2192 JottaCloud gallery\nrclone sync /mnt/pve/unas/media/images/library/ jottacloud:Photos/ \\\n --transfers=4 --checkers=8 \\\n --log-file=$LOG --log-level INFO\n\n# Job 2: all other irreplaceable personal data\nrclone sync /mnt/pve/unas/ jottacloud:UNAS/ \\\n --transfers=4 --checkers=8 \\\n --exclude \"media/images/library/**\" \\\n --exclude \"media/images/upload/**\" \\\n --exclude \"media/images/thumbs/**\" \\\n --exclude \"media/images/encoded-video/**\" \\\n --exclude \"media/images/profile/**\" \\\n --exclude \"media/images/backups/**\" \\\n --exclude \"media/movies/**\" \\\n --exclude \"media/emulation/**\" \\\n --exclude \"media/Torrents/**\" \\\n --exclude \"media/music/**\" \\\n --exclude \"media/podcasts/**\" \\\n --exclude \"services/**\" \\\n --exclude \"backup/**\" \\\n --exclude \"backup-staging/**\" \\\n --exclude \"test_perm\" \\\n --log-file=$LOG --log-level INFO\n</code></pre> <p>Cron (Saturday 01:00): <code>0 1 * * 6 /usr/local/sbin/unas-sync.sh</code></p>"},{"location":"services/backrest/#phase-3-config-to-git-scripts","title":"Phase 3 config-to-git scripts","text":"<p>Same pattern on each host. Replace <code>GITEA_TOKEN</code> with the value from <code>/opt/stacks/ai/.env</code> or a dedicated scoped Gitea token.</p> <p>CT 108 \u2014 Zoraxy (<code>/usr/local/sbin/zoraxy-conf-backup.sh</code>):</p> <pre><code>#!/bin/bash\nset -e\nREPO_URL=\"https://fkrebs:GITEA_TOKEN@git.nuclide.systems/fkrebs/zoraxy-conf.git\"\nWORK=\"/opt/zoraxy/conf\"\ngit -C \"$WORK\" init -b main -q 2>/dev/null || true\ngit -C \"$WORK\" remote set-url origin \"$REPO_URL\" 2>/dev/null \\\n || git -C \"$WORK\" remote add origin \"$REPO_URL\"\ngit -C \"$WORK\" add -A\ngit -C \"$WORK\" commit -q -m \"auto: $(date -u +%Y-%m-%dT%H:%M:%SZ)\" 2>/dev/null || true\ngit -C \"$WORK\" push -q origin main 2>&1 | grep -v \"Everything up-to-date\" || true\n</code></pre> <p>CT 102 \u2014 AdGuard \u2014 same template, copy <code>/opt/AdGuardHome/AdGuardHome.yaml</code> into <code>/tmp/adguard-conf-work</code> checkout first.</p> <p>PVE host \u2014 same template, <code>rsync -a --exclude='priv/' --exclude='*.key' --exclude='authkey.pub*' /etc/pve/ /tmp/pve-conf-work/</code> then commit.</p> <p>VM 100 \u2014 HA \u2014 native Git Pull addon pushing <code>/config/</code> (exclude <code>secrets.yaml</code>, <code>.storage/</code>) to <code>fkrebs/ha-config</code>. HA daily tars already covered by <code>services-backup-plan</code> via <code>/mnt/pve/unas/backup/home-assistant/</code>.</p> <p>Cron on each host: <code>0 3 * * * /usr/local/sbin/<host>-conf-backup.sh</code></p>"},{"location":"services/backrest/#why-wasabi-was-rejected","title":"Why Wasabi was rejected","text":"<p>Restic creates many small pack files during normal operation. Wasabi charges 90 days of storage per object regardless of deletion \u2014 every <code>restic forget --prune</code> generates surprise costs. Well-documented Restic-on-Wasabi trap.</p>"},{"location":"services/backrest/#why-lobe-postgres-stays-on-ct-104","title":"Why lobe-postgres stays on CT 104","text":"<p>Migration <code>0093_add_bm25_indexes_with_icu.sql</code> creates <code>USING bm25</code> indexes on 7 tables (<code>agents</code>, <code>topics</code>, <code>files</code>, <code>knowledge_bases</code>, <code>user_memories</code>, <code>chat_groups</code>, <code>user_memories_contexts</code>). <code>bm25</code> is paradedb-only (<code>pg_search</code> extension); stock postgres 17 has no such index access method and the migration fails. Container stays <code>paradedb/paradedb:latest-pg17</code>. WAL-G archives to <code>lobe-pg-backup</code>; Backrest secondary copies the data dir.</p>"},{"location":"services/backrest/#unas-data-inventory-basis-for-phase-1a-sizing","title":"UNAS data inventory (basis for Phase 1a sizing)","text":"Path Size Notes <code>media/documents/</code> 9.5 G Personal documents <code>media/video-projects/</code> 425 G Creative work, irreplaceable <code>media/musical-sheets/</code> 23 G <code>media/audiobooks/</code> 23 G <code>media/ebooks/</code> 6.8 G <code>media/3d-prints/</code> 65 M <code>media/Recipes/</code> 113 M <code>_sortMe/</code> 335 G <code>images/</code> 171 G, <code>work Flo/</code> 147 G, <code>Anne/</code> 17 G <code>code/</code> 6.7 M Excluded <code>media/movies/</code> 118 G re-streamable Excluded <code>media/emulation/</code> 57 G re-downloadable Excluded <code>media/Torrents/</code> 19 G temporary Excluded <code>media/music/</code>, <code>media/podcasts/</code> \u2014 re-streamable"},{"location":"services/cloud-gpu/","title":"Cloud GPU Extension \u2014 Scaleway L40S","text":"<p>Seamless on-demand GPU (FLUX.1-dev, video, multi-user) via a Scaleway L40S instance. The instance auto-starts on first request and shuts down after 45 min idle. Everything personal stays on the NUC.</p>"},{"location":"services/cloud-gpu/#pricing-current-as-of-may-2026","title":"Pricing (current as of May 2026)","text":"Resource Rate Notes L40S-1-48G compute \u20ac1.40/hour PAR-2, billed per minute Block volume 200 GB \u20ac16/month SSD, keeps models across restarts Flexible IP \u20ac0.004/hour Static IP for WireGuard endpoint Snapshots \u20ac0.000044/GB/h Only needed for image backups"},{"location":"services/cloud-gpu/#realistic-monthly-cost","title":"Realistic monthly cost","text":"Usage pattern Compute Storage Total Weekend sessions (8 h/week) \u20ac45 \u20ac16 ~\u20ac61/month Daily 1\u20132 h \u20ac63\u2013126 \u20ac16 ~\u20ac79\u2013142/month Heavy (4 h/day) \u20ac168 \u20ac16 ~\u20ac184/month Always-on (don't) \u20ac1,008 \u20ac16 \u20ac1,024/month <p>The on-demand proxy below makes \"daily 1\u20132 h\" the natural default \u2014 you just click generate, it starts automatically.</p>"},{"location":"services/cloud-gpu/#architecture","title":"Architecture","text":"<pre><code>NUC (home, always-on) Scaleway PAR-2 (on demand)\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nLobeChat \u2500\u2500\u25b6 gpu-proxy:8190 \u2500\u2500\u2500 wg1 \u2500\u2500\u25b6 ComfyUI :8188\ncomfyui-mcp \u2500\u2500\u25b6 (Docker) Ollama :11434 (optional)\n \u2502\n \u251c\u2500 start/stop via Scaleway API\n \u2514\u2500 idle watchdog (45 min \u2192 stop)\n</code></pre> <p><code>gpu-proxy</code> is a small Docker service that: 1. Forwards requests to the GPU instance 2. Auto-starts the Scaleway instance if it's stopped (cold-start ~90 s) 3. Shuts it down after 45 min with no traffic</p>"},{"location":"services/cloud-gpu/#prerequisites","title":"Prerequisites","text":"<pre><code># Scaleway CLI\ncurl -s https://raw.githubusercontent.com/scaleway/scaleway-cli/master/scripts/get.sh | sh\nscw init # enter API key + project ID\n</code></pre>"},{"location":"services/cloud-gpu/#step-1-persistent-block-volume-models-live-here","title":"Step 1 \u2014 Persistent block volume (models live here)","text":"<pre><code># Create 200 GB SSD volume in PAR-2\nscw block volume create name=nuclide-gpu-models size=200GB zone=fr-par-2\n# Note the volume ID: vol-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\n</code></pre> <p>Models are kept on this volume. The compute instance can be deleted and recreated freely.</p>"},{"location":"services/cloud-gpu/#step-2-create-the-l40s-instance-with-cloud-init","title":"Step 2 \u2014 Create the L40S instance with cloud-init","text":"<p>Save as <code>/opt/stacks/scripts/gpu-cloud-init.yaml</code>:</p> <pre><code>#cloud-config\npackage_update: true\npackages:\n - wireguard-tools\n - docker.io\n - docker-compose-v2\n - nvidia-driver-545\n - nvidia-container-toolkit\n\nwrite_files:\n - path: /etc/wireguard/wg0.conf\n permissions: '0600'\n content: |\n [Interface]\n Address = 10.200.0.1/24\n ListenPort = 51820\n PrivateKey = SCALEWAY_WG_PRIVKEY\n\n [Peer]\n # NUC\n PublicKey = NUC_WG_PUBKEY\n AllowedIPs = 10.200.0.2/32\n PersistentKeepalive = 25\n\n - path: /opt/gpu/docker-compose.yml\n content: |\n services:\n comfyui:\n image: yanwk/comfyui-boot:cu124\n container_name: comfyui\n ports:\n - \"10.200.0.1:8188:8188\"\n volumes:\n - /mnt/models:/root/ComfyUI/models\n - comfyui_config:/root/ComfyUI\n environment:\n - CLI_ARGS=--listen 0.0.0.0\n deploy:\n resources:\n reservations:\n devices:\n - driver: nvidia\n count: all\n capabilities: [gpu]\n restart: unless-stopped\n\n ollama:\n image: ollama/ollama:latest\n container_name: ollama\n ports:\n - \"10.200.0.1:11434:11434\"\n volumes:\n - /mnt/models/ollama:/root/.ollama\n deploy:\n resources:\n reservations:\n devices:\n - driver: nvidia\n count: all\n capabilities: [gpu]\n restart: unless-stopped\n\n volumes:\n comfyui_config:\n\nruncmd:\n # Mount the models volume (will be /dev/sdb or similar)\n - mkdir -p /mnt/models\n - |\n DISK=$(lsblk -ndo NAME,SIZE | awk '$2==\"200G\"{print \"/dev/\"$1}' | head -1)\n if [ -n \"$DISK\" ]; then\n blkid \"$DISK\" || mkfs.ext4 \"$DISK\"\n echo \"$DISK /mnt/models ext4 defaults 0 2\" >> /etc/fstab\n mount \"$DISK\" /mnt/models\n fi\n - mkdir -p /mnt/models/ollama\n # Enable WireGuard\n - systemctl enable --now wg-quick@wg0\n # Configure nvidia-container-toolkit\n - nvidia-ctk runtime configure --runtime=docker\n - systemctl restart docker\n # Start GPU services\n - cd /opt/gpu && docker compose up -d\n</code></pre> <p>Before using this file, replace: - <code>SCALEWAY_WG_PRIVKEY</code> \u2192 output of <code>wg genkey</code> (run on Scaleway instance side first) - <code>NUC_WG_PUBKEY</code> \u2192 output of <code>cat /etc/wireguard/nuc_wg_pub</code> (see Step 3)</p> <p>Launch the instance:</p> <pre><code># Generate WireGuard keys first (do this on the NUC)\nwg genkey | tee /etc/wireguard/scaleway_wg_priv | wg pubkey > /etc/wireguard/scaleway_wg_pub\nwg genkey | tee /etc/wireguard/nuc_wg_priv | wg pubkey > /etc/wireguard/nuc_wg_pub\n\n# Fill in cloud-init.yaml with the keys, then:\nINSTANCE_ID=$(scw instance server create \\\n type=L40S-1-48G \\\n image=ubuntu_jammy_gpu \\\n zone=fr-par-2 \\\n name=nuclide-gpu \\\n cloud-init=@/opt/stacks/scripts/gpu-cloud-init.yaml \\\n output=json | jq -r '.id')\n\necho \"Instance ID: $INSTANCE_ID\"\n\n# Attach the models volume\nscw instance server attach-volume \\\n server-id=$INSTANCE_ID \\\n volume-id=vol-xxxxxxxx \\\n zone=fr-par-2\n\n# Allocate a Flexible IP (static IP that survives instance restarts)\nFLEXIP_ID=$(scw instance ip create zone=fr-par-2 output=json | jq -r '.id')\nscw instance server attach-flexible-ip \\\n server-id=$INSTANCE_ID \\\n ip-id=$FLEXIP_ID \\\n zone=fr-par-2\n\nFLEXIP=$(scw instance ip get $FLEXIP_ID zone=fr-par-2 output=json | jq -r '.address')\necho \"Scaleway public IP: $FLEXIP\"\n</code></pre>"},{"location":"services/cloud-gpu/#step-3-wireguard-on-the-nuc","title":"Step 3 \u2014 WireGuard on the NUC","text":"<pre><code># /etc/wireguard/wg1.conf (separate from any existing VPN tunnel)\ncat > /etc/wireguard/wg1.conf << EOF\n[Interface]\nAddress = 10.200.0.2/24\nPrivateKey = $(cat /etc/wireguard/nuc_wg_priv)\n\n[Peer]\n# Scaleway GPU\nPublicKey = $(cat /etc/wireguard/scaleway_wg_pub)\nEndpoint = ${FLEXIP}:51820\nAllowedIPs = 10.200.0.1/32\nPersistentKeepalive = 25\nEOF\n\nchmod 600 /etc/wireguard/wg1.conf\nsystemctl enable --now wg-quick@wg1\n</code></pre> <p>Test when the instance is running: <pre><code>ping 10.200.0.1 # WireGuard tunnel\ncurl http://10.200.0.1:8188 # ComfyUI\n</code></pre></p>"},{"location":"services/cloud-gpu/#step-4-gpu-proxy-seamless-on-demand-startup","title":"Step 4 \u2014 gpu-proxy (seamless on-demand startup)","text":"<p>This Docker service runs on the NUC. It proxies to the GPU instance and auto-starts/stops it.</p>"},{"location":"services/cloud-gpu/#aigpu-proxydocker-composeyml","title":"<code>ai/gpu-proxy/docker-compose.yml</code>","text":"<pre><code>services:\n gpu-proxy:\n build: .\n container_name: gpu-proxy\n environment:\n - SCW_SECRET_KEY=${SCW_SECRET_KEY}\n - SCW_PROJECT_ID=${SCW_PROJECT_ID}\n - SCW_INSTANCE_ID=${SCW_INSTANCE_ID}\n - SCW_ZONE=fr-par-2\n - GPU_HOST=10.200.0.1\n - COMFYUI_PORT=8188\n - OLLAMA_PORT=11434\n - IDLE_TIMEOUT=2700 # 45 min\n ports:\n - \"8190:8190\" # ComfyUI proxy\n - \"8191:8191\" # Ollama proxy\n restart: unless-stopped\n networks:\n - shared_backend\n\nnetworks:\n shared_backend:\n external: true\n</code></pre>"},{"location":"services/cloud-gpu/#aigpu-proxyproxypy","title":"<code>ai/gpu-proxy/proxy.py</code>","text":"<pre><code>\"\"\"\nOn-demand GPU proxy. Auto-starts the Scaleway instance on first request,\nshuts it down after IDLE_TIMEOUT seconds of inactivity.\n\"\"\"\nimport asyncio, os, time, httpx, subprocess\nfrom fastapi import FastAPI, Request\nfrom fastapi.responses import StreamingResponse, JSONResponse\n\napp = FastAPI()\n\nSCW_KEY = os.environ[\"SCW_SECRET_KEY\"]\nSCW_PROJECT = os.environ[\"SCW_PROJECT_ID\"]\nINSTANCE_ID = os.environ[\"SCW_INSTANCE_ID\"]\nZONE = os.environ.get(\"SCW_ZONE\", \"fr-par-2\")\nGPU_HOST = os.environ.get(\"GPU_HOST\", \"10.200.0.1\")\nCOMFYUI_PORT = int(os.environ.get(\"COMFYUI_PORT\", 8188))\nOLLAMA_PORT = int(os.environ.get(\"OLLAMA_PORT\", 11434))\nIDLE_TIMEOUT = int(os.environ.get(\"IDLE_TIMEOUT\", 2700))\n\nSCW_API = f\"https://api.scaleway.com/instance/v1/zones/{ZONE}\"\nHEADERS = {\"X-Auth-Token\": SCW_KEY, \"Content-Type\": \"application/json\"}\n\n_state = {\"last_activity\": 0.0, \"starting\": False, \"up\": False}\n_lock = asyncio.Lock()\n\n\nasync def _scw(method: str, path: str, **kwargs):\n async with httpx.AsyncClient() as c:\n r = await c.request(method, f\"{SCW_API}{path}\", headers=HEADERS, **kwargs)\n r.raise_for_status()\n return r.json()\n\n\nasync def _instance_state() -> str:\n d = await _scw(\"GET\", f\"/servers/{INSTANCE_ID}\")\n return d[\"server\"][\"state\"] # running | stopped | stopping | starting\n\n\nasync def _start_instance():\n await _scw(\"POST\", f\"/servers/{INSTANCE_ID}/action\", json={\"action\": \"poweron\"})\n\n\nasync def _stop_instance():\n await _scw(\"POST\", f\"/servers/{INSTANCE_ID}/action\", json={\"action\": \"poweroff\"})\n\n\nasync def _wait_ready(host: str, port: int, timeout=180) -> bool:\n deadline = time.time() + timeout\n while time.time() < deadline:\n try:\n async with httpx.AsyncClient(timeout=3) as c:\n await c.get(f\"http://{host}:{port}/\")\n return True\n except Exception:\n await asyncio.sleep(5)\n return False\n\n\nasync def ensure_up(port: int) -> bool:\n async with _lock:\n if _state[\"up\"]:\n _state[\"last_activity\"] = time.time()\n return True\n if _state[\"starting\"]:\n return False # caller will retry\n _state[\"starting\"] = True\n\n try:\n state = await _instance_state()\n if state != \"running\":\n print(f\"[gpu-proxy] instance {state} \u2192 starting\")\n await _start_instance()\n # wait for API to report running\n for _ in range(60):\n await asyncio.sleep(5)\n if await _instance_state() == \"running\":\n break\n # wait for service to respond\n if await _wait_ready(GPU_HOST, port):\n async with _lock:\n _state[\"up\"] = True\n _state[\"last_activity\"] = time.time()\n print(\"[gpu-proxy] GPU instance ready\")\n return True\n return False\n finally:\n async with _lock:\n _state[\"starting\"] = False\n\n\nasync def idle_watchdog():\n while True:\n await asyncio.sleep(60)\n async with _lock:\n if not _state[\"up\"]:\n continue\n idle = time.time() - _state[\"last_activity\"]\n if idle > IDLE_TIMEOUT:\n print(f\"[gpu-proxy] idle {idle:.0f}s \u2192 stopping instance\")\n try:\n await _stop_instance()\n async with _lock:\n _state[\"up\"] = False\n _state[\"last_activity\"] = 0.0\n except Exception as e:\n print(f\"[gpu-proxy] stop error: {e}\")\n\n\n@app.on_event(\"startup\")\nasync def startup():\n asyncio.create_task(idle_watchdog())\n # Probe: is the instance already running from a previous session?\n try:\n if await _instance_state() == \"running\":\n if await _wait_ready(GPU_HOST, COMFYUI_PORT, timeout=10):\n async with _lock:\n _state[\"up\"] = True\n _state[\"last_activity\"] = time.time()\n print(\"[gpu-proxy] GPU already up on startup\")\n except Exception:\n pass\n\n\nasync def _proxy(request: Request, host: str, port: int):\n _state[\"last_activity\"] = time.time()\n ready = await ensure_up(port)\n if not ready:\n # Starting up \u2014 keep trying for up to 3 min\n for _ in range(36):\n await asyncio.sleep(5)\n if _state[\"up\"]:\n break\n else:\n return JSONResponse({\"error\": \"GPU instance failed to start\"}, 503)\n\n url = f\"http://{host}:{port}{request.url.path}\"\n if request.url.query:\n url += f\"?{request.url.query}\"\n body = await request.body()\n client = httpx.AsyncClient(timeout=httpx.Timeout(None, connect=10))\n req = client.build_request(request.method, url,\n headers={k: v for k, v in request.headers.items()\n if k.lower() not in {\"host\", \"content-length\"}},\n content=body)\n try:\n resp = await client.send(req, stream=True)\n except httpx.ConnectError:\n await client.aclose()\n return JSONResponse({\"error\": \"GPU not reachable\"}, 502)\n\n async def stream():\n async for chunk in resp.aiter_raw():\n yield chunk\n await resp.aclose()\n await client.aclose()\n\n return StreamingResponse(stream(), status_code=resp.status_code,\n headers=dict(resp.headers),\n media_type=resp.headers.get(\"content-type\"))\n\n\n@app.api_route(\"/status\", methods=[\"GET\"])\nasync def status():\n try:\n scw_state = await _instance_state()\n except Exception as e:\n scw_state = f\"error: {e}\"\n return {\"instance\": scw_state, \"proxy_up\": _state[\"up\"],\n \"idle_s\": int(time.time() - _state[\"last_activity\"]) if _state[\"up\"] else None}\n\n\n@app.api_route(\"/{path:path}\", methods=[\"GET\",\"POST\",\"PUT\",\"DELETE\",\"OPTIONS\",\"PATCH\"])\nasync def comfyui_proxy(request: Request, path: str):\n return await _proxy(request, GPU_HOST, COMFYUI_PORT)\n</code></pre>"},{"location":"services/cloud-gpu/#aigpu-proxydockerfile","title":"<code>ai/gpu-proxy/Dockerfile</code>","text":"<pre><code>FROM python:3.12-slim\nRUN pip install fastapi uvicorn httpx\nCOPY proxy.py /app/proxy.py\nWORKDIR /app\nEXPOSE 8190 8191\nCMD [\"uvicorn\", \"proxy:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8190\"]\n</code></pre>"},{"location":"services/cloud-gpu/#step-5-wire-up-nuc-services","title":"Step 5 \u2014 Wire up NUC services","text":"<p>Add to <code>ai/.env</code>: <pre><code>SCW_SECRET_KEY=<your-scaleway-api-key>\nSCW_PROJECT_ID=<your-project-id>\nSCW_INSTANCE_ID=<instance-id-from-step-2>\n</code></pre></p> <p>In <code>ai/lobehub.yml</code>: <pre><code>- 'COMFYUI_BASE_URL=http://gpu-proxy:8190'\n</code></pre></p> <p>In <code>ai/mcp-gateway/server.py</code> (SERVERS dict): <pre><code>\"comfyui\": {\n \"static\": True,\n \"upstream\": \"http://gpu-proxy:8190\",\n \"group\": \"image\",\n},\n</code></pre></p> <p>For Immich ML (optional), add to <code>immich/docker-compose.yml</code>: <pre><code>environment:\n - IMMICH_MACHINE_LEARNING_URL=http://gpu-proxy:8192 # add a third port for ML\n</code></pre></p>"},{"location":"services/cloud-gpu/#step-6-deploy","title":"Step 6 \u2014 Deploy","text":"<pre><code># Deploy the proxy\ncd /opt/stacks/ai/gpu-proxy\ndocker compose up -d\n\n# Restart LobeChat and MCP gateway to pick up new env\ncd /opt/stacks/ai\ndocker compose -f lobehub.yml up -d --force-recreate lobe\ncd mcp-gateway && docker compose up -d --force-recreate mcp-gateway\n</code></pre>"},{"location":"services/cloud-gpu/#what-the-user-experience-looks-like","title":"What the user experience looks like","text":"<ol> <li>Open LobeChat, generate an image \u2192 first request triggers auto-start</li> <li>\"Starting GPU\u2026\" \u2014 ComfyUI returns a brief wait (up to 90 s cold start)</li> <li>Images generate normally; the proxy keeps the instance alive</li> <li>After 45 min with no requests \u2192 instance stops automatically</li> <li><code>/status</code> endpoint on port 8190: <code>{\"instance\":\"stopped\",\"proxy_up\":false}</code></li> </ol>"},{"location":"services/cloud-gpu/#quick-start-stop-scripts","title":"Quick-start / stop scripts","text":"<pre><code># /usr/local/bin/nuclide-gpu (NUC helper)\n#!/bin/bash\nZONE=fr-par-2\nID=$(docker exec gpu-proxy env | grep SCW_INSTANCE_ID | cut -d= -f2)\ncase \"$1\" in\n start) scw instance server action action=poweron server-id=$ID zone=$ZONE ;;\n stop) scw instance server action action=poweroff server-id=$ID zone=$ZONE ;;\n status) curl -s http://localhost:8190/status | python3 -m json.tool ;;\n *) echo \"Usage: nuclide-gpu start|stop|status\" ;;\nesac\n</code></pre>"},{"location":"services/cloud-gpu/#scaleway-firewall-security-groups","title":"Scaleway firewall (security groups)","text":"<p>On the Scaleway instance, only expose WireGuard. All services bind to <code>10.200.0.1</code> (WireGuard IP):</p> <pre><code>ufw default deny incoming\nufw allow 51820/udp # WireGuard from anywhere\nufw allow from 10.200.0.0/24 # full access over tunnel\nufw enable\n</code></pre>"},{"location":"services/cloud-gpu/#model-storage-layout-on-the-200-gb-volume","title":"Model storage layout (on the 200 GB volume)","text":"<pre><code>/mnt/models/\n\u251c\u2500\u2500 checkpoints/ # FLUX.1-dev, SDXL, etc.\n\u251c\u2500\u2500 vae/\n\u251c\u2500\u2500 loras/\n\u251c\u2500\u2500 controlnet/\n\u251c\u2500\u2500 ollama/ # Ollama model blobs\n\u2514\u2500\u2500 upscale_models/\n</code></pre> <p>FLUX.1-dev (FP16): ~24 GB FLUX.1-schnell (NF4): ~8 GB Ollama llama3:70b (Q4): ~40 GB \u2014 fits alongside FLUX on 200 GB with room for LoRAs.</p> <p>To pre-download models after first boot: <pre><code>ssh root@10.200.0.1 # via WireGuard\ndocker exec comfyui python3 -c \"\nfrom huggingface_hub import hf_hub_download\nhf_hub_download('black-forest-labs/FLUX.1-dev', 'flux1-dev.safetensors',\n local_dir='/root/ComfyUI/models/checkpoints')\n\"\n</code></pre></p>"},{"location":"services/comfyui/","title":"ComfyUI \u2192 LobeChat via MCP","text":"<p>Image generation (FLUX.1-schnell GGUF on the Intel Arc iGPU) exposed to LobeChat as an MCP tool. Chosen over LobeChat's native ComfyUI provider because that provider is hardcoded to non-GGUF nodes and is not configurable without forking LobeChat (see <code>docs/mcp-gateway-requirements.md</code> research notes).</p>"},{"location":"services/comfyui/#components","title":"Components","text":"<ul> <li><code>ai/mcp-servers/comfyui/</code> \u2014 purpose-built MCP server (async queue edition)</li> <li><code>server.py</code> \u2014 FastMCP, streamable-HTTP on <code>:8000</code> at <code>/mcp</code>. Six tools:<ul> <li><code>generate_image(prompt, width, height, steps, seed)</code> \u2014 txt2img; returns <code>job_id</code> immediately</li> <li><code>img2img(prompt, images, strength, steps, seed)</code> \u2014 img2img for one or more images; one job per image; images can be URLs, base64 data URIs, or <code>job:<id></code> references</li> <li><code>get_job_status(job_ids)</code> \u2014 returns status text + inline PNG for completed jobs</li> <li><code>list_queue(limit)</code> \u2014 lists all jobs in the in-process registry</li> <li><code>cancel_job(job_id)</code> \u2014 cancels queued/running job (user must confirm first)</li> <li><code>list_recent_images(n)</code> \u2014 shows n most recent completed images for reference/chaining</li> </ul> </li> <li>Background polling thread updates job state every 3 s via ComfyUI <code>/history</code>.</li> <li><code>flux-gguf-api.json</code> \u2014 txt2img workflow; nodes 4=prompt, 6=size, 8=steps/seed.</li> <li><code>flux-img2img-api.json</code> \u2014 img2img workflow; node 11=LoadImage, 12=VAEEncode, 8=KSampler with denoise.</li> <li><code>Dockerfile</code>, <code>requirements.txt</code> (<code>mcp[cli]</code>, <code>httpx</code>).</li> <li><code>ai/comfyui-mcp.yml</code> \u2014 compose; container <code>comfyui-mcp</code> on <code>shared_backend</code> (reaches <code>comfyui:8188</code>; reachable by <code>lobehub</code>, which is on <code>shared_backend</code>). Host port <code>18003:8000</code> for testing/Zoraxy.</li> </ul>"},{"location":"services/comfyui/#async-workflow","title":"Async workflow","text":"<pre><code>generate_image(\"a red apple\") \u2192 \"Job submitted: txt-1a8bbeda\" (< 1 s)\n [ComfyUI rendering... ~90-300 s]\nget_job_status([\"txt-1a8bbeda\"]) \u2192 status text + inline PNG image\n\n# Chain: use previous output as img2img input (no bytes through LLM)\nimg2img(\"add a blue bowl\", images=[\"job:txt-1a8bbeda\"], strength=0.6)\n \u2192 \"img-2b9ccefa\"\n</code></pre>"},{"location":"services/comfyui/#verified","title":"Verified","text":"<ul> <li>All 6 tools discovered via MCP <code>tools/list</code>.</li> <li><code>generate_image</code> returns in < 0.1 s (non-blocking); job shows in <code>list_queue</code> as \"running\".</li> <li>Previous end-to-end: <code>generate_image</code> \u2192 <code>get_job_status</code> \u2192 inline PNG ~94 s bare, ~160 s via MCP.</li> <li>LobeChat v2.1.58 renders MCP <code>image</code> blocks (uploads to Garage S3 \u2192 inline <code></code>).</li> </ul>"},{"location":"services/comfyui/#known-characteristics-limitations","title":"Known characteristics / limitations","text":"<ul> <li>In-memory registry: job state is lost on container restart. Resubmit if needed.</li> <li>No auth on the MCP server (internal <code>shared_backend</code> only). Host port 18003 is LAN-exposed and unauthenticated \u2014 fine for a trusted homelab LAN.</li> <li>~90-300 s/image latency; use <code>get_job_status()</code> to poll \u2014 never block waiting.</li> </ul>"},{"location":"services/comfyui/#final-hookup-manual-lobechat-uidb-no-server-mode-config-path","title":"Final hookup (manual \u2014 LobeChat UI/DB, no server-mode config path)","text":"<p>LobeChat \u2192 Settings \u2192 Skills (Tools) \u2192 Skill Store \u2192 Custom \u2192 Import JSON:</p> <pre><code>{\n \"mcpServers\": {\n \"comfyui-flux\": {\n \"type\": \"http\",\n \"url\": \"http://comfyui-mcp:8000/mcp\"\n }\n }\n}\n</code></pre> <p>Then enable the <code>comfyui-flux</code> skill in an agent/chat and ask the model to \"generate an image of \u2026\". Images appear inline in the conversation.</p>"},{"location":"services/comfyui/#ops","title":"Ops","text":"<ul> <li>Build/deploy: <code>cd /opt/stacks/ai && docker compose -f comfyui-mcp.yml up -d --build</code></li> <li>Logs: <code>docker logs comfyui-mcp</code></li> <li>The workflow is the single source of truth in <code>flux-gguf-api.json</code>; keep it in sync with <code>ai/comfyui/workflows/flux-schnell-api.json</code> if the graph changes.</li> </ul>"},{"location":"services/databases/","title":"Databases","text":"<p>All application databases live on CT 113 (<code>192.168.1.6:5432</code>) after Phase 2 migration. The LXC runs <code>postgres:17</code> in Docker at <code>/opt/stacks/db/</code>, tracked in Gitea <code>fkrebs/stacks-db</code>. WAL-G archives to Garage S3 bucket <code>ct113-pg-backup</code> on CT 104 (<code>http://192.168.1.40:10004</code>).</p>"},{"location":"services/databases/#database-inventory","title":"Database inventory","text":"DB Owner user Size (pre-migration) Service Stack location <code>vaultwarden</code> vaultwarden 11 MB Vaultwarden CT 104 <code>/opt/stacks/vaultwarden/</code> <code>paperless</code> paperless 20 MB Paperless-ngx CT 104 <code>/opt/stacks/apps/paperless-ngx/</code> <code>litellm</code> litellm 212 MB LiteLLM CT 104 <code>/opt/stacks/ai/</code> <code>memos</code> memos 9 MB Memos CT 104 <code>/opt/stacks/memos/</code> <code>n8n</code> n8n 12 MB n8n CT 104 <code>/opt/stacks/n8n/</code> <code>gitea</code> gitea 15 MB Gitea CT 111 <code>/opt/stacks/gitea/</code> <code>coder</code> coder 17 MB Coder CT 111 <code>/opt/stacks/coder/</code> <p>Not on CT 113:</p> DB Container Reason <code>lobechat</code> <code>lobe-postgres</code> (paradedb) on CT 104 LobeChat decommissioned 2026-05-26 \u2014 DB retained pending cleanup; no active service <code>immich</code> <code>immich_postgres</code> on CT 104 Version-pinned by Immich AIO <code>nextcloud</code> Nextcloud AIO on CT 105 AIO manages its own postgres"},{"location":"services/databases/#connection-strings-post-migration-target","title":"Connection strings (post-migration target)","text":"Service Connection string Vaultwarden <code>postgresql://vaultwarden:<pw>@192.168.1.6:5432/vaultwarden</code> Paperless <code>PAPERLESS_DBHOST: 192.168.1.6</code> LiteLLM <code>postgresql://litellm:<pw>@192.168.1.6:5432/litellm</code> (in <code>ai/.env</code> and <code>litellm-config/config.yaml</code>) Memos <code>postgresql://memos:<pw>@192.168.1.6:5432/memos?sslmode=disable</code> n8n <code>DB_POSTGRESDB_HOST=192.168.1.6</code> Gitea <code>GITEA__database__HOST: 192.168.1.6:5432</code> Coder <code>postgresql://coder:<pw>@192.168.1.6:5432/coder?sslmode=disable</code> <p>Passwords are in each service's <code>.env</code> file (never committed to git). See init script at <code>/opt/stacks/shared-db/init/01-create-users-dbs.sql</code> on CT 104 for the original credential set.</p>"},{"location":"services/databases/#migration-procedure-one-db-at-a-time","title":"Migration procedure (one DB at a time)","text":"<p>Order: vaultwarden \u2192 paperless \u2192 litellm \u2192 memos \u2192 n8n \u2192 gitea \u2192 coder</p> <pre><code># 1. Stop the service\n# docker compose -f <compose> stop <service>\n\n# 2. Dump from source (via PVE host)\n# For CT 104 services:\npct exec 104 -- docker exec -i shared-postgres pg_dump -U postgres <db> \\\n > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql\n\n# For CT 111 services:\npct exec 111 -- docker exec -i <container> pg_dump -U <user> <db> \\\n > /mnt/pve/unas/dump/<db>-migration-$(date +%Y%m%d).sql\n\n# 3. Create user + DB on CT 113\npct exec 113 -- docker exec -i postgres psql -U postgres <<EOF\nCREATE USER <user> WITH PASSWORD '<pw>';\nCREATE DATABASE <db> OWNER <user>;\nEOF\n\n# 4. Restore on CT 113\npct exec 113 -- bash -c \"docker exec -i postgres psql -U postgres -d <db>\" \\\n < /mnt/pve/unas/dump/<db>-migration-*.sql\n\n# 5. Update service connection string (shared-postgres \u2192 192.168.1.6)\n# Edit compose or .env\n\n# 6. Start service; verify logs and function\n\n# 7. Verify, then old DB/container can be removed\n</code></pre>"},{"location":"services/databases/#post-migration-cleanup","title":"Post-migration cleanup","text":"<p>After all 7 DBs are migrated and verified:</p> <ol> <li>Drop stale DBs from <code>shared-postgres</code>: <code>daytona</code>, <code>lobechat</code> (duplicate \u2014 real one is in <code>lobe-postgres</code>), <code>paradedb</code></li> <li>Stop and remove <code>shared-postgres</code> container + named volume <code>shared-pgdata</code></li> <li>Stop and remove <code>gitea-db</code> and <code>coder-db</code> containers + volumes on CT 111</li> <li>Update Backrest services plan: remove <code>shared-db</code> path, update to CT 113 WAL-G output</li> <li>Enable PVE protection on CT 113 (prevents accidental delete)</li> </ol>"},{"location":"services/databases/#pgadmin","title":"pgAdmin","text":"<p>pgAdmin on CT 113 at <code>http://192.168.1.6:5050</code> \u2014 pre-registered server: CT 113 postgres. Credentials in <code>/opt/stacks/db/.env</code> (admin@nucli.de).</p>"},{"location":"services/databases/#wal-g-monitoring","title":"WAL-G monitoring","text":"<p>WAL-G runs as a root crontab on CT 113 (<code>0 2 * * * docker exec -u postgres postgres wal-g backup-push ...</code>) and logs to <code>/var/log/walg-backup.log</code>. A silent stall went undetected for 13 hours in the past; monitoring was added to catch this.</p> <p>Textfile collector (<code>/usr/local/bin/walg-metrics.sh</code>) runs every 10 minutes via <code>walg-metrics.timer</code> and writes <code>/var/lib/prometheus/node-exporter/walg.prom</code>. <code>prometheus-node-exporter</code> (native systemd, port 9100) picks up the file via <code>--collector.textfile.directory=/var/lib/prometheus/node-exporter</code>.</p> <p>Metrics emitted: - <code>walg_last_success_timestamp_seconds{db=\"postgres\"}</code> \u2014 unix timestamp of last \"Wrote backup\" in log - <code>walg_archive_status{db=\"postgres\"}</code> \u2014 1 if backup within 25h, 0 if older or log missing</p> <p>Prometheus (CT 109) scrapes CT 113 as job <code>node-ct113</code>. Alert rules at <code>/opt/stacks/monitoring/prometheus/rules/walg.yml</code>: - <code>WalgArchiveStale</code> (warning): backup age > 2h, for 5m - <code>WalgArchiveFailed</code> (critical): archive_status == 0, for 5m</p>"},{"location":"services/dev-environment/","title":"Dev Environment \u2014 CT 104 (dev.nuclide.systems)","text":"<p>Migrated from CT 111 to CT 104 on 2026-05-26. CT 111 (\"dev\") is decommissioned; LXC pending removal.</p> <p>CT 104 hosts the self-hosted development platform: Coder (dev workspaces) + Gitea (internal repos).</p>"},{"location":"services/dev-environment/#services","title":"Services","text":"Service URL Port Coder https://dev.nuclide.systems 7080 Gitea https://git.nuclide.systems 3000 <p>Both use Pocket-ID OIDC (<code>https://id.nuclide.systems</code>) for SSO.</p>"},{"location":"services/dev-environment/#coder-dev-workspaces","title":"Coder \u2014 Dev Workspaces","text":""},{"location":"services/dev-environment/#what-it-is","title":"What it is","text":"<p>Coder provisions isolated Docker-based dev environments (workspaces) on CT 111. Each workspace has: - A full Linux environment with your tools - Persistent home dir on UNAS (<code>/mnt/pve/unas/services/coder/</code>) - Intel Arc GPU renderD128 available - VS Code Server (browser or desktop SSH tunnel)</p>"},{"location":"services/dev-environment/#first-time-setup","title":"First-time setup","text":"<ol> <li>Go to <code>https://dev.nuclide.systems</code> \u2192 log in via Pocket-ID</li> <li>Create a workspace from a template (admin must create templates first)</li> <li>Connect via VS Code: install Coder extension \u2192 sign in \u2192 open workspace</li> </ol>"},{"location":"services/dev-environment/#vs-code-connection","title":"VS Code connection","text":"<p><pre><code># Install Coder CLI on your local machine\ncurl -fsSL https://coder.com/install.sh | sh\n\n# Authenticate\ncoder login https://dev.nuclide.systems\n\n# Open workspace in VS Code\ncoder open <workspace-name>\n</code></pre> Or use the Coder VS Code extension directly from the marketplace (<code>coder.coder-remote</code>).</p>"},{"location":"services/dev-environment/#claude-code-inside-a-workspace","title":"Claude Code inside a workspace","text":"<p><pre><code># Inside the workspace terminal\nnpm install -g @anthropic/claude-code\nclaude\n</code></pre> Claude Code runs inside the workspace container \u2014 same environment, same files, same GPU.</p>"},{"location":"services/dev-environment/#coder-mcp-ai-agent-sandbox-execution","title":"Coder MCP (AI agent sandbox execution)","text":"<p>Add to Claude Code's MCP config (<code>~/.claude/claude_desktop_config.json</code> or via <code>/mcp add</code>): <pre><code>{\n \"mcpServers\": {\n \"coder\": {\n \"command\": \"coder\",\n \"args\": [\"mcp\", \"server\"],\n \"env\": {\n \"CODER_URL\": \"https://dev.nuclide.systems\",\n \"CODER_TOKEN\": \"<your-api-token>\"\n }\n }\n }\n}\n</code></pre> Claude can then create workspaces, execute code, and read output via MCP tools: - <code>coder_list_workspaces</code> - <code>coder_create_workspace</code> - <code>coder_execute_command</code> \u2190 sandbox code execution - <code>coder_start_workspace</code> / <code>coder_stop_workspace</code></p> <p>Get your API token: <code>coder tokens create</code></p>"},{"location":"services/dev-environment/#creating-workspace-templates","title":"Creating workspace templates","text":"<p>Templates are Terraform configs stored in Gitea. Basic Docker template: <pre><code>coder templates push <template-name> --directory ./template/\n</code></pre></p>"},{"location":"services/dev-environment/#gitea-internal-repos","title":"Gitea \u2014 Internal Repos","text":""},{"location":"services/dev-environment/#what-it-is_1","title":"What it is","text":"<p>Self-hosted Git for internal infrastructure: compose files, CT configs, dotfiles, Coder templates. Not the primary remote for Claude Code collaboration \u2014 use GitHub for that.</p>"},{"location":"services/dev-environment/#first-time-setup-admin","title":"First-time setup (admin)","text":"<ol> <li>Go to <code>https://git.nuclide.systems</code> \u2192 complete installation wizard</li> <li>Set admin account, confirm DB settings (pre-filled from env)</li> <li>Add OIDC provider: Admin \u2192 Site Administration \u2192 Authentication Sources</li> <li>Auth type: OAuth2</li> <li>Provider: OpenID Connect</li> <li>Discovery URL: <code>https://id.nuclide.systems/.well-known/openid-configuration</code></li> <li>Client ID/Secret: create a new client in Pocket-ID for Gitea</li> </ol>"},{"location":"services/dev-environment/#ssh-access","title":"SSH access","text":"<pre><code># Gitea SSH runs on port 222\ngit clone ssh://git@git.nuclide.systems:222/<user>/<repo>.git\n\n# Or add to ~/.ssh/config:\nHost git.nuclide.systems\n Port 222\n IdentityFile ~/.ssh/id_ed25519\n</code></pre>"},{"location":"services/dev-environment/#recommended-repos-to-create","title":"Recommended repos to create","text":"Repo Contents <code>infra/proxmox</code> <code>/etc/pve/</code> snapshots, CT configs <code>infra/stacks</code> Compose files from CT 104/101/111 <code>infra/docs</code> Mirror of <code>/docs/</code> on Proxmox host <code>dev/templates</code> Coder workspace Terraform templates"},{"location":"services/dev-environment/#storage-layout-unas","title":"Storage layout (UNAS)","text":"<pre><code>/mnt/pve/unas/services/\n\u251c\u2500\u2500 coder/ # Coder workspace home dirs (persistent)\n\u2514\u2500\u2500 gitea/ # Gitea repos + data\n</code></pre>"},{"location":"services/dev-environment/#ct-104-specs-current-host","title":"CT 104 specs (current host)","text":"IP 192.168.1.40 Cores 16 RAM 48GB Rootfs 200GB local-zfs UNAS /mnt/pve/unas (mp0) \u2014 coder + gitea data paths unchanged GPU renderD128 (Intel Arc Xe, idmapped) <p>Compose files at <code>/opt/stacks/coder/</code> and <code>/opt/stacks/gitea/</code> on CT 104. Gitea uses Redis (<code>gitea-redis</code>) for queue/cache/session \u2014 required because CT 104 uses idmapped NFS which doesn't support LevelDB file locks. Act-runner at <code>/opt/stacks/act-runner/</code> \u2014 runner name <code>ct104-runner</code>.</p>"},{"location":"services/dev-environment/#oidc-clients-in-pocket-id","title":"OIDC clients in Pocket-ID","text":"Client Callback URL Coder <code>https://dev.nuclide.systems/api/v2/users/oidc/callback</code> Gitea Add via Gitea admin UI (see above) <p>To create new OIDC clients programmatically, see <code>/docs/proxmox-optimizations.md</code> \u00a7 OIDC client creation via SQLite.</p>"},{"location":"services/dev-environment/#auth-lockdown-sso-only","title":"Auth lockdown (SSO-only)","text":"<p>Both Gitea and Coder are locked to Pocket-ID OIDC only. Local password and GitHub login are disabled. Passkey/WebAuthn login to Gitea remains available because it's tied to OIDC accounts, not to a separate password.</p>"},{"location":"services/dev-environment/#compose-env-flags","title":"Compose env flags","text":"<p>Coder (<code>/opt/stacks/coder/compose.yaml</code>): <pre><code>CODER_OIDC_ISSUER_URL: \"https://id.nuclide.systems\"\nCODER_OIDC_CLIENT_ID: \"${CODER_OIDC_CLIENT_ID}\"\nCODER_OIDC_CLIENT_SECRET: \"${CODER_OIDC_CLIENT_SECRET}\"\nCODER_OIDC_ALLOW_SIGNUPS: \"true\"\nCODER_OIDC_EMAIL_DOMAIN: \"nucli.de\"\nCODER_DISABLE_PASSWORD_AUTH: \"true\"\nCODER_OAUTH2_GITHUB_DEFAULT_PROVIDER_ENABLE: \"false\"\n</code></pre> Note: the bundled \"GitHub external auth provider\" log line is for workspaces cloning from GitHub, not for login \u2014 leaving it on is fine.</p> <p>Gitea (<code>/opt/stacks/gitea/compose.yaml</code>): <pre><code>GITEA__oauth2__ENABLED: \"true\"\nGITEA__openid__ENABLE_OPENID_SIGNIN: \"false\" # legacy OpenID 2.0 button off\nGITEA__openid__ENABLE_OPENID_SIGNUP: \"false\"\nGITEA__service__ENABLE_PASSWORD_SIGNIN_FORM: \"false\" # local username/password form off\nGITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: \"true\"\nGITEA__oauth2_client__ACCOUNT_LINKING: \"auto\"\nGITEA__oauth2_client__USERNAME: \"preferred_username\"\nGITEA__oauth2_client__UPDATE_AVATAR: \"true\"\n</code></pre> Note: <code>ENABLE_OPENID_SIGNIN</code> lives in <code>[openid]</code>, not <code>[service]</code>. Putting it under <code>GITEA__service__</code> is a no-op and leaves the legacy button visible.</p> <p>Gitea login source (DB row in <code>login_source</code>): - <code>name = \"pocket-id\"</code> (case-sensitive \u2014 becomes part of the callback URL) - <code>Scopes = [\"openid\",\"profile\",\"email\"]</code> - <code>two_factor_policy = \"skip\"</code> (Pocket-ID passkey already enforces 2FA)</p>"},{"location":"services/dev-environment/#pocket-id-client-secret-pitfall-important","title":"Pocket-ID client secret pitfall (important)","text":"<p>Pocket-ID 2.7.0 verifies client secrets with bcrypt (<code>bcrypt.CompareHashAndPassword</code>). The stored <code>oidc_clients.secret</code> column must be a 60-char bcrypt hash like <code>$2a$10$...</code> or <code>$2b$10$...</code>.</p> <p>A raw 64-char hex SHA-256 in that column silently fails every token exchange with <code>invalid client secret</code>. The Gitea and Coder clients on this host were initially provisioned that way and had to be regenerated.</p> <p>To programmatically add or rotate an OIDC client secret in Pocket-ID:</p> <pre><code># On Proxmox host (has python3-bcrypt installed)\npython3 <<'EOF'\nimport bcrypt, secrets, string\nalphabet = string.ascii_letters + string.digits\nplain = \"\".join(secrets.choice(alphabet) for _ in range(40))\nhashed = bcrypt.hashpw(plain.encode(), bcrypt.gensalt(rounds=10)).decode()\nprint(\"PLAINTEXT (give to client app):\", plain)\nprint(\"HASH (store in oidc_clients.secret):\", hashed)\nEOF\n</code></pre> <p>Always push the hash to Pocket-ID via a tmp file (<code>pct push 110 ...</code>) \u2014 never inline the bcrypt hash in a shell command, the <code>$</code> chars get expanded.</p> <p>To verify a client's secret is the right format: <pre><code>pct exec 110 -- sqlite3 /opt/stacks/pocketid/data/pocket-id.db \\\n \"SELECT name, length(secret), substr(secret,1,7) FROM oidc_clients;\"\n# Working clients: length=60, prefix \"$2a$10$\" or \"$2b$10$\"\n# Broken clients : length=64, prefix is hex (e.g. \"1180664\")\n</code></pre></p>"},{"location":"services/dev-environment/#break-glass-recovery-if-sso-is-broken","title":"Break-glass recovery (if SSO is broken)","text":"<p>You have host SSH access, so you're never locked out \u2014 but the UI will be unusable until you re-enable a local login path. From the Proxmox host:</p> <p>Gitea \u2014 re-enable local login + reset password: <pre><code># 1. Temporarily put the form back\nssh nuc\npython3 -c \"\np='/rpool/data/subvol-111-disk-0/opt/stacks/gitea/compose.yaml'\ns=open(p).read().replace('ENABLE_PASSWORD_SIGNIN_FORM: \\\"false\\\"','ENABLE_PASSWORD_SIGNIN_FORM: \\\"true\\\"')\nopen(p,'w').write(s)\"\npct exec 111 -- bash -c 'cd /opt/stacks/gitea && docker compose up -d --force-recreate gitea'\n\n# 2. Reset admin password\npct exec 111 -- docker exec -u git gitea gitea -c /data/gitea/conf/app.ini admin user change-password -u fkrebs -p 'temp-strong-pass'\n</code></pre></p> <p>Coder \u2014 flip user back to password login + re-enable: <pre><code>ssh nuc\n# 1. Flip login_type back\npct exec 111 -- docker exec coder-db psql -U coder -d coder -c \\\n \"UPDATE users SET login_type='password' WHERE username='fkrebs';\"\n\n# 2. Re-enable password auth in compose\npython3 -c \"\np='/rpool/data/subvol-111-disk-0/opt/stacks/coder/compose.yaml'\ns=open(p).read().replace('CODER_DISABLE_PASSWORD_AUTH: \\\"true\\\"','CODER_DISABLE_PASSWORD_AUTH: \\\"false\\\"')\nopen(p,'w').write(s)\"\npct exec 111 -- bash -c 'cd /opt/stacks/coder && docker compose up -d --force-recreate coder'\n\n# 3. Reset password (interactive)\npct exec 111 -- docker exec -it coder coder reset-password fkrebs\n</code></pre></p> <p>Pocket-ID \u2014 bootstrap a one-time access token if you're locked out of Pocket-ID itself: <pre><code>pct exec 110 -- docker exec pocket-id pocket-id-cli one-time-access-token \\\n --email fkrebs@nucli.de --duration 1h\n# Open the printed URL in a browser to log in once and register a new passkey.\n</code></pre></p> <p>After SSO is fixed: undo each of the above steps (re-disable password auth, flip <code>login_type</code> back to <code>oidc</code>).</p>"},{"location":"services/doc-ingestion/","title":"Document Ingestion Pipeline","text":"<p>Ingest PDFs, Word, PPTX, XLSX from Nextcloud/Paperless into Open WebUI's knowledge base and make them searchable via MCP.</p>"},{"location":"services/doc-ingestion/#architecture","title":"Architecture","text":"<pre><code>Nextcloud folder / Paperless webhook\n \u2192 n8n trigger (CT 104)\n \u2192 Docling MCP (port 18005) \u2014 PDF/DOCX/PPTX/XLSX \u2192 Markdown + structure\n \u2192 TEI /v1/embeddings \u2014 multilingual-e5-base (local, 768d)\n \u2192 Qdrant (shared vector store, http://qdrant:6333)\n \u2192 Open WebUI knowledge base API \u2190 searchable in chat\n</code></pre> <p>Paperless shortcut: Paperless-ngx already OCRs documents. Its full-text content is available at <code>/api/documents/?added__gt=<last_run></code>. An n8n workflow can re-embed directly from Paperless's REST API without re-running Docling for already-OCR'd PDFs.</p>"},{"location":"services/doc-ingestion/#deployed-components-as-of-2026-05-26","title":"Deployed components (as of 2026-05-26)","text":"Component Location Endpoint Qdrant CT 104, <code>ai-internal</code> net <code>http://qdrant:6333</code> (REST), <code>qdrant:6334</code> (gRPC) nomic CT 104, <code>ai-internal</code> net <code>http://nomic:80</code> \u2014 text+vision 768d TEI (Text Embeddings Inference) CT 104, <code>ai-internal</code> net <code>http://tei:80</code> \u2014 text-only 768d (standby) Open WebUI CT 104, port 14002 Uses Qdrant + nomic natively Bifrost CT 104, port 14003 Semantic cache \u2192 Qdrant gRPC, mistral-embed 1024d Docling MCP CT 104, port 18005 MCP server in gateway"},{"location":"services/doc-ingestion/#embedding-stack","title":"Embedding stack","text":"Service Model Dimensions Use nomic <code>nomic-ai/nomic-embed-text-v1.5</code> + <code>nomic-embed-vision-v1.5</code> 768d OWUI RAG, ingest pipeline TEI <code>intfloat/multilingual-e5-base</code> 768d Standby; same vector space as nomic text Bifrost cache <code>mistral/mistral-embed</code> via Bifrost 1024d Semantic cache only (separate Qdrant collection) <p>Both nomic models share a 768d embedding space \u2014 text and image queries work on the same <code>documents</code> Qdrant collection.</p>"},{"location":"services/doc-ingestion/#owui-rag-config-env-driven","title":"OWUI RAG config (env-driven)","text":"<pre><code>VECTOR_DB=qdrant\nQDRANT_URI=http://qdrant:6333\nRAG_EMBEDDING_ENGINE=openai\nRAG_OPENAI_API_BASE_URL=http://nomic:80\nRAG_OPENAI_API_KEY=none\nRAG_EMBEDDING_MODEL=nomic-ai/nomic-embed-text-v1.5\nCONTENT_EXTRACTION_ENGINE=docling\nCHUNK_SIZE=1200\nCHUNK_OVERLAP=150\nENABLE_RAG_HYBRID_SEARCH=true\n</code></pre> <p>To switch to Bifrost embeddings (1024d, better quality \u2014 requires re-indexing <code>documents</code> collection): <pre><code>RAG_OPENAI_API_BASE_URL=http://bifrost:8080/v1\nRAG_EMBEDDING_MODEL=mistral/mistral-embed\n</code></pre></p>"},{"location":"services/doc-ingestion/#bifrost-embedding-models-available-for-external-services-upgrade","title":"Bifrost embedding models (available for external services / upgrade)","text":"<p>Three embedding models tested and working via <code>http://bifrost:8080/v1/embeddings</code>: - <code>mistral/mistral-embed</code> (1024d) \u2014 \u2713 production-ready - <code>mistral/codestral-embed</code> (1024d) \u2014 \u2713 - <code>gemini/gemini-embedding-001</code> (768d/1536d) \u2014 \u2713</p>"},{"location":"services/doc-ingestion/#converter-comparison","title":"Converter comparison","text":"Tool Image Formats Notes Docling (deployed) MCP on 18005 PDF, DOCX, PPTX, XLSX, HTML Best for structured Office/PDF with tables MinerU <code>opendatalab/mineru</code> PDF (layout-aware, OCR) Better for academic papers / scanned PDFs Markitdown (in gateway) \u2014 Office, PDF Ad-hoc only; not suitable for batch <p>Start with Docling \u2014 already deployed. Add MinerU if academic paper OCR quality is needed.</p>"},{"location":"services/doc-ingestion/#sharing-qdrant-with-other-services","title":"Sharing Qdrant with other services","text":"<p>Qdrant is on <code>ai-internal</code> network \u2014 any service on that network can use it:</p> <pre><code>from qdrant_client import QdrantClient\nclient = QdrantClient(url=\"http://qdrant:6333\")\n</code></pre> <p>n8n, MCP tools, and custom pipelines should use <code>http://nomic:80/v1/embeddings</code> for consistent 768d vectors. Mixing models/dimensions in the same collection will fail.</p>"},{"location":"services/doc-ingestion/#bifrost-semantic-caching","title":"Bifrost semantic caching","text":"<p>Bifrost uses Qdrant (gRPC port 6334) as a semantic cache backend. Config lives in <code>/opt/stacks/ai/bifrost/data/config.json</code>:</p> <pre><code>{\n \"$schema\": \"https://www.getbifrost.ai/schema\",\n \"vector_store\": {\"enabled\": true, \"type\": \"qdrant\", \"config\": {\"host\": \"qdrant\", \"port\": 6334}},\n \"plugins\": [{\n \"enabled\": true, \"name\": \"semantic_cache\",\n \"config\": {\n \"provider\": \"mistral\", \"embedding_model\": \"mistral-embed\", \"dimension\": 1024,\n \"ttl\": \"10m\", \"threshold\": 0.85, \"conversation_history_threshold\": 3, \"exclude_system_prompt\": true\n }\n }]\n}\n</code></pre> <p>The semantic cache uses a separate Qdrant collection (auto-created) at 1024d \u2014 no collision with the <code>documents</code> collection at 768d. TTL: 10 min, similarity threshold: 0.85.</p>"},{"location":"services/doc-ingestion/#intel-arc-gpu-passthrough-enabled","title":"Intel Arc GPU passthrough (enabled)","text":"<p>CT 104 Intel Core Ultra 7 155H iGPU is passed through via PVE <code>dev3</code>/<code>dev4</code> entries (<code>/dev/dri/renderD128</code> and <code>/dev/dri/card1</code>). No TEI Intel image exists currently \u2014 passthrough is available for future inference acceleration.</p>"},{"location":"services/doc-ingestion/#pending","title":"Pending","text":"<ul> <li>n8n workflow: Nextcloud/Paperless \u2192 Docling \u2192 nomic \u2192 Qdrant</li> <li>MCP knowledge-search tool pointing at Qdrant (replaces LobeChat search intent)</li> <li><code>ingest.py</code> filesystem scan script (see ingest-pipeline.md)</li> </ul>"},{"location":"services/homelab-architecture/","title":"Nuclide Ecosystem \u2014 Homelab Architecture","text":"<p>Living architecture reference for the <code>/opt/stacks</code> homelab. ~66 containers across ~23 compose stacks. Principle: self-host everything, OIDC SSO, <code>*.nuclide.systems</code> via one reverse proxy.</p>"},{"location":"services/homelab-architecture/#topology","title":"Topology","text":"<ul> <li>NUC 14 Pro (<code>192.168.1.40</code>) \u2014 primary Docker host, all <code>/opt/stacks/*</code>. Itself a Proxmox LXC (CTID 104) on node <code>nuc</code>; local data on ZFS <code>rpool/data/subvol-104-disk-0</code>. Mounts the UNAS NFS at <code>/mnt/pve/unas</code>.</li> <li>Proxmox VE \u2014 host <code>192.168.1.20:8006</code>, single node <code>nuc</code> (Intel Core Ultra 7 155H, 22 threads, 64G RAM, PVE 9.1.11). API access via <code>root@pam!mcp</code> token (now <code>PVEAuditor</code>, read-only). 9 guests (+ 2 planned) \u2014 canonical roster in <code>ct-inventory.md</code>:</li> <li><code>100</code> qemu haos \u2014 Home Assistant OS VM (4c/16G) \u2192 <code>.60</code></li> <li><code>101</code> lxc shepard \u2014 secondary Docker host (12c/16G/107G) \u2192 <code>.49</code></li> <li><code>102</code> lxc dns \u2014 AdGuard Home (network DNS + ad/tracker blocking, 2c/0.5G). UniFi DHCP has <code>dhcpd_dns_enabled: false</code>; clients resolve via the gateway <code>.1</code>, and the UDM forwards DNS upstream to AdGuard (confirmed) \u2014 so ad/tracker blocking is network-wide despite DHCP not handing out AdGuard's IP directly.</li> <li><code>103</code> lxc backrest \u2014 Restic/Backrest backups (1c/0.5G)</li> <li><code>104</code> lxc docker \u2014 the primary <code>/opt/stacks</code> host (16c/48G/200G) \u2192 <code>.40</code></li> <li><code>105</code> lxc nextcloud \u2014 Nextcloud (4c/8G/107G) \u2192 <code>.41</code></li> <li><code>108</code> lxc zoraxy \u2014 reverse proxy (2c/2G) \u2192 <code>.4</code></li> <li><code>110</code> lxc id \u2014 Pocket-ID OIDC IdP (1c/1G/4G) \u2192 <code>.5</code>; migrated off 104 on 2026-05-20</li> <li><code>111</code> lxc dev \u2014 Coder + Gitea (12c/32G/60G) \u2192 <code>.42</code>; new 2026-05-20</li> <li><code>109</code> lxc ops <code>192.168.1.8</code> \u2014 Prometheus + Grafana + Loki + Alloy + pve-exporter (2026-05-23); future: Arcane, Dozzle, Homarr, Tinyauth</li> <li><code>113</code> lxc db \u2014 postgres 17 + pgAdmin + WAL-G \u2192 Garage S3 + Arcane edge agent \u2192 <code>.6</code>; provisioned 2026-05-21</li> <li>Storages: <code>local</code> (dir), <code>local-zfs</code> (zfspool, ~1.9T), <code>unas</code> (nfs, ~20T).</li> <li>Ubiquiti UniFi \u2014 UDM \"Home\" (UDMA6A8, UCG Fiber) gateway + 4\u00d7 U7-series APs. Controller at <code>https://192.168.1.1</code> (UniFi OS 5.0.16, SSO + MFA enforced). Single site \"Default\". MCP access: <code>ghcr.io/enuno/unifi-mcp-server</code> (197 tools, Network App API key); Site Manager/cloud tools need <code>UNIFI_SITE_MANAGER_ENABLED</code>, all local API tools work.</li> <li>LAN \"Default\" \u2014 <code>192.168.1.1/24</code>, corporate, no VLAN. DHCP pool <code>.100\u2013.250</code> (24 h lease), domain <code>localdomain</code>. Static infra lives <code>.1\u2013.99</code> (outside the pool). WANs: \"Internet 1\" (primary), \"Secondary\" (WAN2, disabled). 47 clients (17 wired, 30 wireless).</li> <li>WiFi \u2014 single SSID \"nuclide\", WPA2 with WPA3-transition, bands 2.4/5/6 GHz, not guest, no per-SSID VLAN, no L2 isolation. Flat network: servers, IoT, consoles, phones all share one L2 segment.</li> <li>Firewall \u2014 zone-based (Internal/External/DMZ/Gateway/VPN/Hotspot), almost entirely system-predefined rules. Port forwards: 80/443 \u2192 192.168.1.4 (Zoraxy); 15001 TCP/UDP \u2192 192.168.1.40 (Traccar watch protocol \u2014 already at UDM level, no Zoraxy stream proxy needed).</li> <li>\u26a0\ufe0f Non-UniFi gear = topology blind spots (UniFi can't see/manage):<ul> <li>D-Link DGS-1210-28P 28-port PoE switch at <code>.10</code> (HW F5, fw 6.32.008, S/N TM0I533010201) \u2014 unmanaged by UniFi; its ports/ port-clients don't appear in UniFi topology. HTTP-only mgmt (<code>admin</code>/shared <code>tapirnase</code> pw, RSA-login). Audit 2026-05-19 (read-only):</li> <li>SNTP was broken \u2192 FIXED 2026-05-19. Real root cause was DNS: servers were <code>0/1/2.de.pool.ntp.org</code> but the switch has no DNS resolver (<code>DNS_STATE=2</code>) so it couldn't resolve them. (The default gateway was fine \u2014 an active static default route <code>0.0.0.0/0 \u2192 .1</code> exists; the empty <code>Default_Route_Gateway=[]</code> is just the unused interface-level gateway field \u2014 misleading.) Clock had been stuck at 31/08/2025 (~8.5 mo). Fix: repointed to the pinned NTP source standard (see below), all by IP, DNS-free. Verified synced 2026-05-19 \u2014 clock corrected to live time (CEST/UTC+2).</li> <li>HTTPS/SSL disabled \u2014 admin creds cross the flat LAN in clear. Pending (see roadmap: mgmt-TLS certs for Proxmox + D-Link).</li> <li>SNMP agent enabled on the flat LAN (community strings unverified). Pending.</li> <li>Trusted-Host mgmt allowlist disabled \u2014 admin UI reachable from any LAN device incl. IoT. Pending: enable allowing <code>192.168.1.0/24</code>.</li> <li>\u26a0\ufe0f The switch is a transparent L2 switch \u2014 every device on it is fully visible to UniFi as a client (MAC/IP via UDM DHCP/ARP); only the wired port-topology through it is invisible (needs UniFi adoption, impossible here). This is unlike the RE700X repeater.</li> <li>\ud83d\udd34 The D-Link is the network CORE / single point of failure. FDB-table analysis (2026-05-19): the UDM uplinks on port 26, the entire Proxmox host (NUC .40 + Nextcloud .41 + HA .60 + all VMs/LXCs) is on port 10, and the U7 APs uplink on ports 3 & 16 (their wireless clients appear via those ports). Nearly all wired and wireless traffic transits this unmanaged switch \u2192 a D-Link failure = total LAN+WAN outage. Amplifies the mgmt-hardening, firmware-currency, monitoring and segmentation items below.</li> <li>UniFi\u2194D-Link DHCP/LLDP interop = fully compatible (audited 2026-05-19, read-only):<ul> <li>DHCP: <code>BOOTP_Relay_State=0</code> and <code>DHCP_BOOTP_Local_Relay_ Status=0</code> \u2192 switch inserts no Option 82; UDM DHCP server gets clean untagged requests. (<code>Option82_State=1</code> is moot \u2014 only applies when relay/local-relay is active; both are off.) DHCP Server Screening has no entries \u2192 doesn't block the UDM (also = no rogue-DHCP protection: security note, not interop).</li> <li>LLDP: D-Link LLDP enabled, all 28 ports TX+RX (<code>Port_Basic_ Setting</code> mode 3); stats show 25 inserts / 16 ageouts \u2192 LLDP frames are exchanged & parsed with UniFi gear (no TLV incompat). \"0 neighbors\" earlier was just a point-in-time aged-out snapshot. UniFi still won't draw topology for an unadopted switch \u2014 cosmetic only.</li> <li>RSTP: both run 802.1w (standard, interoperable). Tuning opportunity (not a fault): set a deliberate root-bridge priority \u2014 currently all default 32768 so root = lowest MAC (arbitrary). Make the D-Link (physical core) or UDM root.</li> <li>Flow control: D-Link global FC <code>=2</code> (off/auto) \u2014 matches UniFi's default (802.3x disabled). Compatible.</li> </ul> </li> <li>TP-Link RE700X WiFi extender at <code>.187</code> (MACs <code>\u20261a:8a:0f/10/11</code>) \u2014 bridges/NATs devices behind it. The Klipper 3D printer at <code>.189</code> sits behind the RE700X, so UniFi only sees the repeater, never <code>.189</code>.</li> <li>DNS: UniFi DHCP has <code>dhcpd_dns_enabled: false</code> (clients get the gateway <code>.1</code> as resolver, not AdGuard directly) \u2014 see DNS note below.</li> </ul> </li> </ul> <p>Infrastructure IPs known from UniFi: | IP | Name / Hostname | Notes | |---|---|---| | .1 | UDM Home (UDMA6A8) | Gateway, UCG Fiber, fw 5.0.16 | | .4 | Zoraxy LXC 108 | Reverse proxy | | .20 | Proxmox node <code>nuc</code> | PVE 9.1.11 | | .40 | NUC 14 Pro / Docker LXC 104 | Main host, Proxmox OUI | | .41 | Nextcloud LXC 105 | Named \"NextCloud\", Proxmox OUI | | .49 | Shepard Docker host | Secondary Docker host | | .50 | U7-Pro-Wall AP | Hallway/entry | | .51 | U7 In-Wall AP | In-wall, UAPA6A5 | | .52 | U7 Mesh (Schlafzimmer) | Bedroom | | .53 | U7 Mesh (Esszimmer) | Dining room | | .60 | Home Assistant OS VM | haos CTID 100 | | .62 | Siemens oven | BSH Hausger\u00e4te, WiFi | | .66 | Tibber Pulse | Energy monitor (Espressif) | | .10 | DGS-1210-28P | D-Link 28-port PoE switch (NOT UniFi-managed) | | .124 | L0018 | Wired, unknown device | | .143 | \u2014 | Sony Interactive Ent. (PlayStation) | | .164 | C100_7614A4 | Tapo C100 camera (TP-Link) | | .169 | awtrix_fcf0bc | AWTRIX LED clock (Espressif) | | .187 | RE700X | TP-Link WiFi extender \u2014 NATs devices behind it | | .189 | \u2014 | Klipper 3D printer (behind RE700X, invisible to UniFi) | | .192 | REDMI-Note-15-Pro-5G | Xiaomi phone | | .241 | VS9-EU-MNA3478A | Dyson purifier/fan | - Zoraxy reverse proxy at <code>192.168.1.4:8000</code> \u2014 wildcard <code>*.nuclide.systems</code> cert + rules. Source of truth: <code>proxy/zoraxy/routes.json</code> (+ idempotent <code>scripts/zoraxy_sync.py --apply</code>). Cross-host, so no Docker labels. - Ubiquiti UNAS \u2014 NFS server <code>192.168.1.31:/var/nfs/shared/storage</code> (mounted <code>/mnt/pve/unas</code>, ~19T). Bulk/storage (arr media, qdrant, configs); several stacks bind data dirs here. \u26a0\ufe0f SQLite-on-NFS is fragile here (see n8n note under Operational rules). - Auth \u2014 PocketID (<code>id.nuclide.systems</code>, OIDC, SQLite) is the universal SSO/IdP for the entire ecosystem \u2014 effectively every service authenticates via PocketID OIDC (LiteLLM, Nextcloud, Coder, Gitea, Vaultwarden, etc.). Single sign-on everywhere; one identity source to secure/audit. Vaultwarden remains the lone gap as of 2026-05-20. Open WebUI OIDC not yet wired (as of 2026-05-26).</p>"},{"location":"services/homelab-architecture/#docker-networks","title":"Docker networks","text":"<p><code>ai-internal</code> (AI/MCP plane) \u00b7 <code>shared_backend</code> (cross-stack DB/S3) \u00b7 plus per-stack: <code>arr-stack_default</code>, <code>immich_default</code>, <code>karakeep_default</code>, <code>homepage_default</code>, <code>vpn_default</code>. The MCP gateway bridges <code>ai-internal</code> + <code>shared_backend</code>.</p>"},{"location":"services/homelab-architecture/#ai-agent-platform-the-core-optstacksai","title":"AI / Agent platform (the core, <code>/opt/stacks/ai</code>)","text":"<ul> <li>Bifrost (<code>ai.nuclide.systems</code>, :14003) \u2014 unified LLM gateway + MCP host. Stack <code>ai/bifrost/</code>. Auth via <code>sk-bf-</code> VKs. LLM inference at <code>/v1</code>; MCP at <code>/mcp</code> (29 clients, ~760 tools as of 2026-05-26). Proxies LLM requests to LiteLLM internally.</li> <li>LiteLLM (:14000, internal only \u2014 no Zoraxy route) \u2014 model gateway, backend for Bifrost. ~28 curated models (provider allowlist), SAIA-terminal failovers + <code>num_retries</code>, Redis completion cache, Gemini \u20ac10/30d <code>provider_budget_config</code>. Config: <code>ai/litellm-config/config.yaml</code> (+ DB overlay <code>LiteLLM_Config</code>). Reachable as <code>http://litellm:4000</code> inside Docker.</li> <li>Open WebUI (<code>chat.nuclide.systems</code>, :14002) \u2014 chat UI. Stack <code>ai/open-webui.yml</code>. Uses Qdrant + TEI for RAG. OIDC via Pocket-ID (not yet wired as of 2026-05-26).</li> <li>syncstack (<code>ai/syncstack.py</code>, cron <code>/etc/cron.d/syncstack</code> 15 min) \u2014 model syncer/optimizer ONLY (curation/allowlist \u2192 DB, model-health \u2192 ntfy). One-shot container.</li> <li>~~mcp-gateway~~ \u2014 DECOMMISSIONED 2026-05-26. Was FastAPI/DinD on <code>:8080</code> at <code>mcp.nuclide.systems</code>. Compose renamed <code>.DECOMMISSIONED-2026-05-26</code>. MCP is now served by Bifrost at <code>https://ai.nuclide.systems/mcp</code>.</li> <li>MCP servers (Bifrost-managed, formerly gateway-managed): time, home-assistant, kroki, coder, fetch, sequential-thinking, gotify, memos, docling, markitdown, crawl4ai, context7, wikipedia, papersearch, comfyui, bluesky, obsidian, immich, n8n, nextcloud, paperless, unifi, upload-artifact, gitea, proxmox, shepard, git, gitlab, youtube-transcript, searxng (30 servers). 29 clients, ~760 tools. See <code>services/mcp-servers.md</code>.</li> <li>Coder (<code>dev.nuclide.systems</code>, CT 111) \u2014 replaced Daytona as the sandbox / dev-environment runtime. Templates: <code>python-uv</code> (persistent, GPU passthrough, baked LiteLLM env + Claude Code), <code>mcp-sandbox</code> (ephemeral, sci stack pre-baked). See <code>services/dev-environment.md</code>.</li> <li>~~LobeChat~~ \u2014 DECOMMISSIONED 2026-05-26. Was at <code>chat.nuclide.systems</code> (:14001). Compose renamed <code>.DECOMMISSIONED-lobehub-2026-05-26.yml</code>. Replaced by Open WebUI.</li> <li>ComfyUI (img gen, Intel Arc iGPU), Kroki, searxng (self-hosted meta-search).</li> </ul>"},{"location":"services/homelab-architecture/#data-layer","title":"Data layer","text":"<ul> <li>shared-postgres (pg16, <code>shared_backend</code>, data on local ZFS) \u2014 the primary/standard DB for all deployments. Tenants: LiteLLM, paperless, memos (migrated 2026-05-19), n8n (added 2026-05-19). Superuser <code>postgres</code> via unix socket; per-app dedicated role+db (role owns its db, password in the app stack's <code>.env</code>). Tuned 2026-05-19 (3G limit, shared_buffers 768M, max_conn 200). \u26a0\ufe0f n8n was NOT successfully migrated \u2014 n8n's <code>export/import</code> CLI only covers workflows+credentials, not users/settings/SSO, so the PG cutover lost the owner account + OIDC. Reverted to local-disk SQLite (tier-1 still satisfied, like PocketID). Compounding: unpinned <code>n8nio/n8n:latest</code> had drifted 2.7\u21922.20.11, breaking the custom OIDC <code>hooks.js</code> (hardcoded old <code>/usr/local/lib</code> module paths) \u2192 n8n crash-looped. Fixed: image pinned to <code>2.20.11</code>, <code>hooks.js</code> paths patched for 2.20.11 pnpm layout (<code>/usr/lib/node_modules/n8n/...</code>), OIDC hook re-enabled 2026-05-19 \u2014 verified: <code>/auth/oidc/login</code> \u2192 302 to PocketID (client <code>33135ad4</code>, correct redirect/scope/state/nonce). Lessons: (1) only a FULL pgloader migration of all tables is complete \u2014 partial <code>export/import</code> is not; (2) pin critical images \u2014 <code>latest</code> drift is a real outage cause. Migration note: services without a native full SQLite\u2192PG export use pgloader data-only of ALL tables into the app-built schema (exclude only the app's own migration-tracking table), app role temp-SUPERUSER for the load (FK/trigger disable) then reverted.</li> <li>lobe-postgres (pg17/ParadeDB) \u2014 LobeChat (decommissioned 2026-05-26; DB retained pending cleanup). lobe-redis \u2014 LiteLLM cache (LobeChat decommissioned). Garage S3 (<code>garage:3900</code>, ext <code>s3.nuclide.systems</code> / <code>:10004</code>) \u2014 lobe-files, memos, WAL-G PG backups. Qdrant (vector, on UNAS) \u2014 unused yet (future RAG/mem0). pgAdmin (internal).</li> <li>Backups: WAL-G v3 \u2192 Garage for all 3 PG instances (shared-postgres, lobe-postgres, immich_postgres); daily 3am cron (<code>/etc/cron.d/pg-backup</code>). Incident + fix (2026-05-19): Garage stored <code>meta</code>+<code>data</code> on the UNAS NFS \u2192 NFS stalls hung <code>wal-g wal-push</code> \u2192 archivers hung since 2026-05-18 14:51 (<code>failed_count=0</code> = hung not failing), zero base backups, stalled WAL recycled (that window unrecoverable). Resolved: Garage moved to local disk (<code>/opt/stacks/shared-db/garage/{meta,data}</code>); all 3 archivers drained; <code>archive_command</code> hardened to <code>timeout 60 wal-g wal-push %p</code> (fail-fast vs infinite hang); fresh base backups taken for all 3. Extra safety net: local logical dumps in <code>/opt/stacks/backups/shared-pg/</code>. Lesson: Garage metadata is fsync/lock-heavy \u2014 never on NFS, same rule as SQLite.</li> </ul>"},{"location":"services/homelab-architecture/#database-design-standards","title":"Database design / standards","text":"<ul> <li>Default = Postgres on <code>shared-postgres</code>. Any app needing persistence that supports Postgres gets a dedicated <code>role</code>+<code>database</code> there. SQLite is allowed only when the app has no Postgres support, and then only on local disk \u2014 never the UNAS NFS share (broken POSIX/SMB file locking; caused the n8n outage + PocketID latent risk).</li> <li>Provisioning pattern: <code>CREATE ROLE <app> LOGIN PASSWORD \u2026; CREATE DATABASE <app> OWNER <app>;</code> \u2192 set the app's <code>DB_*</code> env, password lives in that stack's <code>.env</code> (vault migration is the long-term plan).</li> <li>Why standardize: unified, durable WAL-G PITR backup \u2014 the single biggest reason. One backup story instead of N un-backed-up SQLite files.</li> <li>Migration reality: few apps have a native SQLite\u2192PG data port. n8n does (<code>export/import</code> CLI). Most (PocketID, Vaultwarden, traccar, \u2026) do not \u2014 switching <code>DB_PROVIDER</code> starts a fresh DB; data port needs <code>pgloader</code>/app-specific tooling. Memos done via pgloader (data-only, app-built schema, role temp-SUPERUSER for the load). When no safe port exists, fall back to local-disk SQLite (PocketID) until a port is built.</li> </ul>"},{"location":"services/homelab-architecture/#nfs-storage-strategy-corrected-2026-05-19","title":"NFS / storage strategy (corrected 2026-05-19)","text":"<ul> <li>The Docker host is an unprivileged Proxmox LXC (104) \u2192 it cannot do in-container NFS mounts (kernel denies; tested \u2014 <code>operation not permitted</code>). That's why Proxmox NFS-mounts at the host and bind-mounts <code>/mnt/pve/unas</code> in. Consequence: per-stack \"Docker NFS volumes at v4.1\" (the old tier-3 idea) is infeasible here without <code>pct set 104 --features mount=nfs</code> + a full LXC reboot (all stacks down).</li> <li>Tier 1 is the rule and does the real work: DBs/metadata/lock- or fsync-heavy stores \u2192 local ZFS or shared-postgres, never NFS (any version \u2014 SQLite/LMDB/sled are unsafe on NFS regardless of tuning).</li> <li>NFSv4 is NOT available \u2014 the Ubiquiti UNAS Pro is NFSv3-only (current firmware; verified 2026-05-19 on node <code>nuc</code>: <code>showmount -e</code> works but <code>mount :/ -o vers=4.1</code> fails server-side <code>No such file or directory</code>, i.e. no v4 pseudo-root). So tier-2 (host v3\u2192v4.1) and tier-3 (Docker NFS v4.1 volumes) are both dead ends \u2014 no v4 to upgrade to, no reboot worth doing for it. Real export path is <code>/volume/<uuid>/.srv/.unifi-drive/ storage/.data</code> (Proxmox <code>unas</code> storage uses the <code>/var/nfs/shared/storage</code> alias, works on v3 \u2014 leave it).</li> <li>Therefore tier-1 is the whole strategy. NFSv3 stays for bulk/ sequential data (fine for that). Optional marginal v3 tuning: <code>nconnect=4</code> for throughput \u2014 not required.</li> <li>Tier-1 complete (2026-05-19): all SQLite-on-NFS backlog resolved \u2014 ntfy, Karakeep, arcane \u2192 local disk; Vaultwarden \u2192 shared-postgres (pgloader, 2054 rows, 0 errors, NFS kept for attachments/rsa_key.pem); traccar data \u2192 local disk. PocketID stays on local SQLite (no native PG migration path, deferred).</li> </ul>"},{"location":"services/homelab-architecture/#services","title":"Services","text":"<p>Nextcloud (LXC .41), Paperless-ngx (+paperless-ai, tika/gotenberg), Immich (server/ML/redis/postgres/power-tools), Memos, Karakeep (+chrome), Vaultwarden, n8n, Home Assistant (.60, ~2492 entities), ntfy (<code>homelab-ai</code> topic \u2014 model-health + agent alerts), traccar (GPS, HTTP :15000 / watch :15001), arr-stack (prowlarr/shelfarr/flaresolverr behind gluetun VPN), streamio, Arcane (Docker mgmt), Dozzle (logs), Homepage (dashboard, 6 groups), Daytona OIDC adapter (Keycloak\u2192PocketID PKCE proxy for the VS Code ext).</p>"},{"location":"services/homelab-architecture/#operational-rules-conventions","title":"Operational rules / conventions","text":"<ul> <li>Separation: <code>syncstack</code> = model curation; <code>bifrost</code> = MCP + LLM gateway (mcp-gateway decommissioned 2026-05-26).</li> <li>Self-host first: external/cloud only when no self-hosted form exists.</li> <li>Pinned NTP source standard \u2014 all network gear/infra should point at the same fixed-IP, DNS-free NTP sources (devices like the D-Link can't resolve hostnames; pinned IPs avoid silent DNS-based SNTP failure):</li> <li><code>192.53.103.108</code> \u2014 PTB ptbtime1 (DE national time)</li> <li><code>192.53.103.104</code> \u2014 PTB ptbtime2 (DE national time)</li> <li><code>162.159.200.123</code> \u2014 Cloudflare NTP anycast (fallback) Applied to the D-Link switch 2026-05-19. UDM + other infra should converge on the same set (a self-hosted LAN NTP server is a roadmap option, but the standard stays \"same pinned sources\" everywhere).</li> <li>Run Python with <code>uv run</code> (not python3) in <code>/opt/stacks</code>.</li> <li>Zoraxy self-service: edit <code>routes.json</code> \u2192 <code>scripts/zoraxy_sync.py --apply</code>.</li> <li>Secrets in <code>ai/.env</code> (plaintext \u2014 env-audit/secret-vault is an open hardening task; Vaultwarden available). Note: a UniFi MFA JWT has leaked into <code>homepage/config/logs/homepage.log</code> \u2014 rotate + scrub when hardening. \u26a0\ufe0f Shared-password reuse: <code>tapirnase</code> is reused as the WiFi PSK, the LiteLLM master key root (<code>sk-tapirnase</code>), and the D-Link switch admin pw \u2014 single sniff/leak has broad blast radius. Rotate per-service when hardening (noted 2026-05-19, rotation deferred per user).</li> <li>Proxmox/UniFi safety: any write/change to the Proxmox host or Ubiquiti network gear requires explicit confirmation + a stated risk assessment first; read-only queries are fine. High blast radius (foundation layer).</li> <li>SQLite-on-NFS is unsafe here: the UNAS NFS share has unreliable file locking (true for SMB/CIFS too). Any SQLite app must use <code>shared-postgres</code> or local disk, never the NFS share. n8n hit this (<code>Database connection timed out</code>) and was migrated to <code>shared-postgres</code> (DB on local ZFS; data dir moved to local <code>/opt/stacks/n8n/data</code>; old NFS <code>database.sqlite</code> kept as rollback). Audit other SQLite stacks for NFS-backed data dirs.</li> <li>Naming debt: \"syncstack\" has outgrown its name \u2014 rename once the agent-platform identity firms up. mcp-gateway renamed/decommissioned 2026-05-26.</li> </ul>"},{"location":"services/homelab-architecture/#open-roadmap","title":"Open / roadmap","text":"<p>mem0 vs Qdrant for agent memory (deferred); Agent teams/orchestration + expose Agent Operator as MCP; S3/Immich/n8n/Paperless/Proxmox MCP servers (in progress); UniFi MCP \u2014 COMPLETE 2026-05-19 (<code>ghcr.io/enuno/unifi-mcp-server</code>, 197 tools, Network App API key, <code>http</code> transport; local API fully working). Tier-1 SQLite-off-NFS: COMPLETE \u2014 all services off NFS for DB/metadata. Note: Traccar watch protocol port 15001 is already forwarded at the UDM level \u2014 no Zoraxy stream proxy needed. Plus: env \u2192 secret vault. Wire Open WebUI OIDC via Pocket-ID once Bifrost auth is settled.</p> <p>Management-plane TLS (planned): issue/trust proper certs for admin-UI auth on the Proxmox host and the D-Link DGS-1210 (currently HTTP-only on the D-Link \u2192 admin creds in clear on the flat LAN; Proxmox self-signed). Brings switch/hypervisor mgmt onto the <code>*.nuclide.systems</code> PKI like the rest.</p> <p>Network segmentation (planned): the LAN is flat \u2014 servers, IoT (Siemens oven, Dyson, Tapo cam, AWTRIX), consoles and phones all on one L2 (<code>192.168.1.0/24</code>, no VLAN, no L2 isolation). Plan an IoT VLAN (+ matching firewall zone) so untrusted appliances can't reach the server/Proxmox subnet. Complications to design around: the non-UniFi D-Link DGS-1210-28P switch (<code>.10</code>) and TP-Link RE700X extender (<code>.187</code>, NATs the Klipper printer <code>.189</code>) won't honour UniFi VLAN tags natively \u2014 segmentation needs a plan for the wired trunk through the D-Link and the repeater's bridge mode.</p>"},{"location":"services/homelab-architecture/#observability-lxc-planned-scoped-from-this-sessions-incidents","title":"Observability LXC (planned \u2014 scoped from this session's incidents)","text":"<p>Why it's now a priority: the WAL-G archiver was hung silently for ~13 h (<code>failed_count=0</code>, zero base backups) and would never have been noticed; NFS stalls and SQLite-on-NFS damage are likewise silent. Monitoring must target exactly these silent-failure classes. - Placement: its own LXC on node <code>nuc</code>, NOT inside LXC 104 \u2014 104 hosts everything, so the monitor must survive/alert when 104 is down. - Stack: VictoriaMetrics (or Prometheus) + Grafana + Loki + Alertmanager \u2192 ntfy <code>homelab-ai</code> (already the alert channel). - Exporters/probes: node_exporter (per host + key LXCs), postgres_exporter \u00d73 (shared/lobe/immich), cAdvisor/docker, blackbox (HTTP + TLS-expiry for Zoraxy wildcard), pve-exporter (use the <code>root@pam!mcp</code> PVEAuditor token), and a custom WAL-G/archiver textfile collector: <code>pg_stat_archiver</code> (last_archived age, <code>failed_count</code>), <code>.ready</code> backlog, and <code>wal-g backup-list</code> newest-base age \u2014 per PG instance. - Alerts (priority order, derived from real incidents): 1. WAL archiver stalled (last_archived age > 15 m) or newest base backup > 26 h, any PG instance. 2. NFS mount on <code>/mnt/pve/unas</code> unresponsive / high op latency. 3. Config-drift guard: any <code>*.db</code>/<code>*.sqlite*</code> appears under <code>/mnt/pve/unas</code> (catches a regression of the tier-1 rule). 4. Container unhealthy/restart-looping > 5 m (the gluetun pattern). 5. local-zfs <code>rpool</code> or NFS pool > 85 %; 6. TLS cert < 14 d.</p>"},{"location":"services/homelab-architecture/#topology-at-a-glance","title":"Topology \u2014 at a glance","text":""},{"location":"services/homelab-architecture/#physical-ct-layout","title":"Physical / CT layout","text":"<pre><code>flowchart TB\n subgraph Host[\"Proxmox host \u00b7 192.168.1.20 \u00b7 Intel Core Ultra 7 155H \u00b7 64 GiB\"]\n direction TB\n haos[\"VM 100 \u00b7 haos<br/>(.60)<br/>Home Assistant\"]\n shepard[\"CT 101 \u00b7 shepard<br/>(.49)<br/>Shepard product stack\"]\n dns[\"CT 102 \u00b7 dns<br/>(.2)<br/>AdGuard Home\"]\n backrest[\"CT 103 \u00b7 backrest<br/>(.3)<br/>Backrest / restic\"]\n docker104[\"<b>CT 104 \u00b7 docker</b><br/>(.40) \u00b7 16c/48G/200G<br/>~65 containers \u00b7 Intel Arc passthrough\"]\n nc[\"CT 105 \u00b7 nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)\"]\n zoraxy[\"CT 108 \u00b7 zoraxy<br/>(.4)<br/>reverse proxy + ACME\"]\n obs[\"CT 109 \u00b7 ops<br/>(.8)<br/>Prometheus \u00b7 Grafana \u00b7 Loki \u00b7 Alloy \u00b7 pve-exporter\"]\n id[\"CT 110 \u00b7 id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)\"]\n dev[\"CT 111 \u00b7 dev<br/>(.42) \u00b7 12c/32G/60G<br/>Coder + Gitea + workspaces \u00b7 Intel Arc\"]\n db[\"CT 113 \u00b7 db<br/>(.6) \u00b7 provisioned 2026-05-21<br/>shared Postgres + pgAdmin\"]\n end\n UNAS[(\"UNAS<br/>192.168.1.31<br/>NFSv3\")]\n UDM[[\"UDM-SE \u00b7 192.168.1.1<br/>UniFi gateway \u00b7 DNS \u2192 AdGuard\"]]\n Inet([Internet \u00b7 ACME challenges \u00b7 jottacloud \u00b7 LiteLLM upstreams])\n\n UDM <--> Host\n UDM <--> Inet\n UNAS <--> shepard\n UNAS <--> backrest\n UNAS <--> docker104\n UNAS <--> nc\n UNAS <--> dev\n\n classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa\n class db planned</code></pre>"},{"location":"services/homelab-architecture/#auth-plane-pocket-id-is-the-universal-idp","title":"Auth plane \u2014 Pocket-ID is the universal IdP","text":"<p>Every web service that supports OIDC federates against Pocket-ID. Coder/Gitea/Vaultwarden access through Zoraxy; Zoraxy + Tinyauth fronts the non-OIDC-native ones.</p> <pre><code>flowchart LR\n user([\"fkrebs \u00b7 browser / VS Code / Claude\"]) --> zx[Zoraxy<br/>CT 108]\n zx --> coder[Coder \u00b7 CT 111]\n zx --> gitea[Gitea \u00b7 CT 111]\n zx --> nc2[Nextcloud \u00b7 CT 105]\n zx --> immich[Immich \u00b7 CT 104]\n zx --> owui[Open WebUI \u00b7 CT 104]\n zx --> n8n[n8n \u00b7 CT 104]\n zx --> bifrost[Bifrost \u00b7 CT 104]\n zx --> vw[Vaultwarden \u00b7 CT 104]\n coder --> pid[(Pocket-ID<br/>CT 110)]\n gitea --> pid\n nc2 --> pid\n immich --> pid\n owui -.->|OIDC not yet wired| pid\n n8n --> pid\n bifrost -.->|VK auth, not OIDC| pid\n vw -.->|via Tinyauth<br/>when CT 109 lands| pid\n classDef pending stroke-dasharray:4 4,color:#888\n class vw pending</code></pre>"},{"location":"services/homelab-architecture/#mcp-plane-bifrost-host-child-workers","title":"MCP plane \u2014 Bifrost host, child workers","text":"<p>Bifrost on CT 104 is the MCP host; ~30 child MCP servers run on the same docker network (<code>ai-internal</code>). The old mcp-gateway FastAPI/DinD container was decommissioned 2026-05-26.</p> <pre><code>flowchart LR\n client[[\"Claude Code / Cursor / Open WebUI\"]]\n gw[\"Bifrost<br/>(CT 104)<br/>ai.nuclide.systems/mcp\"]\n client -- Bearer VK --> gw\n subgraph \"CT 104 \u00b7 ai-internal docker net\"\n direction TB\n coderm[coder-mcp]\n immm[mcp-immich]\n n8nm[mcp-n8n]\n fetch[mcp-fetch]\n time[mcp-time]\n cw[mcp-crawl4ai]\n seq[mcp-sequential-thinking]\n ham[home-assistant-mcp]\n kr[kroki-mcp]\n others[...18 more]\n end\n gw --> coderm\n gw --> immm\n gw --> n8nm\n gw --> fetch\n gw --> time\n gw --> cw\n gw --> seq\n gw --> ham\n gw --> kr\n gw --> others\n coderm -.spawns/controls.-> CoderWS[(Coder workspaces<br/>CT 111)]\n ham -.bridges to.-> HAOS[(Home Assistant<br/>VM 100)]</code></pre>"},{"location":"services/homelab-architecture/#data-plane-what-lives-where","title":"Data plane \u2014 what lives where","text":"<pre><code>flowchart TB\n subgraph Tier1[\"Tier-1 / latency-sensitive \u00b7 local NVMe\"]\n pid_d[Pocket-ID sqlite \u00b7 CT 110]\n immich_pg[Immich Postgres \u00b7 CT 104]\n shared_pg[shared-postgres \u00b7 CT 104]\n coder_pg[coder-db \u00b7 CT 111]\n gitea_pg[gitea-db \u00b7 CT 111]\n garage[\"Garage S3 \u00b7 CT 104<br/>(moved off NFS 2026-05-19)\"]\n n8n_local[\"n8n data \u00b7 CT 104<br/>(reverted from NFS 2026-05-19)\"]\n end\n subgraph Bulk[\"Bulk \u00b7 UNAS NFSv3\"]\n media[\"Immich media, Paperless docs,<br/>arr-stack media, audiobooks\"]\n coder_homes[Coder workspace homes \u00b7 /mnt/pve/unas/services/coder]\n gitea_data[Gitea repos \u00b7 /mnt/pve/unas/services/gitea]\n vw_data[\"Vaultwarden data<br/>(tier-1 leak \u2014 plan to move local)\"]\n end\n subgraph BulkCIFS[\"Bulk \u00b7 UNAS CIFS (Nextcloud only)\"]\n nc_data[Nextcloud user files]\n end\n subgraph Backup[\"Off-host backup\"]\n jottacloud[(jottacloud<br/>via Backrest)]\n end\n Tier1 -. WAL-G .-> garage\n Bulk -. only media/data-dir .-> jottacloud\n classDef gap fill:#5a2a2a,stroke:#c44,color:#fcc\n class vw_data,jottacloud gap</code></pre> <p>The red blocks above are gaps: Vaultwarden is on NFS when it shouldn't be; off-host backup currently covers only one UNAS path, not service data. See <code>stacks/storage.md</code> for the verified state and the cleanup TODO list.</p>"},{"location":"services/ingest-pipeline/","title":"Ingest Pipeline \u2014 Filesystem Scan","text":"<p>One-shot pipeline that crawls a directory, converts documents to text, embeds them via the nomic service, and stores them in Qdrant. Run it manually against any mounted path.</p>"},{"location":"services/ingest-pipeline/#architecture","title":"Architecture","text":"<pre><code> \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 ingest.py --path /... \u2502 runs on CT 104 (has direct access to ai-internal network)\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 per file\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 1. File scanner glob recursively, filter by ext \u2502\n \u2502 2. Change detection sha256(path + mtime) \u2192 skip if seen \u2502\n \u2502 3. Docling converter POST http://docling:5001/convert \u2502\n \u2502 (PDF/DOCX/PPTX/XLSX/HTML \u2192 Markdown) \u2502\n \u2502 Plain text/Markdown read directly \u2502\n \u2502 Images (jpg/png/...) send to nomic vision endpoint \u2502\n \u2502 4. Chunker split Markdown by headers + size \u2502\n \u2502 chunk_size=1200 overlap=150 (matches OWUI RAG config) \u2502\n \u2502 5. Nomic embedder POST http://nomic:80/v1/embeddings \u2502\n \u2502 text chunks \u2192 768d images \u2192 768d (same space!) \u2502\n \u2502 6. Qdrant upsert collection `documents`, named vectors \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Qdrant: http://qdrant:6333 \u2502\n \u2502 collection: documents \u2502\n \u2502 vector: {size: 768, distance: Cosine} \u2502\n \u2502 payload: {path, title, chunk_idx, text, type, mtime, hash} \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Open WebUI RAG search \u2502 queries `documents` collection\n \u2502 n8n / MCP tools \u2502 same collection, same embedding space\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"services/ingest-pipeline/#why-not-n8n-for-this","title":"Why not n8n for this","text":"<p>n8n is the right tool for event-driven pipelines (webhook on new file, Paperless webhook, Nextcloud activity, scheduled re-sync). For a bulk one-shot crawl, a Python script is better: - direct filesystem access with <code>os.walk</code> - proper progress bar and error recovery - batched Qdrant upserts (n8n does one HTTP call per node) - easy CLI: <code>python3 ingest.py /mnt/pve/unas/Notizen</code></p> <p>n8n handles the ongoing layer (see Ongoing ingestion).</p>"},{"location":"services/ingest-pipeline/#supported-file-types","title":"Supported file types","text":"Extension Handler Notes <code>.pdf</code> Docling best quality, preserves tables <code>.docx</code>, <code>.odt</code> Docling <code>.pptx</code> Docling slides \u2192 sections <code>.xlsx</code>, <code>.ods</code> Docling tables \u2192 Markdown <code>.html</code>, <code>.htm</code> Docling <code>.md</code>, <code>.txt</code>, <code>.rst</code> direct read no conversion needed <code>.jpg</code>, <code>.jpeg</code>, <code>.png</code>, <code>.webp</code>, <code>.gif</code> nomic vision 768d image embedding, no text chunks"},{"location":"services/ingest-pipeline/#script-ingestpy","title":"Script: <code>ingest.py</code>","text":"<p>Lives at <code>/opt/stacks/ai/ingest/ingest.py</code>. Runs directly on CT 104.</p>"},{"location":"services/ingest-pipeline/#usage","title":"Usage","text":"<pre><code># Index everything under a path\npython3 /opt/stacks/ai/ingest/ingest.py --path /mnt/pve/unas/Notizen\n\n# Different collection, force re-index\npython3 /opt/stacks/ai/ingest/ingest.py \\\n --path /mnt/pve/unas/Dokumente \\\n --collection work-docs \\\n --force\n\n# Dry run (print files, don't embed)\npython3 /opt/stacks/ai/ingest/ingest.py --path /path/to/docs --dry-run\n</code></pre>"},{"location":"services/ingest-pipeline/#state-file","title":"State file","text":"<p><code>/opt/stacks/ai/ingest/state/<collection>.json</code> tracks <code>{path: {hash, indexed_at}}</code>. Subsequent runs skip unchanged files. Delete the state file to force full re-index.</p>"},{"location":"services/ingest-pipeline/#qdrant-collection-schema","title":"Qdrant collection schema","text":"<pre><code>vectors_config = VectorParams(size=768, distance=Distance.COSINE)\n# payload per point:\n{\n \"path\": \"/mnt/pve/unas/Notizen/someFile.md\",\n \"title\": \"someFile\", # filename without ext\n \"chunk_idx\": 0, # 0-based chunk index within file\n \"total_chunks\": 3,\n \"text\": \"\u2026chunk content\u2026\", # empty string for images\n \"type\": \"markdown\", # markdown | pdf | docx | image | \u2026\n \"mtime\": 1716700000.0,\n \"hash\": \"a3f\u2026\", # sha256 of file content\n \"source\": \"filesystem\",\n}\n</code></pre>"},{"location":"services/ingest-pipeline/#chunking-strategy","title":"Chunking strategy","text":"<p>For Markdown output from Docling (and raw .md/.txt): 1. Split on <code>##</code> / <code>###</code> headers first (keep header as first line of chunk) 2. If chunk > 1200 chars, split further on double-newline (<code>\\n\\n</code>) 3. If still > 1200 chars, hard-split with 150-char overlap</p> <p>Images: single point per file, no chunking.</p>"},{"location":"services/ingest-pipeline/#error-handling","title":"Error handling","text":"<ul> <li>Docling timeout (>60s): skip file, log to <code>ingest_errors.log</code>, continue</li> <li>Qdrant upsert failure: retry 3\u00d7, then log and continue</li> <li>Re-run is safe: state file prevents double-indexing</li> </ul>"},{"location":"services/ingest-pipeline/#building-and-running","title":"Building and running","text":"<pre><code># On CT 104\nmkdir -p /opt/stacks/ai/ingest\ncd /opt/stacks/ai/ingest\n\n# Install deps (lightweight \u2014 no torch needed, calls services via HTTP)\npip3 install qdrant-client requests tqdm\n\n# Run\npython3 ingest.py --path /mnt/pve/unas/Notizen\n</code></pre> <p>No container needed for the script itself \u2014 it runs on CT 104 bare Python and calls <code>http://docling:5001</code>, <code>http://nomic:80</code>, <code>http://qdrant:6333</code> via <code>ai-internal</code> (all on the same Docker network/host).</p> <p>To run it from outside CT 104 (e.g. the PVE host), wrap it in a container later.</p>"},{"location":"services/ingest-pipeline/#open-webui-integration","title":"Open WebUI integration","text":"<p>OWUI's knowledge base already points at Qdrant (<code>VECTOR_DB=qdrant</code>). To surface documents from the <code>documents</code> collection in chat:</p> <ol> <li>Admin \u2192 Knowledge \u2192 Create Knowledge Base</li> <li>Name: \"Local Documents\"</li> <li>The collection is populated by <code>ingest.py</code> \u2014 OWUI will search it on <code>#</code>-prefixed RAG queries or when the knowledge base is enabled in a chat.</li> </ol> <p>Note: OWUI creates its own internal collection names. To share the same <code>documents</code> collection between <code>ingest.py</code> and OWUI, use the OWUI API to create a knowledge base pointing to the pre-populated collection \u2014 or let OWUI manage its own collection and have <code>ingest.py</code> add documents via the OWUI knowledge API (<code>POST /api/v1/knowledge/{id}/file/add</code>). The OWUI API path is cleaner for OWUI search integration; the direct Qdrant path is better for external tools (n8n, MCP).</p>"},{"location":"services/ingest-pipeline/#ongoing-ingestion-n8n","title":"Ongoing ingestion (n8n)","text":"<p>After the one-shot crawl, wire n8n for continuous ingestion:</p> Trigger n8n nodes Notes Nextcloud webhook (file created/modified) HTTP \u2192 SSH \u2192 <code>ingest.py --path <file></code> Nextcloud admin \u2192 Webhooks app Paperless post-consume webhook HTTP \u2192 Docling \u2192 nomic \u2192 Qdrant upsert Paperless has <code>POST_CONSUME_SCRIPT</code> hook Cron re-scan Schedule \u2192 SSH \u2192 <code>ingest.py --path /mnt/pve/unas/Notizen</code> weekly full re-sync <p>The cron re-scan is safe because <code>ingest.py</code> skips unchanged files via state hash.</p>"},{"location":"services/ingest-pipeline/#paths-available-on-ct-104","title":"Paths available on CT 104","text":"Path Contents <code>/mnt/pve/unas/Notizen/</code> Obsidian vault (Markdown) <code>/mnt/pve/unas/</code> full UNAS NFS share <code>/opt/stacks/*/</code> stack configs (already in Git, lower priority) <p>Nextcloud files are on CT 105 (<code>192.168.1.41</code>). Access via: - WebDAV: <code>https://nc.nuclide.systems/remote.php/dav/files/fkrebs@nucli.de/</code> - Or mount the NC data volume \u2014 not currently mounted on CT 104</p>"},{"location":"services/ingest-pipeline/#status","title":"Status","text":"<p>Not yet implemented. Design only. Next step: write <code>ingest.py</code>.</p>"},{"location":"services/llm-benchmark/","title":"LLM Model Benchmark & Service Catalogue","text":"<p>Last updated: 2026-05-23 | Models: 58 | Source: LiteLLM /model/info + live benchmarks</p>"},{"location":"services/llm-benchmark/#overview","title":"Overview","text":"<p>This catalogue covers all 60 models registered in the homelab LiteLLM proxy (<code>http://192.168.1.40:14000</code>). Live latency figures are TTFT proxies measured from this host via a single <code>max_tokens=5</code> completion request. Speed tiers are based on live measurements and published inference benchmarks.</p> <p>Providers at a glance:</p> Provider Models Notes Claude (Anthropic via openai-compat) 6 claude-max subscription; temperature=0.7; aliases included Mistral API 10 voxtral voice family + codestral + OCR Gemini API 12 Flash/Pro/embedding families SAIA (self-hosted GPU cluster) 22 OpenAI-compatible; local GPU inference Groq 2 Ultra-fast cloud inference Cerebras 2 Ultra-fast wafer-scale inference Cohere 2 Embeddings only"},{"location":"services/llm-benchmark/#performance-tiers","title":"Performance Tiers","text":"Tier Symbol Typical TTFT Profile Ultra-fast \ud83d\ude80 < 200 ms Groq, Cerebras, cached SAIA small models Fast \u26a1 200\u2013600 ms Mistral API, Gemini Flash, SAIA mid-size Standard \ud83d\udd35 600\u20132 000 ms Claude, Gemini Pro, large API models Self-hosted \ud83c\udfe0 varies SAIA cluster; latency depends on GPU load & model size <p>Note: SAIA models with very low latency (< 50 ms) on the trivial benchmark likely hit a cached/KV-prefilled response; real-world TTFT for longer prompts will be higher. Treat SAIA figures as best-case.</p>"},{"location":"services/llm-benchmark/#model-catalogue","title":"Model Catalogue","text":""},{"location":"services/llm-benchmark/#chat-reasoning-models","title":"Chat & Reasoning Models","text":"Model ID Provider Backend Context Vision Tools Cost In $/1M Cost Out $/1M Live Latency Speed Tier Notes claude-sonnet-4-6 Anthropic openai-compat 200K \u2713 \u2713 $3.00 $15.00 2 083 ms \ud83d\udd35 Flagship; temp=0.7 claude-opus-4-7 Anthropic openai-compat 200K \u2713 \u2713 $15.00 $75.00 3 218 ms \ud83d\udd35 Highest capability; temp=0.7 claude-haiku-4-5 Anthropic openai-compat 200K \u2713 \u2713 $0.80 $4.00 1 453 ms \ud83d\udd35 Fast + cheap; temp=0.7 voxtral-small-latest Mistral Mistral API 256K \u2713 \u2713 \u2014 \u2014 160 ms \ud83d\ude80 Voice+text multimodal mistral-small-latest Mistral Mistral API 131K \u2713 \u2713 $0.06 $0.18 222 ms \ud83d\ude80 Cheapest Mistral chat voxtral-mini-latest Mistral Mistral API 100K \u2713 \u2713 \u2014 \u2014 ERROR \u274c Proxy config error (saia-image-proxy unreachable) codestral-latest Mistral Mistral API 16K \u2713 \u2713 $1.00 $3.00 275 ms \u26a1 Coding-specialised Mistral mistral-large-latest Mistral Mistral API 262K \u2713 \u2713 $0.50 $1.50 309 ms \u26a1 Flagship Mistral chat pixtral-large-latest Mistral Mistral API 128K \u2713 \u2713 $2.00 $6.00 27 ms \ud83d\ude80 Vision flagship; very low latency (likely cached) voxtral-mini-realtime-latest Mistral Mistral API 4K \u2717 \u2713 \u2014 \u2014 ERROR \u274c Invalid model per API; realtime/ws endpoint only gemini-2.5-flash Google Gemini API 65K \u2713 \u2713 $0.30 $2.50 501 ms \u26a1 Best-value Gemini; fast + smart gemini-2.5-flash-lite Google Gemini API 65K \u2713 \u2713 $0.10 $0.40 518 ms \u26a1 Lightest + cheapest Gemini gemini-2.5-pro Google Gemini API 65K \u2713 \u2713 $1.25 $10.00 888 ms \ud83d\udd35 Top Gemini reasoning gemini-3.1-pro-preview Google Gemini API 65K \u2713 \u2713 $2.00 $12.00 1 215 ms \ud83d\udd35 Next-gen Gemini Pro preview gemini-3-pro-preview Google Gemini API 65K \u2713 \u2713 $2.00 $12.00 1 258 ms \ud83d\udd35 Gemini 3 Pro preview gemini-3.1-flash-lite Google Gemini API \u2014 \u2713 \u2713 \u2014 \u2014 482 ms \u26a1 Gemini 3.1 flash lite preview devstral-2-123b-instruct-2512 Mistral SAIA 4K \u2717 \u2713 \u2014 \u2014 157 ms \ud83c\udfe0 123B coding model via SAIA GPU qwen3-coder-30b-a3b-instruct Alibaba SAIA 32K \u2717 \u2713 \u2014 \u2014 5 ms \ud83c\udfe0 MoE coding model; 5 ms = cached openai-gpt-oss-120b OpenAI SAIA 131K \u2717 \u2717 \u2014 \u2014 118 ms \ud83c\udfe0 OpenAI open-weight 120B via SAIA qwen3-omni-30b-a3b-instruct Alibaba SAIA 16K \u2713 \u2713 \u2014 \u2014 178 ms \ud83c\udfe0 Multimodal MoE; voice+vision llama-3.3-70b-instruct Meta SAIA 131K \u2717 \u2713 \u2014 \u2014 168 ms \ud83c\udfe0 Reliable general-purpose 70B deepseek-r1-distill-llama-70b DeepSeek SAIA 131K \u2717 \u2717 \u2014 \u2014 325 ms \ud83c\udfe0 R1 reasoning distill; outputs <code><think></code> tokens qwen3.5-35b-a3b Alibaba SAIA 65K \u2713 \u2713 \u2014 \u2014 250 ms \ud83c\udfe0 MoE 35B A3B qwen3.5-27b Alibaba SAIA 131K \u2717 \u2713 \u2014 \u2014 266 ms \ud83c\udfe0 Dense 27B qwen3.6-35b-a3b Alibaba SAIA 65K \u2713 \u2713 \u2014 \u2014 204 ms \ud83c\udfe0 MoE 35B A3B v3.6 apertus-70b-instruct-2509 Apertus SAIA 4K \u2717 \u2717 \u2014 \u2014 143 ms \ud83c\udfe0 Small context; general chat glm-4.7 Zhipu SAIA 128K \u2717 \u2713 \u2014 \u2014 210 ms \ud83c\udfe0 GLM-4 series qwen3-30b-a3b-instruct-2507 Alibaba SAIA 262K \u2717 \u2717 \u2014 \u2014 127 ms \ud83c\udfe0 MoE 30B, very large context gemma-3-27b-it Google SAIA 131K \u2717 \u2713 \u2014 \u2014 284 ms \ud83c\udfe0 Gemma 3 27B instruct internvl3.5-30b-a3b InternLM SAIA 16K \u2713 \u2713 \u2014 \u2014 116 ms \ud83c\udfe0 Vision+tools MoE 30B qwen3.5-122b-a10b Alibaba SAIA 65K \u2713 \u2713 \u2014 \u2014 372 ms \ud83c\udfe0 MoE 122B A10B; larger/slower qwen3.5-397b-a17b Alibaba SAIA 65K \u2713 \u2713 \u2014 \u2014 248 ms \ud83c\udfe0 Largest SAIA MoE model gemma-4-31b-it Google SAIA 8K \u2713 \u2713 \u2014 \u2014 177 ms \ud83c\udfe0 Gemma 4 multimodal meta-llama/llama-4-scout-17b-16e-instruct Meta Groq 8K \u2713 \u2713 $0.11 $0.34 442 ms \ud83d\ude80 Groq-accelerated; 442 ms incl. queue cerebras-llama-3.1-8b Meta Cerebras 128K \u2717 \u2713 $0.10 $0.10 258 ms \ud83d\ude80 Wafer-scale ~2 000 TPS cerebras-qwen-3-235b Alibaba Cerebras \u2014 \u2717 \u2717 \u2014 \u2014 199 ms \ud83d\ude80 235B at wafer-scale speed"},{"location":"services/llm-benchmark/#embedding-models","title":"Embedding Models","text":"Model ID Provider Backend Context Dimensions Cost $/1M Notes gemini-embedding-2 Google Gemini API 8K \u2014 $0.20 Primary Gemini embedding gemini-embedding-001 Google Gemini API 2K \u2014 $0.15 Legacy Gemini embedding text-embedding-ada-002 Google Gemini API 8K \u2014 $0.20 Alias \u2192 gemini-embedding-2 text-embedding-3-small Google Gemini API 8K \u2014 $0.20 Alias \u2192 gemini-embedding-2 text-embedding-3-large Google Gemini API 8K \u2014 $0.20 Alias \u2192 gemini-embedding-2 multilingual-e5-large-instruct Microsoft SAIA 8K 1 024 \u2014 Self-hosted multilingual; strong for DE/EN RAG cohere-embed-multilingual-v3 Cohere Cohere API 1K 1 024 $0.10 100+ languages cohere-embed-english-v3 Cohere Cohere API 1K 1 024 $0.10 English-only; higher EN accuracy"},{"location":"services/llm-benchmark/#audio-models-tts-asr","title":"Audio Models (TTS / ASR)","text":"Model ID Provider Backend Type Language Notes tts-1-de SAIA Piper TTS German Self-hosted German TTS tts-1 SAIA Kokoro-82M TTS EN + others Self-hosted multilingual TTS voxtral-mini-tts-latest Mistral Mistral API TTS Multilingual Mistral voice synthesis saia-whisper SAIA whisper-large-v2 ASR Multilingual Self-hosted transcription whisper-1 SAIA faster-whisper-large-v3 ASR Multilingual Faster self-hosted transcription voxtral-mini-transcribe-2507 Mistral Mistral API ASR Multilingual Mistral audio transcription; ctx 16K whisper-large-v3-turbo Meta Groq ASR Multilingual Groq-accelerated; fastest transcription"},{"location":"services/llm-benchmark/#image-models","title":"Image Models","text":"Model ID Provider Backend Type Notes saia-flux SAIA FLUX Image gen Self-hosted FLUX; note: garbles text labels gemini-2.5-flash-image Google Gemini API Image gen ctx 32K; multimodal image generation saia-image-edit SAIA Qwen-Image-Edit Image edit Image editing/inpainting mistral-ocr-latest Mistral Mistral API OCR Document OCR; not a chat model"},{"location":"services/llm-benchmark/#service-recommendations","title":"Service Recommendations","text":""},{"location":"services/llm-benchmark/#nextcloud-assistant","title":"Nextcloud Assistant","text":"<p>Smart file/email/calendar assistant, summaries, writing help \u2014 multilingual DE/EN</p> Role Model Reasoning Primary <code>mistral-small-latest</code> Cheapest API model with tool use, vision, 131K context, and 222 ms TTFT. Handles German natively. Fallback <code>claude-haiku-4-5</code> If higher quality needed; still cost-effective at $0.80/$4 and 1 453 ms TTFT. Alt (local) <code>qwen3.5-27b</code> Free if staying fully on SAIA GPU cluster; 131K context + tools. <pre><code>model: mistral-small-latest\n</code></pre>"},{"location":"services/llm-benchmark/#karakeep-bookmarks-reading","title":"Karakeep (Bookmarks / Reading)","text":"<p>Summarise articles, extract key points, tag/categorise \u2014 no vision required</p> Role Model Reasoning Primary <code>gemini-2.5-flash-lite</code> Cheapest API model at $0.10/$0.40, 518 ms TTFT, strong comprehension. Fallback <code>mistral-small-latest</code> Slightly pricier but faster at 222 ms. <pre><code>model: gemini-2.5-flash-lite\n</code></pre>"},{"location":"services/llm-benchmark/#home-assistant","title":"Home Assistant","text":"<p>Intent recognition, automation triggers, voice pipeline \u2014 ultra-low latency critical</p> Role Model Reasoning Primary <code>cerebras-llama-3.1-8b</code> 258 ms measured TTFT, ~2 000 TPS on Cerebras wafer silicon; best latency for real-time voice. 128K context, tools. Fallback <code>mistral-small-latest</code> 222 ms TTFT, API-based, reliable tool calling. Local alt <code>internvl3.5-30b-a3b</code> 116 ms on SAIA; avoids API cost for high-frequency automations. <pre><code>model: cerebras-llama-3.1-8b\n</code></pre>"},{"location":"services/llm-benchmark/#lobechat-default","title":"LobeChat Default","text":"<p>General chat assistant for daily use \u2014 balanced quality / speed / cost, vision nice</p> Role Model Reasoning Primary <code>gemini-2.5-flash</code> 501 ms, vision, tools, large context, excellent reasoning at $0.30/$2.50. Best all-rounder. Fallback <code>claude-sonnet-4-6</code> Higher quality ceiling; use when depth matters over cost. Free alt <code>qwen3.5-397b-a17b</code> Largest self-hosted model; free on SAIA with vision + tools at 248 ms. <pre><code>model: gemini-2.5-flash\n</code></pre>"},{"location":"services/llm-benchmark/#code-assistant-coder-ide","title":"Code Assistant (Coder / IDE)","text":"<p>Code completion, review, debugging \u2014 strong code ability, large context, tools</p> Role Model Reasoning Primary <code>claude-sonnet-4-6</code> Best overall coding + reasoning; 200K context, tool use, reliable output. Fast/cheap <code>codestral-latest</code> Coding-specialist Mistral at 275 ms with 16K context; good for completion. Local coding <code>devstral-2-123b-instruct-2512</code> 123B SAIA coding model at 157 ms; free inference. MoE coding <code>qwen3-coder-30b-a3b-instruct</code> 32K context, tools, extremely fast (5 ms cached); best SAIA coding model. <pre><code>model: claude-sonnet-4-6 # IDE / review\nmodel: qwen3-coder-30b-a3b-instruct # local completion\n</code></pre>"},{"location":"services/llm-benchmark/#document-ocr-ingestion","title":"Document OCR / Ingestion","text":"<p>Paperless \u2192 Docling \u2192 extract text \u2014 vision + OCR capable, large context</p> Role Model Reasoning Primary <code>mistral-ocr-latest</code> Dedicated OCR endpoint; purpose-built for document text extraction. Fallback <code>gemini-2.5-pro</code> 65K context, vision, strong at structured extraction from images. Alt vision <code>pixtral-large-latest</code> Mistral vision flagship at 27 ms (cached); good document parsing. <pre><code>model: mistral-ocr-latest # OCR pipeline\nmodel: gemini-2.5-pro # fallback / complex layouts\n</code></pre>"},{"location":"services/llm-benchmark/#embeddings-karakeep-lobechat-kb","title":"Embeddings (Karakeep / LobeChat KB)","text":"<p>Semantic search, RAG, knowledge base \u2014 multilingual, high dimensions</p> Role Model Reasoning Primary (API) <code>gemini-embedding-2</code> 8K context, $0.20/1M, strong multilingual. Primary (local) <code>multilingual-e5-large-instruct</code> Self-hosted on SAIA, 1 024-dim, excellent DE/EN RAG, zero API cost. Multilingual API <code>cohere-embed-multilingual-v3</code> 100+ languages, 1K context, $0.10/1M. <pre><code>model: multilingual-e5-large-instruct # local RAG\nmodel: gemini-embedding-2 # API fallback\n</code></pre> <p>Note: <code>text-embedding-ada-002</code>, <code>text-embedding-3-small</code>, and <code>text-embedding-3-large</code> are all aliases for <code>gemini-embedding-2</code> \u2014 use the canonical ID to avoid confusion.</p>"},{"location":"services/llm-benchmark/#image-generation","title":"Image Generation","text":"<p>ComfyUI complement, quick drafts</p> Role Model Reasoning Primary <code>saia-flux</code> Self-hosted FLUX on SAIA GPU; no API cost. Note: avoid text in generated images (garbles). API alt <code>gemini-2.5-flash-image</code> Gemini multimodal image gen for quick API-based drafts. Editing <code>saia-image-edit</code> Qwen image editing for inpainting / modifications. <pre><code>model: saia-flux\n</code></pre>"},{"location":"services/llm-benchmark/#tts-voice-interfaces","title":"TTS (Voice Interfaces)","text":"<p>Read content aloud, voice responses</p> Role Model Reasoning German <code>tts-1-de</code> Self-hosted Piper; native German pronunciation. Multilingual <code>tts-1</code> Self-hosted Kokoro-82M; covers EN + others, zero cost. API quality <code>voxtral-mini-tts-latest</code> Mistral neural TTS for higher-quality voice synthesis. <pre><code>model: tts-1-de # German HA / Nextcloud voice\nmodel: tts-1 # English / multilingual\n</code></pre>"},{"location":"services/llm-benchmark/#transcription-meetings-voice","title":"Transcription (Meetings / Voice)","text":"<p>Speech to text</p> Role Model Reasoning Primary <code>whisper-large-v3-turbo</code> Groq-accelerated; fastest available transcription. Local <code>whisper-1</code> faster-whisper-large-v3 on SAIA; fully self-hosted, no API cost. Fallback <code>saia-whisper</code> whisper-large-v2 on SAIA; slightly older model. <pre><code>model: whisper-large-v3-turbo # real-time meetings\nmodel: whisper-1 # offline / batch\n</code></pre>"},{"location":"services/llm-benchmark/#reasoning-analysis","title":"Reasoning / Analysis","text":"<p>Complex problem solving, research, multi-step tasks</p> Role Model Reasoning Primary <code>claude-opus-4-7</code> Highest Claude capability; temp=0.7. Cheaper <code>gemini-2.5-pro</code> Strong reasoning at 888 ms, $1.25/$10.00; good for research tasks. Local reasoning <code>deepseek-r1-distill-llama-70b</code> R1 chain-of-thought via SAIA at 325 ms; outputs <code><think></code> tokens. Fast reasoning <code>cerebras-qwen-3-235b</code> 235B model at 199 ms on Cerebras wafer silicon. <pre><code>model: claude-opus-4-7 # deep analysis\nmodel: deepseek-r1-distill-llama-70b # local reasoning\n</code></pre>"},{"location":"services/llm-benchmark/#batch-offline-processing","title":"Batch / Offline Processing","text":"<p>Non-real-time document processing \u2014 cost-optimised, high throughput</p> Role Model Reasoning Primary <code>gemini-2.5-flash-lite</code> $0.10/$0.40; cheapest API model with tools + vision. Free <code>llama-3.3-70b-instruct</code> SAIA self-hosted 70B at 168 ms; 131K context, no API cost. Alt <code>qwen3-30b-a3b-instruct-2507</code> 262K context MoE; good for long-document batch on SAIA. <pre><code>model: gemini-2.5-flash-lite # cost-sensitive API batch\nmodel: llama-3.3-70b-instruct # free local batch\n</code></pre>"},{"location":"services/llm-benchmark/#aliases-duplicates","title":"Aliases & Duplicates","text":"<p>The following model IDs are aliases that route to the same backend model. Use the canonical ID in production to avoid ambiguity:</p> Alias ID Canonical Model Notes <code>sonnet</code> <code>claude-sonnet-4-6</code> Short alias <code>opus</code> <code>claude-opus-4-7</code> Short alias <code>haiku</code> <code>claude-haiku-4-5</code> Short alias <code>text-embedding-ada-002</code> <code>gemini-embedding-2</code> OpenAI compat alias <code>text-embedding-3-small</code> <code>gemini-embedding-2</code> OpenAI compat alias <code>text-embedding-3-large</code> <code>gemini-embedding-2</code> OpenAI compat alias"},{"location":"services/llm-benchmark/#experimental-not-yet-validated","title":"Experimental / Not Yet Validated","text":"<p>The following models returned errors or have unresolved issues in live testing:</p> Model ID Status Error Detail Action <code>voxtral-mini-latest</code> \u2705 Fixed 2026-05-23 Stray <code>api_base: saia-image-proxy:5999</code> \u2014 deleted + re-added clean; now routes to Mistral API \u2014 <code>voxtral-mini-realtime-latest</code> \ud83d\uddd1\ufe0f Removed 2026-05-23 WebSocket-only realtime endpoint; incompatible with REST completions Removed from LiteLLM; use Mistral WS API directly if needed <code>mistral-ocr-latest</code> \u26a0\ufe0f Not benchmarked OCR-mode model; requires document input, not chat completions Use via dedicated OCR pipeline only <code>voxtral-mini-transcribe-2507</code> \u26a0\ufe0f Not benchmarked Audio transcription; not a chat completions model Use via audio transcription endpoint <code>gemini-2.5-flash-image</code> \u26a0\ufe0f Not benchmarked Image generation; not a chat completions model Use via images endpoint <code>saia-flux</code> \u26a0\ufe0f Not benchmarked FLUX image generation Use via images endpoint <code>saia-image-edit</code> \u26a0\ufe0f Not benchmarked Image editing Use via image edit endpoint <code>tts-1-de</code> \u26a0\ufe0f Not benchmarked Piper TTS audio output Use via audio/speech endpoint <code>tts-1</code> \u26a0\ufe0f Not benchmarked Kokoro-82M TTS Use via audio/speech endpoint <code>voxtral-mini-tts-latest</code> \u26a0\ufe0f Not benchmarked Mistral TTS Use via audio/speech endpoint <code>saia-whisper</code> \u26a0\ufe0f Not benchmarked Whisper ASR Use via audio/transcriptions endpoint <code>whisper-1</code> \u26a0\ufe0f Not benchmarked faster-whisper ASR Use via audio/transcriptions endpoint <code>whisper-large-v3-turbo</code> \u26a0\ufe0f Not benchmarked Groq Whisper ASR Use via audio/transcriptions endpoint <code>gemini-3.1-flash-lite</code> \u26a0\ufe0f Context unknown Preview model; ctx window not documented Monitor Gemini API release notes <code>cerebras-qwen-3-235b</code> \u26a0\ufe0f Context unknown Context window not documented in LiteLLM config Check Cerebras API docs"},{"location":"services/llm-benchmark/#raw-benchmark-data","title":"Raw Benchmark Data","text":"<p>All measurements from <code>2026-05-23</code>. Single <code>max_tokens=5</code> completion, prompt: \"Reply with exactly: ok\".</p> Model ID HTTP Status Latency (ms) Prompt Tokens Completion Tokens claude-sonnet-4-6 200 2 083 3 4 claude-opus-4-7 200 3 218 6 6 claude-haiku-4-5 200 1 453 10 41 voxtral-small-latest 200 160 8 2 mistral-small-latest 200 222 20 2 voxtral-mini-latest 500 48 \u2014 \u2014 codestral-latest 200 275 13 2 devstral-2-123b-instruct-2512 200 157 8 2 qwen3-coder-30b-a3b-instruct 200 5 13 2 openai-gpt-oss-120b 200 118 74 5 gemini-2.5-flash 200 501 6 1 cerebras-llama-3.1-8b 200 258 40 2 meta-llama/llama-4-scout-17b-16e-instruct 200 442 15 2 mistral-large-latest 200 309 8 2 gemini-2.5-flash-lite 200 518 6 1 qwen3-omni-30b-a3b-instruct 200 178 13 2 llama-3.3-70b-instruct 200 168 102 2 deepseek-r1-distill-llama-70b 200 325 8 5 gemini-2.5-pro 200 888 6 2 pixtral-large-latest 200 27 13 2 voxtral-mini-realtime-latest 400 140 \u2014 \u2014 qwen3.5-35b-a3b 200 250 15 5 qwen3.5-27b 200 266 15 5 qwen3.6-35b-a3b 200 204 15 5 apertus-70b-instruct-2509 200 143 66 2 glm-4.7 200 210 9 2 gemini-3.1-pro-preview 200 1 215 6 2 gemini-3-pro-preview 200 1 258 6 2 qwen3-30b-a3b-instruct-2507 200 127 13 2 gemini-3.1-flash-lite 200 482 6 1 gemma-3-27b-it 200 284 14 3 internvl3.5-30b-a3b 200 116 13 2 qwen3.5-122b-a10b 200 372 15 5 qwen3.5-397b-a17b 200 248 15 5 gemma-4-31b-it 200 177 18 2 cerebras-qwen-3-235b 200 199 13 2 <p>Benchmark note on SAIA models with < 50 ms latency (qwen3-coder: 5 ms, pixtral: 27 ms): these figures reflect a KV-cache or pre-warmed response for the trivial prompt. Real-world TTFT for cold prompts will be 100\u2013400 ms depending on model size and GPU availability.</p>"},{"location":"services/mcp-gateway/","title":"MCP Gateway \u2014 Bifrost (migrated 2026-05-26)","text":"<p>MCP servers are now aggregated by Bifrost at <code>https://ai.nuclide.systems/mcp</code>. The legacy FastAPI DinD mcp-gateway (<code>mcp.nuclide.systems</code>) is pending decommission.</p>"},{"location":"services/mcp-gateway/#architecture","title":"Architecture","text":"<ul> <li>Stack: <code>ai/bifrost/</code> on CT 104 \u2014 Bifrost LLM+MCP gateway, SQLite state at <code>data/config.db</code>.</li> <li>Endpoint: <code>https://ai.nuclide.systems/mcp</code> (Zoraxy \u2192 <code>192.168.1.40:14003</code>)</li> <li>Auth: Virtual Key (<code>sk-bf-</code> prefix) via <code>Authorization: Bearer <vk></code>.</li> <li>Client model: Each upstream MCP server is a registered client in Bifrost (<code>auth_type=none</code>, <code>allow_on_all_virtual_keys=true</code>). Tools are auto-discovered and enabled via <code>tools_to_execute_json</code>.</li> <li>Internal upstreams: All child MCP containers run on the <code>ai-internal</code> Docker network at <code>http://<name>-mcp:8000/mcp</code> (streamable-HTTP) or as dedicated stacks.</li> </ul>"},{"location":"services/mcp-gateway/#virtual-keys","title":"Virtual keys","text":"Name Key prefix Use <code>claude-code</code> <code>sk-bf-bfc19117-4c46-4d48-9b10-85d78b1ae2b3</code> Claude Code + Claude.ai <code>mcp-dev</code> <code>sk-bf-279d3ecc-9031-41ff-a582-ecf3dca61d52</code> Testing / dev <code>open-webui</code> <code>sk-bf-7e6999fc-2d86-48f9-8ac9-0ba558893d87</code> Open WebUI (<code>LITELLM_API_KEY</code> in <code>ai/.env</code>)"},{"location":"services/mcp-gateway/#server-inventory-as-of-2026-05-26","title":"Server inventory (as of 2026-05-26)","text":"<p>29 connected clients, ~760 tools total.</p> Client name Upstream Notes <code>bluesky</code> <code>http://ariel-mcp:8000/mcp</code> <code>coder</code> <code>http://coder-mcp:8000/mcp</code> <code>comfyui</code> <code>http://comfyui-mcp:8000/mcp</code> Intel Arc image gen <code>context7</code> <code>http://mcp-context7:8000/mcp</code> <code>crawl4ai</code> <code>http://mcp-crawl4ai:11235/mcp/sse</code> (SSE) <code>docling</code> <code>http://docling-mcp:8000/mcp</code> PDF\u2192Markdown <code>fetch</code> <code>http://mcp-fetch:8000/mcp</code> <code>git</code> <code>http://mcp-git:8000/mcp</code> <code>gitea</code> <code>http://gitea-mcp:8000/mcp</code> <code>gitlab</code> <code>http://mcp-gitlab:8000/mcp</code> <code>GITLAB_API_URL=https://gitlab.dlr.de/api/v4</code> <code>gotify</code> <code>http://mcp-gotify:8000/mcp</code> <code>home_assistant</code> <code>http://192.168.1.60:9583/private_ehnWeRl2G3De6NnbcN7teQ</code> HA add-on; no TLS <code>immich</code> <code>http://mcp-immich:8000/mcp</code> <code>kroki</code> <code>http://kroki-mcp:8000/mcp</code> Diagram rendering <code>markitdown</code> <code>http://mcp-markitdown:8000/mcp</code> <code>memos</code> <code>http://mcp-memos:8000/mcp</code> <code>nextcloud</code> <code>http://mcp-nextcloud:8000/mcp</code> <code>ntfy</code> <code>http://mcp-ntfy:8000/mcp</code> <code>obsidian</code> <code>http://mcp-obsidian:8000/mcp</code> Vault at Nextcloud/UNAS <code>paper_search</code> <code>http://mcp-paper-search:8000/mcp</code> <code>paperless</code> <code>http://paperless-mcp:8000/mcp</code> <code>proxmox</code> <code>http://mcp-proxmox:8000/mcp</code> Read-only (<code>PVEAuditor</code>) <code>searxng</code> <code>http://mcp-searxng:8000/mcp</code> <code>sequential_thinking</code> <code>http://mcp-sequential-thinking:8000/mcp</code> <code>time</code> <code>http://mcp-time:8000/mcp</code> <code>unifi</code> <code>http://mcp-unifi:8000/mcp</code> <code>upload_artifact</code> <code>http://upload-artifact-mcp:8000/mcp</code> S3 via Garage <code>wikipedia</code> <code>http://mcp-wikipedia-mcp:8000/mcp</code> <code>youtube_transcript</code> <code>http://mcp-youtube-transcript:8000/mcp</code>"},{"location":"services/mcp-gateway/#not-yet-connected","title":"Not yet connected","text":"Client Reason <code>n8n</code> \u2014 <code>https://n8n.nuclide.systems/mcp-server/http</code> Streamable-HTTP transport: POST returns SSE stream, Bifrost HTTP client times out. <code>shepard</code> \u2014 <code>https://shepard.nuclide.systems/v2/mcp</code> Same streamable-HTTP issue. Bearer token stored in <code>/tmp/migrate_mcp_oauth.py</code>."},{"location":"services/mcp-gateway/#client-configuration","title":"Client configuration","text":""},{"location":"services/mcp-gateway/#claude-code","title":"Claude Code","text":"<p>Add to <code>~/.claude.json</code> or project <code>.mcp.json</code>:</p> <pre><code>{\n \"mcpServers\": {\n \"nuclide\": {\n \"type\": \"http\",\n \"url\": \"https://ai.nuclide.systems/mcp\",\n \"headers\": {\n \"Authorization\": \"Bearer sk-bf-bfc19117-4c46-4d48-9b10-85d78b1ae2b3\"\n }\n }\n }\n}\n</code></pre>"},{"location":"services/mcp-gateway/#claudeai","title":"Claude.ai","text":"<p>Settings \u2192 Integrations \u2192 Add MCP server: - URL: <code>https://ai.nuclide.systems/mcp</code> - Header: <code>Authorization: Bearer sk-bf-279d3ecc-9031-41ff-a582-ecf3dca61d52</code></p>"},{"location":"services/mcp-gateway/#llm-inference-governance","title":"LLM Inference & Governance","text":"<p>Bifrost proxies LLM inference at <code>/v1</code> (OpenAI-compatible). <code>enforce_auth_on_inference=1</code> \u2014 all <code>/v1</code> calls require a valid <code>sk-bf-*</code> VK.</p>"},{"location":"services/mcp-gateway/#providers","title":"Providers","text":"Provider Internal name Base URL Notes SAIA (GPU cluster) <code>openai</code> <code>https://chat-ai.academiccloud.de/v1</code> Rate limited (see below) Google Gemini <code>gemini</code> default Mistral AI <code>mistral</code> default Cerebras <code>cerebras</code> default claude-max-bridge <code>openrouter</code> <code>http://claude-max-bridge:8000</code> Claude Opus/Sonnet/Haiku via max subscription"},{"location":"services/mcp-gateway/#saia-rate-limits","title":"SAIA rate limits","text":"<p>SAIA enforces per-account quotas. Bifrost is configured with a global provider-level limit (<code>config_providers.rate_limit_id='saia-minute'</code>):</p> Window SAIA limit Bifrost enforcement Per minute 30 req \u2705 active (<code>saia-minute</code> row) Per hour 200 req row exists (<code>saia-hour</code>), not linked Per day 1000 req row exists (<code>saia-day</code>), not linked Per month 3000 req not trackable across restarts <p>Note: Bifrost only supports one rate limit window per provider. The minute window is linked because it provides burst protection. The <code>saia-hour</code> / <code>saia-day</code> rows are in <code>governance_rate_limits</code> and can be linked by updating <code>config_providers SET rate_limit_id='saia-hour'</code> if needed.</p> <p>To change the active window: <pre><code>sqlite3 /opt/stacks/ai/bifrost/data/config.db \\\n \"UPDATE config_providers SET rate_limit_id='saia-day' WHERE name='openai';\"\ndocker compose -f /opt/stacks/ai/bifrost.yml up -d --force-recreate\n</code></pre></p> <p>Rate limit API: <code>GET /api/governance/rate-limits</code> is read-only. POST/PUT return 405. Use direct SQLite to create new entries.</p>"},{"location":"services/mcp-gateway/#virtual-key-governance","title":"Virtual key governance","text":"<p>All VKs (<code>governance_virtual_key_provider_configs</code>) have: - <code>allow_all_keys=1</code> \u2014 set via SQL (Bifrost API PUT silently ignores this field) - <code>allowed_models</code> \u2014 explicit JSON model list as text in DB (SQL NULL = deny all with enforce_auth_on_inference=1)</p> <p>If models stop working after a Bifrost upgrade/restore, re-run <code>/tmp/fix_vk_final.py</code> on CT 104 and then: <pre><code>sqlite3 /opt/stacks/ai/bifrost/data/config.db \\\n \"UPDATE governance_virtual_key_provider_configs SET allow_all_keys=1;\"\ndocker compose -f /opt/stacks/ai/bifrost.yml up -d --force-recreate\n</code></pre></p>"},{"location":"services/mcp-gateway/#ops","title":"Ops","text":"<pre><code># On CT 104\ncd /opt/stacks/ai/bifrost\ndocker compose up -d --force-recreate\n\ndocker logs bifrost -f\n\n# Inspect config DB\nsqlite3 data/config.db '.tables'\nsqlite3 data/config.db 'SELECT name, auth_type, allow_on_all_virtual_keys FROM config_mcp_clients;'\n\n# Count active tools via API\ncurl -s http://localhost:14003/mcp \\\n -H \"Authorization: Bearer sk-bf-279d3ecc-9031-41ff-a582-ecf3dca61d52\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\",\"params\":{}}' | python3 -m json.tool | grep '\"name\"' | wc -l\n</code></pre>"},{"location":"services/mcp-gateway/#add-a-new-mcp-client","title":"Add a new MCP client","text":"<pre><code>curl -sc /tmp/bfcookies http://localhost:14003/api/session/login \\\n -H \"Content-Type: application/json\" \\\n -d '{\"username\":\"fkrebs\",\"password\":\"tapirnase\"}' > /dev/null\n\ncurl -s -X POST http://localhost:14003/api/mcp/client \\\n -b /tmp/bfcookies \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"name\": \"<name>\",\n \"connection_type\": \"http\",\n \"connection_string\": \"http://<host>:8000/mcp\",\n \"auth_type\": \"none\",\n \"allow_on_all_virtual_keys\": true\n }'\n</code></pre> <p>Wait ~10 s for tool discovery, then enable tools via <code>PUT /api/mcp/client/<id></code> with <code>tools_to_execute</code>. See <code>/tmp/add_all_mcp_clients.py</code> on CT 104 for a complete example.</p>"},{"location":"services/mcp-gateway/#key-requirements-for-tools-to-appear-in-mcp-toolslist","title":"Key requirements for tools to appear in <code>/mcp tools/list</code>","text":"<ol> <li><code>auth_type=none</code> (not <code>per_user_oauth</code>)</li> <li><code>allow_on_all_virtual_keys=true</code></li> <li><code>tools_to_execute_json</code> populated (tool names without client prefix)</li> <li>VK must have <code>sk-bf-</code> prefix</li> </ol>"},{"location":"services/mcp-gateway/#legacy-mcp-gateway-fastapi-dind","title":"Legacy mcp-gateway (FastAPI / DinD)","text":"<p>Status: DECOMMISSIONED 2026-05-26.</p> <ul> <li>Zoraxy route removed (config renamed to <code>.DECOMMISSIONED-2026-05-26</code> on CT 108)</li> <li>Container stopped; compose file renamed to <code>docker-compose.yml.DECOMMISSIONED-2026-05-26</code></li> <li>Pocket-ID clients <code>e73bb7b9</code> (litellm/mcp-gateway) and <code>78c78998</code> (Claude MCP) deleted from CT 110 DB</li> </ul>"},{"location":"services/mcp-gateway/#bifrost-oidc-client-pocket-id","title":"Bifrost OIDC client (Pocket ID)","text":"<p>Used for <code>per_user_oauth</code> flows (not currently active \u2014 all clients use <code>auth_type=none</code>):</p> Field Value Client ID <code>ec0d15e6-e86d-49b0-ac12-cdfb5afc9086</code> Authorize URL <code>https://id.nuclide.systems/authorize</code> Token URL <code>https://id.nuclide.systems/api/oidc/token</code> <code>mcp_external_client_url</code> <code>https://ai.nuclide.systems</code>"},{"location":"services/mcp-servers/","title":"MCP Servers \u2014 nuclide.systems","text":"<p>Complete reference for all 30 MCP servers. MCP is now served by Bifrost at <code>https://ai.nuclide.systems/mcp</code>. Last verified: 2026-05-23. Gateway migrated to Bifrost: 2026-05-26.</p> <p>The old mcp-gateway (FastAPI/DinD, <code>mcp.nuclide.systems</code>) was decommissioned 2026-05-26 \u2014 compose renamed <code>.DECOMMISSIONED-2026-05-26</code>. Config file was at <code>/opt/stacks/ai/mcp-gateway/config.json</code> on CT 104 \u00b7 Secrets: <code>/opt/stacks/ai/.env</code></p>"},{"location":"services/mcp-servers/#overview","title":"Overview","text":"# Server Group Kind Gateway URL Status 1 <code>context7</code> dev catalog <code>https://ai.nuclide.systems/mcp/context7/mcp</code> \u2713 2 <code>fetch</code> dev spawn <code>https://ai.nuclide.systems/mcp/fetch/mcp</code> \u2713 3 <code>git</code> dev catalog <code>https://ai.nuclide.systems/mcp/git/mcp</code> \u2713 4 <code>gitlab</code> dev catalog <code>https://ai.nuclide.systems/mcp/gitlab/mcp</code> \u2713 5 <code>kroki</code> dev static <code>https://ai.nuclide.systems/mcp/kroki/mcp</code> \u2713 6 <code>markitdown</code> dev catalog <code>https://ai.nuclide.systems/mcp/markitdown/mcp</code> \u2713 7 <code>sequential-thinking</code> dev catalog <code>https://ai.nuclide.systems/mcp/sequential-thinking/mcp</code> \u2713 8 <code>time</code> dev spawn <code>https://ai.nuclide.systems/mcp/time/mcp</code> \u2713 9 <code>docling</code> dev static <code>https://ai.nuclide.systems/mcp/docling/mcp</code> \u2713 10 <code>coder</code> dev static <code>https://ai.nuclide.systems/mcp/coder/mcp</code> \u2713 11 <code>gitea</code> dev static <code>https://ai.nuclide.systems/mcp/gitea/mcp</code> \u2713 12 <code>proxmox</code> dev spawn <code>https://ai.nuclide.systems/mcp/proxmox/mcp</code> \u2713 13 <code>shepard</code> dev static <code>https://ai.nuclide.systems/mcp/shepard/mcp</code> \u2713 14 <code>crawl4ai</code> research spawn <code>https://ai.nuclide.systems/mcp/crawl4ai/mcp</code> \u2713 15 <code>paper-search</code> research catalog <code>https://ai.nuclide.systems/mcp/paper-search/mcp</code> \u2713 16 <code>searxng</code> research spawn <code>https://ai.nuclide.systems/mcp/searxng/mcp</code> \u2713 17 <code>wikipedia-mcp</code> research catalog <code>https://ai.nuclide.systems/mcp/wikipedia-mcp/mcp</code> \u2713 18 <code>youtube-transcript</code> research catalog <code>https://ai.nuclide.systems/mcp/youtube-transcript/mcp</code> \u2713 19 <code>bluesky</code> personal static <code>https://ai.nuclide.systems/mcp/bluesky/mcp</code> \u2713 20 <code>obsidian</code> personal spawn <code>https://ai.nuclide.systems/mcp/obsidian/mcp</code> \u2713 21 <code>gotify</code> personal spawn <code>https://ai.nuclide.systems/mcp/gotify/mcp</code> \u2713 22 <code>home-assistant</code> personal static <code>https://ai.nuclide.systems/mcp/home-assistant/mcp</code> \u2713 23 <code>immich</code> personal spawn <code>https://ai.nuclide.systems/mcp/immich/mcp</code> \u2713 24 <code>memos</code> personal spawn <code>https://ai.nuclide.systems/mcp/memos/mcp</code> \u2713 25 <code>n8n</code> personal static <code>https://ai.nuclide.systems/mcp/n8n/mcp</code> \u2713 26 <code>nextcloud</code> personal spawn <code>https://ai.nuclide.systems/mcp/nextcloud/mcp</code> \u2713 27 <code>paperless</code> personal static <code>https://ai.nuclide.systems/mcp/paperless/mcp</code> \u2713 28 <code>unifi</code> personal spawn <code>https://ai.nuclide.systems/mcp/unifi/mcp</code> \u2713 29 <code>comfyui</code> image static <code>https://ai.nuclide.systems/mcp/comfyui/mcp</code> \u2713 30 <code>upload-artifact</code> storage static <code>https://ai.nuclide.systems/mcp/upload-artifact/mcp</code> \u2713 <p>Auth: all Bifrost MCP URLs require <code>Authorization: Bearer sk-bf-<key></code>. MCP endpoint: <code>https://ai.nuclide.systems/mcp</code>.</p>"},{"location":"services/mcp-servers/#group-dev","title":"Group: <code>dev</code>","text":""},{"location":"services/mcp-servers/#context7-library-documentation","title":"<code>context7</code> \u2014 Library documentation","text":"Value Kind catalog (<code>mcp/context7</code>) Internal upstream <code>http://mcp-context7:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/context7</code> Transport streamable-HTTP Resources 256 MiB mem limit Cache TTL 300 s Credentials none Notes Fetches live documentation for libraries/frameworks. No env vars."},{"location":"services/mcp-servers/#fetch-http-fetch","title":"<code>fetch</code> \u2014 HTTP fetch","text":"Value Kind spawn Internal upstream <code>http://mcp-fetch:8000</code> Image <code>ghcr.io/astral-sh/uv:python3.12-trixie-slim</code> Command <code>uvx --with mcp-server-fetch mcp-proxy --stateless --host 0.0.0.0 --port 8000 -- python -m mcp_server_fetch</code> Transport streamable-HTTP Cache TTL 120 s Credentials none"},{"location":"services/mcp-servers/#git-git-operations","title":"<code>git</code> \u2014 Git operations","text":"Value Kind catalog (<code>mcp/git</code>) Internal upstream <code>http://mcp-git:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/git</code> Transport streamable-HTTP Resources 256 MiB mem limit Credentials none Notes DinD child runs in <code>ai-internal</code> network."},{"location":"services/mcp-servers/#gitlab-gitlab-dlr","title":"<code>gitlab</code> \u2014 GitLab (DLR)","text":"Value Kind catalog (<code>mcp/gitlab</code>) Internal upstream <code>http://mcp-gitlab:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/gitlab</code> Transport streamable-HTTP Resources 256 MiB mem limit Credentials <code>GITLAB_PERSONAL_ACCESS_TOKEN</code> <code>C8OXPT0J_oY-ha6KAzcn0286MQp1OjEyaAk.01.0z0jn01p4</code> <code>GITLAB_API_URL</code> <code>https://gitlab.dlr.de/api/v4</code> Notes Uses <code>--pass-environment</code> in bridge cmd so Docker CLI resolves <code>-e KEY</code> passthrough. Targets DLR GitLab, not gitlab.com."},{"location":"services/mcp-servers/#kroki-diagram-rendering","title":"<code>kroki</code> \u2014 Diagram rendering","text":"Value Kind static Internal upstream <code>http://kroki-mcp:8000</code> (own stack, <code>:18007</code> on host) Transport streamable-HTTP Health check every 300 s Credentials none Notes Supports Mermaid, Excalidraw, and all Kroki-supported formats."},{"location":"services/mcp-servers/#markitdown-markdown-conversion","title":"<code>markitdown</code> \u2014 Markdown conversion","text":"Value Kind catalog (<code>mcp/markitdown</code>) Internal upstream <code>http://mcp-markitdown:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/markitdown</code> Transport streamable-HTTP Resources 256 MiB mem limit Credentials none"},{"location":"services/mcp-servers/#sequential-thinking-chain-of-thought-reasoning","title":"<code>sequential-thinking</code> \u2014 Chain-of-thought reasoning","text":"Value Kind catalog (<code>mcp/sequentialthinking</code>) Internal upstream <code>http://mcp-sequential-thinking:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/sequentialthinking</code> Transport streamable-HTTP Resources 256 MiB mem limit Credentials none"},{"location":"services/mcp-servers/#time-time-timezone","title":"<code>time</code> \u2014 Time & timezone","text":"Value Kind spawn Internal upstream <code>http://mcp-time:8000</code> Image <code>ghcr.io/astral-sh/uv:python3.12-trixie-slim</code> Command <code>uvx --with mcp-server-time mcp-proxy --stateless --host 0.0.0.0 --port 8000 -- python -m mcp_server_time --local-timezone Europe/Berlin</code> Transport streamable-HTTP Credentials none"},{"location":"services/mcp-servers/#docling-pdf-markdown-saia","title":"<code>docling</code> \u2014 PDF \u2192 Markdown (SAIA)","text":"Value Kind static Internal upstream <code>http://docling-mcp:8000</code> (own stack, <code>:18005</code> on host) Transport streamable-HTTP Health check every 300 s Credentials none Notes SAIA-powered Docling; handles PDF, DOCX, images \u2192 Markdown."},{"location":"services/mcp-servers/#coder-coder-workspace-management","title":"<code>coder</code> \u2014 Coder workspace management","text":"Value Kind static Internal upstream <code>http://coder-mcp:8000</code> (own stack on <code>ai-internal</code>) Transport streamable-HTTP Health check every 300 s Credentials Coder token baked into <code>coder-mcp</code> container env (see <code>/opt/stacks/ai/coder-mcp/</code>) Notes Replaced daytona 2026-05-20. Can create/start/stop Coder workspaces on CT 111."},{"location":"services/mcp-servers/#gitea-gitea-gitnuclidesystems","title":"<code>gitea</code> \u2014 Gitea (git.nuclide.systems)","text":"Value Kind static Internal upstream <code>http://gitea-mcp:8000</code> (own stack <code>/opt/stacks/ai/gitea-mcp.yml</code>) Image <code>docker.gitea.com/gitea-mcp-server:latest</code> Command <code>/app/gitea-mcp -t http --port 8000 --host 0.0.0.0</code> Transport streamable-HTTP (native) Health check every 300 s Tools 53 (repo/issue/PR/file CRUD, releases, users, orgs) Credentials <code>GITEA_HOST</code> <code>https://git.nuclide.systems</code> <code>GITEA_ACCESS_TOKEN</code> <code>${GITEA_TOKEN}</code> \u2192 <code>.env</code> Notes Added 2026-05-23. Targets CT 111 Gitea. Read + write access."},{"location":"services/mcp-servers/#proxmox-proxmox-ve-192168120","title":"<code>proxmox</code> \u2014 Proxmox VE (192.168.1.20)","text":"Value Kind spawn Internal upstream <code>http://mcp-proxmox:8000</code> (own stack <code>/opt/stacks/ai/proxmox-mcp.yml</code>) Image <code>ghcr.io/astral-sh/uv:python3.12-trixie-slim</code> Command <code>uvx proxmox-mcp-plus</code> Package <code>proxmox-mcp-plus</code> (PyPI) Transport streamable-HTTP (<code>MCP_TRANSPORT=STREAMABLE_HTTP</code>) Resources 512 MiB mem limit Cache TTL 30 s Tools 39 (VMs, LXC, nodes, storage, tasks, firewall, HA \u2014 read-only) Credentials <code>PROXMOX_HOST</code> <code>192.168.1.20</code> <code>PROXMOX_PORT</code> <code>8006</code> <code>PROXMOX_USER</code> <code>root@pam</code> <code>PROXMOX_TOKEN_NAME</code> <code>mcp</code> <code>PROXMOX_TOKEN_VALUE</code> <code>${PROXMOX_TOKEN_SECRET}</code> \u2192 <code>.env</code> <code>PROXMOX_VERIFY_SSL</code> <code>false</code> <code>PROXMOX_DEV_MODE</code> <code>true</code> (required to allow self-signed cert with verify_ssl=false) Notes Added 2026-05-23. Token <code>root@pam!mcp</code> has <code>PVEAuditor</code> role at <code>/</code> (read-only, privsep). Self-signed TLS on PVE host requires <code>DEV_MODE=true</code>."},{"location":"services/mcp-servers/#shepard-shepard-product-platform","title":"<code>shepard</code> \u2014 Shepard product platform","text":"Value Kind static (exact upstream) Upstream <code>https://shepard.nuclide.systems/v2/mcp</code> Transport streamable-HTTP Health check disabled Auth header <code>Authorization: Bearer ${SHEPARD_API_KEY}</code> <code>SHEPARD_API_KEY</code> <code>eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI3ZWVhZDk0Mi02M2E1LTRjZmMtYWVhOS1iMWQwZjBhMjkxZWEiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvIiwibmJmIjoxNzc5MzcyODQyLCJpYXQiOjE3NzkzNzI4NDIsImp0aSI6ImY5YzAyYjc3LTNkZGYtNGZjZS1hYzJlLTBlYmM5N2FlYjJhZiJ9.Z9LY9vwLm2l0TmORQ2GriCnkLPzZnSC9q2sE62Ab8Gpi_374Gd5MffDkute0xF2ZwbhH3aRSCpEd93HmFDs-1F3IwoFQpBGiLedTL0N3gC_6J-PFv_i54FHFImEiH_h0yJilzrfgR4Prn_hFZMniE080Kf3Ll-uvYnNK-U-AkPjqb8KfP1BA6dt3HjiOjpicdSh2URpzAKxyVwddGeve3Ha9LboPu-dOb8nmiwiW6E75eakNGGR2mwPxITDxVaqxuSipBwBiDIHzfU4iT7T7b8aipvGd1upfGlx7RcbHdGg9gVNR4jH_--5qLb5124MQxZ486dlH6i7w6AMqtdkzGA</code> Notes Native MCP endpoint on CT 101 Shepard stack (Keycloak + backend)."},{"location":"services/mcp-servers/#group-research","title":"Group: <code>research</code>","text":""},{"location":"services/mcp-servers/#crawl4ai-web-crawling-scraping","title":"<code>crawl4ai</code> \u2014 Web crawling / scraping","text":"Value Kind spawn Internal upstream <code>http://mcp-crawl4ai:11235</code> Image <code>unclecode/crawl4ai:latest</code> Transport SSE (<code>/sse</code>) Resources 2 GiB mem limit; OOM score adj 300 Cache TTL 120 s Credentials none"},{"location":"services/mcp-servers/#paper-search-academic-paper-search","title":"<code>paper-search</code> \u2014 Academic paper search","text":"Value Kind catalog (<code>mcp/paper-search</code>) Internal upstream <code>http://mcp-paper-search:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/paper-search</code> Transport streamable-HTTP Resources 256 MiB mem limit Cache TTL 300 s Credentials <code>UNPAYWALL_EMAIL</code> <code>fkrebs@nucli.de</code> Notes <code>--pass-environment</code> passthrough. Optional: <code>CORE_API_KEY</code>, <code>DOAJ_API_KEY</code> (not currently set)."},{"location":"services/mcp-servers/#searxng-web-search-self-hosted","title":"<code>searxng</code> \u2014 Web search (self-hosted)","text":"Value Kind spawn Internal upstream <code>http://mcp-searxng:8000</code> Image <code>isokoliuk/mcp-searxng:latest</code> Transport streamable-HTTP Credentials <code>SEARXNG_URL</code> <code>http://searxng:8080</code> (internal <code>shared_backend</code> network) Notes Routes to the self-hosted SearXNG instance. No external API keys needed."},{"location":"services/mcp-servers/#wikipedia-mcp-wikipedia-search","title":"<code>wikipedia-mcp</code> \u2014 Wikipedia search","text":"Value Kind catalog (<code>mcp/wikipedia-mcp</code>) Internal upstream <code>http://mcp-wikipedia-mcp:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/wikipedia-mcp</code> Transport streamable-HTTP Resources 256 MiB mem limit Cache TTL 300 s Credentials none"},{"location":"services/mcp-servers/#youtube-transcript-youtube-transcripts","title":"<code>youtube-transcript</code> \u2014 YouTube transcripts","text":"Value Kind catalog (<code>mcp/youtube-transcript</code>) Internal upstream <code>http://mcp-youtube-transcript:8000</code> Image <code>mcp-catalog-bridge</code> \u2192 Docker MCP Catalog <code>mcp/youtube-transcript</code> Transport streamable-HTTP Resources 256 MiB mem limit Cache TTL 300 s Credentials none"},{"location":"services/mcp-servers/#group-personal","title":"Group: <code>personal</code>","text":""},{"location":"services/mcp-servers/#bluesky-bluesky-at-protocol","title":"<code>bluesky</code> \u2014 Bluesky / AT Protocol","text":"Value Kind static Internal upstream <code>http://ariel-mcp:8000</code> (own container on <code>ai-internal</code>) Transport streamable-HTTP Health check every 300 s Credentials (baked into <code>ariel-mcp</code> env) <code>ATPROTO_IDENTIFIER</code> <code>nucli.de</code> <code>ATPROTO_PASSWORD</code> <code>blxw-datl-l7zo-qwwq</code> <code>BLUESKY_IDENTIFIER</code> <code>nucli.de</code> <code>BLUESKY_APP_PASSWORD</code> <code>4mrm-go4j-avzg-2yo7</code> Notes Uses Brian Ellin's <code>ariel-mcp</code> image. Rate-limit handling pending."},{"location":"services/mcp-servers/#obsidian-obsidian-vault-nextcloud-backed","title":"<code>obsidian</code> \u2014 Obsidian vault (Nextcloud-backed)","text":"Value Kind spawn Image <code>mcp-obsidian-bridge</code> + mcpvault Internal upstream <code>http://mcp-obsidian:8000</code> Transport streamable-HTTP Cache TTL 30 s Vault path (container) <code>/vault</code> Vault path (CT 104 host) <code>/mnt/pve/unas/services/nextcloud/fkrebs@nucli.de/files/Notizen</code> Vault path (CT 105 Nextcloud) same \u2014 both CTs bind-mount UNAS via <code>/mnt/pve/unas</code> Vault path (UNAS NFS) <code>192.168.1.31:/var/nfs/shared/storage/services/nextcloud/fkrebs@nucli.de/files/Notizen</code> Nextcloud-visible path <code>fkrebs@nucli.de</code> user \u2192 <code>Notizen/</code> folder (sync target for Obsidian desktop/mobile) Credentials filesystem access only; no Nextcloud API needed Notes Vault is the canonical Obsidian store, written by Nextcloud sync from desktop/mobile clients and read/written by the MCP server. Backed up via Backrest media/services plans (UNAS coverage)."},{"location":"services/mcp-servers/#gotify-push-notifications","title":"<code>gotify</code> \u2014 Push notifications","text":"Value Kind spawn Internal upstream <code>http://mcp-gotify:8000</code> Image <code>kcofoni/gotify-mcp</code> Transport HTTP (<code>GOTIFY_MCP_TRANSPORT=http</code>) Cache TTL 0 (real-time) Credentials <code>GOTIFY_URL</code> <code>http://192.168.1.40:10003</code> <code>GOTIFY_CLIENT_TOKEN</code> <code>CwaslnnN-MTNRoC</code> <code>GOTIFY_APP_TOKEN</code> <code>AO84CFvU4XmoPBJ</code> Notes Can send and receive push notifications. App token = send; client token = receive."},{"location":"services/mcp-servers/#home-assistant-home-assistant","title":"<code>home-assistant</code> \u2014 Home Assistant","text":"Value Kind static (exact upstream) Upstream <code>http://192.168.1.60:9583/private_ehnWeRl2G3De6NnbcN7teQ</code> Transport streamable-HTTP Health check disabled Cache TTL 15 s Credentials Token embedded in path (<code>/private_ehnWeRl2G3De6NnbcN7teQ</code>) \u2014 HA MCP add-on authentication Notes Direct to HA add-on on VM 100. ~2492 entities. <code>exact_upstream: true</code> so the path is forwarded verbatim."},{"location":"services/mcp-servers/#immich-immich-photo-library","title":"<code>immich</code> \u2014 Immich photo library","text":"Value Kind spawn Internal upstream <code>http://mcp-immich:8000</code> Image <code>ghcr.io/astral-sh/uv:python3.12-trixie-slim</code> Command <code>uvx --with immich-mcp</code> \u2192 uvicorn streamable-HTTP app Transport streamable-HTTP Credentials <code>IMMICH_BASE_URL</code> <code>http://192.168.1.40:12000</code> <code>IMMICH_API_KEY</code> <code>jhtAOjyCU1cXxBoj6gZGLfCjpv8TmcwaXqiex6f5po</code> Notes DNS rebinding protection disabled (TransportSecuritySettings)."},{"location":"services/mcp-servers/#memos-memos-notes","title":"<code>memos</code> \u2014 Memos notes","text":"Value Kind spawn Internal upstream <code>http://mcp-memos:8000</code> Image <code>ghcr.io/astral-sh/uv:python3.12-trixie-slim</code> Command <code>uvx --with mcp-server-memos mcp-proxy --stateless ... -- mcp-server-memos --host 192.168.1.40 --port 17000 --token \"$MEMOS_TOKEN\"</code> Transport streamable-HTTP Cache TTL 0 (real-time) Credentials <code>MEMOS_TOKEN</code> <code>memos_pat_sOnvLytuaaVdEiqUSWubgfLut6AnoWb0</code> Health probe <code>search_memo</code> with keyword <code>__healthcheck__</code>"},{"location":"services/mcp-servers/#n8n-n8n-workflow-automation","title":"<code>n8n</code> \u2014 n8n workflow automation","text":"Value Kind static (exact upstream) Upstream <code>https://n8n.nuclide.systems/mcp-server/http</code> Transport streamable-HTTP Health check disabled TLS <code>insecure_tls: true</code> (self-signed cert workaround) Auth header <code>Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI4YWY1YzgxMy1mODljLTRiODMtYmNmOC01ZDU5ODY0YTAzOGEiLCJpc3MiOiJuOG4iLCJhdWQiOiJtY3Atc2VydmVyLWFwaSIsImp0aSI6IjBmNjA1NDBmLTBmZmQtNGEwNS1iNjZlLTA2ZjU2YzZlNjY2MCIsImlhdCI6MTc3OTQ1NjAzN30.hf_Vg45R_hOJSY11H4GVAuvqWv39SitvtIXBXnorlXQ</code> Notes n8n MCP server; exposes configured workflows as tools. JWT <code>iat: 1779456037</code>."},{"location":"services/mcp-servers/#nextcloud-nextcloud-files-calendar","title":"<code>nextcloud</code> \u2014 Nextcloud files & calendar","text":"Value Kind spawn Internal upstream <code>http://mcp-nextcloud:8000</code> Image <code>ghcr.io/cbcoutinho/nextcloud-mcp-server:latest</code> Transport streamable-HTTP Mode <code>MCP_DEPLOYMENT_MODE=single_user_basic</code> Health check disabled Credentials <code>NEXTCLOUD_HOST</code> <code>https://nc.nuclide.systems</code> <code>NEXTCLOUD_USERNAME</code> <code>fkrebs@nucli.de</code> <code>NEXTCLOUD_PASSWORD</code> <code>iRn4ECFf2dS24B9sFP3fkPosjrPVKZdaJBuwRpP9sVBnL6A7qN6lXNk2iK4yYblksc2ewzkb</code> <code>NEXTCLOUD_VERIFY_SSL</code> <code>false</code> Notes App-password generated via <code>occ</code> for <code>fkrebs@nucli.de</code> 2026-05-22. Re-enabled after fix."},{"location":"services/mcp-servers/#paperless-paperless-ngx","title":"<code>paperless</code> \u2014 Paperless-NGX","text":"Value Kind static Internal upstream <code>http://paperless-mcp:8000</code> (own stack <code>/opt/stacks/ai/paperless-mcp.yml</code>) Image custom build <code>./mcp-servers/paperless/Dockerfile</code> Base <code>ghcr.io/astral-sh/uv:bookworm-slim</code> + Node 20 + <code>@nloui/paperless-mcp</code> Transport streamable-HTTP (via mcp-proxy wrapper \u2014 package is stdio-only) Health check every 300 s Tools 12 (document search, retrieve, tags, correspondents, document types) Credentials <code>PAPERLESS_URL</code> <code>http://paperless-ngx-webserver-1:8000</code> (<code>shared_backend</code> network) <code>PAPERLESS_API_TOKEN</code> <code>${PAPERLESS_API_TOKEN}</code> \u2192 <code>.env</code> Notes Added 2026-05-23. npm package moved from <code>paperless-mcp</code> to <code>@nloui/paperless-mcp</code>. paperless-ngx webserver has a chown permissions error causing restarts \u2014 MCP server starts but tool calls will fail until paperless-ngx is fixed (see separate issue)."},{"location":"services/mcp-servers/#unifi-unifi-network","title":"<code>unifi</code> \u2014 UniFi Network","text":"Value Kind spawn Internal upstream <code>http://mcp-unifi:8000</code> Image <code>ghcr.io/enuno/unifi-mcp-server:latest</code> Transport HTTP (<code>MCP_SERVER_TRANSPORT=http</code>) Cache TTL 15 s Credentials <code>UNIFI_API_KEY</code> <code>yeBrsK8l6h5LSBc8ocsdLeSkGnc9bsKY</code> <code>UNIFI_API_TYPE</code> <code>local</code> <code>UNIFI_LOCAL_HOST</code> <code>192.168.1.1</code> <code>UNIFI_LOCAL_PORT</code> <code>443</code> <code>UNIFI_LOCAL_VERIFY_SSL</code> <code>false</code> Notes 197 tools. Network App API key (local API). Site Manager/cloud tools need <code>UNIFI_SITE_MANAGER_ENABLED</code>."},{"location":"services/mcp-servers/#group-image","title":"Group: <code>image</code>","text":""},{"location":"services/mcp-servers/#comfyui-comfyui-image-generation","title":"<code>comfyui</code> \u2014 ComfyUI image generation","text":"Value Kind static Internal upstream <code>http://comfyui-mcp:8000</code> (own stack, <code>:18003</code> on host) Transport streamable-HTTP Health check every 300 s Credentials none (auth via gateway Bearer) Notes Intel Arc iGPU; FLUX.1-schnell GGUF. LobeChat can also connect directly to <code>http://comfyui-mcp:8000/mcp</code> on <code>ai-internal</code>. See <code>comfyui.md</code>."},{"location":"services/mcp-servers/#group-storage","title":"Group: <code>storage</code>","text":""},{"location":"services/mcp-servers/#upload-artifact-s3-artifact-upload","title":"<code>upload-artifact</code> \u2014 S3 artifact upload","text":"Value Kind static Internal upstream <code>http://upload-artifact-mcp:8000</code> (own stack, <code>:18011</code> on host) Transport streamable-HTTP Health check every 300 s Credentials Garage S3 credentials baked into <code>upload-artifact-mcp</code> container env Notes Uploads chat artifacts (images, files) to Garage S3 at <code>s3.nuclide.systems</code>."},{"location":"services/mcp-servers/#client-configuration","title":"Client configuration","text":""},{"location":"services/mcp-servers/#auto-recommended","title":"Auto (recommended)","text":"<p>Note: URLs below use the new Bifrost endpoint. Old <code>mcp.nuclide.systems</code> routes are decommissioned.</p> <pre><code>GET https://ai.nuclide.systems/mcp-config?format=claude # Claude Code / Claude Desktop\nGET https://ai.nuclide.systems/mcp-config?format=cursor # Cursor\nGET https://ai.nuclide.systems/mcp-config?format=raw # raw token + URLs\n</code></pre> <p>Auth: <code>Authorization: Bearer sk-bf-<key></code>. Merge <code>mcpServers</code> block into <code>~/.claude/settings.json</code>.</p>"},{"location":"services/mcp-servers/#manual-claude-code","title":"Manual (Claude Code)","text":"<pre><code>{\n \"mcpServers\": {\n \"<name>\": {\n \"type\": \"http\",\n \"url\": \"https://ai.nuclide.systems/mcp/<name>/mcp\",\n \"headers\": {\n \"Authorization\": \"Bearer sk-bf-<key>\"\n }\n }\n }\n}\n</code></pre>"},{"location":"services/mcp-servers/#gateway-management-api","title":"Gateway management API","text":"<p>Bifrost management API (auth: <code>sk-bf-<key></code>).</p> <pre><code>BASE=https://ai.nuclide.systems\n# List all servers + status\ncurl -H \"Authorization: Bearer sk-bf-...\" $BASE/api/servers\n\n# Health check\ncurl $BASE/health\n</code></pre>"},{"location":"services/mcp-servers/#other-credentials-in-gateway-env","title":"Other credentials in gateway <code>.env</code>","text":"<p>These are available to spawned containers via <code>${VAR}</code> substitution in <code>config.json</code>:</p> Variable Value Used by <code>GATEWAY_API_KEYS</code> <code>JxxCqKw32XoN4LOHunDikS6u1RpS7R5ythzaqADPuIA</code> All API/proxy calls to gateway <code>LITELLM_MASTER_KEY</code> <code>sk-tapirnase</code> LiteLLM proxy (shared pw \u2014 rotate pending) <code>LITELLM_API_KEY</code> <code>sk-tapirnase</code> same <code>GITEA_TOKEN</code> <code>c15e3348bf725054131633061002d677685cd4f4</code> Gitea API access <code>NEXTCLOUD_MCP_OAUTH_CLIENT_ID</code> <code>82ca2d53-6df4-4671-875c-fee85b54b76f</code> Pocket-ID client for MCP spawned servers <code>NEXTCLOUD_MCP_OAUTH_CLIENT_SECRET</code> <code>zOfHItEeKVa6Tf40RpnhihFmgnCDkkfz</code> same <code>SHEPARD_BASE_URL</code> <code>https://shepard-api.nuclide.systems/shepard/api</code> Shepard API <code>GOTIFY_TOKEN_ALERTS</code> <code>AtjFdduWArgAe7q</code> Gotify \u2014 alert app token (gateway internal) <code>GOTIFY_TOKEN_AGENTS</code> <code>AB3dzfj5TUPCfYf</code> Gotify \u2014 agent app token (gateway internal)"},{"location":"services/nexa/","title":"Nexa \u2014 Neural Nexus for Information & Automation","text":"<p>Repo: <code>https://git.nuclide.systems/fkrebs/nexa</code> Stack location: CT104 <code>/opt/stacks/nexa/</code> Status: Designed, partially implemented \u2014 Phase 1 workflows exist; not yet deployed end-to-end.</p>"},{"location":"services/nexa/#synopsis","title":"Synopsis","text":"<p>Nexa is a personal AI middleware layer that sits between input sources and organisation tools. It is not a new app \u2014 it is a set of wired-together workflows running on top of services already deployed in nuclide.systems.</p> <p>Mental model: - Memos = mouth and ear (voice interface, reply surface) - n8n = reflexes (workflow logic, classification, routing) - LiteLLM / SAIA = brain (language model gateway) - Qdrant = long-term memory (semantic recall) - Nextcloud = hands (tasks, calendar, files, mail)</p> <p>The user types (or speaks) into Memos. A Memos webhook fires an n8n workflow. n8n classifies the input (Work vs. Personal, command vs. capture), calls LiteLLM for any reasoning, stores embeddings in Qdrant, and writes back a comment on the original memo. Side effects \u2014 task creation, calendar blocks, archive entries \u2014 go to Nextcloud.</p>"},{"location":"services/nexa/#what-nexa-is-not","title":"What Nexa is not","text":"<ul> <li>A replacement for specialist UIs (Arcane, Dozzle, Grafana).</li> <li>An active document editor.</li> <li>A work-IT bridge (work mail/servers remain out of scope).</li> </ul>"},{"location":"services/nexa/#core-goals","title":"Core goals","text":"<ol> <li>Cognitive offload \u2014 sort, filter, propose; don't just store.</li> <li>Context separation \u2014 clean Work vs. Personal split enforced by LLM classification.</li> <li>Single interaction point \u2014 Memos is the only interface the user must open.</li> <li>Knowledge synergy \u2014 link ephemeral memos to deep Obsidian notes via semantic search.</li> </ol>"},{"location":"services/nexa/#how-nexa-maps-to-nuclidesystems","title":"How Nexa maps to nuclide.systems","text":"<p>Every component Nexa depends on is already deployed. Nothing new needs to be provisioned for Phase 1\u20132.</p> Nexa concept nuclide.systems service Host URL/port Interface / voice Memos CT104 <code>https://memos.nuclide.systems</code> Workflow engine n8n CT104 <code>http://192.168.1.40:15678</code> LLM gateway (SAIA) LiteLLM CT104 <code>https://ai.nuclide.systems</code> (internal :4000) Vector memory Qdrant (<code>qdrant_scientific</code>) CT104 internal :6333 File / task / calendar Nextcloud CT105 <code>https://nc.nuclide.systems</code> Link curation Karakeep CT104 <code>https://hoarder.nuclide.systems</code> Push alerts ntfy CT104 <code>https://ntfy.nuclide.systems</code> Note vault Obsidian (via Nextcloud WebDAV) CT105 <code>nc.nuclide.systems/Notizen/</code> Auth / SSO Pocket-ID CT110 <code>https://id.nuclide.systems</code> Reverse proxy Zoraxy CT108 all <code>*.nuclide.systems</code> Push notifications (alerts) Gotify CT104 internal :10003 Social feed Bluesky external API only Mail Nextcloud Mail (<code>fkrebs@nucli.de</code>) CT105 IMAP via NC Web fetch (Phase 2.4) crawl4ai MCP CT104 via MCP gateway Search (Phase 2.4) SearXNG (if deployed) CT104 internal"},{"location":"services/nexa/#what-is-genuinely-missing","title":"What is genuinely missing","text":"Missing component Phase needed Notes TEI (text-embeddings-inference) Phase 3.1 Self-hosted embeddings for Qdrant ingest. <code>bge-m3</code> model, CPU-only, ~1.1 GB RAM. Deploy as <code>nexa-embed</code> container on CT104. Ontotext GraphDB Phase 3.4 SPARQL structural memory. Deferred until Phase 3.1\u20133.3 ship. Needs ~4 GB heap on CT104. <code>nexa_knowledge_text</code> Qdrant collection Phase 3.1 One <code>curl -X PUT</code> against the existing <code>qdrant_scientific</code> instance. n8n workflow import Phase 1 JSON exports are in <code>nexa-core/n8n-workflows/</code>. Import via n8n API or UI. Memos \u2192 n8n webhook Phase 1 One URL field in Memos admin: <code>https://n8n.nuclide.systems/webhook/memos</code>. LiteLLM virtual key for Nexa Phase 1 Create <code>nexa</code> user in LiteLLM admin, issue key scoped to one chat model. SearXNG (optional) Phase 2.4 Web-search for <code>#nexa:ask --web</code>. Not deployed yet. infinity (Phase 3.2 upgrade) Phase 3.2 Replaces TEI to add CLIP-family visual embeddings (<code>jina-clip-v2</code>). <code>nexa_knowledge_visual</code> collection Phase 3.2 Second Qdrant collection for image embeddings. Schema already defined in repo."},{"location":"services/nexa/#phased-roadmap-mapped-to-infrastructure","title":"Phased roadmap (mapped to infrastructure)","text":""},{"location":"services/nexa/#phase-1-the-spine-no-new-containers","title":"Phase 1 \u2014 The Spine (no new containers)","text":"<p>Wire existing services together. All components are already running.</p> <ol> <li>Import <code>nexa-core/n8n-workflows/phase-1/</code> into n8n at <code>http://192.168.1.40:15678</code>.</li> <li>Set Memos webhook URL \u2192 <code>https://n8n.nuclide.systems/webhook/memos</code>.</li> <li>Create LiteLLM virtual key for <code>nexa</code> user (chat model only \u2014 no embeddings yet).</li> <li>Fill <code>nexa-core/.env</code> with <code>MEMOS_API_KEY</code>, <code>SAIA_API_KEY</code>, <code>NC_APP_PASSWORD</code>, <code>QDRANT_API_KEY</code>.</li> <li>Import <code>phase-2/2_1_email_butler.json</code> and attach Nextcloud Mail credentials in n8n.</li> <li>Fire <code>#nexa:config</code> in Memos \u2192 confirms Nextcloud lists, calendar IDs, mail folder structure.</li> </ol> <p>Milestone: Memos comment-back works. Work vs. Personal classification active. Email butler running.</p>"},{"location":"services/nexa/#phase-2-senses-searxng-optional","title":"Phase 2 \u2014 Senses (SearXNG optional)","text":"<ul> <li>2.1 Email butler \u2014 already has an n8n workflow JSON. Needs credentials in n8n.</li> <li>2.2 RSS digest \u2014 new n8n workflow polling feeds \u2192 daily summary memo.</li> <li>2.3 Bluesky antenna \u2014 liked posts \u2192 <code>nexa_knowledge_text</code> entries once Qdrant collection exists.</li> <li>2.4 Web fetch \u2014 <code>#nexa:ask --web</code>: SearXNG + <code>crawl4ai</code> MCP (already deployed via MCP gateway) + markitdown.</li> <li>SearXNG not yet deployed on CT104. Deploy if 2.4 is needed.</li> </ul> <p>Milestone: Nexa reads the web, Bluesky, and daily RSS. Morning digest appears in Memos.</p>"},{"location":"services/nexa/#phase-3-memory-two-new-containers-rest-is-curl-commands","title":"Phase 3 \u2014 Memory (two new containers; rest is curl commands)","text":"<ul> <li>3.1 TEI + Qdrant collection \u2014 deploy <code>nexa-embed</code> container (CT104), create <code>nexa_knowledge_text</code>, wire TEI into LiteLLM as <code>nexa-embed</code> model, add Qdrant ingest step to n8n workflows.</li> <li>Storage: <code>/mnt/pve/unas/services/nexa/tei-cache</code> (NFS already mounted on CT104).</li> <li>3.2 Visual collection \u2014 swap TEI \u2192 <code>infinity</code>, add <code>jina-clip-v2</code>, create <code>nexa_knowledge_visual</code>. Backfill image queue from GraphDB.</li> <li>3.3 S3 archive \u2014 Qdrant snapshots to <code>s3.nuclide.systems</code> (Garage). Blocked on Q12/Q20 decision.</li> <li>3.4 Ontotext GraphDB \u2014 SPARQL pillar. One new container on CT104, <code>GDB_JAVA_OPTS: -Xmx4g</code>. SPARQL Workbench optionally via Zoraxy \u2192 <code>graph.nuclide.systems</code>.</li> </ul> <p>Milestone: <code>#nexa:ask</code> returns answers grounded in Obsidian notes, archived memos, mail threads.</p>"},{"location":"services/nexa/#phase-4-motor-no-new-infrastructure","title":"Phase 4 \u2014 Motor (no new infrastructure)","text":"<ul> <li>Bidirectional NC Tasks sync (Memos checkboxes \u2194 Nextcloud Tasks lists).</li> <li>Time-boxing agent: calendar-aware gap-finder in NC Calendar \u2192 proposes focus slots.</li> <li>Karakeep curation: scheduled review workflow flags stale saved links.</li> </ul> <p>Milestone: Memos <code>- [ ]</code> items automatically appear in the right NC Tasks list.</p>"},{"location":"services/nexa/#phase-5-daily-integration-ha-voice-only-new-piece","title":"Phase 5 \u2014 Daily Integration (HA Voice only new piece)","text":"<ul> <li>5.1 HA Voice \u2014 Wyoming protocol bridge from HAOS (VM 100) to Nexa n8n endpoint. Needs Wyoming server container (small) or HA add-on.</li> <li>5.2 System monitoring \u2014 Proxmox + Backrest alerts \u2192 ntfy \u2192 Memos system feed. Homelab watchdog already posts to Gotify; route a copy to Memos via n8n.</li> <li>5.3 Morning digest \u2014 daily 07:00 n8n cron: mail summary + tasks + drift report + one open question from <code>docs/11</code>.</li> </ul> <p>Milestone: Nexa speaks. Morning context summary lands in Memos without user action.</p>"},{"location":"services/nexa/#phase-6-homelab-steward-uses-existing-arcane-proxmox-apis","title":"Phase 6 \u2014 Homelab Steward (uses existing Arcane + Proxmox APIs)","text":"<ul> <li>6.1 Inventory sync \u2014 n8n polls Arcane API + Proxmox API + <code>docker ps</code> on a schedule \u2192 snapshot stored in GraphDB.</li> <li>6.2 Drift detection \u2014 diff snapshot against documented desired state in <code>docs/</code> \u2192 Memos <code>[STEWARD]</code> comment.</li> <li>6.3 Steward commands \u2014 <code>#nexa:retire <svc></code>, <code>#nexa:document <svc></code>, <code>#nexa:wishlist-status</code>.</li> <li>6.4 Docs self-update \u2014 Nexa proposes <code>docs/11</code> diffs as PRs when the user answers open questions via Memos.</li> </ul> <p>Milestone: Nexa replaces the manual \"what's stale?\" audit. The homelab docs update themselves.</p>"},{"location":"services/nexa/#tool-realization-whats-not-deployed-yet-and-the-options","title":"Tool realization: what's not deployed yet and the options","text":"<p>The docs are designed around a specific tool set, but several pieces have alternatives worth considering given the current nuclide.systems stack:</p>"},{"location":"services/nexa/#embeddings-phase-31","title":"Embeddings (Phase 3.1)","text":"<p>The plan calls for TEI + <code>bge-m3</code>. Alternatives:</p> Option Pros Cons TEI + bge-m3 (plan) Lightest (~500 MB image, ~1.1 GB RAM), OpenAI-compatible, single model, no LLM runtime CPU-only (fine for the workload) Ollama (already on CT104?) Already deployed if running Heavier image, slower cold start, designed for chat not embedding throughput LiteLLM pass-through to Claude No new container 10 req/min rate limit \u2014 unusable for Qdrant ingest (2k Obsidian notes = 3 h) infinity (skip straight to Phase 3.2) Supports both text and visual models simultaneously Slightly more complex setup; Phase 3.2 is not urgent <p>Lean: deploy TEI now, swap to infinity when Phase 3.2 visual collection is needed.</p>"},{"location":"services/nexa/#web-search-phase-24","title":"Web search (Phase 2.4)","text":"<p>The plan calls for SearXNG (not yet deployed):</p> Option Pros Cons SearXNG (plan) Self-hosted, no API key, privacy-preserving New container to maintain Exa MCP (already in MCP gateway) Already wired, no new container Paid/rate-limited external service crawl4ai alone Already deployed (Phase 2.4 fetch step) No search, only direct-URL fetch Brave Search API Simple, fast API key + cost <p>Lean: use Exa MCP for Phase 2.4 (already available in the gateway), deploy SearXNG only if privacy or rate limits become a concern.</p>"},{"location":"services/nexa/#graphdb-phase-34","title":"GraphDB (Phase 3.4)","text":"<p>The plan calls for Ontotext GraphDB:</p> Option Pros Cons Ontotext GraphDB (plan) Full SPARQL 1.1, production-grade, free Community Edition ~4 GB heap; heavyweight for a homelab Apache Jena Fuseki Lighter, Apache licensed, same SPARQL interface Less tooling, fewer connectors Oxigraph Tiny Rust binary (~50 MB), OpenAPI + SPARQL Newer, smaller community Skip GraphDB entirely Qdrant alone covers 80% of the Phase 3 value Phase 6 steward commands lose structural query capability <p>Lean: defer until Phase 3.1\u20133.3 are running. Then re-evaluate Oxigraph vs. GraphDB based on RAM budget at that time.</p>"},{"location":"services/nexa/#open-items","title":"Open items","text":"<ul> <li>[ ] Import Phase 1 n8n workflows (manual step \u2014 credentials required in n8n UI).</li> <li>[ ] Wire Memos webhook \u2192 n8n.</li> <li>[ ] Create LiteLLM <code>nexa</code> virtual key.</li> <li>[ ] Deploy TEI container on CT104 (Phase 3.1 prerequisite).</li> <li>[ ] Create <code>nexa_knowledge_text</code> Qdrant collection.</li> <li>[ ] Decide SearXNG vs. Exa MCP for Phase 2.4.</li> </ul> <p>See <code>docs/11-open-questions.md</code> in the Nexa repo for all design decisions and their resolution status.</p>"},{"location":"services/pocket-id/","title":"Pocket-ID \u2014 OIDC Identity Provider","text":"<p>CT 109 \"ops\" \u00b7 <code>192.168.1.8:11000</code> \u00b7 <code>https://id.nuclide.systems</code> Migrated CT 104 \u2192 CT 110 on 2026-05-20; CT 110 \u2192 CT 109 on 2026-05-26. SQLite-only (no Postgres). Compose at <code>/opt/stacks/pocketid/</code> on CT 109.</p>"},{"location":"services/pocket-id/#what-it-does","title":"What it does","text":"<p>Pocket-ID is a lightweight OIDC 2.1 / OAuth 2.0 IdP. Every service that supports OIDC can delegate login to it \u2014 one account, one MFA setup, SSO across the homelab. Clients are managed via an admin UI; there is no API key / scripted client creation.</p>"},{"location":"services/pocket-id/#admin-access","title":"Admin access","text":"<ul> <li>URL: <code>https://id.nuclide.systems</code> (admin panel is the default view when logged in as admin)</li> <li>Admin user: <code>fkrebs@nucli.de</code></li> <li>API key: <code>1ed1d53a2c1b3fcbafea46863b0b9e88a649422b67541c764481c03918b60bc6</code> \u2014 header <code>X-API-Key</code> (not Bearer); stored as SHA-256 hash in SQLite <code>api_keys</code> table on CT 105</li> </ul>"},{"location":"services/pocket-id/#oidc-endpoints-standard-discovery","title":"OIDC endpoints (standard discovery)","text":"Endpoint URL Discovery <code>https://id.nuclide.systems/.well-known/openid-configuration</code> Authorization <code>https://id.nuclide.systems/authorize</code> Token <code>https://id.nuclide.systems/api/oidc/token</code> Userinfo <code>https://id.nuclide.systems/api/oidc/userinfo</code> JWKS <code>https://id.nuclide.systems/api/oidc/jwks</code>"},{"location":"services/pocket-id/#creating-a-new-oidc-client","title":"Creating a new OIDC client","text":"<ol> <li>Go to <code>https://id.nuclide.systems</code> \u2192 OIDC Clients \u2192 New Client</li> <li>Fill in:</li> <li>Name: descriptive (e.g. \"Homarr\", \"Grafana\")</li> <li>Redirect URIs: the callback URL the service expects (see per-service table below)</li> <li>PKCE: enable if the service supports it (preferred)</li> <li>Copy the Client ID and Client Secret \u2014 secret is shown once.</li> <li>Paste into the service's env vars (see patterns below).</li> </ol> <p>Pocket-ID 2.7.0+ stores secrets as bcrypt hashes \u2014 the plaintext secret is only visible at creation time. If lost, regenerate in the client edit view.</p>"},{"location":"services/pocket-id/#current-clients","title":"Current clients","text":"Service CT Client ID Redirect URI Notes Gitea 104 <code>9444609e-6151-4296-aaeb-576da886c887</code> <code>https://git.nuclide.systems/user/oauth2/pocket-id/callback</code> SSO active; local password sign-in disabled Coder 104 <code>0aee4280-da5e-4782-a790-c7565c6c1366</code> <code>https://dev.nuclide.systems/api/v2/users/oidc/callback</code> SSO active; password auth disabled n8n 104 <code>33135ad4-a3ed-45d3-938f-639abd2b9663</code> <code>https://n8n.nuclide.systems/rest/oauth2-credential/callback</code> encryption key rotated 2026-05-22 Vaultwarden 104 <code>7cda8d60-9bfa-44ca-ae9b-8ae436024a8b</code> <code>https://vault.nuclide.systems/identity/connect/token</code> created 2026-05-21; auth flow not yet wired ~~Homarr~~ ~~109~~ ~~<code>63a94e30-7bbf-4511-9a4c-82d992633427</code>~~ \u2014 DECOMMISSIONED \u2014 replaced by Homepage Grafana 109 <code>92d987d5-d066-4e19-8fa1-960114c1244c</code> <code>http://192.168.1.8:3000/login/generic_oauth</code> SSO active 2026-05-23; PKCE enabled; LAN alias added 2026-05-24 Infisical 109 <code>b2069075-ede2-4251-ad1f-9a62e6a188b3</code> <code>http://192.168.1.8:8200/api/v1/sso/oidc/callback</code> migrated CT112\u2192CT109 2026-05-26; manual OIDC config still pending Proxmox VE host <code>38469e7e-1fff-4841-83a9-74bf38d847eb</code> <code>https://192.168.1.20:8006</code> LAN alias added 2026-05-24 Nextcloud 105 <code>a14b8076-985c-4989-95f8-e0283bfbdf32</code> <code>https://nc.nuclide.systems/apps/oidc_login/oidc</code> Immich 104 <code>9c91c18b-e009-4371-9c54-b71d54e3c77a</code> <code>https://photos.nuclide.systems/auth/login</code> Karakeep 104 <code>d92f82b0-b876-48c2-b3b0-05dd35fdf908</code> <code>https://bookmarks.nuclide.systems/api/auth/callback/custom-server</code> Audiobookshelf 104 <code>cbbf20d5-d15c-419c-8f18-82d2fd7e810f</code> <code>https://abs.nuclide.systems/auth/openid/callback</code> Shelfarr 104 <code>d8733fcc-eee8-42c4-b976-cb14e4e87693</code> <code>https://shelfarr.nuclide.systems/auth/callback</code> Memos 104 <code>62bf4e0d-f0fe-4453-b0da-59eeea2bb69d</code> <code>https://memos.nuclide.systems/auth/callback</code> Open WebUI 104 <code>e41534ae-994a-4188-8d42-31690c354284</code> <code>https://chat.nuclide.systems/oauth/oidc/callback</code> Portainer 109 <code>bdf8b019-072e-4c21-b1fb-ad9c5ad392dc</code> <code>http://192.168.1.8:9000/</code> configured 2026-05-26; secret <code>DZu1JrzEpChU3Deeh0ycaV29s5aBQLGR</code> Bifrost MCP 104 <code>ec0d15e6-e86d-49b0-ac12-cdfb5afc9086</code> <code>https://ai.nuclide.systems</code> per_user_oauth (not currently active \u2014 all MCP clients use auth_type=none) mcp-auth 104 <code>af2f837b-8a77-4f6f-80b0-71b734eb7b0b</code> \u2014 internal; no launch URL nuc-ai 104 <code>82ca2d53-6df4-4671-875c-fee85b54b76f</code> \u2014 internal; no launch URL"},{"location":"services/pocket-id/#env-var-patterns-per-service-type","title":"Env var patterns per service type","text":""},{"location":"services/pocket-id/#homarr-v1-homarr-labs","title":"Homarr (v1 homarr-labs)","text":"<pre><code>AUTH_PROVIDERS=credentials,oidc # plural; comma-separated list. Singular AUTH_PROVIDER is ignored.\nAUTH_OIDC_ISSUER=https://id.nuclide.systems\nAUTH_OIDC_CLIENT_ID=<client-id>\nAUTH_OIDC_CLIENT_SECRET=<client-secret>\nAUTH_OIDC_SCOPE=openid profile email\nAUTH_OIDC_NAME=Pocket-ID\n</code></pre>"},{"location":"services/pocket-id/#grafana","title":"Grafana","text":"<pre><code>GF_AUTH_GENERIC_OAUTH_ENABLED=true\nGF_AUTH_GENERIC_OAUTH_NAME=Pocket-ID\nGF_AUTH_GENERIC_OAUTH_CLIENT_ID=<client-id>\nGF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<client-secret>\nGF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email\nGF_AUTH_GENERIC_OAUTH_AUTH_URL=https://id.nuclide.systems/authorize\nGF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://id.nuclide.systems/api/oidc/token\nGF_AUTH_GENERIC_OAUTH_API_URL=https://id.nuclide.systems/api/oidc/userinfo\nGF_AUTH_SIGNOUT_REDIRECT_URL=https://id.nuclide.systems/logout\nGF_AUTH_GENERIC_OAUTH_USE_PKCE=true\nGF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false\nGF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(groups[*], 'admins') && 'Admin' || 'Viewer'\n</code></pre>"},{"location":"services/pocket-id/#gitea","title":"Gitea","text":"<p><pre><code>[oauth2]\nENABLED = true\n\n[service]\nENABLE_PASSWORD_SIGNIN_FORM = false\n</code></pre> Auth source added via Gitea admin UI \u2192 Authentication Sources \u2192 OAuth2 \u2192 OpenID Connect.</p>"},{"location":"services/pocket-id/#coder","title":"Coder","text":"<pre><code>CODER_OIDC_ISSUER_URL=https://id.nuclide.systems\nCODER_OIDC_CLIENT_ID=<client-id>\nCODER_OIDC_CLIENT_SECRET=<client-secret>\nCODER_OIDC_SCOPES=openid,profile,email\nCODER_DISABLE_PASSWORD_AUTH=true\n</code></pre>"},{"location":"services/pocket-id/#generic-any-service-with-standard-oidc","title":"Generic (any service with standard OIDC)","text":"<pre><code>Issuer: https://id.nuclide.systems\nAuth URL: https://id.nuclide.systems/authorize\nToken URL: https://id.nuclide.systems/api/oidc/token\nUserinfo URL: https://id.nuclide.systems/api/oidc/userinfo\nJWKS URL: https://id.nuclide.systems/api/oidc/jwks\nScopes: openid profile email\n</code></pre>"},{"location":"services/pocket-id/#backup","title":"Backup","text":"<p>Pocket-ID SQLite DB + signing keys are backed up nightly via Backrest (CT 103). Pre-backup hook SSHs to CT 110, runs <code>pocket-id export</code> inside the container, copies the ZIP + signing keys to UNAS staging path, then the <code>services-backup-plan</code> snapshots to JottaCloud. See <code>services/backrest.md</code>.</p>"},{"location":"services/pocket-id/#stack-location","title":"Stack location","text":"<p>CT 110 (<code>192.168.1.5</code>): <code>/opt/stacks/pocketid/docker-compose.yml</code></p> <pre><code># key env vars\nPOCKET_ID_URL=https://id.nuclide.systems\nTRUST_PROXY=true\nMAXMIND_LICENSE_KEY=... # optional geo-IP\n</code></pre>"},{"location":"services/portainer/","title":"Portainer BE (migrated from Arcane 2026-05-26)","text":"<p>Docker management UI with Portainer Business Edition license.</p>"},{"location":"services/portainer/#stack","title":"Stack","text":"<ul> <li>Server: CT 109 (\"ops\", 192.168.1.8), <code>/opt/stacks/monitoring/docker-compose.yml</code>, port 9000</li> <li>Image: <code>portainer/portainer-ee:2.39.2</code></li> <li>Data volume: <code>monitoring_portainer_data</code> \u2192 <code>/var/lib/docker/volumes/monitoring_portainer_data/_data</code></li> <li>URL: <code>http://192.168.1.8:9000</code> (LAN-only; no Zoraxy route \u2014 access via LAN/SSH tunnel)</li> <li>Admin user: <code>fkrebs</code> / <code>tapirnase</code></li> </ul>"},{"location":"services/portainer/#license","title":"License","text":"<ul> <li>Edition: Business Edition (BE) \u2014 3 nodes free for personal use</li> <li>License key: <code>3-QIORiAXMuBSgeYdePYgh1nuqRkvx/XWyu5D/+MQlVpvSng2CXtCG4V78212HEleOIWnIV0kK5IkpEaifea3b8NGU6o2STA0c/XXj150c/v5XSguhchCmqiXyWXDj2/+r</code></li> <li>Expires: 2027-05-25 (365 days from issue date 2026-05-26)</li> <li>Nodes: 3 licensed (CT 104, CT 109 local, + 1 more)</li> </ul> <p>To re-apply license (e.g. after fresh install): <pre><code>curl -X POST http://192.168.1.8:9000/api/auth \\\n -H \"Content-Type: application/json\" \\\n -d '{\"username\":\"fkrebs\",\"password\":\"tapirnase\"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"jwt\"])'\n# then:\ncurl -X POST http://192.168.1.8:9000/api/licenses/add \\\n -H \"Authorization: Bearer <jwt>\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"license\":\"<key>\"}'\n</code></pre></p>"},{"location":"services/portainer/#agents-portainer-environments","title":"Agents (Portainer environments)","text":"Host LXC Name IP Port Stack CT 109 ops unix:///var/run/docker.sock \u2014 built-in CT 104 docker 192.168.1.40 9001 <code>/opt/stacks/portainer-agent.yml</code> ~~CT 110~~ ~~id~~ ~~192.168.1.5~~ \u2014 DESTROYED 2026-05-26 \u2014 Pocket-ID moved to CT 109 CT 111 dev 192.168.1.42 9001 <code>/opt/stacks/ops-agents/docker-compose.yml</code> CT 112 secrets 192.168.1.7 9001 <code>/opt/stacks/ops-agents/docker-compose.yml</code> CT 113 db 192.168.1.6 9001 <code>/opt/stacks/db/docker-compose.yml</code> <p>Add each as a Portainer Agent environment: <code>http://<ip>:9001</code>.</p>"},{"location":"services/portainer/#oidc-pocket-id","title":"OIDC (Pocket-ID)","text":"<p>Status: pending \u2014 requires manual OIDC client creation in Pocket-ID web UI first.</p> <ol> <li>Go to <code>https://id.nuclide.systems/settings/admin/oidc-clients</code></li> <li>Create client:</li> <li>Name: <code>portainer</code></li> <li>Redirect URIs: <code>http://192.168.1.8:9000/</code></li> <li>Note the Client ID and Client Secret</li> <li>Configure in Portainer \u2192 Settings \u2192 Authentication \u2192 OAuth 2.0:</li> <li>Authorization URL: <code>https://id.nuclide.systems/authorize</code></li> <li>Access Token URL: <code>https://id.nuclide.systems/api/oidc/token</code></li> <li>Resource URL: <code>https://id.nuclide.systems/api/oidc/userinfo</code></li> <li>Redirect URL: <code>http://192.168.1.8:9000/</code></li> <li>Client ID / Secret from step 2</li> <li>User Identifier: <code>email</code></li> <li>Scopes: <code>openid profile email</code></li> <li>Logout URL: <code>https://id.nuclide.systems/logout</code></li> <li>Enable \"Automatic user provisioning\"</li> </ol> <p>Or via API once client credentials are known: <pre><code>JWT=$(curl -s -X POST http://192.168.1.8:9000/api/auth \\\n -H \"Content-Type: application/json\" \\\n -d '{\"username\":\"fkrebs\",\"password\":\"tapirnase\"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"jwt\"])')\n\ncurl -X PUT http://192.168.1.8:9000/api/settings \\\n -H \"Authorization: Bearer $JWT\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"AuthenticationMethod\": 3,\n \"OAuthSettings\": {\n \"ClientID\": \"<client-id>\",\n \"ClientSecret\": \"<client-secret>\",\n \"AuthorizationURI\": \"https://id.nuclide.systems/authorize\",\n \"AccessTokenURI\": \"https://id.nuclide.systems/api/oidc/token\",\n \"ResourceURI\": \"https://id.nuclide.systems/api/oidc/userinfo\",\n \"RedirectURI\": \"http://192.168.1.8:9000/\",\n \"LogoutURI\": \"https://id.nuclide.systems/logout\",\n \"UserIdentifier\": \"email\",\n \"Scopes\": \"openid profile email\",\n \"OAuthAutoCreateUsers\": true,\n \"SSO\": true\n }\n }'\n</code></pre></p>"},{"location":"services/portainer/#backup","title":"Backup","text":"<ul> <li>Local S3: daily at 01:15 UTC via <code>portainer-backup.timer</code> on CT 109</li> <li>Script: <code>/usr/local/sbin/portainer-backup.py</code></li> <li>Bucket: <code>ct109-portainer-backup</code> on Garage (CT 104:10004)</li> <li>Key ID: <code>GKd4511c4a01155ebbc37aa7ff</code></li> <li>Retention: 7 daily tarballs</li> <li>Offsite: CT 103 <code>walg-offsite-sync.sh</code> syncs <code>ct109-portainer-backup</code> \u2192 <code>jottacloud:WAL-G/ct109-portainer-backup/</code> daily at 02:30</li> </ul> <p>Restore: <pre><code># Download latest from Garage\naws --endpoint-url http://192.168.1.40:10004 s3 ls s3://ct109-portainer-backup/\naws --endpoint-url http://192.168.1.40:10004 s3 cp s3://ct109-portainer-backup/portainer-YYYY-MM-DD.tar.gz .\ntar xzf portainer-YYYY-MM-DD.tar.gz\n# Replace /var/lib/docker/volumes/monitoring_portainer_data/_data/ with extracted portainer_data/\n</code></pre></p>"},{"location":"services/portainer/#ops","title":"Ops","text":"<pre><code># CT 109\ncd /opt/stacks/monitoring\ndocker compose up -d --force-recreate portainer\ndocker logs portainer -f\n\n# Trigger manual backup\npython3 /usr/local/sbin/portainer-backup.py\n</code></pre>"},{"location":"services/portainer/#observability","title":"Observability","text":"<p>Portainer metrics scraped by Prometheus on CT 109 at <code>/api/metrics</code>.</p> <ul> <li>Auth: <code>X-API-Key: ptr_tJhUVPuut6wreG6yTkuRZcfM0Rlu8Jehbx7+LHG4unc=</code> (prometheus-scrape API token, user fkrebs)</li> <li>Prometheus job: <code>portainer</code> in <code>/opt/stacks/monitoring/prometheus/prometheus.yml</code></li> <li>Grafana dashboard: <code>http://192.168.1.8:3000/d/portainer-be/portainer-be</code></li> <li>Environment count, health status table, CPU/memory per environment, auth attempt rate</li> <li>Alert rules: <code>/opt/stacks/monitoring/prometheus/rules/portainer.yml</code></li> <li><code>PortainerEnvironmentUnhealthy</code> \u2014 env status == 2 for >2m \u2192 warning</li> <li><code>PortainerDown</code> \u2014 scrape target unreachable for >1m \u2192 critical</li> </ul> <p>Key metrics: | Metric | Description | |---|---| | <code>portainer_environment_count</code> | Total environments by type (docker/k8s/swarm) | | <code>portainer_environment_status</code> | 1=healthy, 2=unhealthy per environment | | <code>portainer_environment_resource_usage</code> | CPU/memory % per environment | | <code>portainer_authentication_total_status</code> | Auth success/fail counters |</p>"},{"location":"services/portainer/#arcane-decommission-2026-05-26","title":"Arcane decommission (2026-05-26)","text":"<p>Arcane was replaced by Portainer on 2026-05-26: - Arcane server (CT 109 <code>/opt/stacks/arcane/docker-compose.yml</code>) \u2192 renamed <code>.DECOMMISSIONED-2026-05-26</code> - Arcane agents removed from: CT 104 ops-agents, CT 105 ops-agents, CT 111 ops-agents, CT 112 ops-agents, CT 113 db stack - Arcane OIDC client <code>81cf4ed0-ea48-4df7-9c2d-cc1704b060f9</code> in Pocket-ID \u2192 to be deleted manually - Zoraxy route <code>arcane.nuclide.systems</code> \u2192 still exists, pending explicit confirmation to remove</p>"},{"location":"services/proton-bridge/","title":"Proton Mail Bridge","text":"<p>Headless Proton Mail Bridge running on CT 104 \u2014 exposes ProtonMail account as SMTP/IMAP endpoints for local services (n8n, Infisical, etc.).</p>"},{"location":"services/proton-bridge/#stack","title":"Stack","text":"<ul> <li>Host: CT 104 (<code>docker</code>, 192.168.1.40), <code>/opt/stacks/proton-bridge/docker-compose.yml</code></li> <li>Image: <code>shenxn/protonmail-bridge:latest</code></li> <li>SMTP: <code>192.168.1.40:1025</code></li> <li>IMAP: <code>192.168.1.40:1143</code></li> <li>Config volume: <code>proton-bridge_proton_config</code></li> </ul>"},{"location":"services/proton-bridge/#initial-login-one-time-interactive","title":"Initial login (one-time, interactive)","text":"<pre><code>ssh root@192.168.1.40\ndocker exec -it proton-bridge /bin/bash\nprotonmail-bridge --cli\n# Commands: login \u2192 (enter Proton credentials) \u2192 list (note bridge SMTP password)\n# exit\n</code></pre> <p>After login the bridge stores credentials in the volume and runs headlessly on restart.</p>"},{"location":"services/proton-bridge/#smtp-credentials-for-other-services","title":"SMTP credentials for other services","text":"<p>Once logged in, run <code>list</code> inside the CLI to get: - SMTP host: <code>192.168.1.40</code> - SMTP port: <code>1025</code> - SMTP user: your Proton email address - SMTP password: the bridge-generated password (not your Proton login password) - IMAP host: <code>192.168.1.40</code> - IMAP port: <code>1143</code></p>"},{"location":"services/proton-bridge/#ops","title":"Ops","text":"<pre><code># CT 104\ncd /opt/stacks/proton-bridge\ndocker compose up -d --force-recreate proton-bridge\ndocker logs proton-bridge -f\n</code></pre>"},{"location":"services/secrets-manager/","title":"Secrets Manager \u2014 Infisical on CT 109","text":"<p>Status: deployed 2026-05-22; migrated CT 112 \u2192 CT 109 on 2026-05-26. Running at <code>http://192.168.1.8:8200</code> (<code>http://secrets.nuclide.lan:8200</code>). LAN-only, no Zoraxy route \u2014 secrets must not be internet-exposed.</p> <p>Stack: <code>infisical/infisical:latest-postgres</code> + Postgres 16 + Redis 7, all on CT 109 (<code>ops</code>, <code>192.168.1.8</code>). Compose at <code>/opt/stacks/infisical/</code> on CT 109. CT 112 (\"secrets\") is decommissioned; LXC pending removal.</p>"},{"location":"services/secrets-manager/#problem-statement","title":"Problem statement","text":"<p>Secrets are currently scattered across:</p> Location Count Risk <code>/opt/stacks/ai/.env</code> on CT 104 ~55 keys Not versioned; duplicated across stacks Per-stack <code>.env</code> files CT 104 ~30 keys Several duplicated (WALG keys, IMMICH_API_KEY, etc.) CT 101, 111, 113 <code>.env</code> files ~30 keys No central rotation story Coder <code>main.tf</code> (hardcoded env vars) 6 Committed to Gitea; visible in template history HA <code>secrets.yaml</code> on HAOS VM 100 unknown Not backed up centrally <p>Goal: single LAN-only secret store that agents, Docker services, and Coder workspaces pull from programmatically. ~85 unique secrets identified in sweep (2026-05-22).</p>"},{"location":"services/secrets-manager/#candidate-solutions","title":"Candidate solutions","text":""},{"location":"services/secrets-manager/#1-infisical-recommended","title":"1. Infisical (recommended)","text":"<p>Open-source HashiCorp Vault alternative. Docker-compose deployable. Native integrations for:</p> <ul> <li>Docker secrets injection (Infisical agent sidecar)</li> <li>Kubernetes/Coder workspace env injection</li> <li>REST API + OIDC machine identities</li> <li>OIDC SSO (can federate with Pocket-ID)</li> <li>Web UI</li> </ul> <p>Deployment: separate LXC (recommended \u2014 see rationale below). Postgres backend \u2192 candidate for CT 113 consolidation once second NVMe lands.</p> <p>Port: 8080 internally; no external exposure needed (LAN-only MCP sidecar pattern).</p>"},{"location":"services/secrets-manager/#2-hashicorp-vault-oss","title":"2. HashiCorp Vault (OSS)","text":"<p>Industry standard. Steeper ops overhead (unsealing, audit logs, lease renewal). Overkill for a homelab unless you need HSM-grade guarantees.</p>"},{"location":"services/secrets-manager/#3-doppler-saas","title":"3. Doppler (SaaS)","text":"<p>Managed; free tier; native CLI and Docker integration. Outbound dependency; secrets leave the homelab. Not suitable given the DLR/LUMEN data handling doctrine.</p>"},{"location":"services/secrets-manager/#4-stay-with-vaultwarden-manual-oidc","title":"4. Stay with Vaultwarden + manual OIDC","text":"<p>Already deployed. Works for human access. No programmatic injection without writing custom code. Dead end for agent/pipeline automation.</p>"},{"location":"services/secrets-manager/#recommendation-infisical-on-a-dedicated-lxc","title":"Recommendation: Infisical on a dedicated LXC","text":""},{"location":"services/secrets-manager/#why-a-separate-lxc","title":"Why a separate LXC?","text":"<ol> <li>Blast radius isolation \u2014 if the CT 104 Docker stack is compromised, the secret store is not on the same attack surface.</li> <li>Minimal footprint \u2014 Infisical + Postgres is the only workload; nothing else can mess with the process space.</li> <li>Simpler audit \u2014 outbound connections from CT 104 are numerous; a dedicated secrets LXC should have near-zero egress.</li> <li>HA restart independence \u2014 CT 104 restarts (image gen, GPU passthrough experiments) don't affect secret availability.</li> </ol> <p>Suggested: CT 112 (next available), 2 vCPU / 2 GB RAM, 8 GB disk on local-zfs.</p>"},{"location":"services/secrets-manager/#architecture","title":"Architecture","text":"<pre><code>flowchart TD\n subgraph CT112[\"CT 112 \u2014 secrets\"]\n infisical[\"Infisical Server\\n:8080\"]\n pg_sec[\"Postgres\\n(infisical DB)\"]\n infisical --- pg_sec\n end\n\n subgraph CT104[\"CT 104 \u2014 Docker host\"]\n agent[\"Infisical Agent\\n(sidecar per stack)\"]\n env_file[\".env (templated)\\nrendered at startup\"]\n agent -->|pull on start| infisical\n agent --> env_file\n end\n\n subgraph CT111[\"CT 111 \u2014 Coder\"]\n coder[\"Coder server\"]\n ws[\"Workspace containers\\n(env injected at provision)\"]\n coder -->|agent token| infisical\n coder --> ws\n end\n\n subgraph CT110[\"CT 110 \u2014 Pocket-ID\"]\n oidc[\"OIDC IdP\"]\n end\n\n oidc -->|machine identity| infisical\n oidc -->|user SSO| infisical</code></pre>"},{"location":"services/secrets-manager/#setup-plan","title":"Setup plan","text":""},{"location":"services/secrets-manager/#phase-1-deploy-infisical-on-ct-112","title":"Phase 1 \u2014 Deploy Infisical on CT 112","text":"<pre><code># On Proxmox host\npvesh create /nodes/nuc/lxc \\\n --ostemplate local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \\\n --vmid 112 --hostname secrets --memory 2048 --cores 2 \\\n --rootfs local-zfs:8 --net0 name=eth0,bridge=vmbr0,ip=dhcp\n\n# Inside CT 112\napt-get install -y docker.io docker-compose-plugin\n# Deploy from https://github.com/Infisical/infisical (official compose)\n</code></pre> <p>Infisical requires a Redis instance alongside Postgres. The official <code>docker-compose.yml</code> bundles both.</p>"},{"location":"services/secrets-manager/#phase-2-migrate-ct-104-secrets","title":"Phase 2 \u2014 Migrate CT 104 secrets","text":"<ol> <li>Create an Infisical project <code>homelab/ct104</code></li> <li>Import existing <code>.env</code> keys via Infisical CLI: <code>infisical import --env prod --path /homelab/ct104 < .env</code></li> <li>Add Infisical agent to each compose stack as a sidecar that renders <code>.env</code> from templates at container start</li> </ol>"},{"location":"services/secrets-manager/#phase-3-coder-workspace-injection","title":"Phase 3 \u2014 Coder workspace injection","text":"<p>Infisical has a native Coder integration (via machine identity token). Replace hardcoded env vars in <code>main.tf</code> with: <pre><code>data \"external\" \"secrets\" {\n program = [\"infisical\", \"export\", \"--env\", \"prod\", \"--path\", \"/coder/workspaces\", \"--format\", \"dotenv-export\"]\n}\n</code></pre></p> <p>Or use the Infisical agent installed in the workspace image.</p>"},{"location":"services/secrets-manager/#phase-4-oidc-sso-via-pocket-id","title":"Phase 4 \u2014 OIDC SSO via Pocket-ID","text":"<p>Infisical supports OIDC SSO under Settings \u2192 Authentication \u2192 OIDC. Register a new client in Pocket-ID with callback <code>https://secrets.nuclide.systems/api/v1/sso/oidc/callback</code> (or LAN-only URL).</p>"},{"location":"services/secrets-manager/#prioritisation","title":"Prioritisation","text":"Item Effort Value Create CT 112, deploy Infisical 1h High \u2014 removes .env sprawl Migrate CT 104 .env 30min High Coder template injection 1h Medium \u2014 workspaces currently work HA secrets.yaml \u2192 Infisical 30min (HA add-on) Medium OIDC SSO via Pocket-ID 30min Low (convenience)"},{"location":"services/secrets-manager/#short-term-mitigation-before-ct-112-is-built","title":"Short-term mitigation (before CT 112 is built)","text":"<ol> <li>Remove secrets from Coder <code>main.tf</code> that appear in Gitea history \u2014 use Coder template variables or env injection instead.</li> <li>Keep <code>/opt/stacks/ai/.env</code> mode <code>0600</code>; ensure it is in <code>.gitignore</code> for any repo that mounts that directory.</li> <li>Use Vaultwarden as the authoritative human copy; rotate any key that has appeared in Claude Code context (see <code>audit-claude-code-meta.md</code>).</li> </ol>"},{"location":"services/secrets-manager/#related","title":"Related","text":"<ul> <li>[[postgres-consolidation]] \u2014 CT 113 will consolidate Postgres instances including Infisical's DB</li> <li>[[feedback_zoraxy_confirm]] \u2014 Zoraxy route for <code>secrets.nuclide.systems</code> will need operator approval when CT 112 is ready</li> <li><code>security/audit-claude-code-meta.md</code> \u2014 lists secrets that transited Claude API this session</li> </ul>"},{"location":"services/zoraxy/","title":"Zoraxy Reverse Proxy","text":"<p>Zoraxy is a Go-based reverse proxy + ACME daemon running as a systemd service on LXC 108 (<code>zoraxy</code>, 192.168.1.4). Config lives in JSON files at <code>/opt/zoraxy/conf/proxy/</code>. Changes take effect after <code>systemctl restart zoraxy</code>.</p>"},{"location":"services/zoraxy/#current-route-table","title":"Current route table","text":"<p>Audited 2026-05-23; updated 2026-05-26 (mcp.nuclide.systems removed, ai/chat backends swapped, arcane.nuclide.systems removed). 20 active public routes have: - <code>EnableAutoHTTPS: true</code> \u2014 Zoraxy requests LE certs for <code>*.nuclide.systems</code> - <code>EnableWebsocketCustomHeaders: true</code> \u2014 preserves WS upgrade headers through the proxy - <code>DisableHopByHopHeaderRemoval: true</code> \u2014 keeps hop-by-hop headers intact for upstream</p> <p><code>SkipWebSocketOriginCheck</code> is enabled on routes that use WebSocket heavily (n8n, Coder, Gotify, Immich, abs, ai, chat, ha, nc, ocpp, s3, shepard*). Disabled on git, hoarder, id, memos (not needed).</p> Domain Upstream SkipWSOrigin Notes abs.nuclide.systems 192.168.1.40:13003 \u2713 Audiobookshelf ai.nuclide.systems 192.168.1.40:14003 \u2713 Bifrost LLM gateway (LiteLLM decommissioned 2026-05-26) ~~arcane.nuclide.systems~~ ~~192.168.1.8:10002~~ \u2014 DECOMMISSIONED 2026-05-26 \u2014 replaced by Portainer chat.nuclide.systems 192.168.1.40:14002 \u2713 Open WebUI (LobeChat decommissioned 2026-05-26) dev.nuclide.systems 192.168.1.40:7080 \u2713 Coder (CT 104; migrated from CT 111 2026-05-26) git.nuclide.systems 192.168.1.40:3000 \u2014 Gitea (CT 104; migrated from CT 111 2026-05-26) gotify.nuclide.systems 192.168.1.40:10003 \u2713 Gotify push notifications ha.nuclide.systems 192.168.1.60:8123 \u2713 Home Assistant (VM 100) hoarder.nuclide.systems 192.168.1.40:17001 \u2014 Karakeep bookmarks id.nuclide.systems 192.168.1.8:11000 \u2014 Pocket-ID OIDC (CT 109; migrated from CT 110 2026-05-26) immich.nuclide.systems 192.168.1.40:12000 \u2713 Immich photos ~~mcp.nuclide.systems~~ ~~192.168.1.40:8080~~ \u2014 DECOMMISSIONED 2026-05-26 \u2014 mcp-gateway removed; MCP now at <code>https://ai.nuclide.systems/mcp</code> (Bifrost) memos.nuclide.systems 192.168.1.40:17000 \u2014 Memos notes n8n.nuclide.systems 192.168.1.40:16000 \u2713 n8n workflows nc.nuclide.systems 192.168.1.41:11000 \u2713 Nextcloud AIO (CT 105) ocpp.nuclide.systems 192.168.1.60:8887 \u2713 OCPP charger endpoint (HAOS) s3.nuclide.systems 192.168.1.40:10004 \u2713 Garage S3 (web endpoint) shepard.nuclide.systems 192.168.1.49:80 \u2713 Shepard frontend (CT 101) shepard-api.nuclide.systems 192.168.1.49:8080 \u2713 Shepard API (CT 101) shepard-auth.nuclide.systems 192.168.1.49:8082 \u2713 Shepard auth (CT 101) traccar.nuclide.systems 192.168.1.40:15000 \u2713 Traccar GPS vault.nuclide.systems 192.168.1.40:11001 \u2713 Vaultwarden <p>Intentionally LAN-only (no Zoraxy route): Dozzle (:10001), Homarr (:7575), Wetty (:4090), docs-server (:13080), paperless (:15003), paperless-ai (:15002), immich-tools. <code>immich-tools.nuclide.systems</code> is listed in portmap but route has not been created \u2014 defer until needed.</p> <p><code>root.config</code> \u2014 ProxyType=0 internal dashboard fallback (127.0.0.1:5487), no ACME.</p>"},{"location":"services/zoraxy/#admin-ui","title":"Admin UI","text":"<pre><code>http://192.168.1.4:8000\n</code></pre> <p>Zoraxy runs with <code>-noauth=true</code> \u2014 no login required on the LAN.</p>"},{"location":"services/zoraxy/#adding-a-new-route","title":"Adding a new route","text":""},{"location":"services/zoraxy/#via-admin-ui","title":"Via admin UI","text":"<ol> <li>Reverse Proxy \u2192 Create Proxy Rules \u2192 New Proxy Rule</li> <li>Enter the domain, set the upstream IP:port</li> <li>TLS \u2192 Enable Auto HTTPS \u2014 tick it</li> <li>Header Rewrite Rules \u2192 enable Disable Hop-by-Hop Removal and WebSocket Custom Headers</li> <li>Save, allow 1\u20132 minutes for LE cert issuance</li> </ol>"},{"location":"services/zoraxy/#via-json-preferred-for-scripted-changes","title":"Via JSON (preferred for scripted changes)","text":"<p>Config files live at <code>/opt/zoraxy/conf/proxy/<domain>.config</code> on CT 108.</p> <p>Minimum template for a new public route:</p> <pre><code>{\n \"ProxyType\": 1,\n \"RootOrMatchingDomain\": \"example.nuclide.systems\",\n \"ActiveOrigins\": [{\"OriginIpOrDomain\": \"192.168.1.40:PORT\", \"Weight\": 1}],\n \"TlsOptions\": {\"EnableAutoHTTPS\": true},\n \"HeaderRewriteRules\": {\"DisableHopByHopHeaderRemoval\": true},\n \"EnableWebsocketCustomHeaders\": true,\n \"AccessFilterUUID\": \"default\"\n}\n</code></pre> <p>After editing, restart Zoraxy:</p> <pre><code>ssh root@192.168.1.4 'systemctl restart zoraxy'\n</code></pre>"},{"location":"services/zoraxy/#backup-restore","title":"Backup / restore","text":"<p>All configs are backed up at <code>/tmp/zoraxy-backup-2026-05-21/</code> on CT 108 (22 files from the 2026-05-21 audit). To restore a single route:</p> <pre><code>ssh root@192.168.1.4 'cp /tmp/zoraxy-backup-2026-05-21/<domain>.config /opt/zoraxy/conf/proxy/ && systemctl restart zoraxy'\n</code></pre>"},{"location":"services/zoraxy/#decommissioned-routes","title":"Decommissioned routes","text":"<p><code>.DECOMMISSIONED-*</code> files in <code>/opt/zoraxy/conf/proxy/</code> are ignored by Zoraxy. Current: - <code>daytona.nuclide.systems.config.DECOMMISSIONED-2026-05-20</code> - <code>id.daytona.nuclide.systems.config.DECOMMISSIONED-2026-05-20</code> - <code>mcp.nuclide.systems.config</code> \u2014 removed 2026-05-26 (mcp-gateway decommissioned; MCP now via Bifrost at <code>ai.nuclide.systems/mcp</code>) - <code>arcane.nuclide.systems.config</code> \u2014 removed 2026-05-26 (Arcane decommissioned; replaced by Portainer at <code>http://192.168.1.8:9000</code> LAN-only)</p>"},{"location":"services/zoraxy/#auth-posture","title":"Auth posture","text":"<p>No routes have <code>ForwardAuthURL</code> set (no Tinyauth/auth middleware). Pocket-ID (<code>id.nuclide.systems</code>) is the OIDC IdP; apps that require auth implement it themselves (Open WebUI, Coder, Gitea, Nextcloud). Public-facing routes like <code>ha.nuclide.systems</code> and <code>ocpp.nuclide.systems</code> have no proxy-level auth \u2014 upstream apps handle it.</p> <p>Pending: Tinyauth or similar on unauthenticated-but-sensitive routes (gotify). Planned for CT 109 deployment.</p>"},{"location":"stacks/CLAUDE/","title":"Notes for agents working in /opt/stacks (CT 104 \"docker\")","text":"<p>Most stacks here are intentional and active. Two things deserve explicit awareness so no agent \"helpfully\" reverts them:</p>"},{"location":"stacks/CLAUDE/#pocket-id-is-gone-from-this-ct-migrated-2026-05-20","title":"Pocket-ID is gone from this CT (migrated 2026-05-20)","text":"<ul> <li>Pocket-ID moved off CT 104 to dedicated CT 110 (\"id\", 192.168.1.5).</li> <li>The compose file <code>/opt/stacks/pocketid/docker-compose.yml</code> was deliberately renamed to <code>docker-compose.yml.MIGRATED-TO-CT110-2026-05-20</code> so accidental <code>docker compose up</code> in that directory is a no-op.</li> <li>The data dir <code>/opt/stacks/pocketid/data/</code> is a migration snapshot only. It is stale; authoritative writes happen on CT 110 since the cutover.</li> <li>Zoraxy upstream for <code>id.nuclide.systems</code> now points at <code>192.168.1.5:11000</code>.</li> <li>Do not restart, <code>compose up</code>, or re-deploy pocket-id here. If something appears broken in OIDC, investigate CT 110 instead. SSH/Console access via <code>pct enter 110</code> on the host, or <code>https://192.168.1.5:11000/</code>.</li> </ul>"},{"location":"stacks/CLAUDE/#dormant-stacks-defined-but-not-running-on-purpose","title":"Dormant stacks \u2014 defined but not running on purpose","text":"<p>Per the audit on 2026-05-20, the following stack directories exist but their containers are intentionally not running. Some are planned to move to CT 109 \"observe\" when that LXC is built; others are legacy experiments waiting on a decision:</p> <ul> <li><code>/opt/stacks/arcane/</code> \u2014 Manager will move to CT 109; edge agent eventually deploys here</li> <li><code>/opt/stacks/dozzle/</code> \u2014 UI will move to CT 109; agent eventually deploys here</li> <li><code>/opt/stacks/homepage/</code> \u2014 replaced by Homarr on CT 109; can be removed once CT 109 lands</li> <li><code>/opt/stacks/arr-stack/</code>, <code>streamio/</code>, <code>nexa/</code>, <code>qdrant/</code>, <code>proxy/</code>, <code>vpn/</code> \u2014 legacy experiments</li> </ul> <p>Do not start these without checking with the operator first.</p>"},{"location":"stacks/CLAUDE/#mcp-gateway-leaves-mcp-child-containers-stay-planned","title":"MCP gateway leaves, MCP child containers stay (planned)","text":"<p>When CT 109 \"observe\" lands, <code>/opt/stacks/ai/mcp-gateway/</code> moves to CT 109 (it's a control plane: OIDC, agent scheduling, token store, usage stats). The ~20 MCP child server containers in this CT (<code>mcp-searxng</code>, <code>mcp-gotify</code>, <code>mcp-immich</code>, <code>mcp-fetch</code>, <code>mcp-time</code>, <code>comfyui-mcp</code>, <code>coder-mcp</code>, <code>kroki-mcp</code>, etc.) stay here on CT 104.</p> <p>The migrated gateway will reach this CT's docker daemon over the planned docker-socket-proxy (see <code>/docs/proxmox-optimizations.md</code> \u00a716). Until CT 109 exists, the gateway runs locally and uses the bind-mounted socket. Do not piecemeal-move the gateway; it's tied to the CT 109 build.</p>"},{"location":"stacks/CLAUDE/#what-is-safe-to-do-here","title":"What is safe to do here","text":"<ul> <li>Anything to do with the actively-running ~65 containers (Immich, n8n, Karakeep, ComfyUI, Vaultwarden, Paperless, LobeChat, LiteLLM, MCP servers, etc.). Daytona was decommissioned 2026-05-20 \u2014 compose files renamed <code>.DECOMMISSIONED-2026-05-20</code>; do not restart.</li> <li>Image pulls + rebuilds for the active stacks (mind the n8n version pin \u2014 see its <code>docker-compose.yaml</code> comment).</li> <li>Rotate exposed secrets in <code>.env</code> files (compose YAMLs themselves are git-tracked).</li> </ul>"},{"location":"stacks/ai/PROMPTING-2026/","title":"Agent system-prompt best practices (2026)","text":"<p>Synthesised from current (2026) prompt-engineering guidance. These are the rules the agent-creator meta-prompt enforces when it drafts a new agent, and the checklist to apply when writing any agent here.</p>"},{"location":"stacks/ai/PROMPTING-2026/#core-philosophy","title":"Core philosophy","text":"<ul> <li>It's information architecture, not magic words. Better context beats longer prompts. Engineer what the model knows and when, not verbosity.</li> <li>A system prompt is operational policy, not a script. Define the agent's identity, job, scope boundaries, success criteria, and stop/escalate rules \u2014 not rigid step-by-step instructions (the model runs many turns/tools).</li> </ul>"},{"location":"stacks/ai/PROMPTING-2026/#structure-in-this-order","title":"Structure (in this order)","text":"<ol> <li>Identity & job \u2014 what the agent is, its single job, and explicitly where the job begins and ends.</li> <li>Operating rules \u2014 tone, language, output format constraints, length.</li> <li>Tools \u2014 which MCP tools it has, when and why to use each, and when not to. Models under-use tools unless told explicitly.</li> <li>Procedure \u2014 the ordered steps/sections to produce (use a numbered list).</li> <li>Boundaries & failure \u2014 permission limits, what to never do, and graceful degradation (\"if a tool fails, write '(nicht verf\u00fcgbar)' and continue \u2014 never invent values\").</li> <li>Closing actions \u2014 side effects (notify, log) stated explicitly.</li> </ol>"},{"location":"stacks/ai/PROMPTING-2026/#mechanics","title":"Mechanics","text":"<ul> <li>Explicit delimiters. Separate rules from content the model processes with stable markers (XML-ish tags or clear headers). Most failures come from the model conflating instructions with data.</li> <li>No assumed state. Dynamic facts (date/time, sensor values) must come from a tool call, never the model's prior. State this in the prompt.</li> <li>Determinism aids. Give concrete IDs/keys; constrain output shape; show one short example only when tone/format matters.</li> <li>Escalation/stop conditions. Say when to stop, when to ask, when to bail.</li> <li>Idempotency/self-logging where relevant (e.g. an agent that logs its own output so a later run can find it).</li> </ul>"},{"location":"stacks/ai/PROMPTING-2026/#anti-patterns","title":"Anti-patterns","text":"<ul> <li>Walls of prose; mixing rules and data; \"be helpful/creative\" with no scope; assuming the current date; vague tool guidance; no failure path.</li> </ul> <p>The Morning Briefing agent (<code>morning-briefing-agent.md</code>) is the worked example that follows all of the above.</p>"},{"location":"stacks/ai/home-info/","title":"Home Info \u2014 seed for the LobeChat morning-briefing agent","text":"<p>Discovered via the Home Assistant MCP (gateway \u2192 <code>home-assistant</code>) on 2026-05-18. Home Assistant: ~2492 entities, 40 domains, 19 areas, 2 floors. Language: German.</p>"},{"location":"stacks/ai/home-info/#layout","title":"Layout","text":"<ul> <li>Floors: Erdgeschoss (level 0) + one upper floor; 19 areas total.</li> <li>Erdgeschoss areas incl.: Badezimmer, B\u00fcro, Esszimmer, Gang, K\u00fcche, Wohnzimmer, Schlafzimmer, Kinderzimmer, Technikraum, Terrasse \u2026 (+ more).</li> </ul>"},{"location":"stacks/ai/home-info/#key-systems","title":"Key systems","text":"<ul> <li>Heat pump: NIBE S1255-6 E (geothermal/brine, BT10 brine-in sensor).</li> <li>Home battery / solar: EcoFlow PowerOcean (<code>PowerOcean emsBpPower</code>, <code>Battery HJ3AZDH5ZG3G0384</code>, switches <code>Hausbatterie: Entladesperre</code>, <code>Hausbatterie: Fahrzeug zuerst</code>). Solar forecast: <code>Solar Prognose</code>, <code>sensor.energy_production_today</code>, <code>sensor.energy_current_hour</code>, <code>sensor.energy_next_hour</code>.</li> <li>EV charging: evcc + PowerPulse wallbox (AC31ZEH4AG130052). Vehicle = BYD.</li> <li>Spa: Whirlpool (heat-pump spa). Filter cycles, thermostat.</li> <li>Other: Philips Hue (lights), smoke detectors (Rauchmelder, w/ temp+humidity), Siemens dishwasher, dryer, Sonos (Playbase/TV), QNAP NAS, 3D printer, grow tent, voice satellites + Piper (local TTS available in HA).</li> </ul>"},{"location":"stacks/ai/home-info/#briefing-entities-use-these-in-the-morning-briefing","title":"Briefing entities (use these in the morning briefing)","text":"Briefing item Entity / source Notes Car state of charge <code>sensor.evcc_byd_configvehicle_soc</code> % (e.g. 98) Car range <code>sensor.evcc_byd_configvehicle_range</code> km (e.g. 422) Car charge limit <code>number.evcc_powerpulse_limit_soc</code> % Solar battery (house) PowerOcean / <code>select.evcc_buffer_soc</code>; use <code>ha_search_entities \"PowerOcean\"</code> / <code>ha_get_state</code> for the live battery-SoC sensor exact SoC sensor: query PowerOcean at briefing time Solar production today <code>sensor.energy_production_today</code> kWh Solar forecast (hour/next) <code>sensor.energy_current_hour</code>, <code>sensor.energy_next_hour</code> Grid import <code>sensor.evcc_powerpulse_charge_total_import</code> kWh Spa temperature <code>sensor.whirlpool_temperatur</code> \u00b0C (e.g. 36.7) Spa thermostat <code>climate.spa_thermostat</code> mode (heat) Spa target temp <code>number.spa_target_desired_temperature</code> \u00b0C Spa pH / bromine NOT in Home Assistant \u2014 no pH/bromine sensors exist omit, or user adds them later / states manually Weather forecast <code>weather.wetter</code> use <code>ha_get_state</code> for forecast attrs Timeline highlights Bluesky MCP (<code>bluesky</code>) \u2014 pending fix once bluesky MCP works Memos recap (yesterday) Memos MCP (<code>memos</code>) \u2014 <code>search_memo</code> / list by date working"},{"location":"stacks/ai/home-info/#systematic-inventory-2026-05-18","title":"Systematic inventory (2026-05-18)","text":"<p>Floor Erdgeschoss (level 0) \u2014 13 areas: Badezimmer, B\u00fcro, Esszimmer, Gang, Garderobe, G\u00e4stezimmer, Kinderzimmer, K\u00fcche, Schlafzimmer, Speis, Technikraum, WC, Wohnzimmer. Floor Au\u00dfen (outdoor) \u2014 6 areas: Eingang, Garage, Grillplatz, Ruheplatz, Terrasse, Grow.</p> <p>Domain counts (2492 entities / 40 domains): sensor 1309, switch 189, update 154, button 152, number 148, binary_sensor 143, select 141, light 62, device_tracker 55, event 33, automation 20, camera 10, notify 9, media_player 8, cover 8, zone 4, fan 4, person 3, image 3, scene 2, climate 2, calendar 1, water_heater 1, weather 1, vacuum 1, humidifier 1, tts 1, stt 1, todo 1, assist_satellite 1, sun 1.</p>"},{"location":"stacks/ai/home-info/#notifications-alerts-for-the-briefing","title":"Notifications / alerts for the briefing","text":"<p>HA has no <code>alert</code> domain; surface alerts from: - Persistent notifications: <code>ha_get_state</code> on <code>persistent_notification.*</code> (or <code>ha_search_entities \"notification\"</code>). - Problem/safety binary_sensors in <code>on</code>: smoke (Rauchmelder), water leak (Wasserticker/\"Batterie fast leer\"), low-battery sensors \u2014 search <code>ha_search_entities \"leer\"</code> / <code>\"rauch\"</code> / <code>\"leak\"</code>, report any <code>state=on</code>. - Automations named like alerts: e.g. <code>automation.low_battery</code> (state on). - Pending updates count: domain <code>update</code> (154 entities; report how many <code>on</code>). The agent should call <code>ha_get_state</code>/<code>ha_search_entities</code> at briefing time and list only items currently in an alert state (don't dump everything).</p>"},{"location":"stacks/ai/home-info/#caveats","title":"Caveats","text":"<ul> <li>Spa pH & bromine are not exposed by the Whirlpool integration \u2014 only temperature/filter. The briefing should skip them (or note \"no chemistry sensors\") unless the user adds manual input_number/sensor entities.</li> <li>House-battery SoC sensor name varies (PowerOcean); the agent should resolve it live via <code>ha_search_entities \"PowerOcean\"</code> + <code>ha_get_state</code> rather than a hardcoded id.</li> </ul>"},{"location":"stacks/ai/morning-briefing-agent/","title":"Morning Briefing \u2014 LobeChat agent","text":""},{"location":"stacks/ai/morning-briefing-agent/#how-to-create-lobechat-ui-2-min","title":"How to create (LobeChat UI \u2014 ~2 min)","text":"<ol> <li>LobeChat \u2192 Create Agent (sidebar +).</li> <li>Title: <code>Morning Briefing</code> \u00b7 Avatar: \u2600\ufe0f</li> <li>Model: <code>qwen3.5-397b-a17b</code> (SAIA, free, strong tool-use), provider OpenAI. LiteLLM auto-fails-over if busy.</li> <li>Plugins / MCP \u2014 enable all of: <code>time</code>, <code>home-assistant</code>, <code>kroki</code>, <code>fetch</code>, <code>sequential-thinking</code>, <code>daytona</code>, <code>ntfy</code>, <code>memos</code>, <code>bluesky</code> (all registered in LobeChat via syncstack).</li> <li>Paste the System Role below.</li> <li>Set the Opening Message below.</li> <li>Optional: LobeChat agent cron for an automatic daily run.</li> </ol>"},{"location":"stacks/ai/morning-briefing-agent/#system-role-paste-verbatim","title":"System Role (paste verbatim)","text":"<pre><code>You are my Morning Briefing assistant for a smart home in Germany (Home\nAssistant, ~2492 entities, floors \"Erdgeschoss\"/\"Au\u00dfen\"). Respond in German,\nterse, dashboard-style, emojis as section headers, one short line per metric,\nround sensibly, never dump raw entity lists. If any tool fails, write\n\"(nicht verf\u00fcgbar)\" for that line and continue \u2014 never invent values.\n\nSTEP 0 (always, silently first):\n- time MCP `get_current_time` \u2192 today + derive yesterday. You do NOT know the\n date; always get it here.\n- Use `sequential-thinking` to plan which tool calls you need, then execute.\n\nTrigger: \"good morning\" / \"briefing\" / chat opened. Produce, in order:\n\n\u25b6 TL;DR \u2014 one punchy line synthesising the day (write this LAST, show it FIRST):\n e.g. \"\u2600\ufe0f guter Solartag, \ud83d\ude97 78 %, laden 13\u201315 Uhr (billig+gr\u00fcn), \ud83d\udd14 1 Hinweis\".\n\n1. \ud83d\ude97 Auto \u2014 SoC `sensor.evcc_byd_configvehicle_soc` %, range\n `sensor.evcc_byd_configvehicle_range` km, limit\n `number.evcc_powerpulse_limit_soc`.\n2. \u2600\ufe0f Solar/Akku \u2014 Hausakku: ha_search_entities \"PowerOcean\" \u2192 ha_get_state;\n `sensor.energy_production_today`, `sensor.energy_current_hour`,\n `sensor.energy_next_hour`; grid import\n `sensor.evcc_powerpulse_charge_total_import`.\n CHART: ha_get_history on the PV sensor for yesterday \u2192 hourly kWh \u2192 render\n via kroki MCP as **Vega-Lite** bar chart (x=Stunde, y=kWh, title with\n yesterday's date). Embed the image.\n3. \u26a1 Energiefluss \u2014 render via kroki a small **D2** (or mermaid) diagram of\n the live flow PV \u2192 Hausakku \u2192 Haus \u2192 Netz \u2192 \ud83d\ude97, annotated with the current\n watts you read in \u00a72. Embed it.\n4. \ud83d\udcb6 Strom & Laden \u2014 fetch MCP GET\n `https://api.awattar.de/v1/marketdata` (German day-ahead prices, no auth).\n Combine the cheapest upcoming hours with the solar forecast (\u00a72) and car\n SoC/limit (\u00a71); via `sequential-thinking` recommend the optimal EV charge\n window today (cheap + green) in one line.\n5. \ud83d\udec1 Whirlpool \u2014 `sensor.whirlpool_temperatur` \u00b0C, `climate.spa_thermostat`,\n `number.spa_target_desired_temperature`. (No pH/Brom sensors \u2014 skip.)\n6. \ud83c\udf26\ufe0f Wetter \u2014 ha_get_state `weather.wetter`: condition, min/max, Regen-%\n from forecast attrs.\n7. \ud83d\udcc5 Heute \u2014 HA `ha_config_get_calendar_events` for today + open items from\n `ha_get_todo`. Max 5 lines; if empty \"nichts angesetzt\".\n8. \ud83d\udd14 Hinweise/Alarme \u2014 ONLY items currently alerting: persistent_notification.*,\n smoke/leak/low-battery binary_sensors \"on\" (search \"leer\",\"rauch\",\"leak\"),\n automation.low_battery if on, count of pending `update` entities on.\n None \u2192 \"keine\".\n9. \ud83d\udce8 ntfy \u2014 ntfy MCP `ntfy_fetch_messages` topic \"homelab-ai\", last 24 h,\n high/urgent first, 1 line each; none \u2192 \"keine\".\n10. \ud83e\udd8b Bluesky \u2014 bluesky MCP: top 3 timeline highlights + 1 line of\n `get-trends`. If auth fails: \"(nicht verf\u00fcgbar)\".\n11. \ud83d\udcdd Memos gestern \u2014 memos MCP `search_memo` for yesterday's date in formats\n \"DD.MM\",\"YYYY-MM-DD\",\"DD.MM.YYYY\"; 2\u20134 bullets; none \u2192 \"keine\".\n12. \ud83e\udde0 Tagesempfehlung \u2014 use `sequential-thinking` to synthesise \u00a71\u20139 into 2\u20133\n concrete actions (Ladefenster, Whirlpool heizen/aus, Lastverschiebung,\n alles aus \u00a78). This is the value \u2014 be specific and practical.\n\nCLOSING ACTIONS (always, after presenting):\n- memos MCP `create_memo`: store a dated PRIVATE memo titled with today's date\n containing the TL;DR + key numbers + Tagesempfehlung. (This makes tomorrow's\n \u00a711 actually find today.)\n- ntfy MCP `ntfy_publish_message` topic \"homelab-ai\", title \"Morning Briefing\",\n priority default: send the TL;DR line so it reaches my phone.\n\nON DEMAND only (if I say \"deep dive\" / \"tiefere analyse\"):\n- daytona MCP: create_sandbox(snapshot \"sciviz-py\") \u2192 write a Python script\n that pulls 7 days of solar production + grid import (give it the figures\n from HA history), renders a matplotlib/seaborn multi-panel trend\n (production vs import, weekday pattern), execute_command to run it, return\n the image, then destroy_sandbox. Embed the figure.\n</code></pre>"},{"location":"stacks/ai/morning-briefing-agent/#opening-message","title":"Opening Message","text":"<pre><code>Guten Morgen! Sag \u201eBriefing\" f\u00fcr dein Dashboard (Auto, Solar + Diagramme,\nStrompreis-Ladeempfehlung, Wetter, Termine, Hinweise, ntfy, Bluesky, Memos\nund eine KI-Tagesempfehlung). \u201eDeep dive\" f\u00fcr die 7-Tage-Energieanalyse.\n</code></pre>"},{"location":"stacks/ai/morning-briefing-agent/#notes","title":"Notes","text":"<ul> <li>Home context + entity IDs: <code>home-info.md</code> (same folder).</li> <li>Uses every server we built: time (date), kroki (Vega-Lite chart + D2/mermaid energy-flow diagram, self-hosted), fetch (aWATTar prices), sequential-thinking (planning + recommendation), daytona (<code>sciviz-py</code> snapshot for the on-demand matplotlib/seaborn deep dive), ntfy (read alerts</li> <li>push digest), memos (recap + self-logging closes the no-date-listing gap), bluesky (timeline+trends), home-assistant (sensors/calendar/todo).</li> <li>Closing memo write is deliberate: the memos MCP can't list by date, so the briefing logs itself \u2192 next day's \u00a711 finds it by date keyword.</li> <li>bluesky degrades gracefully until its app-password rate-limit clears.</li> </ul>"},{"location":"stacks/nexa/","title":"nexa","text":"<p>Neural Nexus for Information & Automation \u2014 central nervous system for a personal IT setup.</p> <p>Memos = voice & ear \u00b7 n8n = reflexes \u00b7 SAIA (LiteLLM) = brain \u00b7 Qdrant (+ optional graph DB) = memory \u00b7 Nextcloud = hands.</p>"},{"location":"stacks/nexa/#start-here","title":"Start here","text":"<p>\ud83d\udcd6 docs/index.md \u2014 TOC, reading paths, repo layout.</p>"},{"location":"stacks/nexa/#repository-layout","title":"Repository layout","text":"<pre><code>nexa/\n\u251c\u2500\u2500 docs/ \u2190 all documentation, numbered for reading order\n\u2514\u2500\u2500 nexa-core/ \u2190 runtime: n8n workflows, configs, prompts, scripts\n</code></pre>"},{"location":"stacks/nexa/CLAUDE/","title":"CLAUDE","text":"<p>STATUS: STALE \u2014 many claims (PVE version, Pocket-ID port, Dockge as docker manager) no longer accurate. Source of truth is /CLAUDE.md and /docs/services/. This file kept for the original nexa-stack design notes only.</p>"},{"location":"stacks/nexa/CLAUDE/#instructions-for-claude-and-other-agents","title":"Instructions for Claude (and other agents)","text":"<p>This file tells future automated runs what they need to know about this repo.</p>"},{"location":"stacks/nexa/CLAUDE/#repo-conventions","title":"Repo conventions","text":"<ul> <li>Documentation: all docs live in <code>/docs/</code> and are numbered. Entry point is <code>docs/index.md</code>. When you add a doc, give it the next free <code>NN-</code> prefix and add a row to the index TOC.</li> <li>Runtime artifacts: live in <code>nexa-core/</code> (workflows, prompts, configs, scripts). Don't put <code>.md</code> documentation in there \u2014 link from <code>/docs/</code> instead.</li> <li>Source-of-truth: if a doc duplicates content from <code>nexa-core/config/*.md</code>, delete the duplicate. Single source of truth.</li> </ul>"},{"location":"stacks/nexa/CLAUDE/#real-infrastructure-verified-from-screenshots-may-2026","title":"Real infrastructure (verified from screenshots, May 2026)","text":"<ul> <li>Proxmox host <code>nuc</code> at <code>192.168.1.20:8006</code> (PVE 9.1.9, kernel 6.17.13-4-pve, EFI).</li> <li>Hardware: 22 threads (Intel Core Ultra 7 155H, 1 socket), 62 GiB RAM, 1.64 TiB disk.</li> <li>Steady state: ~32 GiB used (\u224824 GiB of which is ZFS ARC, tunable via <code>zfs_arc_max</code>), CPU load <2.0, IO delay <0.05%.</li> <li>LXC 102 dns (AdGuard) \u2014 internal DNS, rewrites for <code>*.nuclide.systems</code>.</li> <li>LXC 103 backrest \u2014 backup orchestration.</li> <li>LXC 104 docker \u2014 main docker host at <code>192.168.1.40</code>, hostname <code>docker</code>, OS Debian 13. Unprivileged, originally provisioned from the Dockge helper-script template, but Arcane is the active docker manager today (Dockge is stale, slated for retirement \u2014 see docs/12 #33). Dozzle is the live log viewer. Allocated: 16 CPU, 31.25 GiB RAM (25% used), 8 GiB swap, 200 GiB boot disk (47.7% used). Storage = Ubiquiti UNAS Pro at <code>192.168.1.31</code> (UniFi Drive 4.1.16 on UniFi OS 5.0.17, SFP+ 10 GbE, RAID 6, 19.96 TiB raw / 2.05 TiB used). NFS-exported at <code>/var/nfs/shared/storage</code>, mounted by Proxmox at <code>/mnt/pve/unas</code>, also exposed via SMB at <code>smb://192.168.1.31/<share></code> (Mac) / <code>\\\\192.168.1.31\\<share></code> (Win).</li> </ul> <p>Standard pattern for docker volumes (verified via Karakeep, Q19): plain host bind-mount of <code>/mnt/pve/unas/services/<svc>/<vol></code> from inside LXC 104. No <code>driver_opts</code>, no CIFS, no credentials in the compose. Karakeep, Immich and the rest do exactly this. Nexa follows suit. SMB-as-docker-volume is documented as an escape hatch only (docs/12 #27) for services that hit Nextcloud-style NFS issues \u2014 Nexa doesn't, so we don't use it.</p> <p>Storage-layer snapshots are NOT configured on UNAS Pool 1 (\"Click to Setup\" in the UniFi Drive dashboard). All 2 TB of homelab data has no point-in-time protection at the storage layer \u2014 Backrest covers files, not \"the whole pool last Tuesday\". Highest-leverage fix in the homelab right now (docs/12 #38).</p> <p>UNAS layout conventions (homelab-wide, all docker containers follow them): - <code>services/<svc>/</code> is the general docker config store \u2014 every container in LXC 104 binds its persistent data here. Existing tenants observed: immich, karakeep, nextcloud, ntfy, paperless-ai, pocketid, shelfmark, stremio, traccar, vaultwarden, gluetun. Stale (retire, do not consume): <code>services/siyuan/</code> (migrated to Obsidian), <code>services/open-webui/</code> (unused \u2014 LobeHub is the active LLM UI), and <code>services/dockge/</code> if it exists (retired in favor of Arcane). Nexa MUST follow the same pattern: <code>services/nexa/{qdrant,tei-cache,graphdb,...}</code>. Don't invent a parallel layout. - <code>backup/<svc>/</code> \u2014 per-service backups (existing: home-assistant tars, immich pgdump, nextcloud borg). Nexa snapshots \u2192 <code>backup/nexa/</code>. - <code>media/</code>, <code>code/</code>, <code>_sortMe/</code>, <code>dump/</code>, <code>test_perm</code> \u2014 user data, not Nexa's concern. - Before deploying any Nexa container, READ AN EXISTING STACK in Arcane (e.g. <code>karakeep</code> or <code>immich</code>) to confirm the exact mount syntax in use \u2014 driver name, share path, credential injection pattern. Match it. The actual NFS export root is <code>/var/nfs/shared/storage</code>; the SMB share name is still TBD \u2014 see Q19 in <code>docs/11</code>.</p> <p>Hard-blocklist for any Nexa indexer / agent (never read these paths or matching glob): - <code>_sortMe/wallet/**</code> \u2014 contains PGP keys + bitcoin wallet files. - Any path matching <code>*.gpg</code>, <code>*.asc</code>, <code>*.key</code>, <code>*.pem</code>, <code>id_rsa*</code>, <code>*wallet*</code>, <code>*.kdbx</code>, <code>*credentials*</code>, <code>*secret*</code>. - The Nextcloud appdata dir (<code>services/nextcloud/appdata_*</code>) \u2014 Nextcloud-internal, not user content. Intel iGPU passthrough is configured but currently broken \u2014 see docs/12 #26. - LXC 105 nextcloud \u2014 Nextcloud at <code>nc.nuclide.systems</code>. - LXC 106 octoprint \u2014 currently Exited; flagged in docs/11. - LXC 108 zoraxy \u2014 reverse proxy at <code>192.168.1.4:8000</code>, TLS for <code>*.nuclide.systems</code>. - VM 100 haos \u2014 Home Assistant. - Already-running services on docker host (don't redeploy): - Memos <code>:5230</code>, n8n <code>:5678</code>, LiteLLM <code>:4000</code> (UI LobeHub <code>:3210</code>), Qdrant (<code>qdrant_scientific</code>), ntfy <code>:7998</code>, Karakeep (legacy alias <code>hoarder.nuclide.systems</code>), Vaultwarden <code>:11001</code>, Pocket-ID <code>:1411</code>, Immich, Audiobookshelf, Paperless-ngx, Traccar, Prowlarr, plus MCP containers (<code>crawl4ai-mcp</code>, <code>markitdown-mcp</code>, <code>papersearch-mcp</code>). - Octoprint (LXC 106) is intentionally powered down most of the time. Phase-5 monitoring must skip names matching <code>octoprint*</code> rather than alert on its Exited state. - Obsidian vault lives inside Nextcloud at <code>nc.nuclide.systems/Notizen/</code> (multi-device sync via Nextcloud client). Nexa accesses it via WebDAV \u2014 read-only, no filesystem mount. Ignore list: <code>.copilot/</code>, <code>.copilot-index/</code>, <code>.smart-env/</code>, <code>.caldav-sync/</code>, <code>assets/</code> (visual queue, Phase 3.2), <code>Templates/</code>, <code>BMO/</code>, <code>Excalidraw/</code>. Index target: <code>Notizen/**/*.md</code>. - Nextcloud Tasks lists & calendars (German names, may grow over time): - <code>Pers\u00f6nlich</code> \u2192 Personal context. - <code>DLR</code> \u2192 Work context (DLR is the user's employer). - <code>Einkaufsliste</code> \u2192 Shopping. - <code>Wunschliste</code> \u2192 Wishes. Lists are discovered by name at runtime (Qdrant <code>_config</code> namespace caches <code>name \u2192 id</code>). Never hardcode IDs. The discovery workflow runs daily and on cache-miss; new lists added in Nextcloud are honoured automatically next refresh. - Mail = Nextcloud Mail, single account <code>fkrebs@nucli.de</code>. No separate IMAP entry. The <code>Waiting</code> folder is a manual user signal \u2014 items there are skipped from digests. - Backup model is 3-2-1: UNAS native snapshots \u2192 <code>s3.nuclide.systems</code> (warm, on-site) \u2192 encrypted off-site cold tier (provider TBD, Jottacloud is the user's candidate \u2014 see Q20). Always <code>restic</code>/<code>rclone-crypt</code> before upload \u2014 third-party provider sees only ciphertext. Don't propose alternative backup paths without checking docs/12 #37 first. - Auth = Pocket-ID SSO is global at the Zoraxy layer. Don't add app-level basic-auth to Nexa surfaces; UIs inherit SSO. Machine-to-machine still uses API keys / app passwords. - Decided for Nexa (don't re-litigate without user input): - Vector store: reuse <code>qdrant_scientific</code> with collections suffixed by modality (<code>nexa_knowledge_text</code>, <code>nexa_knowledge_visual</code>). - Embeddings staged: Phase 3.1 TEI + <code>BAAI/bge-m3</code> (text-only, 1024-dim). Phase 3.2 swap to <code>infinity</code> and add <code>jinaai/jina-clip-v2</code> (768-dim, joint text+image space). All forward-compat fields (<code>modality</code>, <code>media_uri</code>, <code>graph_iri</code>, <code>nexa:pendingVisualIndex</code>) exist from 3.1 \u2014 adding the visual collection is additive. - Graph store: Ontotext GraphDB (SPARQL/RDF), Phase 3.4. RDF schema in docs/08 already includes <code>nexa:modality</code> / <code>nexa:mediaUri</code> / <code>nexa:vectorCollection</code> / <code>nexa:pendingVisualIndex</code>. - Chat model: SAIA via LiteLLM virtual key.</p>"},{"location":"stacks/nexa/CLAUDE/#when-working-on-nexa","title":"When working on Nexa","text":"<ol> <li>Read <code>docs/index.md</code> first \u2014 it's the navigator.</li> <li>Open questions first. Before writing code or workflow JSON, scan <code>docs/11-open-questions.md</code>. If your task touches an unanswered Q, stop and ask rather than picking a default. Append new blockers to that doc as <code>[ ] Q-NN</code>.</li> <li>Optimization findings. When you spot infrastructure improvements, add them to <code>docs/12-optimization-opportunities.md</code> as a numbered bullet \u2014 don't just mention them in commit messages.</li> <li>Never inline secrets in workflow JSON or <code>.env</code> committed to git. Use n8n credentials, LiteLLM virtual keys, or (longer term) Vaultwarden.</li> <li>Keep deployment minimal. The default answer to \"do we need a new container?\" is no \u2014 the existing stack covers most needs.</li> </ol>"},{"location":"stacks/nexa/CLAUDE/#branch-policy","title":"Branch policy","text":"<ul> <li>This branch is <code>claude/organize-docs-deployment-7N4v2</code>. Push only here unless told otherwise.</li> <li>New work for an unrelated feature \u2192 new branch under <code>claude/<topic></code>.</li> </ul>"},{"location":"stacks/nexa/CLAUDE/#quick-links","title":"Quick links","text":"<ul> <li>docs/index.md</li> <li>docs/09-deployment.md \u2014 most-touched file during bring-up</li> <li>docs/11-open-questions.md \u2014 read before assuming defaults</li> </ul>"},{"location":"stacks/nexa/docs/","title":"NEXA Documentation","text":"<p>Neural Nexus for Information & Automation \u2014 the central nervous system that ties Memos, n8n, SAIA (LiteLLM), Nextcloud and a vector store into one assistant.</p> <p>This is the documentation entry point. Read top-to-bottom for first-time setup, or jump to the section you need.</p>"},{"location":"stacks/nexa/docs/#table-of-contents","title":"Table of Contents","text":"# Document Read when\u2026 01 Vision & Scope You want to understand what Nexa is and isn't. 02 Roadmap & Phases You want to know the implementation order. 03 Architecture Overview You need a one-page mental model. 04 Integration Matrix You're wiring up a new data source or mapping work-vs-personal flows. 05 Command System You want to know what <code>#nexa:*</code> commands do. 06 Classification Logic You're tuning the work/personal router. 07 Workflow Spec \u2014 Task Router You're building the Phase-2 router workflow. 08 GraphRAG Architecture You're working on Phase 3 (Qdrant + Graph). 09 Deployment You're bringing Nexa up on the real infrastructure. 10 Operations You need backup, monitoring or troubleshooting. 11 Open Questions (user-info-required) Items the user still has to answer before progress. 12 Optimization Opportunities Ideas worth considering for the wider homelab. 13 Information Wishlist What additional system inventory would sharpen future decisions."},{"location":"stacks/nexa/docs/#reading-paths","title":"Reading paths","text":"<ul> <li>First-time installer: 01 \u2192 03 \u2192 09 \u2192 10 \u2192 11</li> <li>Workflow author: 04 \u2192 05 \u2192 07 \u2192 06</li> <li>Phase-3 work (memory): 08 \u2192 09 (\u00a7 Phase 3 add-on)</li> <li>Stakeholder / quick read: 01 \u2192 02 \u2192 12</li> </ul>"},{"location":"stacks/nexa/docs/#repository-layout","title":"Repository layout","text":"<pre><code>nexa/\n\u251c\u2500\u2500 README.md \u2192 points here\n\u251c\u2500\u2500 docs/ \u2192 you are here\n\u2514\u2500\u2500 nexa-core/ \u2192 the actual project\n \u251c\u2500\u2500 ai-prompts/ \u2192 SAIA system prompts\n \u251c\u2500\u2500 config/ \u2192 runtime config (YAML, JSON schema)\n \u251c\u2500\u2500 n8n-workflows/ \u2192 exported workflows, version-controlled\n \u2514\u2500\u2500 scripts/ \u2192 automation helpers\n</code></pre> <p>Source-of-truth for runnable config stays in <code>nexa-core/</code>. Documentation lives here in <code>docs/</code>.</p>"},{"location":"stacks/nexa/docs/01-vision-and-scope/","title":"01 \u2014 Vision & Scope","text":""},{"location":"stacks/nexa/docs/01-vision-and-scope/#vision","title":"Vision","text":"<p>Nexa is the central nervous system of a personal IT setup: an intelligent middleware sitting between input sources (Memos, e-mail, RSS, Karakeep, Bluesky), knowledge stores (Obsidian, Qdrant, Nextcloud) and organization tools (Nextcloud Calendar/Tasks).</p>"},{"location":"stacks/nexa/docs/01-vision-and-scope/#core-goals","title":"Core goals","text":"<ul> <li>Cognitive offload \u2014 Nexa sorts, filters and proposes; it doesn't just store.</li> <li>Context separation \u2014 clean AI-driven split between Work and Personal.</li> <li>Single point of interaction \u2014 Memos is the primary surface (\"voice & ear\").</li> <li>Knowledge synergy \u2014 link ephemeral memos to deep notes via semantic search.</li> </ul>"},{"location":"stacks/nexa/docs/01-vision-and-scope/#scope","title":"Scope","text":""},{"location":"stacks/nexa/docs/01-vision-and-scope/#in-scope","title":"In-scope","text":"<ul> <li>Triaging ~20 personal e-mails per day.</li> <li>Routing tasks to the right Nextcloud list (Work vs. Personal).</li> <li>Suggesting time-boxes in the work calendar.</li> <li>Long-term memory in Qdrant (and optionally a graph DB).</li> <li>Aggregating critical IT alerts (Proxmox, Backrest) into a Memos system feed.</li> </ul>"},{"location":"stacks/nexa/docs/01-vision-and-scope/#out-of-scope","title":"Out-of-scope","text":"<ul> <li>Direct access to work IT (work mail, work servers).</li> <li>Active document editing (metadata extraction only).</li> <li>Replacing specialist UIs (Arcane / Dozzle / Portainer for containers).</li> </ul>"},{"location":"stacks/nexa/docs/01-vision-and-scope/#success-metrics","title":"Success metrics","text":"<ul> <li>Reduced mail interaction \u2014 less time in the mail app, more on the Nexa digest.</li> <li>Calendar fill-rate \u2014 focus slots auto-blocked on the work calendar.</li> <li>Search latency \u2014 old memos / Obsidian notes resurface instantly via <code>#nexa:ask</code>.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/","title":"02 \u2014 Roadmap & Phases","text":"<p>Iterative build-out, value-first. Each phase is shippable on its own.</p>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-1-the-spine-connectivity","title":"Phase 1 \u2014 The Spine (connectivity)","text":"<p>Focus: datapath between Memos, n8n and SAIA. Milestone: Nexa replies on Memos and answers simple questions.</p> <ul> <li>[ ] 1.1 Memos \u2194 n8n bridge \u2014 webhook ingress, content filter, comment-back.</li> <li>[ ] 1.2 SAIA integration \u2014 call LiteLLM proxy from n8n; chat via Memos comments.</li> <li>[ ] 1.3 Git-sync \u2014 periodic export of n8n workflows into this repo.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-2-senses-input-channels","title":"Phase 2 \u2014 Senses (input channels)","text":"<p>Focus: e-mail filter and the Work-vs-Personal router. Milestone: Nexa distinguishes work tasks from personal tasks without manual tags.</p> <ul> <li>[ ] 2.1 Email butler \u2014 filter & summarize the ~20 daily personal mails.</li> <li>[ ] 2.2 RSS morning digest \u2014 curated daily summary in Memos. Items also enter <code>nexa_knowledge_text</code> with <code>source_type=rss, ttl=30d</code> (see 08 \u00a7 Memory sources & retention).</li> <li>[ ] 2.3 Bluesky antenna \u2014 store liked posts as knowledge snippets.</li> <li>[ ] 2.4 On-demand web fetch \u2014 <code>#nexa:ask --web</code> runs a SearXNG query through <code>redis-searxng</code>, fetches top hits via <code>crawl4ai-mcp</code> + <code>markitdown-mcp</code>, embeds and answers. Pages stay in memory with <code>ttl=90d</code> so the next related question doesn't re-fetch.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-3-memory-qdrant-rag","title":"Phase 3 \u2014 Memory (Qdrant & RAG)","text":"<p>Focus: Qdrant + graph layer for retrieval-augmented answers. Milestone: RAG works \u2014 Nexa answers from archived notes; images captured today are queued for visual indexing later.</p> <ul> <li>[ ] 3.1 Text RAG (Path A) \u2014 TEI + <code>bge-m3</code>, single Qdrant collection <code>nexa_knowledge_text</code>. <code>#nexa:ask</code> reads it before answering. Image attachments are recorded as <code>nexa:Note</code> with <code>pendingVisualIndex</code> but not embedded yet.</li> <li>[ ] 3.2 Visual collection (Path C) \u2014 swap TEI \u2192 infinity, add <code>jina-clip-v2</code>, create <code>nexa_knowledge_visual</code>, backfill all queued image notes. RAG workflow gets a parallel branch for visual hits. See 09 \u00a7 \"Phase add-on: visual collection\".</li> <li>[ ] 3.3 S3 long-term archive \u2014 Qdrant snapshots (both collections) to the S3 server. Deferred until Q12 + Q20 are decided. Until then, snapshots stay on UNAS at <code>backup/nexa/snapshots/qdrant/<date>/</code>; UNAS RAID 6 + native pool snapshots (12/#38) cover the data-loss scenarios that matter most for Phase 3.1.</li> <li>[ ] 3.4 Ontotext GraphDB \u2014 SPARQL pillar; structural queries combined with vector hits in <code>#nexa:ask</code>. See 08.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-4-motor-organization-action","title":"Phase 4 \u2014 Motor (organization & action)","text":"<p>Focus: calendar integration and time-boxing. Milestone: Nexa proactively proposes morning focus slots in the work calendar.</p> <ul> <li>[ ] 4.1 Nextcloud Tasks sync \u2014 bidirectional status sync (Memos \u2194 NC Tasks).</li> <li>[ ] 4.2 Time-boxing agent \u2014 calendar-aware slot suggestions.</li> <li>[ ] 4.3 Karakeep curation \u2014 automated review cycle for saved links.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-5-daily-integration-polish","title":"Phase 5 \u2014 Daily integration (polish)","text":"<p>Focus: Home Assistant Voice and monitoring. Milestone: Nexa speaks via HA-Voice and surfaces critical system states proactively.</p> <ul> <li>[ ] 5.1 HA Voice \u2014 Wake-word integration, Wyoming protocol.</li> <li>[ ] 5.2 System monitoring \u2014 Proxmox / Backrest alerts into the Memos system feed via ntfy.</li> <li>[ ] 5.3 Morning digest with self-asking re-loop \u2014 daily 07:00 digest aggregates mail + tasks + drift report. When the digest has spare capacity, Nexa re-asks ONE open question from docs/11 using a backoff schedule (3 d \u2192 7 d \u2192 21 d \u2192 60 d). Answers parsed from the user's reply auto-resolve the question and emit a <code>docs/11</code> diff for review (Phase 6.4 hook). See docs/05 \u00a7 #nexa:digest.</li> </ul>"},{"location":"stacks/nexa/docs/02-roadmap/#phase-6-nexa-as-homelab-steward","title":"Phase 6 \u2014 Nexa as homelab steward","text":"<p>Focus: Nexa actively maintains the homelab inventory instead of being told manually. Milestone: Nexa runs a daily drift report \u2014 what changed, what's stale, what's missing \u2014 and turns it into actionable comments under a pinned <code>[STEWARD]</code> memo.</p> <ul> <li>[ ] 6.1 Inventory sync \u2014 Nexa polls Arcane / <code>docker ps</code> / Proxmox API on a schedule and stores the running-state snapshot in GraphDB (<code>nexa:Service</code>, <code>nexa:LastSeen</code>, <code>nexa:DataPath</code>, \u2026).</li> <li>[ ] 6.2 Drift detection \u2014 diff against the documented desired state in this repo (<code>docs/12</code>, <code>CLAUDE.md</code>, the housekeeping campaign at #35\u201337). Emit findings as structured Memos comments: \"<code>open-webui</code> last used 38 d ago \u2014 retire?\", \"Container <code>foo</code> has no <code>services/foo/</code> bind \u2014 schedule UNAS migration?\".</li> <li>[ ] 6.3 Steward commands \u2014 <code>#nexa:retire <svc></code> (tar to <code>backup/</code>, stop, archive the stack), <code>#nexa:document <svc></code> (template a CLAUDE.md entry), <code>#nexa:wishlist-status</code> (which entries in docs/13 are still needed).</li> <li>[ ] 6.4 Self-update of docs \u2014 when the user resolves an open question via Memos, Nexa proposes the corresponding edit to <code>docs/11-open-questions.md</code> as a PR-ready diff (the doc itself becomes a Nexa-managed surface).</li> </ul> <p>This phase is what turns Nexa from \"an assistant that answers questions\" into \"an assistant that takes care of its own runtime\", and it's the natural home for the cross-cutting housekeeping campaign \u2014 most of those items become semi-automatic once 6.1\u20136.3 ship.</p>"},{"location":"stacks/nexa/docs/03-architecture/","title":"03 \u2014 Architecture Overview","text":"<p>A one-page mental model. For details follow the cross-links.</p>"},{"location":"stacks/nexa/docs/03-architecture/#topology","title":"Topology","text":"<pre><code> \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n voice / typing \u2500\u2500\u2500\u2500\u2500\u2500\u25b6\u2502 Memos \u2502\u25c0\u2500\u2500\u2500\u2500 Nexa replies as comments\n \u2502 (interface) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 webhook (- [ ] / #nexa:*)\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n IMAP / RSS / NC \u2500\u2500\u2500\u2500\u25b6\u2502 n8n \u2502\u25c0\u2500\u2500\u2500\u2500 workflows live in\n Karakeep / Bluesky \u2502 (logic) \u2502 ./nexa-core/n8n-workflows\n \u2514\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2518\n classify \u25b2 \u2502 \u2502 \u25b2 retrieve\n \u2502 \u25bc \u25bc \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 SAIA \u2502 \u2502 Qdrant \u2502\n \u2502 LiteLLM \u2502 \u2502 vector \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 GraphDB \u2502 Ontotext, SPARQL\n \u2502 (RDF) \u2502 Phase 3.4\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc writes\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Nextcloud (Tasks, Calendar, \u2502\n \u2502 Mail, Files / Obsidian) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"stacks/nexa/docs/03-architecture/#components","title":"Components","text":"Component Role Where it runs (today) Memos Interface, voice input, webhook source docker host LXC 104 (unprivileged, 16 CPU / 31 GiB / 200 GiB) \u2192 <code>memos.nuclide.systems</code> n8n Workflow / logic engine docker host LXC 104 \u2192 <code>n8n.nuclide.systems</code> SAIA / LiteLLM Model gateway, embeddings, classification docker host LXC 104 \u2192 <code>ai.nuclide.systems</code> (LiteLLM internal :4000) Qdrant Vector memory (semantic recall) docker host LXC 104 \u2014 reuse <code>qdrant_scientific</code>. Collections: <code>nexa_knowledge_text</code> (Phase 3.1, 1024-dim) and <code>nexa_knowledge_visual</code> (Phase 3.2, 768-dim) Ontotext GraphDB Structural memory via SPARQL (Phase 3.4) not yet deployed; see 09-deployment TEI (HF text-embeddings-inference) Self-hosted text embeddings, <code>BAAI/bge-m3</code>, Phase 3.1 docker host LXC 104, CPU only \u2014 swapped for <code>infinity</code> in Phase 3.2 to add <code>jina-clip-v2</code> Nextcloud Tasks, calendar, mail, files dedicated LXC 105 \u2192 <code>nc.nuclide.systems</code> ntfy Push channel for system alerts docker host \u2192 <code>ntfy.nuclide.systems</code> Backrest Backup orchestration LXC 103 Zoraxy Reverse proxy + TLS LXC 108 (<code>192.168.1.4:8000</code>) AdGuard DNS Internal name resolution LXC 102 Home Assistant Voice + house automation VM 100 (HAOS)"},{"location":"stacks/nexa/docs/03-architecture/#two-pillar-memory","title":"Two-pillar memory","text":"<ul> <li>Qdrant answers \"what is similar / relevant?\" (cosine search over embeddings).</li> <li>Ontotext GraphDB answers \"who, what depends on what, how is it structured?\" (SPARQL over RDF).</li> </ul> <p>Both pillars are queried in parallel for <code>#nexa:ask</code> and merged before SAIA generates the final answer. See 08 \u2014 GraphRAG architecture.</p>"},{"location":"stacks/nexa/docs/03-architecture/#dual-context-routing","title":"Dual-context routing","text":"<p>Every input is classified <code>work</code> or <code>personal</code> before any side effect (task creation, calendar write). See 04 \u2014 Integration matrix and 06 \u2014 Classification logic.</p>"},{"location":"stacks/nexa/docs/04-integration-matrix/","title":"04 \u2014 Integrations-Matrix","text":"<p>Diese Matrix definiert die logische Trennung zwischen privaten und beruflichen Datenstr\u00f6men sowie die Anbindung der Infrastruktur.</p>"},{"location":"stacks/nexa/docs/04-integration-matrix/#1-die-dualitat-arbeit-vs-privat","title":"1. Die Dualit\u00e4t: Arbeit vs. Privat","text":"<p>Nexa muss strikt zwischen zwei Kontexten unterscheiden, da die Datenquellen variieren:</p>"},{"location":"stacks/nexa/docs/04-integration-matrix/#a-bereich-arbeit-work","title":"A. Bereich: ARBEIT (Work)","text":"<ul> <li>Sichtbarkeit: Eingeschr\u00e4nkt (kein Zugriff auf Arbeits-E-Mails oder Firmen-Server).</li> <li>Datenquellen:<ul> <li>Nextcloud Task-Liste <code>DLR</code> (sowie der gleichnamige Kalender).</li> <li>Memos mit Tag <code>#work</code> oder semantischer Erkennung.</li> </ul> </li> <li>Nexa-Fokus: Timeblocking, Fokus-Zeiten, Vorbereitung von Meetings basierend auf Obsidian-Notizen.</li> </ul>"},{"location":"stacks/nexa/docs/04-integration-matrix/#b-bereich-privat-personal","title":"B. Bereich: PRIVAT (Personal)","text":"<ul> <li>Sichtbarkeit: Vollst\u00e4ndig.</li> <li>Datenquellen:<ul> <li>Nextcloud Task-Liste <code>Pers\u00f6nlich</code> und gleichnamiger Kalender.</li> <li>Nextcloud Mail (<code>fkrebs@nucli.de</code>, einziges Konto). Ordner <code>Waiting</code> ist ein manuelles Signal \u2014 Mails dort werden vom Digest ausgeschlossen.</li> <li>Bluesky, RSS, Karakeep.</li> </ul> </li> <li>Nexa-Fokus: Automatisierung des Alltags, Kuratierung von Wissen, E-Mail-Management.</li> </ul>"},{"location":"stacks/nexa/docs/04-integration-matrix/#karakeep-shopping-logik","title":"Karakeep & Shopping-Logik","text":"<ul> <li>Schnittstelle: Karakeep API / RSS.</li> <li>Listen-Routing:<ul> <li>Einkaufsliste: Automatischer Sync f\u00fcr Items mit hoher Frequenz (Lebensmittel, Drogerie).</li> <li>Wunschliste: Speicherort f\u00fcr \"Entdeckungen\". Nexa f\u00fcgt bei Wunschlisten-Items automatisch den aktuellen Preis und eine kurze SAIA-Zusammenfassung hinzu (\"Warum du das speichern wolltest\").</li> </ul> </li> <li>Review-Cycle: Einmal im Monat fragt Nexa bei Wunschlisten-Items nach: \"Immer noch interessiert oder kann das weg?\"</li> </ul>"},{"location":"stacks/nexa/docs/04-integration-matrix/#2-kern-infrastruktur-the-brain-spine","title":"2. Kern-Infrastruktur (The Brain & Spine)","text":"Komponente Rolle im System Memos Zentraler Input (Drafts, Ideen, schnelle Tasks). Schnittstelle f\u00fcr Nexa-Antworten. n8n Logik-Engine. F\u00fchrt die Klassifizierung Arbeit vs. Privat durch. SAIA (LiteLLM) Entscheidet anhand des Inhalts, in welchen Kalender/Liste ein Eintrag geh\u00f6rt. Qdrant Langzeitged\u00e4chtnis. Speichert Projektwissen (Work) und privates Wissen getrennt. Arcane Verwaltung der Docker-Container (n8n, Qdrant, Memos)."},{"location":"stacks/nexa/docs/04-integration-matrix/#3-spezifische-datenflusse-logik","title":"3. Spezifische Datenfl\u00fcsse & Logik","text":""},{"location":"stacks/nexa/docs/04-integration-matrix/#task-routing-das-gehirn-filter","title":"Task-Routing (Das Gehirn-Filter)","text":"<ol> <li>Input: Neues Memo oder Spracheingabe.</li> <li>Analyse: SAIA pr\u00fcft: \"Ist das Business oder Privat?\"</li> <li>Routing: * <code>Arbeits-Kontext</code> -> Eintrag in <code>Work_Tasks</code>.<ul> <li><code>Privat-Kontext</code> -> Eintrag in <code>Personal_Tasks</code>.</li> </ul> </li> <li>Timeboxing: Nexa scannt den <code>Work_Calendar</code> auf L\u00fccken und schl\u00e4gt Slots f\u00fcr <code>Work_Tasks</code> vor.</li> </ol>"},{"location":"stacks/nexa/docs/04-integration-matrix/#e-mail-management-nur-privat","title":"E-Mail Management (Nur Privat)","text":"<ul> <li>Nexa filtert die 20 t\u00e4glichen Privat-Mails.</li> <li>Wichtige private Termine werden in den privaten Kalender extrahiert.</li> <li>Newsletter landen als Zusammenfassung in Memos.</li> </ul>"},{"location":"stacks/nexa/docs/04-integration-matrix/#wissens-management-obsidian-via-nextcloud-qdrant","title":"Wissens-Management (Obsidian via Nextcloud & Qdrant)","text":"<ul> <li>Der Obsidian-Vault liegt im Nextcloud-Ordner <code>Notizen/</code> (Multi-Device-Sync via Nextcloud-Client). Nexa liest read-only \u00fcber WebDAV (<code>/remote.php/dav/files/<user>/Notizen/</code>) \u2014 kein Mount, kein zweiter Sync-Mechanismus.</li> <li>Markdown-Dateien (<code>Notizen/**/*.md</code>) werden in Qdrant (<code>nexa_knowledge_text</code>) indiziert; Bilder unter <code>assets/</code> landen im Phase-3.2-Visual-Queue (siehe docs/11/Q4).</li> <li>Plugin-Internals (<code>.copilot/</code>, <code>.smart-env/</code>, <code>.caldav-sync/</code>, <code>Templates/</code>, <code>BMO/</code>, <code>Excalidraw/</code>) sind explizit von der Indizierung ausgeschlossen.</li> <li>Nexa nutzt dieses Wissen, um Arbeits-Tasks besser zu beschreiben, auch wenn sie keinen Zugriff auf die Arbeits-E-Mails hat.</li> </ul>"},{"location":"stacks/nexa/docs/04-integration-matrix/#4-monitoring-hardware-home-assistant","title":"4. Monitoring & Hardware (Home Assistant)","text":"Dienst Reiz-Typ Nexa-Reaktion Home Assistant Voice / Sensorik Nexa nimmt Sprachbefehle entgegen und meldet Alarme (Haus). Proxmox Stabilit\u00e4t Meldung an Nexa bei Hardware-Problemen. S3 Server Archiv Langzeit-Backups der Wissensdatenbank."},{"location":"stacks/nexa/docs/04-integration-matrix/#5-erganzungen-fur-das-setup-vorschlag","title":"5. Erg\u00e4nzungen f\u00fcr das Setup (Vorschlag)","text":"<ul> <li>Zwei-Faktor-Klassifizierung: Einf\u00fchrung von expliziten Tags (<code>#w</code> f\u00fcr Work, <code>#p</code> f\u00fcr Privat) f\u00fcr Grenzf\u00e4lle, in denen SAIA den Kontext nicht eindeutig bestimmen kann.</li> </ul>"},{"location":"stacks/nexa/docs/05-command-system/","title":"05 \u2014 Command System (<code>#nexa:*</code>)","text":"<p>Nexa \"h\u00f6rt\" auf folgende Kommandos in Memos-Kommentaren oder als Memo-Inhalt mit Hashtag:</p>"},{"location":"stacks/nexa/docs/05-command-system/#konfigurations-kommandos","title":"\ud83d\udd27 Konfigurations-Kommandos","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexaconfig","title":"#nexa:config","text":"<ul> <li>Beschreibung: Triggert Autodiscovery aller Services (Nextcloud Listen/Kalender, Qdrant Collections, LiteLLM-Modelle, Mail-Konto).</li> <li>L\u00e4uft auch automatisch:</li> <li>T\u00e4glich (cron in n8n) \u2014 neue Listen / Kalender werden ohne manuelles Eingreifen aufgenommen.</li> <li>Bei Cache-Miss \u2014 wenn ein Workflow eine Liste/ID nicht mehr findet, l\u00e4uft Discovery einmal, dann Retry.</li> <li>Antwort: Postet Report mit den aktuellen Konfigurationen als Memos-Kommentar.</li> <li>Speichert: Konfiguration in der Qdrant <code>_config</code> Namespace (nicht in <code>.env</code>).</li> </ul> <pre><code>#nexa:config\n</code></pre> <p>Beispiel-Response: <pre><code>\u2705 Autodiscovery abgeschlossen (2026-05-04T23:59):\n- Nextcloud Listen: Pers\u00f6nlich (id=14), DLR (id=dlr-1), Einkaufsliste (id=\u2026), Wunschliste (id=\u2026)\n- Nextcloud Kalender: Pers\u00f6nlich, DLR, Einkaufsliste, Wunschliste\n- Mail-Konto: fkrebs@nucli.de (Posteingang, Archiv, Junk, Waiting)\n- Qdrant Collection: nexa_knowledge_text (1024 dims, Cosine, 0 Punkte)\n- LiteLLM-Modelle: nexa-chat, nexa-embed\n</code></pre></p>"},{"location":"stacks/nexa/docs/05-command-system/#nexastatus","title":"#nexa:status","text":"<ul> <li>Beschreibung: Zeigt den aktuellen System-Status</li> <li>Antwort: Uptime, verbundene Systeme, Memory-Stats</li> </ul> <pre><code>#nexa:status\n</code></pre>"},{"location":"stacks/nexa/docs/05-command-system/#nexasync-obsidian","title":"#nexa:sync-obsidian","text":"<ul> <li>Beschreibung: Triggert sofortige Synchronisierung mit Obsidian Vault</li> <li>Optional Parameter: <code>--vault=/path/to/vault</code> oder <code>--force</code> f\u00fcr Neuindexierung</li> </ul> <pre><code>#nexa:sync-obsidian --force\n</code></pre>"},{"location":"stacks/nexa/docs/05-command-system/#memory-rag-kommandos","title":"\ud83d\udcca Memory & RAG Kommandos","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexaask-question","title":"#nexa:ask [question]","text":"<ul> <li>Beschreibung: Semantische Suche in Qdrant (RAG) basierend auf Obsidian + Memos</li> <li>Antwort: Top-3 Treffer mit Quellen</li> </ul> <pre><code>#nexa:ask Wie implementierten wir das JWT-Middleware-Pattern?\n</code></pre>"},{"location":"stacks/nexa/docs/05-command-system/#nexalearn-topic","title":"#nexa:learn [topic]","text":"<ul> <li>Beschreibung: Explizit neue Information in Qdrant speichern (mit Tags)</li> <li>Optional: <code>--tag=architecture</code> <code>--source=obsidian/notes/arch.md</code> <code>--permanent</code> (\u00fcberschreibt Default-TTL)</li> </ul> <pre><code>#nexa:learn Das Routing-Schema unterscheidet Work vs. Personal via SAIA-Kontext-Analyse --tag=architecture\n</code></pre>"},{"location":"stacks/nexa/docs/05-command-system/#nexaforget-query-or-iri","title":"#nexa:forget [query-or-iri]","text":"<ul> <li>Beschreibung: L\u00f6scht passende Eintr\u00e4ge aus Qdrant + GraphDB sofort.</li> <li>Beispiel: <code>#nexa:forget --source=rss --older-than=14d</code></li> </ul>"},{"location":"stacks/nexa/docs/05-command-system/#nexaretain-source_type-days","title":"#nexa:retain [source_type] [days]","text":"<ul> <li>Beschreibung: Setzt die Default-Retention f\u00fcr einen Source-Type (Default-Werte siehe 08 \u00a7 Memory sources & retention).</li> </ul>"},{"location":"stacks/nexa/docs/05-command-system/#nexaask-question-web","title":"#nexa:ask [question] [--web]","text":"<ul> <li>Beschreibung: Semantische Suche in Qdrant. Mit <code>--web</code> erweitert Nexa die Suche um eine SearXNG-Abfrage (<code>redis-searxng</code>), holt die Top-3 Treffer via <code>crawl4ai-mcp</code> + <code>markitdown-mcp</code>, embedded sie und nutzt das gemeinsame Material f\u00fcr die Antwort. Geholte Seiten bleiben 90 Tage im Speicher (TTL siehe oben).</li> </ul>"},{"location":"stacks/nexa/docs/05-command-system/#workflow-kommandos","title":"\ud83c\udfaf Workflow-Kommandos","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexaroute-test-text","title":"#nexa:route-test [text]","text":"<ul> <li>Beschreibung: Testet die Klassifizierung (Work/Personal) ohne Task zu erstellen</li> </ul> <pre><code>#nexa:route-test Muss morgen die Pr\u00e4sentation f\u00fcr den Client fertigstellen\n</code></pre> <p>Response: <pre><code>\ud83d\udcac Klassifizierung (Test-Mode):\n- Kontext: WORK\n- Vertrauen: 0.95\n- Begr\u00fcndung: \"Client-Pr\u00e4sentation \u2192 professioneller Kontext\"\n</code></pre></p>"},{"location":"stacks/nexa/docs/05-command-system/#nexaemail-digest","title":"#nexa:email-digest","text":"<ul> <li>Beschreibung: Triggert sofortige Email-Zusammenfassung (sonst t\u00e4gl. 7 Uhr)</li> </ul> <pre><code>#nexa:email-digest\n</code></pre>"},{"location":"stacks/nexa/docs/05-command-system/#nexadigest","title":"#nexa:digest","text":"<ul> <li>Beschreibung: T\u00e4gliches Morgen-Digest. L\u00e4uft automatisch (07:00). Inhalt:</li> <li>Kurze Mail-Zusammenfassung (siehe <code>#nexa:email-digest</code>).</li> <li>Anstehende Tasks heute (DLR + Pers\u00f6nlich, nach Urgency).</li> <li>Drift-Report aus Phase 6 (sobald aktiv).</li> <li>Re-ask Block \u2014 siehe unten. Maximal 1 offene Frage pro Digest, nur wenn der Digest sonst kurz genug ist.</li> </ul>"},{"location":"stacks/nexa/docs/05-command-system/#re-ask-der-offenen-fragen-docs11-open-questionsmd","title":"Re-ask der offenen Fragen (<code>docs/11-open-questions.md</code>)","text":"<p>Nexa f\u00fchrt im GraphDB pro offener Frage <code>nexa:askedCount</code>, <code>nexa:lastAskedAt</code>, <code>nexa:nextAskAt</code>. Backoff-Schema:</p> Mal Wartezeit bis zur n\u00e4chsten Frage 1 \u2192 2 3 Tage 2 \u2192 3 7 Tage 3 \u2192 4 21 Tage \u2265 5 60 Tage <p>Im Morgen-Digest wird eine offene Frage angeh\u00e4ngt \u2014 und zwar nur wenn: - Der Digest sonst < 800 Zeichen lang w\u00e4re (Capacity-Guard, damit es nicht nervt). - <code>nextAskAt <= today</code>. - Bevorzugt die Frage mit dem kleinsten <code>askedCount</code> (zuerst neue Fragen, alte selten).</p> <p>Antwortet der User direkt unter dem Digest-Memo, parst Nexa die Antwort, markiert die Frage in <code>docs/11</code> als resolved (Phase 6.4 \u2014 Self-update of docs) und committet einen Diff zur Review.</p>"},{"location":"stacks/nexa/docs/05-command-system/#nexaremind","title":"#nexa:remind <ul> <li>Manuelles Triggern einer Re-Ask, ignoriert Backoff. N\u00fctzlich f\u00fcr R\u00fcckfragen mitten im Tag.</li> </ul>","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexaanswered","title":"#nexa:answered <ul> <li>Beantwortet eine offene Frage explizit. Setzt <code>resolved</code> im GraphDB und schl\u00e4gt einen <code>docs/11</code>-Diff vor.</li> </ul>","text":""},{"location":"stacks/nexa/docs/05-command-system/#admin-kommandos","title":"\ud83d\udd10 Admin-Kommandos","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexareset-config","title":"#nexa:reset-config <ul> <li>Beschreibung: L\u00f6scht gespeicherte Konfiguration, triggert neuen Autodiscovery</li> <li>Warnung: Alle benutzerdefinierten Settings werden vergessen</li> </ul> <pre><code>#nexa:reset-config\n</code></pre>","text":""},{"location":"stacks/nexa/docs/05-command-system/#nexaexport-state","title":"#nexa:export-state <ul> <li>Beschreibung: Exportiert aktuellen State als JSON (f\u00fcr Backups)</li> </ul> <pre><code>#nexa:export-state\n</code></pre>","text":""},{"location":"stacks/nexa/docs/05-command-system/#implementierung-in-n8n","title":"\ud83d\udcdd Implementierung in n8n","text":"<p>Ein Command Parser l\u00e4uft immer mit:</p> <ol> <li>Memos Webhook empf\u00e4ngt alle Memos</li> <li>Regex Check: Sucht nach <code>#nexa:command</code></li> <li>Router: Versendet an entsprechenden n8n-Workflow</li> <li>Antwort: Postet Reply als Kommentar/Edit</li> </ol> <p>Command Parser Regex: <pre><code>^#nexa:(\\w+)(?:\\s+([^\\n]*?))?(?:$|\\s*--)\n</code></pre></p> <p>Extrahiert: <code>[command, parameters]</code></p>"},{"location":"stacks/nexa/docs/05-command-system/#dynamische-konfigurationsspeicherung","title":"\ud83c\udf9b\ufe0f Dynamische Konfigurationsspeicherung","text":"<p>Statt .env zu editieren:</p> <ol> <li>First Run: <code>#nexa:config</code> speichert zu lokalen Metadata</li> <li>Speich-Ziel: </li> <li>Primary: Qdrant Metadata (als <code>_config</code> Namespace)</li> <li>Fallback: <code>nexa-core/config/runtime_config.json</code> (gitignored)</li> <li> <p>Notfall: .env (nur initial)</p> </li> <li> <p>Reload-Logik: Bei jedem Workflow-Start werden Settings aus Qdrant geladen</p> </li> </ol> <p>Das macht Nexa vollst\u00e4ndig selbstst\u00e4ndig nach dem initialem Setup!</p>"},{"location":"stacks/nexa/docs/06-classification-logic/","title":"06 \u2014 Klassifizierungs-Logik","text":"<p>Dieser Fragebogen dient der Feinabstimmung von SAIA, um Tasks korrekt zu routen.</p>"},{"location":"stacks/nexa/docs/06-classification-logic/#a-schlusselworter-projekte-arbeit","title":"A. Schl\u00fcsselw\u00f6rter & Projekte (Arbeit)","text":"<p>Welche Begriffe triggern zwingend die Work-Liste? - [ ] Projekt-Namen (z.B. \"Nexa-Core\", \"Infrastruktur-Audit\") - [ ] Rollenspezifische Begriffe (\"Meeting\", \"Report\", \"Deadline\") - [ ] Tools, die nur im Job vorkommen.</p>"},{"location":"stacks/nexa/docs/06-classification-logic/#b-ausschlusskriterien-privat","title":"B. Ausschlusskriterien (Privat)","text":"<p>Was darf niemals in die Work-Liste? - [ ] Lebensmittel, Rezepte, Haushalt. - [ ] Bluesky-Input (sofern nicht explizit als Recherche markiert). - [ ] Finanz-Mails (Privat-Bank).</p>"},{"location":"stacks/nexa/docs/06-classification-logic/#c-umgang-mit-unscharfe","title":"C. Umgang mit Unsch\u00e4rfe","text":"<ul> <li>Default: Wenn < 70% Sicherheit -> <code>Personal</code>.</li> <li>Interaktion: Nexa erstellt einen Kommentar: \"In welche Liste soll das? [W/P]\".</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/","title":"07 \u2014 Workflow Spec: Phase-2 Task-Router","text":""},{"location":"stacks/nexa/docs/07-workflow-spec/#1-trigger","title":"1. Trigger","text":"<ul> <li>Typ: Webhook (Memos)</li> <li>Filter: <code>payload.content</code> enth\u00e4lt <code>- [ ]</code> ODER <code>#todo</code>.</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/#2-intelligence-node-saia","title":"2. Intelligence Node (SAIA)","text":"<ul> <li>Prompt: \"Analysiere folgende Notiz. Gib als JSON zur\u00fcck: <code>{ \"context\": \"work\" | \"personal\" | \"shopping\" | \"wishes\", \"summary\": \"string\", \"urgency\": 1-5 }</code>.\"</li> <li>Kontext-Zufuhr: Lade die letzten 5 Obsidian-Projekttitel aus Qdrant als Referenz hoch.</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/#3-list-resolver-vor-dem-switch","title":"3. List Resolver (vor dem Switch)","text":"<ul> <li>Lese den Cache <code>nexa._config.lists</code> aus Qdrant: <code>{ \"Pers\u00f6nlich\": <id>, \"DLR\": <id>, \"Einkaufsliste\": <id>, \"Wunschliste\": <id>, discovered_at: <ts> }</code>.</li> <li>Bei Cache-Miss oder 404 vom nachfolgenden Create-Task \u2192 triggere <code>#nexa:config</code> (Discovery), update den Cache, retry einmal.</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/#4-switch-node-4-wege","title":"4. Switch Node (4-Wege)","text":"<ul> <li>Route 1: <code>context == \"work\"</code> \u2192 Nextcloud Create Task auf Liste <code>DLR</code>.</li> <li>Route 2: <code>context == \"personal\"</code> \u2192 Nextcloud Create Task auf Liste <code>Pers\u00f6nlich</code>.</li> <li>Route 3: <code>context == \"shopping\"</code> \u2192 Nextcloud Create Task auf Liste <code>Einkaufsliste</code>.</li> <li>Route 4: <code>context == \"wishes\"</code> \u2192 Nextcloud Create Task auf Liste <code>Wunschliste</code>.</li> <li>Default (Konfidenz < 0.7): <code>Pers\u00f6nlich</code> (siehe 06).</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/#5-feedback-loop","title":"5. Feedback Loop","text":"<ul> <li>Poste die Task-ID und den gew\u00e4hlten Kontext als Kommentar unter das Original-Memo.</li> </ul>"},{"location":"stacks/nexa/docs/07-workflow-spec/#6-listen-diskoverability","title":"6. Listen-Diskoverability","text":"<p>Listennamen sind nicht hardgecodet \u2014 die Resolver-Step liest sie aus dem Runtime-Config-Cache. Wenn der User in Nextcloud eine neue Liste anlegt (z. B. <code>Reisen</code>), erscheint sie nach dem n\u00e4chsten geplanten <code>#nexa:config</code>-Lauf (t\u00e4glich) automatisch als Routing-Ziel \u2014 die System-Prompt der Intelligence Node wird zusammen mit den verf\u00fcgbaren Listen versorgt, sodass SAIA neue Kontexte vorschlagen kann.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/","title":"08 \u2014 GraphRAG: Structural Knowledge & Relations","text":"<p>Decision: graph layer = Ontotext GraphDB with SPARQL (resolved in 11/Q1). Rationale: SPARQL + RDF lets Nexa's memory be browsed and queried with the same standard tooling that's used for any open-data corpus, and it leaves the door open for SHACL / OWL reasoning later.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#two-pillar-memory","title":"Two-pillar memory","text":"Pillar Question it answers Backed by Qdrant (vectors) \"What is similar / relevant?\" Cosine search over embeddings GraphDB (RDF) \"What is connected? What depends on what? Who is involved?\" SPARQL over a typed graph <p>Both pillars are queried in parallel for <code>#nexa:ask</code> and merged before SAIA generates the final answer.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#rdf-schema","title":"RDF schema","text":"<p>Compact, opinionated. One namespace, one ontology file, no v2/v3 inheritance pain.</p> <pre><code>@prefix nexa: <https://nuclide.systems/nexa/ontology#> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n@prefix prov: <http://www.w3.org/ns/prov#> .\n\n# Classes\nnexa:Project a rdfs:Class .\nnexa:Task a rdfs:Class .\nnexa:Person a rdfs:Class .\nnexa:Technology a rdfs:Class .\nnexa:Topic a rdfs:Class .\nnexa:Note a rdfs:Class . # Memos / Obsidian / mail digests\nnexa:File a rdfs:Class .\n\n# Properties\nnexa:owns a rdf:Property ; rdfs:domain nexa:Person ; rdfs:range nexa:Task .\nnexa:uses a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Technology .\nnexa:dependsOn a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Task .\nnexa:childOf a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range nexa:Project .\nnexa:mentions a rdf:Property ; rdfs:domain nexa:Note ; rdfs:range nexa:Topic .\nnexa:scheduledFor a rdf:Property ; rdfs:domain nexa:Task ; rdfs:range xsd:dateTime .\n\n# Datatype properties\nnexa:status a rdf:Property ; rdfs:range xsd:string . # \"needs-action\" | \"in-progress\" | \"done\"\nnexa:context a rdf:Property ; rdfs:range xsd:string . # \"work\" | \"personal\"\nnexa:urgency a rdf:Property ; rdfs:range xsd:integer . # 1\u20135\nnexa:contentHash a rdf:Property ; rdfs:range xsd:string . # for de-dup\n\n# Cross-pillar / multimodality\nnexa:modality a rdf:Property ; rdfs:range xsd:string . # \"text\" | \"image\"\nnexa:mediaUri a rdf:Property ; rdfs:range xsd:anyURI . # memos://\u2026 , nextcloud://\u2026 , obsidian://\u2026\nnexa:vectorCollection a rdf:Property ; rdfs:range xsd:string . # \"nexa_knowledge_text\" | \"nexa_knowledge_visual\"\nnexa:vectorId a rdf:Property ; rdfs:range xsd:string . # Qdrant point ID\nnexa:pendingVisualIndex a rdf:Property ; rdfs:range xsd:boolean . # set true on image notes until Phase 3.2 backfills them\n</code></pre> <p><code>nexa:vectorId</code> + <code>nexa:vectorCollection</code> together are the bridge between graph and vector store. A SPARQL hit can trigger a vector lookup, and a Qdrant payload's <code>graph_iri</code> field walks back the other way.</p> <p><code>nexa:modality</code>, <code>nexa:mediaUri</code> and <code>nexa:pendingVisualIndex</code> exist from Phase 3.1 even though only the text path is wired up. Image attachments captured in 3.1 are recorded as <code>nexa:Note</code> with <code>modality \"image\"</code> and <code>pendingVisualIndex true</code>, then picked up by the Phase-3.2 backfill workflow \u2014 no data loss across phases.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#sync-flows","title":"Sync flows","text":""},{"location":"stacks/nexa/docs/08-graphrag-architecture/#1-memos-graphdb-real-time","title":"1. Memos \u2192 GraphDB (real-time)","text":"<pre><code>Memo content: \"Muss JWT-Middleware f\u00fcr Auth-Service refaktorieren\"\n \u2502\n \u25bc SAIA extracts entities + relations as JSON\n \u2502 { tasks: [{title, urgency}], technologies: [...],\n \u2502 relations: [{type:\"uses\", from:..., to:...}] }\n \u2502\n \u25bc n8n turns JSON into a SPARQL UPDATE\n \u2502\n \u2514\u2500\u2500\u25b6 INSERT DATA { ... } against GraphDB repo \"nexa_knowledge\"\n</code></pre>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#2-obsidian-graphdb-nexasync-obsidian","title":"2. Obsidian \u2192 GraphDB (<code>#nexa:sync-obsidian</code>)","text":"<p>For each Obsidian note: parse front-matter + headings \u2192 emit <code>nexa:Project</code>, <code>nexa:Task</code>, <code>nexa:Note</code> triples; <code>nexa:mentions</code> for <code>[[wikilinks]]</code>.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#3-nextcloud-tasks-graphdb-bidirectional","title":"3. Nextcloud Tasks \u2194 GraphDB (bidirectional)","text":"<p>n8n trigger on Nextcloud CalDAV/Tasks change \u2192 <code>INSERT/DELETE DATA</code> to keep <code>nexa:status</code> and <code>nexa:scheduledFor</code> in sync.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#example-sparql-queries","title":"Example SPARQL queries","text":""},{"location":"stacks/nexa/docs/08-graphrag-architecture/#q1-all-open-tasks-involving-jwt-by-urgency","title":"Q1 \u2014 All open tasks involving JWT, by urgency","text":"<pre><code>PREFIX nexa: <https://nuclide.systems/nexa/ontology#>\n\nSELECT ?taskTitle ?urgency ?projectName\nWHERE {\n ?tech rdfs:label \"JWT\" .\n ?task nexa:uses ?tech ;\n rdfs:label ?taskTitle ;\n nexa:status ?status ;\n nexa:urgency ?urgency .\n FILTER (?status IN (\"needs-action\", \"in-progress\"))\n OPTIONAL { ?task nexa:childOf ?project . ?project rdfs:label ?projectName . }\n}\nORDER BY DESC(?urgency)\n</code></pre>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#q2-what-does-auth-service-transitively-depend-on","title":"Q2 \u2014 What does Auth-Service transitively depend on?","text":"<pre><code>PREFIX nexa: <https://nuclide.systems/nexa/ontology#>\n\nSELECT DISTINCT ?dep ?label\nWHERE {\n ?root rdfs:label \"Auth-Service\" .\n ?root nexa:dependsOn+ ?dep .\n ?dep rdfs:label ?label .\n}\n</code></pre> <p>(<code>+</code> is SPARQL property-paths \u2014 transitive closure, free.)</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#q3-topics-with-the-most-note-mentions-in-the-last-day","title":"Q3 \u2014 Topics with the most note-mentions in the last day","text":"<pre><code>PREFIX nexa: <https://nuclide.systems/nexa/ontology#>\nPREFIX xsd: <http://www.w3.org/2001/XMLSchema#>\n\nSELECT ?topic (COUNT(?note) AS ?n)\nWHERE {\n ?note a nexa:Note ;\n prov:generatedAtTime ?ts ;\n nexa:mentions ?topic .\n FILTER (?ts > NOW() - \"P1D\"^^xsd:duration)\n}\nGROUP BY ?topic\nORDER BY DESC(?n)\nLIMIT 10\n</code></pre>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#q4-cross-pillar-find-vectors-for-tasks-blocking-project-x","title":"Q4 \u2014 Cross-pillar: \"find vectors for tasks blocking project X\"","text":"<pre><code>PREFIX nexa: <https://nuclide.systems/nexa/ontology#>\n\nSELECT ?taskTitle ?vectorId\nWHERE {\n ?proj rdfs:label \"Nexa\" .\n ?task nexa:childOf ?proj ;\n nexa:status \"in-progress\" ;\n nexa:vectorId ?vectorId ;\n rdfs:label ?taskTitle .\n}\n</code></pre> <p>n8n then takes each <code>?vectorId</code>, fetches the embedding from Qdrant, and runs a \"more like this\" search for richer context.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#graphrag-answer-pipeline-nexaask","title":"GraphRAG answer pipeline (<code>#nexa:ask</code>)","text":"<pre><code> #nexa:ask <question>\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc\n [Qdrant] [GraphDB]\n semantic structural\n top-k SPARQL \u2014 auto-generated\n notes paths / dependencies\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n merge + rank\n \u2502\n \u25bc\n SAIA prompt:\n \"Given these passages and these relations, answer \u2026\"\n \u2502\n \u25bc\n comment under the original memo\n</code></pre> <p>Auto-generation of SPARQL: SAIA is given the ontology (above) as a system prompt and asked to emit a <code>SELECT</code>/<code>CONSTRUCT</code> query for the user's natural-language question. n8n executes it, falls back to a templated query on parse failure.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#n8n-integration-sketch","title":"n8n integration sketch","text":""},{"location":"stacks/nexa/docs/08-graphrag-architecture/#workflow-graph-sync-trigger","title":"Workflow: Graph-Sync Trigger","text":"<pre><code>[Memos Webhook]\n \u2502\n[Parse Content]\n \u2502\n[SAIA: Extract entities + relations as JSON]\n \u2502\n[Build SPARQL UPDATE INSERT DATA { ... }]\n \u2502\n[HTTP POST \u2192 /repositories/nexa_knowledge/statements]\n \u2502\n[Index in Qdrant; write Qdrant point id back via second SPARQL UPDATE]\n</code></pre>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#workflow-question-router","title":"Workflow: Question Router","text":"<pre><code>[#nexa:ask Query]\n \u2502\n \u250c\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc\n[SAIA: NL \u2192 SPARQL] [Qdrant: kNN]\n \u2502 \u2502\n[POST \u2192 SPARQL endpoint]\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n rank + merge \u2192 SAIA answer\n</code></pre>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#graph-management-commands","title":"Graph-management commands","text":""},{"location":"stacks/nexa/docs/08-graphrag-architecture/#nexagraph-status","title":"<code>#nexa:graph-status</code>","text":"<p>Returns triple count, class histogram, most-connected entity. Implemented as one SPARQL <code>SELECT (COUNT)</code>.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#nexagraph-trace-entity","title":"<code>#nexa:graph-trace [entity]</code>","text":"<p>Returns the 1-hop (and optionally 2-hop) neighbourhood \u2014 a <code>DESCRIBE <iri></code> plus a templated outgoing/incoming query.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#nexagraph-rebuild","title":"<code>#nexa:graph-rebuild</code>","text":"<p>Clears the named graph and replays Obsidian + Memos. SPARQL: <code>CLEAR GRAPH <https://nuclide.systems/nexa/runtime></code> followed by the import workflow.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#why-two-stores","title":"Why two stores","text":"Scenario Qdrant GraphDB Best \"Which note was similar to this one?\" \u2705 \u274c Qdrant \"What blocks this task?\" \u274c \u2705 GraphDB \"Explain this project\" \u2705 context \u2705 structure both \"All JWT-related open work\" \u2705 semantic \u2705 crisp both <p>Combined: complete understanding rather than a search index or a structure index.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#memory-sources-retention","title":"Memory sources & retention","text":"<p>Not every embedding deserves to live forever. Nexa indexes from several source types and each has its own expected lifetime. The contract: every Qdrant point carries <code>payload.source_type</code> and <code>payload.expires_at</code> (epoch seconds, or <code>null</code> for permanent). A daily prune workflow runs <code>DELETE WHERE expires_at < NOW()</code> on each collection and mirrors the deletion in GraphDB.</p> <code>source_type</code> Where it comes from Default TTL Rationale <code>memo</code> Memos webhook permanent User-authored, low volume, high signal. <code>obsidian</code> Nextcloud <code>Notizen/</code> via WebDAV permanent User-authored knowledge base. <code>mail</code> Nextcloud Mail (single account) 365 d Audit trail + searchable past correspondence. Mail digests are derived, not stored as their own embeddings. <code>mail_digest</code> Daily digest output 90 d Summarised content; the source mails persist longer. <code>karakeep</code> Karakeep saved links permanent User explicitly bookmarked. <code>rss</code> Phase-2.2 morning digest feed items 30 d News signal decays fast; keep recent for \"what was that article last week?\". <code>web_search</code> On-demand fetch via crawl4ai-mcp / markitdown-mcp during <code>#nexa:ask</code> 90 d Useful for \"what did we look at last quarter?\" but not eternal. <code>system</code> Backrest / Proxmox / n8n alerts via <code>nexa.system</code> ntfy topic 30 d Operational telemetry; old alerts have little RAG value. <code>task</code> Nextcloud Tasks \u2194 GraphDB sync until task deleted Mirrors source-of-truth. <p>External sources go through the same pipeline as memos \u2014 fetch \u2192 markitdown-mcp \u2192 embed via TEI \u2192 upsert into <code>nexa_knowledge_text</code> with the appropriate <code>source_type</code> + <code>expires_at</code>. The graph node carries <code>nexa:source</code>, <code>nexa:fetchedAt</code>, <code>nexa:sourceUri</code>, and <code>nexa:contentHash</code> for de-dup (so the same article fetched twice doesn't create two points).</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#web-search-loop","title":"Web search loop","text":"<p><code>#nexa:ask</code> first searches existing memory. If the merged confidence is below a threshold (or the user adds <code>--web</code> to the command), Nexa runs a SearXNG query through <code>redis-searxng</code>, picks the top 3 results, fetches them through <code>crawl4ai-mcp</code> + <code>markitdown-mcp</code>, embeds the cleaned markdown, and answers from the augmented context. The fetched pages stay in memory (TTL 90 d) so the next related question doesn't re-fetch.</p> <p>This means the homelab's existing <code>*-mcp</code> containers are part of Nexa's data plane, not just decoration \u2014 see docs/12 #8.</p>"},{"location":"stacks/nexa/docs/08-graphrag-architecture/#manual-overrides","title":"Manual overrides","text":"<ul> <li><code>#nexa:learn <text> --permanent</code> overrides the default TTL.</li> <li><code>#nexa:forget <iri-or-search></code> triggers an immediate Qdrant delete + GraphDB <code>DELETE WHERE { ?n nexa:vectorId \"...\" . }</code>.</li> <li><code>#nexa:retain <source_type> <days></code> rewrites the default for that source type (stored in the <code>_config</code> namespace, picked up by the next prune run).</li> </ul>"},{"location":"stacks/nexa/docs/09-deployment/","title":"09 \u2014 Deployment","text":"<p>Pragmatic deployment guide that assumes the existing homelab and adds only what's missing.</p>"},{"location":"stacks/nexa/docs/09-deployment/#whats-already-running-no-action-required","title":"What's already running (no action required)","text":"<p>Surveyed from Homepage / Dozzle / Proxmox / Zoraxy:</p> Service Host / port URL Memos docker LXC 104 \u2192 <code>:5230</code> <code>https://memos.nuclide.systems</code> n8n docker LXC 104 \u2192 <code>:5678</code> <code>https://n8n.nuclide.systems</code> LiteLLM (SAIA gateway) docker LXC 104 \u2192 <code>:4000</code> <code>https://ai.nuclide.systems</code> (proxies LobeHub UI :3210; API on :4000) Nextcloud LXC 105 <code>https://nc.nuclide.systems</code> ntfy docker LXC 104 \u2192 <code>:7998</code> <code>https://ntfy.nuclide.systems</code> Karakeep docker LXC 104 \u2192 <code>:3090</code> <code>https://hoarder.nuclide.systems</code> (legacy host alias kept for compatibility) Home Assistant VM 100 (HAOS) <code>https://ha.nuclide.systems</code> Pocket-ID (OAuth/SSO) docker LXC 104 \u2192 <code>:1411</code> <code>https://id.nuclide.systems</code> Vaultwarden docker LXC 104 \u2192 <code>:11001</code> <code>https://vault.nuclide.systems</code> Backrest LXC 103 (internal) AdGuard DNS LXC 102 (internal) Zoraxy reverse proxy LXC 108 \u2192 <code>192.168.1.4:8000</code> TLS for *.nuclide.systems <code>qdrant_scientific</code> (existing) docker LXC 104 reused \u2014 Nexa uses <code>nexa_*</code> collections in this instance <p>The deployment task is not \"spin up the stack\" \u2014 most of the stack is already up. It is wire Nexa across these services + add the small bits that are missing.</p>"},{"location":"stacks/nexa/docs/09-deployment/#whats-missing-for-nexa","title":"What's missing for Nexa","text":"<ol> <li>Qdrant collection for Nexa (<code>nexa_knowledge</code>) inside the existing <code>qdrant_scientific</code> instance \u2014 vector dim follows Q15 (1024 for <code>bge-m3</code>, 768 for <code>nomic-embed-text</code>).</li> <li>TEI (HF text-embeddings-inference) on the docker host for self-hosted embeddings (LiteLLM key is not authorised for OpenAI embeddings \u2014 see 11/Q3+Q15). Lighter than Ollama: single Rust binary, ~500 MB image, no LLM runtime.</li> <li>n8n workflows (<code>./nexa-core/n8n-workflows/</code>) imported into the running n8n.</li> <li>Nextcloud lists & calendars for Work / Personal / Shopping / Wishes (auto-discovered via <code>#nexa:config</code>).</li> <li>Memos webhook \u2192 n8n wired through the Memos config.</li> <li>LiteLLM virtual key for the <code>nexa</code> user with chat-only access (no embeddings \u2014 handled by Ollama).</li> <li>A Zoraxy host entry is not needed \u2014 Memos / n8n / LiteLLM are already proxied.</li> <li>(Phase 3.4) Ontotext GraphDB for the SPARQL pillar \u2014 see add-on at the bottom of this doc.</li> </ol>"},{"location":"stacks/nexa/docs/09-deployment/#step-1-secrets","title":"Step 1 \u2014 Secrets","text":"<p>Copy <code>nexa-core/.env.example</code> \u2192 <code>nexa-core/.env</code> and fill only the secrets:</p> <pre><code>cd nexa-core\ncp .env.example .env\n$EDITOR .env # MEMOS_API_KEY, SAIA_API_KEY, NC_APP_PASSWORD, QDRANT_API_KEY\n</code></pre> <p>The <code>.env</code> is only used at bootstrap time. Everything else (list IDs, calendar IDs, collection sizes) is discovered at runtime via <code>#nexa:config</code> (see 05). No secrets should ever live in n8n workflow JSON \u2014 use n8n credentials instead.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-2-qdrant-collection-nexa_knowledge_text","title":"Step 2 \u2014 Qdrant collection (<code>nexa_knowledge_text</code>)","text":"<p>Phase 3.1 ships Path A (text-only) but the schema and naming already make room for Path C (text + visual) so adding a <code>nexa_knowledge_visual</code> collection later is a pure additive operation \u2014 no rename, no migration, no n8n rewiring.</p> <pre><code># adjust QDRANT_HOST in .env first\nsource nexa-core/.env\n\n# create the text collection from the schema file\ncurl -X PUT \"$QDRANT_HOST/collections/nexa_knowledge_text\" \\\n -H \"Content-Type: application/json\" \\\n -H \"api-key: $QDRANT_API_KEY\" \\\n -d @nexa-core/config/qdrant_schema.json\n</code></pre> <p>The collection name is always suffixed with the modality (<code>_text</code>, <code>_visual</code>) so logic in n8n and SPARQL stays modality-aware from day one. Indexed rows carry these payload fields (source):</p> Field Why it's there now <code>modality</code> Always <code>\"text\"</code> in <code>_text</code>, <code>\"image\"</code> in <code>_visual</code>. Future-proofs cross-modality filters. <code>source_type</code> <code>memo</code> / <code>mail</code> / <code>obsidian</code> / <code>screenshot</code> / <code>image</code> \u2014 used by classification and digest workflows. <code>media_uri</code> <code>memos://\u2026</code>, <code>nextcloud://\u2026</code>, <code>obsidian://\u2026</code>. Empty for text-only rows; populated when Path C ships. <code>graph_iri</code> IRI of the corresponding <code>nexa:Note</code> in GraphDB. The same value is stored on the GraphDB side as <code>nexa:vectorId</code> \u2014 this is the cross-pillar bridge. <code>content_hash</code> de-dup. <code>context</code> <code>work</code> / <code>personal</code>. <p>Targets the existing <code>qdrant_scientific</code> instance \u2014 just an extra collection, no new container. The <code>vectors.size</code> field follows Q15: 1024 for <code>bge-m3</code>, 768 for <code>nomic-embed-text-v1.5</code>.</p>"},{"location":"stacks/nexa/docs/09-deployment/#image-attachments-today-queue-them","title":"Image attachments today (queue them)","text":"<p>Memos can already attach images. Until Phase 3.2 the indexer does not embed them, but it does record them so they can be replayed later:</p> <ul> <li>Memo with an image \u2192 text body still goes into <code>nexa_knowledge_text</code>.</li> <li>The image attachment(s) are written as <code>nexa:Note</code> triples in GraphDB with <code>nexa:modality \"image\"</code> and <code>nexa:vectorId</code> left empty (<code>nexa:pendingVisualIndex true</code>).</li> <li>A Phase-3.2 backfill workflow will pick up everything where <code>?n nexa:pendingVisualIndex true</code> and embed it through the visual collection.</li> </ul> <p>This means no data is lost between 3.1 and 3.2 \u2014 the queue is the GraphDB itself.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-3-self-hosted-embeddings-tei","title":"Step 3 \u2014 Self-hosted embeddings (TEI)","text":"<p>Use HuggingFace text-embeddings-inference \u2014 single Rust binary, ~500 MB image, OpenAI-compatible API, loads exactly one model. Lighter than Ollama because there's no LLM runtime, no GGUF loader, no model registry.</p> <p>The active docker manager on this LXC is Arcane (visible from Homepage as the running container manager \u2014 the LXC was originally provisioned with the Dockge helper-script template, but Dockge is now stale; see 12/#33). Paste the stack into Arcane \u2192 name it <code>nexa</code> \u2192 save \u2192 start. Don't <code>docker compose up -d</code> over SSH; Arcane manages the compose lifecycle.</p> <pre><code># Nexa stack \u2014 paste into Arcane.\n# Storage convention matches the rest of the homelab (verified against\n# the running Karakeep stack, Q19): host bind-mount of\n# /mnt/pve/unas/services/<svc>/<vol>. No volume-driver, no CIFS, no\n# credentials in the compose \u2014 the LXC's NFS mount is already there.\nservices:\n nexa-embed:\n image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5\n container_name: nexa-embed\n restart: unless-stopped\n command: [\"--model-id\", \"BAAI/bge-m3\"]\n ports:\n - \"127.0.0.1:8080:80\"\n volumes:\n - /mnt/pve/unas/services/nexa/tei-cache:/data\n env_file:\n - .env\n\nnetworks: {}\n</code></pre> <p>Pre-deploy step on the docker LXC (one-time):</p> <pre><code>mkdir -p /mnt/pve/unas/services/nexa/{tei-cache,qdrant,graphdb}\nmkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}\n</code></pre> <p>Memory budget: ~1.1 GB resident. First start downloads <code>bge-m3</code> (~1 GB) into <code>/mnt/pve/unas/services/nexa/tei-cache/</code>; subsequent restarts are instant.</p> <p>Secrets (<code>SAIA_API_KEY</code>, <code>MEMOS_API_KEY</code>, <code>QDRANT_API_KEY</code>, <code>NC_APP_PASSWORD</code>) go in the stack's <code>.env</code> next to the compose \u2014 same pattern Karakeep uses (<code>env_file: .env</code>). Arcane has an editor for it. Vaultwarden becomes the source-of-truth long-term (12/#11) but isn't required for the first cut.</p> <p>Why bind-mount and not SMB? Earlier drafts of this doc proposed an SMB-via-docker-volume pattern because of the user's \"had it with Nextcloud\" experience. The Karakeep stack confirms the actual convention is the simpler one: host bind-mount of the LXC's existing <code>/mnt/pve/unas</code> NFS mount. The Nextcloud failure was Nextcloud-specific (its setup tooling chowns the data dir to <code>www-data</code>, which fails against <code>root_squash</code> exports) and doesn't apply to normal containers. SMB-as-docker-volume stays documented in 12/#27 only as an escape hatch if a future service hits Nextcloud-style issues \u2014 Nexa doesn't, so we don't use it.</p> <p>Register it inside LiteLLM (admin UI \u2192 Models) with the OpenAI-compatible adapter:</p> <ul> <li>model name: <code>nexa-embed</code></li> <li>provider: <code>openai</code></li> <li>model: <code>bge-m3</code></li> <li>api_base: <code>http://nexa-embed:80/v1</code></li> <li>api_key: any non-empty string (TEI ignores it)</li> </ul> <p>Now n8n only ever talks to LiteLLM and the model is swappable without touching workflows.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-4-litellm-virtual-key","title":"Step 4 \u2014 LiteLLM virtual key","text":"<p>In the LiteLLM admin UI (<code>ai.nuclide.systems</code>):</p> <ol> <li>Create user <code>nexa</code>.</li> <li>Issue a virtual key with access to:</li> <li>one chat model (already-available model from your SAIA gateway).</li> <li>the <code>nexa-embed</code> model from Step 3.</li> <li>Paste the key into <code>SAIA_API_KEY</code> in <code>.env</code>.</li> </ol>"},{"location":"stacks/nexa/docs/09-deployment/#step-4-n8n-workflows","title":"Step 4 \u2014 n8n workflows","text":"<p>Import the JSON exports \u2014 credentials are filled inside n8n, not in the JSON:</p> <pre><code># n8n personal access token from the n8n UI: Settings \u2192 API\nN8N_URL=https://n8n.nuclide.systems\nN8N_TOKEN=... # from the n8n UI\n\nfor f in nexa-core/n8n-workflows/phase-1/*.json \\\n nexa-core/n8n-workflows/phase-2/*.json; do\n curl -X POST \"$N8N_URL/api/v1/workflows\" \\\n -H \"X-N8N-API-KEY: $N8N_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n --data-binary \"@$f\"\ndone\n</code></pre> <p>Inside n8n, attach credentials to the imported nodes:</p> <ul> <li>Memos \u2192 HTTP header <code>Authorization: Bearer $MEMOS_API_KEY</code></li> <li>LiteLLM \u2192 header <code>Authorization: Bearer $SAIA_API_KEY</code> (chat + <code>nexa-embed</code>)</li> <li>Nextcloud Tasks / Calendar / WebDAV (Obsidian vault under <code>Notizen/</code>) \u2192 one app password (<code>$NC_APP_PASSWORD</code>), reused across all three node types</li> <li>Qdrant \u2192 header <code>api-key: $QDRANT_API_KEY</code></li> </ul> <p>Activate each workflow individually after smoke-test.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-5-memos-webhook","title":"Step 5 \u2014 Memos webhook","text":"<p>In the Memos admin UI, set the webhook URL to the production address of the discovery workflow:</p> <pre><code>https://n8n.nuclide.systems/webhook/memos\n</code></pre> <p>The same URL is the one the workflow exposes; verify with:</p> <pre><code>curl -i https://n8n.nuclide.systems/webhook/memos\n# expect 200 / 405, never 404\n</code></pre>"},{"location":"stacks/nexa/docs/09-deployment/#step-6-bootstrap-commands-via-memos","title":"Step 6 \u2014 Bootstrap commands via Memos","text":"<p>Create a memo with body <code>#nexa:config</code> \u2014 the discovery workflow:</p> <ol> <li>Lists Nextcloud Tasks lists, looking for <code>Pers\u00f6nlich</code>, <code>DLR</code>, <code>Einkaufsliste</code>, <code>Wunschliste</code> (and any new lists added later \u2014 see 05). Caches <code>name \u2192 id</code>.</li> <li>Lists Nextcloud Calendars by the same names; caches IDs.</li> <li>Verifies the Nextcloud Mail account <code>fkrebs@nucli.de</code> and the folder list (<code>Posteingang</code>, <code>Archiv</code>, <code>Junk</code>, <code>Waiting</code>).</li> <li>Counts existing Qdrant points in <code>nexa_knowledge_text</code>.</li> <li>Verifies LiteLLM reachability + lists available models for the Nexa virtual key.</li> <li>Replies as a comment with a runtime-config snapshot that's stored in the Qdrant <code>_config</code> namespace (and mirrored as <code>nexa-core/config/runtime_config.json</code>, gitignored).</li> </ol> <p>The same workflow is also triggered by: - A daily cron inside n8n (so newly added Nextcloud lists become routable without intervention). - Cache-miss in the task-router \u2014 if a list ID 404s, the router fires <code>#nexa:config</code> once and retries.</p> <p>After this point, <code>.env</code> is read-once. Subsequent runs read config from Qdrant.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-7-smoke-tests","title":"Step 7 \u2014 Smoke tests","text":"<pre><code># (1) Memos round-trip \u2014 should produce a comment within ~5 s\ncurl -X POST https://memos.nuclide.systems/api/v1/memos \\\n -H \"Authorization: Bearer $MEMOS_API_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"content\":\"- [ ] testing the router #nexa\"}'\n\n# (2) Classification dry-run\ncurl -X POST https://memos.nuclide.systems/api/v1/memos \\\n -H \"Authorization: Bearer $MEMOS_API_KEY\" \\\n -d '{\"content\":\"#nexa:route-test buy milk\"}'\n\n# (3) RAG test (requires at least one indexed memo/note)\ncurl -X POST https://memos.nuclide.systems/api/v1/memos \\\n -H \"Authorization: Bearer $MEMOS_API_KEY\" \\\n -d '{\"content\":\"#nexa:ask what is the goal of nexa?\"}'\n</code></pre>"},{"location":"stacks/nexa/docs/09-deployment/#step-8-reverse-proxy","title":"Step 8 \u2014 Reverse proxy","text":"<p>Already done \u2014 Zoraxy at <code>192.168.1.4:8000</code> terminates TLS for <code>*.nuclide.systems</code> and forwards to docker LXC 104 (<code>192.168.1.40</code>). No new entry is required for Nexa: every service Nexa talks to already has a host entry.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-9-backups","title":"Step 9 \u2014 Backups","text":"<p>Already covered by Backrest (LXC 103). Add:</p> <ul> <li>n8n workflows \u2192 <code>nexa-core/scripts/backup_workflows.sh</code> (already present) into a Backrest schedule.</li> <li>Qdrant snapshots \u2192 schedule a daily <code>POST /collections/nexa_knowledge/snapshots</code> and rsync to S3 (<code>s3.nuclide.systems</code>). Add as a Backrest pre-hook on the docker host.</li> </ul> <p>For deeper detail: 10 \u2014 Operations.</p>"},{"location":"stacks/nexa/docs/09-deployment/#phase-add-on-ontotext-graphdb-phase-34","title":"Phase add-on: Ontotext GraphDB (Phase 3.4)","text":"<p>Defer until 3.1\u20133.3 ship.</p> <pre><code># nexa-core/docker-compose.graph.yml\nservices:\n graphdb:\n image: ontotext/graphdb:10.7.0\n container_name: nexa-graphdb\n ports: [\"127.0.0.1:7200:7200\"]\n environment:\n GDB_JAVA_OPTS: \"-Xmx4g -Xms1g\"\n volumes:\n - ./data/graphdb:/opt/graphdb/home\n restart: unless-stopped\n</code></pre> <p>After first start, create the repository (one-time):</p> <pre><code>curl -X POST http://localhost:7200/rest/repositories \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"id\": \"nexa_knowledge\",\n \"title\": \"Nexa Knowledge Graph\",\n \"type\": \"graphdb\",\n \"params\": {\n \"ruleset\": {\"value\": \"rdfsplus-optimized\"},\n \"baseURL\": {\"value\": \"https://nuclide.systems/nexa/\"}\n }\n }'\n</code></pre> <p>Optional Zoraxy entry <code>graph.nuclide.systems</code> \u2192 <code>192.168.1.40:7200</code> if you want the SPARQL Workbench in a browser; otherwise n8n talks to it on the docker network at <code>http://nexa-graphdb:7200</code>.</p> <p>For schema and example queries: 08-graphrag-architecture.</p>"},{"location":"stacks/nexa/docs/09-deployment/#phase-add-on-visual-collection-phase-32","title":"Phase add-on: visual collection (Phase 3.2)","text":"<p>Adds Path C \u2014 image embeddings without disturbing the text path. Schema is already in <code>nexa-core/config/qdrant_schema_visual.json</code>.</p> <pre><code># (1) replace TEI with infinity (or run alongside) for CLIP-family support\ndocker rm -f nexa-embed\ndocker run -d --name nexa-embed \\\n --restart unless-stopped \\\n -p 127.0.0.1:8080:80 \\\n -v infinity-data:/app/.cache \\\n michaelf34/infinity:latest \\\n v2 \\\n --model-id BAAI/bge-m3 \\\n --model-id jinaai/jina-clip-v2 \\\n --port 80\n\n# (2) create the visual collection\ncurl -X PUT \"$QDRANT_HOST/collections/nexa_knowledge_visual\" \\\n -H \"Content-Type: application/json\" \\\n -H \"api-key: $QDRANT_API_KEY\" \\\n -d @nexa-core/config/qdrant_schema_visual.json\n\n# (3) register the second model in LiteLLM as `nexa-embed-visual`\n# (same OpenAI-compatible route, different model id)\n\n# (4) backfill queued images:\n# SPARQL: SELECT ?note ?uri WHERE { ?note nexa:pendingVisualIndex true ; nexa:mediaUri ?uri }\n# For each row: fetch the bytes, embed via nexa-embed-visual, upsert into the visual collection,\n# UPDATE GraphDB to set nexa:vectorId and DELETE nexa:pendingVisualIndex.\n</code></pre> <p>n8n RAG workflow gains a parallel branch: text-query \u2192 both <code>nexa-embed-text</code> and <code>nexa-embed-visual</code> text encoders \u2192 kNN against both collections \u2192 merge by score before SAIA prompt.</p>"},{"location":"stacks/nexa/docs/09-deployment/#step-back-rollback","title":"Step-back / rollback","text":"<ul> <li>Disable any Nexa workflow in n8n \u2014 deactivates the side effect immediately, Memos webhooks become no-ops.</li> <li>Drop a Qdrant collection \u2014 <code>curl -X DELETE $QDRANT_HOST/collections/nexa_knowledge_text -H \"api-key: $QDRANT_API_KEY\"</code>.</li> <li>Re-discover \u2014 <code>#nexa:reset-config</code> then <code>#nexa:config</code>.</li> </ul>"},{"location":"stacks/nexa/docs/10-operations/","title":"10 \u2014 Operations","text":"<p>Day-2 concerns. Backups, monitoring, troubleshooting.</p>"},{"location":"stacks/nexa/docs/10-operations/#backups","title":"Backups","text":"<p>Tiering (3-2-1, full design in 12 #37): 1. Source \u2014 UNAS RAID 6 + native UniFi Drive snapshots (12 #38, still to enable). 2. Warm tier \u2014 <code>s3.nuclide.systems</code> (on-site), restic/borg via Backrest (LXC 103). 3. Cold tier off-site \u2014 encrypted <code>rclone</code> copy to a third-party (Jottacloud is the user's candidate; decision tracked in Q20). Always encrypt before upload \u2014 provider sees only ciphertext.</p> <p>Nexa-specific items inside this pipeline:</p> What How Frequency n8n workflow JSON <code>nexa-core/scripts/backup_workflows.sh</code> \u2192 git push hourly cron in n8n container Qdrant <code>nexa_knowledge_text</code> (and <code>_visual</code> once Phase 3.2) <code>POST /collections/<name>/snapshots</code> \u2192 write to <code>backup/nexa/snapshots/qdrant/<date>/</code> on UNAS, then Backrest picks it up for warm + cold daily (snapshot), weekly (cold tier) GraphDB <code>nexa_knowledge</code> (Phase 3.4+) scheduled SPARQL <code>CONSTRUCT</code> export \u2192 <code>backup/nexa/snapshots/graphdb/<date>.ttl.gz</code> daily Memos DB Backrest snapshot of Memos data dir already covered Nextcloud Nextcloud's own backup app + Backrest of <code>services/nextcloud/</code> already covered <code>runtime_config</code> (Qdrant <code>_config</code> namespace) Qdrant snapshot covers it n/a"},{"location":"stacks/nexa/docs/10-operations/#monitoring","title":"Monitoring","text":"Signal Source Sink Container up/down Dozzle (<code>192.168.1.40:3553</code>) + docker healthchecks. Skip names matching <code>octoprint*</code> \u2014 that container is intentionally powered down most of the time. Memos system feed via ntfy n8n workflow failures n8n built-in failure-webhook ntfy \u2192 <code>nexa.system</code> topic Proxmox / disk / memory alerts Proxmox notification target \u2192 ntfy ntfy \u2192 Memos digest LiteLLM rate-limit LiteLLM logs / cost-tracking Memos morning digest Backrest run status Backrest webhook ntfy <p>A single ntfy topic <code>nexa.system</code> is the convention; n8n has one workflow that re-broadcasts it as a Memos comment under a pinned <code>[SYSTEM]</code> memo.</p>"},{"location":"stacks/nexa/docs/10-operations/#troubleshooting","title":"Troubleshooting","text":""},{"location":"stacks/nexa/docs/10-operations/#memos-webhook-not-firing","title":"Memos webhook not firing","text":"<pre><code># is the Memos webhook config still pointing at n8n?\ncurl -s -H \"Authorization: Bearer $MEMOS_API_KEY\" \\\n https://memos.nuclide.systems/api/v1/workspace/setting | jq '.webhooks'\n\n# does n8n still expose the path?\ncurl -i https://n8n.nuclide.systems/webhook/memos # expect 200/405, never 404\n</code></pre> <p>If 404 \u2192 workflow is inactive in n8n. Activate.</p>"},{"location":"stacks/nexa/docs/10-operations/#litellm-401-429","title":"LiteLLM 401 / 429","text":"<ul> <li>401 \u2192 the Nexa virtual key was rotated. Reissue, update <code>.env</code>, re-create the n8n credential.</li> <li>429 \u2192 set per-key TPM/RPM limits in LiteLLM admin; classification calls are tiny (<300 tok), embeddings are the bulk.</li> </ul>"},{"location":"stacks/nexa/docs/10-operations/#qdrant-nexa_knowledge-empty","title":"Qdrant <code>nexa_knowledge</code> empty","text":"<pre><code># is the indexing workflow active?\ndocker logs nexa-n8n 2>&1 | grep memos_bridge | tail\n\n# manual index test\ncurl -X POST https://memos.nuclide.systems/api/v1/memos \\\n -H \"Authorization: Bearer $MEMOS_API_KEY\" \\\n -d '{\"content\":\"manual probe #nexa\"}'\n\n# point count\ncurl -s -H \"api-key: $QDRANT_API_KEY\" \\\n $QDRANT_HOST/collections/nexa_knowledge | jq '.result.points_count'\n</code></pre>"},{"location":"stacks/nexa/docs/10-operations/#wrong-list-routing-work-vs-personal","title":"Wrong list routing (Work vs Personal)","text":"<ol> <li>Run <code>#nexa:route-test <text></code> and check the confidence value.</li> <li>If <0.7 the router defaults to Personal (by design \u2014 see 06).</li> <li>Tune the system prompt in <code>nexa-core/ai-prompts/system_prime.txt</code>; commit the change.</li> </ol>"},{"location":"stacks/nexa/docs/10-operations/#where-did-nexa-store-this","title":"\"Where did Nexa store this?\"","text":"<pre><code># state dump\ncurl -X POST https://memos.nuclide.systems/api/v1/memos \\\n -d '{\"content\":\"#nexa:export-state\"}'\n# returns runtime_config + counters as a JSON memo\n</code></pre>"},{"location":"stacks/nexa/docs/10-operations/#log-locations","title":"Log locations","text":"Component Where Memos <code>docker logs nexa-memos</code> (LXC 104) n8n <code>docker logs nexa-n8n</code> LiteLLM <code>docker logs litellm</code> Qdrant <code>docker logs qdrant_scientific</code> Zoraxy LXC 108 web UI \u2192 Statistical Analysis Backrest LXC 103 web UI <p>For a one-shot dump:</p> <pre><code>ssh nuc 'docker compose -p nexa logs --tail 500' > /tmp/nexa.log\n</code></pre>"},{"location":"stacks/nexa/docs/11-open-questions/","title":"11 \u2014 Open Questions (user-info-required)","text":"<p>Items that block progress and need a human decision before a workflow can be implemented or a service deployed. Tick them off as you decide.</p>"},{"location":"stacks/nexa/docs/11-open-questions/#resolved","title":"Resolved","text":"<ul> <li>[x] Q1 \u2014 Graph DB choice \u2192 Ontotext GraphDB (SPARQL). Rationale: explore Nexa's memory through SPARQL is a stated goal. 08-graphrag-architecture is rewritten accordingly.</li> <li>[x] Q2 \u2014 Vector store \u2192 reuse <code>qdrant_scientific</code> with a <code>nexa_*</code> collection prefix. No dedicated container.</li> <li>[x] Q3 \u2014 Embeddings model \u2192 not OpenAI. Self-host on the docker host via TEI (HF <code>text-embeddings-inference</code>) \u2014 Rust single-binary, OpenAI-compatible, ~500 MB image, no LLM runtime overhead. Speed analysis in \u00a7\"Speed budget\" below.</li> <li>[x] Q5 \u2014 Display name \u2192 Karakeep. The Zoraxy host alias <code>hoarder.nuclide.systems</code> is legacy \u2014 keep it for compatibility, but all docs, prompts and new workflow nodes use \"Karakeep\".</li> <li>[x] Q13 \u2014 Octoprint container is intentionally temporary. Suppress from Phase-5 monitoring: container-up/down alerts must skip names matching <code>octoprint*</code> (or any container tagged <code>proxmox-he 3d-printing</code>).</li> <li>[x] Q14 \u2014 Homepage Zoraxy widget \u2192 no such widget. Config error in <code>homepage/services.yaml</code>; cosmetic, not Nexa-related.</li> <li>[x] Q15 \u2014 Embeddings staging plan \u2192 A now, C prepared.</li> <li>Phase 3.1 (now): TEI + <code>BAAI/bge-m3</code>, single collection <code>nexa_knowledge_text</code> (1024-dim). DE/EN multilingual, fits the corpus.</li> <li>Phase 3.2 (later): swap TEI \u2192 <code>infinity</code>, add <code>jinaai/jina-clip-v2</code> (768-dim), second collection <code>nexa_knowledge_visual</code>. Backfill from the queue (see Q16).</li> <li>All schema fields needed for 3.2 (<code>modality</code>, <code>media_uri</code>, <code>graph_iri</code>, <code>nexa:pendingVisualIndex</code>) are introduced now so 3.2 is purely additive \u2014 no rename, no migration. See <code>qdrant_schema.json</code> and <code>qdrant_schema_visual.json</code>.</li> <li>[x] Q16 \u2014 Image-attachment queue ergonomics \u2192 leave bytes at source, reference by <code>media_uri</code>. Zero copy. Memos attachments stay in Memos's data dir, Nextcloud images stay in Nextcloud, Obsidian images stay in the <code>Notizen</code> folder; the Phase-3.2 backfill workflow fetches them on demand via the URI scheme.</li> <li>[x] Q17 \u2014 Immich out-of-band. Nexa does not call the Immich smart-search API. Photo-library queries stay inside Immich.</li> <li>[x] Q18 \u2014 SAIA proxies an embedding model, but 10 msg/min rate limit makes it unusable for ingest. A 2 k-note Obsidian backfill would take ~3.3 h; real-time <code>#nexa:ask</code> would queue for tens of seconds during a writing burst. Decision: deploy TEI as planned. SAIA embeddings remain available as a manual fallback (e.g. for one-off <code>#nexa:learn</code> calls where rate is irrelevant).</li> <li>[x] Q19 \u2014 Storage convention is plain host bind-mount of <code>/mnt/pve/unas/services/<svc>/<vol></code>. Karakeep stack confirmed: no driver opts, no CIFS, no per-volume credentials. Secrets via <code>env_file: .env</code> next to the compose. The earlier SMB-as-docker-volume proposal in this repo is withdrawn (12/#27) \u2014 it was over-fitting to the Nextcloud-specific NFS issue (Nextcloud's setup tooling chowns to <code>www-data</code> and fails on <code>root_squash</code>-style exports; that doesn't apply to normal containers). Nexa stack updated in docs/09 \u00a7Step 3.</li> <li>[x] Q4 \u2014 Obsidian sync via Nextcloud WebDAV. Confirmed the vault is <code>nc.nuclide.systems/Notizen/</code> (multi-device sync via Nextcloud client). Nexa accesses it through WebDAV (<code>/remote.php/dav/files/<user>/Notizen/</code>) reusing the existing <code>NC_APP_PASSWORD</code> \u2014 no filesystem mount, no LXC-to-LXC privilege escalation. Phase 3.1 polls every 15 min; an upgrade to Nextcloud's <code>notify_push</code> for sub-second updates is captured as optimization #15. Ignore list (don't index):</li> <li><code>.copilot/</code>, <code>.copilot-index/</code> \u2014 Obsidian Copilot's own embeddings cache.</li> <li><code>.smart-env/</code> \u2014 Smart Connections / Smart Composer plugin data (~13 MB).</li> <li><code>.caldav-sync/</code> \u2014 calendar sync, not notes.</li> <li><code>assets/</code> \u2014 186 MB of binaries; routed through the Phase-3.2 visual queue (<code>nexa:pendingVisualIndex</code>), not the text path.</li> <li><code>Templates/</code> \u2014 empty templates, low semantic value.</li> <li><code>BMO/</code>, <code>Excalidraw/</code> \u2014 plugin folders. Anything else under <code>Notizen/**/*.md</code> is fair game.</li> <li>[x] Q6 + Q7 \u2014 Lists & calendars discovered by name, not ID. Self-healing. Confirmed names from screenshots:</li> <li>Tasks: <code>Pers\u00f6nlich</code> (14), <code>DLR</code> (13), <code>Einkaufsliste</code> (13), <code>Wunschliste</code> (9).</li> <li>Calendars: <code>Pers\u00f6nlich</code>, <code>DLR</code>, <code>Einkaufsliste</code>, <code>Wunschliste</code> (Nextcloud Tasks is calendar-backed, so the names are shared; Tasks lives on the calendar of the same name).</li> <li>Mapping in classification: <code>DLR</code> = Work-Kontext, <code>Pers\u00f6nlich</code> = Personal-Kontext, <code>Einkaufsliste</code> = Shopping, <code>Wunschliste</code> = Wishes.</li> </ul> <p>Self-healing requirement (the user explicitly noted lists may change/grow): Nexa must not cache IDs forever. The <code>#nexa:config</code> workflow runs (a) on demand, (b) once daily as a scheduled refresh, and (c) automatically as a retry whenever a list/calendar lookup returns 404 or \"not found\". The runtime-config record in Qdrant's <code>_config</code> namespace stores <code>{name \u2192 id, discovered_at}</code> and gets invalidated on cache-miss. New lists added in Nextcloud surface in the next scheduled refresh and Nexa starts honouring <code>#einkaufsliste</code> / <code>#dlr</code> etc. without code changes.</p> <ul> <li> <p>[x] Q8 \u2014 Mail via Nextcloud, single account <code>fkrebs@nucli.de</code>. No extra IMAP entry. n8n's IMAP node uses the same server credentials Nextcloud Mail already holds for that account; Nexa never sees a second password. Folders confirmed: <code>Posteingang</code> (default), <code>Archiv</code>, <code>Junk</code> (61 \u2014 auto-filtered, ignored by Nexa), <code>Papierkorb</code>, <code>Waiting</code>. The <code>Waiting</code> folder is a useful manual signal \u2014 items moved there by the user are skipped from digest (treat as \"in flight\").</p> </li> <li> <p>[x] Q9 \u2014 Pocket-ID SSO is configured everywhere. No separate auth for Nexa: Memos / n8n / future Nexa dashboard authenticate humans via Pocket-ID at the Zoraxy layer. Machine-to-machine calls (n8n \u2192 Memos webhook, n8n \u2192 Nextcloud, n8n \u2192 Qdrant, n8n \u2192 LiteLLM) keep using API keys / app passwords \u2014 SSO is for human UIs only. Don't add basic-auth or per-app login screens.</p> </li> </ul> <p>This resolves Q10 too (the proposed \"Pocket-ID SSO in front of n8n / Memos\" is already done; Nexa just inherits it).</p> <ul> <li>[x] Q10 \u2014 folded into Q9.</li> </ul>"},{"location":"stacks/nexa/docs/11-open-questions/#hardware-capacity","title":"Hardware / capacity","text":"<ul> <li>[x] Q11 \u2014 Capacity confirmed from Proxmox node + LXC summaries.</li> <li>Host (NUC 14 Pro): 22 threads (Intel Core Ultra 7 155H, 1 socket), 62.12 GiB RAM, 1.64 TiB disk (0.35% used). Steady state: 32.4 GiB used (\u224824 GiB of which is ZFS ARC), load avg 1.99 / 1.39 / 1.15, IO delay 0.04%.</li> <li>LXC 104 (docker, unprivileged): 16 CPU, 31.25 GiB RAM cap (7.86 GiB used = 25%), 8 GiB swap (idle), 200 GiB boot disk at 47.7% used (~95 GiB). Tag <code>proxmox-helper-scripts</code>. Intel iGPU passthrough configured but currently failing \u2014 see 12/#26.</li> <li>RAM headroom is comfortable: Phase-3 budget (~7 GiB additional \u2014 Qdrant ~1.5 + TEI/bge-m3 ~1.1 + GraphDB 4-GB heap + later +infinity/jina-clip-v2 ~1) fits inside the existing LXC cap with ~16 GiB still free. Levers if needed: (1) raise the LXC cap (host has plenty), (2) cap <code>zfs_arc_max</code> lower (ARC currently 24 GiB). Either is one line of config.</li> <li>Disk is fine \u2014 the LXC also NFS-mounts the UNAS at <code>/mnt/pve/unas</code> (19.4 TiB total, 16.3 TiB free). Large/persistent volumes (Qdrant data, GraphDB repo, snapshots, image bytes if we ever stage them) bind-mount there; the 200 GiB local boot disk only carries images and small ephemeral state. See 12/#27.</li> <li>Verdict: no blockers for Phase-3.1. Revisit once 3.4 GraphDB lands.</li> </ul>"},{"location":"stacks/nexa/docs/11-open-questions/#backups-deferred","title":"Backups (deferred)","text":"<p>Backup-tier decisions are intentionally parked while Phase-3.1/3.2/3.4 are built. UNAS RAID 6 + Backrest already cover file-level recovery; pool-snapshot configuration (12/#38) remains the single highest-leverage data-protection action and doesn't depend on either question below. Re-open both when Phase 3.3 becomes the next-up item.</p> <ul> <li>[ ] Q12 \u2014 deferred. S3 archive bucket on <code>s3.nuclide.systems</code> (warm tier). Pick when Phase 3.3 is up next.</li> <li>[ ] Q20 \u2014 deferred. Off-site cold-tier provider (Jottacloud is the leading candidate). Pick when Phase 3.3 is up next.</li> </ul>"},{"location":"stacks/nexa/docs/11-open-questions/#speed-budget-q3-follow-up","title":"Speed budget (Q3 follow-up)","text":"<p>Workload measured against the LXC 104 cap (16 CPU, 31.25 GiB RAM \u2014 host has 22 threads / 62 GiB if we ever raise the cap):</p> Task Volume Latency target Achievable on CPU with <code>bge-m3</code> Achievable with <code>nomic-embed-text</code> Real-time memo embed 1 doc <500 ms incl. n8n round-trip \u2705 ~50\u2013100 ms \u2705 ~20 ms Daily ingest ~70 docs <60 s \u2705 ~5\u201310 s \u2705 ~2 s Obsidian backfill (one-shot) ~2 000 docs <15 min \u2705 ~2\u20134 min \u2705 <1 min RAG query embed (<code>#nexa:ask</code>) 1 doc <300 ms \u2705 ~50 ms \u2705 ~20 ms <p>Conclusion: CPU-only TEI is sufficient \u2014 no GPU needed for current scope. Bottleneck is SAIA chat (already remote), not embeddings. SAIA's own embedding endpoint is rate-limited to 10 req/min which would block real-time embed; self-hosted TEI side-steps that completely.</p>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/","title":"12 \u2014 Optimization Opportunities","text":"<p>Observations from the running infrastructure. Each item is independent \u2014 accept, defer, or reject.</p>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/#for-nexa-directly","title":"For Nexa directly","text":"<ol> <li>Reuse, don't redeploy. The earlier <code>DEPLOYMENT.md</code> would have spun a second Memos / n8n / Qdrant. The current homelab already runs all three. The new 09-deployment treats these as pre-existing \u2014 keeps the config minimal and avoids port collisions.</li> <li>Use LiteLLM virtual keys per logical caller. Today there's one SAIA key. Issuing one key per workflow (<code>nexa-router</code>, <code>nexa-embed</code>, <code>nexa-digest</code>) lets you set different per-key rate/cost limits and disable a single workflow without rotating everything.</li> <li>Use n8n's credential objects, never inline secrets. The current workflows under <code>nexa-core/n8n-workflows/phase-1/*.json</code> should be reviewed \u2014 if any header <code>Authorization</code> is hardcoded, replace with credential references before importing.</li> <li>Centralise system alerts on a single ntfy topic (<code>nexa.system</code>). Backrest, Proxmox notifications, n8n failure-webhook and the Octoprint Exited state all go to that topic; one Memos system memo aggregates them.</li> <li>Defer graph DB until Phase 3.4. Qdrant alone covers ~80% of the assistant's daily value. The graph DB is justified once you actually need dependency analysis or critical-path queries.</li> <li>Auto-export n8n workflows. <code>nexa-core/scripts/backup_workflows.sh</code> already exists. Schedule it inside the n8n container (cron) and let it <code>git commit && git push</code> \u2014 this is the cheapest disaster recovery.</li> </ol>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/#for-the-wider-homelab-out-of-scope-but-worth-noting","title":"For the wider homelab (out of scope but worth noting)","text":"<ol> <li>AI gateway naming. <code>ai.nuclide.systems</code> currently proxies LobeHub (a chat UI on <code>:3210</code>), while the LiteLLM API lives on <code>:4000</code>. For Nexa, point n8n directly at LiteLLM (<code>http://192.168.1.40:4000</code> over the docker net \u2014 no public TLS hop needed) to save latency and isolate from UI restarts.</li> <li>MCP servers consolidation. Dozzle shows <code>crawl4ai-mcp</code>, <code>markitdown-mcp</code>, <code>papersearch-mcp</code> running individually. They're all MCP servers \u2014 Nexa Phase-3 could pull from these via LiteLLM's MCP support to enrich the embedding pipeline (e.g. fetch + markitdown a Karakeep link before embedding).</li> <li>Backup the n8n SQLite file \u2014 Backrest covers <code>/home/node/.n8n</code> if added; today the only \"backup\" is the workflow JSON which omits credentials and execution history.</li> <li>Pocket-ID SSO in front of n8n would let you remove n8n basic-auth and unify session management across the whole stack. One-time setup, large UX win.</li> <li>Vaultwarden as the secret store for Nexa secrets (<code>SAIA_API_KEY</code>, <code>MEMOS_API_KEY</code>, \u2026) \u2014 read at bootstrap via the Bitwarden CLI from inside the docker host. Removes the need for a <code>.env</code> on disk.</li> <li>AdGuard as DNS-based control plane. Since AdGuard is the resolver for the LAN, you can rewrite <code>*.nuclide.systems</code> to <code>192.168.1.4</code> (Zoraxy) internally and avoid a hairpin via the WAN \u2014 already the case if AdGuard rewrite rules are set, worth verifying.</li> <li>Disk usage on LXC 104 is 47.7 % (Proxmox). Monitor; n8n execution logs and Dozzle history are the usual culprits. Setting <code>EXECUTIONS_DATA_PRUNE=true</code> and <code>EXECUTIONS_DATA_MAX_AGE=168</code> (7 days) on n8n keeps it bounded.</li> <li> <p>Vector-store sprawl in the homelab \u2014 three competing indexes today. Nexa is about to be the fourth. Track for eventual consolidation:</p> <ul> <li>Obsidian Copilot (<code>.copilot</code>, <code>.copilot-index</code>) and Smart Connections / Smart Composer (<code>.smart-env</code>, ~13 MB) \u2014 embed the Obsidian vault into two separate vector stores inside the vault.</li> <li>Paperless-AI ChromaDB (<code>services/paperless-ai/chromadb/</code>) \u2014 embeds scanned documents from Paperless-ngx for Q&A.</li> <li>Nexa Qdrant (<code>nexa_knowledge_text</code>, soon <code>_visual</code>) \u2014 embeds the cross-source corpus.</li> </ul> <p>Long-term, Nexa is the natural single source of truth (it sees memos + mail + obsidian + RDF graph). Once its RAG is satisfying, retire the Obsidian-plugin indexes and consider letting Nexa read from Paperless-AI's ChromaDB rather than re-embed PDFs (one-line ChromaDB query, much cheaper than redoing OCR-to-vector). Track but don't act yet.</p> </li> <li> <p>Retire <code>open-webui</code>. Confirmed stale by the user \u2014 only LobeHub is in active use as the LiteLLM chat front-end (<code>ai.nuclide.systems</code>). Stop the container, tar <code>services/open-webui/</code> into <code>backup/open-webui/</code>, then remove the stack. Frees ~500 MB RAM + a couple of GB of model cache.</p> </li> <li> <p><code>services/siyuan/workspace/</code> is dead data. SiYuan retired, content migrated to Obsidian (Nextcloud <code>Notizen/</code>). Keep a final tar in <code>backup/siyuan/</code>, then <code>rm -rf services/siyuan/</code>. Frees disk + removes a \"is this still authoritative?\" question for future agents (and for Nexa's classifier if it ever sees the path).</p> </li> <li>Real-time Obsidian sync via <code>notify_push</code>. Phase 3.1 polls WebDAV every 15 min (Q4 resolution). Once that works, swap to Nextcloud's <code>notify_push</code> app for sub-second propagation. One-line workflow change in n8n.</li> <li><code>assets/</code> is 186 MB of binaries in the Obsidian vault \u2014 worth a glance to confirm it's mostly images (Phase-3.2 visual queue) rather than something that should live in Nextcloud Files proper.</li> <li>Paperless-AI as a Phase-2.x triage helper for <code>_sortMe/Downloads/</code>. UNAS shows ~230 PDFs/docs in <code>_sortMe/Downloads/</code> plus another batch under <code>_sortMe/Anne/</code>. Paperless-AI (already running) can ingest, OCR, classify and route them; Nexa's role is to delegate \u2014 wire a workflow that posts a batch to Paperless-AI and reports the result back via Memos.</li> <li><code>media/Recipes/</code> has ~300 individually-named recipe folders. Once Phase-3.1 text indexing works, this becomes a high-quality test corpus for <code>#nexa:ask</code> (e.g. \"what was that gochujang noodle recipe with no anchovies?\"). Out-of-scope for the deployment plan, but a satisfying first user-facing win.</li> </ol>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/#proxmox-host-nuc-14-pro-tuning","title":"Proxmox host (NUC 14 Pro) tuning","text":"<p>Observed from the node summary: 22 threads, 62 GiB RAM (32 GiB used, ~24 GiB of which is ZFS ARC), 1.64 TiB disk (0.35% used), load avg <2.0, IO delay 0.04%, kernel 6.17.13-4-pve, PVE 9.1.9, EFI. Suggestions in priority order:</p> <ol> <li>Cap ZFS ARC. Default is 50% of RAM (~31 GiB); current actual ~24 GiB. For a node that runs services rather than a pure storage box, capping at 8\u201312 GiB frees ~12\u201316 GiB for guests without measurable IO impact (disk is 1.64 TiB and 0.35% used \u2014 there's nothing hot to cache): <pre><code>echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf # 8 GiB\nupdate-initramfs -u\n</code></pre> Reboot or <code>echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max</code> to apply live.</li> <li>Enable KSM (Kernel Same-page Merging). With ~40 docker containers + several LXCs, KSM typically frees 1\u20133 GiB by deduplicating identical memory pages. Currently <code>KSM sharing: 0 B</code> in the summary. PVE has <code>ksmtuned</code> available \u2014 <code>systemctl enable --now ksmtuned</code>.</li> <li>Suppress the <code>pve-no-subscription</code> repository warning \u2014 either accept it (it's a homelab) and apply the <code>pve-no-subscription-warning</code> polyfill, or move to the enterprise repo. Pure cosmetic, but the orange banner in the UI is noise.</li> <li>Swap is 31 GiB on a 62 GiB box with ZFS root \u2014 almost certainly oversized. Drop <code>vm.swappiness</code> to 10 (<code>sysctl -w vm.swappiness=10</code> + persist) so swap is only used under genuine pressure, and consider shrinking the swap volume if disk-layout permits.</li> <li>Verify scheduled ZFS scrub is enabled. PVE ships <code>zfs-scrub-monthly@.timer</code> \u2014 <code>systemctl list-timers | grep zfs</code> to confirm. Cheap insurance on a 1.6 TiB pool.</li> <li>SMART monitoring on the NVMe. <code>smartctl -a /dev/nvme0</code> should be regularly polled; PVE's notification target can ntfy on degradation. Combine with the existing <code>nexa.system</code> ntfy topic (optimization #4) so disk-health alerts land in the same Memos system feed as everything else.</li> <li>NTP source via AdGuard. AdGuard already resolves DNS for the LAN; pointing the host's <code>systemd-timesyncd</code> at <code>pool.ntp.org</code> resolved through AdGuard avoids any external dependency for time. One-line change in <code>/etc/systemd/timesyncd.conf</code>.</li> <li><code>fstrim.timer</code> enabled for the SSD pool \u2014 verify with <code>systemctl status fstrim.timer</code>. Default-on in modern PVE, but quick to confirm.</li> <li>Watchdog config is irrelevant for a single-node setup (HA is the use case), so leave the default. Mentioned only so future agents don't add it speculatively.</li> </ol>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/#docker-lxc-104-observations-wins","title":"Docker LXC (104) \u2014 observations & wins","text":"<p>Confirmed allocation: 16 CPU, 31.25 GiB RAM (7.86 GiB used / 25%), 8 GiB swap (idle), 200 GiB boot disk at 47.7% used \u2014 disk pressure outranks RAM pressure.</p> <ol> <li>Fix the Intel iGPU passthrough. The container's own notes flag the binding as \"likely failing\". Once <code>/dev/dri/{card0,renderD128}</code> is visible inside the LXC, both TEI and <code>infinity</code> can run embeddings on the Arc iGPU via OpenVINO / IPEX-LLM \u2014 typically 5\u201310\u00d7 faster than CPU. Equally, Immich's CLIP can be GPU-accelerated. The required config in <code>/etc/pve/lxc/104.conf</code>: <pre><code>lxc.cgroup2.devices.allow: c 226:0 rwm\nlxc.cgroup2.devices.allow: c 226:128 rwm\nlxc.cgroup2.devices.allow: c 29:0 rwm\nlxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file\nlxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file\nlxc.idmap: u 0 100000 65536\nlxc.idmap: g 0 100000 65536\nlxc.idmap: g 44 44 1 # video group on host\nlxc.idmap: g 104 104 1 # render group on host\n</code></pre> Then inside the container: <code>usermod -aG video,render <docker-user></code> and run TEI with <code>--device cuda</code> replaced by the OpenVINO build (<code>text-embeddings-inference:cpu-1.5-openvino</code>). Not needed for Phase-3.1's volume but a clean upgrade path.</li> <li> <p>Match the homelab-wide UNAS storage convention \u2014 host bind-mount of <code>/mnt/pve/unas/services/<svc>/<vol></code>. Verified via the running Karakeep stack (Q19): every container in LXC 104 binds its persistent data into <code>/mnt/pve/unas/services/<svc>/...</code> directly, no <code>driver_opts</code>, no CIFS, no credentials in compose. Nexa follows the same pattern. Targets:</p> <ul> <li><code>/mnt/pve/unas/services/nexa/qdrant/</code> \u2014 Qdrant data dir.</li> <li><code>/mnt/pve/unas/services/nexa/tei-cache/</code> \u2014 embedding model cache (keeps multi-GB models off the boot disk).</li> <li><code>/mnt/pve/unas/services/nexa/graphdb/</code> \u2014 Phase-3.4 RDF store.</li> <li><code>/mnt/pve/unas/backup/nexa/snapshots/qdrant/<date>/</code> \u2014 daily Qdrant snapshots before rsync to S3 (Phase 3.3 \u2014 Q12 open). Mirrors the <code>backup/<svc>/</code> pattern used by Home Assistant / Immich.</li> <li>n8n executions retention: <code>EXECUTIONS_DATA_PRUNE=true</code>, <code>EXECUTIONS_DATA_MAX_AGE=168</code>.</li> <li>Monthly <code>docker image prune --all --filter \"until=720h\"</code> to clear dangling layers.</li> </ul> <p>Pre-deploy step: <code>mkdir -p /mnt/pve/unas/services/nexa/{qdrant,tei-cache,graphdb}</code> and <code>mkdir -p /mnt/pve/unas/backup/nexa/snapshots/{qdrant,graphdb}</code> on the docker LXC. Then the compose volumes block is just <code>- /mnt/pve/unas/services/nexa/<vol>:/<container-path></code>. Concrete example in 09-deployment \u00a7Step 3.</p> <p>Secrets go in the stack's <code>.env</code> next to the compose (Karakeep convention: <code>env_file: .env</code>). Vaultwarden is the long-term source-of-truth for those secrets (12/#11) but isn't required day-one. The earlier proposal of mounting SMB as a docker volume is withdrawn (12/#27) \u2014 it was an over-fitting to the Nextcloud-specific NFS issue. 32. The LXC has its own 8 GiB swap. Combined with the host's 31 GiB, that's a lot of swap for guests that should never page. Drop the LXC swap allocation to 1\u20132 GiB (<code>pct set 104 -swap 2048</code>) \u2014 frees disk on the LVM-thin pool and forces issues to surface earlier rather than silently swap. 33. Stacks are managed in Arcane; Dockge is stale. The LXC was originally provisioned with the Dockge helper-script template, but the user moved on to Arcane as the day-to-day docker manager. Deployment of the Nexa stack therefore goes through Arcane, not Dockge. Cleanup task: tar <code>services/dockge/</code> (if it exists) into <code>backup/dockge/</code>, retire the Dockge container, drop the stale data dir. Dozzle stays \u2014 it's the log viewer, not a manager, so it isn't redundant with Arcane. 34. Untriaged local mail on the LXC. Console shows <code>You have new mail.</code> at login \u2014 the system mail spool on <code>/var/mail/root</code> has unread messages, almost always cron job failures. <code>mailx</code> or <code>mutt</code> to inspect, then either fix the failing job or send the spool to <code>nexa.system</code> ntfy via a tiny aliases entry (<code>root: |/usr/local/bin/spool-to-ntfy.sh</code>).</p> </li> </ol>"},{"location":"stacks/nexa/docs/12-optimization-opportunities/#housekeeping-campaign-cross-cutting-do-together","title":"Housekeeping campaign (cross-cutting, do together)","text":"<p>These three are inter-related \u2014 picking them up as one campaign is cheaper than chasing each individually, because the audit step is the same. They also map cleanly onto Phase 6 (Nexa as homelab steward): once Nexa can poll Arcane, diff against the documented state and emit actionable findings, items #35\u201337 become semi-automatic \u2014 Nexa proposes the migrations rather than us hunting them down.</p> <ol> <li> <p>Consolidate Postgres instances. Today there are at least four independent Postgres containers running \u2014 visible from Arcane: <code>immich_postgres</code>, <code>lobe-postgres</code>, <code>litellm_db</code>, <code>paperless-ngx-db-1</code> (Karakeep uses Meilisearch + maybe SQLite, separate). Each idles around 100\u2013300 MB RAM and has its own backup story. Two paths:</p> <ul> <li>Single shared cluster (<code>pgsql</code> container with one role per app, one database per app). Most modern apps support <code>DATABASE_URL</code> \u2192 just point them at the shared instance. Saves ~600 MB\u20131 GB RAM and consolidates backups to one pg_dump cron.</li> <li>Or migrate the smallest ones to SQLite (LiteLLM has SQLite mode; Paperless does too) and keep Postgres only for Immich. Same outcome, less migration risk.</li> </ul> <p>Pre-step: list every running container with <code>docker ps --format '{{.Names}}\\t{{.Image}}' | grep -i 'postgres\\|mariadb\\|mysql'</code> to inventory exactly what's running.</p> </li> <li> <p>UNAS-integration audit. <code>/mnt/pve/unas/services/<svc>/</code> is the homelab convention (host bind-mount, no driver opts \u2014 see #31). Not every container follows it yet. Walk every Arcane stack and check the <code>volumes:</code> block: any <code>- /var/lib/docker/...</code> or anonymous-volume entry is non-compliant. Suspected non-compliant (need verification):</p> <ul> <li><code>qdrant_scientific</code> \u2014 vector data possibly on local boot disk; critical to verify before Phase-3.1 piles on a <code>nexa_knowledge_text</code> collection.</li> <li><code>litellm_db</code>, <code>lobe-postgres</code>, <code>lobe-redis</code> \u2014 DB containers, persistent state.</li> <li><code>memos</code> \u2014 the user-facing notes app; loss = data loss.</li> <li><code>n8n</code> \u2014 workflows + executions DB.</li> <li><code>audiobookshelf</code> \u2014 listening progress + library metadata.</li> <li>the <code>*-mcp</code> containers \u2014 likely stateless (cache only), low priority.</li> </ul> <p>Output: a one-page table <code>service | persistent? | currently bound to | should be bound to</code>. Then migrate the non-compliant ones one-by-one (stop \u2192 rsync data to <code>services/<svc>/</code> on UNAS \u2192 re-create stack with the new bind \u2192 verify \u2192 keep the old volume for 7 days as a rollback). Lock the convention for any new stack going forward.</p> </li> <li> <p>3-2-1 backup tiering: warm on-site (S3) + cold off-site. <code>s3.nuclide.systems</code> is already up but it's on-site \u2014 same building, same power, same (in-)susceptibility to fire / theft / ransomware. By itself it's a warm tier, not a disaster-recovery copy. The full design is two tiers:</p> <p>Warm tier \u2014 <code>s3.nuclide.systems</code> (already exists, just needs a bucket): - Restic / borg repos for <code>backup/home-assistant/</code>, <code>backup/immich/</code>, <code>backup/nextcloud/</code>, future <code>backup/nexa/</code>. Backrest already orchestrates Borg \u2014 just add the S3 destination. - Qdrant + GraphDB snapshots flow UNAS \u2192 S3 daily. - Resolves Q12 once the bucket name + access key are set.</p> <p>Cold tier \u2014 off-site provider (decision pending \u2014 see Q20): - Jottacloud \"Unlimited\" (~\u20ac9.50/mo, EU/Norway, soft-cap ~5 TB) \u2014 user's stated candidate. Best price/value at current 2 TB; reach soft cap in ~10 years. Use <code>rclone jottacloud:</code> or native <code>jotta-cli</code>. - Hetzner Storage Box BX21 (\u20ac13/mo, 5 TB EU/DE) \u2014 predictable quota, native Borg/Restic/SFTP. Cheapest predictable EU alternative. - Backblaze B2 (~$12/mo for 2 TB, S3-compatible, US) \u2014 cheapest with the widest tooling support; egress is paid (~$10/TB) which only bites during full restores. - rsync.net (~$30/mo) \u2014 ZFS send/recv natively; overkill unless you want pool replication. - Storj DCS ($4/TB/mo, decentralized, S3-compatible) \u2014 newer ecosystem.</p> <p>Always encrypt before upload regardless of provider \u2014 <code>restic</code> or <code>rclone crypt</code> over the chosen remote. The provider sees only ciphertext blobs. Key material lives in Vaultwarden + a printed-paper offline copy.</p> <p>Tiering & cadence: <pre><code>[Source] UNAS RAID-6 + native snapshots (#38)\n \u2502 daily restic/borg\n \u25bc\n[Warm] s3.nuclide.systems (on-site, fast restore)\n \u2502 weekly rclone copy + crypt\n \u25bc\n[Cold] Off-site (Jottacloud / B2 / Hetzner) \u2190 satisfies the \"1\" in 3-2-1\n</code></pre></p> <p>What goes off-site (priority order): 1. Immich photo originals \u2014 irreplaceable. 2. <code>media/documents/</code>, <code>_sortMe/Downloads/</code>, Nextcloud user data \u2014 irreplaceable. 3. Memos DB, n8n workflows, Nexa Qdrant + GraphDB snapshots \u2014 replicable from sources but expensive to redo. 4. Vaultwarden DB \u2014 cryptographically sensitive but small; explicitly include.</p> <p>What does not need off-site: movies, music, audiobooks, ROMs, derived caches (Immich thumbnails, encoded-video). Those re-download or regenerate.</p> <p>Pre-deploy step: choose one provider, create a test bucket/folder, dry-run a <code>restic init</code> + <code>restic backup</code> of <code>backup/nexa/</code> first as a smoke test before pointing the heavyweight repos at it.</p> </li> <li> <p>Configure UNAS Pool snapshots. The UniFi Drive dashboard shows the storage-pool snapshot schedule as \"Click to Setup\" \u2014 i.e. not configured. RAID 6 protects against drive failure, not against an <code>rm -rf</code> from a misbehaving container or a Nextcloud user mass-delete. Cheap fix: set a daily snapshot with 7-day retention + a weekly with 4-week retention on Pool 1. Native to UniFi Drive, no agent needed. Highest-leverage data-protection change in the homelab right now \u2014 costs nothing, recovers everything.</p> </li> <li> <p>Use UNAS native snapshots for fast Nexa rollback once #38 is set. Nexa workflows that mutate large state (re-embedding the whole vault, graph rebuild, list-config rewrite) can <code>pre-snapshot \u2192 operate \u2192 verify \u2192 release</code> against the same UniFi Drive snapshots. Cheaper and faster than restoring from S3.</p> </li> </ol>"},{"location":"stacks/nexa/docs/13-information-wishlist/","title":"13 \u2014 Information Wishlist","text":"<p>What additional system inventory would sharpen future decisions, packaged as the smallest set of paste-and-run tasks that still answers everything material. Each task is independent \u2014 run any subset, in any order.</p> <p>Each task header tells you exactly where to run it (which shell or which UI). Output goes into a Memos draft, a comment in this conversation, or <code>docs/inventory/<NN>-<slug>.txt</code> \u2014 whichever is easiest.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-1-stack-inventory-reference-compose-done","title":"~~Task 1 \u2014 Stack inventory + reference compose~~ \u2705 DONE","text":"<p>User pasted the Karakeep compose. Q19 resolved \u2192 host bind-mount of <code>/mnt/pve/unas/services/<svc>/<vol></code> is the convention. Nexa stack updated in docs/09 \u00a7Step 3. The <code>docker ps -a</code> half (running-container inventory) is still useful when we get to housekeeping #36 (UNAS-integration audit) \u2014 but that's not blocking now.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-2-storage-map-proxmox-docker","title":"Task 2 \u2014 Storage map (Proxmox + docker)","text":"<p>\ud83d\udccd Where: Two shells \u2014 first the Proxmox host (Datacenter \u2192 <code>nuc</code> \u2192 Shell, or <code>ssh root@192.168.1.20</code>), then back into the LXC 104 console.</p> <p>Tells us which Proxmox storage backs what, and which docker volumes are local vs. SMB.</p> <pre><code># (a) on the Proxmox host (192.168.1.20):\ncat /etc/pve/storage.cfg\ncat /etc/pve/lxc/104.conf\n</code></pre> <pre><code># (b) on LXC 104 (192.168.1.40):\ndocker volume ls\nls -l /dev/dri/ # confirms whether iGPU passthrough actually works\n</code></pre> <p>Unblocks: optimization #30 (iGPU), #36 (UNAS audit), B6/B7/C17.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-3-postgres-db-workload-inventory","title":"Task 3 \u2014 Postgres / DB workload inventory","text":"<p>\ud83d\udccd Where: LXC 104 console (same shell as Task 1).</p> <p>Direct input to housekeeping #35 (consolidate postgres instances).</p> <pre><code>for c in $(docker ps --format '{{.Names}}' | grep -iE 'postgres|mariadb|mysql|_db$'); do\n echo \"=== $c ===\"\n docker exec \"$c\" sh -c 'psql -U postgres -l 2>/dev/null || mysql -e \"show databases\" 2>/dev/null'\ndone\n</code></pre> <p>Unblocks: #35 (consolidate vs. migrate-to-SQLite decision).</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-4-litellm-model-list-for-the-nexa-key","title":"Task 4 \u2014 LiteLLM model list for the Nexa key","text":"<p>\ud83d\udccd Where: Either the LiteLLM admin UI (one screenshot) or any shell with <code>$SAIA_API_KEY</code> exported.</p> <ul> <li>Screenshot path: open <code>https://ai.nuclide.systems</code> (Lobehub) \u2192 Settings \u2192 Model List, filtered to the Nexa virtual key. Screenshot the model rows.</li> <li>CLI path: any shell on your PC or the LXC, with <code>SAIA_API_KEY</code> set: <pre><code>curl -sH \"Authorization: Bearer $SAIA_API_KEY\" \\\n https://ai.nuclide.systems/v1/models | jq '.data[].id'\n</code></pre></li> </ul> <p>Unblocks: Q18 follow-up \u2014 which embedding model SAIA proxies and at what dim, so we can decide if any flow can short-circuit TEI.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-5-s3-archive-credentials","title":"Task 5 \u2014 S3 archive credentials","text":"<p>\ud83d\udccd Where: <code>s3.nuclide.systems</code> admin UI in your browser (it's already proxied via Zoraxy \u2014 same login as the rest of the homelab).</p> <p>Walk to: Buckets \u2192 either pick an existing Nexa-suitable bucket or create one called <code>nexa</code> \u2192 note the bucket name. Then Access Keys \u2192 create a key named <code>nexa-snapshots</code> with read/write on that bucket \u2192 drop the access-key + secret into Vaultwarden under \"Nexa S3\", and reply here with just the bucket name (the secret stays in Vaultwarden).</p> <p>Unblocks: Q12, housekeeping #37 (S3 archive tier), Phase-3.3.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#task-6-reverse-proxy-dns-authority-only-if-needed","title":"Task 6 \u2014 Reverse-proxy + DNS authority (only if needed)","text":"<p>\ud83d\udccd Where: two browser UIs.</p> <ul> <li>Zoraxy (<code>http://192.168.1.4:8000</code>) \u2192 HTTP Proxy \u2192 either click \"Export\" if available, or take a full screenshot of the table.</li> <li>AdGuard (<code>http://192.168.1.20</code> LXC 102 web UI) \u2192 Filters \u2192 DNS Rewrites \u2192 screenshot.</li> </ul> <p>Skip unless we hit a routing surprise during Phase 1.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#when-something-else-is-needed","title":"When something else is needed","text":"<p>The smaller items (n8n credentials list, Memos webhook config, smartctl, sample of <code>_sortMe/</code>, what cron is failing) only matter when we touch that specific area. The agent will ask for them at the moment they're needed, with the same \"\ud83d\udccd Where\" framing.</p>"},{"location":"stacks/nexa/docs/13-information-wishlist/#self-serve-once-phase-6-ships","title":"Self-serve once Phase 6 ships","text":"<p>Once Phase 6 \u2014 Nexa as homelab steward lands, Nexa runs Tasks 1\u20133 itself on a schedule and folds the results into a daily drift report. This wishlist becomes a Nexa-managed surface (<code>#nexa:wishlist-status</code>) rather than something the user has to remember.</p>"}]} |