fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity
112 Commits 2 Branches 0 Tags
main
Commit Graph

63 Commits

Author SHA1 Message Date
fkrebs ae20382cbe ops: sunset LiteLLM, add owui to Backrest, drop stale DB+role 
- LiteLLM sunset 2026-05-28: container removed, config dir renamed
  .DECOMMISSIONED-2026-05-28, prometheus scrape job deleted, CT 113
  litellm DB + role dropped. Bifrost now connects directly to upstream
  LLM providers (verified: openai → SAIA chat-ai.academiccloud.de).
- Open WebUI data (CT 104 uploads + vector_db, 192K) now staged to
  UNAS /services/open-webui via openwebui-prestage.sh and included in
  Backrest services-backup-plan. Open WebUI chats/users live in CT 113
  Postgres owui DB (already in WAL-G).
- Miniflux confirmed DB-only — already covered by WAL-G.
- backrest.md and homelab-architecture.md updated; services-overview.md
  marks LiteLLM row strikethrough.
 2026-05-28 21:47:50 +02:00
fkrebs 9cbd02e3ff docs: 2026-05-28 ops coverage audit + WAL-G monitor fix 
- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
  cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
  Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
  plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
  timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
  to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
  SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
  noted litellm decommissioned but DB retained).
 2026-05-28 21:20:35 +02:00
fkrebs b26796e30c ops: Loki/Alloy fleet enabled; Backrest+WAL-G+Gitea Gotify hooks; Immich WAL fix 
- Alloy now active on CT 102, 103, 104, 109, 101, 105, 113, nuc (mix of
  Docker container + systemd binary); all hosts shipping journal/docker
  logs to Loki on CT 109. CT 103 alloy newly enabled.
- Backrest plans: actionGotify hooks on SUCCESS (prio 3) + ERROR/WARNING
  (prio 8). Gotify app 'Backrest', token AtIMGOYcZsb5HfO.
- WAL-G staleness monitor on CT 103 (hourly): pg_stat_archiver SSH/exec
  against CT 113 + CT 104 immich_postgres. Found Immich WAL push failing
  for 42h — root cause: garage container on bridge network not
  shared_backend (Garage 2026-05-28 incident aftermath). Reconnected to
  shared_backend; image pinned v0.9.4 in compose to match running.
- Gitea daily digest at 08:00 on PVE host nuc — yesterday's commits.
- Closes fkrebs/n8n-flows #2, #5, #6.
 2026-05-28 15:19:40 +02:00
fkrebs c05f38d008 docs: Diun deployed on CT 109; doc-sync Phase 1a/3 actual state 
- Diun (Docker image update notifier) on CT 109, watching CT 104 via
  socket-proxy, Mondays 08:00 → Gotify app "Diun" (token A34U_7k3biVI_po)
- backrest.md: Phase 1a (rclone sync) confirmed ACTIVE since 2026-05-22
  (Saturdays 01:00). Phase 3 (config-to-git) confirmed ACTIVE across
  CT 102 / CT 103 / PVE host; sync-config-repos rebuilt to drop dead
  CT 110/111/112 refs, auto-discover CT 104 stacks, add CT 109 stacks
- Coverage map updated to reflect actual state (no longer "not deployed")
- Open Phase 3 gap: HAOS Git Pull addon — requires HAOS UI install
- services-overview.md: Diun row added; Traefik replaces Zoraxy
 2026-05-28 10:03:39 +02:00
fkrebs f1ed462c91 fix: resolve 7 of 8 backup blind spots (2026-05-28) 
- Vaultwarden data moved to UNAS /services/vaultwarden (CT 104 local → NFS)
- Garage S3 pre-backup rsync hook (garage-prestage.sh) + path added to plan
- n8n pre-backup rsync hook (n8n-prestage.sh) + confirm path covered
- Paperless-ngx media path added to services-backup-plan
- Stale paths (arcane, arr-stack, gluetun) removed from plan
- Repo passwords confirmed rotated (not tapirnase); LiteLLM blind spot N/A
- CT 103 SSH key deployed to CT 104 + CT 109 for prestage hooks
- pocketid-prestage.sh updated from CT 110 (destroyed) → CT 109
- Coverage map updated with new entries
 2026-05-28 01:35:31 +02:00
fkrebs 24c7d56a1b backup: update coverage map and blind spots (2026-05-28) 
- CT 108 Zoraxy decommissioned -> Traefik on CT 109
- Add critical blind spots: Vaultwarden (local FS, not backed up),
  Garage S3 data (WAL-G archive store, not backed up), Paperless media
- Fix services plan path list to match actual Backrest config.json
- Mark stale plan paths: arcane, arr-stack, gluetun
- Note Garage config incident: garage.toml became a directory after
  Portainer redeploy; fixed 2026-05-28 by manual replace + layout apply
 2026-05-28 01:15:29 +02:00
fkrebs 17b6fc6fcb CT 108 decommissioned: Zoraxy replaced by Traefik v3 on CT 109 
Traefik v3 now handles all *.nuclide.systems traffic (ports 80/443).
ACME DNS-01 via Cloudflare, HTTP/2, Keep-Alive fix for Immich.
CT 108 backup: vzdump-lxc-108-2026_05_28-00_46_45.tar.zst on UNAS.
 2026-05-28 00:49:28 +02:00
fkrebs a0f1707e33 decommission Wallabag: remove from portmap, databases, mcp-gateway docs 
Wallabag replaced by crawl4ai in paywall-bypass n8n flow.
Container, DB, MCP server, and Bifrost client all removed 2026-05-28.
 2026-05-28 00:31:13 +02:00
fkrebs 080fe21329 add wallabag: port 17004, DB entry, Bifrost MCP client 2026-05-27 23:39:58 +02:00
fkrebs 643ee2cd36 Add Miniflux: port 17002, OIDC client, DB entry, Zoraxy route; remove Obsidian drift plugins from RESUME 2026-05-27 20:55:08 +02:00
fkrebs 24d99d18fc backup: fix CT 109 coverage gaps; archive dead Gitea repos 
- ops-backup.timer on CT 109: daily 01:30 tars Pocket-ID data +
  Infisical pg_dump → Garage S3 ct109-portainer-backup (ops-*.tar.gz)
  → already offsite via walg-offsite-sync
- backrest.md: coverage map updated for CT 110/111/112 decommission,
  CT 109 Pocket-ID + Infisical now marked covered
- services plan paths: removed services/pocketid (no longer on UNAS)
- Gitea: archived ct110-pocket-id, ct111-dev, mcp-shepard

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 22:41:18 +02:00
fkrebs fc6507abd2 migrate id (Pocket-ID) CT 110 → CT 109; destroy CT 110 
- Pocket-ID now on CT 109 ops at :11000 (was CT 110 :11000)
- Zoraxy id.nuclide.systems → 192.168.1.8:11000
- AdGuard id.nuclide.lan → 192.168.1.8
- Homepage docker.yaml: removed ct110/ct111/ct112 (all destroyed)
- Homepage services.yaml: pocket-id server: my-local
- pocket-id.md: updated host, client table CTs corrected
- portainer.md: CT 110 row struck through
- portmap, ct-inventory, zoraxy all updated
- CT 110 LXC destroyed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 22:17:03 +02:00
fkrebs a2f1d49170 migrate dev→CT104, secrets→CT109; add proton-bridge; sunset CT111/CT112 
- CT 111 (dev): Gitea + Coder + act-runner migrated to CT 104
  - Gitea uses Redis for queue/cache/session (idmapped NFS LevelDB workaround)
  - Zoraxy routes dev/git updated to 192.168.1.40
  - AdGuard dev.nuclide.lan updated to 192.168.1.40
- CT 112 (secrets): Infisical migrated to CT 109
  - DB dump restored; SITE_URL updated to 192.168.1.8:8200
  - AdGuard secrets.nuclide.lan updated to 192.168.1.8
- Proton Mail Bridge deployed on CT 104 (SMTP :1025, IMAP :1143)
- Homepage updated: Dev group → ct104, Infisical → my-local, Background group added
- portmap, ct-inventory, zoraxy, dev-environment, secrets-manager all updated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 22:08:08 +02:00
fkrebs bcbe9e7a88 portainer: add LXC names to agents table, fix CT 105→CT 110 2026-05-26 21:44:01 +02:00
fkrebs 4f17d99f55 homepage: use ops.nuclide.lan (host alias) not homepage.nuclide.lan 2026-05-26 21:38:09 +02:00
fkrebs ff7bacdaf9 Homepage replaces Homarr: portmap, inventory, DNS docs updated (2026-05-26) 2026-05-26 21:32:14 +02:00
fkrebs 4429b37c49 zoraxy: remove arcane.nuclide.systems route (decommissioned 2026-05-26) 2026-05-26 21:13:57 +02:00
fkrebs 03fd23b37b Update pocket-id.md: full client table with IDs, API key, remove Arcane/lobehub 2026-05-26 21:08:51 +02:00
fkrebs 8d73ae3e30 portainer.md: add Observability section (Prometheus + Grafana) 
Prometheus scrapes /api/metrics via X-API-Key, alert rules for
environment health, Grafana dashboard at uid=portainer-be.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 17:23:09 +02:00
fkrebs f65ceef90a Add Portainer BE doc; update backup coverage for CT 109 
- New services/portainer.md: BE license key, agent inventory, backup
  setup (Garage S3 + JottaCloud offsite), OIDC pending steps
- backrest.md: add CT 109 Portainer to coverage map, remove stale
  services/arcane path from plan, note Arcane decommission

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 17:12:36 +02:00
fkrebs 07b21a7e2e docs: add SAIA rate limits, Bifrost governance, and admin auth notes to mcp-gateway.md 
- SAIA rate limits: minute=30/1m active at provider level; hour/day rows exist but not linked
- VK governance: allow_all_keys SQL workaround, allowed_models must be non-null JSON text
- Fix stale sk-tapirnase reference in add-MCP-client example (now uses cookie auth)
- Add LLM provider table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 14:43:50 +02:00
fkrebs 6c401f2376 docs: reconcile sunset of LiteLLM external, mcp-gateway, and LobeChat 
- Bifrost replaces LiteLLM at ai.nuclide.systems (CT 104:14003); LiteLLM
  now internal-only backend at :14000
- mcp-gateway (FastAPI/DinD, mcp.nuclide.systems) decommissioned 2026-05-26;
  MCP now served by Bifrost at ai.nuclide.systems/mcp (29 clients, ~760 tools)
- LobeChat decommissioned 2026-05-26; chat.nuclide.systems now Open WebUI
  (CT 104:14002, stack ai/open-webui.yml)
- Update portmap, volumes, services-overview, homelab-architecture, zoraxy,
  mcp-servers, databases, README accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 13:35:29 +02:00
fkrebs 550b54085c docs: add open-webui VK to mcp-gateway virtual keys table 2026-05-26 13:11:22 +02:00
fkrebs b065c2b849 docs: mark mcp-gateway fully decommissioned 2026-05-26 2026-05-26 13:10:14 +02:00
fkrebs 0600032b75 docs: migrate MCP gateway docs from mcp-gateway to Bifrost 
- mcp-gateway.md: full rewrite for Bifrost at ai.nuclide.systems/mcp;
  29 clients / ~760 tools; VK auth; n8n+shepard blocked on
  streamable-HTTP; legacy gateway decommission checklist added
- connection-hosts.md: Bifrost at port 14003 is now ai.nuclide.systems;
  LiteLLM demoted to internal-only; mcp.nuclide.systems marked pending
  decommission; OIDC client 78c78998 flagged for removal

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 12:55:33 +02:00
fkrebs fbc6469fc4 doc-ingestion: nomic embedding stack, Bifrost semantic cache, GPU passthrough status 2026-05-26 09:49:09 +02:00
fkrebs 14b21199ee docs: ingest-pipeline design doc; update doc-ingestion with nomic service 2026-05-26 09:20:38 +02:00
fkrebs 3d71824295 auto: doc-ingestion: Qdrant+TEI deployed; OWUI RAG config; Bifrost embed models 2026-05-26 08:56:00 +02:00
fkrebs defc788ab5 2026-05-26: LiteLLM decommissioned, consistency sweep complete 
- zoraxy.md: ai.nuclide.systems note updated (LiteLLM decommissioned)
- LiteLLM stopped, commented out of docker-compose.yml
- Paperless-AI, Nextcloud updated to Bifrost VK
- mcp-gateway sync loops disabled
 2026-05-26 08:07:09 +02:00
fkrebs 7e0082379e 2026-05-26: LobeChat→Open WebUI cutover, Bifrost deployed 
- zoraxy.md: chat.nuclide.systems → Open WebUI :14002, ai.nuclide.systems → Bifrost :14003
- config-to-git.md: CT 104 fkrebs/ct104-conf row added
 2026-05-26 07:50:25 +02:00
fkrebs 067e42c25d ops: cut ai.nuclide.systems over to Bifrost (port 14003) 
LiteLLM remains on :14000 for internal services during migration.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-26 07:29:37 +02:00
fkrebs 13bc1ecc12 RESUME + pocket-id: reflect 2026-05-24 → 26 state 
RESUME:
- Backrest 3-for-3 success on 2026-05-26 (video-projects-plan recovered)
- Vault dedup + 234 link rewrites + cleanup
- Plugin audit + Memos Sync + Obsidian Git + streamlined-ui
- Config-to-git fleet now 8 repos
- AdGuard *.nuclide.lan zone (30 A-records)
- 3 new Homarr boards (home/admin/command)
- Pocket-ID LAN callback URLs
- Immich HTTP/2 Keep-Alive diagnosed
- New constraints: use LAN aliases; daily-note merge policy

pocket-id.md: LAN callback URLs added for Homarr/Grafana/Infisical/Proxmox.
Filed earlier as part of 2026-05-24 work, committing now.
 2026-05-26 06:03:28 +02:00
fkrebs 3638e368b1 adguard: document new *.nuclide.lan zone (30 A-records) 
Per-host + per-service LAN-only aliases added 2026-05-24 on CT 102.
Stable names → host IPs; clients use names instead of bare IPs so
service moves only require an AdGuard edit, not N app updates.

Use case driver: Immich Android HTTP/2 Keep-Alive bug — per-SSID
'nuclide' local URL https://immich.nuclide.lan:12000 bypasses Zoraxy.
 2026-05-24 15:42:10 +02:00
fkrebs a422bbfb13 obsidian MCP: document vault paths across CT 104/105/UNAS 2026-05-24 08:51:04 +02:00
fkrebs 829dd8f339 homarr OIDC: correct env var to AUTH_PROVIDERS (plural) 
Homarr v1 ignores singular AUTH_PROVIDER; OIDC button only appears
when AUTH_PROVIDERS=credentials,oidc is set. Fixed live on CT 109.
 2026-05-24 07:47:13 +02:00
fkrebs e160e3dba8 docs: dozzle is LAN-only, clean up route notes 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 19:10:57 +02:00
fkrebs 895658a115 docs: consistency pass 2026-05-23 (session ×4) 
- zoraxy.md: fix arcane upstream to CT109 (192.168.1.8:10002), add
  SkipWebSocketOriginCheck column, note missing routes (dozzle, etc.)
- arcane.md: reflect CT109 migration complete, expand agent table to
  all 8 environments, fix MANAGER_API_URL and DB paths
- ct-inventory.md: CT103 RAM 512→4096+swap; CT109 footnote complete
- proxmox-memory-audit.md: CT103 bump, add CT109+CT113 rows, fix sum
- portmap.md: add Wetty row, fix arcane backend, WAL-G scrape target,
  homepage decommissioned, dozzle LAN-only note
- mcp-gateway.md: add gitea/paperless/proxmox, count 26→29 servers
- RESUME.md: check off WAL-G, Loki, Zoraxy audit; CT109 in key state
  table; session ×4 completed items block

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 16:42:45 +02:00
fkrebs 14137c32a7 docs(mcp): add gitea, paperless, proxmox MCP servers (2026-05-23) 
Wire three new community MCP servers into the gateway on CT104:
- gitea: official Gitea MCP (53 tools), static upstream gitea-mcp:8000
- paperless: @nloui/paperless-mcp via mcp-proxy (12 tools), static upstream
- proxmox: proxmox-mcp-plus PyPI (39 tools), read-only PVEAuditor token

Total: 27 → 30 servers. Skipped: karakeep (API key unset), audiobookshelf
(Go binary only), vaultwarden (master password exposure risk).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 15:02:26 +02:00
fkrebs 10e138b677 docs: document WAL-G monitoring setup on CT113/CT109 
Add WAL-G monitoring section to databases.md covering the textfile
collector script, walg-metrics.timer, node_exporter config, and
Prometheus alert rules added to prevent silent backup stalls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 14:50:39 +02:00
fkrebs f8d8ee3ac0 docs: add Nexa service doc — synopsis, nuclide.systems mapping, phased roadmap 
Covers what Nexa is, which existing services it depends on, what is
genuinely missing (TEI, Qdrant collection, GraphDB), and alternative
tool choices for embeddings, web-search, and the graph pillar.

Repo: https://git.nuclide.systems/fkrebs/nexa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 14:32:02 +02:00
fkrebs 4c97732720 docs: Homarr + Grafana OIDC wired via Pocket-ID API (2026-05-23) 
Client IDs created programmatically via Pocket-ID /api/oidc/clients endpoint.
Env vars injected into both compose files and force-recreated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 09:14:20 +02:00
fkrebs 474af687af docs: 2026-05-23 session — observability stack, Pocket-ID guide, secrets scrub 
- CHANGELOG: full 2026-05-23 backfill (Loki, Alloy×12 hosts, pve-exporter,
  HA prometheus integration, 5 dashboards, 3 alert rules, Homarr, homelab-configs repo)
- services/pocket-id.md: new — OIDC endpoints, client creation walkthrough,
  per-service env var patterns, current client registry, backup notes
- infra/proxmox-state.md: redact D-Link credentials from plaintext (pelican/pinkpanther)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 08:03:01 +02:00
fkrebs b2cb123414 monitoring: document Loki/Alloy/pve-exporter, dashboards and alerts 
CT 109 ops stack now includes Loki (log agg), Grafana Alloy (log agent on
all 12 hosts), prometheus-pve-exporter. 5 Grafana dashboards imported;
3 alert rules wired to Gotify.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 07:03:51 +02:00
fkrebs a8ed4350f0 ops: CT 109 provisioned — Prometheus + Grafana monitoring 
- Debian 13, Docker 29.5, 4c/4G/32G at 192.168.1.8
- Prometheus :9090 (90d retention), Grafana :3000 (LAN only)
- Scrapes: litellm CT104:14000, node-ct104:9100, node-ct109:9100
- CT 104 standalone node-exporter retained at /opt/stacks/monitoring/
- All 4 targets confirmed up

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 03:44:05 +02:00
fkrebs c7529bd867 docs: Nextcloud completion model → qwen3.5-122b-a10b (vision+tools) 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 03:29:06 +02:00
fkrebs 6309286b96 docs: Nextcloud assistant model config — occ method, current assignments 
integration_openai points to LiteLLM at ai.nuclide.systems/v1.
Fixed completion=llama-3.3-70b-instruct, T2I=saia-flux (was devstral).
occ snippet for future model changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 03:10:58 +02:00
fkrebs c3c40dc07b docs: add model management UI, catalog API, and 4K improvements to mcp-gateway 
- New /ui/models page (sortable model catalog + service assignments)
- model_assignments.json: 35-model catalog with latency/cost/capability metadata
- API: GET/PATCH /api/model-assignments
- Model evaluator agent added to agents.json
- ui.html: widen max-width to 1600px, 4K breakpoints at 1920/2560/3840px

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 02:19:28 +02:00
fkrebs 25da11b53a Add connection-hosts and mcp-servers maintained docs 
- infra/connection-hosts.md: every host, LAN IP, port, and public URL
  across all Proxmox guests, Docker services, network infra, and
  external hardware; OIDC client registry; SSH cheat-sheet
- services/mcp-servers.md: all 27 MCP servers with full credentials,
  upstream URLs, transport, group, gateway management API examples,
  and gateway .env token inventory

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-23 01:03:59 +02:00
fkrebs 396aa0e48e docs: remove claude high-effort variants, set temperature=0.7 on Claude models (58 total) 2026-05-23 00:41:26 +02:00
fkrebs 9ab9033ab0 docs: fix voxtral-mini proxy status + remove realtime model entry 2026-05-23 00:26:32 +02:00