11 Commits

Author SHA1 Message Date
fkrebs ae20382cbe ops: sunset LiteLLM, add owui to Backrest, drop stale DB+role
- LiteLLM sunset 2026-05-28: container removed, config dir renamed
  .DECOMMISSIONED-2026-05-28, prometheus scrape job deleted, CT 113
  litellm DB + role dropped. Bifrost now connects directly to upstream
  LLM providers (verified: openai → SAIA chat-ai.academiccloud.de).
- Open WebUI data (CT 104 uploads + vector_db, 192K) now staged to
  UNAS /services/open-webui via openwebui-prestage.sh and included in
  Backrest services-backup-plan. Open WebUI chats/users live in CT 113
  Postgres owui DB (already in WAL-G).
- Miniflux confirmed DB-only — already covered by WAL-G.
- backrest.md and homelab-architecture.md updated; services-overview.md
  marks LiteLLM row strikethrough.
2026-05-28 21:47:50 +02:00
fkrebs 9cbd02e3ff docs: 2026-05-28 ops coverage audit + WAL-G monitor fix
- homelab-architecture.md: new "Ops coverage matrix" + "Recent ops debt
  cleared" + "Still open" sections. Verifies log aggregation (8/8 hosts),
  Postgres archiving (CT 113 + Immich), Backrest (16 paths exist, all 3
  plans have hooks), Diun, config-to-git, offsite sync, Gotify apps,
  timers and crons across the fleet.
- backrest.md: 16:00 false-positive postmortem (CT 103 had no SSH access
  to CT 113; 2>/dev/null hid the host-key error). Documented retry loop +
  SSH key deployment. Refreshed CT 113 DB list (added owui, miniflux;
  noted litellm decommissioned but DB retained).
2026-05-28 21:20:35 +02:00
fkrebs 6c401f2376 docs: reconcile sunset of LiteLLM external, mcp-gateway, and LobeChat
- Bifrost replaces LiteLLM at ai.nuclide.systems (CT 104:14003); LiteLLM
  now internal-only backend at :14000
- mcp-gateway (FastAPI/DinD, mcp.nuclide.systems) decommissioned 2026-05-26;
  MCP now served by Bifrost at ai.nuclide.systems/mcp (29 clients, ~760 tools)
- LobeChat decommissioned 2026-05-26; chat.nuclide.systems now Open WebUI
  (CT 104:14002, stack ai/open-webui.yml)
- Update portmap, volumes, services-overview, homelab-architecture, zoraxy,
  mcp-servers, databases, README accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-26 13:35:29 +02:00
fkrebs b2cb123414 monitoring: document Loki/Alloy/pve-exporter, dashboards and alerts
CT 109 ops stack now includes Loki (log agg), Grafana Alloy (log agent on
all 12 hosts), prometheus-pve-exporter. 5 Grafana dashboards imported;
3 alert rules wired to Gotify.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 07:03:51 +02:00
fkrebs a8ed4350f0 ops: CT 109 provisioned — Prometheus + Grafana monitoring
- Debian 13, Docker 29.5, 4c/4G/32G at 192.168.1.8
- Prometheus :9090 (90d retention), Grafana :3000 (LAN only)
- Scrapes: litellm CT104:14000, node-ct104:9100, node-ct109:9100
- CT 104 standalone node-exporter retained at /opt/stacks/monitoring/
- All 4 targets confirmed up

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 03:44:05 +02:00
fkrebs 143343decf docs: session continuity file + claude-max-bridge /v1/responses fix
- Add RESUME.md for cross-session continuity (open items, key state, constraints)
- claude-max-bridge: implement /v1/responses (OpenAI Responses API) with
  previous_response_id chaining via server-side history injection into system prompt.
  Root cause of "session already in use": CLI leaves JSONL in un-resumable
  "dequeued" state after each --print run; fix avoids session reuse entirely.
  Also fixed: assistant content must be array-of-blocks not plain string (silent
  JS crash otherwise).
- LiteLLM: add pass_through_endpoints for /v1/responses → claude-max-bridge
- Storage, volumes, architecture docs reconciled (Vaultwarden → local zfs,
  Pocket-ID backup, WAL-G fix, apps/ decommission, Nextcloud CIFS→NFS)
- Add ideas/, proxmox-memory-audit.md, llm-benchmark.md (new docs this session)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 22:42:17 +02:00
fkrebs c2979f7c9d Rename CT 109 observe → ops across all docs
Also updates IP conflict note: .6=CT113, .7=CT112, so CT 109 gets .8.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 10:38:57 +02:00
fkrebs 0d83f83bc0 docs: MCP gateway git/gitlab/paper-search; CT 109 plan; new backrest/db/arcane docs
- mcp-gateway.md: 26 servers (added git, gitlab, paper-search catalog bridge);
  document --pass-environment requirement for env var passthrough in DinD bridges
- ct-inventory.md + homelab-architecture.md: CT 109 IP conflict, CT 113 live
- proxmox-state.md §16: Loki, sshwifty, Alloy, IP conflict note, sidecar table
  expanded to CT 103/111/113; build order updated
- infra/portmap.md: CT 113 db section; QNAP TS-251D (Klipper/Mainsail)
- services/backrest.md: full backup strategy doc (new)
- services/databases.md: CT 113 postgres/WAL-G/pgAdmin doc (new)
- services/arcane.md: Docker management IDE doc (new)
- CHANGELOG.md: 2026-05-22 entry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 09:35:07 +02:00
fkrebs 9f6d0dd195 homelab-architecture: add Mermaid topology / auth / MCP / data plane diagrams 2026-05-21 00:35:24 +02:00
fkrebs 061f09b2d2 docs: top-10 fixes (PORTMAP, storage truth, banners, AdGuard scrub, zoraxy UCI removed, CT 111 section, Daytona purge) 2026-05-21 00:20:18 +02:00
fkrebs 99f93b585f reorg: split into infra/ services/ history/ ideas/ 2026-05-20 23:25:20 +02:00