From c3fd649f68b306e5046b736ec8ab76178f92d80b Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Sat, 23 May 2026 09:44:15 +0200 Subject: [PATCH] =?UTF-8?q?docs:=202026-05-23=20session=20=E2=80=94=20CT10?= =?UTF-8?q?9=20ops,=20Arcane/Dozzle=20migration,=20OIDC=20wired,=20OOM=20f?= =?UTF-8?q?ix,=20n8n=20tracking?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 34 ++++++++++++++++++++++++++++++---- RESUME.md | 12 ++++++++++-- 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cee666e..049d36a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,17 +40,43 @@ All notable infrastructure / service / doc changes. Newest first. - **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff. -### Pocket-ID OIDC (pending user action) +### Pocket-ID OIDC (wired via API) -- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate. +- **Homarr OIDC**: client `63a94e30` created via Pocket-ID API. Redirect URI `http://192.168.1.8:7575/api/auth/callback/oidc`. Homarr compose updated (`AUTH_PROVIDER=oidc`, `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET`, `AUTH_OIDC_ISSUER`, etc.), force-recreated. SSO active 2026-05-23. -- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars. +- **Grafana OIDC**: client `92d987d5` created via Pocket-ID API with PKCE enabled. Redirect URI `http://192.168.1.8:3000/login/generic_oauth`. Grafana compose updated with `GF_AUTH_GENERIC_OAUTH_*` vars, force-recreated. SSO active 2026-05-23; role mapping: `admins` group → Admin, else Viewer. + +### Arcane + Dozzle migrated to CT 109; headless agents on all hosts + +- **Arcane server moved CT 104 → CT 109**: stack at `/opt/stacks/arcane/docker-compose.yml` on CT 109, `APP_URL=http://192.168.1.8:10002`. Zoraxy proxy route for `arcane.nuclide.systems` updated to `192.168.1.8:10002`. Old CT 104 arcane container stopped (renamed decommissioned). +- **Dozzle server moved CT 104 → CT 109**: stack at `/opt/stacks/dozzle/docker-compose.yml` (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113). +- **Headless agents deployed on CT 101, 104, 105, 110, 111, 112** via `/opt/stacks/ops-agents/docker-compose.yml` (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated `MANAGER_API_URL` to CT 109 and added Dozzle agent. +- **docs-server migrated CT 111 → CT 109**: `mkdocs.yml` site_url fixed, nav expanded. docs-server now at `http://192.168.1.8:13080`. + +### HAOS KVM OOM protection + +- **Root cause identified**: karakeep_chrome (CT 104, `gcr.io/zenika-hub/alpine-chrome:123`) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally. +- **shm_size: '512m' added to karakeep chrome service** and force-recreated to reduce Chrome shared memory pressure. +- **oom_score_adj=-300 set on HAOS KVM PID** (protects it from OOM killer — minimum score = hardest to kill). +- **Persistence via systemd timer** (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not. + +### n8n flow tracking + +- **Gitea repo `fkrebs/n8n-flows` created**: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push. +- **IDEAS.md**: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.). +- **10 Gitea issues filed** on `fkrebs/n8n-flows` for each idea — tracked in memory. +- **Homelab health check (bash/systemd)**: interim implementation on nuc (`/usr/local/bin/homelab-health`, `homelab-health.timer`, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: `docker start` for stopped containers, `docker prune` for CT104 /tmp >80%. Escalates to Gotify (`homelab-health` app, token `Az9NpC-m1jjf571`) for crash loops/disk critical. Migration to n8n tracked as issue #1. + +### CT 104 ai repo cleanup + +- **lobehub/data/ untracked**: Postgres data dir was accidentally tracked in git. `git rm --cached -r lobehub/data/` + added to `.gitignore`. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models). ### Docs -- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config). +- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows). - **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section. - **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack. +- **`services/pocket-id.md` created**: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes. ## 2026-05-22 diff --git a/RESUME.md b/RESUME.md index 7e61a00..7fe3452 100644 --- a/RESUME.md +++ b/RESUME.md @@ -24,7 +24,6 @@ Last updated: 2026-05-23. Update this at the end of every session. - [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22. ### 🔵 Planned — requires infrastructure or significant effort -- [ ] **CT 109 "ops"** — LXC for Prometheus, Loki, Grafana, Alloy, Homarr, sshwifty, Gotify. Blocks monitoring stack + Arcane migration. See `infra/proxmox-state.md` §16. - [ ] **Second NVMe** — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items. - [ ] **Infisical secrets migration** — CT 112 deployed (`192.168.1.7:8200`, LAN-only). Phase 1 done. Remaining: - **Phase 2**: create project `homelab/ct104`, import `ai/.env` (~55 keys) via `infisical import`, add agent sidecars to compose stacks @@ -36,9 +35,18 @@ Last updated: 2026-05-23. Update this at the end of every session. - [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`. - [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`. - [ ] **Additional MCP servers** — community MCPs exist for Gitea, Paperless, Karakeep, Vaultwarden, Proxmox-VE, Audiobookshelf. Evaluate + wire in. -- [ ] **Arcane → CT 109 migration** — blocked by CT 109. Procedure documented in `services/arcane.md`. +- [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts. - [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers. +## Recently completed (2026-05-23, session continued) + +- **HAOS KVM OOM root-caused and fixed**: karakeep_chrome on CT 104 triggered host OOM → killed HAOS KVM twice. Applied: `shm_size: '512m'` to chrome, `oom_score_adj=-300` on KVM PID, systemd timer `protect-haos-kvm.timer` (every 5 min) for persistence. +- **Arcane + Dozzle migrated to CT 109**: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080). +- **Homarr + Grafana OIDC wired**: clients created via Pocket-ID API (no manual UI steps). Homarr client `63a94e30`, Grafana client `92d987d5` (PKCE). Both SSO-active. +- **n8n flows tracked in Gitea**: repo `fkrebs/n8n-flows` with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n). +- **CT 104 ai repo cleaned**: `lobehub/data/` untracked from git, `.gitignore` fixed, litellm-config + mcp-gateway changes committed. +- **zoraxy-conf repo**: arcane proxy update committed and pushed (CT 108). + ## Recently completed (2026-05-23) - **Config-to-git crons**: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy → `zoraxy-conf`), CT 102 (AdGuard → `adguard-conf`), PVE (rsync `/etc/pve` → `pve-conf`). Force-push. All three verified with initial push.