fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: 2026-05-23 session — CT109 ops, Arcane/Dozzle migration, OIDC wired, OOM fix, n8n tracking

Browse Source
This commit is contained in:
Florian Krebs
2026-05-23 09:44:15 +02:00
parent 4c97732720
commit c3fd649f68
2 changed files with 40 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+30 -4
View File
@@ -40,17 +40,43 @@ All notable infrastructure / service / doc changes. Newest first.
- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff. - **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.
### Pocket-ID OIDC (pending user action) ### Pocket-ID OIDC (wired via API)
- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate. - **Homarr OIDC**: client `63a94e30` created via Pocket-ID API. Redirect URI `http://192.168.1.8:7575/api/auth/callback/oidc`. Homarr compose updated (`AUTH_PROVIDER=oidc`, `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET`, `AUTH_OIDC_ISSUER`, etc.), force-recreated. SSO active 2026-05-23.
- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars. - **Grafana OIDC**: client `92d987d5` created via Pocket-ID API with PKCE enabled. Redirect URI `http://192.168.1.8:3000/login/generic_oauth`. Grafana compose updated with `GF_AUTH_GENERIC_OAUTH_*` vars, force-recreated. SSO active 2026-05-23; role mapping: `admins` group → Admin, else Viewer.
### Arcane + Dozzle migrated to CT 109; headless agents on all hosts
- **Arcane server moved CT 104 → CT 109**: stack at `/opt/stacks/arcane/docker-compose.yml` on CT 109, `APP_URL=http://192.168.1.8:10002`. Zoraxy proxy route for `arcane.nuclide.systems` updated to `192.168.1.8:10002`. Old CT 104 arcane container stopped (renamed decommissioned).
- **Dozzle server moved CT 104 → CT 109**: stack at `/opt/stacks/dozzle/docker-compose.yml` (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).
- **Headless agents deployed on CT 101, 104, 105, 110, 111, 112** via `/opt/stacks/ops-agents/docker-compose.yml` (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated `MANAGER_API_URL` to CT 109 and added Dozzle agent.
- **docs-server migrated CT 111 → CT 109**: `mkdocs.yml` site_url fixed, nav expanded. docs-server now at `http://192.168.1.8:13080`.
### HAOS KVM OOM protection
- **Root cause identified**: karakeep_chrome (CT 104, `gcr.io/zenika-hub/alpine-chrome:123`) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally.
- **shm_size: '512m' added to karakeep chrome service** and force-recreated to reduce Chrome shared memory pressure.
- **oom_score_adj=-300 set on HAOS KVM PID** (protects it from OOM killer — minimum score = hardest to kill).
- **Persistence via systemd timer** (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.
### n8n flow tracking
- **Gitea repo `fkrebs/n8n-flows` created**: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push.
- **IDEAS.md**: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).
- **10 Gitea issues filed** on `fkrebs/n8n-flows` for each idea — tracked in memory.
- **Homelab health check (bash/systemd)**: interim implementation on nuc (`/usr/local/bin/homelab-health`, `homelab-health.timer`, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: `docker start` for stopped containers, `docker prune` for CT104 /tmp >80%. Escalates to Gotify (`homelab-health` app, token `Az9NpC-m1jjf571`) for crash loops/disk critical. Migration to n8n tracked as issue #1.
### CT 104 ai repo cleanup
- **lobehub/data/ untracked**: Postgres data dir was accidentally tracked in git. `git rm --cached -r lobehub/data/` + added to `.gitignore`. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).
### Docs ### Docs
- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config). - **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).
- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section. - **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack. - **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.
- **`services/pocket-id.md` created**: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.
## 2026-05-22 ## 2026-05-22
RESUME.md
+10 -2
View File
@@ -24,7 +24,6 @@ Last updated: 2026-05-23. Update this at the end of every session.
- [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22. - [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22.
### 🔵 Planned — requires infrastructure or significant effort ### 🔵 Planned — requires infrastructure or significant effort
- [ ] **CT 109 "ops"** — LXC for Prometheus, Loki, Grafana, Alloy, Homarr, sshwifty, Gotify. Blocks monitoring stack + Arcane migration. See `infra/proxmox-state.md` §16.
- [ ] **Second NVMe** — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items. - [ ] **Second NVMe** — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items.
- [ ] **Infisical secrets migration** — CT 112 deployed (`192.168.1.7:8200`, LAN-only). Phase 1 done. Remaining: - [ ] **Infisical secrets migration** — CT 112 deployed (`192.168.1.7:8200`, LAN-only). Phase 1 done. Remaining:
- **Phase 2**: create project `homelab/ct104`, import `ai/.env` (~55 keys) via `infisical import`, add agent sidecars to compose stacks - **Phase 2**: create project `homelab/ct104`, import `ai/.env` (~55 keys) via `infisical import`, add agent sidecars to compose stacks
@@ -36,9 +35,18 @@ Last updated: 2026-05-23. Update this at the end of every session.
- [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`. - [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`.
- [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`. - [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`.
- [ ] **Additional MCP servers** — community MCPs exist for Gitea, Paperless, Karakeep, Vaultwarden, Proxmox-VE, Audiobookshelf. Evaluate + wire in. - [ ] **Additional MCP servers** — community MCPs exist for Gitea, Paperless, Karakeep, Vaultwarden, Proxmox-VE, Audiobookshelf. Evaluate + wire in.
- [ ] **Arcane → CT 109 migration** — blocked by CT 109. Procedure documented in `services/arcane.md`. - [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts.
- [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers. - [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers.
## Recently completed (2026-05-23, session continued)
- **HAOS KVM OOM root-caused and fixed**: karakeep_chrome on CT 104 triggered host OOM → killed HAOS KVM twice. Applied: `shm_size: '512m'` to chrome, `oom_score_adj=-300` on KVM PID, systemd timer `protect-haos-kvm.timer` (every 5 min) for persistence.
- **Arcane + Dozzle migrated to CT 109**: servers on CT 109 (:10002, :10001). Headless agents deployed on CT 101, 104, 105, 110, 111, 112, 113. Zoraxy route updated. docs-server also on CT 109 (:13080).
- **Homarr + Grafana OIDC wired**: clients created via Pocket-ID API (no manual UI steps). Homarr client `63a94e30`, Grafana client `92d987d5` (PKCE). Both SSO-active.
- **n8n flows tracked in Gitea**: repo `fkrebs/n8n-flows` with daily sync cron (CT 104, 03:30). 10 automation ideas filed as Gitea issues. Interim health check bash/systemd on nuc (issue #1 = migrate to n8n).
- **CT 104 ai repo cleaned**: `lobehub/data/` untracked from git, `.gitignore` fixed, litellm-config + mcp-gateway changes committed.
- **zoraxy-conf repo**: arcane proxy update committed and pushed (CT 108).
## Recently completed (2026-05-23) ## Recently completed (2026-05-23)
- **Config-to-git crons**: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy → `zoraxy-conf`), CT 102 (AdGuard → `adguard-conf`), PVE (rsync `/etc/pve` → `pve-conf`). Force-push. All three verified with initial push. - **Config-to-git crons**: scripts + daily 03:00 crons deployed on CT 108 (Zoraxy → `zoraxy-conf`), CT 102 (AdGuard → `adguard-conf`), PVE (rsync `/etc/pve` → `pve-conf`). Force-push. All three verified with initial push.