fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: 2026-05-23 session — CT109 ops, Arcane/Dozzle migration, OIDC wired, OOM fix, n8n tracking

Browse Source
This commit is contained in:
Florian Krebs
2026-05-23 09:44:15 +02:00
parent 4c97732720
commit c3fd649f68
2 changed files with 40 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+30 -4
View File
@@ -40,17 +40,43 @@ All notable infrastructure / service / doc changes. Newest first.
- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.
### Pocket-ID OIDC (pending user action)
### Pocket-ID OIDC (wired via API)
- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate.
- **Homarr OIDC**: client `63a94e30` created via Pocket-ID API. Redirect URI `http://192.168.1.8:7575/api/auth/callback/oidc`. Homarr compose updated (`AUTH_PROVIDER=oidc`, `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET`, `AUTH_OIDC_ISSUER`, etc.), force-recreated. SSO active 2026-05-23.
- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars.
- **Grafana OIDC**: client `92d987d5` created via Pocket-ID API with PKCE enabled. Redirect URI `http://192.168.1.8:3000/login/generic_oauth`. Grafana compose updated with `GF_AUTH_GENERIC_OAUTH_*` vars, force-recreated. SSO active 2026-05-23; role mapping: `admins` group → Admin, else Viewer.
### Arcane + Dozzle migrated to CT 109; headless agents on all hosts
- **Arcane server moved CT 104 → CT 109**: stack at `/opt/stacks/arcane/docker-compose.yml` on CT 109, `APP_URL=http://192.168.1.8:10002`. Zoraxy proxy route for `arcane.nuclide.systems` updated to `192.168.1.8:10002`. Old CT 104 arcane container stopped (renamed decommissioned).
- **Dozzle server moved CT 104 → CT 109**: stack at `/opt/stacks/dozzle/docker-compose.yml` (:10001). Remote agents configured for all 7 Docker hosts (CT 101, 104, 105, 110, 111, 112, 113).
- **Headless agents deployed on CT 101, 104, 105, 110, 111, 112** via `/opt/stacks/ops-agents/docker-compose.yml` (Arcane agent + Dozzle agent). CT 113 already had Arcane agent; updated `MANAGER_API_URL` to CT 109 and added Dozzle agent.
- **docs-server migrated CT 111 → CT 109**: `mkdocs.yml` site_url fixed, nav expanded. docs-server now at `http://192.168.1.8:13080`.
### HAOS KVM OOM protection
- **Root cause identified**: karakeep_chrome (CT 104, `gcr.io/zenika-hub/alpine-chrome:123`) triggered host-level OOM. Linux OOM killer chose HAOS KVM (PID 588858, 10.3 GB RSS) as largest victim. HA was not crashing internally — it was being killed externally.
- **shm_size: '512m' added to karakeep chrome service** and force-recreated to reduce Chrome shared memory pressure.
- **oom_score_adj=-300 set on HAOS KVM PID** (protects it from OOM killer — minimum score = hardest to kill).
- **Persistence via systemd timer** (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.
### n8n flow tracking
- **Gitea repo `fkrebs/n8n-flows` created**: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push.
- **IDEAS.md**: 10 automation ideas documented with node designs (health check flow, Backrest notify, Docling pipeline, HA alerts, Gitea digest, WAL-G monitor, etc.).
- **10 Gitea issues filed** on `fkrebs/n8n-flows` for each idea — tracked in memory.
- **Homelab health check (bash/systemd)**: interim implementation on nuc (`/usr/local/bin/homelab-health`, `homelab-health.timer`, every 1h). Checks: container status per host via SSH, HA VM status, disk usage, key HTTP endpoints. Auto-fix: `docker start` for stopped containers, `docker prune` for CT104 /tmp >80%. Escalates to Gotify (`homelab-health` app, token `Az9NpC-m1jjf571`) for crash loops/disk critical. Migration to n8n tracked as issue #1.
### CT 104 ai repo cleanup
- **lobehub/data/ untracked**: Postgres data dir was accidentally tracked in git. `git rm --cached -r lobehub/data/` + added to `.gitignore`. Committed alongside: litellm-config/config.yaml, mcp-gateway updates (litellm_sync.py, model_assignments.json, models.html, server.py), lobehub.yml (claude-* models).
### Docs
- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config).
- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config, n8n-flows).
- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.
- **`services/pocket-id.md` created**: OIDC endpoints, client creation walkthrough, per-service env var patterns (Homarr, Grafana, Gitea, Coder, Generic), current client registry, backup notes.
## 2026-05-22