From 13bc1ecc12ac80ca64e6f5012dfd62ce31cfaf8e Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Tue, 26 May 2026 06:03:28 +0200 Subject: [PATCH] =?UTF-8?q?RESUME=20+=20pocket-id:=20reflect=202026-05-24?= =?UTF-8?q?=20=E2=86=92=2026=20state?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RESUME: - Backrest 3-for-3 success on 2026-05-26 (video-projects-plan recovered) - Vault dedup + 234 link rewrites + cleanup - Plugin audit + Memos Sync + Obsidian Git + streamlined-ui - Config-to-git fleet now 8 repos - AdGuard *.nuclide.lan zone (30 A-records) - 3 new Homarr boards (home/admin/command) - Pocket-ID LAN callback URLs - Immich HTTP/2 Keep-Alive diagnosed - New constraints: use LAN aliases; daily-note merge policy pocket-id.md: LAN callback URLs added for Homarr/Grafana/Infisical/Proxmox. Filed earlier as part of 2026-05-24 work, committing now. --- RESUME.md | 85 ++++++++++++++++++++++++++++++++----------- services/pocket-id.md | 6 ++- 2 files changed, 68 insertions(+), 23 deletions(-) diff --git a/RESUME.md b/RESUME.md index 8ebea1c..7ca0a13 100644 --- a/RESUME.md +++ b/RESUME.md @@ -1,35 +1,34 @@ # Session Resume -Last updated: 2026-05-23 (session continued Γ—4). +Last updated: 2026-05-26. ## Open items (urgency order) ### πŸ”΄ Critical β€” security risk or unrecoverable data loss -- [ ] **Password rotation: `tapirnase`** β€” shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` Β§operational rules. -- [ ] **UNAS personal data off-host** β€” ~820 GB Immich library partially covered by `media-backup-plan` in Backrest (last OK 2026-05-21 but failing/orphaned). `video-projects-plan` never completed. Backrest is the right vehicle β€” fix reliability first (see 🟠 Backrest item), then verify JottaCloud coverage. See `services/backrest.md` phase 1a. +- [ ] **Password rotation: `tapirnase`** β€” shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password, HAOS Terminal & SSH addon password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` Β§operational rules. - [ ] **Nextcloud Borg passphrase β†’ Vaultwarden** β€” passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See `services/backrest.md` blind spot #4b. +- [ ] **Vault secrets sweep** β€” 32 plaintext secrets in Obsidian vault notes (Anthropic key, Gemini, NC AIO passphrase, etc.). Tracker at `Home/Homelab/Secrets to Move.md` in vault. Migrate to Vaultwarden, redact notes, rotate the high-risk ones. ### 🟠 High β€” known broken / verification needed -- [x] ~~**Arcane OIDC secret**~~ β€” fixed 2026-05-23. New secret generated, Pocket-ID bcrypt updated, Arcane compose updated, force-recreated. Login confirmed working. -- [ ] **Backrest media-repo** β€” `video-projects-plan` has NEVER completed a successful snapshot. `media-backup-plan` last OK 2026-05-21; failed 2026-05-22 01:30; orphaned restic run in progress since 19:10 (Backrest restarted 21:23, orphaned it). Both plans next scheduled 2026-05-25 02:00/01:30. Monitor for completion; if they fail again, investigate rclone:jottacloud:media auth. -- [x] ~~**AdGuard split-horizon DNS**~~ β€” done 2026-05-23. Wildcard rewrite `*.nuclide.systems β†’ 192.168.1.4` added to AdGuardHome.yaml; `rewrites_enabled: true` set. All internal clients now resolve to Zoraxy directly. +- [x] ~~**Arcane OIDC secret**~~ β€” fixed 2026-05-23. +- [x] ~~**Backrest media-repo / video-projects**~~ β€” **all three plans succeeded 2026-05-26** (media 01:30, services 01:04, video-projects 04:00). `video-projects-plan` had never completed before this. βœ… +- [x] ~~**AdGuard split-horizon DNS**~~ β€” done 2026-05-23. - [x] ~~**WAL-G monitoring**~~ β€” done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`. - [x] ~~**Zoraxy audit**~~ β€” done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`. ### 🟑 Medium β€” incomplete migrations / cleanup debt -- [ ] **Docker disk reclaim** β€” ~17 GB reclaimable on CT 104 (10.3 GB unused images, 6.9 GB build cache). `docker system prune -a` after confirming no needed images. See `infra/proxmox-state.md` Β§10c. -- [ ] **Dormant stacks audit** β€” 4 decommissioned 2026-05-23 (daytona, homepage, proxy, streamio β€” composes renamed `.DECOMMISSIONED`). Remaining ~6 still to audit: dozzle (CT 104 copy), arr-stack, qdrant, etc. See `infra/proxmox-state.md` Β§10c. -- [x] ~~**Config-to-git**~~ β€” deployed 2026-05-23. Scripts + daily 03:00 crons on CT 108, CT 102, PVE. HA added 2026-05-24: `fkrebs/home-assistant-config` (existing repo) wired up with `/config/scripts/git-push.sh`. Manual step left: paste `init_commands` block from `/docs/infra/config-to-git.md` into the Terminal & SSH addon config so the cron survives addon updates. Full fleet documented in `/docs/infra/config-to-git.md`. -- [ ] **D-Link hardening** β€” admin UI open to full LAN, SNMP unverified, HTTP-only. Enable trusted-host allowlist, harden SNMP, add TLS. See `services/homelab-architecture.md`. -- [ ] **Vaultwarden OIDC SSO** β€” Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22. - -### 🟑 Medium β€” incomplete migrations / cleanup debt -- [ ] **Infisical OIDC config** β€” Pocket-ID client created (`b2069075-ede2-4251-ad1f-9a62e6a188b3`). Manual step needed: enter OIDC settings in Infisical admin at `http://192.168.1.7:8200/admin` β†’ Settings β†’ OIDC. Issuer: `https://id.nuclide.systems`, ClientID: `b2069075`, Secret: `qsANw95zsza0_tzJR9SzSp4Z6iEIhidgJPHpDih1ffE`. -- [ ] **Infisical secrets migration** β€” Phase 2: import CT104 `.env` keys. Use `infisical import`. See `services/secrets-manager.md` Phase 2. -- [x] ~~**Arcane auto-update notifications**~~ β€” done 2026-05-24. Configured via Arcane UI. -- [x] ~~**Loki dashboard broken in Grafana**~~ β€” fixed 2026-05-23. K8s labels (namespace/pod) replaced with Docker labels (host/container/job) in 14 places across template variables and panel LogQL expressions. Provisioning file updated + pushed via Grafana API. -- [ ] **Grafana HAOS dashboard** β€” HA recorder reconfiguration planned (exclude Prometheus-covered metrics). Reduce recorder include list. -- [ ] **HAOS SSH key** β€” manual step: paste nuc public key in HA terminal. +- [ ] **Docker disk reclaim** β€” ~17 GB reclaimable on CT 104. +- [ ] **Dormant stacks audit** β€” ~6 still to audit (dozzle CT 104 copy, arr-stack, qdrant, etc.). 4 decommissioned 2026-05-23. +- [x] ~~**Config-to-git fleet**~~ β€” 8 repos as of 2026-05-24. PVE/CT 108/CT 102/HA/CT 103/CT 109/CT 113/obsidian-vault/obsidian-config. Full table in `infra/config-to-git.md`. Manual step pending: HA Terminal & SSH addon `init_commands` for cron persistence. +- [ ] **D-Link hardening** β€” admin UI at `http://dlink.nuclide.lan` (`192.168.1.10`). Trusted-host allowlist + SNMP + TLS. +- [ ] **Vaultwarden OIDC SSO** β€” Pocket-ID client ready, env vars not yet set in `/opt/stacks/vaultwarden/.env`. +- [ ] **Infisical OIDC config** β€” manual step in Infisical admin at `http://secrets.nuclide.lan:8200/admin` β†’ Settings β†’ OIDC. Client `b2069075-…`, secret `qsANw95zsza0_…`. +- [ ] **Infisical secrets migration Phase 2** β€” import CT104 `.env` keys via `infisical import`. +- [ ] **Grafana HAOS dashboard** β€” HA recorder reconfiguration (exclude Prometheus-covered metrics). +- [ ] **HAOS SSH key** β€” paste nuc public key in HA terminal. +- [ ] **Obsidian config drift** β€” 3 enabled-but-missing plugins (`obsidian42-brat`, `obsidian-tasks-plugin`, `tasks-caldav-sync`). Reinstall or remove from `community-plugins.json`. See `Home/Homelab/Obsidian Plugin Audit.md` in vault. +- [ ] **Memos Sync plugin install** β€” config pushed to `obsidian-config`, plugin binary needs Community-Plugins install + paste of pre-baked `data.json`. +- [ ] **Immich HTTP/2 Keep-Alive** β€” root-caused 2026-05-24. Workaround live (per-SSID LAN URL). Permanent fix: strip header at Zoraxy. **Needs Zoraxy confirmation.** ### πŸ”΅ Planned β€” requires infrastructure or significant effort - [ ] **Second NVMe** β€” needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items. @@ -43,8 +42,50 @@ Last updated: 2026-05-23 (session continued Γ—4). - [ ] **Document ingestion n8n workflow** β€” Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless β†’ Docling β†’ embeddings β†’ LobeChat KB. See `services/doc-ingestion.md`. - [ ] **ComfyUI async queue** β€” MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`. - [ ] **Additional MCP servers** β€” Gitea βœ“, Paperless βœ“, Proxmox-VE βœ“ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf. -- [x] ~~**Arcane + Dozzle β†’ CT 109**~~ β€” done 2026-05-23. Headless agents on all 7 Docker hosts with unique tokens per environment. 8 environments total: NUC, db, shepard, docker, nextcloud, id, dev, secrets. Auto-update daily 3am. ntfy notifications configured. -- [x] ~~**crawl4ai SSE StreamConsumed**~~ β€” resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers. +- [x] ~~**Arcane + Dozzle β†’ CT 109**~~ β€” done 2026-05-23. +- [x] ~~**crawl4ai SSE StreamConsumed**~~ β€” resolved 2026-05-22. + +## Recently completed (2026-05-26) + +- **Backrest 3-for-3 success**: all plans now completing on schedule. `video-projects-plan` recovered from never-completed state. + +## Recently completed (2026-05-24) + +### Vault + Obsidian +- **Vault dedup**: `Work 1/` promoted β†’ `Work/`, old `Work/` parked as `Work.stale-backup-2026-05-24/` (1-week safety net). Rescued unique `Journal/Journal.md` before swap. Killed dup `Home/BrainBox.md`, `Home/Templates/`, `Willkommen.md` Γ—2, `.caldav-sync/`. +- **234 asset-link rewrites** across 41 files β€” collapsed `assets/`, `../assets/`, `../../foo.pdf` β†’ `.assets/…`. Broken links 242 β†’ 9 (rest non-issues: `tel:`, `about:reader?`, `siyuan://`). +- **`Home/Homelab/Secrets to Move.md`** β€” 32 plaintext secrets inventoried. +- **`Home/Homelab/Obsidian Plugin Audit.md`** β€” 17 installed plugins keep/watch/re-evaluate, 3 config-drift items, 10 new candidates ranked for the stack. +- **`Home/Homelab/App Endpoint Checklist.md`** β€” per-service LAN + external URLs + apps-to-update list. Gotchas: clients sticking to old IP, Immich HTTP/2 Keep-Alive bug. +- **Daily-note merge policy**: Claude edits land in `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## πŸ€– Claude edits` (not parallel folder). +- **Obsidian Git plugin**: wired end-to-end against `fkrebs/obsidian-config`. Auto-commit + push every 60 min, pull on boot, PAT gitignored. Verified working. +- **Streamlined-ui CSS snippet** pushed to `obsidian-config` (hides ribbon icons for non-clickable plugins, compact status bar, tighter file tree). +- **Memos Sync plugin config** pre-pushed (binary install pending user). + +### Config-to-git fleet expanded to 8 repos +- New: `fkrebs/ct103-conf`, `fkrebs/ct109-conf`, `fkrebs/ct113-conf`, `fkrebs/home-assistant-config`, `fkrebs/obsidian-vault` (rolling 2-commit window), `fkrebs/obsidian-config`. +- All daily 03:00 / 04:00. Pattern documented in `infra/config-to-git.md`. +- Rule articulated: every host with hand-edited config must push daily. + +### Networking β€” `*.nuclide.lan` zone +- **30 AdGuard A-records** on CT 102 β€” per-host (`pve`, `nas`, `unifi`, `dlink`, `backrest`, etc.) + per-service aliases (`immich`, `vault`, `karakeep`, `grafana`, `prometheus`, …). +- Documented in `services/adguard-dns.md`. +- Solves "client registered to old IP" gotcha β€” when service moves CT, edit one AdGuard rewrite instead of N apps. +- D-Link IP captured: `192.168.1.10`. +- Filed feedback memory: always use `*.nuclide.lan` aliases in URLs, never bare IPs. + +### Homarr +- **3 new boards** built via SQL: `home` (morning routine, default), `admin` (by-tier), `command` (live-ops iframes + grid). Old "nuclide" board was lost when OIDC user was recreated for password reset. +- Boards owned by `fkrebs@nucli.de`, `is_public=1`. 33 unique apps with ping status. +- Integrations not yet wired (Proxmox / Docker / AdGuard / Gotify widgets β€” need tokens). + +### Pocket-ID +- **LAN callback URLs added** for Homarr, Grafana, Infisical, Proxmox VE so OIDC works via both bare-IP and `.nuclide.lan` alias paths. + +### Other +- **Immich Android upload bug** diagnosed: HTTP/2 + `Keep-Alive` header conflict. Workaround live (per-SSID LAN URL). Permanent fix pending Zoraxy header strip. +- **Nexa #1 filed** at https://git.nuclide.systems/fkrebs/nexa/issues/1 β€” 9 design patterns from Obsidian + Claude session as input for Nexa Phase 1–2. +- **Wi-Fi SSID captured**: `nuclide`. Used for per-network app routing (Immich, HA Companion). ## Recently completed (2026-05-23, session continued Γ—4) @@ -128,3 +169,5 @@ Last updated: 2026-05-23 (session continued Γ—4). - CT 104 has a separate agent reconciling compose files β€” rename decommissioned files (don't just stop), update `/opt/stacks/CLAUDE.md`. - ComfyUI container limit: 24 G. Do not raise without measuring host impact. - Pocket-ID is SQLite-only β€” no Postgres migration possible. +- Use `*.nuclide.lan` aliases when presenting URLs to the user (never bare IPs). Wi-Fi SSID is `nuclide`. +- Vault edits append to `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## πŸ€– Claude edits`; never create a parallel folder. diff --git a/services/pocket-id.md b/services/pocket-id.md index 7601a77..3ce8731 100644 --- a/services/pocket-id.md +++ b/services/pocket-id.md @@ -44,8 +44,10 @@ Pocket-ID is a lightweight OIDC 2.1 / OAuth 2.0 IdP. Every service that supports | Arcane | 104 | `https://arcane.nuclide.systems/auth/callback` | OIDC_CLIENT_SECRET rotated 2026-05-22 | | n8n | 104 | `https://n8n.nuclide.systems/rest/oauth2-credential/callback` | encryption key rotated 2026-05-22 | | Vaultwarden | 104 | `https://vault.nuclide.systems/identity/connect/token` | client created 2026-05-21; auth flow not yet wired | -| Homarr | 109 | `http://192.168.1.8:7575/api/auth/callback/oidc` | SSO active 2026-05-23; client `63a94e30` | -| Grafana | 109 | `http://192.168.1.8:3000/login/generic_oauth` | SSO active 2026-05-23; client `92d987d5`; PKCE enabled | +| Homarr | 109 | `http://192.168.1.8:7575/api/auth/callback/oidc` + `http://homarr.nuclide.lan:7575/...` | SSO active 2026-05-23; client `63a94e30`; LAN alias added 2026-05-24 | +| Grafana | 109 | `http://192.168.1.8:3000/login/generic_oauth` + `http://grafana.nuclide.lan:3000/...` | SSO active 2026-05-23; client `92d987d5`; PKCE enabled; LAN alias added 2026-05-24 | +| Infisical | 112 | `http://192.168.1.7:8200/api/v1/sso/oidc/callback` + `http://secrets.nuclide.lan:8200/...` | LAN alias added 2026-05-24; manual OIDC config still pending | +| Proxmox VE | host | `https://192.168.1.20:8006` + `https://pve.nuclide.lan:8006` | LAN alias added 2026-05-24 | ## Env var patterns per service type