RESUME + pocket-id: reflect 2026-05-24 → 26 state
RESUME: - Backrest 3-for-3 success on 2026-05-26 (video-projects-plan recovered) - Vault dedup + 234 link rewrites + cleanup - Plugin audit + Memos Sync + Obsidian Git + streamlined-ui - Config-to-git fleet now 8 repos - AdGuard *.nuclide.lan zone (30 A-records) - 3 new Homarr boards (home/admin/command) - Pocket-ID LAN callback URLs - Immich HTTP/2 Keep-Alive diagnosed - New constraints: use LAN aliases; daily-note merge policy pocket-id.md: LAN callback URLs added for Homarr/Grafana/Infisical/Proxmox. Filed earlier as part of 2026-05-24 work, committing now.
This commit is contained in:
@@ -1,35 +1,34 @@
|
||||
# Session Resume
|
||||
|
||||
Last updated: 2026-05-23 (session continued ×4).
|
||||
Last updated: 2026-05-26.
|
||||
|
||||
## Open items (urgency order)
|
||||
|
||||
### 🔴 Critical — security risk or unrecoverable data loss
|
||||
- [ ] **Password rotation: `tapirnase`** — shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` §operational rules.
|
||||
- [ ] **UNAS personal data off-host** — ~820 GB Immich library partially covered by `media-backup-plan` in Backrest (last OK 2026-05-21 but failing/orphaned). `video-projects-plan` never completed. Backrest is the right vehicle — fix reliability first (see 🟠 Backrest item), then verify JottaCloud coverage. See `services/backrest.md` phase 1a.
|
||||
- [ ] **Password rotation: `tapirnase`** — shared across WiFi PSK, LiteLLM root key (`sk-tapirnase`), D-Link admin, Backrest repo password, HAOS Terminal & SSH addon password. Single sniff = broad blast radius. Rotate per-service. See `services/homelab-architecture.md` §operational rules.
|
||||
- [ ] **Nextcloud Borg passphrase → Vaultwarden** — passphrase only in container env; CT 105 loss = unrecoverable backup. Move to Vaultwarden. See `services/backrest.md` blind spot #4b.
|
||||
- [ ] **Vault secrets sweep** — 32 plaintext secrets in Obsidian vault notes (Anthropic key, Gemini, NC AIO passphrase, etc.). Tracker at `Home/Homelab/Secrets to Move.md` in vault. Migrate to Vaultwarden, redact notes, rotate the high-risk ones.
|
||||
|
||||
### 🟠 High — known broken / verification needed
|
||||
- [x] ~~**Arcane OIDC secret**~~ — fixed 2026-05-23. New secret generated, Pocket-ID bcrypt updated, Arcane compose updated, force-recreated. Login confirmed working.
|
||||
- [ ] **Backrest media-repo** — `video-projects-plan` has NEVER completed a successful snapshot. `media-backup-plan` last OK 2026-05-21; failed 2026-05-22 01:30; orphaned restic run in progress since 19:10 (Backrest restarted 21:23, orphaned it). Both plans next scheduled 2026-05-25 02:00/01:30. Monitor for completion; if they fail again, investigate rclone:jottacloud:media auth.
|
||||
- [x] ~~**AdGuard split-horizon DNS**~~ — done 2026-05-23. Wildcard rewrite `*.nuclide.systems → 192.168.1.4` added to AdGuardHome.yaml; `rewrites_enabled: true` set. All internal clients now resolve to Zoraxy directly.
|
||||
- [x] ~~**Arcane OIDC secret**~~ — fixed 2026-05-23.
|
||||
- [x] ~~**Backrest media-repo / video-projects**~~ — **all three plans succeeded 2026-05-26** (media 01:30, services 01:04, video-projects 04:00). `video-projects-plan` had never completed before this. ✅
|
||||
- [x] ~~**AdGuard split-horizon DNS**~~ — done 2026-05-23.
|
||||
- [x] ~~**WAL-G monitoring**~~ — done 2026-05-23. Textfile collector (`/usr/local/bin/walg-metrics.sh`) on CT 113 emits `walg_last_success_timestamp_seconds` + `walg_archive_status` to node-exporter textfile. Prometheus alert rules deployed on CT 109: `WalgArchiveStale` (>24h) + `WalgArchiveFailed`.
|
||||
- [x] ~~**Zoraxy audit**~~ — done 2026-05-23. All 22 routes verified; arcane updated to CT 109 backend; `SkipWebSocketOriginCheck` enabled on 13 routes; decommissioned routes noted. Missing: `dozzle.nuclide.systems` route (never created), `immich-tools`, `paperless`, `paperless-ai`. See `services/zoraxy.md`.
|
||||
|
||||
### 🟡 Medium — incomplete migrations / cleanup debt
|
||||
- [ ] **Docker disk reclaim** — ~17 GB reclaimable on CT 104 (10.3 GB unused images, 6.9 GB build cache). `docker system prune -a` after confirming no needed images. See `infra/proxmox-state.md` §10c.
|
||||
- [ ] **Dormant stacks audit** — 4 decommissioned 2026-05-23 (daytona, homepage, proxy, streamio — composes renamed `.DECOMMISSIONED`). Remaining ~6 still to audit: dozzle (CT 104 copy), arr-stack, qdrant, etc. See `infra/proxmox-state.md` §10c.
|
||||
- [x] ~~**Config-to-git**~~ — deployed 2026-05-23. Scripts + daily 03:00 crons on CT 108, CT 102, PVE. HA added 2026-05-24: `fkrebs/home-assistant-config` (existing repo) wired up with `/config/scripts/git-push.sh`. Manual step left: paste `init_commands` block from `/docs/infra/config-to-git.md` into the Terminal & SSH addon config so the cron survives addon updates. Full fleet documented in `/docs/infra/config-to-git.md`.
|
||||
- [ ] **D-Link hardening** — admin UI open to full LAN, SNMP unverified, HTTP-only. Enable trusted-host allowlist, harden SNMP, add TLS. See `services/homelab-architecture.md`.
|
||||
- [ ] **Vaultwarden OIDC SSO** — Pocket-ID client created 2026-05-21, auth flow not wired up. Confirmed no OIDC vars in `/opt/stacks/vaultwarden/.env` as of 2026-05-22.
|
||||
|
||||
### 🟡 Medium — incomplete migrations / cleanup debt
|
||||
- [ ] **Infisical OIDC config** — Pocket-ID client created (`b2069075-ede2-4251-ad1f-9a62e6a188b3`). Manual step needed: enter OIDC settings in Infisical admin at `http://192.168.1.7:8200/admin` → Settings → OIDC. Issuer: `https://id.nuclide.systems`, ClientID: `b2069075`, Secret: `qsANw95zsza0_tzJR9SzSp4Z6iEIhidgJPHpDih1ffE`.
|
||||
- [ ] **Infisical secrets migration** — Phase 2: import CT104 `.env` keys. Use `infisical import`. See `services/secrets-manager.md` Phase 2.
|
||||
- [x] ~~**Arcane auto-update notifications**~~ — done 2026-05-24. Configured via Arcane UI.
|
||||
- [x] ~~**Loki dashboard broken in Grafana**~~ — fixed 2026-05-23. K8s labels (namespace/pod) replaced with Docker labels (host/container/job) in 14 places across template variables and panel LogQL expressions. Provisioning file updated + pushed via Grafana API.
|
||||
- [ ] **Grafana HAOS dashboard** — HA recorder reconfiguration planned (exclude Prometheus-covered metrics). Reduce recorder include list.
|
||||
- [ ] **HAOS SSH key** — manual step: paste nuc public key in HA terminal.
|
||||
- [ ] **Docker disk reclaim** — ~17 GB reclaimable on CT 104.
|
||||
- [ ] **Dormant stacks audit** — ~6 still to audit (dozzle CT 104 copy, arr-stack, qdrant, etc.). 4 decommissioned 2026-05-23.
|
||||
- [x] ~~**Config-to-git fleet**~~ — 8 repos as of 2026-05-24. PVE/CT 108/CT 102/HA/CT 103/CT 109/CT 113/obsidian-vault/obsidian-config. Full table in `infra/config-to-git.md`. Manual step pending: HA Terminal & SSH addon `init_commands` for cron persistence.
|
||||
- [ ] **D-Link hardening** — admin UI at `http://dlink.nuclide.lan` (`192.168.1.10`). Trusted-host allowlist + SNMP + TLS.
|
||||
- [ ] **Vaultwarden OIDC SSO** — Pocket-ID client ready, env vars not yet set in `/opt/stacks/vaultwarden/.env`.
|
||||
- [ ] **Infisical OIDC config** — manual step in Infisical admin at `http://secrets.nuclide.lan:8200/admin` → Settings → OIDC. Client `b2069075-…`, secret `qsANw95zsza0_…`.
|
||||
- [ ] **Infisical secrets migration Phase 2** — import CT104 `.env` keys via `infisical import`.
|
||||
- [ ] **Grafana HAOS dashboard** — HA recorder reconfiguration (exclude Prometheus-covered metrics).
|
||||
- [ ] **HAOS SSH key** — paste nuc public key in HA terminal.
|
||||
- [ ] **Obsidian config drift** — 3 enabled-but-missing plugins (`obsidian42-brat`, `obsidian-tasks-plugin`, `tasks-caldav-sync`). Reinstall or remove from `community-plugins.json`. See `Home/Homelab/Obsidian Plugin Audit.md` in vault.
|
||||
- [ ] **Memos Sync plugin install** — config pushed to `obsidian-config`, plugin binary needs Community-Plugins install + paste of pre-baked `data.json`.
|
||||
- [ ] **Immich HTTP/2 Keep-Alive** — root-caused 2026-05-24. Workaround live (per-SSID LAN URL). Permanent fix: strip header at Zoraxy. **Needs Zoraxy confirmation.**
|
||||
|
||||
### 🔵 Planned — requires infrastructure or significant effort
|
||||
- [ ] **Second NVMe** — needed before rpool mirror + Postgres consolidation (CT 113). Blocks several items.
|
||||
@@ -43,8 +42,50 @@ Last updated: 2026-05-23 (session continued ×4).
|
||||
- [ ] **Document ingestion n8n workflow** — Docling MCP deployed; n8n orchestration not built. Nextcloud/Paperless → Docling → embeddings → LobeChat KB. See `services/doc-ingestion.md`.
|
||||
- [ ] **ComfyUI async queue** — MCP tool blocks 90–300 s. Job-queue pattern: `generate_image()` returns ID; `get_job_status()` polls. See `services/comfyui.md`.
|
||||
- [ ] **Additional MCP servers** — Gitea ✓, Paperless ✓, Proxmox-VE ✓ (done 2026-05-23). Remaining: Karakeep (needs API key from hoarder UI), Vaultwarden, Audiobookshelf.
|
||||
- [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts with unique tokens per environment. 8 environments total: NUC, db, shepard, docker, nextcloud, id, dev, secrets. Auto-update daily 3am. ntfy notifications configured.
|
||||
- [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers.
|
||||
- [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23.
|
||||
- [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22.
|
||||
|
||||
## Recently completed (2026-05-26)
|
||||
|
||||
- **Backrest 3-for-3 success**: all plans now completing on schedule. `video-projects-plan` recovered from never-completed state.
|
||||
|
||||
## Recently completed (2026-05-24)
|
||||
|
||||
### Vault + Obsidian
|
||||
- **Vault dedup**: `Work 1/` promoted → `Work/`, old `Work/` parked as `Work.stale-backup-2026-05-24/` (1-week safety net). Rescued unique `Journal/Journal.md` before swap. Killed dup `Home/BrainBox.md`, `Home/Templates/`, `Willkommen.md` ×2, `.caldav-sync/`.
|
||||
- **234 asset-link rewrites** across 41 files — collapsed `assets/`, `../assets/`, `../../foo.pdf` → `.assets/…`. Broken links 242 → 9 (rest non-issues: `tel:`, `about:reader?`, `siyuan://`).
|
||||
- **`Home/Homelab/Secrets to Move.md`** — 32 plaintext secrets inventoried.
|
||||
- **`Home/Homelab/Obsidian Plugin Audit.md`** — 17 installed plugins keep/watch/re-evaluate, 3 config-drift items, 10 new candidates ranked for the stack.
|
||||
- **`Home/Homelab/App Endpoint Checklist.md`** — per-service LAN + external URLs + apps-to-update list. Gotchas: clients sticking to old IP, Immich HTTP/2 Keep-Alive bug.
|
||||
- **Daily-note merge policy**: Claude edits land in `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## 🤖 Claude edits` (not parallel folder).
|
||||
- **Obsidian Git plugin**: wired end-to-end against `fkrebs/obsidian-config`. Auto-commit + push every 60 min, pull on boot, PAT gitignored. Verified working.
|
||||
- **Streamlined-ui CSS snippet** pushed to `obsidian-config` (hides ribbon icons for non-clickable plugins, compact status bar, tighter file tree).
|
||||
- **Memos Sync plugin config** pre-pushed (binary install pending user).
|
||||
|
||||
### Config-to-git fleet expanded to 8 repos
|
||||
- New: `fkrebs/ct103-conf`, `fkrebs/ct109-conf`, `fkrebs/ct113-conf`, `fkrebs/home-assistant-config`, `fkrebs/obsidian-vault` (rolling 2-commit window), `fkrebs/obsidian-config`.
|
||||
- All daily 03:00 / 04:00. Pattern documented in `infra/config-to-git.md`.
|
||||
- Rule articulated: every host with hand-edited config must push daily.
|
||||
|
||||
### Networking — `*.nuclide.lan` zone
|
||||
- **30 AdGuard A-records** on CT 102 — per-host (`pve`, `nas`, `unifi`, `dlink`, `backrest`, etc.) + per-service aliases (`immich`, `vault`, `karakeep`, `grafana`, `prometheus`, …).
|
||||
- Documented in `services/adguard-dns.md`.
|
||||
- Solves "client registered to old IP" gotcha — when service moves CT, edit one AdGuard rewrite instead of N apps.
|
||||
- D-Link IP captured: `192.168.1.10`.
|
||||
- Filed feedback memory: always use `*.nuclide.lan` aliases in URLs, never bare IPs.
|
||||
|
||||
### Homarr
|
||||
- **3 new boards** built via SQL: `home` (morning routine, default), `admin` (by-tier), `command` (live-ops iframes + grid). Old "nuclide" board was lost when OIDC user was recreated for password reset.
|
||||
- Boards owned by `fkrebs@nucli.de`, `is_public=1`. 33 unique apps with ping status.
|
||||
- Integrations not yet wired (Proxmox / Docker / AdGuard / Gotify widgets — need tokens).
|
||||
|
||||
### Pocket-ID
|
||||
- **LAN callback URLs added** for Homarr, Grafana, Infisical, Proxmox VE so OIDC works via both bare-IP and `.nuclide.lan` alias paths.
|
||||
|
||||
### Other
|
||||
- **Immich Android upload bug** diagnosed: HTTP/2 + `Keep-Alive` header conflict. Workaround live (per-SSID LAN URL). Permanent fix pending Zoraxy header strip.
|
||||
- **Nexa #1 filed** at https://git.nuclide.systems/fkrebs/nexa/issues/1 — 9 design patterns from Obsidian + Claude session as input for Nexa Phase 1–2.
|
||||
- **Wi-Fi SSID captured**: `nuclide`. Used for per-network app routing (Immich, HA Companion).
|
||||
|
||||
## Recently completed (2026-05-23, session continued ×4)
|
||||
|
||||
@@ -128,3 +169,5 @@ Last updated: 2026-05-23 (session continued ×4).
|
||||
- CT 104 has a separate agent reconciling compose files — rename decommissioned files (don't just stop), update `/opt/stacks/CLAUDE.md`.
|
||||
- ComfyUI container limit: 24 G. Do not raise without measuring host impact.
|
||||
- Pocket-ID is SQLite-only — no Postgres migration possible.
|
||||
- Use `*.nuclide.lan` aliases when presenting URLs to the user (never bare IPs). Wi-Fi SSID is `nuclide`.
|
||||
- Vault edits append to `Work/Journal/YYYY/MM/YYYY-MM-DD.md` under `## 🤖 Claude edits`; never create a parallel folder.
|
||||
|
||||
@@ -44,8 +44,10 @@ Pocket-ID is a lightweight OIDC 2.1 / OAuth 2.0 IdP. Every service that supports
|
||||
| Arcane | 104 | `https://arcane.nuclide.systems/auth/callback` | OIDC_CLIENT_SECRET rotated 2026-05-22 |
|
||||
| n8n | 104 | `https://n8n.nuclide.systems/rest/oauth2-credential/callback` | encryption key rotated 2026-05-22 |
|
||||
| Vaultwarden | 104 | `https://vault.nuclide.systems/identity/connect/token` | client created 2026-05-21; auth flow not yet wired |
|
||||
| Homarr | 109 | `http://192.168.1.8:7575/api/auth/callback/oidc` | SSO active 2026-05-23; client `63a94e30` |
|
||||
| Grafana | 109 | `http://192.168.1.8:3000/login/generic_oauth` | SSO active 2026-05-23; client `92d987d5`; PKCE enabled |
|
||||
| Homarr | 109 | `http://192.168.1.8:7575/api/auth/callback/oidc` + `http://homarr.nuclide.lan:7575/...` | SSO active 2026-05-23; client `63a94e30`; LAN alias added 2026-05-24 |
|
||||
| Grafana | 109 | `http://192.168.1.8:3000/login/generic_oauth` + `http://grafana.nuclide.lan:3000/...` | SSO active 2026-05-23; client `92d987d5`; PKCE enabled; LAN alias added 2026-05-24 |
|
||||
| Infisical | 112 | `http://192.168.1.7:8200/api/v1/sso/oidc/callback` + `http://secrets.nuclide.lan:8200/...` | LAN alias added 2026-05-24; manual OIDC config still pending |
|
||||
| Proxmox VE | host | `https://192.168.1.20:8006` + `https://pve.nuclide.lan:8006` | LAN alias added 2026-05-24 |
|
||||
|
||||
## Env var patterns per service type
|
||||
|
||||
|
||||
Reference in New Issue
Block a user