docs: 2026-05-23 session — observability stack, Pocket-ID guide, secrets scrub

- CHANGELOG: full 2026-05-23 backfill (Loki, Alloy×12 hosts, pve-exporter,
  HA prometheus integration, 5 dashboards, 3 alert rules, Homarr, homelab-configs repo)
- services/pocket-id.md: new — OIDC endpoints, client creation walkthrough,
  per-service env var patterns, current client registry, backup notes
- infra/proxmox-state.md: redact D-Link credentials from plaintext (pelican/pinkpanther)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CHANGELOG.md

@@ -2,6 +2,56 @@
 
 All notable infrastructure / service / doc changes. Newest first.
 
+## 2026-05-23
+
+### Observability stack (CT 109 "ops")
+
+- **CT 109 ("ops") fully provisioned**: Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring/` running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in `ct-inventory.md`.
+
+- **Loki deployed on CT 109**: log aggregation at `:3100`, 30-day retention, TSDB v13 schema, filesystem storage at `./loki/data/`. `limits_config` set `reject_old_samples_max_age: 168h` (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via `provisioning/datasources/loki.yml`. Grafana Loki datasource UID: `P8E80F9AEF21F6940`.
+
+- **Prometheus remote_write receiver enabled**: `--web.enable-remote-write-receiver` added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.
+
+- **prometheus-pve-exporter deployed**: `prompve/prometheus-pve-exporter:latest` on CT 109; credentials in `./pve-exporter/pve.yml` (chmod 644 — container runs non-root). PVE `monitor@pve` service user + API token `c4be45dc-...` created with `PVEAuditor` role on `/`. Prometheus scrapes `:9221/pve?target=192.168.1.20`. Metrics: per-VM/CT CPU, memory, disk, network I/O.
+
+- **HA prometheus integration added**: `prometheus:` block in HA `configuration.yaml`; HA exposes ~9211 `hass_*` entity-level metrics at `:8123/api/prometheus`. CT 109 Prometheus scrapes with bearer token (HA long-lived token), `scrape_interval: 60s`. This is distinct from Alloy host-level metrics — both needed.
+
+- **Grafana Alloy deployed across all 12 hosts**: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to `http://192.168.1.8:3100/loki/api/v1/push`. Deployment breakdown:
+  - **Docker Compose** (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, `group_add: ["999"]` for journal access.
+  - **Binary + systemd** (CT 102, 103, 108, nuc): apt install `alloy`, config at `/etc/alloy/config.alloy`, `systemctl enable --now alloy`. Journal-only (no Docker socket on these hosts).
+  - **HA add-on** (VM 100): `wymangr/hassos-addons` Grafana Alloy v0.0.8 — config fills in `prometheus_remote_write` to CT 109 + Loki endpoint. Distinct host label `homeassistant`.
+  - All configs stored in `/opt/homelab-configs/alloy/` (see homelab-configs repo below).
+
+- **5 Grafana dashboards provisioned**:
+  - Node Exporter Full (Grafana ID 1860) — imported for all hosts
+  - LiteLLM (custom-built) — `/d/.../litellm`; panels: request rate, error rate, token throughput, model split, p95 latency
+  - Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
+  - Loki / log explorer — imported
+  - Prometheus target health — imported
+
+- **3 Grafana alert rules created** via `/api/v1/provisioning/alert-rules` (folder `alerts-folder`):
+  - `Node disk > 85%` — per-instance mountpoint disk usage, 5m window
+  - `Prometheus target down` — `count(up == 0) > 0`, 5m window
+  - `LiteLLM error rate > 5%` — 10m rate window, 5m pending
+
+### Homarr (CT 109)
+
+- **Homarr v1 deployed**: `ghcr.io/homarr-labs/homarr:latest` on CT 109 `:7575`. Data at `./data/`, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).
+
+- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.
+
+### Pocket-ID OIDC (pending user action)
+
+- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate.
+
+- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars.
+
+### Docs
+
+- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config).
+- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
+- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.
+
 ## 2026-05-22
 
 - **Pocket-ID backup wired into Backrest**: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.

infra/proxmox-state.md

@@ -819,7 +819,7 @@ Remediation paths in order of effort:
 2. **Add `snmpd` to the host** + generic SNMP device in UniFi → CPU/mem/iface stats from Proxmox visible in UniFi (not in topology, but in monitoring).
 3. **UniFi-managed switch between host and UDM** — clean answer; requires hardware.
 
-D-Link DGS-1210 admin UI lives at `http://192.168.1.10/` (login page, field names `pelican` / `pinkpanther`). Verify the admin password is non-default — DGS-1210 ships with `admin`/blank or `admin`/`admin` on most firmware revisions. A flat-LAN switch with default creds is one of the easier vectors.
+D-Link DGS-1210 admin UI lives at `http://192.168.1.10/`. Verify the admin password is non-default — DGS-1210 ships with `admin`/blank or `admin`/`admin` on most firmware revisions. A flat-LAN switch with default creds is one of the easier vectors. *(Credentials redacted from this doc — check your password manager.)*
 
 Useful inspection from the host any time: `lldpcli show neighbors`.
 

services/pocket-id.md

@@ -0,0 +1,125 @@
+# Pocket-ID — OIDC Identity Provider
+
+**CT 110 "id"** · `192.168.1.5:11000` · `https://id.nuclide.systems`  
+Migrated from CT 104 on 2026-05-20. SQLite-only (no Postgres).
+
+## What it does
+
+Pocket-ID is a lightweight OIDC 2.1 / OAuth 2.0 IdP. Every service that supports OIDC can delegate login to it — one account, one MFA setup, SSO across the homelab. Clients are managed via an admin UI; there is no API key / scripted client creation.
+
+## Admin access
+
+- URL: `https://id.nuclide.systems` (admin panel is the default view when logged in as admin)
+- Admin user: `fkrebs@nucli.de`
+- No API endpoint for client management — use the web UI.
+
+## OIDC endpoints (standard discovery)
+
+| Endpoint | URL |
+|---|---|
+| Discovery | `https://id.nuclide.systems/.well-known/openid-configuration` |
+| Authorization | `https://id.nuclide.systems/authorize` |
+| Token | `https://id.nuclide.systems/api/oidc/token` |
+| Userinfo | `https://id.nuclide.systems/api/oidc/userinfo` |
+| JWKS | `https://id.nuclide.systems/api/oidc/jwks` |
+
+## Creating a new OIDC client
+
+1. Go to `https://id.nuclide.systems` → **OIDC Clients** → **New Client**
+2. Fill in:
+   - **Name**: descriptive (e.g. "Homarr", "Grafana")
+   - **Redirect URIs**: the callback URL the service expects (see per-service table below)
+   - **PKCE**: enable if the service supports it (preferred)
+3. Copy the **Client ID** and **Client Secret** — secret is shown once.
+4. Paste into the service's env vars (see patterns below).
+
+> Pocket-ID 2.7.0+ stores secrets as bcrypt hashes — the plaintext secret is only visible at creation time. If lost, regenerate in the client edit view.
+
+## Current clients
+
+| Service | CT | Redirect URI | Notes |
+|---|---|---|---|
+| Gitea | 111 | `https://git.nuclide.systems/user/oauth2/pocket-id/callback` | SSO active; local password sign-in disabled |
+| Coder | 111 | `https://dev.nuclide.systems/api/v2/users/oidc/callback` | SSO active; password auth disabled |
+| Arcane | 104 | `https://arcane.nuclide.systems/auth/callback` | OIDC_CLIENT_SECRET rotated 2026-05-22 |
+| n8n | 104 | `https://n8n.nuclide.systems/rest/oauth2-credential/callback` | encryption key rotated 2026-05-22 |
+| Vaultwarden | 104 | `https://vault.nuclide.systems/identity/connect/token` | client created 2026-05-21; auth flow not yet wired |
+| Homarr | 109 | `http://192.168.1.8:7575/api/auth/callback/oidc` | **pending** — client not yet created |
+| Grafana | 109 | `http://192.168.1.8:3000/login/generic_oauth` | **pending** — client not yet created |
+
+## Env var patterns per service type
+
+### Homarr (v1 homarr-labs)
+
+```env
+AUTH_PROVIDER=oidc
+AUTH_OIDC_ISSUER=https://id.nuclide.systems
+AUTH_OIDC_CLIENT_ID=<client-id>
+AUTH_OIDC_CLIENT_SECRET=<client-secret>
+AUTH_OIDC_SCOPE=openid profile email
+AUTH_OIDC_NAME=Pocket-ID
+```
+
+### Grafana
+
+```env
+GF_AUTH_GENERIC_OAUTH_ENABLED=true
+GF_AUTH_GENERIC_OAUTH_NAME=Pocket-ID
+GF_AUTH_GENERIC_OAUTH_CLIENT_ID=<client-id>
+GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<client-secret>
+GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
+GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://id.nuclide.systems/authorize
+GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://id.nuclide.systems/api/oidc/token
+GF_AUTH_GENERIC_OAUTH_API_URL=https://id.nuclide.systems/api/oidc/userinfo
+GF_AUTH_SIGNOUT_REDIRECT_URL=https://id.nuclide.systems/logout
+GF_AUTH_GENERIC_OAUTH_USE_PKCE=true
+GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
+GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(groups[*], 'admins') && 'Admin' || 'Viewer'
+```
+
+### Gitea
+
+```ini
+[oauth2]
+ENABLED = true
+
+[service]
+ENABLE_PASSWORD_SIGNIN_FORM = false
+```
+Auth source added via Gitea admin UI → Authentication Sources → OAuth2 → OpenID Connect.
+
+### Coder
+
+```env
+CODER_OIDC_ISSUER_URL=https://id.nuclide.systems
+CODER_OIDC_CLIENT_ID=<client-id>
+CODER_OIDC_CLIENT_SECRET=<client-secret>
+CODER_OIDC_SCOPES=openid,profile,email
+CODER_DISABLE_PASSWORD_AUTH=true
+```
+
+### Generic (any service with standard OIDC)
+
+```
+Issuer:        https://id.nuclide.systems
+Auth URL:      https://id.nuclide.systems/authorize
+Token URL:     https://id.nuclide.systems/api/oidc/token
+Userinfo URL:  https://id.nuclide.systems/api/oidc/userinfo
+JWKS URL:      https://id.nuclide.systems/api/oidc/jwks
+Scopes:        openid profile email
+```
+
+## Backup
+
+Pocket-ID SQLite DB + signing keys are backed up nightly via Backrest (CT 103). Pre-backup hook SSHs to CT 110, runs `pocket-id export` inside the container, copies the ZIP + signing keys to UNAS staging path, then the `services-backup-plan` snapshots to JottaCloud. See `services/backrest.md`.
+
+## Stack location
+
+CT 110 (`192.168.1.5`): `/opt/stacks/pocketid/docker-compose.yml`
+
+```yaml
+# key env vars
+POCKET_ID_URL=https://id.nuclide.systems
+TRUST_PROXY=true
+MAXMIND_LICENSE_KEY=...  # optional geo-IP
+```