fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: 2026-05-23 session — observability stack, Pocket-ID guide, secrets scrub

Browse Source 
- CHANGELOG: full 2026-05-23 backfill (Loki, Alloy×12 hosts, pve-exporter,
  HA prometheus integration, 5 dashboards, 3 alert rules, Homarr, homelab-configs repo)
- services/pocket-id.md: new — OIDC endpoints, client creation walkthrough,
  per-service env var patterns, current client registry, backup notes
- infra/proxmox-state.md: redact D-Link credentials from plaintext (pelican/pinkpanther)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Florian Krebs
2026-05-23 08:03:01 +02:00
parent 3d114c6873
commit 474af687af
3 changed files with 176 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+50
View File
@@ -2,6 +2,56 @@
All notable infrastructure / service / doc changes. Newest first.
## 2026-05-23
### Observability stack (CT 109 "ops")
- **CT 109 ("ops") fully provisioned**: Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring/` running Prometheus (:9090), Grafana (:3000 LAN-only, admin/tapirnase), node-exporter, Loki, Alloy, pve-exporter, and Gotify-bridge. CT info updated in `ct-inventory.md`.
- **Loki deployed on CT 109**: log aggregation at `:3100`, 30-day retention, TSDB v13 schema, filesystem storage at `./loki/data/`. `limits_config` set `reject_old_samples_max_age: 168h` (7 days) to handle log backfill from HA restart. Grafana datasource auto-provisioned via `provisioning/datasources/loki.yml`. Grafana Loki datasource UID: `P8E80F9AEF21F6940`.
- **Prometheus remote_write receiver enabled**: `--web.enable-remote-write-receiver` added to CT 109 Prometheus. Required for HA Alloy add-on to push metrics directly.
- **prometheus-pve-exporter deployed**: `prompve/prometheus-pve-exporter:latest` on CT 109; credentials in `./pve-exporter/pve.yml` (chmod 644 — container runs non-root). PVE `monitor@pve` service user + API token `c4be45dc-...` created with `PVEAuditor` role on `/`. Prometheus scrapes `:9221/pve?target=192.168.1.20`. Metrics: per-VM/CT CPU, memory, disk, network I/O.
- **HA prometheus integration added**: `prometheus:` block in HA `configuration.yaml`; HA exposes ~9211 `hass_*` entity-level metrics at `:8123/api/prometheus`. CT 109 Prometheus scrapes with bearer token (HA long-lived token), `scrape_interval: 60s`. This is distinct from Alloy host-level metrics — both needed.
- **Grafana Alloy deployed across all 12 hosts**: Alloy v1.16.1 from Grafana apt repo. River-syntax configs ship Docker + journal log scraping with host labels to `http://192.168.1.8:3100/loki/api/v1/push`. Deployment breakdown:
- **Docker Compose** (CT 101, 104, 105, 110, 111, 112, 113): container with docker socket + journal mounts, `group_add: ["999"]` for journal access.
- **Binary + systemd** (CT 102, 103, 108, nuc): apt install `alloy`, config at `/etc/alloy/config.alloy`, `systemctl enable --now alloy`. Journal-only (no Docker socket on these hosts).
- **HA add-on** (VM 100): `wymangr/hassos-addons` Grafana Alloy v0.0.8 — config fills in `prometheus_remote_write` to CT 109 + Loki endpoint. Distinct host label `homeassistant`.
- All configs stored in `/opt/homelab-configs/alloy/` (see homelab-configs repo below).
- **5 Grafana dashboards provisioned**:
- Node Exporter Full (Grafana ID 1860) — imported for all hosts
- LiteLLM (custom-built) — `/d/.../litellm`; panels: request rate, error rate, token throughput, model split, p95 latency
- Home Assistant (Grafana ID 12481 + custom panels) — power/energy, climate (temp/humidity/CO₂), home activity, device health/battery
- Loki / log explorer — imported
- Prometheus target health — imported
- **3 Grafana alert rules created** via `/api/v1/provisioning/alert-rules` (folder `alerts-folder`):
- `Node disk > 85%` — per-instance mountpoint disk usage, 5m window
- `Prometheus target down` — `count(up == 0) > 0`, 5m window
- `LiteLLM error rate > 5%` — 10m rate window, 5m pending
### Homarr (CT 109)
- **Homarr v1 deployed**: `ghcr.io/homarr-labs/homarr:latest` on CT 109 `:7575`. Data at `./data/`, docker socket bind-mounted read-only for auto-discovery. LAN-only (no Zoraxy route).
- **homelab-configs Gitea repo**: private repo `fkrebs/homelab-configs` at `git.nuclide.systems`; cloned to `/opt/homelab-configs` on CT 109. Daily cron at 03:00 syncs: Alloy configs per host, Homarr docker-compose, Homarr SQLite dump. Script at `/usr/local/bin/sync-homelab-configs`. Commit + push on any diff.
### Pocket-ID OIDC (pending user action)
- **Homarr OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:7575/api/auth/callback/oidc`. After creation, update Homarr compose with `AUTH_PROVIDER=oidc` + `AUTH_OIDC_*` vars and force-recreate.
- **Grafana OIDC**: client needs to be created manually at `https://id.nuclide.systems`. Redirect URI: `http://192.168.1.8:3000/login/generic_oauth`. After creation, update monitoring compose with `GF_AUTH_GENERIC_OAUTH_*` vars.
### Docs
- **`README.md` expanded**: monitoring quick-links table (Homarr, Grafana, Prometheus, Loki, Alloy UI), public services table by category, operator-only LAN links, related repos table (homelab-configs, klipper-config).
- **`infra/portmap.md` updated**: added Loki :3100, pve-exporter :9221, Alloy :12345, Homarr :7575; Alloy deployment notes across all 12 hosts; updated monitoring section.
- **`services/homelab-architecture.md`** CT 109 description updated to reflect full stack.
## 2026-05-22
- **Pocket-ID backup wired into Backrest**: CT 110 SQLite-only (no Postgres). Pre-backup hook script at `/opt/backrest/scripts/pocketid-prestage.sh` on CT 103: SSHs to CT 110, runs `pocket-id export` inside container, `docker cp`s ZIP to host, scp's ZIP + signing keys to `/mnt/pve/unas/services/pocketid-backup/`. `services-backup-plan` snapshots staging → jottacloud. Verified: hook fires before snapshot, snapshot includes Pocket-ID data.