From a8ed4350f09496b13e058747cdf2943aa56fddfd Mon Sep 17 00:00:00 2001 From: "fkrebs (via Claude)" Date: Sat, 23 May 2026 03:44:05 +0200 Subject: [PATCH] =?UTF-8?q?ops:=20CT=20109=20provisioned=20=E2=80=94=20Pro?= =?UTF-8?q?metheus=20+=20Grafana=20monitoring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Debian 13, Docker 29.5, 4c/4G/32G at 192.168.1.8 - Prometheus :9090 (90d retention), Grafana :3000 (LAN only) - Scrapes: litellm CT104:14000, node-ct104:9100, node-ct109:9100 - CT 104 standalone node-exporter retained at /opt/stacks/monitoring/ - All 4 targets confirmed up Co-Authored-By: Claude Sonnet 4.6 --- ct-inventory.md | 4 +++- infra/portmap.md | 17 +++++++++-------- services/homelab-architecture.md | 6 +++--- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/ct-inventory.md b/ct-inventory.md index e869fb6..b147dda 100644 --- a/ct-inventory.md +++ b/ct-inventory.md @@ -11,7 +11,7 @@ Verified 2026-05-20 via `pct list`, `pct config `, `qm config 100` on `nuc`. | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | | 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 NFS | Intel iGPU (render only) | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | -| 109 | ops | LXC | ⚠️ TBD (`.6`=CT 113, `.7`=CT 112) — likely `.8` | 4 | 8192 | 50 GiB | **Planned** — Prometheus, Loki, Grafana, Arcane Manager, Dozzle, Homarr, Diun, sshwifty, Tinyauth, socket-proxy; Alloy agents on all hosts | — | no | **planned** (see `infra/proxmox-state.md` §16) | +| 109 | ops | LXC unpriv | 192.168.1.8/24 | 4 | 4096 | 32 GiB | **Ops** — Prometheus + Grafana + node-exporter; future: Loki, Arcane, Dozzle, Homarr, Tinyauth | — | no | running (2026-05-23) | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 112 | secrets | LXC unpriv | 192.168.1.7/24 | 2 | 4096 | 20 GiB | **Infisical** secrets manager (own Postgres 16 + Redis 7; replaces scattered .env files) — LAN-only, no Zoraxy route | — | no | running | @@ -48,3 +48,5 @@ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuVqBW3VXg... root@nuc ``` CT 113 ("db") provisioned 2026-05-21; see `services/databases.md`. Future role: consolidate per-stack Postgres instances once second NVMe lands. + +CT 109 ("ops") provisioned 2026-05-23. Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring`: Prometheus (:9090), Grafana (:3000 — LAN only, admin/tapirnase), node-exporter (:9100 host-net). Scrapes: `litellm` on CT 104 `:14000/metrics/`, `node-ct104` `:9100`, `node-ct109` `:9100`. CT 104 runs a standalone node-exporter (`/opt/stacks/monitoring/`) for CT 109 to scrape. diff --git a/infra/portmap.md b/infra/portmap.md index 2de6f0e..8130521 100644 --- a/infra/portmap.md +++ b/infra/portmap.md @@ -26,16 +26,17 @@ --- -## Monitoring (9090 / 9091 / 9100) +## Monitoring — CT 109 ops (192.168.1.8) -| Port | Service | Container | Public URL | Notes | -|---|---|---|---|---| -| [9090](http://192.168.1.40:9090) | Prometheus | `prometheus` | — | LAN only; stack `/opt/stacks/monitoring` | -| [9091](http://192.168.1.40:9091) | Grafana | `grafana` | `grafana.nuclide.systems` (Zoraxy route pending) | admin pw in `.env`; OIDC pending PocketID client | -| 9100 | node-exporter | `node-exporter` | — | host-network mode; scrapes CT 104 | +| Port | Host | Service | Notes | +|---|---|---|---| +| [9090](http://192.168.1.8:9090) | CT 109 | Prometheus | LAN only; 90d retention | +| [3000](http://192.168.1.8:3000) | CT 109 | Grafana | LAN only; admin/tapirnase; import dashboard ID 21501 for LiteLLM | +| 9100 | CT 109 | node-exporter | host-network, self-scrape | +| 9100 | CT 104 | node-exporter | standalone stack `/opt/stacks/monitoring/`; scraped by CT 109 | -Scrape targets: `litellm` (`:14000/metrics/`, bearer `sk-tapirnase`), `node-ct104` (`:9100`), `prometheus` (`:9090`). -LiteLLM metrics enabled via `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-config/config.yaml`. +Scrape targets (CT 109 Prometheus): `litellm` CT104:14000/metrics/ (bearer sk-tapirnase), `node-ct104` :9100, `node-ct109` :9100, `prometheus` self. +LiteLLM metrics: `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-config/config.yaml`. --- diff --git a/services/homelab-architecture.md b/services/homelab-architecture.md index 1e44f68..44066fc 100644 --- a/services/homelab-architecture.md +++ b/services/homelab-architecture.md @@ -24,7 +24,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO, - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4` - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 - - `109` lxc **ops** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md` + - `109` lxc **ops** `192.168.1.8` — Prometheus + Grafana + node-exporter (2026-05-23); future: Loki, Arcane, Dozzle, Homarr, Tinyauth - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21 - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series @@ -394,7 +394,7 @@ flowchart TB docker104["CT 104 · docker
(.40) · 16c/48G/200G
~65 containers · Intel Arc passthrough"] nc["CT 105 · nextcloud
(.41)
Nextcloud AIO (NFS)"] zoraxy["CT 108 · zoraxy
(.4)
reverse proxy + ACME"] - obs(["CT 109 · ops
(.8) · planned
Prom / Graf / Arcane / Homarr / Tinyauth"]) + obs["CT 109 · ops
(.8)
Prometheus · Grafana"] id["CT 110 · id
(.5)
Pocket-ID (moved here 2026-05-20)"] dev["CT 111 · dev
(.42) · 12c/32G/60G
Coder + Gitea + workspaces · Intel Arc"] db["CT 113 · db
(.6) · provisioned 2026-05-21
shared Postgres + pgAdmin"] @@ -412,7 +412,7 @@ flowchart TB UNAS <--> dev classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa - class obs,db planned + class db planned ``` ### Auth plane — Pocket-ID is the universal IdP