ops: CT 109 provisioned — Prometheus + Grafana monitoring

- Debian 13, Docker 29.5, 4c/4G/32G at 192.168.1.8
- Prometheus :9090 (90d retention), Grafana :3000 (LAN only)
- Scrapes: litellm CT104:14000, node-ct104:9100, node-ct109:9100
- CT 104 standalone node-exporter retained at /opt/stacks/monitoring/
- All 4 targets confirmed up

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-23 03:44:05 +02:00
parent 644ba72d77
commit a8ed4350f0
3 changed files with 15 additions and 12 deletions
+3 -1
View File
@@ -11,7 +11,7 @@ Verified 2026-05-20 via `pct list`, `pct config <id>`, `qm config 100` on `nuc`.
| 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running | | 104 | docker | LXC unpriv (idmapped) | 192.168.1.40/24 | 16 | 49152 | 200 GiB | Main Docker host — AI/ML + media + identity-adjacent (~65 containers) | mp0 NFS | Intel iGPU (card+render) | running |
| 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 NFS | Intel iGPU (render only) | running | | 105 | nextcloud | LXC priv | 192.168.1.41/24 | 4 | 8196 | 100 GiB | Nextcloud AIO | mp0 NFS | Intel iGPU (render only) | running |
| 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running | | 108 | zoraxy | LXC unpriv | 192.168.1.4/24 | 2 | 2048 | 6 GiB | **Zoraxy** reverse proxy + ACME (`*.nuclide.systems`) | — | no | running |
| 109 | ops | LXC | ⚠️ TBD (`.6`=CT 113, `.7`=CT 112) — likely `.8` | 4 | 8192 | 50 GiB | **Planned** — Prometheus, Loki, Grafana, Arcane Manager, Dozzle, Homarr, Diun, sshwifty, Tinyauth, socket-proxy; Alloy agents on all hosts | — | no | **planned** (see `infra/proxmox-state.md` §16) | | 109 | ops | LXC unpriv | 192.168.1.8/24 | 4 | 4096 | 32 GiB | **Ops** — Prometheus + Grafana + node-exporter; future: Loki, Arcane, Dozzle, Homarr, Tinyauth | — | no | running (2026-05-23) |
| 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running | | 110 | id | LXC unpriv | 192.168.1.5/24 | 1 | 1024 | 4 GiB | **Pocket-ID** OIDC IdP (`id.nuclide.systems`); migrated off CT 104 on 2026-05-20 | — | no | running |
| 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running | | 111 | dev | LXC unpriv | 192.168.1.42/24 | 12 | 32768 | 60 GiB | Dev host — **Coder + Gitea** + workspaces | mp0 NFS | Intel iGPU (render only; symlinked at `/dev/dri/renderD128`) | running |
| 112 | secrets | LXC unpriv | 192.168.1.7/24 | 2 | 4096 | 20 GiB | **Infisical** secrets manager (own Postgres 16 + Redis 7; replaces scattered .env files) — LAN-only, no Zoraxy route | — | no | running | | 112 | secrets | LXC unpriv | 192.168.1.7/24 | 2 | 4096 | 20 GiB | **Infisical** secrets manager (own Postgres 16 + Redis 7; replaces scattered .env files) — LAN-only, no Zoraxy route | — | no | running |
@@ -48,3 +48,5 @@ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuVqBW3VXg... root@nuc
``` ```
CT 113 ("db") provisioned 2026-05-21; see `services/databases.md`. Future role: consolidate per-stack Postgres instances once second NVMe lands. CT 113 ("db") provisioned 2026-05-21; see `services/databases.md`. Future role: consolidate per-stack Postgres instances once second NVMe lands.
CT 109 ("ops") provisioned 2026-05-23. Debian 13, Docker 29.5.2. Stack `/opt/stacks/monitoring`: Prometheus (:9090), Grafana (:3000 — LAN only, admin/tapirnase), node-exporter (:9100 host-net). Scrapes: `litellm` on CT 104 `:14000/metrics/`, `node-ct104` `:9100`, `node-ct109` `:9100`. CT 104 runs a standalone node-exporter (`/opt/stacks/monitoring/`) for CT 109 to scrape.
+9 -8
View File
@@ -26,16 +26,17 @@
--- ---
## Monitoring (9090 / 9091 / 9100) ## Monitoring — CT 109 ops (192.168.1.8)
| Port | Service | Container | Public URL | Notes | | Port | Host | Service | Notes |
|---|---|---|---|---| |---|---|---|---|
| [9090](http://192.168.1.40:9090) | Prometheus | `prometheus` | — | LAN only; stack `/opt/stacks/monitoring` | | [9090](http://192.168.1.8:9090) | CT 109 | Prometheus | LAN only; 90d retention |
| [9091](http://192.168.1.40:9091) | Grafana | `grafana` | `grafana.nuclide.systems` (Zoraxy route pending) | admin pw in `.env`; OIDC pending PocketID client | | [3000](http://192.168.1.8:3000) | CT 109 | Grafana | LAN only; admin/tapirnase; import dashboard ID 21501 for LiteLLM |
| 9100 | node-exporter | `node-exporter` | — | host-network mode; scrapes CT 104 | | 9100 | CT 109 | node-exporter | host-network, self-scrape |
| 9100 | CT 104 | node-exporter | standalone stack `/opt/stacks/monitoring/`; scraped by CT 109 |
Scrape targets: `litellm` (`:14000/metrics/`, bearer `sk-tapirnase`), `node-ct104` (`:9100`), `prometheus` (`:9090`). Scrape targets (CT 109 Prometheus): `litellm` CT104:14000/metrics/ (bearer sk-tapirnase), `node-ct104` :9100, `node-ct109` :9100, `prometheus` self.
LiteLLM metrics enabled via `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-config/config.yaml`. LiteLLM metrics: `litellm_settings.callbacks: ["prometheus"]` in `ai/litellm-config/config.yaml`.
--- ---
+3 -3
View File
@@ -24,7 +24,7 @@ across ~23 compose stacks. Principle: **self-host everything**, OIDC SSO,
- `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4` - `108` lxc **zoraxy** — reverse proxy (2c/2G) → `.4`
- `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20 - `110` lxc **id** — Pocket-ID OIDC IdP (1c/1G/4G) → `.5`; migrated off 104 on 2026-05-20
- `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20 - `111` lxc **dev** — Coder + Gitea (12c/32G/60G) → `.42`; new 2026-05-20
- `109` lxc **ops** *(planned)* — Prometheus/Grafana/Arcane/Dozzle/Homarr/Tinyauth; see §16 of `infra/proxmox-state.md` - `109` lxc **ops** `192.168.1.8` — Prometheus + Grafana + node-exporter (2026-05-23); future: Loki, Arcane, Dozzle, Homarr, Tinyauth
- `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21 - `113` lxc **db** — postgres 17 + pgAdmin + WAL-G → Garage S3 + Arcane edge agent → `.6`; provisioned 2026-05-21
- Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T). - Storages: `local` (dir), `local-zfs` (zfspool, ~1.9T), `unas` (nfs, ~20T).
- **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series - **Ubiquiti UniFi** — UDM "Home" (UDMA6A8, UCG Fiber) gateway + 4× U7-series
@@ -394,7 +394,7 @@ flowchart TB
docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"] docker104["<b>CT 104 · docker</b><br/>(.40) · 16c/48G/200G<br/>~65 containers · Intel Arc passthrough"]
nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"] nc["CT 105 · nextcloud<br/>(.41)<br/>Nextcloud AIO (NFS)"]
zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"] zoraxy["CT 108 · zoraxy<br/>(.4)<br/>reverse proxy + ACME"]
obs(["CT 109 · ops<br/>(.8) · <i>planned</i><br/>Prom / Graf / Arcane / Homarr / Tinyauth"]) obs["CT 109 · ops<br/>(.8)<br/>Prometheus · Grafana"]
id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"] id["CT 110 · id<br/>(.5)<br/>Pocket-ID (moved here 2026-05-20)"]
dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"] dev["CT 111 · dev<br/>(.42) · 12c/32G/60G<br/>Coder + Gitea + workspaces · Intel Arc"]
db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"] db["CT 113 · db<br/>(.6) · provisioned 2026-05-21<br/>shared Postgres + pgAdmin"]
@@ -412,7 +412,7 @@ flowchart TB
UNAS <--> dev UNAS <--> dev
classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa classDef planned stroke-dasharray:5 5,fill:#222,stroke:#aaa,color:#aaa
class obs,db planned class db planned
``` ```
### Auth plane — Pocket-ID is the universal IdP ### Auth plane — Pocket-ID is the universal IdP