fkrebs
895658a115
- zoraxy.md: fix arcane upstream to CT109 (192.168.1.8:10002), add SkipWebSocketOriginCheck column, note missing routes (dozzle, etc.) - arcane.md: reflect CT109 migration complete, expand agent table to all 8 environments, fix MANAGER_API_URL and DB paths - ct-inventory.md: CT103 RAM 512→4096+swap; CT109 footnote complete - proxmox-memory-audit.md: CT103 bump, add CT109+CT113 rows, fix sum - portmap.md: add Wetty row, fix arcane backend, WAL-G scrape target, homepage decommissioned, dozzle LAN-only note - mcp-gateway.md: add gitea/paperless/proxmox, count 26→29 servers - RESUME.md: check off WAL-G, Loki, Zoraxy audit; CT109 in key state table; session ×4 completed items block Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
83 lines
5.4 KiB
Markdown
83 lines
5.4 KiB
Markdown
# Proxmox Memory Audit
|
||
|
||
Last audited: 2026-05-22
|
||
|
||
## Host physical resources
|
||
|
||
| Resource | Total | Used (idle) | Available |
|
||
|----------|-------|-------------|-----------|
|
||
| RAM | 62 GiB | ~33 GiB | ~28 GiB |
|
||
| Swap | 31 GiB | 0 GiB | 31 GiB |
|
||
|
||
## CT memory allocations
|
||
|
||
| CT | Name | Allocated (MiB) | Swap (MiB) | Typical use | Notes |
|
||
|----|------|----------------|------------|-------------|-------|
|
||
| 101 | shepard | 32,768 | 8,192 | ~4 GB | Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak |
|
||
| 102 | dns | 1,024 | 512 | ~100 MB | AdGuard Home |
|
||
| 103 | backrest | **4,096** | **1,024** | ~200 MB | Restic scheduler — bumped 2026-05-23 for 822 GB initial backup OOM fix |
|
||
| 104 | docker | **49,152** | 32,000 | **6–22 GB** | Main Docker host; FLUX spikes to ~22 GB |
|
||
| 105 | nextcloud | 8,196 | 8,196 | ~2 GB | Nextcloud AIO |
|
||
| 108 | zoraxy | 2,048 | 512 | ~300 MB | Reverse proxy |
|
||
| 109 | ops | 4,096 | 0 | ~1.5 GB | Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr |
|
||
| 110 | id | 1,024 | 512 | ~200 MB | Pocket-ID |
|
||
| 111 | dev | 32,768 | 8,192 | ~3 GB | Coder + Gitea workspaces |
|
||
| 112 | secrets | 4,096 | 512 | ~600 MB | Infisical |
|
||
| 113 | db | 4,096 | 0 | ~800 MB | Postgres 17 + WAL-G |
|
||
| **Sum** | | **141,316 MiB (138 GiB)** | | | **2.2× overprovisioned vs physical RAM** |
|
||
|
||
## Key findings
|
||
|
||
### Overprovisioning is safe — until it isn't
|
||
|
||
Proxmox uses balloon drivers so CTs only consume what they actually use. At idle the host sits at ~33 GB used with 28 GB available. This is healthy. However, two specific CTs represent risk:
|
||
|
||
- **CT 104** (49 GiB limit) + **CT 111** (32 GiB limit) together could claim 81 GiB — well over the 62 GiB physical. If both hit peak simultaneously the host would start swapping heavily.
|
||
- CT 104's ComfyUI (FLUX generation) spikes from 6 GB baseline to ~22 GB. That 16 GB spike on top of the 33 GB idle baseline = 49 GB total, within physical RAM but tight.
|
||
|
||
### Why 40G Docker container limit destabilised the system
|
||
|
||
Setting ComfyUI's container limit to 40 G was the trigger. At generation time:
|
||
- FLUX model + activations: ~22 GB in the container
|
||
- Other ~65 Docker containers on CT 104: ~7 GB
|
||
- CT 104 OS + kernel: ~1 GB
|
||
- Other CTs idle: ~26 GB
|
||
- **Total: ~56 GB** → host started swapping, degrading all services
|
||
|
||
### ComfyUI XPU memory accounting gap
|
||
|
||
ComfyUI's `get_free_memory()` for Intel XPU queries `torch.xpu.get_device_properties().total_memory` = **58 GB** (the full shared memory pool — Arc shares system RAM). It has no awareness of the Docker cgroup limit. Smart memory management (`free_memory()`) calculates `memory_required - get_free_memory()` which is always hugely negative → never evicts models. This makes `--disable-smart-memory` irrelevant for XPU; smart memory is already broken.
|
||
|
||
Consequence: ComfyUI will always try to load the full model into XPU memory regardless of container limit. The cgroup OOM killer is the only backstop.
|
||
|
||
### Container limit recommendation for ComfyUI (CT 104)
|
||
|
||
| Scenario | Limit | Safe? |
|
||
|----------|-------|-------|
|
||
| `--lowvram` (original) | 20 G | ✅ Safe but slow (231 s/image) |
|
||
| No `--lowvram`, FLUX only | **24 G** | ✅ Fits FLUX peak (~22 GB) + 2 GB headroom |
|
||
| No `--lowvram` + img2img after FLUX | 26 G | ✅ FLUX stays resident, SD1.5 loads on top |
|
||
| 28 G | ⚠️ | Marginal — OOM triggered in testing |
|
||
| 40 G | ❌ | Destabilises host when generating |
|
||
|
||
Peak host usage at 24 G container limit during FLUX generation: `24 + 7 (other containers) + 26 (other CTs idle) ≈ 57 GB` — stays under 62 GB physical.
|
||
|
||
## Recommendations
|
||
|
||
1. **ComfyUI container limit**: set to **24 G** when running without `--lowvram`. Current revert to 20 G + `--lowvram` is stable but slower.
|
||
2. **CT 104 LXC allocation (49 GiB)**: appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact.
|
||
3. **CT 111 (dev, 32 GiB)**: Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template.
|
||
4. **Watch list**: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB → host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104.
|
||
5. ~~**Long-term**: when CT 109 (ops) is built, run Prometheus `node_exporter` on the PVE host and alert when host available RAM drops below 8 GiB.~~ Done 2026-05-23 — CT 109 live, node_exporter scraping PVE host via pve-exporter.
|
||
|
||
## ComfyUI memory optimisation log
|
||
|
||
| Date | Change | Effect |
|
||
|------|--------|--------|
|
||
| 2026-05-22 | Removed `--lowvram` + raised limit to 28 G | OOM at 28 G (XPU DRM buffers counted against cgroup) |
|
||
| 2026-05-22 | Raised limit to 40 G | Host destabilised — reverted |
|
||
| 2026-05-22 | Reverted to `--lowvram` + 20 G | Stable, slow (231 s/image) |
|
||
| 2026-05-22 | Downloaded `t5-v1_1-xxl-encoder-Q4_K_S.gguf` (2.6 GB vs 3.2 GB Q5_K_M) | Saves 600 MB at load time |
|
||
| 2026-05-22 | Kept Q5_K_M T5 (Q4_K_S degrades prompt following per city96); removed `--lowvram`; set limit to 24 G; added `--async-offload --force-fp16` | ~7–30 s generation, safe within host memory budget |
|
||
| 2026-05-22 | Removed `--async-offload` | Flag incompatible with GGUF img2img on XPU — caused full CPU fallback (3.5 min/step) and pure-noise output. Removed; XPU generation now correct at ~7 s/step for txt2img. Current CLI: `--listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16` |
|