docs/infra/proxmox-memory-audit.md

Proxmox Memory Audit

Last audited: 2026-05-22

Host physical resources

Resource	Total	Used (idle)	Available
RAM	62 GiB	~33 GiB	~28 GiB
Swap	31 GiB	0 GiB	31 GiB

CT memory allocations

CT	Name	Allocated (MiB)	Swap (MiB)	Typical use	Notes
101	shepard	32,768	8,192	~4 GB	Heavy stack: Mongo, Neo4j, TimescaleDB, Keycloak
102	dns	1,024	512	~100 MB	AdGuard Home
103	backrest	4,096	1,024	~200 MB	Restic scheduler — bumped 2026-05-23 for 822 GB initial backup OOM fix
104	docker	49,152	32,000	6–22 GB	Main Docker host; FLUX spikes to ~22 GB
105	nextcloud	8,196	8,196	~2 GB	Nextcloud AIO
108	zoraxy	2,048	512	~300 MB	Reverse proxy
109	ops	4,096	0	~1.5 GB	Prometheus + Grafana + Loki + Arcane + Dozzle + Homarr
110	id	1,024	512	~200 MB	Pocket-ID
111	dev	32,768	8,192	~3 GB	Coder + Gitea workspaces
112	secrets	4,096	512	~600 MB	Infisical
113	db	4,096	0	~800 MB	Postgres 17 + WAL-G
Sum		141,316 MiB (138 GiB)			2.2× overprovisioned vs physical RAM

Key findings

Overprovisioning is safe — until it isn't

Proxmox uses balloon drivers so CTs only consume what they actually use. At idle the host sits at ~33 GB used with 28 GB available. This is healthy. However, two specific CTs represent risk:

• CT 104 (49 GiB limit) + CT 111 (32 GiB limit) together could claim 81 GiB — well over the 62 GiB physical. If both hit peak simultaneously the host would start swapping heavily.
• CT 104's ComfyUI (FLUX generation) spikes from 6 GB baseline to ~22 GB. That 16 GB spike on top of the 33 GB idle baseline = 49 GB total, within physical RAM but tight.

Why 40G Docker container limit destabilised the system

Setting ComfyUI's container limit to 40 G was the trigger. At generation time:

• FLUX model + activations: ~22 GB in the container
• Other ~65 Docker containers on CT 104: ~7 GB
• CT 104 OS + kernel: ~1 GB
• Other CTs idle: ~26 GB
• Total: ~56 GB → host started swapping, degrading all services

ComfyUI XPU memory accounting gap

ComfyUI's get_free_memory() for Intel XPU queries torch.xpu.get_device_properties().total_memory = 58 GB (the full shared memory pool — Arc shares system RAM). It has no awareness of the Docker cgroup limit. Smart memory management (free_memory()) calculates memory_required - get_free_memory() which is always hugely negative → never evicts models. This makes --disable-smart-memory irrelevant for XPU; smart memory is already broken.

Consequence: ComfyUI will always try to load the full model into XPU memory regardless of container limit. The cgroup OOM killer is the only backstop.

Container limit recommendation for ComfyUI (CT 104)

Scenario	Limit	Safe?
--lowvram (original)	20 G	✅ Safe but slow (231 s/image)
No --lowvram, FLUX only	24 G	✅ Fits FLUX peak (~22 GB) + 2 GB headroom
No --lowvram + img2img after FLUX	26 G	✅ FLUX stays resident, SD1.5 loads on top
28 G	⚠️	Marginal — OOM triggered in testing
40 G	❌	Destabilises host when generating

Peak host usage at 24 G container limit during FLUX generation: 24 + 7 (other containers) + 26 (other CTs idle) ≈ 57 GB — stays under 62 GB physical.

Recommendations

1. ComfyUI container limit: set to 24 G when running without --lowvram. Current revert to 20 G + --lowvram is stable but slower.
2. CT 104 LXC allocation (49 GiB): appropriately sized given Docker workload, but is by far the largest single consumer. Do not raise further without measuring host impact.
3. CT 111 (dev, 32 GiB): Coder workspaces could spike if users run heavy jobs. Consider adding a per-workspace memory limit in the Coder template.
4. Watch list: CT 101 (Shepard, 32 GiB) + CT 104 simultaneously at peak = 54 GB → host would need to swap. Unlikely in practice but possible during CI runs on CT 111 + FLUX generation on CT 104.
5. Long-term: when CT 109 (ops) is built, run Prometheus node_exporter on the PVE host and alert when host available RAM drops below 8 GiB. Done 2026-05-23 — CT 109 live, node_exporter scraping PVE host via pve-exporter.

ComfyUI memory optimisation log

Date	Change	Effect
2026-05-22	Removed --lowvram + raised limit to 28 G	OOM at 28 G (XPU DRM buffers counted against cgroup)
2026-05-22	Raised limit to 40 G	Host destabilised — reverted
2026-05-22	Reverted to --lowvram + 20 G	Stable, slow (231 s/image)
2026-05-22	Downloaded t5-v1_1-xxl-encoder-Q4_K_S.gguf (2.6 GB vs 3.2 GB Q5_K_M)	Saves 600 MB at load time
2026-05-22	Kept Q5_K_M T5 (Q4_K_S degrades prompt following per city96); removed --lowvram; set limit to 24 G; added --async-offload --force-fp16	~7–30 s generation, safe within host memory budget
2026-05-22	Removed --async-offload	Flag incompatible with GGUF img2img on XPU — caused full CPU fallback (3.5 min/step) and pure-noise output. Removed; XPU generation now correct at ~7 s/step for txt2img. Current CLI: --listen 0.0.0.0 --enable-cors-header --use-pytorch-cross-attention --disable-smart-memory --force-fp16