From 3de4d5db63af02538a4ed183e1e169f9ae703a63 Mon Sep 17 00:00:00 2001
From: "fkrebs (via Claude)" <fkrebs@nucli.de>
Date: Sat, 23 May 2026 11:19:21 +0200
Subject: [PATCH] 2026-05-23: PVE optimizations, LXC watchdog, KSM, startup
 order

---
 7.8          | 0
 CHANGELOG.md | 7 +++++++
 RESUME.md    | 7 +++++++
 3 files changed, 14 insertions(+)
 create mode 100644 7.8

diff --git a/7.8 b/7.8
new file mode 100644
index 0000000..e69de29
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 049d36a..dfde518 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -60,6 +60,13 @@ All notable infrastructure / service / doc changes. Newest first.
 - **oom_score_adj=-300 set on HAOS KVM PID** (protects it from OOM killer — minimum score = hardest to kill).
 - **Persistence via systemd timer** (`protect-haos-kvm.timer` on nuc): fires 60s after boot then every 5 min. Checks if KVM PID for VM 100 has oom_score_adj=-300; sets it if not.
 
+### PVE host optimizations
+
+- **LXC startup order set for all CTs**: CT103 (backrest), CT104 (docker), CT105 (nextcloud), CT112 (infisical), CT113 (db) had no startup order defined. Assigned: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full boot order now: DNS(1) → Zoraxy(2) → Pocket-ID(3) → DB(4) → Ops(5) → Infisical(6) → Docker(7) → Backrest(8) → Nextcloud(9) → HAOS(10) → Dev(20). Also set CT109 `onboot=1` (was missing).
+- **KSM (Kernel Samepage Merging) enabled**: `echo 1 > /sys/kernel/mm/ksm/run` active now; persisted via `/etc/systemd/system/ksm-enable.service`. Deduplicates identical pages across LXC containers — effective for shared libc/runtime pages. Zero-risk, online change.
+- **5-minute critical watchdog timer added**: `homelab-health-quick.timer` on nuc runs `QUICK=1 homelab-health` every 5 min. Checks: all LXC status (auto-starts if stopped), HAOS KVM (notify only), critical containers (Pocket-ID, Prometheus, Grafana, LiteLLM, MCP gateway). Complements the hourly full sweep.
+- **Gitea issues filed for backlog PVE opts** (`fkrebs/docs` #2-4): NFS v3→v4 upgrade, ZFS ARC increase to 16 GiB (blocked on CT101 right-sizing), hugepages for HAOS KVM.
+
 ### n8n flow tracking
 
 - **Gitea repo `fkrebs/n8n-flows` created**: exports all n8n workflows as individual JSON files. Daily cron (CT 104, 03:30) syncs via `n8n export:workflow --all` → split → commit + push.
diff --git a/RESUME.md b/RESUME.md
index 7fe3452..8b51985 100644
--- a/RESUME.md
+++ b/RESUME.md
@@ -38,6 +38,13 @@ Last updated: 2026-05-23. Update this at the end of every session.
 - [x] ~~**Arcane + Dozzle → CT 109**~~ — done 2026-05-23. Headless agents on all 7 Docker hosts.
 - [x] ~~**crawl4ai SSE StreamConsumed**~~ — resolved 2026-05-22. Errors were from old code in log history; clean restart confirms no errors. Health: ok (7 tools), lobe-sync: 27/27 servers.
 
+## Recently completed (2026-05-23, session continued ×2)
+
+- **PVE LXC startup order fixed**: CT103/104/105/112/113 had no startup order. Set: CT113=4, CT112=6, CT104=7, CT103=8, CT105=9. Full deterministic boot sequence now in place. Also enabled CT109 onboot=1 (was missing).
+- **KSM enabled on PVE host**: kernel samepage merging active + persisted via `ksm-enable.service`. Zero-cost memory deduplication across LXC containers.
+- **5-minute critical watchdog deployed**: `homelab-health-quick.timer` on nuc; `QUICK=1` mode checks LXC status + critical containers every 5 min. Auto-starts stopped LXCs, notifies Gotify for HAOS KVM issues.
+- **PVE backlog issues filed**: `fkrebs/docs` #2 (NFS v3→v4), #3 (ZFS ARC 16 GiB), #4 (hugepages for KVM).
+
 ## Recently completed (2026-05-23, session continued)
 
 - **HAOS KVM OOM root-caused and fixed**: karakeep_chrome on CT 104 triggered host OOM → killed HAOS KVM twice. Applied: `shm_size: '512m'` to chrome, `oom_score_adj=-300` on KVM PID, systemd timer `protect-haos-kvm.timer` (every 5 min) for persistence.