docs/infra/proxmox-state.md

Proxmox Host Optimization Inventory — nuc

Generated: 2026-05-20 Host: nuc · PVE 9.1.11 · Kernel 6.17.13-4-pve · Debian 13 (trixie) CPU: Intel Core Ultra 7 155H (16C / 22T, hybrid P+E+LP-E) · 1 socket · 1 NUMA RAM: 62 GiB physical · 31 GiB zram swap (50 % of RAM, zstd, prio 100) Storage: single Crucial P3 2 TB NVMe (QLC, DRAM-less) → rpool (ZFS, ashift=12, no redundancy) Workload: 1 VM (HAOS) + 9 LXCs (Docker, AdGuard, Backrest, Nextcloud, Zoraxy, Pocket-ID, Dev, Secrets, DB) + 1 planned (Ops/CT109)

---

TL;DR — top 5 actionable wins

1. Memory overcommit is dangerous. Allocated guest RAM (≈ 290 GiB) is ~4.7× physical (62 GiB). Right-size CT 101 (was 160 → done, now 32) and CT 104 (still 128) — see §2. ✅ applied 2026-05-20
2. ZFS ARC is artificially capped at 6.2 GiB. Default would be ~31 GiB. After §1 settles, raise to 16 GiB. See §3.
3. No redundancy on a QLC SSD with 19 % wear and 59 TB written. Single-disk rpool on a DRAM-less consumer QLC drive is a SPOF. Add a second NVMe and convert to mirror — biggest reliability win available. See §6.
4. Backups never prune. Was configured keep-all=1 — fixed to keep-last=3,keep-daily=7,keep-weekly=4,keep-monthly=6. See §7. ✅ applied 2026-05-20
5. atime and autotrim on ZFS. ✅ applied 2026-05-20
6. No DNS rewrites in AdGuard — every internal target is IP-only; add a split-horizon for nuclide.systems and a .lan shorthand set. See §11a.
7. Self-signed Proxmox web UI cert — front via Zoraxy for free LE. See §11b.

---

1. System snapshot

Resource	State	Notes
Load avg	normal	PSI: CPU some=2.4 % / IO some=1 % over 60 s
Memory	52 / 62 GiB used, 4.5 GiB free	tight; zram swap 15 GiB in use
Swap	zram0 (zstd, 31 GiB) prio 100	working as designed; just a symptom of §1
ARC	6.0 / 6.2 GiB (capped)	hit ratio ~99 % but cap is far below default
NVMe wear	Percentage Used 19 %, 59.2 TB written	~5 % wear/year at current rate; healthy for now
Temperature	56–58 °C	well under the 95 °C critical threshold
Uptime	(see uptime)	scrub clean, no checksum errors
Cluster	standalone	quorum OK, no HA configured

---

2. Memory & VM/CT sizing (measured numbers)

Read from /sys/fs/cgroup/lxc/<id>/memory.{current,peak,max} and free -h inside each guest:

Guest	Cap	Current	Peak	Swap-in-use	Verdict
VM 100 (haos)	16 384 MiB	13 056 MiB	n/a	n/a	balloon disabled; HAOS actually uses what it has
CT 101 (shepard)	160 000 MiB	7.2 GiB	15.6 GiB	706 MiB	wildly over-sized — peak is 10 % of cap
CT 102 (adguard)	512 MiB	343 MiB	509 MiB (99 %)	19 MiB	under-sized — at the cap, AdGuardHome alone is 350 MiB
CT 103 (backrest)	512 MiB	89 MiB	305 MiB	16 MiB	fine
CT 104 (docker/AI)	128 000 MiB	18.8 GiB	29.3 GiB	9.3 GiB	real workload, but currently swapping — likely starved by CT 101
CT 105 (nextcloud)	8 192 MiB	2.1 GiB	3.5 GiB	53 MiB	fine
CT 108 (zoraxy)	2 048 MiB	271 MiB	463 MiB	25 MiB	fine; could halve

Sum of declared caps ≈ 290 GiB on a 62 GiB host. Sum of actual peaks ≈ 49 GiB — totally fits. CT 101's 160 GB cap is the entire problem: it's a phantom that scares the scheduler without using anything close to that.

Concrete CT 101 picture

12 cores, load avg 8.5, ~9 Docker containers (Shepard frontend/backend, Keycloak, Neo4j, MongoDB, MongoExpress, TimescaleDB, Caddy, home-showcase-collector). Peak RSS 15.6 GiB.

→ Drop memory cap to 32 GiB (2× peak headroom). No reboot required for LXC memory changes.

Concrete CT 104 picture

16 cores, load avg 8.0, ~65 Docker containers including Immich (with ML/vectorchord), ComfyUI (image-gen), LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless-ngx+AI, Karakeep, Memos, Gotify, Garage S3, plus a forest of MCP servers, Speaches (OpenVINO using the Arc iGPU). 128 GiB of 200 GiB rootfs used.

Peak RSS 29.3 GiB, but 9.3 GiB sitting in swap — under memory pressure. Two paths:

1. Recommended: cut CT 101 first, then CT 104's pressure mostly disappears on its own. Re-measure peak after CT 101 is fixed. Likely safe to cap at 48 GiB then.
2. Leave the 128 GiB cap as a generous ceiling — harmless once CT 101 is sane.

Other guests

• CT 102 (AdGuard): hitting its cap. Bump to 1 GiB to give DNS room to breathe. (DNS is the single point of failure for the LAN — don't squeeze it.) pct set 102 -memory 1024.
• CT 108 (Zoraxy): cap 1 GiB is plenty.
• VM 100 (HAOS): 13 GiB actually used. Enable ballooning so the host can reclaim idle RAM:

  qm set 100 -balloon 4096
  

  This leaves memory=16384 as a ceiling but lets the host shrink it under pressure.
• KSM is already active (10 255 shared pages) — fine.
• zram swap: keep size at ram / 2 until CT 101 is fixed; reduce to ram / 4 afterwards.

---

3. ZFS tuning

Pool

Setting	Current	Recommend	Why
autotrim	off	on	QLC needs trim; weekly fstrim alone is OK but autotrim is "free"
ashift	12	keep	correct for NVMe
atime	on (relatime)	off	unused on a hypervisor; reduces write amp on QLC
xattr	sa	keep	already optimal
compression	on (lz4)	keep	helping (1.61× on HAOS disk)
dnodesize	legacy	auto	minor; only matters with millions of small files
recordsize (rpool)	128 K	keep for general	tune per-dataset (see below)

ARC

/etc/modprobe.d/zfs.conf currently caps zfs_arc_max=6669991936 (≈ 6.2 GiB).

• After §2 sizing is done, raise this to 16 GiB: options zfs zfs_arc_max=17179869184 and zfs_arc_min=4294967296.
• Apply live without reboot: echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max.

Per-dataset

• VM zvols (rpool/data/vm-*): default volblocksize is 16 K — fine. HAOS disk uses cache=writethrough; on ZFS, switch to cache=none (or unset) — writethrough doubles the sync cost on top of ZFS's own integrity guarantees.
• CT 104 (subvol-104-disk-0, Docker + image-gen): keep recordsize=128K. The workload is dominated by large model files and image outputs, not small-file DB traffic — shrinking the record size would hurt, not help.
• Nextcloud (subvol-105-disk-1): leave at 128 K (mixed sizes, mostly larger files).

Pool features

zpool upgrade rpool was run during this audit and enabled redaction_list_spill + raidz_expansion. Other disabled features (fast_dedup, longname, large_microzap, dynamic_gang_header, block_cloning_endian, physical_rewrite) can be enabled with another zpool upgrade rpool — only do this if you do not need to roll back to an older ZFS.

Commands

zpool set autotrim=on rpool
zfs set atime=off rpool
# (optional, once memory is sane):
echo 'options zfs zfs_arc_max=17179869184' > /etc/modprobe.d/zfs.conf
update-initramfs -u -k all

---

4. Storage & VM disk options

VM 100 (haos)

- scsi0: local-zfs:vm-100-disk-1,cache=writethrough,discard=on,size=32G,ssd=1
+ scsi0: local-zfs:vm-100-disk-1,cache=none,discard=on,iothread=1,size=32G,ssd=1

• cache=none (or remove cache entirely) — let ZFS manage caching.
• iothread=1 with virtio-scsi-pci controller — already using virtio-scsi-pci, just add iothread.
• Already has discard=on and ssd=1 ✔

LXC local-zfs storage

• sparse 1 is set ✔ — thin-provisioned.
• All CTs are on local-zfs rootfs; OK.

UNAS share — current state (measured)

Backend: 192.168.1.31 (looks like a UniFi NAS — exports /volume/.../.unifi-drive/storage/.data, the only NFS export listed is restricted to four allowed clients: the host .20, CT 104 .40, plus .60 and 172.30.33.1).

Two parallel mounts on the host pointed at the same backing data:

Mount	Type	Options (key bits)	Consumers
/mnt/pve/unas	NFS v3	proto=tcp, mountproto=udp, rsize/wsize=1M, hard, relatime, timeo=600	CT 103 (backrest), CT 104 (docker) — bind-mounted to /mnt/pve/unas inside
/mnt/pve/unas_smb	CIFS v3.1.1	cache=strict, actimeo=1, soft, rsize/wsize=4M, uid/gid=33	CT 105 (nextcloud) — bind-mounted to /mnt/pve/unas inside

Issues:

1. CT 105 is on CIFS to the same data CT 104 uses via NFS. Pure duplication. Nextcloud does massive amounts of stat() traffic; actimeo=1 on the CIFS mount forces every metadata lookup to hit the wire, which is slow.
2. NFS is v3, not v4.x. v4 is preferred unless the UDM doesn't export it. v4 fixes locking, removes the separate mountd dance, and supports session trunking.
3. mountproto=udp under packet loss can intermittently fail to (re)mount. Set mountproto=tcp.
4. hard mount with no intr equivalent. If UNAS goes away, anything blocked on it hangs the calling process indefinitely. For non-critical use cases (Nextcloud, but not backrest), soft,timeo=100,retrans=3 is friendlier — Backrest backups should stay hard.
5. CT 105 cannot mount NFS directly because the UNAS export only allows IPs .20/.40/.60/.172.30.33.1 — .41 (CT 105) is missing. So either keep the host-side bind-mount approach (correct) or have UNAS export to .41 too.
6. The bind-mount approach is correct for unprivileged CTs that can't run NFS/CIFS clients themselves. Don't change that pattern.

Recommended consolidation:

# 1. Probe whether the NAS speaks NFSv4
mount -t nfs -o vers=4.2,proto=tcp 192.168.1.31:/var/nfs/shared/storage /mnt/test
# if it works:
pvesm set unas --options vers=4.2,proto=tcp,hard,noatime
# (this re-mounts on next access; or unmount/remount /mnt/pve/unas)

# 2. Switch CT 105 to the NFS bind-mount
pct set 105 --mp0 /mnt/pve/unas,mp=/mnt/pve/unas
# (CT 105 currently uses unas_smb → unas. New line bind-mounts the NFS mount.)
# Then verify nextcloud-aio still sees uid/gid 33 properly — NFS uses host UIDs,
# whereas CIFS was forcing uid=33. May need to chown on the NAS or add an idmap.

# 3. Drop the CIFS storage once CT 105 is migrated
pvesm remove unas_smb   # if it exists as PVE storage
# or remove the entry from /etc/pve/storage.cfg

Notes on perf:

• 1 GbE NIC ceiling is ~118 MB/s. NFS rsize/wsize=1M already saturates that.
• If you have or add a 2.5 GbE / 10 GbE link on either side, also enable jumbo frames (MTU 9000) on vmbr0, the host NIC, and UNAS. That alone can ~double bulk-read throughput.
• For Nextcloud's small-file stats: actimeo=60 (NFS) or cache=loose,actimeo=60 (CIFS, if you stay on it) — dramatically cuts roundtrips at the cost of slightly stale directory listings.

---

5. CPU / boot / kernel

Item	State	Recommend
Governor	performance	keep
HWP EPP	default	set to balance_performance if you want some idle savings without latency cost: echo balance_performance > /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference
intel_iommu=on iommu=pt	set ✔	keep
GPU passthrough (i915.force_probe=!7dd5 xe.force_probe=7dd5)	set for Arc Xe (Meteor Lake)	keep
nvme_core.default_ps_max_latency_us=0	set ✔	disables NVMe power-save — good for stability, costs ~1 W idle
kernel.numa_balancing	0	correct for single socket
Old kernels installed	6.17.13-4 (current) + 7.0.0-3	keep both for now; remove 7.0.0-3 once you've booted 7.0.2-5 successfully after the pending upgrade

Hybrid-core scheduling

The 155H has P-cores (cores 0–11), E-cores (12–17), LP-E cores (18–21). Linux 6.x with intel_pstate=active handles ITD/HWP well; no manual pinning is needed for current workloads. If a CT becomes latency-sensitive, you can pin it with cpuset via lxc.cgroup2.cpuset.cpus (P-cores only).

---

6. Reliability / SPOF

Single disk is the biggest risk

• Current: 1× CT2000P3PSSD8 (QLC, DRAM-less). 19 % wear in (check power_on_hours).
• Recommendation: add a second 2 TB NVMe (ideally a TLC drive with DRAM — Crucial T500, WD SN770, Samsung 990 Pro, Solidigm P44 Pro). Then:

  zpool attach rpool nvme-CT2000P3PSSD8_2429E8BBCFB4-part3 /dev/disk/by-id/<new-disk>-part3
  

  (requires partitioning the new disk to match — sgdisk -R from the existing). Pool becomes a mirror with full self-heal.
• If a second internal slot isn't available, at minimum increase backup frequency for CT 101/104 (the irreplaceable data).

Boot redundancy

proxmox-boot-tool kernel list shows one bootloader entry. After §6 mirror is set up, run proxmox-boot-tool init /dev/<new-disk>-partN so either disk can boot.

---

7. Backups (HIGH PRIORITY — silent risk)

/etc/pve/storage.cfg:

nfs: unas
    prune-backups keep-all=1

keep-all=1 means backups are never deleted automatically. UNAS already holds 2 TB. Set a real policy, e.g.:

pvesm set unas --prune-backups keep-last=3,keep-daily=7,keep-weekly=4,keep-monthly=6

Also: there is no vzdump job configured in /etc/pve/jobs.cfg. Backups are either manual or driven from CT 103 (Backrest). Recommend a scheduled vzdump job for at least VM 100 and CT 101/104 in addition to Backrest, so PVE-native restores remain trivial.

---

8. APT / repositories cleanup

State today:

/etc/apt/sources.list.d/
├── ceph.list                   # all lines commented — fine but consider deleting the file
├── proxmox.sources             # pve-no-subscription (modern deb822) ← keep
├── pve-enterprise.list.bak     # backup, safe to remove
├── pve-enterprise.sources      # Enabled: false ← keep as-is or remove
├── pve-install-repo.list       # pve-no-subscription duplicate
└── pve-no-subscription.list    # pve-no-subscription duplicate

pve-install-repo.list and pve-no-subscription.list duplicate what proxmox.sources already declares. APT deduplicates fetches but the duplication is a foot-gun (one of them will go stale on the next PVE major version transition). Recommended cleanup:

rm /etc/apt/sources.list.d/pve-install-repo.list
rm /etc/apt/sources.list.d/pve-no-subscription.list
rm /etc/apt/sources.list.d/pve-enterprise.list.bak
# keep proxmox.sources and pve-enterprise.sources (already disabled)
apt update

Also: there are 9 pending upgrades including pve-manager 9.1.18 (you're on 9.1.11) and a kernel update. Run apt update && apt full-upgrade at a convenient window.

Unattended-upgrades (configured 2026-05-20)

The host previously had a cron line 0 2 * * * apt-get update && apt-get upgrade -y that was silently no-op'ing on every kernel / PVE point release — apt-get upgrade refuses to install new dependencies, which PVE updates always introduce.

Replaced with unattended-upgrades in a conservative profile:

File	Purpose
/etc/apt/apt.conf.d/52unattended-upgrades-pve	local policy — origins allowlist + email + reboot policy
/etc/apt/apt.conf.d/20auto-upgrades	enables the daily update-list + unattended-upgrade run

Auto-applied:

• origin=Debian,codename=trixie,label=Debian (stable main)
• origin=Debian,codename=trixie-security,label=Debian-Security
• origin=Debian,codename=trixie-updates (stable point updates)

Held for manual apt full-upgrade (intentionally — review release notes first):

• origin=Proxmox,... — pve-manager, kernels, qemu-server, all PVE components

Settings:

• Automatic-Reboot "false" — kernel updates require a manual reboot
• Remove-Unused-Dependencies "true" — autoremove orphans after upgrades
• AutoFixInterruptedDpkg "true" — resume after crash mid-upgrade
• Mail "notify@home.box", MailReport "on-change" — alerts on actual changes

Triggered by:

• apt-daily.timer (daily ~07:00) — refresh package lists
• apt-daily-upgrade.timer (daily ~06:00) — apply unattended upgrades

Caveat: mail delivery isn't reaching you yet. Postfix is up but has relayhost = (none) — change notifications get delivered locally to /var/mail/notify on the host, not to your inbox. Set up a smart-host relay (Gmail/Postmark/etc.) if you want the mails to actually land. Until then, check /var/log/unattended-upgrades/unattended-upgrades.log for history.

Verify any time:

unattended-upgrade --dry-run --debug 2>&1 | grep -E "Allowed origins|would be upgraded|pkgs that look"
systemctl list-timers apt-daily-upgrade.timer
tail /var/log/unattended-upgrades/unattended-upgrades.log

---

9. Networking

• Single bridge vmbr0 on enp86s0 — no VLAN aware (bridge-vlan-aware yes). If you ever want to segment guests by VLAN, add it now (no impact on existing guests as long as you don't tag them):

  bridge-vlan-aware yes
  bridge-vids 2-4094
  

• net.core.rmem_max / wmem_max are at distro defaults (208 KiB). With a 1 GbE NIC the impact is small (link is already saturated at NFS rsize=1M), but with future 2.5/10 GbE bump to 16 MiB:

  cat >/etc/sysctl.d/99-net.conf <<'EOF'
  net.core.rmem_max=16777216
  net.core.wmem_max=16777216
  net.ipv4.tcp_rmem=4096 87380 16777216
  net.ipv4.tcp_wmem=4096 65536 16777216
  EOF
  sysctl --system
  

• TCP congestion control is cubic. bbr is generally better for mixed workloads — change only if you measure a problem.
• wlo1 is present but unused — confirm and disable in BIOS or iface wlo1 inet manual (already done). No action.

---

10. Container-specific issues

CT 104 (docker / AI / image-gen) — 48 GiB cap, 16 cores, GPU passthrough

Measured: 18.8 GiB current, peak 29.3 GiB, 9.3 GiB in swap, load 8.0, ~65 Docker containers (Immich + ML, ComfyUI, LobeChat, n8n, Daytona, LiteLLM, Vaultwarden, Paperless+AI, many MCP servers, Speaches-OpenVINO).

• cores: 16 is justified by the workload (load avg 8 across 16 = ~50 % avg). Don't drop.
• 128 GiB cap is generous but currently 9 GiB is in swap — this is memory pressure from CT 101's phantom cap, not from CT 104's own needs. Fix CT 101 first, re-measure, then either lower CT 104 to 48 GiB or leave it.
• GPU pass-through (/dev/dri/card1, renderD128) confirmed visible inside CT and being used by Speaches via OpenVINO ✔
• Keep recordsize=128K (large files dominate).
• swap: 32000 is high — consider swap: 8192. Heavy CT swap-out on a QLC root SSD adds write amplification.
• Bind-mounted /mnt/pve/unas (NFS) is the right choice ✔
• Bonus: raise scheduler weight if image-gen latency matters: pct set 104 -cpuunits 200.

CT 101 (shepard / docker) — currently 160 GiB cap, peak 15.6 GiB

Workload: ~9 containers — Shepard frontend/backend, Keycloak, Neo4j, MongoDB, MongoExpress, TimescaleDB, Caddy, home-showcase-collector.

• Drop memory cap to 32 GiB: pct set 101 -memory 32768. No restart needed.
• cores: 12 is fine (load 8.5 — close to fully loaded, real work).
• swap: 8192 ✔
• This single change unblocks everything else.

CT 102 (adguard / DNS) — undersized

Measured: 343 MiB used at the 512 MiB cap, AdGuardHome alone is 350 MiB RSS, the CT is one OOM event from killing the LAN's DNS.

• pct set 102 -memory 1024 (bump to 1 GiB).
• Add onboot: 1 (already set) + startup: order=1 (boots first) + protection: 1 (anti-fatfinger). This is the only DNS — treat it like infrastructure.

CT 103 (backrest) — fine

89 MiB used, peak 305 MiB. No changes needed.

CT 105 (nextcloud) — privileged + CIFS

• Privileged container (no unprivileged: 1). For a public-facing app this is the wrong tradeoff. Migration: stop CT, vzdump backup, restore as unprivileged. Be ready to fix file ownership on the bind-mount afterwards (privileged UID 33 → unprivileged needs lxc.idmap).
• Uses CIFS bind-mount to UNAS with actimeo=1 — see §4. Switch to NFS bind-mount, after confirming UID mapping (CIFS forces uid=33; NFS uses host UIDs as-is).
• Memory cap 8 GiB, peak 3.5 GiB — fine as-is.
• Runs Nextcloud AIO stack (mastercontainer-managed: apache, postgres, redis, imaginary, collabora, notify-push).

CT 108 (zoraxy) — slight oversize

Peak 463 MiB on a 2 GiB cap. Lower to 1 GiB if desired (cosmetic).

CT 104 — Docker stacks inventory (/opt/stacks)

CT 104 keeps its Docker workloads in a git-tracked monorepo at /opt/stacks/ with one directory per stack, plus meta-docs (PORTMAP.md, storage.md, volumes.md, docker-networks.md, todo.md). Good practice — this is how to keep ~65 containers manageable. The other CTs (101, 105) don't have /opt/stacks — their compose files live elsewhere.

Stack list (29 dirs):

Stack	Status	Notes
ai/	active — large subtree	comfyui, lobehub, litellm, speaches, mcp-gateway, mcp-servers (many MCP yml files), searxng. Custom syncstack.py to manage cross-file project names.
arr-stack/	dormant (defined, not running)	rdtclient, prowlarr, audiobookshelf, shelfarr, flaresolverr
arcane/	dormant	Docker dashboard
daytona/	active (as daytona-minimal)	dev environments + runner + registry
dozzle/	dormant	container log viewer
gotify/	active	push notifications
homepage/	dormant	dashboard
immich/	active (5 containers)	photo platform + ML
karakeep/	active (3 containers)	bookmark mgr + chrome + meilisearch
memos/	active	notes
n8n/	active	pinned 2.20.11 (good — there's an explicit version-drift comment in the compose)
nexa/	dormant	(?)
paperless_ai/, paperless-ngx/	active (5 containers between them)	OCR pipeline
pocketid/	active	OIDC provider
proxy/	dormant	(?)
qdrant/	dormant	vector DB
shared-db/	active	shared-postgres + garage (S3-compatible) + pgadmin (defined)
streamio/	dormant	media
traccar/	active	GPS tracker
vaultwarden/	active	password mgr
vpn/	dormant	gluetun (intended VPN egress wrapper?)
backups/, docs/, scripts/	meta dirs (no compose)

Observations:

1. ~10 stacks are defined but dormant. No RAM/CPU cost while down, but their images sit on disk and the git repo accrues dead code. Either run them, document why they're parked, or git rm them — repo drift is the silent killer of "I know what's running" confidence.

2. Stack name ≠ compose project name for the ai/ and daytona/ trees (multiple compose files per dir, different project names). The syncstack.py helper exists for this; just be aware that docker compose -f lookups by directory name don't match.

3. Disk-reclaim potential (measured docker system df):

   Asset	Total	Reclaimable
   Images	83 / 61.6 GB	8.3 GB
   Build cache	105 entries / 9.0 GB	4.6 GB
   Volumes	31 / 3.0 GB	940 MB (21 dangling)
   Containers	60 active	0

   pct exec 104 -- docker system prune -a --volumes
   # or non-destructively just the build cache:
   pct exec 104 -- docker builder prune -a
   

   ~13 GB to recover. On a 200 GB rootfs that's 64 % full, this is meaningful.

4. Stack→Zoraxy mapping (§11b): when fronting via Zoraxy, the canonical service endpoints (per PORTMAP.md) are CT 104's IP 192.168.1.40 + port. Worth cross-referencing that file when setting up reverse-proxy entries.

General LXC hygiene

• Strip the giant HTML banners from the top of every /etc/pve/lxc/*.conf — pure noise in pct config.
• All CTs except 105 are unprivileged ✔ with nesting=1,keyctl=1 (appropriate for Docker).

Anti-fat-finger protection (protection: 1) + boot order

Applied across all critical guests (2026-05-20). protection: 1 blocks pct destroy / "Remove" from the UI until manually unset — cheap insurance against the wrong-CT-deleted incident.

Guest	protection	startup order	Rationale
CT 102 (AdGuard / DNS)	✅	order=1 (boots first)	LAN-wide DNS — nothing resolves until this is up
CT 108 (Zoraxy / LE proxy)	✅	order=2	Public-facing reverse proxy + LE; depends on DNS
CT 103 (Backrest)	✅	default	Holds backup config and snapshot metadata
CT 104 (Docker / AI / image-gen)	✅	default	Largest data footprint (200 G rootfs); 65 containers
CT 105 (Nextcloud)	✅	default	User data
VM 100 (HAOS)	✅	default	Home automation state
CT 101 (shepard)	❌ (left optional)	default	Currently a dev/iteration target; protect once stabilized: pct set 101 -protection 1

The onboot: 1 flag was already set on all guests ✔ — they all auto-start on host reboot. The two startup ordered ones now also boot in the right sequence: DNS → Zoraxy → everything else in parallel.

To remove protection on a guest later: pct set <id> -protection 0 (or qm set 100 -protection 0).

---

11a. DNS — AdGuard rewrites & site-wide consistency

Current state (measured):

• AdGuard Home runs on CT 102 (192.168.1.2, hostname dns) — DNS on 0.0.0.0:53/tcp+udp, admin UI on :80.
• Upstream stack is well-configured: load-balanced mix of Telekom-DoT, NextDNS, DNS4EU, Quad9, Cloudflare, Google (DoH/DoQ where supported), bootstrap+fallback on Cloudflare.
• All 6 LXCs resolve via nameserver 192.168.1.2 ✔
• Proxmox host (192.168.1.20) resolves via 192.168.1.2 ✔ — but its search domain is box (probably an install-time leftover; AdGuard's local_domain_name is lan).
• HAOS VM is on 192.168.1.60 — DNS setting unknown without Home Assistant access; verify.
• rewrites: [] and rewrites_enabled: false — no internal name resolution is happening today.

Why this matters

Without rewrites, you address everything by IP. That's brittle (IP changes break links), invisible in logs, and prevents nice tricks like split-horizon DNS for nuclide.systems (so the same name resolves to Zoraxy LAN-internally without going through your public IP / WAN hairpin).

Recommended rewrite set

In AdGuard UI → Filters → DNS rewrites, enable rewrites and add:

# Split-horizon: public domain → Zoraxy on LAN
nuclide.systems       → 192.168.1.4
*.nuclide.systems     → 192.168.1.4

# Service-name shortcuts under the local_domain_name (`.lan`)
pve.lan               → 192.168.1.20      # Proxmox UI
nuc.lan               → 192.168.1.20      # host shorthand
dns.lan               → 192.168.1.2       # AdGuard itself
zoraxy.lan            → 192.168.1.4       # reverse proxy
shepard.lan           → 192.168.1.49      # CT 101
docker.lan            → 192.168.1.40      # CT 104
nextcloud.lan         → 192.168.1.41      # CT 105
haos.lan              → 192.168.1.60      # VM 100
unas.lan              → 192.168.1.31      # NAS
router.lan            → 192.168.1.1       # UniFi gateway

The split-horizon entries are the highest-value: once Zoraxy proxies pve.nuclide.systems (see §11b), the same URL works both from the public internet and from inside the LAN — with no NAT-loopback weirdness and with the LAN traffic never leaving the building.

Edit the YAML directly if preferred (/opt/AdGuardHome/AdGuardHome.yaml inside CT 102), then restart AdGuard. The line rewrites_enabled: false must flip to true.

Verify clients are actually using AdGuard

After rewrites are in, walk the inventory:

Client	Should use DNS	Check
All 6 LXCs	✔ already at .2	pct exec <id> -- cat /etc/resolv.conf
Proxmox host	✔ already at .2	cat /etc/resolv.conf
HAOS VM (192.168.1.60)	unknown	HAOS UI → Settings → System → Network → check DNS servers; should be 192.168.1.2
Router (192.168.1.1, UniFi)	DHCP-hands-out DNS to clients — must serve .2 as primary	UniFi: Settings → Networks → LAN → DHCP DNS: 192.168.1.2
IoT devices (Roborock at .64, others)	inherit via DHCP from router	once UniFi DHCP serves .2, every device that DHCP-renews picks it up. Force-renew or reboot stragglers.
Anything with hard-coded 1.1.1.1 / 8.8.8.8	bypassing the filter	grep service configs for upstream DNS — apps like Pi-hole-aware clients, some Smart TVs, Chromecasts

Optional hardening once the rewrites are stable

• DNSSEC: enable_dnssec: true (currently false). Most upstreams already validate, but flipping this on adds end-to-end checking.
• Block DoH bypass: AdGuard has a "Blocked services" preset for Cloudflare/Google DoH endpoints — useful so clients can't sidestep the filter.
• Resolve box search-domain confusion: edit /etc/resolv.conf (or set it via /etc/network/interfaces) on the host to search lan so it matches AdGuard's local_domain_name.
• Secondary DNS for resilience: CT 102 is currently the sole DNS — if it crashes, the LAN goes blind. Either add protection: 1 (done above) and rely on it, or stand up a tiny secondary AdGuard on a different CT and configure UniFi DHCP to hand out both. (Out of scope for low-hanging fruit, but worth knowing.)
• Conditional forwarding for UNAS: if UNAS announces hostnames (mDNS or its own DNS), point AdGuard at it for the relevant zone via "Upstream DNS servers" using the [/zone/]upstream syntax.

Action checklist

1. AdGuard UI → Filters → DNS rewrites: paste the table above.
2. AdGuard UI → Settings → DNS settings → enable "DNS rewrites".
3. UniFi: confirm DHCP option 6 = 192.168.1.2 (LAN clients get AdGuard).
4. HAOS: confirm Home Assistant has 192.168.1.2 set as DNS.
5. Force-renew DHCP leases on key clients (or just wait — most renew within 24 h).
6. After §11b is done, the public pve.nuclide.systems resolves to .4 from inside the LAN automatically.

---

11b. TLS certificates for the Proxmox web UI

Current state: the PVE web UI on https://192.168.1.20:8006 uses the self-signed certificate generated at install (/etc/pve/local/pveproxy-ssl.pem is absent → falls back to pve-ssl.pem). Every login throws a browser warning.

The wider setup: Zoraxy (CT 108, 192.168.1.4) already handles Let's Encrypt for nuclide.systems (the public domain for this host). So there are three sane options; pick A unless you have a reason not to.

Option A — Reverse-proxy PVE through Zoraxy (recommended)

Pros: single source of LE truth (Zoraxy already renews); no DNS-plugin setup; no exposing the API; nice domain like pve.nuclide.systems. Cons: depends on Zoraxy being up (keep IP:8006 as fallback); needs WebSocket pass-through for the noVNC console and xterm.js shell.

1. Zoraxy host entry
   • Domain: pve.nuclide.systems (or whatever subdomain)
   • Target: https://192.168.1.20:8006
   • Enable WebSocket support (mandatory — noVNC, xterm.js, task log streaming all use it)
   • Skip backend TLS verification (PVE's cert is self-signed)
   • Enable HSTS once you've confirmed the setup works
   • Optionally restrict by source: only LAN + Cloudflare/Tailscale IPs
2. DNS: add an A record pve.nuclide.systems → public IP (or split-horizon to 192.168.1.20 for LAN). Zoraxy will ACME-challenge via whichever method it's configured for (HTTP-01 or DNS-01).
3. Keep https://192.168.1.20:8006 reachable on LAN as an emergency fallback. Don't disable it.
4. Set the PVE redirect-to-HTTPS rules in Zoraxy on for both :80 and :443.

Important caveat: the PVE Mobile app and the pvesh / API clients may not love going through a reverse proxy (they're picky about TLS SNI and cookie domains). Keep direct IP access available for API tooling, or test thoroughly.

Option B — PVE's built-in ACME with DNS-01

Pros: no reverse proxy in the path; PVE renews itself; works for the API too. Cons: requires a DNS provider plugin (your registrar's API token), and an LE-acceptable FQDN that resolves publicly.

1. Register an ACME account:

   pvenode acme account register default you@nuclide.systems
   

2. Configure a DNS plugin. PVE supports acme-dns, cloudflare, route53, desec, etc. via the acme.sh plugin set. Example for Cloudflare:

   pvenode acme plugin add dns cf --api cf --data CF_Token=XXXXXXXX
   

   Replace cf plugin name to match whichever registrar you use for nuclide.systems.
3. Tell the node which domain(s) and how:

   pvenode config set --acme domains=nuc.nuclide.systems
   pvenode config set --acmedomain0 domain=nuc.nuclide.systems,plugin=cf
   

4. Order:

   pvenode acme cert order
   

   PVE drops the cert at /etc/pve/nodes/nuc/pveproxy-ssl.pem and renews ~30 days before expiry via the pve-daily-update timer.

Option C — Push Zoraxy's cert into PVE

Only useful if A and B are off the table. Zoraxy stores its issued certs (location varies by Zoraxy version — typically under its data dir, e.g. /opt/zoraxy/conf/certs/). Cron a script that copies the active cert/key and concatenates them as /etc/pve/local/pveproxy-ssl.pem (cert + chain) and /etc/pve/local/pveproxy-ssl.key, then systemctl reload pveproxy. Brittle — only worth it if you must.

Recommended path

Do A (reverse proxy through Zoraxy) for the web UI. It piggybacks on existing renewal. The mobile-app/API edge cases are usually fine if Zoraxy passes the WebSocket and preserves the Host header. If you later need full ACME on the node itself (e.g. you want valid TLS for pvesh and the API at the node FQDN too), layer B on top — they don't conflict.

Other CTs with web UIs worth fronting via Zoraxy

While you're at it, route through Zoraxy for free LE:

• Backrest (CT 103) — currently IP-only
• AdGuard (CT 102) admin UI — 192.168.1.2:3000
• Nextcloud (CT 105) — almost certainly already exposed; verify it terminates LE in Zoraxy and not internally
• Zoraxy itself (CT 108) — self-hosted, already TLS

For each, add a Zoraxy host entry, set a subdomain, and disable any local TLS / port-exposed listener that bypasses Zoraxy.

---

11. Maintenance / observability

Item	State	Recommend
lm-sensors	not installed	apt install lm-sensors && sensors-detect --auto for CPU/NVMe temps in the UI
Journal size	1.5 GiB	OK; cap at 1 GiB if you want predictability: journalctl --vacuum-size=1G and SystemMaxUse=1G in journald.conf
fstrim.timer	active (weekly)	OK; zfs trim runs separately when autotrim=on
ZFS scrub	last run 2026-05-10, clean	default monthly timer is good
Subscription nag	not removed	If desired, pve-no-nag patch or the proxmox-helper-scripts line — purely cosmetic
Email alerts	(check /etc/pve/user.cfg)	configure root@pam email for failed scrub / failed backup notifications

---

13. Update management — current model

The host previously had 0 2 * * * apt-get update && apt-get upgrade -y (silently no-op'd on every PVE/kernel update) and a weekly bash <(wget tteck/.../update-lxcs-cron.sh) cron that ran dist-upgrade across every LXC. Both removed 2026-05-20 and replaced with the structure below.

Layer 1 — Host packages (Debian + Proxmox)

• unattended-upgrades 2.12 installed on the host.
• /etc/apt/apt.conf.d/52unattended-upgrades-pve allows only Debian, Debian-Security, trixie-updates — Proxmox origin held for manual review.
• Triggered by apt-daily.timer and apt-daily-upgrade.timer (ship with apt, both active enabled).
• Automatic-Reboot "false" — kernel updates wait for a manual reboot.
• Mail "notify@home.box", MailReport "on-change" — Postfix is up but relayhost = (none), so mail is delivered locally to /var/mail/notify (not your inbox until you wire a smart-host).

Layer 2 — CT OS packages (Debian)

unattended-upgrades deployed inside every CT (CT 104 already had it; 101/102/103/105/108/110 added 2026-05-20):

CT	u-u version	Distro	Status
101	2.12	trixie	active
102	2.9.1	bookworm	active
103	2.9.1	bookworm	active
104	2.12	trixie	active
105	2.9.1	bookworm	active
108	2.12	trixie	active
110	2.12	trixie	active

Per-CT allowlist is Debian-only — third-party repos (docker.com, jotta.cloud, claude.ai, cli.github.com, dl.k6.io) are excluded because they ship breaking changes outside Debian's freeze. Upgrade those with explicit apt upgrade <pkg>.

Layer 3 — Helper-script app binaries (AdGuard, Zoraxy)

Each helper-scripts CT ships /usr/bin/update that re-curl|bash's the community-scripts installer. Replaced with proper systemd timers using the apps' own update mechanisms:

CT	Timer	Schedule	Mechanism
102 AdGuard	adguard-update.timer	Wed 03:30 (+15 m jitter)	native AdGuardHome --update flag
108 Zoraxy	zoraxy-update.timer	Wed 03:40 (+15 m jitter)	GitHub releases API, stable semver only (skips RCs), binary swap + 30 s health check + auto-rollback

Both log to /var/log/{adguard,zoraxy}-update.log and journal. Manual invoke: systemctl start <name>-update.service.

Layer 4 — Docker engine inside CTs

docker-ce updates in CT 101, 104, 105, 110 — held by the Debian-only allowlist. Apply with apt upgrade docker-ce docker-ce-cli containerd.io when you want them. Add origin=Docker to the allowlist if you want to auto-apply (not recommended; engine updates occasionally break running containers).

Layer 5 — Docker images (the ~75 containers)

Plan: deploy Diun on CT 109 (ops LXC, see §16). Diun watches image tags on registries, posts to Gotify when a new image is available. Pulls remain manual (docker compose pull && up -d) — protects against latest-tag drift like the n8n incident pinned in /opt/stacks/n8n/docker-compose.yaml.

Layer 6 — Nextcloud-AIO: self-updates via the mastercontainer (CT 105). No external mechanism needed.

---

14. VM 100 (HAOS) auto-restart watchdog

Old approach: */5 * * * * /root/vm100.sh > /dev/null in cron. Script archived to /root/vm100.sh.bak on 2026-05-20.

Replaced with a systemd timer + oneshot:

• /usr/local/sbin/vm100-watchdog.sh — only restarts on status: stopped; skips paused/prelaunch/transitional states; respects /var/lock/qemu-server/lock-100.conf so it doesn't race vzdump or migration
• vm100-watchdog.service (Type=oneshot)
• vm100-watchdog.timer (OnUnitActiveSec=1min, RandomizedDelaySec=15s)

Recovery latency improved from 5 min → 1 min; logging structured in journalctl -u vm100-watchdog.

---

15. Identity — Pocket-ID on its own CT

State as of 2026-05-20

CT 110 "id" created at 192.168.1.5 as the dedicated IdP host. Pocket-ID was previously on CT 104 as one of ~65 docker containers; moved off because:

• Pocket-ID gates auth for everything else; sharing a host with image-gen workloads (9 GiB swap pressure on CT 104) means a single OOM/restart there knocks out SSO across the whole estate
• Pattern match with CT 102 (DNS) and CT 108 (reverse proxy) — each infra-critical concern gets its own CT

CT 110 setting	Value
Hostname / IP	id / 192.168.1.5
Cores / RAM / rootfs	1 / 1 GB / 4 GB
Privilege	unprivileged, nesting=1, keyctl=1
Boot order	onboot=1, startup=order=3 (after DNS=1, Zoraxy=2)
Protection	protection: 1
Auto-updates	unattended-upgrades, Debian-only allowlist
Docker	29.5.1 + compose v5.1.3

Duplication procedure used

1. sqlite3 pocket-id.db ".backup /tmp/pi-snap/pocket-id.db" on CT 104 (online, no downtime to id.nuclide.systems)
2. tar everything except *.db*; restore the live snapshot as pocket-id.db
3. pct pull → pct push to CT 110
4. Adapted compose to drop the shared_backend external network reference (CT 110 uses default bridge)
5. docker compose up -d
6. Verified http://192.168.1.5:11000/healthz returns 200

Zoraxy cutover

id.nuclide.systems upstream needs to change from 192.168.1.40:11000 → 192.168.1.5:11000. Single-line config edit in Zoraxy + reload. Verified live in §11b once executed.

Secrets-rotation list (deferred to cutover day)

• OIDC_CLIENT_SECRET for the Arcane registration (exposed in chat transcript): rotate in Pocket-ID UI, update Arcane env, restart Arcane
• ENCRYPTION_KEY and JWT_SECRET in /opt/stacks/arcane/docker-compose.yml: move to .env (currently empty), regenerate, restart Arcane. Existing user sessions get invalidated — fine, ask everyone to log in again

---

16. CT 109 "ops" — planned observability + ops LXC

Single LXC holding everything monitoring/ops-shaped. Sizing target: 4 cores / 8 GiB RAM / 50 GiB rootfs, unprivileged, nesting=1. RAM bumped from 6 → 8 GiB to accommodate Loki. Disk bumped from 30 → 50 GiB for Loki log retention (30d) alongside Prometheus TSDB.

⚠️ IP conflict: inventory originally assigned 192.168.1.6 but CT 113 (db) took .6 and CT 112 (secrets) took .7. CT 109 needs the next free infra IP — likely .8 (verify against UniFi DHCP table before provisioning).

Access model (initial): LAN-only. No Zoraxy routes until Tinyauth is deployed. Services reachable directly by IP.

Stack to deploy on CT 109

Service	Purpose
Prometheus	metrics TSDB, 30d retention
Loki	log aggregation backend — receives from Alloy agents on all hosts
Grafana	dashboards over Prometheus + Loki (unified metrics + log search)
Alertmanager + alertmanager-gotify-bridge	alert routing → Gotify
Arcane Manager	central docker management UI; edge agents on CT 101 + CT 104 + CT 110 (mTLS, agent-dialed-out)
Dozzle UI	live log tail (quick debugging); agents on CT 101 + CT 104 + CT 110. Complements Loki — Dozzle for live, Loki for historical/search
Homarr	unified dashboard, native Pocket-ID OIDC, Prometheus widget + Grafana iframe support
Diun	docker image update notifier → Gotify
Tinyauth	forward-auth gate for non-OIDC apps (Backrest, raw Dozzle, raw Prometheus, raw Grafana/Loki). OIDC client to Pocket-ID
docker-socket-proxy	local + remote (CT 101/104/110) — hardened read-only docker.sock for Homarr/Arcane discovery
sshwifty	web SSH client, multi-tab — multiple concurrent shells to different hosts (PVE, CT 104, CT 103, CT 111, etc.). LAN-only, port 8182. SSH key auth per host, no password prompt. Zoraxy + Tinyauth gate deferred.
docs-server	mkdocs Material site (/docs git repo → static HTML); migrating here from CT 111 where it currently runs at port 13080. LAN-only, no Zoraxy route.

Sidecars deployed on each host

Two agents per host — node-exporter (metrics) and Alloy (logs). Kept separate: node-exporter metric names are assumed by every Prometheus dashboard/alert; Alloy emitting compatible metrics adds validation risk for no gain.

Host	Sidecars
PVE host	node-exporter, smartctl-exporter, pve-exporter, Alloy (journald → Loki: pve-manager, pveproxy, pvedaemon, LXC/VM lifecycle)
CT 101	node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, Alloy (Docker logs + journald → Loki)
CT 103	node-exporter, Alloy (backrest.service journal + /var/log/rclone-*.log → Loki)
CT 104	node-exporter, cAdvisor, Dozzle agent, Arcane edge agent, docker-socket-proxy, intel_gpu_exporter, Alloy (Docker logs + journald → Loki)
CT 110	node-exporter, Dozzle agent, Arcane edge agent, Alloy (journald → Loki)
CT 111	node-exporter, intel_gpu_exporter, Alloy (journald + Coder/Gitea logs → Loki)
CT 113	node-exporter, postgres_exporter, Alloy (journald + postgres logs → Loki)

Services that stay where they are (NOT on CT 109)

• AdGuard (CT 102), Zoraxy (CT 108), Backrest (CT 103), Pocket-ID (CT 110) — each has its own CT for blast-radius reasons
• All app stacks (Immich, Nextcloud, Shepard, n8n, ComfyUI, etc.)

Migration of Gotify (Tier B — schedule when CT 109 is otherwise stable)

Gotify currently runs in CT 104 docker stack at gotify.nuclide.systems. Moving to CT 109 isolates alerting from CT 104 outages but means updating env vars / webhook targets in ~10 places (MCP servers, Backrest webhooks). Plan: copy DB and app tokens via volume tar, deploy on CT 109, update Zoraxy upstream, then sweep dependents.

Migration of MCP Gateway (Tier B — split control / workload)

/opt/stacks/ai/mcp-gateway/ on CT 104 is the OIDC-gated control plane for the ~20 MCP child containers. Decision (2026-05-20): move only the gateway to CT 109; the child MCP containers stay on CT 104 (they're workload, not control plane).

Mechanics:

• Gateway on CT 109 uses DOCKER_HOST=tcp://<CT 104 socket-proxy>:2375 (the docker-socket-proxy already planned for CT 104) instead of the bind-mounted /var/run/docker.sock. socket-proxy ACL must allow containers, exec, images (read+write).
• Migrate state files via volume tar: config.json, agents.json, prompts/, usage.db, gateway_tokens.json, nc_user_creds.json. Keep them on CT 109 local zfs, not UNAS (per-request latency matters).
• Pocket-ID redirect URI stays https://mcp.nuclide.systems/sso/callback — only Zoraxy's upstream flips from 192.168.1.40:8080 to the CT 109 IP.
• Don't touch the children; gateway still spawns them by name against CT 104's daemon.

Build-order slot: after Arcane + socket-proxy land in §16's checklist, before Tinyauth.

Arcane specifics

• Existing /opt/stacks/arcane/ on CT 104 has the Manager 80 % built: 41 MB SQLite DB carrying Pocket-ID OIDC client + admin user fkrebs@nucli.de
• Container currently stopped (~since May 19)
• Edge mode chosen for agents — agent dials out to Manager over HTTPS/gRPC with mTLS. Manager auto-generates the edge CA on first start; each agent enrolls with a token and gets a 1 yr client cert with auto-renewal
• Same ghcr.io/getarcaneapp/arcane:latest image for Manager and agents; mode is env-driven (ARCANE_EDGE_AGENT=true on agents)
• Intel iGPU is not in Arcane's GPU monitoring catalog (NVIDIA + AMD only). LXC-level passthrough to CT 104 stays untouched; only Arcane's GPU widgets will be blank for the Arc

Build order

1. Create CT 109 (specs above)
2. Deploy node-exporter on host + Prometheus + Grafana first (start collecting baselines)
3. Deploy Arcane Manager with the existing DB restored from CT 104
4. Issue agent tokens, deploy Edge agents on CT 101/104/110
5. Deploy Dozzle Manager + agents
6. Deploy Diun
7. Deploy sshwifty (configure host list + SSH keys for PVE, CT 103, CT 104, CT 111; LAN-only port 8182)
8. Deploy Tinyauth (configure Pocket-ID client first)
9. Deploy Homarr
10. Front the lot via Zoraxy: arcane., dozzle., grafana., prom., home., shell. .nuclide.systems
11. Verify each through the Tinyauth gate where applicable
12. Add Diun watchlist + Alertmanager routing → Gotify

---

17. LLDP / UniFi topology visibility

Investigated 2026-05-20:

• lldpd is installed and active on the host (defaults: advertises + listens)
• Existing /usr/local/bin/update_interface_desc.sh hourly cron consumes LLDP from neighbors and writes # PortDescr: comments into /etc/network/interfaces
• Host is on switch port 10 of the D-Link DGS-1210-28P (SysName: dgs1210, FW 6.32.008).
• /usr/local/bin/update_interface_desc.sh was silently failing — lldpcli not in cron's PATH and the original logic only appended new lines, never replaced stale ones. Rewritten 2026-05-20:
  • PATH set inside the script (/usr/sbin included)
  • Uses lldpcli show neighbors -f keyvalue for machine-readable parsing
  • Single # LLDP: <chassis> :: <port-descr> line per interface; legacy # PortDescr: lines stripped
  • Idempotent (second run = no-op, no backup churn)
  • Daily-keyed backup /etc/network/interfaces.bak.YYYYMMDD
  • Only logs on actual change
  • Old script preserved at /usr/local/bin/update_interface_desc.sh.bak-2026-05-20

The UDM Pro won't see "nuc" in its topology view because LLDP frames use the Nearest-Bridge multicast (01:80:c2:00:00:0e) which any 802.1D-compliant switch terminates by spec — and the D-Link DGS-1210-28P sits between the host and the UDM. LLDP-MED is not a fix for this; it's for endpoint (VoIP/MFP) discovery, not transparent LLDP forwarding.

Remediation paths in order of effort:

1. Enable SNMP v2c/v3 on the DGS-1210 + add as a Generic SNMP device in UniFi → UDM sees the switch and can map port↔MAC. Most practical for this stack.
2. Add snmpd to the host + generic SNMP device in UniFi → CPU/mem/iface stats from Proxmox visible in UniFi (not in topology, but in monitoring).
3. UniFi-managed switch between host and UDM — clean answer; requires hardware.

D-Link DGS-1210 admin UI lives at http://192.168.1.10/ (login page, field names pelican / pinkpanther). Verify the admin password is non-default — DGS-1210 ships with admin/blank or admin/admin on most firmware revisions. A flat-LAN switch with default creds is one of the easier vectors.

Useful inspection from the host any time: lldpcli show neighbors.

---

18a. UNAS access — UID consistency model (post-NFSv4 investigation)

Investigation result (2026-05-20): UNAS Pro advertises NFSv4 in rpcinfo but has no v4 export tree configured. Every v4 mount attempt returns No such file or directory. Ubiquiti has not announced NFSv4 support and the community thread asking for it has no ETA. The official help center also confirms: "UniFi Drive does not support certain NFS export options, such as no_root_squash". So root_squash + v3-only is the long-term reality.

Universal UID landscape on UNAS

Every NFS client write lands as uid 977 / gid 988 (UNAS's all_squash + anon_uid=977/anon_gid=988). The chown probe confirmed no client can change this from the host side. Files written via the legacy CIFS mount appear as uid 33 to the CIFS client but are stored differently on UNAS — the CIFS forceuid=33 mount option lies about ownership client-side.

Per-CT access pattern (canonical)

CT	Mount	Container uid	Effective on disk	Status
103 backrest	NFS bind from host	root	squashes to 977	✓ consistent
104 docker	NFS bind from host	mostly root, n8n=1000 (latent)	squashes to 977	✓ for root containers; n8n latent if it ever writes to UNAS
105 nextcloud	CIFS today (forceuid=33) → NFS + bindfs target	uid 33 (www-data) inside Nextcloud, bindfs translates to 977 on disk	needs migration	🚨 still CIFS

Convention for new containers

Set PUID=977 PGID=988 on any container that writes to UNAS. This pre-aligns with UNAS's enforced mapping and avoids latent permission issues (the n8n class). For images that don't support PUID/PGID, run them as root inside the container — root squashes to 977 cleanly.

Why bindfs for CT 105 specifically

Nextcloud's PHP code hard-checks file ownership against www-data (uid 33). Without remap, NFS reads return uid 977 and Nextcloud refuses to operate normally. CIFS hides this with forceuid=33. NFS+bindfs achieves the same lie with the much faster NFS rail underneath — verified ~5× speed-up on metadata-heavy ops in the non-destructive test on 2026-05-20.

Trigger event to revisit

Watch community.ui.com/RELEASES for a UniFi Drive release that adds:

• NFSv4 export option (would enable idmap)
• no_root_squash support (would enable server-side chown to specific uids)
• Configurable anonuid/anongid (would let us match a real uid)

Any of these would let us simplify the CT 105 stack.

---

18. Homarr inventory — services to include on the dashboard

Captured here so the eventual Homarr config can be assembled in one pass. Groups follow the existing homepage.* label convention used in compose files.

Group: infrastructure

Service	URL	Notes
Proxmox UI	https://192.168.1.20:8006	until LE via Zoraxy lands, see §11b
AdGuard Home (CT 102)	http://192.168.1.2/ (UI on :80)	DNS + admin
Zoraxy (CT 108)	http://192.168.1.4:8000/	reverse proxy admin
Backrest (CT 103)	http://192.168.1.3:9898/	backup orchestration, will be fronted via Tinyauth + Zoraxy
Pocket-ID (CT 110)	https://id.nuclide.systems/	new home, 2026-05-20

Group: network

Service	URL	Notes
UDM Pro	https://192.168.1.1/	UniFi controller
D-Link DGS-1210-28P	http://192.168.1.10/	core L2 switch; host on port 10
UNAS Pro	https://192.168.1.31/	UniFi NAS

Group: ops (to populate when CT 109 lands)

Service	URL
Grafana	https://grafana.nuclide.systems
Arcane	https://arcane.nuclide.systems
Dozzle	https://dozzle.nuclide.systems
Prometheus	https://prom.nuclide.systems (gated by Tinyauth)
Alertmanager	https://alerts.nuclide.systems (gated by Tinyauth)

Group: apps (subset — long list, fill from existing homepage.* labels in /opt/stacks/*/)

Immich, Nextcloud, Vaultwarden, Karakeep, Memos, Paperless-ngx, n8n, ComfyUI, LobeChat, LiteLLM, Traccar, Gotify, Speaches, Daytona, Searxng, Kroki, etc. Pull display labels and icons from the existing homepage.name= / homepage.icon= values per compose.

---

12. Suggested action order

1. Done 2026-05-20 ✅:
   • Memory / sizing: CT 101 160 → 32 GiB; CT 102 512 MiB → 1 GiB; HAOS balloon = 4 GiB
   • Protection: CT 102 startup=1+protection; CT 108 startup=2+protection; CT 103/104/105/110/VM100 protection
   • CT 110 (id) built at 192.168.1.5, Pocket-ID duplicated (online SQLite snapshot)
   • ZFS: autotrim=on, atime=off rpool
   • Backups: retention set on unas
   • APT: duplicate sources removed; host unattended-upgrades deployed (Debian-only)
   • Per-CT u-u: deployed to all 7 CTs with Debian-only allowlist
   • Cron cleanup: removed weekly tteck-LXC-update curl-pipe-bash; removed daily broken apt-get upgrade -y; replaced /root/vm100.sh cron with vm100-watchdog.timer (1 min, lock-aware)
   • Self-updaters: AdGuard (--update flag) Wed 03:30; Zoraxy (GitHub stable releases + rollback) Wed 03:40
   • zpool upgrade rpool ran during the audit (enabled redaction_list_spill, raidz_expansion)
2. Today with a maintenance window:
   • Cut Zoraxy over to CT 110 for id.nuclide.systems (single upstream edit; rollback path = revert one line)
   • apt full-upgrade (kernel 7.0.0-3 → 7.0.2-5, pve-manager 9.1.11 → 9.1.18) + reboot
   • VM 100 disk options (cache=none, iothread=1) — §4. Requires VM stop/start.
   • Switch CT 105 from CIFS → NFS bind-mount (§4). Test UID mapping.
   • Drop the unas_smb storage once CT 105 is migrated.
3. This week:
   • Re-measure CT 104 peak RSS after fixing CT 101 — likely safe to drop to 48 GiB.
   • Convert CT 105 to unprivileged (backup → restore as unprivileged).
   • Raise ARC cap to 16 GiB.
   • Probe NFSv4 against UNAS; switch if supported.
   • Wire Postfix relayhost (Gmail/Postmark/your SMTP) so unattended-upgrades + zfs-zed + cron failures actually mail you.
   • Rotate Backrest plan to back up real data (currently still pointed at /media/data-dir — a 50 KB test file from August 2025); see §7.
4. Medium term:
   • Build CT 109 ops LXC (§16) — Prometheus + Grafana + Arcane Manager + Dozzle + Homarr + Diun + Tinyauth
   • Migrate Gotify from CT 104 to CT 109 (~30 min of env-var updates)
   • Rotate exposed secrets that appeared in this transcript: Arcane OIDC_CLIENT_SECRET, ENCRYPTION_KEY, JWT_SECRET; Immich IMMICH_API_KEY
   • Backrest: enable auth, redesign plans to cover all data tiers (§7), front via Tinyauth for OIDC
5. Next purchase window:
   • Second NVMe → mirror rpool (§6)
   • proxmox-boot-tool init on the new disk
   • 2.5 GbE NIC + matching switch port to UNAS for image-gen / backup speed
   • UniFi-managed switch between host and UDM (or accept SNMP-only visibility from UniFi)

---

---

19. Changes applied 2026-05-20 (session 2)

Optimizations executed

#	Item	Command / action	Result
1	CT 101 protection + boot order	pct set 101 -protection 1 -startup order=10	✅
4	CT 104 memory cap 128→48 GiB	pct set 104 -memory 49152	✅ (swap 32→8 deferred: still 10.4 GB in use)
5	CT 102 rootfs 2→4 GiB	pct resize 102 rootfs 4G	✅ now 28% used
A	VM 100 disk: cache=none + iothread=1	qm set 100 -scsi0 ...,cache=none,iothread=1 + scsihw virtio-scsi-single	✅ HAOS healthy
C	Host apt full-upgrade	kernel 7.0.0→7.0.2-5, pve-manager 9.1.11→9.1.18	✅ installed; reboot pending

Pocket-ID migration completed

• id.nuclide.systems Zoraxy proxy cutover confirmed: 192.168.1.40:11000 → 192.168.1.5:11000
• CT 104 /opt/stacks/pocketid/ directory fully removed (data migrated to CT 110 2026-05-20)
• Backup: /opt/zoraxy/conf/proxy/id.nuclide.systems.config.bak-pre-ct110 (keep as rollback)

Proxmox OIDC via Pocket-ID

Realm pocket-id added; user fkrebs@nucli.de@pocket-id mapped to Administrator role.

pveum realm add pocket-id \
  --type openid \
  --issuer-url https://id.nuclide.systems \
  --client-id 38469e7e-1fff-4841-83a9-74bf38d847eb \
  --client-key <secret> \
  --username-claim email \
  --comment "Pocket-ID OIDC"

pveum user add fkrebs@nucli.de@pocket-id
pveum aclmod / --users fkrebs@nucli.de@pocket-id --roles Administrator

OIDC client inserted directly into Pocket-ID SQLite (API key stored as SHA-256 hash — not reversible):

DB: /opt/stacks/pocketid/data/pocket-id.db on CT 110
Table: oidc_clients
client_id: 38469e7e-1fff-4841-83a9-74bf38d847eb
name: Proxmox VE
callback_urls: ["https://192.168.1.20:8006"]

To add future OIDC clients without UI access:

python3 -c "
import uuid, secrets, bcrypt, json, datetime
client_id = str(uuid.uuid4())
secret_plain = secrets.token_urlsafe(32)
secret_hash = bcrypt.hashpw(secret_plain.encode(), bcrypt.gensalt(rounds=10)).decode()
print(f'id={client_id}')
print(f'secret={secret_plain}')
print(f'hash={secret_hash}')
"
# Then INSERT into oidc_clients with the hash, use secret_plain in the app config
# callback_urls and logout_callback_urls are JSON arrays stored as BLOB
# credentials field is '{}' for standard clients

Note on Pocket-ID API keys: The key column in api_keys stores a SHA-256 hash of the real key (64-char hex). The plaintext key is only shown once at creation time in the UI. If lost, create a new one — there is no recovery path.

Login flow: In PVE web UI, select realm pocket-id at login. You will be redirected to https://id.nuclide.systems for authentication and returned to PVE. The email claim is used as the PVE username.

---

Audit footnote — side effects of this run

• zpool upgrade rpool was executed (not -n). Enabled features: redaction_list_spill, raidz_expansion. Safe on current ZFS version; the pool can no longer be imported by ZFS releases that pre-date these features. No data risk.
• No other state changes were made by this audit.