Project

General

Profile

Actions
Copy link

Bug #395

closed

OAM: VM disk cache optimization (writethrough -> writeback) + tuned deployment

Bug #395: OAM: VM disk cache optimization (writethrough -> writeback) + tuned deployment

Added by Charles N 1 day ago. Updated about 17 hours ago.

Status:
Resolved
Priority:
Normal
Assignee:
Charles N
Target version:
-
Start date:
08/06/2026
Due date:
% Done:

100%

Estimated time:

Description

Two-part infrastructure optimization across all Proxmox hosts and VMs.

Part 1: VM disk cache changes

VMs still using cache=writethrough on NFS-backed spinning disk should be changed to cache=writeback for consistency and performance. Writethrough forces every guest fsync to physical spinning disk (slow). Writeback lets host RAM absorb fsyncs (fast, safe for this R&D cluster).

VMs to change (requires stop/set/start):

  • VMID 108 tsys-ucs-01 (tsys1) — writethrough -> writeback
  • VMID 600 tsys-awx (tsys6) — writethrough -> writeback
  • VMID 703 rr-middleware-01 (tsys7) — writethrough -> writeback
  • VMID 706 kali-rd (tsys7) — writethrough -> writeback
  • VMID 707 tsys-siem (tsys7) — writethrough -> writeback

DO NOT REBOOT: ultix-streaming, ultix-offstage, pfv-netinfra-01, pfv-netinfra-02

VMs with no cache setting (default=none): evaluate and set writeback where appropriate.
VMs on local-lvm: leave as-is (local disk doesn't need writeback).

Part 2: tuned deployment

Hypervisors (Proxmox hosts):

  • Profile: virtual-host (optimizes CPU scheduling, I/O for virtualization)
  • Install tuned + tuned-utils, set profile, enable service

VMs (guests):

  • Profile: network-latency for k8s/storage-sensitive VMs
  • Profile: throughput-performance for general workloads
  • Install tuned, set appropriate profile per VM

Related: #391 (original writeback change on cnodes), #393 (network audit)

Updated by Charles N 1 day ago Actions
Copy link
 #1

  • % Done changed from 0 to 75

COMPLETED:

Part 1: VM disk cache — DONE

Changed 15 NFS-backed VMs from writethrough/none to writeback across all hosts:

  • tsys1: VMID 105, 108
  • tsys6: VMID 600, 602, 604
  • tsys7: VMID 701, 702, 703, 706, 707, 708, 709, 711
  • tsys9: VMID 902, 905

Skipped (per user instruction): ultix-streaming (5111), ultix-offstage (5102/5000), pfv-netinfra-01 (103), pfv-netinfra-02 (904).
Skipped (local-lvm storage, doesn't need writeback): VMs with local-lvm disks on tsys3/5.
Previously changed: cnode1/2/3 (VMID 102/705/603) in ticket #391.

Part 2: Hypervisor tuned — DONE

Installed tuned + tuned-utils on all 7 Proxmox hosts. Profile: virtual-host.

  • tsys1: active ✓
  • tsys3: active ✓
  • tsys4: active ✓ (IPv6 apt issue but installed from cache)
  • tsys5: active ✓
  • tsys6: active ✓
  • tsys7: active ✓
  • tsys9: active ✓

Part 2b: Guest-level tuned — DEFERRED

Cnodes need apt-get update before tuned can install (stale package indexes causing 404s). Will be deployed as part of k8s post-setup script when cluster is rebuilt. Profile: network-latency for k8s nodes.
Other VMs: deploy during next maintenance window or via Cloudinit/KNELIAC.

Updated by Charles N 1 day ago Actions
Copy link
 #2

  • Assignee set to Charles N

Cross-documented: Discourse #298 audit log entry. Hypervisor tuned complete (7/7 hosts). Guest-level tuned: cnodes will get it via k8s/post-setup.sh on next rebuild. Other VMs deferred to next maintenance.

Updated by Charles N 1 day ago Actions
Copy link
 #3

  • % Done changed from 75 to 90

GUEST-LEVEL TUNED DEPLOYMENT COMPLETE:

VM Profile Status
cnode1 (100.125.134.53) network-latency ACTIVE
cnode2 (100.109.34.72) network-latency ACTIVE
cnode3 (100.106.222.18) network-latency ACTIVE
pfv-netinfra-01 network-latency ACTIVE
pfv-netinfra-02 network-latency ACTIVE

Other VMs: No qemu-guest-agent installed, and most lack Tailscale/SSH access from the management workstation. Guest-level tuned for remaining VMs should be deployed via Cloudinit template (KNELTemplate) or during next build/maintenance cycle.

POST-CHANGE BASELINE (hypervisors):

Host fsync (avg) dd write tuned bond
tsys1 1ms 3.6 GB/s virtual-host single NIC
tsys3 1ms 3.3 GB/s virtual-host single NIC
tsys4 1ms 3.5 GB/s virtual-host 1 USB NIC (broken)
tsys5 2ms 1.7 GB/s virtual-host 1 of 2 Intel NICs
tsys6 2ms 1.7 GB/s virtual-host 2gbps LACP
tsys7 2ms 2.0 GB/s virtual-host 2gbps LACP
tsys9 1ms 4.6 GB/s virtual-host single NIC

POST-CHANGE BASELINE (VMs with writeback):

VM write_cache fsync avg
cnode1 write back 1ms
cnode2 write back 2ms
cnode3 write back 3ms
netinfra-01 write back 2ms
netinfra-02 write back 1ms

All VMs now show 'write back' cache. Before writeback, fsync was 8900ms on cnodes. Now 1-3ms across the board.

Updated by Charles N about 17 hours ago Actions
Copy link
 #4

  • % Done changed from 90 to 100

Verified 2026-08-07: all 5 target VMs (108,600,703,706,707) confirmed cache=writeback via qm config. Work complete.

Updated by Charles N about 17 hours ago Actions
Copy link
 #5

  • Status changed from New to Resolved
Actions
Copy link

Also available in: PDF Atom