Project

General

Profile

Actions

Bug #397

open

OAM: Post-change ground-truth end-to-end audit (all hosts + VMs)

Bug #397: OAM: Post-change ground-truth end-to-end audit (all hosts + VMs)

Added by Charles N 1 day ago. Updated about 16 hours ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
-
Start date:
08/06/2026
Due date:
% Done:

40%

Estimated time:

Description

After the physical layer fixes (#394), writeback cache changes (#391/#395), tuned deployment (#395), and k8s rebuild (#392/#367) are complete, run a comprehensive end-to-end audit to verify everything from the ground up.

Scope

Hypervisor level (all 7 Proxmox hosts):

  • Verify tuned profile active (virtual-host)
  • Verify bond status (especially tsys4/5 post-onsite fixes)
  • Verify lldpd running on all hosts (including tsys1)
  • iperf3 multi-flow tests from every host to both storage servers
  • Disk I/O benchmarks (fsync latency, dd throughput)

VM level (all running VMs):

  • Verify cache=writeback on all NFS-backed VMs
  • Verify tuned profile active inside guests (network-latency for k8s, throughput-performance for others)
  • Network connectivity tests (datanet if configured)
  • Disk I/O from inside VMs

Network level:

  • Re-pull all switch configs via conman (verify LAG fixes applied)
  • Verify LLDP neighbors match expected topology
  • Test cross-rack bandwidth (should be >1gbps with fixed bonds)
  • Verify no flapping on any monitored endpoints

k8s level (if rebuilt by then):

  • All nodes Ready
  • etcd quorum stable
  • No API flapping for 30+ minutes
  • CoreDNS running
  • Uptime Kuma monitors green

Output: Update Discourse #298 (audit log) and #299 (network topology) with post-change ground truth.

Related: #391, #392, #393, #394, #395, #367

Updated by Charles N about 16 hours ago Actions #1

  • % Done changed from 0 to 40

Partial 2026-08-07: hypervisor-level audit done (bonds, datanet bridges, VM NICs/IPs, writeback cache all verified fleet-wide). Remaining: re-pull switch configs post-#369 fix, full iperf matrix, k8s node Ready check post-rebuild.

Actions

Also available in: PDF Atom