Bug #391
closed
OAM: cnode VM disk cache changed to writeback (etcd stability fix)
100%
Description
ROOT CAUSE of k8s cluster instability and Uptime Kuma cnode flapping identified and partially fixed.
PROBLEM: All 3 cnode VMs use qcow2 disk images on NFS-backed spinning storage (5400-7200rpm HDD). etcd requires <10ms fsync latency but was seeing 8.9 SECOND fsyncs. This caused:
- etcd leader election failures ('no leader' on all nodes)
- k3s crash-loop (CPU pegged 83-97%)
- API endpoint flapping up/down (triggering Uptime Kuma alerts)
FIX APPLIED: Changed all 3 cnode VM disk cache to writeback on Proxmox:
- cnode1 (VMID 102, pfv-tsys1, storage S2): cache none -> writeback
- cnode2 (VMID 705, pfv-tsys7, storage S1): cache writethrough -> writeback
- cnode3 (VMID 603, pfv-tsys6, storage D1): cache writethrough -> writeback
RESULT: Guest-visible fsync latency dropped from 8900ms to 2-3ms (3000x improvement). Cluster successfully bootstrapped with all 3 nodes Ready. However, etcd heartbeat still occasionally exceeds 500ms default during high-churn periods (888ms stall observed). Requires etcd heartbeat/election timeout tuning or migration to SSD-backed storage.
PHYSICAL DISK INVENTORY (all spinning, no SSD in compute hosts):
- tsys1: WDC WD10EADS 1TB (SATA, 7200rpm)
- tsys6: WDC WD20SDRM 1.8TB (USB, 5400rpm)
- tsys7: WDC WD20SDRW 1.8TB (USB, 5400rpm)
AVAILABLE SSDs (in storage servers, NFS-exported):
- Samsung 860 PRO 234GB (93% full, 16GB free)
- NVMe CT500P1 466GB (not configured)
- SK hynix SSD 477GB (not configured)
RELATED: #367 (k8s cluster rebuild)
Updated by