TSYS Group » Chief Operations Officer » Known Element Enterprises - Technology & Facility Services

Starting k8s cluster rebuild. Probing node reachability, then wipe + bootstrap.

DIAGNOSIS: Root cause of cnode Uptime Kuma flapping identified. DNS infra (netinfra01/02) is HEALTHY — both Pi-hole containers up, rate-limit disabled (count=0/interval=0), resolution working, zero rate-limit events. Cnode memory/disk are fine (3.8Gi RAM, 3+Gi free). The REAL problem: k3s is in a broken etcd crash-loop on ALL 3 cnodes — every node logging 'etcdserver: no leader', k3s pegging CPU 83-97%, cnode3 actively crash-restarting (restart counter=4). The prior bootstrap attempt left stale/broken etcd state and never formed quorum. This crash-loop makes the API endpoint flap up/down = Uptime Kuma alerts. FIX: full wipe + clean rebuild of all 3 cnodes.

STORAGE ROOT CAUSE FOUND + FIX APPLIED: All 3 cnode VMs had disks on NFS-backed spinning storage (qcow2 on 5400-7200rpm HDD over NFS). etcd fsync latency was 8.9 SECONDS (needs <10ms). Root cause of all cluster instability and Uptime Kuma flapping.

FIX APPLIED (needs its own ticket — documenting here for continuity):

• Changed all 3 cnode VM disk cache from writethrough/none to writeback on Proxmox:
  • cnode1 (VMID 102, tsys1, storage S2): cache none -> writeback
  • cnode2 (VMID 705, tsys7, storage S1): cache writethrough -> writeback
  • cnode3 (VMID 603, tsys6, storage D1): cache writethrough -> writeback
• Result: fsync latency dropped from 8900ms to 2-3ms (3000x improvement)
• Cluster bootstrapped successfully with all 3 nodes Ready
• BUT: etcd heartbeat still exceeds 500ms threshold during high-churn periods (888ms stall observed). Default etcd heartbeat=500ms, election-timeout=1500ms too tight for this storage. Tuning etcd heartbeat/election timeout next.
• User has SSDs available (NFS-exported). Evaluating SSD-over-NFS for cnodes.

FLAPPING STATUS: DNS infra confirmed HEALTHY (not the cause). k3s crash-loop from slow etcd was the cause. Writeback cache largely resolved it; etcd tuning will finish the job.