Support #367
open
OAM: k8s cluster rebuild (cnode wiped, no quorum)
25%
Description
k8s cluster is DOWN since 2026-08-05. Operator wiped a cnode, lost etcd quorum.
Scope:
- Wipe remaining cnodes
- Rebuild k3s HA control plane (cnode1/2/3, embedded etcd, Tailscale)
- Restore kubeconfig
- Verify cluster health
Scripts exist: PFVCluster/k8s/ (wipe, install-cp, join-servers, post-setup, verify)
Dependency: complete before worker nodes can join.
Updated by Charles N 1 day ago
- Precedes Support #368: OAM: k8s worker nodes join cluster added
Updated by Charles N 1 day ago
ARCHITECTURE POLICY: All 3 control-plane nodes (cnode1/2/3) are tainted NoSchedule. NO user workloads run on the control plane — only system components (CoreDNS, metrics-server, kube-proxy, flannel) with built-in tolerations. All user workloads schedule on worker nodes exclusively.
Cnodes wiped clean. Starting bootstrap now.
Updated by Charles N 1 day ago
- % Done changed from 10 to 15
DIAGNOSIS: Root cause of cnode Uptime Kuma flapping identified. DNS infra (netinfra01/02) is HEALTHY — both Pi-hole containers up, rate-limit disabled (count=0/interval=0), resolution working, zero rate-limit events. Cnode memory/disk are fine (3.8Gi RAM, 3+Gi free). The REAL problem: k3s is in a broken etcd crash-loop on ALL 3 cnodes — every node logging 'etcdserver: no leader', k3s pegging CPU 83-97%, cnode3 actively crash-restarting (restart counter=4). The prior bootstrap attempt left stale/broken etcd state and never formed quorum. This crash-loop makes the API endpoint flap up/down = Uptime Kuma alerts. FIX: full wipe + clean rebuild of all 3 cnodes.
Updated by Charles N 1 day ago
- % Done changed from 15 to 25
STORAGE ROOT CAUSE FOUND + FIX APPLIED: All 3 cnode VMs had disks on NFS-backed spinning storage (qcow2 on 5400-7200rpm HDD over NFS). etcd fsync latency was 8.9 SECONDS (needs <10ms). Root cause of all cluster instability and Uptime Kuma flapping.
FIX APPLIED (needs its own ticket — documenting here for continuity):
- Changed all 3 cnode VM disk cache from writethrough/none to writeback on Proxmox:
- cnode1 (VMID 102, tsys1, storage S2): cache none -> writeback
- cnode2 (VMID 705, tsys7, storage S1): cache writethrough -> writeback
- cnode3 (VMID 603, tsys6, storage D1): cache writethrough -> writeback
- Result: fsync latency dropped from 8900ms to 2-3ms (3000x improvement)
- Cluster bootstrapped successfully with all 3 nodes Ready
- BUT: etcd heartbeat still exceeds 500ms threshold during high-churn periods (888ms stall observed). Default etcd heartbeat=500ms, election-timeout=1500ms too tight for this storage. Tuning etcd heartbeat/election timeout next.
- User has SSDs available (NFS-exported). Evaluating SSD-over-NFS for cnodes.
FLAPPING STATUS: DNS infra confirmed HEALTHY (not the cause). k3s crash-loop from slow etcd was the cause. Writeback cache largely resolved it; etcd tuning will finish the job.