Project

General

Profile

Actions
Copy link

Bug #393

closed

OAM: Audit 4gbps cross-rack link + NFS pathing from compute hosts to storage

Bug #393: OAM: Audit 4gbps cross-rack link + NFS pathing from compute hosts to storage

Added by Charles N 1 day ago. Updated 1 day ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
08/06/2026
Due date:
% Done:

100%

Estimated time:

Description

Audit the cross-rack network link (4gbps LAG) and NFS storage pathing from compute hosts (tsys1, tsys3, tsys6, tsys7, tsys9) to NFS storage servers (tsys4, tsys5).

Scope:

  • Verify cross-rack link bandwidth, LAG config, failover
  • Test NFS latency/throughput from each compute host to each storage server
  • Identify bottlenecks or asymmetries in NFS pathing
  • Document findings in Discourse #299 (Network Topology) and #300 (Storage Architecture)

Related: #391 (storage perf context)

Updated by Charles N 1 day ago Actions
Copy link
 #1

  • % Done changed from 0 to 50

COMPLETE NETWORK + STORAGE AUDIT FINDINGS:

tsys5 NIC topology (confirmed)

  • nic0 (06:00.0, tg3, Broadcom BCM5761) = motherboard mgmt NIC → vmbr0 (192.168.3.250)
  • nic1 (22:00.0, e1000e, Intel 82571EB port 0) = PCIe dual-port NIC port 1 → bond0 → datanet — UP 1000Mbps
  • nic2 (22:00.1, e1000e, Intel 82571EB port 1) = PCIe dual-port NIC port 2 → bond0 → datanet — DOWN
  • nic2 failure is physical: dmesg shows SmartSpeed fallback 1000→100→10Mbps then link down after 363s. Bad cable or bad switch port.
  • Both storage cables should plug into the Intel PCIe dual-port NIC. Only port 0 (nic1) is working.

Switch LACP findings (root cause of broken bonds)

pfv-core-sw01 (Dell PowerConnect 5448):

  • ch1 (g13-g16): cross-rack uplink to tor3-stor — UP, 4 ports, LACP working. THIS IS the 4gbps link.
  • ch2 (g32-g33): tsys4 storage — ENTIRELY SHUT DOWN on switch side
  • ch3 (g34 only): tsys5 storage — only 1 port
  • CRITICAL: g33 is labeled 'pfv-tsys5-stor' but assigned to channel-group 2 (tsys4's group), NOT ch3 (tsys5's group). g32 (tsys4) and g33 (tsys5) are mixed in the same LAG. This is a cabling/switch config error.

pfv-tor3-stor (actually Dell PowerConnect 5324, NOT Neyland as documented):

  • ch1 (g20-g23): cross-rack to core-sw01 — UP, 4 ports, LACP. Properly configured.
  • ch2 (g8-g9): tsys6 storage — UP, LACP. Properly configured.
  • ch3 (g10-g11): tsys7 storage — UP, LACP. Properly configured.
  • ch4 (g13-g14): tsys8 storage — UP, static mode (no LACP).

Load-balancing mismatch

  • core-sw01: layer-2-3 hash
  • tor3-stor: layer-2-3-4 hash
    Both use Dell's port-channel load-balance, not Linux bonding xmit_hash_policy. No mismatch issue for LACP.

Why 4gbps link delivers 950mbps

The cross-rack uplink IS 4gbps and working. Bottleneck is downstream:

  • tsys4: bond broken (USB NIC, 1 active, switch LAG shut down) = 1gbps max
  • tsys5: bond degraded (1 of 2 NICs up, switch port in wrong channel) = 1gbps max

ONSITE ACTION ITEMS (for 0700):

  1. tsys5 nic2: Swap cable, try different switch port (g34 on core-sw01)
  2. core-sw01: Fix g33 channel assignment (move from ch2 to ch3), unshitdown ch2
  3. tsys4: USB NICs fundamentally unsuitable for LACP. Replace with PCIe NIC if available.
  4. Documentation fix: tor3-stor is Dell PowerConnect 5324, not Neyland 24T

Updated by Charles N 1 day ago Actions
Copy link
 #2

  • % Done changed from 50 to 100

Audit complete. All findings documented in #394 (physical layer bugs) and Discourse #299 (network topology). Cross-linked. Closing.

Updated by Charles N 1 day ago Actions
Copy link
 #3

  • Status changed from New to Resolved
Actions
Copy link

Also available in: PDF Atom