TSYS Group » Chief Operations Officer » Known Element Enterprises - Technology & Facility Services

COMPLETE NETWORK + STORAGE AUDIT FINDINGS:

tsys5 NIC topology (confirmed)¶

• nic0 (06:00.0, tg3, Broadcom BCM5761) = motherboard mgmt NIC → vmbr0 (192.168.3.250)
• nic1 (22:00.0, e1000e, Intel 82571EB port 0) = PCIe dual-port NIC port 1 → bond0 → datanet — UP 1000Mbps
• nic2 (22:00.1, e1000e, Intel 82571EB port 1) = PCIe dual-port NIC port 2 → bond0 → datanet — DOWN
• nic2 failure is physical: dmesg shows SmartSpeed fallback 1000→100→10Mbps then link down after 363s. Bad cable or bad switch port.
• Both storage cables should plug into the Intel PCIe dual-port NIC. Only port 0 (nic1) is working.

Switch LACP findings (root cause of broken bonds)¶

pfv-core-sw01 (Dell PowerConnect 5448):¶

• ch1 (g13-g16): cross-rack uplink to tor3-stor — UP, 4 ports, LACP working. THIS IS the 4gbps link.
• ch2 (g32-g33): tsys4 storage — ENTIRELY SHUT DOWN on switch side
• ch3 (g34 only): tsys5 storage — only 1 port
• CRITICAL: g33 is labeled 'pfv-tsys5-stor' but assigned to channel-group 2 (tsys4's group), NOT ch3 (tsys5's group). g32 (tsys4) and g33 (tsys5) are mixed in the same LAG. This is a cabling/switch config error.

pfv-tor3-stor (actually Dell PowerConnect 5324, NOT Neyland as documented):¶

• ch1 (g20-g23): cross-rack to core-sw01 — UP, 4 ports, LACP. Properly configured.
• ch2 (g8-g9): tsys6 storage — UP, LACP. Properly configured.
• ch3 (g10-g11): tsys7 storage — UP, LACP. Properly configured.
• ch4 (g13-g14): tsys8 storage — UP, static mode (no LACP).

Load-balancing mismatch¶

• core-sw01: layer-2-3 hash
• tor3-stor: layer-2-3-4 hash
  Both use Dell's port-channel load-balance, not Linux bonding xmit_hash_policy. No mismatch issue for LACP.

Why 4gbps link delivers 950mbps¶

The cross-rack uplink IS 4gbps and working. Bottleneck is downstream:

• tsys4: bond broken (USB NIC, 1 active, switch LAG shut down) = 1gbps max
• tsys5: bond degraded (1 of 2 NICs up, switch port in wrong channel) = 1gbps max

ONSITE ACTION ITEMS (for 0700):¶

1. tsys5 nic2: Swap cable, try different switch port (g34 on core-sw01)
2. core-sw01: Fix g33 channel assignment (move from ch2 to ch3), unshitdown ch2
3. tsys4: USB NICs fundamentally unsuitable for LACP. Replace with PCIe NIC if available.
4. Documentation fix: tor3-stor is Dell PowerConnect 5324, not Neyland 24T

Audit complete. All findings documented in #394 (physical layer bugs) and Discourse #299 (network topology). Cross-linked. Closing.