Project

General

Profile

Actions
Copy link

Bug #394

open

BUG: Physical layer wiring + switch LACP misconfiguration on storage network

Bug #394: BUG: Physical layer wiring + switch LACP misconfiguration on storage network

Added by Charles N 1 day ago. Updated about 16 hours ago.

Status:
In Progress
Priority:
Normal
Assignee:
Charles N
Target version:
-
Start date:
08/06/2026
Due date:
% Done:

80%

Estimated time:

Description

Complete Physical Layer Audit — Storage Network (VLAN 1000)

Data sources: LLDP from all 7 Proxmox hosts, switch running-configs + LLDP from core-sw01 (Dell 5448) and tor3-stor (Dell 5324), MAC tables, ARP tables.

CONFIRMED PHYSICAL WIRING MAP (via LLDP)

Management Network (core-sw01, default VLAN 1):

Host NIC NIC MAC → Switch Port LLDP Confirmed
tsys1 nic0 8c:ae:4c:cd:a6:2d g5 No (lldpd inactive on tsys1)
tsys3 nic0 a4:4c:c8:08:d1:b8 g43 Yes
tsys4 nic0 98:90:96:c4:96:9a g8 Yes
tsys5 nic0 (Broadcom BCM5761, motherboard) 18:03:73:43:ce:de g19 Yes
tsys6 nic0 (via tor3-mgmt g7) tor3-mgmt g7 Yes
tsys7 nic0 (via tor3-mgmt g3) tor3-mgmt g3 Yes
tsys9 nic0 a4:bb:6d:e3:56:86 g23 Yes

Storage Network (VLAN 1000):

Host NIC NIC MAC → Switch Port LACP Status
tsys1 nic1 8c:ae:4c:cd:a6:2d core-sw01 g40 none UP, single port
tsys3 enx...a774 8c:ae:4c:cd:a7:74 core-sw01 g27 none UP, single port
tsys4 enx...a926 (USB) 8c:ae:4c:cd:a9:26 core-sw01 g31 none UP, single port (g32/ch2 dead)
tsys4 enx...ec03 (USB) 80:ae:4c:cd:ec:03 core-sw01 g32 ch2 NO-CARRIER, ch2 SHUT DOWN
tsys5 nic1 (Intel 82571EB p0) 00:1b:78:5c:e9:ce core-sw01 g34 ch3 UP
tsys5 nic2 (Intel 82571EB p1) 00:1b:78:5c:e9:cf core-sw01 g33? ch2 DOWN (wrong channel group!)
tsys6 bond0 (nic1+nic2) 00:21:9b:a2:7c:55 tor3-stor g8-g9 ch2 UP, 2gbps LACP
tsys7 bond0 (nic1+nic2) f8:bc:12:34:e0:75 tor3-stor g10-g11 ch3 UP, 2gbps LACP
tsys9 enx...5b6c 9c:69:d3:6a:5b:6c core-sw01 g38 none UP, single port

BUGS FOUND

BUG 1: core-sw01 g33 in WRONG channel group
Port g33 is labeled 'pfv-tsys5-stor' but assigned to channel-group 2 (tsys4's group), NOT channel-group 3 (tsys5's group). Config excerpt:

interface range ethernet g(32-33)   # g32=tsys4 + g33=tsys5 MIXED
  channel-group 2 mode auto
interface ethernet g34              # only g34 in tsys5's group
  channel-group 3 mode auto

FIX: Remove g33 from ch2, add to ch3.

BUG 2: core-sw01 ch2 entirely SHUT DOWN
Channel-group 2 (g32-g33) is administratively shut down. Even if g33 is moved to ch3, g32 (tsys4 2nd USB NIC) is NO-CARRIER anyway.
FIX: After moving g33, either un-shutdown ch2 for g32 alone or leave shut (tsys4 USB NIC is dead).

BUG 3: tsys5 nic2 is physically DOWN
Intel 82571EB port 1 (MAC 00:1b:78:5c:e9:cf) — SmartSpeed fell back to 10Mbps then died after 363s. Likely bad cable or bad switch port. No LLDP neighbor detected (link never came up stable).
FIX: Swap cable. Verify it reaches g33 (after g33 is moved to ch3).

BUG 4: tsys4 USB NICs cannot do LACP
Both USB NICs (enx8cae4ccda926 + enx80ae4ccdec03) fail LACP negotiation. Partner MAC = 00:00:00:00:00:00 (switch gets no LACP response). USB NICs don't reliably pass LACP PDUs. Second NIC is NO-CARRIER.
FIX: Replace with PCIe NIC (if available). Otherwise accept single-port 1gbps.

BUG 5: Ghost 5th cross-rack cable (g17 ↔ g24)
core-sw01 g17 is UP with LLDP neighbor tor3-stor g24, but g17 is NOT in channel-group 1 (which is g13-g16). On tor3-stor, g24 is standalone with spanning-tree disabled + portfast. This cable does nothing useful.
FIX: Either add g17 to ch1 on core-sw01 AND g24 to ch1 on tor3-stor (making it 5gbps), or remove the cable to avoid confusion.

BUG 6: tor3-stor ch4 (g13-g14) configured for non-existent tsys8
Channel-group 4 is labeled 'pfv-tsys8-stor' and uses static mode (no LACP). Both ports are DOWN. No host tsys8 exists in the fleet.
FIX: Remove ch4 config or repurpose ports.

BUG 7: tsys1 lldpd not running
tsys1 is the only host with lldpd inactive. Cannot confirm physical port mapping via LLDP. Port assignments inferred from switch labels only.
FIX: Enable lldpd on tsys1.

BUG 8: tor3-stor documented as 'Neyland 24T' — actually Dell PowerConnect 5324
System Object ID: 1.3.6.1.4.1.674.10895.3004 = Dell. System Description says 'PowerConnect 5324'. Original mapping.txt incorrectly identifies it.
FIX: Update documentation.

ONSITE ACTION PLAN (0700 tomorrow)

Priority 1 — tsys5 storage bond (restore 2gbps):

  1. On core-sw01: Remove g33 from ch2, add to ch3 
    configure
interface ethernet g33
no channel-group
exit
interface ethernet g33
channel-group 3 mode auto
exit
  2. Physically trace cable from tsys5 nic2 (Intel port 1) to switch port g33
  3. Swap cable if bad
  4. Verify: cat /proc/net/bonding/bond0 on tsys5 should show 2 active slaves

Priority 2 — core-sw01 ch2 cleanup:

  1. Leave ch2 shutdown (tsys4 USB NIC is dead anyway)
  2. Or un-shutdown if g32 is repurposed

Priority 3 — tsys4 USB NIC replacement:

  1. If PCIe NIC available: install, configure bond, add switch ports to LAG
  2. If not: accept single-port 1gbps for tsys4

Priority 4 — Ghost cable cleanup:

  1. core-sw01 g17 / tor3-stor g24: add to ch1 or disconnect

Priority 5 — Documentation fixes:

  1. tor3-stor = Dell PowerConnect 5324 (not Neyland)
  2. Enable lldpd on tsys1
  3. Remove dead ch4 config from tor3-stor

Related: #391 (writeback cache), #393 (network audit)

Actions
Copy link

Also available in: PDF Atom