4

Network Equipment: Juniper Ex4600, 40GbE QSFP+ -> 4x 10GbE SFP+ DAC cable, Mellanox ConnectX-2

Problem: One of the SFP+ physical link cannot be brought up (xe-0/0/25:0 Physical Link Down, xe-0/0/25:1,2,3 Physical Link Up), therefore no network access.

Existing Troubleshooting Performed, Information Collected:

  1. Network Card Working:

    • Tested server with xe-0/0/25:1, Physical link up, network access okay, tested with xe-0/0/25:0, Physical link down, no network access
    • When server connected with xe-0/0/25:1 (working), network card's "connection established" light turns green, activity lights soon flashed. Corresponding LED light for xe-0/0/25:1 on EX4600 is on and green.
    • When server connected with xe-0/0/25:1 (faulty), network card's "connection established" light turns green, activity lights stays off. Corresponding LED light for xe-0/0/25:0 on EX4600 is off.

  2. JunOS network setting for xe-0/0:25:1 (working)

root> show interfaces xe-0/0/25:1 detail
Physical interface: xe-0/0/25:1, Enabled, Physical link is Up
  Interface index: 721, SNMP ifIndex: 557, Generation: 214
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: 10Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Media type: Fiber
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 0c:86:10:3d:89:20, Hardware address: 0c:86:10:3d:89:20
  Last flapped   : 2018-07-28 02:36:09 UTC (2w3d 04:09 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :      177488100981384                    0 bps
   Output bytes  :      167345335559587                  176 bps
   Input  packets:         124325557902                    0 pps
   Output packets:         117878406643                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Egress queues: 12 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0         117747387692             57924403
    3                                0                    0                    0
    4                                0                    0                    0
    7                                0             29892590                    0
    8                                0             54714667                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    3                   fcoe
    4                   no-loss
    7                   network-control
    8                   mcast
  Active alarms  : None
  Active defects : None
  Interface transmit statistics: Disabled
  MACSec statistics:
    Output
        Secure Channel Transmitted
        Protected Packets               : 0
        Encrypted Packets               : 0
        Protected Bytes                 : 0
        Encrypted Bytes                 : 0
     Input
        Secure Channel Received
        Accepted Packets                : 0
        Validated Bytes                 : 0
        Decrypted Bytes                 : 0

  Logical interface xe-0/0/25:1.0 (Index 609) (SNMP ifIndex 561)
   (Generation 196)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
     Input  bytes  :             50018849
     Output bytes  :            570590460
     Input  packets:               767221
     Output packets:              1921180
    Local statistics:
     Input  bytes  :             50018849
     Output bytes  :            570590460
     Input  packets:               767221
     Output packets:              1921180
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol eth-switch, MTU: 1514, Generation: 221, Route table: 5
  1. JunOS network setting for xe-0/0:25:0 (faulty)
root> show interfaces xe-0/0/25:0 detail
Physical interface: xe-0/0/25:0, Enabled, Physical link is Down
  Interface index: 720, SNMP ifIndex: 556, Generation: 213
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: 10Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Media type: Fiber
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 0c:86:10:3d:89:1f, Hardware address: 0c:86:10:3d:89:1f
  Last flapped   : 2018-08-14 04:17:55 UTC (02:02:59 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                 8943                    0 bps
   Output bytes  :               128037                    0 bps
   Input  packets:                   73                    0 pps
   Output packets:                 1069                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Egress queues: 12 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    1                    0
    3                                0                    0                    0
    4                                0                    0                    0
    7                                0                  506                    0
    8                                0                  207                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    3                   fcoe
    4                   no-loss
    7                   network-control
    8                   mcast
  Active alarms  : LINK
  Active defects : LINK
  Interface transmit statistics: Disabled
  MACSec statistics:
    Output
        Secure Channel Transmitted
        Protected Packets               : 0
        Encrypted Packets               : 0
        Protected Bytes                 : 0
        Encrypted Bytes                 : 0
     Input
        Secure Channel Received
        Accepted Packets                : 0
        Validated Bytes                 : 0
        Decrypted Bytes                 : 0

  Logical interface xe-0/0/25:0.0 (Index 608) (SNMP ifIndex 558)
   (Generation 195)
    Flags: Device-Down SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
     Input  bytes  :                 4565
     Output bytes  :                10395

     Input  packets:                   48
     Output packets:                   35
    Local statistics:
     Input  bytes  :                 4565
     Output bytes  :                10395
     Input  packets:                   48
     Output packets:                   35
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol eth-switch, MTU: 1514, Generation: 220, Route table: 5

Note: Last flapped : 2018-08-14 04:17:55 UTC (02:02:59 ago), Active alarms : LINK, Active defects : LINK. The server was rebooted machine 2 hours ago, switch detected device attached to the port, however active alarm and defect shows "LINK".

  1. Spanning tree state: Blocked
root> show ethernet-switching interface xe-0/0/25:0
Routing Instance Name : default-switch
Logical Interface flags (DL - disable learning, AD - packet action drop,
                         LH - MAC limit hit, DN - interface down,
                         SCTL - shutdown by Storm-control,
                         MMAS - Mac-move action shutdown ) 

Logical          Vlan          TAG     MAC         STP         Logical           Tagging 
interface        members               limit       state       interface flags  
xe-0/0/25:0.0                          294912                   DN                untagged   
                 default       1       294912      Discarding                     untagged

Updates:

  1. Further debugging revealed issue between two specific network cards (mellanox connectx-2) which cannot connect to the switch simultaneously. Both will work with the switch perfectly by itself. The MAC address of the network cards are different. On the same switch, there are 20+ other mellanox connectx-3 10GbE network cards all happily running concurrently on the same switch.

Update

Problem temporarily fixed by disabling RSTP on problematic port, delete subinterface, add it back. However, sometimes when server reboots, physical network would be down. The interface must be toggled off/on administratively to re-enable physical interface

root> show interfaces xe-0/0/25:0
Physical interface: xe-0/0/25:0, Enabled, Physical link is Down
  Interface index: 720, SNMP ifIndex: 556
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: 10Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Media type: Fiber
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Current address: 0c:86:10:3d:89:1f, Hardware address: 0c:86:10:3d:89:1f
  Last flapped   : 2018-08-16 11:14:14 UTC (16:29:48 ago)
  Input rate     : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : LINK
  Active defects : LINK
  Interface transmit statistics: Disabled

  Logical interface xe-0/0/25:0.0 (Index 608) (SNMP ifIndex 558)
    Flags: Device-Down SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Input packets : 4067
    Output packets: 4827
    Protocol eth-switch, MTU: 1514

{master:0}[edit]
root# set interfaces xe-0/0/25:0 disable

{master:0}[edit]
root# commit
configuration check succeeds
fpc1:
commit complete
commit complete

{master:0}[edit]
root# run show interfaces xe-0/0/25:0
Physical interface: xe-0/0/25:0, Administratively down, Physical link is Down
  Interface index: 720, SNMP ifIndex: 556
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: 10Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Media type: Fiber
  Device flags   : Present Running Down
  Interface flags: Hardware-Down Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Current address: 0c:86:10:3d:89:1f, Hardware address: 0c:86:10:3d:89:1f
  Last flapped   : 2018-08-16 11:14:14 UTC (16:38:47 ago)
  Input rate     : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : LINK
  Active defects : LINK
  Interface transmit statistics: Disabled

  Logical interface xe-0/0/25:0.0 (Index 608) (SNMP ifIndex 558)
    Flags: Device-Down SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Input packets : 4067
    Output packets: 4827
    Protocol eth-switch, MTU: 1514

{master:0}[edit]
root# delete interfaces xe-0/0/25:0 disable

{master:0}[edit]
root# commit
configuration check succeeds
fpc1:
commit complete
commit complete

root# run show interfaces xe-0/0/25:0
Physical interface: xe-0/0/25:0, Enabled, Physical link is Up
  Interface index: 720, SNMP ifIndex: 556
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: 10Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Media type: Fiber
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Current address: 0c:86:10:3d:89:1f, Hardware address: 0c:86:10:3d:89:1f
  Last flapped   : 2018-08-17 03:53:25 UTC (00:00:16 ago)
  Input rate     : 304 bps (0 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : None
  Active defects : None
  Interface transmit statistics: Disabled

  Logical interface xe-0/0/25:0.0 (Index 608) (SNMP ifIndex 558)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Input packets : 4132
    Output packets: 4829
    Protocol eth-switch, MTU: 1514
Improve this question
7
  • Have you tried another breakout cable?
    – Zac67 ♦
    Aug 14, 2018 at 5:04
  • @Zac67 After debugging a bit more, I have isolated the issue. It seems like two and only two network cards (mellanox connectx-2) will not get along with each other. Ex: I have 2 identical machine with mellanox connectx-2. Machine 1 connects to xe-0/0/25:0 while machine 2 connects to xe-0/0/25:1. I manage to get both ports working alone simply by either unplugging or administratively disabling either one, but not simultaneously. I have verified that these two network cards have different MAC addresses.
    – jackluo923
    Aug 14, 2018 at 6:01
  • STP events should be logged, so you can see what's happening. The only reason for STP to block the link is that the downlink server is transmitting BPDUs which isn't common. I'd run a packet capture on the interfaces while they're being plugged in.
    – Zac67 ♦
    Aug 16, 2018 at 16:54
  • @Zac67 , I will do some research and try to re-replicate the problem and do a packet capture (might take day or two).
    – jackluo923
    Aug 16, 2018 at 17:19
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer.
    – Ron Maupin ♦
    Dec 31, 2020 at 4:51

1 Answer 1

Reset to default
-1

Solution (although not a good one) is to disable RSTP, reconfigure the subinterface interface, physically disconnect and reconnect the network cable multiple times. This is the only way with high probability to enabling networking on both network cards.

Note: This is not a hack solution which simply mask the problem, but would temporarily allow both network card to connect. We resolved the situation by replacing one of the network card.

Improve this answer
4
  • 1
    RSTP doesn't (shouldn't) affect the physical link, and it only takes care of redundant links, blocking them logically, which you gave no indication of in your question. Do check the switch's log to find out what goes on when the link is plugged. Is there a switching/bridging application running on your server, possibly causing a loop? If you can't sort this out you'll need to add a network diagram to your question and the sanitized configurations.
    – Zac67 ♦
    Aug 15, 2018 at 5:07
  • @Zac67 There weren't any more logs related to those two particular interface, but I am not an expert so I could have missed some details. Below is what I know so far.
    – jackluo923
    Aug 16, 2018 at 16:47
  • @Zac67 The network diagram is a simple flat network (ToR switch) thus RSTP is not required ( factory default was enabled by default). There are two switches EX4300 and EX4600 configured together as virtual chassis (act as one switch). Each server (25-30) connects to the switch via 1x1GbE and 1x10GbE link . The only "switching/bridging" possibility is the two freshly installed VMware ESxi servers running on them. Prior to Vmware Esxi installation, the two servers are freshly clean installed ubuntu18.04 servers upgraded with 10Gb network cards (strange network issue is already observed here)
    – jackluo923
    Aug 16, 2018 at 16:53
  • @Zac67 It seems the problem had nothing to do with RSTP except making the problem more apparent. The physical link is link is down again after a machine reboot.
    – jackluo923
    Aug 16, 2018 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.