0

Folks,

I am dealing with very strange issue related LACP and not sure if its bug of 6120XG switch or something else. so let me explain my issue here.

Network Diagram:

enter image description here

I have two Cisco Nexus 9396PX TOR switches in Cisco vPC design for milti-chassis connected to HP blade switch 6120XG.

This is in production last 1 years and had no issue. but recently noticed my all LACP links went down even all switches are up. During debug i found Cisco Switch was not receiving any LACP packets to bring this bundle up. (That conclude something is wrong with HP 6120XG switch) Seems like HP switch is freez or something even i can ssh on to switch and run all commands.

HP 6120XG firmware version: Z.14.58

Cisco Nexus 9396PX logs during incident, Here is the logs which i found on Cisco and HP switches.

2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from Ethernet1/3 to Ethernet1/4
2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel412: Ethernet1/3 is down
2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 10000000 Kbit
2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/3 is down (Initializing)
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from Ethernet1/4 to none
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel412: Ethernet1/4 is down
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 100000 Kbit
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/4 is down (Initializing)
2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
2022 Mar 31 05:28:49 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/3: Ethernet1/3 is suspended
2022 Mar 31 05:28:50 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/4: Ethernet1/4 is suspended
2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure)
2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members) (message repeated 2
 times)
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface Ethernet1/3, operational speed changed to 10 Gbps
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/3, operational duplex mode changed to Full
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/3, operational Receive Flow Control state changed to off
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/3, operational Transmit Flow Control state changed to off
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface port-channel412, operational speed changed to 10 Gbps
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface port-channel412, operational duplex mode changed to Full
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel412, operational Receive Flow Control state changed to off
2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel412, operational Transmit Flow Control state changed to off
2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface Ethernet1/4, operational speed changed to 10 Gbps
2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/4, operational duplex mode changed to Full
2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/4, operational Receive Flow Control state changed to off
2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/4, operational Transmit Flow Control state changed to off
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel412: Ethernet1/3 is up
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from none to Ethernet1/3
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel412: Ethernet1/4 is up
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 10000000 Kbit
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode trunk
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface port-channel412 is up in mode trunk
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 20000000 Kbit
2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode trunk

LACP logs

21) FSM:<Ethernet1/3> Transition at 472440 usecs after Thu Mar 31 05:28:39 2022
    Previous state: [LACP_ST_PORT_MEMBER_COLLECTING_AND_DISTRIBUTING_ENABLED]
    Triggered event: [LACP_EV_RECEIVE_PARTNER_PDU_TIMED_OUT]
    Next state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]

22) FSM:<Ethernet1/3> Transition at 485224 usecs after Thu Mar 31 05:28:39 2022
    Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    Triggered event: [LACP_EV_LACP_DOWN_OR_PORT_DOWN]
    Next state: [FSM_ST_NO_CHANGE]

23) FSM:<Ethernet1/3> Transition at 513668 usecs after Thu Mar 31 05:28:39 2022
    Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    Triggered event: [LACP_EV_PORT_HW_PATH_DISABLED]
    Next state: [FSM_ST_NO_CHANGE]

24) FSM:<Ethernet1/3> Transition at 553383 usecs after Thu Mar 31 05:28:39 2022
    Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    Triggered event: [LACP_EV_CLNUP_PHASE_II]
    Next state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]

25) FSM:<Ethernet1/3> Transition at 572723 usecs after Thu Mar 31 05:28:39 2022
    Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    Triggered event: [LACP_EV_LACP_ENABLED_AND_PORT_UP]
    Next state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]

26) FSM:<Ethernet1/3> Transition at 573092 usecs after Thu Mar 31 05:28:40 2022
    Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
    Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
    Next state: [FSM_ST_NO_CHANGE]

27) FSM:<Ethernet1/3> Transition at 574523 usecs after Thu Mar 31 05:28:41 2022
    Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
    Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
    Next state: [FSM_ST_NO_CHANGE]

28) FSM:<Ethernet1/3> Transition at 575875 usecs after Thu Mar 31 05:28:42 2022
    Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
    Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
    Next state: [FSM_ST_NO_CHANGE]

29) FSM:<Ethernet1/3> Transition at 577793 usecs after Thu Mar 31 05:28:43 2022
    Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
    Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
    Next state: [FSM_ST_NO_CHANGE]

On HP 6120XG switch

Mar 31 05:37:25 172.30.9.34 00393 lacp:  Port 18 is blocked - error condition
Mar 31 05:37:25 172.30.9.34 00435 ports:  port 18 is Blocked by LACP
Mar 31 05:37:51 172.30.9.34 00393 lacp:  Port 19 is blocked - error condition
Mar 31 05:37:51 172.30.9.34 00435 ports:  port 19 is Blocked by LACP
Mar 31 05:38:03 172.30.9.34 00393 lacp:  Port 20 is blocked - error condition
Mar 31 05:38:03 172.30.9.34 00435 ports:  port 20 is Blocked by LACP
Mar 31 05:38:36 172.30.9.34 00393 lacp:  Port 21 is blocked - error condition
Mar 31 05:38:36 172.30.9.34 00435 ports:  port 21 is Blocked by LACP
Mar 31 05:38:36 172.30.9.34 00079 ports:  trunk Trk1 is now inactive
Mar 31 05:38:36 172.30.9.34 00002 vlan:  DEFAULT_VLAN virtual LAN disabled
Mar 31 05:41:36 172.30.9.34 00075 system:  Out of pkt buffers; miss count: 2
Mar 31 05:43:57 172.30.9.34 02631 SNTP:  Server not found at 132.163.96.1.
Mar 31 05:44:06 172.30.9.34 02631 SNTP:  Server not found at 132.163.97.1.
Mar 31 05:44:06 172.30.9.34 00414 SNTP:  Unable to reach configured SNTP servers
 

Cisco Nexus switch port configuration

interface port-channel412
  description ** Link to enc-k004-1-b **
  switchport mode trunk
  switchport trunk allowed vlan 60,62-70,100
  mtu 9216
  speed 10000
  vpc 412

HP 6120XG config

hostname "swt-enc-k004-1-b"
interface 23
   disable
   lacp Active
exit
interface 24
   disable
   lacp Active
exit
trunk 18-21 Trk1 LACP
vlan 1
   name "DEFAULT_VLAN"
   untagged 17,22-24,Trk1
   ip address dhcp-bootp
   no untagged 1-16
   exit
vlan 100
   name "ostack_1_public1"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 63
   name "ostack_1_br-vxlan"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 64
   name "ops"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 65
   name "ostack_1_br-mgmt"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 66
   name "ostack_1_ops"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 67
   name "ostack_1_odmz"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 68
   name "ostack_1_idmz"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 69
   name "ostack_lab"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 70
   name "pxe"
   untagged 1-16
   tagged Trk1
   no ip address
   jumbo
   exit
vlan 62
   name "ostack_1_br-lbaas"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit
vlan 60
   name "ostack_1_br-storage"
   tagged 1-16,Trk1
   no ip address
   jumbo
   exit

logging 172.30.8.7 oobm
logging facility syslog
timesync sntp
sntp unicast
sntp server priority 1 132.163.96.1 3 oobm
sntp server priority 2 132.163.97.1 3 oobm
snmp-server community "XXXXXX" operator
spanning-tree
spanning-tree 1 admin-edge-port
spanning-tree 1 bpdu-protection
spanning-tree 2 admin-edge-port
spanning-tree 2 bpdu-protection
spanning-tree 3 admin-edge-port
spanning-tree 3 bpdu-protection
spanning-tree 4 admin-edge-port
spanning-tree 4 bpdu-protection
spanning-tree 5 admin-edge-port
spanning-tree 5 bpdu-protection
spanning-tree 6 admin-edge-port
spanning-tree 6 bpdu-protection
spanning-tree 7 admin-edge-port
spanning-tree 7 bpdu-protection
spanning-tree 8 admin-edge-port
spanning-tree 8 bpdu-protection
spanning-tree 9 admin-edge-port
spanning-tree 9 bpdu-protection
spanning-tree 10 admin-edge-port
spanning-tree 10 bpdu-protection
spanning-tree 11 admin-edge-port
spanning-tree 11 bpdu-protection
spanning-tree 12 admin-edge-port
spanning-tree 12 bpdu-protection
spanning-tree 13 admin-edge-port
spanning-tree 13 bpdu-protection
spanning-tree 14 admin-edge-port
spanning-tree 14 bpdu-protection
spanning-tree 15 admin-edge-port
spanning-tree 15 bpdu-protection
spanning-tree 16 admin-edge-port
spanning-tree 16 bpdu-protection
spanning-tree Trk1 priority 4
oobm
   ip address 172.30.9.34 255.255.248.0
   ip default-gateway 172.30.8.1
   exit
password manager
password operator
 

LACP mode is active on HP 6120XG as you can see in following output. (I do have active on Cisco also)

swt-enc-k004-1-b# show lacp

                           LACP

   PORT   LACP      TRUNK     PORT      LACP      LACP
   NUMB   ENABLED   GROUP     STATUS    PARTNER   STATUS
   ----   -------   -------   -------   -------   -------
   18     Active    Trk1      Up        Yes       Success
   19     Active    Trk1      Up        Yes       Success
   20     Active    Trk1      Up        Yes       Success
   21     Active    Trk1      Up        Yes       Success
   23     Active    23        Down      No        Success
   24     Active    24        Down      No        Success

After reboot HP blade 6120XG LACP came back and traffic started flowing, This happened in past to other racks also but everytime we just reboot switch and it come back. (We don't have HPE support where we can ask to find out if its BUG of HP 6120XG switch or something else in my config)

4
  • 1
    Running out of packet buffers would likely be a bug, or your traffic patterns have changed resulting in an overloaded switch.
    – Ricky
    Apr 5 at 21:07
  • @Ricky you gave me clue and i found this - support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0105744 May be this is because some nasty stuff happened cause spike in traffic. Why does impact LACP ?
    – Satish
    Apr 5 at 21:32
  • 1
    LACP requires PDUs, so if you're out of buffers, they'll get dropped just like everything else. And then the trunk will fail. (personally, I never use a negotiated protocol between switches, just set it to "active" and go with it. it's not like anyone is going to mess with the links between switches. Lines going to a desktop/server should be negotiated as there's no telling what could happen with those.)
    – Ricky
    Apr 5 at 22:25
  • This is very interesting and look like related to my issue - support.hpe.com/hpesc/public/…
    – Satish
    Apr 7 at 19:08

1 Answer 1

0

LACP is an aggregation protocol. Each side needs to be configured for LACP and at least one side needs to actively advertise for the aggregation group to actually form.

You cannot mix Cisco VPC with LACP on the HPE side.

On the 6120XG, try trunk 18-21 Trk1 LACP active. On the Nexus, you need to have LACP activated globally (feature lacp) and use channel-group 1 mode active on each member interface.

3
  • This is running production switches. since last 1 year they are working fine but suddenly we noticed this issue. I don't think its related to vPC vs LACP compatibility or mix technology. (Sorry i have posted wrong diagram which i just fixed it)
    – Satish
    Apr 5 at 18:49
  • In the config snippets you've posted there's no active LACP to be found which matches the error logs. If you think that at least one side is actively negotiating, you should run a debug to verify that.
    – Zac67
    Apr 6 at 8:36
  • I have Active LACP on both end, i have verified. also updated my question with LACP output. (i don't think its active-passive issue here) because this is working link not something i setup today. its been 1 year we had zero issue on this link but just hit this bug last week
    – Satish
    Apr 6 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.