4

I am configuring vPC end to end till my server to get more bandwidth, and following is my scenario

enter image description here

I am seeing one strange thing which is i configured vPC on N3k switch and then i configured my Linux server for link aggregation 802.3ad bonding and then i restart server so far all good, i can see correct bonding configuration in /proc/net/bonding/bind0 and my server also started ping but i found i got packet loss in ping, later i found on switch its showing vpc is down but wondering how i am getting ping?

N3k# show vpc 1


vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
134  Po1  down*  success     success                    -

Later i did shut & no shut on Port-Channel 1 and that immediately bring up vpc

N3k# show vpc 1


vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
131  Po1  up     success     success                    10,20,30

My VPC domain config

vpc domain 204
  peer-switch
  role priority 10
  peer-keepalive destination 10.29.0.51 source 10.29.0.50
  auto-recovery
  ip arp synchronize

This is my vPC config

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30
  spanning-tree port type edge trunk
  spanning-tree bpduguard enable
  speed 10000
  channel-group 1 mode active

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30
  speed 10000
  vpc 1

This is my Linux server config

ifcfg-bond0

NAME=bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="mode=4 miimon=500 downdelay=1000 lacp_rate=1"
NM_CONTROLLED=no

ifcfg-bond0.10

NAME=bond0.10
DEVICE=bond0.10
BOOTPROTO=dhcp
VLAN=yes
ONPARENT=yes
NM_CONTROLLED=no

Question:

  1. How does server pinging even if vpc is down on switch?
  2. Why do i need to shut/no shut vpc to bring it up vpc? is this normal?
  3. I installed 30 servers on same vpc cluster and all had same issue, every time i have to go to switch and need to do port-channel shut/no shut
  4. Am i missing something here?

Update - 1

For testing i reboot server and found server is up but switch vpc is down and on switch i am seeing following logs, This is strange issue.

sw1# show logging | grep "Ethernet1/37"
    2018 Jul  9 14:28:13 sw1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/37 is down (Initializing)
    2018 Jul  9 14:28:13 sw1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/37 is down
    2018 Jul  9 14:28:15 sw1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/37 is down (Initializing)
    2018 Jul  9 14:28:18 sw1 %ETHPORT-5-SPEED: Interface Ethernet1/37, operational speed changed to 10 Gbps
    2018 Jul  9 14:28:18 sw1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/37, operational duplex mode changed to Full
    2018 Jul  9 14:28:18 sw1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/37, operational Receive Flow Control state changed to off
    2018 Jul  9 14:28:18 sw1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/37, operational Transmit Flow Control state changed to off
    2018 Jul  9 14:28:28 sw1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/37 is operationally individual
    2018 Jul  9 14:28:28 sw1 %ETHPORT-5-IF_UP: Interface Ethernet1/37 is up in mode trunk

Server side i am seeing following error

[root@Linux ~]# tail -f /var/log/messages
Jul  9 10:45:47 s_sys@linux kernel: : [  321.299960] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
Jul  9 10:46:11 s_sys@linux kernel: : [  345.300288] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

Linux server side i am seeing following

[root@Linux ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 500
Up Delay (ms): 0
Down Delay (ms): 1000

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 6c:3b:e5:b0:7a:40
Active Aggregator Info:
    Aggregator ID: 2
    Number of ports: 2
    Actor Key: 13
    Partner Key: 32883
    Partner Mac Address: 00:23:04:ee:be:cc
  • What does the log show when the vPC is first brought up/the server is booted? My guess is there's a BPDU from the server and the bpduguard option shuts the bond. – Zac67 Jul 9 '18 at 6:26
  • In logs i am not seeing anything interesting, all standard logs IF up IF down, should i need to increase logging, if you then do you know what logging level i need to set?, i will try to remove bpduguard and try will update you result soon – Satish Jul 9 '18 at 12:25
  • Make sure you've got active LACP configured on at least one side - a passive-passive LACP trunk won't come up (but should as a single interface). – Zac67 Jul 9 '18 at 15:28
  • @Zac67 I have updated output of linux side boding its saying Dynamic link aggregation so does that means auto right? as soon as i set channel-group 1 mode on it brought up vpc but server not pinging – Satish Jul 9 '18 at 15:41
  • I found issue, on switch side i have configured switchport trunk allowed vlan 10,20,30 but on server side i only configured VLAN 10,20 it was the issue, look like LACP negotiating VLAN also, How to stop VLAN negotiation on LACP, reason we set multiple VLAN on trunk so it will allow us to setup server dynamically and we don't need to go to every switch and adjust VLAN, Anyone know how to make it work? – Satish Jul 9 '18 at 17:06
2

Solution

OMG! Problem was downdelay=1000 option from Linux BONDING_OPTS, As soon as i removed that everything started working, I reboot server 10 time and every single time vpc was up.

I don't know why nexus doesn't like downdelay=1000

Now my config look like following after fix:

NAME=bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="mode=4 miimon=500 lacp_rate=1"
NM_CONTROLLED=no
  • Can someone explain in comments why downdelay doesn't work?
  • From mirrors.edge.kernel.org/pub/linux/kernel/people/marcelo/…: "Use integer value for delaying disabling a link by this number (in ms) after the link failure has been detected. Must be a multiple of miimon. Default value is zero. See HA section for additional info." Makes little sense with your problem all right... – Zac67 Jul 11 '18 at 21:02
  • 1
    Please accept your own question, so that it does not keep popping up. – user36472 Nov 9 '18 at 11:55
0

I would remove first the lacp_rate option and check that the LACP rate of the bonding interface changes from 'fast' to 'slow'

Explanation: LACP Rate on the switch is slow (30sec) by default and you set it to fast (1sec) on the server side. That means your server sends and expects LACPPDU's every second where the swich only sends them every 30 seconds. A mismatch tends to cause issues (mostly on the server side), they should be the same on both ends. You could also set it to fast on the switch interface 'lacp rate fast' but the recommendation is using the default (slow) and should be good for most cases - unless you have special requirements.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.