This is my current setup where i have 40Gbps links between all 4 switches running OSPF using L3 links between them but now i want to double my bandwidth between switches so i am planning to add (dotted links) L3 links and let OSPF load-balance traffic on them, Do you think is there any issue to doing that or this is going to be just fine? ( want second set of eyes )

enter image description here

This is what my ospf config looks on all 4 switches.

interface Ethernet2/10
  no switchport
  mtu 9216
  ip address
  no ip ospf passive-interface
  ip router ospf 100 area
  no shutdown

interface Ethernet2/11
  no switchport
  mtu 9216
  ip address
  no ip ospf passive-interface
  ip router ospf 100 area
  no shutdown

More details about current traffic flow

my current traffic flow looks like following diagram at present SW is active BGP switch so all in/out traffic from coming from ISP. then SW1 do load-sharing between two SW3/4 using OSPF ECMP. Last 1 years we have no single complain about voice issue or quality issue ( everyone is happy ). Now when my SW1 is failed then OSPF move BGP route to SW2 and make it active and traffic start flowing from SW2 to SW3/4 ( I have tested this multiple time by manually shifting BGP )

enter image description here

Update - 2

Load-sharing info for OSPF/ECMP

I have following load-sharing configured which is default on cisco nexus switches.

# show ip load-sharing
IPv4/IPv6 ECMP load sharing:
Universal-id (Random Seed): 2223335843
Load-share mode : address source-destination port source-destination
GRE-Outer hash is disabled.
Concatenation is disabled.
  • Be sure you upgrade the links between SW1-2 and SW 3-4 – Ron Trunk May 13 '19 at 17:36
  • Oh yes.. forgot to add that in diagram. good catch! but otherwise do you see any issue to just bring up L3 Link and handover to OSFP ? – Satish May 13 '19 at 17:57
  • As I recall, you do a lot with VoIP. You could end up with a lot of out-of-order packets that would absolutely kill VoIP. – Ron Maupin May 13 '19 at 21:37
  • @RonMaupin I would argue that CEF/ECMP handles "flows" pretty well and will prevent out-of-order delivery of VoIP streams nicely, by assigning a given flow to the same egress interface consistently. – Marc 'netztier' Luethi May 13 '19 at 21:50
  • @Marc'netztier'Luethi, that really depends. Routing protocol per-packet load balancing can really mess up real-time protocols. I was actually thinking of your layer-3 channel suggestion because that will certainly provide per-flow balancing. – Ron Maupin May 13 '19 at 21:55

As these are Point-to-Point links, I would consider using the outage to configure each /30 interface with ip ospf network point-to-point. (New and existing links). This reduces the hello and dead timers. This configuration also reduces the need to negotiate a DR and BDR.

Lastly, I would verify the OSPF neighbor states and routing tables, before and after the cutover. You should see ECMP routes after the cutovers and the appropriate neighborships.

  • getting downtime would be fire drill for us, because we have many many customers so that path would be not easy. but is it possible on SW2 and SW3/4 switches i configured ip ospf network point-to-point and then failover my traffic to SW2 and do same on SW1 and SW3/4? I have updated my question – Satish May 14 '19 at 13:39
  • The plan appears sound to me. Just be cautious that traffic is still flowing as expected in the failover state. – TDurden May 14 '19 at 14:01
  • Also I’m planning to add high ospf cost on new link and wait for sometime until all settle down then reduce cost to start ECMP traffic – Satish May 14 '19 at 15:39
  • Do you think if i change one of idle link of OSPF to p2p will disturb my current traffic flow?\ – Satish May 15 '19 at 13:53
  • No, however I would still CYA. Maintenance window, check preferred routes, look at Netflow, review interface counters and etc. – TDurden May 16 '19 at 14:06

There's two ways to do that.

  • your proposed way, by adding a second link with its own /30 or /31, making sure that OSPF installs multiple equal cost routes in the routing table, and let CEF's ECMP (EqualCostMultiPath) forwarding handle the packet pushing and the distribution of flows across the set of available links. CEF/ECMP uses a different load sharing logic than Port-Channel, and can handle uneven numbers of links a lot better than Port-Channels do. See Ivan Pepelniak's Blog post for reference.

  • use L3 Port-Channels: Move the IP and routing configuration to a port-channel object (which does not have the "switchport" command), and join the given interfaces to that object. Let Port-Channel load distribution logic handle the distribution of flows.

Your proposed idea is more L3/routing oriented, but scaling it might have some issues: You will use a lot of /30 or /31s. You can scale in odd numbers, but you'll have to configure a new link and possibly subnet for every scaling step (or go ip unnumbered). On the upside, individual links with their own subnet are easier to troubleshoot - pinging across a given single link "comes naturally".

On the other hand, L3 Port-Channels don't need more IP subnets and don't actually touch your given routing configuration logic. Port-Channel is a bit more "Nexus style", as the whole history of Nexus switches is based on the VPC concept (which does not quite apply here, I'll admit). Scaling of additional links is easier - just add two more, without touching any IP or routing configuration. However, rules for Port-Channels apply (e.g. keep number of links in powers of 2), while troubleshooting individual links of a Port-Channel is less straightforward (can't ping across an individual link without removing it from the port-channel and reconfiguring it)

ADDON-1: And oh yes, by all means DO follow TDurden's advice to configure point-to-point on the ... ehm.. point-to-point links (really bad pun, I'll admit)

CAVEAT-1: When using port-channels ensure that you select a load-balancing strategy that fits the expected communication patterns for the given link. When connecting a router to a router (essentially only two MAC addresses on the link), then "Src/Dst MAC" might not have the desired results... For reference, see the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide, Release 9.2(x)

ADDON-2: On Nexus 9000, with ECMP/CEF, the load-sharing algorithm can be configured to include L4 properties: ip load-sharing address source-destination port source-destination See Configuring Load-Sharing in the Unicast FIB from the Unicast Routing configuration gude.

CAVEAT-2 When using L3-Port-Channels, keep an eye on the "bandwidth" property of the port-channel interface when a member link goes down. Depending on hardware/software platform, the bandwidth of the port-channel interface might get reduced accordingly, and OSPF might react to that by increasing the given link's cost. This might have (un)intended consequences for the topology.

  • Thanks for chiming in, I have updated my question with more details, currently SW1 is active and that one is doing load-sharing, now if i add one more link between SW1 to SW3/4 why do you think ECMP will be issue? my current ECMP setup running fine last 1 year without any single issue. I am not worried about wasting IP or typing config. I am trying make it less painful without downtime :( – Satish May 14 '19 at 11:54
  • ECMP doesn't support Per-Packet load-sharing, it used Flow based load-sharing. cisco.com/c/en/us/td/docs/ios-xml/ios/mp_l3_vpns/configuration/… – Satish May 15 '19 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.