OSPF increase bandwidth with load-balancing

Question

This is my current setup where i have 40Gbps links between all 4 switches running OSPF using L3 links between them but now i want to double my bandwidth between switches so i am planning to add (dotted links) L3 links and let OSPF load-balance traffic on them, Do you think is there any issue to doing that or this is going to be just fine? ( want second set of eyes )

This is what my ospf config looks on all 4 switches.

interface Ethernet2/10
  no switchport
  mtu 9216
  ip address 192.168.250.9/30
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  no shutdown

interface Ethernet2/11
  no switchport
  mtu 9216
  ip address 192.168.250.13/30
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  no shutdown

More details about current traffic flow

my current traffic flow looks like following diagram at present SW is active BGP switch so all in/out traffic from coming from ISP. then SW1 do load-sharing between two SW3/4 using OSPF ECMP. Last 1 years we have no single complain about voice issue or quality issue ( everyone is happy ). Now when my SW1 is failed then OSPF move BGP route to SW2 and make it active and traffic start flowing from SW2 to SW3/4 ( I have tested this multiple time by manually shifting BGP )

Update - 2

Load-sharing info for OSPF/ECMP

I have following load-sharing configured which is default on cisco nexus switches.

# show ip load-sharing
IPv4/IPv6 ECMP load sharing:
Universal-id (Random Seed): 2223335843
Load-share mode : address source-destination port source-destination
GRE-Outer hash is disabled.
Concatenation is disabled.

Answer 1

As these are Point-to-Point links, I would consider using the outage to configure each /30 interface with ip ospf network point-to-point. (New and existing links). This reduces the hello and dead timers. This configuration also reduces the need to negotiate a DR and BDR.

Lastly, I would verify the OSPF neighbor states and routing tables, before and after the cutover. You should see ECMP routes after the cutovers and the appropriate neighborships.

Answer 2

There's two ways to do that.

• your proposed way, by adding a second link with its own /30 or /31, making sure that OSPF installs multiple equal cost routes in the routing table, and let CEF's ECMP (EqualCostMultiPath) forwarding handle the packet pushing and the distribution of flows across the set of available links. CEF/ECMP uses a different load sharing logic than Port-Channel, and can handle uneven numbers of links a lot better than Port-Channels do. See Ivan Pepelniak's Blog post for reference.

• use L3 Port-Channels: Move the IP and routing configuration to a port-channel object (which does not have the "switchport" command), and join the given interfaces to that object. Let Port-Channel load distribution logic handle the distribution of flows.

Your proposed idea is more L3/routing oriented, but scaling it might have some issues: You will use a lot of /30 or /31s. You can scale in odd numbers, but you'll have to configure a new link and possibly subnet for every scaling step (or go ip unnumbered). On the upside, individual links with their own subnet are easier to troubleshoot - pinging across a given single link "comes naturally".

On the other hand, L3 Port-Channels don't need more IP subnets and don't actually touch your given routing configuration logic. Port-Channel is a bit more "Nexus style", as the whole history of Nexus switches is based on the VPC concept (which does not quite apply here, I'll admit). Scaling of additional links is easier - just add two more, without touching any IP or routing configuration. However, rules for Port-Channels apply (e.g. keep number of links in powers of 2), while troubleshooting individual links of a Port-Channel is less straightforward (can't ping across an individual link without removing it from the port-channel and reconfiguring it)

ADDON-1: And oh yes, by all means DO follow TDurden's advice to configure point-to-point on the ... ehm.. point-to-point links (really bad pun, I'll admit)

CAVEAT-1: When using port-channels ensure that you select a load-balancing strategy that fits the expected communication patterns for the given link. When connecting a router to a router (essentially only two MAC addresses on the link), then "Src/Dst MAC" might not have the desired results... For reference, see the Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide, Release 9.2(x)

ADDON-2: On Nexus 9000, with ECMP/CEF, the load-sharing algorithm can be configured to include L4 properties: ip load-sharing address source-destination port source-destination See Configuring Load-Sharing in the Unicast FIB from the Unicast Routing configuration gude.

CAVEAT-2 When using L3-Port-Channels, keep an eye on the "bandwidth" property of the port-channel interface when a member link goes down. Depending on hardware/software platform, the bandwidth of the port-channel interface might get reduced accordingly, and OSPF might react to that by increasing the given link's cost. This might have (un)intended consequences for the topology.