0

I maintain two cross-connected switches, R1 and R2, which both have a link to an external switch, R3. R1 and R2 both use eBGP to receive routes to various subnets behind R3. Additionally, R1 and R2 use iBGP to exchange routes between each other. The cross-connect between R1 and R2 is considerably larger than the links to R3, and there is enough traffic to completely utilize both links to R3 if traffic is routed appropriately.

Routing diagram

I want to use ECMP to divide R1->R3 traffic so that a share of it follows R1->R3 directly and a share of it follows R1->R2->R3. I want R2->R3 traffic to be similarly divided.

R1 and R2 are Cisco NX-OS switches. R1 sees both potential paths to 192.168.240.0/24 (behind R3) and correctly determines that one was received with eBGP and one with iBGP.

R1# show ip bgp vrf vrf-100
BGP routing table information for VRF vrf-100, address family IPv4 Unicast
BGP table version is 49, Local Router ID is 10.10.100.2
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
* i10.10.100.0/24     10.10.100.3                       100          0 i
*>l                   0.0.0.0                           100      32768 i
* i192.168.240.0/24   10.10.100.3                       100          0 YYYYY i
*>e                   169.254.159.2                                  0 YYYYY i

I tried setting maximum-paths to 2, but Cisco NX-OS switches (and as I understand it, most/all switches) don't consider paths like this equal. eBGP paths are always preferred over iBGP paths and ECMP won't install both in the routing table.

R1# show run bgp
router bgp XXXXX
  vrf vrf-100
    address-family ipv4 unicast
      network 10.10.100.0/24
      maximum-paths 2
    neighbor 10.10.100.3
      remote-as XXXXX
      password 3 b6d56a77617af8e3
      address-family ipv4 unicast
    neighbor 169.254.159.2
      remote-as YYYYY
      password 3 b6d56a77617af8e3
      address-family ipv4 unicast

I tried using "maximum-paths mixed" because my understanding was that this would allow eBGP and iBGP paths to be considered equal.

R1# show run bgp
router bgp XXXXX
  vrf vrf-100
    address-family ipv4 unicast
      network 10.10.100.0/24
      maximum-paths mixed 2
    neighbor 10.10.100.3
      remote-as XXXXX
      password 3 b6d56a77617af8e3
      address-family ipv4 unicast
    neighbor 169.254.159.2
      remote-as YYYYY
      password 3 b6d56a77617af8e3
      address-family ipv4 unicast

In both cases, only the eBGP route is installed in the route table.

R1# show ip route vrf vrf-100
IP Route Table for VRF "vrf-100"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.10.100.0/24, ubest/mbest: 1/0, attached
    *via 10.10.100.2, Vlan100, [0/0], 6w2d, direct
10.10.100.1/32, ubest/mbest: 1/0, attached
    *via 10.10.100.1, Vlan100, [0/0], 6w2d, vrrp_engine
10.10.100.2/32, ubest/mbest: 1/0, attached
    *via 10.10.100.2, Vlan100, [0/0], 6w2d, local
169.254.159.0/30, ubest/mbest: 1/0, attached
    *via 169.254.159.1, Po2.100, [0/0], 1w5d, direct
169.254.159.1/32, ubest/mbest: 1/0, attached
    *via 169.254.159.1, Po2.100, [0/0], 1w5d, local
192.168.240.0/24, ubest/mbest: 1/0, all-best (0x0)
    *via 169.254.159.2, [20/0], 00:31:48, bgp-XXXXXX, external, tag YYYYYY

How do I either:

  1. Convince R1 to multipath over the combination of an eBGP and iBGP route, or
  2. Change what I'm doing to allow for multipathing by some other mechanism?
Improve this question
5
  • Can you use an IGP instead of BGP between the three routers? – Ron Trunk Jul 10 '20 at 18:29
  • With the second part of your requirement ("I want R2->R3 traffic to be similarly divided") you pretty much perfectly described a routing loop. R1 balances its outbound traffic between R2 and R3. R2 balances its outbound traffic between R1 and R3. Why wouldn't R2 balance traffic it receives from R1 back to R1? This is pretty much the reason why BGP prefers external routes by design. Any chance of multi-homing the devices downstream of R1 and R2 or adding another tier of routers? This is one of the benefits of separating border and aggregation functionality in BGP. – rnxrx Jul 10 '20 at 18:58
  • @rnxrx brings up a painfully obvious point that I didn't grasp at first. I spent too much time trying to get the switch to accept my configuration and not enough time understanding why it would be a bad idea. – Eric Weber Jul 20 '20 at 20:23
  • @RonTrunk Unfortunately I cannot use an IGP between the three routers because I don't own all three. R1 and R2 are mine and R3 is an AWS Virtual Private Gateway (which I have abstracted as a single router). The links are actually Direct Connects between my equipment in a colocation facility and AWS. The Virtual Private Gateway (R3) uses ECMP to make use of both links when sending data to the equipment behind R1 and R2, but I haven't found a good way of making use of both links for sessions that originate behind R1 and R2 on their way to AWS. – Eric Weber Jul 20 '20 at 20:33
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer. – Ron Maupin♦ Dec 17 '20 at 19:06
1

Imagine if you had the following network topology, with both R1 and R2 configured for the type of unequal-cost multipath you've described.

Packets bound for the Internet might be sent by R1 to ISP1, or they might be sent to R2. But R2 also has a 50% chance to send those packets, from R1, back to R1. Some packets would loop between R1 and R2 until their TTL reached zero. In other words, you would experience a connectivity outage to some destinations.

ISP1     ISP2
 |        |
 R1------R2

This looping behavior is why BGP does not present options for considering eBGP-learnt paths to be equal to iBGP-learnt ones.

You may know that MPLS labeled-unicast does provide a possible workaround from this problem, but I think that's outside the scope of the current discussion.

Improve this answer
2
  • 1
    You bring up a painfully obvious point that I didn't grasp at first. I spent too much time trying to get the switch to accept my configuration and not enough time understanding why it would be a bad idea. If you have the time, see my comment to @RonTrunk for more information on what I'm actually trying to accomplish. I have two Direct Connects into AWS for redundancy, and I want to make use of both of them for traffic that originates behind R1 and R2. My new understanding is that ECMP would be viable if R1 and R2 merged into one router, but is nonviable in my current setup. – Eric Weber Jul 20 '20 at 20:38
  • You might be trying to micro-optimize for cost (those direct connect ports are relatively cheap, compared to most other connectivity, and probably also, compared to the other costs of any random project/business) in a manner that doesn't have great trade-offs. That might not be a discussion that's on-topic here tbh. However, I believe you can accomplish what you want by provisioning an additional AWS DCGW Virtual Interface over each of your two Direct Connect physical links. Using layer-2, extend the associated VLAN from AWS---via-R1---to-R2 and vice-versa and eBGP sessions. – Jeff Wheeler Jul 20 '20 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.