I have two BGP routers for DIA with an issue with one provider in that the two links to this provider land on the same SP router. My ethernet handoffs via two separate MetroE links from another provider on behalf on my main provider as the one provider already had fiber to the bldg. If someone could also clear up the ISP terminology when one provider transports services for another, I'd appreciate it. The two circuits terminate L3 with the same SP router, so my two routers each peer with the same provider router. I'm assigned PA space from this SP.

I have no issues with outbound load-balancing (or load-sharing as I guess would be more technically accurate). Outbound, I do ECLB at the firewall which picks one of the two edge routers based on a srcip/dstip hash.

This particular carrier -- forget about the carrier just providing transport -- is not load-balancing inbound traffic from their one router across the two links to my two routers, and this is the direction where we could use the combined 5x50Mb BW that we have contracted. The SP sees equal paths to us for the same advertised network and essentially just the first path they learn is what becomes the bestpath.

I've listed what I'm considering as my options below to get traffic over both circuits, and would like to know what the experts here think is best particularly if you're familiar with typical SP SOPs. Since I have a contract, changing the contract at this time is not an option to have this rebuilt some other way.

Allowing maximum-paths 2 in the SP network fixes this, but this applies to all their BGP customers on the same router that I don't think they will allow. At least one option that will work involves static routes, but that’s not what I would prefer.

Below are the options I have considered in my order of preference.

  1. Allow BGP maximum-paths 2 on SP router (affects all BGP customers homed there) so /24 is used when advertised on both circuits

  2. Split my /24 in half and advertise separate /25s over each link along with the /24. The SP has recently stated that a non-documented community could be used for them to accept > /24 prefixes. This requires manipulating NAT on my firewall to use global addresses in both /25’s as most traffic now is destined back to us on just a few addresses in the lower /25.

  3. SP static routes to /24 to force load-balancing w/BGP /24 (floating route).

  4. SP static routes to /25s to force ECLB w/BGP /24 prefix (in RIB, but not used unless failure of /25s).

I think advertising the /25s in BGP is the best option which I only found out recently is possible with the SP undocumented community, but are there other options I haven't considered or concerns about out-of-order packets with some of these choices?

This is sort of the reverse load-balancing problem most people ask with BGP.

  • 1
    Maximum-paths can work even if you're multihomed to two different routers, in the case each ingress PE at operator network needs to have maximum-paths enable and needs to receive both paths from iBGP.
    – ytti
    Jun 9, 2013 at 8:29
  • Had my doubts on that one when I wrote it... Q revised. Jun 9, 2013 at 8:51

3 Answers 3


I would ask for 'maximum-paths' (it's usually called ECMP in standards and documents, not ECLB). And if ECMP is non-starter, then fallback to your /25 plan.

Other acronyms that I couldn't immediately figure out were DIA (dedicated internet access?) and SOP (standard operating procedure?). I'm not sure if these are really so universal acronyms that they should be used in stackexchange without at least hovertext to resolve them.

  • Thanks, I knew ECLB didn't sound right, but I couldn't remember ECMP off the top of my head. And you're right on the TLA's. ;-) DIA is a universal term that every one of the tier 1 ISPs I've used over the last decade use in contracts for Internet service over direct links. SOP is universal as well, but I'll give it to you that it was a stretch using it in a acronym-ladened discpline such as NE. Jun 9, 2013 at 8:50

Talking about terminology you've really already said transport which I think is what that type of service is most commonly referred to as. Sometimes folks will also refer to it as a "tether".

You have another option of moving the interface on your side to the same router and then setting up eBGP multihop*** between you and your provider. Yes, you lose redundancy on your side, but on the opposite side it's going to the same router anyway, so that's kind of a moot point. This also eliminates the need for the provider to do eBGP multipath, in case they're not willing to do that with you (but I've found that in general most providers are fine with turning this on, if it's not turned on already).

If that doesn't seem plausible then announcing two /25's is likely your best bet, unless of course your provider is willing to turn on maximum-paths (again, assuming they haven't done so already).

***Doing load balancing with eBGP multihop in your scenario will involve the following:

  1. Move the second ISP connection to one router on your side.
  2. Both you and the provider configure update-source Loopback0 on each of your sessions - you don't have to use Lo0 if you don't want to, as long as you guys agree on which address to static-route to.
  3. Configure 2x /32 static routes to each others' loopbacks via the connected interfaces (or next-hop IP's) - this is how the load balancing works, as it's really just ECMP.
  4. Configure ebgp-multihop 2 on each others' sessions (You want to keep this number as small as possible to avoid TCP session hijacking).

Voila, load balancing. This also scales per interface, as adding a new port will involve adding another interface and a static route.

  • Can you elaborate how will the operator start sending traffic via both links, by moving the eBGP on same router at customer end, if multipath is not enabled.
    – ytti
    Jun 9, 2013 at 8:26
  • Sure, I'll modify my answer. Jun 9, 2013 at 17:19
  • Strictly speaking, that does not require multi-hop or migrating to same router. It just requires you change your BGP protocol next-hop to be same in both eBGP peers and provider accepting 'remote nexthop' and static route to them.
    – ytti
    Jun 9, 2013 at 17:36
  • I'm not sure what you mean. Using ebgp multihop for load balancing in this fashion does require parallel connections between the same 2 routers. Jun 9, 2013 at 17:39
  • You don't need move the existing eBGP sessions. You could just reset the next-hop in route-map to some address, same in both eBGP (you don't even have to configure the IP anywhere). As long as provider accepts this modified next-hop (needs toggle in IOS and JunOS) and configures the static routes as you explained, it works just fine without multihop or moving the eBGP in customer end.
    – ytti
    Jun 9, 2013 at 17:42

The optimum solution here is BGP ECMP via maximum-paths 2 - however, I will say that "accepting routes > /24" requiring a community tag sounds like epic stupidity - assuming you're a customer of theirs, they should accept whatever you give them up to a maximum number of prefixes and simply filter outbound according to whatever agreements they have with their other peers. one of the upstreams I have puts me in a similar boat with regard to not doing ECMP, I always expect to be allowed to announce any prefix size I desire to a transit provider and have them use it.

So, given that they don't sound terribly competent, to balance traffic, do not withdraw your /24 prefix - instead, leave it be and ensure that your /25 routes are tagged with NO_EXPORT in addition to whatever it is that your provider allegedly requires so that your /25s won't accidentally leak out of their AS (not that it's likely to get very far if it does).

One last note - make sure that this "undocumented community" isn't in fact a blackhole community, because that'd be... y'know, bad.

  • This from a well-known Tier 1, and all the ISPs where I've done BGP don't allow gt 24 as a general policy. I had planned to set NO_EXPORT on the /25s. Jun 10, 2013 at 6:49
  • all of my upstreams will accept > /24 from me, they just won't export it. I actually utilise that effect for shunting traffic since they don't do ECMP.
    – Olipro
    Jun 12, 2013 at 0:53
  • additionally, such a thing would be terrible in the event of an attack - if you cannot annouce a /32 to your upstream with a blackhole community, you're in trouble.
    – Olipro
    Jun 16, 2013 at 11:50
  • They do allow blackhole announcements of any size with the right community. Jun 16, 2013 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.