4

The requirements are

  • Hundreds of sites.
  • Each has a router and firewall and a switch stack.
  • Layer 3 MPLS WAN with BGP to provider VRF.

BGP as an IGP?

iBGP

Would you distribute a public ASN across all branch sites and use iBGP to route inter-branch packets? Route reflector(s); MED based routing?

eBGP

Would you assign a private ASN per branch?

... or Don't Use BGP?

DMVPN

DMVPN (minus encryption?) to hide/abstract away the BGP, allowing use of IGP for all inter-site routing?

Best Design?

Anyone think eBGP or iBGP is preferable? I find DMVPN attractive because

  1. My routing is separated from their routing.
  2. We're able to use a faster converging Interior Gateway Protocol.
  3. We're not using Border Gateway Protocol as an IGP.

however lean towards iBGP because it fits the use case and adds no GRE overhead. What is the best way to route traffic given this topology?

5
  • Are the two different WAN connections on the same MPLS VPN, or are they using two different MPLS VPNs?
    – Ron Maupin
    Oct 3, 2016 at 4:50
  • Opinion based questions are off topic here
    – Ron Trunk
    Oct 3, 2016 at 10:00
  • Chat would be a better place to discuss this.
    – Ron Trunk
    Oct 3, 2016 at 12:13
  • That is like asking for someones favorite something. You should read the What types of questions should I avoid asking? for how to ask a subjective question. There are some ways to ask this sort of question.
    – Ron Maupin
    Oct 3, 2016 at 15:23
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer.
    – Ron Maupin
    Aug 11, 2017 at 18:19

2 Answers 2

2

All I can do is explain how a pretty successful large network does it.

Each of the hundreds to thousands of end-sites on an MPLS VPN is in the same private BGP AS, so site-to-site traffic is switched directly by the carrier MPLS cloud. The data centers each have their own private BGP ASes. So, the WAN is a mixture of iBGP and eBGP. Each end-site and data center runs its own separate IGP, injecting the default and specific routes from the MPLS cloud(s), although the standard defines only one, each site's IGP is independent from all the other sites'.

Some end-sites have one WAN circuit, and some sites have two WAN circuits. Of the sites with two WAN circuits, some have both circuits on one carrier (required to terminate at separate carrier POPs), and some have one circuit on each of two different carriers. Obviously, the data centers have large-pipe connections to all the carriers, but the end-site circuits are right-sized for the traffic to/from the particular site.

Each end-site gets a default route to the MPLS cloud, and a few specific prefixes from the data centers.

This was arrived at after many years of various arrangements. Using an IGP across the WAN for hundreds to thousands of sites just proved too problematic (actually slowing IGP convergence to a crawl), and forcing traffic to a central site, even if the traffic was site-to-site added too much latency.

4
  • Did you ever try deploying a private ASN per site (eBGP only)? Oct 4, 2016 at 5:00
  • That isn't practical (over 10,000 sites). There are some (extremely large) sites, which, for carrier or other reasons, e.g. physical proximity, are connected directly to the data centers, and those sites each have their own private AS.
    – Ron Maupin
    Oct 4, 2016 at 13:31
  • I am reading of several organizations that do it. The number of private ASN's was a limitation at 1,000. I am hearing some providers recommend this ...eBGP w/ Private ASN's (EWPA)? Oct 4, 2016 at 14:30
  • I suppose you could do it that way, but it may be a pain to maintain which sites are using which AS numbers, and adding or removing sites will require selecting and reserving a new AS, or returning an unused AS to the pool for reuse. We also hear that any business should plan for at lest 10x expansion from its current size, because it is so disruptive to re-architect a network if the business grows.
    – Ron Maupin
    Oct 4, 2016 at 14:39
1

In large-scale or "warehouse-scale" data centers, IETF concensus "proposes the use of an EBGP-only design" with a unique ASN for every top of rack (ToR) Ethernet switch rather than using either TRILL (their own layer 2 standard) or M-LAG (VPN, VPS, VCS, etc).

Granted a Layer 3 MPLS is not identical to Ethernet, given the millisecond triggered eBGP reconvergence for link down events, I see added value in a standardized approach. Therefore eBGP, or BGP w Private ASN's, is the best design (because the IETF informs us so).

Experimentation and extensive testing have shown that External BGP (EBGP) [RFC4271] is well suited as a stand-alone routing protocol for these types of data center applications. This is in contrast with more traditional DC designs, which may use simple tree topologies and rely on extending Layer 2 (L2) domains across multiple network devices.

They go onto provide an example ASN scheme.

5.2.1. EBGP Configuration Guidelines and Example ASN Scheme

The diagram below illustrates an example of an ASN allocation scheme. The following is a list of guidelines that can be used:

  • EBGP single-hop sessions are established over direct point-to- point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
  • Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
  • A single ASN is allocated to all of the Clos topology's Tier 1 devices.
  • A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
  • A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.