Does it make sense to deploy OSPF across Metro Ethernet?

Question

When I need to connect a number of branch sites to one another for a customer, I typically recommend an MPLS VPN through a trusted carrier. The CE at each site speaks BGP with its upstream PE and each site is numbered with its own private ASN. This is very convenient for us as BGP provides myriad traffic engineering tools and our adjacencies are limited to n+1 for n sites (the +1 being our colo environment).

Lately, however, I've noticed increasing customer interest in Metro Ethernet solutions. Many of our customers have branch sites within a common metro area and MetroE quotes are coming in at several hundred dollars (USD) less than MPLS service for circuits of the same speeds. While this is appealing, I'm not sure how to best establish routing across a layer two backbone while avoiding turning a mesh topology into hub-and-spoke.

BGP would necessitate a full mesh of adjacencies among branch sites in order to maintain mesh connectivity, which obviously is undesirable from a scalability perspective. The other option is to deploy an IGP, namely OSPF, and have all sites form adjacencies across the WAN. I'd like to address each site as its own area, which seems like overkill but I want to preserve the ability to summarize routes advertised from each site and this can only be done on area borders.

Does this make sense? Are there any caveats to watch out for when deploying OSPF in this manner (for example, should I consider disabling LSA flooding)? Or is there another solution I've overlooked?

Answer 1

This is kind of a complex question, as the two different products are very different. A MPLS L3VPN circuit inherently is a full mesh between all locations that participate, while a MetroE connection is generally a point to point link between two specific sites.

In my experience, a MetroE circuit is a directly provisioned path, with no protect services, unless contracted with a protect path. This means that failure of an interface or router along the specific path would be traffic interrupting between the two sites that are directly linked by the MetroE service. The MPLS L3VPN will heal around interface/routing failures to keep you with a full mesh between your sites. This generally accounts for the price difference between the two.

There is nothing wrong with building your own services on top of a MetroE platform -- you just need to work with customer requirements to determine what type of routing is appropriate. If you are working with a small office network, OSPF/IS-IS/EIGRP could be a wonderful way to exchange routing information between the directly connected sites that you have established. If you're more of an NSP/ISP/*SP, separating infrastructure and customer routes between IGP and EGP becomes much more important as you scale.

As an ISP engineer, we extensively use MetroE and EWAN links to build our backbone, and utilize our knowledge of the physical links to design our iBGP/eBGP environment. In many cases, we use route reflectors, and dual route reflectors (route-reflector-client on both sides of the peering) to reduce our iBGP peer count. However, unless you're dealing with 6+ routers in a POP, iBGP scales pretty well.

In short -- if it's for a single client, that is not offering network services to clients of their own -- stick with an IGP. If external connectivity needs to be shared between sites, with failover/redundancy/etc, closely examine the physical paths you have, and design your eBGP/iBGP to reflect that. No point in having a router in a remote location with only 1 link outside the site to iBGP peer with all other routers in the AS -- use a dual route reflector.

Answer 2

Switching from a managed L3VPN (what I assume you meant by "MPLS VPN") to an L2VPN is a nice step up in that you can run non-IP protocols, and get total control of the routing protocols and routing platforms that define your routing topology.

Assuming that you place just one Ethernet MAC address on the CPE-side of each of your sites, it's much simpler for the provider's equipment to learn and forward 1 MAC addresses per site, than to learn and route potentially-many subnets per-site.

Protocol-wise, this is a bit of a tricky question to answer without a lot more information, as the best choice depends on your traffic and growth plans.

Is this just one big customer that needs internal and internet connectivity, or does it sell connectivity as well? Assuming it's all internal, then you'll just be deploying an IGP and maybe running some eBGP for announcing paths out.

If you don't have very many sites or internal prefixes, a link-state protocol like OSPF or IS-IS makes the most sense, as it will converge quickly and can build the FIB from the RIB quickly if there are just a few prefixes.

If you will have many sites or many prefixes, this will start to tax your routing platforms, as they each need to process each. This is something that starts to take n² time. If you have sites that are coming up and down often, this churn in link-state can also start to tax your router pool.

If you are going to have many sites, many stubby sites (one path "upstream", no other downstream routers), or lots of route churn, you're going to need to look into other protocols or topologies.

In a case like that, I'd recommend using BGP with some route reflectors. This way, you can provision 2+ heavy-duty route-reflectors that spokes announce into, and for which the other spokes can grab a routing table from. This way, you can deploy lightweight CPEs at your many spoke sites that just connect up, announce their space and get an internal table or default route to a router that does.

Approximately, I'd suggest some scales and gear (and assuming sub-Gigabit throughputs):

• 1 - 20 spokes -- OSPFv3 between all sites. Juniper SRX240 or similar for all sites.
• 20 - 100 spokes -- iBGP with route reflectors. Juniper SRX240 in the spokes, Juniper MX5/10/40/80 for route reflection (or Debian Linux/BIRD).
• 100 - 500 spokes -- Start breaking them into different L2 networks, ASes, or areas

Answer 3

Just to add two commonly-overlooked bits to the BGP discussion:

• If you run iBGP, the you usually need another routing protocol to establish connectivity between BGP next hops. From the design perspective, iBGP is more a scalability tool than a routing protocol;
• If you run eBGP, you don't need a full mesh of BGP sessions for optimal end-to-end traffic flows; BGP next hop processing nicely solves those issues.

See http://blog.ioshints.info/2011/08/bgp-next-hop-processing.html for more details

Answer 4

You could very well run OSPF (or other IGP) on a multi-point metro-Ethernet service, and it should work very nicely.

There could be reasons to continue to run BGP, however...though they would be much the same arguments of why you might want to run BGP within your own network as well.

You don't necessarily have to put all of your BGP speakers in the same AS on a "broadcast" network like that. Think of it as a sort of internal IXP run by the telecom where your private ASes can interconnect with each other via a layer 2 mesh. You wouldn't even have to necessarily maintain a full mesh of BGP peerings as BGP updates can carry a next-hop in its update message that isn't the same as where the peer session is coming from.

So, for example, if you have a layer 2 mesh with routers A, B, and C on it and you have BGP peers between A and B, and between B and C, when C gets updates for routes originated at A, it will have A as the next-hop, even though they were learned through the peering session with B. Obviously you would want more route peering than just a single hub-and-spoke, so you're not dependent on the single hub, but you don't need a full mesh by any means.

You still get all the routing-policy benefits of running BGP if you do it this way...and it also, as another respondent mentioned, can use the same private AS number space and could even interconnect with the existing L3VPN so your model and support mechanisms don't need to change.

That being said, I have a couple of metro-E links (point-to-point in my case) that I run OSPF and iBGP across and it works just fine.

Answer 5

What about a route-server configuration with 'spokes' beeing rs-clients of the hub/main routers?

Even though the bgp peerings would be like a hub&spoke topology, all the spokes should be able to send traffic directly to all others.

You could even re-use the same private AS numbers as those used with the MPLS provider to facilitate migration from a service to another.

Answer 6

I would highly recommend reading up on and making a decision regarding alternative OSPF topologies such as starting out as NBMA instead of standard. You will soon realise there is no way for OSPF to correctly select between two paths going to the same router/site within the same metro ethernet, due to the way cost is calculated via outbound interface, both the main and backup WAN links will appear same cost in standard OSPF. However if you choose to use NBMA for example then you can manually define neighbor costs - but now you have to manually define the mesh/adjacencies.

Whatever you do, ROUTE OVER IT DO NOT I REPEAT DO NOT JOIN AT LAYER 2, you're just asking for trouble later on, if you need layer 2 interconnectivity, use OTV or other layer 2 over layer 3 feature.

You'll quickly find you'll learn a lot more about OSPF (and need to know a lot more) than compared to a simple provider MPLS-VPN where the WAN core is hidden from you.

Answer 7

OSPF over metroE works fine but you will need to make sure it fits your needs and you architect accordingly. One caveat I haven't seen mentioned is to make sure you know what MTU your provider supports. I have seen an MTU drop on a metroE link during a provider maintenance cause the OSPF neighbor to not come up. Probably only a problem because they don't really support jumbo frames but they did just for us:) Being nice sometimes doesn't pay off.