I'm looking at implementing a linux-based VPN setup with approx 1200 remote sites connecting into two redundant data centers/DCs. Each remote site has 4 networks, and each DC has ~10. The VPN itself is GRE over IPsec, but what I'm struggling with at present is what is the most appropriate routing protocol to use.

  • Each remote site only needs to know what networks are at the data center, they do not need to know about the networks in use by other remote sites.
  • The networks at each data center may move between each other from time to time, and new networks added (don't want to update static ipsec configs for 1200+ sites each time this happens).
  • The DCs needs to know what networks are in use by every remote site.
  • When the primary DC dies, traffic should route automatically to the backup DC.
  • Most of these remote sites are on dynamic IPs/3G so it's common for sites to come up/down a couple times during the day, and across 1200 sites that equates to a lot of topology changes.
  • The required capacity at present is around 1200, but could grow to 1800-1900 by end of next year.
  • Cannot use closed solutions (Cisco etc, also EIGRP even though it's technically free now).

From my reading of OSPF, it does not appear to support the number of sites.

My questions are:

  • What routing protocol is best suited for this setup?
  • How should I structure it to handle the large number of sites?
  • 1
    What brand/model of hub router will you be using? Your only real choice is OSPF, but there are "tricks" you can use depending on the router type. This link might help too: ciscopress.com/articles/article.asp?p=417092&seqNum=2
    – Ron Trunk
    Commented Jul 8, 2014 at 16:27
  • 1
    If you're landing on a Linux box, have the Linux box learn the routes from the VPN as it comes up, then run OSPF or some bgp daemon to redistribute the connected VPN routes into the 'real' dynamic network? The interesting bit would be the management of learning the client routes without a dynamic protocol across the VPN... high-level help is free, right? Or find something similar to Cisco's ODR for Linux.
    – cpt_fink
    Commented Jul 9, 2014 at 5:03
  • 2
    With more than 1000 vpn sites, I am struggling to understand the decision to build this on a "free" solution via linux, quagga, etc. Compared to commercial offerings, linux routing/vpn is very painful to build and support. Please explain how openwrt is involved and what exactly you are running openwrt on Commented Jul 9, 2014 at 9:58
  • 1
    The sites have existing equipment running on an openwrt variant and can't be changed (hardware can't, software can). Besides the expense of swapping out 1200+ sites, the hardware also provides some very specific telemetry data, so it would still be required even with Cisco in front. I suppose the DC ends could use Cisco, but Linux on the remote sites means it would need to use a standards based approach.
    – Mike
    Commented Jul 9, 2014 at 10:16
  • 2
    @Mike I would imagine eBGP is probably your best bet. Your setup is scaling beyond the recommended (and viable) limitations of most IGPs.
    – Ryan Foley
    Commented Jul 9, 2014 at 10:43

2 Answers 2


I think BGP is probably your best bet in this situation. The number of tail sites you have quickly exceeds the benefits of most IGPs. BGP would allow you to achieve all of your desired implementation requirements, such as limiting tail site knowledge to the primary/backup DC, standards based protocol, ability to efficiently handle over 5,000 different routes, etc.

OSPF is still an option; I just don’t think it’s the best option. You could always setup each leg of your network as a Totally Stubby Area so inter-area routes aren’t propagated down to the each site. Then you could have separate routing instances at each site to handle their networks independently of your head end.

Each SPF recalculation is going to cost you a pretty penny on resources. How volatile these connections are will affect that, this could be often depending on how aggressive your timing is.

Whatever your decision, BGP was designed to scale. And when anyone says they are looking for a way to scale to 7,000+ networks1, BGP is a no-brainer.

1 Forecasting based on growth to 1,900 sites x 4 networks at each site.


Mike, What you want to do is pushing the limits of any protocol implementation, so there is no good answer. Both OSPF or iBGP could be made to work. Any solution we (you) come up with will necessarily be a tradeoff between router processing power, memory (ie. cost), bandwidth, and convergence time. For example, you can break up OSPF into some number of areas to reduce the flooding, but at the expense of memory and CPU.

Convergence time is important, because the faster you want your network to converge, the more CPU and bandwidth you require. As an extreme example, if you want to converge in a second or less, the hub router must generate, send, receive, and process 14400 hello packets per second (2 x 4 x 1800), and you use significant bandwidth doing so. That is in addition to actually routing traffic. That’s a lot to ask of any router.

Both BGP and (Cisco’s implementation of ) OSPF can filter routing updates to the spoke routers, reducing bandwidth. But the hub router will still generate them internally, and that takes a lot of CPU and memory. I don’t know how much you actually save.

I will emphasize @MikePennington’s point that for a network this size, you should really look at a hardware router. If not for your remote sites, then certainly for the hubs – that’s where all the processing is going on.

  • I wouldn't agree that it is pushing the protocol implementations for any protocol - We have purchased routes that have been tested by the vendor to run 2500 BGP session because we expect to get near that figure on them. It becomes a hardware scailing issue and software implementation issue of the protocol, not an issue with the protocol specifically (although I don't agree that IGPs like OSPF are a valid suggestion here).
    – Baldrick
    Commented Jul 9, 2014 at 12:54
  • @jwbensley I agree with you, essentially. That's why I said "protocol implementation" instead of just "protocol."
    – Ron Trunk
    Commented Jul 9, 2014 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.