Network Engineering Stack Exchange is a question and answer site for network engineers. It only takes a minute to sign up.
Sign up to join this community
The carrier is providing Q-in-Q tunneling to all 20+ remote sites and corporate. My questions is: what is the best way to utilize this setup?
Should I only use a transitional VLAN & subnet and route everything?
Should I pass all the VLANs back to the main site and route from that location?
Should I pass only the corporate VLANs out to the remotes and route at each remote?
You didn't really provide enough information to answer this directly (routing protocols, traffic patterns, vendor choices, etc), but this is the way I would evaluate the problem.
If you are working with a routing protocol that may reach "all" the peers on a single VLAN, you may get unusual behavior you didn't expect depending on how your IP addressing is handled. For instance, in a network with default settings, the "highest" IP will be elected in the case of OSPF. This means that, for instance, if you number your networks with .1 being the "core" and .254 being the most distant edge, .254 will be the designated router. So you'll have to carefully consider the choice of routing protocol and the configuration of that protocol. In a case where you may want to split the routing into two areas in the case of OSPF, you'll probably want multiple interfaces.
In the case where the latency is highly variable between sites (e.g. a QinQ service provided through a single "billing" carrier but potentially multiple delivery carriers, fairly common in large metro markets), there may be unusual effects on routing protocol timers as well. This may cause problems in protocols such as EIGRP, which relies heavily on hello timers and acknowledgments of updates. This is going to lead to poor scaling, in the sense that you'll be sending a lot of hellos that every router will have to process. Further, without specific configuration (EIGRP stub routing) each site will also end up acknowledging every other site's routing information, and more importantly, when a site is down HQ will ask every other site for that subnet. In a situation where you have a flapping site, this will also cause issues.
Some vendors may have varying support for features designed to mitigate these design issues, so the vendor choice is a factor.
BGP would solve these problems but it is a higher learning burden for most network engineers not experienced with it. There are ways to solve these problems with OSPF and EIGRP as well, but it requires a sensitive awareness of conditions and requirements.
For 20 sites, I don't think this would be a significant amount of processing or concern. In the case where sites need to communicate to each other frequently it may make more sense. But if the sites mostly need to communicate with the core and very rarely to other sites, I would consider single VLANs used as "transit VLANs" from the core to each site. This means you only have to provision the VLANs once on each location, and once at the core. Doing N*sites*VLANs is quite a big provisioning exercise and will not scale well and will be enormously complex to operate as the number of sites grows without a good network operations team that understands templating and programming.
For a hybrid design with most traffic going through the core and an occasional "side VLAN" for direct communication between two sites, there may be some value to you. But you'll have to carefully consider the routing interactions that might result. It is best to consult with your provider to understand the details behind how the network is built and provisioned on their end (do they own all the resources? Are there "central POPs" that might remain up where other pieces of the network can be down? etc) in order to better understand your choices.
It's a common trope in the industry that routing designs scale better, but this is mostly a response to very flat very large switched data center networks which have a wide failure and broadcast domain.
In the design options you have proposed, I would want to understand the requirements better before making a specific recommendation, but I think that I would lean to individual interfaces from the corporate site to the remote site as "transit" VLANs with each site maintaining its own VLAN database and treating each interface at the core independently (hub and spoke).
An example interface configuration for this option on Cisco gear would be:
int E0/0.201
encaps dot1q 201
no shut
ip address 192.168.1.1 255.255.255.0
int E0/0.202
encaps dot1q 202
no shut
ip address 192.168.2.1 255.255.255.0
int E0/0.201
encaps dot1q 201
no shut
ip address 192.168.1.2 255.255.255.0
int E0/0.202
encaps dot1q 202
no shut
ip address 192.168.2.2 255.255.255.0
Then you would run a routing protocol on each of these interfaces, almost exactly like an individual router would have multiple distinct interfaces to different routers.
The provider can provide you with a list of acceptable VLAN tags or can allow you to run any tags, depending on how their service is provisioned.
If instead what you want is to bridge large broadcast layer 2 domains between multiple sites and have servers/hosts/clients at multiple locations be on the same VLANs so they can communicate to each other "locally", there are many additional concerns. This can create real problems without a thorough understanding of your environment, since providers doing QinQ often fail to pass some important protocols that do not operate well in a QinQ environment.
Sounds like you don't know what your design goal is, or at least haven't explained it clearly.
If you are using a hub and spoke setup, which would effectively be a VLAN per tunnel, then you should just route via the tunnel everywhere. You could run something like OSPF on the hub, and an area per tunnel. Then just advertise sumamry default route to the branch so it goes via the hub. This is providing you want the routing VIA the hub.
It's really not clear what your design goal is or how your QinQ tunnels are setup. You should ask yourself these questions and then give us the answers
Do you have spoke to spoke tunnels? Does each site have local internet breakout? Does everything need to go via the hub?
If the sites have local internet breakout, you could use something like DMVPN, and just advertise a default route to the branch. Your spokes would then just build dynamic vpn tunnels between them via the internet.
I'd do something along the approach Arima mentioned with the sub int per vlan but for IPs I'd probably use /30 or /31 if I could.
Basically an underlying P2P. Then I would absolutely run a routing protocol and in this case I'd probably go with eigrp.
Then at each remote you would have your various voice, data and misc vlan / subnet that would be specific to that site.
Leverage L3 to the access as much as possible.
That's just me anyway. Keep it simple but keep it L3.
hat ultimately depends on what you want to do. The carrier providing Q-in-Q only means that its providing a transparent connection between your sites. You should be designing your network as if the carrier is not even there. Consider the carrier's network a long ethernet cable between your sites, and design accordingly.
