I'm trying to make a PoC about VXLAN, and I have a design issue.
I have one c3750 per client, each 3750 is connected to an N9K with a single fiber (per client).
So each N9Ks are connected to multiple client. I then have 2 ASR9K as border routers doing some eBGP.

I can get 4096 vlan = 1 vxlan with dot1qtunnel, then propagate VXLAN over BGP on N9K, so the clients can use their vlans on their multiple sites.

Although, I need to find a way to give an L3 gateway per client, and I don't find how to reach this goal:

    If I stop the VXLAN before the ASR, I can only give it one VXLAN since I need to use one access vlan for dot1qtunnel to work, and this vlan only can be associated to one vn-segment
    If I propagate the VXLANs to the ASRs, I don't see how I could get one "gateway vlan" per client (at least I can't make it work for now, with an nve up but vni down)

Is there a way where I could, with only one physical link per client site, both use vxlan and provide him internet access?

  • I'm not quite sure that I understand, but each VLAN needs its own gateway. You can't have one gateway for multiple VLANs from a client. A gateway needs to be on the same LAN as the hosts for which it is the gateway. A gateway is the host on a network than knows how to reach other networks, and it needs to be on the same network as the hosts, otherwise the hosts would need a gateway to reach the gateway on a different network.
    – Ron Maupin
    Oct 5, 2017 at 14:57
  • Indeed, what I'm trying to do is defining one "gateway vlan" per client. Client 1 will be free to to whatever he wants with its vlan, but need to use vlan 101 as an internet gateway, client 2 102, and so on. Then, I'd like to have one vlan interface per vlan on the ASR, which will be the client L3 gateway. Oct 5, 2017 at 15:08
  • That's fine, but if a client is using multiple other VLANs, those VLANs each need a gateway. What I think you describe would only allow a host on the gateway VLAN to access the Internet. Hosts on other VLANs would be unable to access the Internet because they do not have gateways.
    – Ron Maupin
    Oct 5, 2017 at 15:12
  • Thanks, I'm fine with providing only one gateway per client. Now I just need to find a way to do the job on the ASR. Oct 5, 2017 at 15:24
  • Simply create subinterfaces of the physical interface. interface GigabitEthernet0/0/0.101
    – Ron Maupin
    Oct 5, 2017 at 15:41

1 Answer 1


I'm trying to decode what you're doing, but I think I'm reading something like the following-

  • You're using VXLAN-EVPN on the N9K's and the ASR9K's. The Nexus boxes are acting as standard VTEP's providing L2/L3 services while the ASR's are basically border leaves.
  • The 3750's are pure L2 devices that pick up trunks from the upstream 9K's and break out access ports to downstream customers.

So - if we assume the above to be true then there are actually a few ways you can approach the problem. The q-in-vni approach mentioned isn't really one of them, though, as that's really more intended to tie together pure L2 environments. This would be applicable if, for example, if you had a client with a few 3750's spread across the network and you wanted to allow them to run a full set of VLAN's among them without having to provision each one on each VTEP. Put another way, there's no gateway involved.

Anyhow - back to what would work. Let's assume you assign a unique VXLAN VNI to each customer network you'll be provisioning. This VNI can show up on any of the VTEP's as needed. Wherever it appears you'll associate it to an SVI which, in turn, would be configured with the customer's vrf and an identical anycast address (i.e. the same default gateway will be present everywhere this customer's network appears). Keep in mind that wherever a given customer's vrf appears you'll also need to create an SVI to associate with an L3VNI. The SVI's don't need to be identically numbered across your VTEP's, as you'll be establishing a mapping on each box between VLAN and VNI numbers.

You'll also need to establish an underlay IP network amongst the N9K's and the ASR9K's. It's possible to do this with pure eBGP but I'd strongly recommend using a limited IGP deployment and then setting up iBGP - either with a full mesh or, preferably, a few route reflectors. There's a great deal written about this on various design guides. There are also some considerations that come in if you'd like to use VPC between pairs of VTEP's to provide redundancy to your downstream clients.

So - the two ways in which you can now manage VLAN numbering on the VTEP is either using the traditional mechanism, which is to globally assign a given VLAN on VTEP-wide basis. You'd say, for example, that VLAN 100 gets mapped to VNI 10100 and then simply allow that VLAN over a trunk to a 3750 that contains said customer. The next customer's VLAN might be VLAN 101 that gets mapped to VNI 10101. Put another way, any reference to VLAN 100 means the same thing on any port on that box.

The other way this can be managed is with port-local VLAN mapping. In this case you can actually map the VLAN identifier on an individual port to any other VLAN on the box. In this case you might create a policy that says that every downstream customer is going to see VLAN 100 but that you'll actually map 100 on the customer facing port to whatever value you want on the box. As an example - perhaps you assign VLAN's 2000-3900 to individual customer VLAN's (..each mapped to their respective VNI, obviously). On the first customer-facing port you might map incoming VLAN 100 to local VLAN 2000. On the next customer-facing port the mapping might be VLAN 100 to local VLAN 2001, and so forth. This potentially simplifies client configuration (...everyone just uses VLAN100 from their point of view, after all) and hides the details of your network configuration.

Now - regardless of your choices above, the ASR side is actually not that big of a deal. The clients are actually routing to default gateways on the Nexus 9K's which, in turn, are using BGP-learned EVPN prefixes to send traffic on to the ASR - which can then be configured, for example, to cross-import these routes to another vrf with Internet access (...again, just an example). The opposite is also true (the N9K's are advertising routes and MAC addresses via BGP to both each other and the ASR's).

I've glossed over a number of details about EVPN design and implementation and would strongly recommend that you read some of the design and configuration guides out there to get a better feel for how these things work together. It's pretty approachable and familiar if you've done a fair amount with non-trivial BGP and self-managed MPLS but might be confusing if these things aren't in your background. It's also worth mentioning that the above solution is going to provide full L2 reachability within common VNI's for a customer and will provide routing between VNI's if a customer has multiple networks within their vrf.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.