I've been trying to search for data on the scaling figures of the various network ASICs with regards to the maximum number of remote VTEPs they can handle when doing ingress-replication (a.k.a. Head-End-Replication). HER is a way of emulating broadcast traffic in an overlay by creating multiple copies of the same packet and send them to all the VTEPs in the same VNI. Some vendors claim some numbers, whereas others claim others that are 1 or even 2 order of magnitude higher.

Any idea? I'd be keen to understand how HER is handled by the ASIC.


I'd be keen to understand how HER is handled by the ASIC.

I'll attempt to answer this bit, but this is only speculation since I don't have first-hand knowledge of vendor ASIC specifications.

Forget about VXLAN for the moment, and consider how an Ethernet switch ASIC works. Ethernet switch ASICs have built-in support for replication. That is, when a BUM packet enters one interface, it is replicated in the ASIC to interfaces on the same vlan. This happens at the "full switching rate" of the ASIC switching engine, i.e. the ASIC forwarding pipeline sees only one packet irrespective of the number of copies that are generated.

So coming back to VXLAN, then, it is possible that the ASIC is able to extend the same replication mechanism to VXLAN as well; only, instead of physical interfaces, the ASIC now has to replicate the packet to multiple VTEPs. If this can happen at the "full switching rate" of the ASIC, i.e. the ASIC forwarding pipeline sees only one packet irrespective of the number of copies that are generated, this is obviously awesome.

Less capable ASICs would rely on recirculation [Digression: Recirculation is the process by which a packet is transmitted out of an interface and it comes back in on another interface - the interfaces in this case are not external interfaces on the front panel of the switch, rather they are "internal" interfaces in the ASIC silicon, but for the purpose of visualization you can think of two physical interfaces connected back-to-back. Switch designers rely on recirculation when the ASIC is incapable of achieving some functionality in one pass, and the packet has to be sent through the ASIC multiple times before the output packet emerges on the desired interface/vlan with the desired encapsulation]. So if the ASIC's VXLAN switching path relies on recirculation, then think about what happens to a BUM packet: if there are 5 VTEPs that the packet is to be replicated to, then the ASIC pipeline sees 6 packets (1 original plus 5 copies), which reduces the ASIC's overall throughput.

So in summary, if your intention is to evaluate and compare the ASIC capabilities from Vendor A and Vendor B, the question to ask them is whether their ASIC employs replication in the ASIC for Head-End Replication of VXLAN packets, or whether their ASIC uses recirculation.

  • Good answer! Some say though that the usual packet replication pipeline doesn't apply as in this case the replication is different and needs manipulation of the outer IP header to reach the various VTEPs. Nevertheless your explanation of normal packet/frane replication is spot on Feb 6 '18 at 17:08
  • @NicolaArnoldi you should accept the answer if it helped you in any way. As DRP has stated, finding an official document will be highly unlikely, as companies typically will protect their secrets. :-)
    – user36472
    Feb 6 '18 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.