Regarding VxLAN Encapsulation

Question

I am reading up on VxLAN and understand the encapsulation process somewhat as below:

Step1: Take your original Ethernet frame.

Step2:Put it inside VxLAN encapsulation.

Step3:VxLAN should then go inside of UDP header.

Step4:UDP goes inside of IP(this should be the transport IP, i guess)

Step5: IP goes inside of whatever the transport is(e.g.Ethernet)

Q1: Please confirm if the above understanding is correct.

Q2:Why do we need VxLAN header to go inside UDP, why not send it over plain IP?

Q3:In other tunnel mechanisms, like the OTV, we don't use any layer 4 protocol(like TCP or UDP), so why use it here? Any specific reasons.

Q4: Why use UDP(since it is best-effort based), why not use TCP?

Q5: Can i look at VxLAN the way i look at HTTP or Telnet(both are applications and operate at layer 7), HTTP uses TCP port 80, telnet uses TCP port 23, what i am trying to understand is VxLAN an application that operates at layer 7 and fits into osi model ? Also, which OSI layer would OTV be operating at and why?One answer below says that OTV can be done using UDP as well rather than MPLS/GRE, does that make OTV a protocol that operates at layer 7?

I have also attached a snapshot of packet capture of VxLAN header from one of the video lectures.

Answer 1

Yes - your understanding of encapsulation is correct: a given frame has a VXLAN header applied. This is carried in a UDP packet.

UDP is used as a convenient format in terms of programming and its use of src/dst port provides a ready means to both multiplex connections as well as a means by which intermediary forwarding elements can hash connections over parallel links. In short, UDP is familiar, has low overhead and is already extremely well understood.

OTV can run over MPLS-GRE or UDP, with UDP being the preferred mechanism for the past few years. Again, one of the big drivers was depolarization of traffic (allowing parallel paths to carry determinstically hashed fractions of overall traffic).

Why not TCP? Excessive overhead on the encapsulating device and added latency are big examples. The bigger point, though, is that to get any value out of TCP would require the ability to concatenate the individual packets to be encapsulated into an overall stream to be managed via sliding windows rather than simply maintaining a 1:1 mapping. Add in the amount of state tracking and buffering issues associated with having the capability for retransmission and it becomes truly unruly.

Answer 2

One additional reason to use UDP vs L2oIP is to enhance entropy. In most cases load-balancing is done per flow/5 tuple, UDP ports are the keys in the hash, VxLAN allows UDP source port to be set to any arbitrarily value (within the range), that could be exposed to the application layer, hence increasing granularity in flow setup and better load-balancing while traversing the network towards remote VTEP

Answer 3

While the accepted answer explains well why UDP is used instead of raw IP, it doesn't explain well why TCP is not used. The short reason is that the Ethernet packets traveling inside a VXLAN tunnel have most commonly already one level of TCP. If you encapsulate those packets inside another level of TCP, you have two levels of TCP in the same packet.

TCP over TCP is a bad idea: http://sites.inka.de/bigred/devel/tcp-tcp.html

While TCP over TCP may appear to work if packet loss is low, having retransmissions at two different levels makes the system increasingly inefficient at high packet loss rates.

So: don't do that! UDP or raw IP is a much better encapsulation, as they don't provide another unnecessary level of retransmissions. Tunneling protocols exist both for UDP (GTP, VXLAN, Geneve) and for raw IP (GRE).

Answer 4

Another reason for using UDP is that it is a 'standard' protocol, so unlike things like GRE/IPSEC and similar, you don't need intermediary devices to support anything exotic.

Consider, too, that you can implement TCP somewhat efficiently on top of UDP, but not the other way round: if you choose TCP, you opt-in to all kinds of extras (as compared to UDP), like error detection and retransmits, that will cost you whether you make full use of them or not.

One reason VxLAN was developed in the first place is that placing headers directly in the frame would effectively make this a GRE or VLAN. It also has limitations, notably linked to the fact you can only pile an extra 20 bytes onto your frame so many times before it requires MTUs to be adjusted throughout your network, or will start impacting throughput (not to mention switching overhead linked to reading/popping/adding VLAN tags).

VxLAN provides VLAN like isolation with point-to-point connectivity, meaning your network can be entirely ignorant of the details, or indeed, of the overlay network.

Not related to VxLAN, but for 'building TCP from UDP + time', see for example https://hpbn.co/building-blocks-of-udp/ (and I would recommend reading the TCP section to understand what the disadvantages of TCP vs UDP can be).

Answer 5

Key to understanding the design of VXLAN is understanding where it is intended to be used.

It is designed for big multi tenant datacenters that need to provide virtual Ethernet networks to their tennants but are outgrowing the conventional VLAN/STP soloution in terms of the number of VLAN numbers or the ability to efficently route traffic.

In such environments it is important that the overlay network plays nice with the underlay network. The underlay network is likely to be very high traffic and thus make use of features like Link aggregation and Equal cost multipath.

ECMP and link aggregation rely on being able to hash the header fields of packets so that packets belonging to the same transport layer session go down the same path and hence are delivered in-order. To maximise efficiency you want different sessions on the overlay network to be regarded as different sessions by the devices on the underlay network.

Most better ECMP and link aggregation implementations understand UDP. VXLAN uses the UDP source port number to communicate session hashing information from the overlay network to the underlay network.