Do LACP MTUs (or other link-level MTUs) need to match?

Question

Does setting a custom MTU on the host side of a LACP bond require also customizing the MTU on the switch side?

Answer 1

Does setting a custom MTU on the host side of a LACP bond require also customizing the MTU on the switch side?

Your data-link MTUs:

• On servers the data-link should always match between servers within the same broadcast domain, or vlan. This isn't critical for TCP services because of TCP MSS negotiation, but UDP services need matched MTUs if they send large packets.
• On switch interfaces facing your servers, the data-link MTU must be at least as large as your host / router MTUs in that vlan
• On other switch to switch / switch to router links, the data-link MTU must be at least large enough for your server MTUs + any other services added by your routers (such as IPSec, MPLS, etc...). By way of example, if your transit switched MTUs are not large enough it can cause strange issues getting network file systems to work correctly.
• On router to router links
  • The routers' data-link MTUs should match your other routers, or you could wind up with strange IGP peering problems.
  • The IP MTU on routed interfaces should be the same size as your server's IP MTUs that must transit them.

If all that looks like jibberish, then just do the simple thing and ensure all your data-link MTUs are the same in your network; however, if you need jumbo or baby-jumbo frames, this can get complicated quickly unless you limit the places where jumbos go. One of the biggest challenges with deploying jumbos is inconsistent jumbo frame support in various network devices; however, most decent enterprise switches will support baby jumbos (for instance, MTUs large enough for MPLS tag stacks two deep).

Detecting MTU problems

Ping something in the same vlan (with DF-set in the IP header) to test your MTU configuration between those hosts.

Linux example, with an 1500 byte MTU...

If you have linux use: ping -M do -s <icmp-payload-size> <some pingable ip address in the same vlan>... This is success, pinging something in the same vlan...

[mpenning@tsunami ~]$ ifbrief
eth0      Link encap:Ethernet  HWaddr 80:ee:de:ad:be:ef
          inet addr:172.16.1.5  Bcast:172.16.1.255  Mask:255.255.255.0

[mpenning@tsunami ~]$ ping -M do -s 1472 -c 1 172.16.1.1
                                 ^^^^^^^
PING 172.16.1.1 (172.16.1.1) 1472(1500) bytes of data.
1480 bytes from 172.16.1.1: icmp_req=1 ttl=255 time=1.30 ms

--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms <------
rtt min/avg/max/mdev = 1.305/1.305/1.305/0.000 ms
[mpenning@tsunami ~]$

This is failure, because we exceed the MTU by one byte...

[mpenning@tsunami ~]$ ping -M do -s 1473 -c 1 172.16.1.1
                                 ^^^^^^^
PING 172.16.1.1 (172.16.1.1) 1473(1501) bytes of data.
From 172.16.1.5 icmp_seq=1 Frag needed and DF set (mtu = 1500) <--------

--- 172.16.1.1 ping statistics ---
0 packets transmitted, 0 received, +1 errors

[mpenning@tsunami ~]$

Windows example, with an 1300 byte MTU...

View your windows mtus with netsh interface ipv4 show subinterfaces and modify with netsh interface ipv4 set subinterface "[windows-intf-name]" mtu=[MTU-size] store=persistent