Jumbo Ethernet Frame Deployment

Question

What are some general guidelines for the successful deployment of jumbo frames in a routed environment?

Answer 1

Terms

Physical interface -- a layer-2 port or switchport. Refers to switch uplinks and downlinks in the context of this document. May or may not be performing 802.1q tagging.

Logical interface -- a layer-3 port or routed port. Refers to server interfaces, router interfaces, switch virtual interfaces and subinterfaces. Anything with an IP address.

Standard domain -- a set of layer-3-adjacent VLANs which operate at the standard Ethernet payload of 1500 bytes.

Jumbo domain -- a set of layer-3-adjacent VLANs which operate at an Ethernet payload MTU greater than 1500 bytes, usually 9000 bytes.

Network Topology

Jumbo frames are not a mix-and-match technology. Jumbo frames are deployed to a clearly delineated segment of the network which is bounded by a router. This layer-3 boundary is essential to support interoperability with the standard domain. The router provides fragmentation and ICMP responses for Path MTU Discovery that are not provided at the lower levels of the networking stack. All communication within the jumbo domain will enjoy support for larger payload, while all communication between standard and jumbo domains will occur at the standard payload size. Communication between discontiguous jumbo domains also occurs at the standard payload size.

Jumbo VLANs

Jumbo frames are not a mix-and-match technology. This point is worth repeating. A given VLAN belongs to the standard domain or the jumbo domain, never both. This means that all physical interfaces that carry a jumbo VLAN must permit jumbo frames. If a jumbo frame encounters a physical interface that does not permit jumbo frames it will be dropped. Likewise, all logical interfaces on a jumbo VLAN must be configured with the exact same MTU value.

Physical Interfaces

Physical interfaces can carry any number of standard and jumbo VLANs as long as jumbo frames are permitted on the interface. Also, both types of VLANs may be trunked to a host or router. It is only important that the logical interfaces on the host or router be configured with the proper MTU value for each respective VLAN. In this sense, there exists a disconnect between physical interfaces and logical interfaces. A physical interface must meet or exceed the jumbo MTU, it does not need to be an exact match.

Logical Interfaces

Jumbo frames are not a mix-and-match technology. There it is again. For all intents and purposes every physical interface in the network can permit jumbo frames whether it carries a standard VLAN, a jumbo VLAN or both. The logical interfaces is where the exact MTU match really matters. This is also where it can get a bit confusing. For instance, configuring a Cisco 6500 SVI for MTU 9000 permits an Ethernet payload of 9000 bytes while configuring an HP NIC team for MTU 9014 permits an Ethernet payload of 9000 bytes. This is because the Cisco value only specifies the payload and the HP value specifies the payload plus Ethernet header. It is important to be aware of these details to get the MTU match exactly right.

Validation

It is useful to be aware of methods for validating that a jumbo VLAN has been properly deployed. Ping tests are sufficient, but one must take care to set the DF bit. Both Cisco and Windows ping utilities offer a method to do this, but once again the method for counting the bytes differs. The length parameter on Cisco includes the ICMP payload, ICMP header and IP header. The length parameter on Windows only specifies the ICMP payload. A length of 9000 on Cisco and a length of 8972 on Windows both produce an Ethernet frame of length 9018 with a payload of 9000 bytes.

Conclusion

Hopefully someone out there will find this diatribe useful. In networking, I think we all know the importance of doing it right the first time. Please feel free to leave comments regarding the accuracy of information, potential improvements and additional references.

References

http://en.wikipedia.org/wiki/Jumbo_frame

Answer 2

Or to summarise:

• Keep it consistent across your environment - check what the maximum MTU value that your infrastructure will support, accounting for the various ways each vendor calculates it (with or without L2 Headers)
• Account for physical interfaces, SVIs and L3 interfaces
• Be aware of protocols like OSPF that enforce MTU matching between devices before they will form an adjacency
• Get the network right before touching any servers. Servers without Jumbos configured will work just fine across a jumbo-enabled network, but the opposite is a disaster.

Answer 3

Depending on your network layout, you should make sure and start from the inside out. Start with links in your core and work out from there through your distribution and access layers (if applicable). Make sure your network completely supports jumbo frames before you start enabling it on your access ports. Most 100/1000 end devices these days don't support jumbo frames, and will (obviously) default to standard mtu if they do.

The best advice I can offer if you are thinking about deploying jumbo frames across your network is make sure you aren't blocking PMTUD (v4 or v6) anywhere within your network. As long as you leave your upstream ports until last, you shouldn't have any problems within your network.

Answer 4

One aspect not covered here is the why.

Jumbo frames can be of great benefit to some applications, but it's worth actually benchmarking your applications to see if you really will get a benefit that covers the costs. These days even for the applications that benefit most it's rare to even get a 10% improvement in throughput, which may not cover the increased operational costs. Inside a backbone using tunneling (MPLS, L2TP, etc.) it can be required, and for little cost as a single backbone is usually a single group of administrators, however once you start extending jumbos to the access layer it quickly gets more complex.

Another thing to consider is that all your documentation and procedures will need to be updated to ensure any new devices added (or RMA replacements) don't become silent black-holes for traffic.