Setting system jumbomtu on Cat 6500

Question

We have an L2 path between a tape drive and a NetApp. The path goes through 3 switches (a 6509 and two N5K) but the two devices are on the same VLAN. All 10 gig fiber.

I found out that while most of the ports in the path are set to an MTU of 9216, one of them (the port on the 6509 core going to the first N5K) is set to 1500. The interface on the other side is set to 9216.

We are getting a max of 750 Mpbs between these two systems and I'm thinking this might be the cause.

I'm guessing that the systems are using PMTUD in this situation and are using 1500 byte frames when talking to each other but this is causing less optimal data transfer.

I've been told by Cisco that to change this, I need to enable system jumbomtu 9216 on the 6509 (it's not set right now) and then set the interface MTU to 9216 as well.

Since this is our core switch I'm hesitant to do this without knowing how the system jumbomtu will affect things. Since it has a redundant non-VSS sister, I should make the change on both of them.

Would there be any ill effects of making this change?

I'm reading things in Cisco's docs that I don't fully understand. For example, the setting jumbo MTU config section says:

"For Layer 2 Ethernet ports, you can configure only the global egress LAN port MTU size" <- (which is set using system jumbomtu)

There are several L2 (non routed, non SVI) ports configured for both 1500 and 9216 yet the system jumbomtu command is not set on this switch. Does that mean that the 9216 setting on these L2 ports is getting ignored?

If I set system jumbomtu 9216, will this effectively change all the L2 ports to an egress MTU of 9216?

The Catalyst switches are running IOS 12.2. No CatOS.

Answer 1

I think you may be confusing the VLAN database MTU and the interface MTU. Pay special attention to the note at the end. Enabling jumbo frames in the VLAN database for a VLAN will not suddenly cause all the switch interfaces in that VLAN to be enabled for jumbo frames.

Jumbo/Giant Frame Support on Catalyst Switches Configuration Example

Catalyst 6000/6500 / Cisco 7600 OSR Series

The Catalyst 6000 series and 7600 Optical Services Router (OSR) platform can support jumbo frame sizes as of release 6.1(1) of CatOS, and 12.1(1)E for Native IOS. However, this is dependent on the type of line cards that you use. There are generally no restrictions to enable the jumbo frame size feature. You can use this feature with trunking/non-trunking and channeling/non-channeling. The default MTU size is 9216 bytes after you enable jumbo frame support on the individual port. However, an Application Specific Integrated Circuit (ASIC) limitation requires that you limit the MTU size to 8092 bytes on these 10/100-based line cards:

• WS-X6248-RJ-45
• WS-X6248A-RJ-45
• WS-X6248-TEL
• WS-X6248A-TEL
• WS-X6348-RJ-45
• WS-X6348-RJ-45V
• WS-X6348-RJ-21

The WS-X6516-GE-TX is also affected at 100 Mbps. At 10/1000 Mbps, up to 9216 bytes can be supported. However, the WS-X6548-RJ-45 line card is not affected, as this line card uses newer ASICs.

Note: Jumbo frame support is available on WS-X6101 ATM modules.

Configure Native IOS

7609(config)#int gigabitEthernet 1/1 
7609(config-if)#mtu ? 
  <1500-9216>  MTU size in bytes 

7609(config-if)#mtu 9216

Verify in Native IOS

7609#show interfaces gigabitEthernet 1/1 
GigabitEthernet1/1 is up, line protocol is up (connected) 
  Hardware is C6k 1000Mb 802.3, address is 0007.0d0e.640a (bia 0007.0d0e.640a) 
  MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec, 
  reliability 255/255, txload 1/255, rxload 1/255

The port ASICs on the Catalyst 6000 count as oversized, which are those frames greater than 1548 bytes but less than the configured jumbo MTU. It uses the rxOversizedPkts counter to track these frames in the output from the show counter command. In this case, the ifInErrors counter value increments and can equal the number of rxOversizedPkts packets value in the show counter command output. On a Catalyst 6000 that runs Cisco Integrated IOS (Native Mode), use the show interface command to check whether the input errors counter increments along with giant counters on the interface that receives these frames.

Note: There is no relationship between the MTU value that you can specify in the VLAN database and the mtu command in the interface configuration mode. With the VLAN database setting, the switch checks for an MTU value greater than 1500. If the switch detects a higher value, puts the VLAN in a non-operational state. Therefore, to support large frames, you only need to change the interface MTU value and not the VLAN database MTU value. If SVI is used, then in order to support large frames, you must increase the MTU on the SVI to route the traffic between VLANs . Refer to the Configuring Jumbo Frame Support section of Interface Configuration for more information about Jumbo Frame support on the Catalyst 6500.

In any case, a switch will not send out jumbo frames to an interface, even on an interface with jumbo frames enabled, unless it receives incoming jumbo frames destined for that interface.

---

Edit:

Your concern about the system jumbomtu command is misplaced. That command sets the non-default MTU size to which you can configure some types of interfaces, but it doesn't actually set the interface MTU size. The warning is about setting the value to the default ethernet MTU value. Doing that will change all the interfaces with non-default MTU sizes back to the default value.

The default for the system jumbomtu is 9216, which seems to be what you want, so you should not need to change that value. If, however, you set your 10 Gb MTU to 9216, then you change the system jumbomtu value to 1500, you will find that any interfaces where you changed the MTU to 9216 will now have an MTU" of 1500 because the system jumbomtu is now different than what you are allowed to set the MTU.

In other words, for some types of interfaces, You can either use the default MTU of 1500, or you can change the MTU to the value set by the system jumbomtu command, but not any other value.