MTU/MSS Oddness?

Question

We have an IKEv2 IPSec tunnel between two Sophos XG Firewall appliances in a corporate system for a remote site.

We don't have any specialized MSS or MTU settings other than what the IPSec tunnel already applies.

Our IPSec configuration profile for the handshake of IKEv2 IPSec tunnels fits the following profile:

... and the tunnel is established.

HOWEVER we're seeing a huge skew from expected traffic - the MSS is 1382 bytes, but we don't seem to be able to replicate this in our calculations.

MTU on the actual network links between everything is 1500 MTU (ISP uplink, Ethernet MTUs internally, etc.) so we're headscratching a little bit here on how the MSS is being calculated here.

If this is, ultimately, a cause of the Sophos XG IPSec tunnel doing MSS clamping internally, then this makes sense, but a breakdown of its encapsulation and calculations would be appreciated if someone is privy to the 'defaults' at play here. Expected MSS per all sane calculations would STILL be a standard 1472, except between the two XGs it's much lower than that, so if anyone's familiar with why this is the case, your insight would be appreciated - it makes zero sense logically.

Answer 1

Cisco has a nice IPSec Overhead calculator (CCO Login required, unfortunately).

https://cway.cisco.com/ipsec-overhead-calculator/

From where we can draw, based on your IPSec settings and a few common optionals thrown in:

 8  PPPoE (optional, but widespread)
20  outer IP header
 8  NAT-T (optional, widespread, sometimes even default-on)
 8  ESP Header (4 SPI and 4 Sequence)
16  ESP IV
20  GRE IP Header (optional, check if implicitely enabled with your product)
 4  GRE Header (optional, check if implicitely enabled with your product)

... <IP MTU>, 
    incl IP (20), plus payload, being either of 
         TCP (20), plus payload
         UDP (8), plus payload
         ICMP (8), plus payload

 N ESP Padding (0-255, to adjust length from ESP Header to ESP ICV to to 32bit boundaries)
 1 Pad Length 
 1 byte  Next Header
16 bytes ESP ICV

On a MTU 1500 link, these settings bring you down from 1500 by 102bytes alone. Add some Padding for 32bit alignment, and your IP MTU is 1392 or even lower. Factor in 20 for IP and 20 for TCP, and that will result in a maximum TCP payload size (a.k.a. MSS) of 1352.

To show a different example, here's a screenshot from Cisco's calculator, based on what would happen if MSS were clamped to 1382 (plus 20b TCP gives the "original IPv4 payload" of 1402) and where none of the "optionals" are at work:

Expected MSS per all sane calculations would STILL be a standard 1472

Nope. That can't ever be the case on a Link with an IP MTU of 1500.

Even two directly connected hosts, with a direct ethernet connection (Ethernet Packet size 1514 bytes and IP MTU of 1500) will never have an MSS of more than 1460.

To actually have a TCP MSS of 1472, you'd need an IP MTU of 1512 (and a L2 MTU of 1526, if on Ethernet). However, 1472 is the maximum size for UDP or ICMP payload in IPv4 within the limits of 1500 bytes of IP-MTU.

With ping, you can't measure a maximum TCP payload size. Using ping for testing requires...

• to understand if the given command line parameter (and test result, too) for "size" is payload only, payload incl. ICMP header or payload incl. ICMP Header and IP header. This may vary by ping implementation.
• to understand that the resulting maximum "size" is not TCP MSS.
• to be certain that PMTUd is at work on the routing hops along the path.
• to be certain that the IPSec tunnel endpoints don't ruin your test result by fragmenting in spite of ping trying to PMTU-discover along the path.
• to keep in mind that while MTU for a given Link/Segment is usually symmetric for both directions, Path MTU is a unidirectional concept and may be different on the return path.

So I suggest you check/verify these:

• be sure to know if either end has PPPoE (-8 bytes)
• check if NAT-T is being used for the given tunnel (-8 bytes). If either tunnel end is behind a NAT device, there will be NAT-T). Check if the given product defaults to NAT-T.
• check if the given product or configuration adds a GRE header (-24bytes).
• be sure to understand if the product or configuration does "df-bit ignore" (or likewise, terms vary by vendor). Be sure to understand how this counteracts any "Let's check MTU for that tunnel" attempts, even cripples PMTUd.
• redo some calculations with the IPSec overhead (see above) and leave some for the padding, or play with cisco's calculator.

For the application, please consider:

• be sure to understand if the application in question uses UDP or TCP, or a combination thereof, and if yes which for which feature/function the application.
• On MTU reduced links, TCP can be helped by MSS clamping.
• On MTU reduced links, UDP (and ICMP, too) must either rely on PMUTd (which is patchy, at best) or on application configuration to reduce max payload per packet.
• Use "df-bit ignore" only as the very last and really ugly non-solution, which breaks PMTUd.

In short: MSS clamping 1382 is not that far off. There are firewall vendors out there that default to 1350.

[added after comment]

About TCP MSS clamping vs PathMDU-Discovery

Yes, TCP MSS clamping is perfectly normal.

Some products do it by default (Cisco ASA, Fortigates), some only after explicit configuration (Juniper ScreenOS based devices, Cisco IOS routers) - I couldn't say for others, from the top of my head.

TCP MSS clamping may be considered a kludge, but in my experience, it does its job more reliably and a lot faster than PMTUd ever will, and therefore I consider it a blessing.

PMTUd depends on ICMP fragmentation needed packet generation by an intermediate hop and successful delivery thereof back to the packet source and correct interpretation of the error message at the source host.

All it takes is one stubborn security admin ("ICMP unreachables are always bad, m'kay?") or a broken NAT implementation ("oups, an incoming ICMP unreachable? I dunno how to match that to any session, let's drop it!") , or even one simplicistic TCP/IP stack (Printers, anyone?) unable to parse the ICMP fragmentation needed message. Any of these along the path or on the end system, and PMTUd dies a horrible death.

TCP MSS clamping however is instantaneous: MSS is only exchanged during the initial handshake, and the ever so slight delay while the clamper manipulates the MSS field in TCP won't be noticed anymore, afterwards. Things just hum along with reduced payload and packet size... [2]

PMTUd on the other hand might come into action with some delay. SYN, SYN-ACK and the client's request [3] packet(s) might still be way below PathMTU. Even the first few packets of the server's response might be small enough to sneak through without fragmentation. And when the bulk transfer of data is about to start, this get sluggish.

Also: please bear in mind that "recieving ICMP fragmentation needed" packets and the vendor's way of doing "df-bit ignore" might be independent.

An implementation might still choose to implement a strategy of "let's fragment that thing anyway, BUT ALSO send a 'fragmentation needed' and hope they react to it and send smaller chunks next time".

To spot such behaviour, you might have to pull packet dumps at the destination, and see if and in how many fragments the packets near/at/over the PathMTU's size arrive. [4]

about application behavior

Yes by all means, investigate the application which has trouble to work across that MTU reduced path.

If TCP is in use, be sure to check the MSS value in both SYN and SYN-ACK on both ends of the TCP session. If either sees an incoming MSS value that's larger than the one on the reduced MTU path, things will get nasty.

Also: Double check that PathMTU discovery works in both directions: It's fine that a spoke site's firewall tells the client to send smaller packets. At the HQ's/datacenter's end, someone should tell the server, too. Again, PathMTU is an unidirectional concept.

Also: Verify if there are load balancers or "WAN accelerators" along the path (or anything else that acts as an unexpected "TCP endpoint in the middle". I've seen some of these do horrible things to TCP, crippling applications to a degree that makes me want to do unspeakable things to delevopers and clueless operators running them.

In short: Capture a few seconds of traffic at the TCP speakers on both endpoints, and look at what happens once things start to go wrong. Compare the packets as they leave and (if and) how they arrive at the respective other end. Look for fragments, changed IP and TCP header fields, missing packets, retransmissions.

[2] ... and go haywire if the path's MTU changes, because rerouting just happened out somewhere in the WAN and a link with even lower MTU is being used. Can't have everything, can you?

[3] Please note: TCP as such has no notion of "client" nor "server". To TCP, there is just an "initiator" and a "responder". It is however common that an application client acts as initiator and the server is usually the responder.

[4] If you really need to dive deep: Replicate the setup in a lab enviroment, and use "ESP-NULL" instead of AES-256. Give or take a few changed field lengths and probably slightly less overhead, the tunnel endpoints will do the entire IKEv2 and IPSec thing (well, without a few "Sec" parts) and then you can packet-capture (not quite) encrypted traffic and see what's actually happening on the WAN/Internet side of the firewall.

[added after comment]

Answer 2

REALLY REALLY late update, but I got busy with other tasks (MS365 migrations, recent software vulnerabilities, etc.), so whoops that i left this unsolved.

The MTU information helped guide us down a path of hunting, but then it took a turn to "What the heck?" land when we discussed this with Sophos - we actually got an engineer on the line. TURNS OUT that all VPN tunnels share a slice of the bandwidth and MTU as well as transfer rates are all limited when you have multiple IPSec tunnels enabled on XG. Which is not documented behavior.

With a firewall rules cleanup AND a plan to move to the XG RED tunnel software inbuilt to the XG firewall, this will clean up the speed issues.

The MTU issue did appear to be a red herring though, as it led us to finding other information. But it was indicative of the other chaos that XG has caused and why they prefer you use their proprietary RED mechanisms instead... Sophos does things weirdly, I guess.